USE 558 - Write fulltext records to TIMDEX dataset

[ghukill]

NOTE: this PR relies on MITLibraries/timdex-dataset-api#189, and will get a force push to revert the timdex-dataset-api version back from a pinned branch.

Purpose and background context

Why these changes are being introduced:

The final leg of the journey of harvesting fulltext for this app is writing fulltexts back to the TIMDEX dataset to live alongside the records they are associated with. We should still support writing locally to JSONLines for debugging.

How this addresses that need:

In direct support of this goal,

• functions from harvest.py now yield DatasetFulltext instances which are expected by timdex-dataset-api (TDA) library for writing to the "fulltexts" data type
• refactor TIMDEXThesesRecords to accept an instantiated TIMDEXDataset instance such that it can be created once, then reused for reading + writing
• update harvest CLI command to default to writing back to the TIMDEX dataset

Related to this work was some refactoring,

• updated record_and_fulltext_iter() to use the joblib library to offload the complexities of parallel work to a well known library

How can a reviewer manually see the effects of these changes?

1- Set TimdexMangers Dev1 credentials

2- Set env vars (DSpace credentials shared offline of PR):

TDA_LOG_LEVEL=DEBUG
WARNING_ONLY_LOGGERS=asyncio,botocore,urllib3,s3transfer,boto3,MARKDOWN

DSPACE_API_BASE="..."
DSPACE_USERNAME="..."
DSPACE_PASSWORD="..."

TIMDEX_DATASET_LOCATION=s3://timdex-extract-dev-222053980223/dataset

3- Run CLI, omitting --output-jsonl which will now default to TIMDEX dataset:

uv run --env-file .env dfh --verbose \
harvest \
--run-id="5e11f27e-fe05-44f4-8bcb-5acbdb032cf3" \
--record-limit=100

This CLI command:

1. finds and limits to 100 records from a specific TIMDEX ETL run
2. writes records back to TIMDEX dataset

4- Start an ipython shell:

uv run --env-file .env ipython

5- Read the fulltext records we just wrote:

import os

from timdex_dataset_api import TIMDEXDataset
from timdex_dataset_api.config import configure_dev_logger

configure_dev_logger()

td = TIMDEXDataset(os.environ["TIMDEX_DATASET_LOCATION"])

# load three records from new "fulltext"
df = td.fulltexts.read_dataframe(table="current_fulltexts", run_id="5e11f27e-fe05-44f4-8bcb-5acbdb032cf3", limit=3)

# observe fulltext for one:
df.iloc[0]
"""
Out[2]: 
timdex_record_id                                    dspace:1721.1-58259
source                                                           dspace
run_date                                            2026-05-13 00:00:00
run_type                                                           full
action                                                            index
run_id                             5e11f27e-fe05-44f4-8bcb-5acbdb032cf3
run_record_offset                                                100109
run_timestamp                          2026-05-13 09:49:33.846930-04:00
filename              s3://timdex-extract-dev-222053980223/dataset/d...
fulltext_timestamp                     2026-05-18 16:35:52.548253-04:00
fulltext_md5                           95a4f7c63b78c6d7518f416df1e258c0
fulltext              b'THE RUBIDIUM-STRONTIUM \nGEOCHRONOLOGY\nOF A...
Name: 0, dtype: object
"""

# see just fulltext, noting it's binary data to avoid encoding issues during read/write
df.iloc[0].fulltext[:200]
"""b'THE RUBIDIUM-STRONTIUM \nGEOCHRONOLOGY\nOF ARGILLACEOUS SEDIMENTS\nby\nPHILIP ROY WHITNEY\nS. B., Massachusetts Institute\nof Technology (1956)\nSUBMITTED IN PARTIAL FULFILLMENT\nOF THE REQUIREMENTS FOR THE\nD'"""

Includes new or updated dependencies?

YES

Changes expectations for external applications?

YES

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/USE-558

Code review

• Code review best practices are documented here and you are encouraged to have a constructive dialogue with your reviewers about their preferences and expectations.

[ghukill]

NOTE: this PR relies on MITLibraries/timdex-dataset-api#189, and will get a force push to revert the timdex-dataset-api version back from a pinned branch.

[ghukill]

This is the joblib library in action here.

In a way, it's more intuitive in that you are basically calling your function, with args, over an iterator of items.

What's not immediately intuitive:

• use of delayed(...) should wrap your "worker" function
• Parallel(...) arguments are really important, setting threads vs processes, if unordered is okay, etc.

But overall, really liking it so far, and very nice to offload this finicky parallel work.

[ehanson8]

Definitely cleans the code up! Noted for potential future use

[ehanson8]

Looks great, approved!

[ehanson8]

Definitely cleans the code up! Noted for potential future use