USE 557 - identify records for harvest and parse bitstream download information

[ghukill]

Purpose and background context

This PR introduces the mechanism by which this application will identify records to harvest fulltext for. And, a lightweight DSpace client to generate pre-signed URLs for known bitstream UUIDs.

For now, this application only wants to harvest fulltext bitstreams for DSpace theses. Ultimately, it will want to write the fulltext back to the TIMDEX dataset under a new data type called "fulltexts" (see USE-531), associated with specific TIMDEX records from the "records" data type. As such, it makes sense to read records from the TIMDEX dataset as the primary driver for what to harvest.

Additionally, the DSpace theses records in the TIMDEX dataset have their original DSpace METS which conveniently includes bitstream information! By utilizing that, we can avoid 3-4 DSpace APIs calls per item, and skip directly to generating a pre-signed S3 URL for a known bitstream UUID as the only DSpace API call we'll need to make.

Functionality added:

• Identify DSpace records to harvest fulltext for
• Extracting bitstream information from those record's METS files
• Lighweight DSpace client that supports authentication + pre-signed URL generation

Functionality coming based on this:

• A harvesting loop that performs all above
• Downloading of the bitstream with a pre-signed URL in hand
• Writing fulltext back to TIMDEX dataset

How can a reviewer manually see the effects of these changes?

1- Test new class TIMDEXThesesRecords

Set Dev1 TimdexManagers credentials.

Create .env file:

TDA_LOG_LEVEL=DEBUG
WARNING_ONLY_LOGGERS=asyncio,botocore,urllib3,s3transfer,boto3,MARKDOWN
TIMDEX_DATASET_LOCATION=s3://timdex-extract-dev-222053980223/dataset

Start ipython shell:

uv run --env-file .env ipython

Init TIMDEXThesesRecords instance, limited to 10 records:

import logging
from dfh.timdex_dataset import TIMDEXThesesRecords

# bootstrap logging
logging.basicConfig(level=logging.DEBUG)

tdt = TIMDEXThesesRecords(limit=10)

Note that no run_id filtering is applied here, thereby defaulting to all, current dspace records that are deemed to be theses.

Retrieve all records as a list from the iterator:

records = list(tdt.record_and_bitstream_metadata_iter())

Observe first record:

records[0]
"""
{'timdex_record_id': 'dspace:1721.1-150510',
 'run_id': '5e11f27e-fe05-44f4-8bcb-5acbdb032cf3',
 'run_record_offset': 100001,
 'run_date': datetime.datetime(2026, 5, 13, 0, 0),
 'run_timestamp': datetime.datetime(2026, 5, 13, 9, 49, 33, 846930, tzinfo=<DstTzInfo 'America/Detroit' EDT-1 day, 20:00:00 DST>),
 'fulltext_bitstream': {'uuid': 'b0e3c854-de3e-44ce-94a3-6088615a5eeb',
  'href': 'https://..../bitstreams/b0e3c854-de3e-44ce-94a3-6088615a5eeb/download',
  'size': 74970,
  'checksum': '2e3d96be8538d9c1a549d7292055ee78',
  'checksum_type': 'MD5',
  'mimetype': 'text/plain'}}
"""

Note: ignore the testing DSpace host found in the record.

With this in hand from this class + iterator, we have enough information to:

1. Request a pre-signed S3 URL for the bitstream UUID b0e3c854-de3e-44ce-94a3-6088615a5eeb
2. Download that fulltext
3. Write fulltext back to TIMDEX dataset associated with this specific record (timdex_record_id="dspace:1721.1-150510", run_id="5e11f27e-fe05-44f4-8bcb-5acbdb032cf3", run_record_offset="100001")

This iterator is quite quick. The I/O bottleneck will be in future work of parallelizing the requests for pre-signed bitstream URLs from this information.

Note

This section has been updated based on our pivot to using dspace-rest-python client versus a lightweight, custom client for DSpace API interactions.

2- Test generation of pre-signed URLs for bitstream downloading

Add the following to your .env file and restart ipython shell, with the values shared outside of this PR:

DSPACE_API_BASE="..."
DSPACE_USERNAME="..."
DSPACE_PASSWORD="..."

NOTE: at this time, this application is expecting these env vars to authenticate if API base + username + password is not explicitly passed during DSpaceClient init. Still in flux how this app will receive the credentials. It's possible they will be parsed from another secret, and likely passed to the client init. The client itself will likely remain defaulting to these three env vars for ease of testing.

Get a DSpaceClient instance::

from dfh.dspace import *

dspace_client = get_dspace_client()
# 2026-05-14 14:17:57,542 - Authenticated successfully as *********

Note that authentication is automatic due to auth_on_init defaulting to True in function.

Generate a pre-signed S3 URL for the bitstream shown above:

get_presigned_url_for_bitstream(dspace_client, "b0e3c854-de3e-44ce-94a3-6088615a5eeb")

The link string that is returned is only valid for a single use and for a limited amount of time; both of which are easily satisfied by this harvester as it loops through records and utilizes the URLs.

Includes new or updated dependencies?

YES

Changes expectations for external applications?

NO

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/USE-557

Code review

• Code review best practices are documented here and you are encouraged to have a constructive dialogue with your reviewers about their preferences and expectations.

[ghukill]

Convenience method for a pre-signed URL, leaning on a self.api_get() call.

[ghukill]

This is the first TIMDEX application that optionally uses a TIMDEX_DATASET_LOCATION env var. The CLI will likely still require it get passed, to align with other TIMDEX apps in the pipeline... but looking to establish this alternate pattern here and now with the hopes perhaps we can move to an env var for most/all apps at some point.

[ghukill]

In the event we broaden our harvesting of fulltext beyond theses, it would not be difficult to add some inheritance here:

• TIMDEXRecords
• TIMDEXThesesRecords(TIMDEXRecords)
  • all shared except opionated filtering to theses records only

I mention this only to point out that while we have some theses-opinionation in the business logic here, it will be easy to extend if and when needed. Opted to focus on the immediate need.

[Copilot]

Pull request overview

Introduces the initial harvesting “selection + lookup” layer by (1) iterating TIMDEX DSpace thesis records, (2) extracting TEXT bitstream metadata from embedded DSpace METS, and (3) providing a lightweight DSpace REST client to generate pre-signed download URLs for known bitstream UUIDs.

Changes:

• Add TIMDEXThesesRecords to fetch/filter DSpace thesis records from the TIMDEX dataset and parse TEXT bitstream metadata from METS.
• Add DSpaceClient to authenticate against DSpace and request signed URLs for bitstreams.
• Add pytest coverage and fixtures for METS parsing edge-cases; adjust lint/typechecker configuration.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
dfh/timdex_dataset.py	New TIMDEX dataset helper for filtering thesis records and extracting TEXT bitstream info from METS XML.
dfh/dspace.py	New lightweight DSpace REST client for auth + signed URL retrieval.
tests/test_dataset.py	New tests covering TIMDEXThesesRecords iteration and METS parsing error cases.
tests/conftest.py	Adds fixtures for TIMDEX record JSON + various METS XML inputs.
tests/fixtures/timdex/dspace-1721.1-139336.json	TIMDEX thesis record fixture used by dataset tests.
tests/fixtures/dspace/mets/1721.1-139336.xml	“Happy path” METS fixture with TEXT fileGrp.
tests/fixtures/dspace/mets/1721.1-139336-missing-text-filegrp.xml	METS fixture missing TEXT fileGrp (error case).
tests/fixtures/dspace/mets/1721.1-139336-text-filegrp-missing-file.xml	METS fixture with empty TEXT fileGrp (error case).
tests/fixtures/dspace/mets/1721.1-139336-text-file-missing-flocat.xml	METS fixture missing FLocat for TEXT file (error case).
tests/fixtures/dspace/mets/1721.1-139336-text-flocat-missing-href.xml	METS fixture missing href in FLocat (error case).
pyproject.toml	Updates mypy excludes/overrides and ruff configuration (exclusions/ignore list).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ehanson8]

Works great and the code looks good, a few suggestions!

[ehanson8]

Was there a reason to not use FulltextBitstreamInfo? I see you've bounced between text and fulltext in the names, comments, and docstrings

[ghukill]

Nice catch - no good reason. I'll try and normalize on "Fulltext".

[ghukill]

You know, on second thought, maybe there was rhyme and reason.

We know that we're going after "fulltext" for theses. But technically the bitstream is TEXT.

If you're okay with it @ehanson8, I'd kind of like to keep as-is. Tried to use "fulltext" in places where we're speaking about it conceptually, but often "text" in classes like this and methods that are quite literally interacting with the TEXT bitstream.

[ehanson8]

That makes sense, a docstring clarifying this could be helpful but not required

[ghukill]

No rush here at all @ehanson8 for re-review, just confirming that we did pivot to using the dspace-rest-python client: 6e35ea1.

[ehanson8]

Looks great, excellent work!