USE 58 - Lazy load current records

[ghukill]

Purpose and background context

Why these changes are being introduced:

Current TIMDEX records has been a consistent source of complexity and performance concerns. And, one of the defining features of TDA, so it's worth getting right.

The previous approach was to materialize lightweight metadata records about all current versions of each record in memory as a DuckDB temp table. This made repeated read queries pulling only current records more efficient, but was unnecessarily loading that data into memory for operations like writing data or reading a specific run (not current record associated).

It turns out that reading current records is somewhat rare, and when it does happen, it's usually a one-off request as part of a larger operation like re-indexing a source in TIM.

How this addresses that need:

The new approach is a hybrid between a view only (lazy evaluation) and a temporary table in memory (eager evaluation).

By default, a view is created, which either a) does not get used or b) is used but only 1-2 times per session and the lazy evaluation of a view is okay.

Alternatively, TIMDEXDataset can now be initialized with preload_current_records=True if it's known that a multiple requests for current records will be needed in the session and it's worth the time and memory hit upfront. In this scenario, a temporary, in-memory table is created as before.

Effect:

• memory usage hovers around 300-500mb for most operations, instead of 2.8-3gb when current record metadata was loaded into memory
• load time for TIMDEXDataset is much quicker, benefitting operations like writing to the dataset that has nothing to do with current records
• one can extend this idea to writing embeddings in the coming months, which have nothing to do with current records per se

How can a reviewer manually see the effects of these changes?

1- Set AWS credentials for production (more data)

2- Set env vars:

TDA_LOG_LEVEL=DEBUG
WARNING_ONLY_LOGGERS=asyncio,botocore,urllib3,s3transfer,boto3,MARKDOWN
TIMDEX_DATASET_LOCATION=s3://timdex-extract-prod-300442551476/dataset

3- Start Ipython shell

pipenv run ipython

4- Preliminary setup:

import os

from timdex_dataset_api import TIMDEXDataset
from timdex_dataset_api.config import configure_dev_logger

configure_dev_logger()

For both of the next steps, we'll be finding a single the current version of a single alma record which is know to have 30+ versions; a "needle in a haystack" query.

5- Init a TIMDEXDataset without pre-loading current records and perform read:

td = TIMDEXDataset(os.environ["TIMDEX_DATASET_LOCATION"])
td.read_dataframe(timdex_record_id="alma:9935128664406761", table="current_records")

• memory peaks around 350mb ✅
• TIMDEXDataset load is fast ✅
• current records is acceptable, ~5-8 seconds 🚨

6- Init a TIMDEXDataset with pre-loading current records and perform read:

td = TIMDEXDataset(os.environ["TIMDEX_DATASET_LOCATION"], preload_current_records=True)
td.read_dataframe(timdex_record_id="alma:9935128664406761", table="current_records")

• memory peaks around 2.8gb 🚨
• TIMDEXDataset load is acceptable 🚨
• current records is fast, ~1 seconds ✅

Includes new or updated dependencies?

NO

Changes expectations for external applications?

YES: For most operations in the TIMDEX ETL pipeline, which don't use current records, the load time and memory usage is fairly dramatically decreased.

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/USE-58
  • prep work for new methods

[ghukill]

The changes to this method are the key here.

We breakout the SQL query that identifies current records into a string. Then depending on the preload_current_records flag, we either:

• create a view directly from that
• or, create a temporary table and a view based on the temporary table

[ghukill]

A table type of LOCAL TEMPORARY is indication of a DuckDB temporary table.

[ehanson8]

Looks good!