TIMX 530 - rebuild TIMDEXDatasetMetadata with static database file approach

[ghukill]

Purpose and background context

This PR begins effectively a rewrite of TIMDEXMetadataDataset, and the process of incrementally updating TIMDEXDataset to align with those changes. Major changes:

1. Start rewrite of TIMDEXMetadataDataset, building up functionality over the next few PRs
2. TIMDEXDataset temporarily loses the ability to filter to only "current" records
3. Minimally implemented, but paving the way for changing how TIMDEXDataset is loaded to expect the root of the dataset, then internally know where the ETL parquet records are

This is probably the trickiest PR for epic TIMX-515. To begin building up both TIMDEXMetadataDataset and TIMDEXDataset to more fully utilize dataset metadata, it requires temporarily stripping back functionality and unit testing.

For this review, I would ask reviewers to focus most heavily on TIMDEXMetadataDataset creating and attaching to a static metadata file as the foundation for dataset metadata. Future PRs will layer on append deltas, create views, adjust how we read data, and the associated tests.

Planned, ordered next steps:

• Write append deltas, TIMX-527
• Create metadata projections over static DB file and append deltas, TIMX-526
• Fully rework TIMDEXDataset.load() and dataset "location", TIMX-533
• Rework read methods to use SQL + metadata, TIMX-529
• Merge append deltas into static DB, TIMX-528

How can a reviewer manually see the effects of these changes?

Despite things being in a liminal state, manual testing is possible.

1- Set AWS Dev TimdexManagers credentials

2- Set env vars:

TDA_LOG_LEVEL=DEBUG
WARNING_ONLY_LOGGERS=asyncio,botocore,urllib3,s3transfer,boto3,MARKDOWN
TIMDEX_DATASET_LOCATION=s3://timdex-extract-dev-222053980223/dataset_scratch

3- Start Ipython shell with pipenv run ipython and do some setup:

import os

from timdex_dataset_api import TIMDEXDataset, TIMDEXDatasetMetadata
from timdex_dataset_api.config import configure_dev_logger

configure_dev_logger()

td = TIMDEXDataset(os.environ["TIMDEX_DATASET_LOCATION"])
tdm = TIMDEXDatasetMetadata(os.environ["TIMDEX_DATASET_LOCATION"])

4- First, note small update to TIMDEXDataset where it knows both the dataset root location and the location of the ETL parquet files:

td.location
# Out[5]: 's3://timdex-extract-dev-222053980223/dataset_scratch'

td.data_records_root
# Out[6]: 's3://timdex-extract-dev-222053980223/dataset_scratch/data/records'

5- Recreate the dataset metadata:

tdm.recreate_static_database_file()

6- Perform very simple query demonstrating the metadata database file was created and is successfully attached:

tdm.conn.query("""show databases;""")
"""
Out[8]: 
┌───────────────┐
│ database_name │
│    varchar    │
├───────────────┤
│ memory        │
│ static_db     │
└───────────────┘
"""

tdm.conn.query("""select count(*) from static_db.records;""")
"""
Out[9]: 
┌──────────────┐
│ count_star() │
│    int64     │
├──────────────┤
│      4151483 │
└──────────────┘
"""

7- If interested, reload the TIMDEXDatasetMetadata instance and see the payoff of all this work in nearly instant dataset metadata available!

tdm = TIMDEXDatasetMetadata(os.environ["TIMDEX_DATASET_LOCATION"])
tdm.conn.query("""select * from static_db.records;""")

Ticket TIMX-526 will create all the helpful tables and views once the append deltas exist as well.

Includes new or updated dependencies?

NO

Changes expectations for external applications?

NO: still only merging to feature branch epic-TIMX-515

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/TIMX-530

[coveralls]

Pull Request Test Coverage Report for Build 16780878765

Details

• 80 of 88 (90.91%) changed or added relevant lines in 3 files are covered.
• 1 unchanged line in 1 file lost coverage.
• Overall coverage decreased (-2.0%) to 94.038%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
timdex_dataset_api/utils.py	13	16	81.25%
timdex_dataset_api/metadata.py	61	66	92.42%

Files with Coverage Reduction	New Missed Lines	%
timdex_dataset_api/dataset.py	1	96.45%

Totals
Change from base Build 16725129203:	-2.0%
Covered Lines:	347
Relevant Lines:	369

---

💛 - Coveralls

[jonavellecuerdo]

Looks great! Just have a couple questions. My main change request / question was on whether we should keep referring to the DuckDB file as a "static" DuckDB file (see comment for more details).

I also wondered:

1. Whether the performance on my local machine was similar to yours:

2. Whether we should add a DEBUG log also describing the time elapsed after the "Attaching to static database file" log message

3. Where the 'database_name_varchar' comes from:

[ghukill]

Looks great! Just have a couple questions. My main change request / question was on whether we should keep referring to the DuckDB file as a "static" DuckDB file (see comment for more details).

I also wondered:

1. Whether the performance on my local machine was similar to yours:

Yep! Looks about right. The static metadata file recreation takes anywhere from 30-45 seconds. And that will grow over time, but we don't need to perform very often.

2. Whether we should add a DEBUG log also describing the time elapsed after the "Attaching to static database file" log message

If you're comfortable with it, I'll add during TIMX-526 which is focused heavily on setting up the DuckDB context! In fact, might be in there already...

3. Where the 'database_name_varchar' comes from:

That's just a DuckDB convention for the DuckDB query `"show databases;". For a lot of DuckDB (and other DBs for that matter) their "meta" commands like this often have kind of awkward column names because it's just communicating data.

Actually, I'm realizing that's kind of two lines! It's "database_name" as the column name, and "varchar" as the column type.