USE 288 - update embeddings write schema#27
Merged
Conversation
Why these changes are being introduced: The embeddings schema column `timestamp` was changed to `embedding_timestamp`, requiring a change in this application that writes to it. Additionally, we had a unit test with a TODO to revisit after some read methods were available in TDA. How this addresses that need: * Updates write schema to `embedding_timestamp` * Updates test `test_create_embeddings_writes_to_timdex_dataset` Side effects of this change: * None Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/USE-288
ghukill
commented
Dec 16, 2025
| embedding_strategy=embedding.embedding_strategy, | ||
| embedding_vector=embedding.embedding_vector, | ||
| embedding_object=json.dumps(embedding.embedding_token_weights).encode(), | ||
| embedding_timestamp=embedding.embedding_timestamp.isoformat(), |
Collaborator
Author
There was a problem hiding this comment.
This is probably the most important change: setting the renamed column embedding_timestamp on write. Formerly we weren't setting a timestamp at all, relying on defaults in the TDA DatasetEmbedding class, but this is more explicit.
ghukill
commented
Dec 16, 2025
|
|
||
|
|
||
| @pytest.fixture | ||
| def dataset_with_records(tmp_path) -> TIMDEXDataset: |
Collaborator
Author
There was a problem hiding this comment.
This new fixture gives us a real dataset, with real records, that allows for writing embeddings associated with real records and thereby supporting read methods.
jonavellecuerdo
approved these changes
Dec 16, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose and background context
When writing embeddings to the TIMDEX dataset, update the column
timestamptoembedding_timestampto match a schema change in thetimdex-dataset-apilibrary.How can a reviewer manually see the effects of these changes?
Probably not worth the effort getting everything up and running. Suggested to analyze the unit test
test_create_embeddings_writes_to_timdex_datasetwhich shows a write to the dataset.Includes new or updated dependencies?
YES
Changes expectations for external applications?
NO
What are the relevant tickets?
Code review