Raise errors when documents fail to add to dataset#50
Merged
stevevanhooser merged 2 commits intoMar 25, 2026
Conversation
…wnload Four locations were silently swallowing exceptions (except: pass) when documents failed to convert or be added to the dataset database. This caused downloaded datasets to have fewer documents than expected with no indication of what went wrong. Changes: - jsons2documents(): warn with document IDs and error details for each conversion failure - ndi_dataset_dir.__init__(): raise RuntimeError when documents fail to add to the database (FileExistsError for duplicates is still silently skipped) - downloadDataset(): raise RuntimeError at the end if any documents were lost during conversion or database insertion - _sync_download_new(): raise RuntimeError when sync loses documents - add_ingested_session(): raise RuntimeError instead of DEBUG-only logging https://claude.ai/code/session_015oe6ucKwCFznpu1PN1nAfz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
except: pass) when documents failed to convert or be added to the database, causing datasets to have fewer documents than expected with zero indication of what went wrongjsons2documents()now warns with specific document IDs and error messages for each conversion failurendi_dataset_dir.__init__(),downloadDataset(),_sync_download_new(), andadd_ingested_session()now raiseRuntimeErrorwith details when any documents are lost (duplicateFileExistsErroris still safely skipped)Context
When downloading a dataset with 7221 documents, only 7123 ended up in the dataset — 98 documents were silently dropped. This was caused by bare
except: passblocks in the conversion and database-add paths.Test plan
https://claude.ai/code/session_015oe6ucKwCFznpu1PN1nAfz