Skip to content

Complete level 2 cleanup reorganization#1669

Merged
kmharrington merged 9 commits into
masterfrom
cleanup-cleanup
Jun 27, 2026
Merged

Complete level 2 cleanup reorganization#1669
kmharrington merged 9 commits into
masterfrom
cleanup-cleanup

Conversation

@kmharrington

Copy link
Copy Markdown
Member

I'm going to admit to having used Claude to help reorganize this. It's going to substantially reduce the number of times we run through the file system and databases as we are cleaning things up. (I don't think this will be enough to shorten LATs runtime but it's a start).

This changes how the cleanup loops work:

Before:

for each timecode:
    check_timecode_complete()
for each timecode:
    check_timecode_complete()
    delete_staged()
for each timecode:
    check_timecode_complete()
    delete_level2()

After

for each timecode:
    check_timecode_complete()
    if ready:
        delete_staged()
    if ready:
        delete_level2()

Other changes

  • add option to set a maximum allowable runtime (although only checked once per loop)
  • add dry run print outs

kmharrington and others added 9 commits June 18, 2026 22:06
…3thk loop (#1671)

* Only log in the cases of unexpected exceptions (not TypeErrors) for g3thk loop

* Include error in log message
* expand update_g3tsmurf_db to accept minimum and maximum ctimes

* print time differences

* change variable names and expose delay warning and error to argparse

* comment formatting

* Update sotodlib/io/imprinter.py

Co-authored-by: Josh Borrow <josh@joshborrow.com>

* Update sotodlib/io/imprinter.py

Co-authored-by: Josh Borrow <josh@joshborrow.com>

* Apply suggestions from code review

Co-authored-by: Josh Borrow <josh@joshborrow.com>

* remove int casting and set defaults through the system

* add option to automatically register books as bad when removing a level 2 obs. closes #1438

* fix counter logic. closes #1666

* prevent missing files from throwing deletion errors

* add additional counter checks

* improve handling for BadTimeSamples since I've realized missing files is a bigger issue

* close the bookbinder log

* add timing counter errors to autofixing

* remove log statement

* other binders need a close

* only close bookbinder if it exists

---------

Co-authored-by: Josh Borrow <josh@joshborrow.com>
@kmharrington kmharrington marked this pull request as ready for review June 27, 2026 14:13
@kmharrington

Copy link
Copy Markdown
Member Author

I've now tested and run this over everything. Ready to deploy.

@kmharrington kmharrington merged commit 8b37fa4 into master Jun 27, 2026
3 of 5 checks passed
@kmharrington kmharrington deleted the cleanup-cleanup branch June 27, 2026 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants