Skip to content

Recover corrupted checkpoint downloads#55

Open
kiwigitops wants to merge 1 commit into
apple:mainfrom
kiwigitops:fix-atomic-checkpoint-downloads
Open

Recover corrupted checkpoint downloads#55
kiwigitops wants to merge 1 commit into
apple:mainfrom
kiwigitops:fix-atomic-checkpoint-downloads

Conversation

@kiwigitops

Copy link
Copy Markdown

Summary

  • download checkpoints to a temporary file before replacing the final .ckpt
  • redownload an existing checkpoint once when torch.load hits a truncated archive error
  • use the same checkpoint-loading path for the CLI and wrapper APIs

Refs #11

Tests

  • $env:PYTHONPATH='src/simplefold'; python -m pytest tests/test_checkpoint_utils.py -q
  • python -m compileall -q src/simplefold/utils/checkpoint_utils.py src/simplefold/inference.py src/simplefold/wrapper.py tests/test_checkpoint_utils.py
  • git diff --check

@kiwigitops

Copy link
Copy Markdown
Author

Hi, checking in on this when you have a chance. The change is scoped to making checkpoint downloads atomic and retrying once when an existing .ckpt is detected as truncated, so interrupted downloads do not leave a permanently bad cache file.

@kiwigitops

Copy link
Copy Markdown
Author

Quick follow-up here. The atomic checkpoint download/retry path is still current, and the checkpoint utility tests passed on my side. Happy to adjust if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant