fix(dataset): compare on-the-wire bytes so gzip responses aren't flag…#17
Conversation
|
Heads-up: I'm pushing two small maintainer-edit cleanups directly onto this branch (thanks for leaving "Allow edits by maintainers" on). The gzip fix itself is correct and approved — these are just polish:
One follow-up for #11: it currently carries a copy of this same docstring with the old |
…ged as truncated downloads The truncation check in url.py compared bytes *written to disk* against Content-Length. That breaks for Content-Encoding responses: when the server sends gzip, Content-Length is the compressed size while iter_bytes() yields the larger decompressed body, so written bytes always exceed Content-Length and the download is falsely rejected. Compare r.num_bytes_downloaded (raw on-the-wire bytes, before httpx decompresses) against Content-Length instead — the two are now measured on the same, compressed scale, so gzip-served files pass while genuinely truncated transfers still fail. Regression: SQuAD's train-v2.0.json is gzip-served; Content-Length=9551051 but the decoded body is ~42MB, which the old written-bytes check rejected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…s dev-v2.0.json source The byte-count guard rejects any num_bytes_downloaded != Content-Length mismatch, including an over-read, so "truncated download" was a misnomer for that direction — reword to "size mismatch". Also align test_download_accepts_compressed_response's docstring to the file RULER actually fetches (dev-v2.0.json: Content-Length=800683, decoded ~4.4MB) instead of train-v2.0.json, so the Regression note describes the real download path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
b0bd8a1 to
7b96904
Compare
2 participants
Type
Summary
url:gzip truncation fix — the download verifier compared byteswritten against
Content-Length, but when the server sendsContent-Encoding: gzip,Content-Lengthis the compressed size while thedecoded body is larger, so valid downloads were falsely flagged as truncated.
Now compares against
num_bytes_downloaded(on-the-wire bytes).Related Issues
Test Plan
Automated
ruff check && ruff format --check)ty check)pdm run pytest)Checklist
Required (all PRs)
type(scope): description)AI-Generated Code - <model> (<provider>)in module docstringcore/