Is your feature request related to a problem or challenge?
The Avro spec places a 16-byte sync marker after every block in an object container file specifically so corrupt blocks can be detected, and the Java reference reader rejects a mismatch (DataFileStream throws IOException("Invalid sync!")). Both arrow-avro readers parse each block's trailing marker into Block.sync but never compare it against the file header's marker, so a desynced or corrupted stream can decode silently into wrong values. This is not hypothetical for compressed files: avro-deflate block data is raw DEFLATE with no checksum, so a damaged block either errors structurally or decodes into plausible garbage. We observed transient object-store read corruption in production (Azure, hadoop-azure readahead faults) where only structural damage was detectable; sync validation shrinks the silent-corruption window and matches the reference implementation.
This continues the recent hardening of the container decode path (#10237 block size/count bounds, #10407 varint bounds).
Describe the solution you'd like
After BlockDecoder::flush() returns a block (one site in the sync Reader, one in the async reader), error if block.sync != header.sync(). Any file the Java reader accepts still reads fine — appended files reuse the file's marker, so only spec-violating or damaged data fails.
Depends on the sync marker assembly fix (#10494): with the current assembly bug, validation false-positives whenever a block's marker straddles a chunk boundary. Happy to submit a PR covering both readers plus tests (corrupted-marker file errors in both readers; existing suites stay green).
Are there any user-facing changes?
Corrupt files that previously decoded silently now error, matching the Java reader.
Is your feature request related to a problem or challenge?
The Avro spec places a 16-byte sync marker after every block in an object container file specifically so corrupt blocks can be detected, and the Java reference reader rejects a mismatch (
DataFileStreamthrowsIOException("Invalid sync!")). Both arrow-avro readers parse each block's trailing marker intoBlock.syncbut never compare it against the file header's marker, so a desynced or corrupted stream can decode silently into wrong values. This is not hypothetical for compressed files: avro-deflate block data is raw DEFLATE with no checksum, so a damaged block either errors structurally or decodes into plausible garbage. We observed transient object-store read corruption in production (Azure, hadoop-azure readahead faults) where only structural damage was detectable; sync validation shrinks the silent-corruption window and matches the reference implementation.This continues the recent hardening of the container decode path (#10237 block size/count bounds, #10407 varint bounds).
Describe the solution you'd like
After
BlockDecoder::flush()returns a block (one site in the syncReader, one in the async reader), error ifblock.sync != header.sync(). Any file the Java reader accepts still reads fine — appended files reuse the file's marker, so only spec-violating or damaged data fails.Depends on the sync marker assembly fix (#10494): with the current assembly bug, validation false-positives whenever a block's marker straddles a chunk boundary. Happy to submit a PR covering both readers plus tests (corrupted-marker file errors in both readers; existing suites stay green).
Are there any user-facing changes?
Corrupt files that previously decoded silently now error, matching the Java reader.