GH-49966: [C++] Detect different endianness between IPC file and stream in IPC file fuzzer by pitrou · Pull Request #49968 · apache/arrow

pitrou · 2026-05-12T15:31:13Z

Rationale for this change

In the IPC file format, the IPC file footer has a copy of the embedded IPC stream's schema. However, the two copies may be different in case of an invalid/corrupted IPC file. The fuzzer would then fail with differing contents between IPC file and stream, as only one of them would have undergone endianness swapping.

Issue found by OSS-Fuzz: https://issues.oss-fuzz.com/issues/506111650

What changes are included in this PR?

Add a member variable in ipc::ReadStats that reflects the original endianness of the IPC schema (before the endianness was normalized to native)
Compare the original endianness of the IPC stream and file schemas in the IPC file fuzzer, and skip comparing the contents if the endianness differs
Add unit tests for endianness detection and conversion based on the corresponding "golden" integration files

Are these changes tested?

Yes, by additional unit tests and additional fuzz regression file.

Are there any user-facing changes?

No.

GitHub Issue: [C++] IPC file fuzzer fails when footer schema has differing endianness #49966

pitrou · 2026-05-12T16:08:38Z

@github-actions crossbow submit -g cpp

pitrou · 2026-05-12T16:09:16Z

@kiszk Is it possible for you to run the new unit test arrow-ipc-endianness-test on a big endian machine?

kiszk · 2026-05-12T16:28:54Z

Thank you. I will do it next week since I am taking a vacation this week.

pitrou · 2026-05-12T16:36:41Z

Also @paleolimbot FYI

pitrou · 2026-05-12T16:41:08Z

It looks like the Valgrind errors are related, I'll take a look.

paleolimbot

Thank you! These changes seem reasonable pending CI.

I'm wondering if I need to update anything in nanoarrow's reader to cover this case. I don't think so because we always byte swap to native endian on read (i.e., no option to skip this) but I may be misunderstanding the failure.

pitrou · 2026-05-13T07:19:02Z

I'm wondering if I need to update anything in nanoarrow's reader to cover this case. I don't think so because we always byte swap to native endian on read (i.e., no option to skip this) but I may be misunderstanding the failure.

I don't think there's anything to do, because the problem was in the fuzz target, not in the IPC reader itself. Valid IPC files should have the same schema in the file footer and in the embedded IPC stream.

…d stream in IPC file fuzzer

pitrou · 2026-06-03T13:45:06Z

Thank you. I will do it next week since I am taking a vacation this week.

@kiszk Ping :)

pitrou · 2026-06-03T13:57:10Z

@github-actions crossbow submit -g cpp

github-actions · 2026-06-03T14:00:29Z

Revision: 86203c1

Submitted crossbow builds: ursacomputing/crossbow @ actions-8f178ec382

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

pitrou · 2026-06-03T14:56:28Z

@github-actions crossbow submit test-conda-cpp-valgrind

github-actions · 2026-06-03T14:58:45Z

Revision: 4c61110

Submitted crossbow builds: ursacomputing/crossbow @ actions-5600ef63e7

Task	Status
test-conda-cpp-valgrind

pitrou · 2026-06-04T16:39:34Z

I managed to cross-compile Arrow for s390x and then run the unit tests under emulation. The new arrow-ipc-endianness-test passed successfully.

conbench-apache-arrow · 2026-06-04T23:20:55Z

After merging your PR, Conbench analyzed the 2 benchmarking runs that have been run so far on merge-commit de5b933.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

kiszk · 2026-06-13T04:24:04Z

@pitrou Sorry, I was busy after a vacation. While it was merged, I will check it this weekend.

kiszk · 2026-06-13T06:01:23Z

@pitrou It works well on big-endian machine 🎉 Thank you very much for taking care of big-endian.

% git clone git@github.com:apache/arrow
% cd arrow
% git submodule update --init --recursive
...
% mkdir build && cd build
% cmake .. --preset ninja-debug-flight
...
% cmake --build .
...
% export ARROW_TEST_DATA=/home/ishizaki/arrow/testing/data
% debug/arrow-ipc-endianness-test 
Running main() from gmock_main.cc
[==========] Running 38 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 38 tests from TestEndianness/TestEndianness
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_datetime
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_datetime (3 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_decimal
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_decimal (35 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_decimal256
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_decimal256 (28 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_dictionary
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_dictionary (1 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_dictionary_unsigned
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_dictionary_unsigned (1 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_extension
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_extension (0 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_interval
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_interval (0 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_map
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_map (0 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_nested
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_nested (0 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_nested_dictionary
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_nested_dictionary (1 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_nested_large_offsets
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_nested_large_offsets (0 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_null
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_null (0 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_null_trivial
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_null_trivial (0 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_primitive
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_primitive (2 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_primitive_large_offsets
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_primitive_large_offsets (0 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_primitive_no_batches
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_primitive_no_batches (0 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_primitive_zerolength
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_primitive_zerolength (2 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_recursive_nested
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_recursive_nested (0 ms)
[ RUN      ] TestEndianness/TestEndianness.StreamReader/generated_union
[       OK ] TestEndianness/TestEndianness.StreamReader/generated_union (1 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_datetime
[       OK ] TestEndianness/TestEndianness.FileReader/generated_datetime (2 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_decimal
[       OK ] TestEndianness/TestEndianness.FileReader/generated_decimal (31 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_decimal256
[       OK ] TestEndianness/TestEndianness.FileReader/generated_decimal256 (27 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_dictionary
[       OK ] TestEndianness/TestEndianness.FileReader/generated_dictionary (1 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_dictionary_unsigned
[       OK ] TestEndianness/TestEndianness.FileReader/generated_dictionary_unsigned (0 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_extension
[       OK ] TestEndianness/TestEndianness.FileReader/generated_extension (0 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_interval
[       OK ] TestEndianness/TestEndianness.FileReader/generated_interval (0 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_map
[       OK ] TestEndianness/TestEndianness.FileReader/generated_map (0 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_nested
[       OK ] TestEndianness/TestEndianness.FileReader/generated_nested (0 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_nested_dictionary
[       OK ] TestEndianness/TestEndianness.FileReader/generated_nested_dictionary (1 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_nested_large_offsets
[       OK ] TestEndianness/TestEndianness.FileReader/generated_nested_large_offsets (0 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_null
[       OK ] TestEndianness/TestEndianness.FileReader/generated_null (0 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_null_trivial
[       OK ] TestEndianness/TestEndianness.FileReader/generated_null_trivial (0 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_primitive
[       OK ] TestEndianness/TestEndianness.FileReader/generated_primitive (2 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_primitive_large_offsets
[       OK ] TestEndianness/TestEndianness.FileReader/generated_primitive_large_offsets (0 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_primitive_no_batches
[       OK ] TestEndianness/TestEndianness.FileReader/generated_primitive_no_batches (0 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_primitive_zerolength
[       OK ] TestEndianness/TestEndianness.FileReader/generated_primitive_zerolength (2 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_recursive_nested
[       OK ] TestEndianness/TestEndianness.FileReader/generated_recursive_nested (0 ms)
[ RUN      ] TestEndianness/TestEndianness.FileReader/generated_union
[       OK ] TestEndianness/TestEndianness.FileReader/generated_union (1 ms)
[----------] 38 tests from TestEndianness/TestEndianness (159 ms total)

[----------] Global test environment tear-down
[==========] 38 tests from 1 test suite ran. (159 ms total)
[  PASSED  ] 38 tests.

%

github-actions Bot added awaiting review Awaiting review Component: C++ Component: FlightRPC labels May 12, 2026

pitrou force-pushed the gh49966-fuzz-ipc-file-endianness branch from ece9732 to 7dbefac Compare May 12, 2026 15:40

This comment was marked as outdated.

Sign in to view

pitrou marked this pull request as ready for review May 12, 2026 16:38

pitrou requested a review from lidavidm as a code owner May 12, 2026 16:38

pitrou force-pushed the gh49966-fuzz-ipc-file-endianness branch from 7dbefac to b26a93d Compare May 12, 2026 16:40

paleolimbot approved these changes May 12, 2026

View reviewed changes

github-actions Bot added awaiting merge Awaiting merge and removed awaiting review Awaiting review labels May 12, 2026

lidavidm approved these changes May 13, 2026

View reviewed changes

apacheGH-49966: [C++] Detect different endianness between IPC file an…

86203c1

…d stream in IPC file fuzzer

pitrou force-pushed the gh49966-fuzz-ipc-file-endianness branch from b26a93d to 86203c1 Compare June 3, 2026 13:44

pitrou requested review from assignUser, jonkeane, kou and raulcd as code owners June 3, 2026 14:50

Fix Valgrind failure

4c61110

pitrou force-pushed the gh49966-fuzz-ipc-file-endianness branch from 8ba9a96 to 4c61110 Compare June 3, 2026 14:52

pitrou merged commit de5b933 into apache:main Jun 4, 2026
55 checks passed

pitrou removed the awaiting merge Awaiting merge label Jun 4, 2026

pitrou mentioned this pull request Jun 4, 2026

[C++] IPC file fuzzer fails when footer schema has differing endianness #49966

Closed

pitrou deleted the gh49966-fuzz-ipc-file-endianness branch June 4, 2026 16:43

Uh oh!

Conversation

pitrou commented May 12, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

pitrou commented May 12, 2026

Uh oh!

pitrou commented May 12, 2026

Uh oh!

This comment was marked as outdated.

kiszk commented May 12, 2026

Uh oh!

pitrou commented May 12, 2026

Uh oh!

pitrou commented May 12, 2026

Uh oh!

paleolimbot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou commented May 13, 2026

Uh oh!

pitrou commented Jun 3, 2026

Uh oh!

pitrou commented Jun 3, 2026

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

pitrou commented Jun 3, 2026

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

pitrou commented Jun 4, 2026

Uh oh!

Uh oh!

conbench-apache-arrow Bot commented Jun 4, 2026

Uh oh!

kiszk commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kiszk commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pitrou commented May 12, 2026 •

edited by github-actions Bot

Loading

paleolimbot left a comment •

edited

Loading

kiszk commented Jun 13, 2026 •

edited

Loading

kiszk commented Jun 13, 2026 •

edited

Loading