feat: symbolically represent IPC message bodies in IR

[Benjamin-Philip]

What issue does this PR close?

Closes #112.

The body of a message needs to represented symbolically in Arrow JSON. We
previously serialized it within our internal representation, making it
impossible to produce Arrow JSON from the IR.

What's Changed

We now leave the body as-is in the IR and only serialize it on to_ipc.

[github-actions]

Preview URL: https://Benjamin-Philip.github.io/arrow-erlang-1

If the preview URL doesn't work, you may have forgotten to configure your fork repository for preview.
See https://github.com/apache/arrow-erlang/blob/main/CONTRIBUTING.md#forks for instructions on how to configure.

[Copilot]

Pull request overview

Updates the IPC message IR so RecordBatch message bodies are kept as symbolic Arrow arrays (to support Arrow JSON) and are only serialized when producing IPC (EMF) bytes.

Changes:

• Store RecordBatch message bodies as [arrow_array:array()] in the IR and serialize them in to_ipc/1.
• Add arrow_ipc_record_batch:body_length/1 and use it to populate #message.body_length.
• Update test fixtures and suites to expect symbolic bodies and validate computed body length.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
test/arrow_ipc_record_batch_SUITE.erl	Adds a new test asserting the computed RecordBatch body length matches the serialized body size.
test/arrow_ipc_message_SUITE.erl	Updates expectations so RecordBatch message bodies remain symbolic (arrays) in the IR.
test/arrow_ipc_marks_data.hrl	Updates test macros to construct RecordBatch messages with symbolic bodies (columns) instead of pre-serialized binaries.
src/arrow_ipc_record_batch.erl	Exports and implements body_length/1 for RecordBatch bodies.
src/arrow_ipc_message.hrl	Changes #message.body type to `[arrow_array:array()]
src/arrow_ipc_message.erl	Updates message construction and IPC serialization to serialize body arrays at to_ipc/1.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/arrow_ipc_message.erl:36

• The module doc still describes #message.body as a single arrow_array:array/0, but the record field and from_erlang/2 now use a list of arrays. This mismatch can confuse users and will make the docs inconsistent with the new symbolic representation approach.

5.  `body`: The actual body. Can be undefined (in the case of Schema)
    or a `t:arrow_array:array/0` (in the case of Record Batch).

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/arrow_ipc_message.erl:76

• from_erlang/1 is now specified/documented as taking only a schema header, but the implementation will still accept any term (including a record batch header) and silently produce a message with body_length = 0 and no body. That can lead to invalid IPC messages that are hard to debug; it’s safer to reject non-schema headers at runtime.

from_erlang(Header) ->
    #message{header = Header, body_length = 0}.

src/arrow_ipc_message.erl:102

• to_ipc/1 serializes the Flatbuffers metadata before computing the body binary, and relies on the Message#message.body_length already being correct. Since from_erlang/2 now accepts a symbolic body (arrays) and computes body_length from the record batch header, it’s possible for body_length to drift from the actual serialized body size (or for callers to accidentally pass a mismatched header/body). Compute the body first and set body_length from byte_size(Body) when calling arrow_format_nif:serialize_message/1 so the EMF metadata stays consistent with the emitted bytes.

    Metadata = arrow_format_nif:serialize_message(Message#message{body = undefined}),
    MetadataSize = <<(byte_size(Metadata)):32>>,
    Body =
        case Message#message.body of
            undefined ->

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/arrow_ipc_message.erl:105

• to_ipc/1 now assumes Message#message.body is always a list of arrays and will crash with badarg if a caller constructs a #message{body = Bin} (binary) directly (which previously worked). If you want this change to be strictly backward compatible and/or produce a clearer error, add guards for list vs binary (and a fallback error) in the body serialization case expression.

        case Message#message.body of
            undefined ->
                <<>>;
            Arrays ->
                body_from_erlang(Arrays)