GH-49946: [Format] Better document equivalence between IPC file and streams

[pitrou]

Rationale for this change

As discussed in https://lists.apache.org/thread/jpxl3yzm96wkxzb1clokxklsy32b3plh, we want to better document the rough equivalence between IPC files and streams.

What changes are included in this PR?

• Recommendation around emitting "nice" IPC file footers that don't reorder, omit or repeat batches
• Better outlining of deviations of the IPC file format vs. the IPC streaming format
• Assorted minor wording and presentation improvements in the columnar / IPC doc

Are these changes tested?

N/A.

Are there any user-facing changes?

No.

• GitHub Issue: [Format] Better document IPC file and stream equivalence #49946

[pitrou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 6ea2485

Submitted crossbow builds: ursacomputing/crossbow @ actions-0208c5a8f4

Task	Status
preview-docs

[pitrou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 28765e5

Submitted crossbow builds: ursacomputing/crossbow @ actions-6cfd80b0e4

Task	Status
preview-docs

[pitrou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 51fb5a1

Submitted crossbow builds: ursacomputing/crossbow @ actions-3cd6e7c16a

Task	Status
preview-docs

[pitrou]

Preview URL: https://s3.amazonaws.com/arrow-data/pr_docs/49947/format/Columnar.html

[paleolimbot]

Thank you for writing this up!

[paleolimbot]

It may be nice at some point to indicate in the "features" section of a flatbuffers Schema that the stream can definitely be read as an IPC stream (i.e., doesn't differ between what one would get from reading using the blocks in the footer). The fact that nanoarrow does this blindly is not great and I'll fix it, but it is a cool feature that you can do full scans without random access in most cases.

[pitrou]

What do you call the features section? Is it

arrow/format/Schema.fbs

Lines 71 to 81 in a0d2885

	enum Feature : long {
	/// Needed to make flatbuffers happy.
	UNUSED = 0,
	/// The stream makes use of multiple full dictionaries with the
	/// same ID and assumes clients implement dictionary replacement
	/// correctly.
	DICTIONARY_REPLACEMENT = 1,
	/// The stream makes use of compressed bodies as described
	/// in Message.fbs.
	COMPRESSED_BODY = 2
	}

? I don't think anyone is reading this table currently.

[paleolimbot]

That's what I was thinking of (but no need to deal with this now 🙂 )

[pitrou]

@alamb @tustvold @CurtHagenlocher @zeroshade @jbonofre

[pitrou]

I've reworded the equivalence section to also mention the Schema and custom metadata in the IPC file footer.

Again @alamb @tustvold @CurtHagenlocher @zeroshade @jbonofre @lidavidm do you want to take a look? This is a (minor) spec update, I would rather find consensus on this.

[pitrou]

Also @joellubi

[pitrou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 2f21cc8

Submitted crossbow builds: ursacomputing/crossbow @ actions-49a37905b6

Task	Status
preview-docs

[CurtHagenlocher]

Thanks! These improvements are great!

[alamb]

Looks like a very nice improvement to me. Thank you @pitrou

[alamb]

this is much nicer

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 35eed28.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.