Variant shredding has no array-level entry points: every consumer re-writes the same VariantArray loops

[CurtHagenlocher]

Describe the enhancement requested

Apache.Arrow.Operations.Shredding implements the parquet VariantShredding spec one value at a time, but a shredded variant is a COLUMN-level layout. There is no entry point in either direction that takes or returns a VariantArray:

• read: VariantArrayShreddingExtensions.GetLogicalVariantValue(VariantArray, int) resolves one row's typed_value + residual, and VariantUnshredder.Reconstruct works on a single ShredResult. Nothing turns a shredded VariantArray into its canonical unshredded equivalent.
• write: ShredSchemaInferer.Infer / VariantShredder.Shred / ShreddedVariantArrayBuilder.Build take and return values and ShredResults. Nothing takes a VariantArray.

So every consumer that holds a column writes the same two loops. Ours, from a parquet implementation that reassembles on read and shreds on write:

public static VariantArray Reassemble(VariantArray array)
{
    if (!array.IsShredded) return array;

    var builder = new VariantArray.Builder();
    for (int i = 0; i < array.Length; i++)
    {
        if (array.IsNull(i)) { builder.AppendNull(); continue; }
        builder.Append(array.GetLogicalVariantValue(i));
    }
    return builder.Build(allocator: null);
}

and, on the way in, a loop that decodes every row to VariantValue (plus a null mask) purely to hand the sequence to Infer/Shred.

Neither loop encodes any judgement — they are the obvious implementation, which is the argument for having them once, upstream, next to the code whose invariants they depend on. As it stands each consumer re-derives details like "read the LOGICAL value so an already-shredded input re-shreds rather than losing its typed columns", and gets to find out the hard way that VariantArray is not an Apache.Arrow.Array.

Suggested API

// read
public static VariantArray Reassemble(this VariantArray array);

// write
public static VariantArray Shred(this VariantArray array, ShredSchema schema);
public static bool TryShred(this VariantArray array, ShredOptions options, out VariantArray shredded);

Splitting inference from shredding matters for the write side, and is why Shred(array, schema) is listed separately from TryShred: a parquet file has ONE schema while a writer sees one batch at a time, so a caller must infer once over a representative batch and shred every later batch into that same layout. An API that only offered infer-and-shred-together would quietly produce row groups whose layouts disagree.

These also compose with #398 — an array-level entry point has somewhere natural to carry validity, which the value-level one does not.

Happy to open a PR for any of this if the shape sounds right.

Component(s)

C#