perf: parquet LevelInfoBuilder::write_list can be optimized?

[mapleFU]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

LevelInfoBuilder::write_list would handle element by element without any batching, can this being optimized?

Describe the solution you'd like

Handle it with batching

Describe alternatives you've considered

Find the null or not null element and batching them

Additional context

no

[mapleFU]

Oh, this might be solved by #9752

[mapleFU]

@alamb @HippoBaro The simplest way for this is to optimize last-level list, when list rep-level reaches the childs level, it's easy to calculating the child-levels not-null levels, which could easily batching the results.

For example:

[1, 2, 3], [4, 5], [], null

If the write_list found itself the last list writer, it could batching and get the offsets:

offsets: [0, 3, 5, 5, 5]

And writes [1, 2, 3], [4, 5] leaf in one pass, then by using offsets, it can mark the first and the 4th level as the start of the level.

I didn't think this carefully for deep nested list, but generally this would accelerate case like non-empty lists. What do you think of this?

[alamb]

I didn't think this carefully for deep nested list, but generally this would accelerate case like non-empty lists. What do you think of this?

maybe @HippoBaro has some thoughts

[HippoBaro]

Yes, for runs of valid, non-empty list rows, we can write the child values as one contiguous range, then use the parent list offsets to go back and mark the list starts in the repetition levels. That avoids calling into the child writer once per list row.

I am working on an updated #9653 which contains this!