Skip to content

Potential performance optimization using BatchCoalescer::push_batch_with_filter #3208

Description

@albertlockett

Since apache/arrow-rs#9755 the performance of coalescing batches that have been prefiltered has been improved.

There are some places we may be able to use this in query-engine for better performance ...

We use the batch coaescer when concatenating OTAP batches in the batch processor and in output of if/else operator in OPL -- but we don't use this push_batch_with_filter function because in none of these locations do we have a selection vector readily available for the batches we're concatenating.

If we had an OPL program like this (an if with no else)..

// ...
| if (<condition>) {
  // ...
}
| // ...

What happens here is we materialize an batch for things that passed the if condition, and also a batch for things that go into the default noop else branch, then concatenate them together (eventually using BatchCoalescer). We could avoid materializing the batch for the else, and instead just keep the selection vectors we would have used to materialize this OTAP batch and drill them into our batch concatenate code.

If we wanted to be really fancy, we could also recognize two additinoal patterns:

  • a) the if condition is extremely selective, so it might be faster to just feed the original batch into the branch and then prune rows when concatenating afterward.
  • b) when the branch ends with a where, we could just get the selection vector produced by the predicate and use this when concatenating instead of having the where clause actually materialize an OTAP batch.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions