format: explicit cursors and backpressure?

[lidavidm]

What feature or improvement would you like to see?

We have this implicitly from ArrowArrayStream but maybe we need something more explicit? Or would the async stream and its backpressure handle this?

[zeroshade]

I assume that the Async idea and it's backpressure would handle this, at least IMO

[lidavidm]

Putting things together we need the following:

• Executing a SQL query can either give you partitions or (multiple) result sets
  • Maybe this can be unified to giving you a list of "promise" objects that can be either immediately consumed to get a (partition of a) result set, or serialized into a partition (if supported)
  • This does lead to more API calls, though...
• Each result set needs to have a requested schema bound (so the driver can convert if it's able to/if there are potentially multiple valid Arrow types for a column)
• Result streams should use the async device interface, which handles backpressure already
• Cancelling the statement will cancel the entire query?
• Dropping a substream shouldn't cancel the entire query?
• Do we need some way to potentially retry a stream, and ideally retrying a stream from a given (row) offset?
  • Some of these questions may be specific to Flight SQL, but I think it generally applies to any sort of newfangled system with a REST or gRPC API instead of an old-school database protocol
• We won't support seeking through streams (aka no actual cursors)
• We should standardize that readers should be safe to use if ADBC objects are released (but they may error instead of producing data and/or ADBC objects may error on release if there are unclosed streams)