ESQL: Parallelize newline split discovery for large text files#155009
Open
julian-elastic wants to merge 4 commits into
Open
ESQL: Parallelize newline split discovery for large text files#155009julian-elastic wants to merge 4 commits into
julian-elastic wants to merge 4 commits into
Conversation
Collaborator
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
FROM <ndjson dataset> | LIMIT 10over a 9.8 GB S3 file spent nearly all of itstime in planning: split discovery walked the file with ~146 serial blocking
ranged GETs to find newline boundaries, one thread parked on
Socket.read. Thecost scales with file size rather than with
LIMIT, so no read-side change canavoid it.
Change
Discovery is now two phases. Planning does no I/O: it either produces splits
directly or emits a descriptor of the probe offsets a file needs. Every probe
across every file is then flattened into one
BoundedParallelGather, so asingle global budget bounds in-flight GETs regardless of file count.
produce identical boundaries; the gather preserves file order. An offset
that finds no boundary costs only its own split: the splits either side of
it merge, and every other offset's outcome stands.
aborting, so the connection returns to the pool for the next probe.
windows overlap the same records anyway, and planning would materialize
fileLength / strideprobe tasks — unbounded below. The floor caps a file'sprobes at
fileLength / window.Without an executor the probe phase walks the same offsets serially on the
calling thread and produces identical boundaries.
gather joins; a cancelled probe aborts its stream instead of draining it.
Measurements
split_discovery_nanosfrom the profile, same query and file, 147 splits at thedefault 64 MB target split size: 61.0s on
main(n=1) versus 2.5s here (n=3,2.48-3.58s). End-to-end
tookis not a clean comparison: the read side nowdominates and varies several-fold run to run, which is a separate defect.
Coordinator-local planning change, so no wire format, settings, or cross-node
contract changes. Unit tests cover the probe arithmetic, boundary reduction,
drain-versus-abort, the concurrency ceiling, the stride floor, and cancellation
in each phase.
Closes https://github.com/elastic/esql-planning/issues/1528