Skip to content

ESQL: Parallelize newline split discovery for large text files#155009

Open
julian-elastic wants to merge 4 commits into
elastic:mainfrom
julian-elastic:ndjsonSlowSplit
Open

ESQL: Parallelize newline split discovery for large text files#155009
julian-elastic wants to merge 4 commits into
elastic:mainfrom
julian-elastic:ndjsonSlowSplit

Conversation

@julian-elastic

@julian-elastic julian-elastic commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Problem

FROM <ndjson dataset> | LIMIT 10 over a 9.8 GB S3 file spent nearly all of its
time in planning: split discovery walked the file with ~146 serial blocking
ranged GETs to find newline boundaries, one thread parked on Socket.read. The
cost scales with file size rather than with LIMIT, so no read-side change can
avoid it.

Change

Discovery is now two phases. Planning does no I/O: it either produces splits
directly or emits a descriptor of the probe offsets a file needs. Every probe
across every file is then flattened into one BoundedParallelGather, so a
single global budget bounds in-flight GETs regardless of file count.

  • Probe offsets are fixed multiples of the stride, so serial and parallel
    produce identical boundaries; the gather preserves file order. An offset
    that finds no boundary costs only its own split: the splits either side of
    it merge, and every other offset's outcome stands.
  • Each probe reads a bounded 256 KB window and drains before closing rather than
    aborting, so the connection returns to the pool for the next probe.
  • The effective stride is floored to the 256 KB probe window: below it, probe
    windows overlap the same records anyway, and planning would materialize
    fileLength / stride probe tasks — unbounded below. The floor caps a file's
    probes at fileLength / window.
  • Concurrency is capped at 16 and clamped to the node's blob-store permits.
    Without an executor the probe phase walks the same offsets serially on the
    calling thread and produces identical boundaries.
  • Cancellation is checked between phases, inside each probe, and after the
    gather joins; a cancelled probe aborts its stream instead of draining it.

Measurements

split_discovery_nanos from the profile, same query and file, 147 splits at the
default 64 MB target split size: 61.0s on main (n=1) versus 2.5s here (n=3,
2.48-3.58s). End-to-end took is not a clean comparison: the read side now
dominates and varies several-fold run to run, which is a separate defect.

Coordinator-local planning change, so no wire format, settings, or cross-node
contract changes. Unit tests cover the probe arithmetic, boundary reduction,
drain-versus-abort, the concurrency ceiling, the stride floor, and cancellation
in each phase.

Closes https://github.com/elastic/esql-planning/issues/1528

@elasticsearchmachine elasticsearchmachine added v9.6.0 needs:triage Requires assignment of a team area label labels Jul 25, 2026
@julian-elastic julian-elastic self-assigned this Jul 25, 2026
@julian-elastic julian-elastic added :Analytics/ES|QL AKA ESQL >bug and removed needs:triage Requires assignment of a team area label labels Jul 25, 2026
@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jul 25, 2026
@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL >bug Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v9.6.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants