Non-multitask 2D gintervals.quantiles(intervals.set.out=) truncates to the first ~1000 intervals (spurious NaN at scale)

[aviezerl]

Summary

gintervals.quantiles() on a 2D track with intervals.set.out= (the streaming
big-set path) returns mostly NaN for a large scope when options(gmultitasking = FALSE).
Only the first ~1000 intervals (a single contiguous block, the start of the first
chromosome pair) get finite quantiles; every interval after that is NaN. The in-memory
result and the default multitasking big-set result are both correct.

This is not a regression from the 5.11.9 audit fixes - it reproduces with a single
percentile, where the 5.11.9 quantiles buffer fix is a no-op (size*1 == size). It was
found while adversarially testing those fixes.

Reproduce (misha 5.11.9, internal test DB)

devtools::load_all(export_all = FALSE)
source("tests/testthat/helper-test_db.R")
gdb.init(create_isolated_test_db())

scope <- gscreen("test.rects > 40", gintervals.2d(c(1, 2), 0, -1))   # 245193 2D intervals
nrow(scope)
# 245193

# in-memory: correct
im <- gintervals.quantiles("test.rects", 0.5, scope, iterator = c(1, 1))
sum(is.finite(im[[7]]))
# 245193   (all finite)

# non-multitask streaming big-set: WRONG
withr::with_options(list(gmultitasking = FALSE, gbig.intervals.size = 10), {
  gintervals.quantiles("test.rects", 0.5, scope, iterator = c(1, 1), intervals.set.out = "test.cs")
  fd <- gintervals.load("test.cs"); gintervals.rm("test.cs", force = TRUE)
  sum(is.finite(fd[[7]]))                       # 1000   <-- only the first 1000
  all(which(is.finite(fd[[7]])) == 1:1000)      # TRUE   <-- first contiguous block
})

# default multitask streaming big-set: correct
withr::with_options(list(gmultitasking = TRUE, gbig.intervals.size = 10, gmax.processes = 3), {
  gintervals.quantiles("test.rects", 0.5, scope, iterator = c(1, 1), intervals.set.out = "test.cs2")
  fd2 <- gintervals.load("test.cs2"); gintervals.rm("test.cs2", force = TRUE)
  sum(is.finite(fd2[[7]]))                      # 245193  (all finite)
})

Expected vs actual

path	finite quantiles
in-memory (gintervals.quantiles no intervals.set.out)	245193 / 245193 ✓
big-set, default gmultitasking = TRUE	245193 / 245193 ✓
big-set, gmultitasking = FALSE	1000 / 245193 ✗

Notes / likely location

• The finite block is exactly the first 1000 intervals (contiguous), so the
  non-multitask 2D streaming scan appears to stop early and the remaining chromosome
  pairs are written via the "skipped scope chromosome" branch as all-NaN.
• Path: the do_big_intervset_out 2D branch of gintervals_quantiles in
  src/GenomeTrackQuantiles.cpp (the non-multitask streaming writer). The multitask
  twin (gintervals_quantiles_multitask) is unaffected.
• 1D appears unaffected; small 2D scopes are correct. Only large (>~1000 interval)
  2D scopes on the non-default gmultitasking = FALSE path are wrong.

Affects: misha 5.11.9 (and earlier).