Skip to content

F5 (cheap follow-up): replace tokio::io::copy merge with std::fs::copy or sendfile(2) #305

@crippledgeek

Description

@crippledgeek

From the 2026-05-04 download-optimization audit (F5 in docs/superpowers/specs/2026-05-04-download-optimization-audit.md).

The full F5 proposal (pre-allocate + pwrite chunks directly into output, skip the merge step entirely) was dropped after Round-2 research showed:

  • Merge cost is 2.7s on NVMe ext4, ~17s on HDD, never on user-perceived latency path
  • Windows SetFileValidData requires SE_MANAGE_VOLUME_PRIVILEGE
  • btrfs fallocate permanently disables compression
  • NFS falls back to write-zeros (blocking)
  • No user has hit the 30-min merge timeout
  • AIMD ramps to 8 MB chunks → 128 chunks for 1 GB (not 16k as audit claimed)

The cheap micro-optimization that remains: replace tokio::io::copy in merge_chunks_ordered (crates/rdlp-downloader/src/http/parallel.rs:520-594) with std::fs::copy (spawn_blocking-wrapped) or sendfile(2) to eliminate the userspace buffer hop.

When to act

Only if someone reports merge slowness in practice. Until then, low priority.

Acceptance

  • 5-line change replacing tokio::io::copy call with spawn_blocking(|| std::fs::copy(...))
  • No resume-marker changes (keep current per-chunk-file architecture)
  • Pre-push gate green

References

  • Spec non-goal: docs/superpowers/specs/2026-05-21-f3-f6-download-optimization-design.md (F5 dropped after research)
  • Audit: docs/superpowers/specs/2026-05-04-download-optimization-audit.md (F5 section)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions