Skip to content

[codex] fix sync oversized raw uploads#218

Draft
upsetbit wants to merge 1 commit into
masterfrom
codex/sync-stream-upload
Draft

[codex] fix sync oversized raw uploads#218
upsetbit wants to merge 1 commit into
masterfrom
codex/sync-stream-upload

Conversation

@upsetbit
Copy link
Copy Markdown
Contributor

@upsetbit upsetbit commented Jun 6, 2026

Summary

  • Add SessionsService.PushChunk for bounded unary raw uploads.
  • Use chunked uploads from the CLI for raw files above 32 MiB, and as fallback when unary Push hits resource_exhausted.
  • Keep existing unary Push for small raws.
  • Validate full raw size and SHA-256 on the server before uploading to object storage or committing metadata/sync_state.
  • Reject gaps, overlaps, out-of-order chunks, truncated raws, and hash mismatches.

Validation

  • just gen
  • gofmt on changed Go files
  • go test ./internal/cli ./internal/server/...
  • just build
  • Local server started from this worktree with the repo .env.
  • CLI auth was temporarily pointed from http://100.100.253.82:7070 to http://127.0.0.1:7070 for local validation, then restored. 100.100.253.82 is not this host's Tailscale IP; the local Tailscale IP is 100.101.225.96.
  • ./bin/prosa sync --verbose completed locally with Catch-up: sent 2 · skipped 0 · errors 0 for the two previously failing large sessions.

Integrity checks

Compared local SQLite/raw files against Postgres and MinIO for:

  • 019e7f15-7f1d-7b60-a05a-f9292ed63841

    • local size/hash: 81729232, bd66ae61e2f7f2cfc92529a509f90dfca1495bbc666347a1db217ca3d9b3bdac
    • Postgres size/hash/sync_state: matched
    • MinIO object size/hash streamed with aws s3 cp ... - | shasum -a 256: matched
    • turns/tools counts local vs Postgres: 531 / 9, matched
  • 08dc7d62-210e-4f11-b677-3b1c1b8cb391

    • local size/hash: 93724672, 1d940811b1265a66845cf3f40e177803157f53d32bc8893695215486d06f832c
    • Postgres size/hash/sync_state: matched
    • MinIO object size/hash streamed with aws s3 cp ... - | shasum -a 256: matched
    • turns/tools counts local vs Postgres: 106 / 10, matched

View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant