feat(issue4): add real NCCL bitwise reproducibility diagnostics

src/code/issue4/bitwise_diagnostics/.gitignore

@@ -0,0 +1,3 @@
+__pycache__/
+diagnostic-output/
+sweep-output/

src/code/issue4/bitwise_diagnostics/EXPERIMENT.md

@@ -0,0 +1,100 @@
+# Four-GPU reproducibility experiment
+
+Date: 2026-07-23
+
+The machine-readable summary is
+[`results/autodl-4x3090.json`](results/autodl-4x3090.json).
+
+## Environment
+
+- 4 × NVIDIA GeForce RTX 3090 24 GiB, driver 580.105.08
+- PyTorch 2.8.0+cu128, CUDA 12.8, NCCL 2.27.3
+- GPU 0↔1 and GPU 2↔3: `PXB`; traffic between the pairs: `SYS`
+- two NUMA nodes; no NVLink reported by `nvidia-smi topo -m`
+- fixed seed 2026 and rank-distinct float32 inputs
+
+Each reported run is a fresh four-rank `torchrun --standalone` launch. This
+matters: repeated calls inside one communicator do not test communicator
+initialization or run-to-run selection.
+
+## Same-configuration results
+
+At 1 MiB input per rank, AllReduce covered nine
+`NCCL_ALGO × NCCL_PROTO` configurations:
+
+| Algorithm | Protocols | Independent runs | Calls/run | Result |
+|---|---|---:|---:|---|
+| automatic | automatic, Simple, LL | 5 each | 20 | bitwise identical |
+| Ring | automatic, Simple, LL | 5 each | 20 | bitwise identical |
+| Tree | automatic, Simple, LL | 5 each | 20 | bitwise identical |
+
+Reduce-Scatter used the same input size and repetition count:
+
+| Algorithm | Protocols | Result |
+|---|---|---|
+| automatic | automatic, Simple, LL | bitwise identical |
+| Ring | automatic, Simple, LL | bitwise identical |
+| Tree | automatic, Simple, LL | rejected with `ncclInvalidUsage` |
+
+Tree is therefore not presented as a Reduce-Scatter mitigation on this NCCL
+version. Importantly, the runner did not silently fall back to Ring.
+
+No same-configuration run-to-run divergence was observed. This is a bounded
+negative result for this exact software and PCIe topology, not a universal
+claim that these algorithms are deterministic.
+
+## Granularity
+
+NCCL automatic selection was also checked across message sizes:
+
+| Bytes/rank | Independent runs | Calls/run | Result |
+|---:|---:|---:|---|
+| 1 KiB | 5 | 50 | bitwise identical |
+| 64 KiB | 5 | 30 | bitwise identical |
+| 1 MiB | 5 | 20 | bitwise identical |
+| 16 MiB | 5 | 3 | bitwise identical |
+
+The call count is lower for large messages because the diagnostic retains raw
+bytes until comparison. Every row still compares independent launches with
+the same call count within the row.
+
+## Reproducible algorithm-change case
+
+To verify localization and demonstrate the numerical consequence of reduction
+order, one factor was changed from Ring to Tree while seed, rank inputs,
+hardware, dtype, message size, and call sequence stayed fixed.
+
+The first difference appeared at call 0, rank 0, byte 24 (float32 element 6).
+For that 1 MiB output:
+
+- 94,656 bytes / 192,767 bits changed;
+- maximum absolute error was `9.5367431640625e-07`;
+- maximum reported ULP distance was 49,152;
+- Ring SHA-256:
+  `c5797c4ea5b9d3a387fab7d4e3a6405fa60f3397d19a050070e896433b99d513`;
+- Tree SHA-256:
+  `3f3470571b53caf023710336ca21a493c7a27e27e40871280a5c85a44d44a991`.
+
+This is a controlled configuration-change case, **not** evidence of
+nondeterminism within either fixed configuration. It demonstrates why
+algorithm/topology selection must remain stable when bitwise continuity
+between jobs is required.
+
+## Conclusions by acceptance dimension
+
+1. **Granularity:** no run-to-run difference was observed from 1 KiB through
+   16 MiB under automatic selection. Larger captures cost proportionally more
+   memory; sampling call count is an explicit experimental trade-off.
+2. **Hardware:** results apply to a four-GPU, dual-NUMA PCIe host. The `SYS`
+   boundary between GPU pairs is materially different from NVLink/NVSwitch;
+   NVLS was therefore not tested or recommended.
+3. **Software flow:** fresh process groups, fixed rank mapping, fixed inputs,
+   and explicit algorithm/protocol settings produced stable bytes. Switching
+   Ring to Tree immediately changed results. For reproducible jobs, pin the
+   tested software stack and selection controls, but only use algorithms that
+   support the target collective.
+
+The evidence supports fixed Ring (including Simple or LL on this host) as a
+measured reproducible option for both tested collectives. This is deliberately
+scoped to the recorded environment rather than framed as a universal NCCL
+guarantee.

src/code/issue4/bitwise_diagnostics/README.md

@@ -0,0 +1,121 @@
+# NCCL bitwise reproducibility diagnostics
+
+This tool launches the same collective workload in **separate `torchrun`
+process groups**, captures every output as raw bytes, and compares all ranks
+and calls against the first run. A JSON report identifies the first divergent
+call, rank, byte/element offset, changed bits, absolute error, and ULP error.
+
+Unlike an arithmetic simulation, this exercises the installed PyTorch, NCCL,
+CUDA runtime, topology discovery, and actual communication path.
+
+## Requirements
+
+- Linux, Python 3.10+, PyTorch with NCCL, and at least two CUDA GPUs
+- All tested ranks must see the same GPUs in every independent run
+- For multi-node experiments, invoke `worker.py` with the site's normal
+  rendezvous command and compare its capture files with `core.compare_runs`
+
+No claim of determinism is inferred from an algorithm name. The report only
+describes the measured hardware/software configuration.
+
+See [`EXPERIMENT.md`](EXPERIMENT.md) for measured four-GPU results covering
+algorithm/protocol selection, message granularity, both collectives, and a
+controlled Ring-versus-Tree difference.
+
+## Quick start
+
+```bash
+cd src/code/issue4/bitwise_diagnostics
+
+# NCCL automatic selection
+python diagnose.py --nproc-per-node 8 --runs 5 \
+  --op all_reduce --elements 1048576 --calls 20 \
+  --output-dir results/default
+
+# Pin one supported algorithm/protocol for an A/B comparison
+python diagnose.py --nproc-per-node 8 --runs 5 \
+  --algo Ring --proto Simple --op all_reduce \
+  --elements 1048576 --calls 20 --output-dir results/ring-simple
+
+# Unified A/B matrix; unsupported combinations are recorded, not hidden
+python sweep.py --nproc-per-node 8 --runs 5 \
+  --algos default,Ring,Tree --protos default,Simple,LL
+```
+
+Exit status is `0` for bitwise-identical runs, `2` when a divergence is
+detected, and non-zero on invalid configuration or launch failure. Use
+`--keep-payloads` when forensic inspection is needed; otherwise multi-MB raw
+captures are deleted after `report.json` is written.
+
+Existing captures can also be compared offline (including captures copied
+from different nodes):
+
+```bash
+python core.py run-a.json run-b.json --output comparison.json
+```
+
+Run CPU-only unit tests with:
+
+```bash
+python -m unittest -v test_core.py
+```
+
+## Controlled experiment matrix
+
+Change one factor at a time and retain `report.json` for each row:
+
+1. **Message granularity:** 1 KiB, 64 KiB, 1 MiB, 16 MiB, 128 MiB. Since
+   raw captures scale as `elements × dtype bytes × calls × ranks × runs`,
+   reduce `--calls` for the largest cases and keep the value identical across
+   compared configurations.
+2. **Collective:** `all_reduce` and `reduce_scatter`.
+3. **Selection:** NCCL default, then supported `NCCL_ALGO` values with
+   `Simple`, `LL`, and (only on supported platforms) `LL128`.
+4. **Resources:** record GPU model/count, NVLink/NVSwitch, NIC, node count,
+   PyTorch/CUDA/NCCL versions, and topology. Worker metadata records the
+   software versions and effective experiment overrides automatically.
+5. **Stability:** at least five independent launches and 50 calls per launch.
+
+Do not force unsupported combinations. NCCL 2.24+ fails on invalid algorithm
+tokens, and NVIDIA warns that forcing LL128 on unsupported platforms can cause
+data corruption. `NCCL_ALGO` and `NCCL_PROTO` are diagnostic controls, not
+universal production recommendations.
+
+## Interpreting results across the three acceptance dimensions
+
+- **Granularity:** small messages emphasize launch/protocol behavior; large
+  messages exercise more chunks/channels and expose more reduction sites.
+  Compare divergence rate and first-call position alongside latency measured
+  by a dedicated benchmark such as `nccl-tests`.
+- **Hardware resources:** rank count and topology determine valid algorithms
+  and reduction paths. A result on NVSwitch must not be generalized to PCIe or
+  multi-node fabrics. A pinned `NCCL_TOPO_FILE` is useful only when it
+  accurately represents the tested system.
+- **Software flow:** fresh process groups test run-to-run reproducibility,
+  which repeated calls inside one communicator cannot establish. Pin package
+  versions and seeds; serialize collectives in the same order on every rank.
+  PyTorch deterministic-algorithm settings and
+  `CUBLAS_WORKSPACE_CONFIG=:4096:8` matter for surrounding compute, but they do
+  not constitute evidence that a collective is bitwise reproducible.
+
+## Reproducible non-determinism case
+
+The tool deliberately does not fabricate nondeterminism. To document a case:
+
+1. run the default selection matrix on the target cluster;
+2. retain the first report with `bitwise_identical=false`;
+3. repeat with one variable pinned at a time;
+4. report a mitigation only if repeated measurements turn identical;
+5. attach NCCL `INFO` logs to show the selected algorithm/topology.
+
+This separates observed evidence from assumptions about Ring, Tree, PAT, or
+NVLS internals and avoids recommending undocumented environment variables.
+
+## References
+
+- NVIDIA NCCL environment variables:
+  https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html
+- PyTorch reproducibility:
+  https://docs.pytorch.org/docs/stable/notes/randomness.html
+- PyTorch deterministic algorithms:
+  https://docs.pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html