Throughput drops to 0 with timeouts when scaling from 16 to 32 replicas on a single machine — is large-scale (30+) testing expected to require multiple physical machines?

[iifishing]

I'm running consensus throughput experiments with this repo and running into a scaling issue when increasing the replica count on a single machine.
Setup: All replicas, clients, and the controller are running on a single machine (no --replica-hosts/--client-hosts, i.e. local mode).
Working command (16 replicas, runs fine):
bash./hotstuff run --replicas 16 --clients 1 --batch-size 96 --max-concurrent 400
--crypto ecdsa --view-timeout 5s --client-timeout 10s --duration 60s
--measurement-interval 1s --output experiment_data --log-level debug
Failing command (32 replicas, same parameters otherwise):
bash./hotstuff run --replicas 32 --clients 1 --batch-size 96 --max-concurrent 400
--crypto ecdsa --view-timeout 5s --client-timeout 10s --duration 60s
--measurement-interval 1s --output experiment_data --log-level debug
Observed behavior: At 32 replicas, the run frequently produces repeated view timeouts, and the client reports 0 (or near-0) executed commands — throughput effectively drops to 0, even though the exact same parameters work fine at 16 replicas.
My questions:

Is this a known limitation of running 30+ replicas on a single machine, or is it expected to scale linearly with more resources?
For large-scale experiments (30+, 50+, 100+ replicas), what parameter tuning would you recommend (e.g. --view-timeout, --client-timeout, --max-concurrent, --batch-size) to keep consensus stable as replica count grows?
Does this codebase's design assume/require distributing replicas across multiple physical machines for larger-scale experiments (via --replica-hosts/--client-hosts), rather than running everything locally on one machine? If so, is there a recommended minimum replica-count threshold beyond which single-machine testing is no longer representative/reliable?

Any guidance on recommended configurations or known scaling limits for large-N experiments would be very helpful. Happy to share full debug logs if useful.

[meling]

These are good questions, although there isn't a single answer because the behavior depends heavily on the hardware and the workload.

A few general observations:

• HotStuff (and consensus protocols in general) is designed to run on a cluster of machines. The ability to run all replicas on a single machine is primarily intended to simplify development, testing, and small-scale benchmarking.
• Once all replicas share the same CPU, memory subsystem, scheduler, and network stack, you're no longer measuring a realistic distributed deployment. Instead, the bottleneck often becomes local resource contention (CPU scheduling, goroutines, logging, gRPC, kernel networking, etc.) rather than the consensus protocol itself.
• For that reason, results obtained on a single machine are generally not representative when comparing different protocols or evaluating scalability. They can, however, be useful when evaluating changes to the implementation itself, since the environment is reproducible.

Regarding your specific experiment, I ran the same commands on an Apple M2 Max, and both the 16- and 32-replica configurations completed successfully with sustained throughput. I did not observe throughput dropping to zero or repeated view timeouts.

That suggests the issue is likely environment-dependent rather than a fundamental limitation in the implementation.

A few things I'd recommend trying:

• Reduce the log level to info (or even warn). debug logging generates a substantial amount of output, and with many replicas, the terminal itself can become a bottleneck. I ran the experiments in the Ghostty terminal, the fastest available.
• Try redirecting the output to a file instead of rendering everything in the terminal.
• Monitor CPU utilization while the experiment is running. If the machine is CPU-saturated, view timeouts are expected because replicas simply don't get scheduled frequently enough.
• If the machine is heavily loaded, increasing --view-timeout (and possibly --client-timeout) may help avoid unnecessary view changes. There is no universally correct timeout—it depends on how quickly replicas can make progress on your hardware.

As for parameter tuning, there is unfortunately no formula that scales with the number of replicas. The optimal values depend on many factors, including:

• CPU performance and number of cores
• cryptographic backend (ecdsa vs. threshold signatures, etc.)
• batch size
• client concurrency
• workload characteristics

In general, increasing the replica count increases communication and cryptographic work, so it is often necessary to relax timeouts and/or reduce the offered load (--max-concurrent) if the machine becomes saturated.

Finally, for experiments involving a large number of replicas, I would generally recommend distributing replicas across multiple physical machines. That more closely matches the deployment model the implementation is designed for and avoids many artifacts that arise when running every process on a single machine.

Hope this helps.