A lean, predictable, and blazingly fast Remote Build Execution (RBE) server for Bazel, written in Rust.
Most RBE solutions are built on the JVM, requiring constant GC tuning and 4GB+ memory just to idle. When your build cache server needs its own dedicated node, something is wrong.
FerrisRBE takes a different approach:
- Zero GC Pauses: Rust's ownership model eliminates garbage collection. Predictable p99 latencies without JVM tuning.
- O(1) Memory CAS Streaming: Stream 10GB artifacts with constant ~50MB RAM usage. No more OOM kills during large uploads.
- 12-Factor by Default: No XML, no YAML, no properties files. Just environment variables that operators already know how to manage.
- Adaptive Resilience: Workers auto-tune keepalive intervals based on network conditions. Transient failures don't fail builds.
Complete RBE stack with workers, cache, BES, and execution on your machine.
git clone https://github.com/xangcastle/ferrisrbe.git
cd ferrisrbe
# Build and load images with Bazel
bazel run //oci:server.amd64.image.load
bazel run //oci:worker.amd64.image.load
bazel run //oci:cache.amd64.image.load
bazel run //oci:bes.amd64.image.load
# Start the stack with Podman Compose
podman-compose -f podman-compose.yml up -d
# Verify the containers are running and healthy
curl -I http://localhost:9092 || echo "Server is up"
curl -I http://localhost:9096 || echo "BES UI is up"
# Configure Bazel
echo 'build:remote --remote_cache=grpc://localhost:9094' >> ~/.bazelrc
echo 'build:remote --remote_upload_local_results=true' >> ~/.bazelrc
echo 'build:remote --remote_executor=grpc://localhost:9092' >> ~/.bazelrc
echo 'build:remote --remote_default_exec_properties=OSFamily=linux' >> ~/.bazelrc
bazel build --config=remote //...# Create namespace
kubectl create namespace rbe
# Helm install from local chart
helm install ferrisrbe ./charts/ferrisrbe \
--namespace rbe --create-namespace
# Or with NodePort for local testing
helm install ferrisrbe ./charts/ferrisrbe \
--namespace rbe --create-namespace \
--set server.service.type=NodePort \
--set server.service.nodePort=30092 \
--set cache.service.type=NodePort \
--set cache.service.nodePortGrpc=30094 \
--set bes.service.type=NodePort \
--set bes.service.nodePortUi=30096For advanced deployments requiring extensive configuration, view the direct Helm deployment values.yaml.
FerrisRBE isn't a toy implementation; it's designed to handle the thundering herd of a massive monorepo CI pipeline.
- Multi-Level Queuing: Fast, medium, and slow queues automatically determined by action size. No more head-of-line blocking. Fast actions (<1s) never wait behind slow ones (>10s).
- Lock-Free Concurrency: Leveraging
DashMapwith 64 shards for L1 action cache and in-flight operations, ensuring high throughput without lock contention. Action cache reads in microseconds, not milliseconds. - Event-Driven Workers: Eliminates busy-waiting CPU cycles using
tokio::sync::Notify. Your cluster's CPU is for building, not polling. - Smart Materialization: Automatically degrades from zero-copy hardlinks to standard copies on
EXDEVcross-device volume mounts (perfect for containerized executors). - Zero-GC Runtime: Rust's ownership model eliminates garbage collection pauses. Predictable p99 latencies under any load.
FerrisRBE implements a tiered caching strategy:
┌─────────────────────────────────────────────────────────────┐
│ L1 Cache: DashMap (in-memory) │
│ - 64 shards, lock-free concurrent access │
│ - Microsecond-level reads (~50-100μs) │
│ - Lost on server restart │
├─────────────────────────────────────────────────────────────┤
│ L2 Cache: rbe-cache DiskActionCache │
│ - Persistent across restarts │
│ - Shared across server replicas │
│ - Millisecond-level reads (~1-3ms) │
├─────────────────────────────────────────────────────────────┤
│ L3 Storage: CAS (rbe-cache) │
│ - Persistent blob storage │
│ - Gigabyte-scale artifacts │
└─────────────────────────────────────────────────────────────┘
Current Status: L1 (DashMap) is implemented and provides exceptional performance for action cache hits. L2 integration is planned to provide persistence and horizontal scalability.
The following metrics were obtained through reproducible comparative benchmarks against other RBE solutions (Buildfarm, Buildbarn, BuildBuddy). Benchmark scripts are available in benchmark/.
| Solution | Language | Idle Memory | Relative to FerrisRBE | GC Pauses |
|---|---|---|---|---|
| FerrisRBE 🦀 | Rust | 6.7 MB | 1x (baseline) | None ✅ |
| Buildbarn | Go | ~120-200 MB | ~18-30x | Minimal |
| Buildfarm | Java | ~800-1200 MB | ~120-180x | Yes (G1GC) |
| BuildBuddy | Java/Go | ~1.2-2 GB | ~180-300x | Yes (JVM) |
Note: FerrisRBE is 120-300x more memory efficient than JVM-based solutions, and 18-30x more efficient than Go.
For a 20-node RBE cluster:
| Solution | Total Memory (Idle) | Est. Monthly Cost* |
|---|---|---|
| FerrisRBE | ~134 MB | $50-100 |
| Buildbarn | ~2.4-4 GB | $150-250 |
| Buildfarm | ~16-24 GB | $800-1,200 |
| BuildBuddy | ~24-40 GB | $1,200-2,000 |
*AWS/GCP estimate for instances required to support baseline memory.
| Component | Memory (Idle) | Memory (Peak) | CPU (Idle) |
|---|---|---|---|
| Server | ~5-10 MB | ~50-150 MB | ~0.01 cores |
| Worker | ~10-15 MB | ~100-200 MB | ~0.01 cores |
| Total | ~15-25 MB | ~150-350 MB | ~0.02 cores |
Compare to Java-based alternatives that idle at 500MB+ and spike to 4GB+ during GC.
| Metric | FerrisRBE | JVM Solutions |
|---|---|---|
| p50 Latency | 12 ms | 45 ms |
| p99 Latency | 18 ms | 180-500 ms* |
| p99.9 Latency | 25 ms | 500-2000 ms* |
| Large File Streaming | O(1) memory | O(n) memory, OOM risk |
| Connection Cleanup | Immediate | Delayed (zombie threads) |
| Cache Stampede | Coalesced | Backend overload |
| Consistency | ✅ High |
* Latency spikes caused by GC pauses
| Test | FerrisRBE | JVM Solutions |
|---|---|---|
| O(1) Streaming | Constant memory regardless of file size | Memory scales with file size |
| Connection Churn | Immediate task cancellation | Zombie threads, resource leaks |
| Cache Stampede | Request coalescing prevents overload | Backend overwhelmed |
| Multi-level Scheduling | Fast actions never blocked | Head-of-line blocking (FIFO) |
The benchmark suite tests eight critical dimensions of RBE performance:
cd benchmark
# 1. Memory footprint (baseline)
./scripts/benchmark.sh
# 2. Execution API throughput (Zero-GC advantage)
./scripts/execution-load-test.py --actions 1000 --concurrent 50
# 3. Action Cache performance (L1 DashMap + L2 disk-backed)
./scripts/action-cache-test.py --operations 10000 --concurrent 100
# 4. Multi-level scheduler (no head-of-line blocking)
./scripts/noisy-neighbor-test.py --slow 10 --fast 50
# 5. O(1) Streaming (constant memory with large files)
./scripts/o1-streaming-test.py --large-sizes 5 10 --small-count 1000
# 6. Connection churn (resource cleanup)
./scripts/connection-churn-test.py --connections 1000 --disconnect-rate 0.3
# 7. Cache stampede (thundering herd protection)
./scripts/cache-stampede-test.py --requests 10000 --concurrent 100
# 8. Cold start time (<100ms vs 5-30s JVM)
./scripts/cold-start-test.shSee benchmark/README.md for detailed benchmark documentation and benchmark/results/BENCHMARK_RESULTS.md for results.
Add to your .bazelrc:
# Remote Cache (works from any OS)
build:remote-cache --remote_cache=grpc://localhost:9094
build:remote-cache --remote_upload_local_results=true
# Remote Execution (requires Linux toolchains)
build:remote-exec --config=remote-cache
build:remote-exec --remote_executor=grpc://localhost:9092
build:remote-exec --remote_default_exec_properties=OSFamily=linux# Cache only
bazel build --config=remote-cache //...
# Full remote execution
bazel build --config=remote-exec //...# You should see "remote cache hit" and "remote" execution in the output
bazel build --config=remote //... 2>&1 | grep -E "(remote cache hit|processes)"FerrisRBE strictly follows 12-Factor App methodology. No cryptic XML or YAML files required.
| Env Variable | Default | Description |
|---|---|---|
RBE_PORT |
9092 |
Server listening port |
RBE_L1_CACHE_CAPACITY |
100000 |
Max entries in the in-memory action cache |
RBE_INLINE_OUTPUT_THRESHOLD |
1048576 |
Size (bytes) below which outputs are sent inline |
RBE_MAX_CONCURRENT_DOWNLOADS |
10 |
Concurrency limit for materializing execroots |
See docs/configuration.md for the complete reference.
- Architecture - System design and components
- Deployment - Kubernetes, Helm, and Docker deployment
- Configuration - Environment variables and tuning
- Bazel Integration -
.bazelrcconfiguration - API Reference - REAPI v2.4 endpoints
- Monitoring - Metrics and logging
- Troubleshooting - Common issues and solutions
ferrisrbe/
├── src/
│ ├── server/ # gRPC services (REAPI v2.4)
│ ├── execution/ # Scheduler, state machine, results
│ ├── worker/ # Worker registry and management
│ ├── cas/ # Content Addressable Storage backends
│ └── cache/ # L1 Action Cache (DashMap)
├── charts/ # Helm charts for Kubernetes
├── k8s/ # Raw Kubernetes manifests
├── examples/ # Test projects (Bazel 7.4, 8.x, 9.x)
└── docs/ # Full documentation
- REAPI v2.4 Capabilities Service
- Action merging (deduplication of identical in-flight actions)
- HTTP/2 adaptive keepalive for resilient worker connections
- Multi-level scheduler (Fast/Medium/Slow queues)
- DashMap-based L1 Action Cache (lock-free, 64 shards)
- O(1) CAS streaming with async I/O
- Comprehensive benchmark suite (8 dimensions)
- Persistent L2 Cache (disk-backed ActionCache in rbe-cache)
- Implemented: DiskActionCache provides persistence across server restarts and cache sharing across multiple server replicas.
- Prometheus / OpenTelemetry metrics exposition
- Why: Production observability for SRE teams
- Web UI for build monitoring
- Real-time build queue visualization
- Worker status and health
- Cache hit/miss analytics
- Remote Build Without the Bytes (BwoB) support
- Skip downloading outputs when not needed
- Compressed CAS transfers (zstd)
- Reduce network bandwidth for large artifacts
PRs are welcome. We value:
- Clean abstractions
- Explicit error handling
- Comprehensive documentation
- Avoiding
unwrap()in critical paths
See docs/project-structure.md for codebase orientation.
Built with 🦀 for engineers who value predictability.