feat: gVisor DST integration with syscall-level fault injection - #1
Merged
Conversation
Research into alternative hypervisors (bhyve, Firecracker, Cloud Hypervisor) and the fundamental question of whether hypervisors are needed in the age of AI-generated DST-compatible code. Key findings: - Antithesis chose bhyve for clean codebase, took 5 years to build - Library-based DST (TigerBeetle) is 1000x faster but requires code changes - AI could generate DST-compatible code by default - Hybrid approach: fast library DST + thorough hypervisor validation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds a hypervisor abstraction layer and Firecracker implementation: - traits.rs: Common Hypervisor trait for QEMU, Firecracker, Cloud Hypervisor - VmState, SnapshotType, SnapshotInfo types - HypervisorType enum with performance characteristics - Async trait with start/stop/pause/resume/snapshot/restore - firecracker/: Complete Firecracker microVM integration - REST API client over Unix socket (hyper + hyperlocal) - Snapshot create/restore with ~5ms latency target - Full VM lifecycle management - Statistics tracking for latencies Performance comparison: - QEMU TCG: ~2-5s boot, ~200ms restore, full determinism - Firecracker: ~125ms boot, ~5ms restore, KVM only This provides a fast alternative to QEMU TCG for rapid iteration while keeping QEMU for full deterministic testing. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds Cloud Hypervisor implementation with same Hypervisor trait: - cloud_hypervisor/config.rs: VM configuration - cloud_hypervisor/client.rs: REST API client - PUT /vm.create, /vm.boot, /vm.pause, /vm.resume - PUT /vm.snapshot, /vm.restore - cloud_hypervisor/vm.rs: VM lifecycle with Hypervisor trait Cloud Hypervisor vs Firecracker: - Same performance (~150ms boot, ~5ms restore) - Additional features: Windows guests, GPU passthrough, device hotplug - Slightly different API but same concepts Both are KVM-only, so no deterministic time control. Use QEMU TCG for full determinism, microVMs for fast iteration. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds comprehensive Firecracker integration tests: - Basic lifecycle (start, stop, pause, resume) - Snapshot create and restore - Multi-cycle snapshot/restore - Restore latency benchmarking - Exploration branching pattern Also adds guest/scripts/build-firecracker-rootfs.sh: - Builds Alpine Linux-based ext4 rootfs - Includes minimal init for Firecracker - Supports optional redis-rust binary Tests are #[ignore] by default - require: - KVM (/dev/kvm) - Firecracker binary - Kernel and rootfs images Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Research into fast alternatives to QEMU TCG for deterministic execution: - rr: 1.2-2x overhead but single-core only - Hermit: Near-native but maintenance mode, unreliable - Intel PT: Trace-only, not for replay - Direct Binary Translation: Research stage, promising 2-3x speedup Conclusion: No fast deterministic alternative exists today. Antithesis took 5 years to build one. Recommended hybrid approach: 1. Harness mode for unit tests (fast, simulated) 2. Firecracker for VM tests (fast, non-deterministic) 3. QEMU TCG for validation (slow, deterministic) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add gVisor as a container-based hypervisor backend for Bloodhound, enabling deterministic simulation testing without hardware virtualization. Key features: - GvisorVm implementing Hypervisor trait with full DST support - Unix socket client for DST control protocol (JSON-RPC) - FaultProbabilities for syscall-level fault injection - Fast CoW snapshots (~5ms restore) - ~50ms container boot time New files: - src/hypervisor/gvisor/ - Complete gVisor backend implementation - config.rs: GvisorConfig, DstConfig, FaultProbabilities - client.rs: GvisorClient with DST command protocol - vm.rs: GvisorVm implementing Hypervisor trait - tests/gvisor_integration.rs - 20+ integration tests - examples/redis-rust-gvisor/ - Example with gVisor-specific features - syscall fault injection (EIO, EAGAIN, EINTR, ENOMEM) - time fault injection (clock skew, clock jump) - syscall-resilience property check Note: Requires building the gVisor DST fork (gvisor/) with Bazel. See gvisor/BUILD_DST.md for build instructions. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add gvisor-patches/ directory containing the DST patch series for gVisor, following the same pattern as qemu-patches/. The actual gVisor fork with these patches applied is maintained at https://github.com/nerdsane/gvisor (branch: bloodhound-dst). Patch series: - 001: Virtual time (VirtualClocks, DST integration) - 002: Deterministic RNG (ChaCha20-based) - 003: Cooperative scheduling (deterministic goroutines) - 004: Deterministic network (ordered packets) - 005: Deterministic filesystem (inode alloc, dir ordering) - 006: Save/Restore (snapshot tree, CoW pages) - 007: Bloodhound integration (fault injection, properties) Also adds gvisor/ to .gitignore since the fork should be cloned separately for development. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add syscall-level fault injection for Read/Write operations - Add FD filtering (0-4) to protect stdio and async runtimes - Add deterministic replay support (verified with seed=42) - Add ADR-009 for gVisor DST integration architecture - Update ADR-004 and ADR-005 with gVisor implementation details - Update CLAUDE.md with: - Systematic debugging approach for multi-process systems - gVisor DST mode documentation - ADR quick reference updates - Add Python DST control client for external control - Update redis-rust-gvisor example with healthchecks Key gVisor changes (in ./gvisor fork): - runsc/config/flags.go: Fix FaultDiskRead flag propagation - pkg/sentry/dst/bloodhound.go: Add fault injection helpers - runsc/boot/loader.go: Initialize DST coordinator on startup Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a third execution mode to Bloodhound: gVisor DST Mode - container-native deterministic simulation testing without VM overhead.
Key Features
Changes
Documentation
Code
dst_control_client.py)gVisor Fork (separate repo: nerdsane/gvisor-dst)
Test plan
🤖 Generated with Claude Code