A detailed information of the working + results and observation can be viewed on https://docs.google.com/document/d/1-uFeC7B5LJ4hgEFsKNKpbJyWdwnLC1os/edit?usp=sharing&ouid=110504837487987153614&rtpof=true&sd=true
Team Project by Aiori Team (PS06)
Build a toolchain to simulate Anycast address flipping effects on CDN-driven web performance, with particular focus on metrics such as First Contentful Paint (FCP).
- Dataset Analysis: Use real-world flip rates (approximately 3.2% of vantage points) and observed RTT penalties (≥50 ms) to model flips.
- Emulation Framework: Integrate flipping probability and latency overhead into network-based page load tests.
- Performance Impact: Measure changes in FCP for HTTP/1.1 and HTTP/2 under flip scenarios.
- Mitigation Ideas: Propose routing or caching strategies to reduce performance degradation.
This modular framework simulates and quantifies the effects of Anycast flips on CDN-driven web performance. Each layer focuses on a defined function, from modeling flip data to reproducing network behavior and validating mitigation strategies.
Represent real-world Anycast flip dynamics—situations where a user's connection is rerouted to a different Point of Presence (POP), typically increasing RTT and degrading performance.
- Flip Probability: Approximately 3.2% of TCP connections experience a routing flip.
- Latency Penalty: A flipped connection incurs an additional +50 ms RTT.
- Per-Connection Independence: Each TCP flow samples the flip probability independently.
A Bernoulli process determines if a new connection is routed to the high-latency POP:
- p = 0.032
- If a flip occurs, an additive delay ΔRTT = 50 ms is applied.
- Empirical Mode: Realistic parameters (3.2% flip rate, +50 ms penalty).
- Stress Mode: Elevated flip probability (e.g., 50%) for visible impact.
- Base RTT per POP
- Flip probability (p)
- Latency penalty (ΔRTT)
- Mode (empirical / stress)
This feeds into the emulation setups below.
Two complementary emulation approaches are implemented to reproduce Anycast flips under controlled and realistic conditions.
Recreate Anycast flipping behavior under reproducible and configurable conditions using Mahimahi network shells, Python-based proxy logic, and browser automation for accurate FCP measurement.
This approach operates at the application layer, allowing precise control over latency, connection handling, and flip probabilities. Two CDN Points of Presence (POPs) are emulated using Mahimahi shells:
- POP A (Low latency): 30 ms RTT
- POP B (High latency): 80 ms RTT
A flip-aware proxy (written in Python) probabilistically routes new connections to either POP A or POP B, based on the modeled flip probability.
-
POP Emulation Each POP runs an independent Nginx server under Mahimahi with different delays, providing distinct RTT environments.
-
Flip Proxy (
flip_proxy.py) Implements per-connection flipping using a Bernoulli process. For each new TCP connection:if random.random() < 0.032: route_to_high_latency_pop() else: route_to_low_latency_pop()
Logs each flip decision along with response timings.
-
Test Driver (
test_proxy.py) Generates repeated HTTP requests to the proxy, records latency, and compiles results into CSV logs for later analysis. -
Browser-Level Testing (Puppeteer) Uses Puppeteer automation to open static test pages through the proxy and record:
- First Contentful Paint (FCP)
- Page Load Time
- RTT and resource timing data
-
Log Analysis The latency and FCP distributions are compared between flipped and non-flipped flows to verify empirical consistency with the model (≈3.2% flips).
src/
└── anycast-flip-experiment/
├── flip_proxy.py
├── logs/
│ ├── access-high.log
│ ├── access-low.log
│ ├── error-high.log
│ └── error-low.log
├── nginx-configs/
│ ├── mime.types
│ ├── nginx-high.conf
│ └── nginx-low.conf
├── public/
├── test-page/
│ ├── images/
│ ├── index.html
│ ├── scripts/
│ │ ├── script1.js
│ │ └── script2.js
│ └── styles/
│ └── style1.css
├── analyze_webvitals.py
├── test_proxy.py
└── start_experiment_fixed.sh
- Configurable flip probability and latency penalty.
- Independent TCP flow handling using Python sockets.
- Detailed access and error logging for each backend.
- Reproducible browser-side measurements (FCP and total load).
- Automated post-processing via
analyze_webvitals.py.
This approach effectively captures client-side performance degradation under controlled Anycast-like conditions and supports both HTTP/1.1 and HTTP/2 scenarios.
Simulate real Anycast routing behavior with BGP convergence and failover using Docker and Free Range Routing (FRR), focusing on the network-layer causes of Anycast flips.
This setup emulates Anycast routing across multiple Points of Presence (POPs) in a fully virtualized environment.
Two nodes advertise the same Anycast prefix (10.0.0.100/32) via BGP to a router container.
A client container accesses this Anycast IP; routing decisions determine which node is currently serving the traffic.
-
Router (FRR) Runs BGP and learns routes from both nodes, selecting the preferred one based on BGP attributes.
-
Node A / Node B Each runs FRR and Nginx, advertising
10.0.0.100/32with unique preferences to the router. -
Client Sends HTTP requests to
10.0.0.100. Traffic is automatically routed to the active node without any manual redirection. -
Flip Trigger Achieved by stopping one node or changing route attributes (Local Pref / MED), simulating real-world route withdrawal and reconvergence.
When a flip occurs (for example, when Node A stops or withdraws its route):
-
BGP Convergence The router withdraws the route from Node A and selects Node B as the new next-hop for the Anycast prefix.
-
Client Experience Subsequent requests to
10.0.0.100are transparently redirected to Node B without changing the client’s target IP. -
Measured Metrics
- Route propagation and convergence time.
- Session persistence across BGP changes.
- Failover delay and potential packet loss.
Anycast-CDN-Setup/
├── README.md
├── client-scripts/
├── docker-compose.yml
├── frr/
│ ├── daemons
│ ├── nodea/
│ │ └── frr.conf
│ ├── nodeb/
│ │ └── frr.conf
│ ├── router/
│ │ └── frr.conf
│ └── vtysh.conf
├── s1/
│ └── index.html
├── s2/
│ └── index.html
└── web/
├── Dockerfile
└── index.html
- True BGP-based Anycast simulation using FRR.
- Dynamic routing flip behavior based on path preference or node failure.
- Captures network-layer failover and BGP convergence metrics.
- Extensible to multiple POPs and variable routing attributes.
This approach provides a realistic foundation to study the causal mechanism of Anycast flips at the routing layer and complements the application-level observations of Approach 1.
| Aspect | Approach 1: Mahimahi + Proxy | Approach 2: Docker + FRR |
|---|---|---|
| Layer | Application / Transport | Network / Routing |
| Control | Deterministic latency and probability | Real BGP-driven routing |
| Dependencies | Mahimahi, Python, Nginx, Puppeteer | Docker, FRR, Docker Compose |
| Outputs | Browser-level metrics (FCP, load time) | Routing tables, failover time |
| Goal | Controlled flip emulation | Realistic routing validation |
Together, these approaches form a comprehensive framework for reproducing and analyzing Anycast flip phenomena across layers, connecting routing-level instability with end-user performance metrics.
- Mahimahi Integration: Provides latency-controlled POPs and flip proxy for reproducible connection behavior.
- BGP Testbed Integration: Uses containerized FRR routers and nodes to demonstrate actual routing flips.
- Protocol-Aware Testing: HTTP/1.1 and HTTP/2 comparison for flip sensitivity.
- Browser Automation: Puppeteer scripts for metric collection.
- Reproducibility: Deterministic seeds and static test pages.
- Flexibility: Adjustable flip probabilities, latency penalties, and routing attributes.
- Performance Metrics: FCP, Total Page Load Time, and Time to First Byte (TTFB).
- Network Metrics: RTT, TCP handshake time, retransmissions, and BGP convergence delay.
- Controlled Comparison: Baseline vs. flip scenarios for both HTTP versions.
- Statistical Analysis: Mean, deviation, and percentile distribution.
- Visualization: Graphical comparison of FCP variance under differing flip probabilities and routing configurations.
- Route Pinning: Preserve POP consistency across session lifetime.
- Adaptive Caching: Prefetch and retain frequently requested content to offset latency.
- Protocol-Aware Handling: Employ HTTP/2 or QUIC multiplexing and connection reuse to limit user impact.
Mitigation techniques are re-evaluated under identical conditions across both setups to quantify performance improvements and stability recovery.
Comparative data illustrating performance recovery and reduced FCP variance in optimized configurations.
We are actively refining the simulation of Anycast flips through both frameworks.
- The Mahimahi-based setup accurately emulates per-connection flips with configurable delay models.
- The Docker + FRR setup successfully demonstrates real BGP route flipping and failover.
Both environments are now integrated into the overall toolchain, enabling end-to-end correlation between routing flips and user-visible web performance.
The framework remains under continuous development to improve realism, expand metric coverage, and incorporate further optimization techniques.
The repository consists of two independent but complementary submodules:
FunnyCast/
├── README.md # Main documentation (this file)
├── anycast-flip-experiment/ # Approach 1: Mahimahi-based emulation
│ ├── README.md # Detailed setup, configuration, and usage
│ ├── flip_proxy.py
│ ├── test_proxy.py
│ └── analyze_webvitals.py
└── Anycast-CDN-Setup/ # Approach 2: Docker + FRR routing simulation
├── README.md # Setup and commands for network-level testbed
├── docker-compose.yml
└── frr/
Each subdirectory includes a dedicated README.md with environment setup, step-by-step execution commands, and log collection details.
The main README provides the conceptual and architectural overview connecting both approaches under the unified FunnyCast framework.