An end-to-end A/B testing case study on a simulated fintech payment checkout, demonstrating experiment design, statistical analysis, and business decision-making when conversion gains come with increased fraud risk.
- The simplified checkout flow lifts conversion by 1.6 percentage points (p = 0.005), but fraud among completed transactions doubles from ~2% to ~3.9% (p < 0.001).
- At realistic fraud cost assumptions (2.5x chargeback multiplier), the increased fraud wipes out the revenue gain from higher conversion. The experiment is a net negative.
- Fraud risk is concentrated on desktop transactions, suggesting a mobile-only rollout as a viable alternative.
- Experiment design: hypothesis definition, metric hierarchy (primary, guardrail, secondary), power analysis (analytical and simulation-based), sample size calculation.
- Data simulation: realistic user-level data with segment-specific conversion rates, log-normal transaction values, and differential fraud rates across treatment groups.
- Statistical analysis: two-proportion z-tests, confidence intervals, randomisation checks.
- Guardrail analysis: fraud rate testing, cost-benefit analysis with sensitivity to fraud cost multipliers.
- Segment analysis: heterogeneous treatment effects by device type and country.
- Common pitfalls: simulation showing inflated false positive rates from daily peeking, and how ignoring guardrail metrics leads to wrong decisions.
The scenario is informed by experience working with payment fraud and transaction data at fintech companies. The simulation models a European payments company operating across Germany, France, the Netherlands, and Spain. All data is simulated; no proprietary information is used.
Interactive Tableau dashboard showing conversion lift, fraud increase by segment, and the revenue vs fraud cost tradeoff.
Python, NumPy, Pandas, SciPy, statsmodels, Matplotlib, Seaborn, Tableau
pip install numpy pandas scipy statsmodels matplotlib seaborn
jupyter lab ab-test-checkout-fraud.ipynb