Lagged correlations reveal when the past of one asset influences another's future; at the high frequency level, order-book data are asynchronous, making standard discrete-lag techniques unsuitable. Here, we study limit order books of multiple cryptocurrencies to uncover cross-crypto lead-lag signals. The task is to design and backtest a strategy that uses order-book features (e.g., bid-ask imbalance, depth shifts) of one coin to predict short-term returns on another, assessing out-of-sample performance (Sharpe, hit rate) and net profitability after transaction costs.
This project is organized into several modules, each with its own documentation. Below is a summary and quick links to each module's documentation:
- Backtesting (backtesting/README.md)
- Event-driven engine for simulating trading strategies on historical order-book data. Includes portfolio management, transaction cost modeling, and extensibility for custom strategies.
- Feature Extraction (feature_extraction/README.md)
- Framework for generating financial features from preprocessed order book data. Supports a wide range of features and easy extension.
- Preprocessing (preprocessing/README.md)
- Tools for converting raw crypto CSV data into structured formats for analysis and ML. Includes both importable modules and CLI scripts.
- See also preprocessing/IMPLEMENTATION_SUMMARY.md for implementation details.
- Prediction Model (prediction_model/README.md)
- Core components for training, evaluating, and managing ML models for financial prediction. Includes data preprocessing, model management, and example workflows.
- Scripts (scripts/README.md)
- Utility scripts for data processing, feature engineering, model management, and automated backtesting. Includes batch processing and feature generation tools.
- Strategies (strategies/README.md)
- Collection of trading strategies for backtesting and evaluation. Each strategy is documented with its logic and requirements.
- Performance Analysis (perf_analysis/README_random_baseline.md)
- Mathematical background and implementation details for random baseline recall confidence intervals.
- External Resources (found_documentation.md)
- Curated links to related projects and documentation for backtesting, lead-lag detection, and crypto trading bots.
- Project Overview
- Installation
- Quick Start
- Project Structure
- Strategy Development
- Contributing
- License
- Contact
A modular framework for analyzing information flow in crypto order-books, featuring event-driven backtesting, advanced data preprocessing, and support for custom trading strategies.
- Clone the repository:
git clone <repo-url> cd ifcob
- Install dependencies:
pip install -r requirements.txt
Use the automated backtesting script to quickly test strategies:
# Run a single strategy with data index 1
python scripts/run_backtest.py --strategy TFCumulativeReturnStrategy --data-index 1
# Run multiple strategies with different data versions
python scripts/run_backtest.py --strategy Mateo2StartStrategy --strategy RFPredAllSignedStratMateoCheating --data-index 2
# Save detailed results
python scripts/run_backtest.py --strategy MyStrategy --data-index 1 --output-dir results --save-trades --save-portfoliopython scripts/run_backtest.py --list-strategiesThe backtesting module evaluates trading strategies using historical order-book data.
Key Features:
- Event-driven simulation engine
- Support for custom strategies and signals
- Realistic transaction cost modeling
- Performance metrics: Sharpe ratio, hit rate, drawdown, and net P&L
- Modular design for easy extension
Quick Start:
from backtesting import Backtester, Strategy
# Define your strategy
class MyStrategy(Strategy):
def generate_signals(self, data):
# Implement signal logic
pass
# Run backtest
bt = Backtester(data, MyStrategy, transaction_costs)
results = bt.run()
print(results.summary())See backtesting/README.md for detailed documentation.
The preprocessing module provides tools for converting raw cryptocurrency CSV data into a structured format suitable for feature extraction and analysis.
Key Features:
- Vectorized preprocessing with configurable block sizes
- Automatic duplicate removal
- Conversion to wide format for analysis
- Support for both single file and batch processing
- Importable module with clean API
Quick Start:
from preprocessing import preprocess_crypto_data, preprocess_data_folder
# Process single file
df = preprocess_crypto_data('data/raw/DATA_0/XBT_EUR.csv', 'data/preprocessed/XBT_EUR.parquet', 'XBT')
# Process entire folder
results = preprocess_data_folder('data/raw/DATA_0', 'data/preprocessed/DATA_0')Command Line Usage:
# Process all files in a folder
python preprocessing/preprocess_script.py -i data/raw/DATA_0 -o data/preprocessed/DATA_0
# Process specific coins
python preprocessing/preprocess_script.py -i data/raw/DATA_0 -o data/preprocessed/DATA_0 --coins XBT ETHSee preprocessing/README.md for detailed documentation.
The scripts/run_backtest.py provides a comprehensive command-line interface for running backtests with automatic strategy discovery and data versioning.
Key Features:
- Automatic Strategy Discovery: Dynamically finds all strategies in the
strategies/folder - Data Index Management: Mandatory
--data-indexparameter ensures explicit data versioning - Automatic Feature Generation: Checks for and generates precomputed features when needed
- Precomputed Features Support: Automatic detection and handling of strategies that load precomputed features
- Flexible Configuration: Support for custom data sources, fees, and strategy parameters
- Comprehensive Results: Detailed output with performance metrics, trade logs, and portfolio evolution
- Profiling Support: Built-in performance profiling for optimization
The project uses a data index system to manage different versions of preprocessed data:
data/features/
├── DATA_1/ # First dataset version
│ ├── XBT_EUR.parquet
│ └── ETH_EUR.parquet
├── DATA_2/ # Second dataset version
│ ├── XBT_EUR.parquet
│ └── ETH_EUR.parquet
└── ...
Strategy Compatibility:
- ✅ Supports
--data-index: Strategies withdata_indexparameter automatically use the specified dataset ⚠️ Hardcoded Paths: Strategies withoutdata_indexparameter may use hardcoded paths and need manual updates- ℹ️ No Precomputed Features: Strategies that don't load external data work with any dataset
The backtesting script automatically handles precomputed features:
- Detection: Analyzes strategy code to detect if precomputed features are needed
- Verification: Checks if feature files exist for the specified data index
- Generation: Automatically generates missing features using
scripts/generate_features.py - Interactive Mode: Prompts user before generating features (unless
--auto-generate-featuresis used) - Error Handling: Provides clear error messages and manual generation commands if automatic generation fails
Feature Generation Options:
--auto-generate-features: Generate features automatically without prompting (non-interactive mode)--force-feature-check: Check for features even for strategies that may not need them--ignore-feature-errors: Continue with backtesting even if feature generation fails
Basic Usage:
# Required: Always specify data index
python scripts/run_backtest.py --strategy MyStrategy --data-index 1
# Multiple strategies
python scripts/run_backtest.py --strategy Strategy1 --strategy Strategy2 --data-index 2Automatic Feature Generation:
# Non-interactive mode (auto-generates features without prompting)
python scripts/run_backtest.py --strategy Mateo2StartStrategy --data-index 2 --auto-generate-features
# Force feature check for all strategies
python scripts/run_backtest.py --strategy SimpleStrategy --data-index 1 --force-feature-check
# Continue even if feature generation fails
python scripts/run_backtest.py --strategy MyStrategy --data-index 1 --ignore-feature-errorsAdvanced Configuration:
# Custom parameters
python scripts/run_backtest.py \
--strategy RFPredAllSignedStratMateo \
--data-index 1 \
--window-size 10 \
--threshold 0.05 \
--initial-capital 500000 \
--fee-percentage 0.15
# Custom data sources (override defaults)
python scripts/run_backtest.py \
--strategy MyStrategy \
--data-index 1 \
--data-sources XBT:data/custom/XBT.parquet ETH:data/custom/ETH.parquet
# Save detailed results
python scripts/run_backtest.py \
--strategy MyStrategy \
--data-index 1 \
--output-dir backtest_results \
--save-trades \
--save-portfolioPerformance Analysis:
# Profile execution
python scripts/run_backtest.py --strategy MyStrategy --data-index 1 --profile
# Custom split ratio for calibration/validation
python scripts/run_backtest.py --strategy MyStrategy --data-index 1 --split-ratio 0.8When running a strategy that requires precomputed features:
$ python scripts/run_backtest.py --strategy Mateo2StartStrategy --data-index 2
============================================================
BACKTESTING ENGINE
============================================================
Using default data sources for DATA_2: [('XBT', 'data/features/DATA_2/XBT_EUR.parquet'), ('ETH', 'data/features/DATA_2/ETH_EUR.parquet')]
Data index: 2
Loaded strategy: Mateo2StartStrategy
→ Uses precomputed features with data_index=2
============================================================
CHECKING PRECOMPUTED FEATURES
============================================================
Strategy 'Mateo2StartStrategy' requires precomputed features
⚠ Precomputed features for DATA_2 not found
Generate precomputed features for DATA_2? [Y/n]: y
Generating precomputed features for DATA_2...
Generating features for XBT...
Generating features for ETH...
✓ Successfully generated precomputed features for DATA_2
Trading fees: 0.1%
Dataloader initialized successfully
...When using --output-dir, the script generates:
backtest_summary_TIMESTAMP.csv: High-level performance metricsbacktest_results_TIMESTAMP.json: Detailed results in JSON formattrades_details_STRATEGY_PHASE_TIMESTAMP.csv: Individual trade records (with--save-trades)portfolio_evolution_STRATEGY_PHASE_TIMESTAMP.csv: Portfolio value over time (with--save-portfolio)
To make your strategy compatible with the data index system, add a data_index parameter:
from backtesting.strategy import Strategy
import pandas as pd
import joblib
class MyStrategy(Strategy):
def __init__(self, data_index=1, window_size=5):
super().__init__()
# Use data_index for loading precomputed features
self.features_df = pd.read_parquet(f"data/features/DATA_{data_index}/XBT_EUR.parquet")
self.model = joblib.load(f"predictors/my_model_{window_size}ms.joblib")
def get_action(self, data, current_portfolio, fees_graph):
# Your strategy logic here
passThe automated script discovers strategies by:
- Scanning all
.pyfiles in thestrategies/folder - Finding classes that inherit from
Strategy - Checking for
data_indexparameter support - Detecting precomputed feature usage patterns
- Always use
data_indexparameter for loading precomputed features - Test with multiple data versions to ensure robustness
- Use descriptive strategy names and documentation
- Handle missing features gracefully in your strategy logic
- Profile your strategies to optimize execution time
- Use
--auto-generate-featuresfor automated CI/CD pipelines
When developing strategies that use precomputed features:
- Dependency Check: The backtesting script automatically detects if your strategy loads precomputed features
- Automatic Generation: Missing features are generated automatically using the
generate_features.pyscript - Data Versioning: Features are generated for the specific
data_indexyou specify - Error Recovery: If automatic generation fails, manual commands are provided
# Example strategy with automatic feature dependency detection
class MyMLStrategy(Strategy):
def __init__(self, data_index=1):
super().__init__()
# This will be automatically detected as requiring precomputed features
self.model = joblib.load("predictors/my_model.joblib")
self.features = pd.read_parquet(f"data/features/DATA_{data_index}/XBT_EUR.parquet")Contributions are welcome! Please open issues or submit pull requests for improvements or bug fixes.
This project is licensed under the terms of the LICENSE file in this repository.
For questions or collaboration, please contact the project maintainer.