Know what's wrong with your robot data before training starts.
Open-source dataset observability for robot learning — integrity, quality, coverage, and coreset optimization in one pipeline.
$ calibra integrity /data/my_demos.h5
─── Dataset Integrity ────────────────────────────────────
my_demos · 120 episodes
Critical (1)
❌ camera_freeze_events: 1 of 120 episodes (0.8%) contain a run of ≥5
consecutive near-identical camera frames (episode ep_17).
Warnings (1)
⚠️ blurry_episode_fraction: camera frames markedly blurrier than the
rest of the dataset in 1 episode.
Passed (8)
✅ timestamp_jitter_cv ✅ timestamp_dropout_rate ✅ short_episode_fraction
✅ action_dropout_rate ✅ duplicate_frame_rate ✅ ldlj
✅ jerk_spike_rate ✅ velocity_discontinuity_rate
Integrity Score: 85/100 · Status: Warning
- ✅ Integrity first — catch broken timestamps, duplicate/frozen/blurry camera frames, jittery/jerky motion, and incomplete episodes before they reach training, with
calibra integrity(LeRobot v1, HDF5/Isaac Lab, robomimic) - 🔍 Audit robot datasets for quality, synchrony, coverage, and task structure
- ✂️ Reduce training data by up to 75% with quality-aware coreset selection
- 📊 Audit 30+ public LeRobot datasets with an interactive Hugging Face Space
- 🤖 Supports LeRobot, Isaac Lab, RLDS, HDF5, MCAP, and Hugging Face Hub
- 📦 Install with
pip install calibra-robotics
Calibra answers the questions practitioners ask about a new dataset, in the order they ask them:
| Step | Question | Command |
|---|---|---|
| 1. Integrity | Can I trust this dataset? | calibra integrity |
| 2. Quality | Is it clean? | calibra audit / calibra score |
| 3. Coverage | Is it diverse enough? | calibra review |
| 4. Optimize | Can I train faster/cheaper? | calibra prune |
pip install calibra-robotics
calibra integrity /data/my_demos.h5
calibra audit lerobot/pushtNo installation required.
🔗 Calibra — Dataset Integrity (Hugging Face Space)
- Check any LeRobot dataset's Integrity first — timestamps, sync, completeness, duplicate/frozen/blurry frames, jittery motion
- Then see its Quality & Coverage score and percentile
- Compare against community benchmarks
- Download a full audit report
Most robot learning labs collect thousands of demonstrations and train on all of them — silently ingesting bad data, wasting compute on near-duplicates, and producing undiagnosable policy failures.
| Problem | What Calibra does |
|---|---|
| Jerk spikes, dropped frames, sync errors | Audit — detect and flag bad episodes before training |
| 60–80% of episodes are near-duplicates | Select — maximize behavioral coverage in a smaller coreset |
| Policy failure cause is unknown | Score — decompose quality into measurable, falsifiable metrics |
Random selection picks a clustered subset. Calibra's coverage-based selector spreads selections across the behavioral space — ensuring the policy sees every behavioral mode, even rare ones.
On real PushT data: at 10% retention, Calibra achieves lower prediction error than training on the full dataset, while random selection degrades sharply. The shaded region is Calibra's advantage over random at every budget.
Ablation across 5 seeds on ALOHA mobile (keep 30%): Calibra full pipeline and diversity-only both outperform all published baselines. Herding is the only method that reliably underperforms random.
Mean improvement over random selection (5 seeds, 30% retention, 3 datasets):
| Method | BC-MLP | ACT | Diffusion Policy |
|---|---|---|---|
| Diversity-only | +29.5% | +26.5% | +11.9% |
| Calibra full | +24.5% | +23.7% | +13.8% |
| K-Center | +24.0% | +23.1% | +10.1% |
| Facility Location | +21.5% | +18.4% | +8.7% |
| Random | 0.0% | 0.0% | 0.0% |
Method rankings are stable across all three policy families (Spearman ρ ≥ 0.86). → Full benchmarks and ablations
Inspect dataset health, identify problematic demonstrations with root causes, and generate a training-ready coreset — all from one interface. Generated with calibra audit lerobot/columbia_cairlab_pusht_real --html-out report.html.
| Command | Description |
|---|---|
calibra integrity |
"Can I trust this dataset?" — timestamps, sync, episode completeness, duplicate/frozen/blurry camera frames, jittery/jerky motion (--decode-images for LeRobot v1) |
calibra audit |
Full diagnostic report with bootstrap CIs and per-episode outlier detection |
calibra review |
Ranked episode review queue — separates anomaly, quality-risk, and coverage-value signals |
calibra prune |
Two-stage coreset: quality filter + greedy max-coverage selection |
calibra certify |
Structured CERTIFIED / PROVISIONAL / NOT CERTIFIED; --json for CI |
calibra predict |
Estimate training outcome before spending GPU time |
calibra watch |
Real-time quality feedback during teleoperation |
calibra score |
Composite 0–100 score across Quality, Synchrony, Coverage, Task Structure |
calibra compare |
Evidence-backed cross-dataset comparison with falsifiable claims |
calibra corrupt |
Inject synthetic corruptions to validate metric sensitivity |
calibra card |
Generate a HuggingFace dataset quality card |
calibra sim2real |
Quantify sim-to-real distribution gap and transfer risk |
calibra transfer |
Cross-embodiment compatibility scoring |
calibra cure |
Automatic data remediation (smoothing, resampling, trimming) |
calibra audit-all |
Bulk-audit an entire HF org; writes CalibraReport JSONs |
calibra site |
Generate a static leaderboard website from audit results |
calibra serve |
Local REST API server and web dashboard |
v0.7.1 (current) — Dataset Integrity: calibra integrity — timestamps, sync, episode completeness, duplicate/frozen/blurry camera frames, jittery/jerky motion, with LeRobot v1 image support via --decode-images. Full details in CHANGELOG.md.
Next — Vision Integrity for video-backed LeRobot (v2/v3): decode sampled frames from LeRobot's mp4-encoded v2/v3 datasets so duplicate-frame/camera-freeze/blur detection work there too.
# 1. Record demos
lerobot-record --robot-type so100 --repo-id $HF_USER/my_dataset
# 2. Curate and write the report
calibra prune /path/to/my_dataset --keep 0.3 --report results/my_dataset/latest.json
# 3. Train on the coreset
lerobot-train policy=act dataset_repo_id=./my_dataset_coresetfrom calibra.integrations.lerobot import load_dataset
ds = load_dataset("lerobot/pusht", report_path="results/pusht/latest.json")
# ds is a datasets.Dataset with only Calibra-approved episodesfrom calibra.integrations.isaac_lab import export_gr00t_manifest, filter_hdf5
export_gr00t_manifest("results/franka/latest.json", demos_path="demos.hdf5")
filter_hdf5("demos.hdf5", "results/franka/latest.json", "demos_coreset.hdf5")calibra prune demos.hdf5 --keep 0.3 --policy gr00t --report results/franka/latest.json
python -m gr00t.train --manifest gr00t_manifest.json --demo-file demos_coreset.hdf5from calibra.ingestion.registry import load
from calibra.pipeline import Pipeline
from calibra.pruning import CoresetSelector
batch = load("lerobot/pusht")
report = Pipeline().run(batch, policy_family="diffusion")
selector = CoresetSelector(keep_fraction=0.3)
result = selector.select(batch, report)
# result.keep_episode_ids → filter your datasetPyPI package name:
calibra-robotics(thecalibraname on PyPI is an unrelated package)
pip install calibra-robotics # core (numpy + pydantic only)
pip install 'calibra-robotics[lerobot]' # LeRobot / HuggingFace Hub (recommended)
pip install 'calibra-robotics[hdf5]' # HDF5 (Isaac Lab, Robomimic)
pip install 'calibra-robotics[rlds]' # RLDS / TF Datasets
pip install 'calibra-robotics[mcap]' # MCAP / ROS2 bags
pip install 'calibra-robotics[all]' # everythingFormats supported: LeRobot v1/v2/v3 (Parquet), HuggingFace Hub IDs, HDF5 (Isaac Lab, Robomimic), RLDS/TF Datasets, MCAP/ROS2 bags.
Camera-frame checks (duplicate_frame_rate, camera_freeze_events, blurry_episode_fraction in calibra integrity) work out of the box on HDF5/Isaac Lab/robomimic data, and on LeRobot v1 datasets via calibra integrity <path> --decode-images (opt-in — decodes HuggingFace Image-feature columns, off by default since it increases load time/memory). Not yet supported for LeRobot v2/v3 (video-encoded).
Coming soon. The central empirical finding — that the optimal coreset selection strategy depends on the data-retention budget — will be described in full detail.
Available after paper release.
Calibra is not open to external pull requests or contributions at this time.
git clone https://github.com/omertt27/Calibra
pip install -e '.[all,dev]'
pytest # 679 tests
ruff check . # zero errors expectedBusiness Source License 1.1 — free for research and internal use, converts to Apache 2.0 on 2030-06-30. Commercial hosting requires a separate license. See LICENSE and LICENSING.md. Contact: omertahtaci05@gmail.com