Rescored

Convert YouTube videos into editable sheet music using AI.

Deployment

• Backend: https://calebhan-rescored.hf.space
• Frontend: https://rescored.vercel.app

For best results, use the local deployment.

How it works

1. Submit a YouTube URL
2. Audio is extracted, separated by instrument, and transcribed to MIDI
3. Edit the notation in the browser
4. Export as MIDI

Tech stack

• Backend: Python/FastAPI, Celery, Redis
• Frontend: React, VexFlow (notation), Tone.js (playback), Zustand
• ML: BS-RoFormer + Demucs (source separation), YourMT3+ + ByteDance ensemble + BiLSTM refinement (transcription)

Piano transcription achieves 96.1% F1 on the MAESTRO test set.

Setup

See CONTRIBUTING.md for local development setup.

Roadmap

• Phase 1: Piano transcription, source separation, interactive editor, MIDI export, BiLSTM refinement
• Phase 2: Multi-instrument transcription, grand staff, advanced editing, PDF export
• Phase 3: User accounts, cloud storage, collaboration

License

MIT License. See LICENSE for details.

Acknowledgments

• YourMT3+ (KAIST): Paper
• ByteDance Piano Transcription: GitHub
• BS-RoFormer: GitHub
• Demucs (Meta AI Research): Paper
• audio-separator, librosa, madmom, pretty_midi, VexFlow, Tone.js

---

This is an educational project. Users are responsible for copyright compliance when transcribing YouTube content.