Music Score Analyser

A music score viewer with real-time playback and an AI chat assistant that can reason about and highlight notes.

Setup

Prerequisites: Node.js 18+, an OpenAI API key.

# 1. Install dependencies
npm install

# 2. Configure environment
cp .env.example .env
# Edit .env and set OPENAI_API_KEY

.env variables:

Variable	Description
VITE_API_URL	Backend base URL (default: http://localhost:8000)
OPENAI_API_KEY	Required for the chat and evals
LANGCHAIN_API_KEY	Optional — enables LangSmith tracing for evals

Adding score files: Drop .xml or .mxl MusicXML files into public/. The server auto-discovers them and serves them via the score select dropdown.

Running

npm run dev      # Vite frontend (http://localhost:5173)
npm run server   # Express backend (http://localhost:8000)

Both must be running simultaneously for the full app to work.

Testing

npm test         # Vitest unit tests (domain layer)
npm run cy:open  # Cypress E2E — interactive
npm run cy:run   # Cypress E2E — headless CI
npm run evals    # LLM evals against dataset (requires OPENAI_API_KEY)

Architecture

flowchart TD
    subgraph Providers["React Context Providers"]
        PE[PlayEngineProvider]
        SR[ScoreRendererProvider]
        NS[NotesServiceProvider]
        TK[TickerProvider]
        CS[ChatSidebarProvider]
    end

    subgraph Domain["Domain Layer"]
        PlayEngine[PlayEngine]
        ScoreRenderer[ScoreRenderer]
        NotesService[NotesService]
        Ticker[Ticker]
    end

    subgraph Orchestration["Orchestration"]
        PO[usePlayOrchestrator]
    end

    subgraph Chat["Chat Feature"]
        Agent[server/agent.ts]
        UseChat[useChat]
        UseHistory[useHistory]
    end

    subgraph Evals["Evals"]
        EvalDataset[evals/dataset.ts]
        EvalRunner[evals/run.ts]
    end

    TK -->|onPlayStateChange| PlayEngine
    PO -->|registerParts / scheduleFinish| PlayEngine
    PO -->|advanceCursor / highlightNotes| ScoreRenderer
    PO -->|getNoteIdsWithinRange| NotesService
    PO -->|registerTickCallback| Ticker
    ScoreRenderer -->|noteMap| NotesService
    UseChat -->|notesToJSON| NotesService
    UseChat -->|POST /chat| Agent
    UseHistory -->|GET /history| Agent
    EvalRunner -->|runChatAgent| Agent
    EvalDataset --> EvalRunner

Loading

Component	Responsibility
PlayEngine	Tone.js Transport, Sampler/Synth Parts, PlayState pub/sub
ScoreRenderer	OSMD rendering, note extraction, cursor advancement, highlightNotes()
NotesService	B-tree note timeline, getNoteIdsWithinRange(), notesToJSON()
Ticker	requestAnimationFrame loop, registerTickCallback(), self-managing start/stop
usePlayOrchestrator	Mediates PlayEngine + ScoreRenderer; bridges Tone.js audio timing with rAF visual updates
useChat	experimental_useObject streaming, integrates history
useHistory	React Query seed from /history, local addToHistory()
server/agent.ts	SYSTEM_PROMPT, buildMessages(), traceable runChatAgent()

Key Design Decisions

Single Ticker, pub/sub play state

Ticker is a singleton in context that runs requestAnimationFrame. It self-starts and self-stops by subscribing to PlayEngine.onPlayStateChange — no external management needed. All components that need tick events call ticker.registerTickCallback().

B-tree in NotesService

Notes are indexed in a B-tree keyed by tick position. getNoteIdsWithinRange(startTicks, endTicks) efficiently finds all notes that should have been highlighted since the last tick, ensuring no notes are missed even if a tick fires late.

PlayOrchestrator as mediator

usePlayOrchestrator is the only place that knows about both PlayEngine and ScoreRenderer. It registers Tone.js Part objects for accurate audio scheduling and uses the Ticker callback for visual updates (cursor + note highlighting). These are intentionally separate concerns: audio timing is handled by Tone.js internals, visual timing by rAF.

Chat citations

The AI response schema includes a highlights array. Each highlight renders as a chip that calls scoreRenderer.highlightNotes(). If the highlighted notes are behind the current cursor position, the cursor resets and re-advances to the target tick.

Agent isolation for evals

server/agent.ts exports the SYSTEM_PROMPT and a traceable runChatAgent function that uses generateObject (non-streaming). The HTTP route uses streamObject with the same prompt via buildMessages(). This lets evals call the agent directly without an HTTP server.

Project Structure

src/
  components/
    ui/          # Presentational components (no context hooks)
    features/    # Components connected to global context
  domain/        # Framework-free business logic + unit tests
  hooks/         # React context providers and custom hooks
  models/        # Plain TypeScript types (Note, PlayState, Instrument)
  pages/         # Route-level components
  types/         # Global type declarations (window augmentation)
server/
  index.ts       # Express routes (/score-files, /chat, /history)
  agent.ts       # LLM agent (traceable, reusable for evals)
evals/
  dataset.ts     # Eval cases (AgentInput + expected output)
  run.ts         # Eval runner (substring match scoring)
cypress/
  e2e/           # End-to-end tests (score-page/)
  support/       # Custom commands, consts, type declarations
  fixtures/      # MusicXML fixture files for tests
public/          # MusicXML score files served by the backend