feat: text-to-music mode (generate from prompt alone, no input audio)

[leszko]

Summary

Adds a text-to-music mode: generate music in realtime from the text prompt alone, with no input audio — selectable at session start and switchable mid-session, in both the backend and the web app.

ACE-Step's trained "no reference audio" signal is the checkpoint's canonical silence latent (the model forward uses it to simulate text2music mode), so a text-only session is a normal streaming session whose source latent and structure context are the silence latent, diffusing from pure noise at denoise=1.0. No model or pipeline changes — only session construction and wiring.

Changes

Contract (registry-first):

• SessionConfig gains text2music: bool and text2music_duration_s: float = 60.0 — auto-projected into GET /api/protocol and the generated TS types.
• swap_source command gains a text2music field (no binary PCM frame, mirrors use_server_source).
• wireContract.gen.ts regenerated.

Engine (acestep/streaming):

• source.py: text2music_waveform() — silent placeholder that seeds the playback ring (the user hears generated slices stream in over silence) — and resolve_text2music_source() — canonical-silence PreparedSource via EmptyLatent plus fixed 120 BPM / C major / 4 conditioning defaults. Skips VAE encode, semantic extract, librosa beat-tracking (returns 0 BPM on silence, which would poison the text conditioning), CNN key detection, and stem extraction.
• session.py: create() and the swap path branch on the flag; swap_source() accepts text2music.

Transport + SDK:

• ws_adapter.py synthesizes the silent source server-side for both the init handshake and t2m swaps; no audio frame crosses the wire.
• web/sdk/protocol.ts: connect() skips the PCM frame when config.text2music is set; new sendSwapTextToMusic().

Web app:

• New client-side sentinel source (web/lib/text2music.ts) so every picker surface works unchanged: pinned TEXT TO MUSIC sleeve in the crate fan, pinned entry in the CORE-tab track picker, option in the lite select.
• useStartSession / useFixtureSwap translate the sentinel into the wire flag; text sessions bypass the hear-the-source-first denoise gate (the source is silence) and snap denoise to 1.0.

Testing

• tests/unit (149 passed; contract drift guards included), npm run typecheck, npm run build.
• Headless WS session against a live pod: silent initial buffer in → non-silent generated slices out (rms ~0.27), live prompt re-encode applied mid-stream.
• Browser: start session → swap to Text to music (denoise snaps to 1, music fills the silent buffer) → live prompt change via Tags panel → swap back to a fixture. Reconnect path re-derives the t2m config from the store sentinel.

🤖 Generated with Claude Code

[ryanontheinside]

Thanks for pushing this forward. The UI/protocol direction here is useful as a first sketch of a ?prompt-only source? mode, but I don?t think this should merge as ACE-Step text-to-music yet. The missing piece is not a small bug in the PR; it is the core ACE-Step text-to-music path.

Right now the PR mostly does three things:

• Adds a text2music flag to the startup and swap_source wire protocol.
• Skips sending PCM from the browser when that flag is set.
• Synthesizes a silent source on the backend and creates an EmptyLatent for the streaming session.

That is a reasonable placeholder for ?no uploaded source audio?, but it is not how ACE-Step?s text-to-music generation works.

In this branch, resolve_text2music_source() returns an EmptyLatent as both source.latent and context_latent, and the rest of the session continues through the existing streaming path. Separately, encode_cond_pair() still hardcodes TASK_INSTRUCTIONS["cover"] and [Instrumental]. So the model is not receiving LM-generated semantic/audio codes. It is essentially the existing cover/remix engine running against a silent placeholder.

The important distinction is that ?no reference audio? is not the same thing as ?text-to-music semantic plan?. ACE-Step text-to-music involves a planning stage that generates 5 Hz semantic/audio codes from the prompt, lyrics, and metadata. Those codes then feed the diffusion model. This PR skips that stage entirely.

Relevant ACE-Step references:

• The official inference quick start initializes both AceStepHandler and LLMHandler, then calls generate_music(dit_handler, llm_handler, params, config, ...).
  https://github.com/ace-step/ACE-Step-1.5/blob/main/docs/en/INFERENCE.md#quick-start

• The official GenerationParams include audio_codes and LM-specific parameters such as thinking, lm_temperature, lm_cfg_scale, use_cot_metas, etc.
  https://github.com/ace-step/ACE-Step-1.5/blob/main/docs/en/INFERENCE.md#generationparams-parameters

• The official LM usage notes say text2music, lego, and complete use the LM when enabled, while cover, repaint, and extract skip it.
  https://github.com/ace-step/ACE-Step-1.5/blob/main/docs/en/INFERENCE.md#4-lm-usage

• The ACE-Step tutorial describes the DiT as receiving semantic codes and conditions generated by the LM, and says ?No LM? means the user does the planning.
  https://github.com/ace-step/ACE-Step-1.5/blob/main/docs/en/Tutorial.md#choosing-the-planner-lm-models

• The ACE-Step 1.5 paper frames the LM as the planner and describes Text-to-Code / Code-to-Text over generated audio codes.
  https://arxiv.org/html/2602.00744v1

What I think a real implementation needs:

1. Backend LM integration

We need a real ACE-Step LM runtime in the backend, likely via LLMHandler or a local wrapper around the same components. The streaming session currently owns the DiT/VAE/text encoder path only. Real text-to-music needs to load/select the 5 Hz LM model, manage its memory, and expose its capability in the server contract.

2. A planning phase before streaming generation

For text-to-music, the prompt/lyrics/BPM/key/time signature/duration should first produce audio semantic codes. That may take a while and should be represented as its own state, not hidden inside swap_source. The UI should expect ?planning/generating semantic codes? before the live stream can meaningfully start.

3. Feeding generated codes into the DiT path

The ACE-Step model code already has hooks for this: audio_codes and precomputed_lm_hints_25Hz flow into condition preparation. A correct implementation should feed the LM output there, or convert/cache it into the equivalent 25 Hz hints, rather than cloning an EmptyLatent into context_latent.

4. Task-specific conditioning

The text-to-music path should not reuse the cover instruction by accident. TASK_INSTRUCTIONS["text2music"] may be part of the final path, but changing the instruction alone is not sufficient. The key missing work is still the semantic-code generation and wiring.

5. Runtime lifecycle and UX

Because code generation can be slow, the backend should publish explicit progress/status/failure events. We probably need cancellation, timeout handling, and a clear transition from ?planning? to ?streaming?. This should not feel like an instant fixture swap, because it is not doing the same amount of work.

6. Capability/version guarding

The client currently skips the binary PCM frame when config.text2music is true. That is a new handshake shape. Server-side fixtures already have a probe because mixed frontend/backend deploys can otherwise hang. Text-to-music needs the same kind of capability/version guard; otherwise an older pod will ignore text2music and block waiting for the audio frame.

7. Clear product semantics

If we keep a placeholder mode around, it should be named as such. Calling this ?text-to-music? will mislead future work because the hard part is not present yet. The reusable pieces here are probably the picker affordance, the no-PCM handshake idea, and some UI state transitions, but the engine path needs a larger design pass.

So my recommendation is: do not merge this as text-to-music. Either re-scope it as UI/protocol scaffolding behind a disabled/experimental flag, or replace the backend path with a real LM/audio-code implementation. The next PR should start from the ACE-Step LM planning flow and then decide how that output fits into DEMON?s streaming engine.

• Codex

[ryanontheinside]

please see main comment