Skip to content

Add CustomVoice, VoiceDesign, native clone, and fixed-mode HTTP server support#5

Open
AntNoHuabei wants to merge 9 commits into
yet-another-ai:mainfrom
AntNoHuabei:main
Open

Add CustomVoice, VoiceDesign, native clone, and fixed-mode HTTP server support#5
AntNoHuabei wants to merge 9 commits into
yet-another-ai:mainfrom
AntNoHuabei:main

Conversation

@AntNoHuabei

Copy link
Copy Markdown

Summary

This PR expands Qwen3-TTS runtime coverage and packaging around three practical synthesis modes:

  • CustomVoice: built-in speaker IDs / speaker names with optional style instruction
  • VoiceDesign: instruction-driven voice design models
  • VoiceClone: Python-free WAV / prompt based voice cloning

It also adds a fixed-mode HTTP server binary (qts_server) with OpenAI-compatible speech output and async job progress APIs.

Major changes

Voice modes

  • Add CustomVoice model metadata loading and speaker-token conditioning.
  • Add VoiceDesign synthesis APIs and CLI support.
  • Add native WAV voice-clone prompt construction without requiring Python at runtime.
  • Support both x-vector-only clone and ICL clone with reference text/audio codes.
  • Add support for reusable protobuf voice-clone prompt files.

Runtime and model compatibility

  • Add VoiceDesign model kind detection.
  • Support VoiceDesign GGUF variants that use code_pred.hidden_proj.* naming.
  • Add Q8_0 tensor dequantization for F32 loading paths used by VoiceDesign/custom projection paths.
  • Add vocoder artifact fallback discovery for sibling model directories.
  • Add native soxr-based resampling and audio-code extraction path.

Performance and Vulkan stability

  • Improve CustomVoice Vulkan talker stability.
  • Optimize CustomVoice code predictor path.
  • Add optional preprojected code predictor embeddings.
  • Fuse CustomVoice code predictor projections where possible.
  • Keep CPU fallback behavior for paths that are not safe or profitable on Vulkan.

Server mode

  • Add standalone qts_server.exe binary.
  • Server mode is fixed at startup: none, custom, design, or clone.
  • Add OpenAI-compatible synchronous endpoint:
    • POST /v1/audio/speech
  • Add QTS async job APIs:
    • POST /v1/qts/audio/jobs
    • GET /v1/qts/audio/jobs/{job_id}
    • GET /v1/qts/audio/jobs/{job_id}/audio
  • Add synthesis progress reporting with rollout frame progress.
  • Serialize requests through a single worker so the loaded ggml/Vulkan engine is not concurrently accessed.

CLI / server examples

qts_server --mode design \
  --model-dir models/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
  --backend vulkan \
  --vocoder-ep cpu \
  --language-id 2055
qts_server --mode custom \
  --speaker serena \
  --model-dir models/Qwen3-TTS-12Hz-1.7B-CustomVoice
qts_server --mode clone \
  --voice-clone-wav ref.wav \
  --voice-clone-ref-text "reference text" \
  --model-dir models/Qwen3-TTS-12Hz-0.6B-Base

Validation

Tested locally on Windows with Vulkan:

  • CustomVoice synthesis
  • VoiceDesign synthesis
  • WAV clone / ICL clone path
  • qts_server fixed design mode
  • /health
  • /v1/audio/speech
  • async job creation, progress polling, and WAV download
  • fixed-mode rejection when a request tries to switch mode

Build/check commands used:

cargo check -p qts_cli --no-default-features
cargo build --release -p qts_cli --bin qts_server --no-default-features --features vulkan

Notes

  • qts_server intentionally fixes conditioning mode at startup to keep server behavior predictable for wrappers and API gateways.
  • The OpenAI-compatible endpoint remains synchronous; progress is exposed through the QTS async job APIs.
  • Runtime packaging for Windows server currently needs only:
    • qts_server.exe
    • onnxruntime.dll
    • soxr.dll

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant