Skip to content

Document HostedVoiceCloneConfig for pipecat and livekit plugins - #26

Open
serhatzx wants to merge 2 commits into
mainfrom
serhat/sdk-29-update-sdk-docs-to-include-voice-clone-tts
Open

Document HostedVoiceCloneConfig for pipecat and livekit plugins#26
serhatzx wants to merge 2 commits into
mainfrom
serhat/sdk-29-update-sdk-docs-to-include-voice-clone-tts

Conversation

@serhatzx

@serhatzx serhatzx commented Jun 4, 2026

Copy link
Copy Markdown
Member

Summary by CodeRabbit

  • Documentation
    • Updated the RealtimeModel configuration reference to document expanded tts_config support, including hosted voice cloning via HostedVoiceCloneConfig.
    • Added a new documentation section covering required audio sample metadata (PCM bytes, sample rate, channels, reference transcript, synthesis mode) and usage guidance.
    • Updated WebSocket protocol documentation for hosted voice cloning, including revised recommended clean speech duration for audio sample data.
    • Added a “Voice Cloning” feature card highlighting hosted pre-cloned voices and on-the-fly cloning from raw audio samples.

@serhatzx
serhatzx requested a review from Jannek312 June 4, 2026 11:22
@coderabbitai

coderabbitai Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This PR adds documentation for the new HostedVoiceCloneConfig feature across LiveKit and Pipecat integration guides, updates the WebSocket API reference specifications with revised audio duration requirements, and announces the voice cloning capability in feature cards.

Changes

Voice Cloning Configuration Documentation

Layer / File(s) Summary
HostedVoiceCloneConfig in integration guides
livekit.mdx, pipecat.mdx
The RealtimeModel tts_config type expands to include HostedVoiceCloneConfig. Both guides document the new config with parameter descriptions for PCM audio data, sample rate, channels, reference transcript, and synthesis mode, plus usage examples and implementation tips.
HostedVoiceCloneV1 API reference specification
api-reference/realtime.mdx
WebSocket protocol documentation is updated to specify that clean speech audio samples should be 10–15 seconds in duration (reduced from 20–25 seconds) for the voice_clone_v1 guidance and the audio_data field description.
Voice Cloning feature cards
livekit.mdx, pipecat.mdx
Feature cards are added to both integration guides' feature grids, announcing voice cloning capabilities including both pre-cloned hosted voices and on-the-fly cloning from raw audio without external TTS credentials.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

  • deepslate-labs/deepslate-docs#23: Both PRs document the same hosted voice-cloning configuration, introducing the feature across integration guides with configuration fields and usage examples.

Suggested reviewers

  • Janrupf

Poem

🐰 A voice so fine, now cloned with care,
Raw audio samples drift through the air,
No external keys, just truth to tell,
Shorter samples sing—ten seconds ring—
Configuration whispers, the feature bells! 🎵

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The pull request title accurately describes the main change: documenting HostedVoiceCloneConfig support across pipecat and livekit plugins, which is reflected in all three modified documentation files.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch serhat/sdk-29-update-sdk-docs-to-include-voice-clone-tts

Comment @coderabbitai help to get the list of available commands and usage tips.

@mintlify

mintlify Bot commented Jun 4, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
Deepslate 🟢 Ready View Preview Jun 4, 2026, 11:24 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@livekit.mdx`:
- Line 153: The documented duration for the HostedVoiceClone audio sample is
incorrect; update the `audio_data` field description (the table entry currently
saying "10 to 15 seconds of speech") to "20 to 25 seconds of speech" in both
places it appears (the occurrences around the HostedVoiceClone/audio_data
examples), and ensure any sample snippets or examples referencing "10 to 15
seconds" are changed to "20 to 25 seconds" so the docs match the upstream schema
and avoid untested example misconfiguration.

In `@pipecat.mdx`:
- Line 185: Update the documented required duration for the `audio_data`
clone-sample from "10 to 15 seconds" to "20 to 25 seconds" in both places where
it appears (the two entries describing `audio_data` in the clone-sample/hosted
clone schema) so the docs match the hosted clone schema; do not add any example
code snippets that are untested while making this text change.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0b9681ff-5966-4824-81e8-6b8d0bb48232

📥 Commits

Reviewing files that changed from the base of the PR and between 3f735c8 and fa8fe74.

📒 Files selected for processing (2)
  • livekit.mdx
  • pipecat.mdx

Comment thread livekit.mdx
Comment thread pipecat.mdx

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@api-reference/realtime.mdx`:
- Line 620: The documentation changes at lines 620 and 661 in
api-reference/realtime.mdx now specify "10 to 15 seconds" for
HostedVoiceCloneV1.audio_data, but the HostedVoiceCloneV1.audio_data field
definition in api-reference/realtime.proto still documents "20 to 25 seconds".
Resolve this contract mismatch by choosing one as the source of truth: either
update the protobuf schema specification for HostedVoiceCloneV1.audio_data in
api-reference/realtime.proto to "10 to 15 seconds" to match the documentation,
or revert the documentation changes at lines 620 and 661 back to "20 to 25
seconds" to match the proto specification. Ensure all references across both
files use the same duration range consistently.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0b2a0305-fe16-4810-a9d6-79bddaa9a566

📥 Commits

Reviewing files that changed from the base of the PR and between fa8fe74 and 321317b.

📒 Files selected for processing (1)
  • api-reference/realtime.mdx


<Note>
Use 20 to 25 seconds of clean speech for `voice_clone_v1.audio_data`. In JSON, protobuf `bytes` fields are base64-encoded; binary protobuf clients send the raw audio bytes.
Use 10 to 15 seconds of clean speech for `voice_clone_v1.audio_data`. In JSON, protobuf `bytes` fields are base64-encoded; binary protobuf clients send the raw audio bytes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Align HostedVoiceCloneV1.audio_data duration with the protobuf contract.

Line 620 and Line 661 now document 10–15 seconds, but api-reference/realtime.proto for HostedVoiceCloneV1.audio_data still specifies 20–25 seconds. This creates a contract mismatch between API reference and schema for the same field.

Please make one source of truth and update the other side in the same change (either docs back to 20–25 or proto+related docs to 10–15 consistently).

Also applies to: 661-661

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@api-reference/realtime.mdx` at line 620, The documentation changes at lines
620 and 661 in api-reference/realtime.mdx now specify "10 to 15 seconds" for
HostedVoiceCloneV1.audio_data, but the HostedVoiceCloneV1.audio_data field
definition in api-reference/realtime.proto still documents "20 to 25 seconds".
Resolve this contract mismatch by choosing one as the source of truth: either
update the protobuf schema specification for HostedVoiceCloneV1.audio_data in
api-reference/realtime.proto to "10 to 15 seconds" to match the documentation,
or revert the documentation changes at lines 620 and 661 back to "20 to 25
seconds" to match the proto specification. Ensure all references across both
files use the same duration range consistently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant