Skip to content

docs(saa-js): document turnReady event payload field types (#22) #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
MunawarJohar wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs(saa-js): document turnReady event payload field types (#22) #43

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion packages/saa-js/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ await client.start({ videoElement: videoEl });
| `prediction` | `{ cls, rawCls, confidence, source, numFaces, responding }` |
| `vad` | `{ probability, isSpeech }` |
| `state` | `{ state }` (one of `listening`, `sending`, `cancelled`, `idle`) |
| `turnReady` | `{ audioBase64, audioPcm16, durationSec, frames, context }` |
| `turnReady` | `{ audioBase64, audioPcm16, durationSec, serverTurnReadyTsMs, frames, context }` — see [field types](#turnready-fields) below |
| `config` | `{ modelClass2Threshold }` |
| `stats` | `{ rttMs, bufferedAmount, sentVideo, skippedVideo, sentAudio, uptimeMs }` |
| `interrupt` | `{ fadeMs, confidence }` |
Expand All @@ -82,6 +82,24 @@ await client.start({ videoElement: videoEl });

`warmupComplete` fires once the server model has warmed up and is producing real predictions; use it to drop any loading UI. `prediction.responding` is `true` while your app is mid-response (see `markResponding`), and `interjection` fires when the agent should volunteer after humans go quiet.

### `turnReady` fields

| Field | Type | Description |
| --- | --- | --- |
| `audioBase64` | `string` | Turn audio, PCM16 @ 16 kHz mono, base64-encoded. |
| `audioPcm16` | `Int16Array` | Same audio, decoded to a typed array. |
| `durationSec` | `number` | Turn duration in seconds. |
| `serverTurnReadyTsMs` | `number \| null` | Server-clock timestamp (ms) when the turn was emitted; useful for latency measurement. `null` on older servers that don't send it. |
| `frames` | `Array<{ tsOffsetS: number; imageBase64: string }>` | Video frames captured during the turn. Empty array in audio-only mode. `tsOffsetS` is seconds from turn start. |
| `context` | `string \| null` | Set by the server for special cases, e.g. `"interjection_follow_up"`. `null` for a normal turn. |

```ts
client.on("turnReady", (turn) => {
yourSTT.send(turn.audioBase64);
console.log(`turn was ${turn.durationSec}s, ${turn.frames.length} video frames`);
});
```

## LLM integration

The SDK captures speech but does **not** route it to an LLM. SAA is model-agnostic and drop-in: use the `turnReady` event to forward only device-directed audio to any model, ASR, or voice stack you like.
Expand Down