Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions examples/twilio/media_streams/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,12 @@
from dataclasses import dataclass, field
from typing import Any, Awaitable, Callable, Optional, Union

from pathlib import Path

from dotenv import load_dotenv

load_dotenv(Path(__file__).resolve().parents[1] / ".env")

try:
from fastapi import FastAPI, HTTPException, Request, WebSocket, WebSocketDisconnect
from fastapi.responses import JSONResponse, PlainTextResponse, Response
Expand Down
8 changes: 4 additions & 4 deletions packages/saa-js/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ await client.start({ videoElement: videoEl });

| Method | Description |
| --------------------------- | ----------- |
| `start({ videoElement, mediaStream? })` | Start streaming + connect. Calls `getUserMedia` unless `mediaStream` is supplied. `videoElement` is required when video capture is enabled. |
| `start(options?)` | Start streaming + connect. Pass `{ videoElement }` when video capture is enabled; `{ mediaStream }` to reuse an existing stream. Calls `getUserMedia` unless `mediaStream` is supplied. With both `enableAudio` and `enableVideo` false, call with no args. |
| `stop()` | Stop streaming and disconnect. |
| `feedAudio(audio, sampleRate?)` | Push externally-captured audio (requires `enableAudio: false`). Accepts Float32 `[-1,1]`, Int16 PCM, or a raw int16 buffer; re-chunked + resampled to the wire's 16 kHz / 100 ms blocks. See [External capture](#external-capture). |
| `feedVideo(jpeg)` | Push an externally-captured JPEG frame (requires `enableVideo: false`). Accepts a `Blob`, `ArrayBuffer`, or view. |
Expand All @@ -70,13 +70,13 @@ await client.start({ videoElement: videoEl });
| `prediction` | `{ cls, rawCls, confidence, source, numFaces, responding }` |
| `vad` | `{ probability, isSpeech }` |
| `state` | `{ state }` (one of `listening`, `sending`, `cancelled`, `idle`) |
| `turnReady` | `{ audioBase64, audioPcm16, durationSec, frames, context }` |
| `turnReady` | `{ audioBase64, audioPcm16, durationSec, serverTurnReadyTsMs, frames, context }` |
| `config` | `{ modelClass2Threshold }` |
| `stats` | `{ rttMs, bufferedAmount, sentVideo, skippedVideo, sentAudio, uptimeMs }` |
| `interrupt` | `{ fadeMs, confidence }` |
| `interjection` | `{ reason, audioBase64, audioPcm16, durationSec }` |
| `error` | `{ title, message, detail }` |
| `disconnected` | `{ code, reason }` |
| `error` | `{ title, message, detail, code? }` |
| `disconnected` | `{ code, reason, wasClean }` |

`warmupComplete` fires once the server model has warmed up and is producing real predictions; use it to drop any loading UI. `prediction.responding` is `true` while your app is mid-response (see `markResponding`), and `interjection` fires when the agent should volunteer after humans go quiet.

Expand Down
30 changes: 30 additions & 0 deletions packages/saa-js/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

87 changes: 87 additions & 0 deletions packages/saa-js/test/client.test.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
// node:test suite for @attenlabs/saa-js AttentionClient edge cases.
// Uses a fake WebSocket so tests run without a live broker or browser APIs.

import test, { afterEach } from "node:test";
import assert from "node:assert/strict";
import { AttentionClient } from "../dist/index.js";

/** @type {typeof globalThis.WebSocket | undefined} */
let savedWebSocket;

/** @type {MockWebSocket | null} */
let activeSocket = null;

class MockWebSocket {
static OPEN = 1;

constructor(url, protocols) {
this.url = url;
this.protocols = protocols;
this.readyState = MockWebSocket.OPEN;
this.binaryType = "arraybuffer";
activeSocket = this;
// Defer open until the client assigns onopen/onclose handlers.
setTimeout(() => this.onopen?.({}), 0);
}

send() {}

close(code = 1000, reason = "") {
setTimeout(
() =>
this.onclose?.({
code,
reason,
wasClean: code === 1000,
}),
0,
);
}
}

afterEach(async () => {
if (savedWebSocket !== undefined) {
globalThis.WebSocket = savedWebSocket;
} else {
delete globalThis.WebSocket;
}
savedWebSocket = undefined;
activeSocket = null;
});

function installMockWebSocket() {
savedWebSocket = globalThis.WebSocket;
// @ts-expect-error test double
globalThis.WebSocket = MockWebSocket;
}

test("mid-session disconnect delivers error when prediction listener was never registered", async () => {
installMockWebSocket();

const errors = [];
const client = new AttentionClient({
token: "bad-token",
url: "wss://test.example/ws",
enableAudio: false,
enableVideo: false,
});

client.on("error", (event) => {
errors.push(event);
});

await client.start();
assert.ok(activeSocket);
assert.deepEqual(activeSocket.protocols, ["bad-token"]);

activeSocket.close(1006, "connection dropped");

await new Promise((resolve) => setTimeout(resolve, 0));

assert.equal(errors.length, 1);
assert.equal(errors[0].title, "Connection Failed");
assert.match(errors[0].message, /Could not reach the server/);
assert.equal(errors[0].code, 1006);

await client.stop();
});
19 changes: 19 additions & 0 deletions packages/saa-py/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ client = AttentionClient(
enable_audio=True, # Set False to skip mic capture
enable_video=True, # Set False to skip webcam capture
server_profile=None, # Override server profile; auto "audio_only" when enable_video=False
auto_reconnect=True, # Reconnect on retriable WebSocket drops
)
```

Expand Down Expand Up @@ -157,6 +158,8 @@ def handle(event):
| `@on_interjection` | `InterjectionEvent` | Proactive AI volunteer after humans go quiet |
| `@on_error` | `AttentionErrorEvent` | Connection, auth, or server error |
| `@on_disconnected` | `DisconnectedEvent` | WebSocket closes |
| `@on_reconnecting` | `ReconnectingEvent` | Backing off before a reconnect attempt |
| `@on_reconnected` | `ReconnectedEvent` | WebSocket reopened after a drop |

### Event types

Expand Down Expand Up @@ -244,6 +247,22 @@ title: str # error category ("Auth Failed", "Connection Stalled
message: str # human-readable message
detail: str | None = None # technical detail
code: int | None = None # WebSocket close code, if applicable
kind: str | None = None # transport | auth | rate_limit | audio | server | environment
retriable: bool = False # whether auto_reconnect will retry this class of error
```

#### `ReconnectingEvent`

```python
attempt: int # 1-based attempt counter
delay_s: float # backoff before this attempt
last_code: int # close code that triggered the reconnect
```

#### `ReconnectedEvent`

```python
attempts: int # attempts it took to reconnect
```

#### `DisconnectedEvent`
Expand Down
5 changes: 4 additions & 1 deletion packages/saa-py/src/saa/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -907,7 +907,10 @@ def _close_to_error(code: int, reason: str, was_clean: bool) -> Optional[Attenti
return AttentionErrorEvent(
title="Connection Failed",
message="Could not reach the server.",
detail=f"The server may be down or unreachable. (close code {code})",
detail=(
f"The server may be down or unreachable."
f" (close code {code}, reason={reason or 'none'})"
),
code=code,
kind="transport",
retriable=True,
Expand Down