Skip to content

RSDK-14075 Add wake-word-miss sensor #24

Merged
oliviamiller merged 8 commits into
viam-modules:mainfrom
oliviamiller:data-cap
Jun 9, 2026
Merged

RSDK-14075 Add wake-word-miss sensor #24
oliviamiller merged 8 commits into
viam-modules:mainfrom
oliviamiller:data-cap

Conversation

@oliviamiller

@oliviamiller oliviamiller commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Add wakeword-miss-sensor

Summary

  • New viam:filtered-audio:wakeword-miss-sensor sensor model that captures wake-word near-miss audio + metadata to the Viam Data tab — binary WAV in the binary store, tabular row joined by binary_data_id. Useful for diagnosing alse negatives and building wake-word retraining data.
  • wake-word-filter gains two optional config attrs (wakeword_miss_sensor, near_miss_threshold). When both are set and OWW's peak segment score lands in the band near_miss_threshold ≤ max_oww_score < oww_threshold, the filter pushes the segment to the sensor via DoCommand
  • Disabled by default — only fires when wakeword_miss_sensor + near_miss_threshold are both configured. Existing detection path unchanged.

Testing

  • Manually verified on a real robot: wake-miss captures fire, WAVs land in the binary store, tabular rows show binary_data_id populated.

Configuration example added to README under the new model section.

@oliviamiller oliviamiller changed the title Save speech segments as wav files Add wake-word-miss sensor Jun 9, 2026
@oliviamiller oliviamiller marked this pull request as ready for review June 9, 2026 15:53
@oliviamiller oliviamiller requested a review from seanavery June 9, 2026 15:53
@oliviamiller oliviamiller changed the title Add wake-word-miss sensor RSDK-14075 Add wake-word-miss sensor Jun 9, 2026
if yielded:
self._refresh_conversation_window()

async def _maybe_push_miss(self, pcm_bytes: bytes, max_score: float) -> None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A little confused where this gets called?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it gets called in _run_detection

Comment thread src/models/wake_word_filter.py Outdated
miss_sensor_name = str(attrs.get("wakeword_miss_sensor", "")).strip()
instance.miss_sensor = None
if miss_sensor_name:
try:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the sensor dep handling happen here or the config validation?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved to validation

"wake-miss captures disabled",
miss_sensor_name,
)
nm_thresh = attrs.get("near_miss_threshold", None)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to validate this threshold further? What happens if it is greater than oww_threshold?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its it greater then no miss data would be saved, i added a check to validate to error if configured that way

"duration_ms": duration_ms,
"audio_wav_b64": wav_b64,
}
await self.miss_sensor.do_command(cmd)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any issues awaiting the cloud upload here? If this is slow does it interfere with mic streaming?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, I didn't notice any issues while testing but this should run in the background to not block the audio loop, updating

@oliviamiller oliviamiller merged commit f697155 into viam-modules:main Jun 9, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants