feat: hand-grab overlay control (fist to grab, move to drag, open to drop)#26
Merged
Conversation
# Conflicts: # README.md
MediaPipe Hands via ONNX Runtime CPU, all C#: pointing-pose arming + fingertip-velocity nudge moves the overlay. No shim changes, no NVIDIA dependency (AR SDK has no hand features; TAO GestureNet re-enters the TRT co-version hazard). Non-RTX capable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
8 tasks: model I/O verification gate first, then TDD Core units (palm decode, pose classifier, nudge state machine), config/VM plumbing, App ONNX inference loop, UI wiring, human-gate verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nsor I/O Palm detection (192x192, post-processed boxes) + hand landmark (224x224, 21pt + presence + handedness), sourced from PINTO0309/hand-gesture- recognition-using-onnx (Apache-2.0) after rejecting the Qualcomm AI Hub primary source (license: other, not Apache-2.0/MIT). README.md records source URLs, license evidence, exact input/output tensor names, shapes, layout, value range, and output semantics for later tasks to copy verbatim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rewrite the third-source deviation rationale as plain project reasoning (the quoted allowance came from controller dispatch instructions, not the task brief file — a brief grep would find nothing); loosen the sqn_ = square-normalized aside to read as naming inference, not a source quote; drop a second brief attribution on the anchor-config reference table. No technical values changed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… output (Core, tested) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l function) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ore, tested) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…und-trip Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… CPU, 15 Hz) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed indicator Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…et tracker on stop; surface inference failures Final-review findings: cos_get_frame is consume-on-read, so the inference loop was stealing frames from the present pump (overlay judder) — the pump is now the sole consumer and republishes into HandInference. Tracker.Reset on Stop kills the stale-armed teleport + stuck handle; a Failed event greys the toggle with a note when the loop dies (spec §7). *.onnx marked binary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ea clamp, post-processed palm path Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… after loop death Final-review residuals: a Failed loop greys the toggle but left it enabled, so a camera stop/start resurrected the dead loop (SyncFingerControl now also requires FingerControlAvailable); and the pump kept copying ~8 MB/tick into the tap for a loop that would never read it (_running = false in the catch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nting is inert Human-gate feedback: pointing collided with the presenter's primary gesture (pointing at content) and felt like a mode toggle. Fist = grab, move = drag, open hand = drop; tracked point moves to landmark 9 (palm-center proxy — the index tip is buried in a fist). Classifier gains a three-way split (Pointing / Fist / Other); the tracker arms on Fist only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…exTip const Review follow-ups on the fist pivot: new test proves a Pointing frame while armed counts as a lost frame (disarm on the 5th, same hysteresis as NoHand); test name fixed to ArmsAfterThreeConsecutiveFistFrames; spec banner intro made count-neutral; IndexTip const removed (unused since the tracked point moved to PalmCenter). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hands-free overlay repositioning for teaching/showcasing: make a fist ✊ at the webcam to grab the overlay, move your hand to drag it, open your hand to drop it. Pointing ☝ is deliberately inert so presenters can point at screen content freely.
src/CameraOnScreen.App/Assets/models/hand/README.md).cos_get_frameconsumer — it's consume-on-read; a second caller steals frames and judders the overlay).Design/plan/reviews
Spec:
docs/superpowers/specs/2026-07-08-finger-pointing-overlay-control-design.md(amended — see banner)Plan:
docs/superpowers/plans/2026-07-08-finger-pointing-overlay-control.mdExecuted subagent-driven: per-task spec+quality reviews, final whole-branch review (1 Critical frame-stealing integration bug found + fixed), gesture pivot from live human-gate feedback.
Test plan
-DryRunconfirmsmodels\hand\rides in the staged build output🤖 Generated with Claude Code