Build an Action Validator for Physical Task Safety

[sahoo-tech]

Labels: hard enhancement gssoc-2026

Before Execra confirms a step as complete in the physical domain, validate that the user performed the action correctly and safely.

What you'll code:

• Create core/physical/action_validator.py:
  • ActionValidator.validate(step: str, detections, hand_results, depth_map) -> ValidationResult
  • ValidationResult model: is_valid: bool, confidence: float, issues: list[str], corrections: list[str]
  • Validation rules for each task type:
    • Cooking: knife used near cutting board (not near body) for cutting steps
    • Hardware: screwdriver oriented correctly (vertical, not horizontal) for assembly
    • Form filling: pen in dominant hand near paper (not phone)
  • If is_valid=False: dispatch a corrective GuidanceInstruction before advancing the step
• Write unit tests with synthetic detection scenarios for each validation rule

Skills needed: Python · spatial reasoning · domain logic · CV integration

👉 Claim this issue on GitHub →

---

[purvask2006-collab]

Hi @sahoo-tech

I would love to take on this issue as part of GSSoC 2026! Implementing the physical action validation layer is a critical step in turning Execra into a proactive, safety-first assistant. I have strong experience with Python, spatial geometry/reasoning, and integrating computer vision models like YOLO and MediaPipe.

Here is my target execution plan for this module:

Proposed Implementation Plan

1. Architecture & Data Models (core/physical/action_validator.py):

• Define the ValidationResult schema using Pydantic (or standard Python dataclasses depending on project standards) to strongly enforce types for is_valid, confidence, issues, and corrections.
• Implement the structural interface for ActionValidator.validate(step, detections, hand_results, depth_map).

2. Domain-Specific Spatial Reasoning Rules:

• Cooking Domain: Parse bounding box coordinates from detections to compute spatial proximity vectors between identified knives and cutting boards, checking absolute bounding boundaries relative to the detected user body frame.
• Hardware Domain: Use hand_results landmarks and depth_map matrices to calculate the orientation vector/slope of a screwdriver, enforcing threshold logic for true vertical alignment ($75^\circ \text{ to } 90^\circ$ relative to the tool surface plane).
• Form Filling Domain: Check intersection regions of pen coordinates with paper bounding contours while verifying hand tracking states to flag phone distractions.

3. Proactive Intervention:

• Connect the validation engine failure states directly to the GuidanceDispatcher system to immediately override progress and throw a corrective GuidanceInstruction overlay whenever is_valid == False.

4. Testing Architecture:

• Write mock scenarios inside tests/unit/ using synthetic bounding boxes, landmark matrices, and mock depth fields to ensure $100%$ validation accuracy across both safe and unsafe scenarios.

I am fully available to start working on this immediately according to the project's workflow patterns and conventional commit guidelines. Could you please assign this issue to me?

Thank you!

[sahoo-tech]

@purvask2006-collab please start working on it

[purvask2006-collab]

Thank you for assigning this to me, @sahoo-tech Excited to dive into this.

Before I begin implementing the ActionValidator, I want to clarify a few architectural and data contract details to ensure it aligns perfectly with the existing codebase:

Data Contract & Schema Preferences: For the ValidationResult model, should I use Pydantic (v2) or standard Python dataclasses? Additionally, since the instructions mention dispatching a corrective GuidanceInstruction when is_valid=False, should ActionValidator.validate() return this instruction inside the corrections list as a string, or should the validator directly interact with an existing GuidanceDispatcher component?

Expected Data Structure for CV Inputs: Could you clarify the expected structures for detections and hand_results? For example, are detections structured as standard YOLO-style bounding boxes ([x_min, y_min, x_max, y_max, confidence, class_id]), and is hand_results modeled after the MediaPipe Hands landmark topology (21 points with x, y, z coordinates)? Knowing this will help me construct accurate synthetic scenarios for the unit tests.

Depth Map & Coordinate Frame Integration: For the hardware domain's screwdriver orientation rule, you mentioned using the depth_map matrix to calculate alignment relative to the tool surface plane. Do we have existing geometric helper utilities within Execra to project 2D pixel coordinates + depth values into a 3D camera-frame vector, or should I implement those spatial projection matrices from scratch inside action_validator.py?

Let me know your thoughts on these, and I'll kick off the implementation right away.

[purvask2006-collab]

Hi, I'm a GSSoC 2026 contributor. I have completed this issue and raised a PR.

PR: (#274)

Changes made:

• core/physical/action_validator.py — ActionValidator with ValidationResult model and domain-specific rules (cooking, hardware, form filling)
• tests/test_action_validator.py — 55 unit tests, all passing (98% coverage)

Please review when you get a chance. Happy to make any changes based on feedback!