Plate Food Identifier

[seehorne]

Feature Description
Identify and describe food items on a plate using clock-face directions

Problem It Solves
Visually impaired users lack spatial awareness of their food

Proposed Solution
Provide precise, clock-based audio feedback using a photo of the plate

Implementation details

Assume each generated tool implements one user-facing task. If this issue enumerates multiple stages, execute one ordered copilot_llm_call(...) per stage, pass useful structured artifacts to later calls with metadata={"previous_stage_artifact": ...}, and pass the original image to every later visual stage. If an earlier result is empty, failed, or low-confidence, pass only the original image—not the failed response or artifact—so the next model can inspect it independently. Use the stage capability as capability. Choose only from these capabilities: general_reasoning, ocr, object_detection_localization, structured_visual_understanding, spatial_reasoning, navigation, camera_motion, or temporal_reasoning. Never use visual_reasoning. The backend may evaluate and escalate reasoning capabilities according to the execution policy. Generated tools must not choose implementations, models, providers, detector backends, fallback order, retries, or verification logic. Do not implement detection, OCR, VLM, LLM, model loading, or provider calls inside generated tool files. Generated tools must not create routers, capability registries, detector/OCR/LLM wrappers, new model-router clients, provider-specific DEFAULT_MODEL constants, COCO_CLASSES, .pt model loading/discovery logic, or direct provider calls.

Alternatives Considered

Example usage
{'input': 'Photo of a three-compartment plate containing rice, mixed vegetables, and sesame chicken', 'output': {'description': "Rice from 11 o'clock to 1 o'clock, mixed vegetables from 1 o'clock to 7 o'clock, and sesame chicken from 7 o'clock to 11 o'clock"}}

Live Mode

Live Query

Additional Context

Brainstorming Session

Q: What might happen if the user's plate contains multiple food items that overlap with each other, and how would you want the Plate Food Identifier to handle such a scenario?
A: If multiple things occupy the same clockface portion you can acknowledge their overlap. Start by listing the one first in clockwise order, then the next one. If they both start at the same spot clockface wise, say the one that ends soonest first. If their overlap is exact, order doesn’t matter

Q: What might happen if the user's plate contains multiple food items that overlap with each other, and how would you want the Plate Food Identifier to handle such a scenario?
A: If multiple things occupy the same clockface portion you can acknowledge their overlap. Start by listing the one first in clockwise order, then the next one. If they both start at the same spot clockface wise, say the one that ends soonest first. If their overlap is exact, order doesn’t matter
Unless otherwise specified, in streaming mode, any verbal/text response should be limited to 15 words. No such limit applies to one-shot output.

Video Summary

A person demonstrates a hypothetical visual assistance tool that identifies and locates food items on a plate using a clock-face analogy.

In this scenario, after a user takes a single photo of their meal, the tool would provide an audio description of the plate's contents based on their positions. The following examples from the video illustrate this functionality:

• Input: A photo of a three-compartment plate containing rice, mixed vegetables, and sesame chicken.
• Generated Output: An audio response describing the plate's layout: "Rice from 11 o'clock to 1 o'clock, mixed vegetables from 1 o'clock to 7 o'clock, and sesame chicken from 7 o'clock to 11 o'clock."

The key functionality of this desired tool is to provide visually impaired users with spatial awareness of their food through precise, clock-based audio feedback.

Write the code for this tool inside the tools folder. Assume the tool implements one user-facing task. For a Task Stages section, make one ordered copilot_llm_call(...) per stage and explicitly pass useful artifacts to later calls. Use only these capabilities: general_reasoning, ocr, object_detection_localization, structured_visual_understanding, spatial_reasoning, navigation, camera_motion, temporal_reasoning. Never use visual_reasoning. Do not import litellm, call litellm.completion(), create new model-router clients, create ModelRouter classes, resolve model names, resolve API keys, import detector libraries/provider SDKs/YOLO, define COCO_CLASSES, hardcode model names, load .pt files, or call YOLO(...). Do not select implementations, providers, fallback order, retries, or verification logic in generated tools.