Skip to content

Add image input to the AI Coach (multimodal)#31

Merged
saksham2001 merged 2 commits into
mainfrom
feature/coach-multimodal-images
Jun 29, 2026
Merged

Add image input to the AI Coach (multimodal)#31
saksham2001 merged 2 commits into
mainfrom
feature/coach-multimodal-images

Conversation

@saksham2001

Copy link
Copy Markdown
Owner

Summary

Lets users attach a photo to a coach message, sent alongside text to whichever provider is active: OpenAI (Responses), Gemini (generateContent), or OpenRouter (Chat Completions). Off by default behind a new Settings toggle.

  • Providers: images enter at the one canonical request builder; each adapter translates to its own shape (input_image / inlineData / image_url). Text content stays a String, so existing text + tool-call flows are untouched.
  • Storage: images compressed to JPEG in Documents/coach_attachments/; CoachMessage stores only a small JSON ref (additive SwiftData field).
  • UI: camera button in the composer (only when enabled) → staged thumbnail → image rendered in the user bubble. Image-only sends allowed.
  • Tests: 172 existing tests still pass; 7 new tests cover each provider's image shape + text-only regression.

Related issues

#27

Type of change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 📟 New / improved wearable support (BLE driver layer)
  • 🤖 Coach / LLM change (tools, prompts, orchestration)
  • 🎨 UI / DesignSystem change
  • 🧹 Refactor / chore (no behavior change)
  • 📝 Docs only
  • ⚠️ Breaking change (existing data, settings, or APIs change)

How was this tested?

  • Added / updated unit tests (PulseLoopTests)
  • Ran the test suite locally (⌘U in Xcode)
  • Tested on a physical device with a real ring — model:
  • Tested with demo data (-seedDemo YES, no hardware)
  • N/A (docs / non-code change)

Privacy & data

  • This change does not send health data off-device without explicit user action.
  • No secrets, API keys, or personal data are committed.
  • N/A

Screenshots / recordings

Checklist

  • My code follows the project's style (SwiftLint passes).
  • I ran the tests and they pass.
  • I updated docs / README where relevant.
  • I read the Contributing guide.

@saksham2001 saksham2001 self-assigned this Jun 29, 2026
@saksham2001 saksham2001 linked an issue Jun 29, 2026 that may be closed by this pull request
@saksham2001 saksham2001 merged commit cbb2487 into main Jun 29, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Make AI coach multimodal

1 participant