β οΈ Archaeology only. This document describes the legacy MITRA v2 architecture, which has been removed. The current runtime is the MHA v3 stack documented in../html-to-markdown.mdand../CLAUDE.md. Use this file to understand how the service used to behave (e.g. when reading old PRs or migrating data); do not rely on it to reason about today's code.
Audience. A founder, a clinician, a designer, or a new engineer. No jargon, no hidden assumptions. Diagrams first, words second.
MindMitra is an AI mental-health companion for Indian youth. A user opens the app, types or speaks a message, and gets back a warm, personalised reply within ~2 seconds. Behind the scenes a single backend service runs a five-stage pipeline that:
- understands what the user just said,
- checks for any safety signal (self-harm, suicide),
- recalls the right memories about this person,
- drafts a response using two AI passes (one fast, one careful),
- streams the answer back word-by-word and quietly writes new memories in the background.
Everything else in this doc is detail on those five steps.
βββββββββββββββ HTTPS / SSE ββββββββββββββββββββββββββββββ
β Browser β ββββββββββββββββββββββββββΆ β FastAPI (chatbotAgent) β
β (React) β ββββββββββββββββββββββββββ β /chat /chat/stream β
βββββββββββββββ stream of text chunks β /chat/greeting /transcribeβ
ββββββββββββββ¬ββββββββββββββββ
β
ββββββββββββββββββββββββΌβββββββββββββββββββββββ
β β β
βββββββββΌβββββββββ ββββββββββΌβββββββββ ββββββββββΌββββββββββ
β MITRA Pipeline β β Memory Services β β External LLMs β
β (one per turn) βββββΆβ Postgres + Qdrantβββββ Azure / Gemini /β
β β β β β Groq / GLM / BGE β
βββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ
β
βββββββββΌβββββββββββββββββ
β Background workers β
β β’ Consolidation worker β
β β’ Reflection worker β
ββββββββββββββββββββββββββ
Three things to remember from this picture:
- The browser only ever talks to FastAPI. Everything else is private.
- A single chat turn goes through one pipeline (the MITRA Pipeline).
- Memory and background work happen next to the pipeline β not inside the user's response time.
chatbotAgent/
βββ app/
β βββ main.py # FastAPI startup, request middleware, CORS
β βββ api/
β β βββ chat.py # /chat, /chat/stream, /chat/greeting,
β β β # /chat/end-session, /transcribe
β β βββ health.py # /health, /debug/memory
β β βββ onboarding.py # First-time user data capture
β β βββ therapist_bridge.py # Clinician handoff
β β βββ me_memory.py # GET/DELETE on a user's own memories
β βββ pipeline/
β β βββ crisis_fast_path.py # 2-stage safety detector (lex + LLM)
β β βββ mitra/
β β βββ orchestrator.py # β runs ONE turn end-to-end
β β βββ classifier.py # what is the user trying to do?
β β βββ retriever.py # parallel memory fetch (deadlined)
β β βββ assembler.py # builds the system + user prompt
β β βββ stance_selector.py # tone / max-questions / disclosure rules
β β βββ generator.py # two-pass draft β critique β repair
β β βββ dual_track.py # optional Track-B: deeper rewrite
β β βββ dispatch.py # the route entry-point
β βββ memory/
β β βββ episodic.py # write/search episodic memories
β β βββ qdrant_v2.py # vector store wrapper (real + in-mem fake)
β β βββ repositories.py # Supabase tables (CRUD)
β β βββ identity_card.py # who is this person? (single JSON doc)
β β βββ working.py # short-term sliding window (last N turns)
β β βββ affective.py # mood time-series
β β βββ relational.py # who appears in their life (graph)
β β βββ procedural.py # what coping strategies they actually use
β β βββ relationship_state.py # stage: stranger β acquaintance β trusted
β β βββ importance.py # 0β1 score for "should we remember this?"
β β βββ decay.py # Ebbinghaus forgetting curve
β β βββ salience.py # blended scoring used at retrieval
β βββ jobs/
β β βββ extractor.py # LLM extracts memory candidates from a turn
β β βββ consolidation_worker.py # offline: dedup + decay + reflect
β βββ providers/
β β βββ azure_openai_client.py # gpt-5-mini (primary generator)
β β βββ gemini_client.py # gemini-2.5-flash (track-B + fallback)
β β βββ groq_client.py # llama-3.1 8B (cheap classifier)
β β βββ glm_client.py # GLM-4-32B (long-context optional)
β β βββ embeddings_bge.py # local BGE-M3
β β βββ embeddings_gemini.py # text-embedding-004 fallback
β βββ core/
β β βββ auth.py # Supabase JWT validation, SKIP_AUTH dev mode
β β βββ config.py # env config + model registry
β β βββ models.py # Roles β ModelConfig mapping
β β βββ prompts/ # stance, critic, crisis prompts
β β βββ pii.py # redact emails / phone before logs
β β βββ rate_limit.py # per-user / per-route limits
β β βββ telemetry.py # OpenTelemetry spans + SLO checks
β β βββ logging.py # β structured logs + banner helpers
β βββ services/
β β βββ supabase_service.py # Postgres client + helpers
β β βββ greeting_service.py # time-aware greeting line
β β βββ voice_prosody.py # Praat-based prosody features
β β βββ helplines.py # localised crisis helpline registry
β β βββ therapist_*.py # clinician-facing bridge
β βββ utils/ # json, constants, small helpers
βββ scripts/
β βββ eval_locomo_lite.py # memory eval harness
βββ tests/
βββ health/ # boot + contract checks (no live LLMs)
βββ (integration tests) # opt-in with RUN_INTEGRATION=1
The two stars (β) are the files you will open most often:
pipeline/mitra/orchestrator.pyβ the storyboard for a chat turn.core/logging.pyβ every log line in the terminal flows through here.
This is the single most important diagram in the codebase.
sequenceDiagram
autonumber
participant U as π€ User (browser)
participant API as FastAPI /chat/stream
participant ORCH as MITRA orchestrator
participant CLS as Classifier (Groq llama-3.1 8B)
participant CRI as Crisis fast-path
participant RET as Retriever (parallel)
participant ASM as Prompt assembler
participant GEN as Generator (Azure gpt-5-mini)
participant CW as Consolidation worker (background)
participant DB as Supabase + Qdrant
U->>API: POST /chat/stream { message, session_id }
API->>API: validate JWT, log "π₯ POST /chat/stream"
API->>ORCH: run_mitra_turn(...)
par classify and check safety in parallel
ORCH->>CLS: what intent + needs_memory?
ORCH->>CRI: lexical scan; if ambiguous β LLM confirmer
end
alt π¨ crisis triggered
ORCH-->>API: localised crisis safety reply
API-->>U: SSE chunks (helpline + grounding)
else π normal turn
ORCH->>RET: fetch identity_card + episodes + affect + procedural
RET->>DB: parallel reads (deadlined ~600ms)
ORCH->>ASM: build system + user prompt with stance constraints
ORCH->>GEN: two-pass: draft β critic β repair (optional Track-B)
GEN-->>API: stream chunks via stream_callback
API-->>U: SSE chunks (sentence-buffered)
ORCH->>DB: write turn trace + working memory append
end
Note over API,CW: After response, in background:
API->>CW: every N messages β run_once_for_user
CW->>DB: extract candidates β dedup β write episodic memories
| # | Stage | What it does | Where it lives |
|---|---|---|---|
| 1 | Classify + safety | Looks at the message and asks two questions in parallel: (a) what is the user trying to do β vent, share, ask for advice, etc.? and (b) is this a safety situation? | mitra/classifier.py + pipeline/crisis_fast_path.py |
| 2 | Crisis short-circuit | If safety says hard, stop everything. Reply with the localised helpline + grounding script. No memory reads, no LLM generation. | pipeline/crisis_fast_path.py |
| 3 | Retrieve | Pull only what the next prompt actually needs: who this person is (identity card), their last few highly-relevant memories, their recent mood pattern, what coping moves have worked before. All four reads run in parallel, with a hard ~600 ms deadline. | mitra/retriever.py + memory/* |
| 4 | Assemble + stance | Build a system prompt that says "you are Mitra, this user is X, here are 3 relevant memories, your tone today is holding not advising, ask at most 1 question". | mitra/assembler.py + mitra/stance_selector.py |
| 5 | Generate | First pass writes a draft. A tiny critic checks for style violations (too long, too clinical, too many questions, missed groundingβ¦). If anything fails, the model rewrites. The critic is bounded β at most 1 repair pass on the hot path. | mitra/generator.py, optional mitra/dual_track.py |
After the response is sent, two background things happen so the user never waits for them:
- Consolidation worker (
jobs/consolidation_worker.py) extracts new memory candidates from the last few turns, dedupes against existing memories, scores them withimportance.py, and writes the survivors to Postgres + Qdrant. - Reflection (also in the worker) every few sessions writes a cross-session insight ("they cope better when their sister is around").
Different jobs need different models. The mapping lives in
core/models.py and is the single source of truth.
| Role | Model | Provider | Why this one |
|---|---|---|---|
| Classifier (intent + needs-memory) | llama-3.1-8b-instant |
Groq | Fast, cheap, structured-output friendly |
| Crisis confirmer (LLM second-stage) | gemini-2.5-flash |
Gemini | Low latency, multilingual |
| Generator (primary) | gpt-5-mini |
Azure OpenAI | Best instruction-following + true streaming |
| Generator (Track B / fallback) | gemini-2.5-flash |
Gemini | Long context, cheaper rewrite path |
| Long-context (only when prompt > 32k) | glm-4-32b-0414-128k |
Z.AI | Cheap 128k window |
| Memory extractor (offline) | llama-3.1-8b-instant |
Groq | Cheap; runs in background |
| Embeddings (primary) | BGE-M3 |
local | Multilingual, no per-call cost |
| Embeddings (fallback) | text-embedding-004 |
Gemini | Drop-in if BGE host is down |
| Speech-to-text | whisper-large-v3-turbo |
Groq | Cheap fallback when Azure STT returns empty |
If you swap a model, change exactly one row in core/models.py. The
pipeline picks it up on the next startup.
Memory is the hardest part to explain, so we use a simple metaphor: a notebook, a journal, and a habit log.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Per-user memory model β
ββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Identity cardβ One short JSON: name, pronouns, languages, key β
β (the cover) β relationships, soft preferences. Read every turn. β
ββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Episodic β Up to a few hundred short "moments" per user. β
β (the journal)β Each has: summary, importance, mood (V-A-D), tags, β
β β embedding. Lives in Postgres + Qdrant. β
ββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Affective β Time-series of mood scores over the last N days. β
β (the mood β Used to detect patterns ("worse on Sundays"). β
β graph) β β
ββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Relational β Tiny graph: people in the user's life, their roles. β
β (who's who) β Stops Mitra from confusing "didi" with "boss". β
ββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Procedural β Coping strategies that the user has tried + worked. β
β (the habit β Used by the assembler to suggest things they already β
β log) β trust. β
ββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Working β A sliding window of the last few turns of THIS chat. β
β (scratchpad) β In-process; flushed at session end. β
ββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Relationship β A single label per user: stranger β acquaintance β β
β state β trusted β close. Controls how much self-disclosure β
β β the assistant offers. β
ββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
flowchart LR
A[Chat turn happens] -->|inline trace| B[Working memory<br/>append]
A -.->|every N turns| C[Consolidation worker<br/>kicks off in background]
C --> D[LLM extractor<br/>jobs/extractor.py]
D --> E[Importance score<br/>memory/importance.py]
E -->|score < 0.3| X[Drop]
E -->|score β₯ 0.3| F[Dedupe vs existing<br/>cosine similarity]
F -->|duplicate| G[Reinforce existing<br/>+1 strength]
F -->|new| H[Write to Postgres<br/>+ upsert into Qdrant]
H --> I[Decay over time<br/>memory/decay.py]
I -->|strength β 0| J[Archive]
H -->|recalled in a future turn| K[mark_recalled<br/>strength bump]
Two things to internalise:
- Memories get stronger every time we use them. Recall = reinforcement.
- Memories that are never recalled fade. This is by design β it prevents the database from drowning the model in stale context.
Safety is the only stage that is allowed to skip the rest of the pipeline. If anything looks like self-harm or acute suicidal ideation, we never run the generator.
ββββββββββββββββββββββββββββ
β User message arrives β
ββββββββββββββββ¬ββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββ
β Stage 1 β lexical scan β multilingual phrases, regex,
β (instant, no network) β romanised Hindi included
ββββββββ¬βββββββββββββ¬βββββββ
β safe β ambiguous β hard
βΌ βΌ βΌ
continue Stage 2 β Gemini CRISIS FAST-PATH
pipeline confirmer (β€ 600ms) β’ play helpline message
β β’ do grounding script
βΌ β’ log crisis_event
triggered? ββββ yes βββΆ β’ return immediately
β
βΌ
no β continue pipeline
Notes:
- The lexical list is hand-curated, multilingual, and intentionally over-flags. The LLM second stage cleans up false positives.
- The fast-path never waits on memory or generation.
- Every trigger writes a row to
crisis_eventsfor audit + clinician follow-up.
The chat UI is the part most likely to feel broken if a single piece of plumbing is wrong, so this section is precise.
Provider (Azure / Gemini)
β yields text deltas
βΌ
azure_openai_client._complete_stream
β’ producer runs in a worker thread
β’ puts deltas into asyncio.Queue
β’ CRITICAL: producer is fire-and-forget
(asyncio.create_task, NOT awaited)
β
βΌ
mitra/dispatch._make_llm_caller
β’ async-iterates the queue
β’ calls stream_callback(delta) for each chunk
β
βΌ
api/chat.py event_generator
β’ on_chunk pushes into another asyncio.Queue
(loop.call_soon_threadsafe β thread safe)
β’ main loop reads queue, buffers until a
sentence boundary, then yields:
event: text_chunk_delta
data: {"chunk": "..."}\n\n
β
βΌ
Browser EventSource
β’ parses each `data:` line as JSON
β’ appends `chunk` to the live message bubble
If you ever see "the response only appears at the end", the bug is
almost certainly in step 2 β someone re-introduced an await before
the producer task. The fix is asyncio.create_task(asyncio.to_thread(...)),
never await asyncio.to_thread(...).
Every important step in the backend now prints a coloured banner or a single-line stage marker. A healthy chat turn produces logs like:
2026-04-20 14:01:12 [INFO] app.api.chat req=ab12 βββ π₯ POST /chat/stream
user : a0778b19
session : 9f3d
persona : mitra
language : en
avatar : True
msg_len : 47 chars
2026-04-20 14:01:12 [INFO] app.mitra req=ab12 βββ π§ MITRA TURN START
...
2026-04-20 14:01:12 [INFO] app.mitra req=ab12 [STAGE] classify+safety intent=vent safety=safe crisis=False dur_ms=180
2026-04-20 14:01:12 [INFO] app.mitra req=ab12 [STAGE] retrieve_memories episodes=4 identity_card=True affect=low_mood dur_ms=210
2026-04-20 14:01:12 [INFO] app.mitra req=ab12 [STAGE] select_stance stance=holding max_q=1
2026-04-20 14:01:13 [INFO] app.mitra req=ab12 [STAGE] assemble_prompt sys_chars=2310 user_chars=412 episodes_used=3 dur_ms=8
2026-04-20 14:01:13 [INFO] app.mitra req=ab12 [STAGE] generate_response generator=TwoPassGenerator stream=True
2026-04-20 14:01:14 [INFO] app.mitra req=ab12 [STAGE] generate_done chars=312 accepted_on_pass=1 fallback=False dur_ms=1100
2026-04-20 14:01:14 [INFO] app.mitra req=ab12 βββ β
MITRA TURN COMPLETE
intent : vent
stance : holding
chars : 312
episodes : 3
accepted_on : pass 1
fallback : False
timings_ms : classify_ms=180 retrieve_ms=210 assemble_ms=8 generate_ms=1100 total_ms=1500
2026-04-20 14:01:15 [INFO] app.api.chat req=ab12 π§ͺ [CONSOLIDATE] user=a0778b19 candidates=2 written=1 archived=0 reflections=0
Three things make this useful in production:
req=correlation id β every log line in one HTTP turn shares the same request id (set by middleware inmain.py, propagated viaContextVarandbind_request_context).- Banner + stage style β banners frame the start/end of a unit of work; stages are dense single-line markers between them.
- Background threads inherit context. Use
spawn_correlated_thread(fromcore/logging.py) instead ofthreading.Threaddirectly β the consolidation worker uses this so its logs still carryreq=.
Set LOG_FORMAT=json in production to get structured JSON instead of
the colour banners β same fields, different renderer.
| Variable | Effect | Default |
|---|---|---|
MITRA_STACK_ENABLED |
Master switch. If off, /chat returns 503. |
1 (on) |
MITRA_CRISIS_V2_ENABLED |
Use the 2-stage crisis detector. | 1 |
MITRA_DUAL_TRACK_ENABLED |
Run optional Track-B rewrite for tougher turns. | 0 |
MITRA_CONSOLIDATION_INTERVAL |
Run consolidation every N user turns. | 12 |
LOG_LEVEL |
INFO / DEBUG / WARNING. | INFO |
LOG_FORMAT |
colour (dev) or json (prod). |
colour |
SKIP_AUTH |
Dev only. Replaces JWT validation with DEV_USER_ID. |
unset |
QDRANT_URL |
Vector store. | required in prod |
SUPABASE_URL / SUPABASE_SERVICE_KEY |
Postgres + auth. | required in prod |
The full list lives in chatbotAgent/.env.example.
Q: Where do I add a new "intent"?
mitra/classifier.py defines Intent. Add the value, update the
classifier prompt, and decide in stance_selector.py what stance + max
questions that intent should trigger.
Q: Why are there two "generators"?
The default is TwoPassGenerator β one draft, one critic, optional
repair. The richer DualTrackGenerator runs an additional Track-B
rewrite for emotionally heavy turns. Both implement the same generate()
interface so the orchestrator doesn't care which is wired.
Q: How do I disable memory entirely for a test?
Pass an in-memory Qdrant (InMemoryQdrant from memory/qdrant_v2.py)
and a no-op EpisodicRepo. The retriever degrades gracefully β empty
context is fine.
Q: What happens if Azure / Gemini is down?
The provider raises; the orchestrator catches and returns a soft
fallback message and logs fallback=True. The critic also has a
"soft fallback" template it can return without an LLM call.
Q: Why isn't there a workflow.py any more?
There used to be a legacy linear pipeline. It was deleted in this
migration. All chat now flows through pipeline/mitra/.
docs/platform.mdβ memory v2 modules, Qdrant, pipeline, runbook.docs/EVALUATION.mdβ how we test (pytest + LOCOMO-lite + LLM judge).docs/api_contracts.mdβ every HTTP endpoint, request, response.docs/product.mdβ MindGym, therapist bridge, analytics, security headers.chatbotAgent/app/pipeline/mitra/orchestrator.pyβ when in doubt, read the orchestrator. It is intentionally short.
This section documents deferred improvements with concrete recipes. Anyone picking these up should be able to ship them in one PR.
Prosody analysis is OFF on the chat path (April 2026). The LLM never
consumed the features and import parselmouth has hung the FastAPI
lifespan under macOS Gatekeeper while holding the GIL. The synchronous
analyser still lives in app/services/voice_prosody.py for offline jobs.
When you actually want the LLM to see acoustic features, follow this
3-step recipe β never call analyze_prosody from an async handler:
- Carry the field through the pipeline. Add a
prosody: Optional[Dict[str, Any]]field toTurnInput(app/pipeline/mitra/orchestrator.py) so the structured features reach the assembler. - Render it into the system prompt. Add a small block in the
assembler (
app/pipeline/mitra/assembler.py) that, whenprosodyis present, callsfrom app.services.voice_prosody import format_prosody_for_promptand inlines the resulting block under a## Acoustic contextheading. Keep it short β one paragraph. - Extract Praat in a sandboxed subprocess. Create
app/tools/prosody_cli.pythat reads a base64 WAV from stdin, runsanalyze_prosody, and writes JSON to stdout. From the chat endpoint:subprocess.run(["python", "-m", "app.tools.prosody_cli"], input=wav_b64, capture_output=True, text=True, timeout=4). If the timeout fires you kill the subprocess β the FastAPI process is unaffected.
Also re-enable the dependency: uncomment praat-parselmouth==0.4.7 in
chatbotAgent/requirements.txt and re-add the audio_data field to the
frontend POST in src/components/chat/ChatGPTInterface.tsx.
The backend now forwards every provider delta straight to the SSE
stream β no sentence-boundary buffer (app/api/chat.py event_generator).
First audible byte should land within ~1.5 s of the request reaching
the orchestrator on a healthy Azure connection.
If you ever re-introduce server-side buffering (sentence batching, JSON
chunking, etc.), gate it behind a feature flag and document the
trade-off here. Frontend Presence mode already does its own
Unicode-aware sentence segmentation in
src/components/chat/ChatGPTInterface.tsx (COMPLETED_SENTENCES_RE).
SKIP_AUTH=0β the dev override bypasses Supabase JWT validation.- Multi-worker uvicorn:
uvicorn app.main:app --workers 4 --worker-class uvicorn.workers.UvicornWorker. - Wire
app/core/rate_limit.pyinto the chat router (currently defined but not enforced). - Apply the
mitra_turn_tracesschema migration soaccepted_on_pass,fallback_used,stance,callback_budget,critic,used_episodesexist as real columns. The repository's schema-drift fallback then becomes a no-op (no warning, no JSON demotion). - Set
OTEL_SERVICE_NAMEso spans aren't no-op. - Validate Gemini quota and consider flipping
MITRA_DUAL_TRACK_ENABLED=1for parallel speculative generation. - Pick a faster
GENERATOR_PRIMARYmodel ifgpt-5-minip95 latency remains > 3 s βgpt-4o-miniand Groq Llama-3.1-70B are reasonable candidates with their own quality bake-offs.
Never import a native-code library at module top-level inside a route
handler, the FastAPI lifespan, or any code path reachable from an
async request. Native dynamic loaders (dyld on macOS, ld-linux on
Linux) can stall on signature checks, license probes, or model-file
downloads while holding the GIL β and a single stalled import freezes
the entire ASGI process.
If you must use such a library:
- Wrap it in a CLI subprocess called via
subprocess.run(..., timeout=N)so a hang can be killed without taking down the server. - Or run it in a one-shot worker (
spawn_correlated_thread,concurrent.futures.ProcessPoolExecutor) gated by a hard timeout and a circuit breaker. - Or push it entirely off-line into a job (
app/jobs/) that the chat path never awaits.
This rule is what got broken when prosody was inlined into the chat endpoint. Treat any new native dependency (Whisper, faiss, llama-cpp, sentence-transformers, etc.) the same way.
Last revised: April 2026. If anything in this doc disagrees with the code, the code is right β please update this file in the same PR.