Skip to content

Latest commit

Β 

History

History
572 lines (478 loc) Β· 30.1 KB

File metadata and controls

572 lines (478 loc) Β· 30.1 KB

MindMitra β€” How the Backend Actually Works

⚠️ Archaeology only. This document describes the legacy MITRA v2 architecture, which has been removed. The current runtime is the MHA v3 stack documented in ../html-to-markdown.md and ../CLAUDE.md. Use this file to understand how the service used to behave (e.g. when reading old PRs or migrating data); do not rely on it to reason about today's code.

Audience. A founder, a clinician, a designer, or a new engineer. No jargon, no hidden assumptions. Diagrams first, words second.


1. The 30-second pitch

MindMitra is an AI mental-health companion for Indian youth. A user opens the app, types or speaks a message, and gets back a warm, personalised reply within ~2 seconds. Behind the scenes a single backend service runs a five-stage pipeline that:

  1. understands what the user just said,
  2. checks for any safety signal (self-harm, suicide),
  3. recalls the right memories about this person,
  4. drafts a response using two AI passes (one fast, one careful),
  5. streams the answer back word-by-word and quietly writes new memories in the background.

Everything else in this doc is detail on those five steps.


2. The big picture (one diagram)

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”        HTTPS / SSE         β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚   Browser   β”‚ ─────────────────────────▢ β”‚  FastAPI (chatbotAgent)    β”‚
β”‚  (React)    β”‚ ◀───────────────────────── β”‚  /chat   /chat/stream      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   stream of text chunks    β”‚  /chat/greeting  /transcribeβ”‚
                                            β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                                         β”‚
                                  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                                  β”‚                      β”‚                      β”‚
                          β”Œβ”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                          β”‚  MITRA Pipeline β”‚    β”‚ Memory Services β”‚    β”‚ External LLMs    β”‚
                          β”‚  (one per turn) │◀──▢│  Postgres + Qdrant│◀──│ Azure / Gemini /β”‚
                          β”‚                 β”‚    β”‚                  β”‚   β”‚ Groq / GLM / BGE β”‚
                          β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                  β”‚
                          β”Œβ”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                          β”‚ Background workers     β”‚
                          β”‚ β€’ Consolidation worker β”‚
                          β”‚ β€’ Reflection worker    β”‚
                          β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Three things to remember from this picture:

  • The browser only ever talks to FastAPI. Everything else is private.
  • A single chat turn goes through one pipeline (the MITRA Pipeline).
  • Memory and background work happen next to the pipeline β€” not inside the user's response time.

3. What lives where (the file map)

chatbotAgent/
β”œβ”€β”€ app/
β”‚   β”œβ”€β”€ main.py                       # FastAPI startup, request middleware, CORS
β”‚   β”œβ”€β”€ api/
β”‚   β”‚   β”œβ”€β”€ chat.py                   # /chat, /chat/stream, /chat/greeting,
β”‚   β”‚   β”‚                             # /chat/end-session, /transcribe
β”‚   β”‚   β”œβ”€β”€ health.py                 # /health, /debug/memory
β”‚   β”‚   β”œβ”€β”€ onboarding.py             # First-time user data capture
β”‚   β”‚   β”œβ”€β”€ therapist_bridge.py       # Clinician handoff
β”‚   β”‚   └── me_memory.py              # GET/DELETE on a user's own memories
β”‚   β”œβ”€β”€ pipeline/
β”‚   β”‚   β”œβ”€β”€ crisis_fast_path.py       # 2-stage safety detector (lex + LLM)
β”‚   β”‚   └── mitra/
β”‚   β”‚       β”œβ”€β”€ orchestrator.py       # ⭐ runs ONE turn end-to-end
β”‚   β”‚       β”œβ”€β”€ classifier.py         # what is the user trying to do?
β”‚   β”‚       β”œβ”€β”€ retriever.py          # parallel memory fetch (deadlined)
β”‚   β”‚       β”œβ”€β”€ assembler.py          # builds the system + user prompt
β”‚   β”‚       β”œβ”€β”€ stance_selector.py    # tone / max-questions / disclosure rules
β”‚   β”‚       β”œβ”€β”€ generator.py          # two-pass draft β†’ critique β†’ repair
β”‚   β”‚       β”œβ”€β”€ dual_track.py         # optional Track-B: deeper rewrite
β”‚   β”‚       └── dispatch.py           # the route entry-point
β”‚   β”œβ”€β”€ memory/
β”‚   β”‚   β”œβ”€β”€ episodic.py               # write/search episodic memories
β”‚   β”‚   β”œβ”€β”€ qdrant_v2.py              # vector store wrapper (real + in-mem fake)
β”‚   β”‚   β”œβ”€β”€ repositories.py           # Supabase tables (CRUD)
β”‚   β”‚   β”œβ”€β”€ identity_card.py          # who is this person? (single JSON doc)
β”‚   β”‚   β”œβ”€β”€ working.py                # short-term sliding window (last N turns)
β”‚   β”‚   β”œβ”€β”€ affective.py              # mood time-series
β”‚   β”‚   β”œβ”€β”€ relational.py             # who appears in their life (graph)
β”‚   β”‚   β”œβ”€β”€ procedural.py             # what coping strategies they actually use
β”‚   β”‚   β”œβ”€β”€ relationship_state.py     # stage: stranger β†’ acquaintance β†’ trusted
β”‚   β”‚   β”œβ”€β”€ importance.py             # 0–1 score for "should we remember this?"
β”‚   β”‚   β”œβ”€β”€ decay.py                  # Ebbinghaus forgetting curve
β”‚   β”‚   └── salience.py               # blended scoring used at retrieval
β”‚   β”œβ”€β”€ jobs/
β”‚   β”‚   β”œβ”€β”€ extractor.py              # LLM extracts memory candidates from a turn
β”‚   β”‚   └── consolidation_worker.py   # offline: dedup + decay + reflect
β”‚   β”œβ”€β”€ providers/
β”‚   β”‚   β”œβ”€β”€ azure_openai_client.py    # gpt-5-mini (primary generator)
β”‚   β”‚   β”œβ”€β”€ gemini_client.py          # gemini-2.5-flash (track-B + fallback)
β”‚   β”‚   β”œβ”€β”€ groq_client.py            # llama-3.1 8B (cheap classifier)
β”‚   β”‚   β”œβ”€β”€ glm_client.py             # GLM-4-32B (long-context optional)
β”‚   β”‚   β”œβ”€β”€ embeddings_bge.py         # local BGE-M3
β”‚   β”‚   └── embeddings_gemini.py      # text-embedding-004 fallback
β”‚   β”œβ”€β”€ core/
β”‚   β”‚   β”œβ”€β”€ auth.py                   # Supabase JWT validation, SKIP_AUTH dev mode
β”‚   β”‚   β”œβ”€β”€ config.py                 # env config + model registry
β”‚   β”‚   β”œβ”€β”€ models.py                 # Roles β†’ ModelConfig mapping
β”‚   β”‚   β”œβ”€β”€ prompts/                  # stance, critic, crisis prompts
β”‚   β”‚   β”œβ”€β”€ pii.py                    # redact emails / phone before logs
β”‚   β”‚   β”œβ”€β”€ rate_limit.py             # per-user / per-route limits
β”‚   β”‚   β”œβ”€β”€ telemetry.py              # OpenTelemetry spans + SLO checks
β”‚   β”‚   └── logging.py                # ⭐ structured logs + banner helpers
β”‚   β”œβ”€β”€ services/
β”‚   β”‚   β”œβ”€β”€ supabase_service.py       # Postgres client + helpers
β”‚   β”‚   β”œβ”€β”€ greeting_service.py       # time-aware greeting line
β”‚   β”‚   β”œβ”€β”€ voice_prosody.py          # Praat-based prosody features
β”‚   β”‚   β”œβ”€β”€ helplines.py              # localised crisis helpline registry
β”‚   β”‚   └── therapist_*.py            # clinician-facing bridge
β”‚   └── utils/                        # json, constants, small helpers
β”œβ”€β”€ scripts/
β”‚   └── eval_locomo_lite.py           # memory eval harness
└── tests/
    β”œβ”€β”€ health/                       # boot + contract checks (no live LLMs)
    └── (integration tests)           # opt-in with RUN_INTEGRATION=1

The two stars (⭐) are the files you will open most often:

  • pipeline/mitra/orchestrator.py β€” the storyboard for a chat turn.
  • core/logging.py β€” every log line in the terminal flows through here.

4. One chat turn, end-to-end

This is the single most important diagram in the codebase.

sequenceDiagram
  autonumber
    participant U as πŸ‘€ User (browser)
    participant API as FastAPI /chat/stream
    participant ORCH as MITRA orchestrator
    participant CLS as Classifier (Groq llama-3.1 8B)
    participant CRI as Crisis fast-path
    participant RET as Retriever (parallel)
    participant ASM as Prompt assembler
    participant GEN as Generator (Azure gpt-5-mini)
    participant CW as Consolidation worker (background)
    participant DB as Supabase + Qdrant

    U->>API: POST /chat/stream { message, session_id }
    API->>API: validate JWT, log "πŸ“₯ POST /chat/stream"
    API->>ORCH: run_mitra_turn(...)

    par classify and check safety in parallel
        ORCH->>CLS: what intent + needs_memory?
        ORCH->>CRI: lexical scan; if ambiguous β†’ LLM confirmer
    end

    alt 🚨 crisis triggered
        ORCH-->>API: localised crisis safety reply
        API-->>U: SSE chunks (helpline + grounding)
    else πŸ™‚ normal turn
        ORCH->>RET: fetch identity_card + episodes + affect + procedural
        RET->>DB: parallel reads (deadlined ~600ms)
        ORCH->>ASM: build system + user prompt with stance constraints
        ORCH->>GEN: two-pass: draft β†’ critic β†’ repair (optional Track-B)
        GEN-->>API: stream chunks via stream_callback
        API-->>U: SSE chunks (sentence-buffered)
        ORCH->>DB: write turn trace + working memory append
    end

    Note over API,CW: After response, in background:
    API->>CW: every N messages β†’ run_once_for_user
    CW->>DB: extract candidates β†’ dedup β†’ write episodic memories
Loading

The five stages, in plain English

# Stage What it does Where it lives
1 Classify + safety Looks at the message and asks two questions in parallel: (a) what is the user trying to do β€” vent, share, ask for advice, etc.? and (b) is this a safety situation? mitra/classifier.py + pipeline/crisis_fast_path.py
2 Crisis short-circuit If safety says hard, stop everything. Reply with the localised helpline + grounding script. No memory reads, no LLM generation. pipeline/crisis_fast_path.py
3 Retrieve Pull only what the next prompt actually needs: who this person is (identity card), their last few highly-relevant memories, their recent mood pattern, what coping moves have worked before. All four reads run in parallel, with a hard ~600 ms deadline. mitra/retriever.py + memory/*
4 Assemble + stance Build a system prompt that says "you are Mitra, this user is X, here are 3 relevant memories, your tone today is holding not advising, ask at most 1 question". mitra/assembler.py + mitra/stance_selector.py
5 Generate First pass writes a draft. A tiny critic checks for style violations (too long, too clinical, too many questions, missed grounding…). If anything fails, the model rewrites. The critic is bounded β€” at most 1 repair pass on the hot path. mitra/generator.py, optional mitra/dual_track.py

After the response is sent, two background things happen so the user never waits for them:

  • Consolidation worker (jobs/consolidation_worker.py) extracts new memory candidates from the last few turns, dedupes against existing memories, scores them with importance.py, and writes the survivors to Postgres + Qdrant.
  • Reflection (also in the worker) every few sessions writes a cross-session insight ("they cope better when their sister is around").

5. The brains: which model does what

Different jobs need different models. The mapping lives in core/models.py and is the single source of truth.

Role Model Provider Why this one
Classifier (intent + needs-memory) llama-3.1-8b-instant Groq Fast, cheap, structured-output friendly
Crisis confirmer (LLM second-stage) gemini-2.5-flash Gemini Low latency, multilingual
Generator (primary) gpt-5-mini Azure OpenAI Best instruction-following + true streaming
Generator (Track B / fallback) gemini-2.5-flash Gemini Long context, cheaper rewrite path
Long-context (only when prompt > 32k) glm-4-32b-0414-128k Z.AI Cheap 128k window
Memory extractor (offline) llama-3.1-8b-instant Groq Cheap; runs in background
Embeddings (primary) BGE-M3 local Multilingual, no per-call cost
Embeddings (fallback) text-embedding-004 Gemini Drop-in if BGE host is down
Speech-to-text whisper-large-v3-turbo Groq Cheap fallback when Azure STT returns empty

If you swap a model, change exactly one row in core/models.py. The pipeline picks it up on the next startup.


6. Memory: how Mitra "remembers" you

Memory is the hardest part to explain, so we use a simple metaphor: a notebook, a journal, and a habit log.

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                        Per-user memory model                        β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Identity cardβ”‚ One short JSON: name, pronouns, languages, key       β”‚
β”‚ (the cover)  β”‚ relationships, soft preferences. Read every turn.    β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Episodic     β”‚ Up to a few hundred short "moments" per user.        β”‚
β”‚ (the journal)β”‚ Each has: summary, importance, mood (V-A-D), tags,   β”‚
β”‚              β”‚ embedding. Lives in Postgres + Qdrant.               β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Affective    β”‚ Time-series of mood scores over the last N days.     β”‚
β”‚ (the mood    β”‚ Used to detect patterns ("worse on Sundays").        β”‚
β”‚  graph)      β”‚                                                      β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Relational   β”‚ Tiny graph: people in the user's life, their roles.  β”‚
β”‚ (who's who)  β”‚ Stops Mitra from confusing "didi" with "boss".       β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Procedural   β”‚ Coping strategies that the user has tried + worked.  β”‚
β”‚ (the habit   β”‚ Used by the assembler to suggest things they already β”‚
β”‚  log)        β”‚ trust.                                                β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Working      β”‚ A sliding window of the last few turns of THIS chat. β”‚
β”‚ (scratchpad) β”‚ In-process; flushed at session end.                  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Relationship β”‚ A single label per user: stranger β†’ acquaintance β†’   β”‚
β”‚ state        β”‚ trusted β†’ close. Controls how much self-disclosure   β”‚
β”‚              β”‚ the assistant offers.                                β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

How a memory is born and how it dies

flowchart LR
    A[Chat turn happens] -->|inline trace| B[Working memory<br/>append]
    A -.->|every N turns| C[Consolidation worker<br/>kicks off in background]
    C --> D[LLM extractor<br/>jobs/extractor.py]
    D --> E[Importance score<br/>memory/importance.py]
    E -->|score &lt; 0.3| X[Drop]
    E -->|score β‰₯ 0.3| F[Dedupe vs existing<br/>cosine similarity]
    F -->|duplicate| G[Reinforce existing<br/>+1 strength]
    F -->|new| H[Write to Postgres<br/>+ upsert into Qdrant]
    H --> I[Decay over time<br/>memory/decay.py]
    I -->|strength β†’ 0| J[Archive]
    H -->|recalled in a future turn| K[mark_recalled<br/>strength bump]
Loading

Two things to internalise:

  • Memories get stronger every time we use them. Recall = reinforcement.
  • Memories that are never recalled fade. This is by design β€” it prevents the database from drowning the model in stale context.

7. Safety: the crisis fast-path

Safety is the only stage that is allowed to skip the rest of the pipeline. If anything looks like self-harm or acute suicidal ideation, we never run the generator.

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ User message arrives     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
               β”‚
               β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Stage 1 β€” lexical scan   β”‚  multilingual phrases, regex,
β”‚ (instant, no network)    β”‚  romanised Hindi included
β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜
       β”‚ safe       β”‚ ambiguous          β”‚ hard
       β–Ό            β–Ό                    β–Ό
   continue     Stage 2 β€” Gemini     CRISIS FAST-PATH
   pipeline    confirmer (≀ 600ms)   β€’ play helpline message
                    β”‚                  β€’ do grounding script
                    β–Ό                  β€’ log crisis_event
              triggered? ──── yes ──▢  β€’ return immediately
                    β”‚
                    β–Ό
                  no β†’ continue pipeline

Notes:

  • The lexical list is hand-curated, multilingual, and intentionally over-flags. The LLM second stage cleans up false positives.
  • The fast-path never waits on memory or generation.
  • Every trigger writes a row to crisis_events for audit + clinician follow-up.

8. Streaming: how the words actually reach the browser

The chat UI is the part most likely to feel broken if a single piece of plumbing is wrong, so this section is precise.

Provider (Azure / Gemini)
        β”‚  yields text deltas
        β–Ό
azure_openai_client._complete_stream
   β€’ producer runs in a worker thread
   β€’ puts deltas into asyncio.Queue
   β€’ CRITICAL: producer is fire-and-forget
     (asyncio.create_task, NOT awaited)
        β”‚
        β–Ό
mitra/dispatch._make_llm_caller
   β€’ async-iterates the queue
   β€’ calls stream_callback(delta) for each chunk
        β”‚
        β–Ό
api/chat.py event_generator
   β€’ on_chunk pushes into another asyncio.Queue
     (loop.call_soon_threadsafe β€” thread safe)
   β€’ main loop reads queue, buffers until a
     sentence boundary, then yields:
       event: text_chunk_delta
       data: {"chunk": "..."}\n\n
        β”‚
        β–Ό
Browser EventSource
   β€’ parses each `data:` line as JSON
   β€’ appends `chunk` to the live message bubble

If you ever see "the response only appears at the end", the bug is almost certainly in step 2 β€” someone re-introduced an await before the producer task. The fix is asyncio.create_task(asyncio.to_thread(...)), never await asyncio.to_thread(...).


9. Logging: what you should see in the terminal

Every important step in the backend now prints a coloured banner or a single-line stage marker. A healthy chat turn produces logs like:

2026-04-20 14:01:12 [INFO] app.api.chat req=ab12 ─── πŸ“₯ POST /chat/stream
                                                      user        : a0778b19
                                                      session     : 9f3d
                                                      persona     : mitra
                                                      language    : en
                                                      avatar      : True
                                                      msg_len     : 47 chars
2026-04-20 14:01:12 [INFO] app.mitra   req=ab12 ─── 🧠 MITRA TURN START
                                                      ...
2026-04-20 14:01:12 [INFO] app.mitra   req=ab12 [STAGE] classify+safety intent=vent safety=safe crisis=False dur_ms=180
2026-04-20 14:01:12 [INFO] app.mitra   req=ab12 [STAGE] retrieve_memories episodes=4 identity_card=True affect=low_mood dur_ms=210
2026-04-20 14:01:12 [INFO] app.mitra   req=ab12 [STAGE] select_stance stance=holding max_q=1
2026-04-20 14:01:13 [INFO] app.mitra   req=ab12 [STAGE] assemble_prompt sys_chars=2310 user_chars=412 episodes_used=3 dur_ms=8
2026-04-20 14:01:13 [INFO] app.mitra   req=ab12 [STAGE] generate_response generator=TwoPassGenerator stream=True
2026-04-20 14:01:14 [INFO] app.mitra   req=ab12 [STAGE] generate_done chars=312 accepted_on_pass=1 fallback=False dur_ms=1100
2026-04-20 14:01:14 [INFO] app.mitra   req=ab12 ─── βœ… MITRA TURN COMPLETE
                                                      intent      : vent
                                                      stance      : holding
                                                      chars       : 312
                                                      episodes    : 3
                                                      accepted_on : pass 1
                                                      fallback    : False
                                                      timings_ms  : classify_ms=180 retrieve_ms=210 assemble_ms=8 generate_ms=1100 total_ms=1500
2026-04-20 14:01:15 [INFO] app.api.chat req=ab12 πŸ§ͺ [CONSOLIDATE] user=a0778b19 candidates=2 written=1 archived=0 reflections=0

Three things make this useful in production:

  • req= correlation id β€” every log line in one HTTP turn shares the same request id (set by middleware in main.py, propagated via ContextVar and bind_request_context).
  • Banner + stage style β€” banners frame the start/end of a unit of work; stages are dense single-line markers between them.
  • Background threads inherit context. Use spawn_correlated_thread (from core/logging.py) instead of threading.Thread directly β€” the consolidation worker uses this so its logs still carry req=.

Set LOG_FORMAT=json in production to get structured JSON instead of the colour banners β€” same fields, different renderer.


10. Configuration & feature flags

Variable Effect Default
MITRA_STACK_ENABLED Master switch. If off, /chat returns 503. 1 (on)
MITRA_CRISIS_V2_ENABLED Use the 2-stage crisis detector. 1
MITRA_DUAL_TRACK_ENABLED Run optional Track-B rewrite for tougher turns. 0
MITRA_CONSOLIDATION_INTERVAL Run consolidation every N user turns. 12
LOG_LEVEL INFO / DEBUG / WARNING. INFO
LOG_FORMAT colour (dev) or json (prod). colour
SKIP_AUTH Dev only. Replaces JWT validation with DEV_USER_ID. unset
QDRANT_URL Vector store. required in prod
SUPABASE_URL / SUPABASE_SERVICE_KEY Postgres + auth. required in prod

The full list lives in chatbotAgent/.env.example.


11. Common questions

Q: Where do I add a new "intent"? mitra/classifier.py defines Intent. Add the value, update the classifier prompt, and decide in stance_selector.py what stance + max questions that intent should trigger.

Q: Why are there two "generators"? The default is TwoPassGenerator β€” one draft, one critic, optional repair. The richer DualTrackGenerator runs an additional Track-B rewrite for emotionally heavy turns. Both implement the same generate() interface so the orchestrator doesn't care which is wired.

Q: How do I disable memory entirely for a test? Pass an in-memory Qdrant (InMemoryQdrant from memory/qdrant_v2.py) and a no-op EpisodicRepo. The retriever degrades gracefully β€” empty context is fine.

Q: What happens if Azure / Gemini is down? The provider raises; the orchestrator catches and returns a soft fallback message and logs fallback=True. The critic also has a "soft fallback" template it can return without an LLM call.

Q: Why isn't there a workflow.py any more? There used to be a legacy linear pipeline. It was deleted in this migration. All chat now flows through pipeline/mitra/.


12. Where to read next

  • docs/platform.md β€” memory v2 modules, Qdrant, pipeline, runbook.
  • docs/EVALUATION.md β€” how we test (pytest + LOCOMO-lite + LLM judge).
  • docs/api_contracts.md β€” every HTTP endpoint, request, response.
  • docs/product.md β€” MindGym, therapist bridge, analytics, security headers.
  • chatbotAgent/app/pipeline/mitra/orchestrator.py β€” when in doubt, read the orchestrator. It is intentionally short.

13. Future Work

This section documents deferred improvements with concrete recipes. Anyone picking these up should be able to ship them in one PR.

13.1 Re-enabling voice prosody (Praat / parselmouth)

Prosody analysis is OFF on the chat path (April 2026). The LLM never consumed the features and import parselmouth has hung the FastAPI lifespan under macOS Gatekeeper while holding the GIL. The synchronous analyser still lives in app/services/voice_prosody.py for offline jobs.

When you actually want the LLM to see acoustic features, follow this 3-step recipe β€” never call analyze_prosody from an async handler:

  1. Carry the field through the pipeline. Add a prosody: Optional[Dict[str, Any]] field to TurnInput (app/pipeline/mitra/orchestrator.py) so the structured features reach the assembler.
  2. Render it into the system prompt. Add a small block in the assembler (app/pipeline/mitra/assembler.py) that, when prosody is present, calls from app.services.voice_prosody import format_prosody_for_prompt and inlines the resulting block under a ## Acoustic context heading. Keep it short β€” one paragraph.
  3. Extract Praat in a sandboxed subprocess. Create app/tools/prosody_cli.py that reads a base64 WAV from stdin, runs analyze_prosody, and writes JSON to stdout. From the chat endpoint: subprocess.run(["python", "-m", "app.tools.prosody_cli"], input=wav_b64, capture_output=True, text=True, timeout=4). If the timeout fires you kill the subprocess β€” the FastAPI process is unaffected.

Also re-enable the dependency: uncomment praat-parselmouth==0.4.7 in chatbotAgent/requirements.txt and re-add the audio_data field to the frontend POST in src/components/chat/ChatGPTInterface.tsx.

13.2 First-byte SLO

The backend now forwards every provider delta straight to the SSE stream β€” no sentence-boundary buffer (app/api/chat.py event_generator). First audible byte should land within ~1.5 s of the request reaching the orchestrator on a healthy Azure connection.

If you ever re-introduce server-side buffering (sentence batching, JSON chunking, etc.), gate it behind a feature flag and document the trade-off here. Frontend Presence mode already does its own Unicode-aware sentence segmentation in src/components/chat/ChatGPTInterface.tsx (COMPLETED_SENTENCES_RE).

13.3 Production deployment checklist

  • SKIP_AUTH=0 β€” the dev override bypasses Supabase JWT validation.
  • Multi-worker uvicorn: uvicorn app.main:app --workers 4 --worker-class uvicorn.workers.UvicornWorker.
  • Wire app/core/rate_limit.py into the chat router (currently defined but not enforced).
  • Apply the mitra_turn_traces schema migration so accepted_on_pass, fallback_used, stance, callback_budget, critic, used_episodes exist as real columns. The repository's schema-drift fallback then becomes a no-op (no warning, no JSON demotion).
  • Set OTEL_SERVICE_NAME so spans aren't no-op.
  • Validate Gemini quota and consider flipping MITRA_DUAL_TRACK_ENABLED=1 for parallel speculative generation.
  • Pick a faster GENERATOR_PRIMARY model if gpt-5-mini p95 latency remains > 3 s β€” gpt-4o-mini and Groq Llama-3.1-70B are reasonable candidates with their own quality bake-offs.

13.4 Architectural rule β€” native libs and the request hot path

Never import a native-code library at module top-level inside a route handler, the FastAPI lifespan, or any code path reachable from an async request. Native dynamic loaders (dyld on macOS, ld-linux on Linux) can stall on signature checks, license probes, or model-file downloads while holding the GIL β€” and a single stalled import freezes the entire ASGI process.

If you must use such a library:

  • Wrap it in a CLI subprocess called via subprocess.run(..., timeout=N) so a hang can be killed without taking down the server.
  • Or run it in a one-shot worker (spawn_correlated_thread, concurrent.futures.ProcessPoolExecutor) gated by a hard timeout and a circuit breaker.
  • Or push it entirely off-line into a job (app/jobs/) that the chat path never awaits.

This rule is what got broken when prosody was inlined into the chat endpoint. Treat any new native dependency (Whisper, faiss, llama-cpp, sentence-transformers, etc.) the same way.


Last revised: April 2026. If anything in this doc disagrees with the code, the code is right β€” please update this file in the same PR.