Skip to content

AGENTS.md

Latest commit

 

History

History
106 lines (75 loc) · 26 KB

AGENTS.md

File metadata and controls

106 lines (75 loc) · 26 KB
Raw

AGENTS.md

This file provides guidance to agents when working with code in this repository.

Project Overview

Mimora is a local, offline pronunciation trainer (Python 3.11/3.12, Tkinter GUI). It speaks an LLM-generated phrase aloud (TTS backend per language: Kokoro for English, Supertonic 3 for Spanish - see mimora/tts.py), records the user repeating it, then scores the attempt against the reference with the active pronunciation engine plus prosody. The engine is selected by config.ENGINE (settings.json "engine"; see mimora/engine.py): the default is phoneme (espeak reference phonemes vs a wav2vec2 phoneme recognizer, feature-weighted edit distance, calibrated 0–5 grade); the alternative acoustic engine uses Wav2Vec2-embedding cosine-DTW plus phoneme/word error rates; none disables scoring entirely (no recognizer model is loaded, every take is accepted, the GUI shows a neutral "scoring off" read-out - for slow machines). The user repeats the same phrase until the score passes, then generates the next one.

The practice language is data, not an assumption: every language is one entry in config.LANGUAGE_PROFILES (display name, FLORES-200 code, engines, variants with Kokoro/espeak wiring and voices, phrase-generation prompts, startup greeting, preview/warm-up phrases, default practice text and its unreadable-file fallback). The active language/variant comes from settings.json "practice_language" / "accent" (the legacy "english_accent" is read as a fallback and dropped on the next save); both apply after a restart. English (variants american/british) is fully calibrated; Spanish (variant castilian) runs the phoneme engine as experimental until pronunciation/phoneme/es_model_calibration.json lands (scoring falls back to the English calibration; the app logs a startup warning and the settings window shows a notice - see config.engine_experimental). The English-only acoustic engine is offered per profile (config.available_engines); an unavailable engine falls back to the language's first available one. Adding a language = one profile module in mimora/languages/ (a pure-data PROFILE dict) registered in the LANGUAGE_PROFILES assembly in config.py, plus an engine calibration, never a new if language == ... branch (see tasks/multilingual_support_task.md).

The pronunciation-scoring core in pronunciation/acoustic/ is adapted from OpenPronounce (MIT) and reused as a GUI-agnostic library.

Running the App

# Root requirements.txt pulls in the subproject files via -r (llm_server,
# pronunciation/acoustic and pronunciation/phoneme), so one install covers all:
pip install -r requirements.txt
python main.py

Also requires the native espeak-ng binary on PATH (used by phonemizer) and a GGUF chat model at config.EXTERNAL_MODEL_PATH.

Default LLM backend: local_server - llm_server/server.py is launched automatically as a subprocess. Alternatives: set "llm_backend": "lm-studio" in config/settings.json and run LM Studio on http://localhost:1234 (or on another machine in the local network - set "lm_studio_host" to its address as host or host:port, port defaults to 1234; also editable in the settings window); or "llm_backend": "off" - no LLM is loaded or started at all, practice phrases are the source text's own sentences taken verbatim in order (mimora/phrase_source.py); the phrase-length choice is disabled in the settings window in this mode (sentences are never shortened into fragments).

Architecture

  • main.py - PronunciationTrainerGUI: Tkinter GUI, recording, the Prompt→Record→Analyze→Feedback→Loop state machine, threading orchestration; the LLM-server subprocess lifecycle is delegated to mimora/llm_server_ctl.py.
  • mimora/bootstrap.py - early process setup for the entry point, stdlib-only. early_init() runs BEFORE the heavy mimora.* imports in main.py (UTF-8 console/env hints, HF symlink-warning suppression, library warning filters); setup_logging(log_file) runs AFTER them (basicConfig with force=True replaces handlers auto-installed during the imports). The two-phase split is load-bearing - do not merge or reorder the calls.
  • mimora/session.py - SessionState: per-run score tally (distinct-phrase count, mean of per-phrase best scores with raw/graded formatting, the current phrase's attempt list for the ring's dot column) and the bounded attempt history with trend arrows. Pure data, no Tk; main.py owns the instance and forwards its outputs to the view (update_session_stats / render_history). Unit-tested in tests/test_session.py.
  • mimora/playback.py - PlaybackController: the per-playback stop-event lifecycle plus the talking-mouth coupling. new_event() installs a fresh stop event per playback (Tk main thread only), stop() supersedes the current one, play_with_face() is the blocking chokepoint around TTSManager.play_array that also drives the FaceWidget's loudness track, play_async() the fire-and-forget wrapper, finished() the identity-guarded Ready-status restore. Composed by main.py with the Tk root, the view facade, the TTSManager and the shutdown event.
  • mimora/ui.py - TrainerView: the view facade main.py composes. Owns the window chrome (header, status bar, tip line), the control row with the mic button, the enter_* intent methods and the feedback orchestration (show_feedback), and delegates panel-local work to per-panel classes: mimora/ui_practice.py (PracticePanel, collapsible source-text editor), mimora/ui_hero.py (HeroCard: phrase with clickable words, translation, and the score row - the FaceWidget on the left, then the SCORE column and WORK ON badges, and the ProgressRing on the right - session average of per-phrase bests plus the current phrase's attempt dots), mimora/ui_prosody.py (ProsodyPanel, pitch/energy sparklines with the cached last result) and mimora/ui_history.py (HistoryPanel, scrollable attempt list). Shared palette/fonts/wheel helpers and the hover tooltip live in mimora/ui_theme.py (importing it also disables ttkbootstrap's classic-widget autostyle hook). The controller-facing API is unchanged by the split; ui.py re-exports THEME, FONT_FAMILY, WHEEL_EVENTS and wheel_scroll_units for settings_window.py.
  • mimora/engine.py - engine dispatcher: binds the backend chosen by config.ENGINE (phoneme default, acoustic alternative, none = scoring off) and exposes one analyze(...) interface, so main.py is engine-agnostic and only the selected engine's weights load. pronunciation/none/ is a no-op engine returning PronunciationResult(scored=False, passed=True); the GUI renders unscored results neutrally (ui.py _show_unscored_feedback).
  • pronunciation/phoneme/speech.py - default pronunciation engine (text-only reference): espeak reference phonemes vs a wav2vec2 phoneme recognizer, feature-weighted edit distance, mapped to a calibrated 0–5 grade. Model calibration lives in pronunciation/phoneme/<lang>_model_calibration.json (committed, selected by espeak language); a per-user phoneme_good override in pronunciation/phoneme/calibration.json (gitignored, shaped {lang: {users: {name: ...}}}). Samples appended to logs/phoneme_samples.jsonl. No GUI dependency.
  • pronunciation/acoustic/speech.py - alternative pronunciation engine (adapted from OpenPronounce). Single entry point analyze(user_audio, expected_text, reference_audio) -> PronunciationResult. Wav2Vec2 embeddings + per-step cosine DTW, phoneme/word error rates. Scoring uses a calibratable acoustic floor (pronunciation/acoustic/calibration.json overrides config.PRONUNCIATION_ACOUSTIC_GOOD); every attempt's raw components are appended to logs/acoustic_samples.jsonl. No GUI/Tkinter dependency. Prosody is no longer computed here (returns prosody={}) - see mimora/prosody.py.
  • mimora/prosody.py - engine-agnostic prosody layer: F0/energy contour extraction (librosa/sklearn, no torch). main.py calls compute_prosody(user, reference) after analyze (via _compute_prosody_safe, skipped entirely while the prosody block is collapsed (show_prosody) - pyin pitch tracking is expensive on slow machines) and fills result.prosody, so the pitch/energy charts work identically regardless of the active engine. Pure plotting helpers (to_semitones, resample_series) stay in mimora/prosody_utils.py.
  • pronunciation/acoustic/calibrate.py - on-request semi-automatic calibration: fits the acoustic floor from collected samples and writes pronunciation/acoustic/calibration.json (--dry-run to preview).
  • mimora/audio_io.py - shared audio-device infrastructure depended on by both the mic and speaker paths (so neither depends on the other). Exports reset_portaudio() (shared PortAudio reset used by recording and playback), uses_winsound() (single source of truth for which playback path is taken), WINSOUND_AVAILABLE and the winsound lead-in constants. The coordinating AUDIO_LOCK and pipeline AUDIO_SAMPLE_RATE stay in config.py with the other AUDIO_* settings. (The synthesis rate is no longer a constant here: it is a property of the active TTS backend, TTSManager.sample_rate.)
  • mimora/tts.py - TTSManager: facade over per-variant synthesis backends selected by profile data (config.TTS_BACKEND, registry TTS_BACKENDS): KokoroBackend (torch, 24 kHz; English) and SupertonicBackend (ONNX, 44.1 kHz; Spanish - weights OpenRAIL-M, cached in model_cache/supertonic3/ via SUPERTONIC_CACHE_DIR). synthesize() returns the waveform at TTSManager.sample_rate (the backend's native rate - main.py reads this property everywhere instead of a constant); play_array(waveform, sample_rate) plays any waveform. Also exports loudness_envelope(waveform, sample_rate, fps) (per-frame RMS → 0..1 track that drives the talking mouth - see face_widget.py). The PortAudio/winsound device helpers it uses live in mimora/audio_io.py.
  • mimora/face_widget.py - FaceWidget: cartoon articulation face shown on a Tk Canvas. A filled talking mouth opens/closes during playback; a smiley (plus eyebrows and blinking) reflects the score while idle. Frames are rendered by Pillow at 4x supersampling (LANCZOS downscale = antialiasing Tk Canvas lacks) and cached per quantized state, so steady-state animation is an image swap. Deps: stdlib tkinter + Pillow. The mouth is driven by play_levels(levels, fps) - a pre-computed loudness track the widget's own after-loop replays by wall-clock - rather than a live audio callback (see the Windows-audio note below).
  • mimora/progress_widget.py - ProgressRing: circular session-average gauge on the right of the hero score row (mirrors the face on the left), with a vertical attempt-dot column on its left flank - one dot per take of the current phrase, bottom-up, coloured by quality (theme good/warn/bad from the score fraction); only the last few fit (column height = ring), older dots drop off the bottom. A tk.Frame holding a dots Canvas, the ring Canvas and a count Label (grid, so the count centres under the ring). The ring (track + fill arc with round caps) and the dot column are rendered by Pillow at 4x supersampling (LANCZOS downscale for the antialiasing Tk Canvas lacks); the numbers (average, "/ max", phrase count) are plain Tk text in the app font, so no TrueType font file is needed. Driven by set_progress(value, maximum, count) and set_attempts(scores) (empty list clears the dots - ui.py enter_reference_playing does this per new phrase); starts empty ("0 phrases", no fill, no dots). The file is named generically on purpose - ProgressRing is the current shape; a later redesign swaps the class without moving the import. Deps: stdlib tkinter + Pillow. This holds the session tally that previously sat in the status bar (see ui.py update_session_stats).
  • mimora/lifecycle.py - process-exit helpers, Tk-free: hard_exit() (TerminateProcess on Windows to dodge the CUDA DLL-detach crash, os._exit elsewhere) and spawn_replacement() (detached self-relaunch for the settings restart). main.py keeps the orchestration: quit_app / restart_app run _shutdown_runtime first (playback, recorder, LLM-server subprocess), then call these.
  • mimora/llm.py - LLMManager: OpenAI-compatible client. generate_phrase() produces one practice phrase per request (non-streaming).
  • mimora/llm_server_ctl.py - LLMServerController: lifecycle of the local LLM-server subprocess (the local_server backend). start(llm_mgr) launches llm_server/server.py and blocks until it answers (readiness polled via LLMManager.check_connection); shutdown() terminates it with a 5-second kill fallback and is safe to call from two threads at once (loader thread + Tk thread in quit_app).
  • mimora/recorder.py - AudioRecorder: microphone capture thread with silence-based auto-stop, device selection, normalization and the diagnostic WAV dumps; returns the take as one 16 kHz mono array.
  • mimora/translator.py - TranslatorManager: offline NLLB-200 translation of the practice phrase for the translation panel (loaded lazily, CPU by default - config.TRANSLATOR_DEVICE).
  • mimora/loader.py - pure, stateless config-loading helpers (JSON parsing, setting validation, device probe) used by config.py. Unit-tested in tests/test_loader.py.
  • mimora/phoneme_examples.py - static IPA phoneme → example word table backing the "WORK ON" badges (tooltip text and the example word spoken on a badge click), per language.
  • mimora/phrase_source.py - SourceTextPhraseProvider: the "off" LLM backend's phrase source, stdlib-only (importable without openai). Duck-typed drop-in for LLMManager in the generation path (main.py only calls generate_phrase on it): returns the source text's sentences verbatim, sequentially with wraparound; a text edit (detected by hash) restarts from the first sentence. Also home of the shared split_sentences helper LLMManager._split_sentences delegates to. Unit-tested in tests/test_phrase_source.py.
  • mimora/config.py - all configuration (device, model names, score threshold, practice-text path, phrase-generation settings, audio settings) plus the language model: LANGUAGE_PROFILES (language → profile with variants, engines, prompts, texts), assembled from the per-language pure-data modules in mimora/languages/ (one PROFILE dict each, e.g. english.py, spanish.py), and the derived per-run constants (PRACTICE_LANGUAGE, ACCENT, TARGET_LANGUAGE, ESPEAK_LANGUAGE, TTS_*, SOURCE_FLORES_CODE, PHONEME_EXPERIMENTAL). Public helpers: language_choices(), accent_choices(language), accent_voices(accent, language), accent_default_voice(accent, language), default_accent(language), available_engines(language), engine_experimental(engine, language, accent), translation_targets(language). User overrides live in config/settings.json; UI themes in config/themes/. Unit-tested in tests/test_language_profiles.py.
  • mimora/settings_ctl.py - SettingsGlue: the persistence mechanics around settings.json - persist (with suppression during reset), persist_and_apply_live (table-driven live config attrs), sync_window (mirror a main-window change into the open settings dialog) and reset_to_defaults (the "Default" button, dispatching only defaults that differ from the live values). Owns the LIVE_CONFIG_ATTRS / SETTING_LIVE_ATTRS tables; the dispatch itself (on_setting_changed, the on_*_changed handlers) stays in main.py, injected as callbacks. Unit-tested in tests/test_settings_ctl.py.
  • mimora/settings_window.py - settings dialog (gear button in the header). Declarative model: every editable settings.json key is one Field (kind, choices, range, restart flag) grouped into Sections; SettingsWindow renders them into a scrollable column and stays passive - changes go to main.py on_setting_changed, which applies live effects and persists once via mimora/settings_ctl.py (restart-only keys are just persisted). Restart-only changes surface a "Restart now" offer that relaunches the process (main.py restart_app). The Language field lists config.language_choices(); the Accent field follows the selected language and is hidden when it has a single variant; the Engine field offers only config.available_engines; an experimental notice appears when the selected engine+language runs uncalibrated (config.engine_experimental); "Random voice per phrase" is disabled when the running variant has fewer than two voices. Includes a per-voice "Listen" preview (running language/variant only), "Cancel" (reverts everything changed while the window was open, then closes) and "Default" (SettingsGlue.reset_to_defaults: removes every override from settings.json - config.USER_SETTING_DEFAULTS / reset_user_settings - and applies the built-in defaults live without re-persisting them).
  • llm_server/server.py - standalone FastAPI server loading GGUF via llama_cpp; runs as a separate process to avoid GPU contention.
  • texts/practice_text.txt - default source text loaded into the input panel at startup; the default path comes from the active language profile (texts/practice_text_es.txt for Spanish). texts/ holds additional practice texts.

State Machine (pronunciation loop)

  1. Prompt - llm_mgr.generate_phrase(source_text) → tts_mgr.synthesize(phrase). The synthesized array is stored as self.reference_audio and played for the user. Phrase generation + synth + playback all run in one daemon thread (_generate_and_prompt).
  2. Record - shared recording path (AudioRecorder._record_loop → AudioRecorder.get_audio), 16 kHz mono. Gated by _can_record() (a phrase must be ready and nothing else busy).
  3. Analyze - _finalize_recording → analyze_recording (daemon thread) calls engine.analyze(...) - the dispatcher in mimora/engine.py, which selects pronunciation.acoustic or pronunciation.phoneme by config.ENGINE.
  4. Feedback - _show_feedback (via root.after) fills the hero card (score, verdict, "WORK ON" phoneme badges, underlined problem words in the phrase) and appends the take to the attempt-history list; enables replay buttons.
  5. Loop - the user decides when to move on: generate the next phrase, or record the same one again (the phrase/reference are retained until a new phrase is generated). result.passed is not enforced yet - no code in main.py/ui.py reads it, so nothing auto-advances or forces a repeat based on the verdict. See the Pass-threshold note under Key Patterns & Gotchas.

Key Patterns & Gotchas

  • Threading: Recording, analysis, model loading, phrase generation, and playback run in daemon threads. Always update the GUI via root.after(); never read/write Tk widgets from a background thread. Source text is read on the main thread and passed into the worker.

  • Reference audio is synthesized once (mimora/tts.py synthesize()): the same waveform is both played to the user and passed to analyze() as the reference. Only ONE synthesis backend runs per session (the one the active variant selects); the backends never mix within a run.

  • Sample rates: recording uses 16 kHz; the TTS backend outputs its native rate (Kokoro 24 kHz, Supertonic 44.1 kHz) - main.py reads it from TTSManager.sample_rate, never a constant; Wav2Vec2 needs 16 kHz. engine.analyze takes user_sr and reference_sr and _prepare_waveform resamples to 16 kHz internally. play_array plays the reference at the backend rate and the user recording at 16 kHz.

  • Pronunciation model lifecycle (pronunciation/acoustic/speech.py): models load lazily; load_models() makes loading explicit (call in a background thread at startup) and warm_up() removes first-call latency - mirroring mimora/tts.py. Device follows config.WAV2VEC2_DEVICE (defaults to config.DEVICE). speech.py reads config via getattr(..., default) so it stays usable without config edits.

  • Pass threshold / result.passed are currently inert (reserved for a future gating loop): config.PRONUNCIATION_SCORE_THRESHOLD (settings.json "pronunciation_score_threshold"; not exposed in the settings window) feeds each engine's score_threshold and produces result.passed - the acoustic engine always as score >= threshold (pronunciation/acoustic/speech.py), the phoneme engine only as a fallback when no calibrated bucket exists (pronunciation/phoneme/speech.py; normally passed = bucket >= PASS_BUCKET, so the threshold is bypassed on the default calibrated path). But result.passed is not consumed by the app: ui.py show_feedback derives the score read-out, quality band, face and history from the score/bucket, not from passed, and the phoneme feedback string ("(passed)"/"(try again)") is never rendered. So today the Pass-threshold value has no visible effect - it is computed and logged only, kept as a hook for a future pass/repeat gate. When that gate is implemented, wire it to result.passed (and account for the phoneme calibrated-bucket path, where the raw threshold does not apply).

  • GPU contention: Wav2Vec2, Kokoro, and llama_cpp can compete for VRAM. Mitigations: the LLM runs in a separate process (llm_server/), and the loop's phases (LLM → Kokoro → Wav2Vec2) run sequentially. If VRAM is tight, set WAV2VEC2_DEVICE = "cpu" in mimora/config.py.

  • Phrase generation (mimora/llm.py): generate_phrase() is a stateless, non-streaming completion with its own system prompt (config.PHRASE_GEN_SYSTEM_PROMPT, a {min_words}/{max_words} format template); each request builds its messages from scratch (there is no conversational history). To keep phrases varied, the prompt only includes a sliding window of the source text (_current_window, advanced every PHRASE_GEN_WINDOW_REPEATS calls) plus a random focus word and opening-style hint; _clean_phrase strips quotes/list markers. A proficiency level 0-5 (config.PHRASE_GEN_LEVEL, settings.json "phrase_gen_level", live) selects per-language constraints from the profile's phrase_gen["levels"] (vocab/grammar hints folded into the system prompt - kept stable across calls for the llama.cpp prefix cache - word range, wordfreq Zipf floor); the generated phrase is validated against them (_fits_level) with at most PHRASE_GEN_LEVEL_RETRIES (=1) regeneration, then accepted as-is (soft degradation; samples logged to logs/phrase_level_samples.jsonl for threshold tuning). See tasks/phrase_level_task.md (in prototypes-pronunciation).

  • Audio normalization (main.py): peaks are normalized before analysis; silence below AUDIO_MIN_PEAK_THRESHOLD = 0.01 skips gain adjustment.

  • Windows audio: TTS/playback uses winsound to bypass PortAudio/MME issues; a sounddevice fallback exists for other platforms. A ~150 ms silence lead-in is prepended to avoid clipping the first audio. config.AUDIO_LOCK serialises PortAudio init/teardown between the mic and speaker paths.

  • Talking mouth without an audio callback: winsound.PlaySound plays the whole buffer with no per-frame hook, so the face cannot follow live amplitude on Windows. Instead the loudness envelope is pre-computed from the (fully known) waveform via tts.loudness_envelope() and the face replays it on its own wall-clock after-loop (FaceWidget.play_levels). PlaybackController.play_with_face() (mimora/playback.py) is the single chokepoint wrapping every play_array call: it starts the track, plays, then closes the mouth - and prepends closed-mouth frames matching TTSManager.playback_lead_in_seconds() so the animation lines up with the Windows warm-up silence. The same path is used on all platforms (no live-RMS branch). Because the envelope uses the same sample_rate as playback, the slowed-reference speed (lowered sample rate) stretches the mouth track automatically. _rest_face_if_current() stops the mouth on finish/interrupt without clobbering a playback that superseded it (same identity-guard idea as PlaybackController.finished).

  • LLM server subprocess (mimora/llm_server_ctl.py): started from load_components() via LLMServerController.start(), which polls LLMManager.check_connection() until ready; terminated via LLMServerController.shutdown() (called from _shutdown_runtime on quit/restart) with a 5-second kill fallback.

  • Device detection (mimora/config.py): CUDA auto-detected via torch.cuda.is_available().

  • espeak-ng is a native binary dependency of phonemizer (separate install, not pip) - required for phoneme extraction and word-error analysis.

Testing

python -m unittest discover -s tests -v              # all fast unit tests, no model download
python tests/test_speech.py user.wav [ref.wav]       # optional end-to-end (loads the model)

Code Style (Python)

  • No linting/formatting config - follow PEP 8.
  • Type hints used throughout (from typing import Optional, List).
  • Logging via logging: %(asctime)s [%(levelname)s] (%(threadName)s) %(message)s.
  • Use explicit RuntimeError with a descriptive message for runtime validation, not assert.
  • Library deprecation warnings are filtered in mimora/bootstrap.py (early_init(), called at the top of main.py before the heavy imports).