IELTS speaking & reading trainer with on-device Whisper ASR

README.md

@@ -1,8 +1,34 @@
-# KittenTTS Android
-
-On-device text-to-speech Android app powered by [KittenML](https://github.com/KittenML) neural TTS models. Runs entirely offline — no internet required after install.
-
-Ported from the [iOS version](https://github.com/user/kitten-tts-ios) (Swift/SwiftUI) to Kotlin/Jetpack Compose.
+# KittenTTS Android — IELTS Speaking & Reading Trainer
+
+An on-device English learning app for **IELTS speaking and reading practice**, built on
+[KittenML](https://github.com/KittenML) neural TTS plus an on-device **Whisper** speech
+recognizer. Read a passage aloud, and the app transcribes your voice and scores your
+**reading accuracy (Word Error Rate)** and an **IELTS band estimate** across the four
+official criteria — all fully offline, no internet required after install.
+
+It also keeps the original **Voice Studio** (text-to-speech) so you can hear a model read
+any passage before you try it yourself.
+
+## How it works
+
+The app has three tabs: **Reading**, **Speaking**, and **Voice Studio**.
+
+**Reading (read-aloud)**
+1. Pick from 200 bundled passages, graded across IELTS bands and topics.
+2. **Listen** — a neural voice reads the passage aloud (KittenTTS).
+3. **Record & read aloud** — your microphone audio is captured at 16 kHz.
+4. **Score** — Whisper-tiny.en (ONNX, on-device) transcribes your speech; it is aligned
+   word-by-word against the passage to compute Word Error Rate, reading accuracy, and an
+   IELTS-style band breakdown.
+
+**Speaking (free response)**
+1. Pick an IELTS-style prompt (Part 1 questions, Part 2 cue cards, Part 3 discussion).
+2. **Record your answer** in your own words.
+3. **Score** — Whisper transcribes it, then a fluency analyzer (pace, pauses, fillers) and
+   a free-speech scorer estimate all four official criteria — now genuinely from *your own*
+   vocabulary and grammar: **Task Response · Fluency & Coherence · Lexical Resource ·
+   Grammatical Range & Accuracy**, plus an overall band. It also shows which prompt points
+   you covered and a transcript.
 
 ## Screenshots
 
@@ -14,6 +40,21 @@ Ported from the [iOS version](https://github.com/user/kitten-tts-ios) (Swift/Swi
 
 ## Features
 
+**IELTS reading practice**
+- 200 bundled read-aloud passages graded by IELTS band (4.5–9) and topic
+- On-device speech recognition (Whisper-tiny.en, ONNX) — fully offline
+- Word Error Rate scoring with word-level highlighting (correct / misread / skipped)
+- IELTS band estimate across the four official criteria
+- "Listen" button — hear a neural voice read the passage first
+
+**IELTS speaking practice (free response)**
+- IELTS-style prompts across Part 1, Part 2 (cue cards), and Part 3
+- Answer in your own words; Whisper transcribes on-device
+- Fluency analysis from the audio: words-per-minute, long pauses, filler words
+- Full four-criteria band estimate from your own vocabulary and grammar
+- Prompt-point coverage feedback + transcript
+
+**Voice Studio (text-to-speech)**
 - 3 model sizes: **Nano** (15M), **Micro** (40M), **Mini** (80M)
 - 8 voices: Rosie, Bella, Jasper, Luna, Bruno, Hugo, Kiki, Leo
 - Adjustable speed (0.5x – 2.0x)
@@ -24,6 +65,31 @@ Ported from the [iOS version](https://github.com/user/kitten-tts-ios) (Swift/Swi
 
 ## Architecture
 
+### Speech recognition + scoring (IELTS practice)
+
+```
+Microphone (16 kHz mono PCM)
+  → Log-mel spectrogram (80 bins, Whisper spec)
+  → Whisper encoder ONNX  → hidden states
+  → Whisper decoder ONNX  → greedy token decode (30 s windows)
+  → Byte-level BPE detokenize → transcript
+  → WER alignment vs. passage → accuracy + highlighting
+  → IELTS four-criteria band estimate
+```
+
+The Whisper ONNX models (quantized whisper-tiny.en, ~41 MB) are **bundled in the APK**
+(`assets/asr/`) and run on the same ONNX Runtime used for TTS. Fetch them with
+[`tools/download_whisper_onnx.py`](tools/download_whisper_onnx.py) (no PyTorch needed) —
+see [`app/src/main/assets/asr/README.md`](app/src/main/assets/asr/README.md). The pipeline
+(mel spectrogram + greedy decode + byte-level tokenizer) is validated to match the
+reference HuggingFace implementation exactly.
+
+> The IELTS band estimate is an honest *approximation* derived from reading accuracy,
+> coverage, and pace. A read-aloud task cannot fully measure free-speech lexical/grammatical
+> range; those proxies are documented in `IeltsScorer.kt`.
+
+### Text-to-speech (Voice Studio)
+
 ```
 Text Input (any length)
   → Auto-chunking (max 400 chars at sentence boundaries)
@@ -39,10 +105,12 @@ Text Input (any length)
 
 | Component | Technology |
 |-----------|-----------|
-| UI | Kotlin + Jetpack Compose |
-| ML Inference | ONNX Runtime Android |
+| UI | Kotlin + Jetpack Compose + Navigation Compose |
+| ML Inference | ONNX Runtime Android (TTS + Whisper ASR) |
+| Speech recognition | Whisper-tiny.en (ONNX, on-device) |
 | Phonemization | espeak-ng (C via JNI/NDK) |
-| Audio | AudioTrack (24kHz Float32 PCM) |
+| Audio | AudioTrack playback (24kHz) · AudioRecord capture (16kHz) |
+| Scoring | Word Error Rate (Levenshtein) + IELTS band heuristics |
 | Build | Gradle KTS, Android NDK, CMake |
 
 ## Download
@@ -72,7 +140,12 @@ Get the latest APK from [Releases](https://github.com/rockerritesh/kitten-tts-an
    ./build-espeak-ng.sh
    ```
 
-3. Open in Android Studio and build, or:
+3. Fetch the Whisper ASR model into `assets/asr/` (needed for speaking/reading scoring):
+   ```bash
+   python3 tools/download_whisper_onnx.py
+   ```
+
+4. Open in Android Studio and build, or:
    ```bash
    ./gradlew assembleDebug
    ```
@@ -82,16 +155,35 @@ Get the latest APK from [Releases](https://github.com/rockerritesh/kitten-tts-an
 ```
 app/src/main/
 ├── java/com/kittenml/tts/
-│   ├── MainActivity.kt           # Entry point
-│   ├── engine/
+│   ├── MainActivity.kt           # Entry point + bottom-nav (Practice / Voice Studio)
+│   ├── engine/                   # TTS
 │   │   ├── KittenTTSEngine.kt    # Core TTS pipeline
 │   │   ├── EspeakBridge.kt       # JNI wrapper
 │   │   └── AudioPlayer.kt        # AudioTrack playback
+│   ├── asr/                      # Speech recognition
+│   │   ├── AudioRecorder.kt      # 16 kHz mic capture
+│   │   ├── MelSpectrogram.kt     # 80-bin log-mel features
+│   │   ├── WhisperTokenizer.kt   # byte-level BPE decode
+│   │   ├── WhisperAsrEngine.kt   # encoder/decoder ONNX inference
+│   │   └── AsrState.kt
+│   ├── scoring/
+│   │   ├── WerScorer.kt          # Word Error Rate + alignment
+│   │   ├── ScoreResult.kt
+│   │   ├── IeltsScorer.kt        # read-aloud four-criteria estimate
+│   │   ├── IeltsAssessment.kt
+│   │   ├── BandUtil.kt           # quality → IELTS band helpers
+│   │   ├── FluencyAnalyzer.kt    # pace / pauses / fillers from audio
+│   │   └── FreeSpeechScorer.kt   # free-response four-criteria estimate
+│   ├── data/
+│   │   ├── Paragraph.kt / ParagraphRepository.kt
+│   │   └── SpeakingPrompt.kt / SpeakingPromptRepository.kt
 │   ├── ui/
 │   │   ├── theme/                 # Dark theme (Color, Theme, Type)
 │   │   └── screen/
-│   │       ├── TTSScreen.kt      # Main UI
-│   │       └── TTSViewModel.kt   # State management
+│   │       ├── TTSScreen.kt      # Voice Studio UI
+│   │       ├── TTSViewModel.kt
+│   │       ├── practice/         # reading: list + record/score + ViewModel
+│   │       └── speaking/         # free speaking: list + record/score + ViewModel
 │   └── model/
 │       ├── TTSModel.kt           # Model enum
 │       └── EngineState.kt        # Engine state
@@ -100,11 +192,23 @@ app/src/main/
 │   ├── espeak-jni.c              # JNI glue layer
 │   └── CMakeLists.txt            # NDK build config
 └── assets/
-    ├── models/                    # ONNX model files (~168 MB)
+    ├── models/                    # TTS ONNX model files (~168 MB)
     ├── voices/                    # Voice embedding JSONs (~51 MB)
-    └── espeak-ng-data/            # Phoneme data files (~1 MB)
+    ├── espeak-ng-data/            # Phoneme data files (~1 MB)
+    ├── asr/                       # Whisper ONNX + vocab (see asr/README.md)
+    └── ielts/
+        ├── paragraphs.json        # 200 read-aloud passages
+        └── speaking_prompts.json  # free-speaking prompts (Parts 1–3)
 ```
 
+## IELTS passages
+
+The bundled passages live in [`app/src/main/assets/ielts/paragraphs.json`](app/src/main/assets/ielts/paragraphs.json),
+each with `id`, `title`, `topic`, `band`, and `text`. The app ships with 200 graded
+passages across 15 topics and IELTS bands 4.5–9.0. Add or edit passages by changing this
+file — no code change is needed, as the list is read at runtime. Free-speaking prompts
+live alongside it in `speaking_prompts.json`.
+
 ## License
 
 Apache 2.0

app/build.gradle.kts

@@ -70,6 +70,7 @@ dependencies {
     implementation(libs.androidx.compose.ui.tooling.preview)
     implementation(libs.androidx.lifecycle.viewmodel.compose)
     implementation(libs.androidx.lifecycle.runtime.compose)
+    implementation(libs.androidx.navigation.compose)
     implementation(libs.onnxruntime.android)
     implementation(libs.gson)
     debugImplementation(libs.androidx.compose.ui.tooling)

app/src/main/AndroidManifest.xml

@@ -1,6 +1,8 @@
 <?xml version="1.0" encoding="utf-8"?>
 <manifest xmlns:android="http://schemas.android.com/apk/res/android">
 
+    <uses-permission android:name="android.permission.RECORD_AUDIO" />
+
     <application
         android:allowBackup="true"
         android:icon="@mipmap/ic_launcher"

app/src/main/assets/asr/README.md

@@ -0,0 +1,60 @@
+# ASR models (speech recognition)
+
+The IELTS speaking practice feature scores your reading by transcribing your
+voice with an on-device **Whisper-tiny.en** ONNX model and comparing the
+transcript to the passage (Word Error Rate). The models are **bundled inside
+the APK** and run fully offline — the same approach as the TTS models.
+
+This folder must contain four files before scoring will work. The default is
+the **quantized (int8) whisper-tiny.en** — the small build, validated to match
+the reference feature extractor exactly and to transcribe correctly:
+
+| File | What it is | Approx size |
+|------|------------|-------------|
+| `whisper_encoder.onnx` | audio (log-mel) → hidden states | ~10 MB |
+| `whisper_decoder.onnx` | hidden states + tokens → logits | ~30 MB |
+| `vocab.json` | byte-level BPE vocabulary (token → id) | ~1 MB |
+| `asr_config.json` | I/O names, decoder prompt, special token ids | < 1 KB |
+
+## How to get them (recommended: direct download, no PyTorch)
+
+The simplest path downloads a pre-converted model from Hugging Face
+(`onnx-community/whisper-tiny.en`) — no torch/optimum needed:
+
+```bash
+python3 tools/download_whisper_onnx.py                 # quantized (small, default)
+python3 tools/download_whisper_onnx.py --precision fp32  # full precision (larger)
+```
+
+This writes all four files here and reads the decoder prompt + special-token
+ids straight from the model config. The non-merged `decoder_model` it fetches
+matches the engine's no-KV-cache greedy decode loop.
+
+## Alternative: export it yourself
+
+`tools/export_whisper_onnx.py` exports via `optimum` (note: needs
+`optimum<2.0`, since 2.x removed the `exporters` module). The direct download
+above is preferred.
+
+Track the `.onnx` files with **git-lfs** (the repo already uses it for the TTS
+models):
+
+```bash
+git lfs track "app/src/main/assets/asr/*.onnx"
+git add .gitattributes app/src/main/assets/asr
+```
+
+## If the files are missing
+
+The app still builds and runs. Recording works, but the engine reports
+`Unavailable` and the Practice screen shows a hint instead of a score. Drop
+the files in and rebuild to enable scoring.
+
+## Notes
+
+- `aaptOptions { noCompress("onnx") }` in `app/build.gradle.kts` already keeps
+  the ONNX files uncompressed in the APK.
+- Audio is captured at 16 kHz mono and processed in 30-second windows, so
+  passages of any length are supported.
+- For higher accuracy at the cost of size/speed, swap in `whisper-base.en`;
+  the Kotlin side reads dimensions from the model, so no code change is needed.

app/src/main/assets/asr/asr_config.json

@@ -0,0 +1,16 @@
+{
+  "encoderModel": "whisper_encoder.onnx",
+  "decoderModel": "whisper_decoder.onnx",
+  "vocabFile": "vocab.json",
+  "decoderStartIds": [
+    50257,
+    50362
+  ],
+  "eosTokenId": 50256,
+  "maxNewTokens": 224,
+  "inputFeaturesName": "input_features",
+  "encoderHiddenName": "last_hidden_state",
+  "decoderInputIdsName": "input_ids",
+  "decoderEncoderHiddenName": "encoder_hidden_states",
+  "decoderLogitsName": "logits"
+}