From bdda7139090b0853089b87b9b0354d7faa105cba Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sat, 6 Jun 2026 14:53:48 -0700 Subject: [PATCH 01/15] docs: adaptive typing design note (learned per-user key geometry) Captures the opt-in learned key-geometry feature: one per-user model behind both taps (KeyDetector) and gestures (ProximityInfo sweet spots, Java-only / no native rebuild), the context-prior + learned-geometry layers, hard caps to avoid wrong literals, privacy (content-free, incognito-gated), leantype.db persistence riding the existing backup, a stats page, and a phased build order. Co-Authored-By: Claude Opus 4.8 --- docs/ADAPTIVE_TYPING.md | 199 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 199 insertions(+) create mode 100644 docs/ADAPTIVE_TYPING.md diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md new file mode 100644 index 000000000..895d1930a --- /dev/null +++ b/docs/ADAPTIVE_TYPING.md @@ -0,0 +1,199 @@ +# Adaptive Typing — learned per-user key geometry (taps + gestures) + +> Status: **design / in progress** (opt-in feature). Tracking issue: see the +> "Adaptive learned key geometry" issue on the fork. This note is the source of +> truth for the design; update it as the implementation evolves. + +## Goal + +Make the keyboard *feel like it learns how you type* — for both **tapping** and +**gesture/glide** input — by adapting each key's effective touch geometry to where +your finger actually lands and how the current context makes some keys more likely. +This is the technique popular keyboards use ("internally resize keys"); the academic +framing is a spatial model with a context prior (Bayesian touch). + +Explicitly **not** this feature: making autocorrect pick a better *word* after the +fact. We want it to feel like "the keyboard adapted to me," not "I still miss keys +but autocorrect cleans up." (That word-ranking idea is captured separately in +`SUGGESTION_RANKING.md`.) + +## How the engine works today (why this is feasible) + +Two independent geometry systems decide what your input becomes: + +1. **Literal tap → key** — `KeyDetector.detectHitKey` (`keyboard/KeyDetector.java`) + picks the key with the smallest edge-distance (`Key.squaredDistanceToEdge`). Pure + geometry, no weighting. The exact tapped letter is committed immediately; + mistakes are fixed downstream by autocorrect. +2. **Spatial model (gestures + tap-correction)** — the native recognizer scores + candidates using per-key **sweet spots** (effective center + radius). Those sweet + spots are **computed in Java** in `ProximityInfo.createNativeProximityInfo` + (`com/android/inputmethod/keyboard/ProximityInfo.java:168-220`) from key hit-box + centers + the static per-row `TouchPositionCorrection`, then pushed across the + existing JNI (`setProximityInfoNative`). + +**Key consequence:** because the sweet spots are produced in Java and passed through +the existing JNI, we can change what a swipe resolves to **without a native (C++) +rebuild** — we just feed adjusted centers/radii. This is what lets the feature reach +gesture compose, which is a hard requirement. + +There is no dynamic or learned key resizing today. A *static* per-row +`TouchPositionCorrection` exists (sweet-spot Y offset + radius per row); its +X-correction is disabled/obsolete. + +## The model: one learned model, two consumers + +A single per-user model, keyed by **(key, layout, orientation)**, stores content-free +geometry: + +``` +TOUCH_MODEL( + key_code, layout, orientation, + mean_dx, mean_dy, -- where you land relative to the key center (EMA) + var_dx, var_dy, -- how consistent you are (for confidence + radius scaling) + count, -- samples seen (gates confidence; powers the stats page) + updated_at +) +``` + +No characters, words, or sequences are ever stored — only aggregate geometry per key. + +Two parts read it: + +- **Taps** → `KeyDetector.detectHitKey`: measure distance to each key's **learned** + center (center + mean offset) instead of the raw center → borderline taps resolve + the way *you* type. +- **Gestures** → `ProximityInfo` sweet spots: shift each key's center by its learned + offset and scale its radius by your consistency → the recognizer matches your swipe + against keys positioned where your hand actually goes. + +Same model behind both ⇒ coherent "this is how I type." + +### Layer A — dynamic context prior (tap-only, ephemeral, stores nothing) + +Each keystroke, read the in-progress word's top-N completion candidates (the engine +already produces ~18 internally; the visible 3 is only a display limit) and project +them to a *next-character* distribution: for each candidate, take the char at the +current position weighted by the candidate's score; sum per letter. Example: typed +`H`, candidates `Hello`/`Hey`/`He` → `e` dominates → **E's tap target grows slightly +for the next tap.** Recomputed live; never persisted. + +For **gestures** there is no single "next key" to enlarge mid-stroke, so the context +prior is tap-only. The contextual-likelihood part for gestures is already handled by +the native language model (word-level). The new gesture win is Layer B (learned +geometry). + +### Layer B — learned per-user touch model (persisted, the "learning") + +- **Record:** on a *confident* tap (committed and not immediately backspaced / + autocorrected away), compute `dx = touchX - keyCenterX`, `dy = touchY - keyCenterY` + and fold into that key's running mean/variance via an exponential moving average + (recent behavior weighted more; old data decays). +- **Apply:** effective center = `center + mean_offset`; effective radius scales with + consistency (tighter for keys you nail, more forgiving for scattered keys). + +## Safety: never "I pressed W but it typed E" + +The literal tap is committed as-is, so the bias is **hard-capped** and confidence-gated: + +- Max center shift ≤ ~25% of key width/height (tunable via a strength slider). +- Radius scale bounded (e.g. 0.7–1.4×). +- A key's bias only ramps in after enough samples (`count`) and low-enough variance; + the applied magnitude scales with confidence — no sudden jumps. +- The context prior only breaks *ambiguous* taps within the cap band. **Beyond the cap + into a neighbor's territory, the neighbor always wins.** Strength = 0 ⇒ pure learned + geometry, no context flipping. + +## Privacy & security + +- Layer A persists nothing. +- Layer B persists only per-key geometry + counts — **no text content.** The only weak + signal is per-key `count` (letter-usage frequency), which never leaves the device. +- **Respect existing learning gates:** do not record while `mIncognitoModeEnabled` + (always-incognito pref, framework no-learning fields, or password fields) — the same + gate user-history learning uses. Opt-in master toggle on top. +- The settings backup already contains clipboard text + user dictionaries, so adding + content-free geometry does not widen the backup's sensitivity. We may still + quantize/round counts for extra caution. + +## Persistence, export/import, backup compatibility + +Local data lives in a single SQLite DB, `leantype.db` +(`latin/database/Database.kt`, raw `SQLiteOpenHelper`, currently VERSION 2; clipboard +lives there). The settings backup (`settings/preferences/BackupRestorePreference.kt`, +Advanced tab) **already zips the entire `leantype.db` and restores it**, and restore +is lenient (unknown zip entries skipped; missing columns handled by schema checks in +`Database.copyFromDb` + `onUpgrade`). + +Therefore: + +- **Single export/import path (satisfied for free):** add the touch model as a **new + table in `leantype.db`** → it is automatically part of the existing Advanced-tab + backup and restored the same way. No second mechanism. +- **Move behaviors to another device:** backup → restore. +- **Delete:** a "Reset learned typing model" button clears the table. +- **Don't break existing setting files:** purely additive — bump `Database.VERSION`, + add an `onUpgrade` that `CREATE`s the table; old code reads only tables it knows, so + an old backup (no table → created empty) and a new backup on an older app (unknown + table ignored) both restore safely. The format is unversioned but tolerant by design. + +## Stats / "your learned keyboard" page (Settings) + +A visualization page (trust + the reset control live here): + +- **Heatmap**: each key with its learned offset (arrow) and a variance/confidence + ellipse — literally "what your learned keyboard looks like." +- **Per-key accuracy stats**: + - *Consistency* = how tightly you cluster on the key (low variance). + - *Correction rate* = how often a tap on the key was immediately backspaced / + autocorrected away — an **approximate** accuracy proxy (we never know true intent). + e.g. "Z corrected ~18% vs E ~2%." +- **Reset** button. + +## Configurability + +- Master toggle (opt-in, default off). +- Strength slider (off → gentle tie-break → aggressive). +- Reset learned model. +- Honors incognito / no-learning fields. + +## Implementation footprint + +- **Gestures:** `ProximityInfo.createNativeProximityInfo` (`:185-198`) — add the + learned per-key offset to `sweetSpotCenterXs/Ys` and scale `sweetSpotRadii`; when the + feature is on, generate sweet spots from key centers even for layouts lacking + `TouchPositionCorrection` data (so it doesn't depend on the layout shipping it). + Re-push on keyboard reload / model update (learning is slow → no per-keystroke native + churn). **No C++ rebuild.** +- **Taps:** `KeyDetector.detectHitKey` — distance to the learned center + the capped + context prior tie-break. +- **Store:** new table in `leantype.db` + a DAO (follow `ClipboardDao`) + a + `TouchModelManager` that computes capped effective geometry and applies the EMA + update. +- **Learning hook:** record confident-tap offsets (incognito-gated) from the input + path. +- **Settings:** 5-file pref pattern (toggle + strength), reset, stats page. + +## Caveat to validate on-device + +The default sweet spots are tuned. We must confirm the learned shifts *improve* gesture +recognition rather than destabilize it — hence confidence-gating, caps, opt-in, and the +stats page to keep it honest. Validate the gesture path explicitly (it's the priority). + +## Phased build order + +1. **Foundation:** opt-in pref (5-file) + `leantype.db` table + DAO + `TouchModelManager` + (EMA update, capped effective-geometry API). Compiles; no behavior yet. +2. **Learning + gesture injection:** record confident taps; feed learned geometry into + `ProximityInfo` sweet spots (the gesture priority) + the capped `KeyDetector` + tie-break for taps. Testable on-device. +3. **Context prior (Layer A):** completion-derived next-char boost for taps. +4. **Stats page + reset UI + strength tuning.** + +## Open questions (tracked) + +1. Counts in backup: keep raw (slightly richer stats, faint usage signal) vs + quantize/omit. Current plan: keep (needed for the stats page), local-only + opt-in. +2. Learning scope confirmed: per layout + orientation (not per size/one-handed/floating + for v1). +3. Strength default and cap magnitude — tune on-device. From 056c0407a3a1dd202d48ec6287b24edde7a16f3f Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sat, 6 Jun 2026 15:00:32 -0700 Subject: [PATCH 02/15] feat(typing): adaptive key geometry - foundation (pref + touch-model store) Phase 1 of #1 (no behavior yet; nothing reads the model). Adds the opt-in prefs PREF_ADAPTIVE_KEY_GEOMETRY (+ strength) via the 5-file pattern, and a content-free per-(key, layout, orientation) touch-model table in leantype.db with a cached DAO (EMA running mean/variance + count, restore, clear). DB VERSION 2->3 with an additive onUpgrade, and copyFromDb extended to carry the model across the existing settings backup/restore (guarded so older backups without the table restore fine). Compiles: :app:compileStandardDebugJavaWithJavac. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/latin/database/Database.kt | 23 ++- .../keyboard/latin/database/TouchModelDao.kt | 169 ++++++++++++++++++ .../keyboard/latin/settings/Defaults.kt | 3 + .../keyboard/latin/settings/Settings.java | 5 + .../latin/settings/SettingsValues.java | 9 + 5 files changed, 208 insertions(+), 1 deletion(-) create mode 100644 app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt diff --git a/app/src/main/java/helium314/keyboard/latin/database/Database.kt b/app/src/main/java/helium314/keyboard/latin/database/Database.kt index 0e9c785cd..15b1c8f5b 100644 --- a/app/src/main/java/helium314/keyboard/latin/database/Database.kt +++ b/app/src/main/java/helium314/keyboard/latin/database/Database.kt @@ -10,17 +10,21 @@ import java.io.File class Database private constructor(context: Context, name: String = NAME) : SQLiteOpenHelper(context, name, null, VERSION) { override fun onCreate(db: SQLiteDatabase) { db.execSQL(ClipboardDao.CREATE_TABLE) + db.execSQL(TouchModelDao.CREATE_TABLE) } override fun onUpgrade(db: SQLiteDatabase, oldVersion: Int, newVersion: Int) { if (oldVersion < 2) { db.execSQL("ALTER TABLE CLIPBOARD ADD COLUMN IMAGE_URI TEXT") } + if (oldVersion < 3) { + db.execSQL(TouchModelDao.CREATE_TABLE) + } } companion object { private val TAG = Database::class.java.simpleName - private const val VERSION = 2 + private const val VERSION = 3 const val NAME = "leantype.db" private var instance: Database? = null fun getInstance(context: Context): Database { @@ -55,6 +59,23 @@ class Database private constructor(context: Context, name: String = NAME) : SQLi clipDao.addClip(it.getLong(0), it.getInt(1) != 0, it.getString(2) ?: "", imageUri) } } + // Touch model (adaptive typing): present only in backups from versions that have it. + val touchDao = TouchModelDao.getInstance(context) + if (touchDao != null) { + val hasTouchModel = otherDb.readableDatabase.rawQuery( + "SELECT name FROM sqlite_master WHERE type='table' AND name='${TouchModelDao.TABLE}'", null + ).use { it.moveToNext() } + if (hasTouchModel) { + touchDao.clear() + otherDb.readableDatabase.rawQuery(TouchModelDao.SELECT_ALL, null).use { + while (it.moveToNext()) { + touchDao.restore(TouchModelDao.Stat( + it.getInt(0), it.getString(1), it.getInt(2), it.getFloat(3), it.getFloat(4), + it.getFloat(5), it.getFloat(6), it.getInt(7), it.getLong(8))) + } + } + } + } otherDb.close() file.delete() } diff --git a/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt b/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt new file mode 100644 index 000000000..d53f3f2ca --- /dev/null +++ b/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt @@ -0,0 +1,169 @@ +// SPDX-License-Identifier: GPL-3.0-only +package helium314.keyboard.latin.database + +import android.content.ContentValues +import android.content.Context +import android.database.sqlite.SQLiteDatabase +import helium314.keyboard.latin.utils.Log + +/** + * Cached access to the learned per-user touch-model table (see docs/ADAPTIVE_TYPING.md). + * + * Stores, per (key code, layout, orientation), the running mean landing offset relative to the + * key center and its variance, plus a sample count. **Content-free**: no characters/words are + * stored, only aggregate touch geometry. Used to bias tap resolution (KeyDetector) and gesture + * sweet spots (ProximityInfo) toward where the user actually types. + * + * Lives in the shared [Database] ("leantype.db"), so it rides the existing settings + * backup/restore. Lookups are O(1) from an in-memory cache for the input hot path. + */ +class TouchModelDao private constructor(private val db: Database) { + + /** One key's learned stats. Offsets/variance are in pixels (relative to the key center). */ + data class Stat( + val keyCode: Int, + val layout: String, + val orientation: Int, + var meanDx: Float, + var meanDy: Float, + var varDx: Float, + var varDy: Float, + var count: Int, + var updatedAt: Long, + ) + + private val cache = HashMap() + + init { + db.readableDatabase.query( + TABLE, + arrayOf(COLUMN_KEY_CODE, COLUMN_LAYOUT, COLUMN_ORIENTATION, COLUMN_MEAN_DX, COLUMN_MEAN_DY, + COLUMN_VAR_DX, COLUMN_VAR_DY, COLUMN_COUNT, COLUMN_UPDATED_AT), + null, null, null, null, null + ).use { + while (it.moveToNext()) { + val s = Stat(it.getInt(0), it.getString(1), it.getInt(2), it.getFloat(3), it.getFloat(4), + it.getFloat(5), it.getFloat(6), it.getInt(7), it.getLong(8)) + cache[key(s.keyCode, s.layout, s.orientation)] = s + } + } + } + + /** + * Fold a new landing offset (touch position minus key center, in px) into the running model + * for this key, using an exponential moving average so recent behavior is weighted more and + * old data decays. Persists immediately. Callers must gate on incognito + the opt-in pref. + */ + @Synchronized + fun record(keyCode: Int, layout: String, orientation: Int, dx: Float, dy: Float, now: Long) { + val k = key(keyCode, layout, orientation) + val s = cache[k] + if (s == null) { + val ns = Stat(keyCode, layout, orientation, dx, dy, 0f, 0f, 1, now) + cache[k] = ns + write(ns) + return + } + val a = EMA_ALPHA + val oldMeanDx = s.meanDx + val oldMeanDy = s.meanDy + s.meanDx = (1 - a) * oldMeanDx + a * dx + s.meanDy = (1 - a) * oldMeanDy + a * dy + // Exponentially-weighted variance around the pre-update mean. + s.varDx = (1 - a) * (s.varDx + a * (dx - oldMeanDx) * (dx - oldMeanDx)) + s.varDy = (1 - a) * (s.varDy + a * (dy - oldMeanDy) * (dy - oldMeanDy)) + if (s.count < Int.MAX_VALUE) s.count++ + s.updatedAt = now + write(s) + } + + /** Learned stats for a key, or null if none recorded yet. */ + @Synchronized + fun get(keyCode: Int, layout: String, orientation: Int): Stat? = + cache[key(keyCode, layout, orientation)] + + /** Snapshot of all stats (for the stats page / debugging). */ + @Synchronized + fun all(): List = cache.values.map { it.copy() } + + /** Insert a full stat verbatim (no EMA) — used when restoring from a backup. */ + @Synchronized + fun restore(s: Stat) { + cache[key(s.keyCode, s.layout, s.orientation)] = s + write(s) + } + + /** Forget everything (the "reset learned typing model" action). */ + @Synchronized + fun clear() { + if (cache.isEmpty()) return + cache.clear() + db.writableDatabase.delete(TABLE, null, null) + } + + private fun write(s: Stat) { + val cv = ContentValues(9) + cv.put(COLUMN_KEY_CODE, s.keyCode) + cv.put(COLUMN_LAYOUT, s.layout) + cv.put(COLUMN_ORIENTATION, s.orientation) + cv.put(COLUMN_MEAN_DX, s.meanDx) + cv.put(COLUMN_MEAN_DY, s.meanDy) + cv.put(COLUMN_VAR_DX, s.varDx) + cv.put(COLUMN_VAR_DY, s.varDy) + cv.put(COLUMN_COUNT, s.count) + cv.put(COLUMN_UPDATED_AT, s.updatedAt) + db.writableDatabase.insertWithOnConflict(TABLE, null, cv, SQLiteDatabase.CONFLICT_REPLACE) + } + + companion object { + private const val TAG = "TouchModelDao" + // ~ last 20 samples dominate; tuned later on-device. + private const val EMA_ALPHA = 0.05f + /** Below this many samples a key's learned bias should not be applied (confidence gate). */ + const val MIN_CONFIDENT_SAMPLES = 20 + + const val TABLE = "TOUCH_MODEL" + private const val COLUMN_KEY_CODE = "KEY_CODE" + private const val COLUMN_LAYOUT = "LAYOUT" + private const val COLUMN_ORIENTATION = "ORIENTATION" + private const val COLUMN_MEAN_DX = "MEAN_DX" + private const val COLUMN_MEAN_DY = "MEAN_DY" + private const val COLUMN_VAR_DX = "VAR_DX" + private const val COLUMN_VAR_DY = "VAR_DY" + private const val COLUMN_COUNT = "COUNT" + private const val COLUMN_UPDATED_AT = "UPDATED_AT" + const val CREATE_TABLE = """ + CREATE TABLE $TABLE ( + $COLUMN_KEY_CODE INTEGER NOT NULL, + $COLUMN_LAYOUT TEXT NOT NULL, + $COLUMN_ORIENTATION INTEGER NOT NULL, + $COLUMN_MEAN_DX REAL NOT NULL, + $COLUMN_MEAN_DY REAL NOT NULL, + $COLUMN_VAR_DX REAL NOT NULL, + $COLUMN_VAR_DY REAL NOT NULL, + $COLUMN_COUNT INTEGER NOT NULL, + $COLUMN_UPDATED_AT INTEGER NOT NULL, + PRIMARY KEY ($COLUMN_KEY_CODE, $COLUMN_LAYOUT, $COLUMN_ORIENTATION) + ) + """ + const val SELECT_ALL = + "SELECT $COLUMN_KEY_CODE, $COLUMN_LAYOUT, $COLUMN_ORIENTATION, $COLUMN_MEAN_DX, $COLUMN_MEAN_DY, " + + "$COLUMN_VAR_DX, $COLUMN_VAR_DY, $COLUMN_COUNT, $COLUMN_UPDATED_AT FROM $TABLE" + + private fun key(keyCode: Int, layout: String, orientation: Int) = "$keyCode|$layout|$orientation" + + private var instance: TouchModelDao? = null + + /** Returns the instance, or null if it can't be created (e.g. device locked). */ + @Synchronized + fun getInstance(context: Context): TouchModelDao? { + if (instance == null) + try { + instance = TouchModelDao(Database.getInstance(context)) + } catch (e: Throwable) { + Log.e(TAG, "can't create TouchModelDao", e) + } + return instance + } + } +} diff --git a/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt b/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt index b948de0b1..344c0b422 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt +++ b/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt @@ -157,6 +157,9 @@ object Defaults { const val PREF_MULTIPART_FULL_WORD_SUGGESTIONS = true const val PREF_MULTIPART_TAP_SEED_GESTURE = true const val PREF_MULTIPART_RERECOGNIZE_TAPS = false + // Adaptive typing (opt-in). Off by default; strength is a 0..100 percentage of the cap. + const val PREF_ADAPTIVE_KEY_GEOMETRY = false + const val PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH = 50 const val PREF_SHOW_SETUP_WIZARD_ICON = true const val PREF_USE_CONTACTS = false const val PREF_USE_APPS = false diff --git a/app/src/main/java/helium314/keyboard/latin/settings/Settings.java b/app/src/main/java/helium314/keyboard/latin/settings/Settings.java index 1b07fb3e7..29d1830a7 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/Settings.java +++ b/app/src/main/java/helium314/keyboard/latin/settings/Settings.java @@ -183,6 +183,11 @@ public final class Settings implements SharedPreferences.OnSharedPreferenceChang // re-recognize the whole word, instead of literally appending it to a (possibly // mis-resolved) fragment. Makes a slow tap-after-swipe behave like a fast one. Default off. public static final String PREF_MULTIPART_RERECOGNIZE_TAPS = "multipart_rerecognize_taps"; + // Adaptive typing (opt-in): learn per-user key landing offsets and bias both tap resolution + // and gesture sweet-spots toward where the user actually types. Content-free (geometry only), + // incognito-gated. Strength scales the cap (0 = off). See docs/ADAPTIVE_TYPING.md. + public static final String PREF_ADAPTIVE_KEY_GEOMETRY = "adaptive_key_geometry"; + public static final String PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH = "adaptive_key_geometry_strength"; public static final String PREF_SHOW_SETUP_WIZARD_ICON = "show_setup_wizard_icon"; public static final String PREF_USE_CONTACTS = "use_contacts"; public static final String PREF_USE_APPS = "use_apps"; diff --git a/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java b/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java index e0081fac6..563929a6d 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java +++ b/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java @@ -145,6 +145,9 @@ public class SettingsValues { public final boolean mMultipartFullWordSuggestions; public final boolean mMultipartTapSeedGesture; public final boolean mMultipartRerecognizeTaps; + // Adaptive typing (opt-in): learn where the user lands and bias tap/gesture key geometry. + public final boolean mAdaptiveKeyGeometry; + public final int mAdaptiveKeyGeometryStrength; public final boolean mSlidingKeyInputPreviewEnabled; public final int mKeyLongpressTimeout; public final boolean mEnableEmojiAltPhysicalKey; @@ -395,6 +398,12 @@ public SettingsValues(final Context context, final SharedPreferences prefs, fina mMultipartRerecognizeTaps = prefs.getBoolean( Settings.PREF_MULTIPART_RERECOGNIZE_TAPS, Defaults.PREF_MULTIPART_RERECOGNIZE_TAPS); + mAdaptiveKeyGeometry = prefs.getBoolean( + Settings.PREF_ADAPTIVE_KEY_GEOMETRY, + Defaults.PREF_ADAPTIVE_KEY_GEOMETRY); + mAdaptiveKeyGeometryStrength = prefs.getInt( + Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH, + Defaults.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH); mSuggestionStripHiddenPerUserSettings = mToolbarMode == ToolbarMode.HIDDEN || mToolbarMode == ToolbarMode.TOOLBAR_KEYS; mOverrideShowingSuggestions = mInputAttributes.mMayOverrideShowingSuggestions From 73e4baf7090cc9bb109f950d61efd67b6cbe161c Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sat, 6 Jun 2026 15:03:07 -0700 Subject: [PATCH 03/15] feat(typing): adaptive key geometry - TouchModelManager policy + tests Phase 2 policy layer for #1: capped, confidence- and strength-scaled landing-offset math (pure functions). The cap (MAX_SHIFT_FRACTION of key size) is the safety bound so a learned bias can never flip a clear press to a neighbor; confidence ramps the bias in with sample count; strength scales it (0 = off). 7 unit tests. Co-Authored-By: Claude Opus 4.8 --- .../latin/database/TouchModelManager.kt | 46 ++++++++++++++ .../latin/database/TouchModelManagerTest.kt | 60 +++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt create mode 100644 app/src/test/java/helium314/keyboard/latin/database/TouchModelManagerTest.kt diff --git a/app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt b/app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt new file mode 100644 index 000000000..62b814a33 --- /dev/null +++ b/app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt @@ -0,0 +1,46 @@ +// SPDX-License-Identifier: GPL-3.0-only +package helium314.keyboard.latin.database + +/** + * Policy layer over [TouchModelDao]: turns a key's raw learned stats into a *capped*, + * confidence- and strength-scaled landing offset that the input paths apply + * (see docs/ADAPTIVE_TYPING.md). Pure functions — no Android / DB / threading — so the + * caps and ramp are unit-testable and the input hot paths just consume the result. + * + * The cap is the safety guarantee: a learned bias can shift a key's effective center by at + * most [MAX_SHIFT_FRACTION] of the key dimension, so a clearly-on-target press can never flip + * to a neighbor. The bias also ramps in with sample count (no sudden jumps from sparse data) + * and scales with the user's strength setting (0 = off). + */ +object TouchModelManager { + /** Max center shift as a fraction of the key's width/height. */ + const val MAX_SHIFT_FRACTION = 0.25f + /** Confidence reaches full strength at this many samples; it is 0 at/below MIN_CONFIDENT_SAMPLES. */ + const val FULL_CONFIDENCE_SAMPLES = 60 + + /** + * Capped, confidence- and strength-scaled landing offset for a key, in pixels. + * @return a fresh {dx, dy}; {0, 0} when there is not enough data or strength is 0. + */ + fun adjustedOffset(stat: TouchModelDao.Stat?, keyWidth: Int, keyHeight: Int, + strengthPercent: Int): FloatArray { + if (stat == null || strengthPercent <= 0 || keyWidth <= 0 || keyHeight <= 0) { + return floatArrayOf(0f, 0f) + } + val scale = confidence(stat.count) * (strengthPercent.coerceIn(0, 100) / 100f) + if (scale <= 0f) return floatArrayOf(0f, 0f) + val maxX = MAX_SHIFT_FRACTION * keyWidth + val maxY = MAX_SHIFT_FRACTION * keyHeight + val dx = (stat.meanDx * scale).coerceIn(-maxX, maxX) + val dy = (stat.meanDy * scale).coerceIn(-maxY, maxY) + return floatArrayOf(dx, dy) + } + + /** 0 below [TouchModelDao.MIN_CONFIDENT_SAMPLES], ramps linearly to 1 at [FULL_CONFIDENCE_SAMPLES]. */ + fun confidence(count: Int): Float { + val min = TouchModelDao.MIN_CONFIDENT_SAMPLES + if (count <= min) return 0f + if (count >= FULL_CONFIDENCE_SAMPLES) return 1f + return (count - min).toFloat() / (FULL_CONFIDENCE_SAMPLES - min) + } +} diff --git a/app/src/test/java/helium314/keyboard/latin/database/TouchModelManagerTest.kt b/app/src/test/java/helium314/keyboard/latin/database/TouchModelManagerTest.kt new file mode 100644 index 000000000..b815e83f2 --- /dev/null +++ b/app/src/test/java/helium314/keyboard/latin/database/TouchModelManagerTest.kt @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: GPL-3.0-only +package helium314.keyboard.latin.database + +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertTrue + +/** Unit tests for the capped/confidence/strength policy in [TouchModelManager]. */ +class TouchModelManagerTest { + + private fun stat(meanDx: Float, meanDy: Float, count: Int) = + TouchModelDao.Stat(0x65 /* 'e' */, "qwerty", 0, meanDx, meanDy, 0f, 0f, count, 0L) + + @Test fun nullStatGivesNoOffset() { + val o = TouchModelManager.adjustedOffset(null, 100, 120, 100) + assertEquals(0f, o[0]); assertEquals(0f, o[1]) + } + + @Test fun belowConfidenceThresholdGivesNoOffset() { + // count <= MIN_CONFIDENT_SAMPLES => confidence 0 => no bias even with a large mean + val o = TouchModelManager.adjustedOffset(stat(40f, -30f, TouchModelDao.MIN_CONFIDENT_SAMPLES), 100, 120, 100) + assertEquals(0f, o[0]); assertEquals(0f, o[1]) + } + + @Test fun zeroStrengthGivesNoOffset() { + val o = TouchModelManager.adjustedOffset(stat(40f, -30f, 1000), 100, 120, 0) + assertEquals(0f, o[0]); assertEquals(0f, o[1]) + } + + @Test fun largeOffsetIsCappedToKeyFraction() { + // Full confidence + full strength, but a huge mean must be clamped to +/-25% of the key. + val o = TouchModelManager.adjustedOffset(stat(9999f, -9999f, 1000), 100, 120, 100) + assertEquals(TouchModelManager.MAX_SHIFT_FRACTION * 100, o[0], 0.001f) // +25 + assertEquals(-TouchModelManager.MAX_SHIFT_FRACTION * 120, o[1], 0.001f) // -30 + } + + @Test fun moderateOffsetPassesThroughWithinCap() { + // count=60 => confidence 1; strength 100% => full mean, well within the cap. + val o = TouchModelManager.adjustedOffset(stat(5f, -4f, TouchModelManager.FULL_CONFIDENCE_SAMPLES), 100, 120, 100) + assertEquals(5f, o[0], 0.001f) + assertEquals(-4f, o[1], 0.001f) + } + + @Test fun strengthScalesTheOffset() { + val full = TouchModelManager.adjustedOffset(stat(10f, 0f, 1000), 100, 120, 100)[0] + val half = TouchModelManager.adjustedOffset(stat(10f, 0f, 1000), 100, 120, 50)[0] + assertEquals(full / 2f, half, 0.001f) + } + + @Test fun confidenceRampIsMonotonic() { + val c20 = TouchModelManager.confidence(20) + val c40 = TouchModelManager.confidence(40) + val c60 = TouchModelManager.confidence(60) + val c100 = TouchModelManager.confidence(100) + assertEquals(0f, c20, 0.001f) + assertTrue(c40 > c20 && c40 < c60) + assertEquals(1f, c60, 0.001f) + assertEquals(1f, c100, 0.001f) + } +} From 094f484f973ef1ae439e18039f7fa9738eaf8641 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sat, 6 Jun 2026 19:00:09 -0700 Subject: [PATCH 04/15] feat(typing): adaptive key geometry - learning hook + gesture injection + settings Phase 2 of #1 (testable end-to-end). Closes the loop: - PointerTracker records each letter tap's landing offset into the touch model (opt-in + incognito gated; async DB write so typing stays fast). - ProximityInfo shifts per-key sweet spots by the capped learned offset, so gesture recognition AND tap-correction follow where the user actually types. Generated even when a layout has no static touch-position-correction data. Element id passed from Keyboard; computed in Java, crosses the existing JNI - no native change. - Gesture typing -> Advanced: opt-in toggle + strength slider (reload keyboard on change). - @JvmStatic on the Kotlin DAO factory / manager so the Java hot paths can call them. Deferred: literal-tap KeyDetector tie-break (marginal/risky), the completion-derived context prior, and the stats page. Builds: :app:assembleStandardDebug. Co-Authored-By: Claude Opus 4.8 --- .../inputmethod/keyboard/ProximityInfo.java | 52 ++++++++++++++++--- .../helium314/keyboard/keyboard/Keyboard.java | 2 +- .../keyboard/keyboard/PointerTracker.java | 22 ++++++++ .../keyboard/latin/database/TouchModelDao.kt | 19 ++++++- .../latin/database/TouchModelManager.kt | 1 + .../settings/screens/GestureTypingScreen.kt | 18 +++++++ app/src/main/res/values/strings.xml | 4 ++ 7 files changed, 108 insertions(+), 10 deletions(-) diff --git a/app/src/main/java/com/android/inputmethod/keyboard/ProximityInfo.java b/app/src/main/java/com/android/inputmethod/keyboard/ProximityInfo.java index 9015f3e6d..04a974753 100644 --- a/app/src/main/java/com/android/inputmethod/keyboard/ProximityInfo.java +++ b/app/src/main/java/com/android/inputmethod/keyboard/ProximityInfo.java @@ -6,6 +6,7 @@ package com.android.inputmethod.keyboard; +import android.content.Context; import android.graphics.Rect; import helium314.keyboard.latin.utils.Log; @@ -14,6 +15,10 @@ import helium314.keyboard.keyboard.Key; import helium314.keyboard.keyboard.internal.TouchPositionCorrection; import helium314.keyboard.latin.common.Constants; +import helium314.keyboard.latin.database.TouchModelDao; +import helium314.keyboard.latin.database.TouchModelManager; +import helium314.keyboard.latin.settings.Settings; +import helium314.keyboard.latin.settings.SettingsValues; import helium314.keyboard.latin.utils.JniUtils; import java.util.ArrayList; @@ -48,12 +53,17 @@ public class ProximityInfo { private final List mSortedKeys; @NonNull private final List[] mGridNeighbors; + // Adaptive typing: the keyboard element this proximity info is for (e.g. alphabet vs symbols). + // Keys the learned touch model by (key, this element, orientation) so layouts stay separate. + private final int mLayoutElementId; @SuppressWarnings("unchecked") public ProximityInfo(final int gridWidth, final int gridHeight, final int minWidth, final int height, final int mostCommonKeyWidth, final int mostCommonKeyHeight, @NonNull final List sortedKeys, - @NonNull final TouchPositionCorrection touchPositionCorrection) { + @NonNull final TouchPositionCorrection touchPositionCorrection, + final int layoutElementId) { + mLayoutElementId = layoutElementId; mGridWidth = gridWidth; mGridHeight = gridHeight; mGridSize = mGridWidth * mGridHeight; @@ -165,14 +175,34 @@ private long createNativeProximityInfo(@NonNull final TouchPositionCorrection to infoIndex++; } - if (touchPositionCorrection.isValid()) { + // Adaptive typing (opt-in): when enabled, shift each key's sweet-spot center by the + // learned per-user landing offset (capped) so the native recognizer matches swipes — and + // tap-correction candidates — against keys where the user's hand actually goes. This + // path runs even when the layout ships no static touch-position-correction data, so we + // still generate sweet spots in that case. The shift is computed in Java and crosses the + // existing JNI; no native change. See docs/ADAPTIVE_TYPING.md. + final boolean tpcValid = touchPositionCorrection.isValid(); + final SettingsValues sv = Settings.getValues(); + final boolean adaptiveOn = sv != null && sv.mAdaptiveKeyGeometry; + final int adaptiveStrength = sv != null ? sv.mAdaptiveKeyGeometryStrength : 0; + TouchModelDao adaptiveDao = null; + int adaptiveOrientation = 0; + if (adaptiveOn && adaptiveStrength > 0) { + final Context ctx = Settings.getCurrentContext(); + if (ctx != null) { + adaptiveDao = TouchModelDao.getInstance(ctx); + adaptiveOrientation = ctx.getResources().getConfiguration().orientation; + } + } + final boolean applyAdaptive = adaptiveDao != null; + if (tpcValid || applyAdaptive) { if (DEBUG) { - Log.d(TAG, "touchPositionCorrection: ON"); + Log.d(TAG, "sweet spots: ON (tpc=" + tpcValid + " adaptive=" + applyAdaptive + ")"); } sweetSpotCenterXs = new float[keyCount]; sweetSpotCenterYs = new float[keyCount]; sweetSpotRadii = new float[keyCount]; - final int rows = touchPositionCorrection.getRows(); + final int rows = tpcValid ? touchPositionCorrection.getRows() : 0; final float defaultRadius = DEFAULT_TOUCH_POSITION_CORRECTION_RADIUS * (float)Math.hypot(mMostCommonKeyWidth, mMostCommonKeyHeight); for (int infoIndex = 0, keyIndex = 0; keyIndex < sortedKeys.size(); keyIndex++) { @@ -186,7 +216,7 @@ private long createNativeProximityInfo(@NonNull final TouchPositionCorrection to sweetSpotCenterYs[infoIndex] = hitBox.exactCenterY(); sweetSpotRadii[infoIndex] = defaultRadius; final int row = hitBox.top / mMostCommonKeyHeight; - if (row < rows) { + if (tpcValid && row < rows) { final int hitBoxWidth = hitBox.width(); final int hitBoxHeight = hitBox.height(); final float hitBoxDiagonal = (float)Math.hypot(hitBoxWidth, hitBoxHeight); @@ -197,11 +227,19 @@ private long createNativeProximityInfo(@NonNull final TouchPositionCorrection to sweetSpotRadii[infoIndex] = touchPositionCorrection.getRadius(row) * hitBoxDiagonal; } + if (applyAdaptive) { + final TouchModelDao.Stat stat = adaptiveDao.get(key.getCode(), + Integer.toString(mLayoutElementId), adaptiveOrientation); + final float[] off = TouchModelManager.adjustedOffset( + stat, key.getWidth(), key.getHeight(), adaptiveStrength); + sweetSpotCenterXs[infoIndex] += off[0]; + sweetSpotCenterYs[infoIndex] += off[1]; + } if (DEBUG) { Log.d(TAG, String.format(Locale.US, " [%2d] row=%d x/y/r=%7.2f/%7.2f/%5.2f %s code=%s", infoIndex, row, sweetSpotCenterXs[infoIndex], sweetSpotCenterYs[infoIndex], - sweetSpotRadii[infoIndex], (row < rows ? "correct" : "default"), + sweetSpotRadii[infoIndex], (tpcValid && row < rows ? "correct" : "default"), Constants.printableCode(key.getCode()))); } infoIndex++; @@ -209,7 +247,7 @@ private long createNativeProximityInfo(@NonNull final TouchPositionCorrection to } else { sweetSpotCenterXs = sweetSpotCenterYs = sweetSpotRadii = null; if (DEBUG) { - Log.d(TAG, "touchPositionCorrection: OFF"); + Log.d(TAG, "sweet spots: OFF"); } } diff --git a/app/src/main/java/helium314/keyboard/keyboard/Keyboard.java b/app/src/main/java/helium314/keyboard/keyboard/Keyboard.java index c89423538..807c33828 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/Keyboard.java +++ b/app/src/main/java/helium314/keyboard/keyboard/Keyboard.java @@ -113,7 +113,7 @@ public Keyboard(@NonNull final KeyboardParams params) { mProximityInfo = new ProximityInfo(params.GRID_WIDTH, params.GRID_HEIGHT, mOccupiedWidth, mOccupiedHeight, mMostCommonKeyWidth, mMostCommonKeyHeight, - mSortedKeys, params.mTouchPositionCorrection); + mSortedKeys, params.mTouchPositionCorrection, mId.mElementId); mProximityCharsCorrectionEnabled = params.mProximityCharsCorrectionEnabled; } diff --git a/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java b/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java index 427b88e6b..23943c6b9 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java +++ b/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java @@ -1591,6 +1591,7 @@ eventTime, getActivePointerTrackerCount(), graceMs, this, sLastLetterTapTime = eventTime; sLastLetterTapCodepoint = code; pushTapDebugPoint(mKeyX, mKeyY, mPointerId, eventTime); + recordAdaptiveTouchSample(currentKey, mKeyX, mKeyY); } } if (isInSlidingKeyInput) { @@ -1598,6 +1599,27 @@ eventTime, getActivePointerTrackerCount(), graceMs, this, } } + // Adaptive typing (opt-in, see docs/ADAPTIVE_TYPING.md): fold this letter tap's landing + // offset (touch minus key center) into the learned per-(key, layout, orientation) model so + // the spatial model can later bias gesture sweet-spots toward where the user actually types. + // Content-free (geometry only); gated on the opt-in pref and incognito. The DAO write is async. + private void recordAdaptiveTouchSample(final Key key, final int x, final int y) { + if (key == null || x < 0 || y < 0 || mKeyboard == null) return; + final SettingsValues sv = Settings.getValues(); + if (sv == null || !sv.mAdaptiveKeyGeometry || sv.mIncognitoModeEnabled) return; + final android.content.Context context = Settings.getCurrentContext(); + if (context == null) return; + final helium314.keyboard.latin.database.TouchModelDao dao = + helium314.keyboard.latin.database.TouchModelDao.getInstance(context); + if (dao == null) return; + final android.graphics.Rect hitBox = key.getHitBox(); + final float dx = x - hitBox.exactCenterX(); + final float dy = y - hitBox.exactCenterY(); + dao.record(key.getCode(), Integer.toString(mKeyboard.mId.mElementId), + context.getResources().getConfiguration().orientation, dx, dy, + System.currentTimeMillis()); + } + @Override public void cancelTrackingForAction() { if (isShowingPopupKeysPanel()) { diff --git a/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt b/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt index d53f3f2ca..951acf5fc 100644 --- a/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt +++ b/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt @@ -5,6 +5,7 @@ import android.content.ContentValues import android.content.Context import android.database.sqlite.SQLiteDatabase import helium314.keyboard.latin.utils.Log +import java.util.concurrent.Executors /** * Cached access to the learned per-user touch-model table (see docs/ADAPTIVE_TYPING.md). @@ -33,6 +34,11 @@ class TouchModelDao private constructor(private val db: Database) { ) private val cache = HashMap() + // Persist off the input thread: record() runs on every letter tap, so the DB write must not + // block typing. The cache (source of truth at runtime) is updated synchronously; the disk + // write is serialized on this single thread (order preserved). Losing the last sample or two + // on a crash is acceptable for a learning model. + private val writeExecutor = Executors.newSingleThreadExecutor() init { db.readableDatabase.query( @@ -61,7 +67,7 @@ class TouchModelDao private constructor(private val db: Database) { if (s == null) { val ns = Stat(keyCode, layout, orientation, dx, dy, 0f, 0f, 1, now) cache[k] = ns - write(ns) + persistAsync(ns.copy()) return } val a = EMA_ALPHA @@ -74,7 +80,15 @@ class TouchModelDao private constructor(private val db: Database) { s.varDy = (1 - a) * (s.varDy + a * (dy - oldMeanDy) * (dy - oldMeanDy)) if (s.count < Int.MAX_VALUE) s.count++ s.updatedAt = now - write(s) + persistAsync(s.copy()) + } + + private fun persistAsync(snapshot: Stat) { + try { + writeExecutor.execute { write(snapshot) } + } catch (e: Throwable) { + Log.e(TAG, "touch model write rejected", e) + } } /** Learned stats for a key, or null if none recorded yet. */ @@ -155,6 +169,7 @@ class TouchModelDao private constructor(private val db: Database) { private var instance: TouchModelDao? = null /** Returns the instance, or null if it can't be created (e.g. device locked). */ + @JvmStatic @Synchronized fun getInstance(context: Context): TouchModelDao? { if (instance == null) diff --git a/app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt b/app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt index 62b814a33..1918f91b3 100644 --- a/app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt +++ b/app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt @@ -22,6 +22,7 @@ object TouchModelManager { * Capped, confidence- and strength-scaled landing offset for a key, in pixels. * @return a fresh {dx, dy}; {0, 0} when there is not enough data or strength is 0. */ + @JvmStatic fun adjustedOffset(stat: TouchModelDao.Stat?, keyWidth: Int, keyHeight: Int, strengthPercent: Int): FloatArray { if (stat == null || strengthPercent <= 0 || keyWidth <= 0 || keyHeight <= 0) { diff --git a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt index 489b036a4..b2930a362 100644 --- a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt +++ b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt @@ -73,6 +73,9 @@ fun GestureTypingScreen( add(Settings.PREF_SPACE_VERTICAL_SWIPE) add(Settings.PREF_TOUCHPAD_SENSITIVITY) add(Settings.PREF_DELETE_SWIPE) + add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY) + if (prefs.getBoolean(Settings.PREF_ADAPTIVE_KEY_GEOMETRY, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY)) + add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH) add(Settings.PREF_SHORTCUT_ROWS) if (prefs.getBoolean(Settings.PREF_SHORTCUT_ROWS, Defaults.PREF_SHORTCUT_ROWS)) { add(Settings.PREF_SHORTCUT_TOP_ROW) @@ -172,6 +175,21 @@ fun createGestureTypingSettings(context: Context) = listOf( Setting(context, Settings.PREF_DELETE_SWIPE, R.string.delete_swipe, R.string.delete_swipe_summary) { SwitchPreference(it, Defaults.PREF_DELETE_SWIPE) }, + Setting(context, Settings.PREF_ADAPTIVE_KEY_GEOMETRY, + R.string.adaptive_key_geometry, R.string.adaptive_key_geometry_summary) { + SwitchPreference(it, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY) { + KeyboardSwitcher.getInstance().setThemeNeedsReload() // rebuild keyboard so sweet spots refresh + } + }, + Setting(context, Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH, R.string.adaptive_key_geometry_strength) { def -> + SliderPreference( + name = def.title, + key = def.key, + default = Defaults.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH, + range = 0f..100f, + description = { it.toString() } + ) { KeyboardSwitcher.getInstance().setThemeNeedsReload() } + }, Setting(context, Settings.PREF_SHORTCUT_ROWS, R.string.shortcut_rows, R.string.shortcut_rows_summary) { SwitchPreference(it, Defaults.PREF_SHORTCUT_ROWS) }, diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml index da85fb934..b613493d6 100644 --- a/app/src/main/res/values/strings.xml +++ b/app/src/main/res/values/strings.xml @@ -367,6 +367,10 @@ Delete swipe Perform a swipe from the delete key to select and remove bigger portions of text at once + + Adaptive key geometry (experimental) + Learn where you actually tap each key and nudge gesture and tap recognition toward your style. Stored on-device only, no text is saved. + Adaptive geometry strength Shortcut rows Enable temporary layout-defined shortcut rows opened by vertical swipes Top shortcut row swipe From 2ef0d0f1b79a3e7cf1b72790495e4053fa1624f0 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sat, 6 Jun 2026 20:16:25 -0700 Subject: [PATCH 05/15] feat(typing): gesture-endpoint learning + learned-typing-model stats page #1, follow-ups to the adaptive-geometry feature: - Gestures now teach the model (not just consume it) via their clean endpoints: finger-down -> word's first letter, finger-up -> last letter. Interior keys skipped (corner-cutting); fresh single strokes only. InputLogic.maybeRecordGestureEndpoints. - New stats screen (AdaptiveTypingStatsScreen) reachable from Gesture typing -> Advanced when the feature is on: per-key average offset, spread (consistency), sample count, plus a Reset button. Wired via SettingsDestination + a clickable Preference row. Build + SettingsContainerTest green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/latin/inputlogic/InputLogic.java | 40 +++++++ .../keyboard/settings/SettingsContainer.kt | 2 + .../keyboard/settings/SettingsNavHost.kt | 5 + .../screens/AdaptiveTypingStatsScreen.kt | 100 ++++++++++++++++++ .../settings/screens/GestureTypingScreen.kt | 17 ++- app/src/main/res/values/strings.xml | 5 + docs/ADAPTIVE_TYPING.md | 30 ++++-- 7 files changed, 187 insertions(+), 12 deletions(-) create mode 100644 app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java index af91356e7..0faf3cb7e 100644 --- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java +++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java @@ -3909,6 +3909,9 @@ public void onUpdateTailBatchInputCompleted(final SettingsValues settingsValues, if (settingsValues.mMultipartRerecognizeTaps) { mLiveStroke.set(mWordComposer.getInputPointers()); } + // Adaptive typing: learn this gesture's clean endpoints so swipes teach the model too. + maybeRecordGestureEndpoints(settingsValues, composedText, extendExistingCompose, + usedMergedTrail, keyboardSwitcher); mWordComposer.setBatchInputWord(composedText); setComposingTextInternal(composedText, 1); if (extendExistingCompose) { @@ -3973,6 +3976,43 @@ public void onUpdateTailBatchInputCompleted(final SettingsValues settingsValues, enterCombiningMode(settingsValues, false /* fromTap, unused — kept for clarity */); } + // Adaptive typing (opt-in, see docs/ADAPTIVE_TYPING.md): a gesture's first point (finger-down) + // and last point (finger-up) are clean "I aimed here" samples for the word's first and last + // letters — unlike interior keys, which suffer corner-cutting. Fold those two offsets into the + // learned model so swipes teach it too. Only for fresh single strokes (merged/extended trails + // have ambiguous endpoints). Gated on the opt-in pref + incognito; the DAO write is async. + private void maybeRecordGestureEndpoints(final SettingsValues sv, final String word, + final boolean extend, final boolean usedMergedTrail, + final KeyboardSwitcher keyboardSwitcher) { + if (sv == null || !sv.mAdaptiveKeyGeometry || sv.mIncognitoModeEnabled) return; + if (extend || usedMergedTrail || word == null || word.isEmpty()) return; + final InputPointers pts = mWordComposer.getInputPointers(); + final int n = pts.getPointerSize(); + if (n < 2) return; // need a real stroke; taps are handled in PointerTracker + final Keyboard keyboard = keyboardSwitcher.getKeyboard(); + if (keyboard == null) return; + final int[] xs = pts.getXCoordinates(); + final int[] ys = pts.getYCoordinates(); + recordGestureEndpoint(keyboard, Character.toLowerCase(word.codePointAt(0)), xs[0], ys[0]); + recordGestureEndpoint(keyboard, Character.toLowerCase(word.codePointBefore(word.length())), + xs[n - 1], ys[n - 1]); + } + + private void recordGestureEndpoint(final Keyboard keyboard, final int codePoint, + final int x, final int y) { + if (!Character.isLetter(codePoint) || x < 0 || y < 0) return; + final helium314.keyboard.keyboard.Key key = keyboard.getKey(codePoint); + if (key == null) return; + final helium314.keyboard.latin.database.TouchModelDao dao = + helium314.keyboard.latin.database.TouchModelDao.getInstance(mLatinIME); + if (dao == null) return; + final android.graphics.Rect hitBox = key.getHitBox(); + dao.record(codePoint, Integer.toString(keyboard.mId.mElementId), + mLatinIME.getResources().getConfiguration().orientation, + x - hitBox.exactCenterX(), y - hitBox.exactCenterY(), + System.currentTimeMillis()); + } + /** * Commit the typed string to the editor. *

diff --git a/app/src/main/java/helium314/keyboard/settings/SettingsContainer.kt b/app/src/main/java/helium314/keyboard/settings/SettingsContainer.kt index 5dc52e6a5..6f4097e32 100644 --- a/app/src/main/java/helium314/keyboard/settings/SettingsContainer.kt +++ b/app/src/main/java/helium314/keyboard/settings/SettingsContainer.kt @@ -96,6 +96,8 @@ object SettingsWithoutKey { const val LOAD_GESTURE_LIB = "load_gesture_library" const val TWO_THUMB_SPACING_MODE = "two_thumb_spacing_mode" const val TWO_THUMB_BACKSPACE_BEHAVIOR = "two_thumb_backspace_behavior" + const val ADAPTIVE_TYPING_STATS = "adaptive_typing_stats" // entry row (navigates) + const val ADAPTIVE_TYPING_STATS_CONTENT = "adaptive_typing_stats_content" // the stats body const val BACKGROUND_IMAGE = "background_image" const val BACKGROUND_IMAGE_LANDSCAPE = "background_image_landscape" const val CUSTOM_FONT = "custom_font" diff --git a/app/src/main/java/helium314/keyboard/settings/SettingsNavHost.kt b/app/src/main/java/helium314/keyboard/settings/SettingsNavHost.kt index 9b8b9952a..eea1a388d 100644 --- a/app/src/main/java/helium314/keyboard/settings/SettingsNavHost.kt +++ b/app/src/main/java/helium314/keyboard/settings/SettingsNavHost.kt @@ -18,6 +18,7 @@ import helium314.keyboard.latin.settings.SettingsSubtype.Companion.toSettingsSub import helium314.keyboard.latin.settings.getTransitionAnimationScale import helium314.keyboard.settings.screens.AIIntegrationScreen import helium314.keyboard.settings.screens.AboutScreen +import helium314.keyboard.settings.screens.AdaptiveTypingStatsScreen import helium314.keyboard.settings.screens.AdvancedSettingsScreen import helium314.keyboard.settings.screens.AppearanceScreen import helium314.keyboard.settings.screens.ColorsScreen @@ -106,6 +107,9 @@ fun SettingsNavHost( composable(SettingsDestination.TwoThumbTyping) { TwoThumbTypingScreen(onClickBack = ::goBack) } + composable(SettingsDestination.AdaptiveTypingStats) { + AdaptiveTypingStatsScreen(onClickBack = ::goBack) + } composable(SettingsDestination.Advanced) { AdvancedSettingsScreen(onClickBack = ::goBack) } @@ -183,6 +187,7 @@ object SettingsDestination { const val Toolbar = "toolbar" const val GestureTyping = "gesture_typing" const val TwoThumbTyping = "two_thumb_typing" + const val AdaptiveTypingStats = "adaptive_typing_stats_screen" const val Advanced = "advanced" const val Libraries = "libraries_hub" const val AIIntegration = "ai_integration" diff --git a/app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt b/app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt new file mode 100644 index 000000000..7aa78442b --- /dev/null +++ b/app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt @@ -0,0 +1,100 @@ +// SPDX-License-Identifier: GPL-3.0-only +package helium314.keyboard.settings.screens + +import android.content.Context +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.padding +import androidx.compose.material3.Button +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Text +import androidx.compose.runtime.Composable +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue +import androidx.compose.ui.Modifier +import androidx.compose.ui.platform.LocalContext +import androidx.compose.ui.res.stringResource +import androidx.compose.ui.text.font.FontFamily +import androidx.compose.ui.unit.dp +import helium314.keyboard.latin.R +import helium314.keyboard.latin.database.TouchModelDao +import helium314.keyboard.settings.SearchSettingsScreen +import helium314.keyboard.settings.SettingsWithoutKey +import kotlin.math.sqrt + +/** + * "Learned typing model" page: shows what the adaptive-key-geometry feature has learned per key + * (mean landing offset, spread/consistency, sample count) and lets the user reset it. Reuses the + * standard settings scaffold by rendering one content [Setting]; see [AdaptiveTypingStatsContent]. + */ +@Composable +fun AdaptiveTypingStatsScreen(onClickBack: () -> Unit) { + SearchSettingsScreen( + onClickBack = onClickBack, + title = stringResource(R.string.adaptive_key_geometry_stats_title), + settings = listOf(SettingsWithoutKey.ADAPTIVE_TYPING_STATS_CONTENT) + ) +} + +private data class StatRow(val label: String, val dx: Int, val dy: Int, val spread: Int, val count: Int) + +private fun loadStatRows(context: Context): List { + val dao = TouchModelDao.getInstance(context) ?: return emptyList() + return dao.all() + .sortedByDescending { it.count } + .map { s -> + val letter = try { String(Character.toChars(s.keyCode)) } catch (e: Throwable) { "?" } + val orient = if (s.orientation == 2) "L" else "P" // 2 == landscape + val spread = sqrt(((s.varDx + s.varDy) / 2f).coerceAtLeast(0f)).toInt() + StatRow("$letter ($orient)", s.meanDx.toInt(), s.meanDy.toInt(), spread, s.count) + } +} + +/** The dynamic stats body. Rendered as the content of a single registered Setting. */ +@Composable +fun AdaptiveTypingStatsContent() { + val context = LocalContext.current + var rows by remember { mutableStateOf(loadStatRows(context)) } + Column(Modifier.fillMaxWidth().padding(horizontal = 12.dp, vertical = 8.dp)) { + Text( + stringResource(R.string.adaptive_key_geometry_stats_explanation), + style = MaterialTheme.typography.bodySmall, + color = MaterialTheme.colorScheme.onSurfaceVariant, + modifier = Modifier.padding(bottom = 12.dp) + ) + if (rows.isEmpty()) { + Text( + stringResource(R.string.adaptive_key_geometry_stats_empty), + modifier = Modifier.padding(vertical = 8.dp) + ) + } else { + rows.forEach { r -> + Row( + modifier = Modifier.fillMaxWidth().padding(vertical = 3.dp), + horizontalArrangement = Arrangement.SpaceBetween + ) { + Text(r.label, fontFamily = FontFamily.Monospace, style = MaterialTheme.typography.bodyLarge) + Text( + "Δ(${r.dx}, ${r.dy})px ±${r.spread}px n=${r.count}", + fontFamily = FontFamily.Monospace, + style = MaterialTheme.typography.bodyMedium, + color = MaterialTheme.colorScheme.onSurfaceVariant + ) + } + } + } + Button( + onClick = { + TouchModelDao.getInstance(context)?.clear() + rows = emptyList() + }, + modifier = Modifier.padding(top = 16.dp) + ) { + Text(stringResource(R.string.adaptive_key_geometry_reset)) + } + } +} diff --git a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt index b2930a362..ab7e3912a 100644 --- a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt +++ b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt @@ -19,6 +19,8 @@ import helium314.keyboard.latin.utils.prefs import helium314.keyboard.settings.Setting import helium314.keyboard.settings.SearchSettingsScreen import helium314.keyboard.settings.SettingsActivity +import helium314.keyboard.settings.SettingsDestination +import helium314.keyboard.settings.preferences.Preference import helium314.keyboard.settings.preferences.SliderPreference import helium314.keyboard.settings.preferences.SwitchPreference import helium314.keyboard.settings.Theme @@ -74,8 +76,10 @@ fun GestureTypingScreen( add(Settings.PREF_TOUCHPAD_SENSITIVITY) add(Settings.PREF_DELETE_SWIPE) add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY) - if (prefs.getBoolean(Settings.PREF_ADAPTIVE_KEY_GEOMETRY, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY)) + if (prefs.getBoolean(Settings.PREF_ADAPTIVE_KEY_GEOMETRY, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY)) { add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH) + add(SettingsWithoutKey.ADAPTIVE_TYPING_STATS) + } add(Settings.PREF_SHORTCUT_ROWS) if (prefs.getBoolean(Settings.PREF_SHORTCUT_ROWS, Defaults.PREF_SHORTCUT_ROWS)) { add(Settings.PREF_SHORTCUT_TOP_ROW) @@ -190,6 +194,17 @@ fun createGestureTypingSettings(context: Context) = listOf( description = { it.toString() } ) { KeyboardSwitcher.getInstance().setThemeNeedsReload() } }, + Setting(context, SettingsWithoutKey.ADAPTIVE_TYPING_STATS, + R.string.adaptive_key_geometry_stats_title, R.string.adaptive_key_geometry_stats_summary) { + Preference( + name = it.title, + description = stringResource(R.string.adaptive_key_geometry_stats_summary), + onClick = { SettingsDestination.navigateTo(SettingsDestination.AdaptiveTypingStats) } + ) + }, + Setting(context, SettingsWithoutKey.ADAPTIVE_TYPING_STATS_CONTENT, R.string.adaptive_key_geometry_stats_title) { + AdaptiveTypingStatsContent() + }, Setting(context, Settings.PREF_SHORTCUT_ROWS, R.string.shortcut_rows, R.string.shortcut_rows_summary) { SwitchPreference(it, Defaults.PREF_SHORTCUT_ROWS) }, diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml index b613493d6..26e09d80c 100644 --- a/app/src/main/res/values/strings.xml +++ b/app/src/main/res/values/strings.xml @@ -371,6 +371,11 @@ Adaptive key geometry (experimental) Learn where you actually tap each key and nudge gesture and tap recognition toward your style. Stored on-device only, no text is saved. Adaptive geometry strength + Learned typing model + See what the keyboard has learned about your typing + Per key: average offset from the key center (where you land), spread (how consistent you are), and number of samples. Higher samples and lower spread mean a stronger, more confident adjustment. Nothing here ever leaves your device. + Nothing learned yet. Turn on adaptive key geometry and type for a while, then come back. + Reset learned model Shortcut rows Enable temporary layout-defined shortcut rows opened by vertical swipes Top shortcut row swipe diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md index 895d1930a..ef6024021 100644 --- a/docs/ADAPTIVE_TYPING.md +++ b/docs/ADAPTIVE_TYPING.md @@ -85,10 +85,15 @@ geometry). ### Layer B — learned per-user touch model (persisted, the "learning") -- **Record:** on a *confident* tap (committed and not immediately backspaced / - autocorrected away), compute `dx = touchX - keyCenterX`, `dy = touchY - keyCenterY` - and fold into that key's running mean/variance via an exponential moving average - (recent behavior weighted more; old data decays). +- **Record (taps):** on a letter tap, compute `dx = touchX - keyCenterX`, + `dy = touchY - keyCenterY` and fold into that key's running mean/variance via an + exponential moving average (recent behavior weighted more; old data decays). + Implemented in `PointerTracker.recordAdaptiveTouchSample`. +- **Record (gestures):** a swipe also teaches the model, but only via its clean + **endpoints**: finger-down ≈ the word's first letter, finger-up ≈ its last letter. + Interior keys are skipped (corner-cutting makes them unreliable) and only fresh single + strokes count (merged/extended trails have ambiguous ends). Implemented in + `InputLogic.maybeRecordGestureEndpoints`. - **Apply:** effective center = `center + mean_offset`; effective radius scales with consistency (tighter for keys you nail, more forgiving for scattered keys). @@ -182,13 +187,16 @@ stats page to keep it honest. Validate the gesture path explicitly (it's the pri ## Phased build order -1. **Foundation:** opt-in pref (5-file) + `leantype.db` table + DAO + `TouchModelManager` - (EMA update, capped effective-geometry API). Compiles; no behavior yet. -2. **Learning + gesture injection:** record confident taps; feed learned geometry into - `ProximityInfo` sweet spots (the gesture priority) + the capped `KeyDetector` - tie-break for taps. Testable on-device. -3. **Context prior (Layer A):** completion-derived next-char boost for taps. -4. **Stats page + reset UI + strength tuning.** +1. ✅ **Foundation:** opt-in pref (5-file) + `leantype.db` table + DAO + `TouchModelManager` + (EMA update, capped effective-geometry API). +2. ✅ **Learning + gesture injection:** record letter taps; feed learned geometry into + `ProximityInfo` sweet spots (gestures + tap-correction). The capped literal-tap + `KeyDetector` tie-break is deferred (marginal/risky — hitbox-gated). +3. ✅ **Gesture-endpoint learning:** swipes teach the model via their start/end keys. +4. ✅ **Stats / "learned typing model" page** (`AdaptiveTypingStatsScreen`) + reset. +5. ⬜ **Context prior (Layer A):** completion-derived next-char boost for taps. +6. ⬜ **Strength/cap tuning** + optional heatmap visualization + interior-key gesture + learning (needs corner-cutting handling or native alignment). ## Open questions (tracked) From 8eda439b92ae65ff4c4881767e9f8a4ddf7fd19a Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sat, 6 Jun 2026 08:26:26 -0700 Subject: [PATCH 06/15] fix(two-thumb): clear stale merged-trail extend-base on delete + gesture lifecycle WordComposer.mExtendBatchInputBase (the multi-part merged-trail base) was only cleared by a normally-completing gesture, so an abnormal end (cancel / empty-top recognition) left it armed, and no deletion path cleared it. A later fresh swipe then merged with the ghost trail - most visibly at the start of a text box. Clear it at the same word-end sites where mLiveStroke is already dropped (handleBackspaceEvent, resetComposingState, commitChosenWord, desync) plus the two gesture-lifecycle origins (onStartBatchInput top, onCancelBatchInput). Does not touch the hot WordComposer.reset() path. Adds 8 tests: base cleared after each backspace mode (character/fragment/whole-word), the delete slider, fresh onStartBatchInput, and onCancelBatchInput; plus 2 guards pinning the (currently dead) static-seed interlock. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/latin/inputlogic/InputLogic.java | 26 ++++ .../keyboard/latin/InputLogicTest.kt | 115 ++++++++++++++++++ dev-log.md | 89 ++++++++++++++ 3 files changed, 230 insertions(+) diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java index 0faf3cb7e..69bd9f087 100644 --- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java +++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java @@ -745,6 +745,15 @@ public void onStartBatchInput(final SettingsValues settingsValues, // Combining mode: snapshot before cancel; the gesture will re-arm the timer on completion. final boolean wasInCombiningMode = mInCombiningMode; cancelCombiningMode(); + // Two-thumb typing: defensively drop any stale merged-trail extend-base before this + // gesture's extend decision. The base is meant to live for exactly one gesture (set in + // the extend branch below, cleared in onUpdateTailBatchInputCompleted), but an abnormal + // prior gesture end — a cancel, or an empty-top recognition that early-returns before + // that clear — can leave it armed. A fresh gesture would then merge with the ghost + // trail; most visibly at the start of a text box, where no composing word exists to + // re-set it. Clearing here guarantees every gesture begins from a clean base; the + // extend branch re-arms it from the real composing word when appropriate. + mWordComposer.setExtendBatchInputBase(null); mConnection.beginBatchEdit(); // Two-thumb typing (#1.1 + combining-mode): two ways the gesture can EXTEND an existing // composing word instead of replacing it: @@ -895,6 +904,10 @@ public void onCancelBatchInput(final LatinIME.UIHandler handler) { // Drop any seed codepoint stashed by PointerTracker so the next gesture doesn't // strip its first letter against a stale seed. helium314.keyboard.keyboard.PointerTracker.consumeGestureSeedCodepoint(); + // Two-thumb typing: a cancelled gesture never reaches onUpdateTailBatchInputCompleted, + // which is the only routine site that clears the merged-trail extend-base. Drop it here + // so this cancelled gesture's trail can't leak into the next one. + mWordComposer.setExtendBatchInputBase(null); // Tap-promotion-extend was a per-gesture decision; clear on cancel so the NEXT // gesture re-evaluates against current timing. mGestureExtendsByTapPromotion = false; @@ -2360,6 +2373,12 @@ private void handleBackspaceEvent(final Event event, final InputTransaction inpu // stroke. The whole-word and batch delete branches below call mWordComposer.reset() // directly rather than resetComposingState(), so clear here to cover every path. mLiveStroke.reset(); + // Two-thumb typing: a backspace invalidates the merged-trail extend-base for the same + // reason it invalidates mLiveStroke above — re-doing the word must start from a clean + // stroke, not merge with the geometry we just partially deleted. This single clear + // covers all three backspace modes (character / fragment / whole-word), since it runs + // before any of their branches. + mWordComposer.setExtendBatchInputBase(null); // Typing-insight overlay: a backspace edits/clears the gesture word, so its trail is now // stale. Drop it so it doesn't linger. final MainKeyboardView backspaceKv = KeyboardSwitcher.getInstance().getMainKeyboardView(); @@ -3547,6 +3566,9 @@ private void resetComposingState(final boolean alsoResetLastComposedWord) { mCombiningWordHasGestureFragment = false; // Live-converge (#1.7): the word is gone, so its accumulated stroke is stale. mLiveStroke.reset(); + // Two-thumb typing: likewise drop the merged-trail extend-base. This path also covers + // the delete slider (finishInput -> resetComposingState) and cursor-move resets. + mWordComposer.setExtendBatchInputBase(null); // Combining mode is keyed on a composing word existing; if we're wiping it, the // pending timer would commit nothing useful, so cancel. cancelCombiningMode(); @@ -4217,6 +4239,8 @@ private void commitChosenWord(final SettingsValues settingsValues, final String mCombiningWordHasGestureFragment = false; // Live-converge (#1.7): word committed — its accumulated stroke no longer applies. mLiveStroke.reset(); + // Two-thumb typing: the merged-trail extend-base belongs to the just-committed word too. + mWordComposer.setExtendBatchInputBase(null); if (DebugFlags.DEBUG_ENABLED) { long runTimeMillis = System.currentTimeMillis() - startTimeMillis; Log.d(TAG, "commitChosenWord() : " + runTimeMillis + " ms to run " @@ -4379,6 +4403,8 @@ private void setComposingTextInternalWithBackgroundColor(final CharSequence newC // Live-converge (#1.7): composing was force-cancelled due to a desync — the stored // stroke no longer matches anything on screen, so drop it. mLiveStroke.reset(); + // Two-thumb typing: the merged-trail extend-base is equally stale after a desync. + mWordComposer.setExtendBatchInputBase(null); } } diff --git a/app/src/test/java/helium314/keyboard/latin/InputLogicTest.kt b/app/src/test/java/helium314/keyboard/latin/InputLogicTest.kt index 3ef7bb4fc..b73ab10fe 100644 --- a/app/src/test/java/helium314/keyboard/latin/InputLogicTest.kt +++ b/app/src/test/java/helium314/keyboard/latin/InputLogicTest.kt @@ -39,9 +39,12 @@ import org.robolectric.shadows.ShadowLog import java.util.* import kotlin.math.min import kotlin.streams.asSequence +import helium314.keyboard.latin.common.InputPointers import kotlin.test.BeforeTest import kotlin.test.Test import kotlin.test.assertEquals +import kotlin.test.assertFalse +import kotlin.test.assertTrue @RunWith(RobolectricTestRunner::class) @Config(shadows = [ @@ -607,6 +610,107 @@ class InputLogicTest { assertEquals("", composingText) } + // --- Two-thumb typing: the merged-trail extend-base (WordComposer.mExtendBatchInputBase) + // must never outlive the word it belonged to. Before the fix it was only cleared on a + // normally-completing gesture, so an abnormal end (cancel / empty-top recognition) left it + // armed, and NO deletion path cleared it. A later fresh swipe — most visibly at the start of + // a text box — then merged with that ghost trail. Each test below arms the base, performs one + // user action, and asserts the base is dropped. (We arm the base directly because the JVM + // harness has no native recognizer to produce a real merged trail.) Each fails pre-fix. --- + + @Test fun extendBaseClearedByCharacterBackspace() { + reset() + latinIME.prefs().edit { + putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) + // Character mode = neither fragment-pop nor whole-word delete. Both default ON + // (Defaults.kt), so they must be explicitly disabled to exercise the per-char path. + putBoolean(Settings.PREF_GESTURE_FRAGMENT_BACKSPACE, false) + putBoolean(Settings.PREF_COMBINING_BACKSPACE_DELETES_GESTURE_WORD, false) + } + chainInput("hello") + armExtendBase() + functionalKeyPress(KeyCode.DELETE) // character delete; word stays composing as "hell" + assertEquals("hell", composingText) + assertFalse(composer.isExtendBatchInputBaseSet) + } + + @Test fun extendBaseClearedByFragmentBackspace() { + reset() + latinIME.prefs().edit { + putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) + putBoolean(Settings.PREF_GESTURE_FRAGMENT_BACKSPACE, true) + putBoolean(Settings.PREF_COMBINING_BACKSPACE_DELETES_GESTURE_WORD, false) + } + chainInput("hello") // each tap records a fragment boundary under manual spacing + armExtendBase() + functionalKeyPress(KeyCode.DELETE) // fragment pop + assertFalse(composer.isExtendBatchInputBaseSet) + } + + @Test fun extendBaseClearedByWholeWordBackspace() { + reset() + latinIME.prefs().edit { + putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) + putBoolean(Settings.PREF_COMBINING_BACKSPACE_DELETES_GESTURE_WORD, true) + putBoolean(Settings.PREF_COMBINING_BACKSPACE_DELETES_COMPOSING_TEXT, true) + } + chainInput("hello") + armExtendBase() + functionalKeyPress(KeyCode.DELETE) // whole-word delete of the composing word + assertEquals("", composingText) + assertFalse(composer.isExtendBatchInputBaseSet) + } + + @Test fun extendBaseClearedByDeleteSlider() { + reset() + latinIME.prefs().edit { putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) } + chainInput("hello") + armExtendBase() + // The delete slider (swipe-from-backspace) routes through inputLogic.finishInput() in + // KeyboardActionListenerImpl.onMoveDeletePointer / onUpWithDeletePointerActive. + inputLogic.finishInput() + assertFalse(composer.isExtendBatchInputBaseSet) + } + + @Test fun extendBaseClearedByFreshGestureStart() { + reset() + latinIME.prefs().edit { putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) } + // No composing word: the next gesture is fresh and must not inherit a leaked base. + armExtendBase() + inputLogic.onStartBatchInput(settingsValues, KeyboardSwitcher.getInstance(), latinIME.mHandler) + handleMessages() + assertFalse(composer.isExtendBatchInputBaseSet) + } + + @Test fun extendBaseClearedByGestureCancel() { + reset() + latinIME.prefs().edit { putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) } + armExtendBase() + inputLogic.onCancelBatchInput(latinIME.mHandler) + handleMessages() + assertFalse(composer.isExtendBatchInputBaseSet) + } + + // Static-seed reachability guard. PointerTracker's tap-seed path (sLastLetterTap*) is gated + // on (!isMultipartComposeActive() && mCombiningGraceMs > 0). But grace > 0 forces multi-part + // composition active, so that conjunction is unsatisfiable and the seed is currently + // unreachable dead code. These pin the interlock: if a future settings refactor decouples + // them and re-arms the seed, it must first add the stale-static cleanup (the seed statics are + // process-global and never reset on delete / commit / field switch). + @Test fun graceImpliesMultipartComposeActive_keepsSeedPathDead() { + reset() + latinIME.prefs().edit { putInt(Settings.PREF_COMBINING_GRACE_MS, 1000) } + setText("") // force a settings reload + assertTrue(settingsValues.isMultipartComposeActive) + } + + @Test fun manualSpacingImpliesMultipartComposeActive() { + reset() + latinIME.prefs().edit { putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) } + setText("") + assertTrue(settingsValues.isMultipartComposeActive) + } + @Test fun forceAutoCapWorksWhenAutoCapIsOff() { reset() latinIME.prefs().edit { @@ -1309,6 +1413,17 @@ class InputLogicTest { checkConnectionConsistency() } + // Arm the merged-trail extend-base with a non-empty trail, simulating the state left by a + // prior gesture fragment. (A single empty InputPointers is treated as "clear", so two real + // points are required for isExtendBatchInputBaseSet to become true.) + private fun armExtendBase() { + val base = InputPointers(8) + base.addPointer(10, 20, 0, 0) + base.addPointer(30, 40, 0, 0) + composer.setExtendBatchInputBase(base) + assertTrue(composer.isExtendBatchInputBaseSet) + } + private fun checkConnectionConsistency() { // RichInputConnection only has composing text up to cursor, but InputConnection has full composing text val expectedConnectionComposingText = if (composingStart == -1 || composingEnd == -1) "" diff --git a/dev-log.md b/dev-log.md index 35da9bbfa..5a976bcda 100644 --- a/dev-log.md +++ b/dev-log.md @@ -825,3 +825,92 @@ The whole-word branch for a *composing* word used `mConnection.deleteTextBeforeC - Whole-word delete through a sentence deletes word-by-word with NO mashing/desync. - Re-doing a word after deletion recognizes cleanly (no ever-growing `th…`). - Swipe-on-backspace bulk delete leaves no stale stroke for the next word. + +--- + +## 2026-06-06 — Fix stale merged-trail extend-base leaking into the next gesture + +### Context +Bug report: gestures occasionally get "stored" after a deletion and bleed into the +next swipe — most visibly when swiping a fresh word at the start of a text box, +which produced a word fused with previously-deleted gesture geometry. Branch: +`fix/extend-base-leak-on-delete` off `main`. + +### Root cause +There are two per-word gesture-geometry stores. `mLiveStroke` (live-converge, in +`InputLogic`) was already cleared on every word-end path. But its WordComposer-side +counterpart — `mExtendBatchInputBase` (the multi-part merged-trail base consumed by +`WordComposer.setBatchInputPointers`) — was only ever cleared by a *normally +completing* gesture (`onUpdateTailBatchInputCompleted`, the lone routine clear, +which sits AFTER two empty-recognition early-returns). An abnormal gesture end left +the `mExtendBatchInputBaseSet` flag armed: +- `onCancelBatchInput` never cleared it, +- the empty-top-word early-returns in `onUpdateTailBatchInputCompleted` skip the clear, +- `WordComposer.reset()` does not touch it, so NO deletion path cleared it either. + +Once leaked, the flag survived every backspace / selection-delete / commit. A later +fresh gesture (no composing word ⇒ the `onStartBatchInput` extend branch never runs, +so it neither set nor cleared the base) hit the merge guard in +`setBatchInputPointers` and prepended the ghost trail. Start-of-text-box made it +maximally visible because there is no composing word there to legitimately re-set +the base. + +### Actions Taken +- `latin/inputlogic/InputLogic.java`: clear the merged-trail base + (`mWordComposer.setExtendBatchInputBase(null)`) at the same word-end sites where + `mLiveStroke` is already dropped, plus the two gesture-lifecycle origins: + - `onStartBatchInput` (top, before the extend decision — guarantees every gesture + starts from a clean base; the extend branch re-arms it from the real composing + word when appropriate), + - `onCancelBatchInput` (a cancelled gesture never reaches the routine clear), + - `handleBackspaceEvent` (next to the existing `mLiveStroke.reset()`, before any + mode branch — covers all three backspace modes in one place), + - `resetComposingState` (covers the delete slider's `finishInput` and cursor-move + resets), + - `commitChosenWord` and the composing-desync recovery path. + Deliberately did NOT add the clear to the hot `WordComposer.reset()` to avoid any + interaction with the mid-gesture merge timing; mirroring the reviewed `mLiveStroke` + sites is equivalent and lower-risk. +- `app/src/test/.../InputLogicTest.kt`: 6 base-clear regression tests (one per + backspace mode — character / fragment / whole-word — plus delete slider, fresh + `onStartBatchInput`, and `onCancelBatchInput`), each arming the base then asserting + it is dropped (each fails pre-fix). Added `armExtendBase()` helper and `InputPointers` + / `assertTrue` / `assertFalse` imports. +- Also added two reachability-guard tests pinning the (currently dead) PointerTracker + static-seed interlock: `grace > 0 ⇒ isMultipartComposeActive()` and the + manual-spacing equivalent. If a future settings refactor decouples them and re-arms + the seed, these fail and force adding the missing stale-static cleanup first. + +### Decisions Made +- Threading was verified safe: every `mExtendBatchInputBase` mutation runs on the UI + thread (the recognizer runs on the suggestions HandlerThread but only reads a + snapshot), so the new clears need no locking and can't race the merge. +- Scoped to dropping the base. The *related* "re-extend after a partial (character / + fragment) delete still merges the un-trimmed `mInputPointers`" issue is left for a + later ticket per the established "drop, don't trim" philosophy — the fix neither + causes nor worsens it. + +### Testing status +- Built and ran on this machine (newly installed JDK 21 + Android SDK platform-35 / + build-tools 35.0.0). `:app:testStandardDebugUnitTest` filtered to the 8 new tests + + `WordComposerTest`: **BUILD SUCCESSFUL, 11/11 passing.** Full-suite still carries the + 3 pre-existing known failures (`insertLetterIntoWordHangulFails`, + `revert autocorrect on delete`, `tapOnlyCombiningWordDoesNotShowAutospaceIndicator…`), + unrelated to this change. +- On-device validation still recommended: reproduce the start-of-text-box ghost-merge + (swipe a word, trigger an abnormal gesture end, delete, swipe a fresh word) and + confirm it no longer fuses; re-check all three backspace modes + the delete slider. + +### Manual Tests — Extend-base leak +| # | Steps | Expected Result | +|---|---|---| +| 1 | Two-thumb on. Swipe a word; cancel/abort a follow-up gesture; delete everything; swipe a fresh word at the start of the box. | Fresh word recognizes alone — no fusion with the deleted gesture. | +| 2 | Repeat #1 with backspace mode = Delete one character. | Same; no ghost merge. | +| 3 | Repeat #1 with backspace mode = Delete last fragment. | Same; no ghost merge. | +| 4 | Repeat #1 with backspace mode = Delete whole word. | Same; no ghost merge. | +| 5 | Repeat #1 using swipe-from-backspace (delete slider) to clear. | Same; no ghost merge. | + +### Open Questions / Next Steps +- Build the standard debug APK and install for on-device validation. +- Decide whether to also tackle the re-extend-after-partial-delete (stale + `mInputPointers`) follow-up. From 2acee3314e947873f79968b74608778990c74b2a Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sat, 6 Jun 2026 23:46:51 -0700 Subject: [PATCH 07/15] feat(typing): mock-keyboard heatmap on the learned-typing-model stats page #1. The raw px-delta list was hard to interpret, so show the data spatially: - Store each key's size (KEY_WIDTH/KEY_HEIGHT) with its offset so the offset can be expressed as a fraction of the key (DB VERSION 3->4; additive ALTER on upgrade; copyFromDb reads by column name so older backups restore fine). - Stats page now renders a mock QWERTY where each key shows a dot at where you tend to land (offset as a fraction of the key) plus a faint spread ring; confident keys in the accent color, still-learning keys faded. The numeric list stays below for exact values. - Recorders (tap + gesture-endpoint) now pass the key's hitbox size. Build + extendBase/manager/settings tests green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/keyboard/PointerTracker.java | 2 +- .../keyboard/latin/database/Database.kt | 34 +++++- .../keyboard/latin/database/TouchModelDao.kt | 28 +++-- .../keyboard/latin/inputlogic/InputLogic.java | 2 +- .../screens/AdaptiveTypingStatsScreen.kt | 110 +++++++++++++++--- app/src/main/res/values/strings.xml | 1 + 6 files changed, 144 insertions(+), 33 deletions(-) diff --git a/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java b/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java index 23943c6b9..dff85db69 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java +++ b/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java @@ -1617,7 +1617,7 @@ private void recordAdaptiveTouchSample(final Key key, final int x, final int y) final float dy = y - hitBox.exactCenterY(); dao.record(key.getCode(), Integer.toString(mKeyboard.mId.mElementId), context.getResources().getConfiguration().orientation, dx, dy, - System.currentTimeMillis()); + hitBox.width(), hitBox.height(), System.currentTimeMillis()); } @Override diff --git a/app/src/main/java/helium314/keyboard/latin/database/Database.kt b/app/src/main/java/helium314/keyboard/latin/database/Database.kt index 15b1c8f5b..60059bfd3 100644 --- a/app/src/main/java/helium314/keyboard/latin/database/Database.kt +++ b/app/src/main/java/helium314/keyboard/latin/database/Database.kt @@ -18,13 +18,18 @@ class Database private constructor(context: Context, name: String = NAME) : SQLi db.execSQL("ALTER TABLE CLIPBOARD ADD COLUMN IMAGE_URI TEXT") } if (oldVersion < 3) { + // Created fresh with the current schema (already includes the key-size columns). db.execSQL(TouchModelDao.CREATE_TABLE) + } else if (oldVersion < 4) { + // Upgrading from the v3 table, which lacked the per-key size columns. + db.execSQL("ALTER TABLE ${TouchModelDao.TABLE} ADD COLUMN ${TouchModelDao.COLUMN_KEY_WIDTH} INTEGER NOT NULL DEFAULT 0") + db.execSQL("ALTER TABLE ${TouchModelDao.TABLE} ADD COLUMN ${TouchModelDao.COLUMN_KEY_HEIGHT} INTEGER NOT NULL DEFAULT 0") } } companion object { private val TAG = Database::class.java.simpleName - private const val VERSION = 3 + private const val VERSION = 4 const val NAME = "leantype.db" private var instance: Database? = null fun getInstance(context: Context): Database { @@ -67,11 +72,28 @@ class Database private constructor(context: Context, name: String = NAME) : SQLi ).use { it.moveToNext() } if (hasTouchModel) { touchDao.clear() - otherDb.readableDatabase.rawQuery(TouchModelDao.SELECT_ALL, null).use { - while (it.moveToNext()) { - touchDao.restore(TouchModelDao.Stat( - it.getInt(0), it.getString(1), it.getInt(2), it.getFloat(3), it.getFloat(4), - it.getFloat(5), it.getFloat(6), it.getInt(7), it.getLong(8))) + // Read by column name so an older backup that lacks the key-size columns + // restores fine (those default to 0). + otherDb.readableDatabase.rawQuery("SELECT * FROM ${TouchModelDao.TABLE}", null).use { c -> + val iCode = c.getColumnIndex("KEY_CODE") + val iLayout = c.getColumnIndex("LAYOUT") + val iOrient = c.getColumnIndex("ORIENTATION") + val iMdx = c.getColumnIndex("MEAN_DX") + val iMdy = c.getColumnIndex("MEAN_DY") + val iVdx = c.getColumnIndex("VAR_DX") + val iVdy = c.getColumnIndex("VAR_DY") + val iCount = c.getColumnIndex("COUNT") + val iUpd = c.getColumnIndex("UPDATED_AT") + val iW = c.getColumnIndex(TouchModelDao.COLUMN_KEY_WIDTH) + val iH = c.getColumnIndex(TouchModelDao.COLUMN_KEY_HEIGHT) + if (iCode >= 0 && iLayout >= 0 && iOrient >= 0) { + while (c.moveToNext()) { + touchDao.restore(TouchModelDao.Stat( + c.getInt(iCode), c.getString(iLayout), c.getInt(iOrient), + c.getFloat(iMdx), c.getFloat(iMdy), c.getFloat(iVdx), c.getFloat(iVdy), + c.getInt(iCount), c.getLong(iUpd), + if (iW >= 0) c.getInt(iW) else 0, if (iH >= 0) c.getInt(iH) else 0)) + } } } } diff --git a/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt b/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt index 951acf5fc..9717c67f5 100644 --- a/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt +++ b/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt @@ -20,7 +20,9 @@ import java.util.concurrent.Executors */ class TouchModelDao private constructor(private val db: Database) { - /** One key's learned stats. Offsets/variance are in pixels (relative to the key center). */ + /** One key's learned stats. Offsets/variance are in pixels (relative to the key center). + * keyWidth/keyHeight are the key's size in px at record time, so a viewer can express the + * offset as a fraction of the key (e.g. "18px = 20% toward the lower-left of E"). */ data class Stat( val keyCode: Int, val layout: String, @@ -31,6 +33,8 @@ class TouchModelDao private constructor(private val db: Database) { var varDy: Float, var count: Int, var updatedAt: Long, + var keyWidth: Int = 0, + var keyHeight: Int = 0, ) private val cache = HashMap() @@ -44,12 +48,12 @@ class TouchModelDao private constructor(private val db: Database) { db.readableDatabase.query( TABLE, arrayOf(COLUMN_KEY_CODE, COLUMN_LAYOUT, COLUMN_ORIENTATION, COLUMN_MEAN_DX, COLUMN_MEAN_DY, - COLUMN_VAR_DX, COLUMN_VAR_DY, COLUMN_COUNT, COLUMN_UPDATED_AT), + COLUMN_VAR_DX, COLUMN_VAR_DY, COLUMN_COUNT, COLUMN_UPDATED_AT, COLUMN_KEY_WIDTH, COLUMN_KEY_HEIGHT), null, null, null, null, null ).use { while (it.moveToNext()) { val s = Stat(it.getInt(0), it.getString(1), it.getInt(2), it.getFloat(3), it.getFloat(4), - it.getFloat(5), it.getFloat(6), it.getInt(7), it.getLong(8)) + it.getFloat(5), it.getFloat(6), it.getInt(7), it.getLong(8), it.getInt(9), it.getInt(10)) cache[key(s.keyCode, s.layout, s.orientation)] = s } } @@ -61,11 +65,12 @@ class TouchModelDao private constructor(private val db: Database) { * old data decays. Persists immediately. Callers must gate on incognito + the opt-in pref. */ @Synchronized - fun record(keyCode: Int, layout: String, orientation: Int, dx: Float, dy: Float, now: Long) { + fun record(keyCode: Int, layout: String, orientation: Int, dx: Float, dy: Float, + keyWidth: Int, keyHeight: Int, now: Long) { val k = key(keyCode, layout, orientation) val s = cache[k] if (s == null) { - val ns = Stat(keyCode, layout, orientation, dx, dy, 0f, 0f, 1, now) + val ns = Stat(keyCode, layout, orientation, dx, dy, 0f, 0f, 1, now, keyWidth, keyHeight) cache[k] = ns persistAsync(ns.copy()) return @@ -80,6 +85,8 @@ class TouchModelDao private constructor(private val db: Database) { s.varDy = (1 - a) * (s.varDy + a * (dy - oldMeanDy) * (dy - oldMeanDy)) if (s.count < Int.MAX_VALUE) s.count++ s.updatedAt = now + if (keyWidth > 0) s.keyWidth = keyWidth + if (keyHeight > 0) s.keyHeight = keyHeight persistAsync(s.copy()) } @@ -116,7 +123,7 @@ class TouchModelDao private constructor(private val db: Database) { } private fun write(s: Stat) { - val cv = ContentValues(9) + val cv = ContentValues(11) cv.put(COLUMN_KEY_CODE, s.keyCode) cv.put(COLUMN_LAYOUT, s.layout) cv.put(COLUMN_ORIENTATION, s.orientation) @@ -126,6 +133,8 @@ class TouchModelDao private constructor(private val db: Database) { cv.put(COLUMN_VAR_DY, s.varDy) cv.put(COLUMN_COUNT, s.count) cv.put(COLUMN_UPDATED_AT, s.updatedAt) + cv.put(COLUMN_KEY_WIDTH, s.keyWidth) + cv.put(COLUMN_KEY_HEIGHT, s.keyHeight) db.writableDatabase.insertWithOnConflict(TABLE, null, cv, SQLiteDatabase.CONFLICT_REPLACE) } @@ -146,6 +155,8 @@ class TouchModelDao private constructor(private val db: Database) { private const val COLUMN_VAR_DY = "VAR_DY" private const val COLUMN_COUNT = "COUNT" private const val COLUMN_UPDATED_AT = "UPDATED_AT" + const val COLUMN_KEY_WIDTH = "KEY_WIDTH" + const val COLUMN_KEY_HEIGHT = "KEY_HEIGHT" const val CREATE_TABLE = """ CREATE TABLE $TABLE ( $COLUMN_KEY_CODE INTEGER NOT NULL, @@ -157,12 +168,11 @@ class TouchModelDao private constructor(private val db: Database) { $COLUMN_VAR_DY REAL NOT NULL, $COLUMN_COUNT INTEGER NOT NULL, $COLUMN_UPDATED_AT INTEGER NOT NULL, + $COLUMN_KEY_WIDTH INTEGER NOT NULL DEFAULT 0, + $COLUMN_KEY_HEIGHT INTEGER NOT NULL DEFAULT 0, PRIMARY KEY ($COLUMN_KEY_CODE, $COLUMN_LAYOUT, $COLUMN_ORIENTATION) ) """ - const val SELECT_ALL = - "SELECT $COLUMN_KEY_CODE, $COLUMN_LAYOUT, $COLUMN_ORIENTATION, $COLUMN_MEAN_DX, $COLUMN_MEAN_DY, " + - "$COLUMN_VAR_DX, $COLUMN_VAR_DY, $COLUMN_COUNT, $COLUMN_UPDATED_AT FROM $TABLE" private fun key(keyCode: Int, layout: String, orientation: Int) = "$keyCode|$layout|$orientation" diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java index 69bd9f087..4413319df 100644 --- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java +++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java @@ -4032,7 +4032,7 @@ private void recordGestureEndpoint(final Keyboard keyboard, final int codePoint, dao.record(codePoint, Integer.toString(keyboard.mId.mElementId), mLatinIME.getResources().getConfiguration().orientation, x - hitBox.exactCenterX(), y - hitBox.exactCenterY(), - System.currentTimeMillis()); + hitBox.width(), hitBox.height(), System.currentTimeMillis()); } /** diff --git a/app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt b/app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt index 7aa78442b..e59120b34 100644 --- a/app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt +++ b/app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt @@ -2,9 +2,11 @@ package helium314.keyboard.settings.screens import android.content.Context +import androidx.compose.foundation.Canvas import androidx.compose.foundation.layout.Arrangement import androidx.compose.foundation.layout.Column import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.aspectRatio import androidx.compose.foundation.layout.fillMaxWidth import androidx.compose.foundation.layout.padding import androidx.compose.material3.Button @@ -16,6 +18,12 @@ import androidx.compose.runtime.mutableStateOf import androidx.compose.runtime.remember import androidx.compose.runtime.setValue import androidx.compose.ui.Modifier +import androidx.compose.ui.geometry.CornerRadius +import androidx.compose.ui.geometry.Offset +import androidx.compose.ui.geometry.Size +import androidx.compose.ui.graphics.nativeCanvas +import androidx.compose.ui.graphics.toArgb +import androidx.compose.ui.platform.LocalConfiguration import androidx.compose.ui.platform.LocalContext import androidx.compose.ui.res.stringResource import androidx.compose.ui.text.font.FontFamily @@ -40,25 +48,85 @@ fun AdaptiveTypingStatsScreen(onClickBack: () -> Unit) { ) } -private data class StatRow(val label: String, val dx: Int, val dy: Int, val spread: Int, val count: Int) - -private fun loadStatRows(context: Context): List { +private fun loadStats(context: Context): List { val dao = TouchModelDao.getInstance(context) ?: return emptyList() - return dao.all() - .sortedByDescending { it.count } - .map { s -> - val letter = try { String(Character.toChars(s.keyCode)) } catch (e: Throwable) { "?" } - val orient = if (s.orientation == 2) "L" else "P" // 2 == landscape - val spread = sqrt(((s.varDx + s.varDy) / 2f).coerceAtLeast(0f)).toInt() - StatRow("$letter ($orient)", s.meanDx.toInt(), s.meanDy.toInt(), spread, s.count) + return dao.all().sortedByDescending { it.count } +} + +/** A mock QWERTY keyboard with, on each key, a dot showing where the user tends to land + * (offset from the key center, as a fraction of the key) and a faint ring for the spread. + * Confident keys (enough samples) are drawn in the accent color, still-learning keys faded. */ +@Composable +private fun MockKeyboardHeatmap(stats: List) { + val orientation = LocalConfiguration.current.orientation + val pref = stats.filter { it.orientation == orientation } + val byCode = (if (pref.isNotEmpty()) pref else stats).associateBy { it.keyCode } + + val keyBg = MaterialTheme.colorScheme.surfaceVariant + val labelColor = MaterialTheme.colorScheme.onSurfaceVariant + val accent = MaterialTheme.colorScheme.primary + val faded = MaterialTheme.colorScheme.outline + val rows = listOf("qwertyuiop", "asdfghjkl", "zxcvbnm") + val labelPaint = remember { + android.graphics.Paint().apply { + isAntiAlias = true + textAlign = android.graphics.Paint.Align.CENTER + } + } + + Canvas( + modifier = Modifier + .fillMaxWidth() + .aspectRatio(10f / (3f * 1.35f)) + .padding(vertical = 8.dp) + ) { + val cellW = size.width / 10f + val cellH = cellW * 1.35f + labelPaint.color = labelColor.toArgb() + labelPaint.textSize = cellW * 0.42f + rows.forEachIndexed { r, row -> + val startX = (size.width - row.length * cellW) / 2f + val top = r * cellH + row.forEachIndexed { i, ch -> + val left = startX + i * cellW + val cx = left + cellW / 2f + val cy = top + cellH / 2f + drawRoundRect( + color = keyBg, + topLeft = Offset(left + 3f, top + 3f), + size = Size(cellW - 6f, cellH - 6f), + cornerRadius = CornerRadius(8f, 8f) + ) + drawContext.canvas.nativeCanvas.drawText( + ch.uppercase(), cx, + cy - (labelPaint.ascent() + labelPaint.descent()) / 2f, labelPaint + ) + val s = byCode[ch.code] + if (s != null && s.keyWidth > 0 && s.keyHeight > 0) { + val fx = (s.meanDx / s.keyWidth).coerceIn(-0.5f, 0.5f) + val fy = (s.meanDy / s.keyHeight).coerceIn(-0.5f, 0.5f) + val dotX = cx + fx * cellW + val dotY = cy + fy * cellH + val col = if (s.count >= TouchModelDao.MIN_CONFIDENT_SAMPLES) accent else faded + val spreadFrac = sqrt( + (((s.varDx / (s.keyWidth.toFloat() * s.keyWidth)) + + (s.varDy / (s.keyHeight.toFloat() * s.keyHeight))) / 2f).coerceAtLeast(0f) + ).coerceIn(0f, 0.5f) + if (spreadFrac > 0f) + drawCircle(col.copy(alpha = 0.15f), spreadFrac * cellW, Offset(dotX, dotY)) + drawLine(col, Offset(cx, cy), Offset(dotX, dotY), strokeWidth = 2.5f) + drawCircle(col, cellW * 0.07f, Offset(dotX, dotY)) + } + } } + } } /** The dynamic stats body. Rendered as the content of a single registered Setting. */ @Composable fun AdaptiveTypingStatsContent() { val context = LocalContext.current - var rows by remember { mutableStateOf(loadStatRows(context)) } + var stats by remember { mutableStateOf(loadStats(context)) } Column(Modifier.fillMaxWidth().padding(horizontal = 12.dp, vertical = 8.dp)) { Text( stringResource(R.string.adaptive_key_geometry_stats_explanation), @@ -66,20 +134,30 @@ fun AdaptiveTypingStatsContent() { color = MaterialTheme.colorScheme.onSurfaceVariant, modifier = Modifier.padding(bottom = 12.dp) ) - if (rows.isEmpty()) { + if (stats.isEmpty()) { Text( stringResource(R.string.adaptive_key_geometry_stats_empty), modifier = Modifier.padding(vertical = 8.dp) ) } else { - rows.forEach { r -> + MockKeyboardHeatmap(stats) + Text( + stringResource(R.string.adaptive_key_geometry_stats_legend), + style = MaterialTheme.typography.bodySmall, + color = MaterialTheme.colorScheme.onSurfaceVariant, + modifier = Modifier.padding(vertical = 8.dp) + ) + stats.forEach { s -> + val letter = try { String(Character.toChars(s.keyCode)) } catch (e: Throwable) { "?" } + val orient = if (s.orientation == 2) "L" else "P" // 2 == landscape + val spread = sqrt(((s.varDx + s.varDy) / 2f).coerceAtLeast(0f)).toInt() Row( modifier = Modifier.fillMaxWidth().padding(vertical = 3.dp), horizontalArrangement = Arrangement.SpaceBetween ) { - Text(r.label, fontFamily = FontFamily.Monospace, style = MaterialTheme.typography.bodyLarge) + Text("$letter ($orient)", fontFamily = FontFamily.Monospace, style = MaterialTheme.typography.bodyLarge) Text( - "Δ(${r.dx}, ${r.dy})px ±${r.spread}px n=${r.count}", + "Δ(${s.meanDx.toInt()}, ${s.meanDy.toInt()})px ±${spread}px n=${s.count}", fontFamily = FontFamily.Monospace, style = MaterialTheme.typography.bodyMedium, color = MaterialTheme.colorScheme.onSurfaceVariant @@ -90,7 +168,7 @@ fun AdaptiveTypingStatsContent() { Button( onClick = { TouchModelDao.getInstance(context)?.clear() - rows = emptyList() + stats = emptyList() }, modifier = Modifier.padding(top = 16.dp) ) { diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml index 26e09d80c..0de94eba0 100644 --- a/app/src/main/res/values/strings.xml +++ b/app/src/main/res/values/strings.xml @@ -375,6 +375,7 @@ See what the keyboard has learned about your typing Per key: average offset from the key center (where you land), spread (how consistent you are), and number of samples. Higher samples and lower spread mean a stronger, more confident adjustment. Nothing here ever leaves your device. Nothing learned yet. Turn on adaptive key geometry and type for a while, then come back. + On each key, the dot shows where you tend to land relative to the center, and the ring shows your spread. Solid dots are confident; faded dots are still learning. Reset learned model Shortcut rows Enable temporary layout-defined shortcut rows opened by vertical swipes From 320cbe17e99e20610d80bda23569ae259d08e36c Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 00:04:17 -0700 Subject: [PATCH 08/15] chore(db): collapse experimental touch-model migration into a single recreate-on-upgrade step --- .../helium314/keyboard/latin/database/Database.kt | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/app/src/main/java/helium314/keyboard/latin/database/Database.kt b/app/src/main/java/helium314/keyboard/latin/database/Database.kt index 60059bfd3..f1ec827af 100644 --- a/app/src/main/java/helium314/keyboard/latin/database/Database.kt +++ b/app/src/main/java/helium314/keyboard/latin/database/Database.kt @@ -17,13 +17,13 @@ class Database private constructor(context: Context, name: String = NAME) : SQLi if (oldVersion < 2) { db.execSQL("ALTER TABLE CLIPBOARD ADD COLUMN IMAGE_URI TEXT") } - if (oldVersion < 3) { - // Created fresh with the current schema (already includes the key-size columns). + if (oldVersion < 4) { + // The learned touch model is experimental and disposable, so on any upgrade from + // before it stabilized we just recreate it with the current schema instead of + // carrying per-version column migrations. No real release shipped the table, so + // this loses nothing in practice. + db.execSQL("DROP TABLE IF EXISTS ${TouchModelDao.TABLE}") db.execSQL(TouchModelDao.CREATE_TABLE) - } else if (oldVersion < 4) { - // Upgrading from the v3 table, which lacked the per-key size columns. - db.execSQL("ALTER TABLE ${TouchModelDao.TABLE} ADD COLUMN ${TouchModelDao.COLUMN_KEY_WIDTH} INTEGER NOT NULL DEFAULT 0") - db.execSQL("ALTER TABLE ${TouchModelDao.TABLE} ADD COLUMN ${TouchModelDao.COLUMN_KEY_HEIGHT} INTEGER NOT NULL DEFAULT 0") } } From d43311043ebca0f2a40a666e43dcbc42340b9dd0 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 06:51:17 -0700 Subject: [PATCH 09/15] feat(typing): next-key context prior + adaptive tap biasing (#1) Completes adaptive typing on the tap side: - AdaptiveKeyContext builds a "likely next key" prior from the top-5 suggestions, weighted equally (averaged, not score-skewed): the next char of the in-progress word's completions, or the first char of next-word predictions for a fresh word. Rebuilt between keystrokes in InputLogic.setSuggestedWords (off the tap hot path), read lock-free per tap. - KeyDetector.detectHitKey now biases the tapped key by the learned per-key landing offset (shifts the effective center) AND the context prior (enlarges likely keys). Both capped; the prior cap (~18% of key) is deliberately below the learned cap (~25%) so it nudges rather than dominates. Only near-boundary taps can flip; clear presses are untouched. - Suppressed during gestures/swipes (PointerTracker.isInGestureOrKeySwipe); gestures are not context-biased since suggestions don't change mid-swipe. Applies to both current words being built and the first key of a new word (predictions). Build + extendBase/manager/settings tests green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/keyboard/AdaptiveKeyContext.java | 94 +++++++++++++++++++ .../keyboard/keyboard/KeyDetector.java | 81 ++++++++++++++++ .../keyboard/keyboard/PointerTracker.java | 6 ++ .../keyboard/latin/inputlogic/InputLogic.java | 10 ++ docs/ADAPTIVE_TYPING.md | 14 ++- 5 files changed, 202 insertions(+), 3 deletions(-) create mode 100644 app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java diff --git a/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java b/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java new file mode 100644 index 000000000..e61d14076 --- /dev/null +++ b/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java @@ -0,0 +1,94 @@ +/* + * SPDX-License-Identifier: GPL-3.0-only + */ +package helium314.keyboard.keyboard; + +import helium314.keyboard.latin.SuggestedWords; + +import java.util.HashMap; +import java.util.Map; + +/** + * Holds a transient "likely next key" prior derived from the current suggestion strip, used to + * gently enlarge the touch target of likely next keys (adaptive typing, see + * docs/ADAPTIVE_TYPING.md). + * + *

It is rebuilt between keystrokes — whenever the suggestions change — on the UI thread, and + * read per tap in {@link KeyDetector}. Reads are lock-free via {@code volatile} parallel arrays, + * which are tiny (at most a handful of distinct next-characters), so the tap hot path stays fast + * even for very fast typists. Because suggestions are computed asynchronously, the prior may lag + * the latest keystroke by one tap under very fast typing; that is harmless, as the prior is only + * a soft, capped bias. + * + *

Suggestions are weighted EQUALLY (averaged) rather than by score, so the result is not skewed + * toward the single top suggestion. + */ +public final class AdaptiveKeyContext { + /** How many suggestions to average over. */ + private static final int TOP_N = 5; + + private static volatile int[] sCodes; + private static volatile float[] sWeights; + + private AdaptiveKeyContext() {} + + /** + * Rebuild the prior from the top suggestions. + * + * @param words the current suggestion strip contents. + * @param position index of the NEXT character within each suggestion — the current + * composing-word length while a word is being built, or 0 for a new word + * (using next-word predictions, whose first letter is the likely next key). + */ + public static void update(final SuggestedWords words, final int position) { + if (words == null || words.isEmpty() || position < 0) { + clear(); + return; + } + final int n = Math.min(TOP_N, words.size()); + final HashMap tally = new HashMap<>(); + int considered = 0; + for (int i = 0; i < n; i++) { + final String w = words.getWord(i); + if (w == null || position >= w.length()) continue; // typed word itself / too short + final int cp = Character.toLowerCase(w.charAt(position)); + if (!Character.isLetter(cp)) continue; + tally.merge(cp, 1, Integer::sum); + considered++; + } + if (considered == 0) { + clear(); + return; + } + final int[] codes = new int[tally.size()]; + final float[] weights = new float[tally.size()]; + int j = 0; + for (final Map.Entry e : tally.entrySet()) { + codes[j] = e.getKey(); + weights[j] = (float) e.getValue() / considered; // 0..1, equal-weight average + j++; + } + sCodes = codes; + sWeights = weights; + } + + public static void clear() { + sCodes = null; + sWeights = null; + } + + /** Prior weight in [0, 1] for the given key code (0 if none / no prior). */ + public static float weight(final int code) { + final int[] c = sCodes; + final float[] w = sWeights; + if (c == null) return 0f; + for (int i = 0; i < c.length; i++) { + if (c[i] == code) return w[i]; + } + return 0f; + } + + public static boolean hasPrior() { + return sCodes != null; + } +} diff --git a/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java b/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java index 400511cda..caf093e25 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java +++ b/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java @@ -6,10 +6,25 @@ package helium314.keyboard.keyboard; +import android.content.Context; +import android.graphics.Rect; + +import helium314.keyboard.latin.database.TouchModelDao; +import helium314.keyboard.latin.database.TouchModelManager; +import helium314.keyboard.latin.settings.Settings; +import helium314.keyboard.latin.settings.SettingsValues; + /** * This class handles key detection. */ public class KeyDetector { + // Adaptive typing: the context prior may enlarge a likely key's effective target by at most + // this fraction of the key (deliberately a bit less than the learned-geometry cap, so the + // prior nudges rather than dominates). See docs/ADAPTIVE_TYPING.md. + private static final float PRIOR_MAX_FRACTION = 0.18f; + // A neighbor is only allowed to win a tap if the touch is within this fraction of its hitbox. + private static final float CONSIDER_MARGIN_FRACTION = 0.40f; + private final int mKeyHysteresisDistanceSquared; private final int mKeyHysteresisDistanceForSlidingModifierSquared; @@ -101,6 +116,72 @@ public Key detectHitKey(final int x, final int y) { primaryKey = key; } } + // Adaptive typing (opt-in): for a plain tap (not a gesture/swipe), let the learned + // per-key landing offset and the current next-key prior gently bias which key wins — + // bounded so only genuinely ambiguous, near-boundary taps can flip. + if (primaryKey != null && !PointerTracker.isInGestureOrKeySwipe()) { + final Key biased = applyAdaptiveBias(touchX, touchY, primaryKey); + if (biased != null) return biased; + } return primaryKey; } + + /** Returns a key that should win this tap instead of {@code geo} due to learned/prior bias, + * or {@code null} to keep the plain geometric result. */ + private Key applyAdaptiveBias(final int touchX, final int touchY, final Key geo) { + final SettingsValues sv = Settings.getValues(); + if (sv == null || !sv.mAdaptiveKeyGeometry || sv.mAdaptiveKeyGeometryStrength <= 0) return null; + final Context ctx = Settings.getCurrentContext(); + if (ctx == null || mKeyboard == null) return null; + final TouchModelDao dao = TouchModelDao.getInstance(ctx); + final boolean hasPrior = AdaptiveKeyContext.hasPrior(); + if (dao == null && !hasPrior) return null; // nothing to bias with + final String layout = Integer.toString(mKeyboard.mId.mElementId); + final int orientation = ctx.getResources().getConfiguration().orientation; + final int strength = sv.mAdaptiveKeyGeometryStrength; + + Key best = geo; + float bestScore = adjustedDistance(geo, touchX, touchY, dao, layout, orientation, strength); + for (final Key k : mKeyboard.getNearestKeys(touchX, touchY)) { + if (k == geo) continue; + final int code = k.getCode(); + if (code <= 0 || !Character.isLetter(code)) continue; + // Only a genuinely-favored neighbor (a confident learned offset and/or a next-key + // prior) may steal a near-boundary tap; otherwise leave the geometric result alone. + final float prior = AdaptiveKeyContext.weight(code); + final TouchModelDao.Stat st = (dao == null) ? null : dao.get(code, layout, orientation); + final boolean hasLearned = st != null && st.getCount() >= TouchModelDao.MIN_CONFIDENT_SAMPLES; + if (prior <= 0f && !hasLearned) continue; + // Bound: the touch must be within a margin of the neighbor's hitbox. + final float margin = CONSIDER_MARGIN_FRACTION * k.getWidth(); + if (k.squaredDistanceToEdge(touchX, touchY) > margin * margin) continue; + final float s = adjustedDistance(k, touchX, touchY, dao, layout, orientation, strength); + if (s < bestScore) { + bestScore = s; + best = k; + } + } + return best == geo ? null : best; + } + + /** Distance from the touch to the key's effective center (center shifted by the learned + * landing offset) minus the capped next-key prior boost. Smaller wins. */ + private float adjustedDistance(final Key k, final int touchX, final int touchY, + final TouchModelDao dao, final String layout, final int orientation, final int strength) { + final Rect hb = k.getHitBox(); + float cx = hb.exactCenterX(); + float cy = hb.exactCenterY(); + if (dao != null) { + final TouchModelDao.Stat st = dao.get(k.getCode(), layout, orientation); + final float[] off = TouchModelManager.adjustedOffset(st, k.getWidth(), k.getHeight(), strength); + cx += off[0]; + cy += off[1]; + } + final float dx = touchX - cx; + final float dy = touchY - cy; + final float dist = (float) Math.sqrt(dx * dx + dy * dy); + final float boost = AdaptiveKeyContext.weight(k.getCode()) + * PRIOR_MAX_FRACTION * k.getWidth() * (strength / 100f); + return dist - boost; + } } diff --git a/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java b/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java index dff85db69..cb81b315a 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java +++ b/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java @@ -162,6 +162,12 @@ public static int consumeGestureSeedCodepoint() { return seed; } + /** True while a gesture, key-swipe (space/delete), or shortcut-row swipe is in progress. + * Adaptive tap biasing must be suppressed in these states (it only applies to plain taps). */ + public static boolean isInGestureOrKeySwipe() { + return sInGesture || sInKeySwipe || sInShortcutRowSwipe; + } + private static TypingTimeRecorder sTypingTimeRecorder; // The position and time at which first down event occurred. diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java index 4413319df..449a0a12b 100644 --- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java +++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java @@ -1376,6 +1376,16 @@ public void setSuggestedWords(final SuggestedWords suggestedWords) { mWordComposer.setAutoCorrection(suggestedWordInfo); } mSuggestedWords = suggestedWords; + // Adaptive typing: refresh the "likely next key" prior from the strip so the next tap can + // bias toward likely keys. The next-character index is the current composing-word length + // (or 0 for a new word, where the predictions' first letters are the likely next keys). + // Cheap, and runs on suggestion updates rather than the tap hot path. + final SettingsValues svAdaptive = Settings.getValues(); + if (svAdaptive != null && svAdaptive.mAdaptiveKeyGeometry) { + final int nextCharIndex = mWordComposer.isComposingWord() + ? mWordComposer.getTypedWord().length() : 0; + helium314.keyboard.keyboard.AdaptiveKeyContext.update(suggestedWords, nextCharIndex); + } final boolean newAutoCorrectionIndicator = suggestedWords.mWillAutoCorrect; // Put a blue underline to a word in TextView which will be auto-corrected. diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md index ef6024021..aa84f11d9 100644 --- a/docs/ADAPTIVE_TYPING.md +++ b/docs/ADAPTIVE_TYPING.md @@ -190,11 +190,19 @@ stats page to keep it honest. Validate the gesture path explicitly (it's the pri 1. ✅ **Foundation:** opt-in pref (5-file) + `leantype.db` table + DAO + `TouchModelManager` (EMA update, capped effective-geometry API). 2. ✅ **Learning + gesture injection:** record letter taps; feed learned geometry into - `ProximityInfo` sweet spots (gestures + tap-correction). The capped literal-tap - `KeyDetector` tie-break is deferred (marginal/risky — hitbox-gated). + `ProximityInfo` sweet spots (gestures + tap-correction). 3. ✅ **Gesture-endpoint learning:** swipes teach the model via their start/end keys. 4. ✅ **Stats / "learned typing model" page** (`AdaptiveTypingStatsScreen`) + reset. -5. ⬜ **Context prior (Layer A):** completion-derived next-char boost for taps. +5. ✅ **Tap biasing + context prior (Layer A):** `KeyDetector` now biases the tapped key by + the learned per-key offset AND a next-key prior. The prior is built in + `AdaptiveKeyContext` from the top-5 suggestions, weighted **equally** (averaged, not + score-skewed): the next char of the in-progress word's completions, or the first char of + the next-word predictions for a fresh word. It is rebuilt between keystrokes (in + `InputLogic.setSuggestedWords`, off the tap path) and read lock-free per tap. The prior's + cap (`PRIOR_MAX_FRACTION` ≈ 18% of key) is deliberately a bit **below** the learned cap + (≈ 25%), so it nudges rather than dominates. Bias is suppressed during gestures/swipes + (`PointerTracker.isInGestureOrKeySwipe`) and only flips near-boundary taps. Gestures are + intentionally NOT context-biased (suggestions don't change mid-swipe). 6. ⬜ **Strength/cap tuning** + optional heatmap visualization + interior-key gesture learning (needs corner-cutting handling or native alignment). From d2a5235dc59b66712780e6cf26cb80fc49dee1f9 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 07:02:52 -0700 Subject: [PATCH 10/15] docs(adaptive): correct rationale for tap-only context prior; clarify swipe-start behavior --- docs/ADAPTIVE_TYPING.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md index aa84f11d9..9aeee09a4 100644 --- a/docs/ADAPTIVE_TYPING.md +++ b/docs/ADAPTIVE_TYPING.md @@ -201,8 +201,20 @@ stats page to keep it honest. Validate the gesture path explicitly (it's the pri `InputLogic.setSuggestedWords`, off the tap path) and read lock-free per tap. The prior's cap (`PRIOR_MAX_FRACTION` ≈ 18% of key) is deliberately a bit **below** the learned cap (≈ 25%), so it nudges rather than dominates. Bias is suppressed during gestures/swipes - (`PointerTracker.isInGestureOrKeySwipe`) and only flips near-boundary taps. Gestures are - intentionally NOT context-biased (suggestions don't change mid-swipe). + (`PointerTracker.isInGestureOrKeySwipe`) and only flips near-boundary taps. + + The context prior is **tap-only by design** — not because swipe suggestions are + unavailable (they do update live during a swipe), but because the prior is a *per-key* + mechanism (it enlarges the next key's tap target), and a swipe resolves a *whole word* + holistically via the recognizer, which already incorporates previous-word context through + its language model. So there is no single "next key" to enlarge mid-stroke. Making swipe + *word selection* more context-aware is the separate word-level re-ranking lever in + `SUGGESTION_RANKING.md`, not this prior. Note the resulting asymmetry: for a fresh word the + first *tap* is context-biased, but the first point of a *swipe* is not. + + The **learned geometry** (Layer B), by contrast, applies to the entire swipe including its + start — every key the stroke passes is matched against its learned-shifted sweet spot — and + a swipe's endpoints also feed the model (gesture-endpoint learning). 6. ⬜ **Strength/cap tuning** + optional heatmap visualization + interior-key gesture learning (needs corner-cutting handling or native alignment). From e8fdf9aee652af8c34e4badf4e78c464a613d4f5 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 07:26:32 -0700 Subject: [PATCH 11/15] Add independent context-prior toggle and group adaptive settings Split the next-key context prior into its own opt-in setting (adaptive_context_prior, default off) alongside the existing learned key-geometry toggle, and group both under a single "Adaptive typing" section in the Gesture typing settings. - Settings/Defaults/SettingsValues: new mAdaptiveContextPrior flag. - KeyDetector: independently gate learned-offset bias vs prior boost; either alone can bias a near-boundary tap, both share the strength slider. adjustedDistance now takes usePrior and gates the boost. - InputLogic.setSuggestedWords: build the prior only when the prior toggle is on; clear AdaptiveKeyContext otherwise. - GestureTypingScreen: new "Adaptive typing" category holding both toggles; strength slider shown if either is on; stats shown if learning is on. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/keyboard/KeyDetector.java | 28 ++++++++++++------- .../keyboard/latin/inputlogic/InputLogic.java | 4 ++- .../keyboard/latin/settings/Defaults.kt | 1 + .../keyboard/latin/settings/Settings.java | 3 ++ .../latin/settings/SettingsValues.java | 4 +++ .../settings/screens/GestureTypingScreen.kt | 18 ++++++++---- app/src/main/res/values/strings.xml | 3 ++ 7 files changed, 45 insertions(+), 16 deletions(-) diff --git a/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java b/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java index caf093e25..b7aa54c3d 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java +++ b/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java @@ -130,32 +130,37 @@ public Key detectHitKey(final int x, final int y) { * or {@code null} to keep the plain geometric result. */ private Key applyAdaptiveBias(final int touchX, final int touchY, final Key geo) { final SettingsValues sv = Settings.getValues(); - if (sv == null || !sv.mAdaptiveKeyGeometry || sv.mAdaptiveKeyGeometryStrength <= 0) return null; + if (sv == null || sv.mAdaptiveKeyGeometryStrength <= 0) return null; + // The two halves are independently toggleable: learned per-key offset, and the + // context prior. Either alone is enough to bias a tap; both share the strength slider. + final boolean learn = sv.mAdaptiveKeyGeometry; + final boolean usePrior = sv.mAdaptiveContextPrior; + if (!learn && !usePrior) return null; final Context ctx = Settings.getCurrentContext(); if (ctx == null || mKeyboard == null) return null; - final TouchModelDao dao = TouchModelDao.getInstance(ctx); - final boolean hasPrior = AdaptiveKeyContext.hasPrior(); + final TouchModelDao dao = learn ? TouchModelDao.getInstance(ctx) : null; + final boolean hasPrior = usePrior && AdaptiveKeyContext.hasPrior(); if (dao == null && !hasPrior) return null; // nothing to bias with final String layout = Integer.toString(mKeyboard.mId.mElementId); final int orientation = ctx.getResources().getConfiguration().orientation; final int strength = sv.mAdaptiveKeyGeometryStrength; Key best = geo; - float bestScore = adjustedDistance(geo, touchX, touchY, dao, layout, orientation, strength); + float bestScore = adjustedDistance(geo, touchX, touchY, dao, layout, orientation, strength, usePrior); for (final Key k : mKeyboard.getNearestKeys(touchX, touchY)) { if (k == geo) continue; final int code = k.getCode(); if (code <= 0 || !Character.isLetter(code)) continue; // Only a genuinely-favored neighbor (a confident learned offset and/or a next-key // prior) may steal a near-boundary tap; otherwise leave the geometric result alone. - final float prior = AdaptiveKeyContext.weight(code); + final float prior = usePrior ? AdaptiveKeyContext.weight(code) : 0f; final TouchModelDao.Stat st = (dao == null) ? null : dao.get(code, layout, orientation); final boolean hasLearned = st != null && st.getCount() >= TouchModelDao.MIN_CONFIDENT_SAMPLES; if (prior <= 0f && !hasLearned) continue; // Bound: the touch must be within a margin of the neighbor's hitbox. final float margin = CONSIDER_MARGIN_FRACTION * k.getWidth(); if (k.squaredDistanceToEdge(touchX, touchY) > margin * margin) continue; - final float s = adjustedDistance(k, touchX, touchY, dao, layout, orientation, strength); + final float s = adjustedDistance(k, touchX, touchY, dao, layout, orientation, strength, usePrior); if (s < bestScore) { bestScore = s; best = k; @@ -165,9 +170,11 @@ private Key applyAdaptiveBias(final int touchX, final int touchY, final Key geo) } /** Distance from the touch to the key's effective center (center shifted by the learned - * landing offset) minus the capped next-key prior boost. Smaller wins. */ + * landing offset, when enabled) minus the capped next-key prior boost (when enabled). + * Smaller wins. */ private float adjustedDistance(final Key k, final int touchX, final int touchY, - final TouchModelDao dao, final String layout, final int orientation, final int strength) { + final TouchModelDao dao, final String layout, final int orientation, final int strength, + final boolean usePrior) { final Rect hb = k.getHitBox(); float cx = hb.exactCenterX(); float cy = hb.exactCenterY(); @@ -180,8 +187,9 @@ private float adjustedDistance(final Key k, final int touchX, final int touchY, final float dx = touchX - cx; final float dy = touchY - cy; final float dist = (float) Math.sqrt(dx * dx + dy * dy); - final float boost = AdaptiveKeyContext.weight(k.getCode()) - * PRIOR_MAX_FRACTION * k.getWidth() * (strength / 100f); + final float boost = usePrior + ? AdaptiveKeyContext.weight(k.getCode()) * PRIOR_MAX_FRACTION * k.getWidth() * (strength / 100f) + : 0f; return dist - boost; } } diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java index 449a0a12b..5536ba519 100644 --- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java +++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java @@ -1381,10 +1381,12 @@ public void setSuggestedWords(final SuggestedWords suggestedWords) { // (or 0 for a new word, where the predictions' first letters are the likely next keys). // Cheap, and runs on suggestion updates rather than the tap hot path. final SettingsValues svAdaptive = Settings.getValues(); - if (svAdaptive != null && svAdaptive.mAdaptiveKeyGeometry) { + if (svAdaptive != null && svAdaptive.mAdaptiveContextPrior) { final int nextCharIndex = mWordComposer.isComposingWord() ? mWordComposer.getTypedWord().length() : 0; helium314.keyboard.keyboard.AdaptiveKeyContext.update(suggestedWords, nextCharIndex); + } else { + helium314.keyboard.keyboard.AdaptiveKeyContext.clear(); } final boolean newAutoCorrectionIndicator = suggestedWords.mWillAutoCorrect; diff --git a/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt b/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt index 1d76bd78a..d899ec83a 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt +++ b/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt @@ -160,6 +160,7 @@ object Defaults { // Adaptive typing (opt-in). Off by default; strength is a 0..100 percentage of the cap. const val PREF_ADAPTIVE_KEY_GEOMETRY = false const val PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH = 50 + const val PREF_ADAPTIVE_CONTEXT_PRIOR = false const val PREF_SHOW_SETUP_WIZARD_ICON = true const val PREF_USE_CONTACTS = false const val PREF_USE_APPS = false diff --git a/app/src/main/java/helium314/keyboard/latin/settings/Settings.java b/app/src/main/java/helium314/keyboard/latin/settings/Settings.java index abff08a16..93f68771a 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/Settings.java +++ b/app/src/main/java/helium314/keyboard/latin/settings/Settings.java @@ -188,6 +188,9 @@ public final class Settings implements SharedPreferences.OnSharedPreferenceChang // incognito-gated. Strength scales the cap (0 = off). See docs/ADAPTIVE_TYPING.md. public static final String PREF_ADAPTIVE_KEY_GEOMETRY = "adaptive_key_geometry"; public static final String PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH = "adaptive_key_geometry_strength"; + // Context prior: enlarge the touch target of the next key the suggestions predict. Independent + // of the learned-geometry toggle above; both share the strength slider. See ADAPTIVE_TYPING.md. + public static final String PREF_ADAPTIVE_CONTEXT_PRIOR = "adaptive_context_prior"; public static final String PREF_SHOW_SETUP_WIZARD_ICON = "show_setup_wizard_icon"; public static final String PREF_USE_CONTACTS = "use_contacts"; public static final String PREF_USE_APPS = "use_apps"; diff --git a/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java b/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java index e1add5869..02d3a3962 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java +++ b/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java @@ -148,6 +148,7 @@ public class SettingsValues { // Adaptive typing (opt-in): learn where the user lands and bias tap/gesture key geometry. public final boolean mAdaptiveKeyGeometry; public final int mAdaptiveKeyGeometryStrength; + public final boolean mAdaptiveContextPrior; public final boolean mSlidingKeyInputPreviewEnabled; public final int mKeyLongpressTimeout; public final boolean mEnableEmojiAltPhysicalKey; @@ -405,6 +406,9 @@ public SettingsValues(final Context context, final SharedPreferences prefs, fina mAdaptiveKeyGeometryStrength = prefs.getInt( Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH); + mAdaptiveContextPrior = prefs.getBoolean( + Settings.PREF_ADAPTIVE_CONTEXT_PRIOR, + Defaults.PREF_ADAPTIVE_CONTEXT_PRIOR); mSuggestionStripHiddenPerUserSettings = mToolbarMode == ToolbarMode.HIDDEN || mToolbarMode == ToolbarMode.TOOLBAR_KEYS; mOverrideShowingSuggestions = mInputAttributes.mMayOverrideShowingSuggestions diff --git a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt index ab7e3912a..505ac04c4 100644 --- a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt +++ b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt @@ -75,16 +75,20 @@ fun GestureTypingScreen( add(Settings.PREF_SPACE_VERTICAL_SWIPE) add(Settings.PREF_TOUCHPAD_SENSITIVITY) add(Settings.PREF_DELETE_SWIPE) - add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY) - if (prefs.getBoolean(Settings.PREF_ADAPTIVE_KEY_GEOMETRY, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY)) { - add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH) - add(SettingsWithoutKey.ADAPTIVE_TYPING_STATS) - } add(Settings.PREF_SHORTCUT_ROWS) if (prefs.getBoolean(Settings.PREF_SHORTCUT_ROWS, Defaults.PREF_SHORTCUT_ROWS)) { add(Settings.PREF_SHORTCUT_TOP_ROW) add(Settings.PREF_SHORTCUT_BOTTOM_ROW) } + + // Adaptive typing — both sub-features grouped in one section. + add(R.string.adaptive_typing_category) + add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY) + add(Settings.PREF_ADAPTIVE_CONTEXT_PRIOR) + val learnOn = prefs.getBoolean(Settings.PREF_ADAPTIVE_KEY_GEOMETRY, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY) + val priorOn = prefs.getBoolean(Settings.PREF_ADAPTIVE_CONTEXT_PRIOR, Defaults.PREF_ADAPTIVE_CONTEXT_PRIOR) + if (learnOn || priorOn) add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH) + if (learnOn) add(SettingsWithoutKey.ADAPTIVE_TYPING_STATS) // the learned model only fills with the geometry half } SearchSettingsScreen( onClickBack = onClickBack, @@ -185,6 +189,10 @@ fun createGestureTypingSettings(context: Context) = listOf( KeyboardSwitcher.getInstance().setThemeNeedsReload() // rebuild keyboard so sweet spots refresh } }, + Setting(context, Settings.PREF_ADAPTIVE_CONTEXT_PRIOR, + R.string.adaptive_context_prior, R.string.adaptive_context_prior_summary) { + SwitchPreference(it, Defaults.PREF_ADAPTIVE_CONTEXT_PRIOR) + }, Setting(context, Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH, R.string.adaptive_key_geometry_strength) { def -> SliderPreference( name = def.title, diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml index 08110c704..2009204a1 100644 --- a/app/src/main/res/values/strings.xml +++ b/app/src/main/res/values/strings.xml @@ -387,6 +387,9 @@ Adaptive key geometry (experimental) Learn where you actually tap each key and nudge gesture and tap recognition toward your style. Stored on-device only, no text is saved. Adaptive geometry strength + Adaptive typing + Anticipate likely keys (experimental) + Make the next key the suggestions predict slightly easier to press. Averages the top suggestions and is capped so it never overrides a clear press. Applies to taps, not swipes. Learned typing model See what the keyboard has learned about your typing Per key: average offset from the key center (where you land), spread (how consistent you are), and number of samples. Higher samples and lower spread mean a stronger, more confident adjustment. Nothing here ever leaves your device. From eb0b118040f5dc3835f6fd04f00b3538e0884242 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 07:37:05 -0700 Subject: [PATCH 12/15] Add live adaptive-typing debug overlay A debug toggle ("Show adaptive targets on keyboard") that visualizes the adaptive model directly on the live keyboard, so the feature is visible as you type: - Learned geometry: each letter key shows a faint geometric-center ring, an arrow to its learned landing target, and a dot at that target. - Context prior: keys the suggestions predict get a translucent halo whose radius grows with the prior weight; halos morph between keystrokes. Implemented as AdaptiveTargetsDrawingPreview (an AbstractDrawingPreview, same mechanism as the gesture-debug overlay), drawn on the DrawingPreviewPlacerView above the keys. It is purely visual, reads the same live model / prior / settings the engine uses, and is gated on its pref each frame (zero cost when off). The halo radius is exaggerated vs the engine's sub-key boost for legibility; the visible keys are deliberately not reflowed. - New pref PREF_ADAPTIVE_DEBUG_OVERLAY (5-file), shown under "Adaptive typing" when either adaptive toggle is on. - AdaptiveKeyContext gains a change listener fired on update()/clear(); the overlay repaints on it. MainKeyboardView registers the listener and feeds the overlay the keyboard + padding so markers align with rendered keys. - docs/ADAPTIVE_TYPING.md: document the overlay, the independent context-prior toggle, the grouped settings, the built heatmap, and fix the DB version. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/keyboard/AdaptiveKeyContext.java | 20 ++ .../keyboard/keyboard/MainKeyboardView.java | 14 ++ .../AdaptiveTargetsDrawingPreview.java | 194 ++++++++++++++++++ .../keyboard/latin/settings/Defaults.kt | 1 + .../keyboard/latin/settings/Settings.java | 3 + .../latin/settings/SettingsValues.java | 4 + .../settings/screens/GestureTypingScreen.kt | 7 + app/src/main/res/values/strings.xml | 2 + docs/ADAPTIVE_TYPING.md | 60 +++++- 9 files changed, 297 insertions(+), 8 deletions(-) create mode 100644 app/src/main/java/helium314/keyboard/keyboard/internal/AdaptiveTargetsDrawingPreview.java diff --git a/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java b/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java index e61d14076..26871f2f9 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java +++ b/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java @@ -30,8 +30,26 @@ public final class AdaptiveKeyContext { private static volatile int[] sCodes; private static volatile float[] sWeights; + /** + * Optional observer notified whenever the prior changes, on the same (UI) thread that mutates + * it. Used only by the debug overlay (see AdaptiveTargetsDrawingPreview) to repaint the live + * keyboard as the prior shifts between keystrokes; null in normal operation. Volatile so the + * keyboard view can register/clear it from its own lifecycle without extra locking. + */ + private static volatile Runnable sChangeListener; + private AdaptiveKeyContext() {} + /** Register (or clear, with {@code null}) the debug repaint observer. */ + public static void setChangeListener(final Runnable listener) { + sChangeListener = listener; + } + + private static void fireChanged() { + final Runnable l = sChangeListener; + if (l != null) l.run(); + } + /** * Rebuild the prior from the top suggestions. * @@ -70,11 +88,13 @@ public static void update(final SuggestedWords words, final int position) { } sCodes = codes; sWeights = weights; + fireChanged(); } public static void clear() { sCodes = null; sWeights = null; + fireChanged(); } /** Prior weight in [0, 1] for the given key code (0 if none / no prior). */ diff --git a/app/src/main/java/helium314/keyboard/keyboard/MainKeyboardView.java b/app/src/main/java/helium314/keyboard/keyboard/MainKeyboardView.java index c642688db..c10eda072 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/MainKeyboardView.java +++ b/app/src/main/java/helium314/keyboard/keyboard/MainKeyboardView.java @@ -37,6 +37,7 @@ import helium314.keyboard.compat.ConfigurationCompatKt; import helium314.keyboard.keyboard.internal.DrawingPreviewPlacerView; import helium314.keyboard.keyboard.internal.DrawingProxy; +import helium314.keyboard.keyboard.internal.AdaptiveTargetsDrawingPreview; import helium314.keyboard.keyboard.internal.GestureDebugPointsDrawingPreview; import helium314.keyboard.keyboard.internal.GestureFloatingTextDrawingPreview; import helium314.keyboard.keyboard.internal.GestureTrailsDrawingPreview; @@ -126,6 +127,9 @@ public final class MainKeyboardView extends KeyboardView implements DrawingProxy private final SlidingKeyInputDrawingPreview mSlidingKeyInputDrawingPreview; // Debug overlay for two-thumb point hinting (#2.1), toggled by PREF_GESTURE_DEBUG_DRAW_POINTS. private final GestureDebugPointsDrawingPreview mGestureDebugPointsDrawingPreview; + // Debug overlay visualizing adaptive typing (learned offsets + next-key prior), toggled by + // PREF_ADAPTIVE_DEBUG_OVERLAY. See docs/ADAPTIVE_TYPING.md. + private final AdaptiveTargetsDrawingPreview mAdaptiveTargetsDrawingPreview; // Key preview private final KeyPreviewDrawParams mKeyPreviewDrawParams; @@ -233,6 +237,13 @@ public MainKeyboardView(final Context context, final AttributeSet attrs, final i // Debug overlay last so it draws ON TOP of the gesture trail / floating preview. mGestureDebugPointsDrawingPreview = new GestureDebugPointsDrawingPreview(); mGestureDebugPointsDrawingPreview.setDrawingView(drawingPreviewPlacerView); + // Adaptive-typing visualization overlay. Always "enabled" at the preview layer; the actual + // drawing is gated on its live pref each frame, so it costs nothing when toggled off. It + // repaints between keystrokes via the AdaptiveKeyContext change listener registered below. + mAdaptiveTargetsDrawingPreview = new AdaptiveTargetsDrawingPreview(); + mAdaptiveTargetsDrawingPreview.setDrawingView(drawingPreviewPlacerView); + mAdaptiveTargetsDrawingPreview.setPreviewEnabled(true); + AdaptiveKeyContext.setChangeListener(mAdaptiveTargetsDrawingPreview::onAdaptiveContextChanged); mainKeyboardViewAttr.recycle(); mDrawingPreviewPlacerView = drawingPreviewPlacerView; @@ -394,6 +405,9 @@ public void setKeyboard(@NonNull final Keyboard keyboard) { keyboard, -getPaddingLeft(), -getPaddingTop() + getVerticalCorrection()); PointerTracker.setKeyDetector(mKeyDetector); mPopupKeysKeyboardCache.clear(); + // Keys render at getX()+paddingLeft / getY()+paddingTop; hand those to the adaptive overlay + // so its markers line up with the drawn keys. + mAdaptiveTargetsDrawingPreview.setKeyboard(keyboard, getPaddingLeft(), getPaddingTop()); mSpaceKey = keyboard.getKey(Constants.CODE_SPACE); final int keyHeight = keyboard.mMostCommonKeyHeight - keyboard.mVerticalGap; diff --git a/app/src/main/java/helium314/keyboard/keyboard/internal/AdaptiveTargetsDrawingPreview.java b/app/src/main/java/helium314/keyboard/keyboard/internal/AdaptiveTargetsDrawingPreview.java new file mode 100644 index 000000000..0f4d5fe05 --- /dev/null +++ b/app/src/main/java/helium314/keyboard/keyboard/internal/AdaptiveTargetsDrawingPreview.java @@ -0,0 +1,194 @@ +/* + * SPDX-License-Identifier: GPL-3.0-only + */ + +package helium314.keyboard.keyboard.internal; + +import android.content.Context; +import android.graphics.Canvas; +import android.graphics.Color; +import android.graphics.Paint; + +import androidx.annotation.NonNull; + +import helium314.keyboard.keyboard.AdaptiveKeyContext; +import helium314.keyboard.keyboard.Key; +import helium314.keyboard.keyboard.Keyboard; +import helium314.keyboard.keyboard.PointerTracker; +import helium314.keyboard.latin.database.TouchModelDao; +import helium314.keyboard.latin.database.TouchModelManager; +import helium314.keyboard.latin.settings.Settings; +import helium314.keyboard.latin.settings.SettingsValues; + +/** + * Debug visualization for adaptive typing (opt-in via {@code PREF_ADAPTIVE_DEBUG_OVERLAY}). Drawn + * on top of the live keyboard, it makes the two halves of the feature visible as you type: + * + *

    + *
  • Learned key geometry — for each letter key with a confident learned landing offset, + * a faint ring marks the key's geometric centre, an arrow points to where the user's taps + * actually land on average, and a filled dot marks that learned target. This is the same + * offset {@link helium314.keyboard.keyboard.KeyDetector} biases taps toward.
  • + *
  • Next-key context prior — keys the current suggestions predict get a translucent + * halo whose radius grows with the prior weight, centred on the (possibly shifted) target. + * The halo appears/grows/shrinks between keystrokes, so the keyboard visibly "leans" toward + * the likely next key.
  • + *
+ * + *

The overlay is purely visual — it never changes detection. It reads the same live model + * ({@link TouchModelDao}), prior ({@link AdaptiveKeyContext}) and {@link SettingsValues} the engine + * uses, so what you see is what the engine does (the halo radius is exaggerated relative to the + * engine's sub-key boost so the effect is legible). Drawing is gated on the live pref each frame, + * so it costs nothing when the toggle is off. + * + *

Threading mirrors the other previews: {@link #setKeyboard} runs on the keyboard-view layout + * path and {@link #drawPreview} on {@code DrawingPreviewPlacerView}'s {@code onDraw}, both on the + * main thread. Repaints between keystrokes are driven by {@link AdaptiveKeyContext}'s change + * listener (also fired on the main thread), wired up by {@code MainKeyboardView}. + */ +public final class AdaptiveTargetsDrawingPreview extends AbstractDrawingPreview { + // Halo radius for a fully-agreed prior (weight 1.0), as a fraction of key width. Deliberately + // larger than KeyDetector's PRIOR_MAX_FRACTION (0.18) — this is a visualization, so the bulge + // is exaggerated to read clearly; smaller weights scale down linearly. + private static final float HALO_MAX_FRACTION = 0.55f; + private static final float GEOM_DOT_RADIUS_PX = 3f; + private static final float EFF_DOT_RADIUS_PX = 6f; + private static final float MIN_ARROW_LENGTH_PX = 1.5f; + + /** Current keyboard, set on the layout path; iterated for keys at draw time. */ + private Keyboard mKeyboard; + /** Keyboard-view padding: keys render at {@code getX()+paddingLeft, getY()+paddingTop}, and the + * placer canvas is already translated to the keyboard-view origin, so we add padding here. */ + private int mPaddingLeft; + private int mPaddingTop; + + private final Paint mGeomPaint = new Paint(); // reference: geometric key centre + private final Paint mArrowPaint = new Paint(); // shift from centre to learned target + private final Paint mEffPaint = new Paint(); // learned landing target + private final Paint mHaloFill = new Paint(); // prior bulge (fill) + private final Paint mHaloStroke = new Paint(); // prior bulge (outline) + + public AdaptiveTargetsDrawingPreview() { + mGeomPaint.setAntiAlias(true); + mGeomPaint.setColor(Color.WHITE); + mGeomPaint.setAlpha(0x66); + mGeomPaint.setStyle(Paint.Style.STROKE); + mGeomPaint.setStrokeWidth(2f); + + mArrowPaint.setAntiAlias(true); + mArrowPaint.setColor(Color.rgb(255, 171, 0)); // amber + mArrowPaint.setAlpha(0xCC); + mArrowPaint.setStyle(Paint.Style.STROKE); + mArrowPaint.setStrokeWidth(3f); + mArrowPaint.setStrokeCap(Paint.Cap.ROUND); + + mEffPaint.setAntiAlias(true); + mEffPaint.setColor(Color.rgb(255, 171, 0)); // amber + mEffPaint.setAlpha(0xEE); + mEffPaint.setStyle(Paint.Style.FILL); + + mHaloFill.setAntiAlias(true); + mHaloFill.setColor(Color.rgb(0, 200, 83)); // green = "predicted, easier to hit" + mHaloFill.setStyle(Paint.Style.FILL); + + mHaloStroke.setAntiAlias(true); + mHaloStroke.setColor(Color.rgb(0, 200, 83)); + mHaloStroke.setStyle(Paint.Style.STROKE); + mHaloStroke.setStrokeWidth(2.5f); + } + + /** Hand the overlay the current keyboard and the view padding at which keys are rendered. */ + public void setKeyboard(final Keyboard keyboard, final int paddingLeft, final int paddingTop) { + mKeyboard = keyboard; + mPaddingLeft = paddingLeft; + mPaddingTop = paddingTop; + invalidateDrawingView(); + } + + /** Repaint hook for {@link AdaptiveKeyContext}'s change listener (fires on each keystroke). */ + public void onAdaptiveContextChanged() { + invalidateDrawingView(); + } + + @Override + public void onDeallocateMemory() { + mKeyboard = null; + } + + @Override + public void setPreviewPosition(@NonNull final PointerTracker tracker) { + // No-op: the overlay is derived from the keyboard + model, not a single pointer. + } + + @Override + public void drawPreview(@NonNull final Canvas canvas) { + if (!isPreviewEnabled()) return; // geometry valid + final Keyboard keyboard = mKeyboard; + if (keyboard == null) return; + final SettingsValues sv = Settings.getValues(); + if (sv == null || !sv.mAdaptiveDebugOverlay) return; + final boolean learn = sv.mAdaptiveKeyGeometry; + final boolean prior = sv.mAdaptiveContextPrior; + if (!learn && !prior) return; + final int strength = sv.mAdaptiveKeyGeometryStrength; + + // Learned-offset lookup is keyed by layout + orientation, like KeyDetector. + TouchModelDao dao = null; + String layout = null; + int orientation = 0; + if (learn) { + final Context ctx = Settings.getCurrentContext(); + if (ctx != null) { + dao = TouchModelDao.getInstance(ctx); + layout = Integer.toString(keyboard.mId.mElementId); + orientation = ctx.getResources().getConfiguration().orientation; + } + } + + for (final Key key : keyboard.getSortedKeys()) { + if (key.isSpacer()) continue; + final int code = key.getCode(); + if (code <= 0 || !Character.isLetter(code)) continue; + final int w = key.getWidth(); + final int h = key.getHeight(); + if (w <= 0 || h <= 0) continue; + final float cx = mPaddingLeft + key.getX() + w / 2f; + final float cy = mPaddingTop + key.getY() + h / 2f; + + float effX = cx; + float effY = cy; + boolean haveLearned = false; + if (dao != null) { + final TouchModelDao.Stat st = dao.get(code, layout, orientation); + if (st != null && st.getCount() >= TouchModelDao.MIN_CONFIDENT_SAMPLES) { + final float[] off = TouchModelManager.adjustedOffset(st, w, h, strength); + effX = cx + off[0]; + effY = cy + off[1]; + haveLearned = true; + } + } + + // Prior halo first, so the learned dot/arrow sit on top of it. + if (prior) { + final float weight = AdaptiveKeyContext.weight(code); + if (weight > 0f) { + final float r = weight * HALO_MAX_FRACTION * w; + mHaloFill.setAlpha(0x33); + canvas.drawCircle(effX, effY, r, mHaloFill); + mHaloStroke.setAlpha(0x99); + canvas.drawCircle(effX, effY, r, mHaloStroke); + } + } + + if (haveLearned) { + canvas.drawCircle(cx, cy, GEOM_DOT_RADIUS_PX, mGeomPaint); + final float dx = effX - cx; + final float dy = effY - cy; + if (dx * dx + dy * dy > MIN_ARROW_LENGTH_PX * MIN_ARROW_LENGTH_PX) { + canvas.drawLine(cx, cy, effX, effY, mArrowPaint); + } + canvas.drawCircle(effX, effY, EFF_DOT_RADIUS_PX, mEffPaint); + } + } + } +} diff --git a/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt b/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt index d899ec83a..c02b2079c 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt +++ b/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt @@ -161,6 +161,7 @@ object Defaults { const val PREF_ADAPTIVE_KEY_GEOMETRY = false const val PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH = 50 const val PREF_ADAPTIVE_CONTEXT_PRIOR = false + const val PREF_ADAPTIVE_DEBUG_OVERLAY = false const val PREF_SHOW_SETUP_WIZARD_ICON = true const val PREF_USE_CONTACTS = false const val PREF_USE_APPS = false diff --git a/app/src/main/java/helium314/keyboard/latin/settings/Settings.java b/app/src/main/java/helium314/keyboard/latin/settings/Settings.java index 93f68771a..a0e3c05fd 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/Settings.java +++ b/app/src/main/java/helium314/keyboard/latin/settings/Settings.java @@ -191,6 +191,9 @@ public final class Settings implements SharedPreferences.OnSharedPreferenceChang // Context prior: enlarge the touch target of the next key the suggestions predict. Independent // of the learned-geometry toggle above; both share the strength slider. See ADAPTIVE_TYPING.md. public static final String PREF_ADAPTIVE_CONTEXT_PRIOR = "adaptive_context_prior"; + // Debug visualization: draw the effective per-key targets (learned-offset shift + prior bulge) + // on the live keyboard so the adaptive behavior is visible as you type. See ADAPTIVE_TYPING.md. + public static final String PREF_ADAPTIVE_DEBUG_OVERLAY = "adaptive_debug_overlay"; public static final String PREF_SHOW_SETUP_WIZARD_ICON = "show_setup_wizard_icon"; public static final String PREF_USE_CONTACTS = "use_contacts"; public static final String PREF_USE_APPS = "use_apps"; diff --git a/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java b/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java index 02d3a3962..4b86f4618 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java +++ b/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java @@ -149,6 +149,7 @@ public class SettingsValues { public final boolean mAdaptiveKeyGeometry; public final int mAdaptiveKeyGeometryStrength; public final boolean mAdaptiveContextPrior; + public final boolean mAdaptiveDebugOverlay; public final boolean mSlidingKeyInputPreviewEnabled; public final int mKeyLongpressTimeout; public final boolean mEnableEmojiAltPhysicalKey; @@ -409,6 +410,9 @@ public SettingsValues(final Context context, final SharedPreferences prefs, fina mAdaptiveContextPrior = prefs.getBoolean( Settings.PREF_ADAPTIVE_CONTEXT_PRIOR, Defaults.PREF_ADAPTIVE_CONTEXT_PRIOR); + mAdaptiveDebugOverlay = prefs.getBoolean( + Settings.PREF_ADAPTIVE_DEBUG_OVERLAY, + Defaults.PREF_ADAPTIVE_DEBUG_OVERLAY); mSuggestionStripHiddenPerUserSettings = mToolbarMode == ToolbarMode.HIDDEN || mToolbarMode == ToolbarMode.TOOLBAR_KEYS; mOverrideShowingSuggestions = mInputAttributes.mMayOverrideShowingSuggestions diff --git a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt index 505ac04c4..1ea93a322 100644 --- a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt +++ b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt @@ -88,6 +88,7 @@ fun GestureTypingScreen( val learnOn = prefs.getBoolean(Settings.PREF_ADAPTIVE_KEY_GEOMETRY, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY) val priorOn = prefs.getBoolean(Settings.PREF_ADAPTIVE_CONTEXT_PRIOR, Defaults.PREF_ADAPTIVE_CONTEXT_PRIOR) if (learnOn || priorOn) add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH) + if (learnOn || priorOn) add(Settings.PREF_ADAPTIVE_DEBUG_OVERLAY) // visualize the effect on the live keyboard if (learnOn) add(SettingsWithoutKey.ADAPTIVE_TYPING_STATS) // the learned model only fills with the geometry half } SearchSettingsScreen( @@ -193,6 +194,12 @@ fun createGestureTypingSettings(context: Context) = listOf( R.string.adaptive_context_prior, R.string.adaptive_context_prior_summary) { SwitchPreference(it, Defaults.PREF_ADAPTIVE_CONTEXT_PRIOR) }, + Setting(context, Settings.PREF_ADAPTIVE_DEBUG_OVERLAY, + R.string.adaptive_debug_overlay, R.string.adaptive_debug_overlay_summary) { + SwitchPreference(it, Defaults.PREF_ADAPTIVE_DEBUG_OVERLAY) { + KeyboardSwitcher.getInstance().setThemeNeedsReload() // rebuild so the overlay attaches/detaches at once + } + }, Setting(context, Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH, R.string.adaptive_key_geometry_strength) { def -> SliderPreference( name = def.title, diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml index 2009204a1..40df2caa1 100644 --- a/app/src/main/res/values/strings.xml +++ b/app/src/main/res/values/strings.xml @@ -390,6 +390,8 @@ Adaptive typing Anticipate likely keys (experimental) Make the next key the suggestions predict slightly easier to press. Averages the top suggestions and is capped so it never overrides a clear press. Applies to taps, not swipes. + Show adaptive targets on keyboard (debug) + Draw each key\'s learned landing point and a halo that grows on keys the suggestions predict, so you can watch the adaptive model work as you type. Visual only — does not change typing. Learned typing model See what the keyboard has learned about your typing Per key: average offset from the key center (where you land), spread (how consistent you are), and number of samples. Higher samples and lower spread mean a stronger, more confident adjustment. Nothing here ever leaves your device. diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md index 9aeee09a4..ce45003fe 100644 --- a/docs/ADAPTIVE_TYPING.md +++ b/docs/ADAPTIVE_TYPING.md @@ -124,8 +124,8 @@ The literal tap is committed as-is, so the bias is **hard-capped** and confidenc ## Persistence, export/import, backup compatibility Local data lives in a single SQLite DB, `leantype.db` -(`latin/database/Database.kt`, raw `SQLiteOpenHelper`, currently VERSION 2; clipboard -lives there). The settings backup (`settings/preferences/BackupRestorePreference.kt`, +(`latin/database/Database.kt`, raw `SQLiteOpenHelper`, currently VERSION 4 — clipboard + +the `TOUCH_MODEL` table, which also stores per-key width/height for fraction-of-key math). The settings backup (`settings/preferences/BackupRestorePreference.kt`, Advanced tab) **already zips the entire `leantype.db` and restores it**, and restore is lenient (unknown zip entries skipped; missing columns handled by schema checks in `Database.copyFromDb` + `onUpgrade`). @@ -155,12 +155,48 @@ A visualization page (trust + the reset control live here): e.g. "Z corrected ~18% vs E ~2%." - **Reset** button. +The built page (`AdaptiveTypingStatsScreen`) renders the heatmap on a **mock keyboard** +(`MockKeyboardHeatmap`): each key shows its learned offset as a dot displaced from center +and a spread indicator, both expressed as a **fraction of the key**, so "18px on E" reads +as a visible nudge rather than an abstract number. + +## Live debug overlay ("see it in action") + +A debug toggle — **"Show adaptive targets on keyboard"** — draws the adaptive model +*directly on the live keyboard* so you can watch it work as you type. Implemented as +`AdaptiveTargetsDrawingPreview` (an `AbstractDrawingPreview`, the same overlay mechanism +as the gesture-debug points), drawn on the `DrawingPreviewPlacerView` above the keys: + +- **Learned geometry (Layer B):** for each letter key with a confident learned offset, a + faint ring marks the geometric center, an arrow points to the learned landing target, + and a filled dot marks it — the same shift `KeyDetector` biases taps toward. +- **Context prior (Layer A):** keys the current suggestions predict get a translucent + green halo whose radius grows with the prior weight. Because the prior is rebuilt + between keystrokes, the halos appear/grow/shrink **live as you type** — the keyboard + visibly "leans" toward the likely next key. + +It is purely visual (never changes detection), reads the same live model / prior / +settings the engine uses, and is gated on its pref each frame (zero cost when off). +Repaints are driven by an `AdaptiveKeyContext` change listener fired on each keystroke; +`MainKeyboardView` registers it and feeds the overlay the current keyboard + padding so +markers align with the rendered keys. The halo radius is intentionally exaggerated +relative to the engine's sub-key boost so the effect is legible. We deliberately do **not** +reflow/resize the visible keys (jarring, breaks muscle memory) — an overlay communicates +the same thing without destabilizing typing. + ## Configurability -- Master toggle (opt-in, default off). -- Strength slider (off → gentle tie-break → aggressive). -- Reset learned model. -- Honors incognito / no-learning fields. +All adaptive controls live under one **"Adaptive typing"** section in *Gesture typing* +settings. The two halves are **independently toggleable** (either alone, both, or neither): + +- **Adaptive key geometry** (Layer B, learned offsets) — opt-in, default off. +- **Anticipate likely keys** (Layer A, context prior) — opt-in, default off. +- **Strength slider** — shared by both; shown when either toggle is on (off → gentle + tie-break → aggressive). +- **Learned typing model** stats page (heatmap + reset) — shown when learning is on. +- **Show adaptive targets on keyboard (debug)** — live visualization overlay (below); + shown when either toggle is on. +- Honors incognito / no-learning fields; learning records nothing in those contexts. ## Implementation footprint @@ -215,8 +251,16 @@ stats page to keep it honest. Validate the gesture path explicitly (it's the pri The **learned geometry** (Layer B), by contrast, applies to the entire swipe including its start — every key the stroke passes is matched against its learned-shifted sweet spot — and a swipe's endpoints also feed the model (gesture-endpoint learning). -6. ⬜ **Strength/cap tuning** + optional heatmap visualization + interior-key gesture - learning (needs corner-cutting handling or native alignment). +6. ✅ **Independent context-prior toggle + grouped settings:** Layer A (context prior) is now + its own opt-in toggle ("Anticipate likely keys"), separate from Layer B (learned geometry); + both live under one "Adaptive typing" section and share the strength slider. `KeyDetector` + gates the learned-offset bias and the prior boost independently. +7. ✅ **Heatmap stats page:** `AdaptiveTypingStatsScreen` renders the learned model on a mock + keyboard (offsets as a fraction of each key) + reset. +8. ✅ **Live debug overlay:** `AdaptiveTargetsDrawingPreview` draws learned targets + prior + halos on the real keyboard, morphing as you type (toggle: "Show adaptive targets on keyboard"). +9. ⬜ **Strength/cap tuning** + interior-key gesture learning (needs corner-cutting handling or + native alignment). ## Open questions (tracked) From 06dd2e20f0b668882ba873421253f7cadd2b3672 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 08:21:44 -0700 Subject: [PATCH 13/15] Fix context-prior overlay lag and capital-letter bias On-device tracing showed the next-key bias is applied correctly in practice (each tap is biased by the prediction for the prefix typed so far), but two issues made it look/behave wrong: 1. Overlay trailed by ~one key. The prior is rebuilt only when the suggestion strip refreshes, which is debounced ~100 ms; the trace showed a consistent ~113 ms gap between a keystroke and its setSuggested, so for most of the inter-key interval the overlay still showed the previous prediction. When the context prior is enabled, shorten that debounce (PROMPT_PRIOR_UPDATE_DELAY_MS = 30 ms in LatinIME.UIHandler) so the prediction lands before the next tap. 2. Bias silently skipped capital letters. The prior stores lowercase next-characters, but a shifted keyboard reports uppercase key codes, so AdaptiveKeyContext.weight() missed them (priorOnGeo=0.0 on the sentence-initial capital). weight() now folds the queried code to lowercase, so the bias and the overlay halos work on the shifted layout too. Also adds debug-overlay-gated tracing (AdaptivePrior tag) in setSuggestedWords and KeyDetector, plus AdaptiveKeyContext.debugString(), to make this diagnosable on-device, and documents both fixes in docs/ADAPTIVE_TYPING.md. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/keyboard/AdaptiveKeyContext.java | 25 ++++++++++++++++--- .../keyboard/keyboard/KeyDetector.java | 8 ++++++ .../helium314/keyboard/latin/LatinIME.java | 15 ++++++++++- .../keyboard/latin/inputlogic/InputLogic.java | 13 ++++++++++ docs/ADAPTIVE_TYPING.md | 7 ++++++ 5 files changed, 63 insertions(+), 5 deletions(-) diff --git a/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java b/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java index 26871f2f9..b98efca1d 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java +++ b/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java @@ -97,13 +97,16 @@ public static void clear() { fireChanged(); } - /** Prior weight in [0, 1] for the given key code (0 if none / no prior). */ + /** Prior weight in [0, 1] for the given key code (0 if none / no prior). Case-insensitive: + * the prior stores lowercase next-characters, but a shifted keyboard reports uppercase key + * codes, so we fold to lowercase to match (otherwise the bias/overlay miss capital letters). */ public static float weight(final int code) { final int[] c = sCodes; final float[] w = sWeights; - if (c == null) return 0f; - for (int i = 0; i < c.length; i++) { - if (c[i] == code) return w[i]; + if (c == null || w == null) return 0f; + final int lower = Character.toLowerCase(code); + for (int i = 0; i < c.length && i < w.length; i++) { + if (c[i] == lower) return w[i]; } return 0f; } @@ -111,4 +114,18 @@ public static float weight(final int code) { public static boolean hasPrior() { return sCodes != null; } + + /** Human-readable snapshot of the current prior, e.g. {@code [e=0.60,o=0.40]}, for debug logs. */ + public static String debugString() { + final int[] c = sCodes; + final float[] w = sWeights; + if (c == null || w == null) return "(none)"; + final StringBuilder sb = new StringBuilder("["); + for (int i = 0; i < c.length && i < w.length; i++) { + if (i > 0) sb.append(','); + sb.append((char) c[i]).append('=') + .append(String.format(java.util.Locale.US, "%.2f", w[i])); + } + return sb.append(']').toString(); + } } diff --git a/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java b/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java index b7aa54c3d..720fed290 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java +++ b/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java @@ -13,6 +13,7 @@ import helium314.keyboard.latin.database.TouchModelManager; import helium314.keyboard.latin.settings.Settings; import helium314.keyboard.latin.settings.SettingsValues; +import helium314.keyboard.latin.utils.Log; /** * This class handles key detection. @@ -166,6 +167,13 @@ private Key applyAdaptiveBias(final int touchX, final int touchY, final Key geo) best = k; } } + if (sv.mAdaptiveDebugOverlay && hasPrior) { + Log.d("AdaptivePrior", "tap geo='" + (char) geo.getCode() + + "' priorOnGeo=" + AdaptiveKeyContext.weight(geo.getCode()) + + " chosen='" + (char) best.getCode() + "'" + + (best == geo ? "" : " (FLIPPED by bias)") + + " prior=" + AdaptiveKeyContext.debugString()); + } return best == geo ? null : best; } diff --git a/app/src/main/java/helium314/keyboard/latin/LatinIME.java b/app/src/main/java/helium314/keyboard/latin/LatinIME.java index cec4c8537..4c4f1210a 100644 --- a/app/src/main/java/helium314/keyboard/latin/LatinIME.java +++ b/app/src/main/java/helium314/keyboard/latin/LatinIME.java @@ -210,6 +210,11 @@ public static final class UIHandler extends LeakGuardHandlerWrapper { private static final int ARG2_UNUSED = 0; private static final int ARG1_TRUE = 1; + // When the adaptive context prior is enabled, suggestions are refreshed with this shorter + // debounce instead of mDelayInMillisecondsToUpdateSuggestions, so the prior (and its debug + // overlay) keep up with the keystroke instead of trailing it by ~one key. See ADAPTIVE_TYPING.md. + private static final long PROMPT_PRIOR_UPDATE_DELAY_MS = 30; + private int mDelayInMillisecondsToUpdateSuggestions; private int mDelayInMillisecondsToUpdateShiftState; @@ -294,8 +299,16 @@ public void handleMessage(@NonNull final Message msg) { } public void postUpdateSuggestionStrip(final int inputStyle) { + long delay = mDelayInMillisecondsToUpdateSuggestions; + final LatinIME latinIme = getOwnerInstance(); + if (latinIme != null) { + final SettingsValues sv = latinIme.mSettings.getCurrent(); + if (sv != null && sv.mAdaptiveContextPrior) { + delay = Math.min(delay, PROMPT_PRIOR_UPDATE_DELAY_MS); + } + } sendMessageDelayed(obtainMessage(MSG_UPDATE_SUGGESTION_STRIP, inputStyle, - 0 /* ignored */), mDelayInMillisecondsToUpdateSuggestions); + 0 /* ignored */), delay); } public void postReopenDictionaries() { diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java index 5536ba519..da89829f2 100644 --- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java +++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java @@ -1385,6 +1385,19 @@ public void setSuggestedWords(final SuggestedWords suggestedWords) { final int nextCharIndex = mWordComposer.isComposingWord() ? mWordComposer.getTypedWord().length() : 0; helium314.keyboard.keyboard.AdaptiveKeyContext.update(suggestedWords, nextCharIndex); + if (svAdaptive.mAdaptiveDebugOverlay) { + final StringBuilder words = new StringBuilder(); + for (int i = 0; i < Math.min(5, suggestedWords.size()); i++) { + if (i > 0) words.append('|'); + words.append(suggestedWords.getWord(i)); + } + Log.d("AdaptivePrior", "setSuggested style=" + suggestedWords.mInputStyle + + " composing=" + mWordComposer.isComposingWord() + + " typed='" + mWordComposer.getTypedWord() + "'" + + " pos=" + nextCharIndex + + " words=[" + words + "]" + + " -> prior=" + helium314.keyboard.keyboard.AdaptiveKeyContext.debugString()); + } } else { helium314.keyboard.keyboard.AdaptiveKeyContext.clear(); } diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md index ce45003fe..75ccaf2a6 100644 --- a/docs/ADAPTIVE_TYPING.md +++ b/docs/ADAPTIVE_TYPING.md @@ -78,6 +78,13 @@ current position weighted by the candidate's score; sum per letter. Example: typ `H`, candidates `Hello`/`Hey`/`He` → `e` dominates → **E's tap target grows slightly for the next tap.** Recomputed live; never persisted. +The prior is rebuilt from the suggestion strip, which is normally debounced ~100 ms; that +made the prior (and the debug overlay) trail the keystroke by ~one key. When the prior is +enabled we shorten that debounce (`PROMPT_PRIOR_UPDATE_DELAY_MS` in `LatinIME.UIHandler`) +so the prediction is ready before the next tap. Lookups are case-insensitive +(`AdaptiveKeyContext.weight` folds to lowercase) so the bias also applies on the shifted +keyboard, where keys report uppercase codes. + For **gestures** there is no single "next key" to enlarge mid-stroke, so the context prior is tap-only. The contextual-likelihood part for gestures is already handled by the native language model (word-level). The new gesture win is Layer B (learned From f6657db6ed8125fc0857fc4e5f318d01f6b27fca Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 08:50:38 -0700 Subject: [PATCH 14/15] Scope prior debounce reduction to the debug overlay only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The suggestion-update debounce (~100 ms) exists because the suggestion compute blocks the UI thread (performUpdateSuggestionStripSync waits on holder.get with a 200 ms timeout); the debounce coalesces that expensive compute so fast typing isn't hit by one block per keystroke. The functional context-prior bias already works at the full debounce (at normal speed the prior is ready before the next tap). Only the debug overlay visibly trails. So instead of shortening the debounce whenever the prior is enabled, shorten it only while the debug overlay is also on — a temporary diagnostic — and raise the value from 30 ms to 50 ms. Ordinary typing (overlay off) keeps the full debounce and is unaffected. Co-Authored-By: Claude Opus 4.8 --- .../java/helium314/keyboard/latin/LatinIME.java | 14 +++++++++----- docs/ADAPTIVE_TYPING.md | 16 ++++++++++------ 2 files changed, 19 insertions(+), 11 deletions(-) diff --git a/app/src/main/java/helium314/keyboard/latin/LatinIME.java b/app/src/main/java/helium314/keyboard/latin/LatinIME.java index 4c4f1210a..e0d81325b 100644 --- a/app/src/main/java/helium314/keyboard/latin/LatinIME.java +++ b/app/src/main/java/helium314/keyboard/latin/LatinIME.java @@ -210,10 +210,12 @@ public static final class UIHandler extends LeakGuardHandlerWrapper { private static final int ARG2_UNUSED = 0; private static final int ARG1_TRUE = 1; - // When the adaptive context prior is enabled, suggestions are refreshed with this shorter - // debounce instead of mDelayInMillisecondsToUpdateSuggestions, so the prior (and its debug - // overlay) keep up with the keystroke instead of trailing it by ~one key. See ADAPTIVE_TYPING.md. - private static final long PROMPT_PRIOR_UPDATE_DELAY_MS = 30; + // While the adaptive debug overlay is on, suggestions are refreshed with this shorter + // debounce instead of mDelayInMillisecondsToUpdateSuggestions, so the visualized prior keeps + // up with the keystroke instead of trailing it by ~one key. Scoped to the overlay (a + // temporary diagnostic) on purpose: the normal ~100 ms debounce coalesces the UI-blocking + // suggestion compute, so we don't shorten it during ordinary typing. See ADAPTIVE_TYPING.md. + private static final long PROMPT_PRIOR_UPDATE_DELAY_MS = 50; private int mDelayInMillisecondsToUpdateSuggestions; private int mDelayInMillisecondsToUpdateShiftState; @@ -303,7 +305,9 @@ public void postUpdateSuggestionStrip(final int inputStyle) { final LatinIME latinIme = getOwnerInstance(); if (latinIme != null) { final SettingsValues sv = latinIme.mSettings.getCurrent(); - if (sv != null && sv.mAdaptiveContextPrior) { + // Only when actively visualizing the prior: overlay on AND prior on. Ordinary + // typing keeps the full debounce so fast typists aren't hit by extra computes. + if (sv != null && sv.mAdaptiveDebugOverlay && sv.mAdaptiveContextPrior) { delay = Math.min(delay, PROMPT_PRIOR_UPDATE_DELAY_MS); } } diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md index 75ccaf2a6..fea5bd888 100644 --- a/docs/ADAPTIVE_TYPING.md +++ b/docs/ADAPTIVE_TYPING.md @@ -78,12 +78,16 @@ current position weighted by the candidate's score; sum per letter. Example: typ `H`, candidates `Hello`/`Hey`/`He` → `e` dominates → **E's tap target grows slightly for the next tap.** Recomputed live; never persisted. -The prior is rebuilt from the suggestion strip, which is normally debounced ~100 ms; that -made the prior (and the debug overlay) trail the keystroke by ~one key. When the prior is -enabled we shorten that debounce (`PROMPT_PRIOR_UPDATE_DELAY_MS` in `LatinIME.UIHandler`) -so the prediction is ready before the next tap. Lookups are case-insensitive -(`AdaptiveKeyContext.weight` folds to lowercase) so the bias also applies on the shifted -keyboard, where keys report uppercase codes. +The prior is rebuilt from the suggestion strip, which is normally debounced ~100 ms (that +debounce exists because the suggestion compute *blocks the UI thread* — see +`InputLogic.performUpdateSuggestionStripSync`, which waits on `holder.get(..., 200 ms)` — so +it coalesces the expensive compute during fast typing). At that cadence the prior is still +ready before the next tap at normal speed (the functional bias works), but the *debug +overlay* visibly trails by ~one key. So **only while the debug overlay is on** we shorten +the debounce to 50 ms (`PROMPT_PRIOR_UPDATE_DELAY_MS` in `LatinIME.UIHandler`) so the +visualization keeps up; ordinary typing (overlay off) keeps the full debounce and is +unaffected. Lookups are case-insensitive (`AdaptiveKeyContext.weight` folds to lowercase) +so the bias also applies on the shifted keyboard, where keys report uppercase codes. For **gestures** there is no single "next key" to enlarge mid-stroke, so the context prior is tap-only. The contextual-likelihood part for gestures is already handled by From 6ae3996620995e68a16e8c2fb535c309c83c201d Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 09:12:02 -0700 Subject: [PATCH 15/15] Compute prior immediately while debug overlay is on MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 50 ms (and even 30 ms) overlay debounce was easy to out-type. A truly non-blocking async refresh isn't safe here: the suggestion compute reads the non-thread-safe WordComposer on a background thread, and the existing design keeps that safe by blocking the UI during the compute (the only existing async compute runs during gestures, when there's no concurrent typing). Doing it async during typing would risk a torn read / crash and would need a composer snapshot — too much for a debug visualization. Instead, drop the suggestion debounce to 0 while the debug overlay + the context prior are both on. The overlay already repaints the instant the prior updates (the AdaptiveKeyContext change listener fires inside setSuggestedWords), so computing immediately is the safe equivalent of "update as soon as the suggestion is made". The only remaining latency is the compute itself (~5-10 ms release, ~20 ms debug build). Ordinary typing (overlay off) keeps the full, smooth debounce. Reverts the earlier PROMPT_PRIOR_UPDATE_DELAY_MS approach. Co-Authored-By: Claude Opus 4.8 --- .../helium314/keyboard/latin/LatinIME.java | 28 +++++++++---------- docs/ADAPTIVE_TYPING.md | 12 ++++++-- 2 files changed, 23 insertions(+), 17 deletions(-) diff --git a/app/src/main/java/helium314/keyboard/latin/LatinIME.java b/app/src/main/java/helium314/keyboard/latin/LatinIME.java index e0d81325b..378ff33cd 100644 --- a/app/src/main/java/helium314/keyboard/latin/LatinIME.java +++ b/app/src/main/java/helium314/keyboard/latin/LatinIME.java @@ -210,13 +210,6 @@ public static final class UIHandler extends LeakGuardHandlerWrapper { private static final int ARG2_UNUSED = 0; private static final int ARG1_TRUE = 1; - // While the adaptive debug overlay is on, suggestions are refreshed with this shorter - // debounce instead of mDelayInMillisecondsToUpdateSuggestions, so the visualized prior keeps - // up with the keystroke instead of trailing it by ~one key. Scoped to the overlay (a - // temporary diagnostic) on purpose: the normal ~100 ms debounce coalesces the UI-blocking - // suggestion compute, so we don't shorten it during ordinary typing. See ADAPTIVE_TYPING.md. - private static final long PROMPT_PRIOR_UPDATE_DELAY_MS = 50; - private int mDelayInMillisecondsToUpdateSuggestions; private int mDelayInMillisecondsToUpdateShiftState; @@ -302,13 +295,20 @@ public void handleMessage(@NonNull final Message msg) { public void postUpdateSuggestionStrip(final int inputStyle) { long delay = mDelayInMillisecondsToUpdateSuggestions; - final LatinIME latinIme = getOwnerInstance(); - if (latinIme != null) { - final SettingsValues sv = latinIme.mSettings.getCurrent(); - // Only when actively visualizing the prior: overlay on AND prior on. Ordinary - // typing keeps the full debounce so fast typists aren't hit by extra computes. - if (sv != null && sv.mAdaptiveDebugOverlay && sv.mAdaptiveContextPrior) { - delay = Math.min(delay, PROMPT_PRIOR_UPDATE_DELAY_MS); + // While the debug overlay + context prior are both on, compute the prediction + // immediately (no debounce) so the visualization keeps up with fast typing. The overlay + // repaints the instant the prior updates, so "no debounce" is the safe equivalent of + // "update as soon as the suggestion is made". The remaining floor is the suggestion + // compute itself, which briefly blocks the UI thread by design (see + // performUpdateSuggestionStripSync) — fast on release, slower on debug builds. Scoped to + // the overlay so ordinary typing keeps the full, smooth debounce. + if (inputStyle == SuggestedWords.INPUT_STYLE_TYPING) { + final LatinIME latinIme = getOwnerInstance(); + if (latinIme != null) { + final SettingsValues sv = latinIme.mSettings.getCurrent(); + if (sv != null && sv.mAdaptiveDebugOverlay && sv.mAdaptiveContextPrior) { + delay = 0; + } } } sendMessageDelayed(obtainMessage(MSG_UPDATE_SUGGESTION_STRIP, inputStyle, diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md index fea5bd888..23565e5fa 100644 --- a/docs/ADAPTIVE_TYPING.md +++ b/docs/ADAPTIVE_TYPING.md @@ -83,9 +83,15 @@ debounce exists because the suggestion compute *blocks the UI thread* — see `InputLogic.performUpdateSuggestionStripSync`, which waits on `holder.get(..., 200 ms)` — so it coalesces the expensive compute during fast typing). At that cadence the prior is still ready before the next tap at normal speed (the functional bias works), but the *debug -overlay* visibly trails by ~one key. So **only while the debug overlay is on** we shorten -the debounce to 50 ms (`PROMPT_PRIOR_UPDATE_DELAY_MS` in `LatinIME.UIHandler`) so the -visualization keeps up; ordinary typing (overlay off) keeps the full debounce and is +overlay* visibly trails by ~one key. So **only while the debug overlay is on** we drop the +debounce to 0 in `LatinIME.UIHandler.postUpdateSuggestionStrip` (compute the prediction +immediately); since the overlay repaints the instant the prior updates, that is the safe +equivalent of "update as soon as the suggestion is made". The remaining floor is the +compute itself (~5–10 ms release, ~20 ms debug), which briefly blocks the UI by design. A +fully non-blocking async refresh would be snappier still, but the suggestion compute reads +the non-thread-safe `WordComposer` on the background thread — the blocking design exists to +avoid that race during typing — so doing it safely would need a composer snapshot (not worth +it for a debug visualization). Ordinary typing (overlay off) keeps the full debounce and is unaffected. Lookups are case-insensitive (`AdaptiveKeyContext.weight` folds to lowercase) so the bias also applies on the shifted keyboard, where keys report uppercase codes.