feat(typing): adaptive learned key geometry (taps + gestures)#3
Merged
Conversation
Captures the opt-in learned key-geometry feature: one per-user model behind both taps (KeyDetector) and gestures (ProximityInfo sweet spots, Java-only / no native rebuild), the context-prior + learned-geometry layers, hard caps to avoid wrong literals, privacy (content-free, incognito-gated), leantype.db persistence riding the existing backup, a stats page, and a phased build order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…store) Phase 1 of #1 (no behavior yet; nothing reads the model). Adds the opt-in prefs PREF_ADAPTIVE_KEY_GEOMETRY (+ strength) via the 5-file pattern, and a content-free per-(key, layout, orientation) touch-model table in leantype.db with a cached DAO (EMA running mean/variance + count, restore, clear). DB VERSION 2->3 with an additive onUpgrade, and copyFromDb extended to carry the model across the existing settings backup/restore (guarded so older backups without the table restore fine). Compiles: :app:compileStandardDebugJavaWithJavac. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 2 policy layer for #1: capped, confidence- and strength-scaled landing-offset math (pure functions). The cap (MAX_SHIFT_FRACTION of key size) is the safety bound so a learned bias can never flip a clear press to a neighbor; confidence ramps the bias in with sample count; strength scales it (0 = off). 7 unit tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…on + settings Phase 2 of #1 (testable end-to-end). Closes the loop: - PointerTracker records each letter tap's landing offset into the touch model (opt-in + incognito gated; async DB write so typing stays fast). - ProximityInfo shifts per-key sweet spots by the capped learned offset, so gesture recognition AND tap-correction follow where the user actually types. Generated even when a layout has no static touch-position-correction data. Element id passed from Keyboard; computed in Java, crosses the existing JNI - no native change. - Gesture typing -> Advanced: opt-in toggle + strength slider (reload keyboard on change). - @JvmStatic on the Kotlin DAO factory / manager so the Java hot paths can call them. Deferred: literal-tap KeyDetector tie-break (marginal/risky), the completion-derived context prior, and the stats page. Builds: :app:assembleStandardDebug. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Owner
Author
|
Phases 1 + 2 landed and build/test green:
Deferred (follow-ups): literal-tap |
…page #1, follow-ups to the adaptive-geometry feature: - Gestures now teach the model (not just consume it) via their clean endpoints: finger-down -> word's first letter, finger-up -> last letter. Interior keys skipped (corner-cutting); fresh single strokes only. InputLogic.maybeRecordGestureEndpoints. - New stats screen (AdaptiveTypingStatsScreen) reachable from Gesture typing -> Advanced when the feature is on: per-key average offset, spread (consistency), sample count, plus a Reset button. Wired via SettingsDestination + a clickable Preference row. Build + SettingsContainerTest green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ure lifecycle WordComposer.mExtendBatchInputBase (the multi-part merged-trail base) was only cleared by a normally-completing gesture, so an abnormal end (cancel / empty-top recognition) left it armed, and no deletion path cleared it. A later fresh swipe then merged with the ghost trail - most visibly at the start of a text box. Clear it at the same word-end sites where mLiveStroke is already dropped (handleBackspaceEvent, resetComposingState, commitChosenWord, desync) plus the two gesture-lifecycle origins (onStartBatchInput top, onCancelBatchInput). Does not touch the hot WordComposer.reset() path. Adds 8 tests: base cleared after each backspace mode (character/fragment/whole-word), the delete slider, fresh onStartBatchInput, and onCancelBatchInput; plus 2 guards pinning the (currently dead) static-seed interlock. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… page #1. The raw px-delta list was hard to interpret, so show the data spatially: - Store each key's size (KEY_WIDTH/KEY_HEIGHT) with its offset so the offset can be expressed as a fraction of the key (DB VERSION 3->4; additive ALTER on upgrade; copyFromDb reads by column name so older backups restore fine). - Stats page now renders a mock QWERTY where each key shows a dot at where you tend to land (offset as a fraction of the key) plus a faint spread ring; confident keys in the accent color, still-learning keys faded. The numeric list stays below for exact values. - Recorders (tap + gesture-endpoint) now pass the key's hitbox size. Build + extendBase/manager/settings tests green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…recreate-on-upgrade step
Completes adaptive typing on the tap side: - AdaptiveKeyContext builds a "likely next key" prior from the top-5 suggestions, weighted equally (averaged, not score-skewed): the next char of the in-progress word's completions, or the first char of next-word predictions for a fresh word. Rebuilt between keystrokes in InputLogic.setSuggestedWords (off the tap hot path), read lock-free per tap. - KeyDetector.detectHitKey now biases the tapped key by the learned per-key landing offset (shifts the effective center) AND the context prior (enlarges likely keys). Both capped; the prior cap (~18% of key) is deliberately below the learned cap (~25%) so it nudges rather than dominates. Only near-boundary taps can flip; clear presses are untouched. - Suppressed during gestures/swipes (PointerTracker.isInGestureOrKeySwipe); gestures are not context-biased since suggestions don't change mid-swipe. Applies to both current words being built and the first key of a new word (predictions). Build + extendBase/manager/settings tests green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… swipe-start behavior
Split the next-key context prior into its own opt-in setting (adaptive_context_prior, default off) alongside the existing learned key-geometry toggle, and group both under a single "Adaptive typing" section in the Gesture typing settings. - Settings/Defaults/SettingsValues: new mAdaptiveContextPrior flag. - KeyDetector: independently gate learned-offset bias vs prior boost; either alone can bias a near-boundary tap, both share the strength slider. adjustedDistance now takes usePrior and gates the boost. - InputLogic.setSuggestedWords: build the prior only when the prior toggle is on; clear AdaptiveKeyContext otherwise. - GestureTypingScreen: new "Adaptive typing" category holding both toggles; strength slider shown if either is on; stats shown if learning is on. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A debug toggle ("Show adaptive targets on keyboard") that visualizes the
adaptive model directly on the live keyboard, so the feature is visible as
you type:
- Learned geometry: each letter key shows a faint geometric-center ring, an
arrow to its learned landing target, and a dot at that target.
- Context prior: keys the suggestions predict get a translucent halo whose
radius grows with the prior weight; halos morph between keystrokes.
Implemented as AdaptiveTargetsDrawingPreview (an AbstractDrawingPreview, same
mechanism as the gesture-debug overlay), drawn on the DrawingPreviewPlacerView
above the keys. It is purely visual, reads the same live model / prior /
settings the engine uses, and is gated on its pref each frame (zero cost when
off). The halo radius is exaggerated vs the engine's sub-key boost for
legibility; the visible keys are deliberately not reflowed.
- New pref PREF_ADAPTIVE_DEBUG_OVERLAY (5-file), shown under "Adaptive typing"
when either adaptive toggle is on.
- AdaptiveKeyContext gains a change listener fired on update()/clear(); the
overlay repaints on it. MainKeyboardView registers the listener and feeds
the overlay the keyboard + padding so markers align with rendered keys.
- docs/ADAPTIVE_TYPING.md: document the overlay, the independent context-prior
toggle, the grouped settings, the built heatmap, and fix the DB version.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On-device tracing showed the next-key bias is applied correctly in practice (each tap is biased by the prediction for the prefix typed so far), but two issues made it look/behave wrong: 1. Overlay trailed by ~one key. The prior is rebuilt only when the suggestion strip refreshes, which is debounced ~100 ms; the trace showed a consistent ~113 ms gap between a keystroke and its setSuggested, so for most of the inter-key interval the overlay still showed the previous prediction. When the context prior is enabled, shorten that debounce (PROMPT_PRIOR_UPDATE_DELAY_MS = 30 ms in LatinIME.UIHandler) so the prediction lands before the next tap. 2. Bias silently skipped capital letters. The prior stores lowercase next-characters, but a shifted keyboard reports uppercase key codes, so AdaptiveKeyContext.weight() missed them (priorOnGeo=0.0 on the sentence-initial capital). weight() now folds the queried code to lowercase, so the bias and the overlay halos work on the shifted layout too. Also adds debug-overlay-gated tracing (AdaptivePrior tag) in setSuggestedWords and KeyDetector, plus AdaptiveKeyContext.debugString(), to make this diagnosable on-device, and documents both fixes in docs/ADAPTIVE_TYPING.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The suggestion-update debounce (~100 ms) exists because the suggestion compute blocks the UI thread (performUpdateSuggestionStripSync waits on holder.get with a 200 ms timeout); the debounce coalesces that expensive compute so fast typing isn't hit by one block per keystroke. The functional context-prior bias already works at the full debounce (at normal speed the prior is ready before the next tap). Only the debug overlay visibly trails. So instead of shortening the debounce whenever the prior is enabled, shorten it only while the debug overlay is also on — a temporary diagnostic — and raise the value from 30 ms to 50 ms. Ordinary typing (overlay off) keeps the full debounce and is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 50 ms (and even 30 ms) overlay debounce was easy to out-type. A truly non-blocking async refresh isn't safe here: the suggestion compute reads the non-thread-safe WordComposer on a background thread, and the existing design keeps that safe by blocking the UI during the compute (the only existing async compute runs during gestures, when there's no concurrent typing). Doing it async during typing would risk a torn read / crash and would need a composer snapshot — too much for a debug visualization. Instead, drop the suggestion debounce to 0 while the debug overlay + the context prior are both on. The overlay already repaints the instant the prior updates (the AdaptiveKeyContext change listener fires inside setSuggestedWords), so computing immediately is the safe equivalent of "update as soon as the suggestion is made". The only remaining latency is the compute itself (~5-10 ms release, ~20 ms debug build). Ordinary typing (overlay off) keeps the full, smooth debounce. Reverts the earlier PROMPT_PRIOR_UPDATE_DELAY_MS approach. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SHAWNERZZ
force-pushed
the
feat/adaptive-key-geometry
branch
from
219b964 to
5f709ef
Compare
SHAWNERZZ
changed the title
Owner
Author
|
Merging the adaptive typing feature to main — validated on-device and happy with the behavior. Rebuilt on the latest main (which now includes the refined live-converge casing fix from #6); unit suite green except the 3 documented pre-existing failures. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft / work-in-progress. Tracks #1.
Opt-in adaptive learned key geometry for taps and gestures. Design:
docs/ADAPTIVE_TYPING.md.Phase checklist
leantype.dbtouch-model table + DAO +TouchModelManagerProximityInfosweet spots) + capped tap tie-break (KeyDetector)Will stay in draft until it's testable end-to-end.