Skip to content

Adopt Maestro-style state-change-aware touch retry and lazy settle policy #597

@thymikee

Description

@thymikee

Parent

Follow-up from #581.

What to build

Model Maestro's most useful execution behavior as a generic Agent Device reliability/performance policy: after a touch command that is expected to mutate UI, detect whether the surface actually changed, retry likely no-op taps within a small budget, and then settle lazily only when the next command needs fresh UI state.

This should not be implemented as more Maestro-only compatibility branching. The goal is to turn the behavior we validated through Maestro suites into a reusable Agent Device runtime primitive used by replay/test flows and available to normal command paths where it improves reliability without adding fixed sleeps.

Why

The React Navigation and Bluesky validation showed that Agent Device is already fast when it avoids unnecessary snapshots and fixed waits. The remaining Maestro behavior worth incorporating is not YAML syntax; it is the runner model:

  • taps can be issued before the UI is fully ready, so a successful low-level tap can still be a semantic no-op
  • some flows rely on "tap until the app moved" rather than "tap returned successfully"
  • settling should be demand-driven, because unconditional post-command waits erase Agent Device's performance advantage
  • diagnostics should explain when retries or slow snapshots are the reason a flow got slower

We already have partial pieces:

  • Maestro tapOn retries while target resolution/click fails
  • assertVisible caches a snapshot for a following tapOn
  • post-gesture stabilization waits for stable snapshots after scroll/swipe
  • Android snapshot freshness retries suspicious post-action snapshots

Those pieces should be reconciled into one explicit interaction outcome policy.

Acceptance criteria

  • Add an internal interaction outcome policy that can compare a pre-action and post-action surface signature cheaply enough to detect "tap returned, but UI did not change".
  • Reuse existing snapshot/reference-frame caches where possible so the policy does not add an extra full snapshot in the common passing path.
  • Retry no-op touch actions only for bounded, explicit contexts such as replay/test or an internal action flag; do not make every manual press slower by default.
  • Make settle demand-driven: if the next command needs fresh UI state, wait for a stable signature; if it does not, do not sleep.
  • Fold the existing Maestro-only tap retry and generic post-gesture stabilization paths into this policy where practical, without losing current behavior.
  • Emit diagnostics for interaction_no_change_retry, interaction_settled, and interaction_settle_timeout, including attempts and duration.
  • Add provider-backed integration coverage for a first tap that is a semantic no-op and succeeds after retry.
  • Add a regression benchmark or timing assertion showing the common successful path does not regress from the current Maestro replay timings.

Non-goals

  • Do not copy Maestro's coordinate quirks or app-specific workarounds into generic Agent Device behavior.
  • Do not add fixed sleeps after every command.
  • Do not expand Maestro YAML syntax surface in this issue unless it falls out naturally from the shared policy.

Blocked by

None - can start immediately.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions