feat: Android Agent MVP — native app + reverse connection + server integration#336
feat: Android Agent MVP — native app + reverse connection + server integration#336suyiiyii wants to merge 50 commits into
Conversation
…t-unified # Conflicts: # tests/test_cancel_device_release.py
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🚀 Lint & Format Check Results: ✅ PASSEDWhat was checked:
✅ All checks passed!Your code follows the project's style guidelines. Checks performed:
This is an automated check. For questions, ask in the PR discussion. |
…bug UI Adds a real end-to-end test that boots an Android emulator, installs the debug APK, pairs it with a live backend and asserts that screenshot / current_app / tap actually work through the reverse command channel — the one path that exercises the real Kotlin app (MediaProjection + AccessibilityService), which unit/contract tests can't cover. - MainActivity: debug-only intent hook (test_server_url / test_pairing_code) so CI can claim a pairing without driving the UI. Gated on FLAG_DEBUGGABLE, ignored entirely in release builds. - scripts/android_agent_e2e_check.py: deterministic setup (accessibility via settings, MediaProjection pre-granted via appops to skip the consent dialog) + device-effect assertions; dumps logcat/screenshot on failure. - .github/workflows/android-agent-e2e.yml: emulator-runner job (api-35, x86_64), non-blocking by design. - UI: move the technical/debug panel (Technical state, Start/Stop, Tap target, swipe sandbox) behind a "developer options" toggle so the main flow ends at pairing. The e2e tap assertion uses this toggle as its target.
android-emulator-runner executes its `script` line-by-line, so the backslash line-continuation split the backend launch command (autoglm-gui got a literal `\` argument) and per-line `export`s did not persist. Move the whole flow into scripts/run_android_agent_e2e_ci.sh and invoke it in one line. The previous run confirmed the emulator infra works in CI (x86_64 boots under KVM in ~40s); only the inline shell was broken.
|
Claude finished @suyiiyii's task in 6m 19s —— View job Review completeI re-read the server registry/API, device integration, Android native app, frontend sidebar, and contract tests at HEAD ( I attempted to run FindingsP0 — Pairing endpoints remain unauthenticated
P1 — Android local HTTP server has no authentication
P1 — Agent token stored in plaintext SharedPreferences
P1 —
|
In CI the websocket reaches "connected" ~1.4s before the appops-approved MediaProjection consent activity returns RESULT_OK, so the first screenshot command raced ahead and got "Screen capture permission has not been granted". Poll the screenshot command until it succeeds, same as the current_app gate. CI run 27553475670 confirmed appops auto-grant works on the x86_64 emulator (logcat: capturePermissionLauncher resultCode=-1); the only issue was timing.
|
Claude finished @suyiiyii's task in 2m 49s —— View job Review completeI re-read the server registry/API, device integration, Android native app, frontend sidebar, and contract tests at HEAD ( FindingsP0 — Pairing endpoints remain unauthenticated
P1 — Android local HTTP server has no authentication
P1 — Agent token stored in plaintext SharedPreferences
P1 —
|
current_app failed in CI with accessibility_service_not_connected: setting enabled_accessibility_services before the freshly-installed app had ever run does not bind the service, and re-writing the same value is a no-op. Now: - enable accessibility, but force a rebind (toggle accessibility_enabled 0->1) only after the app process is alive (post am start); - re-kick on every current_app retry until the service connects. Verified locally from a cleared-accessibility state: ALL CHECKS PASSED.
|
Claude finished @suyiiyii's task in 2m 59s —— View job Review plan
Review completeI re-read the server registry/API, device integration, Android native app, frontend sidebar, and contract tests at HEAD ( FindingsP0 — Pairing endpoints remain unauthenticated
P1 — Android local HTTP server has no authentication
P1 — Agent token stored in plaintext SharedPreferences
P1 —
|
The tap assertion scrolls to developerToggleButton, but the swipe used
hard-coded large-screen coordinates (540,1800). CI emulators run at 320x640,
so those land off-screen and the toggle never scrolls into view
("could not find on-screen node developerToggleButton"). Compute swipe
coordinates from `wm size` instead.
CI progress: screenshot + current_app now pass on x86_64; this was the last
hard-coded assumption. Verified locally (1080x2400): ALL CHECKS PASSED.
|
Claude finished @suyiiyii's task in 3m 45s —— View job Review completeI re-read the server registry/API, device integration, Android native app, frontend sidebar, and contract tests at HEAD ( I attempted to run FindingsP0 — Pairing endpoints remain unauthenticated
P1 — Android local HTTP server has no authentication
P1 — Agent token stored in plaintext SharedPreferences
P1 —
|
The tap assertion kept failing on the CI emulator (320x640). Three fixes: - scroll_to now accepts the node only when its center is in the on-screen tappable band and nudges up/down, instead of tapping whatever the dump reports first (uiautomator lists off-screen scroll children too). - find_node matches negative bounds, so an overshoot (node scrolled above the fold) is detectable and recoverable. - screen_size prefers "Override size" over "Physical size". Reproduced CI's 320x640 locally (adb wm size 320x640) and passed twice.
|
Claude finished @suyiiyii's task in 5m 18s —— View job
Review plan
Review completeI re-read the server registry/API, device integration, Android native app, frontend sidebar, and contract tests at HEAD ( Several earlier P0/P1 issues were already addressed in I did not run FindingsP0 — Pairing endpoints remain unauthenticated
P1 — Android local HTTP server has no authentication
P1 — Agent token stored in plaintext SharedPreferences
P1 —
|
The e2e test step itself passed on the x86_64 CI emulator (run 27556963480, step "Run Android Agent E2E" = success). The job was only marked failed by the setup-uv post-job step, where `uv cache prune` hung for 5 minutes and was killed (exit 2). Disable uv caching for this job; uv sync is fast enough.
|
Claude finished @suyiiyii's task in 3m 12s —— View job Review plan
Review completeI read the server registry/API, device integration, Android native app, frontend sidebar, and contract tests at HEAD ( Several earlier P0/P1 issues were already addressed in previous commits: the Android HTTP server now binds to The remaining blockers are mostly security boundaries, missing FindingsP0 — Pairing endpoints remain unauthenticated
P1 — Android local HTTP server has no authentication
P1 — Agent token stored in plaintext SharedPreferences
P1 —
|
|
Claude finished @suyiiyii's task in 3m 28s —— View job Review completeI read the server registry/API, device integration, Android native app, frontend sidebar, and tests at HEAD ( I was unable to run FindingsP0 — Pairing endpoints remain unauthenticated
P1 — Android local HTTP server has no authentication
P1 — Agent token stored in plaintext SharedPreferences
P1 —
|
…tests
- ReverseAgentDevice._send now checks result.success and raises a clear error
(with result.error) instead of returning an empty payload that crashes
callers like get_screenshot/get_current_app with KeyError. (P1)
- ReverseAgentRegistry.remove_agent closes the WebSocket safely when called
from a sync route: it records each session's event loop and uses
run_coroutine_threadsafe when no loop is running in the caller thread,
instead of asyncio.create_task (which raises off the event loop). (P1)
- DELETE /api/reverse_agents/registry/{id} now also removes the device from
DeviceManager, so it no longer lingers in /api/devices as DISCONNECTED. (P2)
- DeviceManager.unregister_reverse_agent drops the ReverseAgentDevice instance
so disconnect/reconnect cycles don't leak. (P2)
- _cleanup_stale_agents_locked now also prunes claimed-but-never-connected
agents stuck in "paired" past the stale TTL. (P2)
- Restore the cancel-device-release regression tests (issue #172 / PR #317)
that were dropped, and keep the new chat-cancel test (8 tests total). (P1)
|
Claude finished @suyiiyii's task in 2m 39s —— View job Review completeI read the server registry/API, device integration, Android native app, frontend sidebar, and contract tests at HEAD ( The remaining blockers are mostly security boundaries, missing FindingsP0 — Pairing endpoints remain unauthenticated
P1 — Android local HTTP server has no authentication
P1 — Agent token stored in plaintext SharedPreferences
P1 —
|
# Conflicts: # docs/docs/configuration.md # docs/docs/intro.md
|
Claude finished @suyiiyii's task in 3m 57s —— View job Review completeI read the server registry/API, device integration, Android native app, frontend sidebar, and contract tests at the current HEAD. Several earlier P0/P1 issues have already been fixed in prior commits: the Android HTTP server binds to I was unable to run FindingsP0 — Pairing endpoints remain unauthenticated
P1 — Android local HTTP server has no authentication
P1 — Agent token stored in plaintext SharedPreferences
P1 —
|

Summary
android-agent/): Kotlin app with HTTP Server, Screen Capture (MediaProjection), Reverse Agent Client (pairing/token persistence/reconnect), Accessibility Service, Foreground Service, Material You/3 UI with setup checklist flow, Chinese + English supportReverseAgentRegistryfor agent pairing/heartbeat/WebSocket sessions,reverse_agentsAPI endpoints for registration/pairing/status/commandsThis allows users to control Android devices without ADB setup — just install the APK and pair with AutoGLM-GUI.
Key Components
Android App (
android-agent/)MainActivity.kt— Material 3 setup checklist UI with step-by-step permissionsAgentHttpServer.kt— Embedded HTTP server implementing Remote Device protocolScreenCaptureController.kt— MediaProjection-based screenshot captureReverseAgentClient.kt— Reverse connection client with token persistence and auto-reconnectAgentForegroundService.kt— Foreground service for long-running agentDeviceAccessibilityService.kt— Accessibility service integrationBackend (
AutoGLM_GUI/)reverse_agent_registry.py— Agent registry with pairing, heartbeat, WebSocket sessionsapi/reverse_agents.py— REST API endpoints for reverse agent managementFrontend (
frontend/)DeviceSidebar.tsx— Added "Add Android Agent" entry with pairing flowapi.ts— Reverse agent API client functionslocales/zh.ts,locales/en.ts— i18n stringsTest plan
./gradlew assembleDebuguv run pytest tests/test_reverse_agents_api.py -vuv run pytest tests/test_cancel_device_release.py -vcd frontend && pnpm run build🤖 Generated with Claude Code