Skip to content

feat: page_text + auth_handoff tools and Chrome profile persistence#4

Merged
todie merged 1 commit into
mainfrom
feat/page-text-auth-handoff
Apr 7, 2026
Merged

feat: page_text + auth_handoff tools and Chrome profile persistence#4
todie merged 1 commit into
mainfrom
feat/page-text-auth-handoff

Conversation

@todie

@todie todie commented Apr 7, 2026

Copy link
Copy Markdown
Member

Summary

Improvements driven by a real-world test case: scraping a JS-heavy SPA (Threads.com) gated behind a login wall. The existing toolset made this awkward — scrape (Scrapling) struggles with JS-heavy SPAs, browse requires manual screenshot/click chains, and there was no way to hand off auth to a human or persist Chrome sessions across sandbox restarts.

  • page_text — Playwright-based "navigate and extract rendered text" tool. Handles JS-heavy SPAs that Scrapling can't reach. Supports wait_for selector, selector filter, custom timeout, and use_profile for persistent sessions.
  • auth_handoff — opens a URL in headed Chrome and returns the noVNC URL so a human can log in. Optionally polls (via a single launch_persistent_context Playwright process) for a wait_for_selector or wait_for_url_contains signal indicating auth is complete.
  • --persist-profile <name> flag on reach create mounts a host directory as Chrome user data. Profiles live under $XDG_DATA_HOME/reach/profiles/<name> (or config override), exposed in the container at /home/sandbox/.config/google-chrome-profiles/<name>. Sessions survive sandbox restarts.
  • MCP tool count: 8 → 10
  • Docs: updated CLAUDE.md, docs/mcp-tools.md, docs/scraping.md (with a Threads-style auth handoff example).

Design notes

  • Auth handoff polling: single Playwright process owns the whole lifetime via launch_persistent_context and polls inside the same Python process. Sidesteps Playwright's flakey CDP-attach story. Trade-off: the docker exec call blocks for up to timeout_seconds, which is fine for human-in-the-loop use.
  • Python script execution: helpers are embedded as pub const &str and passed via python3 -c, with the JSON payload threaded through env vars (REACH_PAGE_TEXT_PAYLOAD / REACH_AUTH_HANDOFF_PAYLOAD) to dodge shell quoting hell. Both helpers always exit 0 with a single JSON object on stdout — uniform parsing path on the Rust side.
  • Profile mounts: container path is stable (/home/sandbox/.config/google-chrome-profiles/<name>); host root resolves via SandboxDefaults::resolved_profile_dir() honoring $XDG_DATA_HOME and config overrides. Profile name propagated as a reach.profile Docker label.
  • noVNC URL resolution: auth_handoff reads the actual mapped noVNC port via docker.find(target), falling back to 6080. URL format centralized in novnc_url().

Test plan

  • cargo fmt --all -- --check clean
  • cargo clippy --workspace --all-targets -- -D warnings zero warnings
  • cargo test --workspace — 78 unit tests passing
  • cargo build --workspace --release succeeds
  • 3 new e2e tests added (t37_page_text_basic, t38_page_text_selector, t39_auth_handoff_returns_vnc_url) — #[ignore]d like the existing e2e suite, run with cargo test --workspace -- --ignored
  • Manual verification against Threads.com login wall (pending Docker daemon)

🤖 Generated with Claude Code

…sistence

Improvements driven by a real-world test case (scraping a JS-heavy
SPA behind a login wall):

- page_text: Playwright-based "navigate and extract text" tool that
  handles SPAs Scrapling can't reach
- auth_handoff: returns the noVNC URL and (optionally) polls for an
  auth-complete signal so an agent can hand off login to a human
- --persist-profile flag on `reach create` mounts a host directory
  as Chrome user data, so sessions survive sandbox restarts
- E2E tests for the new tools (ignored by default)
- Updated MCP tool docs and scraping guide

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@todie
todie force-pushed the feat/page-text-auth-handoff branch from 5ab4991 to 4cb9a38 Compare April 7, 2026 10:17
@todie
todie merged commit 1874977 into main Apr 7, 2026
7 of 9 checks passed
@todie
todie deleted the feat/page-text-auth-handoff branch April 7, 2026 10:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant