Epic: Improve the Testing Deployment Process

[RayBB]

Epic: Improve the Testing Deployment Process

Problem Statement

The Open Library testing deployment page is painfully slow to load. When pushing multiple PRs, removing some, or updating others, the friction is high and the iteration speed sucks. We want to make the whole process fast enough that deploying to testing stops being a bottleneck.

Since this is not user-facing, API stability is a non-concern. We can move fast and choose whatever shape makes the most internal sense.

---

Current System

The entire testing deployment system lives in a single file: openlibrary/plugins/openlibrary/status.py (557 lines), with its template at openlibrary/templates/status.html (253 lines). It is served entirely via legacy Web.py (Infogami). No FastAPI endpoints or Lit components exist for this system yet.

Routes

Method	Path	Class	Purpose
GET	/status	status	Main page — renders full HTML with testing state, drift, build results, flags
POST	/status/add	status_add	Queue PR(s) by number or URL
POST	/status/remove	status_remove	Remove PR(s) from queue
POST	/status/enable	status_enable	Stage PR(s) for enabling (applied on next deploy)
POST	/status/disable	status_disable	Stage PR(s) for disabling (applied on next deploy)
POST	/status/pull-latest	status_pull_latest	Stage fetching latest SHA for PR(s) (applied on next deploy)
POST	/status/deploy	status_deploy	Apply all pending changes + trigger Jenkins rebuild
POST	/status/refresh	status_refresh	Evict GitHub drift cache, forcing refresh on next load

Data Models

All defined in status.py as dataclasses:

TestingState
  last_deploy_at: str          # ISO timestamp
  prs: list[TestingPR]

TestingPR
  pr: int                      # PR number
  commit: str                  # pinned commit SHA
  active: bool
  title: str
  added_at: str                # ISO timestamp
  added_by: str                # OL username
  pull_latest_sha: str         # pending SHA update
  pending_active: bool | None  # pending enable/disable
  author: str                  # GitHub login
  author_avatar: str
  assignee: str
  assignee_avatar: str

DevMergedStatus                # CI build result (from _dev-merged_status.txt)
  git_status: str
  pr_statuses: list[PRStatus]
  footer: str

PRStatus
  pull_line: str
  status: str
  body: str

Persistence

Store	Path / Key	Contents
JSON file	./_testing-prs.json	Full TestingState (PRs + last_deploy_at)
JSON file	./_dev-merged_status.txt	CI build result, parsed into DevMergedStatus
Memcache	status.github_pr_drift (5 min TTL)	Per-PR drift info (head_sha, drift count, merged status)

Auth

_is_maintainer() checks the current user against /usergroup/maintainers or /usergroup/admin. All mutating POST endpoints require this check. The GET /status page hides the testing section from non-maintainers.

External Dependencies

• GitHub API (api.github.com/repos/internetarchive/openlibrary) — fetches PR info (title, author, assignee, head SHA) and drift info (commits behind). Uses config.github_api_token for auth. Fallback fields on error.
• Jenkins (jenkins.openlibrary.org/job/testing-deploy/buildWithParameters) — triggered on deploy with the full active PR list as GH_REPO_AND_BRANCH parameter. Uses config.jenkins_token for auth.

Current Performance Issues

The page loads synchronously on every GET:

1. Reads _testing-prs.json from disk
2. Calls _get_drift_info() which hits memcache (fast) or GitHub API (slow — one request per PR)
3. Reads _dev-merged_status.txt from disk
4. Renders full HTML server-side with inline <script> and <style>

The main bottleneck is the GitHub API calls on cache miss — each PR needs at minimum one API call (GET /pulls/{num}), and drifted PRs need a second (GET /compare/{base}...{head}). These calls are currently made synchronously and sequentially (one blocking HTTP request per PR). Moving to async httpx in FastAPI would let the server concurrently fetch drift data from GitHub — turning an O(n) sequential wait into a single round-trip.

---

Proposed Phases

Phase 1 — Read-only FastAPI Endpoints

Expose structured JSON APIs for the current testing state. Build alongside the existing Web.py handlers — no need to migrate the whole file at once.

Design notes:

• Expose structured JSON endpoints that cover what the page currently shows: the PR list with metadata and drift, the build result, server info, feature flags
• Reuse the existing data models (TestingState, TestingPR, etc.) — they already have to_dict() methods
• Share the same persistence layer (_load_testing_state(), memcache) — FastAPI and Web.py run in the same container and can both read _testing-prs.json
• Auth: Use the same _is_maintainer() check for now (it calls get_current_user() which works in both contexts)
• Cache drift info in the response with Cache-Control headers (respect the existing 5-minute drift TTL)
• Use async httpx to fetch GitHub data concurrently (the current Web.py code calls _github_get sequentially per PR)

Tech: FastAPI route in openlibrary/fastapi/ (new file, e.g. testing.py)

Success criteria:

• A client can fetch everything the current testing page displays in a few lightweight JSON requests
• FastAPI endpoints are registered and return correct data
• Existing Web.py page continues to work unchanged

Phase 2 — Lit Front End (Read-only)

Rebuild the testing page UI using Lit components that consume the Phase 1 APIs.

Key considerations from the current template (status.html):

• The page has 9 distinct sections: deploy banner, add-PR form, PR table with drift indicators, action buttons, deploy button, build results, system info, feature flags, refresh button
• The PR table is the most complex piece — per-row state (merged/inactive/new/pending-enable/pending-disable), drift levels, and checkbox selection
• Currently all interaction is via synchronous form POSTs with full page reloads

Goals:

• Much faster initial load (fetch JSON, render client-side)
• No full-page reloads on actions
• Cleaner, maintainable components
• Same visual layout, slightly friendlier UX

Out of scope: Writes still hit the old Web.py POST handlers; this phase is view-only.

Phase 3 — Write APIs

Add FastAPI endpoints for mutating operations: queue PRs, remove PRs, stage enable/disable, pull latest SHA, trigger deploy, evict drift cache.

Design notes:

• Batch operations should be first-class — accept multiple PRs in add/remove/update operations
• Only one deploy at a time (Jenkins queue handles serialization)
• Auth: Require maintainer group for all write endpoints
• File-based persistence: both Web.py and FastAPI read/write _testing-prs.json — use a file lock or atomic write pattern (write + rename)

Phase 4 — Lit UI + Write APIs

Wire the Lit front end to the Phase 3 write APIs. At the end of this phase, the old testing page is fully replaced.

UX wins: Instant feedback, no full-page reloads, batch operations feel snappy.

---

Implementation Notes

Decision	Rationale
FastAPI over Web.py	Modern async support, auto-generated docs, easier to maintain
Lit over Vue	We're broadly moving away from Vue; Lit is lighter and keeps us framework-agnostic
API shape	Internal-only, so optimize for speed and clarity over backwards compatibility
Share persistence layer	FastAPI and Web.py run in the same container; reuse _load_testing_state() / _save_testing_state() directly
Dev-merged status file	CI writes _dev-merged_status.txt; FastAPI APIs should read it the same way Web.py does
Memcache for drift	FastAPI can share the same memcache pool — reuse _DRIFT_CACHE_KEY to avoid duplicate GitHub API calls
GitHub API token	Shared via config.github_api_token — already works for both Web.py and FastAPI contexts

---

Rollout Strategy

1. Phase 1 + 2 ship at /status/v2 (read-only, Lit front-end) without touching the legacy page
2. Phase 3 + 4 add writes to /status/v2; once parity is proven, retire /status and move /status/v2 to /status
3. During migration, the existing /status page remains the source of truth for writes

---

Success Criteria

• Testing page loads in < 1s (vs. current multi-second wait)
• Queuing / removing / updating PRs feels instantaneous (no full reload)
• Read-only FastAPI endpoints return the same data as the current page
• Old testing page can be safely deleted

---

Settled Questions

• [x] Do we want to expose any of these APIs for automation (e.g., CLI or bot-driven deploys), or keep them internal and undocumented? — Internal and documented by FastAPI's auto-generated OpenAPI schema. A CLI tool would be nice to have one day but is out of scope for now.
• [x] Should batch operations (queue multiple PRs at once) be a first-class API concept? — Yes. Accept multiple PRs in add/remove/update operations. However there will only ever be one deploy active at a time (Jenkins serializes the queue).

[mekarpeles]

Context

• PR feat: interactive testing environment dashboard on /status (#11506) #12241 (feat: interactive testing environment dashboard on /status, merged Apr 2026) is the origin of the current status.py implementation described in this epic. Reading that diff alongside this issue is useful background for Phase 1 contributors.

• _save_testing_state() is a non-atomic write (status.py:386): it calls TESTING_STATE_FILE.write_text(...) directly with no locking. This is safe today under Web.py's single-threaded-per-request model, but will be a data-loss risk once Phase 3 introduces concurrent async writers. The Phase 3 notes mention an atomic write+rename pattern — that design decision should be made before Phase 3 ships, not after.

• httpx==0.28.1 is already a pinned dependency (requirements.txt). No new dependency required for the async GitHub API calls in Phase 1.

• FastAPI auth gap for Phase 1: _is_maintainer() (status.py:390) checks user.is_member_of_any(["/usergroup/maintainers", "/usergroup/admin"]), but the existing FastAPI auth patterns in openlibrary/fastapi/auth.py only expose is_admin() and is_librarian(). There's no existing require_maintainer dependency. Phase 1 implementers will need to either call _is_maintainer() directly (verify it's callable in the FastAPI request context) or add a new require_maintainer Depends following the existing pattern.

Issue Assessment

Open questions

• Does get_current_user() reliably return the session user in the async FastAPI request context? The existing auth.py calls it at line 157, suggesting it works — but confirming this (or noting any caveats) would help Phase 1 implementers avoid guessing.

Full triage checklist (maintainers / Richy)

• Triage — correct labels applied or removed
• Problem / opportunity — problem is clear, audience identified, actionable
• Scope — issue is well-scoped; phased rollout prevents big-bang rewrites
• Justification — synchronous-sequential GitHub API calls on cache miss are a real latency bottleneck
• Success & testing criteria — < 1s load time, no full reloads, old page deletable
• Proposal & breakdown — four phases with concrete scope, data models, persistence decisions, and rollout strategy
• Risks, concerns, open questions — file-locking race (Phase 3), FastAPI auth gap (Phase 1) not fully specified yet
• Richy context — PR feat: interactive testing environment dashboard on /status (#11506) #12241 reviewed, current status.py read, FastAPI auth.py checked, httpx dependency confirmed

Note

This comment was automatically generated by Richy, via PAM, Open Library's Project AI Manager, on behalf of @mekarpeles. Richy is designed to provide status visibility, perform basic project management functions and relevant codebase research, and provide actionable feedback so contributors aren't left waiting.