Implement multi-user YouTube architecture and approval hardening

[djbclark]

Objective

Turn the current single-operator YouTube integration into a genuinely multi-user, hosted implementation that is secure, quota-aware, auditable, and easier for Google/YouTube reviewers to approve.

This ticket is based on a code review of commit c1aa205 and recent YouTube work through 2026-07-31. It is the implementation plan beneath #1 and is blocked by #1 (production model, Cloud project, policies, reviews, and rollout prerequisites).

The product promise remains: help a user organize videos related to their subscriptions into private, user-controlled topics so they can switch among interests and return to YouTube to watch. It is not creator growth, public video scoring, surveillance, or replacement playback.

What the code does today

The current design is internally consistent for one self-hosted operator, but almost every important object is global:

Area	Current implementation	Multi-user/approval problem
App authentication	One 8-character token in backend/api.py, accepted in a header or query string	No user identity, weak shared credential, query-string leakage, and no tenant authorization
Storage	One SQLite database; analyses, queues, collections, WebSub rows, and YouTube mappings have no owner key in backend/core/database.py	Same video/category/channel keys collide across users; every read/write is cross-tenant by construction
OAuth credentials	One client and one YOUTUBE_OAUTH_REFRESH_TOKEN in the process environment/SecretSpec in backend/core/youtube_oauth.py	Only one YouTube account can be connected; token lookup is not bound to the requesting user
OAuth transaction	In-memory pending state and a localhost redirect in backend/api.py	Not durable across workers/restarts and not associated with an authenticated application account
Consent and mutations	OAuth callback calls activate_after_oauth, enabling sync and creating/adopting playlists immediately (callback, activation)	OAuth consent is not the same as express consent to the described class of bulk playlist actions; users are not shown channel identity, titles, visibility, or automation rules before writes
Taxonomy/sync settings	One filesystem categories.toml and one global [youtube_playlists] section	Users cannot have isolated topics, preferences, or sync state
Playlist mappings	Category is the primary key and video ID is the primary key in the mapping tables	One user's mappings overwrite another user's mappings
WebSub	One row per channel and one queue/cache result per video in backend/api.py	No per-user channel membership or fan-out; first user's cached video suppresses work for everyone else
Work execution	One in-process queue worker and max_concurrent = 1 (backend/api.py); quota exhaustion can sleep a CLI process until reset	Does not scale horizontally or enforce per-user/system quota budgets
YouTube data acquisition	Hosted-capable paths use page scraping, yt-dlp, browser cookies, caption extraction, and audio downloads (analyzer, transcripts, playlist importer)	Cookie collection and restriction-circumvention concerns could undermine an otherwise compliant OAuth review; source provenance is unclear
AI processing	YouTube URLs go to Gemini; metadata/transcripts may go to Groq, OpenRouter, or other configured models	Hosted production needs explicit disclosure, consent, subprocessor controls, and a defensible Limited Use story
User controls	Mobile app configures a server URL and the same 8-character token; there is no connected-channel, OAuth consent, disconnect, or delete-account UI	Reviewers cannot see complete user notice/control in the submitted product
Public policy	docs/PRIVACY.md and docs/TERMS.md explicitly describe self-hosting and local storage	These statements would be inaccurate for a hosted multi-user release

One existing choice should be preserved: the app opens YouTube videos in YouTube/system browser rather than implementing replacement playback (PostDetailScreen.tsx).

P0 — Establish a production-safe YouTube feature boundary

• Keep local/self-hosted research and import tools available only in an explicitly separate local mode; create a production feature flag/build profile that cannot invoke browser-cookie extraction, private list=WL imports, yt-dlp metadata/caption/audio downloads, or YouTube HTML scraping.
• Do not upload or store browser cookies in the hosted service.
• Inventory every YouTube-related input and label its provenance in code and storage: YouTube Data API authorized data, public WebSub feed data, user-entered URL, model-derived summary, user-authored topic, and local-only importer data.
• Prevent model-derived or scraped fields from being represented as official YouTube API metadata.
• Obtain an explicit policy answer in the production audit for private AI topic suggestions. YouTube's policy guide warns against inferring a video's content category/type from API data. Until approved, do not use YouTube API fields as classification input; describe labels as private, editable organizational suggestions—not authoritative categories, safety ratings, metrics, or public channel profiling.
• Keep all AI-derived labels private to the authorizing user and never aggregate them into public video/channel insights.
• Decide whether Gemini native YouTube URL analysis is in the approved production scope and document its terms/data flow separately from YouTube Data API access.
• Add a CI test that production configuration fails closed if cookie, scraping, subtitle-download, media-download, or local playlist-import switches are enabled.

P0 — Add real application identity and tenant authorization

• Add normal application accounts and authenticated sessions suitable for mobile and web clients. Replace the shared 8-character API token in hosted mode with revocable, expiring sessions/access tokens.
• Stop accepting credentials in query parameters in hosted mode. Use secure, HTTP-only cookies for web or Authorization headers/secure storage for mobile.
• Make CORS an explicit production allowlist; never fall back to * outside development.
• Introduce immutable users/tenant IDs and require an authenticated user context in every API handler, background job, database method, and audit event.
• Move hosted storage from the single SQLite singleton to a migration-managed database suitable for concurrent workers (for example PostgreSQL). Retain SQLite only for self-hosted mode if desired.
• Add user_id (or a connection/account owner key) to analyses, collections, deleted rows, queues, taxonomy settings, consent records, playlist mappings, and sync jobs.
• Replace global primary keys with tenant-safe keys, for example (user_id, shortcode), (user_id, category_name), (youtube_connection_id, video_id), and (user_id, collection_id).
• Add database row-level constraints and tests proving that cross-user reads, edits, deletes, exports, retries, and playlist mutations are impossible.
• Make export/import, thumbnails, logs, static assets, and cached data tenant-aware; avoid guessable cross-user object URLs.

P0 — Redesign YouTube OAuth per user

• Create a youtube_connections model containing at least: application user, stable Google/YouTube channel identity, encrypted refresh token, granted scopes, consent/policy versions, connection status, created/refreshed/revoked timestamps, and last authorization verification.
• Encrypt refresh tokens with a managed key; separate token-encryption keys from the database; support rotation and never log tokens or authorization codes.
• Replace the global environment refresh token and parameterless refresh_access_token() with a per-connection credential provider required by every YouTube API client.
• Persist OAuth state/PKCE transactions in short-lived server storage, bind each to the authenticated user and intended return URI, make it single-use, and support multiple web workers/restarts.
• Use an exact production HTTPS redirect URI on the verified domain; keep loopback OAuth only in local/self-hosted mode.
• Handle refresh-token rotation, invalid_grant, revoked scopes, partial grants, reconnects, and concurrent refreshes safely.
• After authorization, call channels.list(mine=true, part=id,snippet) (and add it to the audit/API inventory) so the UI can clearly identify the YouTube channel that will be modified.
• Prototype and test the smallest scope set that supports subscriptions.list(mine=true), playlists.list/insert, and playlistItems.insert/delete. Replace the current full youtube scope if a narrower reviewed scope works; record endpoint-by-endpoint test evidence rather than assuming.

P0 — Require specific consent before YouTube writes

• Remove activate_after_oauth() from the OAuth callback. A successful OAuth grant must not automatically enable sync or create/adopt playlists.
• Add a post-OAuth review screen showing: connected YouTube channel, requested features, each proposed playlist title, privacy status, what future automatic changes will occur, data sent to AI providers, and links to privacy/terms/revocation.
• Default created playlists to private and make the visibility explicit before confirmation.
• Require a separate affirmative confirmation before the first playlists.insert or playlistItems.insert/delete call. Store the consent version and timestamp.
• Do not silently adopt an existing same-titled playlist. Show it and require explicit selection, or create an app-owned playlist with a stable marker and retain its ID.
• Give users separate controls for subscription discovery, WebSub updates, AI topic suggestions, and automatic playlist synchronization; OAuth should not force-enable all of them.
• Before enabling ongoing automatic moves, explain that changing a topic may remove an item from one YouTube playlist and add it to another. Provide pause and preview/dry-run controls.
• Design consent for bulk, ongoing automation, not per-channel or per-video prompts. A user may select all subscriptions (or many with search/filter/multi-select), review the classes of actions once, and authorize SuperBrain to process those selected channels and resulting playlist changes automatically until paused or revoked.
• Do not require separate approval for every subscribed channel, discovered upload, categorized video, playlist insertion, or routine retry when it is already covered by the user's recorded bulk consent.
• Require a new confirmation only when the service materially expands beyond that consent—for example enabling a new data recipient, changing playlist visibility, adding a new class of mutation, or including channels the user did not select under an “all current and future subscriptions” choice.
• Provide a bulk preview with counts and representative examples before activation, followed by progress, completion summary, failures, undo/correction where feasible, and a searchable action history.
• Add an immutable audit trail for user consent and each requested YouTube mutation, excluding secrets and raw sensitive payloads.

Policy basis: YouTube requires API clients to clearly identify authorized actions, associated channels/content owners, and visibility changes, and to obtain express consent before insert/update/delete actions.

P1 — Make playlist and subscription behavior tenant-safe and correct

• Refactor YouTubePlaylistClient to require a youtube_connection_id and inject the correct per-user access token on every request.
• Pass the authenticated owner through ensure_category_playlists, sync_video_category, category-change hooks, backfill jobs, status endpoints, and all database mapping calls.
• Replace global TOML playlist settings with per-user persisted settings; version taxonomy changes and queue explicit reconciliation jobs.
• Reconcile playlist mappings against YouTube before mutation, including deletion, renamed/missing playlists, and missing playlist items.
• Fix the current duplicate handling that records a synthetic duplicate:<video_id> as a playlist-item ID. Locate and store the real item ID before a future delete.
• Make writes idempotent with operation keys and transactional state so retries cannot duplicate or misapply changes.
• Redesign WebSub as a global channel subscription plus a per-user membership/fan-out table, or as separately isolated subscriptions. A delivery must enqueue work for every opted-in user and only those users.
• Fetch subscriptions with pagination and support bulk “all subscriptions,” multi-select, select-by-filter, and “include future subscriptions” rules without channel-by-channel authorization screens.
• Execute large selections as resumable bulk jobs with bounded concurrency, progress counts, partial-failure reporting, cancellation, and safe resume. Preserve one recorded user authorization for the job rather than prompting on each item.
• Deduplicate analysis work without sharing private user data: reusable public/non-authorized artifacts may be cached separately, while user summaries, topics, subscriptions, and playlists remain isolated.

P1 — Implement revocation, deletion, retention, and freshness

• Add Disconnect YouTube: stop jobs, unsubscribe/fan-out as appropriate, revoke the Google grant, delete the refresh token, and delete or de-identify associated Authorized Data and mappings within policy timelines.
• Add Delete account/data in product UI and API; make it usable without contacting support.
• Record deletion jobs and completion receipts, including primary storage, queues, caches, logs, exports, and backup expiry.
• On revoked/invalid authorization, disable mutations immediately and start the required data cleanup clock.
• Add source, fetched_at, last_refreshed_at, and expires_at metadata to stored YouTube API data.
• Refresh or delete covered YouTube API data at least every 30 days and verify relevant video existence/current metadata. Never retain a stale snapshot indefinitely.
• Re-prompt users when material data uses or policy terms change.
• Add automated lifecycle tests using an accelerated clock: expiry, refresh, revocation, account deletion, backup aging, and retry-queue removal.

P1 — Control quota, jobs, and production operations

• Replace the in-process worker/thread and quota-reset sleep with a durable job queue and horizontally safe workers.
• Store every YouTube call's endpoint, quota cost, user, job, result class, and timestamp without storing tokens.
• Enforce per-user, per-connection, per-minute, and project-wide budgets before making a call.
• Reserve quota for interactive disconnect/correction operations and fail gracefully when background quota is exhausted.
• Use bounded exponential backoff with jitter for transient errors; do not retry authorization, policy, or invalid-request failures blindly.
• Avoid repeatedly listing every playlist during item sync. Cache mappings subject to freshness rules and reconcile on explicit triggers/errors.
• Add dashboards/alerts for daily units, minute rate, endpoint mix, failed/retried writes, refresh failures, revoked accounts, deletion deadlines, and tenant-isolation errors.
• Load-test the API, queue, database, token refresh, and WebSub fan-out at expected beta and launch sizes.

P1 — Build reviewer-visible mobile/web controls

• Replace server URL/shared-token setup in hosted mode with sign-up/sign-in and an account screen.
• Add a YouTube connection screen showing the channel identity, granted features/scopes in plain language, last sync, quota delay/error state, and links to privacy/terms/Google permissions.
• Add the explicit playlist preview/confirmation workflow described above.
• Make bulk management a first-class UX: select all/many channels or videos, apply/include/exclude topic rules in bulk, preview aggregate effects, start once, then monitor or cancel the background job.
• Add controls for pause sync, reconnect, disconnect/revoke, export, delete YouTube data, and delete account.
• Preserve the current Open in YouTube behavior and unmodified official thumbnails/metadata; do not add replacement/background playback.
• Clearly distinguish official YouTube metadata, user-authored labels, and AI-generated private suggestions.
• Add an in-product privacy notice adjacent to OAuth and AI features, not only on a website.

P1 — Hosted privacy and AI-provider controls

• Replace self-hosted-only privacy/terms text before production and version the accepted documents.
• Publish a data-flow diagram and retention table matching actual fields, queues, caches, backups, logs, and subprocessors.
• Disclose every hosted model/infrastructure provider that may receive a YouTube URL, title, description, transcript, summary, or account-derived data.
• Obtain explicit consent for required data transfers, configure providers not to train on user data, minimize payloads, and document contracts/retention.
• Do not send subscription lists, channel identity, OAuth tokens, playlist IDs, or other Authorized Data to an LLM unless specifically necessary, disclosed, consented, and approved.
• Add automated tests comparing declared data flows/scopes/endpoints with code configuration so documentation cannot silently drift.

Approval package generated from the implementation

• Maintain a machine-readable inventory of all scopes, endpoints, parts/fields, quota costs, stored fields, retention, and user-visible features.
• Update the production YouTube audit to include channels.list and any other endpoint actually used; remove endpoints/features absent from production.
• Provide screenshots/video of: pre-OAuth explanation, complete English consent screen, connected channel identity, playlist preview and privacy, affirmative mutation consent, successful private playlist result, pause/disconnect, and deletion.
• Demonstrate that production cannot access browser cookies or invoke yt-dlp/scraping/media downloads.
• Demonstrate tenant isolation with two test users and two YouTube channels.
• Demonstrate 30-day refresh/deletion behavior and immediate disablement after revocation.
• Make the approval narrative and UI consistently emphasize private subscription/topic organization and returning to YouTube to watch—not creator analytics/growth.

Current official references:

• YouTube Developer Policies: https://developers.google.com/youtube/terms/developer-policies
• YouTube policy guide: https://developers.google.com/youtube/terms/developer-policies-guide
• OAuth verification requirements: https://support.google.com/cloud/answer/13464321?hl=en
• OAuth production submission: https://support.google.com/cloud/answer/13461325?hl=en
• OAuth scope catalog: https://developers.google.com/identity/protocols/oauth2/scopes#youtube
• Server-side OAuth: https://developers.google.com/youtube/v3/guides/auth/server-side-web-apps
• playlists.insert: https://developers.google.com/youtube/v3/docs/playlists/insert
• playlistItems.insert: https://developers.google.com/youtube/v3/docs/playlistItems/insert
• subscriptions.list: https://developers.google.com/youtube/v3/docs/subscriptions/list

Required tests and acceptance criteria

• Two concurrent users can connect different YouTube channels and never see or mutate each other's data, jobs, topics, playlists, tokens, exports, or WebSub results.
• OAuth callback never performs a YouTube write; no write occurs before a separately recorded, specific confirmation.
• One explicit bulk consent can cover all selected channels/videos and the described ongoing automatic sync; tests prove the normal workflow never degrades into per-channel or per-video approval prompts.
• Every YouTube client call requires an explicit connection owner; a missing/mismatched owner fails closed.
• Production build contains no callable cookie, yt-dlp, scraping, caption-download, or audio-download path.
• The scope set is the smallest empirically verified set for the shipped endpoints, and consent-screen scopes match code and audit inventory exactly.
• Disconnect/revocation prevents further calls immediately and lifecycle tests prove required deletion/refresh behavior.
• Per-user/project quota controls and idempotency survive retries, worker restarts, and concurrent requests.
• Reviewer demo and public policies match the exact production binary and backend behavior.
• Security review and a two-account penetration test report no cross-tenant access or token leakage.
• Production readiness: make YouTube subscription organization available to many users #1 is complete and both OAuth verification and the revised YouTube compliance/quota review approve the hosted use case before broad launch.