Remove Convex: Flask + SQLite + SSE becomes the only backend#15
Merged
Conversation
…ript (P1+P7) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports the chatInternal.ts state machine (transition guards, idempotency, COALESCE fallbacks, stale-generation reaper) to backend/db/chat_store.py, user app-state to app_users.py, and schoologyCache.ts diff/tombstone logic to schoology_cache_store.py, with a permanent smoke gate in scripts/smoke_stores.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Chat generation now writes directly to SQLite via chat_store (launcher replaces the internal_chat blueprint, reaper thread replaces the Convex cron); schoology/user/auth routes read and write local stores; the frontend replaces Convex reactive queries with useLiveQuery over the /api/events SSE channel; chat page keeps streamed text on terminal instead of waiting for a reactive re-push. Deletes frontend/convex/, the bridge layers, and the aud=convex JWT surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…_to_cache (P8) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…c and polling - Order messages by insertion (rowid) so a turn's user prompt always precedes its assistant reply; migration sorts by _creationTime - Publish chat.thread.updated on existing-thread send so the sidebar reorders instantly in every tab - Preserve partial streamed text when the reaper terminates a stale generation; skip the write lock on idle reaper ticks - Dedupe the optimistic user bubble against its own message event; resync messages and active-generation state after SSE reconnect - Batch N+1 course queries; stop background-tab polling and double focus refetches; drop stale convex dep from nested requirements Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Redis-snapshot-only check fails open when the snapshot is absent (LRU eviction, post-terminal state), streaming a generation to any authenticated user. Check the authoritative owner in chat.db before streaming — restores the fail-closed guarantee PR #4 provided via Convex, now against the new SQLite source of truth. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Integrate the Convex removal with everything on dev (#1-#14). Resolutions: - api/routes.py: keep #15's fail-closed SQLite ownership check (restores PR #4's IDOR guarantee against the new source of truth). - services/schoology/client.py: graft PR #3's full SSRF hardening (validate_outbound_url + per-hop re-validating redirect loop) onto #15's rewritten client, which only had host-scoped auth. - auth/routes.py: keep PR #8's log-injection fix (log arg keys, not values). - Convex modules/frontend deleted by #15 (convex_sync, internal_chat, convex/, ConvexClientProvider): keep deleted. - chat/page.tsx: take #15's SSE rewrite, which already re-implements PR #5's input-preservation and loading-vs-not-entitled behavior. Tool-call rehydration on reconnect remains a known follow-up (needs an SSE-era endpoint). - tests/test_jwt_utils.py: drop tests for the Convex JWT audience and JWKS document #15 removed; algorithm-confusion coverage retargeted to the mobile audience, not deleted. Verified on the merged tree: ruff clean, 142 passed + 2 xpassed, smoke_stores 36/36, app imports clean, frontend tsc --noEmit clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| def _fetch_user_row(executor, user_id: int): | ||
| return executor.execute( | ||
| f"SELECT {_USER_COLUMNS} FROM users WHERE id = ?", (user_id,) |
| conn = get_conn(Config.MAIN_DB_PATH) | ||
| cursor = conn.cursor() | ||
| cursor.execute( | ||
| f"UPDATE users SET {assignments}, app_state_updated_at = ? WHERE id = ?", |
| if course_ids: | ||
| placeholders = ",".join("?" * len(course_ids)) | ||
| for row in conn.execute( | ||
| f"SELECT * FROM schoology_courses WHERE course_id IN ({placeholders})", |
Comment on lines
+426
to
+427
| f"SELECT * FROM schoology_assignments" | ||
| f" WHERE course_id IN ({placeholders}) AND due_at_ms >= ?", |
| if course_ids: | ||
| placeholders = ",".join("?" * len(course_ids)) | ||
| for row in conn.execute( | ||
| f"SELECT * FROM schoology_courses WHERE course_id IN ({placeholders})", |
Comment on lines
+975
to
+976
| except Exception: | ||
| pass |
| subprocess with a watcher thread. | ||
| """ | ||
| import logging | ||
| import subprocess |
Comment on lines
+116
to
+117
| process = subprocess.Popen( | ||
| [sys.executable, "-m", "services.chat.worker", generation_id], |
| assert verify_convex_token(mobile_token) is None | ||
| assert verify_mobile_access_token(convex_token) is None | ||
| other_token = create_token(**USER, audience=other_audience) | ||
| assert verify_token(mobile_token, audience=other_audience) is None |
| assert verify_mobile_access_token(convex_token) is None | ||
| other_token = create_token(**USER, audience=other_audience) | ||
| assert verify_token(mobile_token, audience=other_audience) is None | ||
| assert verify_mobile_access_token(other_token) is None |
| ) | ||
| except Exception: | ||
| logger.exception( | ||
| "chat_launcher_mark_failed_error generation_id=%s", generation_id |
| ) | ||
| except Exception: | ||
| logger.warning( | ||
| "chat_launcher_publish_terminal_error generation_id=%s", generation_id |
| """ | ||
| if not _mark_generation_started(generation_id): | ||
| logger.info( | ||
| "chat_generation_already_running generation_id=%s", generation_id |
| except Exception as exc: | ||
| _mark_generation_finished(generation_id) | ||
| message = str(exc) or exc.__class__.__name__ | ||
| logger.exception("chat_generation_spawn_failed generation_id=%s", generation_id) |
These files predate the ruff config #8 added, so CI's ruff gate flagged them once the branch was merged with dev. Pure formatting — import sorting and one placeholder-less f-string; no logic change (compile + full pytest still green). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Integrate TinyFish's free Search and Fetch APIs as chat tools so the assistant can read external links (e.g. a course-catalog PDF that Schoology only exposes as a link) and search the public web — capabilities the chat did not have before. - services/tinyfish.py: thin requests-based client for the Fetch (POST api.fetch.tinyfish.ai) and Search (GET api.search.tinyfish.ai) REST endpoints, authenticated with a single shared X-API-Key. No new dependency and no Python 3.11+ floor (avoids the tinyfish SDK). - services/chat/web_tools.py: fetch_url and web_search tool definitions + execution, returning the existing ToolExecutionResult shape. Output is bounded (per-URL char cap, capped URL/result counts). - services/chat/service.py: offer the web tools on every generation, independent of schoology_connected (fetching a public URL needs no Schoology auth), and dispatch web tool names to execute_web_tool. - config.py: TINYFISH_API_KEY env var + a validate() notice when unset. - prompts/chat_system.txt: tell the model about the new tools. Set TINYFISH_API_KEY in backend/.env to enable. Key from agent.tinyfish.ai/api-keys. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Web-tool error messages raised by the TinyFish client are streamed to the chat UI as errorText, so "TinyFish fetch error: ..." etc. would surface to end users. Reword the raised messages to generic "Web fetch"/"Web search" wording and make the not-configured message user-friendly. Internal identifiers (module, exception classes, env var, endpoints, comments) are not user-facing and are left unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add TinyFish web tools (fetch_url + web_search) to chat
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes Convex entirely. Flask + SQLite is now the single source of truth, and a new per-user SSE event channel (
GET /api/events, Redis Streams over the same Upstash infra as chat token streaming) preserves full live-update parity — every surface that updated reactively under Convex stays push-driven: onboarding auto-advance, thread list reordering, generation/cancel state across tabs, message visibility, upcoming refresh, profile picture.Design doc:
MIGRATION_SPEC.md(committed at repo root).What changed
Backend
db/chat_store.py— ports the Convex chat state machine (transition guards, idempotency viaUNIQUE(user_id, client_request_id), terminal no-ops,BEGIN IMMEDIATEin place of OCC) to a newchat.db; Convex_idstrings preserved so old/chat/#<id>URLs keep workingdb/app_users.py+ columns onusers— onboarding/consent/profile state (was Convexuserstable); chat entitlement now enforced server-sidedb/schoology_cache_store.py— portsschoologyCache.tsdiff/tombstone logic (batched queries)services/events.py+events/blueprint — the app-wide SSE channel; stores publish typed events after commit so no call site can forgetchat/blueprint +services/chat/launcher.py—POST /api/chat/messagesreplaces the sendMessage → Convex action → internal-secret hop (one hop instead of three);reaper.pydaemon thread replaces thefailStaleGenerationscron and now also terminates attached SSE clients, preserving partial streamed textservices/convex_bridge.py, all threeconvex_sync.pybridges,internal_chat/, theaud=convexJWT/JWKS surface,CONVEX_*/CHAT_INTERNAL_SECRETconfigFrontend
hooks/useAppEvents.ts(singleton SSE reader, Last-Event-ID reconnect) +hooks/useLiveQuery.ts(fetch-on-mount, apply-on-event, focus/reconnect resync, visibility-aware backstop poll) +lib/api.tsfrontend/convex/,ConvexClientProvider, 4 dead hooks, theconvexdependencyMigration
backend/scripts/migrate_convex_export.py— one-time import of anpx convex exportzip (chat history, user state, preferences; schoology cache deliberately self-heals via refresh). Run once against prod data before deploying:python -m scripts.migrate_convex_export <export.zip>Verification
python -m compileallclean;python -m scripts.smoke_stores— 36 assertions over the ported state machine (idempotency, guards, reaper, tombstoning) all passpnpm buildclean; zeroconvexreferences left infrontend/srcMIGRATION_SPEC.md— worth a pass before merging, especially chat resume/cancel across tabs and onboarding advancePost-merge follow-ups
~/Developer/one/CLAUDE.md(outside this repo) — still documents the Convex architectureCONVEX_URL,CONVEX_BRIDGE_SECRET,CHAT_INTERNAL_SECRET,NEXT_PUBLIC_CONVEX_URL; optionally setAPP_EVENTS_*)last_convex_check_atcolumn name (schema, out of scope),CHAT_CONVEX_HEARTBEAT_MSenv fallback (intentional back-compat)🤖 Generated with Claude Code