Skip to content

fix(gateway): offload SessionStore calls off the event loop via asyncio.to_thread#61842

Closed
kenyonxu wants to merge 2 commits into
NousResearch:mainfrom
kenyonxu:fix/pr1-session-store-off-event-loop
Closed

fix(gateway): offload SessionStore calls off the event loop via asyncio.to_thread#61842
kenyonxu wants to merge 2 commits into
NousResearch:mainfrom
kenyonxu:fix/pr1-session-store-off-event-loop

Conversation

@kenyonxu

Copy link
Copy Markdown
Contributor

Summary

PR #55159 added AsyncSessionDB to offload SessionDB calls from the gateway event loop, and even included an AST guard to prevent new raw on-loop calls. But SessionStore in gateway/session.py creates its own SessionDB() directly (line 949), bypassing the AsyncSessionDB facade entirely. So every get_or_create_session_is_session_ended_in_dbdb.get_session chain still runs synchronously on the event loop.

On a large state.db (~1.4 GB in production), this blocks the loop for seconds to minutes, starving Discord/Slack heartbeats and delaying session activation (#53297 — 15-30s delay for existing sessions).

This PR follows the exact same approach as #55159: SessionStore internals stay fully synchronous (zero changes), and all hot-path callers wrap calls with await asyncio.to_thread(self.session_store.method, ...).

Changes

Commit 1: Offload session store calls (gateway/run.py, gateway/slash_commands.py)

~24 call sites wrapped with asyncio.to_thread:

  • get_or_create_session, switch_session, update_session
  • load_transcript, rewrite_transcript, rewind_session
  • reset_session, set_model_override, _save

Commit 2: Async-ify compression-in-flight check (gateway/run.py)

_session_has_compression_in_flight blocked the event loop twice on the message hot path:

  1. Under session_store._lock during _ensure_loaded_locked (JSON read)
  2. Via db.get_compression_lock_holder (SQLite SELECT)

Both sources are now offloaded via asyncio.to_thread. The method signature changed from def to async def; the call site at _handle_active_session_busy_message adds await.

Why not fix this inside SessionStore?

The Threading.Lock contract on SessionStore is preserved — all internals remain synchronous and thread-safe. The fix is purely at the call boundary, matching the pattern established by #55159 for _session_db.

Related

Verification

  • tests/gateway/test_compression_in_flight_check.py — 7 tests (includes thread-offload verification)
  • tests/gateway/test_session.py — 100 tests
  • tests/gateway/test_session_store_runtime_stale_guard.py — 11 tests
  • tests/gateway/test_compression_interrupt_demotion_56391.py — existing tests adapted
  • 119 passed across the affected test files

kenyonxu added 2 commits July 10, 2026 11:21
…cio.to_thread

Every inbound message calls get_or_create_session which synchronously
executes _is_session_ended_in_db → db.get_session → conn.execute on
the asyncio event loop. On a ~1.4GB state.db, this blocks the loop
for seconds to minutes, starving Discord heartbeats.

Upstream NousResearch#55159 fixed the same pattern for self._session_db in
gateway/run.py but missed SessionStore._db in gateway/session.py.

This follows the exact same approach as NousResearch#55159:
- session.py internals stay fully synchronous (zero changes)
- Threading.Lock contract is preserved
- All hot-path callers in run.py and slash_commands.py wrap calls
  with await asyncio.to_thread(self.session_store.method, ...)

Affected: get_or_create_session, switch_session, update_session,
load_transcript, rewrite_transcript, rewind_session, reset_session,
set_model_override, _save — ~24 call sites across 2 files.

# Conflicts:
#	gateway/slash_commands.py
…check (NousResearch#5)

The sync _session_has_compression_in_flight sat on the message hot path
and blocked the event loop twice: under session_store._lock during
_ensure_loaded_locked (JSON read) and via db.get_compression_lock_holder
(SQLite SELECT). Async-ify the method and offload both sources via
asyncio.to_thread; await the call site in _handle_active_session_busy_message.
@alt-glitch alt-glitch added type/perf Performance improvement or optimization comp/gateway Gateway runner, session dispatch, delivery sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages P1 High — major feature broken, no workaround labels Jul 10, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related: #60980 (api_server SessionDB offload) and the merged event-loop-starvation cluster #60984 / #53603 / #52090. This PR closes a distinct hole -- SessionStore in gateway/session.py instantiates its own SessionDB() directly, bypassing the AsyncSessionDB facade #55159 added, so the get_or_create_session hot path still ran sync SQLite on the loop. Same asyncio.to_thread approach, different call site; not a duplicate.

@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Merged via #61905 with your commits and authorship preserved. The salvage also incorporated #61847, fixed the missing awaited compression check, and added a single async SessionStore boundary plus concurrency guards. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P1 High — major feature broken, no workaround sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/perf Performance improvement or optimization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Telegram existing sessions can take 15-30s to activate while new chats respond immediately

3 participants