Skip to content

fix(gateway): harden Telegram outbound delivery against PoolTimeout#2

Open
theplatformx wants to merge 10 commits into
mainfrom
fix/telegram-outbound-queue-health
Open

fix(gateway): harden Telegram outbound delivery against PoolTimeout#2
theplatformx wants to merge 10 commits into
mainfrom
fix/telegram-outbound-queue-health

Conversation

@theplatformx

Copy link
Copy Markdown
Owner

Fixes #1

Summary

  • routes Telegram reply/edit/media/typing outbound Bot API calls through one bounded delivery queue
  • adds queue depth, last-success, and PoolTimeout counters to runtime status details
  • marks repeated Telegram outbound PoolTimeouts as retryable fatal so gateway supervision can recover instead of silently consuming updates
  • changes active-session follow-up behavior toward FIFO queueing instead of default interrupt storms
  • adds incident runbook and regression coverage for queueing, media/edit routing, PoolTimeout classification, and degraded health display

Verification

  • PASS: python3 -m py_compile gateway/platforms/telegram.py gateway/run.py gateway/status.py hermes_cli/gateway.py tests/gateway/test_telegram_delivery_gateway.py tests/hermes_cli/test_gateway_runtime_health.py
  • BLOCKED locally: python3 -m pytest tests/gateway/test_telegram_delivery_gateway.py tests/hermes_cli/test_gateway_runtime_health.py -q (pytest not installed)
  • BLOCKED locally: uv run pytest tests/gateway/test_telegram_delivery_gateway.py tests/hermes_cli/test_gateway_runtime_health.py -q (uv cache write failed: No space left on device)

Deployment gate

Do not deploy directly to production. Build a staging image and test with a safe Telegram bot/chat first.

Rollback

  • revert this PR and rebuild
  • runtime rollback: redeploy previous known-good image tag/digest
  • emergency operational recovery while rollback deploys: restart affected gateway/container to clear wedged PTB/httpx pool

Notes

This PR intentionally does not patch the live VPS. It follows the GitHub source-of-truth path requested for the Viewport Hermes incident.

@theplatformx

Copy link
Copy Markdown
Owner Author

Deployment gate: do not merge or deploy this PR until it passes independent verification. Required before live: clean test run in a fresh environment, staging image build, staging Telegram bot/chat smoke test, forced PoolTimeout recovery test, Docker/gateway health verification, and rollback command confirmed. Local author verification was limited to py_compile; pytest was blocked locally by disk space, and this fork currently reports no GitHub checks/workflows.

@theplatformx

Copy link
Copy Markdown
Owner Author

Verification update for commit 041d973:

Passed:

  • PR Telegram Gateway Checks / telegram-gateway-tests
  • Nix / nix (ubuntu-latest)
  • Nix / nix (macos-latest)
  • Tests / e2e
  • Supply Chain Audit

Still red:

  • Tests / test

I inspected the failed broad test log after the compatibility fix. The Telegram thread-fallback failures caused by the first PR revision are gone. Remaining broad-suite failures are outside this Telegram change set: Matrix nio mock/spec failures, missing OpenAI credentials in streaming/token tests, logging handler isolation, CLI/provider tests, skill-manager tests, and existing gateway/session/OpenClaw setup failures.

Gate decision: do not deploy directly to production from this PR. The next safe step is review/merge only after accepting the unrelated broad-suite baseline, then build a staging/new container and run a real Telegram smoke/soak test before promoting to the live Hermes_Viewport_Bot container.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Telegram gateway outbound send pool can wedge after PoolTimeout and consume updates without replies

1 participant