Skip to content

Fix Worker LISTEN/NOTIFY connection leak on shutdown#79

Closed
TkTech with Copilot wants to merge 3 commits into
mainfrom
copilot/fix-worker-stop-connection-issue
Closed

Fix Worker LISTEN/NOTIFY connection leak on shutdown#79
TkTech with Copilot wants to merge 3 commits into
mainfrom
copilot/fix-worker-stop-connection-issue

Conversation

Copilot AI commented Jan 23, 2026

Copy link
Copy Markdown

Worker's dedicated LISTEN/NOTIFY connection wasn't being closed during shutdown, preventing clean database teardown in tests and potentially exhausting connections in production.

Changes

  • Store LISTEN/NOTIFY connection in Worker._notification_connection instance variable
  • Close connection explicitly in Worker.stop() before canceling tasks
  • Add test verifying connection cleanup via pg_stat_activity queries

Implementation

# In _maintain_notifications()
self._notification_connection = await AsyncConnection.connect(
    self.chancy.dsn, autocommit=True
)

# In stop()
if self._notification_connection:
    await self._notification_connection.close()
    self._notification_connection = None

Test creates a standalone Chancy instance, measures active connections before/after worker lifecycle, and verifies no leaks independent of connection pool behavior.

Original prompt

This section details on the original issue you should resolve

<issue_title>Worker seems to be not closing the LISTEN/NOTIFY connection on stop</issue_title>
<issue_description>Hey,

I'm starting some workers in tests to e2e validate the system and then I have trouble tearing down the database. It seems the Worker.stop does not stop the connection created in _maintain_notifications is not stored:

chancy/chancy/worker.py

Lines 445 to 470 in b0e3e94

async def _maintain_notifications(self):
"""
Listen for notifications from the database.
Improves the reactivity of a worker by allowing it to almost immediately
react to cluster events using Postgres's LISTEN/NOTIFY feature.
.. note::
This feature utilizes a permanent connection to the database
separate from the shared connection pool and is not counted against
the pool's connection limit.
"""
connection = await AsyncConnection.connect(
self.chancy.dsn, autocommit=True
)
await connection.execute(
sql.SQL("LISTEN {channel};").format(
channel=sql.Identifier(f"{self.chancy.prefix}events")
)
)
self.chancy.log.info("Started listening for realtime notifications.")
self._notifications_ready_event.set()
async for notification in connection.notifies():
j = json.loads(notification.payload)
await self.hub.emit(j.pop("t"), j)

so later in stop it is not closed:

chancy/chancy/worker.py

Lines 805 to 849 in b0e3e94

async def stop(self) -> bool:
"""
Stop the worker.
Attempts to stop the worker gracefully, sending a CancelledError to all
running tasks and waiting up to `shutdown_timeout` seconds for them to
complete before returning.
Returns True if the worker was stopped cleanly, or False if the worker
returned due to the timeout expiring.
"""
try:
async with asyncio.timeout(self.shutdown_timeout) as cm:
# Stop accepting new queues and queue changes.
try:
await self.manager.cancel("queues")
except KeyError:
pass
# Delete all the queues we know about so the executors can
# clean up.
self._queues.clear()
while self._executors:
await asyncio.sleep(0.1)
# And finally axe everything else started by this worker.
await self.manager.cancel_all()
except TimeoutError:
# We check this instead of depending on the exception in case the
# exception wasn't really raised by us but a nested timeout.
if cm.expired():
await self.hub.emit(
"worker.shutdown_timeout",
{
"worker": self,
},
)
return False
raise
await self.hub.emit(
"worker.stopped",
{
"worker": self,
},
)
return True

I'd like to confirm it is not intentional before I make a PR to fix it.</issue_description>

<agent_instructions>Our worker's long-running connection for LISTEN/NOTIFY is not managed by a pool and does not get cleaned up properly when a worker is stopped. We additionally need a test to ensure we don't leak postgres connections.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@TkTech Definite oversight. I'll need to think about how to structure a connection leakage test so we can catch this kind of issue in the future.

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 2 commits January 23, 2026 12:09
Co-authored-by: TkTech <72590+TkTech@users.noreply.github.com>
Co-authored-by: TkTech <72590+TkTech@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix LISTEN/NOTIFY connection not closing on worker stop Fix Worker LISTEN/NOTIFY connection leak on shutdown Jan 23, 2026
Copilot AI requested a review from TkTech January 23, 2026 12:20
@TkTech

TkTech commented Jan 30, 2026

Copy link
Copy Markdown
Owner

Gave copilot a try, 👎

@TkTech TkTech closed this Jan 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Worker seems to be not closing the LISTEN/NOTIFY connection on stop

2 participants