Commit 2a97e77
fix(server): handle Redis idle disconnects in session-store client (#20143)
## Summary
The session-store node-redis client doesn't attach an `'error'` event
listener, so when Redis closes an idle connection (server-side `timeout`
setting), node-redis emits an unhandled `'error'` event and the entire
Node process crashes with `SocketClosedUnexpectedlyError`.
## Reproduction
1. Deploy twenty-server against a Redis instance with `timeout 300` (5
min idle close).
2. Don't log in (or otherwise keep the session store completely idle).
3. ~5 minutes after `Nest application successfully started`, the process
crashes:
```
node:events:487
throw er; // Unhandled 'error' event
^
SocketClosedUnexpectedlyError: Socket closed unexpectedly
at Socket.<anonymous> (/app/node_modules/@redis/client/dist/lib/client/socket.js:194:118)
...
Emitted 'error' event on Commander instance at:
at RedisSocket._RedisSocket_onSocketError (/app/node_modules/@redis/client/dist/lib/client/socket.js:218:10)
```
Kubernetes restarts the pod and the loop repeats every ~5 minutes (12
restarts in 95 min in our environment).
`twenty-worker` is unaffected — BullMQ's ioredis client has its own
keep-alive and the queue keeps it busy.
## Root cause
`packages/twenty-server/src/engine/core-modules/session-storage/session-storage.module-factory.ts`
constructs the node-redis client with no error listener:
```ts
const redisClient = createClient({ url: connectionString });
redisClient.connect().catch((err) => {
throw new Error(`Redis connection failed: ${err}`);
});
```
In Node.js, an unhandled `'error'` event on an `EventEmitter` becomes an
uncaught exception. node-redis emits `'error'` on socket close. With no
listener, the process exits 1 — even though node-redis would otherwise
reconnect on its own.
## Fix
1. Attach a `client.on('error', ...)` listener so disconnect errors are
logged. node-redis' built-in `reconnectStrategy` then takes over.
2. Set `pingInterval: 60_000` so the connection is never idle long
enough to be reaped by any reasonable Redis `timeout`. Defense in depth.
## Verification
Reproduced locally with Redis `CONFIG SET timeout 30` (30s for fast
reproduction). Without the fix: process exits 30s after boot. With the
fix: client logs the disconnect, reconnects, and the process keeps
running.
## Notes / out of scope
- `cache-storage.module-factory.ts` uses `cache-manager-redis-yet`
(which wraps node-redis under the hood). It may exhibit the same
vulnerability under sufficiently idle conditions; recommend a follow-up
to confirm and similarly harden it.
- `redis-client.service.ts` uses ioredis, which has built-in keepalive
and reconnect — no immediate crash risk, but adding error logging there
would be a nice consistency win.
## Test plan
- [ ] Existing tests still pass
- [ ] Manual: deploy with low Redis `timeout` (e.g. `30`), confirm
process survives
- [ ] Manual: kill Redis briefly, confirm twenty-server reconnects
instead of exiting
---------
Co-authored-by: Charles Bochet <charles@twenty.com>1 parent bbd9720 commit 2a97e77
2 files changed
Lines changed: 36 additions & 3 deletions
File tree
- packages/twenty-server/src/engine/core-modules
- cache-storage
- session-storage
Lines changed: 25 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
1 | 3 | | |
2 | 4 | | |
3 | | - | |
| 5 | + | |
| 6 | + | |
4 | 7 | | |
5 | 8 | | |
6 | 9 | | |
7 | 10 | | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
8 | 15 | | |
9 | 16 | | |
10 | 17 | | |
| |||
30 | 37 | | |
31 | 38 | | |
32 | 39 | | |
33 | | - | |
34 | | - | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
35 | 57 | | |
36 | 58 | | |
37 | 59 | | |
| |||
Lines changed: 11 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
| 4 | + | |
3 | 5 | | |
4 | 6 | | |
5 | 7 | | |
| |||
8 | 10 | | |
9 | 11 | | |
10 | 12 | | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
11 | 17 | | |
12 | 18 | | |
13 | 19 | | |
| |||
57 | 63 | | |
58 | 64 | | |
59 | 65 | | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
60 | 71 | | |
61 | 72 | | |
62 | 73 | | |
| |||
0 commit comments