Skip to content

Chain don't get re-scheduled after being killed with notify_chain_stop() #789

Description

@0xgouda

Describe the bug
We have many chains with max_instances := 1, sometimes we want to disable one chain and kill the currently active execution of it - because it takes long to finish -, we use select timetable.notify_stop_chain(chain_id, worker_name);

We observe that the chain has stopped executing, but active_chain table still doesn't remove the old killed execution and hence after re-activating the chain subsequent runs keep logging Cannot proceed. Sleeping as they see the chain active.

To Reproduce
Steps to reproduce the behavior:

  1. Add the chain: SELECT timetable.add_job('test-finished-last_run', '* * * * *', 'select pg_sleep(300)', job_max_instances := 1);
  2. Inspect timetable.active_chain
  3. When the chain is active, kill it with select timetable.notify_chain_stop(chain_id, worker_name);
  4. observe that timetable.active_chain won't change at all
  5. see Cannot proceed, sleeping log messages in pg_timetable logs

Expected behavior
pg_timetable should correctly update timetable.active_chain when a chain gets killed by notify_chain_stop()

Screenshots

postgres=# SELECT timetable.add_job('test-finished-last_run', '* * * * *', 'select pg_sleep(300)', job_max_instances := 1);
 add_job
---------
      15
(1 row)

postgres=# select * from timetable.active_chain ;
 chain_id | client_name  |          started_at
----------+--------------+------------------------------
       15 | pg_timetable | 2026-07-10 17:22:31.32798+03
(1 row)

postgres=# select timetable.notify_chain_stop(15, 'pg_timetable');
 notify_chain_stop
-------------------

(1 row)

postgres=# update timetable.chain set live = false where chain_id = 15 ;
UPDATE 1
postgres=# update timetable.chain set live = true where chain_id = 15 ;
UPDATE 1
postgres=# select * from timetable.active_chain ;
 chain_id | client_name  |          started_at
----------+--------------+------------------------------
       15 | pg_timetable | 2026-07-10 17:22:31.32798+03
(1 row)

---------------------------------------------------------------

2026-07-10 17:22:31.329 [INFO] [chain:15] [task:14] [vxid:25769804593] Starting task
2026-07-10 17:22:31.331 [INFO] [chain:15] [task:14] [vxid:25769804593] Switching to autonomous task mode
2026-07-10 17:22:31.335 [INFO] [chain:1] [task:1] [vxid:30064771470] Starting task
2026-07-10 17:22:31.341 [INFO] [chain:1] [task:1] [vxid:30064771470] {"Description": "Log Execution"}
2026-07-10 17:22:31.348 [INFO] [chain:1] [task:1] [vxid:30064771470] Task executed successfully
2026-07-10 17:22:31.348 [INFO] [chain:1] [vxid:30064771470] Chain executed successfully
2026-07-10 17:23:31.319 [INFO] [notification:{119301 pg_timetable {"ConfigID": 15, "Command": "STOP", "Ts": 1783693360}}] [pid:155156] [signal:{15 STOP 1783693360 0}] Adding asynchronous chain to working queue
2026-07-10 17:23:31.319 [ERROR] [chain:15] [task:14] [sql:select pg_sleep(300)] [args:[]] [err:context canceled] [pid:155157] [time:59.988433701s] [vxid:25769804593] Query
2026-07-10 17:23:31.320 [INFO] [count:0] Retrieve interval chains to run
2026-07-10 17:23:31.323 [INFO] [count:1] Retrieve scheduled chains to run
2026-07-10 17:23:31.324 [INFO] [chain:1] Starting chain
2026-07-10 17:23:31.326 [INFO] [chain:1] [task:1] [vxid:30064771481] Starting task
2026-07-10 17:23:31.326 [INFO] [chain:15] [task:14] [vxid:25769804593] Closing standalone session
2026-07-10 17:23:31.326 [ERROR] [chain:15] [task:14] [error:context canceled] [vxid:25769804593] Task execution failed
2026-07-10 17:23:31.326 [INFO] [chain:15] [task:14] [vxid:25769804593] Ignoring task failure
2026-07-10 17:23:31.326 [ERROR] [chain:15] [sql:commit] [args:[]] [err:timeout: context already done: context canceled] [pid:155159] [time:2.333µs] [vxid:25769804593] Query
2026-07-10 17:23:31.326 [ERROR] [chain:15] [error:timeout: context already done: context canceled] [vxid:25769804593] Application cannot commit after job finished
2026-07-10 17:23:31.326 [INFO] [chain:15] [vxid:25769804593] Chain executed successfully
2026-07-10 17:23:31.326 [ERROR] [chain:15] [err:context canceled] [time:902ns] [vxid:25769804593] Acquire
2026-07-10 17:23:31.326 [ERROR] [error:context canceled] Cannot save information about the chain run status
2026-07-10 17:23:31.333 [INFO] [chain:1] [task:1] [vxid:30064771481] {"Description": "Log Execution"}
2026-07-10 17:23:31.333 [INFO] [chain:1] [task:1] [vxid:30064771481] Task executed successfully
2026-07-10 17:23:31.333 [INFO] [chain:1] [vxid:30064771481] Chain executed successfully
2026-07-10 17:24:31.321 [INFO] [count:0] Retrieve interval chains to run
2026-07-10 17:24:31.327 [INFO] [count:2] Retrieve scheduled chains to run
2026-07-10 17:24:31.329 [INFO] [chain:15] Cannot proceed. Sleeping

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Fields

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions