fix: Cap WorkerLock timeout intervals to 60 seconds

[jason-famedly]

Fixes the symptoms of #19315 / #19588 but not the underlying reason causing the number to grow so large in the first place.

ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit

Copied from the original pull request on Famedly's Synapse repo (with some edits):

Basing the time interval around a 5 seconds leaves a big window of waiting especially as this window is doubled each retry, when another worker could be making progress but can not.

Right now, the retry interval in seconds looks like [0.2, 5, 10, 20, 40, 80, 160, 320, (continues to double)] after which logging should start about excessive times and (relatively quickly) end up with an extremely large retry interval with an unrealistic expectation past the heat death of the universe. 1 year in seconds = 31,536,000.

With this change, retry intervals in seconds should look more like:

[
0.2, 
0.4, 
0.8, 
1.6, 
3.2, 
6.4, 
12.8, 
25.6, 
51.2, 
60, < never goes higher than this
]

Logging about excessive wait times will start at 10 minutes.

Previous breakdown when we were using 15 minutes

[
0.2, 
0.4, 
0.8, 
1.6, 
3.2, 
6.4, 
12.8, 
25.6, 
51.2, 
102.4,  # 1.7 minutes
204.8,  # 3.41 minutes
409.6,  # 6.83 minutes
819.2,  # 13.65 minutes  < logging about excessive times will start here, 13th iteration
900,  # 15 minutes < never goes higher than this
]

Further suggested work in this area could be to define the cap, the retry interval starting point and the multiplier depending on how frequently this lock should be checked. See data below for reasons why. Increasing the jitter range may also be a good idea

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct (run the linters)

[CLAassistant]

All committers have signed the CLA.

[denzs]

After the issue occured again in our prod:

2026-01-22 13:36:53.725errorsynapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 5120s. There may be a deadlock.
2026-01-22 13:36:53.981errorsynapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 10240s. There may be a deadlock.
2026-01-22 13:36:54.560errorsynapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 20480s. There may be a deadlock.
2026-01-22 13:36:54.798errorsynapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 40960s. There may be a deadlock.
2026-01-22 13:36:56.342errorsynapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 81920s. There may be a deadlock.

My hypothesis would be: the issue is not primarily about the dimensions of the growing timeout but, about the timeout being ignored at all?

At least the logged timeout is not reflected in the timestamp deltas of the log lines?!

[jason-famedly]

After the issue occured again in our prod:

2026-01-22 13:36:53.725errorsynapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 5120s. There may be a deadlock.
2026-01-22 13:36:53.981errorsynapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 10240s. There may be a deadlock.
2026-01-22 13:36:54.560errorsynapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 20480s. There may be a deadlock.
2026-01-22 13:36:54.798errorsynapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 40960s. There may be a deadlock.
2026-01-22 13:36:56.342errorsynapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 81920s. There may be a deadlock.

My hypothesis would be: the issue is not primarily about the dimensions of the growing timeout but, about the timeout being ignored at all?

At least the logged timeout is not reflected in the timestamp deltas of the log lines?!

Yes there is more than one thing going on here. This fix(switch max() to min() and adjust iteration assumptions) is only to fix obnoxiously long strings of numbers that are trying to reach infinity not being introduced in the first place. The underlying cause is something else: timeouts seem to not be honored as well what the request that triggers the situation at all is doing to cause the locks to pile up and not make progress

[jason-famedly]

Added some additional work onto this:

1. Adjusted for the fact that the retry interval was actually for time outs, by renaming it
2. Adjusted that a timeout interval should only be increased if an actual timeout was reached
3. Added a warning level log to the generic Exception to see if it is even being hit
4. Added/adjusted tests to check for time outs being hit(there is a typo in one that I will fix after the first test series is run)

This allows that a normal notification of another lock being released should not increment a timeout, when a time out has not actually occurred. It should cut down on what may end up otherwise being excessive log spam about locks having a long timeout duration when such is not true. The jitter value is maintained for the timeout, to help avoid a "thundering herd" situation when all locks may time out at the same time.

[MadLittleMods]

To be clear, nothing for you to do here yet. Just noting this discussion so we can think about what to do for these scenarios.

Since #19602, pump() is deprecated and we can be more precise here by using self.reactor.advance(...)

But actually trying this out here and there some internal details leaking through and we would end up with this which isn't great.

Suggested change

	# Recall that pump() will advance time of the given amount 100 times, this
	# amounts to about 10 seconds passing
	self.pump(0.1)
	# Wait for database queries to run so we get past the
	# `try_acquire_lock(...)` in `WaitingLock` and the timeout actually gets
	# scheduled
	self.reactor.advance(0)
	# Wait some time which should trigger waiting loop to timeout and retry
	# acquiring the lock a few times
	self.reactor.advance(10)

Related comments around how the database queries are ran (added in matrix-org/synapse#8497):

synapse/tests/server.py

Lines 660 to 669 in 76b4fdc

	# check for more "callLater" callbacks added by the thread callback
	# This isn't required in a regular reactor, but it ends up meaning that
	# our database queries can complete in a single call to `advance` [1] which
	# simplifies tests.
	#
	# [1]: we replace the threadpool backing the db connection pool with a
	# mock ThreadPool which doesn't really use threads; but we still use
	# reactor.callFromThread to feed results back from the db functions to the
	# main thread.
	super().advance(0)

cc @reivilibre (someone who also has been in the loop around pump() vs advance())

I can maybe see how pump(...) became a thing 🤔 but it's still imprecise.

[MadLittleMods]

Just to note for people who might come back to look at this PR, the implementation I recommended here isn't quite precise as it may advance time by one by increment beyond the amount. But it's fine for our use case ⏩

[reivilibre]

Generally fine other than a little footgun with our Duration class!

[reivilibre]

Thank you!

[MadLittleMods]

Fixes the symptoms of #19315 / #19588 but not the underlying reason causing the number to grow so large in the first place.

I think this fixes the actual problems, not just the symptoms.

The root cause is that _retry_interval can grow past Duration(timedelta)'s max value and raise an OverflowError. When this occurs, there is a tight-loop where the OverflowError propagates up from Duration() → clock.call_later() → timeout_deferred() → await timeout_deferred(...), where it is caught by the bare except Exception: pass at line 254-255. The loop immediately continues to the next iteration with no waiting at all.

Even the initially proposed change here fixed the problem of the _retry_interval/timeout growing too large. Additionally, this PR has since evolved to only increase the timeout when we actually encounter a defer.TimeoutError which is another fix that would solve this.

It's possible that this code could tight-loop again if we encounter some other exception over and over but I guess that's expected and our "critical" section is already as small as possible around timeout_deferred(...). We could have some tight-loop detection but feels overkill as at some point, we have to trust timeout_deferred(...) to do the right thing. Overall, the situation is better here as this PR adds some logging for the general exception case instead of swallowing the error and moving on so you can actually notice any tight-loop root cause now.

Random dev notes

Reproduce Duration(timedelta) raising OverflowError

$ poetry run python
>>> Duration(seconds=10 ** 14)
Traceback (most recent call last):
  File "<python-input-5>", line 1, in <module>
    Duration(seconds=10 ** 14)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^
OverflowError: days=1157407407; must have magnitude <= 999999999