catalog.node can hang indefinitely even when a wait is set

[jinnatar]

What I expected

For this call to always return a result or an exception:

            try:
                self._services_index[node_name], services = self.consul.catalog.node(
                    node_name,
                    index=self._services_index[node_name],
                    wait="5s",
                )
            except ConsulException as e:
                logger.error(f"Node {node_name} went away and was skipped: {e}")
                self.active_nodes.remove(node_name)
                continue  # skip to next node

What happened instead

Every now and then under circumstances I'm not able to pin down, the call hangs indefinitely without returning a result or an exception.

Further info

• It could be related to a consul node restart at the wrong time, or anything really, but it only happens on the order of weeks and is thus hard to pin down.
• Strace output:

strace: Process 3283546 attached
recvfrom(4,

• I can't get pdb injected after the fact because it's stuck in a syscall.

Happy to put in whatever hooks necessary to troubleshoot further, just not yet skilled enough to do that at the moment unguided. I didn't see any debug logging in the code that I could enable either.