Vault::ConnectionPool::PoolShuttingDownError

[PTcrusher]

Hi,

We are noticing a lot of Vault::ConnectionPool::PoolShuttingDownError lately.
After looking at the stack traces we saw the exception is raised in the pop method whenever the shutdown_block is set (which by looking at the code would only occur if we are in a middle of a client.shutdown call).

@mutex.synchronize do
  loop do
    raise ConnectionPool::PoolShuttingDownError if @shutdown_block
    (...)
    @resource.wait(@mutex, to_wait)
  end
end

Given so, could this mean that the resource is release before the connections are properly closed (waking up all treads waiting for the resource)?

  def shutdown(&block)
    raise ArgumentError, "shutdown must receive a block" unless block_given?
    @mutex.synchronize do
      @shutdown_block = block
      @resource.broadcast
      shutdown_connections
    end
  end

Shouldn't we make sure that all the connections are shutdown through the shutdown_connections method before the @resource.broadcast can actually be called?

Could the order of this two lines be the reason for the Vault::ConnectionPool::PoolShuttingDownError errors?

STACK_TRACE.log
CALLER_JOIN.log

[chrisarcand]

Two fixes in v0.19.0 (December 2024) likely resolved this:

1. Upgraded connection-pool from vendored 2.2.0 to upstream gem 2.4+ (5+ years of bug fixes)
2. Made shutdown lock-protected [http client's shutdown should be lock-protected #329] - ensures thread-safe access to @nhp

Closing as fixed in v0.19.0. If you still experience this issue on v0.19.0+, open a new issue with details.