When tests/integration/test_k8s.py::test_remove_non_leader_control_plane is executed on three k8s node deployment one of ceph-csi units ends in error
Model Controller Cloud/Region Version SLA Timestamp
canonicalk8s-maas-ceph foundations-maas maas_cloud/default 3.6.14 unsupported 15:55:59Z
App Version Status Scale Charm Channel Rev Exposed Message
ceph-csi v3.9.0,v0,v3... active 3 ceph-csi latest/edge 97 no Versions: cephfs=v3.9.0, config=v0, rbd=v3.9.0
ceph-mon 17.2.9 active 3 ceph-mon quincy/stable 396 no Unit is ready and clustered
ceph-osd 17.2.9 active 3 ceph-osd quincy/stable 761 no Unit is ready (1 OSD)
k8s 1.35.0 waiting 2/3 k8s 1.35/beta 1823 yes Waiting to verify features
k8s-worker 1.35.0 active 3 k8s-worker 1.35/beta 1821 no Ready
Unit Workload Agent Machine Public address Ports Message
ceph-mon/0 active idle 7 10.241.144.92 Unit is ready and clustered
ceph-mon/1* active idle 9 10.241.144.86 Unit is ready and clustered
ceph-mon/2 active idle 10 10.241.144.88 Unit is ready and clustered
ceph-osd/0 active idle 11 10.241.144.93 Unit is ready (1 OSD)
ceph-osd/1* active idle 12 10.241.144.91 Unit is ready (1 OSD)
ceph-osd/2 active idle 13 10.241.144.95 Unit is ready (1 OSD)
k8s-worker/0 active idle 1 10.241.144.84 Ready
k8s-worker/2* active idle 3 10.241.144.94 Ready
k8s-worker/3 active idle 14 10.241.144.96 Ready
k8s/0 waiting failed 4 10.241.144.87 6443/tcp Waiting to verify features
ceph-csi/1 error idle 10.241.144.87 hook failed: "kubernetes-info-relation-departed"
k8s/1* active idle 5 10.241.144.90 6443/tcp Ready
ceph-csi/0* active idle 10.241.144.90 Ready
k8s/2 active idle 6 10.241.144.85 6443/tcp Ready
ceph-csi/2 active idle 10.241.144.85 Ready
Machine State Address Inst id Base AZ Message
1 started 10.241.144.84 node1 ubuntu@22.04 zone1 Deployed
3 started 10.241.144.94 node2 ubuntu@22.04 zone2 Deployed
4 started 10.241.144.87 microk8s-1 ubuntu@22.04 zone1 Deployed
5 started 10.241.144.90 microk8s-3 ubuntu@22.04 zone3 Deployed
6 started 10.241.144.85 microk8s-2 ubuntu@22.04 zone2 Deployed
7 started 10.241.144.92 vault-1 ubuntu@22.04 zone1 Deployed
9 started 10.241.144.86 vault-3 ubuntu@22.04 zone3 Deployed
10 started 10.241.144.88 vault-2 ubuntu@22.04 zone2 Deployed
11 started 10.241.144.93 node5 ubuntu@22.04 zone2 Deployed
12 started 10.241.144.91 node4 ubuntu@22.04 zone1 Deployed
13 started 10.241.144.95 node6 ubuntu@22.04 zone3 Deployed
14 started 10.241.144.96 node3 ubuntu@22.04 zone3 Deployed
2026-02-16 09:23:10 DEBUG juju.worker.uniter.runner runner.go:753 starting jujuc server {unix /var/lib/juju/agents/unit-ceph-csi-2/agent.socket <nil>}
2026-02-16 09:23:11 DEBUG unit.ceph-csi/2.juju-log server.go:405 ceph-client:12: ops 3.5.2 up and running.
2026-02-16 09:23:11 DEBUG unit.ceph-csi/2.juju-log server.go:405 ceph-client:12: Emitting Juju event ceph_client_relation_departed.
2026-02-16 09:23:11 INFO unit.ceph-csi/2.juju-log server.go:405 ceph-client:12: Installing ceph-common to version: 17.2.9-0ubuntu0.22.04.1.amd64
2026-02-16 09:23:11 DEBUG unit.ceph-csi/2.juju-log server.go:405 ceph-client:12: connect_tcp.started host='127.0.0.1' port=6443 local_address=None timeout=10.0 socket_options=None
2026-02-16 09:23:11 DEBUG unit.ceph-csi/2.juju-log server.go:405 ceph-client:12: connect_tcp.failed exception=ConnectError(ConnectionRefusedError(111, 'Connection refused'))
2026-02-16 09:23:11 INFO unit.ceph-csi/2.juju-log server.go:405 ceph-client:12: Status context closed with: []
2026-02-16 09:23:11 ERROR unit.ceph-csi/2.juju-log server.go:405 ceph-client:12: Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
yield
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_transports/default.py", line 250, in handle_request
resp = self._pool.handle_request(req)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpcore/_sync/connection_pool.py", line 256, in handle_request
raise exc from None
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpcore/_sync/connection_pool.py", line 236, in handle_request
response = connection.handle_request(
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpcore/_sync/connection.py", line 101, in handle_request
raise exc
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpcore/_sync/connection.py", line 78, in handle_request
stream = self._connect(request)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpcore/_sync/connection.py", line 124, in _connect
stream = self._network_backend.connect_tcp(**kwargs)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpcore/_backends/sync.py", line 207, in connect_tcp
with map_exceptions(exc_map):
File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/./src/charm.py", line 611, in <module>
ops.main.main(CephCsiCharm)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/__init__.py", line 363, in main
return _legacy_main.main(
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/main.py", line 39, in main
return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/_main.py", line 507, in main
manager.run()
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/_main.py", line 491, in run
self._emit()
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/_main.py", line 426, in _emit
self._emit_charm_event(self.dispatcher.event_name)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/_main.py", line 470, in _emit_charm_event
event_to_emit.emit(*args, **kwargs)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/framework.py", line 337, in emit
framework._emit(event)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/framework.py", line 910, in _emit
self._reemit(event_path)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/framework.py", line 1028, in _reemit
custom_handler(event)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/charms/reconciler.py", line 45, in reconcile
self.reconcile_function(event)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/./src/charm.py", line 531, in reconcile
self.check_namespace()
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/./src/charm.py", line 374, in check_namespace
self._client.get(Namespace, name=namespace)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/lightkube/core/client.py", line 229, in get
return self._client.request("get", res=res, name=name, namespace=namespace)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/lightkube/core/generic_client.py", line 341, in request
resp = self.send(req)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/lightkube/core/generic_client.py", line 304, in send
return self._client.send(req, stream=stream)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_client.py", line 914, in send
response = self._send_handling_auth(
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_client.py", line 942, in _send_handling_auth
response = self._send_handling_redirects(
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_client.py", line 979, in _send_handling_redirects
response = self._send_single_request(request)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_client.py", line 1014, in _send_single_request
response = transport.handle_request(request)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_transports/default.py", line 249, in handle_request
with map_httpcore_exceptions():
File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 111] Connection refused
When tests/integration/test_k8s.py::test_remove_non_leader_control_plane is executed on three k8s node deployment one of ceph-csi units ends in error
example
link to logs
I saw also
hook failed: "ceph-client-relation-departed"In that case logs showed