Skip to content

ceph-csi goes to error after k8s unit is removed and added #70

Description

@marosg42

When tests/integration/test_k8s.py::test_remove_non_leader_control_plane is executed on three k8s node deployment one of ceph-csi units ends in error

example

Model                   Controller        Cloud/Region        Version  SLA          Timestamp
canonicalk8s-maas-ceph  foundations-maas  maas_cloud/default  3.6.14   unsupported  15:55:59Z

App         Version          Status   Scale  Charm       Channel         Rev  Exposed  Message
ceph-csi    v3.9.0,v0,v3...  active       3  ceph-csi    latest/edge      97  no       Versions: cephfs=v3.9.0, config=v0, rbd=v3.9.0
ceph-mon    17.2.9           active       3  ceph-mon    quincy/stable   396  no       Unit is ready and clustered
ceph-osd    17.2.9           active       3  ceph-osd    quincy/stable   761  no       Unit is ready (1 OSD)
k8s         1.35.0           waiting    2/3  k8s         1.35/beta      1823  yes      Waiting to verify features
k8s-worker  1.35.0           active       3  k8s-worker  1.35/beta      1821  no       Ready

Unit           Workload  Agent   Machine  Public address  Ports     Message
ceph-mon/0     active    idle    7        10.241.144.92             Unit is ready and clustered
ceph-mon/1*    active    idle    9        10.241.144.86             Unit is ready and clustered
ceph-mon/2     active    idle    10       10.241.144.88             Unit is ready and clustered
ceph-osd/0     active    idle    11       10.241.144.93             Unit is ready (1 OSD)
ceph-osd/1*    active    idle    12       10.241.144.91             Unit is ready (1 OSD)
ceph-osd/2     active    idle    13       10.241.144.95             Unit is ready (1 OSD)
k8s-worker/0   active    idle    1        10.241.144.84             Ready
k8s-worker/2*  active    idle    3        10.241.144.94             Ready
k8s-worker/3   active    idle    14       10.241.144.96             Ready
k8s/0          waiting   failed  4        10.241.144.87   6443/tcp  Waiting to verify features
  ceph-csi/1   error     idle             10.241.144.87             hook failed: "kubernetes-info-relation-departed"
k8s/1*         active    idle    5        10.241.144.90   6443/tcp  Ready
  ceph-csi/0*  active    idle             10.241.144.90             Ready
k8s/2          active    idle    6        10.241.144.85   6443/tcp  Ready
  ceph-csi/2   active    idle             10.241.144.85             Ready

Machine  State    Address        Inst id     Base          AZ     Message
1        started  10.241.144.84  node1       ubuntu@22.04  zone1  Deployed
3        started  10.241.144.94  node2       ubuntu@22.04  zone2  Deployed
4        started  10.241.144.87  microk8s-1  ubuntu@22.04  zone1  Deployed
5        started  10.241.144.90  microk8s-3  ubuntu@22.04  zone3  Deployed
6        started  10.241.144.85  microk8s-2  ubuntu@22.04  zone2  Deployed
7        started  10.241.144.92  vault-1     ubuntu@22.04  zone1  Deployed
9        started  10.241.144.86  vault-3     ubuntu@22.04  zone3  Deployed
10       started  10.241.144.88  vault-2     ubuntu@22.04  zone2  Deployed
11       started  10.241.144.93  node5       ubuntu@22.04  zone2  Deployed
12       started  10.241.144.91  node4       ubuntu@22.04  zone1  Deployed
13       started  10.241.144.95  node6       ubuntu@22.04  zone3  Deployed
14       started  10.241.144.96  node3       ubuntu@22.04  zone3  Deployed

link to logs

I saw also hook failed: "ceph-client-relation-departed"
In that case logs showed

2026-02-16 09:23:10 DEBUG juju.worker.uniter.runner runner.go:753 starting jujuc server  {unix /var/lib/juju/agents/unit-ceph-csi-2/agent.socket <nil>}
2026-02-16 09:23:11 DEBUG unit.ceph-csi/2.juju-log server.go:405 ceph-client:12: ops 3.5.2 up and running.
2026-02-16 09:23:11 DEBUG unit.ceph-csi/2.juju-log server.go:405 ceph-client:12: Emitting Juju event ceph_client_relation_departed.
2026-02-16 09:23:11 INFO unit.ceph-csi/2.juju-log server.go:405 ceph-client:12: Installing ceph-common to version: 17.2.9-0ubuntu0.22.04.1.amd64
2026-02-16 09:23:11 DEBUG unit.ceph-csi/2.juju-log server.go:405 ceph-client:12: connect_tcp.started host='127.0.0.1' port=6443 local_address=None timeout=10.0 socket_options=None
2026-02-16 09:23:11 DEBUG unit.ceph-csi/2.juju-log server.go:405 ceph-client:12: connect_tcp.failed exception=ConnectError(ConnectionRefusedError(111, 'Connection refused'))
2026-02-16 09:23:11 INFO unit.ceph-csi/2.juju-log server.go:405 ceph-client:12: Status context closed with: []
2026-02-16 09:23:11 ERROR unit.ceph-csi/2.juju-log server.go:405 ceph-client:12: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_transports/default.py", line 250, in handle_request
    resp = self._pool.handle_request(req)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpcore/_sync/connection.py", line 101, in handle_request
    raise exc
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpcore/_sync/connection.py", line 78, in handle_request
    stream = self._connect(request)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpcore/_backends/sync.py", line 207, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/./src/charm.py", line 611, in <module>
    ops.main.main(CephCsiCharm)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/__init__.py", line 363, in main
    return _legacy_main.main(
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/main.py", line 39, in main
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/_main.py", line 507, in main
    manager.run()
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/_main.py", line 491, in run
    self._emit()
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/_main.py", line 426, in _emit
    self._emit_charm_event(self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/_main.py", line 470, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/framework.py", line 337, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/framework.py", line 910, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/ops/framework.py", line 1028, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/charms/reconciler.py", line 45, in reconcile
    self.reconcile_function(event)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/./src/charm.py", line 531, in reconcile
    self.check_namespace()
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/./src/charm.py", line 374, in check_namespace
    self._client.get(Namespace, name=namespace)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/lightkube/core/client.py", line 229, in get
    return self._client.request("get", res=res, name=name, namespace=namespace)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/lightkube/core/generic_client.py", line 341, in request
    resp = self.send(req)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/lightkube/core/generic_client.py", line 304, in send
    return self._client.send(req, stream=stream)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/var/lib/juju/agents/unit-ceph-csi-2/charm/venv/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 111] Connection refused

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions