Detector HDF plugin

[yannachen]

Jupy 15 20:30

uid: ['f25e8ac0-c419-4464-967c-5bb356e344fa']

Big energy moving

Plan failed: <WatchableAsyncStatus, device: ge_13element, task: <coroutine object AsyncStatusBase.__init__.<locals>.wait_with_error_message at 0x7ff09d9cb970>, errored: TimeoutError("Timeout Error while waiting 11.0s to update ['ca://XSP_Ge_13elem:HDF1:NumCaptured_RBV']. Last observed signal and value were (<ophyd_async.core._signal.SignalR object at 0x7ff0a56fe150>, 2)")>

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/tasks.py", line 507, in wait_for
    return await fut
           ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/queues.py", line 186, in get
    await getter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_signal.py", line 498, in observe_signals_value
    item = await asyncio.wait_for(q.get(), iteration_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/tasks.py", line 506, in wait_for
    async with timeouts.timeout(timeout):
               ~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/timeouts.py", line 116, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_status.py", line 37, in wait_with_error_message
    await awaitable
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_status.py", line 205, in _notify_watchers_from
    async for update in iterator:
    ...<6 lines>...
            self._update_watcher(watcher, self._last_update)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_detector.py", line 637, in trigger
    async for update in self._wait_for_index(
    ...<6 lines>...
        yield update
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_detector.py", line 508, in _wait_for_index
    async for sig, value in observe_signals_value(
    ...<15 lines>...
            break
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_signal.py", line 500, in observe_signals_value
    raise TimeoutError(
    ...<3 lines>...
    ) from exc
TimeoutError: Timeout Error while waiting 11.0s to update ['ca://XSP_Ge_13elem:HDF1:NumCaptured_RBV']. Last observed signal and value were (<ophyd_async.core._signal.SignalR object at 0x7ff0a56fe150>, 2)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky_queueserver/manager/worker.py", line 296, in _execute_plan
    result = func()
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky_queueserver/manager/worker.py", line 486, in start_plan_func
    return self._RE(plan_func(*plan_args, **plan_kwargs), {"all": [self._run_reg_cb]}, **plan_meta)
           ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 976, in __call__
    plan_return = self._resume_task(init_func=_build_task)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 1121, in _resume_task
    raise exc
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 1756, in _run
    raise err
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 1610, in _run
    msg = self._plan_stack[-1].send(resp)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1399, in __call__
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1250, in baseline_wrapper
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 863, in monitor_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 925, in fly_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1399, in __call__
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1250, in baseline_wrapper
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 863, in monitor_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 925, in fly_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/utils/__init__.py", line 1279, in dec_inner
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 548, in finalize_wrapper
    ret = yield from plan
          ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/src/haven/preprocessors/aps_suspenders.py", line 122, in install_suspenders
    yield from plan
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/utils/__init__.py", line 1279, in dec_inner
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/src/haven/preprocessors/open_shutters.py", line 88, in open_shutters_wrapper
    return_val = yield from new_plan
                 ^^^^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 548, in finalize_wrapper
    ret = yield from plan
          ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/src/haven/preprocessors/open_shutters.py", line 159, in open_on_kickoff_wrapper
    response = yield msg
               ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 213, in plan_mutator
    inner_ret = yield msg
                ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 213, in plan_mutator
    inner_ret = yield msg
                ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 213, in plan_mutator
    inner_ret = yield msg
                ^^^^^^^^^
  [Previous line repeated 5 more times]
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 2378, in _status_object_completed
    raise FailedStatus(ret) from exc
bluesky.utils.FailedStatus: <WatchableAsyncStatus, device: ge_13element, task: <coroutine object AsyncStatusBase.__init__.<locals>.wait_with_error_message at 0x7ff09d9cb970>, errored: TimeoutError("Timeout Error while waiting 11.0s to update ['ca://XSP_Ge_13elem:HDF1:NumCaptured_RBV']. Last observed signal and value were (<ophyd_async.core._signal.SignalR object at 0x7ff0a56fe150>, 2)")>

Plan failed: <WatchableAsyncStatus, device: ge_13element, task: <coroutine object AsyncStatusBase.__init__.<locals>.wait_with_error_message at 0x7ff09dae18a0>, errored: TimeoutError("Timeout Error while waiting 11.0s to update ['ca://XSP_Ge_13elem:HDF1:NumCaptured_RBV']. Last observed signal and value were (<ophyd_async.core._signal.SignalR object at 0x7ff0a56fe150>, 26)")>

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/tasks.py", line 507, in wait_for
    return await fut
           ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/queues.py", line 186, in get
    await getter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_signal.py", line 498, in observe_signals_value
    item = await asyncio.wait_for(q.get(), iteration_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/tasks.py", line 506, in wait_for
    async with timeouts.timeout(timeout):
               ~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/timeouts.py", line 116, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_status.py", line 37, in wait_with_error_message
    await awaitable
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_status.py", line 205, in _notify_watchers_from
    async for update in iterator:
    ...<6 lines>...
            self._update_watcher(watcher, self._last_update)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_detector.py", line 637, in trigger
    async for update in self._wait_for_index(
    ...<6 lines>...
        yield update
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_detector.py", line 508, in _wait_for_index
    async for sig, value in observe_signals_value(
    ...<15 lines>...
            break
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_signal.py", line 500, in observe_signals_value
    raise TimeoutError(
    ...<3 lines>...
    ) from exc
TimeoutError: Timeout Error while waiting 11.0s to update ['ca://XSP_Ge_13elem:HDF1:NumCaptured_RBV']. Last observed signal and value were (<ophyd_async.core._signal.SignalR object at 0x7ff0a56fe150>, 26)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky_queueserver/manager/worker.py", line 296, in _execute_plan
    result = func()
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky_queueserver/manager/worker.py", line 486, in start_plan_func
    return self._RE(plan_func(*plan_args, **plan_kwargs), {"all": [self._run_reg_cb]}, **plan_meta)
           ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 976, in __call__
    plan_return = self._resume_task(init_func=_build_task)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 1121, in _resume_task
    raise exc
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 1756, in _run
    raise err
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 1610, in _run
    msg = self._plan_stack[-1].send(resp)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1399, in __call__
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1250, in baseline_wrapper
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 863, in monitor_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 925, in fly_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1399, in __call__
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1250, in baseline_wrapper
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 863, in monitor_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 925, in fly_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/utils/__init__.py", line 1279, in dec_inner
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 548, in finalize_wrapper
    ret = yield from plan
          ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/src/haven/preprocessors/aps_suspenders.py", line 122, in install_suspenders
    yield from plan
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/utils/__init__.py", line 1279, in dec_inner
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/src/haven/preprocessors/open_shutters.py", line 88, in open_shutters_wrapper
    return_val = yield from new_plan
                 ^^^^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 548, in finalize_wrapper
    ret = yield from plan
          ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/src/haven/preprocessors/open_shutters.py", line 159, in open_on_kickoff_wrapper
    response = yield msg
               ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 213, in plan_mutator
    inner_ret = yield msg
                ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 213, in plan_mutator
    inner_ret = yield msg
                ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 213, in plan_mutator
    inner_ret = yield msg
                ^^^^^^^^^
  [Previous line repeated 5 more times]
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 2378, in _status_object_completed
    raise FailedStatus(ret) from exc
bluesky.utils.FailedStatus: <WatchableAsyncStatus, device: ge_13element, task: <coroutine object AsyncStatusBase.__init__.<locals>.wait_with_error_message at 0x7ff09dae18a0>, errored: TimeoutError("Timeout Error while waiting 11.0s to update ['ca://XSP_Ge_13elem:HDF1:NumCaptured_RBV']. Last observed signal and value were (<ophyd_async.core._signal.SignalR object at 0x7ff0a56fe150>, 26)")>

# Xspress3 is now ready to use!
epics> xsp4_system_start_count_enb: Xsp3Sys[path].run_flags=0x10C
xsp4_system_start_count_enb: Xsp3Sys[path].run_flags=0x10C
2026/07/15 20:40:11.999 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:40:12.000 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA0
2026/07/15 20:40:12.000 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:40:12.000 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA1
2026/07/15 20:40:12.000 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:40:12.000 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA2
2026/07/15 20:40:12.000 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:40:12.000 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA3
2026/07/15 20:40:12.000 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:40:12.000 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA4
2026/07/15 20:40:12.000 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:40:12.001 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA5
2026/07/15 20:40:12.001 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:40:12.001 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA6
2026/07/15 20:40:12.001 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:40:12.001 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA7
2026/07/15 20:40:12.001 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:40:12.001 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_EVENT_WIDTH
2026/07/15 20:40:12.001 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:40:12.001 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_DTFACTOR
2026/07/15 20:40:12.001 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:40:12.001 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_DTPERCENT

Before scan:

after failed:

# Xspress3 is now ready to use!
epics> 2026/07/15 20:50:46.622 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:50:46.623 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA0
2026/07/15 20:50:46.623 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:50:46.623 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA1
2026/07/15 20:50:46.623 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:50:46.624 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA2
2026/07/15 20:50:46.624 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:50:46.624 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA3
2026/07/15 20:50:46.624 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:50:46.624 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA4
2026/07/15 20:50:46.624 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:50:46.624 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA5
2026/07/15 20:50:46.624 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:50:46.624 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA6
2026/07/15 20:50:46.624 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:50:46.624 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA7
2026/07/15 20:50:46.624 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:50:46.624 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_EVENT_WIDTH
2026/07/15 20:50:46.624 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:50:46.624 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_DTFACTOR
2026/07/15 20:50:46.624 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:50:46.624 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_DTPERCENT
CAS: request from 10.54.135.159:59568 => put call back time out
CAS: Request from 10.54.135.159:59568 => cmmd=19 cid=0x1a8 type=4 count=1 postsize=8
CAS: Request from 10.54.135.159:59568 =>   available=0xab2b6 	N=0 paddr=0x7f29ec002f38
2026/07/15 20:52:27.285 NDFileHDF5:writeInt32: ERROR status=3, function=172, value=1 old=0
2026/07/15 20:52:27.285 XSP_Ge_13elem:HDF1:FlushNow devBusyAsyn::processCallbackOutput process error 
2026/07/15 20:52:28.541 NDFileHDF5:writeInt32: ERROR status=3, function=172, value=1 old=0
2026/07/15 20:52:28.541 XSP_Ge_13elem:HDF1:FlushNow devBusyAsyn::processCallbackOutput process error 
2026/07/15 20:52:29.411 NDFileHDF5:writeInt32: ERROR status=3, function=172, value=1 old=0
2026/07/15 20:52:29.411 XSP_Ge_13elem:HDF1:FlushNow devBusyAsyn::processCallbackOutput process error 
CAS: request from 10.54.135.159:59568 => put call back time out
CAS: Request from 10.54.135.159:59568 => cmmd=19 cid=0x19a type=0 count=1 postsize=8
CAS: Request from 10.54.135.159:59568 =>   available=0xab2b7 	N=0 paddr=0x7f29ec0034e8
2026/07/15 20:53:24.407 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:53:24.407 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA0
2026/07/15 20:53:24.407 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:53:24.407 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA1
2026/07/15 20:53:24.407 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:53:24.407 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA2
2026/07/15 20:53:24.407 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:53:24.407 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA3
2026/07/15 20:53:24.407 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:53:24.407 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA4
2026/07/15 20:53:24.407 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:53:24.407 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA5
2026/07/15 20:53:24.408 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:53:24.408 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA6
2026/07/15 20:53:24.408 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:53:24.408 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_SCA7
2026/07/15 20:53:24.408 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:53:24.408 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_EVENT_WIDTH
2026/07/15 20:53:24.408 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:53:24.408 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_DTFACTOR
2026/07/15 20:53:24.408 asynPortDriver:getAddress: XSP3 invalid address=13, must be in range 0 to 12
2026/07/15 20:53:24.408 paramAttribute:paramAttribute: ERROR, cannot find parameter XSP3_CHAN_DTPERCENT

[canismarko]

Hi, @yannachen. Anything to fix in haven based on this? If not, can it be closed?

[yannachen]

I am not sure what the problem is. What I can do is restart the hardware. If you confirmed it is not Haven-related, please close it.

[canismarko]

I'd rather not assume it's a problem in haven, we should figure out the problem first, then we can open an issue to fix it if it's in haven.

I will close this for now, if you have more information that points to something in haven, please feel free to re-open.