IPython Kernel Configuration in ipython_kernel_config.py breaks `worker`

[kivel]

Description

The IPython profile connection configuration for the kernel found in <ipython_dir>/<profile_dir>/ipython_kernel_config.py is being ignored. Specifically, settings such as:

c.ConnectionFileMixin.connection_file = "/tmp/kernel-tst-fixed-ports.json"
c.ConnectionFileMixin.ip = "10.17.0.113"
c.ConnectionFileMixin.control_port = 60001
c.ConnectionFileMixin.hb_port = 60002
c.ConnectionFileMixin.iopub_port = 60003
c.ConnectionFileMixin.shell_port = 60004
c.ConnectionFileMixin.stdin_port = 60005

are being replaced by random ports instead of the specified values.
This causes the environments fail to load consistently.

Why is this problematic?

Firewall Restrictions: If only certain ports are open in a firewall, using ipython_kernel_config.py allows binding the kernel to fixed ports, ensuring connectivity.
Deterministic Connection: By defining a deterministic connection file, remote connections (e.g., jupyter console and jupyter lab) can reconnect after the environment is closed and re-opened.

Expected behavior

The IPython kernel should respect all configuration settings specified in ipython_kernel_config.py. This includes binding to the specified IP address and ports, and using the defined connection file. This behavior ensures that users can manage their network configurations effectively and maintain stable remote connections.

Steps to Reproduce:

connection file only

# ---  ~/.ipython/profile_dynamic_config/ipython_kernel_config.py
# Configuration file for ipython-kernel.
from pathlib import Path

KERNEL_PATH = "/tmp/jupyter/kernels"
BEAMLINE = "TST"
# Create the path
connection_file = Path(KERNEL_PATH) / f"kernel-{BEAMLINE.lower()}-dynamic.json"

# Ensure the folder exists
connection_file.parent.mkdir(parents=True, exist_ok=True)

c = get_config()  #noqa

c.ConnectionFileMixin.connection_file = str(connection_file)

start-re-manager --use-ipython-kernel=ON --startup-profile=dynamic_config

In case this is the first time the profile is used, or the connection file has been manually removed, the environment will open and the connection file will be created.
However, when the environment is closed and re-opened, the worker will attempt to create a new ipython kernel, with newly assigned random ports. This will raise a ValueError exception:

[I 2025-01-27 12:56:41,081 bluesky_queueserver.manager.worker] Preparing to start IPython kernel ...
[IPKernelApp] ERROR | Invalid Message
...
ValueError: Invalid Signature: b'c428a2e3bfe97ebfefe5aac7b472b84235d713dd4100b81c54ad3c77d454eb50'

This is due to a mismatch of the key in the existing connection file.

fully configured network

Similarly to the previous example, the ipython_kernel_config.py is extended to fully configure the network and a different connection file.

# --- ~/.ipython/profile_static_config/ipython_kernel_config.py
# Configuration file for ipython-kernel.
from pathlib import Path

KERNEL_PATH = "/tmp/jupyter/kernels"
BEAMLINE = "TST"
# Create the path
connection_file = Path(KERNEL_PATH) / f"kernel-{BEAMLINE.lower()}-static.json"

# Ensure the folder exists
connection_file.parent.mkdir(parents=True, exist_ok=True)

c = get_config()  #noqa

c.ConnectionFileMixin.connection_file = str(connection_file)
c.ConnectionFileMixin.ip = "10.37.0.33"
c.ConnectionFileMixin.control_port = 60001
c.ConnectionFileMixin.hb_port = 60002
c.ConnectionFileMixin.iopub_port = 60003
c.ConnectionFileMixin.shell_port = 60004
c.ConnectionFileMixin.stdin_port = 60005

This will not throw an exception, but will fail to fully load the environment, also the reported connection info during the startup is wrong.
Attempting to connect to the partially running kernel via

jupyter console --existing /tmp/jupyter/kernels/kernel-tst-static.json

will succeed. But closing the environment, fails with:

qserver environment close
Arguments: ['environment', 'close']
13:13:48 - MESSAGE: 
{'success': False, 'msg': 'RE Worker environment does not exist.'}

[dmgav]

The issue is resolved in #317
Extra configuration parameters for RE Manager were added that allow setting fixed 0MQ ports and connection file name.