Adopt conmon-rs as default container monitor

[saschagrunert]

Propose switching CRI-O from the traditional per-container conmon to the pod-level conmon-rs monitor in OpenShift.

• Introduce ConmonRS feature gate in TechPreviewNoUpgrade
• Extend ContainerRuntimeConfig with a containerMonitor field (PascalCase enum: Conmon, ConmonRS)
• Move conmon-rs package from RHCOS layer to the Node Layer (OCP-aligned versioning)
• Graduate from Tech Preview (opt-in) to GA (default) over one minor release cycle
• Tracking: OCPNODE-1288

/cc @haircommander @rphillips

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign rvanderp3 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[saschagrunert]

@bitoku @harche @QiWang19 PTAL, too

[dgrisonnet]

should we also verify that the containers in the pods are gracefully shutdown?

[saschagrunert]

Good catch. Added a "Graceful shutdown" test case that verifies SIGTERM delivery, preStop hook execution, and graceful exit before SIGKILL, compared against a conmon baseline. This is important since conmon-rs changes the signal forwarding path (pod-level vs. per-container).

[dgrisonnet]

What would happen to clusters using the machine config override during the upgrade to the version where the conmon binary is removed?

Also, is there a plan to remove the capacity to set runtime_type = "oci" in cri-o or will upstream continue to support both?

[saschagrunert]

Updated the "Removing a deprecated feature" section to address both questions.

For the override case: before the conmon binary is removed, the MCO must detect MachineConfig overrides referencing runtime_type = "oci" or a conmon monitor_path and block the upgrade during preflight checks. This prevents clusters from upgrading into a state where the override references a missing binary.

For upstream CRI-O: runtime_type = "oci" will continue to be supported upstream for the foreseeable future. conmon remains the default monitor in upstream CRI-O and in other distributions (Fedora, RHEL standalone). Removing the conmon binary from the OpenShift node image is an OCP-specific decision that doesn't affect upstream.

[dgrisonnet]

do we have a way to know why conmon-rs crashed if it ever happens?

[saschagrunert]

Added a "Crash diagnostics" bullet to the support procedures. Three main sources of information:

1. CRI-O logs the exit code and signal for the conmon-rs process when it exits unexpectedly.
2. conmon-rs logs its own operational output to journald by default (LogDriver::Systemd), so journalctl -t conmonrs on the affected node shows activity leading up to the crash. Panic messages go to stderr, which CRI-O captures. conmon-rs uses panic = "abort" in release builds, so panics terminate with SIGABRT immediately.
3. On RHCOS, systemd-coredump captures core dumps via the default kernel.core_pattern pipe. coredumpctl list / coredumpctl info on the node can be used for post-crash analysis.

[openshift-ci]

@saschagrunert: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.