Add cloud-connector as a native foremanctl feature

[jeremylenz]

Summary

• Re-implements the upstream satellite_operations.cloud_connector role natively in foremanctl (SAT-45966 / SAT-44641)
• New cloud-connector feature gated behind --add-feature cloud-connector, with rh-cloud dependency
• Installs rhc and yggdrasil-worker-forwarder, configures the worker, starts rhcd, sets rhc_instance_id via the Foreman API, and announces to Sources
• Works with both foremanctl deploy and forge deploy-dev
• Uses OAuth for all Foreman API calls during setup, matching the pattern used by other roles (foreman_proxy, iop_core, pulp, backup)
• Creates a dedicated cloud_connector_user service user with limited dispatch_cloud_requests permission for runtime worker authentication
• Adds a reusable foremanctl_api custom module for OAuth-authenticated API calls where no FAM module exists
• Early pre-checks validate consumer certificate presence and mutual exclusion with the iop feature
• Adds Foreman CA to system trust store so the worker binary can verify Foreman's certificate
• Optional --cloud-connector-http-proxy flag for environments without direct internet access

Companion PR: theforeman/foreman_rh_cloud#1214 (adds the announce_to_sources API endpoint)

Test plan

• foremanctl deploy --add-feature cloud-connector completes successfully
• foremanctl features --list-enabled includes cloud-connector
• systemctl status rhcd shows active/running
• /etc/rhc/workers/foreman_rh_cloud.toml has correct content with service user credentials
• hammer settings info --name rhc_instance_id shows the consumer cert CN
• hammer user info --login cloud_connector_user shows the service user with Cloud Connector role
• Second foremanctl deploy is idempotent
• forge deploy-dev --add-feature cloud-connector completes successfully
• Enabling both cloud-connector and iop fails early with a clear error

🤖 Generated with Claude Code

[ehelms]

Is this how we do it today? This is an anti-pattern we have been trying to avoid.

[jeremylenz]

foreman-installer does the same thing via puppet-certs:

• templates/rhsm-katello-reconfigure.erb — copies to /etc/pki/ca-trust/source/anchors/ and runs update-ca-trust
• manifests/katello.pp — uses trusted_ca::ca to do the same

We need it here because yggdrasil-worker-forwarder is a Go binary that uses the OS trust store — no way to pass it a CA path. That said, we could move this into the certificates role so it's done once globally instead of per-feature. Would that be cleaner?

[jeremylenz]

I was apprehensive about this too. I figured we can change it later..
Claude reply above

[ehelms]

• templates/rhsm-katello-reconfigure.erb — copies to /etc/pki/ca-trust/source/anchors/ and runs update-ca-trust
• manifests/katello.pp — uses trusted_ca::ca to do the same

These are both 404s because they were moved quite a bit ago. I think we need to consider udpating yggdrasil-worker-forwarder vs. starting this trend of relying on the system store. Let's phone a friend for another opinion. @evgeni ☎️

[jeremylenz]

We need to update it anyway to use DBUS (so it can run on RHEL 10), so we can probably just tack that change on there.

[jeremylenz]

https://github.com/theforeman/yggdrasil-worker-forwarder doesn't seem to use Github Issues, so I will create an internal Jira for that.

[jeremylenz]

I updated the description of https://redhat.atlassian.net/browse/SAT-27307. However, ideally I would like this PR to be merged and use the "incorrect" approach temporarily, to move things forward,.

[ehelms]

The problem is -- if this makes this change, and then another feature relies on it, we end up with a snowball effect of things relying on this paradigm. I think we need to come to a consensus about whether we configure the system trust with our CA or not.

[evgeni]

Yeah, seems my ☎️ was broken and I never seen this ping.

But I do agree with Eric, we do not want to add certs to the system trust, as

• it's not inherited by containers, so they need the explicit file passed anyway
• it makes tools we do not manage trust this CA
• it's a pattern we are trying to avoid as it makes things inconsistent (because of the above)

[ehelms]

This is the part we still need to work out how to properly handle, that is, the authentication.

[jeremylenz]

As I see it our options are
1 - Keep things as they are here, feeding admin/changeme to rhc - Not acceptable because the admin password ends up in the worker config.toml.
2 - Create a service user with limited permissions - Doable in the existing PRs here and in foreman_rh_cloud.
3 - Create a service user + personal access token and use that, same as the previous architecture - FAM doesn't have a module for personal access tokens, so this would be a fair bit of work

I think Option 2 seems promising and am pushing that update now.

[jeremylenz]

@ehelms Updated to use OAuth, but we still need the service user because yggdrasil-worker-forwarder only supports HTTP basic auth, and doesn't support OAuth.

[jeremylenz]

Pushed updates addressing review feedback (1bfe640):

• Handler: switched from ansible.builtin.service to ansible.builtin.systemd_service since we use daemon_reload
• Workers directory: removed the task — confirmed rhc package on EL9 creates /etc/rhc/workers/
• Settings API: replaced raw uri calls with theforeman.foreman.setting module for rhc_instance_id and allow_auto_inventory_upload
• Lint fix: added noqa: no-static-secrets on the defaults password (same pattern as foreman_development)

[jeremylenz]

rebased & fixed conflicts

[qcjames53]

I'm not enough in the loop to give this a full review but had a few comments that might be helpful. Consider this 80% good points 20% ass-talking. Here's what I was able to test directly:

• The iop conflict and missing consumer cert checks both work on the positive and negative cases (used manual subscription manager commands for the latter).
• The yggdrasil-worker-forwarder check works in the negative (was unable to figure out package install on centos9).
• The cloud connector role works.

Beyond these tests, I have questions about the whole approach of cloud connector even after reading the planning docs and such.

1. How are we splitting RHEL and Centos 9 deploys? I think the differentiation needs to be clearer since (best I can tell) yggdrasil-worker-forwarder is RHEL only. If the packaged container images are centos9 and the rpms needed for cloud connector are RHEL-only, that seems like a big problem to me!
2. deploy-dev needs to work with cloud connector. At minimum a procedure but ideally out of the box. I think we really need a procedure for setting up a centos9 deploy-dev environment to build/test cloud connector. If that's not possible, then we need a good procedure for setting up a RHEL environment as deploy-dev.
3. If it's possible to pass in subscription manager credentials safely, I would greatly prefer if we could deploy an environment with cloud connector immediately instead of requiring manual registration for the quadlet VM, THEN adding the feature. I understand this won't be an issue for customers but this would be HUGE for testing purposes.

I have to gracefully bow out of reviewing this for a few reasons: capacity + artemis stuff + this baby could come any day! Sorry to not complete testing but I put in an honest effort.

[qcjames53]

yggdrasil-worker-forwarder - Is this available in centos9? Where is it shipped? I know it won't be a problem for shipping pre-built images but for dev environments and such we should really document a pathway for getting this installed.

[jeremylenz]

I checked and it's in https://yum.theforeman.org/plugins/3.19/el9/x86_64/

[qcjames53]

Same as above but opposite. Idempotency?

[jeremylenz]

This is correct — changed_when: false means "this command never changes anything," which is right for a read-only openssl x509 call. Standard Ansible pattern for info-gathering tasks.

[jeremylenz]

I'll respond with thoughts off the top of my head :)

1. How are we splitting RHEL and Centos 9 deploys? I think the differentiation needs to be clearer since (best I can tell) yggdrasil-worker-forwarder is RHEL only. If the packaged container images are centos9 and the rpms needed for cloud connector are RHEL-only, that seems like a big problem to me!

In order for cloud connector to work you need available on your base system:

1. ansible-runner command - Downstream, from the Satellite repo, e.g. satellite-6.19-for-rhel-9-x86_64-rpms. Upstream, from the Foreman Plugins repo, e.g. https://yum.theforeman.org/plugins/3.19/el9/x86_64/
2. yggdrasil-worker-forwarder - Downstream, from the Satellite repo. Upstream, from the Foreman Plugins repo, e.g. https://yum.theforeman.org/plugins/3.19/el9/x86_64/
3. rhc - from AppStream (CentOS or RHEL)

In the previous architecture, these weren't even listed as prerequisites, even in upstream docs. Downstream, you'd already have all of these since the Satellite system must be registered and have those repos enabled. But upstream, anything can happen. It seems this is a documentation gap. cc @Lennonka

2. deploy-dev needs to work with cloud connector. At minimum a procedure but ideally out of the box. I think we really need a procedure for setting up a centos9 deploy-dev environment to build/test cloud connector. If that's not possible, then we need a good procedure for setting up a RHEL environment as deploy-dev.

I was testing with deploy-dev on RHEL (with --target-host localhost). I think it works fine, as long as you have the repos available that I mentioned above.

[jeremylenz]

@qcjames53 Addressed 2 comments and replied to the third

[lfu]

Tested cloud connector feature in development mode. All core functionality works correctly. Found 3 necessary configuration changes for development environment.

---

🔍 Testing Methodology

Environment: Development mode using forge deploy-dev with Rails source code
Test Scope:

• Cloud connector deployment
• Service user authentication
• API endpoint functionality
• Worker configuration

🔗 Related/Dependent PRs

Foreman RH Cloud Plugin

• PR: Repurpose CloudConnectorAnnounceTask for daily Sources registration foreman_rh_cloud#1214
• Purpose: Cloud connector service user and role support
• Status: Pending
• Dependency: Required for cloud connector service user authentication functionality

Foremanctl Branch Preservation

• PR: Add --preserve-plugin-branches flag to deploy-dev #568
• Purpose: Preserve plugin branches during development deployment
• Status: Pending
• Dependency: Needed for --preserve-plugin-branches flag used in testing

Note: These PRs should be merged before or alongside this cloud connector feature PR.

---

✅ What Works

1. Cloud Connector Deployment

• ✓ Cloud connector role created successfully
• ✓ Required packages present (rhc, yggdrasil-worker-forwarder) - installed as prerequisite
• ✓ Worker config file created: /etc/rhc/workers/foreman_rh_cloud.toml
• ✓ rhcd service running and enabled

Note: The cloud connector role does not install packages - it assumes rhc and yggdrasil-worker-forwarder are already present (installed manually or by prerequisite roles).

2. Service User Authentication

Created service user cloud_connector_user with proper credentials:

User ID: 5
Role: Cloud Connector
Auth Source: Internal
Last Login: Successfully authenticated via API

3. API Endpoint Verification

Manual test of /api/v2/rh_cloud/cloud_request endpoint:

Request:

curl -u cloud_connector_user:$PASSWORD \
  -X POST \
  http://localhost:3000/api/v2/rh_cloud/cloud_request \
  -H 'Content-Type: application/json' \
  -d '{"directive":"foreman_rh_cloud", ...}'

Result:

✓ Authenticated user cloud_connector_user against INTERNAL authentication source
✓ Authorized user cloud_connector_user(cloud_connector_user)
✓ Request processed (playbook URL validation occurred as expected)

4. Configuration Validation

• ✓ rhc_instance_id setting configured: 5fc35187-ee43-48a8-b61a-01c0153d0562
• ✓ Worker URL correct: http://localhost:3000/api/v2/rh_cloud/cloud_request
• ✓ Credentials properly stored in worker config

---

🔧 Required Code Changes

Found 3 necessary changes for development environment support:

Change 1: Development URL Configuration

File: development/playbooks/deploy-dev/deploy-dev.yaml

Issue: Development uses Rails dev server on port 3000, not containerized Foreman

Fix:

vars:
  cloud_connector_url: "http://localhost:3000"
  cloud_connector_admin_user: "{{ foreman_development_admin_user }}"
  cloud_connector_admin_password: "{{ foreman_development_admin_password }}"

Why: Points worker to the correct development Rails server endpoint

---

Change 2: Disable Certificate Validation for HTTP

File: src/roles/cloud_connector/tasks/main.yaml

Issue: Development uses HTTP (no SSL), certificate validation fails

Fix:

- name: Create cloud connector service user
  theforeman.foreman.user:
    # ... existing parameters ...
    validate_certs: false  # <-- Add this

Why: HTTP connections don't have valid certificates in development

---

Change 3: Make CA Certificate Path Optional

File: src/roles/cloud_connector/tasks/main.yaml

Issue: No CA certificate exists for HTTP connections, undefined variable error

Fix:

- name: Create cloud connector service user
  theforeman.foreman.user:
    # ... existing parameters ...
    ca_path: "{{ foreman_ca_certificate | default(omit) }}"  # <-- Change this

Why: Prevents Ansible error when foreman_ca_certificate is undefined (HTTP mode)

---

⚠️ Known Limitations

Automated pytest Suite Skipped

Issue: All 6 cloud connector tests were skipped:

tests/cloud_connector_test.py::test_rhc_package_installed SKIPPED
tests/cloud_connector_test.py::test_yggdrasil_worker_forwarder_package_installed SKIPPED
tests/cloud_connector_test.py::test_workers_directory_exists SKIPPED
tests/cloud_connector_test.py::test_worker_config_exists SKIPPED
tests/cloud_connector_test.py::test_worker_script_exists SKIPPED
tests/cloud_connector_test.py::test_rhcd_service_running SKIPPED

Root Cause:

• pytest hardcoded to check ./foremanctl features (production state)
• Development uses ./forge deploy-dev (separate state)
• Feature flags from --add-feature are temporary, not persisted

Impact:

• ❌ Automated test suite cannot verify development deployments
• ✅ Manual verification confirms all test criteria pass

---

Testing completed with: Claude Code

[jeremylenz]

Re: Change 1 - cloud_connector_url defaults to foreman_url which is https://{{ fqdn }},
and httpd proxies to the Rails server. The HTTPS URL through httpd works fine in dev — we tested it. Using http://localhost:3000 directly would bypass httpd and also break the ca_path on
all the FAM module calls. The worker TOML would also have an HTTP URL, which means the runtime cloud requests would go over unencrypted HTTP.

Change 2 is not necessary unless I were to do Change 1, which I think is not needed for the reason above.

Updating now with Change 3.

[jeremylenz]

Btw, Our role does install the packages — line 2-7 of tasks/main.yaml:

  - name: Install rhc and yggdrasil-worker-forwarder
    ansible.builtin.package:
      name:
        - rhc
        - yggdrasil-worker-forwarder

[jeremylenz]

We decided that foremanctl should not rely on the system CA trust store; rather, the applications consuming the certificate must be configured to point to the certificate directly. This means the consuming application (yggdrasil in this case) must be able to be configured in this way.

Converting to draft while we wait for that to be implemented in yggdrasil.

[ehelms]

You can use the has_feature filter.

[ehelms]

What does announce to sources do?

[jeremylenz]

It's an API call made to the cloud to identify this Foreman instance as a source in a service that was previously called Sources - I'm not sure what the name of the service is now. My understanding is that it's needed in order for cloud remediations to show your registered host as "connected"

[ehelms]

Why do we have to set this? Is the default value false?

[jeremylenz]

Setting up cloud connector has always required turning on this setting, and in the previous architecture there was a message in the UI to that effect. The default is true, but the user may have turned it off and then tried to enable cloud connector.

[jeremylenz]

cc @ShimShtein in case you can add more color here