PagerDuty Operator

• PagerDuty Operator
  • About
  • How the PagerDuty Operator works
  • Development
    • Set up local openshift cluster
    • Deploy dependencies
    • Option 1: Run pagerduty-operator outside of OpenShift
    • Option 2: Run local built operator in CRC
      • Generate secret with quay.io creds
      • Deploy pagerduty-operator from custom repo
    • Create PagerDutyIntegration
    • Create ClusterDeployment
    • Delete ClusterDeployment
  • Testing
    • Unit tests
    • PKO template tests
    • Regenerating test fixtures

About

The PagerDuty operator is used to automate integrating OpenShift Dedicated clusters with PagerDuty that are provisioned via https://cloud.redhat.com/.

This operator runs on Hive and watches for new cluster deployments. Hive is an API driven OpenShift cluster providing OpenShift Dedicated provisioning and management.

How the PagerDuty Operator works

• The PagerDutyIntegration controller watches for changes to PagerDutyIntegration CRs, and also for changes to appropriately labeled ClusterDeployment CRs (and ConfigMap/Secret/SyncSet resources owned by such a ClusterDeployment).
• For each PagerDutyIntegration CR, it will get a list of matching ClusterDeployments that have the spec.installed field set to true.
• For each of these ClusterDeployments, PagerDuty creates a secret which contains the integration key required to communicate with PagerDuty Web application.
• The PagerDuty operator then creates syncset with the relevant information for hive to send the PagerDuty secret to the newly provisioned cluster.
• This syncset is used by hive to deploy the pagerduty secret to the provisioned cluster so that the relevant SRE team get notified of alerts on the cluster.
• The pagerduty secret is deployed to the coordinates specified in the spec.targetSecretRef field of the PagerDutyIntegration CR.

Development

Set up local OpenShift cluster

For example install crc as described in its documentation.

Deploy dependencies

Create hive CRDs. To do so, clone hive repo and run

oc apply -f config/crds

Deploy namespace, role, etc from pagerduty-operator

oc apply -f manifests/01-namespace.yaml
oc apply -f manifests/02-role.yaml
oc apply -f manifests/03-service_account.yaml
oc apply -f manifests/04-role_binding.yaml
oc apply -f deploy/crds/pagerduty.openshift.io_pagerdutyintegrations.yaml

Create secret with pagerduty api key, for example using a trial account. You can then create an API key at https://{your-account}.pagerduty.com/api_keys.

Following is an example secret to adjust and apply with oc apply -f ${FILENAME}.

apiVersion: v1
data:
  PAGERDUTY_API_KEY: bXktYXBpLWtleQ== #echo -n ${PAGERDUTY_API_KEY} | base64
kind: Secret
metadata:
  name: pagerduty-api-key
  namespace: pagerduty-operator
type: Opaque

Option 1: Run pagerduty-operator outside of OpenShift

export OPERATOR_NAME=pagerduty-operator
go run main.go

In another terminal create namespace pagerduty-operator if needed:

oc create namespace pagerduty-operator

Continue to Create PagerDutyIntegration.

Option 2: Run local built operator in CRC

Build local code modifications and push image to your own quay.io account.

$ ALLOW_DIRTY_CHECKOUT=true IMAGE_REPOSITORY=${USER_ID} make docker-build
[...]
Successfully tagged quay.io/${USER_ID}/pagerduty-operator:latest

$ podman login quay.io
$ podman push quay.io/${USER_ID}/pagerduty-operator:latest

Generate secret with quay.io creds

• visit account page https://quay.io/user/{userid}?tab=settings
• click generate encrypted password
• Re-enter password
• download ${USER_ID}-secret.yml
• Deploy quay.io secret

oc project pagerduty-operator
oc apply -f ~/Downloads/${USER_ID}-secret.yml -n pagerduty-operator

Deploy pagerduty-operator from custom repo

Create a copy of manifests/05-operator.yaml and modify it to use your image from quay.io

...
      imagePullSecrets:
        - name: ${USER_ID}-pull-secret
      containers:
        - name: pagerduty-operator
          image: quay.io/${USER_ID}/pagerduty-operator
...

Deploy modified operator manifest

oc apply -f path/to/modified/operator.yaml

Note: In some cases, the pagerduty-operator pod in the pagerduty-operator namespace doesn't start with the following error:

Warning  FailedScheduling  3m5s  default-scheduler  0/1 nodes are available:
1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption
victims found for incoming pod.

To remedy this, lower the requested resources in the manifests/05-operator.yaml deployment (e.g. lower memory from 2G to 0.5G).

Create the PagerDutyIntegration object

There's an example at deploy-extras/pagerduty_v1alpha1_pagerdutyintegration_cr.yaml that you can edit and apply to your cluster.

You'll need to use a valid escalation policy ID from your PagerDuty account. You can get this by clicking on your policy at https://{your-account}.pagerduty.com/escalation_policies#. The ID will be visible in the URL after the # character.

Create the ClusterDeployment object

pagerduty-operator doesn't start reconciling clusters until spec.installed is set to true.

You can create a dummy ClusterDeployment by copying a real one from an active hive

real-hive$ oc get cd -n ${NAMESPACE} ${CD_NAME} -o yaml > /tmp/fake-clusterdeployment.yaml

...

$ oc create namespace ${NAMESPACE}
$ oc apply -f /tmp/fake-clusterdeployment.yaml

Perform the following modifications:

oc edit clusterdeployment ${CD_NAME} -n ${NAMESPACE}

• Add the following finalizer (i.e. metadata.finalizers):

  - pd.managed.openshift.io/example-pagerdutyintegration

  (the suffix comes from the PagerDutyIntegration name)

• Add the following label:

  api.openshift.com/test: "true"

  (this is used by the ClusterDeployment selector in the default PagerDutyIntegration)

• Set spec.installed to true.

Delete the ClusterDeployment object

To trigger pagerduty-operator to remove the service in pagerduty, delete the clusterdeployment.

oc delete clusterdeployment fake-cluster -n fake-cluster-namespace

You may need to remove dangling finalizers from the clusterdeployment object.

oc edit clusterdeployment fake-cluster -n fake-cluster-namespace

Testing

Unit tests

Run the full test suite:

make test

This runs all Go unit tests including controller tests, utility tests, and PKO template tests.

PKO template tests

The operator is deployed via Package Operator (PKO). Deployment manifests live in deploy_pko/ as Go templates (.gotmpl files) that PKO renders using config values from the ClusterPackage.

Template tests validate that all .gotmpl files render correctly with different configurations. There are two layers of testing:

Go unit tests (pkg/pko/template_test.go):

Structural validation of rendered templates — verifies correct kind, metadata, phase annotations, config substitution (image, fedramp, escalation policies), array rendering via toJson, JSON blob quoting, and that osd-silent/osd-scale-test PDIs correctly include or omit fields relative to the main osd PDI.

go test ./pkg/pko/... -v

PKO snapshot tests (deploy_pko/manifest.yaml test.template section):

The manifest.yaml defines test contexts with different config values (production, integration, empty arrays, fedramp, orchestration-disabled). kubectl-package validate renders all templates with each context and compares the output against snapshot fixtures in deploy_pko/.test-fixtures/.

kubectl-package validate deploy_pko/

If the output doesn't match the fixtures, validation fails — this catches unintended changes to template rendering.

Regenerating test fixtures

After intentionally modifying a .gotmpl file or the manifest.yaml config schema, the snapshot fixtures need to be regenerated:

rm -rf deploy_pko/.test-fixtures
kubectl-package validate deploy_pko/

This regenerates the fixture directories. Review the changes with git diff to confirm only the expected fields changed, then commit the updated fixtures alongside the template changes.

If kubectl-package is not installed, download it from the package-operator releases.