DPC-5482: Datadog dashboard, monitors and synthetic tests by jscott-nava · Pull Request #3022 · CMSgov/dpc-app

jscott-nava · 2026-06-03T21:28:39Z

🎫 Ticket

https://jira.cms.gov/browse/DPC-5482
https://jira.cms.gov/browse/DPC-5252
https://jira.cms.gov/browse/DPC-5507

🛠 Changes

This PR adds the following:

Datadog dashboard and monitor modules and initial configurations.
Github actions workflows for the new modules, and an update to the pre-existing SOPS workflow (for the sake of consistency but also to be compatible with the shared root.tofu.tf file.
Updates to the shared root.tofu.tf file to improve the handling of the env variable.
Synthetic tests that mirror what exists in New Relic.

ℹ️ Context

The addition of the Datadog dashboard, monitors and synthetic tests are tasks in the greater epic of migrating from New Relic to Datadog.

🧪 Validation

A sample of relevant workflow runs is as follows:

https://github.com/CMSgov/dpc-app/actions/runs/27039018299 (dashboard apply)
https://github.com/CMSgov/dpc-app/actions/runs/27038929251 (dashboard plan)
https://github.com/CMSgov/dpc-app/actions/runs/27043646951/ (sops sandbox plan)
https://github.com/CMSgov/dpc-app/actions/runs/27043453954 (sops test apply)
https://github.com/CMSgov/dpc-app/actions/runs/27043257131 (sops test plan)
https://github.com/CMSgov/dpc-app/actions/runs/27031155558 (monitors apply prod)

jscott-nava · 2026-06-04T16:45:54Z

Moving back to draft to add monitors code to this PR.

lukey-luke · 2026-06-09T18:52:19Z

@@ -0,0 +1,66 @@
+name: tf-00-datadog-dashboard
+run-name: tf-00-datadog-dashboard


Looks like BCDA uses -40- for datadog
(e.g. 40-datadog-monitors and 40-datadog-dashboards)

What is the advantage of using this "##-xyz" format? Should we be consistent across teams?

My understanding is that the prefixes are intended to create a sort of dependency tree so that it is clear in which order each TF module should be applied. In AB2D, for example, 10-core contains all IAM resources, and therefore clearly should be applied prior to 30-api which depends on those resources.

There is not currently a strict standard shared between repos (i.e. BCDA uses 40 for Datadog but AB2D uses 60 and 65), but since Datadog does not depend on anything else and is a team wide resource (as opposed to account or environment wide) it seems as though 00 is as good a place as any. (These numbers can also adjusted very easily in the future as the larger Tofu refactoring effort progresses.)

I might make sense to follow BCDA and AB2D and give it a higher rating, since wouldnt the dashboard not depend on our services to be up and running to accurately report data?

@Jose-verdance Sure, let me know what number makes sense and I'll make that change.

I am ok with 40s for the datadog and maybe call it observability.

lukey-luke · 2026-06-09T19:14:56Z

@@ -0,0 +1,106 @@
+name: tf-10-config


👍

We still need to figure out a strategy for handling free-floating parameters in upper environments.
As a long-term solution, making this part of the Build and deploy workflow, would avoid the need to perform extra steps for updating SSM values

I drafted a new ticket to address this: https://jira.cms.gov/browse/DPC-5519

This is relevant to multi-csp work as we'll want to use the sops module for managing things like CLEAR secret key

As discussed, we will sidebar on this with the team to figure out the SOPS deployment strategy. As far as the scope of this PR is concerned however, the SOPS-related changes are limited to fixes to allow it to continue to work with changes made to the shared root.tofu.tf file which were required by the dashboard and monitors modules.

lukey-luke · 2026-06-09T19:16:42Z

+  lambda: true
+  s3: true
+  rds: true
+lambda:


values are the same across dev/test/sandbox/prod.yml, so we could probably include these as part of defaults.yml

This code was borrowed from the AB2D implementation of the monitors module and seems like a reasonable starting point for the monitors deployment with the expectation that the environment specific values will likely change with ongoing Datadog tasks as we realize that all monitors do not need to be in each environment.

Also, while all configuration could be contained in the defaults.yml file for this PR, leaving this current implementation helps to clarify how each environment can be independently configured.

lukey-luke · 2026-06-09T19:18:13Z

+     See https://terraform-docs.io/user-guide/configuration/ for more information.
+-->
+## Outputs
+


Could you provide context where I can find the new datadog dashboard?

@lukey-luke You should have received an email from Datadog HQ on (or around) May 26th with details on how to register and log in to the CMS Datadog instance, and from there you can browse all the DASG dashboards and monitors. Please reach out to Bishoy if you have not received that email.

Thanks!
For other folks reviewing, DPC Metrics Dashboard can be found HERE

Jose-verdance · 2026-06-10T13:47:32Z

+    permissions:
+      contents: read
+      id-token: write
+    runs-on: codebuild-dpc-app-non-prod-${{ github.run_id }}-${{ github.run_attempt }}


Hey @jscott-nava, why is this run with the non prod version? Will there be a separate dashboard for prod added later?

@Jose-verdance There is one dashboard per team (not account), and that dashboard covers all environments.

Jose-verdance · 2026-06-10T13:49:23Z

+  env         = var.env
+  root_module = "https://github.com/CMSgov/bcda-app/tree/main/ops/services/10-sops"


Why is this bcda-app and not cdap or dpc-ops?

Jose-verdance · 2026-06-10T13:51:29Z

@@ -0,0 +1,5 @@
+variable "env" {


If this is restricted to test then where are the sandbox and prod monitor metrics connected to?

jscott-nava · 2026-06-12T18:33:42Z

Converting this PR back to draft to add Datadog synthetic tests (DPC-5507).

jscott-nava force-pushed the jscott/DPC-5482 branch from b33f6a3 to 6c34699 Compare June 3, 2026 21:29

jscott-nava marked this pull request as ready for review June 3, 2026 22:12

jscott-nava requested a review from a team as a code owner June 3, 2026 22:12

jscott-nava marked this pull request as draft June 4, 2026 16:45

jscott-nava force-pushed the jscott/DPC-5482 branch from 2b28cdc to b1eeba3 Compare June 5, 2026 22:48

jscott-nava marked this pull request as ready for review June 5, 2026 22:50

jscott-nava changed the title ~~DPC-5482: Datadog dashboard~~ DPC-5482: Datadog dashboard and monitors Jun 9, 2026

lukey-luke reviewed Jun 9, 2026

View reviewed changes

jscott-nava requested a review from lukey-luke June 9, 2026 22:54

Jose-verdance reviewed Jun 10, 2026

View reviewed changes

jscott-nava marked this pull request as draft June 12, 2026 18:32

jscott-nava added 15 commits June 12, 2026 11:35

Adding Datadog dashboard TF and workflow.

fc11e5a

Adding temporary push trigger.

acb8fc6

Updating runons for DD dashboard workflow.

c8cb082

Fixing GH secret name.

f556c51

Refactoring root.tofu env value.

8d5f110

Various DD dashboard workflow updates.

ba7b75d

DD dashboard workflow fixes.

c157595

Adding terraform-docs.yml and DD dashboard README.

44f1c1c

More DD dashboard workflow fixes.

a10d09e

Fixing assume role request arn in DD dashboard workflow.

d150eab

Cleaning up workflow branch ref.

a9103d2

Renaming workflow file.

648f7b4

Workflow updates.

c7f12fd

Workflow update.

9b212c1

Workflow update

1393fcb

jscott-nava added 28 commits June 12, 2026 11:35

Workflow updates.

37d5e6e

Workflow updates.

616befb

Workflow update.

b730f9b

Adding Datadog monitor configuration.

52d5836

Adding Datadog monitors and workflow

cc639cf

Adding branch refs

acff9a2

Renaming workflows

cf8a20e

Renaming workflows

6c2b6d7

Renaming workflow:

b0a5a83

Renaming workflow file

beeba66

Unquoting branch ref

b1fad42

Adding pull_request trigger

2e92569

Adding pull_request trigger temporarily

5816196

Removing temporary pull_request triggers

dd077bf

Removing env var defaults

fbb62c2

Adding env to tf commands

1cab7d5

Removing obsolete code block

cadf14f

DD dashboard runbook URL update

0731995

Adding DD monitors README

eaecc27

Updating SOPS workflow

3c99799

Config workflow fix

41f3809

Adding setup steps for sops and yq to config workflow

e1b14e2

Renaming 10-config to 10-sops for clarity

cf2a779

Renaming TF dir from 10-config to 10-sops for clarity

6a518b9

Reverting 10-config name change due to state mismatch

0ee7525

Adding temporary pull_request trigger

6a8e2df

Workflow step name fix

ee8409a

Adding Datadog synthetic tests.

9ac3ff9

jscott-nava force-pushed the jscott/DPC-5482 branch from 15e819f to 9ac3ff9 Compare June 12, 2026 18:35

jscott-nava changed the title ~~DPC-5482: Datadog dashboard and monitors~~ DPC-5482: Datadog dashboard, monitors and synthetic tests Jun 12, 2026

		@@ -0,0 +1,66 @@
		name: tf-00-datadog-dashboard
		run-name: tf-00-datadog-dashboard

		env = var.env
		root_module = "https://github.com/CMSgov/bcda-app/tree/main/ops/services/10-sops"

Conversation

jscott-nava commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎫 Ticket

🛠 Changes

ℹ️ Context

🧪 Validation

Uh oh!

jscott-nava commented Jun 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jscott-nava commented Jun 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jscott-nava commented Jun 3, 2026 •

edited

Loading