Make yace embeddable in OTel Collectors by ArthurSens · Pull Request #1837 · prometheus-community/yet-another-cloudwatch-exporter

ArthurSens · 2026-04-06T20:41:45Z

This PR makes YACE embeddable in custom OTel Collector distributions through OCB. Implementing prometheus/proposals#69.

To facilitate this PR's review, I've opened prometheus/prometheus-opentelemetry-collector#4, which embeds YACE into a custom collector distribution. That repository's Makefile should contain everything needed to build and run the collector with YACE embedded.

tristanburgess · 2026-04-14T20:37:23Z

Thanks for submitting this! This is really cool. I'll plan on taking a look at this tomorrow for a thorough review.

tristanburgess · 2026-04-16T19:22:37Z

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package otelcollector


WDYT about adding a godoc comment here or in a separate doc.go file as described here to convey that this package is experimental and subject to changes?

Comments on package declarations should provide general package documentation. These comments can be short, like the sort package’s brief description:

// Package sort provides primitives for sorting slices and user-defined // collections. package sort

They can also be detailed like the gob package’s overview. That package uses another convention for packages that need large amounts of introductory documentation: the package comment is placed in its own file, doc.go, which contains only those comments and a package clause.

On that note anything we should add to the README or is it too early? https://github.com/prometheus-community/yet-another-cloudwatch-exporter?tab=readme-ov-file#yace---yet-another-cloudwatch-exporter

tristanburgess · 2026-04-16T19:43:45Z

+		return nil, err
+	}
+
+	interval, err := cfg.awsScrapeInterval()


How do you think this parameter should play with https://github.com/prometheus/opentelemetry-collector-bridge/blob/5e92765d01563b43d343bc0054c2bc65c6bb6e8e/config.go#L32?

Typically users expect would expect a default AWS scrape interval of 5 minutes/300s. Can we set the default for the AWS scrape interval for this as well?

What should happen if someone sets the AWS scrape interval to less than the configured bridge scrape interval? I would think that's not a desirable situation as there could be data loss and wasted AWS calls so perhaps we should validate against it?

tristanburgess · 2026-04-17T15:11:53Z

+	"go.opentelemetry.io/collector/receiver"
+)
+
+var receiverType = component.MustNewType("yace_exporter")


Wondering if you think the _exporter naming has a chance of confusing users with https://opentelemetry.io/docs/collector/components/exporter/? Should we do yace_receiver instead? Or perhaps just yace? Looking at https://opentelemetry.io/docs/collector/components/receiver/ it seems a lot of them are named in the config files as a single word.

tristanburgess · 2026-04-17T16:01:59Z

+
+	registry := prometheus.NewRegistry()
+	for _, metric := range exporter.Metrics {
+		if err := registry.Register(metric); err != nil {


I think we have to consider tightening the scope of the YACE-wide metrics that get collected about the run health

yet-another-cloudwatch-exporter/pkg/promutil/prometheus.go

Lines 26 to 87 in cbafcec

CloudwatchAPIErrorCounter = prometheus.NewCounterVec(prometheus.CounterOpts{

Name: "yace_cloudwatch_request_errors",

Help: "Help is not implemented yet.",

}, []string{"api_name"})

CloudwatchAPICounter = prometheus.NewCounterVec(prometheus.CounterOpts{

Name: "yace_cloudwatch_requests_total",

Help: "Number of calls made to the CloudWatch APIs",

}, []string{"api_name"})

CloudwatchGetMetricDataAPICounter = prometheus.NewCounter(prometheus.CounterOpts{

Name: "yace_cloudwatch_getmetricdata_requests_total",

Help: "DEPRECATED: replaced by yace_cloudwatch_requests_total with api_name label",

})

CloudwatchGetMetricDataAPIMetricsCounter = prometheus.NewCounter(prometheus.CounterOpts{

Name: "yace_cloudwatch_getmetricdata_metrics_requested_total",

Help: "Number of metrics requested from the CloudWatch GetMetricData API which is how AWS bills",

})

CloudwatchGetMetricStatisticsAPICounter = prometheus.NewCounter(prometheus.CounterOpts{

Name: "yace_cloudwatch_getmetricstatistics_requests_total",

Help: "DEPRECATED: replaced by yace_cloudwatch_requests_total with api_name label",

})

ResourceGroupTaggingAPICounter = prometheus.NewCounter(prometheus.CounterOpts{

Name: "yace_cloudwatch_resourcegrouptaggingapi_requests_total",

Help: "Help is not implemented yet.",

})

AutoScalingAPICounter = prometheus.NewCounter(prometheus.CounterOpts{

Name: "yace_cloudwatch_autoscalingapi_requests_total",

Help: "Help is not implemented yet.",

})

TargetGroupsAPICounter = prometheus.NewCounter(prometheus.CounterOpts{

Name: "yace_cloudwatch_targetgroupapi_requests_total",

Help: "Help is not implemented yet.",

})

APIGatewayAPICounter = prometheus.NewCounter(prometheus.CounterOpts{

Name: "yace_cloudwatch_apigatewayapi_requests_total",

})

APIGatewayAPIV2Counter = prometheus.NewCounter(prometheus.CounterOpts{

Name: "yace_cloudwatch_apigatewayapiv2_requests_total",

})

Ec2APICounter = prometheus.NewCounter(prometheus.CounterOpts{

Name: "yace_cloudwatch_ec2api_requests_total",

Help: "Help is not implemented yet.",

})

ShieldAPICounter = prometheus.NewCounter(prometheus.CounterOpts{

Name: "yace_cloudwatch_shieldapi_requests_total",

Help: "Help is not implemented yet.",

})

ManagedPrometheusAPICounter = prometheus.NewCounter(prometheus.CounterOpts{

Name: "yace_cloudwatch_managedprometheusapi_requests_total",

Help: "Help is not implemented yet.",

})

StoragegatewayAPICounter = prometheus.NewCounter(prometheus.CounterOpts{

Name: "yace_cloudwatch_storagegatewayapi_requests_total",

Help: "Help is not implemented yet.",

})

DmsAPICounter = prometheus.NewCounter(prometheus.CounterOpts{

Name: "yace_cloudwatch_dmsapi_requests_total",

Help: "Help is not implemented yet.",

})

DuplicateMetricsFilteredCounter = prometheus.NewCounter(prometheus.CounterOpts{

Name: "yace_cloudwatch_duplicate_metrics_filtered",

Help: "Help is not implemented yet.",

})

If I configure an OTel collector process with say

receivers: yace/prod: config_file: /etc/yace/prod.yml aws_scrape_interval: 300s yace/staging: config_file: /etc/yace/staging.yml aws_scrape_interval: 300s exporters: otlphttp: endpoint: https://otlp.example.com service: pipelines: metrics: receivers: [yace/prod, yace/staging] exporters: [otlphttp]

Then the resulting metrics e.g.

yet-another-cloudwatch-exporter/pkg/promutil/prometheus.go

Line 31 in cbafcec

Name: "yace_cloudwatch_requests_total",

are package-global counters. There would result two duplicate series of the sum of the two receivers counter values with no differentiation and thus I think not useful.

Oh, that's so interesting! This sounds like the classic trade-off between Push vs Pull. If we used separate exporters for prod/staging, the Prometheus scrape process would add the job/instance labels, and that would be enough to differentiate the two. Now that we're enabling push, the scrape-related labels are being lost!

I think we need to solve this in the bridge library, since this will be a problem to all exporters that are being made embeddable 🤔

I've opened prometheus/opentelemetry-collector-bridge#27

tristanburgess · 2026-04-17T16:48:59Z

+	if _, err := c.Options(); err != nil {
+		return err
+	}
+	_, err := c.jobsConfig(discardLogger())


I think this discards warning validation logs right? Such as this one. I think it makes sense to keep them for caller's FYI but open to thoughts.

Hmmmm, that's true... but all these functions are also being called during Start() and in Start() we are passing a real logger.

This made me think... why do we need to call those in Validate() and also during Start()? 🤔

tristanburgess · 2026-04-17T16:50:22Z

+func NewFactory() receiver.Factory {
+	return prombridge.NewFactory(
+		receiverType,
+		newLifecycleManager(slog.Default()),


What actually sets the default logger in this case? Like I want to configure debug level YACE logging in the OTel receiver config, how does that look and does this actually build the receiver with the logger hooked back into the collector?

ArthurSens · 2026-04-17T19:36:44Z

+    groups:
+      aws-sdk-v2:
+        patterns:
+          - "github.com/aws/aws-sdk-go-v2*"


Not sure if this applies in this module as well 🤔

yeah AFAIK it just groups related updates together into one PR, instead of opening a PR for each individual package. Since go.mod includes many github.com/aws/aws-sdk-go-v2 packages it will help

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

ArthurSens · 2026-04-23T18:39:24Z

prometheus/opentelemetry-collector-bridge#28 was merged to address the log wiring and fc15644 implements it

SuperQ · 2026-04-29T16:10:14Z

@@ -0,0 +1,100 @@
+module github.com/prometheus-community/yet-another-cloudwatch-exporter/otelcollector


Please remove this package. There is no good reason to have to maintain multiple Go modules within Prometheus repos.

ArthurSens · 2026-05-07T19:26:14Z

I'm closing this PR since we have agreed on keeping the new module outside of this repo :)

ArthurSens mentioned this pull request Apr 6, 2026

Add YACE exporter prometheus/prometheus-opentelemetry-collector#4

Closed

tristanburgess self-requested a review April 14, 2026 18:30

tristanburgess reviewed Apr 14, 2026

View reviewed changes

Comment thread otelcollector/testdata/config.yml Outdated

tristanburgess reviewed Apr 16, 2026

View reviewed changes

Comment thread pkg/exporter.go

tristanburgess reviewed Apr 16, 2026

View reviewed changes

Comment thread pkg/runtime_config.go Outdated

tristanburgess reviewed Apr 16, 2026

View reviewed changes

Comment thread otelcollector/go.mod

tristanburgess reviewed Apr 16, 2026

View reviewed changes

tristanburgess reviewed Apr 17, 2026

View reviewed changes

Comment thread pkg/exporter.go

tristanburgess reviewed Apr 17, 2026

View reviewed changes

Comment thread otelcollector/factory.go Outdated

tristanburgess reviewed Apr 17, 2026

View reviewed changes

Comment thread otelcollector/lifecycle.go

tristanburgess reviewed Apr 17, 2026

View reviewed changes

Comment thread otelcollector/config.go Outdated

ArthurSens force-pushed the embed-in-collector branch from d322e29 to 2dca175 Compare April 17, 2026 18:45

ArthurSens commented Apr 17, 2026

View reviewed changes

ArthurSens added 11 commits April 23, 2026 15:28

Make yace embeddable in OTel Collectors

97bd597

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

Continue using config file path instead of embedding config

0a89ed2

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

Fix config parsing

e1c4cc4

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

Return error when updating metrics failed

df6530b

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

Add changelog entry

8c3a950

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

Add dependabot config for the new module

b3f9240

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

Add package documentation

19e5414

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

Update embedding.md

a8334bb

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

Update code comment

8491888

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

Re-use existing testdata config file

799bcb8

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

Use mapstructure Decoder to decode config

2b6c698

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

ArthurSens added 4 commits April 23, 2026 15:29

Ensure lifecycle is separate when running multiple instances at once

823cad8

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

Properly wire collector logger to exporter logger

fc15644

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

go mod tidy after rebase

e13e1d9

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

Remove year from copyright headers

8836c86

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

ArthurSens force-pushed the embed-in-collector branch from 4b30e91 to 8836c86 Compare April 23, 2026 18:37

ArthurSens marked this pull request as ready for review April 23, 2026 18:37

SuperQ requested changes Apr 29, 2026

View reviewed changes

ArthurSens mentioned this pull request May 5, 2026

Make Config and Prometheus Registry generation usable as standalone packages #1857

Merged

ArthurSens closed this May 7, 2026

	CloudwatchAPIErrorCounter = prometheus.NewCounterVec(prometheus.CounterOpts{
	Name: "yace_cloudwatch_request_errors",
	Help: "Help is not implemented yet.",
	}, []string{"api_name"})
	CloudwatchAPICounter = prometheus.NewCounterVec(prometheus.CounterOpts{
	Name: "yace_cloudwatch_requests_total",
	Help: "Number of calls made to the CloudWatch APIs",
	}, []string{"api_name"})
	CloudwatchGetMetricDataAPICounter = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "yace_cloudwatch_getmetricdata_requests_total",
	Help: "DEPRECATED: replaced by yace_cloudwatch_requests_total with api_name label",
	})
	CloudwatchGetMetricDataAPIMetricsCounter = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "yace_cloudwatch_getmetricdata_metrics_requested_total",
	Help: "Number of metrics requested from the CloudWatch GetMetricData API which is how AWS bills",
	})
	CloudwatchGetMetricStatisticsAPICounter = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "yace_cloudwatch_getmetricstatistics_requests_total",
	Help: "DEPRECATED: replaced by yace_cloudwatch_requests_total with api_name label",
	})
	ResourceGroupTaggingAPICounter = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "yace_cloudwatch_resourcegrouptaggingapi_requests_total",
	Help: "Help is not implemented yet.",
	})
	AutoScalingAPICounter = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "yace_cloudwatch_autoscalingapi_requests_total",
	Help: "Help is not implemented yet.",
	})
	TargetGroupsAPICounter = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "yace_cloudwatch_targetgroupapi_requests_total",
	Help: "Help is not implemented yet.",
	})
	APIGatewayAPICounter = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "yace_cloudwatch_apigatewayapi_requests_total",
	})
	APIGatewayAPIV2Counter = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "yace_cloudwatch_apigatewayapiv2_requests_total",
	})
	Ec2APICounter = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "yace_cloudwatch_ec2api_requests_total",
	Help: "Help is not implemented yet.",
	})
	ShieldAPICounter = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "yace_cloudwatch_shieldapi_requests_total",
	Help: "Help is not implemented yet.",
	})
	ManagedPrometheusAPICounter = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "yace_cloudwatch_managedprometheusapi_requests_total",
	Help: "Help is not implemented yet.",
	})
	StoragegatewayAPICounter = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "yace_cloudwatch_storagegatewayapi_requests_total",
	Help: "Help is not implemented yet.",
	})
	DmsAPICounter = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "yace_cloudwatch_dmsapi_requests_total",
	Help: "Help is not implemented yet.",
	})
	DuplicateMetricsFilteredCounter = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "yace_cloudwatch_duplicate_metrics_filtered",
	Help: "Help is not implemented yet.",
	})

		@@ -0,0 +1,100 @@
		module github.com/prometheus-community/yet-another-cloudwatch-exporter/otelcollector

Uh oh!

Uh oh!

Conversation

ArthurSens commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

tristanburgess commented Apr 14, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tristanburgess Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tristanburgess Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArthurSens commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArthurSens commented May 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ArthurSens commented Apr 6, 2026 •

edited

Loading

tristanburgess Apr 17, 2026 •

edited

Loading

tristanburgess Apr 17, 2026 •

edited

Loading

ArthurSens commented Apr 23, 2026 •

edited

Loading