diff --git a/src/content/docs/guides/analytics/collect-metrics.mdx b/src/content/docs/guides/analytics/collect-metrics.mdx index 24bf654f0..a24e94b95 100644 --- a/src/content/docs/guides/analytics/collect-metrics.mdx +++ b/src/content/docs/guides/analytics/collect-metrics.mdx @@ -31,7 +31,7 @@ the snapshot of all historical metrics. ## Summarize metrics You can [transform](/guides/transformation/filter-and-select-data) metrics like -ordinary data, e.g., write aggregations over metrics to compute runtime +ordinary data, for example, write aggregations over metrics to compute runtime statistics suitable for reporting or dashboarding: ```tql @@ -41,8 +41,8 @@ summarize runtime=sum(duration), pipeline_id sort -runtime ``` -The above example computes the total runtime over all pipelines grouped by their -unique ID. +The previous example computes the total runtime over all pipelines grouped by +their unique ID. ## Inspect pipeline throughput @@ -57,3 +57,16 @@ sort -egress_events The `events` fields count records that passed through the pipeline during the metric period. The `bytes` fields approximate the amount of data transferred. + +## Send metrics to Prometheus + +Use `shape="prometheus"` with metrics and to_prometheus to +send live metrics to a Prometheus-compatible Remote Write receiver: + +```tql +metrics live=true, shape="prometheus" +to_prometheus "https://prometheus.example/api/v1/write" +``` + +The prometheus integration page covers endpoint +setup, authentication headers, and additional examples. diff --git a/src/content/docs/integrations/index.mdx b/src/content/docs/integrations/index.mdx index 9e3899017..e3ac55493 100644 --- a/src/content/docs/integrations/index.mdx +++ b/src/content/docs/integrations/index.mdx @@ -33,8 +33,9 @@ communication over numerous protocols and APIs: - **Message queues**: kafka, nats, amazon/sqs, amqp -- **Databases**: snowflake, - clickhouse, mysql +- **Databases and analytics**: snowflake, + clickhouse, mysql, + prometheus - **Network protocols**: tcp, udp, http, syslog diff --git a/src/content/docs/integrations/prometheus.mdx b/src/content/docs/integrations/prometheus.mdx new file mode 100644 index 000000000..0da0bb290 --- /dev/null +++ b/src/content/docs/integrations/prometheus.mdx @@ -0,0 +1,67 @@ +--- +title: Prometheus +--- + +[Prometheus](https://prometheus.io/) is an open-source monitoring system and +time-series database. Tenzir can send metric events to Prometheus-compatible +Remote Write receivers, including Prometheus, Grafana Mimir, Cortex, Thanos +Receive, and VictoriaMetrics. + +Use metrics with `shape="prometheus"` and to_prometheus to +send Tenzir node metrics to a Prometheus Remote Write endpoint. This integration +doesn't expose a scrape endpoint for the Prometheus pull model. + +## Prerequisites + +Before you send metrics, configure your receiver to accept Remote Write traffic. +For Prometheus, start the server with the Remote Write receiver enabled and use +the `/api/v1/write` endpoint. + +Store endpoint URLs and authentication headers as +[secrets](/explanations/secrets) when they contain credentials. + +## Examples + +### Send canonical metric events + +Send events that already use the default metric shape expected by +to_prometheus: + +```tql +from { + metric: "http_requests_total", + value: 42, + timestamp: 2026-05-15T10:00:00Z, + labels: { + method: "GET", + status: 200, + }, +} +to_prometheus "https://prometheus.example/api/v1/write" +``` + +### Export Tenzir node metrics + +Convert live Tenzir node metrics to Prometheus samples: + +```tql +metrics live=true, shape="prometheus" +to_prometheus "https://prometheus.example/api/v1/write" +``` + +### Add authentication headers + +Pass additional HTTP headers when your receiver requires authentication. Header +values are resolved as secrets. + +```tql +metrics live=true, shape="prometheus" +to_prometheus "prometheus-remote-write-url", + headers={Authorization: "prometheus-remote-write-token"} +``` + +## See Also + +- metrics +- to_prometheus +- analytics/collect-metrics diff --git a/src/content/docs/reference/operators.mdx b/src/content/docs/reference/operators.mdx index 22af14ac1..a65c5938b 100644 --- a/src/content/docs/reference/operators.mdx +++ b/src/content/docs/reference/operators.mdx @@ -723,6 +723,10 @@ operators: description: 'Sends events to an OpenSearch-compatible Bulk API.' example: 'to_opensearch "localhost:9200", …' path: 'reference/operators/to_opensearch' + - name: 'to_prometheus' + description: 'Sends metric events to a Prometheus Remote Write receiver.' + example: 'to_prometheus "https://prometheus.example/api/v1/write"' + path: 'reference/operators/to_prometheus' - name: 'to_s3' description: 'Writes events to one or multiple objects in Amazon S3.' example: 'to_s3 "s3://my-bucket/data/{uuid}.json" { write_ndjson }' @@ -2050,6 +2054,14 @@ to_opensearch "localhost:9200", … + + +```tql +to_prometheus "https://prometheus.example/api/v1/write" +``` + + + ```tql diff --git a/src/content/docs/reference/operators/metrics.mdx b/src/content/docs/reference/operators/metrics.mdx index 8211c2518..d6b779d2c 100644 --- a/src/content/docs/reference/operators/metrics.mdx +++ b/src/content/docs/reference/operators/metrics.mdx @@ -69,7 +69,8 @@ Controls the output shape. The default is `"raw"`, which preserves the native `tenzir.metrics.*` schemas listed below. Set `shape="prometheus"` to transform metrics into canonical records that are -compatible with Prometheus-oriented pipelines: +compatible with Prometheus-oriented pipelines. You can send this shape directly +to to_prometheus. ```tql { @@ -725,6 +726,13 @@ select timestamp, used_bytes {timestamp: 2023-12-21T12:55:32.916200, used_bytes: 461842751488} ``` +### Send metrics to Prometheus + +```tql +metrics live=true, shape="prometheus" +to_prometheus "https://prometheus.example/api/v1/write" +``` + ### Get the memory usage over time ```tql @@ -782,4 +790,7 @@ select timestamp, port, handle, reads, bytes ## See Also - diagnostics +- to_prometheus +- analytics/collect-metrics - plot-data-with-charts +- prometheus diff --git a/src/content/docs/reference/operators/to_prometheus.mdx b/src/content/docs/reference/operators/to_prometheus.mdx new file mode 100644 index 000000000..227e18a27 --- /dev/null +++ b/src/content/docs/reference/operators/to_prometheus.mdx @@ -0,0 +1,277 @@ +--- +title: to_prometheus +category: Outputs/Events +example: 'to_prometheus "https://prometheus.example/api/v1/write"' +--- + +import Op from '@components/see-also/Op.astro'; +import TLSOptions from '@partials/operators/TLSOptions.mdx'; + +Sends metric events to a Prometheus Remote Write receiver. + +```tql +to_prometheus url:string, [protobuf_message=string, name=string, + value=number, timestamp=time, labels=record, type=string, help=string, + unit=string, family=string, start_timestamp=time, headers=record, + tls=record, timeout=duration, connection_timeout=duration, + max_retry_count=int, retry_delay=duration, flush_interval=duration, + max_samples_per_request=int, max_uncompressed_bytes=int, + sanitize_names=bool] +``` + +## Description + +The `to_prometheus` operator converts incoming events to Prometheus Remote +Write requests and sends them with HTTP `POST`. The request body is a protobuf +message compressed with Snappy. + +By default, the operator sends Remote Write v1 requests with the +`prometheus.WriteRequest` protobuf message. Set +`protobuf_message="io.prometheus.write.v2.Request"` to send Remote Write v2 +requests. + +The default input shape is: + +```tql +{ + metric: "http_requests_total", + value: 42, + timestamp: 2026-05-15T10:00:00Z, + labels: { + method: "GET", + status: 200, + }, + type: "counter", + help: "HTTP requests", + unit: "requests", + family: "http_requests", + start_timestamp: 2026-05-15T09:00:00Z, +} +``` + +You can use expression options such as `name=metric_name` or +`labels={tenant: tenant_id}` to send existing schemas without reshaping them +first. + +The metrics operator can produce the default input shape with +`shape="prometheus"`. + +The operator writes the metric-name expression as the reserved Prometheus +`__name__` label. The expression defaults to the `metric` field. Providing +`labels.__name__` is invalid. Samples with the same label set are batched into +one time series per request, and samples in each time series are sorted by +timestamp before serialization. + +### `url: string` + +The Remote Write endpoint URL. + +The URL is resolved as a [secret](/explanations/secrets), so you can pass a +secret name to avoid hardcoding sensitive URLs. + +### `protobuf_message = string (optional)` + +The protobuf message to send. + +Supported values: + +- `prometheus.WriteRequest` +- `io.prometheus.write.v2.Request` + +Defaults to `prometheus.WriteRequest`. + +### `name = string (optional)` + +Expression for the metric name. + +Defaults to `metric`. + +### `value = number (optional)` + +Expression for the sample value. + +Defaults to `value`. + +### `timestamp = time (optional)` + +Expression for the sample timestamp. + +If the expression evaluates to `null` or the field is absent, Tenzir uses the +processing time. + +Defaults to `timestamp`. + +### `labels = record (optional)` + +Expression for metric labels. + +The record field names become label names. Label values are converted to +strings. Empty label names, empty label values, duplicate label names, and +`__name__` labels are invalid. + +Defaults to `labels`. If the field is absent, the operator sends no custom +labels. + +### `type = string (optional)` + +Expression for Prometheus metric metadata type. + +Supported values are `counter`, `gauge`, `histogram`, `gaugehistogram`, +`summary`, `info`, and `stateset`. + +Defaults to `type`. + +### `help = string (optional)` + +Expression for the metric help text. + +Defaults to `help`. + +### `unit = string (optional)` + +Expression for the metric unit. + +Defaults to `unit`. + +### `family = string (optional)` + +Expression for the metric family name. + +If the expression evaluates to `null` or the field is absent, Tenzir uses the +final metric name. + +Defaults to `family`. + +### `start_timestamp = time (optional)` + +Expression for the Remote Write v2 sample start timestamp. + +This field is only sent for Remote Write v2 requests. Defaults to +`start_timestamp`. + +### `headers = record (optional)` + +Additional HTTP headers. Header values are resolved as +[secrets](/explanations/secrets). + +The following Remote Write headers are reserved and cannot be overridden: + +- `Content-Encoding` +- `Content-Length` +- `Content-Type` +- `User-Agent` +- `X-Prometheus-Remote-Write-Version` + +### `timeout = duration (optional)` + +Timeout for each HTTP request. + +Defaults to `30s`. + +### `connection_timeout = duration (optional)` + +Timeout for establishing the connection. + +Defaults to `5s`. + +### `max_retry_count = int (optional)` + +Maximum number of retry attempts per request. + +A request is retried on transient transport failures and HTTP `429` and `5xx` +responses. + +Defaults to `5`. + +### `retry_delay = duration (optional)` + +Base duration between retry attempts. + +Tenzir uses exponential backoff starting at `retry_delay` and capping at +`16 * retry_delay`. A `Retry-After` response header overrides this delay. + +Defaults to `1s`. + + + +### `flush_interval = duration (optional)` + +Maximum amount of time to buffer samples before sending a request. + +Defaults to `1s`. + +### `max_samples_per_request = int (optional)` + +Maximum number of samples to include in one request. + +Defaults to `2000`. + +### `max_uncompressed_bytes = int (optional)` + +Maximum uncompressed protobuf request size in bytes. + +Defaults to `4Mi`. + +### `sanitize_names = bool (optional)` + +Whether to replace invalid metric-name and label-name characters with `_`. + +Defaults to `true`. + +## Examples + +### Send canonical metric events + +```tql +from { + metric: "http_requests_total", + value: 42, + timestamp: 2026-05-15T10:00:00Z, + labels: { + method: "GET", + status: 200, + }, +} +to_prometheus "https://prometheus.example/api/v1/write" +``` + +### Map an existing schema + +```tql +from { + metric_name: "cpu_usage", + usage: 0.82, + host: "sensor-a", +} +to_prometheus "https://prometheus.example/api/v1/write", + name=metric_name, + value=usage, + labels={host: host} +``` + +### Send Remote Write v2 metadata + +```tql +from { + metric: "requests_total", + value: 100, + type: "counter", + help: "Total requests", + unit: "requests", +} +to_prometheus "https://prometheus.example/api/v1/write", + protobuf_message="io.prometheus.write.v2.Request" +``` + +## Limitations + +The operator currently supports scalar samples only. It does not support native +histograms, exemplars, persistent queues, automatic protocol negotiation, +OAuth, SigV4, or round-robin DNS behavior. + +## See Also + +- metrics +- to_http +- analytics/collect-metrics +- prometheus diff --git a/src/sidebar.ts b/src/sidebar.ts index ddbffd657..5b40dff8f 100644 --- a/src/sidebar.ts +++ b/src/sidebar.ts @@ -398,6 +398,7 @@ export const integrations = [ "integrations/elasticsearch", "integrations/mysql", "integrations/opensearch", + "integrations/prometheus", "integrations/snowflake", "integrations/splunk", ],