[akamai] - Add OpenTelemetry collection mode via the Akamai Siem Receiver

[ShourieG]

Type of change

• Enhancement

Proposed commit message

akamai: add OpenTelemetry collection mode via the akamai_siem receiver

• New siem_otel data stream: native OTel receiver as third input
  beside CEL/GCS (Tech Preview). Requires stack 9.5.0;
  format_version 3.6.0 (new SVR00006/SVR00009 checks excluded —
  legacy pipeline).

• Separate data stream because Fleet gives any otelcol-containing
  data stream a .otel-suffixed template pattern, which would orphan
  logs-akamai.siem-* for existing users.

• Routing: data_stream.dataset var + bodymap docs → events land in
  logs-akamai.siem-<ns>, existing pipeline reused.
  dynamic_dataset/namespace widen the API key (Fleet scopes otelcol
  grants to .otel datasets that bodymap never writes to).

• Template quotes string vars (all-digit values ≠ YAML ints);
  enable_storage var (off) drafts file_storage extension — no
  storage: ref on the receiver, pending auto-bind support.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Screenshots

All Inputs

Receiver Config

[github-actions]

Elastic Docs Style Checker (Vale)

Summary: 1 warning, 1 suggestion found

⚠️ Warnings (1): Fix when the suggestion improves clarity or correctness.

File	Line	Rule	Message
packages/akamai/_dev/build/docs/README.md	25	Elastic.DirectionalLanguage	Don't use directional language. Use 'the following steps' instead of 'the below steps'.

💡 Suggestions (1): Optional style improvements. Apply when helpful.

File	Line	Rule	Message
packages/akamai/data_stream/siem_otel/manifest.yml	103	Elastic.WordChoice	Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'Disabled', unless the term is in the UI.

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

Changelog link mismatch — expected https://github.com/elastic/integrations/pull/19496 in the following file(s):

• packages/akamai/changelog.yml

Tip

If expected, add the changelog-link-check:skip label to skip this check. Or, if an issue link was intended, use .../issues/<n> instead.

View Buildkite build
@ShourieG

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: fc275fc

Failed CI Steps

• 📜 Check changelog PR links
• Check integrations akamai

History

cc @ShourieG

[github-actions]

TL;DR

buildkite/integrations failed primarily because packages/akamai/changelog.yml contains a PR link to .../pull/99999 instead of this PR (.../pull/19496). Update that link and re-run CI first; the second failed job log is truncated and does not include its underlying error.

Remediation

• In packages/akamai/changelog.yml, replace the new entry’s link value https://github.com/elastic/integrations/pull/99999 with https://github.com/elastic/integrations/pull/19496.
• Re-run the Buildkite build after the changelog fix. If Check integrations akamai still fails, pull the full step log/artifacts for that job (the attached excerpt ends at teardown and omits the original failure line).

Investigation details

Root Cause

The :scroll: Check changelog PR links step validates that modified changelog.yml entries point to the current PR number. It failed on packages/akamai/changelog.yml because the file contains a hardcoded mismatched PR link (99999).

The separate Check integrations akamai log provided in this workflow artifact does not contain the actual failing command output (only teardown + artifact upload), so its exact root cause cannot be confirmed from the available data.

Evidence

• Build: https://buildkite.com/elastic/integrations/builds/44413
• Job/step: :scroll: Check changelog PR links
• Key log excerpt:

  ERROR: unexpected link: 'https://github.com/elastic/integrations/pull/99999'
         expected:         'https://github.com/elastic/integrations/pull/19496'
  --- 1 changelog link(s) do not match this PR
  - `packages/akamai/changelog.yml`
  

• Job/step: Check integrations akamai
• Key log excerpt:

  --- [akamai] failed
  🚨 Error: The command exited with status 1
  user command error: exit status 1
  

  (No preceding stack trace/assertion/error line is present in the provided log file.)

Verification

• Not run locally: the Buildkite wrapper script requires CI-provided environment/tooling not present in this runner (YQ_VERSION was unset), so local reproduction was not representative of CI.

Follow-up

After correcting the changelog link, if the Akamai integration step still fails, share the full untruncated step log (or failed JUnit XML content) and I can map it to the exact source file/line for a second-pass fix recommendation.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• #19496 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• [akamai] - Add OpenTelemetry collection mode via the Akamai Siem Receiver #19496 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[ShourieG]

@jamiehynds, wanted to get your opinion on this change from an user perspective.

With the current limitations imposed by Fleet, having an otel receiver config within an existing data stream breaks it for existing users since fleet renames the index template by appending an .otel suffix to the template name. Because of this I needed to create a separate data stream for the otel receiver and then route the data to the existing data stream to avoid ingest pipeline duplication.

The user sees this as a new input type in addition to the existing cel and gcs inputs as shown in the screenshots. Upon configuring and using it the data lands in the original akamai.siem data stream. The issue is now we have almost two similar inputs with almost identical configurations, the CEL API based Input and the OpenTelemetry receiver that also uses the api underneath but with custom Go code that makes it way more stable and performant compared to the CEL input. We went the OTEL route since Akamai pull api has a lot of issues and the CEL implementation suffers a lot from these limitations with various reported and escalated in the past.

The issue now is about user perception. We end up having two similar looking inputs from the outside CEL API and OpenTelemetry receiver but the latter is much more stable an performant.

Do we keep both of these or phase out the older CEL implementation slowly or what pitch do we make to users to it's less confusing on their side.

cc: @narph