Add Message Attestation for ServerToAgent messages

[liustanley]

Resolves #265, see proposal document for context.

Summary

This PR introduces an optional, end-to-end integrity mechanism for ServerToAgent messages based on X.509 certificate chains. When both the Server and the Agent opt in, every ServerToAgent message sent after the initial handshake carries a signature that the Agent verifies against a pre-configured trust anchor (a CA certificate) that is distinct from the TLS certificate authority used to establish the transport.

The mechanism allows OpAMP deployments to separate the distribution server from the authoritative source of OpAMP messages. A compromised distribution server cannot, in this model, push arbitrary configuration or commands to an Agent, because every message must be signed by a key whose trust chain validates against the Agent's pre-configured root.

Strict opt-in: implementations that do not implement Message Attestation require no changes. Existing OpAMP deployments are unaffected until both sides opt in.

Changes

Wire-level changes (all Status: [Development]):

• AgentCapabilities.RequiresPayloadTrustVerification = 0x00010000
• ServerCapabilities.OffersPayloadTrustVerification = 0x00000080
• ServerToAgent.trust_chain_response = 12 (new TrustChainResponse message)
• ServerToAgent.signature = 13

The new "Message Attestation" section in specification.md covers the threat model, trust model, opt-in semantics, capability negotiation, connection-time handshake, in-session per-message verification, algorithm selection (any X.509-supported), certificate requirements, failure modes (all result in agent disconnect), and out-of-scope items.

Client-facing changes

To enable verification of messages, the OpAMP client is configured with a root certificate to trust at startup via a config value:

extensions:
  opamp:
    server:
      payload_ca: /etc/opamp/securecorp-ca.pem
      ws:
        endpoint: wss://127.0.0.1:4320/v1/opamp

When provided, the client will indicate to the server that it requires authenticated payloads, and use the provided root certificate to verify messages the server sends.

Motivation

The OpAMP Server is currently treated as both a point of distribution and the authoritative source of OpAMP messages. If an OpAMP Server is compromised, clients can be subject to significant security vulnerabilities such as forced restarts, breaches of sensitive information, and unintended binary updates. By requiring messages to be signed by an authoritative source, an attacker gaining access to the OpAMP Server would not be able to sign messages that do not meet strict validation/type requirements. OpAMP Agents will also have the capability to reject messages with invalid certificates, and can immediately terminate the connection, preventing further attacks.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: liustanley / name: Stanley Liu (04e8442, 187263b, 4ac3b0a, 8da5ad4, d296b93)
• ✅ login: pierreprinetti / name: Pierre Prinetti (d296b93)
• ✅ login: tigrannajaryan / name: Tigran Najaryan (d296b93)
• ✅ login: truthbk / name: Jaime Fullaondo (7a8dfcf, 8da5ad4, 1f683f7)

[tigrannajaryan]

I am blocking this for now to prevent accidental merging.

This needs a thorough review.

I am also not sure we want this now, it is not aligned with anything we have on the roadmap: #321

If we make the call we want this in the spec, this is going to require a prototype demonstrating it (not needed for now) and we will need to decide when we want this to be added (possibly after 1.0).

[domodwyer]

Disclaimer: I provided technical help for the proposal.

I agree - I think getting a working prototype nailed down would really help shake out any issues 👍

I understand the roadmap to V1 is already being worked on so the timing isn't great, but I do think there's value in at least investigating what a more secure protocol might look like.

The primary concern with OpAMP as it proposed for V1, is that it's a security single-point-of-failure. If you compromise the distribution server (which necessarily sits in a risky network position, reachable by many machines) then it's game over: the attacker has full control of the fleet.

This is especially bad if you're a 3rd party provider like Datadog (my employer) as a single nginx exploit or similar could yield control of many thousands of machines across many distinct organisations. This kind of exposure becomes a significant business risk and isn't something that's easy to accept - it unfortunately fundamentally restricts our adoption of the open protocol we're otherwise big fans of!

We're definitely motivated to help develop any solutions 🙏

[domodwyer]

So I think there's two other properties that this proposal doesn't currently address that we might want to consider - they're both part of a class of attacks we expect for our internal protocol to mitigate:

• Message replay: if an attacker obtains a valid and signed message, they can reuse that message and send it to the same agent again, or a different agent that was not the intended recipient.
• Rollback attack: related to the above, a previously captured message can be sent to revert any subsequent changes (like "roll back a config change").

A classic replay attack in this context would be capturing a validly signed ServerToAgentCommand.type = Restart message for an Agent, and then resending it every time that agent connects, causing a DoS of the Agent using the attested message.

A more subtle attack would be to use the above replay trick to obtain a legitimate remote config message that compels an Agent to upload sensitive files to the server¹ or perform some similarly risky action, and then broadcast that command to all connected agents instead of just the intended recipient.

Notably an attacker can't change the content of these messages (that would invalidate the signature). They're constrained to only replaying messages previously sent by the server.

Fixes are relatively easy:

• To prevent cross-session replay attacks, you can make use of a session nonce ("number used once") that is randomly generated by the Agent, unique to the connection and is part of the signed ServerToAgent payloads sent to that Agent. Upon receiving a payload, the ID is checked to match. That prevents a message destined for Agent A from being replayed to Agent B.

• To prevent intra-session replay attacks: maintain a sequence number such that an old message cannot be resent later in the session to roll back the effects of a message (e.g. cannot "undo" a "change config" request) - much like your car probably does. There's a similar mechanism already in place for AgentToServer messages already, but for a different purpose.

Note: there is the instance_id field which goes a long way to preventing cross-Agent replay attacks, but the protocol allows the server to push an "instance ID change" request (ref) which could be used to switch the Agent to an instance ID that is part of a previously captured request. I'm not familiar enough with the implementation to know if this is a viable path, and if not, the existing instance ID may suffice as a session nonce, but it should be documented as a load-bearing part of the protocol security if used as such.

Footnotes

1. https://opentelemetry.io/docs/specs/opamp/#security-considerations ↩

[tigrannajaryan]

To prevent intra-session replay attacks: maintain a sequence number such that an old message cannot be resent later in the session to roll back the effects of a message (e.g. cannot "undo" a "change config" request) - much like your car probably does. There's a similar mechanism already in place for AgentToServer messages already, but for a different purpose.

We can look into adding sequence_num to ServerToAgent to prevent the replay attacks.

There is a a bit of nuance here that needs some thought:

• We cannot require for agent-generated and server-generated sequence_num to be in sync, they will be incremented independently, particularly when WebSocket transport is in use. Nevertheless the agent can correctly verify monotonic increase of server-side generated sequence_num.
• Need to specify what happens on reconnects, Server restarts and also when multiple Server instances are behind load balancer.

but the protocol allows the server to push an "instance ID change" request (ref) which could be used to switch the Agent to an instance ID that is part of a previously captured request.

This can't happen if the attestation is used. The "instance ID change" itself is subject to attestation. We must assume if the signing server is uncompromised then "instance ID change" message is legitimate.

[liustanley]

We can look into adding sequence_num to ServerToAgent to prevent the replay attacks.

Should we scope this to a separate PR or address it in this one?

[tigrannajaryan]

Let's do in a separate PR. sequence_num is also interesting for other reasons, and I want to have that discussed separately. There is a bit of nuance there (e.g. how are the server instances expected to generate sequential numbers in a horizontally scaled deployment) that warrants its own discussion.

Once we have sequence_num we can update attestation section to make use of it.

[domodwyer]

I think this presents an interesting future development if justified by usage - being able to safely delegate parts of operating the fleet to 3rd parties or higher-risk infra, while retaining control over risk factors is a nice feature.

[tigrannajaryan]

This is indeed interesting. We can imagine a simple fallback mechanism where the Agent upon detecting the Server does not support attestation or if signature cannot be verified, goes into "readonly" mode where it continues sending AgentToServer messages with status reports, etc, but ignores any remote config, commands, etc received from the Server.

This is also useful for situations when certificates expire. Instead of resulting in total loss of communication with the Agents this would result in readonly mode, greatly helping with understanding and troubleshooting the situation.

This may be a mode of operation that we allow in the spec and let implementations decide how strict they want to be in this failure mode.

[liustanley]

I agree that a "readonly" mode would be an interesting fallback mechanism, but I'm inclined to leave this mechanism as out of scope for the initial message attestation spec. We are already defining message attestation as an opt-in capability, so users who opt-in will likely be interfacing with servers that support attestation and will expect full read-write capabilities.

If a "readonly" mode is desired in the future, it may make sense to propose this as a separate spec from message attestation, since it could apply to users who are opt-out as well.

[tigrannajaryan]

I took a quick look at the proposal and it is not clear to me exactly what kind of attacks this proposal prevents.

If I understand the server is expected to sign all payloads so that the client can verify. This requires the server to have access to the certificate that is used for signing. If the attack that we are attempting to prevent is performed by compromising the server, what exactly is prevented here? The attacker who can tamper with the ServerToAgent messages on the server will also gain access to the signing certificate and can sign the tampered payloads.

Unless the signing process and access to the signing certificate is separated from the server that creates ServerToAgent messages, I am not sure I understand what purpose this serves. If that separation is the intent then I do not see this described in the specification. Are you aiming for having 2 different servers, one that creates ServerToAgent payloads and another server that signs the payloads? Only the second server has access to the certificate presumably?

Please describe the mechanics of how you intend this to work in more details. An architecture diagram would be very helpful.

[domodwyer]

I took a quick look at the proposal and it is not clear to me exactly what kind of attacks this proposal prevents.

So the spec proposal doc outlines some "attack vectors" it helps mitigate (which are more accurately "security concerns" - they're not all attacks) but the TLDR is that there's some benefits to traceability and auditing by cryptographically tying changes to users / infra operator identities, and additional hardening of data at rest, etc, but realistically these are probably "nice to haves" for most use OpAMP users.

The real pain point today is that a compromise of the OpAMP server (or TLS terminating proxy) yields full control of the OpAMP fleet connected to it - I think this is probably the biggest concern. It is worth noting that since my previous message an nginx bug that enables remote code execution in specific configurations was identified, which if successfully exploited, would give an attacker the access they'd need to take control of the fleet today. nginx is commonly deployed as a load balancer / ingress gateway in front of the actual services like an OpAMP server.

If I understand the server is expected to sign all payloads so that the client can verify. This requires the server to have access to the certificate that is used for signing. ...

I completely agree with your general sentiment in this paragraph - this configuration would make no sense as it provides no benefit. Opting into attestation here adds nothing over using the existing TLS setup.

If that separation is the intent then I do not see this described in the specification. Are you aiming for having 2 different servers, one that creates ServerToAgent payloads and another server that signs the payloads? Only the second server has access to the certificate presumably?

Yes, this is generally how this would work. The split helps isolate any compromise of the distribution server, which necessarily sits at a relatively high exposure position on the "edge" of the network, where it is widely reachable (by at least every member of the fleet).

I've sketched a quick diagram of an example deployment for the sake of discussion:

In the above:

1. The OpAMP agent connects to the reachable OpAMP server at the network / infra team's edge.
2. The OpAMP server decides what message it wants to send to the agent (omitted from this diagram is data sourcing, etc). It builds a full ServerToAgent payload.
3. The OpAMP server submits this ServerToAgent payload to the internal policy engine; this server looks at the request, and decides if it is suitable to be sent to the specific OpAMP agent.

This is the core of the additional security - it's structurally isolating the "what can be done" decision away from the edge server that is relatively easier to gain control over. The policy engine should obviously be deployed such that it is considered secure and trustworthy - limited network access, limited authorised users, additional reviews for code changes, etc.

This policy engine can decide on what is allowed to be sent to the OpAMP agent using whatever criteria is relevant to the organisation.

That might include factors such as:

• Denying specific messages outright (e.g. no ServerToAgentCommand allowed).
• Enforcing specific customer opt-ins / opt-outs ("I only want to allow using feature X, not Y").
• Enforcing security constraints on messages ("the SIEM pipeline must never be disabled").
• Limiting the types and content of custom messages / plugins.
• Only configs originating from the infra management team can be pushed to OpAMP agents.
• Only the latest version of tracers can be installed (mitigating "bring your own vulnerability" type attacks)
• etc

These constraints could be added client side, but the level of flexibility necessary to express the above (and more needed in prod envs) would require a fair bit of complexity in the Agent code, shifting the burden onto OTel rather than the organisation itself. With an attestation approach, the protocol defines a way to stamp a message as "allowed" and the client knows how to validate it - all the bespoke decision logic is maintained in the backend, by the organisation itself.

6. If accepted, the policy engine submits a signature request to a secure signer (typically a tightly controlled hardware security module) to cryptographically "stamp" the message as approved.

7. The now signed ServerToAgent is sent back to the agent, along with the signature it can use to validate the message was authorised.

This is of course only an example, and any actual deployment may be simpler or more complex depending what the organisation has already in place, and what risk mitigation is deemed necessary. We could definitely write some of the deployment guidelines into the spec, but as-is today it doesn't really prescribe deployments - we would definitely be happy to contribute some documentation as part of this effort though! (either in the spec or as general docs).

[tigrannajaryan]

@domodwyer thank you for your comment. It is very helpful and greatly clarifies the operation.

@liustanley do you mind adding a "supplementary-guidelines.md" to this PR and incorporate there the description from here? We usually include non-normative explanations like that in Otel specs (for example here). This is a complex feature and I think it warrants a guideline to help implementors.

We need more reviewers and more feedback on this proposal and having these guidelines in the PR rather than in a comment also makes it easier to review.

[pierreprinetti]

This appears like a solid spec, with the clear premise that it wants to be lightweight. In my head I like to frame it that while TLS protects the message on the wire, message attestation is a powerful jump base for attribution and auditability of OpAMP changes. This is going to become extremely useful the more operations become AI-driven.

No algorithm negotiation also means that the burden to set up quantum-resistant crypto (very touchy topic for asymmetric primitives) remains orthogonal to this spec.

Just a couple comments inline. My third comment would have been to future-proof the signature block with something like:

message Signature {
    oneof kind {
        bytes detached = 1;
    }
}
Signature signature = 2;

but I understand that introducing new signatures would be a backwards-compatible change in gRPC even with the current code (thanks @domodwyer for educating me on the matter).

I hope this proposal finds its way to the main branch.

[pierreprinetti]

Why is the signature optional in the first SignedServerToAgent, considering that it may have a payload?

[pavankrish123]

Thanks @liustanley

What's the "break glass" story for CA compromise?

If the signing CA (or an intermediate) is compromised, the operator needs agents to immediately stop trusting it. But agents will only accept messages signed by that very CA — creating a deadlock where the operator can't push a trust rotation via OpAMP, while the attacker can still produce valid signatures.

Should the spec define an out-of-band revocation mechanism, a pre-configured backup CA, or a time-bounded trust (e.g., short-lived signing certs that naturally expire)? Without this, I am guessing the failure mode of a CA compromise is arguably worse than not having attestation at all — we will loose fleet control and the attacker retains it.? Correct me if I am wrong

[pierreprinetti]

Thanks @liustanley

What's the "break glass" story for CA compromise?

If the signing CA (or an intermediate) is compromised, the operator needs agents to immediately stop trusting it. But agents will only accept messages signed by that very CA — creating a deadlock where the operator can't push a trust rotation via OpAMP, while the attacker can still produce valid signatures.

Should the spec define an out-of-band revocation mechanism, a pre-configured backup CA, or a time-bounded trust (e.g., short-lived signing certs that naturally expire)? Without this, I am guessing the failure mode of a CA compromise is arguably worse than not having attestation at all — we will loose fleet control and the attacker retains it.? Correct me if I am wrong

@pavankrish123

If an intermediate signing CA is compromised, standard X.509 revocation (CRL/OCSP) handles it; the spec already recommends this. If instead the root payload trust anchor itself is compromised, it must be replaced via the same out-of-band mechanism used to set it up. Certificate maintenance is indeed an additional operational burden; maybe it's worth mentioning that the operator should set up a CA cert rotation mechanism. However, the CA alone wouldn't let an attacker send messages to the Agent: for that, the control of the OpAMP server endpoint is required.

If an attacker gains control of both the signing CA and the OpAMP server endpoint, then it is indistinguishable from the legitimate operator, and the escape hatch has to be external to OpAMP. Even if there were a secondary CA, the Agent would still be fetching config from the compromised endpoint. In that scenario, the escape hatch to be same out-of-band mechanism (read: not OpAMP) that was used to set up the legitimate CA in the first place.

You seem to require the same escape hatch, with or without message attestation, in case of complete compromission. Except that with message attestation the scenario is arguably less likely to occur.

[liustanley]

@liustanley do you mind adding a "supplementary-guidelines.md" to this PR and incorporate there the description from here? We usually include non-normative explanations like that in Otel specs (for example here). This is a complex feature and I think it warrants a guideline to help implementors.

We need more reviewers and more feedback on this proposal and having these guidelines in the PR rather than in a comment also makes it easier to review.

Thanks for the suggestion @tigrannajaryan, that makes sense with me. Just added details from @domodwyer's comment in a new file supplementary-guidelines.md.

[tigrannajaryan]

@andykellr (who is the other spec maintainer) and I have discussed this proposal.

We believe it is an important missing capability of OpAMP and we would like to have it added.

We recently agreed on a roadmap for OpAMP. This capability is something we don't have on the roadmap. However, I think it is important enough to modify the roadmap and include it, provided that we have a good solution.

I reviewed the proposal and overall the suggested solution seems reasonable to me. I will post my detailed comments on the specifics a bit later.

To go ahead with this I would like to see the following:

• Create a prototype in opamp-go. We want to see Client and Server implementations. This needs to be a prototype, so you can skip the tests for now. Create an early draft PR to discuss the API and callbacks first.
• Modify the example in opamp-go to demonstrate how attestation works. Show the best-practice approach, introduce the separate out-of-process Policy server as described in supplementary-guidelines.md.
• Once the prototype and the example are in place, maintainers and approvers will review the whole package (including this PR) and provided everything is in order the capability will be added in [Development] status.
• Modify Collector opampextension and Collector Supervisor to support attestation. I need additional confirmation from @evan-bradley on this one, that he is able to review the PRs against these components, since he is the most active maintainer.

I expect that the prototype, example and final production version of opamp-go, opamp extension and Supervisor will be done by the authors of the proposal. This is a complicated feature. Current maintainers will not be able to provide the manpower to implement it themselves. We can commit to reviewing and long-term maintenance, provided that someone else implements it initially, at production quality, with necessary test coverage.

Note: this message is not a final approval and acceptance of the proposal. This is a commitment to review it and accept it if it meets all requirements, including new requirements that may come up in the future. It is possible that the proposal is rejected at some stage in the process I outlined above, although I commit to doing by best to make it successful, since I believe this capability is needed.

[tigrannajaryan]

To prevent intra-session replay attacks: maintain a sequence number such that an old message cannot be resent later in the session to roll back the effects of a message (e.g. cannot "undo" a "change config" request) - much like your car probably does. There's a similar mechanism already in place for AgentToServer messages already, but for a different purpose.

We can look into adding sequence_num to ServerToAgent to prevent the replay attacks.

There is a a bit of nuance here that needs some thought:

• We cannot require for agent-generated and server-generated sequence_num to be in sync, they will be incremented independently, particularly when WebSocket transport is in use. Nevertheless the agent can correctly verify monotonic increase of server-side generated sequence_num.
• Need to specify what happens on reconnects, Server restarts and also when multiple Server instances are behind load balancer.

but the protocol allows the server to push an "instance ID change" request (ref) which could be used to switch the Agent to an instance ID that is part of a previously captured request.

This can't happen if the attestation is used. The "instance ID change" itself is subject to attestation. We must assume if the signing server is uncompromised then "instance ID change" message is legitimate.

[tigrannajaryan]

@liustanley please sign EasyCLA.