Message Attestation: Client + Server prototype

[liustanley]

Overview

Implements the Message Attestation feature from opamp-spec PR 333. Every ServerToAgent is wrapped in a SignedServerToAgent envelope carrying a detached X.509 signature. Agents that opt in can verify that responses originate from an authorised distribution server and have not been tampered with in transit — even when TLS is terminated at a load balancer.

Fully opt-in and backward-compatible. Agents and servers that do not configure attestation use the existing wire format unchanged.

This PR contains only the core implementation — no tests, no examples. See message-attestation-example for a runnable demo.

---

API

Server — server.Settings

type Settings struct {
    // ... existing fields ...

    // PayloadSigner produces detached signatures over every outbound
    // ServerToAgent when an Agent has declared
    // AgentCapabilities_RequiresPayloadTrustVerification. When set, the
    // server automatically advertises
    // ServerCapabilities_OffersPayloadTrustVerification on all responses.
    // Nil disables attestation entirely (default, backward-compatible).
    PayloadSigner signing.Signer
}

Client — client/types.StartSettings

type StartSettings struct {
    // ... existing fields ...

    // PayloadVerifier validates the X.509 trust chain and per-message
    // signatures delivered by the server. When set, the client
    // automatically sets AgentCapabilities_RequiresPayloadTrustVerification.
    // Mutually exclusive with PayloadTOFUStore; this field takes precedence.
    PayloadVerifier signing.Verifier

    // PayloadTOFUStore enables Trust On First Use enrollment.
    // On startup: if Load() returns an anchor it is promoted to a
    // PayloadVerifier; otherwise the client sets both
    // RequiresPayloadTrustVerification and AcceptsPayloadTrustAnchorTOFU,
    // accepts the root CA from the first TrustChainResponse, and persists
    // it via Save(). Mutually exclusive with PayloadVerifier.
    //
    // WARNING: TOFU provides no security on the first connection. Prefer
    // PayloadVerifier with an out-of-band provisioned CA when possible.
    PayloadTOFUStore signing.TOFUStore
}

---

signing package — interfaces

// Signer is implemented by the OpAMP server (or its delegate).
type Signer interface {
    Sign(ctx context.Context, payload []byte) ([]byte, error)
    ChainDER(ctx context.Context) ([][]byte, error)
}

// TrustAnchorProvider is an optional extension of Signer for TOFU enrollment.
// The server checks for this interface via type assertion.
type TrustAnchorProvider interface {
    TrustAnchorPEM(ctx context.Context) ([]byte, error)
}

// Verifier is implemented by the OpAMP agent.
type Verifier interface {
    ValidateChain(ctx context.Context, chainDER [][]byte, now time.Time) (*x509.Certificate, error)
    Verify(ctx context.Context, payload, signature []byte, leaf *x509.Certificate) error
}

// TOFUStore persists the enrolled root CA across restarts.
type TOFUStore interface {
    Load() ([]byte, error)   // nil if no anchor stored yet
    Save(pem []byte) error   // idempotent — O_EXCL write, subsequent saves are no-ops
}

Provided implementations

Type	Usage
signing.LocalSigner	In-process signing from a crypto.Signer (file-loaded key)
signing.LocalVerifier	In-process verification against an *x509.CertPool
signing.RemoteSigner	Delegates Sign/ChainDER/TrustAnchorPEM to an HTTP policy server
signing.FileTOFUStore	Persists the enrolled root CA to a file

Constructor helpers

signing.LocalSignerFromFiles(keyPath, chainPath string) (*LocalSigner, error)
signing.VerifierFromFile(caPath string) (*LocalVerifier, error)
signing.VerifierFromPEM(pemBytes []byte) (*LocalVerifier, error)
signing.NewFileTOFUStore(path string) *FileTOFUStore
signing.NewRemoteSigner(baseURL string) *RemoteSigner

---

Callbacks

No new callbacks are introduced. Existing callbacks now receive attestation-related errors through the existing surfaces:

// ConnectionCallbacks (set inside OnConnecting)

// OnReadMessageError fires when the server cannot read or deserialise
// an inbound message. Now also covers oversized-message rejections.
OnReadMessageError func(conn Connection, mt int, msgByte []byte, err error)

// OnMessageResponseError fires when the server fails to send a response.
// Now also fires when a response exceeds MaxMessageSize.
OnMessageResponseError func(conn Connection, message *protobufs.ServerToAgent, err error)

On the client side, any attestation failure (missing trust chain, chain validation failure, SAN mismatch, bad signature) terminates the connection immediately. The WS client applies exponential backoff before reconnecting. The HTTP polling client resets attestation state and applies the same backoff before the next poll.

---

New error sentinel (server)

// ErrSendBeforeNegotiated is returned from Connection.Send when called
// before the first AgentToServer is received (agent capabilities unknown).
// Push responses from OnMessage callbacks, not OnConnected.
var ErrSendBeforeNegotiated = errors.New("...")

---

Capability bits (auto-managed)

Operators do not set these manually — they are derived from configuration:

Bit	Auto-set by
AgentCapabilities_RequiresPayloadTrustVerification	Client, when PayloadVerifier or PayloadTOFUStore is set
AgentCapabilities_AcceptsPayloadTrustAnchorTOFU	Client, when TOFU store has no persisted anchor yet
ServerCapabilities_OffersPayloadTrustVerification	Server, when PayloadSigner != nil

---

Minimal wiring example

Server:

signer, _ := signing.LocalSignerFromFiles("server.key", "server-chain.pem")
srv := server.New(logger)
srv.Start(server.StartSettings{
    Settings: server.Settings{
        PayloadSigner: signer,
        Callbacks:     ...,
    },
})

Agent (pre-provisioned CA):

verifier, _ := signing.VerifierFromFile("ca.pem")
client := client.NewWebSocket(logger)
client.Start(context.Background(), types.StartSettings{
    PayloadVerifier: verifier,
    ...
})

Agent (TOFU enrollment):

client.Start(context.Background(), types.StartSettings{
    PayloadTOFUStore: signing.NewFileTOFUStore("/var/lib/opamp/trust-anchor.pem"),
    ...
})

[truthbk]

Looks consistent with the spec.

[truthbk]

We should maybe call out in the comments what algorithms are FIPS 140-3 compatible. If they're all FIPS 140-3 valid, we should also make a mention of that.

[truthbk]

Added a couple more comments. Just dropping some thoughts that came up during the review. I think the implementation closely matches the spec with respect to attestation failures and cert expiration, but we want to be very confident about the behavior on these events we know will eventually, and regularly, occur. For instance, if mid-attestation we have a failure, I think we reset the attestation and start "fresh", I also think this is fully transparent to the user. We should also maybe make sure these events are logged.

[truthbk]

Can you please check if the backoff logic also adds jitter to the backoff period. I think it does, but please double-check 🙏

[liustanley]

Just confirmed that backoff adds jitter with a randomized interval: https://github.com/cenkalti/backoff/blob/v4/exponential.go#L90. NextBackOff() returns getRandomValueFromInterval(...) → a value in [interval*0.5, interval*1.5]. So each attestation retry is jittered ±50%, which avoids a fleet retrying in lockstep after a synchronized failure.

[truthbk]

We know the cached leaf cert will expire at some point - and we know that attestation state is kept per connection, but the OpAMP connection behavior is not totally clear to me. Are OpAMP client-server connections long-lived and reused across requests? Or is a new connection started with every OpAMP flow?

[liustanley]

Are OpAMP client-server connections long-lived and reused across requests? Or is a new connection started with every OpAMP flow?

The answer to this differs by transport:

• WebSocket: OpAMP client-server connection is long-lived and persistent, has one attestationState per connection. The leaf is validated once on the first message and pinned for the connection's lifetime, then subsequent messages are signature-only checks. On attestation failure, the connection will terminate which triggers a reconnect, leading to a fresh state and fresh handshake with the new chain.
• HTTP: Relies on request/response polling so there is no persistent connection, and a new connection is started with every OpAMP flow. However, the same attestationState is reused across polls to avoid re-doing the full chain validation on every single request. First poll handshakes, later polls verify against the pinned leaf. On any failure, Reset() is called and the next poll re-handshakes.

We know the cached leaf cert will expire at some point

In the case that the server rotates its signing key (or the leaf expires and the server starts sending a renewed one), a signature mismatch error will occur which causes the client to reset/reconnect to trigger a fresh handshake with the new chain. But if the leaf expires with no server-side change, the expired leaf may not be re-checked since Verify is signature only. In this case, recovery would rely on the connection dropping/a failure, or a manual reset which could be problematic if this is a valid scenario.

[truthbk]

I think this is how we transparently recover from an expired leaf cert - which is great.