diff --git a/CHANGELOG.md b/CHANGELOG.md index 85d42fa05ed..4fd9e0b8687 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,12 @@ release. ### Logs +### Audit + +- Introduce Audit-Logging signal and related specification sections. + ([#1](https://github.com/apeirora/opentelemetry-specification/pull/1)) + ([#6](https://github.com/apeirora/opentelemetry-specification/pull/6)) + ### Baggage ### Profiles diff --git a/oteps/0267-audit-logging.md b/oteps/0267-audit-logging.md new file mode 100644 index 00000000000..e97b57dbf66 --- /dev/null +++ b/oteps/0267-audit-logging.md @@ -0,0 +1,373 @@ +# Audit Logging Signal + +Introduce a dedicated Audit Logging signal to OpenTelemetry that guarantees +lossless, tamper-evident delivery of security-relevant events and satisfies +compliance requirements such as ISO 27001. + + + + +## Motivation + +### The problem with re-using the existing Log signal + +OpenTelemetry Logs are designed for general-purpose observability. They share +the same pipeline as traces and metrics, which means they are subject to: + +- **Sampling** – processors and exporters may drop records for performance + reasons. +- **Back-pressure shedding** – SDK queues may overflow and silently discard + records. +- **Transformation** – processors may modify, redact, or aggregate records + before export. + +These behaviours are intentional and desirable for observability logs. They are +**incompatible** with audit logging, where every record MUST be delivered +exactly once to the designated audit sink, in order, without modification. + +Furthermore, compliance frameworks such as ISO 27001, SOC 2, PCI-DSS and HIPAA +require: + +- Guaranteed delivery to a tamper-protected sink. +- Proof of record integrity (i.e. the record was not altered in transit). +- Separation of audit records from operational logs. +- Clock synchronisation across all emitting services. + +None of these guarantees are made today by any OTel signal. + +### Use cases + +| Actor | Audit event | Compliance driver | +|-----------------|-------------------------------|------------------------------| +| End user | Successful / failed login | ISO 27001 A.8.15, SOC 2 CC6 | +| End user | Access to sensitive data | GDPR Art. 30, HIPAA §164.312 | +| Operator | Configuration change | ISO 27001 A.8.15 | +| Service account | Elevated-privilege action | ISO 27001 A.5.18 | +| Service | Outbound call to external API | PCI-DSS Req. 10 | +| Runtime | Antivirus / IDs rule fired | ISO 27001 A.8.16 | + +### Why a new signal and not a new Log feature flag? + +A feature flag (e.g. `audit=true` on a `LogRecord`) still shares the existing +pipeline and therefore inherits all the behaviours listed above. A dedicated +signal allows: + +- A purpose-built SDK pipeline with no sampling, no transformation, and + at-least-once delivery semantics. +- A distinct OTLP service name / endpoint so the audit sink can be isolated + from the observability backend. +- A different data model with mandatory integrity metadata. + +## Explanation + +### Signal overview + +The new signal is called **Audit Logging**. It is modelled structurally after +the existing Log signal and reuses the `LogRecord` data model as its transport +layer, adding a small set of mandatory fields. + +``` +Application code + │ + ▼ + AuditLogger.emit(AuditRecord) + │ + ▼ + AuditProvider (SDK-level, no sampling, no dropping) + │ └── AttributeProcessor (enrich only, never redact/drop) + ▼ + AuditExporter ──► Audit sink (e.g. OpenSearch SIEM, Splunk, S3 WORM) + │ + └── returns SHA-256 receipt hash ◄──────┐ + │ + emit() return value (delivery proof) +``` + +### AuditProvider + +`AuditProvider` is the entry point, analogous to `LoggerProvider`. It: + +- Holds a reference to a configured `AuditExporter`. +- Provides named `AuditLogger` instances. +- MUST NOT expose a sampling configuration option. +- MUST guarantee at-least-once delivery: records are queued durably (or + synchronously) until the exporter acknowledges receipt. + +### AuditLogger + +`AuditLogger` is obtained from `AuditProvider` and is scoped to an +instrumentation scope (library name + version). It exposes a single method: + +``` +receipt = AuditLogger.emit(AuditRecord) → AuditReceipt +``` + +`emit` is **synchronous by default**: it MUST block until the exporter +acknowledges the record, or return an error. An asynchronous variant MAY be +provided, but MUST still guarantee delivery and MUST NOT silently discard on +failure. + +### AuditRecord data model + +`AuditRecord` is a `LogRecord` that uses the standard dedicated +`LogRecord` fields for universal concepts and carries all +audit-specific data as `Attributes` following the `audit.*` semantic +convention namespace. + +#### LogRecord fields used + +| Field | Type | Description | +|---------------------|-------------------------|-----------------------------------------------------------------------------------| +| `Timestamp` | `fixed64` | Time the auditable action occurred. MUST be set. | +| `ObservedTimestamp` | `fixed64` | Time the SDK observed the event. MUST be set. | +| `EventName` | `string` | Semantic name of the audit event. MUST be set. | +| `Body` | `AnyValue` | Free-form or structured event details. MAY be set. | +| `Attributes` | `map` | Carries audit semantic attributes (see below). MUST set all mandatory attributes. | +| `Resource` | – | Emitting service / host; carries integrity attributes. | + +`SeverityNumber`, `SeverityText`, and `InstrumentationScope` SHOULD NOT +be set / SHOULD be left empty for audit records. + +#### Audit semantic attributes + +Audit-specific data is carried in `Attributes` using the `audit.*` +semantic convention namespace, or by reusing existing OTel semantic +convention attributes where a direct mapping exists. + +**Mandatory attributes:** + +| Attribute name | Type | Description | +|--------------------|----------|-------------------------------------| +| `audit.record.id` | `string` | Unique stable identifier (UUID v4). | +| `audit.actor.id` | `string` | Identity that performed the action. | +| `audit.actor.type` | `string` | `user`, `service`, or `system`. | +| `audit.action` | `string` | Verb: `LOGIN`, `READ`, `DELETE`, … | +| `audit.outcome` | `string` | `success`, `failure`, or `unknown`. | + +**Optional / recommended attributes:** + +| Attribute name | Type | Description | +|-------------------------|----------|-------------------------------------------------------| +| `audit.target.id` | `string` | Identifier of the resource acted upon. | +| `audit.target.type` | `string` | Type of the target resource. | +| `audit.source.id` | `string` | Network address or identifier of the source. | +| `audit.source.type` | `string` | Type of the source (e.g. `ipv4`, `ipv6`, `hostname`). | +| `audit.integrity.value` | `string` | Base64-encoded signature or HMAC. | +| `audit.sequence.number` | `int` | Monotonic counter for hash-chain continuity. | +| `audit.sequence.prev_hash` | `string` | SHA-256 of the preceding record. | +| `audit.schema.version` | `string` | Schema version (e.g. `1.0.0`). | + +The `Resource` carries `audit.integrity.algorithm` and +`audit.integrity.certificate` (unchanged per-service metadata). + +### AuditReceipt + +`AuditReceipt` is returned by `emit` once the record has been acknowledged by +the exporter/sink: + +| Field | Type | Description | +|-----------------|-----------|--------------------------------------------------------------------------------------------------------------------------------------------| +| `RecordId` | `string` | Echoes the `audit.record.id` attribute of the persisted record. | +| `IntegrityHash` | `string` | SHA-256 of the canonical serialization of the `AuditRecord`, computed by the sink after successful write. Used for integrity verification. | +| `SinkTimestamp` | `fixed64` | Nanoseconds since UNIX epoch when the sink persisted the record. | + +The `IntegrityHash` is computed by the **sink**, not the SDK, so it reflects the +record as actually persisted – not as emitted. The emitting service MAY store +the receipt for its own tamper-detection purposes. The emitting service SHOULD compute the same hash locally and compare it to the receipt to verify integrity. A mismatch indicates the record was altered in transit or by the sink, which should trigger an alert (e.g. log an error, raise an incident). + +### OTLP transport + +Audit records are exported using a dedicated OTLP endpoint (distinct from the +standard `/v1/logs` endpoint), tentatively `/v1/audit`. This allows: + +- Network-level isolation (firewall rules, separate credentials). +- Independent QoS policies (no back-pressure shedding). +- Clear separation in the sink's data model. + +The payload is the standard `ExportLogsServiceRequest` proto with an additional +`audit` boolean flag in the `ResourceLogs` message. + +#### No partial success + +The OTLP receiver MUST NOT respond with a `partial_success` for an audit log +export request. If the receiver cannot process one or more audit records in a +batch, it MUST reject the **entire** batch and return an error. The exporter +MUST treat any `partial_success` response as a hard failure, retain all +records, and retry the full batch. + +This constraint ensures that no audit record can be silently lost by a receiver +that processes only part of a batch and acknowledges the rest. + +#### Instrumentation Scope is not applicable + +The `InstrumentationScope` field present in the standard `ScopeLogs` message +has no meaning for audit logging. Audit records are not emitted by +instrumentation libraries – they are emitted by application or system code +acting on behalf of an actor. Exporters MUST leave the `InstrumentationScope` +field empty (or populated with only the SDK name/version as a technical marker) +and receivers MUST NOT use it for routing, filtering, or processing audit +records. + +The three OTLP envelope layers that **do** apply to audit logging are: + +| OTLP layer | Role in audit logging | +|--------------|--------------------------------------------------------------------| +| `Resource` | Emitting service / host; carries integrity attrs. | +| `LogRecord` | Carries `AuditRecord` via dedicated fields and `Attributes`. | +| `Attributes` | Mandatory and optional `audit.*` attributes; see data model above. | + +The `Resource` MUST carry the following attributes whenever any +record in the batch includes an `audit.integrity.value`: + +| Attribute | Type | Description | +|-------------------------------|----------|-------------------------------------------------------------------------------------------------------------| +| `audit.integrity.algorithm` | `string` | Algorithm used to compute `IntegrityValue` (JWA identifier for signatures, IANA MAC identifier for HMACs). | +| `audit.integrity.certificate` | `string` | Key reference: full DER cert (base64), fingerprint, Key ID, SKI, or Issuer+Serial. Not for HMAC algorithms. | + +See +[Integrity Resource Attributes](../specification/audit/data-model.md#integrity-resource-attributes) +for the full field descriptions. + +#### Durability at the sink is out of scope + +This OTEP does not prescribe how the audit sink (e.g. OpenSearch, Splunk, +S3 WORM, a SIEM) stores, retains, or protects records once they have been +successfully acknowledged. Requirements such as write-once/read-many storage, +retention periods, or geographic replication are deployment concerns that depend +on the specific sink technology and the applicable compliance framework. + +The boundary of the OTel audit signal ends at the moment the sink acknowledges +receipt (and, optionally, returns an `AuditReceipt`). Everything beyond that +point – including long-term durability, tamper-proof storage, and retention +enforcement – is outside the scope of this specification. + +## Internal details + +### No sampling, no dropping + +The SDK MUST reject any attempt to configure a sampler on the `AuditProvider`. +The SDK MUST use an unbounded (or disk-backed) queue. If the in-memory queue is +full, the SDK MUST either block the caller or persist the record to a local +durable buffer rather than silently drop it. + +### Processor restrictions + +Only **additive** processors (enrich, tag, forward) are permitted in the audit +pipeline. Processors that delete attributes, filter records, or aggregate +records MUST be rejected at configuration time. + +### Clock synchronisation + +The SDK SHOULD warn at startup if the system clock is not synchronised (e.g. +NTP offset > 1 second). The `ObservedTimestamp` field provides a second +timestamp from the SDK's perspective, which allows the sink to detect skewed +emitters. + +### Interaction with the existing Log signal + +Audit records MUST NOT be routed through the standard `LoggerProvider` +pipeline. The two signals are independent. An instrumentation library MAY emit +both an audit record (for compliance) and a regular log record (for +observability) for the same event. + +### Failure handling + +If the exporter cannot reach the sink: + +1. The SDK MUST buffer the record (in memory or on disk, depending on + configuration). +2. The SDK MUST retry with exponential back-off. +3. If the buffer is exhausted (disk full, etc.) the SDK MUST surface a + hard error to the caller rather than silently drop the record. +4. The SDK MUST expose a metric (`audit.records.dropped`) that counts + any records lost due to unrecoverable errors. + +## Trade-offs and mitigations + +| Trade-off | Mitigation | +|-----------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------| +| Synchronous `emit` adds latency to the calling thread | An async variant with a durable local queue mitigates this; the queue size and flush interval are configurable. | +| A separate pipeline increases SDK complexity | The audit pipeline deliberately reuses `LogRecord` as its wire format, so OTLP exporters can be reused with minimal changes. | +| SHA-256 receipt requires a round-trip to the sink | The receipt is optional for callers that do not need proof-of-delivery; a fire-and-forget async mode MAY omit it. | +| Disk-backed queue introduces a dependency on localStorage | This is opt-in; the default is an in-memory queue with blocking back-pressure. | + +## Prior art and alternatives + +### Alternative 1 – Existing Log signal with a dedicated exporter + +Tag `LogRecord`s with `audit=true` and route them to a dedicated exporter via a +filtering processor. **Rejected**: still subject to SDK queue overflow and +requires consumers to filter records; does not provide integrity receipts. + +### Alternative 2 – Out-of-band audit library + +Bypass OTel entirely and use a dedicated audit library (e.g. OWASP Enterprise +Security API Logger). **Rejected**: loses OTel's context propagation, semantic +conventions, and OTLP transport; introduces a second observability stack. + +### Alternative 3 – W3C Audit Vocabulary / CEF + +Map to an existing audit format. **Not rejected**: the `AuditRecord` semantic +conventions SHOULD map cleanly to Common Event Format (CEF) and the W3C audit +vocabulary. This OTEP does not prescribe a wire format beyond OTLP. + +### Prior art in OTel + +- [OTEP 0092 – Logs Vision](logs/0092-logs-vision.md): defines the overall + logging architecture on which this proposal builds. +- [OTEP 0097 – Log Data Model](logs/0097-log-data-model.md): `AuditRecord` + extends the `LogRecord` data model defined here. +- [OTEP 0202 – Events and Logs API](0202-events-and-logs-api.md): the + `AuditLogger`/`AuditProvider` hierarchy mirrors the `Logger`/`LoggerProvider` + hierarchy introduced here. + +## Open questions + +1. **OTLP endpoint path** – Should audit records share `/v1/logs` with a + distinguishing flag, or use a dedicated `/v1/audit` path? A dedicated + path makes routing and access control simpler but requires changes to all + OTLP receivers. + +2. **Receipt hash computation** – Should the hash be computed by the SDK + (guaranteeing what was sent) or by the sink (guaranteeing what was stored)? + The current proposal says sink, but this requires a synchronous + acknowledgement that may not be supported by all exporters. + +3. **Mandatory vs. optional receipt** – Should `emit` always block for a + receipt, or should this be configurable? Blocking guarantees delivery proof + but impacts throughput. + +4. **Retention metadata** – Should the `AuditRecord` carry a `RetentionPolicy` + field (e.g. `keep_for: 7_years`) so the sink can enforce retention + automatically? This is application-domain logic that may be out of scope for + OTel. + +5. **Redaction / PII** – ISO 27001 requires protecting logs from unauthorised + access but does not require storing PII in plain text. How should the signal + interact with attribute-level encryption or redaction at the SDK layer? + +6. **Multi-sink fan-out** – Some compliance frameworks require logs to be + written to two independent sinks simultaneously. Should this be a first-class + configuration option of `AuditProvider`? + +## Prototypes + +No prototype exists yet. A reference implementation is planned for: + +- **Java** – extending the existing `io.opentelemetry:opentelemetry-sdk-logs` + module. +- **Go** – extending `go.opentelemetry.io/otel/sdk/log`. + +## Future possibilities + +- **Semantic conventions** for common audit event names + (`user.login.success`, `resource.delete`, `config.change`, ...) following the + same pattern as existing OTel semantic conventions. +- **Integrity chain** – each record carries the hash of the previous record + (blockchain-style) so gaps in the audit trail are detectable without a + centralized sequence counter. +- **Declarative configuration** support so that audit pipeline parameters + (sink URL, queue depth, retry policy, retention) can be specified in the + [OTel configuration file schema](https://github.com/open-telemetry/opentelemetry-configuration). +- **Collector support** – a dedicated `auditlog` receiver/exporter pair in + the OpenTelemetry Collector that enforces the no-drop guarantee end-to-end. diff --git a/specification/audit/README.md b/specification/audit/README.md new file mode 100644 index 00000000000..23f973cae35 --- /dev/null +++ b/specification/audit/README.md @@ -0,0 +1,163 @@ + + +# OpenTelemetry Audit Logging + +**Status**: [Development](../document-status.md) + +
+Table of Contents + + + +- [OpenTelemetry Audit Logging](#opentelemetry-audit-logging) + - [Introduction](#introduction) + - [Why a Dedicated Audit Logging Signal?](#why-a-dedicated-audit-logging-signal) + - [Compliance Use Cases](#compliance-use-cases) + - [Signal Overview](#signal-overview) + - [Relationship to the Log Signal](#relationship-to-the-log-signal) + - [Specifications](#specifications) + - [References](#references) + + + +
+ +## Introduction + +Audit logging is a compliance-critical signal that records +security-relevant events for regulatory and accountability purposes. +Compliance frameworks such as ISO 27001, SOC 2, PCI-DSS, and HIPAA +require that audit records be delivered without loss, modification, or +sampling to a tamper-protected audit sink. + +The OpenTelemetry Audit Logging signal provides a purpose-built pipeline +with the delivery and integrity guarantees that compliance frameworks +demand. It extends OpenTelemetry's unified observability approach to +the domain of security and compliance auditing. + +## Why a Dedicated Audit Logging Signal? + +The existing OpenTelemetry [Log signal](../logs/README.md) is designed +for general-purpose observability. Its pipeline is intentionally subject +to behaviours that are incompatible with audit logging: + +- **Sampling** – processors and exporters may drop records for + performance reasons. +- **Back-pressure shedding** – SDK queues may overflow and silently + discard records. +- **Transformation** – processors may modify, redact, or aggregate + records before export. + +For audit logging, every record MUST be delivered to the designated audit +sink without modification or loss. A feature flag on a `LogRecord` cannot +fulfil this requirement because it still shares the existing pipeline and +inherits all the above behaviours. + +A dedicated Audit Logging signal provides: + +- A purpose-built SDK pipeline with no sampling, no transformation, and + at-least-once delivery semantics. +- A distinct OTLP endpoint (`/v1/audit`) so the audit sink can be + isolated from the observability backend, enabling independent access + control and QoS policies. +- A dedicated data model with mandatory integrity fields. +- Clear separation of audit records from operational logs, as required + by ISO 27001 Annex A and similar frameworks. + +## Compliance Use Cases + +The following table illustrates representative audit events and the +compliance drivers that motivate them: + +| Actor | Audit event | Compliance driver | +|-----------------|-------------------------------|------------------------------| +| End user | Successful / failed login | ISO 27001 A.8.15, SOC 2 CC6 | +| End user | Access to sensitive data | GDPR Art. 30, HIPAA §164.312 | +| Operator | Configuration change | ISO 27001 A.8.15 | +| Service account | Elevated-privilege action | ISO 27001 A.5.18 | +| Service | Outbound call to external API | PCI-DSS Req. 10 | +| Runtime | Antivirus / IDs rule fired | ISO 27001 A.8.16 | + +## Signal Overview + +The Audit Logging signal is modelled structurally after the Log signal. +It reuses the OTLP `LogRecord` as its wire-format transport and adds a +small set of mandatory fields that satisfy compliance requirements. + +``` +Application code + │ + ▼ + AuditLogger.emit(AuditRecord) ──► returns AuditReceipt + │ RecordId (caller-generated) + ▼ + AuditProvider (no sampling, no dropping) + │ └── AuditRecordProcessor (enrich only – never redact or drop) + │ ├── Simple processor (sync, default) + │ ├── Batching processor (async, high-volume) + │ └── Signing processor (adds IntegrityValue) + ▼ + AuditExporter ──► Tier-2 Collector (optional) ──► Audit sinks + or ──► Audit sink directly │ + │ (OpenSearch, Splunk, S3 WORM …) + └── returns AuditReceipt + (RecordId + IntegrityHash + SinkTimestamp) +``` + +The principal API components are: + +- **AuditProvider** – the entry point of the API. It provides named + `AuditLogger` instances and MUST NOT expose a sampler configuration + option. See [Audit Logging API](./api.md#auditprovider). +- **AuditLogger** – emits `AuditRecord`s and returns an `AuditReceipt` + once the sink has acknowledged the record. + See [Audit Logging API](./api.md#auditlogger). + +The principal data components are: + +- **AuditRecord** – the audit event payload containing mandatory fields + for actor, action, outcome, timestamps, and optional integrity + metadata. See [Audit Record Data Model](./data-model.md#auditrecord-definition). +- **AuditReceipt** – proof-of-delivery returned by the sink, containing + a unique record identifier and a SHA-256 integrity hash. + See [Audit Record Data Model](./data-model.md#auditreceipt-definition). + +## Relationship to the Log Signal + +The Audit Logging signal is **independent** of the Log signal. Audit +records MUST NOT be routed through the standard `LoggerProvider` +pipeline. + +An application or instrumentation library MAY emit both an `AuditRecord` +(for compliance) and a regular `LogRecord` (for observability) for the +same event. The two records travel through independent pipelines to +independent backends. + +The OTLP wire format reuses the `LogRecord` protobuf message as the +payload carrier. The following OTLP envelope layers apply: + +| OTLP layer | Role in audit logging | +|------------------------|--------------------------------------------------------| +| `Resource` | Emitting service / host; integrity attrs on Resource. | +| `LogRecord` | Carries AuditRecord via dedicated fields + Attributes. | +| `Attributes` | Mandatory and optional `audit.*` semantic attributes. | +| `InstrumentationScope` | **Not applicable.** MUST be left empty by exporters. | + +## Specifications + +- [Audit Logging API](./api.md) +- [Audit Logging SDK](./sdk.md) +- [Audit Record Data Model](./data-model.md) +- [Audit Logging Collector (Tier-2)](./collector.md) + +## References + +- [OTEP 0267 – Audit Logging Signal](../../oteps/0267-audit-logging.md) +- [OTEP 0092 – OpenTelemetry Logs Vision](../../oteps/logs/0092-logs-vision.md) +- [OTEP 0097 – Log Data Model](../../oteps/logs/0097-log-data-model.md) +- [OTEP 0202 – Events and Logs API](../../oteps/0202-events-and-logs-api.md) diff --git a/specification/audit/api.md b/specification/audit/api.md new file mode 100644 index 00000000000..7fa6331ed31 --- /dev/null +++ b/specification/audit/api.md @@ -0,0 +1,204 @@ + + +# Audit Logging API + +**Status**: [Development](../document-status.md) + +
+Table of Contents + + + +- [Audit Logging API](#audit-logging-api) + - [AuditProvider](#auditprovider) + - [AuditProvider operations](#auditprovider-operations) + - [Get an AuditLogger](#get-an-auditlogger) + - [AuditLogger](#auditlogger) + - [Emit an AuditRecord](#emit-an-auditrecord) + - [Optional and required parameters](#optional-and-required-parameters) + - [Concurrency requirements](#concurrency-requirements) + - [References](#references) + + +
+ +The Audit Logging API provides application code with a means to emit +[`AuditRecord`s](./data-model.md#auditrecord-definition) that are guaranteed to +reach the configured audit sink without loss or modification. + +The API consists of these main components: + +- [**AuditProvider**](#auditprovider) – the entry point of the API. + Provides access to `AuditLogger` instances. +- [**AuditLogger**](#auditlogger) – emits `AuditRecord`s and returns + an [`AuditReceipt`](./data-model.md#auditreceipt-definition). + +```mermaid +graph TD + A[AuditProvider] -->|Get| B(AuditLogger) + B -->|Emit| C(AuditRecord) + C -->|returns| D(AuditReceipt) +``` + +Unlike the [Logs API](../logs/api.md), the Audit Logging API: + +- Does **not** provide an `Enabled` check: audit records are always + emitted regardless of configuration, because dropping audit records + is prohibited. +- Does **not** accept a sampling-related configuration: the pipeline + MUST NOT sample or drop records. +- Has `emit` return an [`AuditReceipt`](./data-model.md#auditreceipt-definition) + as proof-of-delivery once the sink acknowledges the record. + +## AuditProvider + +`AuditLogger`s are obtained from an `AuditProvider`. + +The `AuditProvider` is expected to be accessed from a central place. +The API SHOULD provide a way to set and access a global default +`AuditProvider`. + +### AuditProvider operations + +The `AuditProvider` MUST provide the following function: + +- Get an `AuditLogger` + +#### Get an AuditLogger + +This API MUST accept the following parameter: + +- `name` (required): A string identifying the component, library, or + subsystem that is emitting audit records (for example, + `"com.example.auth"` or `"payment-service"`). The name is used as a + diagnostic label and is stored as an SDK-internal marker. It MUST NOT + be empty. If an empty string is provided, the SDK SHOULD log a + warning and use a fallback name rather than failing. + + Note: unlike `LoggerProvider.GetLogger`, this `name` is NOT mapped + to an OTLP `InstrumentationScope`. See + [README – OTLP Envelope Layers](./README.md#relationship-to-the-log-signal). + +This API MAY accept the following optional parameters: + +- `version` (optional): A string specifying the version of the + emitting component (for example, `"1.2.3"`). +- `schema_url` (optional): A Schema URL to be recorded in emitted + records for semantic convention versioning. + +The term *identical* applied to `AuditLogger`s describes instances +where all parameters are equal. The term *distinct* describes instances +where at least one parameter has a different value. + +## AuditLogger + +The `AuditLogger` is responsible for emitting `AuditRecord`s to the +audit pipeline. + +The `AuditLogger` MUST provide a function to: + +- [Emit an `AuditRecord`](#emit-an-auditrecord) + +The `AuditLogger` MUST NOT provide an `Enabled` or sampling-related +function. Conditional emission is the sole responsibility of +application code; the SDK MUST NOT silently suppress records based on +any sampling or filtering configuration. + +### Emit an AuditRecord + +The effect of calling this API is to submit an `AuditRecord` to the +audit pipeline and to block until the audit sink acknowledges receipt, +returning an `AuditReceipt`. + +The API MUST accept the following parameters: + +- [`Timestamp`](./data-model.md#logrecord-field-usage) (required): the + time at which the auditable action occurred. +- [`EventName`](./data-model.md#logrecord-field-usage) (required): the + semantic name of the audit event. +- [`Attributes`](./data-model.md#audit-semantic-attributes) (required): + a map of key-value pairs that carries all audit-specific data. + The following attribute keys MUST be present and non-empty: + + | Attribute key | Type | Description | + |--------------------|----------|-------------------------------------| + | `audit.actor.id` | `string` | Identity that performed the action. | + | `audit.actor.type` | `string` | `user`, `service`, or `system`. | + | `audit.action` | `string` | Verb describing what was done. | + | `audit.outcome` | `string` | `success`, `failure`, or `unknown`. | + + The attribute key `audit.record.id` is also required in every + persisted record. If the caller omits it, the SDK MUST generate a + UUID v4 and inject it into `Attributes` before any further + processing. The value MUST be stable across retries of the same + event. + +The API MUST accept the following optional parameters: + +- [`ObservedTimestamp`](./data-model.md#logrecord-field-usage) + (optional): if not provided, the SDK MUST set it to the current time. +- [`Body`](./data-model.md#logrecord-field-usage) (optional): free-form + or structured event details. + +Additional optional attributes MAY be provided in `Attributes`. See +the [Audit Record Data Model](./data-model.md#audit-semantic-attributes) +for the full list, which includes `audit.target.id`, `audit.target.type`, +`audit.source.id`, `audit.source.type`, `audit.integrity.value`, +`audit.sequence.number`, +`audit.sequence.prev_hash`, and `audit.schema.version`. + +**Return value**: The API MUST return an +[`AuditReceipt`](./data-model.md#auditreceipt-definition) when the +audit sink acknowledges that the record has been persisted. The receipt +echoes the caller's `audit.record.id` and contains an `IntegrityHash` +and a `SinkTimestamp`. + +**Delivery semantics**: `emit` is synchronous by default. It MUST block +the calling thread until the exporter receives a successful +acknowledgement from the audit sink. This guarantees at-least-once +delivery and provides the caller with a `RecordId` that can be logged +or stored for audit trail completeness. + +An asynchronous variant MAY be provided by the SDK. An asynchronous +`emit`: + +- MUST still guarantee at-least-once delivery through an internal + durable buffer. +- MUST NOT silently discard records on failure. +- MAY return a future, promise, or channel through which the + `AuditReceipt` will be delivered once the sink acknowledges the + record. + +**Error handling**: If the audit sink cannot be reached and the SDK's +retry budget is exhausted, `emit` MUST surface a hard error to the +caller (for example, by raising an exception or returning an error +code). It MUST NOT return silently or return a partial / empty receipt. + +## Optional and required parameters + +Required parameters are those not marked optional in the +[Emit an AuditRecord](#emit-an-auditrecord) section. The API MUST be +structured to require these parameters (for example, as positional +arguments or a required record type). + +For each optional parameter, the API MUST be structured to accept it, +but MUST NOT require the caller to provide it. + +## Concurrency requirements + +For languages that support concurrent execution, the Audit Logging API +provides the following guarantees: + +- **AuditProvider** – all methods MUST be documented as safe for + concurrent use. +- **AuditLogger** – `emit` MUST be documented as safe for concurrent + use. Concurrent `emit` calls MUST be serialized internally by the SDK + so that records are enqueued in the order they are received. + +## References + +- [OTEP 0267 – Audit Logging Signal](../../oteps/0267-audit-logging.md) +- [OTEP 0202 – Events and Logs API](../../oteps/0202-events-and-logs-api.md) diff --git a/specification/audit/collector.md b/specification/audit/collector.md new file mode 100644 index 00000000000..959199a55fc --- /dev/null +++ b/specification/audit/collector.md @@ -0,0 +1,306 @@ + + +# Audit Logging Collector (Tier-2) + +**Status**: [Development](../document-status.md) + +
+Table of Contents + + + +- [Audit Logging Collector (Tier-2)](#audit-logging-collector-tier-2) + - [Overview](#overview) + - [Endpoint Definition](#endpoint-definition) + - [Single-Record Ingest](#single-record-ingest) + - [Batch Ingest](#batch-ingest) + - [Request Model](#request-model) + - [Required Record Fields](#required-record-fields) + - [Optional Record Fields](#optional-record-fields) + - [Response Model](#response-model) + - [Single-Record Response Fields](#single-record-response-fields) + - [Batch Response Fields](#batch-response-fields) + - [Domain Status Values](#domain-status-values) + - [HTTP Status Codes](#http-status-codes) + - [Idempotency and Duplicate Handling](#idempotency-and-duplicate-handling) + - [Verification and Sink Delivery](#verification-and-sink-delivery) + - [Integrity Verification](#integrity-verification) + - [Hash-Chain Validation](#hash-chain-validation) + - [Per-Sink Delivery Status](#per-sink-delivery-status) + - [Record Lifecycle](#record-lifecycle) + - [Retry Guidance](#retry-guidance) + - [Security Requirements](#security-requirements) + - [References](#references) + + + +
+ +## Overview + +This document specifies the behavior of an OpenTelemetry Collector +acting as a Tier-2 intermediary in an audit logging pipeline. The +collector receives `AuditRecord`s from SDK exporters (Tier-1), +verifies their integrity, and distributes them to one or more required +audit sinks. + +The Tier-2 collector is optional. SDK exporters MAY deliver directly +to the audit sink when no collector is deployed. When a collector is +present it MUST satisfy the same no-drop and at-least-once delivery +guarantees as the SDK itself. + +``` +SDK Exporter ──POST /v1/audit──► Tier-2 Collector + │ + verify integrity + validate hash-chain + │ + ┌────────────┼────────────┐ + ▼ ▼ ▼ + Primary sink Secondary SIEM / archive + (OpenSearch) (S3 WORM) (Splunk) +``` + +## Endpoint Definition + +The Tier-2 collector MUST expose an HTTP endpoint that accepts audit +records from SDK exporters. + +### Single-Record Ingest + +- **Path**: `/v1/audit` (MUST; matches the SDK exporter default). +- **Method**: `POST`. +- **Body**: one serialized `AuditRecord` payload. + +### Batch Ingest + +- **Path**: `/v1/audit` (same endpoint; batch indicated by content). +- **Method**: `POST`. +- **Body**: batch payload containing multiple `AuditRecord`s. + +The collector MUST support both single-record and batch requests on +the same endpoint. The request content type MUST follow the OTLP +serialization convention (protobuf or JSON). + +## Request Model + +Request payloads MUST follow the +[Audit Record Data Model](./data-model.md). + +### Required Record Fields + +Each record MUST include the following LogRecord fields: + +- `Timestamp` +- `EventName` + +And the following mandatory attributes in the `Attributes` map: + +- `audit.record.id` +- `audit.actor.id` +- `audit.actor.type` +- `audit.action` +- `audit.outcome` + +### Optional Record Fields + +Each record MAY include any attribute defined in the +[Audit Semantic Attributes](./data-model.md#audit-semantic-attributes) +section, including `audit.schema.version`, `audit.integrity.value`, +`audit.sequence.number`, and `audit.sequence.prev_hash`. The signing algorithm +and key reference are carried as `Resource` attributes +`audit.integrity.algorithm` and `audit.integrity.certificate`. + +## Response Model + +Tier-2 responses MUST include: + +- A transport-level HTTP status code. +- A domain-level processing status in the response body. + +### Single-Record Response Fields + +- `record_id` – echoes the `RecordId` from the request. +- `status` – domain status value (see [Domain Status Values](#domain-status-values)). +- `integrity_hash` – SHA-256 of the record as persisted by the + collector (same semantics as `AuditReceipt.IntegrityHash`). +- `sink_timestamp` – nanoseconds since UNIX epoch when the collector + persisted the record to all required sinks. +- `verify_status` – `passed`, `failed`, or `deferred`. +- `sink_status_map` – map of sink name → delivery status (see + [Per-Sink Delivery Status](#per-sink-delivery-status)). +- `reason` – human-readable explanation when `status` is not success. +- `retry_after` – seconds to wait before retrying, when applicable. + +### Batch Response Fields + +- `batch_id` – collector-assigned identifier for the batch. +- `status` – aggregate domain status. +- `record_status_map` – map of `RecordId` → single-record response. +- `sink_status_aggregate` – aggregate delivery status across all sinks. + +### Domain Status Values + +The collector SHOULD use the following domain status values: + +- `accepted_pending_verify` – received; integrity check in progress. +- `verified_queued` – integrity passed; queued for sink delivery. +- `verified_exporting` – actively delivering to required sinks. +- `delivered_all_sinks` – every required sink confirmed receipt. +- `delivered_partial` – at least one required sink failed; treated as + non-success (see [Verification and Sink Delivery](#verification-and-sink-delivery)). +- `rejected_verify_failed` – integrity check failed. +- `rejected_schema_invalid` – schema validation failed. +- `rejected_authz` – authentication or authorisation failure. +- `failed_internal` – unexpected internal collector error. + +## HTTP Status Codes + +The collector SHOULD use the following HTTP status codes: + +| Code | Meaning | +|---------------|----------------------------------------------------------------| +| `200` | Fully processed; all required sinks confirmed. | +| `202` | Accepted for async verify or export; final state pending. | +| `400` | Malformed payload, schema mismatch, or canonicalization error. | +| `401` / `403` | Authentication or authorisation failure. | +| `409` | Duplicate `RecordId` with differing payload hash. | +| `413` | Payload too large. | +| `429` | Intake throttled by policy. | +| `500` | Unexpected internal error. | +| `503` | Temporarily unavailable or no reliable intake capacity. | + +The collector MUST NOT return HTTP 207 (Multi-Status) to indicate +partial success. If any required sink fails the collector MUST reject +the entire batch (see +[No partial success](./sdk.md#no-partial-success)). + +## Idempotency and Duplicate Handling + +The collector MUST treat `audit.record.id` as the primary idempotency +key. + +When a duplicate `audit.record.id` is received: + +- If the canonical payload hash is **identical**, the collector SHOULD + return a deterministic success response for the existing record + without creating a duplicate entry. +- If the canonical payload hash **differs**, the collector MUST return + HTTP 409. The SDK exporter MUST treat HTTP 409 as a non-retryable + `Failure`. + +## Verification and Sink Delivery + +### Integrity Verification + +When `audit.integrity.value` is present the collector SHOULD verify +it against the algorithm declared in the `Resource` attribute +`audit.integrity.algorithm` and the key material referenced by +`audit.integrity.certificate` in the collector's trust policy. +Before verifying, the collector MUST serialize the `AuditRecord` to +JSON and canonicalize it using +[RFC 8785 – JSON Canonicalization Scheme (JCS)][rfc8785]; the +`audit.integrity.*` attributes MUST be excluded from the canonical +form — they carry the proof itself and MUST NOT be part of the +verified payload. The canonical byte sequence of the remaining +record is the input to the verification operation. + +[rfc8785]: https://www.rfc-editor.org/rfc/rfc8785 + +The collector MAY defer verification for low-latency ingest (returning +`accepted_pending_verify`) but MUST complete verification before +acknowledging delivery to any required sink. + +If verification fails the collector MUST: + +1. Reject the record with domain status `rejected_verify_failed`. +2. NOT forward the record to any sink. +3. Return HTTP 400 with a descriptive `reason`. + +### Hash-Chain Validation + +When `audit.sequence.number` and `audit.sequence.prev_hash` are present the +collector SHOULD validate chain continuity: + +- `audit.sequence.number` MUST be strictly greater than the previous + record's `audit.sequence.number` in the same audit stream. +- `audit.sequence.prev_hash` MUST equal the `IntegrityHash` of the preceding + record. + +A broken chain SHOULD be surfaced as a warning metric and logged as a +security event. The collector MAY still accept and forward a broken +chain record but MUST annotate the delivery status with a chain +validation warning. + +### Per-Sink Delivery Status + +For each required sink the collector SHOULD report: + +- `exporter_name` – configured name of the sink exporter. +- `sink_type` – type of sink (e.g. `opensearch`, `s3`, `splunk`). +- `delivery_status` – `success`, `failed`, `timeout`, `retrying`, or + `unknown`. +- `attempt_count` – number of delivery attempts made. +- `last_attempt_at` – timestamp of the most recent attempt. +- `sink_ack_id` – acknowledgement identifier returned by the sink, if + available. +- `error_code` – sink-specific error code on failure. +- `error_message` – human-readable error description on failure. + +A record outcome is successful only when every required sink reports +`delivery_status: success`. + +### Record Lifecycle + +Recommended state transitions: + +``` +received + → accepted_pending_verify + → verified_queued + → verified_exporting + → delivered_all_sinks (terminal: success) +``` + +Alternative terminal states: + +- `rejected_schema_invalid` +- `rejected_verify_failed` +- `rejected_authz` +- `delivered_partial` (treated as non-success) +- `failed_internal` + +## Retry Guidance + +SDK exporters SHOULD use the `retry_after` field when present. + +Retry-ability by status code: + +| Status | Behaviour | +|-----------------------------------|--------------------------------------| +| `202`, `429`, `503` | Retryable with exponential back-off. | +| `400`, `401`, `403`, `409`, `413` | Non-retryable. | +| `500` | Implementation and policy dependent. | + +## Security Requirements + +The collector ingress MUST require secure transport and SHOULD support: + +- TLS with server authentication (minimum TLS 1.2). +- mTLS for stronger client identity where required. +- Token-based authentication (e.g. Bearer token) where mTLS is not + used. + +The collector SHOULD validate `IntegrityValue` according to the +algorithm in `audit.integrity.algorithm` and the configured +verification policy before forwarding records to sinks. + +## References + +- [Audit Logging API](./api.md) +- [Audit Logging SDK](./sdk.md) +- [Audit Record Data Model](./data-model.md) +- [OTEP 0267 – Audit Logging Signal](../../oteps/0267-audit-logging.md) diff --git a/specification/audit/data-model.md b/specification/audit/data-model.md new file mode 100644 index 00000000000..d672f931e9e --- /dev/null +++ b/specification/audit/data-model.md @@ -0,0 +1,585 @@ + + +# Audit Record Data Model + +**Status**: [Development](../document-status.md) + +
+Table of Contents + + + +- [Audit Record Data Model](#audit-record-data-model) + - [Design Notes](#design-notes) + - [Requirements](#requirements) + - [Relationship to LogRecord](#relationship-to-logrecord) + - [OTLP Envelope Layers](#otlp-envelope-layers) + - [AuditRecord Definition](#auditrecord-definition) + - [LogRecord Field Usage](#logrecord-field-usage) + - [Field: `Timestamp`](#field-timestamp) + - [Field: `ObservedTimestamp`](#field-observedtimestamp) + - [Field: `EventName`](#field-eventname) + - [Field: `Body`](#field-body) + - [Field: `Attributes`](#field-attributes) + - [Audit Semantic Attributes](#audit-semantic-attributes) + - [Mandatory Attributes](#mandatory-attributes) + - [Attribute: `audit.record.id`](#attribute-auditrecordid) + - [Actor Attributes](#actor-attributes) + - [Attribute: `audit.action`](#attribute-auditaction) + - [Attribute: `audit.outcome`](#attribute-auditoutcome) + - [Optional Attributes](#optional-attributes) + - [Target Attributes](#target-attributes) + - [Source Attributes](#source-attributes) + - [Integrity Attributes](#integrity-attributes) + - [Ordering Attributes](#ordering-attributes) + - [Integrity Resource Attributes](#integrity-resource-attributes) + - [Attribute: `audit.integrity.algorithm`](#attribute-auditintegrityalgorithm) + - [Attribute: `audit.integrity.certificate`](#attribute-auditintegritycertificate) + - [AuditReceipt Definition](#auditreceipt-definition) + - [Field: `RecordId`](#field-recordid) + - [Field: `IntegrityHash`](#field-integrityhash) + - [Field: `SinkTimestamp`](#field-sinktimestamp) + - [Example AuditRecords](#example-auditrecords) + - [Successful user login](#successful-user-login) + - [Failed privileged configuration change](#failed-privileged-configuration-change) + - [References](#references) + + + +
+ +## Design Notes + +### Requirements + +The Audit Record Data Model was designed to satisfy the following +requirements: + +- Every audit record MUST carry sufficient information to identify the + actor, the action performed, the target of the action, and the + outcome. This is the minimum required by ISO 27001 Annex A (event + logging) and equivalent compliance frameworks. + +- The data model MUST include timestamps that allow clock-skew detection + across distributed services (ISO 27001 – clock synchronisation). + +- The data model MUST support optional cryptographic integrity proofs + (asymmetric digital signatures or symmetric HMACs) so that records + can be verified for integrity after delivery. + +- The data model MUST be efficiently representable in OTLP by reusing + the existing `LogRecord` proto message as a transport container. + +- The data model MUST be extensible via arbitrary key-value attributes + to accommodate application-specific compliance requirements. + +### Relationship to LogRecord + +An `AuditRecord` is a `LogRecord` that uses the standard dedicated +`LogRecord` fields for the universal concepts they represent, and +carries all audit-specific information as `Attributes` following the +`audit.*` semantic convention namespace. + +This follows the established OTel convention: universal concepts +(timestamps, event name, body, context) receive dedicated top-level +`LogRecord` fields; signal- and domain-specific information is +expressed as semantic-convention attributes. Audit records are +therefore processable by any OTel tooling that understands `LogRecord`s. + +The `SeverityNumber` and `SeverityText` fields of the underlying +`LogRecord` SHOULD NOT be set for audit records. Severity is not a +meaningful concept for audit events; the `audit.outcome` attribute +serves a structurally similar purpose. + +### OTLP Envelope Layers + +The following OTLP envelope layers apply to audit records: + +| OTLP layer | Role in audit logging | +|------------------------|------------------------------------------------------------------| +| `Resource` | Emitting service / host; integrity attrs (see below). | +| `LogRecord` | Carries the `AuditRecord` via dedicated fields and `Attributes`. | +| `InstrumentationScope` | **Not applicable.** MUST be left empty by exporters. | + +The `Resource` also carries `audit.integrity.algorithm` and +`audit.integrity.certificate` — see +[Integrity Resource Attributes](#integrity-resource-attributes). + +## AuditRecord Definition + +An `AuditRecord` is a single security-relevant event emitted by +application or system code on behalf of an actor. It is transported +as a `LogRecord`; the following two tables summarise how `LogRecord` +fields are used and which semantic attributes carry audit-specific +data. + +### LogRecord Field Usage + +| LogRecord Field | Audit Usage | Required | +|------------------------|----------------------------------------|--------------------------------| +| `Timestamp` | Time the auditable action occurred | MUST be set | +| `ObservedTimestamp` | Time the SDK observed the event | MUST be set | +| `EventName` | Semantic name of the audit event type | MUST be set; MUST NOT be empty | +| `Body` | Free-form or structured event details | MAY be set | +| `Attributes` | Audit semantic attributes (see below) | MUST contain mandatory attrs | +| `Resource` | Emitting service / host metadata | MUST be set | +| `TraceId` | Correlation to an active trace, if any | MAY be set | +| `SpanId` | Correlation to an active span, if any | MAY be set | +| `SeverityNumber` | Not applicable for audit records | SHOULD NOT be set | +| `SeverityText` | Not applicable for audit records | SHOULD NOT be set | +| `InstrumentationScope` | Not applicable for audit records | MUST be left empty | + +#### Field: `Timestamp` + +| Property | Value | +|------------|----------------------------------| +| Type | `fixed64` | +| Required | MUST be set | +| Constraint | `Timestamp <= ObservedTimestamp` | + +The time at which the audit event occurred, expressed as nanoseconds +since the UNIX epoch (UTC). The application MUST set this field to the +time the auditable action actually took place. + +If the application cannot determine the precise event time, it SHOULD +use the current time at the moment of the `emit` call and document +this in the `Body` or `Attributes`. + +The clock used for `Timestamp` MUST be a wall-clock source +synchronised via NTP or an equivalent protocol. The SDK SHOULD warn at +startup if the system clock offset exceeds one second. + +#### Field: `ObservedTimestamp` + +| Property | Value | +|------------|----------------------------------------------------| +| Type | `fixed64` | +| Required | MUST be set | +| Constraint | `ObservedTimestamp >= Timestamp` and `<= now(UTC)` | + +The time at which the OpenTelemetry SDK observed the event, expressed +as nanoseconds since the UNIX epoch (UTC). The SDK MUST set this field +to the current time when `emit` is called if the application does not +provide it. + +`ObservedTimestamp` enables the audit sink to detect clock skew between +the emitting service and the SDK host. A significant difference between +`Timestamp` and `ObservedTimestamp` SHOULD be treated as a clock +synchronisation warning. + +#### Field: `EventName` + +| Property | Value | +|----------|--------------------------------| +| Type | `string` | +| Required | MUST be set; MUST NOT be empty | + +A short, dot-separated semantic name that uniquely identifies the type +of audit event. `EventName` MUST be stable across releases and SHOULD +follow a hierarchical naming convention, for example: + +- `user.login.success` +- `user.login.failure` +- `resource.file.read` +- `config.change` +- `privilege.escalation` + +Semantic conventions for common `EventName` values SHOULD be defined +in the +[OpenTelemetry Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/). + +#### Field: `Body` + +| Property | Value | +|----------|------------| +| Type | `AnyValue` | +| Required | MAY be set | + +Free-form additional information about the audit event. The value MAY +be a string, a byte buffer, or a structured object. Applications SHOULD +store information that does not fit naturally into the named mandatory +attributes here. + +The `Body` MUST NOT duplicate information already present in the +mandatory `Attributes`. + +#### Field: `Attributes` + +| Property | Value | +|----------|-----------------------------------------| +| Type | `map` | +| Required | MUST contain mandatory audit attributes | + +Key-value pairs that carry audit-specific context. The mandatory and +optional attribute names are defined in +[Audit Semantic Attributes](#audit-semantic-attributes) below. +Applications MAY add additional attributes beyond those listed here +using standard OTel +[attribute naming conventions](../common/attribute-naming.md). + +Attribute keys MUST be unique within a record. If the same key is set +multiple times, the last value MUST be used. + +### Audit Semantic Attributes + +All audit-specific data that does not map to a standard `LogRecord` +dedicated field is expressed as `Attributes` following the `audit.*` +semantic convention namespace, or by reusing existing semantic +convention attributes where a direct mapping exists. + +#### Mandatory Attributes + +| Attribute name | Type | Required | Description | +|--------------------|----------|----------|------------------------------------------------| +| `audit.record.id` | `string` | MUST | Unique stable identifier; UUID v4 recommended. | +| `audit.actor.id` | `string` | MUST | Identity that performed the action. | +| `audit.actor.type` | `string` | MUST | `user`, `service`, or `system`. | +| `audit.action` | `string` | MUST | Verb describing what was done. | +| `audit.outcome` | `string` | MUST | `success`, `failure`, or `unknown`. | + +#### Attribute: `audit.record.id` + +A caller-generated unique identifier for this `AuditRecord`. The value +MUST be immutable once set and MUST remain stable across retries so +that receivers can deduplicate records delivered more than once. + +The caller SHOULD use a UUID v4 or an equivalent unpredictable unique +identifier. The SDK MUST generate a UUID v4 if the caller does not +provide one. + +`audit.record.id` is echoed back unchanged in the +[`AuditReceipt`](#auditreceipt-definition), allowing the emitting +application to correlate receipts with the original `emit` call. + +#### Actor Attributes + +Actor attributes identify the entity that performed the auditable +action. Together, `audit.actor.id` and `audit.actor.type` provide the +minimum identity context required by ISO 27001 Annex A item 1 (user +IDs). + +**`audit.actor.id`** + +The identity of the entity that performed the action. This MAY be a +user ID (string or integer cast to string), a service account name, +or a fully qualified system identifier. If the actor identity cannot +be determined (for example, an unauthenticated request), +`audit.actor.id` MUST still be set to a value indicating the unknown +state (e.g. `"anonymous"` or `"unknown"`). + +Applications where the actor is an authenticated end user MAY set +`enduser.id` in addition to `audit.actor.id` when the values differ +(for example, when `audit.actor.id` is a stable internal identifier +and `enduser.id` is the authentication subject claim). + +**`audit.actor.type`** + +Classifies the kind of entity that performed the action: + +| Value | Meaning | +|-----------|--------------------------------------------------------| +| `user` | A human user, identified by a user account. | +| `service` | An automated service, daemon, or service account. | +| `system` | The operating system or a privileged system component. | + +Applications SHOULD choose the most appropriate value based on context. +Use `service` for non-human actors that operate autonomously, and +`user` for human-initiated actions, even if performed by an AI agent +on behalf of a user. + +#### Attribute: `audit.action` + +A short verb or verb phrase that describes what the actor did. +Applications SHOULD use uppercase verbs to distinguish action values +from event names. Common values include: + +`LOGIN`, `LOGOUT`, `READ`, `WRITE`, `CREATE`, `UPDATE`, `DELETE`, +`EXECUTE`, `APPROVE`, `REJECT`, `EXPORT`, `IMPORT`. + +Applications MAY define domain-specific action verbs. Action names +SHOULD be stable across releases. + +#### Attribute: `audit.outcome` + +The result of the auditable action: + +| Value | Meaning | +|-----------|--------------------------------------------------------------| +| `success` | The action completed successfully. | +| `failure` | The action was attempted but did not complete successfully. | +| `unknown` | The outcome could not be determined at the time of emission. | + +ISO 27001 Annex A items 5 and 6 require logging both successful and +unsuccessful access attempts. Using `unknown` SHOULD be avoided except +for fire-and-forget actions where acknowledgement is not possible. + +#### Optional Attributes + +| Attribute name | Type | Required | Description | +|-------------------------|----------|----------|-------------------------------------------------------| +| `audit.target.id` | `string` | SHOULD | Identifier of the resource acted upon. | +| `audit.target.type` | `string` | SHOULD | Type of the target resource. | +| `audit.source.id` | `string` | MAY | Network address or identifier of the source. | +| `audit.source.type` | `string` | MAY | Type of the source (e.g. `ipv4`, `ipv6`, `hostname`). | +| `audit.integrity.value` | `string` | MAY | Base64-encoded cryptographic integrity proof. | +| `audit.sequence.number` | `int` | MAY | Monotonic counter for hash-chain continuity. | +| `audit.sequence.prev_hash` | `string` | MAY | SHA-256 of the previous record in the stream. | +| `audit.schema.version` | `string` | SHOULD | Schema version of the audit payload. | + +#### Target Attributes + +**`audit.target.id`** SHOULD be set to an identifier for the object +upon which the action was performed. Examples: + +- A file path (`/etc/passwd`) +- A database table name (`users`) +- A REST endpoint path (`/api/v1/orders/42`) +- A cloud resource ARN or ID + +**`audit.target.type`** SHOULD classify the type of resource, for +example `file`, `database.table`, `http.endpoint`, `k8s.configmap`. +When applicable, values SHOULD align with existing OTel semantic +convention resource types. + +If the action was not directed at a specific resource (for example, a +system startup event), both attributes MAY be omitted. + +#### Source Attributes + +**`audit.source.id`** SHOULD be set for network-initiated actions (for +example, a login attempt over HTTPS). The value SHOULD be an IPv4 +address in dotted-decimal notation (`203.0.113.42`) or a full IPv6 +address (`2001:db8::1`), or a hostname. + +**`audit.source.type`** SHOULD classify the address format. Recommended +values are `ipv4`, `ipv6`, and `hostname`. When the source cannot be +determined, both attributes MAY be omitted. + +#### Integrity Attributes + +**`audit.integrity.value`** + +A base64-encoded cryptographic integrity proof over the canonical +serialization of the `AuditRecord`. The proof is either an asymmetric +digital signature or a symmetric HMAC, as indicated by the +`audit.integrity.algorithm` `Resource` attribute (see +[Integrity Resource Attributes](#integrity-resource-attributes)). + +Before signing or verifying, the `AuditRecord` MUST be serialized to +JSON and then canonicalized using +[RFC 8785 – JSON Canonicalization Scheme (JCS)][rfc8785]. The +`audit.integrity.*` attributes MUST be excluded from the canonical +form before signing — they carry the proof itself and MUST NOT be +part of the signed payload. The canonical byte sequence of the +remaining record is the input to the signing or HMAC operation. +Implementations MUST NOT use any other serialization or +canonicalization method for this purpose. + +[rfc8785]: https://www.rfc-editor.org/rfc/rfc8785 + +When `audit.integrity.value` is set, `audit.integrity.algorithm` MUST +be set as a `Resource` attribute. + +#### Ordering Attributes + +The optional ordering attributes enable hash-chain validation across a +sequence of `AuditRecord`s. When populated, receivers can detect +whether records have been deleted, inserted, or reordered by verifying +that the sequence numbers are monotonically increasing and that each +`audit.sequence.prev_hash` matches the `IntegrityHash` returned in the preceding +record's `AuditReceipt`. + +Implementations that require strong tamper-evidence for ordered +sequences SHOULD populate both `audit.sequence.number` and +`audit.sequence.prev_hash`. + +**`audit.sequence.number`** + +A monotonically increasing integer counter, scoped to a single +`AuditLogger` instance or a named audit stream. The first record in a +stream SHOULD have `audit.sequence.number` equal to `1`. A gap between +two consecutive values indicates that one or more records were lost or +deleted and SHOULD trigger an alert. + +**`audit.sequence.prev_hash`** + +The `IntegrityHash` of the immediately preceding record in the same +audit stream (as returned in the preceding `AuditReceipt`). The first +record in a stream SHOULD set `audit.sequence.prev_hash` to the SHA-256 hash of +the empty string +(`e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`). + +A `audit.sequence.prev_hash` that does not match the stored `IntegrityHash` of +the previous record indicates tampering and MUST be treated as a +critical integrity violation. + +### Integrity Resource Attributes + +The signing algorithm and key reference are identical for every +`AuditRecord` emitted by a given service instance. They are therefore +carried once as OTel `Resource` attributes instead of being repeated +in every record. + +#### Attribute: `audit.integrity.algorithm` + +| Property | Value | +|----------|--------------------------------------------------------------------------| +| Type | `string` | +| Required | MUST be set if `audit.integrity.value` is set on any record in the batch | + +The algorithm used to compute `audit.integrity.value` for all records +produced by this resource. + +- For asymmetric signatures the value SHOULD be a registered JWA + algorithm identifier, for example `RS256`, `ES256`, or `EdDSA`. +- For symmetric HMACs the value SHOULD be a registered IANA MAC + algorithm identifier, for example `HMAC-SHA256` or `HMAC-SHA512`. + +#### Attribute: `audit.integrity.certificate` + +| Property | Value | +|----------|------------| +| Type | `string` | +| Required | MAY be set | + +A reference to the signing key used to produce `audit.integrity.value`. +MUST NOT be set when `audit.integrity.algorithm` denotes a symmetric +MAC algorithm. + +The value MUST be one of the following forms, listed from most to +least self-contained: + +- **Full certificate** – base64 (standard encoding, no line wrapping) + of the DER-encoded X.509 public-key certificate. Relying parties + can verify `audit.integrity.value` without any out-of-band lookup. +- **Fingerprint** – a hash string in the form `sha256:` or + `sha1:`, where `` is the lowercase hex encoding of + the DER certificate hash **without** any separators + (e.g. `sha256:3a5f9c2e117b4d8f…`). Colons or other separator + characters within the hex portion are NOT permitted. Requires + matching against a locally trusted certificate. +- **Key ID** – an opaque string identifier (e.g. a JWK `kid` claim + such as `key-2024-01`) agreed between emitter and relying party. + Requires key retrieval via JWK Set URI or equivalent. +- **SKI** – Subject Key Identifier as lowercase hex without separators + (e.g. `3cf012ab…`). Requires a PKI lookup. +- **Issuer + Serial** – Issuer Distinguished Name and serial number + separated by a slash (e.g. `CN=MyCA,O=Acme Corp/12345`). Requires + a PKI lookup. + +If `audit.integrity.certificate` is omitted and +`audit.integrity.algorithm` denotes an asymmetric algorithm, the +relying party MUST obtain the public key through a separately +configured trust anchor. + +## AuditReceipt Definition + +An `AuditReceipt` is returned by `AuditLogger.emit` once the audit +sink has successfully persisted the record. It serves as proof-of-delivery +and enables integrity verification by the emitting application. + +| Field | Type | Required | Description | +|-----------------|-----------|-------------|--------------------------------------------------------------| +| `RecordId` | `string` | MUST be set | Echoes the caller's `audit.record.id` attribute. | +| `IntegrityHash` | `string` | MUST be set | SHA-256 of the record as persisted by the sink. | +| `SinkTimestamp` | `fixed64` | MUST be set | Nanoseconds since UNIX epoch when the sink wrote the record. | + +### Field: `RecordId` + +The value of the `audit.record.id` attribute from the corresponding +`AuditRecord` as provided by the caller. The sink MUST echo this value +unchanged. Callers use it to correlate the receipt with the original +`emit` call and to confirm that the correct record was persisted. + +### Field: `IntegrityHash` + +The SHA-256 hash of the canonical serialization of the `AuditRecord` +as it was written to persistent storage, computed by the sink. The +record MUST be serialized to JSON and canonicalized using +[RFC 8785 – JSON Canonicalization Scheme (JCS)][rfc8785] before +hashing. Returned to the emitting application so that it can verify +that the record was not altered between emission and persistence. + +The emitting application SHOULD compute the same hash locally +immediately after calling `emit` and compare it to the received +`IntegrityHash`. A mismatch indicates that the record was altered in +transit or by the sink and SHOULD trigger an alert. + +### Field: `SinkTimestamp` + +Nanoseconds since the UNIX epoch (UTC) at which the audit sink +persisted the record. This value MAY differ from `ObservedTimestamp` +due to buffering or network latency, and MAY be used to detect +abnormally long delivery times. + +## Example AuditRecords + +The examples below show the `LogRecord` representation of an +`AuditRecord`. Resource attributes are shown separately. + +### Successful user login + +```json +{ + "Timestamp": 1714041600000000000, + "ObservedTimestamp": 1714041600001000000, + "EventName": "user.login.success", + "Body": null, + "Attributes": { + "audit.record.id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "audit.actor.id": "u8472", + "audit.actor.type": "user", + "audit.action": "LOGIN", + "audit.outcome": "success", + "audit.target.id": "/api/auth/session", + "audit.target.type": "http.endpoint", + "audit.source.id": "203.0.113.42", + "audit.source.type": "ipv4", + "audit.schema.version": "1.0.0", + "session.id": "sess-abc123", + "tls.version": "TLSv1.3" + } +} +``` + +Resource attributes: + +```json +{ + "service.name": "auth-service", + "service.version": "2.3.1" +} +``` + +### Failed privileged configuration change + +```json +{ + "Timestamp": 1714041700000000000, + "ObservedTimestamp": 1714041700002000000, + "EventName": "config.change", + "Body": "Authorization denied: missing role 'config-writer'", + "Attributes": { + "audit.record.id": "f7e8d9c0-b1a2-3456-cdef-0987654321ab", + "audit.actor.id": "svc-deployer", + "audit.actor.type": "service", + "audit.action": "UPDATE", + "audit.outcome": "failure", + "audit.target.id": "production/app-secrets", + "audit.target.type": "k8s.configmap", + "audit.schema.version": "1.0.0", + "request.id": "req-xyz789", + "k8s.namespace": "production" + } +} +``` + +## References + +- [OTEP 0267 – Audit Logging Signal](../../oteps/0267-audit-logging.md) +- [OTEP 0097 – Log Data Model](../../oteps/logs/0097-log-data-model.md) +- [ISO 27001:2022 Annex A – Audit Logging controls](https://www.iso.org/standard/27001) +- [Common Event Format (CEF) specification](https://www.microfocus.com/documentation/arcsight/arcsight-smartconnectors-8.4/cef-implementation-standard/) diff --git a/specification/audit/sdk.md b/specification/audit/sdk.md new file mode 100644 index 00000000000..81fc3f28dfc --- /dev/null +++ b/specification/audit/sdk.md @@ -0,0 +1,599 @@ + + +# Audit Logging SDK + +**Status**: [Development](../document-status.md) + +
+Table of Contents + + + +- [Audit Logging SDK](#audit-logging-sdk) + - [AuditProvider](#auditprovider) + - [AuditProvider Creation](#auditprovider-creation) + - [AuditLogger Creation](#auditlogger-creation) + - [Configuration](#configuration) + - [No Sampling](#no-sampling) + - [Shutdown](#shutdown) + - [ForceFlush](#forceflush) + - [AuditLogger](#auditlogger) + - [Emit an AuditRecord](#emit-an-auditrecord) + - [AuditRecord Queue](#auditrecord-queue) + - [Durability guarantees](#durability-guarantees) + - [Back-pressure handling](#back-pressure-handling) + - [AuditRecordProcessor](#auditrecordprocessor) + - [Processor restrictions](#processor-restrictions) + - [AuditRecordProcessor operations](#auditrecordprocessor-operations) + - [OnEmit](#onemit) + - [Shutdown](#shutdown-1) + - [ForceFlush](#forceflush-1) + - [Built-in processors](#built-in-processors) + - [Simple processor](#simple-processor) + - [Batching processor](#batching-processor) + - [Signing processor (Sig)](#signing-processor-sig) + - [AuditRecordExporter](#auditrecordexporter) + - [AuditRecordExporter operations](#auditrecordexporter-operations) + - [Export](#export) + - [ForceFlush](#forceflush-2) + - [Shutdown](#shutdown-2) + - [OTLP transport requirements](#otlp-transport-requirements) + - [Dedicated endpoint](#dedicated-endpoint) + - [No partial success](#no-partial-success) + - [Idempotency](#idempotency) + - [InstrumentationScope](#instrumentationscope) + - [Failure handling](#failure-handling) + - [Clock synchronisation](#clock-synchronisation) + - [Interaction with the Log signal](#interaction-with-the-log-signal) + - [Collector pipeline (Tier-2)](#collector-pipeline-tier-2) + - [Observability](#observability) + - [Concurrency requirements](#concurrency-requirements) + - [References](#references) + + + +
+ +The Audit Logging SDK is the implementation of the +[Audit Logging API](./api.md). It provides the guarantee that every +emitted `AuditRecord` is delivered to the configured audit sink without +loss, modification, or sampling. + +All language implementations of OpenTelemetry that support the Audit +Logging signal MUST provide an SDK conforming to this specification. + +## AuditProvider + +A `AuditProvider` MUST provide a way to associate a +[Resource](../resource/sdk.md) with all `AuditRecord`s produced by any +`AuditLogger` obtained from it. + +### AuditProvider Creation + +The SDK SHOULD allow the creation of multiple independent +`AuditProvider` instances. Although most applications will only need one. + +Each `AuditProvider` instance MUST have its own independent queue, +exporter pipeline, and failure counter. + +### AuditLogger Creation + +`AuditLogger` instances MUST only be created through an +`AuditProvider`. + +The `AuditProvider` MUST implement the +[Get an AuditLogger](./api.md#get-an-auditlogger) operation. The +`name` and optional `version` / `schema_url` parameters are stored +internally on the created `AuditLogger` for diagnostic purposes. + +If an invalid `name` (null or empty string) is provided, a working +`AuditLogger` MUST be returned rather than null or an exception. The +`name` SHOULD retain the invalid value and the SDK SHOULD emit a +warning. + +### Configuration + +The `AuditProvider` owns the following configuration: + +- One or more [AuditRecordProcessor](#auditrecordprocessor) instances. +- The [AuditRecord Queue](#auditrecord-queue) parameters. +- The retry and back-pressure policy (see + [Failure handling](#failure-handling)). + +Configuration MUST be applied to all already-returned `AuditLogger`s +when it is changed after provider creation (that is, adding a processor +after the provider is created MUST affect all existing loggers). + +### No Sampling + +The `AuditProvider` MUST NOT accept or expose any sampler or +sampling-rate configuration. Any attempt to configure sampling on the +audit pipeline MUST be rejected at configuration time with an +explanatory error. + +The SDK MUST NOT apply any record filtering, dropping, or aggregation +that reduces the number of `AuditRecord`s reaching the exporter. + +### Shutdown + +On the first call, `Shutdown` MUST: + +1. Call `ForceFlush` to ensure all buffered records are exported. +2. Call `Shutdown` on every registered + `AuditRecordProcessor`. + +`Shutdown` SHOULD complete or abort within a configurable timeout. +After `Shutdown` returns, subsequent calls to `GetAuditLogger` SHOULD +return an error or throw an exception. Any call to `emit` on an +`AuditLogger` obtained from a shut-down `AuditProvider` MUST raise a +hard error or throw an exception and MUST NOT silently drop the record. + +`Shutdown` SHOULD provide a way for the caller to determine whether it +succeeded, failed, or timed out. + +Subsequent calls to `Shutdown` or `ForceFlush` after the first +`Shutdown` call MUST be no-ops and MUST NOT return an error. + +### ForceFlush + +`ForceFlush` requests that all buffered `AuditRecord`s be exported +immediately. + +`ForceFlush` MUST invoke `ForceFlush` on all registered +`AuditRecordProcessor` instances. + +`ForceFlush` SHOULD complete or abort within a configurable timeout and +SHOULD provide a way for the caller to determine whether it succeeded, +failed, or timed out. + +If `Shutdown` has already been called, `ForceFlush` MUST be a no-op +and MUST NOT return an error. + +## AuditLogger + +### Emit an AuditRecord + +If `emit` is called after the owning `AuditProvider` has been shut +down, the SDK MUST raise a hard error or throw an exception. The SDK +MUST NOT silently drop the record. + +When `emit` is called, the SDK MUST: + +1. If `Attributes` does not contain `audit.record.id`, generate a + UUID v4 and inject it into `Attributes` before any further + processing. +2. Set `ObservedTimestamp` to the current time if the caller did not + provide it. +3. Validate that the required LogRecord fields (`Timestamp`, + `EventName`) and the mandatory attributes (`audit.actor.id`, + `audit.actor.type`, `audit.action`, `audit.outcome`) are present + and non-empty. If any required field or attribute is missing, + `emit` MUST surface a hard error to the caller and MUST NOT + silently drop the record. +4. Enqueue the `AuditRecord` in the [AuditRecord Queue](#auditrecord-queue). +5. Pass the record through all registered + `AuditRecordProcessor` instances via + [`OnEmit`](#onemit). +6. Block until the exporter returns a successful acknowledgement from + the audit sink. +7. Return the [`AuditReceipt`](./data-model.md#auditreceipt-definition) provided + by the sink. + +If the synchronous `emit` cannot obtain an acknowledgement within the +configured timeout, it MUST surface a hard error. It MUST NOT return +successfully without a valid receipt. It may implement retries with back-off, +but this MUST be transparent to the caller and MUST NOT cause `emit` to block +indefinitely. + +## AuditRecord Queue + +### Durability guarantees + +The SDK MUST NOT use a bounded queue that silently drops records when +full. The SDK MUST use one of the following strategies: + +- **Disk-backed queue** (default): records are persisted to local + storage before acknowledgement by the exporter. This guarantees + delivery across SDK restarts and process crashes. +- **Unbounded in-memory queue** (opt-in): the queue grows without a + fixed limit. The SDK MUST expose a configurable high-water-mark + warning threshold. When the queue depth exceeds this threshold, the + SDK SHOULD emit a warning via the SDK's internal logging. The SDK SHOULD + provide meaningfull metrics (for example, `audit.queue.depth`) to allow + operators to monitor the queue depth and set up alerts. +- **Other lossless queuing strategy**: if the SDK implements a different + strategy that guarantees no record loss (for example, DB persistence queue + with back-pressure blocking), it MUST provide the same + durability guarantees and operational visibility as the above options. + The SDK MUST NOT use a strategy that can result in silent record loss + under any circumstances. + +### Back-pressure handling + +If the in-memory queue is exhausted (for example, because the exporter +is unavailable for an extended period): + +- The SDK MUST either block the calling thread (applying back-pressure + to the application) or persist the record to a local durable buffer. +- The SDK MUST NOT silently discard the record. + +The back-pressure strategy (blocking vs. disk-backed) MUST be +configurable. + +## AuditRecordProcessor + +`AuditRecordProcessor` is an interface that allows hooks into the +`AuditRecord` pipeline for enrichment and forwarding. + +### Processor restrictions + +The Audit Logging pipeline enforces strict restrictions on processors +to guarantee that no record is lost or altered: + +- Only **additive** operations are permitted: processors MAY enrich + records by adding `Attributes`, but MUST NOT remove existing + attributes, filter records, aggregate records, or alter the mandatory + LogRecord fields (`Timestamp`, `EventName`) or the mandatory + attributes (`audit.actor.id`, `audit.actor.type`, `audit.action`, + `audit.outcome`). +- Processors that delete attributes, suppress records, or aggregate + records MUST be rejected at configuration time with an explanatory + error. +- Processors MUST NOT introduce sampling or conditional forwarding. + +### AuditRecordProcessor operations + +#### OnEmit + +`OnEmit` is called synchronously on the thread that called `emit`, +after the `AuditRecord` has been queued and before the response is +returned to the caller. + +**Parameters:** + +- `auditRecord` – a mutable `AuditRecord` for the emitted record. + Processors MAY enrich the record (add attributes) but + MUST NOT remove mandatory fields or filter the record. +- `context` – the resolved `Context` at the time of emission. + +**Returns:** `Void` + +`OnEmit` MUST NOT block indefinitely. It SHOULD complete quickly to +minimise the latency of synchronous `emit` calls. + +Mutations made by a processor to `auditRecord` MUST be visible to +subsequently registered processors. + +If `OnEmit` is called after the processor has been shut down, the +processor MUST raise a hard error or throw an exception and MUST NOT +silently accept or forward the record. + +#### Shutdown + +Shuts down the processor. Called when the `AuditProvider` is shut down. + +On the first call, `Shutdown` MUST include the effects of `ForceFlush`. + +`Shutdown` SHOULD complete or abort within a configurable timeout. + +Subsequent calls to `Shutdown` or `ForceFlush` after the first +`Shutdown` call MUST be no-ops and MUST NOT return an error. + +#### ForceFlush + +Requests that the processor export all buffered records as soon as +possible, preferably before returning. + +`ForceFlush` SHOULD provide a way for the caller to determine whether +it succeeded, failed, or timed out. + +If `Shutdown` has already been called, `ForceFlush` MUST be a no-op +and MUST NOT return an error. + +### Built-in processors + +#### Simple processor + +The simple processor passes each `AuditRecord` directly to the +configured `AuditRecordExporter` synchronously on the calling thread. +This guarantees that the exporter acknowledges the record before +`emit` returns. + +The processor MUST synchronize calls to the exporter's `Export` to +prevent concurrent invocations. + +**Configurable parameters:** + +- `exporter` – the `AuditRecordExporter` to which records are pushed. + +The SDK MUST provide the simple processor as a built-in. It is the +default processor for synchronous `emit`. + +#### Batching processor + +The batching processor accumulates `AuditRecord`s in an internal queue +and exports them in batches asynchronously. It reduces per-record +round-trip overhead while retaining the no-drop guarantee: records +MUST NOT be discarded when the batch queue is full; the processor MUST +apply back-pressure or switch to disk-backed storage instead. + +The batching processor MUST still satisfy the at-least-once delivery +requirement: if the process exits before a batch is exported, buffered +records MUST be recoverable (e.g. via a disk-backed queue). + +**Configurable parameters:** + +- `exporter` – the `AuditRecordExporter` to which batches are pushed. +- `maxQueueSize` – maximum number of records held in memory before + back-pressure is applied. Default: 2048. +- `scheduledDelayMillis` – how long the processor waits before + exporting an incomplete batch. Default: 5000 ms. +- `exportTimeoutMillis` – maximum time allowed for a single `Export` + call. Default: 30 000 ms. +- `maxExportBatchSize` – maximum number of records per exported batch. + Default: 512. +- `maxRetryCount` – maximum export retry attempts before surfacing a + hard error. Default: 5. +- `initialBackoffMillis` – initial wait between retries; doubled on + each attempt (exponential back-off). Default: 1000 ms. + +The SDK SHOULD provide the batching processor as a built-in. + +#### Signing processor (Sig) + +The signing processor computes an `audit.integrity.value` attribute +for each `AuditRecord` to provide tamper-evidence. The value is a +base64-encoded asymmetric digital signature or symmetric HMAC, as +determined by the `audit.integrity.algorithm` `Resource` attribute +configured on the `AuditProvider`. Before signing, the processor +MUST serialize the `AuditRecord` to JSON and canonicalize it using +[RFC 8785 – JSON Canonicalization Scheme (JCS)][rfc8785]; the +`audit.integrity.*` attributes MUST be excluded from the canonical +form — they carry the proof itself and MUST NOT be part of the +signed payload. The canonical byte sequence of the remaining record +is the input to the signing or HMAC operation. It MUST be implemented as a +separate processor that can be added to the pipeline in addition to +the simple processor, to allow flexibility in the choice of signing +algorithm and key management strategy. + +[rfc8785]: https://www.rfc-editor.org/rfc/rfc8785 + +## AuditRecordExporter + +`AuditRecordExporter` is the interface that protocol-specific exporters +implement to transmit `AuditRecord`s to the audit sink. + +Each implementation MUST document its concurrency characteristics. + +### AuditRecordExporter operations + +#### Export + +Exports `AuditRecord`s to the audit sink. + +`Export` MUST NOT be called concurrently on the same exporter instance. + +`Export` MUST NOT block indefinitely. There MUST be a configurable +upper time limit after which the call times out with a `Failure` result. + +**Parameters:** + +- Optional `batch` – a collection of `AuditRecord`s to export. The concrete + type is language-specific. + +**Returns:** `ExportResult` + +`ExportResult` has values of either `Success` or `Failure`: + +- `Success` – all records in the batch have been acknowledged by the + audit sink. The exporter MUST include the `AuditReceipt` for each + record in the result. +- `Failure` – the export failed. The SDK MUST retain all records in + the batch and retry according to the configured retry policy. The + SDK MUST NOT drop records on a `Failure` result. + +#### ForceFlush + +Requests that any buffered `AuditRecord`s be exported immediately. + +`ForceFlush` SHOULD provide a way for the caller to determine whether +it succeeded, failed, or timed out. + +#### Shutdown + +Shuts down the exporter. Called when the `AuditProvider` is shut down. + +On the first call, `Shutdown` MUST flush any internally buffered records +and release all resources. After `Shutdown` completes, subsequent calls +to `Export` MUST return `Failure`. + +Subsequent calls to `Shutdown` or `ForceFlush` after the first +`Shutdown` call MUST be no-ops and MUST NOT return an error. + +### OTLP transport requirements + +#### Dedicated endpoint + +Audit records MUST be exported to a dedicated OTLP endpoint, distinct +from the standard `/v1/logs` endpoint. The default path is `/v1/audit`. + +Using a dedicated endpoint enables: + +- Network-level isolation (firewall rules, separate TLS certificates + and credentials). +- Independent QoS policies with no back-pressure shedding from the + observability pipeline. +- Clear separation in the sink's data model. + + +The payload MUST use the standard `ExportLogsServiceRequest` proto, +with the `audit` flag set to `true` in the `ResourceLogs` message to +identify the batch as audit records. + +#### No partial success + +The OTLP receiver MUST NOT respond with a `partial_success` for an +audit log export request. + +If the receiver cannot process one or more records in a batch, it MUST +reject the **entire** batch and return an error status. The SDK exporter +MUST treat any `partial_success` response as a hard `Failure`, retain +**all** records in the batch (including those that would have been +accepted), and retry the full batch. + +This constraint ensures that no audit record can be silently lost by a +receiver that acknowledges only part of a batch. + +#### Idempotency + +The OTLP receiver SHOULD treat `audit.record.id` as an idempotency +key. + +When a record with an already-known `audit.record.id` arrives: + +- If the canonical payload hash is **identical**, the receiver SHOULD + return a deterministic success response for the existing record + without creating a duplicate entry. +- If the canonical payload hash **differs**, the receiver MUST return + a conflict error (HTTP 409 or equivalent). The SDK exporter MUST + treat a conflict error as a non-retryable `Failure`. + +This ensures that retries after transient failures do not produce +duplicate audit entries, provided the caller uses a stable `RecordId` +across retry attempts. + +#### InstrumentationScope + +`InstrumentationScope` has no meaning for audit logging. Audit records +are emitted by application or system code acting on behalf of an actor, +not by instrumentation libraries. + +Exporters MUST leave the `InstrumentationScope` field empty, or MAY +populate it with the SDK name and version as a technical marker. +Receivers MUST NOT use `InstrumentationScope` for routing, filtering, +or processing audit records. + +## Failure handling + +If the exporter cannot reach the audit sink, the SDK MUST follow this +failure-handling sequence: + +1. **Buffer**: Retain the record in the SDK's queue (disk-backed or + in-memory, depending on configuration). The record MUST NOT be + discarded. +2. **Retry**: Attempt reexport with exponential back-off and a + configurable maximum number of retries. The initial retry interval, + back-off multiplier, and maximum interval SHOULD be configurable. +3. **Hard error**: If the retry budget is exhausted or the buffer is + full (for example, the disk is full), the SDK MUST surface a hard + error to the caller (exception, error code, or equivalent). It MUST + NOT silently drop the record. +4. **Metric**: The SDK MUST increment the `audit.records.dropped` + metric counter for every record that is lost due to an unrecoverable + error. See [Observability](#observability). + +## Clock synchronisation + +Accurate timestamps are a requirement of ISO 27001 Annex A (clock +synchronisation). The SDK SHOULD: + +- Warn at startup if the system clock NTP offset exceeds one second. +- Warn when `Timestamp` and `ObservedTimestamp` differ by more than a + configurable threshold (default: 5 seconds), which may indicate clock + skew between the event source and the SDK host. + +The SDK MUST set `ObservedTimestamp` to the wall-clock time at the +moment `emit` is called, using the SDK host's system clock. + +## Interaction with the Log signal + +Audit records MUST NOT be routed through the standard `LoggerProvider` +pipeline. The two signals are independent and MUST use separate +providers, queues, and exporters. + +An application or library MAY emit both an `AuditRecord` (for +compliance) and a regular `LogRecord` (for observability) for the same +event. The records travel through independent pipelines to independent +backends and MUST NOT interfere with each other. + +## Collector pipeline (Tier-2) + +In enterprise deployments an OpenTelemetry Collector typically sits +between the SDK exporter and the final audit sink. This collector acts +as a **Tier-2** intermediary that can verify record integrity, +distribute records to multiple sinks, and provide delivery tracking. + +``` +Application + │ + ▼ /v1/audit (SDK exporter) +OTel Collector ──► verify hash/signature + │ └── forward to required sinks + ├──► Primary sink (OpenSearch, Splunk, …) + ├──► Secondary sink (S3 WORM, cold archive) + └──► SIEM + +Collector returns delivery status per sink to the SDK exporter. +``` + +The SDK exporter interacts with the Tier-2 collector via the same +`/v1/audit` endpoint; no API changes are required at the SDK level. + +The Tier-2 collector SHOULD: + +- Verify `audit.integrity.value` of each received record against the + algorithm declared in the `Resource` attribute + `audit.integrity.algorithm` and the key material referenced by + `audit.integrity.certificate`. +- Validate `audit.sequence.number` continuity and `audit.sequence.prev_hash` + chain integrity when these optional attributes are present. +- Deliver each record to every configured required sink before + returning a success response to the SDK exporter. +- Return a conflict error (HTTP 409) when a duplicate `RecordId` with + a differing payload hash is received. +- Expose per-sink delivery status in its response so that operators can + detect partial delivery failures. + +The Tier-2 collector MUST NOT respond with partial success (see +[No partial success](#no-partial-success)). If any required sink +rejects the record the collector MUST reject the entire request. + +For the full Tier-2 collector specification see +[Audit Logging Collector](./collector.md). + +## Observability + +The SDK MUST expose the following internal metrics to allow operators +to monitor the health of the audit pipeline: + +| Metric name | Type | Description | +|--------------------------|-----------|---------------------------------------------| +| `audit.records.emitted` | Counter | Total records passed to the SDK pipeline. | +| `audit.records.exported` | Counter | Records successfully acknowledged by sink. | +| `audit.records.dropped` | Counter | Records lost due to unrecoverable errors. | +| `audit.queue.depth` | Gauge | Current number of records in the SDK queue. | +| `audit.export.duration` | Histogram | Round-trip time from `Export` to receipt. | + +A non-zero value for `audit.records.dropped` MUST be treated as a +critical operational alert. SDK authors SHOULD provide a mechanism (for +example, a log message or a callback) to notify operators immediately +when this counter is incremented. + +## Concurrency requirements + +For languages that support concurrent execution: + +- **AuditProvider** – logger creation, `ForceFlush`, and `Shutdown` + MUST be safe for concurrent use. +- **AuditLogger** – `emit` MUST be safe for concurrent use. +- **AuditRecordExporter** – `ForceFlush` and `Shutdown` MUST be safe + for concurrent use. `Export` MUST NOT be called concurrently on the + same instance. + +## References + +- [OTEP 0267 – Audit Logging Signal](../../oteps/0267-audit-logging.md) +- [OTEP 0150 – Logging Library SDK Prototype Specification](../../oteps/logs/0150-logging-library-sdk.md)