Audit Logging signal#5059
Conversation
| | `Algorithm` | `string` | Algorithm used for the digital signature (e.g., `RS256`, `ES256`). | | ||
| | `Certificate` | `bytes` | Digital public certificate used for the signature verification. | |
There was a problem hiding this comment.
To increase efficiency wouldn't this make sense to be captured at a higher level ie instrumentation scope?
There was a problem hiding this comment.
As far as I know instrumentation scopes do not allow additional attributes. But you are right, we can move it to the resource entity.
There was a problem hiding this comment.
Instrumentation scopes do allow attributes https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/common/v1/common.proto
I was more thinking of making them first class fields.
| | `Outcome` | `enum` | `SUCCESS`, `FAILURE`, `UNKNOWN`. MUST be set. | | ||
| | `Resource` | `AnyValue` | The target resource (file, endpoint, table, …). SHOULD be set. | | ||
| | `Action` | `string` | Verb describing what was done (e.g. `READ`, `WRITE`, `DELETE`, `LOGIN`). MUST be set. | | ||
| | `SourceIP` | `string` | Source network address, if applicable. | |
There was a problem hiding this comment.
Why not just use the attribute?
| | `Timestamp` | `fixed64` | Nanoseconds since UNIX epoch (UTC). MUST be set. `<= ObservedTimestamp <= now(UTC)` | | ||
| | `ObservedTimestamp` | `fixed64` | Nanoseconds since UNIX epoch when the SDK observed the event. Used for clock skew detection. MUST be set. `>= Timestamp <= now(UTC)` | | ||
| | `EventName` | `string` | Semantic name of the audit event (e.g. `user.login.success`). MUST be set. | | ||
| | `Actor` | `AnyValue` | Identity that performed the action (user ID, service account, …). MUST be set. | |
There was a problem hiding this comment.
Would this be better as a map?
There was a problem hiding this comment.
Would this be better as a map?
Why? I do understand the attribute thing (above), but don't get your idea behind why this should be a map?
There was a problem hiding this comment.
If it was a map, it would behave like attributes & enable us to define more structure to it while we enabling linking etc. Alternatively they just end as attributes.
There was a problem hiding this comment.
ok, gotcha - now it's an attribute combination: audit.actor.type + audit.actor.id
| | `Actor` | `AnyValue` | Identity that performed the action (user ID, service account, …). MUST be set. | | ||
| | `ActorType` | `enum` | `USER`, `SERVICE`, `SYSTEM`. MUST be set. | | ||
| | `Outcome` | `enum` | `SUCCESS`, `FAILURE`, `UNKNOWN`. MUST be set. | | ||
| | `Resource` | `AnyValue` | The target resource (file, endpoint, table, …). SHOULD be set. | |
There was a problem hiding this comment.
Could attributes make more sense so it can be correlated with semconv attributes & other signals?
| | SHA-256 receipt requires a round-trip to the sink | The receipt is optional for callers that do not need proof-of-delivery; a fire-and-forget async mode MAY omit it. | | ||
| | Disk-backed queue introduces a dependency on localStorage | This is opt-in; the default is an in-memory queue with blocking back-pressure. | | ||
|
|
||
| ## Prior art and alternatives |
There was a problem hiding this comment.
Has it been considered adding an additional event_kind property to the log record and having dedicated queues based on event kind that way we don't need new api but rather just new method/parameter. This would address queue overflow.
We could extend exporters to only process the defined event_kind that way we could have different endpoints based on event_kind.
There was a problem hiding this comment.
Yes, we have evaluated the option to just add one 'flag': open-telemetry/semantic-conventions#3469
We could extend exporters to only process the defined event_kind that way we could have different endpoints based on event_kind.
Could you elaborate on this a bit more? Would this different endpoint allow us to get the AuditReceipt? How would the receiver (e.g. in OTel Collector) look like?
There was a problem hiding this comment.
Alternative 1 is the only documented similiar approach however it differs significantly. Difference being rather than using a single shared queue and being at risk of overflow, the queue is split based on event_kind hence removes that risk.
Audit Receipt is something that could be added as an additional field to the current log response. This field would only be sent if requested via the request.
For the collector if the event_kind is audit, then it should either not allow processing or only a simplified processor which can only add attributes.
This comment was marked as resolved.
This comment was marked as resolved.
|
Thanks a lot @thompson-tomo for all your early inputs! |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Signed-off-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
Signed-off-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
Durability at the sink is out of scope Signed-off-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
Signed-off-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
Signed-off-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
Signed-off-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
Signed-off-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
#2 Signed-off-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
replaces #1 Signed-off-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
This reverts commit 869a23c.
This reverts commit a5550bc.
Signed-off-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
... that uses the standard dedicated `LogRecord` fields for universal concepts and carries all audit-specific data as `Attributes` following the `audit.*` semantic convention namespace Signed-off-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
Signed-off-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
Signed-off-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
|
Hi @hilmarf, thank you for pushing on this! I have a stepping back question - since this PR is trying to expand the scope of the OpenTelemetry project, we need to agree on the direction first. Do you have updates on open-telemetry/community#2409, do we see more demands to make it a priority for OpenTelemetry? I don't think we can accept this PR before the project proposal is accepted. |
|
This PR was marked stale. It will be closed in 14 days without additional activity. |
|
Closed as inactive. Feel free to reopen if this PR is still being worked on. |
4 participants
Overview
This pull request introduces a new Audit Logging signal to OpenTelemetry—a purpose-built observability pipeline for security-relevant events that satisfies compliance requirements such as ISO 27001, SOC 2, PCI-DSS, and HIPAA.
Motivation
The existing OpenTelemetry Logs signal is designed for general-purpose observability and intentionally permits sampling, back-pressure shedding, and record transformation. These behaviors are incompatible with audit logging, where:
A dedicated Audit Logging signal provides:
/v1/audit) for network-level isolation from observability backendsSIG request
open-telemetry/community#2409
Changes
This PR adds comprehensive specification documentation and establishes the complete audit logging framework:
New Documents
Key Features
✅ Guaranteed Delivery – At-least-once delivery with durable (disk-backed) queuing
✅ Integrity Protection – Optional asymmetric signatures or symmetric HMACs for tamper-evidence
✅ No Partial Success – OTLP receivers reject entire batches if any record fails
✅ Clock Skew Detection – Dual timestamps (event vs. observed) enable clock synchronization verification
✅ Idempotency – RecordId as idempotency key prevents duplicate audit entries on retries
✅ Hash-Chain Support – Optional SequenceNo and PrevHash fields for ordered stream integrity
✅ Compliance Ready – ISO 27001 Annex A requirements built into the core design
Data Model Highlights
AuditRecord – Mandatory fields:
RecordId,Timestamp,ObservedTimestamp,EventNameActor,ActorType,Action,OutcomeTargetResource,SourceIP,Signature,Algorithm,Certificate,Hmac,SequenceNo,PrevHashAuditReceipt – Proof-of-delivery returned by sink:
RecordId(echoed),IntegrityHash(SHA-256),SinkTimestampSpecification Status
LogRecordprotobuf withaudit=trueflag; dedicated/v1/auditendpointReferences
LogRecordas transport container for efficiency