TDH Decoding Support for Manifest-Based ETW Events

[utpilla]

Summary

Extend the existing etw::tdh::TdhDecoder (added for TraceLogging / TraceLoggingDynamic in #270) to also decode manifest-based ETW events. Manifest support was explicitly deferred as future work in #270; the design there was built to accommodate it "without rework," and this issue delivers that extension.

The change reuses the entire existing decode pipeline: TdhGetEventInformation, the EVENT_PROPERTY_INFO -> EventFormat walker, the LocationType skip-chain, and the EventData construction. The only manifest-specific additions are (a) how a schema is identified for caching and (b) where the event name is read from.

Motivation

The TdhDecoder currently handles only self-describing events, those carrying an inline EVENT_HEADER_EXT_TYPE_EVENT_SCHEMA_TL blob. Manifest-based providers (the bulk of Windows' own instrumentation: Microsoft-Windows-Kernel-*, .NET, countless first- and third-party providers authored with mc.exe) do not embed their schema in the event. Instead, the event carries only an EVENT_DESCRIPTOR, and the layout lives in an XML manifest registered with the OS.

TdhGetEventInformation is the same entry point for both sources, so consuming manifest events through the existing code path is a small, localized extension rather than a new subsystem. Adding it lets one_collect decode the large population of manifest providers through one uniform EventFormat / EventData interface.

Background: how manifest events differ

	TraceLogging / TraceLoggingDynamic	Manifest-based
Where the schema lives	Inline in the event (EVENT_HEADER_EXT_TYPE_EVENT_SCHEMA_TL)	In an XML manifest compiled into a DLL and registered with the OS
Schema identity	Raw schema-TL bytes	(Provider GUID, Id, Version) from EVENT_DESCRIPTOR
Who registers it, where	Nobody; it's in-band	Installer/admin via wevtutil im (or MOF/WBEM); lands in HKLM\...\WINEVT\Publishers pointing at a resource DLL
Needed on the decoding machine	No (self-describing)	Yes, else TdhGetEventInformation returns ERROR_NOT_FOUND
TDH entry point	TdhGetEventInformation	TdhGetEventInformation (identical)

TraceLogging vs. TraceLoggingDynamic differ only on the producer side (compile-time vs. runtime-built metadata); they share one wire format and one decode path. Manifest is the genuinely different case, and it is what this issue adds.

Scope

In scope

• Decoding manifest-based events via TdhGetEventInformation (same call as TraceLogging).
• A manifest schema cache keyed by (Provider GUID, Id, Version), split by pointer width, alongside the existing schema-TL cache.
• Transparent dispatch: TdhDecoder::decode() picks the schema source automatically (inline SCHEMA_TL -> TraceLogging; otherwise -> manifest).
• Event-name resolution appropriate to the source (EventNameOffset for TraceLogging, TaskNameOffset for manifest/WPP), selected via DecodingSource.
• Reuse of the existing struct-flattening, scalar/string type mapping, and 32-/64-bit handling.

Out of scope (future work)

• Map / enum value resolution (manifest providers commonly emit enums and bitmasks; today only the raw integer reaches the consumer).
• Array-typed properties and properties whose length or count is given by another property.
• MOF/WBEM and WPP-specific decoding niceties beyond what TDH returns as TRACE_EVENT_INFO.

As in #270, an unsupported property kind is emitted as an "unsupported" placeholder field so it never disables the rest of the schema or stops the event stream.

Design

Cache: shared schema arena, per-source index maps

The cache was recently optimized in #308 to an append-only arena of decoded schemas plus per-(pointer-width) index maps that key raw TL bytes to a usize index into that arena. This hashes the key exactly once on the hot path (the index is copied out, ending the map borrow, then the schema is read from the arena). Manifest support slots into this shape cleanly:

struct SchemaCache {
    schemas: Vec<CachedSchema>,                 // ONE shared arena for both sources
    tl_64:       HashMap<Vec<u8>, usize>,       // TraceLogging bytes -> arena index
    tl_32:       HashMap<Vec<u8>, usize>,
    manifest_64: HashMap<ManifestKey, usize>,   // (provider, id, version) -> arena index
    manifest_32: HashMap<ManifestKey, usize>,
}

Why manifest needs its own maps. The two sources are identified by different key types (Vec<u8> schema bytes vs. ManifestKey { provider, id, version }), and a Rust HashMap is monomorphic in its key. Unifying them into one map would require either an enum key (which forces an allocation on every lookup to borrow &[u8] as the enum, defeating the single-hash zero-alloc hot path) or synthesizing a byte key for the tuple (which conflates two identity namespaces and risks cross-source collisions). The identity semantics also differ: TL bytes are the schema content; a ManifestKey is a reference to an externally registered schema. Separate maps keep the two keyspaces cleanly isolated and each lookup on its natural, borrowable key. This mirrors the existing 32-/64-bit split, which is likewise "separate maps, not a composite key."

Why the arena is shared. A decoded CachedSchema ({ event_name, format, schema_id }) is source-agnostic (nothing about it remembers whether it came from TL or a manifest), so storage has no reason to be split. The maps are the index; the Vec is the storage. The arena is append-only, so an index handed out by either map stays valid for the decoder's lifetime, and inserts always push to the end and return distinct indices, so TL and manifest entries can never alias the same slot. Sharing the arena also unifies the SchemaId space, so exporters keying on schema_id see one coherent numbering across both sources. (Analogy: two database indexes over one table, two ways to find rows, one table.)

Cache key rationale

• TraceLogging / Dynamic: keyed by the raw schema-TL bytes. EVENT_DESCRIPTOR.Id is unreliable (usually 0), so the embedded bytes are the only trustworthy identity.
• Manifest: keyed by (Provider GUID, Id, Version). The manifest pins this tuple to a fixed layout

Source selection

decode() first looks for an inline SCHEMA_TL item. If present -> TraceLogging path (unchanged). If absent -> build a ManifestKey from the record's EVENT_DESCRIPTOR and take the manifest path. Both paths converge on the same cache-miss handler (TdhGetEventInformation + walker) and the same EventData construction.

Reused unchanged

call_tdh_get_event_information, build_cached_schema, walk_properties, intype_to_field_info, and the framework's field-reading machinery in event/mod.rs. TDH normalizes every source into one TRACE_EVENT_INFO, so the byte-level decoding is source-agnostic: scalars, strings, and nested structs decode identically for manifest events.

Event name

read_event_name checks TRACE_EVENT_INFO.DecodingSource: DecodingSourceTlg reads EventNameOffset; otherwise (manifest / WPP) it reads TaskNameOffset.

Behavior notes

• Registration dependency. Manifest decoding requires the provider's manifest to be registered on the decoding machine (HKLM\...\WINEVT\Publishers -> resource DLL). If it isn't, TdhGetEventInformation returns ERROR_NOT_FOUND and decode() returns TdhDecodeError::NotFound. Reading that registry key does not require admin; only registration does. (Consuming a live ETW session is a separate, privileged concern.)
• No registry access in our code. We never call registry APIs. The Publishers lookup + resource-DLL load happens internally inside tdh.dll, behind the TdhGetEventInformation call in call_tdh_get_event_information.
• No per-event registry cost. That call runs only on a cache miss. Cache hits are a single hashmap probe + EventData::new, identical to the TraceLogging path.
• Dynamic pickup. Negative results are not cached, so a manifest registered after the decoder starts is picked up on the next event of that (Provider, Id, Version). A tuple that already resolved is not re-queried (correct per the manifest contract that a fixed tuple means a fixed layout; layout changes bump Version, which is a new key). A never-registered chatty provider will re-issue the TDH call per event; a bounded-negative-cache optimization could be added later if needed.

Public API

No new public API. TdhDecoder::decode() gains manifest handling transparently. TdhDecodedEvent::event_name now carries the manifest task name for manifest events. TdhDecodeError is unchanged (NotFound now also covers "manifest not registered").

[swashtek]

Thank you @utpilla for putting it together, I will start working on it.

[swashtek]

Current dispatch predicate can route MOF/WPP into the manifest cache and not TraceLogging is not equivalent to manifest.
Today source selection is effectively a negative test, if no inline SCHEMA_TL, it goes to manifest path, but that branch includes non-TL sources. That is correctness risk.

For classsic/MOF events, EVENT_DESCRIPTOR.Id is commonly 0; discrimination is by event-class GUID + Opcode.
With keying on ManifestKey { provider, id, version }, multiple classic events from one provider can collapse to {provider, 0, version}.

Two guards would address this:

1. Pre-TDH fast reject: if EVENT_HEADER_FLAG_CLASSIC_HEADER is set, do not take manifest path.
2. On cache miss, require TRACE_EVENT_INFO.DecodingSource == DecodingSourceXMLFile before inserting into manifest cache; otherwise return Unsupported (also blocks WPP).

[brianrob]

Question - do you plan to also implement support for EventSource manifest events?

TraceEvent support reference: https://github.com/microsoft/perfview/blob/main/src/TraceEvent/DynamicTraceEventParser.cs#L200

[utpilla]

Current dispatch predicate can route MOF/WPP into the manifest cache and not TraceLogging is not equivalent to manifest. Today source selection is effectively a negative test, if no inline SCHEMA_TL, it goes to manifest path, but that branch includes non-TL sources. That is correctness risk.

For classsic/MOF events, EVENT_DESCRIPTOR.Id is commonly 0; discrimination is by event-class GUID + Opcode. With keying on ManifestKey { provider, id, version }, multiple classic events from one provider can collapse to {provider, 0, version}.

Two guards would address this:

1. Pre-TDH fast reject: if EVENT_HEADER_FLAG_CLASSIC_HEADER is set, do not take manifest path.
2. On cache miss, require TRACE_EVENT_INFO.DecodingSource == DecodingSourceXMLFile before inserting into manifest cache; otherwise return Unsupported (also blocks WPP).

@swashtek Good catch!. "No inline SCHEMA_TL" is a negative test that also captures classic/MOF and WPP, and keying those on ManifestKey { provider, id, version } collapses classic events (Id == 0) onto {provider, 0, version}. I'll adopt both guards that you suggested.

I'll update the issue so the scoping is enforced rather than accidental. Thanks!

[utpilla]

Question - do you plan to also implement support for EventSource manifest events?

TraceEvent support reference: https://github.com/microsoft/perfview/blob/main/src/TraceEvent/DynamicTraceEventParser.cs#L200

@brianrob We definitely want to but, thanks for flagging this! I hadn't realized EventSource manifest events would need a separate mechanism until digging in. EventSource's default mode emits its manifest in-band, not registered with the OS, so TdhGetEventInformation returns ERROR_NOT_FOUND and the existing TDH path can't decode them, exactly what your DynamicTraceEventParser reference handles.

I'll file it as a separate issue so #310 stays self-contained. Thanks again for the pointer.

[brianrob]

I'll file it as a separate issue so #310 stays self-contained. Thanks again for the pointer.

Sounds good. EventSources can also write their events using TraceLogging, but you're already covered there. Manifest-based using the "manifest in the event stream" is the default.

[utpilla]

Sounds good. EventSources can also write their events using TraceLogging, but you're already covered there. Manifest-based using the "manifest in the event stream" is the default.

Is there any requirement (a specific .NET version etc.) for a user to switch the event type to Tracelogging for EventSource?