Skip to content

in_winevtlog: Enrich EventData key-value map if enabled - #12082

Merged
edsiper merged 2 commits into
masterfrom
cosmo0920-enrich-event-data-on-in_winevtlog
Jul 10, 2026
Merged

in_winevtlog: Enrich EventData key-value map if enabled#12082
edsiper merged 2 commits into
masterfrom
cosmo0920-enrich-event-data-on-in_winevtlog

Conversation

@cosmo0920

@cosmo0920 cosmo0920 commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

In the current implementation, we didn't handle EventData key-value map even if an event template insists there's a structure that contains EventData structure. This can be retrieving via EventTemplate.

Closes #12080.


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
pipeline:
  inputs:
    - name: winevtlog
      tag: some-tag
      alias: Windows alias
      channels: Application,System,Security
      interval_sec: 5
      read_existing_events: false
      render_event_as_xml: false
      read_limit_per_cycle: 2m
      event_data_as_map: true
      event_template_cache_size: 256
  outputs:
    - name: stdout
  • Debug log output from testing the change

Then, we can handle EventData key-value map in each of records.
For example, we'll have as follows:

Fluent Bit v5.1.0
* Copyright (C) 2015-2026 The Fluent Bit Authors
* Fluent Bit is a CNCF graduated project under the Fluent organization
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____
|  ___| |                | |   | ___ (_) |         |  ___||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   _|___ \ | |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \|  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V //\__/ /\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)\___/


[2026/07/10 19:58:54.156] [ info] Configuration:
[2026/07/10 19:58:54.156] [ info]  flush time     | 1.000000 seconds
[2026/07/10 19:58:54.156] [ info]  grace          | 5 seconds
[2026/07/10 19:58:54.156] [ info]  daemon         | 0
[2026/07/10 19:58:54.156] [ info] ___________
[2026/07/10 19:58:54.156] [ info]  inputs:
[2026/07/10 19:58:54.156] [ info]      winevtlog
[2026/07/10 19:58:54.156] [ info] ___________
[2026/07/10 19:58:54.156] [ info]  filters:
[2026/07/10 19:58:54.156] [ info] ___________
[2026/07/10 19:58:54.156] [ info]  outputs:
[2026/07/10 19:58:54.156] [ info]      stdout.0
[2026/07/10 19:58:54.156] [ info] ___________
[2026/07/10 19:58:54.156] [ info]  collectors:
[2026/07/10 19:58:54.158] [ info] [fluent bit] version=5.1.0, commit=ff52d4ad78, pid=81008
[2026/07/10 19:58:54.158] [debug] [engine] maxstdio set: 512
[2026/07/10 19:58:54.158] [debug] [engine] coroutine stack size: 98302 bytes (96.0K)
[2026/07/10 19:58:54.158] [ info] [storage] ver=1.5.4, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2026/07/10 19:58:54.158] [ info] [simd    ] SSE2
[2026/07/10 19:58:54.158] [ info] [cmetrics] version=2.1.5
[2026/07/10 19:58:54.158] [ info] [ctraces ] version=0.7.1
[2026/07/10 19:58:54.159] [ info] [input:winevtlog:Windows alias] initializing
[2026/07/10 19:58:54.159] [ info] [input:winevtlog:Windows alias] storage_strategy='memory' (memory only)
[2026/07/10 19:58:54.159] [debug] [winevtlog:Windows alias] created event channels: read=808 write=812
[2026/07/10 19:58:54.159] [debug] [input:winevtlog:Windows alias] EventData named maps enabled; template cache size=256
[2026/07/10 19:58:54.159] [debug] [input:winevtlog:Windows alias] connect to local machine
[2026/07/10 19:58:54.159] [ warn] [input:winevtlog:Windows alias] read limit per cycle cannot exceed 1.8M. Set up to 1.8M
[2026/07/10 19:58:54.160] [debug] [stdout:stdout.0] created event channels: read=888 write=892
[2026/07/10 19:58:54.162] [ info] [sp] stream processor started
[2026/07/10 19:58:54.162] [ info] [output:stdout:stdout.0] worker #0 started
[2026/07/10 19:58:54.162] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[2026/07/10 19:59:14.198] [debug] [input:winevtlog:Windows alias] could not enumerate publisher event metadata: 2
[2026/07/10 19:59:14.225] [debug] [input:winevtlog:Windows alias] could not enumerate publisher event metadata: 2
[2026/07/10 19:59:14.247] [debug] [input:winevtlog:Windows alias] could not enumerate publisher event metadata: 2
[2026/07/10 19:59:14.262] [debug] [input:winevtlog:Windows alias] could not enumerate publisher event metadata: 1813
[2026/07/10 19:59:14.262] [debug] [input:winevtlog:Windows alias] read 1992 bytes from 'Application'
[2026/07/10 19:59:14.265] [debug] [input:winevtlog:Windows alias] loaded EventData template for Microsoft-Windows-Kernel-Power, event 105 version 1: 3 fields
[2026/07/10 19:59:14.265] [debug] [input:winevtlog:Windows alias] read 480 bytes from 'System'
[2026/07/10 19:59:14.271] [debug] [input:winevtlog:Windows alias] loaded EventData template for Microsoft-Windows-Security-Auditing, event 4798 version 0: 9 fields
[2026/07/10 19:59:14.283] [debug] [input:winevtlog:Windows alias] read 13312 bytes from 'Security'
[2026/07/10 19:59:15.162] [debug] [task] created task=000001D1251D9A90 id=0 OK
[2026/07/10 19:59:15.163] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2026/07/10 19:59:15.239] [debug] [out flush] cb_destroy coro_id=0
[2026/07/10 19:59:15.239] [debug] [task] destroy task=000001D1251D9A90 (task_id=0)
[2026/07/10 19:59:19.205] [debug] [input:winevtlog:Windows alias] read 1992 bytes from 'Application'
[2026/07/10 19:59:19.206] [debug] [input:winevtlog:Windows alias] read 480 bytes from 'System'
[2026/07/10 19:59:19.212] [debug] [input:winevtlog:Windows alias] read 6656 bytes from 'Security'
[2026/07/10 19:59:20.178] [debug] [task] created task=000001D1251DA0D0 id=0 OK
[2026/07/10 19:59:20.178] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] some-tag: [[1783681159.180823900, {}], {"ProviderName"=>"Alienware SupportAssist Remediation", "ProviderGuid"=>"", "Qualifiers"=>0, "EventID"=>0, "Version"=>0, "Level"=>4, "Task"=>0, "Opcode"=>0, "Keywords"=>"0x80000000000000", "TimeCreated"=>"2026-07-10 19:59:14 +0900", "EventRecordID"=>406115, "ActivityID"=>"", "RelatedActivityID"=>"", "ProcessID"=>36372, "ThreadID"=>0, "Channel"=>"Application", "Computer"=>"DESKTOP-JLLFF9D", "UserID"=>"", "Message"=>"PowerEvent handled successfully by the service.", "StringInserts"=>["PowerEvent handled successfully by the service."]}]
[1] some-tag: [[1783681159.190378400, {}], {"ProviderName"=>"DellDigitalDelivery", "ProviderGuid"=>"", "Qualifiers"=>0, "EventID"=>0, "Version"=>0, "Level"=>4, "Task"=>0, "Opcode"=>0, "Keywords"=>"0x80000000000000", "TimeCreated"=>"2026-07-10 19:59:14 +0900", "EventRecordID"=>406116, "ActivityID"=>"", "RelatedActivityID"=>"", "ProcessID"=>18416, "ThreadID"=>0, "Channel"=>"Application", "Computer"=>"DESKTOP-JLLFF9D", "UserID"=>"", "Message"=>"PowerEvent handled successfully by the service.", "StringInserts"=>["PowerEvent handled successfully by the service."]}]
[2] some-tag: [[1783681159.198715300, {}], {"ProviderName"=>"DellTechHub", "ProviderGuid"=>"", "Qualifiers"=>0, "EventID"=>0, "Version"=>0, "Level"=>4, "Task"=>0, "Opcode"=>0, "Keywords"=>"0x80000000000000", "TimeCreated"=>"2026-07-10 19:59:14 +0900", "EventRecordID"=>406117, "ActivityID"=>"", "RelatedActivityID"=>"", "ProcessID"=>6128, "ThreadID"=>0, "Channel"=>"Application", "Computer"=>"DESKTOP-JLLFF9D", "UserID"=>"", "Message"=>"PowerEvent handled successfully by the service.", "StringInserts"=>["PowerEvent handled successfully by the service."]}]
[3] some-tag: [[1783681159.205411200, {}], {"ProviderName"=>"IntelTraceAgentService", "ProviderGuid"=>"", "Qualifiers"=>16386, "EventID"=>257, "Version"=>0, "Level"=>4, "Task"=>1, "Opcode"=>0, "Keywords"=>"0x80000000000000", "TimeCreated"=>"2026-07-10 19:59:14 +0900", "EventRecordID"=>406118, "ActivityID"=>"", "RelatedActivityID"=>"", "ProcessID"=>6256, "ThreadID"=>0, "Channel"=>"Application", "Computer"=>"DESKTOP-JLLFF9D", "UserID"=>"", "Message"=>"system power changed", "StringInserts"=>["system power changed"]}]
[4] some-tag: [[1783681159.206378900, {}], {"ProviderName"=>"Microsoft-Windows-Kernel-Power", "ProviderGuid"=>"{331C3B3A-2005-44C2-AC5E-77220C37D6B4}", "Qualifiers"=>"", "EventID"=>105, "Version"=>1, "Level"=>4, "Task"=>100, "Opcode"=>0, "Keywords"=>"0x8000000000000404", "TimeCreated"=>"2026-07-10 19:59:14 +0900", "EventRecordID"=>117323, "ActivityID"=>"", "RelatedActivityID"=>"", "ProcessID"=>4, "ThreadID"=>38328, "Channel"=>"System", "Computer"=>"DESKTOP-JLLFF9D", "UserID"=>"NT AUTHORITY\SYSTEM", "Message"=>"電源の変更。", "EventData"=>{"AcOnline"=>true, "RemainingCapacity"=>90324, "FullChargeCapacity"=>94676}, "StringInserts"=>[true, 90324, 94676]}]
<snip>
[1] some-tag: [[1783681189.178071200, {}], {"ProviderName"=>"Microsoft-Windows-Security-Auditing", "ProviderGuid"=>"{54849625-5478-4994-A5BA-3E3B0328C30D}", "Qualifiers"=>"", "EventID"=>4672, "Version"=>0, "Level"=>0, "Task"=>12548, "Opcode"=>0, "Keywords"=>"0x8020000000000000", "TimeCreated"=>"2026-07-10 19:59:46 +0900", "EventRecordID"=>5660641, "ActivityID"=>"{0630C5C6-0E8E-0001-4CC6-30068E0EDD01}", "RelatedActivityID"=>"", "ProcessID"=>1840, "ThreadID"=>5340, "Channel"=>"Security", "Computer"=>"DESKTOP-JLLFF9D", "UserID"=>"", "Message"=>"新しいログオンに特権が割り当てられました。

サブジェクト:
        セキュリティ ID:                S-1-5-18
        アカウント名:           SYSTEM
        アカウント ドメイン:            NT AUTHORITY
        ログオン ID:            0x3E7

特権:           SeAssignPrimaryTokenPrivilege
                        SeTcbPrivilege
                        SeSecurityPrivilege
                        SeTakeOwnershipPrivilege
                        SeLoadDriverPrivilege
                        SeBackupPrivilege
                        SeRestorePrivilege
                        SeDebugPrivilege
                        SeAuditPrivilege
                        SeSystemEnvironmentPrivilege
                        SeImpersonatePrivilege
                        SeDelegateSessionUserImpersonatePrivilege", "EventData"=>{"SubjectUserSid"=>"S-1-5-18", "SubjectUserName"=>"SYSTEM", "SubjectDomainName"=>"NT AUTHORITY", "SubjectLogonId"=>"0x3e7", "PrivilegeList"=>"SeAssignPrimaryTokenPrivilege
                        SeTcbPrivilege
                        SeSecurityPrivilege
                        SeTakeOwnershipPrivilege
                        SeLoadDriverPrivilege
                        SeBackupPrivilege
                        SeRestorePrivilege
                        SeDebugPrivilege
                        SeAuditPrivilege
                        SeSystemEnvironmentPrivilege
                        SeImpersonatePrivilege
                        SeDelegateSessionUserImpersonatePrivilege"}, "StringInserts"=>["S-1-5-18", "SYSTEM", "NT AUTHORITY", "0x3e7", "SeAssignPrimaryTokenPrivilege
                        SeTcbPrivilege
                        SeSecurityPrivilege
                        SeTakeOwnershipPrivilege
                        SeLoadDriverPrivilege
                        SeBackupPrivilege
                        SeRestorePrivilege
                        SeDebugPrivilege
                        SeAuditPrivilege
                        SeSystemEnvironmentPrivilege
                        SeImpersonatePrivilege
                        SeDelegateSessionUserImpersonatePrivilege"]}]
[2026/07/10 19:59:50.182] [debug] [out flush] cb_destroy coro_id=2
[2026/07/10 19:59:50.182] [debug] [task] destroy task=000001D1251D9BD0 (task_id=0)
[2026/07/10 20:00:06] [engine] caught signal (SIGINT)
[2026/07/10 20:00:06.792] [ warn] [engine] service will shutdown in max 5 seconds
[2026/07/10 20:00:06.792] [ info] [engine] pausing all inputs..
[2026/07/10 20:00:06.792] [ info] [input] pausing Windows alias
[2026/07/10 20:00:07.812] [ info] [engine] service has stopped (0 pending tasks)
[2026/07/10 20:00:07.812] [ info] [input] pausing Windows alias
[2026/07/10 20:00:07.812] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2026/07/10 20:00:07.813] [ info] [output:stdout:stdout.0] thread worker #0 stopped
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

  • New Features

    • Added an option to emit Windows Event Log EventData as structured maps using template field names.
    • Added configurable event-template caching (event_template_cache_size), defaulting to 256 entries.
    • Extended structured event-data handling consistently across XML, text, and standard output formats.
  • Bug Fixes

    • Improved resilience when event values can’t be rendered by emitting null instead of failing.
    • Strengthened template parsing, validation, and cleanup during startup and shutdown.

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@cosmo0920
cosmo0920 requested a review from edsiper as a code owner July 10, 2026 11:01
@cosmo0920 cosmo0920 added this to the Fluent Bit v5.1 milestone Jul 10, 2026
@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 15776971-a3b8-4cb7-a8eb-a0bfbe058c9c

📥 Commits

Reviewing files that changed from the base of the PR and between c7dbd50 and 5f4b7fa.

📒 Files selected for processing (1)
  • plugins/in_winevtlog/winevtlog.c
🚧 Files skipped from review as they are similar to previous changes (1)
  • plugins/in_winevtlog/winevtlog.c

📝 Walkthrough

Walkthrough

The winevtlog input adds optional named EventData maps using Windows event templates. It introduces template caching, validates cache configuration, parses metadata field names, updates all packing formats, and strengthens string-insert rendering cleanup.

Changes

Windows EventData mapping

Layer / File(s) Summary
Template contract and cache lifecycle
plugins/in_winevtlog/winevtlog.h, plugins/in_winevtlog/in_winevtlog.c
Adds event-template metadata and cache fields, configures event_data_as_map and cache sizing, validates initialization, allocates the cache, and destroys it during exit.
Template parsing and retrieval
plugins/in_winevtlog/winevtlog.c
Adds wide-character XML parsing, event metadata template loading, cache lookup and eviction, and template destruction.
Event data packing and read integration
plugins/in_winevtlog/pack.c, plugins/in_winevtlog/winevtlog.c
Serializes variants through shared helpers, emits compatible EventData maps keyed by template names, preserves structured text output, passes templates through all packing paths, and hardens string-insert rendering cleanup.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant winevtlog_read
  participant winevtlog_event_template_get
  participant get_string_inserts
  participant winevtlog_pack_event
  winevtlog_read->>winevtlog_event_template_get: retrieve event template
  winevtlog_read->>get_string_inserts: render event values
  winevtlog_read->>winevtlog_pack_event: pass template and string inserts
  winevtlog_pack_event->>winevtlog_pack_event: emit named EventData map
Loading

Possibly related PRs

Suggested reviewers: edsiper, fujimotos, koleini

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: adding optional EventData key-value enrichment in in_winevtlog.
Linked Issues check ✅ Passed The PR adds the requested event_data_as_map mode and derives EventData maps from event templates, preserving Data Name fields as named pairs.
Out of Scope Changes check ✅ Passed The added template cache and packing refactor support the requested EventData mapping feature and stay within the issue scope.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch cosmo0920-enrich-event-data-on-in_winevtlog

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c7dbd50d8a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/in_winevtlog/winevtlog.c Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
plugins/in_winevtlog/winevtlog.c (1)

731-825: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoff

Metadata enumeration cost on cache miss + permanent negative caching.

event_template_load opens publisher metadata and linearly scans every event via EvtNextEventMetadata until it matches event_id/version. For providers with large event manifests this is an O(N) walk per unique (provider,event_id,version) on the collector thread. It is bounded by the cache so amortized cost is low, but two points are worth confirming:

  • The scan happens synchronously in the read path; a provider with thousands of events will pay a one-time spike on first sight of each event.
  • A failed/partial load (valid == FLB_FALSE) is cached and never retried, so transient metadata unavailability permanently suppresses the EventData map for that key until restart.

Neither is a blocker; consider a short TTL / re-validation for invalid entries if transient failures are a concern.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/in_winevtlog/winevtlog.c` around lines 731 - 825, Review the cache
handling around event_template_load and the code that consumes
event_template->valid; failed or partial metadata loads must not be permanently
cached. Add a short retry/expiration policy for invalid entries, re-invoking
event_template_load after the TTL while retaining successful entries
indefinitely, and ensure the retry state is synchronized with existing cache
access.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@plugins/in_winevtlog/winevtlog.c`:
- Around line 731-825: Review the cache handling around event_template_load and
the code that consumes event_template->valid; failed or partial metadata loads
must not be permanently cached. Add a short retry/expiration policy for invalid
entries, re-invoking event_template_load after the TTL while retaining
successful entries indefinitely, and ensure the retry state is synchronized with
existing cache access.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 21a4ee60-471e-4be7-9774-9653e6e1a8c6

📥 Commits

Reviewing files that changed from the base of the PR and between 5fd6b28 and c7dbd50.

📒 Files selected for processing (4)
  • plugins/in_winevtlog/in_winevtlog.c
  • plugins/in_winevtlog/pack.c
  • plugins/in_winevtlog/winevtlog.c
  • plugins/in_winevtlog/winevtlog.h

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

in_winevtlog: option to emit EventData as named key/value pairs instead of the positional StringInserts array

2 participants