WIP: prototype for logging aggregation of output views and counts.

[cfkoehler]

Add drop-off view-count stats (placeStats for output)

Motivation

When someone changes the drop-off/output config (OUTPUT_TYPE / DENYLIST) that controls which views get written, there's currently no way to see the effect at runtime — you either diff output files by hand or wait for a downstream consumer to notice. This adds an always-available, low-overhead counter that answers "in the last N minutes, how many objects had view X written, broken down by file type and status?" — for live verification of drop-off config changes.

It mirrors the existing placeStats/ResourceWatcher pattern: accumulate counts in memory, flush an aggregated summary to a dedicated logfile on an interval, then reset — so each line is a rolling window.

What it does

A new Namespace-bound singleton, emissary.output.stats.ViewOutputStats, accumulates counts keyed by viewName × fileType × status and a daemon scheduler flushes-and-resets them to a dedicated dropOffStats logfile every interval. Each flush emits one structured (Logstash JSON) line per non-zero key:

{"node":"localhost","viewName":"NormalizedText","fileType":"PDF","status":"OK","count":42,"windowSeconds":60}

Hooks are wired into the framework's default output filters so it works out of the box:

• DataFilter (raw output) — records each view with OK / WRITE_FAILED when written, and NOT_OUTPUTTABLE for views present but suppressed by config (the key config-verification signal, since DataFilter gates per-view).
• AbstractRollableFilter (JsonOutputFilter, XmlOutputFilter) — records each output object's primary view + non-empty alternate views as OK. These filters gate at the object level (all views or none), so there's no NOT_OUTPUTTABLE breakdown on this path.

The API is deliberately schema-agnostic (record(viewName, fileType, status) takes plain strings), so downstream projects can feed their own view names and statuses without emissary depending on their types.

Enablement

Off by default. Enable with env var LOG_DROPOFF_STATS=true, which both wires up the aggregator (EmissaryNode) and enables the logback file appender. Flush cadence is configured in emissary/output/stats/ViewOutputStats.cfg (INTERVAL, INTERVAL_UNIT; currently 1 MINUTE). When disabled, ViewOutputStats.record(...) is a cheap no-op (singleton not bound).

Files

• New: emissary/output/stats/ViewOutputStats.java, ViewOutputStats.cfg, ViewOutputStatsTest.java
• directory/EmissaryNode.java — create/bind when enabled
• server/EmissaryServer.java — final flush + shutdown alongside ResourceWatcher
• output/filter/DataFilter.java, output/filter/AbstractRollableFilter.java — counting hooks
• config/logback.xml — dropOffStats appender + logger (mirrors objectTrace)

How to test / evaluate

1. Start with the feature on:

   LOG_DROPOFF_STATS=true ./emissary server -a 2

2. Confirm the startup line:
   INFO emissary.directory.EmissaryNode - Started drop-off view stats...ViewOutputStats flushing every 1 MINUTES
3. Drop files into target/data/InputData/, wait ≤1 minute.
4. Inspect the logfile: ${emissary.output.root}/dropOffStats/<node>-<port>-dropoff-stats.log — expect one JSON line per viewName × fileType × status window.
5. Verify the use case: flip an OUTPUT_TYPE/DENYLIST entry for the active filter and confirm the next window shows counts move between OK and NOT_OUTPUTTABLE (on the DataFilter path).

Unit coverage: ViewOutputStatsTest (accumulation, flush-and-reset window, concurrent counting with exact totals, no-op-when-unbound, lifecycle); DataFilterTest#testViewOutputStatsRecorded and JsonOutputFilterTest#testViewOutputStatsRecorded drive the hooks end-to-end.

Notes

• A zero-count window writes nothing (no empty lines). Counts also flush on graceful server shutdown.
• No new dependencies (reuses Codahale-style pattern, logback Logstash encoder already used by objectTrace).

AI Disclosure

• AI assistance used
• Tool(s): Open Code (qwen3-coder-30b)
• Scope:
  • Explored the codebase and created the implementation plan
  • Implemented the feature (aggregator, filter hooks, lifecycle wiring, logback)
  • Suggested and wrote unit tests
  • Summarized PR Description
• Human verification:
  • I reviewed and edited all generated code
  • I ran tests, checkstyle, and PMD locally, and evaluated the feature end-to-end locally with upstream intergration
  • I understand and can explain all changes
• Risk areas touched:
  • Output filter write path and server startup/shutdown wiring — all gated behind LOG_DROPOFF_STATS and off by default