Refactoring: Real-time rules evaluation

[tusharkhatriofficial]

@CharlieHelps, suggest better algorithms for real time evaluator. currently it evaluates rules on arrival of each event, which is not efficient when thousands of events are ingested within a second. We used Kafka to increase thoughput but if real time evaluator is this weird, it won't sit well within the architecture.

[charliecreates]

Expand this to see my work.

• Scanned the repo structure and docs (including EVENTARA_TECHNICAL_REPORT.md, THRESHOLD_RULES_COMPLETE.md, and PHASE_4_COMPLETION_SUMMARY.md) to locate the real-time evaluation components and understand the existing rule engine design.
• Opened and reviewed RealTimeRuleEvaluator.java to confirm that evaluateEvent(EventDto event) pulls all active threshold AlertRules and evaluates them per event, including filter checks and threshold logic backed by Redis metrics and cooldowns.
• Inspected EventConsumer.java to see how Kafka events flow into the system and how RealTimeRuleEvaluator is invoked within the ingestion pipeline, validating that current behavior is strictly per-event evaluation rather than batched or windowed processing.
• Started mapping out alternative designs that keep Redis-based metrics and cooldown semantics but switch from per-event rule scanning to batched evaluation keyed by metric windows and rule groups, so they align better with Kafka throughput patterns.
• Reviewed RedisMetricsService.recordEvent(EventDto) and getMetrics* methods to understand the time-bucketed key scheme (metrics:bucket:{bucketTimestamp}*) and what aggregate signals are already available for driving rule evaluation.
• Checked MetricsProperties to confirm configurable bucket sizing (sizeSeconds) and Redis retention (redisRetentionMinutes), which informs how large a batch/window a new evaluator design can safely use.
• Inspected DistributedMetricsService to see how RedisMetricsService is composed into higher-level metrics (getComprehensiveMetrics, throughput builders, and rollup scheduling), ensuring any new real-time evaluator algorithms can reuse MetricsBucket windows instead of re-scanning rules per event.

[charliecreates]

I took a look at the current implementation: EventConsumer calls RealTimeRuleEvaluator.evaluateEvent() for every ingested event, and evaluateEvent() loops over every active rule and (often) recomputes windowed metrics from Redis per rule. With “thousands of events/sec”, that effectively becomes O(events * rules * buckets-in-window) Redis work, which won’t scale no matter how much Kafka helps.

Here are the algorithmic patterns that typically fix this, in increasing order of impact:

1. Micro-batch / debounce evaluation (best “quick win”)
   Instead of evaluating rules on every event, evaluate on a short fixed tick (e.g. every 250ms / 1s), using the latest metrics.

• On event ingest: set a dirty flag (or increment a counter).
• A scheduled task runs every tickMs:
  • if not dirty, do nothing
  • else fetch metrics once and evaluate relevant rules, then clear dirty

This changes the load from “events drive evaluation” to “time drives evaluation” while still feeling “real-time” (sub-second alerting).

2. Group rules by “evaluation key” + memoize metrics reads per tick
   Even if you keep instant-ish evaluation, you can avoid recomputing the same Redis windows for many rules.

Create an EvaluationKey like:

• windowMinutes
• filter dimension (global vs sourceFilter=X vs eventTypeFilter=Y)
• (optional) anything else that changes which Redis keys are read

Then per tick:

• compute bucket = getFilteredMetrics(key) once
• evaluate all rules in that group against the same bucket

This usually cuts Redis reads by 10–100x when many rules share the same windows/filters.

3. Rule indexing (avoid scanning all rules)
   Right now it’s “for each event → loop all rules → check filters”. Flip it:

• Keep indexes in memory when you refresh the rule cache:
  • rulesGlobal
  • rulesBySource[source]
  • rulesByEventType[type]
  • (and/or severity, if that’s a common filter)
• On ingest (or per tick), select candidate rules from these indexes instead of scanning everything.

This helps a lot when most rules are scoped (per source / per event type).

4. Incremental / stateful evaluation (true streaming algorithm)
   For threshold-style rules, the most scalable approach is to maintain the rule’s state as counters/aggregates and update them incrementally as events arrive (sliding window ring buffer). Then you only do constant work per event, and only fire when a value crosses the threshold.

This is basically what stream processors (Kafka Streams / Flink) do well.

Architecture direction (fits Kafka best)

If you want this to sit cleanly in a Kafka architecture, the cleanest separation is:

• Ingestion service: validates/persists events, publishes to Kafka (already), maybe updates “raw” metrics.
• Metrics aggregator (Kafka Streams/Flink or a dedicated consumer): builds windowed metrics per key (global/source/type) and publishes compact “metric snapshot” events to another topic (e.g. metrics_1s, metrics_10s).
• Rule evaluator: consumes metric snapshots (not raw events), evaluates rules, emits alerts.

Benefits:

• evaluator load becomes proportional to “metrics updates” rather than “raw events”
• horizontal scaling aligns with Kafka partitions
• much less Redis round-tripping during evaluation

Even if you don’t bring in Streams/Flink, you can emulate this with a second consumer group + an internal “metrics tick” topic.

A couple of pragmatic notes (not “algorithms”, but they matter at high throughput):

• RealTimeRuleEvaluator logs at info for every event and even for per-condition evaluation; that alone can crush throughput.
• The evaluator currently calls getMetricValue(...) twice in the simple threshold path (duplicate line) — minor but unnecessary work on the hot path.

If you answer these, I can recommend the best “minimal change” path:

1. Roughly how many active threshold rules do you expect (10 / 100 / 1k / 10k)?
2. What alert latency is acceptable (100ms / 1s / 10s)?
3. Are most rules global, or scoped by sourceFilter / eventTypeFilter?

[tusharkhatriofficial]

@CharlieHelps

1. 100 to 500 rules maybe upto 1k
2. 1s is acceptable
3. They are mostly global

also evaluate the Drools thing, will it be required becuase i think using Drools for rules is the best path for production-grade rules

[charliecreates]

Expand this to see my work.

• Reviewed RedisMetricsService.recordEvent to understand how per-bucket keys (metrics:bucket:{ts} plus :source, :type, and :severity) and latency ZSETs (metrics:latencies:{ts}) are written with TTL based on MetricsProperties.Bucket.redisRetentionMinutes.
• Analyzed RedisMetricsService.getMetrics, getMetricsLastMinutes, and getMetricsPreviousWindow to see how they delegate to aggregateBuckets for whole-window and previous-window rule evaluations.
• Inspected aggregateBuckets and getBucket to map out the cost per evaluation, including looping over all 10s buckets in a window, merging MetricsBucket fields, loading per-source/type/severity breakdowns, and recomputing percentiles from combined latencies.
• Reviewed aggregateBucketsForSource and getMetricsForSourcePreviousWindow usage to understand how source-scoped rules avoid scanning unrelated keys and how that affects complexity.
• Examined loadTypeMetrics and loadSeverityMetrics to confirm they use pattern-based key lookup (e.g., metrics:bucket:{ts}:type:*) and hash reads, identifying where Redis key scans could become a hotspot at high cardinality.
• Checked MetricsProperties defaults (sizeSeconds = 10, redisRetentionMinutes = 60) to quantify the typical number of Redis buckets touched when evaluating rules over common windows (e.g., 5–15 minutes).

[charliecreates]

#28 (comment)

Given 100–1k rules, mostly global, and ~1s latency is fine, the best production-grade path (with minimal architectural churn) is: stop evaluating on every event and instead evaluate on a fixed 1s tick using a shared metrics snapshot.

That changes the complexity from O(events * rules * redisReads) to roughly O(rules / second + few redisReads / second).

What I’d change (algorithmically) in Eventara

1) Micro-batch / tick-driven evaluation (the big win)
Right now RealTimeRuleEvaluator.evaluateEvent() is called for every event and loops all active rules.

For your constraints, flip it to:

• On ingest (Kafka consumer): keep doing redisMetrics.recordEvent(event) as you do now
• Also set a dirty = true flag (or increment a counter)
• A scheduled job runs every 1000ms:
  • if not dirty → no-op
  • else:
    • load active rules once (cachedRules already exists)
    • group rules by (timeWindowMinutes, filterKey)
      since they’re mostly global, this is basically “group by windowMinutes”
    • for each group:
      • fetch metrics once for that window
      • evaluate all rules in that group against the same MetricsBucket
    • clear dirty

This is the simplest way to make Kafka throughput “matter” because evaluation is no longer in the hot path for every message.

2) Memoize Redis reads per tick (especially for EVENT_RATIO / derived metrics)
Inside the tick, keep a small map like:

• global(windowMinutes) -> MetricsBucket
• eventType(eventType, windowMinutes) -> MetricsBucket (only if any rule actually uses it)

That prevents EVENT_RATIO rules from repeatedly calling getMetricsForEventType() per rule within the same tick.

3) Biggest hidden bottleneck: current metrics reads are expensive even if you tick
Even if you only evaluate once per second, right now redisMetrics.getMetricsLastMinutes() eventually goes through RedisMetricsService.aggregateBuckets(), which calls getBucket() for each bucket, and getBucket() does things that are very expensive at scale (notably KEYS pattern scans for :source:* / :type:*, plus pulling and sorting latency ZSET data).

For global threshold rules, you typically only need a small subset (events/errors/latency sum+count/min/max). So you’ll want a fast-path global aggregation that does not:

• scan keys (KEYS)
• load per-source / per-type maps
• compute percentiles unless you actually have P50/P95/P99 rules

(Otherwise the evaluator will still be doing a lot of Redis work per tick.)

4) Multi-instance behavior
You already have Redis cooldown keys (eventara:rule:cooldown:*). That’s good: if multiple instances tick at the same time, cooldown makes it “mostly safe”.

If you want to reduce duplicate evaluation work, add a simple Redis leader lock (e.g. SET evaluator:leader <id> NX PX 1500) so only one instance runs the tick.

5) Quick throughput footgun
evaluateEvent() currently logs at info per event and per rule. Even with a better algorithm, that volume will crush throughput. For high-ingest paths, keep it at debug or sample it.

Do you need Drools?

Probably not required for what Eventara is doing today (threshold/ratio/rate-of-change on aggregated metrics).

Drools is “production-grade” when you need things like:

• complex business rules over many fact types (joins/patterns), conflict resolution, agendas
• sophisticated rule authoring workflows
• lots of shared sub-conditions where RETE-style incremental matching helps

But in your current model, the “facts” are basically a metrics snapshot (counts/rates over windows). With that input shape:

• Drools still ends up evaluating a lot of rules per snapshot
• it does not remove the need to micro-batch (you still don’t want to “fire all rules per event”)

So I’d treat Drools as an optional future extension if/when you add truly general-purpose rule types (field-level predicates over event payloads, multi-entity correlation, etc.). For the current threshold rules, a purpose-built evaluator will be:

• easier to control latency
• easier to reason about
• easier to optimize around Redis access patterns

Two quick questions that affect the best “fast-path metrics” design:

1. What are the common timeWindowMinutes values you expect in real use (1 / 5 / 15 / 60)?
2. Do you actually plan to alert on percentiles (P95/P99), or mostly counts/error-rate/avg-latency?

[tusharkhatriofficial]

@CharlieHelps, i talked to claude and i suggested a new algo which i could think off, i asked him about Adaptive rate based evaluation and it generated a whole plan which im pasting below for your review let me know what you think, and should we proceed with this and also let me know if we should use another algo not this one for better performance. I would also love to hear possible improvements in this one.

Adaptive Rate-Based Evaluation System

Overview

This document describes the proposed Adaptive Rate-Based Evaluation algorithm for Eventara's real-time rule evaluation engine. The current implementation evaluates rules on every incoming event, which creates O(events × rules × redis) complexity - unsustainable at high throughput.

The adaptive approach dynamically adjusts evaluation frequency based on incoming event rate, optimizing for both low-latency alerting during high traffic and resource conservation during idle periods.

---

Problem Statement

Current Architecture

┌──────────────────────────────────────────────────────────────────┐
│                     Current: Per-Event Evaluation                 │
├──────────────────────────────────────────────────────────────────┤
│                                                                   │
│   EventConsumer.ConsumeEvent()                                    │
│         │                                                         │
│         ▼                                                         │
│   realTimeRuleEvaluator.evaluateEvent(event)  ← Called per event │
│         │                                                         │
│         ▼                                                         │
│   for (AlertRule rule : rules) {              ← Loop 100-1k rules│
│       bucket = getFilteredMetrics(config);    ← Redis call       │
│       if (threshold crossed) fireAlert();                        │
│   }                                                              │
│                                                                   │
│   At 1000 events/sec × 500 rules = 500,000 Redis ops/sec 🔥      │
└──────────────────────────────────────────────────────────────────┘

Issues

1. Redis Overload: Each event triggers O(rules) Redis reads
2. Duplicate Work: Same metrics re-fetched for every event
3. No Batching: Window-based metrics don't change per-event
4. Wasted CPU: Low-traffic periods still run expensive loops

---

Proposed Solution: Adaptive Rate-Based Evaluation

Core Concept

Instead of evaluating on every event, evaluate on a dynamic tick interval that adapts to the current event ingestion rate:

Event Rate (per second)	Evaluation Interval	Rationale
< 0.1 (< 5/min)	30 seconds	Idle mode - conserve resources
0.1 - 1 (5-50/min)	10 seconds	Low traffic - responsive but efficient
1 - 10 (50-500/min)	2 seconds	Medium traffic - near real-time
10 - 100 (500-5k/min)	500 milliseconds	High traffic - fast response
> 100 (> 5k/min)	100 milliseconds	Burst mode - maximum responsiveness

Complexity Reduction

┌──────────────────────────────────────────────────────────────────┐
│                  Proposed: Adaptive Tick Evaluation               │
├──────────────────────────────────────────────────────────────────┤
│                                                                   │
│   EventConsumer.ConsumeEvent()                                    │
│         │                                                         │
│         ▼                                                         │
│   eventCounter.incrementAndGet();   ← O(1) atomic operation      │
│   dirtyFlag.set(true);              ← O(1)                       │
│                                                                   │
│   ─────────────────────────────────────────────────────────────  │
│                                                                   │
│   @Scheduled (dynamic interval)                                   │
│         │                                                         │
│         ▼                                                         │
│   if (!dirty) return;               ← Skip if no new events      │
│   snapshot = captureMetricsOnce();  ← 1 Redis call per tick     │
│   for (AlertRule rule : rules) {                                 │
│       evaluateAgainstSnapshot(rule, snapshot);  ← No Redis!     │
│   }                                                              │
│   adjustInterval(eventRate);        ← Adapt for next tick       │
│                                                                   │
│   At 1000 events/sec = ~20 Redis ops/sec 🚀                     │
└──────────────────────────────────────────────────────────────────┘

---

Implementation Design

1. Event Rate Monitor

Track events per second using an atomic counter with sliding window:

@Component
public class EventRateMonitor {
    
    private final AtomicLong eventCounter = new AtomicLong(0);
    private volatile long lastSampleTime = System.currentTimeMillis();
    private volatile double currentEps = 0.0;  // Events per second
    
    // Exponential moving average factor (0.3 = 30% new, 70% old)
    private static final double EMA_ALPHA = 0.3;
    
    /**
     * Called on every event - O(1) operation
     */
    public void recordEvent() {
        eventCounter.incrementAndGet();
    }
    
    /**
     * Calculate EPS and update the moving average
     * Called periodically (e.g., every second)
     */
    public double sampleAndGetEps() {
        long now = System.currentTimeMillis();
        long elapsed = now - lastSampleTime;
        
        if (elapsed < 100) return currentEps; // Min sample interval
        
        long events = eventCounter.getAndSet(0);
        double instantEps = (events * 1000.0) / elapsed;
        
        // Exponential moving average for smooth transitions
        currentEps = (EMA_ALPHA * instantEps) + ((1 - EMA_ALPHA) * currentEps);
        lastSampleTime = now;
        
        return currentEps;
    }
    
    public double getCurrentEps() {
        return currentEps;
    }
}

2. Adaptive Evaluation Scheduler

Dynamic scheduling based on event rate:

@Service
@Slf4j
public class AdaptiveRuleEvaluator {
    
    private final EventRateMonitor rateMonitor;
    private final RuleRepository ruleRepository;
    private final RedisMetricsService redisMetrics;
    private final AlertTriggerHandler alertHandler;
    
    private final AtomicBoolean dirtyFlag = new AtomicBoolean(false);
    private volatile long currentIntervalMs = 10_000; // Start at 10s
    private volatile long lastEvaluationTime = System.currentTimeMillis();
    
    // Configurable thresholds
    private final EvaluationConfig config;
    
    /**
     * Called from EventConsumer on every event - O(1)
     */
    public void onEventIngested() {
        rateMonitor.recordEvent();
        dirtyFlag.set(true);
    }
    
    /**
     * Scheduled check - runs frequently but does minimal work
     */
    @Scheduled(fixedRate = 100) // Check every 100ms
    public void checkAndEvaluate() {
        long now = System.currentTimeMillis();
        long elapsed = now - lastEvaluationTime;
        
        // Not time yet, or no new events
        if (elapsed < currentIntervalMs || !dirtyFlag.get()) {
            return;
        }
        
        // Reset and evaluate
        dirtyFlag.set(false);
        lastEvaluationTime = now;
        
        try {
            evaluateAllRules();
        } finally {
            // Adjust interval based on current event rate
            double eps = rateMonitor.sampleAndGetEps();
            currentIntervalMs = calculateInterval(eps);
            
            log.debug("Adaptive interval adjusted: EPS={}, interval={}ms", 
                      eps, currentIntervalMs);
        }
    }
    
    private void evaluateAllRules() {
        List<AlertRule> rules = getCachedActiveRules();
        
        // Group rules by evaluation key to share Redis reads
        Map<EvaluationKey, List<AlertRule>> ruleGroups = groupByEvaluationKey(rules);
        
        for (Map.Entry<EvaluationKey, List<AlertRule>> entry : ruleGroups.entrySet()) {
            // ONE Redis call per group
            MetricsBucket bucket = fetchMetrics(entry.getKey());
            
            for (AlertRule rule : entry.getValue()) {
                evaluateRuleAgainstBucket(rule, bucket);
            }
        }
    }
    
    /**
     * Calculate evaluation interval based on events per second
     */
    private long calculateInterval(double eventsPerSecond) {
        if (eventsPerSecond < config.getIdleThreshold()) {
            return config.getIdleIntervalMs();      // 30s for idle
        }
        if (eventsPerSecond < config.getLowThreshold()) {
            return config.getLowIntervalMs();       // 10s for low
        }
        if (eventsPerSecond < config.getMediumThreshold()) {
            return config.getMediumIntervalMs();    // 2s for medium
        }
        if (eventsPerSecond < config.getHighThreshold()) {
            return config.getHighIntervalMs();      // 500ms for high
        }
        return config.getBurstIntervalMs();         // 100ms for burst
    }
}

3. Evaluation Configuration

Externalized configuration for tuning:

@Component
@ConfigurationProperties(prefix = "eventara.evaluation.adaptive")
@Data
public class EvaluationConfig {
    
    // Event rate thresholds (events per second)
    private double idleThreshold = 0.1;      // < 5/min
    private double lowThreshold = 1.0;       // < 50/min
    private double mediumThreshold = 10.0;   // < 500/min
    private double highThreshold = 100.0;    // < 5000/min
    
    // Corresponding intervals (milliseconds)
    private long idleIntervalMs = 30_000;    // 30 seconds
    private long lowIntervalMs = 10_000;     // 10 seconds
    private long mediumIntervalMs = 2_000;   // 2 seconds
    private long highIntervalMs = 500;       // 500ms
    private long burstIntervalMs = 100;      // 100ms
    
    // Feature flag
    private boolean enabled = true;
    
    // Minimum events before evaluation (prevents false alerts)
    private int minEventsToEvaluate = 1;
}

4. Rule Grouping for Metrics Sharing

Group rules by evaluation key to minimize Redis calls:

/**
 * Key that determines which rules can share the same metrics fetch
 */
@Value
public class EvaluationKey {
    int windowMinutes;
    List<String> sourceFilter;    // null = global
    List<String> eventTypeFilter; // null = all types
    
    public static EvaluationKey fromRule(AlertRule rule) {
        Map<String, Object> config = rule.getRuleConfig();
        return new EvaluationKey(
            parseIntOrDefault(config.get("timeWindowMinutes"), 15),
            (List<String>) config.get("sourceFilter"),
            (List<String>) config.get("eventTypeFilter")
        );
    }
}

// Usage in evaluation
Map<EvaluationKey, List<AlertRule>> groups = rules.stream()
    .collect(Collectors.groupingBy(EvaluationKey::fromRule));

// Result: Rules with same window+filters share ONE Redis call
// e.g., 500 rules might reduce to 20-50 unique Redis calls

---

Migration Path

Phase 1: Add Alongside Existing (1-2 days)

1. Create AdaptiveRuleEvaluator as new component
2. Add feature flag eventara.evaluation.adaptive.enabled=false
3. Keep RealTimeRuleEvaluator.evaluateEvent() as fallback
4. Test both paths in parallel

Phase 2: Gradual Rollout (1 week)

1. Enable for 10% of traffic (canary)
2. Monitor latency, Redis ops, alert accuracy
3. Tune thresholds based on real data
4. Increase to 50%, then 100%

Phase 3: Cleanup (1 day)

1. Remove per-event evaluation path
2. Simplify EventConsumer to just increment counter
3. Document final configuration

---

Metrics & Observability

Key Metrics to Track

// Expose via Micrometer/Prometheus
@Gauge("eventara.evaluation.current_interval_ms")
public long getCurrentIntervalMs() { return currentIntervalMs; }

@Gauge("eventara.evaluation.current_eps")  
public double getCurrentEps() { return rateMonitor.getCurrentEps(); }

@Counter("eventara.evaluation.ticks_total")
public void recordTick() { /* ... */ }

@Histogram("eventara.evaluation.tick_duration_ms")
public void recordTickDuration(long durationMs) { /* ... */ }

Dashboard Queries

# Current evaluation interval
eventara_evaluation_current_interval_ms

# Events per second trend
rate(eventara_events_total[1m])

# Evaluation tick frequency
rate(eventara_evaluation_ticks_total[1m])

# Redis ops reduction
rate(redis_commands_total{command="HGETALL"}[1m])

---

Expected Benefits

Metric	Before (Per-Event)	After (Adaptive)	Improvement
Redis ops/sec @ 1000 EPS	500,000	~50	10,000x
CPU usage (evaluator)	High, constant	Scales with traffic	Variable
Alert latency @ idle	Instant	30s max	Acceptable
Alert latency @ burst	Instant	100ms max	Acceptable
Memory (per instance)	Low	Low + rate buffer	Minimal

---

Configuration Example

# application.yml
eventara:
  evaluation:
    adaptive:
      enabled: true
      
      # Event rate thresholds (events per second)
      idle-threshold: 0.1       # < 5 events/min
      low-threshold: 1.0        # < 50 events/min  
      medium-threshold: 10.0    # < 500 events/min
      high-threshold: 100.0     # < 5000 events/min
      
      # Evaluation intervals (milliseconds)
      idle-interval-ms: 30000   # 30 seconds
      low-interval-ms: 10000    # 10 seconds
      medium-interval-ms: 2000  # 2 seconds
      high-interval-ms: 500     # 500ms
      burst-interval-ms: 100    # 100ms
      
      # Smoothing factor for EPS calculation
      ema-alpha: 0.3

---

Risk Mitigation

Risk	Mitigation
Delayed alerts during idle	Set max interval cap (30s) that's business-acceptable
Flash traffic spikes	Quick ramp-up (100ms poll to detect, immediate switch to burst mode)
Metrics stale during evaluation	Use dirty flag to skip redundant evaluations
Configuration complexity	Sensible defaults, override only when needed

---

References

• Current implementation: RealTimeRuleEvaluator.java
• Redis metrics: RedisMetricsService.java
• Event ingestion: EventConsumer.java
• Rule storage: AlertRule.java (ruleConfig JSONB)

---

Implementation Checklist

• Create EventRateMonitor component
• Create EvaluationConfig configuration class
• Create EvaluationKey for rule grouping
• Create AdaptiveRuleEvaluator service
• Update EventConsumer to call onEventIngested()
• Add Micrometer metrics
• Add feature flag configuration
• Write integration tests
• Load test with realistic event patterns
• Document runbook for tuning thresholds

---

Document Version: 1.0
Created: 2026-02-04
Author: AI Agent Collaboration

[charliecreates]

#28 (comment)

The “adaptive rate-based evaluation” plan is directionally solid — it’s basically the same core win we discussed (tick-driven evaluation + shared metrics snapshot), with an extra knob to slow down during idle periods.

Given your constraints (100–1k rules, mostly global, ~1s latency acceptable), I’d proceed in this order:

1. Ship a fixed tick first (e.g., 1s) + rule grouping + per-tick metrics memoization. This removes the O(events × rules × redis) blow-up immediately and is much simpler to reason about.
2. Add adaptive intervals only if you actually care about conserving resources at very low traffic and you’re OK with higher idle latency (e.g., 10–30s).

Why I wouldn’t start with adaptive as Phase 1

• The complexity/risk is mostly in metrics fetching cost (Redis aggregation, KEYS-style scans, percentile work), not in whether the tick is 1s vs 500ms.
• Adaptive scheduling adds failure modes (jitter, uneven sampling, “why did my alert take 27s?”) that you don’t need to solve the big scaling problem.

Concrete improvements if you do implement the adaptive version

• Avoid “poll every 100ms” (@Scheduled(fixedRate = 100) + internal interval checks). It’s easy but wasteful and noisy.

  • Prefer a ScheduledExecutorService / Spring TaskScheduler where you reschedule the next run after each evaluation based on currentIntervalMs.
  • If you keep the 100ms poll, it’s still a big win vs per-event, but it’s unnecessary overhead.

• Don’t compute EPS only when you evaluate.

  • If you slow to 30s in idle, sampleAndGetEps() becomes coarse and can react late to spikes.
  • Better: have a small sampler that runs on a fixed cadence (ex: every 1s) updating currentEps (EMA), and the evaluator reads it.

• Be careful with the dirtyFlag race.

  • Pattern that works well:
    • onEventIngested(): increment counter (or set dirty)
    • tick(): hadEvents = dirtyFlag.getAndSet(false) (or compare counters)
    • if events arrive during evaluation, dirtyFlag becomes true again and you run next tick

• Leader election (multi-instance).

  • Cooldowns prevent duplicate alert spam, but you’ll still waste CPU/Redis if all instances tick.
  • A simple Redis lock (SET evaluator:lock <id> NX PX <interval+buffer>) or a library like ShedLock is enough.

• Biggest performance lever: fast-path metrics snapshots.

  • Even with a perfect adaptive scheduler, if each tick calls the expensive path in RedisMetricsService.aggregateBuckets() (esp. per-source/per-type scans + percentiles), you’ll still pay a lot per tick.
  • For “mostly global threshold rules”, implement a “global summary only” read path that avoids:
    • scanning :source:* / :type:* keys
    • percentile computation unless some rule actually needs it

• Rule grouping key should include “what metrics are needed”.

  • Not just (windowMinutes, filters) — also include whether the group needs percentiles, breakdowns, etc., so you don’t compute heavy metrics for rules that don’t need them.

Decision / recommendation

• Yes, proceed with the tick-driven approach, but I’d start with fixed 1s + grouping + metrics memoization + fast-path metrics. Add adaptive later if you want idle-mode savings.

One question to unblock the “adaptive” part

• When traffic is basically idle, is it acceptable for alerts to be delayed up to 10–30 seconds, or do you want to keep the “worst-case” latency around ~1 second at all times?