`labeled_counter` metrics use sample counts instead of counter values

[omansfeld]

The Bug

While looking at the networking.http_3_congestion_event_reason labeled_counter (labels: loss, ecn_ce) I noticed a significant discrepancy between GLAMs view (~16% ecn_ce earlier this year) vs what I remembered (~3% ecn_ce at the time).

See https://glam.telemetry.mozilla.org/fog/probe/networking_http_3_congestion_event_reason/explore?activeBuckets=%5B%22loss%22%2C%22ecn-ce%22%5D&timeHorizon=ALL vs the screenshot below that I used in a talk earlier this year:

Investigation

I looked into this and it seems like GLAM now uses sample counts instead of the actual counter values when calculating percentages.

I.e. a client sending a ping with loss = 9 and ecn_ce = 1 has one sample for each, thus it records it as 50% loss and 50% ecn_ce instead of the real 9/1 -> 90%/10% proportion.

This is also visible in those two (ai assisted) queries that I wrote to understand what is happening (and what the data actually looks like):

1. Using actual proportions (in a hacky way because I tried to use the glam aggregate table): https://sql.telemetry.mozilla.org/queries/121187#295620

versus

2. trying to mirror what GLAM seems to do (using client counts): https://sql.telemetry.mozilla.org/queries/121188#295622

Impact

I assume this affects all labeled counters on GLAM. I have checked a few others where I remember rough values and they all don't match what I expected them to show.

I guess (hope) this is a bug and not intended. At least for me GLAM is not useful for labeled counters with this. And I am in general a big fan and try to use GLAM over STMO (and design metrics so they look useful on GLAM) whenever possible.

┆Issue is synchronized with this Jira Task

[edugfilho]

This is intended per https://mozilla-hub.atlassian.net/browse/DENG-7764.
The good news is, it's not a hard change to make. @omansfeld would it help if I added a dropdown where you could pick the aggregation type? The right aggregation varies per metric. In the case of networking_http_3_congestion_event_reason I believe sum would be the right one to use.

[omansfeld]

Hey, thanks for the quick answer.

It sounds unintuitive for me to ignore the counter values for labeled_counter and the approach discussed in the JIRA ticket where client A submits {label1: 1}, {label2: 1} and client B submits {label1: 10} and we get {label1: 1.5}, label2: 0.5} seems more like Client ID normalization to me, which as far as I can tell is already it's own option in the top row of dropdowns.

Anyways, I might be lacking context here and only have the narrow view of how we use labeled counters in my scope. But imo the dangerous thing is that the current aggregation looks correct without very in depth knowledge about how we record the metric in question or remembering it's historical values. Which then can lead to very wrong assumptions (or a lot of wasted time trying to get custom queries and the GLAM display to match in my case).

Having a dropdown would definitely help and then I think to some extend the person viewing GLAM has the responsibility to check that they are using the correct options for the data they are viewing (or ask an expert if unsure), so in the grand scheme of things it would probably be fine.

[edugfilho]

@omansfeld thank you for that input. I dug for more context and here are a few points:

• I realized adding the dropdown won't help: sum is already what's being used... exposing avg/max etc would only produce other per-client weightings, none more correct as a category comparison.

• Why we don't tend to use raw SUM of values for the proportions: A value sum is frequently dominated by a few hyper-active clients, so the bars stop reflecting the typical client. E.g. 1M clients each report category A once, while 10 stuck clients report B 1M× each → SUM says "B = 91%" when really ~all clients are in A.

• What I fixed (normalized view): The categorical % was being normalized by sample/ping count, not clients. So chatty clients were over-weighted. It now uses distinct clients per category (total_users(key) / Σ total_users), which is robust to activity volume and keeps the bars summing to 100%. The y-axis is also pinned to 0–100% with whole-percent ticks.

• Your expected behavior was also introduced, now under "non-normalized.": The sum-of-values-by-category view you were after is now what the non-normalized toggle shows. Heads up that it's an approximation (reconstructed from bucketed data, so coarsest in the tail) and outlier-sensitive... which is why it's not the default view.

Ready for review: #3525

[omansfeld]

Got it, thanks! The raw sums being bad for the reasons you outlined definitely makes a lot of sense. We probably need some type of per client normalization.

One question for me to better understand the default view I'll be seeing and to make sure I properly articulated my point: Is the normalization done over the label sums per client, or do we just count label occurrence per client?

I.e if we have

clientA: {label1: 9, label2: 1},
clientB: {label1: 1, label2: 1}

do these normalize to option A:

{label1: 2 occurrences}, [label2: 2 occurrences} --> 50% each

or option B:

clientA: {90%, 10%}, clientB: {50%, 50%} 
--> label1 = (0.9+0.5)/2 = 0.7
--> label2 = (0.1+0.5)/2 = 0.3

Both ways are accounting and normalizing for your example of a few clients submitting a very high {label1: 10mil), but only option B would correctly display if most clients were to report {label1: 9, label2: 1} or a similar label distribution.

That was the case for my metric networking_http_3_congestion_event_reason. Most clients reported many loss and only a few clients reported ecn_ce. After we fixed a loss detection bug now most clients should either report loss or ecn or in rare cases multiple of each or a mixture. But that change wasn't visible in GLAM because (afaict) a client with {loss: 10, ecn_ce: 1} got normalized to 50% each, which I think is misleading.

[edugfilho]

Yea, unfortunately it's doing A now.
And I see it can get misleading. B is the true client normalization (the type we do with histograms), but it isn't going to be straightforward to implement, because it involves ETL changes and potentially a big backfill.

In case you're curious: To implement B I'll need to make changes in the ETL that account specifically for labeled counters that are categorical, which currently follow the same path as non-categorical labeled_counters, and that path buckets each label independently and collapses clients into counts early on, thus removing the possibility of computing a per-client cross-label ratio.
But that's the easy part. Where it gets more complicated is the fact I'll probably have to backfill a ton of data, unless we're ok with having a chart that shows a different type of aggregation from a certain point (I'm interested to hear your thoughts on that btw)... In any case, it's a thing I'll have to discuss with people. But given that that's the original intention of that Jira ticket and I feel bad for having cut corners there (maybe we wouldn't need a backfill then?), I'm willing to do this now-ish.