Implement updates tracking OWID grapher#6
Open
xrendan wants to merge 1279 commits into
Open
Conversation
🔨 (typescript) check for unused locals
Pass the full grapher config to getDatapageDataV2 for multi-dim views, instead of the partial view config from the multi-dim config. This seems like a general oversight that slipped through until now. It was uncovered thanks to an incorrect citation in "How to cite this data", which wasn't using the title from the full grapher config, where the title comes from the variable grapher config, and instead fell back to variable.display.name.
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Mirror analytics fixes from cloudflare-workers PR Ports the GA4 analytics changes from owid/cloudflare-workers#27 so the two mirrored analytics.ts files stay in sync. - Namespace Grapher query params under q_* in cf_function_invocation events. Until now `?country=DE` (a real Grapher param used by chart embeds, thumbnails, and downloads) overwrote the structured `country` field sourced from cf-ipcountry, silently corrupting the country column in dbt's raw_traffic / worker_traffic. Other query params pass through bare, except for collisions with structured names which are dropped via a reserved-key set (built from Object.keys(params) plus "status_code"). The Grapher key list comes from GRAPHER_QUERY_PARAM_KEYS in @ourworldindata/types — same source-of-truth used elsewhere in this repo (e.g. functions/_common/urlTools.ts). - Add a `host` param so the same event can carry which hostname served the request (e.g. ourworldindata.org vs api.owid.io). Drop the pathname argument from getCommonEventParams; it's derivable from request.url and was always passed by the single call site as such. - Switch from the hand-rolled parseCookies in cookieTools.ts to parseCookie from the `cookie` npm package (jshttp). Bumps the functions workspace dep to ^1.1.1 to pick up the parseCookie alias (parse is now deprecated upstream). - Add the reverse-link header comment pointing at the cloudflare-workers mirror, so future readers of either file know to update both. * 🐛🤖 Address analytics review comments * 🔨🤖 Inline analytics cookie helper
✨ hide tick marks when x-axis is shared
🔨 standardise AxisView components
🎉 (admin) add last edited note for map brackets
🔨 update regions file
Adds support for shared x-axis for faceted slope charts. This is a bit more involved because this involves a rewrite of using HorizontalAxis in slope charts.
🐝 revert changes to MultiDimRedirect
✨ touchups ✨cleanup code
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Marcel Gerber <mgerber59@gmail.com>
…stom color section is open
* Capture ASN (as_org/asn) on cf_function_invocation analytics events
Adds the request's network operator to the GA4 events emitted by the
analytics middleware, derived from Cloudflare's `request.cf` (same source as
the existing country lookup — NOT the client IP, which we don't forward):
- as_org: request.cf.asOrganization e.g. "Amazon.com", "Google Cloud", "Hetzner Online GmbH"
- asn: request.cf.asn e.g. 16509
Motivation: on the direct data-download channels (e.g. /grapher/*.csv) a large
chunk of traffic has a normal browser user-agent but no referrer and no
browsing footprint — undetectable as bots by UA alone. A hosting/cloud ASN
behind a browser UA is a strong, clean bot signal that lets the analytics
pipeline separate datacenter scrapers from real visitors. ASN is the network
operator, not PII, and needs no IP handling.
Defaults to "" / 0 when request.cf is absent (e.g. local dev).
NOTE: analytics.ts is mirrored in cloudflare-workers/utils/analytics.ts so the
proxy workers emit the same events — a matching change goes there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(analytics): narrow request.cf before reading ASN fields
request.cf is a union type (incoming metadata | request-init), so
request.cf?.asOrganization widened to {} and broke `tsgo -b -f` (TS2339 /
TS2322) — which gates the Functions deploy. Narrow request.cf to the two
fields we read, matching how api/detect-country casts request.cf?.country.
Addresses Codex P1 review on #6566.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Type request.cf as IncomingRequestCfProperties instead of casting
* Improve getCommonEventParams tests
The cf properties are always set on incoming requests so we only need
one test where we check all the attributes.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Martin Račák <martin.racak@ourworldindata.org>
Preview/staging Pages deployments were sampled at 100% (vs prod's 1%), so their traffic — dominated by bulk crawlers like ClaudeBot hitting *.owid.pages.dev — badly skewed raw GA4 event counts and mixed preview noise into prod analytics. Set the preview-env sampling rate to "0" so the worker sends nothing from previews (validation still passes; shouldSample is always false). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Capture Cloudflare bot-detection signals on analytics events Adds bot_score, verified_bot, verified_bot_category to cf_function_invocation events, read from request.cf (same source as as_org/asn). We're on Super Bot Fight Mode, not Enterprise Bot Management, so score/JA fingerprints are likely absent — these are captured to empirically confirm what populates on our plan, and would carry real data automatically if we ever add the Enterprise add-on. bot_score uses a -1 sentinel so "absent" stays distinguishable from a real 1–99 score. Not PII. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * 🤖 style: format code --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Marigold <1550888+Marigold@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Links to issues, Figma, Slack, and a technical introduction to the work.
Screenshots / Videos / Diagrams
Add if relevant, i.e. might not be necessary when there are no UI changes.
Testing guidance
Step-by-step instructions on how to test this change
Reminder to annotate the PR diff with design notes, alternatives you considered, and any other helpful context.
Checklist
(delete all that do not apply)
Before merging
If DB migrations exists:
After merging