Document how the dashboard scoring works #667
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,60 @@ | ||||||
| # Interop Dashboard | ||||||
|
|
||||||
| Our dashboard is available at https://wpt.fyi/interop. | ||||||
|
|
||||||
| This document explains how tests are run and how score are computed. There are several layers of async operations with varying amounts of delay, so this might also serve as a guide to identify where the problem is if the dashboard is not being updated. | ||||||
|
|
||||||
| ## Focus Area Definitions | ||||||
|
|
||||||
| The focus areas are defined by the combination of three data sources: | ||||||
|
|
||||||
| - [wpt-metadata](https://github.com/web-platform-tests/wpt-metadata) labels specific tests with labels like `interop-2022-subgrid`. These labels are added manually after review of the test lists. The easiest way to enumerate all labeled tests is through the [wpt.fyi API](https://wpt.fyi/api/metadata?includeTestLevel=true&product=chrome). | ||||||
| - [`category-data.json`](https://github.com/web-platform-tests/results-analysis/blob/main/interop-scoring/category-data.json) defines the focus areas for each year as an identifier like `interop-2024-layout` and a set of labels like `interop-2021-flexbox`, `interop-2022-subgrid`, etc. | ||||||
| - [`interop-data.json`](https://github.com/web-platform-tests/wpt.fyi/blob/main/webapp/static/interop-data.json) provides human readable names for the focus area identifiers, such as "Layout" for `interop-2024-layout`. This file also includes investigation efforts. The same data is available in [`interop-data.js`](https://wpt.fyi/components/interop-data.js). | ||||||
|
Member
There was a problem hiding this comment. It's also where |
||||||
|
|
||||||
| ## Focus Area Scoring | ||||||
|
|
||||||
| Scores for each focus area are computed in [results-analysis](https://github.com/web-platform-tests/results-analysis) and published as CSV ([example](https://raw.githubusercontent.com/web-platform-tests/results-analysis/gh-pages/data/interop-2024/interop-2024-experimental-v2.csv)) to GitHub pages. The CSV data and `interop-data.js` is fetched by the wpt.fyi frontend to populate the scores. | ||||||
|
Member
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| ## End-to-End Pipeline | ||||||
|
|
||||||
| The pipeline for updating scores is as follows: | ||||||
|
|
||||||
| - All of WPT is run on Chrome, Edge, Firefox, and Safari across two different CI systems: Taskcluster and Azure Pipelines. The cadence varies: | ||||||
|
Member
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| - Chrome Canary: [every commit](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/tools/ci/tc/tasks/test.yml#L188-L193) | ||||||
|
foolip marked this conversation as resolved.
|
||||||
| - Chrome Stable: [daily](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/tools/ci/tc/tasks/test.yml#L206-L211) | ||||||
| - Edge Dev: [every 3 hours](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/.azure-pipelines.yml#L380) | ||||||
| - Edge Stable: [daily](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/.azure-pipelines.yml#L344) | ||||||
| - Firefox Nightly: [every commit](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/tools/ci/tc/tasks/test.yml#L163-L168) | ||||||
| - Firefox Stable: [daily](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/tools/ci/tc/tasks/test.yml#L175-L180) | ||||||
| - Safari Technology Preview: [every 3 hours](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/.azure-pipelines.yml#L491) | ||||||
| - Safari Stable: [daily](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/.azure-pipelines.yml#L451) | ||||||
|
Comment on lines
+25
to
+32
Member
There was a problem hiding this comment. web-platform-tests/wpt@6f491c4 is over two years old — why are we linking to that commit?
Comment on lines
+31
to
+32
Member
There was a problem hiding this comment. Safari hasn't run on Azure pipelines in a year and a half: web-platform-tests/wpt@8870a21. https://github.com/web-platform-tests/wpt/blob/ec31a461d352a7a67ae972b347aea77d5ca9160b/.github/workflows/safari_technology_preview.yml#L24 and https://github.com/web-platform-tests/wpt/blob/ec31a461d352a7a67ae972b347aea77d5ca9160b/.github/workflows/safari_stable.yml#L24 are the relevant links now. |
||||||
|
|
||||||
| The 3 hourly and daily cadences are implemented using "epoch" branches which | ||||||
| are updated by a [GitHub Actions workflow](https://github.com/web-platform-tests/wpt/blob/master/.github/workflows/epochs.yml). | ||||||
|
Member
There was a problem hiding this comment. Why is this |
||||||
|
|
||||||
| Runs take 1-3 hours to complete. Webhooks notify wpt.fyi of the results, which downloads them and makes them available within minutes of completion. (Some CI setups instead use a results upload API, but not those discussed here.) | ||||||
|
Member
There was a problem hiding this comment. Only TC uses a webhook (see https://github.com/web-platform-tests/wpt/settings/hooks, though I realise most people can't see that). Azure Pipelines just sends a very simple request to wpt.fyi: https://github.com/web-platform-tests/wpt/blob/master/tools/ci/azure/fyi_hook.yml As does GitHub Actions: https://github.com/web-platform-tests/wpt/blob/master/.github/workflows/wpt_fyi_notify.yml The time it takes wpt.fyi to ingest results is also relatively variable; I don't know if we have good data on that anywhere? |
||||||
|
|
||||||
| Delay: 1-6 hours for experimental, 1-27 hours for stable | ||||||
|
|
||||||
| - [results-analysis-cache](https://github.com/web-platform-tests/results-analysis-cache) downloads all runs from wpt.fyi and coverts them into a more efficient format to speed up results scoring. This runs [hourly](https://github.com/web-platform-tests/results-analysis-cache/blob/683fe10f5f58c096e2a0dc3214c53ce949192c39/.github/workflows/results.yml#L7-L9) and takes less than 10 minutes. | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| Delay: 10-70 minutes | ||||||
|
|
||||||
| - [results-analysis](https://github.com/web-platform-tests/results-analysis) fetches results from `results-analysis-cache` and produces the CSV files used by the dashboard. This runs [every 3 hours](https://github.com/web-platform-tests/results-analysis/blob/67c603a9b306c71da37c4798301935318949c45e/.github/workflows/update_gh_pages.yml#L7-L9) and takes a few minutes. | ||||||
|
Member
There was a problem hiding this comment. https://github.com/web-platform-tests/results-analysis/blob/da8fe496b093e0c81cd66274594e2febe6b515fe/build.sh#L20 starts off by running the |
||||||
|
|
||||||
| Delay: 0-3 hours | ||||||
|
foolip marked this conversation as resolved.
|
||||||
|
|
||||||
| - The wpt.fyi frontend always fetches the latest CSV files, so there is no additional delay before the changes appear on the dashboard. | ||||||
|
|
||||||
| ## Troubleshooting | ||||||
|
|
||||||
| It's normal for experimental results to take up 10 hours to update and stable results over 30 hours. If things take longer than this it's time to troubleshoot. | ||||||
|
|
||||||
| First check if new runs are appearing in [wpt.fyi/runs](https://wpt.fyi/runs)? If not, the runs are probably not completing in Taskcluster or Azure Pipelines. Diagnosing why this happens is not always easy, so please [file an issue](https://github.com/web-platform-tests/wpt/issues/new) and poke someone for help. | ||||||
|
|
||||||
| Once wpt.fyi has the results it's rare for anything to go wrong, but it could be in one of the GitHub Actions workflows: | ||||||
|
|
||||||
| - [results-analysis-cache](https://github.com/web-platform-tests/results-analysis-cache/actions) | ||||||
| - [results-analysis](https://github.com/web-platform-tests/results-analysis/actions) | ||||||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want to call this
interop-dashboard.mdto make it clear, given other metrics are within the scope of the Interop team?