Add automated JSON schema validation by ajlende · Pull Request #8039 · Automattic/themes

ajlende · 2024-08-27T23:11:49Z

Changes proposed in this Pull Request:

Adds an npm script and GitHub action for validating JSON for themes.

Gets the schema from the "Requires at least" minimum WP version.
Warns when the $schema version doesn't match the minimum WP version.
Validates the theme.json and style variations against the minimum WP version schema.

Validation script

I may have had a bit too much fun with building the tables and adding colors, so the code exploded a bit. 😅

$ node theme-utils.mjs help validate-theme                    

validate-theme [--format=FORMAT] [--color=WHEN] [--table-width=COLUMNS] <array of theme slugs>

Validates a theme against the WordPress theme requirements.

--format=FORMAT

        Output format. Possible values: *table*, json, dir.

--color=WHEN

        Colorize the output for table or dir formats. The automatic mode only enables colors if 
        an interactive terminal is detected. Possible values: *auto*, always, never.

--table-width=COLUMNS

        Explicitly set the width of the table format instead of determining it automatically.
        Will default to 120 if omitted and width cannot be determined automatically.

But I think the new result is really nice. ✨

And it works with pagers.

$ # It can take a while to process longer lists of themes
$ node theme-utils.mjs validate-theme --color=always --table-width=$(( $(tput cols) )) atheme,adventurer,grammer,skatepark | less -R

Or without if you only have a theme or two to validate.

$ node theme-utils.mjs validate-theme grammer                                                                  

Progress: [ '■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■', 100 ] 1/1

╔═════════════════════════════════════════════════════════════╤══════════════════════════╗
║ ERROR                                                       │ Message : missing 'Requi ║
║ Theme : Grammer                                             │ res at least' header met ║
║ File  : style.css                                           │ adata                    ║
╟─────────────────────────────────────────────────────────────┼──────────────────────────╢
║ WARNING                                                     │ Actual   : undefined     ║
║ Theme : Grammer                                             │ Expected : 5.9 or greate ║
║ File  : style.css                                           │ r                        ║
║                                                             │ Message  : the 'Requires ║
║                                                             │  at least' version does  ║
║                                                             │ not support theme.json   ║
╟─────────────────────────────────────────────────────────────┼──────────────────────────╢
║ WARNING                                                     │ Instance path : /setting ║
║ Theme  : Grammer                                            │ s/spacing/spacingScale   ║
║ File   : theme.json                                         │ Schema path   : #/defini ║
║ Schema : https://schemas.wp.org/trunk/theme.json            │ tions/settingsSpacingPro ║
║                                                             │ perties/properties/spaci ║
║                                                             │ ng/properties/spacingSca ║
║                                                             │ le/additionalProperties  ║
║                                                             │ Keyword       : addition ║
║                                                             │ alProperties             ║
║                                                             │ Params        : { additi ║
║                                                             │ onalProperty: 'theme' }  ║
║                                                             │ Message       : must NOT ║
║                                                             │ have additional properti ║
║                                                             │ es                       ║
╟─────────────────────────────────────────────────────────────┼──────────────────────────╢
║ WARNING                                                     │ Instance path :          ║
║ Theme  : Grammer                                            │ Schema path   : #/requir ║
║ File   : assets/fonts/font-collection.json                  │ ed                       ║
║ Schema : https://schemas.wp.org/wp/6.5/font-collection.json │ Keyword       : required ║
║                                                             │ Params        : { missin ║
║                                                             │ gProperty: 'slug' }      ║
║                                                             │ Message       : must hav ║
║                                                             │ e required property 'slu ║
║                                                             │ g'                       ║
║                                                             ├──────────────────────────╢
║                                                             │ Instance path :          ║
║                                                             │ Schema path   : #/requir ║
║                                                             │ ed                       ║
║                                                             │ Keyword       : required ║
║                                                             │ Params        : { missin ║
║                                                             │ gProperty: 'name' }      ║
║                                                             │ Message       : must hav ║
║                                                             │ e required property 'nam ║
║                                                             │ e'                       ║
╚═════════════════════════════════════════════════════════════╧══════════════════════════╝

Validation passed with warnings.

And the added --format=json option is also super helpful when combined with jq to drill down into the data.

For example, this is the breakdown of all the current themes and a count of the types of problems that they have:

$ node theme-utils.mjs validate-theme --format=json $(find . -name 'theme.json' | awk -F/ '{print $2}' | uniq | sort | paste -s -d, -) | jq '.[].data[].message' | sort | uniq -c | sort -bgr
5815 "must NOT have additional properties"
 696 "must be object"
 551 "must be string"
 328 "the $schema version does not match style.css 'Requires at least' version"
 153 "must match exactly one schema in oneOf"
  73 "must be equal to constant"
  71 "the 'Requires at least' version does not support theme.json"
  71 "must be equal to one of the allowed values"
  47 "Missing $schema URI: undefined"
   6 "property name must be valid"
   5 "must be number"
   2 "must be >= 1"
   2 "missing 'Requires at least' header metadata"
   1 "must have required property 'version'"
   1 "must have required property 'slug'"
   1 "must have required property 'name'"

GitHub Action

The GitHub action only runs on the modified themes.

It can be tested by committing some modifications to a theme and running like this:

$ 
$ act -s GITHUB_TOKEN="$(gh auth token)" pull_request -j validate-theme
INFO[0000] Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock' 
WARN  ⚠ You are using Apple M-series chip and you have not specified container architecture, you might encounter issues while running act. If so, try running it with '--container-architecture linux/amd64'. ⚠  
[Validate Themes/validate-theme] 🚀  Start image=catthehacker/ubuntu:act-latest
[Validate Themes/validate-theme]   🐳  docker pull image=catthehacker/ubuntu:act-latest platform= username= forcePull=true
[Validate Themes/validate-theme]   🐳  docker create image=catthehacker/ubuntu:act-latest platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host"
[Validate Themes/validate-theme]   🐳  docker run image=catthehacker/ubuntu:act-latest platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host"
[Validate Themes/validate-theme]   ☁  git clone 'https://github.com/actions/setup-node' # ref=v4
[Validate Themes/validate-theme] Non-terminating error while running 'git clone': some refs were not updated
[Validate Themes/validate-theme] ⭐ Run Main Checkout
[Validate Themes/validate-theme]   🐳  docker cp src=/Users/ajlende/Documents/a8c-themes/. dst=/Users/ajlende/Documents/a8c-themes
[Validate Themes/validate-theme]   ✅  Success - Main Checkout
[Validate Themes/validate-theme] ⭐ Run Main Setup Node
[Validate Themes/validate-theme]   🐳  docker cp src=/Users/ajlende/.cache/act/actions-setup-node@v4/ dst=/var/run/act/actions/actions-setup-node@v4/
[Validate Themes/validate-theme]   🐳  docker exec cmd=[node /var/run/act/actions/actions-setup-node@v4/dist/setup/index.js] user= workdir=
| Resolved .nvmrc as 20
[Validate Themes/validate-theme]   💬  ::debug::isExplicit: 
[Validate Themes/validate-theme]   💬  ::debug::explicit? false
[Validate Themes/validate-theme]   💬  ::debug::isExplicit: 18.20.4
[Validate Themes/validate-theme]   💬  ::debug::explicit? true
[Validate Themes/validate-theme]   💬  ::debug::isExplicit: 20.10.0
[Validate Themes/validate-theme]   💬  ::debug::explicit? true
[Validate Themes/validate-theme]   💬  ::debug::isExplicit: 20.15.0
[Validate Themes/validate-theme]   💬  ::debug::explicit? true
[Validate Themes/validate-theme]   💬  ::debug::evaluating 3 versions
[Validate Themes/validate-theme]   💬  ::debug::matched: 20.15.0
[Validate Themes/validate-theme]   💬  ::debug::checking cache: /opt/hostedtoolcache/node/20.15.0/arm64
[Validate Themes/validate-theme]   💬  ::debug::Found tool in cache node 20.15.0 arm64
| Found in cache @ /opt/hostedtoolcache/node/20.15.0/arm64
[Validate Themes/validate-theme]   ❓  ::group::Environment details
| node: v20.15.0
| npm: 10.7.0
| yarn: 
[Validate Themes/validate-theme]   ❓  ::endgroup::
| [command]/opt/hostedtoolcache/node/20.15.0/arm64/bin/npm config get cache
| /root/.npm
[Validate Themes/validate-theme]   💬  ::debug::npm's cache folder "/root/.npm" configured for the root directory
[Validate Themes/validate-theme]   💬  ::debug::followSymbolicLinks 'true'
[Validate Themes/validate-theme]   💬  ::debug::followSymbolicLinks 'true'
[Validate Themes/validate-theme]   💬  ::debug::implicitDescendants 'true'
[Validate Themes/validate-theme]   💬  ::debug::matchDirectories 'true'
[Validate Themes/validate-theme]   💬  ::debug::omitBrokenSymbolicLinks 'true'
[Validate Themes/validate-theme]   💬  ::debug::Search path '/Users/ajlende/Documents/a8c-themes/package-lock.json'
[Validate Themes/validate-theme]   💬  ::debug::/Users/ajlende/Documents/a8c-themes/package-lock.json
[Validate Themes/validate-theme]   💬  ::debug::Found 1 files to hash.
[Validate Themes/validate-theme]   💬  ::debug::primary key is node-cache-Linux-npm-9df091025ae38d8512d20d182f34873fc5425eec86d0742d8c8bdbe8dd0f3e69
[Validate Themes/validate-theme]   💬  ::debug::Resolved Keys:
[Validate Themes/validate-theme]   💬  ::debug::["node-cache-Linux-npm-9df091025ae38d8512d20d182f34873fc5425eec86d0742d8c8bdbe8dd0f3e69"]
[Validate Themes/validate-theme]   💬  ::debug::Checking zstd --quiet --version
[Validate Themes/validate-theme]   💬  ::debug::1.4.8
[Validate Themes/validate-theme]   💬  ::debug::zstd version: 1.4.8
[Validate Themes/validate-theme]   💬  ::debug::Resource Url: http://192.168.68.55:64143/_apis/artifactcache/cache?keys=node-cache-Linux-npm-9df091025ae38d8512d20d182f34873fc5425eec86d0742d8c8bdbe8dd0f3e69&version=b3f0cb83629d634645a5146420c017462ebb5229bd60271a7a86e489a6066469
[Validate Themes/validate-theme]   💬  ::debug::Failed to delete archive: Error: ENOENT: no such file or directory, unlink ''
| npm cache is not found
[Validate Themes/validate-theme]   ❓ add-matcher /run/act/actions/actions-setup-node@v4/.github/tsc.json
[Validate Themes/validate-theme]   ❓ add-matcher /run/act/actions/actions-setup-node@v4/.github/eslint-stylish.json
[Validate Themes/validate-theme]   ❓ add-matcher /run/act/actions/actions-setup-node@v4/.github/eslint-compact.json
[Validate Themes/validate-theme]   ✅  Success - Main Setup Node
[Validate Themes/validate-theme]   ⚙  ::set-output:: node-version=v20.15.0
[Validate Themes/validate-theme]   ⚙  ::set-output:: cache-hit=false
[Validate Themes/validate-theme]   ⚙  ::add-path:: /opt/hostedtoolcache/node/20.15.0/arm64/bin
[Validate Themes/validate-theme] ⭐ Run Main Install dependencies
[Validate Themes/validate-theme]   🐳  docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/2] user= workdir=
| npm warn deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
| npm warn deprecated rimraf@2.7.1: Rimraf versions prior to v4 are no longer supported
| npm warn deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
| npm warn deprecated npmlog@5.0.1: This package is no longer supported.
| npm warn deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
| npm warn deprecated har-validator@5.1.5: this library is no longer supported
| npm warn deprecated intl-messageformat-parser@1.8.1: We've written a new parser that's 6x faster and is backwards compatible. Please use @formatjs/icu-messageformat-parser
| npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
| npm warn deprecated gauge@3.0.2: This package is no longer supported.
| npm warn deprecated domexception@4.0.0: Use your platform's native DOMException instead
| npm warn deprecated are-we-there-yet@2.0.0: This package is no longer supported.
| npm warn deprecated abab@2.0.6: Use your platform's native atob() and btoa() methods instead
| npm warn deprecated @npmcli/move-file@1.1.2: This functionality has been moved to @npmcli/fs
| npm warn deprecated @humanwhocodes/object-schema@2.0.3: Use @eslint/object-schema instead
| npm warn deprecated @humanwhocodes/config-array@0.11.14: Use @eslint/config-array instead
| npm warn deprecated rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
| npm warn deprecated rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
| npm warn deprecated rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
| npm warn deprecated npmlog@6.0.2: This package is no longer supported.
| npm warn deprecated are-we-there-yet@3.0.1: This package is no longer supported.
| npm warn deprecated gauge@4.0.4: This package is no longer supported.
| npm warn deprecated rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
| npm warn deprecated glob@7.1.7: Glob versions prior to v9 are no longer supported
| npm warn deprecated rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
| npm warn deprecated rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
| npm warn deprecated node-sass@7.0.3: Node Sass is no longer supported. Please use `sass` or `sass-embedded` instead.
| npm warn deprecated core-js@2.6.12: core-js@<3.23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js.
| 
| > themes@1.0.566 prepare
| > husky
| 
| HUSKY=0 skip install
| added 1909 packages, and audited 1945 packages in 1m
| 
| 288 packages are looking for funding
|   run `npm fund` for details
| 
| 9 vulnerabilities (4 moderate, 5 high)
| 
| To address all issues (including breaking changes), run:
|   npm audit fix --force
| 
| Run `npm audit` for details.
[Validate Themes/validate-theme]   ✅  Success - Main Install dependencies
[Validate Themes/validate-theme] ⭐ Run Main Validate Themes
[Validate Themes/validate-theme]   🐳  docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/3] user= workdir=
| 
| Progress: [ '■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■', 100 ] 1/1
| 
| ╔═════════════════════════════════════════════════════════════╤════════════════════════════════════════════════════════╗
| ║ warning                                                     │ actual   : https://schemas.wp.org/trunk/theme.json     ║
| ║ file   : grammer/theme.json                                 │ expected : https://schemas.wp.org/wp/6.5/theme.json    ║
| ║ schema : https://schemas.wp.org/wp/6.5/theme.json           │ message  : mismatched $schema with "Requires at least" ║
| ║                                                             │ metadata in style.css                                  ║
| ╟─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────╢
| ║ error                                                       │ instance path : /version                               ║
| ║ file   : grammer/theme.json                                 │ schema path   : #/properties/version/enum              ║
| ║ schema : https://schemas.wp.org/wp/6.5/theme.json           │ keyword       : enum                                   ║
| ║                                                             │ message       : must be equal to one of the allowed va ║
| ║                                                             │ lues                                                   ║
| ║                                                             ├────────────────────────────────────────────────────────╢
| ║                                                             │ instance path : /settings/color                        ║
| ║                                                             │ schema path   : #/definitions/settingsPropertiesColor/ ║
| ║                                                             │ properties/color/additionalProperties                  ║
| ║                                                             │ keyword       : additionalProperties                   ║
| ║                                                             │ message       : must NOT have additional properties    ║
| ║                                                             ├────────────────────────────────────────────────────────╢
| ║                                                             │ instance path : /settings/spacing/spacingScale         ║
| ║                                                             │ schema path   : #/definitions/settingsPropertiesSpacin ║
| ║                                                             │ g/properties/spacing/properties/spacingScale/additiona ║
| ║                                                             │ lProperties                                            ║
| ║                                                             │ keyword       : additionalProperties                   ║
| ║                                                             │ message       : must NOT have additional properties    ║
| ║                                                             ├────────────────────────────────────────────────────────╢
| ║                                                             │ instance path : /settings/typography                   ║
| ║                                                             │ schema path   : #/properties/typography/additionalProp ║
| ║                                                             │ erties                                                 ║
| ║                                                             │ keyword       : additionalProperties                   ║
| ║                                                             │ message       : must NOT have additional properties    ║
| ╟─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────╢
| ║ warning                                                     │ actual   : https://schemas.wp.org/trunk/theme.json     ║
| ║ file   : grammer/styles/lightmode.json                      │ expected : https://schemas.wp.org/wp/6.5/theme.json    ║
| ║ schema : https://schemas.wp.org/wp/6.5/theme.json           │ message  : mismatched $schema with "Requires at least" ║
| ║                                                             │ metadata in style.css                                  ║
| ╟─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────╢
| ║ error                                                       │ instance path : /version                               ║
| ║ file   : grammer/styles/lightmode.json                      │ schema path   : #/properties/version/enum              ║
| ║ schema : https://schemas.wp.org/wp/6.5/theme.json           │ keyword       : enum                                   ║
| ║                                                             │ message       : must be equal to one of the allowed va ║
| ║                                                             │ lues                                                   ║
| ╟─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────╢
| ║ error                                                       │ instance path :                                        ║
| ║ file   : grammer/assets/fonts/font-collection.json          │ schema path   : #/required                             ║
| ║ schema : https://schemas.wp.org/wp/6.5/font-collection.json │ keyword       : required                               ║
| ║                                                             │ message       : must have required property 'slug'     ║
| ║                                                             ├────────────────────────────────────────────────────────╢
| ║                                                             │ instance path :                                        ║
| ║                                                             │ schema path   : #/required                             ║
| ║                                                             │ keyword       : required                               ║
| ║                                                             │ message       : must have required property 'name'     ║
| ╚═════════════════════════════════════════════════════════════╧════════════════════════════════════════════════════════╝
| 
| Validation failed.
[Validate Themes/validate-theme]   ❌  Failure - Main Validate Themes
[Validate Themes/validate-theme] exitcode '1': failure
[Validate Themes/validate-theme] 🏁  Job failed
Error: Job 'validate-theme' failed

Related issue(s):

madhusudhand

Thanks @ajlende for the changes. It is amazing 🥇
I dropped few comments inline.

…iles

madhusudhand

Excellent work! 🙇

The new table format looks neat and is very helpful to quickly fix the changes.
Changes looks good to me. 🚀

madhusudhand · 2024-08-30T07:57:55Z

I've detected changes to the following themes in this PR: Eventual, Pique.

@vcanales Bot posted preview links even though there are no relevant changes for these themes. (or may be one of the commit had those changes).
May be the script should add/remove preview links based on the latest commit?

Edit:

Hmm, I did a quick test of the script, and it seems to be removing links based on the latest push.
But it is failing to update or remove the comment when all theme changes are removed. (when the PR has zero theme changes)

Created an issue here: WordPress/action-wp-playground-pr-preview#19

ajlende added the Automation label Aug 27, 2024

ajlende requested review from MaggieCabrera and vcanales August 27, 2024 23:11

ajlende self-assigned this Aug 27, 2024

madhusudhand reviewed Aug 28, 2024

View reviewed changes

Comment thread theme-utils.mjs Outdated

Comment thread theme-utils.mjs Outdated

ajlende mentioned this pull request Aug 28, 2024

Update Node/npm and dependencies #8045

Merged

ajlende added 14 commits August 29, 2024 10:02

Install AJV and AJV Draft 4 validators

b1b32d3

Add validate schema theme util script

2c0ae95

Add schema:validate npm script

b40b909

Run validate-schema on lint-staged

b38e67e

Add GitHub workflow for schema validation

8413d70

Fix buildComZip console log

5ca634a

Add fast-glob dev dependency

e83b893

Update to validate an entire theme at a time rather than individual f…

93476e5

…iles

Remove lint-staged rules

09524be

Rename npm script

cdfe78a

Install chalk dev dependency

c84ab84

Update the validate-theme script

91b520e

Update the validate-theme script

c9068fc

Update the validate-theme script

69ac8fe

ajlende force-pushed the add/schema-validation branch from 0c21163 to 69ac8fe Compare August 30, 2024 06:03

madhusudhand approved these changes Aug 30, 2024

View reviewed changes

Comment thread theme-utils.mjs Outdated

ajlende added 4 commits August 30, 2024 13:16

Update the validate-theme script

b1ce050

Update the validate-theme script

f004e53

Update CONTRIBUTING.md

e2c3456

Update CONTRIBUTING.md

91023db

ajlende merged commit dd40a6e into trunk Aug 30, 2024

ajlende deleted the add/schema-validation branch August 30, 2024 21:34

madhusudhand mentioned this pull request Sep 4, 2024

fix a minor issue with theme.json validation error reporting #8065

Merged

madhusudhand mentioned this pull request Sep 4, 2024

Theme validation: add style.css metadata validations #8068

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add automated JSON schema validation#8039

Add automated JSON schema validation#8039
ajlende merged 18 commits into
trunkfrom
add/schema-validation

ajlende commented Aug 27, 2024 •

edited

Loading

Uh oh!

madhusudhand left a comment

Uh oh!

Uh oh!

Uh oh!

madhusudhand left a comment

Uh oh!

Uh oh!

madhusudhand commented Aug 30, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ajlende commented Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed in this Pull Request:

Validation script

GitHub Action

Related issue(s):

Uh oh!

madhusudhand left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

madhusudhand left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

madhusudhand commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ajlende commented Aug 27, 2024 •

edited

Loading

madhusudhand commented Aug 30, 2024 •

edited

Loading