[command-reference]: add diff script, CI check, and update reference docs

[saehejkang]

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Motivation and Context

Related to #1534

Thought it would be worthwhile to add a sort of mandatory check to the CI to fail, if the command-reference docs are not updated appropriately, whenever changes are made to the CLI. When the lint command is ran, there is an output that shows the diff between the current CLI build and what is written in the docs. Hopefully this will help keep the docs up to date and reduce tech debt over time.

NOTE: The scripts/generate-command-reference.py file was generated with AI but reviewed by me.

Testing

• Tested locally
• Added/updated tests
• Added/updated docs

[kiwigitops]

One generated-doc issue looks worth fixing before this becomes the drift baseline: --max-concurrent-downloads currently renders with the default twice, for example Maximum number of concurrent downloads (default: 3) (default: 3) in the run, create, and image pull sections.

It looks like the option help text already includes the default and the generator appends the parsed default again.

[katiewasnothere]

@saehejkang I wonder if we can use ArgumentParser's experimental help dump feature here instead of needing to manually parse the help with python. What do you think?

[jglogan]

I think I'm more inclined toward a two-layer approach for this:

• (As developers and reviewers) we all need to be vigilant about CLI-related changes and ensure that docs get updated.
• (As maintainers) we have a recurring task to burn some API tokens on correcting the drift (and an agent can use the experimental dump help metadata to establish a baseline).

Having a script do things can get the job done for the command reference but isn't as adaptable as using a natural language based approach to solving a natural language related problem. Neither approach will be perfect but I feel the result will be better if we just lean into LLMs for this.

[saehejkang]

@jglogan @katiewasnothere

I wonder if we can use ArgumentParser's experimental help dump feature here instead of needing to manually parse the help with python. What do you think?

LOL, I completely missed this feature. I can see it being something that simplifies a lot of the script and then we don't need to add any python stuff.

Having a script do things can get the job done for the command reference but isn't as adaptable as using a natural language based approach to solving a natural language related problem.

FWIW, I did use an LLM to actually go through and make the necessary updates to the command reference docs. The script made it a lot quicker for the LLM to see the drift and make the updates. Saves a lot more tokens and we only have to prompt it to:

run the diff script and make updates to the docs

Furthermore, if there is ever a time when a developer wants to see the drift (and has no AI), they can read the output of the script and make updates promptly. Also, the CI, would now have a way to check for the drift automatically (and fail if updates were missed), because I assume we can't use an LLM in the pipeline (yet)?