Define MCS-API conformance report program details in the sig-mc website

[MrFreezeex]

In order to graduate to stable, the MCS-API needs a conformance report program allowing users to find various implementations and see what they support from MCS-API/what test they are passing from our conformance suite.

So far, this was mainly discussed in the SIG-Multicluster meeting on 01/06 (link to the notes: https://docs.google.com/document/d/18mk62nOXE_MCSSnb4yJD_8UadtzJrYyJxFwbrgabHe8/edit?tab=t.0#heading=h.sszecla3gmyd)

We know that we want to:

• Update this page https://multicluster.sigs.k8s.io/guides/ (the source is here: https://github.com/kubernetes-sigs/sig-multicluster-site/blob/main/site-src/guides/index.md) that currently lists implementations (we could rename it because it's currently named "guide," which isn't very representative of what it actually is) and be able to link the last conformance report of each implementation here (note that the conformance report output is a simple standalone html file with a matrix of test passed) with at least a date and the project version tested (and possibly the commit/version of the conformance test suite).
• Assume good faith in submissions from implementors (essentially, that they don't modify the results manually) + that they will test using the latest commit of the MCS-API conformance suite at that time.
• Consider report staleness based on the date of submission rather than the MCS-API version/project version. In the last discussion in the SIG meeting, we discussed removing implementations that haven't submitted a conformance report for more than a year, and that this would be done twice a year around the time of KubeCon EU/NA. Gateway API accepts non-conformant reports, and it's probably a good idea here too. We discussed removing implementations from the existing page after a grace period of 6 months once the conformance report program has been set up and properly announced.
• Encode all the info for a submission in a (semi-)structured way via a GitHub issue template or directly in a PR (and possibly a PR template to follow or by committing some metadata in the repo about the submission). Note that we don't expect to have a high volume, so structuring absolutely everything in a form isn't technically mandatory besides the basics
• Regarding all the above (and mostly the previous point), Gateway API is much more mature in this regard, so try to reasonably get some inspiration from or copy some of their process.

Feel free to reach out in the sig-multicluster slack channel or in our meetings if you need further guidance/discussions (and it would be great if you can present what you did in one of our sig meeting too once you are done but we can also just review the PRs/proposal async too)!

[MrFreezeex]

/help

[k8s-ci-robot]

@MrFreezeex:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

Details

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[AnupamGhosh]

Hi @MrFreezeex, I can work on it. Based on the description can I assume that all the changes need to be made in https://github.com/kubernetes-sigs/sig-multicluster-site repo?

[MrFreezeex]

Hi @AnupamGhosh, yes mostly and perhaps some small tweak to the conformance report if we need to (not sure we need to do further tweaks there though).

[AnupamGhosh]

I have some questions regarding this

1. Can I assume there will be 3 categories of conformance status: Conformant, Partially Conformant and Stale?
2. What happens after an implementer submits a conformance report via a Github issue?
   • Will a maintainer review the report and update the conformance status by raising a PR?
3. With PR template (I assume the template to be in a .github/PULL_REQUEST_TEMPLATE/<conformance-template>.md) users have to manually append ?template=conformance-report.md to the compare URL to get the template — it's not presented as a selectable option in the UI the way issue templates are. For simplicity is it okay to skip PR templating for conformance report?
4. How can a implementer run the conformance test against their implementation? Should that be added in the website or is it easy enough to figure that out?
5. Having a html report is good but Gateway API generates report in yaml format which is easy to parse. They have this page https://gateway-api.sigs.k8s.io/implementations/wizard/ where an end user can check which implementation is conformant with which version of the Gateway API. Should there be an option in the mcs api conformance report to generate report in yaml file also?

[MrFreezeex]

1. Can I assume there will be 3 categories of conformance status: Conformant, Partially Conformant and Stale?

Sure that sounds fine!

What happens after an implementer submits a conformance report via a Github issue?

Well part of this is determining what's the process should be and there's probably need to have only one method to submit whether it's a PR or an issue. A PR sounds easier at first glance though.

With PR template (I assume the template to be in a .github/PULL_REQUEST_TEMPLATE/.md) users have to manually append ?template=conformance-report.md to the compare URL to get the template — it's not presented as a selectable option in the UI the way issue templates are. For simplicity is it okay to skip PR templating for conformance report?

Eitherway it sounds relatively ok, but I would try to see what other project in the ecosystem do and try to match some of their approach. A pull request template could be ok if documented properly or just forcing this to be updated in the content of the PR itself for instance 🤷‍♂️.

How can a implementer run the conformance test against their implementation? Should that be added in the website or is it easy enough to figure that out?

It's not super hard but it probably deserves a bit more documentation somewhere indeed! There are multiple ways but for instance one is running go test conformance/ in this repo with some -contexts param specifying two clusters in your current kubeconfig. I think you can run the ginkgo CLI too and one other way could be to integrate directly within a project via go dependency (example: https://github.com/cilium/cilium/blob/main/pkg/clustermesh/mcsapi/conformance/conformance_test.go)

Should there be an option in the mcs api conformance report to generate report in yaml file also?

It depends what we want to do about it, Gateway-API generate matrix comparing each implementation from this. We could possibly have the same, it would probably more be a (welcomed) nice to have than something strictly required I would say. We have much less implementation than GW-API so while a matrix would be nice it shouldn't be that much of a nightmare to open up to 5 reports for a user and eyeball what they need from it 😅

[AnupamGhosh]

Thanks @MrFreezeex for the quick reply! I also feel that PR is a better way to go.

I looked at https://github.com/kubernetes-sigs/verify-conformance a GitHub bot designed to act as a checker for conformance submissions, ensuring that PRs contain the correct data, valid structures, and passing results before a human ever reviews them.

For eg, In this PR cncf/k8s-conformance#4091 the title was wrong and the bot report that. Also looking at the README.md file in the PR the contributor mentions how the tests are executed.

I think in an ideal form MCS API conformance report could be something like this

1. The conformance test suite outputs a standardized, machine-readable report.yaml.
2. The contributor forks the mcs-api repo and commits their report to a specific directory structure (e.g., conformance/reports/<mcs-api-version>/<provider-name>/report.yaml) with a README that says how the test was performed.
3. Once a PR is opened. In the MCS API a simple CI checks against that directory. Similar to how the verify-conformance code automatically parses JUnit files, the MCS API CI would automatically validate the schema of the submitted report.yaml, ensure all required core tests are marked as passed (and from the ginko logs catch any manual tampering or formatting errors).
4. After the PR lands, with mcdocs hooks from the reports a conformance matrix can be generated dynamically.

Since MCS API has much less implementation the workflow can be simplified by skipping the automatic CI checker and asking the contributor to manually update the comparison matrix (with that mcdocs hooks is also not required). The PR will contain the report.html and a README.md that says how the test was performed.

[MrFreezeex]

Thanks for work and research on this!

So I would say that if we want to do more than just linking the HTML, like having a report matrix the yaml output sounds better. I am not sure what the matrix would look like but we have llike 25-ish tests so if it lists all the tests individually it sounds a bit tedious to manually update.

About generating matrix I had seen more this being made in some independent scripts, like in that case it could be a script that generate a markdown file based on the YAML reports which would be then rendered by mkdocs. Those could also be committed with a simple verify script in CI making sure contributors would have correctly run within the PR. I am not super experienced in mkdocs so hooks might be better too I don't really know though.

When you say "The contributor forks the mcs-api repo and commits their report to a specific directory structure", do you mean the sig-multicluster-site repo and not the mcs-api one?

[skitt]

We discussed this in the APAC call today, thanks for joining @AnupamGhosh!

Regarding machine-readable output, Ginkgo (the framework used in the conformance tests) can produce a JSON report, perhaps we can use that directly.

If we follow the Gateway API pattern, reports would go alongside the conformance tests in https://github.com/kubernetes-sigs/mcs-api/, and only the resulting matrix would go to the web site.

[AnupamGhosh]

When you say "The contributor forks the mcs-api repo and commits their report to a specific directory structure", do you mean the sig-multicluster-site repo and not the mcs-api one?

yes @MrFreezeex you are correct I meant in the sig-multicluster-site repo. Sorry for the typo.

I created two sub task for this issue

• Establish conformance report directory and auto-generate implementation matrix sig-multicluster-site#52
• Support machine-readable output for conformance test reports #144

[AnupamGhosh]

I was thinking that embedding the metadata directly into the JSON/YAML report eliminates the need for a manual PR template and allows the information to be automatically extracted for the comparison matrix. The conformance reporting tool can support this by accepting metadata via optional CLI flags. During the PR submission process, reviewers need to check that the required metadata are present within the submitted JSON/YAML file.