Skip to content

[FEATURE] Validate config/samples against the CRD schemas (catch hallucinated manifests) #1021

Description

@Defilan

Problem

The Foreman gate runs Go-only checks (make fmt/vet/lint/test + bite check). It does not validate sample manifests against the CRD OpenAPI schemas. So a coder can produce a config/samples/** change with invalid or hallucinated CRD fields (wrong types, non-existent fields) and it GATE-PASSes, because nothing checks YAML against the schema.

Observed (dogfood #699): a coder rewrote config/samples/model_amd_vulkan_igpu.yaml using Model.spec.url (field is source), spec.parameters.{contextSize,embeddingSize,layerCount} (no such block), InferenceService.spec.modelRef: {name: …} (it's a string), runtime: {…} (it's a string), spec.service (it's endpoint). The manifest would fail kubectl apply, yet it GATE-PASSed because the change touched no Go.

Proposed fix

Add a make validate-samples target that:

  1. Extracts JSON Schemas from config/crd/bases/*.yaml (kubeconform's CRD extractor / openapi2jsonschema).
  2. Runs kubeconform on config/samples/**.yaml against those schemas plus the built-in Kubernetes schemas.

Wire it into CI (a workflow step on config/{samples,crd}/** changes). Optionally add it to the gate's check list so a coder's sample change is validated in the self-gate, not just at CI.

Acceptance

  • make validate-samples fails on a sample with an unknown/mistyped CRD field and passes on the current valid samples.
  • CI runs it on PRs touching samples or CRDs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/foremanForeman: the agentic fleet orchestrator add-onenhancementNew feature or requestpriority/mediumMedium priority

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions