Feat/generate ubproject and schema validation

[arnoox]

📌 Description

• Add sn_schemas.py which translates metamodel.yaml into a schemas.json file consumable by sphinx-needs. This enables schema-based validation at parse time: required fields, regex patterns on option values, mandatory link presence (minItems: 1), and mandatory link target type checks (validate.network).
• Because ubCode (the VS Code extension for sphinx-needs) evaluates these schemas during editing, all metamodel violations now surface as IDE diagnostics directly in the editor, errors are caught early with lightweight, fast rendering, without waiting for a full Sphinx build.
• Add unit tests (test_sn_schemas.py) and integration tests (test_sn_schemas_integration.py) that validate generated schemas against the real S-CORE metamodel using jsonschema_rs, the same engine sphinx-needs uses at runtime.
• Add developer documentation (README.md) covering the two validation layers (schema vs. post-build Python checks) with a detailed coverage comparison table.

🚨 Impact Analysis

• This change does not violate any tool requirements and is covered by existing tool requirements
• This change does not violate any design decisions
• Otherwise I have created a ticket for new tool qualification

✅ Checklist

• Added/updated documentation for new or changed features
• Added/updated tests to cover the changes
• Followed project coding standards and guidelines

[ubmarco]

I tested this and it looks functionally working well. Here are some finding, a mix of AI, and my personal findings.

Found 2 issues:

1. The error message in _classify_links says only the first regex will be used when multiple regex patterns are supplied for the same link field, but the implementation unconditionally overwrites on each iteration so the last regex actually wins. The companion test even asserts the last value is kept (# Last regex overwrites previous ones), confirming the message is wrong.

   docs-as-code/src/extensions/score_metamodel/sn_schemas.py

   Lines 110 to 120 in 5f1263b

   	for link_value in link_values:
   	if link_value.startswith("^"):
   	if field in regexes:
   	LOGGER.error(
   	f"Multiple regex patterns for {label} link field "
   	f"'{field}' in need type '{type_name}'. "
   	"Only the first one will be used in the schema."
   	)
   	regexes[field] = link_value
   	else:
   	if field not in targets:

   Test that confirms the actual (last-wins) behavior:

   docs-as-code/src/extensions/score_metamodel/tests/test_sn_schemas.py

   Lines 155 to 163 in 5f1263b

   	def test_multiple_regex_for_same_field_logs_error(self) -> None:
   	links = {"field": "^regex1$, ^regex2$"}
   	with patch("src.extensions.score_metamodel.sn_schemas.LOGGER") as mock_logger:
   	regexes, _ = _classify_links(links, "my_type", mandatory=True)
   	mock_logger.error.assert_called_once()
   	# Last regex overwrites previous ones
   	assert regexes == {"field": "^regex2$"}

   Fix: either change the message to say "the last one will be used", or continue after logging so the first one is actually retained (and update the test accordingly).

2. needs_metamodel_generated.toml should not be written into confdir. _generate_needs_types_toml, _generate_needs_fields_toml and _generate_needs_links_toml build TOML strings purely in memory, but the setup then writes the concatenation to Path(app.confdir) / "needs_metamodel_generated.toml" solely so the path can be appended to needscfg_merge_toml_files. needs_config_writer then merges the same content into ubproject.toml, so the on-disk file contains nothing the final ubproject.toml doesn't already have — and it lands inside the user's source tree (docs/), forcing a .gitignore entry. Running bazel run //:docs_check produces both files side-by-side with overlapping contents, which is confusing and serves no consumer.

   docs-as-code/src/extensions/score_sync_toml/__init__.py

   Lines 154 to 167 in 5f1263b

   	# Generate TOML fragments for types, fields, and links from the metamodel.
   	# needs_config_writer cannot serialise these structures itself, so we combine
   	# them into a single file and register it for merging.
   	# Write to a deterministic path so it's always overwritten (no temp file leak).
   	metamodel_toml = (
   	_generate_needs_types_toml(app)
   	+ _generate_needs_fields_toml(app)
   	+ _generate_needs_links_toml(app)
   	)
   	metamodel_path = Path(app.confdir) / "needs_metamodel_generated.toml"
   	metamodel_path.write_text(metamodel_toml, encoding="utf-8")
   	# Merge the generated metamodel TOML (types, fields, links) into the final ubproject.toml.
   	app.config.needscfg_merge_toml_files.append(str(metamodel_path))

   .gitignore entry added to hide the generated file (also covers the redundancy):

   https://github.com/eclipse-score/docs-as-code/blob/5f1263b1fccd9acde0d608e29e672f448e53979e/.gitignore

   README explicitly labels the file as an "Intermediate merge input for needs_config_writer":

   https://github.com/eclipse-score/docs-as-code/blob/5f1263b1fccd9acde0d608e29e672f448e53979e/src/extensions/score_sync_toml/README.md#L130-L137

   Fix options (in order of preference):

   • Write the intermediate to app.outdir (or a tempfile.mkdtemp() directory cleaned up via a build-finished handler) so it never appears under confdir. The .gitignore entry can then be removed.
   • Better still: extend needs_config_writer to accept an in-memory TOML fragment (string or dict) so the intermediate file is not needed at all. The current "deterministic path so no temp file leak" rationale (__init__.py:157) is solving the wrong problem — the leak only exists because confdir was chosen as the target.

---

Lower-confidence findings (below the 80-point bar)

The 5 parallel review agents surfaced these additional candidates. Each was scored 0–100 by an independent agent; only items at ≥80 are reported as confirmed issues above. The rest are listed here for completeness — they are either latent (no current trigger), intentional, or pre-existing.

2. Optional-field regex alternation ^$|{pattern} (score: 60)

In get_pattern_schema, optional fields wrap the metamodel pattern as f"^$|{pattern}". JSON Schema's pattern keyword is defined as a substring search (ECMA-262 semantics, not full match), whereas the post-build Python checker uses re.fullmatch. For patterns that are not fully ^…$ anchored (e.g. ^https://github.com/.* used in source_code_link / testlink fields in metamodel.yaml), the alternation ^$|^https://github.com/.* may behave differently in jsonschema-rs (used by sphinx-needs / ubCode) than in the Python checker, allowing values one engine accepts and the other rejects.

docs-as-code/src/extensions/score_metamodel/sn_schemas.py

Lines 300 to 320 in 5f1263b

	"""Return a JSON schema that validates a string against a regex pattern.
	For optional fields, allows either an empty string OR a string matching
	the pattern, matching the Python checker's behavior where empty strings
	are treated as "absent" and not validated.
	"""
	if is_optional:
	# Allow empty strings for optional fields (Python checker treats "" as absent)
	# Use regex alternation to match either empty string or the original pattern
	return {
	"type": "string",
	"pattern": f"^$|{pattern}",
	}
	return {
	"type": "string",
	"pattern": pattern,
	}
	def get_array_pattern_schema(pattern: str, is_optional: bool = False) -> dict[str, Any]:
	"""Return a JSON schema that validates an array where each item matches a regex.

Why scored 60: The current metamodel mostly uses fully anchored patterns, so this is a real but latent inconsistency rather than a bug hitting today. Worth a TODO and a test for an unanchored pattern.

3. Missing TOML escaping in _generate_needs_types_toml (score: 60)

_generate_needs_fields_toml and _generate_needs_links_toml apply .replace("\\", "\\\\").replace('"', '\\"') before interpolating values into TOML strings. _generate_needs_types_toml does not — directive, title, and prefix are written raw. If any need-type title or prefix ever contains a backslash or double quote, the generated TOML will be syntactically invalid and downstream parsing will fail.

https://github.com/eclipse-score/docs-as-code/blob/5f1263b1fccd9acde0d608e29e672f448e53979e/src/extensions/score_sync_toml/__init__.py

Why scored 60: Real divergence from the sibling helpers, but no current metamodel value contains the problem characters, so the bug is latent. Easy to fix by mirroring the escape calls used in the field/link helpers.

4. Default ID pattern broadened to allow uppercase (score: 65)

The auto-generated fallback ID pattern in yaml_parser.py widened from ^{prefix}[0-9a-z_]+$ (lowercase only) to ^{prefix}[0-9a-zA-Z_]+$ (mixed case). This affects every need type that does not declare an explicit id: pattern.

docs-as-code/src/extensions/score_metamodel/yaml_parser.py

Lines 130 to 140 in 5f1263b

	# Ensure ID regex is set
	if "id" not in t["mandatory_options"]:
	prefix = t["prefix"]
	t["mandatory_options"]["id"] = f"^{prefix}[0-9a-zA-Z_]+$"
	if "color" in yaml_data:
	t["color"] = yaml_data["color"]
	if "style" in yaml_data:
	t["style"] = yaml_data["style"]

Why scored 65: The change appears to have ridden in on an earlier sphinx-needs 6.3.0 upgrade commit on the same branch (not introduced in this PR's primary diff hunks), and there is no commit message or comment explaining the policy shift. Confirm intentional; if so, document; if not, restore lowercase.

5. needscfg_exclude_vars may duplicate needs_config_writer defaults (score: 25)

In score_sync_toml/__init__.py, the setup code does list(getattr(app.config, "needscfg_exclude_vars", None) or []) + ["needs_from_toml", "needs_from_toml_table", "needs_schema_definitions_from_json"]. If needs_config_writer (loaded earlier in the extension order) already registers those three entries as the default value, the result will contain duplicates.

https://github.com/eclipse-score/docs-as-code/blob/5f1263b1fccd9acde0d608e29e672f448e53979e/src/extensions/score_sync_toml/__init__.py

Why scored 25: Could not confirm needs_config_writer actually registers these defaults; even if duplicates appear, exclusion-list duplication is typically benign. Worth a quick check during review.

6. Optional-link validate.network intentionally omitted (score: 5 — false positive)

sn_schemas.py deliberately skips emitting validate.network for optional links, with an in-source explanation: the post-build Python checker reports optional-link target-type violations at treat_as_info=True, while sphinx-needs schema validators only support a single violation severity per schema rule. Including optional links in the schema would escalate info-level findings to hard errors. A possible solution is to write optional links into dedicated rules, so they can be assigned a dedicated severity. The README's coverage table documents this trade-off.

Why scored 5: Documented intentional design decision with rationale and tests. Not an issue.

---

Follow-up suggestions (not blocking this PR)

These are improvements to track in subsequent PRs — they are not regressions in #394, but the surrounding code makes them natural next steps.

F1. Replace empty-string sentinels with nullable fields

_generate_needs_fields_toml always writes default = "" for any field that has no metamodel-supplied default. Throughout the codebase this empty string is then re-interpreted as "field not given" in later checks. Modelling absence with nullable fields (null / Optional) is the cleaner alternative — distinguishing a deliberately empty value from an unset one — and avoids the readability and type-safety problems that come with overloaded empty strings.

docs-as-code/src/extensions/score_sync_toml/__init__.py

Lines 20 to 43 in 5f1263b

	def _generate_needs_fields_toml(app: Sphinx) -> str:
	"""Serialize ``app.config.needs_fields`` as ``[needs.fields.*]`` TOML entries.
	``needs_config_writer`` cannot serialize the nested dicts that make up
	``needs_fields`` (e.g. ``{"schema": {"type": "string"}, "default": ""}``),
	so custom fields like ``safety``, ``security``, and ``reqtype`` are missing
	from ``ubproject.toml``. Without them, ubCode does not know these are valid
	options and will report them as unknown fields.
	Returns only the ``default`` value for each field, which is sufficient for
	ubCode to recognise the field as valid.
	Must be called *after* ``score_metamodel.setup()`` has run (i.e. after
	``app.config.needs_fields`` has been extended with the metamodel fields).
	"""
	lines: list[str] = []
	for field_name, field_config in sorted(app.config.needs_fields.items()):
	default = field_config.get("default", "")
	# TOML-escape the default value (handle quotes)
	escaped = str(default).replace("\\", "\\\\").replace('"', '\\"')
	lines.append(f"[needs.fields.{field_name}]\n")
	lines.append(f'default = "{escaped}"\n')
	lines.append("\n")
	return "".join(lines)

F2. Promote shared enum values to field-level schemas

Where a field carries the same enum across every need type that uses it, the per-type duplication can be hoisted into a single [needs.fields.<name>] schema entry, e.g.

[needs.fields.status]
schema.enum = ["draft", "review", "approved", "done"]

This would shrink the generated schemas.json, make ubCode autocompletion show enum values for the field everywhere it appears, and keep the metamodel DRY. _generate_needs_fields_toml currently emits only default — extending it with schema.enum (and possibly schema.type) when the same constraint holds across every type that uses the field is the natural follow-up.

docs-as-code/src/extensions/score_sync_toml/__init__.py

Lines 20 to 43 in 5f1263b

	def _generate_needs_fields_toml(app: Sphinx) -> str:
	"""Serialize ``app.config.needs_fields`` as ``[needs.fields.*]`` TOML entries.
	``needs_config_writer`` cannot serialize the nested dicts that make up
	``needs_fields`` (e.g. ``{"schema": {"type": "string"}, "default": ""}``),
	so custom fields like ``safety``, ``security``, and ``reqtype`` are missing
	from ``ubproject.toml``. Without them, ubCode does not know these are valid
	options and will report them as unknown fields.
	Returns only the ``default`` value for each field, which is sufficient for
	ubCode to recognise the field as valid.
	Must be called *after* ``score_metamodel.setup()`` has run (i.e. after
	``app.config.needs_fields`` has been extended with the metamodel fields).
	"""
	lines: list[str] = []
	for field_name, field_config in sorted(app.config.needs_fields.items()):
	default = field_config.get("default", "")
	# TOML-escape the default value (handle quotes)
	escaped = str(default).replace("\\", "\\\\").replace('"', '\\"')
	lines.append(f"[needs.fields.{field_name}]\n")
	lines.append(f'default = "{escaped}"\n')
	lines.append("\n")
	return "".join(lines)

F3. Emit $schema header in generated ubproject.toml

The generated ubproject.toml should start with

"$schema" = "https://ubcode.useblocks.com/ubproject.schema.json"

so that editors with TOML schema support give schema-aware completion and validation while editing the file. See ubCode configuration docs. This would likely live as a top-of-file fragment in shared.toml or as a needs_config_writer option, depending on how the merge tool orders keys.

https://github.com/eclipse-score/docs-as-code/blob/5f1263b1fccd9acde0d608e29e672f448e53979e/src/extensions/score_sync_toml/shared.toml

F4. Convert needs_schema_debug_path to a relative path

The generated ubproject.toml currently contains an absolute, machine-specific path:

schema_debug_path = "/home/marco/git/eclipse-score/docs-as-code/docs/schema_debug"

needs_config_writer already supports rewriting absolute paths via needscfg_relative_path_fields, and its own docstring explicitly names needs_schema_debug_path as the example field for this option:

"List of config path patterns to convert to relative paths. Can be strings (e.g., 'needs_schema_debug_path') ..."
— needs_config_writer/main.py:94

The fix is to add "needs_schema_debug_path" to the existing needscfg_relative_path_fields.extend([...]) call. If sphinx-needs cannot consume a relative value for this field, that is an upstream bug to file against sphinx-needs / needs-config-writer; otherwise this is a one-line change.

docs-as-code/src/extensions/score_sync_toml/__init__.py

Lines 168 to 177 in 5f1263b

	# Relative paths to confdir for Bazel provided absolute paths.
	app.config.needscfg_relative_path_fields.extend(
	[
	"needs_external_needs[*].json_path",
	{
	"field": "needs_flow_configs.score_config",
	"prefix": "!include ",
	},
	]
	)

[ubmarco]

I am approving this PR, the noted review comments can be addressed in follow-up PRs. Functionally all good.

[AlexanderLanin]

bazel build //:needs_json fails with WARNING: Failed to write schemas.json: [Errno 30] Read-only file system: '/home/alex/.cache/bazel/_bazel_alex/ec58ecca617edd0b2864120a40405cd7/sandbox/linux-sandbox/271/execroot/_main/bazel-out/k8-fastbuild/bin/_needs_json/_sources/docs/schemas.json'. Schema validation will be unavailable.

Don't we test that in CI? 😨

[a-zw]

I don't think it makes sense to create this README here which seems to be AI slop more or less duplicating info from our actual documentation.

[arnoox]

fair point. I will remove this.

[a-zw]

Why is the metamodel extended here? Was that a gap before? Why didn't we find that before then?

[AlexanderLanin]

I was just reviewing that. It was a gap. Why we didnt find that ourselves is a very valid question.

[a-zw]

Even more files getting generated into the workspace?

This clashes with my proposal here: eclipse-score/score#2826

Technically, this is just a draft, so not really relevant for this PR though. Just want to mention it.

[arnoox]

• docs/needs_metamodel_generated.toml can definitely be moved to some build dir. I'll fix as part of this PR.
• docs/schemas.json is maybe required in the workspace. I can check if it is possible for ubproject.toml to reference to something from within the build dir.

[a-zw]

We should trim down the worst of this AI generated stuff. Some tests here are not really useful.

[arnoox]

Agreed, the tests for the trivial dict building helpers add no real value. I will keep only the tests that cover actual branching logic.

[a-zw]

Same as in the other test file: Some tests are pointless and should be removed.

[a-zw]

Shouldn't his be a docstring? Comment before method makes no sense.

[arnoox]

Fixed.

[AlexanderLanin]

I'm not so sure about the architecture here. score_metamodel writes the json, and the toml is written by a separate extension. However the toml has a hard dependency on having a json file.

Don't the toml and the json also overlap in their content?

[AlexanderLanin]

Can we figure out some common naming scheme for the generated files? With this change we'll need to touch gitignore for all users, so some wildcard pattern would be nice.