Replace nix-env metadata scanning with targeted metadata lookup

README.md

@@ -9,7 +9,6 @@ SPDX-License-Identifier: CC-BY-SA-4.0
 This repository is home to various command line tools and Python libraries that aim to help with software supply chain challenges:
 - [`sbomnix`](#generate-sbom) is a utility that generates SBOMs given a [Nix](https://nixos.org/) flake reference or store path.
 - [`nixgraph`](./doc/nixgraph.md) helps query and visualize dependency graphs for [Nix](https://nixos.org/) packages.
-- [`nixmeta`](./doc/nixmeta.md) summarizes nixpkgs meta-attributes from the given nixpkgs version.
 - [`vulnxscan`](./doc/vulnxscan.md) is a vulnerability scanner demonstrating the usage of SBOMs in running vulnerability scans.
 - [`repology_cli`](./doc/repology_cli.md) and [`repology_cve`](./doc/repology_cli.md#repology-cve-search) are command line clients to [repology.org](https://repology.org/).
 - [`nix_outdated`](./doc/nix_outdated.md) is a utility that finds outdated nix dependencies for given out path, listing the outdated packages in priority order based on how many other packages depend on the given outdated package.
@@ -71,7 +70,10 @@ $ cd sbomnix
 $ nix develop
 ```
 
-The devshell adds all CLI entry points (`sbomnix`, `nixgraph`, `nixmeta`, `vulnxscan`, `repology_cli`, `repology_cve`, `nix_outdated`, `provenance`) to `PATH`. They run against the local source tree, so any edits are picked up immediately without reinstalling.
+The devshell adds all CLI entry points (`sbomnix`, `nixgraph`,
+`vulnxscan`, `repology_cli`, `repology_cve`, `nix_outdated`,
+`provenance`) to `PATH`. They run against the local source tree, so any
+edits are picked up immediately without reinstalling.
 
 All tools support a consistent verbosity flag: no flag or `--verbose=0`
 shows INFO output, `-v` or `--verbose=1` enables VERBOSE progress
@@ -103,7 +105,7 @@ $ sbomnix github:NixOS/nixpkgs?ref=nixos-unstable#wget
 ```
 
 #### Generate SBOM Based on Derivation File or Out-path
-Flake references are the recommended target for `sbomnix`. When the target is a flake reference, `sbomnix` can resolve the nixpkgs version used to build the package and enrich the SBOM with metadata such as descriptions, licenses, maintainers, and homepage links. When the target is a store path, there is no information about which nixpkgs version produced it, so metadata enrichment is skipped by default; see [Nixpkgs Metadata Source Selection](#nixpkgs-metadata-source-selection).
+Flake references are the recommended target for `sbomnix`. When the target is a flake reference, `sbomnix` can resolve the nixpkgs version used to build the package and enrich the SBOM with metadata such as descriptions, licenses, maintainers, and homepage links. When the target is a store path, there is no information about which nixpkgs version produced it, so metadata enrichment is skipped; see [Nixpkgs Metadata Source Selection](#nixpkgs-metadata-source-selection).
 
 By default `sbomnix` scans the given target and generates an SBOM including the runtime dependencies.
 Notice: determining the target runtime dependencies in Nix requires building the target.
@@ -135,7 +137,7 @@ $ sbomnix github:NixOS/nixpkgs/nixos-unstable#wget --buildtime
 ```bash
 $ sbomnix /path/to/result
 ```
-Note: store paths carry no record of which nixpkgs version produced them, so nixpkgs metadata enrichment is skipped by default. Pass `--meta-nixpkgs` to supply a nixpkgs source explicitly, or see [Nixpkgs Metadata Source Selection](#nixpkgs-metadata-source-selection).
+Note: store paths carry no record of which nixpkgs version produced them, so nixpkgs metadata enrichment is skipped. Use a flakeref target when nixpkgs metadata is required.
 
 #### Nixpkgs Metadata Source Selection
 `sbomnix` enriches packages with nixpkgs metadata, such as descriptions,
@@ -147,19 +149,18 @@ toplevel flakerefs are handled through the selected NixOS package set, so
 overlays, package overrides, nixpkgs config, and system-specific package-set
 changes can be represented.
 
-Store-path targets skip nixpkgs metadata by default; pass `--meta-nixpkgs` to
-choose the source explicitly.
-
-`--meta-nixpkgs <flakeref-or-path>` scans an explicit nixpkgs source.
-`--meta-nixpkgs nix-path` scans the `nixpkgs=` entry from `NIX_PATH` as an
-explicit opt-in source. `--exclude-meta` disables this enrichment and cannot be
-combined with `--meta-nixpkgs`.
+Store-path targets skip nixpkgs metadata because the store path does not
+identify the nixpkgs source that produced it. `--exclude-meta` disables metadata
+enrichment.
 
 CycloneDX and SPDX outputs record the selected metadata source in document
 metadata, including fields such as `nixpkgs:metadata_source_method`,
 `nixpkgs:path`, `nixpkgs:rev`, `nixpkgs:flakeref`, `nixpkgs:version`, and
 `nixpkgs:message`.
 
+See [sbomnix metadata enrichment](./doc/sbomnix_metadata.md) for a short
+overview of source selection, component matching, and metadata caching.
+
 #### Visualize Package Dependencies
 `sbomnix` uses structured Nix JSON to find package dependencies where
 available. `nixgraph` can also be used as a stand-alone tool for visualizing

doc/sbomnix_metadata.md

@@ -0,0 +1,53 @@
+<!--
+SPDX-FileCopyrightText: 2026 Technology Innovation Institute (TII)
+
+SPDX-License-Identifier: CC-BY-SA-4.0
+-->
+
+# sbomnix metadata enrichment
+
+`sbomnix` can enrich SBOM components with nixpkgs package metadata such as
+descriptions, licenses, maintainers, and homepage links. Metadata enrichment is
+best-effort: an SBOM can still be generated when metadata is unavailable or when
+no metadata candidate matches a component exactly.
+
+## Metadata source selection
+
+For flakeref targets, `sbomnix` selects a nixpkgs metadata source from the
+target context. For NixOS toplevel flakerefs, it uses the evaluated
+`nixosConfigurations.<name>.pkgs` package set, so overlays, package overrides,
+nixpkgs config, and system-specific settings can be represented.
+
+For other flakeref targets, `sbomnix` uses the target flake lock graph to find
+the pinned nixpkgs source. Store-path targets skip nixpkgs metadata because a
+store path does not identify which nixpkgs source produced it.
+
+## Component matching
+
+Component names, pnames, and versions are used as lookup hints to find possible
+nixpkgs derivations. Those names are not trusted as the final match. The
+metadata helper returns derivation and output identities, and `sbomnix` accepts
+metadata only when the returned `drvPath` or `outPath` matches an SBOM component
+exactly.
+
+This keeps metadata enrichment tied to the actual components in the SBOM while
+still allowing targeted lookups in package sets.
+
+## Flake input packages
+
+Some targets include packages that come from flake inputs rather than the
+primary nixpkgs package set. After scanning the selected package set, `sbomnix`
+may scan package roots exported by the target flake inputs for components that
+remain unmatched.
+
+## Caching and diagnostics
+
+Package metadata lookups are cached by metadata source identity, lookup set,
+lookup mode, and helper implementation fingerprint. With `-v`, `sbomnix` logs
+the selected metadata source, lookup counts, and whether package metadata came
+from a cache hit or a fresh scan.
+
+CycloneDX and SPDX outputs record the selected metadata source in document
+metadata, including fields such as `nixpkgs:metadata_source_method`,
+`nixpkgs:path`, `nixpkgs:rev`, `nixpkgs:flakeref`, `nixpkgs:version`, and
+`nixpkgs:message`.

nix/apps.nix

@@ -29,9 +29,6 @@
           # nix run .#nixgraph
           nixgraph = mkApp "${sbomnix}/bin/nixgraph" "Visualize nix package dependencies";
 
-          # nix run .#nixmeta
-          nixmeta = mkApp "${sbomnix}/bin/nixmeta" "Summarize nixpkgs meta-attributes";
-
           # nix run .#vulnxscan
           vulnxscan = mkApp "${sbomnix}/bin/vulnxscan" "Scan nix artifacts or SBOMs for vulnerabilities";
 

nix/packages.nix

@@ -103,7 +103,6 @@
         ++ [
           (mkDevEntry "sbomnix" "sbomnix.main")
           (mkDevEntry "nixgraph" "nixgraph.main")
-          (mkDevEntry "nixmeta" "nixmeta.main")
           (mkDevEntry "nix_outdated" "nixupdate.nix_outdated")
           (mkDevEntry "vulnxscan" "vulnxscan.vulnxscan_cli")
           (mkDevEntry "repology_cli" "repology.repology_cli")

pyproject.toml

@@ -43,7 +43,6 @@ Homepage = "https://github.com/tiiuae/sbomnix"
 [project.scripts]
 sbomnix = "sbomnix.main:main"
 nixgraph = "nixgraph.main:main"
-nixmeta = "nixmeta.main:main"
 nix_outdated = "nixupdate.nix_outdated:main"
 vulnxscan = "vulnxscan.vulnxscan_cli:main"
 repology_cli = "repology.repology_cli:main"
@@ -59,6 +58,9 @@ version = { file = ["VERSION"] }
 [tool.setuptools.packages.find]
 where = ["src"]
 
+[tool.setuptools.package-data]
+sbomnix = ["*.nix"]
+
 [tool.ruff]
 line-length = 88
 target-version = "py310"
@@ -87,7 +89,6 @@ select = [
 known-first-party = [
   "common",
   "nixgraph",
-  "nixmeta",
   "nixupdate",
   "provenance",
   "repology",

src/common/flakeref.py

@@ -7,6 +7,7 @@
 import logging
 import pathlib
 import re
+import time
 
 from common.errors import FlakeRefRealisationError, FlakeRefResolutionError
 from common.log import LOG, LOG_VERBOSE
@@ -65,21 +66,29 @@ def try_resolve_flakeref(  # noqa: PLR0913
     else:
         log.log(LOG_VERBOSE, "Evaluating '%s'", flakeref)
     cmd = nix_cmd("eval", "--raw", flakeref, impure=impure)
+    started = time.perf_counter()
     ret = exec_cmd_fn(cmd, raise_on_error=False, return_error=True, log_error=False)
+    elapsed = time.perf_counter() - started
     if ret is None or ret.returncode != 0:
+        log.debug("nix eval failed for '%s' after %.3fs", flakeref, elapsed)
         if looks_like_flakeref:
             raise FlakeRefResolutionError(flakeref, ret.stderr if ret else "")
         log.debug("not a flakeref: '%s'", flakeref)
         return None
     nixpath = ret.stdout.strip()
+    log.log(LOG_VERBOSE, "Evaluated '%s' in %.3fs", flakeref, elapsed)
     log.debug("flakeref='%s' maps to path='%s'", flakeref, nixpath)
     if not force_realise:
         return nixpath
     log.info("Realising flakeref '%s'", flakeref)
     cmd = nix_cmd("build", "--no-link", flakeref, impure=impure)
+    started = time.perf_counter()
     ret = exec_cmd_fn(cmd, raise_on_error=False, return_error=True, log_error=False)
+    elapsed = time.perf_counter() - started
     if ret is None or ret.returncode != 0:
+        log.debug("nix build failed for '%s' after %.3fs", flakeref, elapsed)
         raise FlakeRefRealisationError(flakeref, ret.stderr if ret else "")
+    log.log(LOG_VERBOSE, "Realised flakeref '%s' in %.3fs", flakeref, elapsed)
     return nixpath
 
 
@@ -92,8 +101,15 @@ def _resolve_flakeref_derivation(flakeref, *, impure, exec_cmd_fn, log):
     """Return the derivation path for a flakeref target."""
     log.info("Evaluating flakeref '%s'", flakeref)
     cmd = nix_cmd("derivation", "show", flakeref, impure=impure)
+    started = time.perf_counter()
     ret = exec_cmd_fn(cmd, raise_on_error=False, return_error=True, log_error=False)
+    elapsed = time.perf_counter() - started
     if ret is None or ret.returncode != 0:
+        log.debug(
+            "nix derivation show failed for flakeref '%s' after %.3fs",
+            flakeref,
+            elapsed,
+        )
         raise FlakeRefResolutionError(flakeref, ret.stderr if ret else "")
     drv_paths = parse_nix_derivation_show(ret.stdout)
     drv_path = next(iter(drv_paths), "")
@@ -102,22 +118,19 @@ def _resolve_flakeref_derivation(flakeref, *, impure, exec_cmd_fn, log):
             flakeref,
             "nix derivation show returned no derivation path",
         )
+    log.log(
+        LOG_VERBOSE,
+        "Resolved flakeref derivation in %.3fs to '%s'",
+        elapsed,
+        drv_path,
+    )
     log.debug("flakeref='%s' maps to derivation='%s'", flakeref, drv_path)
     return drv_path
 
 
 def _force_realise_flakeref(flakeref, *, impure, exec_cmd_fn, log):
     """Return the realized output path for a runtime flakeref target."""
     log.info("Evaluating flakeref '%s'", flakeref)
-    nixpath = _realised_flakeref_path(
-        flakeref,
-        impure=impure,
-        exec_cmd_fn=exec_cmd_fn,
-    )
-    if nixpath:
-        log.debug("flakeref='%s' maps to realised path='%s'", flakeref, nixpath)
-        return nixpath
-
     return _build_flakeref_path(
         flakeref,
         impure=impure,
@@ -136,28 +149,28 @@ def _build_flakeref_path(flakeref, *, impure, exec_cmd_fn, log):
         flakeref,
         impure=impure,
     )
+    started = time.perf_counter()
     ret = exec_cmd_fn(cmd, raise_on_error=False, return_error=True, log_error=False)
+    elapsed = time.perf_counter() - started
     if ret is None or ret.returncode != 0:
+        log.debug("nix build failed for flakeref '%s' after %.3fs", flakeref, elapsed)
         raise FlakeRefRealisationError(flakeref, ret.stderr if ret else "")
     nixpath = _first_output_path(ret.stdout)
     if not nixpath:
         raise FlakeRefRealisationError(
             flakeref,
             "nix build returned no output path",
         )
+    log.log(
+        LOG_VERBOSE,
+        "Resolved flakeref to built path '%s' in %.3fs",
+        nixpath,
+        elapsed,
+    )
     log.debug("flakeref='%s' maps to path='%s'", flakeref, nixpath)
     return nixpath
 
 
-def _realised_flakeref_path(flakeref, *, impure, exec_cmd_fn):
-    """Return the already-realised output path for ``flakeref`` when available."""
-    cmd = nix_cmd("path-info", flakeref, impure=impure)
-    ret = exec_cmd_fn(cmd, raise_on_error=False, return_error=True, log_error=False)
-    if ret is None or ret.returncode != 0:
-        return None
-    return _first_output_path(ret.stdout)
-
-
 def parse_nixos_configuration_ref(
     flakeref: str,
     *,

src/common/proc.py

@@ -8,6 +8,7 @@
 import os
 import shlex
 import subprocess
+import time
 from collections.abc import Callable, Sequence
 from shutil import which
 from typing import IO, Literal, overload
@@ -20,6 +21,18 @@
 ExecCmdFn = Callable[..., ExecCmdResult]
 
 
+def _stream_summary(value: str | None) -> str:
+    if value is None:
+        return "not captured"
+    return f"{len(value)} byte(s), {value.count(chr(10))} line(s)"
+
+
+def _stream_preview(value: str, limit: int = 4000) -> str:
+    if len(value) <= limit:
+        return value.rstrip()
+    return f"{value[:limit].rstrip()}\n... truncated {len(value) - limit} byte(s)"
+
+
 @overload
 def exec_cmd(
     cmd: Sequence[CommandPart],
@@ -62,6 +75,7 @@ def exec_cmd(
         raise TypeError("cmd must be an argv sequence, not a string-like value")
     argv = [os.fspath(part) for part in cmd]
     command_str = shlex.join(argv)
+    started = time.perf_counter()
     LOG.debug("Running: %s", command_str)
     try:
         if stdout:
@@ -73,8 +87,26 @@ def exec_cmd(
                 encoding="utf-8",
                 check=True,
             )
+        LOG.debug(
+            "Finished in %.3fs (rc=%s, stdout=%s, stderr=%s): %s",
+            time.perf_counter() - started,
+            ret.returncode,
+            _stream_summary(ret.stdout),
+            _stream_summary(ret.stderr),
+            command_str,
+        )
+        if ret.stderr:
+            LOG.debug("Command stderr:\n%s", _stream_preview(ret.stderr))
         return ret
     except subprocess.CalledProcessError as error:
+        LOG.debug(
+            "Failed in %.3fs (rc=%s, stdout=%s, stderr=%s): %s",
+            time.perf_counter() - started,
+            error.returncode,
+            _stream_summary(error.stdout),
+            _stream_summary(error.stderr),
+            command_str,
+        )
         if log_error:
             LOG.error(
                 "Error running shell command:\n cmd:   '%s'\n stdout: %s\n stderr: %s",