Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,5 +63,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Exit codes: `0` completed (regardless of findings), `1` scan failed or partial, `2` usage
error. Findings alone never fail a build.
- Maven graphs are marked approximate — they are reconstructed, not read, and we never shell
out to `mvn`.
out to `mvn`. Each module of a multi-module build resolves against its own POM, so module
boundaries hold: one module's nearest-wins choice cannot decide another's versions, and each
transitive is attributed to the `<dependency>` declaration that introduced it.
- Maven `<exclusions>` are honoured on declared dependencies as well as inherited ones,
including entries supplied by `dependencyManagement` and Maven's whole-segment wildcards.
Excluded coordinates are absent from the report, and the `exclusions` field on each
dependency records what was removed.
- Licensed under Apache 2.0.
28 changes: 25 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,8 +123,22 @@ as version `(123456,)` and orders as a real version.
**Maven is approximate and says so.** Parent inheritance, BOM imports, nearest-wins and exclusions
— not profiles, mirrors, relocation or version ranges. Never shell out to `mvn`: running a
project's build to discover its dependencies is itself a supply chain risk.
`MavenResolver._backfill` re-reads `pom.xml` to apply BOM-supplied versions to direct dependencies,
which the synchronous parser cannot do because BOMs live on Central.
`MavenResolver._backfill` re-reads `pom.xml` to apply BOM-supplied versions and exclusions to
direct dependencies, which the synchronous parser cannot do because BOMs live on Central.

**Maven resolves one module at a time.** `_maven_units` in `scanner.py` regroups the flat
dependency list back onto its scan units by `source.path`, and `expand_units` walks each
separately on one shared resolver — shared cache, shared connection pool, shared node budget.
Merging a reactor into one traversal gives whichever module is walked first the deciding vote on
every shared coordinate, and leaves every transitive stamped with a POM that need not lead to it.

**Exclusions are the one thing here that removes packages.** Everything else in this codebase
fails towards over-reporting; `is_excluded` fails the other way, so a bug in it hides a real
dependency instead of adding noise. Hence: wildcards are whole-segment only (`*:*`, `g:*`,
`*:a` — never a prefix glob), half-declared `<exclusion>` elements are dropped at parse time so
they cannot become accidental wildcards, and `Dependency.exclusions` is emitted in JSON so every
removal stays auditable. The negative tests in `tests/test_maven.py` that assert a wildcard does
*not* fire are load-bearing.

**Ranges, SARIF and CycloneDX are hand-written** rather than pulled from packages — a dependency
scanner with a large dependency tree of its own is a poor advertisement. Correctness is held by
Expand Down Expand Up @@ -159,6 +173,14 @@ and versions. This has caught two real bugs (a missing `coverage[toml]` extra tr
- No SBOM *ingest*.
- npm/yarn v1 lockfiles record no scope, so dev transitives are reported as runtime unless a
`package.json` sits alongside.
- Gradle sees only literal declarations — no version catalogues or computed versions.
- Gradle sees only literal declarations — no version catalogues or computed versions, and no
`exclude` handling, so a Gradle graph over-reports where a Maven one would not.
- A parent POM that exists only on disk is never read. `_effective` fetches parents from
Central and `_backfill` reads a module's own POM, so an unpublished aggregator parent supplies
neither versions nor exclusions to its modules; the dependencies affected stay `unresolved`.
This is the largest remaining gap for real reactors.
- `dependencyManagement` exclusions supplied by an imported BOM reach a project only when that
project also has a version to resolve — `_backfill` is gated on an unresolved version so a
fully-versioned project pays no network cost.
- OSV's unversioned second pass for `MAL-` advisories on yanked packages is not implemented;
malicious advisories affecting the installed version still surface normally.
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ files that introduced it:
the report describes what is actually installed, not what would resolve today. Only when no
lockfile is present does it fall back to resolving version ranges against the registry, and
those findings are labelled `resolved` rather than `pinned`. Java, which has no lockfile, gets
its graph reconstructed from Maven Central and is marked `~` for approximate.
its graph reconstructed from Maven Central and is marked `~` for approximate — one module at a
time, so a reactor's module boundaries survive into the report.

**A clean result is never implied unless it was earned.** Most of the rest of the design
follows from this:
Expand Down
12 changes: 12 additions & 0 deletions src/icebergsca/.agents/skills/icebergsca/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,18 @@ anything was checked.
* **Maven and Gradle graphs are approximate.** Java has no lockfile, so the graph is
reconstructed from Maven Central. Affected manifests carry `"approximate": true`. Mention
this when reporting Java results.
* **Multi-module Maven provenance is trustworthy — you do not need to re-derive it.** Each
module resolves against its own POM, so a transitive's `source` is the declaration that
actually introduced it, `source.line` points at that `<dependency>` element, and `parents`
names the package it came through. Two modules may legitimately report different versions of
the same coordinate; that is Maven's per-module nearest-wins, not a bug. What `approximate`
still covers: profiles, mirrors, `<relocation>`, version ranges, and a parent POM that lives
only on disk and was never published to Central — management supplied that way is missed, and
the dependencies affected show up as `pin: "unresolved"`.
* **`<exclusions>` are applied, so an excluded artifact is absent by design.** Declared and
`dependencyManagement`-supplied exclusions both count, including Maven's whole-segment
wildcards. If a package you expected is missing, check the `exclusions` list on the
dependency that would have pulled it in before concluding the scan missed it.
* **Go reads `go.mod`, never `go.sum`.** `go.sum` lists versions merely *considered* during
resolution, so scanning it reports vulnerabilities in code that was never built.
* **`yarn.lock` v1 records no scope.** Without a sibling `package.json`, dev transitives are
Expand Down
16 changes: 12 additions & 4 deletions src/icebergsca/.agents/skills/icebergsca/references/json-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,9 @@ One entry per ecosystem-and-directory scan unit.
* `parsed` is what was **actually read**, which differs from `lockfiles` when a lockfile could
not be parsed and the manifest was used instead. Report `parsed`, not `lockfiles`.
* `from_lockfile: false` means versions are declared, not installed.
* `approximate: true` means the graph was reconstructed rather than read — currently Maven.
* `approximate: true` means the graph was reconstructed rather than read — currently Maven. It
does *not* mean provenance is guesswork: each module is resolved on its own, so a
transitive's `source` names the declaration that introduced it.
* `dependency_count` values sum to `summary.dependencies`.

## `dependencies`
Expand All @@ -100,7 +102,8 @@ One entry per ecosystem-and-directory scan unit.
"pin": "pinned",
"constraint": null,
"source": { "path": "requirements.txt", "line": 4 },
"parents": ["pkg:pypi/requests@2.19.1"]
"parents": ["pkg:pypi/requests@2.19.1"],
"exclusions": []
}
```

Expand All @@ -112,8 +115,13 @@ One entry per ecosystem-and-directory scan unit.
* `constraint` is the raw declared text (`">=2.0,<3.0"`, `"^4.17.21"`) when the version was not
pinned outright.
* `source.line` is `null` for JSON and XML manifests, where a line number would be guesswork.
* `parents` is populated only for lockfile formats that record edges. Empty means "unknown",
never "nothing depends on it".
* `parents` is populated for lockfile formats that record edges and for reconstructed Maven
transitives. Empty means "unknown", never "nothing depends on it".
* `source` on a reconstructed Maven transitive is the declaration that introduced it — the
module's own POM and the `<dependency>` line — not merely the first file scanned.
* `exclusions` lists the `group:artifact` keys this declaration removes from its own subtree
(Maven `<exclusions>`, wildcards included). A package named here is absent from the graph
deliberately, which is the one case where something is missing without being an error.

## `findings`

Expand Down
6 changes: 6 additions & 0 deletions src/icebergsca/core/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,12 @@ class Dependency:
#: Packages that pull this one in. Only populated for lockfiles that record
#: edges; empty is "we don't know", never "nothing depends on it".
parents: tuple[PackageRef, ...] = ()
#: ``group:artifact`` keys this dependency must not drag in — Maven
#: ``<exclusions>`` today, Gradle ``exclude`` and npm ``overrides`` later.
#: Empty means "nothing was excluded", and deliberately not ``None`` for "we
#: could not tell": a parser that cannot read exclusions leaves this empty and
#: the graph over-reports, which is the safe way round.
exclusions: frozenset[str] = frozenset()

@property
def is_runtime(self) -> bool:
Expand Down
48 changes: 44 additions & 4 deletions src/icebergsca/core/scanner.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
)
from icebergsca.ecosystems import get as get_ecosystem
from icebergsca.ecosystems.base import FileParser
from icebergsca.ecosystems.maven import MavenResolver, MavenResult
from icebergsca.ecosystems.maven import MavenResolver, MavenResult, MavenUnit
from icebergsca.osv.client import USER_AGENT, OSVClient, OSVResult
from icebergsca.registry import RegistryClient
from icebergsca.resolve import Resolver
Expand Down Expand Up @@ -89,7 +89,7 @@ async def scan(root: Path, options: ScanOptions | None = None) -> ScanReport:
discovery = discover(root, options.discovery)
kept, results, skipped, warnings = _parse_all(discovery, options.scopes)

outcome = await _resolve_and_check(kept, options, discovery.root)
outcome = await _resolve_and_check(kept, discovery.units, options, discovery.root)

return ScanReport(
root=discovery.root,
Expand Down Expand Up @@ -138,7 +138,10 @@ class _VulnOutcome:


async def _resolve_and_check(
dependencies: tuple[Dependency, ...], options: ScanOptions, root: Path
dependencies: tuple[Dependency, ...],
units: tuple[ScanUnit, ...],
options: ScanOptions,
root: Path,
) -> _VulnOutcome:
"""Stages 3 and 4: resolve unpinned constraints, then look everything up in OSV.

Expand Down Expand Up @@ -173,7 +176,9 @@ async def _resolve_and_check(
cache,
offline=options.offline,
concurrency=options.concurrency,
).expand(dependencies, root=root)
).expand_units(
dependencies, _maven_units(dependencies, units), root=root
)
dependencies = tuple(
dep for dep in maven.dependencies if dep.scope in options.scopes
)
Expand Down Expand Up @@ -237,6 +242,41 @@ async def _resolve_and_check(
)


def _maven_units(
dependencies: tuple[Dependency, ...], units: tuple[ScanUnit, ...]
) -> tuple[MavenUnit, ...]:
"""Regroup the flat dependency list back onto the scan units it came from.

Discovery assigns every file to exactly one (ecosystem, directory) unit, and the
parsers stamp each dependency with the file it was declared in, so ``source.path``
identifies a dependency's unit exactly. Regrouping here rather than threading units
through the parse stage keeps ``_parse_all`` unchanged and — because it happens
after ``_dedupe`` — guarantees the resolver sees the same list the report will.

A Maven dependency whose path belongs to no unit still gets a unit of its own.
Silently dropping it would remove packages from the report, which is the one thing
this stage must never do.
"""
owner = {
path: PurePath(unit.directory)
for unit in units
if unit.ecosystem is EcosystemId.MAVEN
for path in unit.files
}

grouped: dict[PurePath, list[Dependency]] = defaultdict(list)
for dep in dependencies:
if dep.ref.ecosystem is not EcosystemId.MAVEN:
continue
path = PurePath(dep.source.path)
grouped[owner.get(path, path.parent)].append(dep)

return tuple(
MavenUnit(directory=directory, dependencies=tuple(deps))
for directory, deps in grouped.items()
)


def _finalise_manifests(
results: tuple[ManifestResult, ...],
dependencies: tuple[Dependency, ...],
Expand Down
4 changes: 2 additions & 2 deletions src/icebergsca/ecosystems/maven/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from icebergsca.core.models import EcosystemId
from icebergsca.ecosystems.base import EcosystemSpec, FileSpec, unimplemented
from icebergsca.ecosystems.maven.parser import parse_manifest
from icebergsca.ecosystems.maven.resolver import MavenResolver, MavenResult
from icebergsca.ecosystems.maven.resolver import MavenResolver, MavenResult, MavenUnit

SPEC = EcosystemSpec(
id=EcosystemId.MAVEN,
Expand All @@ -19,4 +19,4 @@
parse_lockfile=unimplemented("Maven lockfile"),
)

__all__ = ["SPEC", "MavenResolver", "MavenResult", "parse_manifest"]
__all__ = ["SPEC", "MavenResolver", "MavenResult", "MavenUnit", "parse_manifest"]
21 changes: 21 additions & 0 deletions src/icebergsca/ecosystems/maven/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,27 @@ def key(self) -> str:
return f"{self.group}:{self.artifact}"


def is_excluded(key: str, exclusions: frozenset[str]) -> bool:
"""True when a ``group:artifact`` key matches any exclusion pattern.

Maven 3 allows ``*`` in either position, but only as a *whole* segment: ``org.foo*``
is not a prefix glob in Maven and must not become one here. That restraint matters
more than the feature does. An exclusion is the one thing in this tool that removes
a package from the report rather than adding one, so an over-eager pattern hides a
real dependency instead of merely making noise.
"""
if key in exclusions:
# The overwhelmingly common case, and the only one that costs nothing.
return True

group, _, artifact = key.partition(":")
return any(
pattern in ("*:*", f"{group}:*", f"*:{artifact}")
for pattern in exclusions
if "*" in pattern
)


@dataclass(frozen=True, slots=True)
class Pom:
"""One parsed POM file, with nothing inherited or interpolated yet."""
Expand Down
58 changes: 48 additions & 10 deletions src/icebergsca/ecosystems/maven/parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -158,13 +158,21 @@ def _dependencies(container: Element | None) -> tuple[RawDependency, ...]:
parsed: list[RawDependency] = []
for element in _children(container, "dependency"):
exclusions_element = _child(element, "exclusions")
exclusions = (
frozenset(
f"{_text(item, 'groupId')}:{_text(item, 'artifactId')}"
for item in _children(exclusions_element, "exclusion")
)
# A half-declared exclusion is discarded rather than kept as ``g:`` or ``:a``.
# Exclusions remove packages from the report, and a half-key sits one wildcard
# rule away from matching everything — dropping it over-reports, which is the
# direction to fail in.
items = (
_children(exclusions_element, "exclusion")
if exclusions_element is not None
else frozenset()
else []
)
exclusions = frozenset(
f"{group}:{artifact}"
for group, artifact in (
(_text(item, "groupId"), _text(item, "artifactId")) for item in items
)
if group and artifact
)

parsed.append(
Expand Down Expand Up @@ -241,29 +249,59 @@ def _parse_pom_manifest(path: Path, content: str) -> list[Dependency]:
for entry in pom.managed
if entry.version
}
# Maven merges a managed entry's exclusions with the dependency's own rather than
# letting either replace the other, which is how a parent centralises an exclusion
# for a dependency that declares none of its own.
managed_exclusions = {
entry.key: entry.exclusions for entry in pom.managed if entry.exclusions
}

dependencies: list[Dependency] = []
for entry in pom.dependencies:
group = interpolate(entry.group, properties) or entry.group
artifact = interpolate(entry.artifact, properties) or entry.artifact
version = interpolate(entry.version, properties) or managed_versions.get(
f"{group}:{artifact}"
)
key = f"{group}:{artifact}"
version = interpolate(entry.version, properties) or managed_versions.get(key)

usable = version if version and not has_unresolved_property(version) else None
dependencies.append(
Dependency(
ref=PackageRef(EcosystemId.MAVEN, f"{group}:{artifact}", usable),
ref=PackageRef(EcosystemId.MAVEN, key, usable),
scope=maven_scope(entry.scope or DEFAULT_SCOPE),
direct=True,
source=SourceLocation(path, _find_line(content, artifact)),
pin=Pin.PINNED if usable else Pin.UNRESOLVED,
constraint=version,
exclusions=_interpolated_exclusions(
entry.exclusions | managed_exclusions.get(key, frozenset()),
properties,
),
)
)
return dependencies


def _interpolated_exclusions(
exclusions: frozenset[str], properties: dict[str, str]
) -> frozenset[str]:
"""Resolve ``${...}`` in exclusion keys, which are coordinates like any other.

A key left with an unresolved property is kept verbatim: it will simply not match,
which leaves the package in the report rather than removing it on a guess.
"""
if not exclusions:
return frozenset()

resolved: set[str] = set()
for key in exclusions:
group, _, artifact = key.partition(":")
resolved.add(
f"{interpolate(group, properties) or group}"
f":{interpolate(artifact, properties) or artifact}"
)
return frozenset(resolved)


def _parse_gradle(path: Path, content: str) -> list[Dependency]:
"""Read declared coordinates out of a Gradle build script.

Expand Down
Loading