Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,14 @@ Each case supports both benchmark directions:
advertising a narrower, language-specific implementation. It is not a
scoring fallback; after the operation is authored, its response is still
judged strictly and never substituted with the other operation.
`compatibleOperations` may explicitly list a separately reviewed alternate
endpoint that reaches the same complete authored target. Runners execute it
as a second result: the canonical operation remains the case `status`, while
`requiredDestinationStatus` reports whether the raw canonical or reviewed
compatible results include the destination. That recall-forward field also
tolerates line-only or containing ranges and additional results. It is never
a silent retry and cannot change strict endpoint precision or capability
results.
`allowedExtraTargets` may list conservative alternate targets that can
accompany the required `expectedDeclaration` but cannot replace it. An
alternate may have a different symbol kind when navigation legitimately
Expand Down
3 changes: 3 additions & 0 deletions benchmarks/cases/go-baseline.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@ cases:
displayName: Last
usageLookups:
- operation: declaration
compatibleOperations: [definition]
usage:
location:
uri: benchmark://source/src/example/service_test.go
Expand Down Expand Up @@ -228,6 +229,7 @@ cases:
displayName: DefaultPrefix
usageLookups:
- operation: declaration
compatibleOperations: [definition]
usage:
location:
uri: benchmark://source/src/example/service_test.go
Expand Down Expand Up @@ -271,6 +273,7 @@ cases:
displayName: DefaultRepository
usageLookups:
- operation: declaration
compatibleOperations: [definition]
usage:
location:
uri: benchmark://source/src/example/service_test.go
Expand Down
1 change: 1 addition & 0 deletions benchmarks/cases/go-lsp-parity.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,7 @@ cases:
displayName: Last
usageLookups:
- operation: declaration
compatibleOperations: [definition]
usage:
location:
uri: benchmark://source/cmd/app/main.go
Expand Down
1 change: 1 addition & 0 deletions benchmarks/cases/go-precision.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ cases:
displayName: Record
usageLookups:
- operation: declaration
compatibleOperations: [definition]
usage:
location: { uri: benchmark://source/cmd/app/main.go, range: { start: { line: 10, character: 10 }, end: { line: 10, character: 16 } } }
kind: method
Expand Down
1 change: 1 addition & 0 deletions benchmarks/cases/javascript-baseline.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,7 @@ cases:
displayName: title
usageLookups:
- operation: declaration
compatibleOperations: [definition]
usage:
location:
uri: benchmark://source/src/components.js
Expand Down
2 changes: 2 additions & 0 deletions benchmarks/cases/php-baseline.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,7 @@ cases:
displayName: last
usageLookups:
- operation: declaration
compatibleOperations: [definition]
usage:
location:
uri: benchmark://source/src/Consumer.php
Expand Down Expand Up @@ -238,6 +239,7 @@ cases:
displayName: PREFIX
usageLookups:
- operation: declaration
compatibleOperations: [definition]
usage:
location:
uri: benchmark://source/src/Consumer.php
Expand Down
1 change: 1 addition & 0 deletions benchmarks/cases/php-lsp-parity.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,7 @@ cases:
displayName: "$sent"
usageLookups:
- operation: declaration
compatibleOperations: [definition]
usage:
location:
uri: benchmark://source/src/Consumer.php
Expand Down
1 change: 1 addition & 0 deletions benchmarks/cases/php-precision.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ cases:
displayName: send
usageLookups:
- operation: declaration
compatibleOperations: [definition]
usage:
location: { uri: benchmark://source/src/Consumer.php, range: { start: { line: 11, character: 15 }, end: { line: 11, character: 19 } } }
kind: method
Expand Down
1 change: 1 addition & 0 deletions benchmarks/cases/typescript-baseline.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,7 @@ cases:
displayName: name
usageLookups:
- operation: declaration
compatibleOperations: [definition]
usage:
location:
uri: benchmark://source/src/components.tsx
Expand Down
1 change: 1 addition & 0 deletions benchmarks/cases/typescript-lsp-parity.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,7 @@ cases:
displayName: name
usageLookups:
- operation: declaration
compatibleOperations: [definition]
usage:
location:
uri: benchmark://source/src/app.ts
Expand Down
1 change: 1 addition & 0 deletions benchmarks/cases/typescript-precision.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ cases:
displayName: Widget
usageLookups:
- operation: declaration
compatibleOperations: [definition]
usage:
location: { uri: benchmark://source/src/app.ts, range: { start: { line: 2, character: 14 }, end: { line: 2, character: 20 } } }
kind: interface
Expand Down
71 changes: 40 additions & 31 deletions docs/src/content/docs/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -36,88 +36,97 @@ those decisions as executable regression contracts, and the daily workflow runs
them against Bifrost `master` so a previously satisfied semantic edge cannot
quietly disappear.

The current result starts from the synchronized UsageBench `78e66e5` run and
Bifrost `782522b2`. The Java profile and pinned Bifrost revision were rerun
after correcting JDT workspace isolation and three navigation operations.
Metals was then rerun at 1.6.8; its Scala results were unchanged. The other
eight reference profiles retain their synchronized reports. Every runner
completed without an error. The report shape remains analyzer-neutral:
language servers are the first comparison set, not a permanent limit, and
future analyzers can be added without changing the authored ground truth.
The current result uses UsageBench parent revision `d01a4aa` and pinned Bifrost
`782522b2`. Bifrost, gopls, TypeScript LS, and Intelephense were freshly rerun
after adding the reviewed compatible-operation evidence and report-derived
destination scorer. The corrected Java and refreshed Metals reports are
retained, as are the five other synchronized reference reports whose cases and
operations did not change. Every fresh report records zero runner errors. The
report shape remains analyzer-neutral: language servers are the first
comparison set, not a permanent limit, and future analyzers can be added without
changing the authored ground truth.

## Required destinations found

The headline metric asks the question closest to normal editor use: **did the
operation surface every reviewed destination the user needed?** A case counts
when all required references are present and every navigation or type lookup
includes its expected target. A containing range or additional result does not
turn that into a failure.
turn that into a failure; neither does a line-only location on the expected
line.

This is intentionally recall-forward. It does not say that imprecise ranges,
unrelated results, or a wrong first choice are harmless; the stricter table
below measures those differences. The denominator contains the **131 cases
scoreable by both Bifrost and the corresponding reference server**.
below measures those differences. The denominator contains the **144 cases
scoreable by both sides using either the canonical operation or an explicitly
reviewed compatible operation**. It does not
silently retry a request: the canonical endpoint remains the source of truth
in the strict table below.

<div class="result-summary" aria-label="Current corrected result summary">
<div class="result-metric">
<span>Required destinations found</span>
<div class="metric-comparison" role="group" aria-label="Bifrost 90.1 percent; reference LSPs 90.1 percent">
<div class="metric-comparison" role="group" aria-label="Bifrost 88.9 percent; reference LSPs 91.0 percent">
<div class="metric-side metric-side--bifrost">
<span>Bifrost</span>
<strong>90.1%</strong>
<strong>88.9%</strong>
</div>
<div class="metric-side metric-side--reference">
<span>Reference LSPs</span>
<strong>90.1%</strong>
<strong>91.0%</strong>
</div>
</div>
<p>118/131 destinations each</p>
<p>128/144 · 131/144 destinations</p>
</div>
<div class="result-metric">
<span>Equal-profile mean</span>
<div class="metric-comparison" role="group" aria-label="Bifrost 90.2 percent; reference LSPs 91.4 percent">
<div class="metric-comparison" role="group" aria-label="Bifrost 88.9 percent; reference LSPs 91.6 percent">
<div class="metric-side metric-side--bifrost">
<span>Bifrost</span>
<strong>90.2%</strong>
<strong>88.9%</strong>
</div>
<div class="metric-side metric-side--reference">
<span>Reference LSPs</span>
<strong>91.4%</strong>
<strong>91.6%</strong>
</div>
</div>
<p>equal weight for each of ten profiles</p>
</div>
<div class="result-metric">
<span>Median paired profile gap</span>
<strong>0.0 <small>points</small></strong>
<p>the middle cross-profile difference is a tie</p>
<strong>−3.1 <small>points</small></strong>
<p>the middle profile favors the reference server</p>
</div>
<div class="result-metric">
<span>Profile outcomes</span>
<strong>3–3–4</strong>
<strong>3–2–5</strong>
<p>Bifrost leads · ties · trails</p>
</div>
</div>

| Profile | Shared | Bifrost destinations | Reference destinations | Gap | Reference unsupported |
| Profile | Shared destination-scoreable | Bifrost destinations | Reference destinations | Gap | Reference unavailable |
|---|---:|---:|---:|---:|---:|
| [C++](languages/cpp/) | 15 | 12/15 (80.0%) | 14/15 (93.3%) | −13.3 pp | 1 |
| [C#](languages/csharp/) | 16 | 14/16 (87.5%) | 15/16 (93.8%) | −6.3 pp | 0 |
| [Go](languages/go/) | 6 | 5/6 (83.3%) | 6/6 (100.0%) | −16.7 pp | 6 |
| [Go](languages/go/) | 11 | 9/11 (81.8%) | 11/11 (100.0%) | −18.2 pp | 1 |
| [Java](languages/java/) | 11 | 11/11 (100.0%) | 11/11 (100.0%) | 0.0 pp | 0 |
| [JavaScript, TypeScript](languages/javascript-typescript/) | 17 | 15/17 (88.2%) | 15/17 (88.2%) | 0.0 pp | 5 |
| [PHP](languages/php/) | 10 | 10/10 (100.0%) | 10/10 (100.0%) | 0.0 pp | 4 |
| [JavaScript, TypeScript](languages/javascript-typescript/) | 21 | 19/21 (90.5%) | 19/21 (90.5%) | 0.0 pp | 1 |
| [PHP](languages/php/) | 14 | 12/14 (85.7%) | 14/14 (100.0%) | −14.3 pp | 0 |
| [Python](languages/python/) | 13 | 11/13 (84.6%) | 13/13 (100.0%) | −15.4 pp | 0 |
| [Ruby](languages/ruby/) | 16 | 15/16 (93.8%) | 11/16 (68.8%) | +25.0 pp | 4 |
| [Rust](languages/rust/) | 15 | 14/15 (93.3%) | 13/15 (86.7%) | +6.7 pp | 0 |
| [Scala](languages/scala/) | 12 | 11/12 (91.7%) | 10/12 (83.3%) | +8.3 pp | 3 |
| **Pooled** | **131** | **118/131 (90.1%)** | **118/131 (90.1%)** | **0.0 pp** | **23** |
| **Pooled** | **144** | **128/144 (88.9%)** | **131/144 (91.0%)** | **−2.1 pp** | **10** |

Fresh `requiredDestinationStatus` reports reproduce the three affected profile
rows exactly. The pinned full-corpus Bifrost report finds 135/152 individually
scoreable cases; intersecting scoreable case IDs with each reference report
produces the paired 128/144 total shown here.

The two sides reach the same pooled result through different strengths. The
reference servers lead on C++, C#, Go, and Python; Bifrost leads on Ruby, Rust,
and Scala. Java, JavaScript/TypeScript, and PHP tie. On Java specifically, JDT
LS surfaces every required destination even though only four cases satisfy the
stricter range-and-identity contract below.
The reference servers lead on C++, C#, Go, PHP, and Python; Bifrost leads on
Ruby, Rust, and Scala. Java and JavaScript/TypeScript tie. On Java
specifically, JDT LS surfaces every required destination even though only four
cases satisfy the stricter range-and-identity contract below.

<div class="evidence-note">
“Required destinations found” tolerates result clutter. For example, Ruby
Expand Down
21 changes: 14 additions & 7 deletions docs/src/content/docs/languages/go.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,18 @@ title: Go — Bifrost and gopls
description: Compare package, embedding, receiver, and interface-method reference behavior.
---

| Runner | Exact | Position unverified | Hard | Unsupported |
|---|---:|---:|---:|---:|
| Bifrost | 9 | 0 | 2 | 1 |
| gopls | 6 | 0 | 0 | 6 |
| Runner | Required destinations found | Strict exact | Position unverified | Hard | Strict unsupported |
|---|---:|---:|---:|---:|---:|
| Bifrost | 9/11 | 9 | 0 | 2 | 1 |
| gopls | 11/11 | 6 | 0 | 0 | 6 |

## Strong agreement

Both analyzers can score six common cases and are exact together on five.
gopls alone is exact on the dot-import concrete-receiver call; Bifrost currently
misclassifies the imported `Record` selectors as shadowed local bindings.
In the compatibility-aware editor view, eleven cases are destination-scoreable
by both analyzers. gopls reaches all eleven destinations; Bifrost misses the
dot-import concrete-receiver call and the interface-family reference set. The
strict canonical comparison still has only six shared cases, where gopls is
exact on all six and Bifrost is exact on five.

## The interface-family split

Expand All @@ -22,6 +24,11 @@ Definition. Bifrost can score five of those six, but its interface receiver case
still misses the two conservative concrete candidates and the declaration
lookup.

Five of those canonical Declaration contracts also record reviewed Definition
compatibility. They raise the shared destination-scoreable denominator from six
to eleven, but do not change the strict gopls Declaration denominator or
capability result.

This consistent behavior may reflect an intentional method-family policy. It is
not enough to claim object insensitivity: the case does not vary allocation or
receiver contexts while keeping every other factor fixed.
Expand Down
16 changes: 12 additions & 4 deletions docs/src/content/docs/languages/javascript-typescript.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@ title: JavaScript and TypeScript — Bifrost and TypeScript LS
description: Compare ES modules, CommonJS, barrels, types, JSX, and conservative receiver evidence.
---

| Runner | Exact | Position unverified | Hard | Unsupported | Not planned |
|---|---:|---:|---:|---:|---:|
| Bifrost | 20 | 0 | 2 | 0 | 1 |
| TypeScript LS | 13 | 2 | 2 | 5 | 1 |
| Runner | Required destinations found | Strict exact | Position unverified | Hard | Strict unsupported | Not planned |
|---|---:|---:|---:|---:|---:|---:|
| Bifrost | 19/21 | 20 | 0 | 2 | 0 | 1 |
| TypeScript LS | 19/21 | 13 | 2 | 2 | 5 | 1 |

## TypeScript agreement after import policy

Expand All @@ -15,6 +15,14 @@ policy near misses no longer obscure semantic agreement. On the 17 cases both
sides can score, 11 are exact for both, 4 only for Bifrost, and 2 only for the
TypeScript language server.

The class-property contract, the two TypeScript-property contracts, and the
type-annotation contract retain their reviewed Declaration operation while
recording Definition as explicit compatibility evidence. Those four cases raise
the shared destination-scoreable denominator from 17 to 21 without rewriting
the strict endpoint comparison. The CommonJS exported-member contract is
intentionally excluded: its Definition target is a distinct
function-expression token.

## CommonJS split

Bifrost satisfies the destructured-function and barrel-class CommonJS cases
Expand Down
13 changes: 9 additions & 4 deletions docs/src/content/docs/languages/php.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,22 @@ title: PHP — Bifrost and Intelephense
description: Compare namespaces, imports, traits, interfaces, receivers, properties, and static members.
---

| Runner | Exact | Position unverified | Hard | Unsupported |
|---|---:|---:|---:|---:|
| Bifrost | 12 | 0 | 2 | 0 |
| Intelephense | 9 | 0 | 1 | 4 |
| Runner | Required destinations found | Strict exact | Position unverified | Hard | Strict unsupported |
|---|---:|---:|---:|---:|---:|
| Bifrost | 12/14 | 12 | 0 | 2 | 0 |
| Intelephense | 14/14 | 9 | 0 | 1 | 4 |

## Shared strengths

Both analyzers can score ten common cases and are exact together on nine.
Bifrost alone is exact on class construction because Intelephense returns both
the class and explicit constructor as navigation targets.

Four canonical Declaration contracts—properties, a constant, and a bodyless
interface member—also record Definition as reviewed compatibility evidence.
They raise the shared destination-scoreable denominator from ten to fourteen;
strict operation precision and capability counts remain unchanged.

## Bifrost recall edge

Bifrost also scores four Declaration-oriented cases that Intelephense does not
Expand Down
24 changes: 19 additions & 5 deletions docs/src/content/docs/methodology.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,14 @@ The user-facing metric counts a case when:
2. every navigation lookup includes its expected destination; and
3. every type lookup includes its expected type destination.

A broader containing range and additional returned results are tolerated. This
models the basic editor question—“did the operation take me to, or list, the
code I needed?”—without pretending that range precision and result clutter are
invisible to users. Unsupported operations remain outside the shared
denominator.
A line-only location, a broader containing range, and additional returned
results are tolerated. This models the basic editor question—“did the operation
take me to, or list, the code I needed?”—without pretending that range
precision and result clutter are invisible to users. A case remains in this
shared denominator when both analyzers can execute every required lookup
through either the canonical operation or an explicitly reviewed compatible
operation. Operations unavailable through both paths remain outside it; the
strict metric continues to use only the authored canonical operation.

This is best read as **required-destination recall**, not an all-purpose quality
score. A response containing the expected target among many unrelated targets
Expand Down Expand Up @@ -72,6 +75,17 @@ capability-advertisement bias. It does **not** make the scorer lenient: once a
case selects Declaration or Definition, that exact request must satisfy the
strict singleton target contract, with no fallback to the other endpoint.

Cases may additionally declare reviewed `compatibleOperations` when another
endpoint reaches the exact same authored target. The runner records these as
separate alternate results: `status` remains the canonical protocol-specific
score, while `requiredDestinationStatus` is computed from the raw canonical
and reviewed-compatible locations. It records `found` when all expected
destinations are present even if ranges contain the expected token or the
response includes extras. An alternate is never queried silently, never merged
into the canonical response, and cannot improve the strict endpoint result.
Per-report `totals.requiredDestinations` makes this metric machine-readable
without changing the canonical counters.

The strict result is not labeled generic LSP compliance. The current
[`Location`](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.18/specification/#location)
shape requires a URI and range, and the protocol defines how ranges are
Expand Down
Loading
Loading