Skip to content

Test262: 62 deterministic compiled-mode regressions vs stale baseline (14 compiled-only codegen bugs) #882

Description

@nickna

Summary

A clean run of the compiled Test262 baseline (Test262CompiledTests.CompiledBaseline) against the committed baselines/compiled.txt shows 62 regressions and 108 new passes (net +46; suite still improved overall). All 62 regressions are deterministic — the 20 sampled in the run log reproduced 20/20, and the full set was reconstructed and independently re-verified (882 candidate paths re-run → exactly 62, zero flake).

The root context: baselines/compiled.txt is 187 commits stale — last regenerated at 0494a763 (#693, 2026-06-15), while baselines/interpreted.txt is current as of b9d3aa3f (#802, 2026-06-17). So this drift is the accumulated compiled-mode delta across 187 commits, never recorded.

Cross-referencing each regressed path against the (current) interpreted baseline splits the 62 cleanly:

Class Count Meaning
Compiled-only (interp still Pass) 14 Genuine compiled/codegen regressions — interpreter gets these right, compiled does not. Priority.
Shared (interp baseline already Fail/RuntimeError) 48 Behavior changes already accepted on the interpreted side at #802; the stale compiled baseline just never recorded them.

Buckets: 37 RuntimeError + 25 Fail.

Reproduce

dotnet build SharpTS.Test262.Worker/SharpTS.Test262.Worker.csproj -c Release   # worker is NOT rebuilt by dotnet test
dotnet test SharpTS.Test262/SharpTS.Test262.csproj -c Release \
  --filter "FullyQualifiedName~CompiledBaseline"        # ~17 min; run backgrounded
# Regenerate (only after the 14 compiled-only are triaged — don't bury them):
#   SHARPTS_TEST262_UPDATE_BASELINE=1 dotnet test ... --filter "FullyQualifiedName~CompiledBaseline"

The 14 compiled-only regressions (priority — codegen bugs)

These pass in the interpreter but fail compiled. Three themes:

  • Sloppy-mode this not bound to the global objectArray/prototype/filter/15.4.4.20-5-1, some/15.4.4.17-5-1 (callback reads this._x where this should be global), language/expressions/call/11.2.3-3_8 (this.bar(foo()) at top level). Compiled throws instead of resolving this → globalThis.

  • Object.defineProperty/defineProperties/create reading descriptor fields off an attributes object that has an accessor without a getter15.2.3.6-3-215..217, 15.2.3.7-5-b-201..203, 15.2.3.5-4-241..243. e.g. Object.defineProperty(attrs, "get", {set(){}}) then defineProperty(obj, "p", attrs); reading attrs.get must yield undefined, compiled throws (RuntimeError).

  • eval with exotic whitespace between new and the operandlanguage/expressions/new/S11.2.2_A1.1, A1.2 evaluate new<WS>Number for WS ∈ {U+0009 tab, U+000B VT, U+000C FF, U+00A0 NBSP, U+2028, U+2029, …} via eval(...) and expect 0.

  • test/built-ins/Array/prototype/filter/15.4.4.20-5-1.js Pass -> RuntimeError (interp: Pass)

  • test/built-ins/Array/prototype/some/15.4.4.17-5-1.js Pass -> RuntimeError (interp: Pass)

  • test/built-ins/Object/create/15.2.3.5-4-241.js Pass -> RuntimeError (interp: Pass)

  • test/built-ins/Object/create/15.2.3.5-4-242.js Pass -> RuntimeError (interp: Pass)

  • test/built-ins/Object/create/15.2.3.5-4-243.js Pass -> RuntimeError (interp: Pass)

  • test/built-ins/Object/defineProperties/15.2.3.7-5-b-201.js Pass -> RuntimeError (interp: Pass)

  • test/built-ins/Object/defineProperties/15.2.3.7-5-b-202.js Pass -> RuntimeError (interp: Pass)

  • test/built-ins/Object/defineProperties/15.2.3.7-5-b-203.js Pass -> RuntimeError (interp: Pass)

  • test/built-ins/Object/defineProperty/15.2.3.6-3-215.js Pass -> RuntimeError (interp: Pass)

  • test/built-ins/Object/defineProperty/15.2.3.6-3-216.js Pass -> RuntimeError (interp: Pass)

  • test/built-ins/Object/defineProperty/15.2.3.6-3-217.js Pass -> RuntimeError (interp: Pass)

  • test/language/expressions/call/11.2.3-3_8.js Pass -> Fail (interp: Pass)

  • test/language/expressions/new/S11.2.2_A1.1.js Pass -> Fail (interp: Pass)

  • test/language/expressions/new/S11.2.2_A1.2.js Pass -> Fail (interp: Pass)

The 48 shared regressions (interp baseline already records these failing)

Lower priority — these mirror behavior already accepted in interpreted.txt at #802; regenerating the compiled baseline will fold them in. They cluster in the same Object.* descriptor area, sloppy-this Promise reaction handlers (*-nonstrict/*-sloppy), sparse-array inherited-accessor iteration (Array/prototype/{reduce,reduceRight,...}), and for-in/propertyIsEnumerable DontEnum attribute tests (Number.MAX_VALUE/MIN_VALUE, Boolean/Error/String prototype).

Full list of all 48 shared regressions
  • test/built-ins/Array/prototype/every/15.4.4.16-7-c-i-22.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/filter/15.4.4.20-5-30.js Pass -> RuntimeError (interp: Fail)
  • test/built-ins/Array/prototype/filter/15.4.4.20-9-c-i-22.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/forEach/15.4.4.18-5-25.js Pass -> RuntimeError (interp: Fail)
  • test/built-ins/Array/prototype/forEach/15.4.4.18-7-c-i-22.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/indexOf/15.4.4.14-9-b-i-21.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/lastIndexOf/15.4.4.15-8-b-i-20.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/lastIndexOf/15.4.4.15-8-b-i-21.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/map/15.4.4.19-8-c-i-22.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/reduce/15.4.4.21-8-b-iii-1-18.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/reduce/15.4.4.21-8-b-iii-1-20.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/reduce/15.4.4.21-8-b-iii-1-22.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/reduce/15.4.4.21-8-c-4.js Pass -> RuntimeError (interp: RuntimeError)
  • test/built-ins/Array/prototype/reduce/15.4.4.21-9-c-i-18.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/reduce/15.4.4.21-9-c-i-20.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/reduce/15.4.4.21-9-c-i-22.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/reduceRight/15.4.4.22-8-b-iii-1-22.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/reduceRight/15.4.4.22-8-c-4.js Pass -> RuntimeError (interp: RuntimeError)
  • test/built-ins/Array/prototype/reduceRight/15.4.4.22-9-c-i-18.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/reduceRight/15.4.4.22-9-c-i-20.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/reduceRight/15.4.4.22-9-c-i-22.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Array/prototype/some/15.4.4.17-5-25.js Pass -> RuntimeError (interp: Fail)
  • test/built-ins/Array/prototype/some/15.4.4.17-7-c-i-22.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Boolean/prototype/S15.6.3.1_A4.js Pass -> RuntimeError (interp: RuntimeError)
  • test/built-ins/Error/prototype/S15.11.3.1_A2_T1.js Pass -> RuntimeError (interp: RuntimeError)
  • test/built-ins/Error/the-initial-value-of-errorprototypemessage-is-the-empty-string.js Pass -> RuntimeError (interp: RuntimeError)
  • test/built-ins/Number/MAX_VALUE/S15.7.3.2_A4.js Pass -> RuntimeError (interp: RuntimeError)
  • test/built-ins/Number/MIN_VALUE/S15.7.3.3_A4.js Pass -> RuntimeError (interp: RuntimeError)
  • test/built-ins/Object/defineProperties/15.2.3.7-6-a-102.js Pass -> RuntimeError (interp: Fail)
  • test/built-ins/Object/defineProperties/15.2.3.7-6-a-92.js Pass -> RuntimeError (interp: Fail)
  • test/built-ins/Object/defineProperties/15.2.3.7-6-a-93.js Pass -> RuntimeError (interp: RuntimeError)
  • test/built-ins/Object/defineProperty/15.2.3.6-4-108.js Pass -> RuntimeError (interp: Fail)
  • test/built-ins/Object/defineProperty/15.2.3.6-4-470.js Pass -> RuntimeError (interp: Fail)
  • test/built-ins/Object/preventExtensions/15.2.3.10-3-10.js Pass -> RuntimeError (interp: RuntimeError)
  • test/built-ins/Object/preventExtensions/15.2.3.10-3-18.js Pass -> RuntimeError (interp: Fail)
  • test/built-ins/Object/preventExtensions/15.2.3.10-3-19.js Pass -> RuntimeError (interp: Fail)
  • test/built-ins/Object/preventExtensions/15.2.3.10-3-20.js Pass -> RuntimeError (interp: RuntimeError)
  • test/built-ins/Object/preventExtensions/15.2.3.10-3-8.js Pass -> RuntimeError (interp: Fail)
  • test/built-ins/Object/preventExtensions/15.2.3.10-3-9.js Pass -> RuntimeError (interp: Fail)
  • test/built-ins/Object/prototype/S15.2.3.1_A2.js Pass -> RuntimeError (interp: RuntimeError)
  • test/built-ins/Promise/executor-call-context-sloppy.js Pass -> Fail (interp: RuntimeError)
  • test/built-ins/Promise/prototype/finally/rejected-observable-then-calls-argument.js Pass -> RuntimeError (interp: RuntimeError)
  • test/built-ins/Promise/prototype/then/rxn-handler-fulfilled-invoke-nonstrict.js Pass -> RuntimeError (interp: RuntimeError)
  • test/built-ins/Promise/prototype/then/rxn-handler-rejected-invoke-nonstrict.js Pass -> RuntimeError (interp: RuntimeError)
  • test/built-ins/String/prototype/S15.5.3.1_A2.js Pass -> RuntimeError (interp: RuntimeError)
  • test/language/expressions/new/spread-err-mult-err-expr-throws.js Pass -> Fail (interp: Fail)
  • test/language/expressions/new/spread-err-sngl-err-expr-throws.js Pass -> Fail (interp: Fail)
  • test/language/expressions/property-accessors/S11.2.1_A4_T1.js Pass -> Fail (interp: RuntimeError)

Recommended next steps

  1. Triage/fix the 14 compiled-only regressions — start with the sloppy-mode this → globalThis binding in compiled call/codegen, which spans the Array, call, and (in the shared set) Promise-reaction clusters.
  2. Spot-check the Object.defineProperty descriptor-extraction path (accessor-without-getter on the attributes object) — contiguous test numbers suggest one codegen path.
  3. Then regenerate baselines/compiled.txt (SHARPTS_TEST262_UPDATE_BASELINE=1) so the 108 genuine new passes (incl. parser gains: ParseError 88→10) are recorded and the baseline stops being 187 commits stale.

Provenance

  • Branch: main @ 33f266f9; full compiled run: 11,384 tests in 1007.5 s → Pass=7942 Fail=1843 ParseError=10 RuntimeError=690 Timeout=2 HarnessError=18 Skipped=879.
  • Subset config: config/subset.json (timeout 15 s). Env: win32, .NET 10, 12 logical cores.
  • Regressions independently re-verified deterministic (882-path targeted re-run, recycle-safe).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions