From e7bba3cb1cf0cbc60f43a45f26146508970333d4 Mon Sep 17 00:00:00 2001 From: Nick Nassiri Date: Mon, 22 Jun 2026 12:19:38 -0700 Subject: [PATCH] =?UTF-8?q?Docs:=20clarify=20benchmark=20suites=20?= =?UTF-8?q?=E2=80=94=20rename=20SharpTS.Benchmarks=20=E2=86=92=20SharpTS.M?= =?UTF-8?q?icrobenchmarks,=20add=20READMEs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The repo has two benchmark projects with near-identical names but distinct purposes, which made them easy to conflate: - benchmarks/ (PowerShell + shared bench.ts) — external/competitive timing vs Node.js and Bun. Goal: meet or exceed Node. - SharpTS.Benchmarks/ (BenchmarkDotNet, in-process) — internal/headroom timing vs the idiomatic-C# ceiling and the object?/boxing dynamic-typing tax, with allocation profiling. Backs the compiler perf work. Rename SharpTS.Benchmarks → SharpTS.Microbenchmarks to remove the name collision, pin RootNamespace/AssemblyName so the by-string embedded-resource names stay stable, and document the two-axis split: - benchmarks/README.md, SharpTS.Microbenchmarks/README.md - new 'Benchmarking' section in CLAUDE.md (parallel to Conformance Suites) The shared algorithms.ts stays byte-identical between the suites (embedded as SharpTS.Microbenchmarks.algorithms.ts). Verified: Release build clean, all 5 manifest resource names resolve, solution parses, no remaining references. --- CLAUDE.md | 9 ++ .../Baselines/EquivalentCSharp.cs | 2 +- .../Baselines/IdiomaticCSharp.cs | 2 +- .../Benchmarks/ArrayHelpersBenchmarks.cs | 6 +- .../Benchmarks/ComputationalBenchmarks.cs | 8 +- .../Benchmarks/ObjectLiteralsBenchmarks.cs | 6 +- .../Benchmarks/PropertyAccessBenchmarks.cs | 6 +- .../Benchmarks/RegexBenchmarks.cs | 6 +- .../Benchmarks/StarterWorkloadBenchmarks.cs | 4 +- .../Infrastructure/BenchmarkHarness.cs | 2 +- .../Infrastructure/CompilationCache.cs | 2 +- .../Program.cs | 2 +- SharpTS.Microbenchmarks/README.md | 80 +++++++++++++++++ .../SharpTS.Microbenchmarks.csproj | 8 +- .../TypeScriptSources/ArrayHelpers.ts | 0 .../TypeScriptSources/ObjectLiterals.ts | 0 .../TypeScriptSources/PropertyAccess.ts | 0 .../TypeScriptSources/Regex.ts | 0 SharpTS.csproj | 6 +- SharpTS.sln | 4 +- benchmarks/README.md | 88 +++++++++++++++++++ benchmarks/scripts/lib/algorithms.ts | 2 +- docs/plans/regexp-102-scope.md | 2 +- 23 files changed, 214 insertions(+), 31 deletions(-) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/Baselines/EquivalentCSharp.cs (99%) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/Baselines/IdiomaticCSharp.cs (98%) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/Benchmarks/ArrayHelpersBenchmarks.cs (96%) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/Benchmarks/ComputationalBenchmarks.cs (94%) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/Benchmarks/ObjectLiteralsBenchmarks.cs (92%) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/Benchmarks/PropertyAccessBenchmarks.cs (94%) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/Benchmarks/RegexBenchmarks.cs (93%) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/Benchmarks/StarterWorkloadBenchmarks.cs (96%) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/Infrastructure/BenchmarkHarness.cs (99%) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/Infrastructure/CompilationCache.cs (95%) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/Program.cs (96%) create mode 100644 SharpTS.Microbenchmarks/README.md rename SharpTS.Benchmarks/SharpTS.Benchmarks.csproj => SharpTS.Microbenchmarks/SharpTS.Microbenchmarks.csproj (76%) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/TypeScriptSources/ArrayHelpers.ts (100%) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/TypeScriptSources/ObjectLiterals.ts (100%) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/TypeScriptSources/PropertyAccess.ts (100%) rename {SharpTS.Benchmarks => SharpTS.Microbenchmarks}/TypeScriptSources/Regex.ts (100%) create mode 100644 benchmarks/README.md diff --git a/CLAUDE.md b/CLAUDE.md index c686f34e..ffc8ad70 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -159,6 +159,15 @@ Two standalone test projects pin SharpTS against external corpora. Neither is in - **`SharpTS.Test262/`** — TC39 ECMA-262 (interpreter + compiled). Update baseline: `SHARPTS_TEST262_UPDATE_BASELINE=1`. - **`SharpTS.TypeScriptConformance/`** — `microsoft/TypeScript` conformance (type-checker only). Update baseline: `SHARPTS_TSCONFORMANCE_UPDATE_BASELINE=1`. Bucket model: Pass/Fail/ParseError/TypeCheckError/Skipped/HarnessError. Match strategy: `(line, tsCode)` tuples — `tsCode` is the canonical `TSnnnn` code carried on every type-checker `Diagnostic` (see `Diagnostics/Diagnostic.cs`). See `SharpTS.TypeScriptConformance/README.md`. +## Benchmarking + +Two **complementary** benchmark suites measure different things — they are not redundant and cannot be merged (one drives external runtime executables; the other must run in-process against managed code). Pick by the question you're answering. + +- **`benchmarks/`** — *external / competitive.* "Are we as fast as Node/Bun?" PowerShell harness (`run-benchmarks.ps1`) runs `scripts/*.ts` whole-program across the SharpTS interpreter, SharpTS compiled, Node.js, and Bun via a shared `scripts/lib/bench.ts`. Wired into CI (`.github/workflows/benchmarks.yml`, `workflow_dispatch` only). Goal: **meet or exceed Node.** See `benchmarks/README.md`. +- **`SharpTS.Microbenchmarks/`** — *internal / headroom.* "How close are we to the C# ceiling, and where's the overhead?" BenchmarkDotNet project in `SharpTS.sln`; compiles TS in-process and compares against idiomatic C# (native-type ceiling) and "equivalent" C# (`object?`/boxing tax), with `MemoryDiagnoser` allocation profiling. This is the harness behind compiler perf work. See `SharpTS.Microbenchmarks/README.md`. + +`benchmarks/scripts/lib/algorithms.ts` is **shared byte-identical** between the two (embedded into the microbenchmark assembly as `SharpTS.Microbenchmarks.algorithms.ts`) so both measure the same source. Embedded-resource names are referenced by string and `RootNamespace`/`AssemblyName` are pinned in the `.csproj` — a wrong name compiles but throws at `[GlobalSetup]`. + ## See Also - `STATUS.md` - Feature implementation status and known bugs diff --git a/SharpTS.Benchmarks/Baselines/EquivalentCSharp.cs b/SharpTS.Microbenchmarks/Baselines/EquivalentCSharp.cs similarity index 99% rename from SharpTS.Benchmarks/Baselines/EquivalentCSharp.cs rename to SharpTS.Microbenchmarks/Baselines/EquivalentCSharp.cs index c5c36e0d..0382f079 100644 --- a/SharpTS.Benchmarks/Baselines/EquivalentCSharp.cs +++ b/SharpTS.Microbenchmarks/Baselines/EquivalentCSharp.cs @@ -1,4 +1,4 @@ -namespace SharpTS.Benchmarks.Baselines; +namespace SharpTS.Microbenchmarks.Baselines; /// /// C# implementations using object/dynamic types to simulate SharpTS runtime overhead. diff --git a/SharpTS.Benchmarks/Baselines/IdiomaticCSharp.cs b/SharpTS.Microbenchmarks/Baselines/IdiomaticCSharp.cs similarity index 98% rename from SharpTS.Benchmarks/Baselines/IdiomaticCSharp.cs rename to SharpTS.Microbenchmarks/Baselines/IdiomaticCSharp.cs index e9e25990..20deec5c 100644 --- a/SharpTS.Benchmarks/Baselines/IdiomaticCSharp.cs +++ b/SharpTS.Microbenchmarks/Baselines/IdiomaticCSharp.cs @@ -1,4 +1,4 @@ -namespace SharpTS.Benchmarks.Baselines; +namespace SharpTS.Microbenchmarks.Baselines; /// /// Idiomatic C# implementations using native types (int, long, bool[], etc.). diff --git a/SharpTS.Benchmarks/Benchmarks/ArrayHelpersBenchmarks.cs b/SharpTS.Microbenchmarks/Benchmarks/ArrayHelpersBenchmarks.cs similarity index 96% rename from SharpTS.Benchmarks/Benchmarks/ArrayHelpersBenchmarks.cs rename to SharpTS.Microbenchmarks/Benchmarks/ArrayHelpersBenchmarks.cs index 45d952c7..0cf49f32 100644 --- a/SharpTS.Benchmarks/Benchmarks/ArrayHelpersBenchmarks.cs +++ b/SharpTS.Microbenchmarks/Benchmarks/ArrayHelpersBenchmarks.cs @@ -1,9 +1,9 @@ using System.Reflection; using BenchmarkDotNet.Attributes; using BenchmarkDotNet.Order; -using SharpTS.Benchmarks.Infrastructure; +using SharpTS.Microbenchmarks.Infrastructure; -namespace SharpTS.Benchmarks.Benchmarks; +namespace SharpTS.Microbenchmarks.Benchmarks; /// /// Iterator-helper benchmarks for the issue #90 hot path. Measures the @@ -43,7 +43,7 @@ public void Setup() { var assembly = typeof(ArrayHelpersBenchmarks).Assembly; using var stream = assembly.GetManifestResourceStream( - "SharpTS.Benchmarks.TypeScriptSources.ArrayHelpers.ts") + "SharpTS.Microbenchmarks.TypeScriptSources.ArrayHelpers.ts") ?? throw new InvalidOperationException("Could not find embedded resource ArrayHelpers.ts"); using var reader = new StreamReader(stream); var tsSource = reader.ReadToEnd(); diff --git a/SharpTS.Benchmarks/Benchmarks/ComputationalBenchmarks.cs b/SharpTS.Microbenchmarks/Benchmarks/ComputationalBenchmarks.cs similarity index 94% rename from SharpTS.Benchmarks/Benchmarks/ComputationalBenchmarks.cs rename to SharpTS.Microbenchmarks/Benchmarks/ComputationalBenchmarks.cs index c2b5050e..c3ececad 100644 --- a/SharpTS.Benchmarks/Benchmarks/ComputationalBenchmarks.cs +++ b/SharpTS.Microbenchmarks/Benchmarks/ComputationalBenchmarks.cs @@ -1,10 +1,10 @@ using System.Reflection; using BenchmarkDotNet.Attributes; using BenchmarkDotNet.Order; -using SharpTS.Benchmarks.Baselines; -using SharpTS.Benchmarks.Infrastructure; +using SharpTS.Microbenchmarks.Baselines; +using SharpTS.Microbenchmarks.Infrastructure; -namespace SharpTS.Benchmarks.Benchmarks; +namespace SharpTS.Microbenchmarks.Benchmarks; // Computational algorithm benchmarks: SharpTS-compiled TypeScript vs idiomatic // C# (native types — the performance ceiling) vs "equivalent" C# (object?/boxing @@ -23,7 +23,7 @@ namespace SharpTS.Benchmarks.Benchmarks; /// Shared embedded-resource loading + compiled-delegate resolution. public abstract class ComputationalBenchmarkBase { - private const string ResourceName = "SharpTS.Benchmarks.algorithms.ts"; + private const string ResourceName = "SharpTS.Microbenchmarks.algorithms.ts"; protected static Func LoadCompiled(string functionName) { diff --git a/SharpTS.Benchmarks/Benchmarks/ObjectLiteralsBenchmarks.cs b/SharpTS.Microbenchmarks/Benchmarks/ObjectLiteralsBenchmarks.cs similarity index 92% rename from SharpTS.Benchmarks/Benchmarks/ObjectLiteralsBenchmarks.cs rename to SharpTS.Microbenchmarks/Benchmarks/ObjectLiteralsBenchmarks.cs index 6ef776e6..f35cc455 100644 --- a/SharpTS.Benchmarks/Benchmarks/ObjectLiteralsBenchmarks.cs +++ b/SharpTS.Microbenchmarks/Benchmarks/ObjectLiteralsBenchmarks.cs @@ -1,9 +1,9 @@ using System.Reflection; using BenchmarkDotNet.Attributes; using BenchmarkDotNet.Order; -using SharpTS.Benchmarks.Infrastructure; +using SharpTS.Microbenchmarks.Infrastructure; -namespace SharpTS.Benchmarks.Benchmarks; +namespace SharpTS.Microbenchmarks.Benchmarks; /// /// Object-literal allocation benchmarks. Each { ... } in TS source @@ -29,7 +29,7 @@ public void Setup() { var assembly = typeof(ObjectLiteralsBenchmarks).Assembly; using var stream = assembly.GetManifestResourceStream( - "SharpTS.Benchmarks.TypeScriptSources.ObjectLiterals.ts") + "SharpTS.Microbenchmarks.TypeScriptSources.ObjectLiterals.ts") ?? throw new InvalidOperationException("Could not find embedded resource ObjectLiterals.ts"); using var reader = new StreamReader(stream); var tsSource = reader.ReadToEnd(); diff --git a/SharpTS.Benchmarks/Benchmarks/PropertyAccessBenchmarks.cs b/SharpTS.Microbenchmarks/Benchmarks/PropertyAccessBenchmarks.cs similarity index 94% rename from SharpTS.Benchmarks/Benchmarks/PropertyAccessBenchmarks.cs rename to SharpTS.Microbenchmarks/Benchmarks/PropertyAccessBenchmarks.cs index f1b794d7..82cd4c20 100644 --- a/SharpTS.Benchmarks/Benchmarks/PropertyAccessBenchmarks.cs +++ b/SharpTS.Microbenchmarks/Benchmarks/PropertyAccessBenchmarks.cs @@ -1,9 +1,9 @@ using System.Reflection; using BenchmarkDotNet.Attributes; using BenchmarkDotNet.Order; -using SharpTS.Benchmarks.Infrastructure; +using SharpTS.Microbenchmarks.Infrastructure; -namespace SharpTS.Benchmarks.Benchmarks; +namespace SharpTS.Microbenchmarks.Benchmarks; /// /// Property-access benchmarks. Measures cost of obj.foo lookup in @@ -31,7 +31,7 @@ public void Setup() { var assembly = typeof(PropertyAccessBenchmarks).Assembly; using var stream = assembly.GetManifestResourceStream( - "SharpTS.Benchmarks.TypeScriptSources.PropertyAccess.ts") + "SharpTS.Microbenchmarks.TypeScriptSources.PropertyAccess.ts") ?? throw new InvalidOperationException("Could not find embedded resource PropertyAccess.ts"); using var reader = new StreamReader(stream); var tsSource = reader.ReadToEnd(); diff --git a/SharpTS.Benchmarks/Benchmarks/RegexBenchmarks.cs b/SharpTS.Microbenchmarks/Benchmarks/RegexBenchmarks.cs similarity index 93% rename from SharpTS.Benchmarks/Benchmarks/RegexBenchmarks.cs rename to SharpTS.Microbenchmarks/Benchmarks/RegexBenchmarks.cs index 582d2573..314bd0ed 100644 --- a/SharpTS.Benchmarks/Benchmarks/RegexBenchmarks.cs +++ b/SharpTS.Microbenchmarks/Benchmarks/RegexBenchmarks.cs @@ -1,9 +1,9 @@ using System.Reflection; using BenchmarkDotNet.Attributes; using BenchmarkDotNet.Order; -using SharpTS.Benchmarks.Infrastructure; +using SharpTS.Microbenchmarks.Infrastructure; -namespace SharpTS.Benchmarks.Benchmarks; +namespace SharpTS.Microbenchmarks.Benchmarks; /// /// Measures regex literal compilation overhead. Each TS regex literal @@ -35,7 +35,7 @@ public void Setup() { var assembly = typeof(RegexBenchmarks).Assembly; using var stream = assembly.GetManifestResourceStream( - "SharpTS.Benchmarks.TypeScriptSources.Regex.ts") + "SharpTS.Microbenchmarks.TypeScriptSources.Regex.ts") ?? throw new InvalidOperationException("Could not find embedded resource Regex.ts"); using var reader = new StreamReader(stream); var tsSource = reader.ReadToEnd(); diff --git a/SharpTS.Benchmarks/Benchmarks/StarterWorkloadBenchmarks.cs b/SharpTS.Microbenchmarks/Benchmarks/StarterWorkloadBenchmarks.cs similarity index 96% rename from SharpTS.Benchmarks/Benchmarks/StarterWorkloadBenchmarks.cs rename to SharpTS.Microbenchmarks/Benchmarks/StarterWorkloadBenchmarks.cs index 55c07dba..181c61d5 100644 --- a/SharpTS.Benchmarks/Benchmarks/StarterWorkloadBenchmarks.cs +++ b/SharpTS.Microbenchmarks/Benchmarks/StarterWorkloadBenchmarks.cs @@ -1,8 +1,8 @@ using BenchmarkDotNet.Attributes; using BenchmarkDotNet.Order; -using SharpTS.Benchmarks.Baselines; +using SharpTS.Microbenchmarks.Baselines; -namespace SharpTS.Benchmarks.Benchmarks; +namespace SharpTS.Microbenchmarks.Benchmarks; // Starter-set workloads that broaden coverage beyond the original arithmetic // kernels: a builtin-heavy JSON round-trip, a data-parallel typed-array kernel, diff --git a/SharpTS.Benchmarks/Infrastructure/BenchmarkHarness.cs b/SharpTS.Microbenchmarks/Infrastructure/BenchmarkHarness.cs similarity index 99% rename from SharpTS.Benchmarks/Infrastructure/BenchmarkHarness.cs rename to SharpTS.Microbenchmarks/Infrastructure/BenchmarkHarness.cs index a961316b..b8ecbbfd 100644 --- a/SharpTS.Benchmarks/Infrastructure/BenchmarkHarness.cs +++ b/SharpTS.Microbenchmarks/Infrastructure/BenchmarkHarness.cs @@ -3,7 +3,7 @@ using SharpTS.Parsing; using SharpTS.TypeSystem; -namespace SharpTS.Benchmarks.Infrastructure; +namespace SharpTS.Microbenchmarks.Infrastructure; /// /// Harness for compiling TypeScript to .NET assemblies and invoking compiled methods diff --git a/SharpTS.Benchmarks/Infrastructure/CompilationCache.cs b/SharpTS.Microbenchmarks/Infrastructure/CompilationCache.cs similarity index 95% rename from SharpTS.Benchmarks/Infrastructure/CompilationCache.cs rename to SharpTS.Microbenchmarks/Infrastructure/CompilationCache.cs index bc70f4cb..cac199f2 100644 --- a/SharpTS.Benchmarks/Infrastructure/CompilationCache.cs +++ b/SharpTS.Microbenchmarks/Infrastructure/CompilationCache.cs @@ -1,6 +1,6 @@ using System.Collections.Concurrent; -namespace SharpTS.Benchmarks.Infrastructure; +namespace SharpTS.Microbenchmarks.Infrastructure; /// /// Manages pre-compilation of TypeScript sources during benchmark initialization. diff --git a/SharpTS.Benchmarks/Program.cs b/SharpTS.Microbenchmarks/Program.cs similarity index 96% rename from SharpTS.Benchmarks/Program.cs rename to SharpTS.Microbenchmarks/Program.cs index 9b678af7..20411888 100644 --- a/SharpTS.Benchmarks/Program.cs +++ b/SharpTS.Microbenchmarks/Program.cs @@ -5,7 +5,7 @@ using BenchmarkDotNet.Columns; using BenchmarkDotNet.Order; -namespace SharpTS.Benchmarks; +namespace SharpTS.Microbenchmarks; /// /// Entry point for SharpTS benchmark suite. diff --git a/SharpTS.Microbenchmarks/README.md b/SharpTS.Microbenchmarks/README.md new file mode 100644 index 00000000..cbea585f --- /dev/null +++ b/SharpTS.Microbenchmarks/README.md @@ -0,0 +1,80 @@ +# Microbenchmarks (`SharpTS.Microbenchmarks/`) + +**Internal / headroom benchmarks.** This [BenchmarkDotNet](https://benchmarkdotnet.org/) +suite answers a different question from the cross-runtime suite: *how close is +SharpTS-compiled TypeScript to the performance ceiling, and where does the +overhead go?* It is the harness behind the compiler perf work (object-shape +structs, typed-array fast paths, the merge-sort `Array.sort`, etc.). + +Each workload is measured three ways: + +- **SharpTS** — TypeScript compiled to IL by `ILCompiler`, invoked in-process. +- **Idiomatic** — hand-written C# using native types. The **performance ceiling**. +- **Equivalent** — C# written with `object?`/boxing to approximate the + **dynamic-typing tax** SharpTS pays for JS semantics. + +It is part of `SharpTS.sln` and runs **in-process**: the TypeScript is compiled +once to a DLL at `[GlobalSetup]`, then reached through a cached strongly-typed +`Func` delegate so reflection and argument boxing stay **out of +the timed region**. + +> For the *external* comparison against Node.js and Bun, see +> [`../benchmarks`](../benchmarks). That suite has a table explaining why the two +> are kept separate ("Why two benchmark suites?"). + +## Layout + +| Path | Purpose | +|------|---------| +| `Program.cs` | BenchmarkDotNet entry point (GitHub-Markdown + HTML exporters, `MemoryDiagnoser`, rank/ops-per-sec columns). | +| `Benchmarks/*.cs` | One file per workload family (computational, starter workloads, array helpers, property access, object literals, regex). | +| `Baselines/IdiomaticCSharp.cs` | Native-type C# baselines — the ceiling. | +| `Baselines/EquivalentCSharp.cs` | `object?`/boxing C# baselines — the dynamic-typing tax. | +| `Infrastructure/BenchmarkHarness.cs` | Compile TS → DLL, load it, resolve compiled methods/delegates. | +| `Infrastructure/CompilationCache.cs` | Compile each TS source once, reuse across benchmark classes. | +| `TypeScriptSources/*.ts` | TS bodies for the non-computational workloads (embedded as resources). | + +The computational/starter workloads load their TS from +`../benchmarks/scripts/lib/algorithms.ts`, embedded as the resource +`SharpTS.Microbenchmarks.algorithms.ts`. That file is **shared byte-identical** +with the cross-runtime shell harness, so both suites measure the same source. + +## Running + +```bash +# From the repo root. BenchmarkDotNet requires a Release build. +dotnet run -c Release --project SharpTS.Microbenchmarks + +# Interactive picker, or filter to a subset: +dotnet run -c Release --project SharpTS.Microbenchmarks -- --filter '*Fibonacci*' +dotnet run -c Release --project SharpTS.Microbenchmarks -- --list flat +``` + +Results (Markdown + HTML, plus allocation columns) are written under +`BenchmarkDotNet.Artifacts/`. + +## Conventions + +- **One algorithm per class**, each with a single `[Params]` axis — a single + class with multiple independent `[Params]` would run BenchmarkDotNet's full + Cartesian product and waste ~Nx of the work. +- Compiled functions are reached via `ComputationalBenchmarkBase.LoadCompiled`, + which returns a cached `Func` — keep reflection/boxing outside + `[Benchmark]` methods so the measurement reflects the generated IL, not the + invocation plumbing. +- Embedded-resource names are referenced **by string** (e.g. + `"SharpTS.Microbenchmarks.TypeScriptSources.Regex.ts"`). `RootNamespace`/ + `AssemblyName` are pinned in the `.csproj` so those names stay stable; if you + rename the project, keep the strings and the pinned names in sync. + +## Embedded-resource gotcha + +A wrong resource name **compiles fine** but throws at `[GlobalSetup]` +(`GetManifestResourceStream` returns null). After adding a `.ts` source or +renaming anything, verify the manifest names resolve: + +```powershell +$asm = [System.Reflection.Assembly]::LoadFrom( + (Get-ChildItem -Recurse SharpTS.Microbenchmarks/bin -Filter SharpTS.Microbenchmarks.dll)[0].FullName) +$asm.GetManifestResourceNames() +``` diff --git a/SharpTS.Benchmarks/SharpTS.Benchmarks.csproj b/SharpTS.Microbenchmarks/SharpTS.Microbenchmarks.csproj similarity index 76% rename from SharpTS.Benchmarks/SharpTS.Benchmarks.csproj rename to SharpTS.Microbenchmarks/SharpTS.Microbenchmarks.csproj index 9cd0468d..b2b3c5cb 100644 --- a/SharpTS.Benchmarks/SharpTS.Benchmarks.csproj +++ b/SharpTS.Microbenchmarks/SharpTS.Microbenchmarks.csproj @@ -7,6 +7,12 @@ enable false + + SharpTS.Microbenchmarks + SharpTS.Microbenchmarks + true portable @@ -28,7 +34,7 @@ - + diff --git a/SharpTS.Benchmarks/TypeScriptSources/ArrayHelpers.ts b/SharpTS.Microbenchmarks/TypeScriptSources/ArrayHelpers.ts similarity index 100% rename from SharpTS.Benchmarks/TypeScriptSources/ArrayHelpers.ts rename to SharpTS.Microbenchmarks/TypeScriptSources/ArrayHelpers.ts diff --git a/SharpTS.Benchmarks/TypeScriptSources/ObjectLiterals.ts b/SharpTS.Microbenchmarks/TypeScriptSources/ObjectLiterals.ts similarity index 100% rename from SharpTS.Benchmarks/TypeScriptSources/ObjectLiterals.ts rename to SharpTS.Microbenchmarks/TypeScriptSources/ObjectLiterals.ts diff --git a/SharpTS.Benchmarks/TypeScriptSources/PropertyAccess.ts b/SharpTS.Microbenchmarks/TypeScriptSources/PropertyAccess.ts similarity index 100% rename from SharpTS.Benchmarks/TypeScriptSources/PropertyAccess.ts rename to SharpTS.Microbenchmarks/TypeScriptSources/PropertyAccess.ts diff --git a/SharpTS.Benchmarks/TypeScriptSources/Regex.ts b/SharpTS.Microbenchmarks/TypeScriptSources/Regex.ts similarity index 100% rename from SharpTS.Benchmarks/TypeScriptSources/Regex.ts rename to SharpTS.Microbenchmarks/TypeScriptSources/Regex.ts diff --git a/SharpTS.csproj b/SharpTS.csproj index 7a998571..4074d2c9 100644 --- a/SharpTS.csproj +++ b/SharpTS.csproj @@ -1,4 +1,4 @@ - + Exe @@ -57,8 +57,8 @@ - - + + diff --git a/SharpTS.sln b/SharpTS.sln index 6616e0e3..7a5f1bc1 100644 --- a/SharpTS.sln +++ b/SharpTS.sln @@ -1,4 +1,4 @@ - + Microsoft Visual Studio Solution File, Format Version 12.00 # Visual Studio Version 18 VisualStudioVersion = 18.3.11312.210 @@ -9,7 +9,7 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SharpTS.Tests", "SharpTS.Te EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SharpTS.Example.Interop", "Examples\Interop\SharpTS.Example.Interop.csproj", "{189D6325-E224-0097-E8A5-0FC6C44652A0}" EndProject -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SharpTS.Benchmarks", "SharpTS.Benchmarks\SharpTS.Benchmarks.csproj", "{27E0C313-AB29-43BC-AA2C-E52F13E634AE}" +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SharpTS.Microbenchmarks", "SharpTS.Microbenchmarks\SharpTS.Microbenchmarks.csproj", "{27E0C313-AB29-43BC-AA2C-E52F13E634AE}" EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SharpTS.Sdk", "SharpTS.Sdk\SharpTS.Sdk.csproj", "{35E3B550-4DE0-4D5F-ADC3-C80DF7092F17}" EndProject diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 00000000..06acccce --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,88 @@ +# Cross-Runtime Benchmarks (`benchmarks/`) + +**External / competitive benchmarks.** This suite answers one question: *how does +SharpTS compare to other JavaScript/TypeScript runtimes?* The bar is to **meet or +exceed Node.js**. + +It runs the same TypeScript workloads across four runtimes and prints a +side-by-side table: + +- SharpTS **interpreter** (`dotnet run -- script.ts`) +- SharpTS **compiled** (`dotnet run -- --compile script.ts` → run the DLL) +- **Node.js** (with `--experimental-strip-types` on Node < 23) +- **Bun** (if installed) + +Measurement is **whole-program, process-level** (includes startup + the full +pipeline), so it reflects what a user actually experiences invoking each runtime. + +> For the *internal* benchmarks — SharpTS-compiled vs the idiomatic-C# +> performance ceiling, with allocation/GC profiling — see +> [`../SharpTS.Microbenchmarks`](../SharpTS.Microbenchmarks). The two suites are +> complementary and measure different things; see "Why two suites?" below. + +## Layout + +| Path | Purpose | +|------|---------| +| `run-benchmarks.ps1` | Builds SharpTS (Release), runs every `scripts/*.ts` on all runtimes, writes `results.txt`. | +| `format-results.ps1` | Renders `results.txt` as a Markdown table (used for the CI job summary). | +| `scripts/*.ts` | One workload per file (fibonacci, sort, json, regex, closures, …). | +| `scripts/lib/bench.ts` | Shared cross-runtime timing harness (auto-batching, warmup, mean/min/stdev). | +| `scripts/lib/algorithms.ts` | Algorithm bodies **shared byte-identical** with the microbenchmark suite (embedded there as a resource). | + +## Running + +```powershell +# Run everything; results land in $TEMP/bench-results/results.txt +./benchmarks/run-benchmarks.ps1 + +# Render the table from a results file +./benchmarks/format-results.ps1 -ResultsFile $env:TEMP/bench-results/results.txt +``` + +Override the output directory with `$env:OUTPUT_DIR`. Node and Bun are detected +automatically; Bun is skipped if not on `PATH`. + +## How timing works + +Each workload calls `bench(name, param, fn)` from `scripts/lib/bench.ts`, which: + +1. Probes once; if a single call is already ≥ 1 ms it samples one call at a time + (honest for slow cases like the tree-walking interpreter on big inputs). +2. Otherwise warms the JIT, calibrates an inner batch until a sample spans ≥ 1 ms + (lifting fast cases above the timer noise floor), then samples to a budget. +3. Emits one line per case, consumed by `format-results.ps1`: + + ``` + BENCH::::: + ``` + +`performance.now()` (sub-microsecond, monotonic) is used everywhere so the +methodology is identical across runtimes. A `guard` accumulator defeats +dead-code elimination in both SharpTS modes and the JS engines. + +If a runtime produces no `BENCH:` line (crash, parse error, missing API), +`run-benchmarks.ps1` warns loudly and echoes the tail of its output rather than +silently leaving a blank cell. + +## CI + +`.github/workflows/benchmarks.yml` runs this suite on `workflow_dispatch` and +publishes the formatted table to the job summary, with the raw `results.txt` +uploaded as an artifact. It is **not** run on every push (timing on shared CI +runners is noisy and the full sweep is slow). + +## Why two benchmark suites? + +| | `benchmarks/` (this suite) | `SharpTS.Microbenchmarks/` | +|---|---|---| +| **Question** | Are we as fast as Node/Bun? | How close are we to the C# ceiling, and where's the overhead? | +| **Compares against** | Node.js, Bun | Idiomatic C# (native types) + "equivalent" C# (`object?`/boxing) | +| **Tool** | PowerShell + shared `bench.ts` | BenchmarkDotNet | +| **Scope** | Whole-program, process-level | In-process, per-function (delegate-invoked) | +| **Profiling** | Wall-clock mean/min/stdev | + allocations/GC (`MemoryDiagnoser`) | + +They can't be merged: BenchmarkDotNet must run in-process against managed code +(it can't drive the `node`/`bun` executables), and the cross-runtime comparison +must be black-box at the process boundary. Keeping them separate is intentional; +the shared `scripts/lib/algorithms.ts` ensures both measure identical source. diff --git a/benchmarks/scripts/lib/algorithms.ts b/benchmarks/scripts/lib/algorithms.ts index 4210f43e..732b2886 100644 --- a/benchmarks/scripts/lib/algorithms.ts +++ b/benchmarks/scripts/lib/algorithms.ts @@ -4,7 +4,7 @@ // * the cross-runtime shell harness (benchmarks/run-benchmarks.ps1) imports // these into its driver scripts and runs them under the SharpTS // interpreter, the SharpTS compiler, Node.js, and Bun; -// * the BenchmarkDotNet suite (SharpTS.Benchmarks) embeds this file as a +// * the BenchmarkDotNet suite (SharpTS.Microbenchmarks) embeds this file as a // resource, compiles it, and invokes the functions via reflection. // // Keep every export a plain `number -> number` function (no top-level diff --git a/docs/plans/regexp-102-scope.md b/docs/plans/regexp-102-scope.md index 945c38ce..3d1837b8 100644 --- a/docs/plans/regexp-102-scope.md +++ b/docs/plans/regexp-102-scope.md @@ -422,6 +422,6 @@ gaps (real features to implement), not compiled divergences. None of the above threatens the existing regex perf work: both runtimes already delegate matching to native `System.Text.RegularExpressions.Regex` and share a process-lifetime compile cache keyed by `(pattern, options)`. The one remaining native-Regex opportunity is `RegexOptions.Compiled` on the -cached engines (one-time JIT cost amortized by the cache; A/B with `SharpTS.Benchmarks/RegexBenchmarks`). +cached engines (one-time JIT cost amortized by the cache; A/B with `SharpTS.Microbenchmarks/RegexBenchmarks`). Only cluster 3 touches a hot path, and its fast-path-preserving design keeps the common `r.exec(s)` / `r.test(s)` literal call site on typed fields.