Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .benchmark-history/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Benchmark History

This directory stores the benchmark history in `history.json`. The file is
gitignored.

## Format

See `history.example.json` for the format. Each entry contains:

- Timestamp of the benchmark run
- System information (Deno, V8, TypeScript versions)
- Hardware details (OS, architecture, CPU cores)
- Package versions for each workspace
- Benchmark results for each function

## Usage

History is automatically managed by the benchmark runner:

```bash
deno task bench
```

The history file keeps the last 100 benchmark runs.
38 changes: 38 additions & 0 deletions .benchmark-history/history.example.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
[
{
"timestamp": "2024-01-15T10:00:00.000Z",
"denoVersion": "2.5.6",
"v8Version": "14.0.365.5",
"typescriptVersion": "5.9.2",
"os": "linux",
"arch": "x86_64",
"cpuModel": "Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz",
"cpuCores": 4,
"memoryTotal": 0,
"packages": {
"array": "5.0.0",
"date": "5.0.0",
"function": "5.0.0",
"math": "5.0.0",
"object": "5.0.0",
"text": "5.0.0",
"type": "5.0.0"
},
"benchmarks": {
"text/capitalize": [
{
"name": "capitalize - short string",
"measuredRunsAvgMs": 0.0000447,
"measuredRunsMs": [],
"totalMs": 50.2,
"iterationsPerSec": 22371364,
"min": 0.0000446,
"max": 0.0001284,
"p75": 0.0000472,
"p99": 0.0000583,
"p995": 0.0000899
}
]
}
}
]
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
coverage
docs
.DS_Store
.benchmark-history/history.json
219 changes: 219 additions & 0 deletions BENCHMARK.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,219 @@
# Benchmarking System

This repository includes a comprehensive benchmarking system to track and
monitor the performance of all utility functions across workspaces.

## Overview

The benchmark system consists of:

1. **Benchmark files** (`.bench.ts`) - Individual benchmark files for each
function
2. **Benchmark runner** - Automated tool to run all benchmarks and collect
results
3. **Benchmark history** - JSON file tracking performance metrics over time
4. **Generator tool** - Script to generate benchmark file templates

## Quick Start

### Running Benchmarks

To run all benchmarks and save results to history:

```bash
deno task bench
```

Or using the full command:

```bash
deno task bench:run
```

### Generating Benchmark Templates

If you've added new functions and need benchmark templates:

```bash
deno task bench:generate
```

This will create `.bench.ts` files for any functions that don't have benchmarks
yet.

## Benchmark File Structure

Each benchmark file follows this pattern:

```typescript
/**
* Benchmark for functionName function
*/

import { functionName } from './function-name.ts'

// Test data
const testData = {/* ... */}

Deno.bench('functionName - basic case', () => {
functionName(testData)
})

Deno.bench('functionName - edge case', () => {
functionName(edgeCaseData)
})
```

### Best Practices for Writing Benchmarks

1. **Use realistic data** - Test with data that represents real-world usage
2. **Test different scales** - Small, medium, and large datasets
3. **Cover edge cases** - Empty inputs, extreme values, etc.
4. **Be descriptive** - Use clear benchmark names that explain what's being
tested
5. **Avoid side effects** - Benchmarks should be pure and repeatable

## Benchmark History

The benchmark history is stored in `.benchmark-history/history.json`
(gitignored).

### History Entry Structure

Each benchmark run creates a history entry with:

```json
{
"timestamp": "2024-01-01T12:00:00.000Z",
"denoVersion": "2.5.6",
"v8Version": "14.0.365.5",
"typescriptVersion": "5.9.2",
"os": "linux",
"arch": "x86_64",
"cpuModel": "N/A",
"cpuCores": 4,
"memoryTotal": 0,
"packages": {
"array": "5.0.0",
"date": "5.0.0",
...
},
"benchmarks": {
"array/group-by": [
{
"name": "groupBy - small dataset (5 items)",
"measuredRunsAvgMs": 0.0023,
"iterationsPerSec": 434782,
"min": 0.0020,
"max": 0.0030,
"p75": 0.0024,
"p99": 0.0028,
"p995": 0.0029
}
]
}
}
```

### Metrics Tracked

- **measuredRunsAvgMs**: Average time per operation in milliseconds
- **iterationsPerSec**: Number of operations per second
- **min/max**: Minimum and maximum execution times
- **p75/p99/p995**: Percentile metrics for consistency analysis

## Workspaces

Benchmarks are organized by workspace:

- `array/` - Array manipulation functions
- `date/` - Date utilities
- `function/` - Function utilities
- `math/` - Mathematical operations
- `object/` - Object utilities
- `text/` - Text/string processing
- `type/` - Type utilities

## Running Individual Benchmarks

To run benchmarks for a specific file:

```bash
deno bench array/group-by.bench.ts
```

To run all benchmarks in a workspace:

```bash
deno bench array/*.bench.ts
```

## Continuous Integration

The benchmark system can be integrated into CI/CD pipelines to:

1. Track performance over time
2. Detect performance regressions
3. Compare performance across different versions
4. Monitor the impact of code changes

## Performance Regression Detection

To detect performance regressions, compare the latest benchmark results with
historical data:

1. Run benchmarks on the main branch
2. Run benchmarks on your feature branch
3. Compare the `measuredRunsAvgMs` values
4. Investigate any significant increases (>10% slower)

## Example Output

```

Check notice on line 172 in BENCHMARK.md

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

BENCHMARK.md#L172

Fenced code blocks should have a language specified
🏃 Running benchmarks...

System Information:
Deno: 2.5.6
V8: 14.0.365.5
TypeScript: 5.9.2
OS: linux
Arch: x86_64
CPU Cores: 4

Running benchmarks for array...
Running benchmarks for math...
Running benchmarks for text...

✅ Completed 12 benchmark(s)

📊 Results saved to .benchmark-history/history.json

Benchmark Summary:
array/group-by: 0.0023ms avg (434782 ops/sec)
array/sort-by: 0.0156ms avg (64102 ops/sec)
math/sum: 0.0008ms avg (1250000 ops/sec)
text/capitalize: 0.0003ms avg (3333333 ops/sec)
```

## Maintenance

- History is limited to the last 100 runs to keep file size manageable
- The `.benchmark-history` directory is gitignored
- Benchmark files are version-controlled alongside their source files
- Update benchmarks when function signatures or behavior changes

## Contributing

When adding new functions:

1. Run `deno task bench:generate` to create template benchmark files
2. Update the generated templates with appropriate test cases
3. Run `deno task bench` to verify benchmarks work
4. Commit the benchmark files with your changes

## Notes

- Benchmarks should be run on a quiet system for consistent results
- Results may vary across different hardware and OS
- Use benchmark trends over time rather than absolute values
- Consider running multiple times and averaging for critical performance tests
Loading
Loading