Skip to content

Profile remaining slow binary benchmark cases #41

Description

@GiggleLiu

Context

PR #29 added the benchmark harnesses and multiple CPU contraction optimizations. The latest full benchmark refresh on that PR shows the network scenarios are faster than the one-thread Julia baseline, but several binary benchmark scenarios remain substantially slower.

Full benchmark refresh command set:

make bench-binary
make bench-network
JULIA_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 make bench-julia
python3 benchmarks/compare.py
make check

Current binary rows from the PR #29 full benchmark refresh:

| Benchmark  | Scenario                         |   Julia (ns) |    Rust (ns) | Julia/Rust | Drop vs Julia |
|------------|----------------------------------|--------------|--------------|------------|---------------|
| binary     | batched_matmul_8x8_batch_4       |         3000 |        26247 |    0.114 |  774.90% |
| binary     | high_d_12x12_contract_4_batch_4  |         8709 |        42167 |    0.207 |  384.18% |
| binary     | high_d_12x12_contract_6          |        22875 |        54340 |    0.421 |  137.55% |
| binary     | high_d_15x15_contract_5_batch_5  |        52958 |       119670 |    0.443 |  125.97% |
| binary     | high_d_15x15_contract_7          |       364417 |       375286 |    0.971 |    2.98% |
| binary     | high_d_18x18_contract_8          |     10882416 |     10720317 |    1.015 |    0.00% |
| binary     | high_d_20x20_contract_9          |     84399125 |     78107208 |    1.081 |    0.00% |
| binary     | matmul_10x10                     |         4333 |        35552 |    0.122 |  720.50% |

Problem

The largest remaining gaps are in small, batched, and materialization-heavy binary cases:

  • batched_matmul_8x8_batch_4: Rust is +774.90% slower.
  • matmul_10x10: Rust is +720.50% slower.
  • high_d_12x12_contract_4_batch_4: Rust is +384.18% slower.

The larger pure-contraction cases are now competitive or faster, so the remaining cost is likely not the large GEMM kernel itself.

Suggested investigation

  • Add or reuse a focused binary profiling harness similar to profile_network so setup cost and Criterion overhead do not obscure hotspots.
  • Profile matmul_10x10 and batched_matmul_8x8_batch_4 first.
  • Check whether remaining time is dominated by contraction setup, layout analysis, small-GEMM dispatch overhead, permutation/materialization, or tensor allocation.
  • Decide whether small Standard contractions need a separate fast path from the large faer-backed path.

Acceptance criteria

  • Record a focused profile for at least matmul_10x10 and batched_matmul_8x8_batch_4.
  • Identify the dominant cost bucket for each.
  • Either implement a targeted optimization with before/after benchmark numbers, or document why the current behavior is acceptable.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions