Add RaceBench workload (blackscholes, streamcluster, fluidanimate) by nkaretnikov · Pull Request #57 · oss-garage/bedrock

nkaretnikov · 2026-07-10T20:20:57Z

This PR copies part of the RaceBench dataset and adds it as a workload, so we could use it to benchmark various scheduling strategies as well as detection oracles. The files are copied for security and reproducibility reasons. For security reasons, I also do not copy the provided binaries, which are used to demonstrate that the bugs are reachable.

Information on RaceBench:

https://github.com/rb130/RaceBench
https://github.com/rb130/RaceBenchData (.1 samples copied from here, with some data removed)
https://siqima.me/publications/RaceBench.pdf (the paper explains how the original dataset was constructed and references the above repositories).

The programs added are part of the PARSEC benchmark suite (Princeton Application Repository for Shared-memory Computers), which is a standard set of multithreaded programs used to study parallel/shared-memory performance, but they have bugs injected into them by the RaceBench authors.

Why RaceBench:

it contains programs that are easy to compile (small size, no external deps, relatively modern source code)
it has a clear detection oracle, which is important since Bedrock doesn't have any out of the box:
- rc=0: no bug with this seed (expected common case)
- rc=134 (SIGABRT): a bug fired, the RACEBENCH_STAT file records which of the target's 20 injected bugs (trigger_num[bug_id]) was hit. With 3 files, we get 60 bugs total.

My tests outside of Bedrock:

when running with multiple CPUs, I was able to trigger up to 20/60 bugs just by running the programs:

$ for i in $(seq -w 1 10); do cat workloads/racebench/scripts/baseline-galactus-2026-07-07-allcores-run${i}.txt ; done

when limiting to a single CPU, I was able to trigger up to 1/60 bugs just by running the programs:

$ for i in $(seq -w 1 10); do cat workloads/racebench/scripts/baseline-galactus-2026-07-07-cpuset0-run${i}.txt ; done
[...]
fluidanimate: reached 1/20 in 974 runs -> bug_ids: 7

My tests with Bedrock:

I was able to trigger up to 1/60 bugs by walking from seed 1 to seed 3000 and running the programs (the same bug as was found on a single CPU on the host outside of Bedrock):

$ cat workloads/racebench/scripts/bedrock-galactus-2026-07-09-night.txt
[...]
boot 158 seed 158: coverage 1/60; slowest-plateau 0/1000000 NEW: fluidanimate+{7}

Now that we have these results, we should check the following:

Is the bug reproducible on Bedrock with this exact seed? For me, it currently doesn't, but I see that I have max pebs skid > margin_for_host_cpu (9 > 8), so there's non-determinism):

$ cd workloads/racebench/scripts
$ SEED_BASE=158 PLATEAU=1 MAX=1 ./bedrock.sh
--- bedrock coverage: seed_base=158 plateau=1 max=1 ---
boot 1 seed 158: coverage 0/60; slowest-plateau 1/1
blackscholes: reached 0/20 in 1 boots -> bug_ids:
streamcluster: reached 0/20 in 1 boots -> bug_ids:
fluidanimate: reached 0/20 in 1 boots -> bug_ids:
TOTAL bedrock coverage: 0/60

or the following from root (for full logs):

$ RDRAND_SEED=158 nix run .#test-racebench-workload 2>&1 | tee /tmp/repro-seed158.txt

Can we improve on the 1/60 result by changing the scheduler or the detection oracle? Can we at least get to 20/60 as on multi-core?
We could also work on improving speed, then we could measure N bugs per unit of time. You could find tons of bugs with a dumb fuzzer that is very very fast.

Instead of adding this to CI, the proposed workflow would be to run this locally and share results in PR comments (compared to main) because:

code changes could affect reproducibility, so bugs might no longer trigger on the same seed as before
if you want to test multiple seeds, you would want to run this for several hours, which is not practical on CI.

The following files provide documentation on the workload and scripts:

workloads/racebench/README.md
workloads/racebench/scripts/README.md.

Comparing the files in this PR vs the files in RaceBench (cb79cc578e064e026e0ed041c8a24501c4d91f58):

Bedrock:

$ for i in $(git diff origin/master..@ --name-only); do sha256sum $i; done > /tmp/br.txt

RaceBench:

$ for i in $(find blackscholes.1 -type f | egrep "code|input"); do sha256sum $i; done >> /tmp/rb.txt
$ for i in $(find streamcluster.1 -type f | egrep "code|input"); do sha256sum $i; done >> /tmp/rb.txt
$ for i in $(find fluidanimate.1 -type f | egrep "code|input"); do sha256sum $i; done >> /tmp/rb.txt

Diff:

$ awk 'NR==FNR{rb[$1]=rb[$1](rb[$1]?", ":"")$2;next}
       {if($1 in rb)printf "SAME  %s\n        = %s\n",$2,rb[$1];
        else printf "NEW   %s  (%s)\n",$2,substr($1,1,12)}' /tmp/rb.txt /tmp/br.txt

nkaretnikov · 2026-07-10T21:56:42Z

note: these inputs are binary files

nkaretnikov · 2026-07-10T21:59:54Z

note: these inputs are also binary files

nkaretnikov · 2026-07-10T22:27:42Z

Note: I will try to do the Bedrock run again after setting the margin to 16 on my CPU and see if I can get reproducible results with the same seed.

nkaretnikov · 2026-07-11T21:35:37Z

I have a local patch which causes Bedrock to abort immediately when PEBS skid exceeds host margin (the current behavior can be achieved with an environment variable BEDROCK_IGNORE_PEBS_MARGIN). I'm testing it locally now to see if it's sufficient to avoid non-reproducible runs on racebench.

Otherwise, I might need to add a similar one for APIC timer late injects. Will submit as separate PRs once I'm done with testing.

I'm also thinking about bumping the margin from 8 to 16 on Ice Lake-SP because every other run now aborts due to the skid. This is with 8:

SEED_BASE=1 PLATEAU=1000000 MAX=3000 ./bedrock.sh 2>&1 | tee "bedrock-galactus-$(date +%F)-night.txt"
--- bedrock coverage: seed_base=1 plateau=1000000 max=3000 ---
boot 1 seed 1: coverage 0/60; slowest-plateau 1/1000000
boot 2 seed 2: WARNING boot did not complete cleanly (no OK marker)
boot 2 seed 2: coverage 0/60; slowest-plateau 2/1000000
boot 3 seed 3: coverage 0/60; slowest-plateau 3/1000000
boot 4 seed 4: coverage 0/60; slowest-plateau 4/1000000
boot 5 seed 5: coverage 0/60; slowest-plateau 5/1000000
boot 6 seed 6: WARNING boot did not complete cleanly (no OK marker)
boot 6 seed 6: coverage 0/60; slowest-plateau 6/1000000
boot 7 seed 7: coverage 0/60; slowest-plateau 7/1000000
boot 8 seed 8: WARNING boot did not complete cleanly (no OK marker)
boot 8 seed 8: coverage 0/60; slowest-plateau 8/1000000
boot 9 seed 9: coverage 0/60; slowest-plateau 9/1000000
boot 10 seed 10: coverage 0/60; slowest-plateau 10/1000000
boot 11 seed 11: coverage 0/60; slowest-plateau 11/1000000
boot 12 seed 12: WARNING boot did not complete cleanly (no OK marker)
boot 12 seed 12: coverage 0/60; slowest-plateau 12/1000000
boot 13 seed 13: WARNING boot did not complete cleanly (no OK marker)
boot 13 seed 13: coverage 0/60; slowest-plateau 13/1000000
boot 14 seed 14: WARNING boot did not complete cleanly (no OK marker)
boot 14 seed 14: coverage 0/60; slowest-plateau 14/1000000
boot 15 seed 15: coverage 0/60; slowest-plateau 15/1000000
boot 16 seed 16: WARNING boot did not complete cleanly (no OK marker)
boot 16 seed 16: coverage 0/60; slowest-plateau 16/1000000
boot 17 seed 17: coverage 0/60; slowest-plateau 17/1000000
boot 18 seed 18: WARNING boot did not complete cleanly (no OK marker)
boot 18 seed 18: coverage 0/60; slowest-plateau 18/1000000
boot 19 seed 19: WARNING boot did not complete cleanly (no OK marker)
boot 19 seed 19: coverage 0/60; slowest-plateau 19/1000000
boot 20 seed 20: coverage 0/60; slowest-plateau 20/1000000

These warnings are runs aborting at boot due to the skid.

nkaretnikov · 2026-07-12T12:05:31Z

I've pushed the code that triggers an abort when PEBS margin is exceeded and bumped the margin to 16 on Ice Lake-SP, which allows to clear that check. I then let RaceBench run overnight:

$ SEED_BASE=1 PLATEAU=1000000 MAX=3000 ./bedrock.sh 2>&1 | tee "bedrock-galactus-$(date +%F)-night.txt"

This produced the same crash as earlier, but on a different seed:

$ grep NEW workloads/racebench/scripts/bedrock-galactus-2026-07-11-night.txt
boot 203 seed 203: coverage 1/60; slowest-plateau 0/1000000 NEW: fluidanimate+{7}

Unfortunately, this is still not reproducible:

$ RDRAND_SEED=203 nix run .#test-racebench-workload

The reason is most likely PEBS late injects, so I will add an abort on that condition as well. As mentioned earlier elsewhere, these tend to happen during early boot, so I will also need to do what I did on the mptest branch and restructure the runner to use snapshot fuzzing: boot the guest with a boot seed and halt after Linux is loaded, then attach a child guest with a different seed.

I will then do another run and see if the seed would be reproducible with that config.

If so, I'll submit the abort patches and the PEBS margin increase as separate PRs, so that they could be merged independently. Then I will rebase this branch, so that only RaceBench changes are part of this diff.

nkaretnikov · 2026-07-12T15:09:47Z

I've just pushed 3 more commits:

abort on APIC timer late injects (can be bypassed with an env var to get old behavior, which I also use to start the parent; this will be also needed to make CI pass as the runners do not rely on PEBS)
switch to snapshot fuzzing: parent is allowed to run with late injects to clear early boot, child aborts on late injects
disable guest vDSO clock (vdso=0): this is important to get deterministic results.

With these, I conducted several tests:

Determinism: Same boot seed, same child seed. Boot the parent. Boot the child 5 times and diff the runs. Only stats should differ. OK.
Determinism: Same boot seed (but the parent was restarted), same child seed. Restart the parent. Boot the child 5 times and diff the runs against 1. Only stats should differ. OK.
Seed affects scheduling: Repeat 1 with a different child seed. Diff against 1. Schedules should differ. OK.
Seed affects scheduling: Repeat 1 but change the boot seed and restart the parent. Diff against 1. Schedules should differ. OK.

Commands used:

To boot the parent:

BOOT_SEED=<parent_seed> nix run .#test-racebench-fork-parent

To boot the child:

BEDROCK_PARENT_ID=<id> RDRAND_SEED=<child_seed> nix run .#test-racebench-fork-child | tee run<N>.txt

nkaretnikov · 2026-07-12T16:01:42Z

CI passes again and determinism issues are fixed, so I'm doing another night run to see if (1) I'm able to reproduce the same bug (2) the bug is reproducible given a (boot seed, child seed) pair.

nkaretnikov · 2026-07-12T22:18:22Z

The night run finished (which I have not pushed yet) and triggered the same bug, now at seed 77:

$ grep NEW bedrock-galactus-2026-07-12-night.txt
fork 77 seed 77: coverage 1/60; slowest-plateau 0/1000000 NEW: fluidanimate+{7}

$ head -n3 bedrock-galactus-2026-07-12-night.txt
--- booting fork parent (boot_seed=0); tolerating early-boot late injects ---
--- fork parent ready: vm_id=39 (boot_seed=0) ---
--- bedrock coverage (fork): boot_seed=0 seed_base=1 plateau=1000000 max=3000 ---

But I couldn't reproduce this with standalone parent/child workflows or by rerunning the above command from seed 76 to 78.

BOOT_SEED=0 nix run .#test-racebench-fork-parent

BEDROCK_PARENT_ID=3040 RDRAND_SEED=77 nix run .#test-racebench-fork-child

I did notice, however, that when I ran the fork-parent workload, the image was rebuilt.

The workload is scheduler based, but I wonder if a different guest image could be causing this, so I'm going to make sure that is reproducible as well.

The reason this likely happens is because workload-monitor uses src = ./.., while the files in the repo change.

There's also this code:

find . -print0 | cpio --null -o -H newc | gzip -9 > $out

where find emits files in filesystem (unsorted) order, files carry build-time mtimes, and gzip -9 stamps the current time into the gzip header.

nkaretnikov · 2026-07-13T06:57:54Z

The new run that was executing during the night did not trigger any bugs, so I'm running one more, from guest seed 3001.

nkaretnikov added 9 commits July 6, 2026 23:20

Add RaceBench workload (blackscholes, streamcluster, fluidanimate)

cd6a0ed

Update docs

03dd593

Add baseline script and docs

f050fe0

Add baseline stats

823c178

Remove previous stats

b3a9789

Add scripts for Bedrock

238239f

Fix issue with capturing progress info

1ae795c

Fix module detection

0e267e3

Add bedrock night run results

f864449

nkaretnikov commented Jul 10, 2026

View reviewed changes

nkaretnikov mentioned this pull request Jul 10, 2026

Thread pausing #20

Open

Abort when PEBS skid exceeds host margin

0d94d62

nkaretnikov added 3 commits July 11, 2026 21:37

Bump Ice Lake-SP and default host margins to 16

79d00d2

Add night run with margin 16 on Ice Lake-SP

e17aa25

Delete old night run

4ef4204

nkaretnikov added 3 commits July 12, 2026 12:36

Abort when APIC timer injects late

b3fefbf

Switch to snapshot fuzzing for determinism

6fc7abc

Determinism: disable guest vDSO clock (vdso=0)

fb27471

nkaretnikov added 3 commits July 12, 2026 15:11

Lint

43628b2

Pass BEDROCK_IGNORE_LATE_INJECT on CI

02da802

Lint

550c315

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add RaceBench workload (blackscholes, streamcluster, fluidanimate)#57

Add RaceBench workload (blackscholes, streamcluster, fluidanimate)#57
nkaretnikov wants to merge 19 commits into
oss-garage:masterfrom
nkaretnikov:racebench

nkaretnikov commented Jul 10, 2026 •

edited

Loading

Uh oh!

nkaretnikov Jul 10, 2026

Uh oh!

nkaretnikov Jul 10, 2026

Uh oh!

nkaretnikov commented Jul 10, 2026

Uh oh!

nkaretnikov commented Jul 11, 2026

Uh oh!

nkaretnikov commented Jul 12, 2026

Uh oh!

nkaretnikov commented Jul 12, 2026

Uh oh!

nkaretnikov commented Jul 12, 2026

Uh oh!

nkaretnikov commented Jul 12, 2026 •

edited

Loading

Uh oh!

nkaretnikov commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

nkaretnikov commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nkaretnikov Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

nkaretnikov Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

nkaretnikov commented Jul 10, 2026

Uh oh!

nkaretnikov commented Jul 11, 2026

Uh oh!

nkaretnikov commented Jul 12, 2026

Uh oh!

nkaretnikov commented Jul 12, 2026

Uh oh!

nkaretnikov commented Jul 12, 2026

Uh oh!

nkaretnikov commented Jul 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nkaretnikov commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

nkaretnikov commented Jul 10, 2026 •

edited

Loading

nkaretnikov commented Jul 12, 2026 •

edited

Loading