ci: add focused PR benchmark labels by connortsui20 · Pull Request #9018 · vortex-data/vortex

connortsui20 · 2026-07-27T16:41:19Z

Adds focused PR benchmark triggers:

action/bench-random-access
action/bench-compress
action/bench-string (benchmarks: add string compression benchmark #9060)
action/bench-sql (excludes Vortex Compact)
action/bench-sql-compact (Vortex Compact plus Parquet controls)
action/bench-all (random access, compression, string encoding, and SQL; GPU remains separate)

Each trigger dispatches a role-specific reusable workflow. SQL coverage stays in the Python catalog and is resolved into a matrix at runtime. The all-CPU path uses a single pr-all preset (the exact union of pr and pr-compact), so shared SQL jobs run once instead of twice.

Benchmark comparisons now:

Compare the PR merge commit with the exact base commit.
Check the base commit's develop benchmark status once before SQL matrix fan-out.
Fail early when the baseline run failed instead of polling the full history from every runner.
Namespace remote datasets by preset to prevent concurrent jobs from clobbering each other.
Use Parquet controls and aggregate file-size totals for more stable reports.

TODO: After merge, delete the legacy action/benchmark* labels once develop uses the new dispatcher.

connortsui20 · 2026-07-27T17:39:45Z

note that we probably dont want all of these labels, but I feel like a bit more than what we have on develop now would be nice?

connortsui20 · 2026-07-27T17:48:04Z

actually Im going to make them all disjoint, lets see how that works

codspeed-hq · 2026-07-27T18:50:13Z

Merging this PR will not alter performance

✅ 1842 untouched benchmarks
⏩ 55 skipped benchmarks¹

_{Comparing ct/benchmark-labels (b1b5ca5) with develop (08c336f)}

55 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

myrrc · 2026-07-28T08:42:19Z

Let's use bench-random-access, ra isn't straight obvious

0ax1

couple of q's

0ax1 · 2026-07-28T15:10:10Z

+      # v4 (Postgres) ingest -- REQUIRED (see develop-bench.yml rationale). Empty records:
      # post-ingest.py --postgres upserts the commit row only. Gated on the
      # ingest-role ARN var (the assume-role input that MUST exist for OIDC to
      # succeed).


Probably need a action/bench-gpu-compress label for GPU compress?

yeah I am going to run this once this merges:

gh label edit "action/benchmark-gpu-compress" --name "action/bench-gpu-compress" \ --description "Run only the GPU compression benchmark on this PR" gh label delete action/benchmark --yes gh label delete action/benchmark-sql --yes gh label delete action/benchmark-sql-full --yes

0ax1 · 2026-07-28T15:11:30Z

+            "pr-compact": COMPACT,
            "pr-full": FULL_PR,
            "develop": FULL_LOCAL,
        },


is there any label reaching pr full?

no since I think we should just keep everything disjoint, if someone wants to run more benchmarks I think they should just add all 4 (or 5 for gpu) labels.

connortsui20 · 2026-07-28T15:30:28Z

          cat comment.md >> "$GITHUB_STEP_SUMMARY"
      - name: Comment PR
-        if: github.event.pull_request.head.repo.fork == false
+        if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false


i think this is a qol change

github-actions · 2026-07-29T12:55:20Z

Benchmarks: TPC-H SF=10 on S3 📖

Verdict: No clear signal (environment too noisy confidence)
Attributed non-control impact: -6.7%
Engines: DataFusion No clear signal (-2.8%, environment too noisy confidence) · DuckDB No clear signal (-10.5%, environment too noisy confidence)
Vortex (geomean): 0.986x ➖
Parquet (geomean): 1.057x ➖
Shifts: Parquet (control) +5.7% · Median polish +1.1%

How to read Verdict and Engines

Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
Engines: Per-engine attribution using that engine's Parquet rows as controls. This answers whether each engine improved or regressed independently.
Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-compact (1.022x ➖, 0↑ 1↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
tpch_q01/datafusion:vortex-compact	808724314	842862012	0.96
tpch_q02/datafusion:vortex-compact	576594266	545310807	1.06
tpch_q03/datafusion:vortex-compact 🚨	968406825	648963384	1.49
tpch_q04/datafusion:vortex-compact	435936463	394267252	1.11
tpch_q05/datafusion:vortex-compact	923043916	809398552	1.14
tpch_q06/datafusion:vortex-compact	495436006	477579211	1.04
tpch_q07/datafusion:vortex-compact	930024697	944027530	0.99
tpch_q08/datafusion:vortex-compact	1107188600	1036401071	1.07
tpch_q09/datafusion:vortex-compact	1249486193	1227430298	1.02
tpch_q10/datafusion:vortex-compact	855438137	1000342254	0.86
tpch_q11/datafusion:vortex-compact	444681886	476074409	0.93
tpch_q12/datafusion:vortex-compact	670528541	646803443	1.04
tpch_q13/datafusion:vortex-compact	410766642	396995153	1.03
tpch_q14/datafusion:vortex-compact	469783959	467221527	1.01
tpch_q15/datafusion:vortex-compact	814580427	888213740	0.92
tpch_q16/datafusion:vortex-compact	301718460	291651551	1.03
tpch_q17/datafusion:vortex-compact	1044851696	1032519203	1.01
tpch_q18/datafusion:vortex-compact	1209256831	1123679708	1.08
tpch_q19/datafusion:vortex-compact	700770127	770757622	0.91
tpch_q20/datafusion:vortex-compact	731317989	729111269	1.00
tpch_q21/datafusion:vortex-compact	1296367460	1307033397	0.99
tpch_q22/datafusion:vortex-compact	382422199	400687012	0.95

datafusion / parquet (1.052x ➖, 1↑ 1↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
tpch_q01/datafusion:parquet	744965011	692981416	1.08
tpch_q02/datafusion:parquet	747835995	596699021	1.25
tpch_q03/datafusion:parquet	915235416	877733494	1.04
tpch_q04/datafusion:parquet	603603145	474717983	1.27
tpch_q05/datafusion:parquet 🚨	1558582065	1061796481	1.47
tpch_q06/datafusion:parquet	638413696	518919828	1.23
tpch_q07/datafusion:parquet	1404378870	1147880139	1.22
tpch_q08/datafusion:parquet	1404199638	1402241027	1.00
tpch_q09/datafusion:parquet	1599926794	1718502346	0.93
tpch_q10/datafusion:parquet	1969907570	1873659515	1.05
tpch_q11/datafusion:parquet	452431171	412573437	1.10
tpch_q12/datafusion:parquet	609326338	646370987	0.94
tpch_q13/datafusion:parquet	693656914	673816962	1.03
tpch_q14/datafusion:parquet	826879858	811701159	1.02
tpch_q15/datafusion:parquet	1317613185	1380195977	0.95
tpch_q16/datafusion:parquet	369982219	309089289	1.20
tpch_q17/datafusion:parquet	1300371936	1344351358	0.97
tpch_q18/datafusion:parquet	1442739589	1470278292	0.98
tpch_q19/datafusion:parquet	937369818	882733370	1.06
tpch_q20/datafusion:parquet	1119540189	1108598004	1.01
tpch_q21/datafusion:parquet	1518249664	1675957148	0.91
tpch_q22/datafusion:parquet 🚀	733393529	1065361068	0.69

duckdb / vortex-compact (0.950x ➖, 0↑ 0↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
tpch_q01/duckdb:vortex-compact	860488085	877654368	0.98
tpch_q02/duckdb:vortex-compact	1130241274	1130172089	1.00
tpch_q03/duckdb:vortex-compact	1263861749	1231405170	1.03
tpch_q04/duckdb:vortex-compact	806146817	854124281	0.94
tpch_q05/duckdb:vortex-compact	1553691869	1483101169	1.05
tpch_q06/duckdb:vortex-compact	1063913698	1084616387	0.98
tpch_q07/duckdb:vortex-compact	1632367632	1665612265	0.98
tpch_q08/duckdb:vortex-compact	1843894373	1880105936	0.98
tpch_q09/duckdb:vortex-compact	2036544156	2061830872	0.99
tpch_q10/duckdb:vortex-compact	1706306857	1659383815	1.03
tpch_q11/duckdb:vortex-compact	641228172	717551975	0.89
tpch_q12/duckdb:vortex-compact	1725039923	1984717139	0.87
tpch_q13/duckdb:vortex-compact	1044317185	1203299614	0.87
tpch_q14/duckdb:vortex-compact	912193700	933207494	0.98
tpch_q15/duckdb:vortex-compact	930755034	980280502	0.95
tpch_q16/duckdb:vortex-compact	413367715	461980278	0.89
tpch_q17/duckdb:vortex-compact	1329337519	1439886408	0.92
tpch_q18/duckdb:vortex-compact	1148540040	1380651302	0.83
tpch_q19/duckdb:vortex-compact	1217739214	1260384235	0.97
tpch_q20/duckdb:vortex-compact	1780761689	1866460079	0.95
tpch_q21/duckdb:vortex-compact	2558798610	2775554922	0.92
tpch_q22/duckdb:vortex-compact	417247776	448727919	0.93

duckdb / parquet (1.062x ➖, 0↑ 0↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
tpch_q01/duckdb:parquet	661059584	658092844	1.00
tpch_q02/duckdb:parquet	871838152	792355084	1.10
tpch_q03/duckdb:parquet	1346104663	1255880403	1.07
tpch_q04/duckdb:parquet	779550748	684543508	1.14
tpch_q05/duckdb:parquet	1608744630	1483211935	1.08
tpch_q06/duckdb:parquet	608491902	571889272	1.06
tpch_q07/duckdb:parquet	1384021169	1348807627	1.03
tpch_q08/duckdb:parquet	1989300040	1724316576	1.15
tpch_q09/duckdb:parquet	1718329340	1780287041	0.97
tpch_q10/duckdb:parquet	2709743289	2721323791	1.00
tpch_q11/duckdb:parquet	510304948	471923626	1.08
tpch_q12/duckdb:parquet	898888710	801824177	1.12
tpch_q13/duckdb:parquet	1071383592	1000729106	1.07
tpch_q14/duckdb:parquet	1038243801	1005478010	1.03
tpch_q15/duckdb:parquet	764023093	713548205	1.07
tpch_q16/duckdb:parquet	796410289	654892124	1.22
tpch_q17/duckdb:parquet	964708830	913151372	1.06
tpch_q18/duckdb:parquet	1120917120	1020798393	1.10
tpch_q19/duckdb:parquet	1146371095	1153111560	0.99
tpch_q20/duckdb:parquet	1478429237	1512622690	0.98
tpch_q21/duckdb:parquet	1289734801	1295397220	1.00
tpch_q22/duckdb:parquet	955968975	890542918	1.07

github-actions · 2026-07-29T12:56:32Z

Benchmarks: Statistical and Population Genetics 📖

Verdict: No clear signal (low confidence)
Attributed non-control impact: -0.1%
Engines: DuckDB No clear signal (-0.1%, low confidence)
Vortex (geomean): 1.016x ➖
Parquet (geomean): 1.017x ➖
Shifts: Parquet (control) +1.7% · Median polish +1.4%

How to read Verdict and Engines

Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
Engines: Per-engine attribution using that engine's Parquet rows as controls. This answers whether each engine improved or regressed independently.
Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

duckdb / vortex-compact (1.016x ➖, 0↑ 0↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
statpopgen_q00/duckdb:vortex-compact	12481322	12139709	1.03
statpopgen_q01/duckdb:vortex-compact	16791099	15655074	1.07
statpopgen_q02/duckdb:vortex-compact	591261886	587268739	1.01
statpopgen_q03/duckdb:vortex-compact	1168544147	1161428606	1.01
statpopgen_q04/duckdb:vortex-compact	1193791513	1179536805	1.01
statpopgen_q05/duckdb:vortex-compact	601677562	598222435	1.01
statpopgen_q06/duckdb:vortex-compact	1513235268	1500671058	1.01
statpopgen_q07/duckdb:vortex-compact	911067620	908040518	1.00
statpopgen_q08/duckdb:vortex-compact	941445199	935343940	1.01
statpopgen_q09/duckdb:vortex-compact	948774574	927077281	1.02
statpopgen_q10/duckdb:vortex-compact	2637280379	2630828062	1.00

duckdb / parquet (1.017x ➖, 0↑ 0↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
statpopgen_q00/duckdb:parquet	338761550	340151722	1.00
statpopgen_q01/duckdb:parquet	424188925	416824826	1.02
statpopgen_q02/duckdb:parquet	814666876	799568694	1.02
statpopgen_q03/duckdb:parquet	1243971499	1234671178	1.01
statpopgen_q04/duckdb:parquet	1260591989	1224167105	1.03
statpopgen_q05/duckdb:parquet	870592633	851006209	1.02
statpopgen_q06/duckdb:parquet	1510073204	1452768088	1.04
statpopgen_q07/duckdb:parquet	906993996	901446097	1.01
statpopgen_q08/duckdb:parquet	920137610	917145688	1.00
statpopgen_q09/duckdb:parquet	1063806093	1049717187	1.01
statpopgen_q10/duckdb:parquet	2300640198	2235094467	1.03

No file size changes detected.

github-actions · 2026-07-29T12:56:34Z

Benchmarks: Clickbench on NVME 📖

Verdict: No clear signal (low confidence)
Attributed non-control impact: +3.5%
Engines: DataFusion No clear signal (+9.2%, low confidence) · DuckDB No clear signal (-1.9%, low confidence)
Vortex (geomean): 1.020x ➖
Parquet (geomean): 0.985x ➖
Shifts: Parquet (control) -1.5% · Median polish -0.2%

How to read Verdict and Engines

Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
Engines: Per-engine attribution using that engine's Parquet rows as controls. This answers whether each engine improved or regressed independently.
Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-compact (1.060x ➖, 0↑ 12↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
clickbench_q00/datafusion:vortex-compact	1659990	1636222	1.01
clickbench_q01/datafusion:vortex-compact	24241852	25108341	0.97
clickbench_q02/datafusion:vortex-compact	92784719	94799011	0.98
clickbench_q03/datafusion:vortex-compact	56156796	56713716	0.99
clickbench_q04/datafusion:vortex-compact	254124381	255442657	0.99
clickbench_q05/datafusion:vortex-compact	362918252	375912526	0.97
clickbench_q06/datafusion:vortex-compact 🚨	1889234	1686707	1.12
clickbench_q07/datafusion:vortex-compact	39538277	38960256	1.01
clickbench_q08/datafusion:vortex-compact	413068061	406202739	1.02
clickbench_q09/datafusion:vortex-compact	582887569	596867798	0.98
clickbench_q10/datafusion:vortex-compact	190149019	190366880	1.00
clickbench_q11/datafusion:vortex-compact	255977453	256313691	1.00
clickbench_q12/datafusion:vortex-compact	396815904	365830975	1.08
clickbench_q13/datafusion:vortex-compact	619261696	574616330	1.08
clickbench_q14/datafusion:vortex-compact	371706579	379172348	0.98
clickbench_q15/datafusion:vortex-compact	303794076	301094547	1.01
clickbench_q16/datafusion:vortex-compact	766219161	757042335	1.01
clickbench_q17/datafusion:vortex-compact	757695767	767476436	0.99
clickbench_q18/datafusion:vortex-compact	1581402455	1459237090	1.08
clickbench_q19/datafusion:vortex-compact	44838058	41730859	1.07
clickbench_q20/datafusion:vortex-compact 🚨	440553722	397407864	1.11
clickbench_q21/datafusion:vortex-compact	666162940	606378982	1.10
clickbench_q22/datafusion:vortex-compact 🚨	1092456335	962238746	1.14
clickbench_q23/datafusion:vortex-compact 🚨	2597400484	2333998184	1.11
clickbench_q24/datafusion:vortex-compact	109486916	100294370	1.09
clickbench_q25/datafusion:vortex-compact	200411170	185856891	1.08
clickbench_q26/datafusion:vortex-compact 🚨	111506528	98783616	1.13
clickbench_q27/datafusion:vortex-compact 🚨	606295459	525002277	1.15
clickbench_q28/datafusion:vortex-compact	2735734738	2509840888	1.09
clickbench_q29/datafusion:vortex-compact	107506239	99625476	1.08
clickbench_q30/datafusion:vortex-compact	462215938	450472489	1.03
clickbench_q31/datafusion:vortex-compact	512264298	476851541	1.07
clickbench_q32/datafusion:vortex-compact 🚨	1311260317	1131474863	1.16
clickbench_q33/datafusion:vortex-compact 🚨	1673331616	1486842504	1.13
clickbench_q34/datafusion:vortex-compact 🚨	1714747492	1496132234	1.15
clickbench_q35/datafusion:vortex-compact 🚨	310480846	268673122	1.16
clickbench_q36/datafusion:vortex-compact	62327763	60882971	1.02
clickbench_q37/datafusion:vortex-compact	29312584	27121248	1.08
clickbench_q38/datafusion:vortex-compact	17793008	17537298	1.01
clickbench_q39/datafusion:vortex-compact	133728788	130990500	1.02
clickbench_q40/datafusion:vortex-compact	15606204	14199328	1.10
clickbench_q41/datafusion:vortex-compact 🚨	17988453	14897857	1.21
clickbench_q42/datafusion:vortex-compact 🚨	12825105	11474461	1.12

datafusion / parquet (0.971x ➖, 1↑ 0↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
clickbench_q00/datafusion:parquet	1590779	1557716	1.02
clickbench_q01/datafusion:parquet	18998698	19825174	0.96
clickbench_q02/datafusion:parquet	43388861	44249677	0.98
clickbench_q03/datafusion:parquet	34249381	35174137	0.97
clickbench_q04/datafusion:parquet	269256225	291088438	0.92
clickbench_q05/datafusion:parquet	317180802	337226836	0.94
clickbench_q06/datafusion:parquet	1542522	1554232	0.99
clickbench_q07/datafusion:parquet	21000347	20771769	1.01
clickbench_q08/datafusion:parquet	335464350	339705140	0.99
clickbench_q09/datafusion:parquet	462340620	470982677	0.98
clickbench_q10/datafusion:parquet	90788017	89598845	1.01
clickbench_q11/datafusion:parquet	115936095	118046529	0.98
clickbench_q12/datafusion:parquet	300789397	304707319	0.99
clickbench_q13/datafusion:parquet	469922536	482497862	0.97
clickbench_q14/datafusion:parquet	310205589	312060161	0.99
clickbench_q15/datafusion:parquet	285950544	284651797	1.00
clickbench_q16/datafusion:parquet	641960419	664904723	0.97
clickbench_q17/datafusion:parquet	655611918	647556272	1.01
clickbench_q18/datafusion:parquet	1383731064	1449923714	0.95
clickbench_q19/datafusion:parquet	26483035	28836207	0.92
clickbench_q20/datafusion:parquet	573718910	568148859	1.01
clickbench_q21/datafusion:parquet	632205985	629780081	1.00
clickbench_q22/datafusion:parquet	910180674	912825568	1.00
clickbench_q23/datafusion:parquet	4026900969	4129821952	0.98
clickbench_q24/datafusion:parquet	53845048	54398130	0.99
clickbench_q25/datafusion:parquet	129924790	132506196	0.98
clickbench_q26/datafusion:parquet	53933782	55798275	0.97
clickbench_q27/datafusion:parquet	645967712	682924706	0.95
clickbench_q28/datafusion:parquet	2453097523	2503809181	0.98
clickbench_q29/datafusion:parquet 🚀	45058531	54599831	0.83
clickbench_q30/datafusion:parquet	319179670	330996568	0.96
clickbench_q31/datafusion:parquet	344393866	358590950	0.96
clickbench_q32/datafusion:parquet	1095310795	1119832366	0.98
clickbench_q33/datafusion:parquet	1510951681	1518703432	0.99
clickbench_q34/datafusion:parquet	1499731018	1535920519	0.98
clickbench_q35/datafusion:parquet	251604449	259928759	0.97
clickbench_q36/datafusion:parquet	107214973	114423623	0.94
clickbench_q37/datafusion:parquet	43584969	44770663	0.97
clickbench_q38/datafusion:parquet	62283252	65198957	0.96
clickbench_q39/datafusion:parquet	213002966	215005839	0.99
clickbench_q40/datafusion:parquet	22632464	23548777	0.96
clickbench_q41/datafusion:parquet	21738268	23008888	0.94
clickbench_q42/datafusion:parquet	21594996	23280117	0.93

duckdb / vortex-compact (0.981x ➖, 2↑ 0↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
clickbench_q00/duckdb:vortex-compact	9425905	9508606	0.99
clickbench_q01/duckdb:vortex-compact	20498348	22189687	0.92
clickbench_q02/duckdb:vortex-compact	81686835	82932078	0.98
clickbench_q03/duckdb:vortex-compact	47600773	47503392	1.00
clickbench_q04/duckdb:vortex-compact	228223246	231678155	0.99
clickbench_q05/duckdb:vortex-compact	243086535	250659107	0.97
clickbench_q06/duckdb:vortex-compact	17625435	17316853	1.02
clickbench_q07/duckdb:vortex-compact	35490252	36640416	0.97
clickbench_q08/duckdb:vortex-compact	350699393	347626710	1.01
clickbench_q09/duckdb:vortex-compact	541690986	544102484	1.00
clickbench_q10/duckdb:vortex-compact	187575039	192268220	0.98
clickbench_q11/duckdb:vortex-compact	254351755	257936718	0.99
clickbench_q12/duckdb:vortex-compact	315492478	318919345	0.99
clickbench_q13/duckdb:vortex-compact	576184860	583721582	0.99
clickbench_q14/duckdb:vortex-compact	363217478	367771778	0.99
clickbench_q15/duckdb:vortex-compact	279206325	283164580	0.99
clickbench_q16/duckdb:vortex-compact	649298443	651764276	1.00
clickbench_q17/duckdb:vortex-compact	552185758	555669042	0.99
clickbench_q18/duckdb:vortex-compact	1052874299	1055012429	1.00
clickbench_q19/duckdb:vortex-compact	33330607	34430342	0.97
clickbench_q20/duckdb:vortex-compact	389000556	393172609	0.99
clickbench_q21/duckdb:vortex-compact	580644306	600883907	0.97
clickbench_q22/duckdb:vortex-compact	997051329	1020407182	0.98
clickbench_q23/duckdb:vortex-compact	339343445	356214182	0.95
clickbench_q24/duckdb:vortex-compact 🚀	64874255	76270735	0.85
clickbench_q25/duckdb:vortex-compact	195661776	199206129	0.98
clickbench_q26/duckdb:vortex-compact	68557302	72109927	0.95
clickbench_q27/duckdb:vortex-compact	741932526	738788383	1.00
clickbench_q28/duckdb:vortex-compact	3187801632	3171853944	1.01
clickbench_q29/duckdb:vortex-compact	80125171	78355692	1.02
clickbench_q30/duckdb:vortex-compact	433784787	438117043	0.99
clickbench_q31/duckdb:vortex-compact	532929931	535192397	1.00
clickbench_q32/duckdb:vortex-compact	1213390537	1219436244	1.00
clickbench_q33/duckdb:vortex-compact	1142832204	1145509991	1.00
clickbench_q34/duckdb:vortex-compact	1267470724	1264738798	1.00
clickbench_q35/duckdb:vortex-compact	414431867	412104198	1.01
clickbench_q36/duckdb:vortex-compact	33178026	35228806	0.94
clickbench_q37/duckdb:vortex-compact 🚀	24891064	29254985	0.85
clickbench_q38/duckdb:vortex-compact	29345707	27122880	1.08
clickbench_q39/duckdb:vortex-compact	53440811	54199057	0.99
clickbench_q40/duckdb:vortex-compact	24839657	24653312	1.01
clickbench_q41/duckdb:vortex-compact	23084031	24520577	0.94
clickbench_q42/duckdb:vortex-compact	20548723	20697236	0.99

duckdb / parquet (1.000x ➖, 0↑ 0↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
clickbench_q00/duckdb:parquet	22927304	24157881	0.95
clickbench_q01/duckdb:parquet	30723536	30399276	1.01
clickbench_q02/duckdb:parquet	50963061	50697120	1.01
clickbench_q03/duckdb:parquet	40833569	40343862	1.01
clickbench_q04/duckdb:parquet	205566001	204505333	1.01
clickbench_q05/duckdb:parquet	260565266	258217241	1.01
clickbench_q06/duckdb:parquet	48241730	48128871	1.00
clickbench_q07/duckdb:parquet	32579319	32049529	1.02
clickbench_q08/duckdb:parquet	278115224	271836326	1.02
clickbench_q09/duckdb:parquet	405325081	400536005	1.01
clickbench_q10/duckdb:parquet	85585638	85073869	1.01
clickbench_q11/duckdb:parquet	103320625	102109916	1.01
clickbench_q12/duckdb:parquet	286313717	281211219	1.02
clickbench_q13/duckdb:parquet	478791264	472638438	1.01
clickbench_q14/duckdb:parquet	317535985	321255770	0.99
clickbench_q15/duckdb:parquet	260759993	255068156	1.02
clickbench_q16/duckdb:parquet	606305462	596293696	1.02
clickbench_q17/duckdb:parquet	501776246	497732043	1.01
clickbench_q18/duckdb:parquet	1039539688	1022006856	1.02
clickbench_q19/duckdb:parquet	29741386	29252940	1.02
clickbench_q20/duckdb:parquet	414119611	420407654	0.99
clickbench_q21/duckdb:parquet	540007497	536728809	1.01
clickbench_q22/duckdb:parquet	926040706	920133391	1.01
clickbench_q23/duckdb:parquet	262003568	278518592	0.94
clickbench_q24/duckdb:parquet	74046397	73311989	1.01
clickbench_q25/duckdb:parquet	163109364	163126047	1.00
clickbench_q26/duckdb:parquet	56723833	54227693	1.05
clickbench_q27/duckdb:parquet	476498852	472481740	1.01
clickbench_q28/duckdb:parquet	4746243954	4788004480	0.99
clickbench_q29/duckdb:parquet	42294988	44327529	0.95
clickbench_q30/duckdb:parquet	313328002	312767372	1.00
clickbench_q31/duckdb:parquet	376577012	376050136	1.00
clickbench_q32/duckdb:parquet	1112752694	1112064466	1.00
clickbench_q33/duckdb:parquet	1114092005	1111345893	1.00
clickbench_q34/duckdb:parquet	1153778063	1151501596	1.00
clickbench_q35/duckdb:parquet	374528166	369451117	1.01
clickbench_q36/duckdb:parquet	48040753	48193509	1.00
clickbench_q37/duckdb:parquet	35443854	35639157	0.99
clickbench_q38/duckdb:parquet	36914789	36489618	1.01
clickbench_q39/duckdb:parquet	82014952	90137763	0.91
clickbench_q40/duckdb:parquet	21499052	21400859	1.00
clickbench_q41/duckdb:parquet	19838309	21335470	0.93
clickbench_q42/duckdb:parquet	23180954	22888599	1.01

No file size changes detected.

github-actions · 2026-07-29T12:59:15Z

Benchmarks: Appian on NVME 📖

Verdict: No clear signal (low confidence)
Attributed non-control impact: +1.5%
Engines: DataFusion No clear signal (+2.5%, environment too noisy confidence) · DuckDB No clear signal (+0.6%, low confidence)
Vortex (geomean): 1.077x ➖
Parquet (geomean): 1.061x ➖
Shifts: Parquet (control) +6.1% · Median polish +7.4%

How to read Verdict and Engines

Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
Engines: Per-engine attribution using that engine's Parquet rows as controls. This answers whether each engine improved or regressed independently.
Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-compact (1.076x ➖, 0↑ 2↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
appian_q01/datafusion:vortex-compact	131958145	132460200	1.00
appian_q02/datafusion:vortex-compact 🚨	652764828	586903352	1.11
appian_q03/datafusion:vortex-compact 🚨	380676742	344908615	1.10
appian_q04/datafusion:vortex-compact	47361489233	44474511014	1.06
appian_q05/datafusion:vortex-compact	266104781	245584180	1.08
appian_q06/datafusion:vortex-compact	360491977	332211289	1.09
appian_q07/datafusion:vortex-compact	458529176	424923259	1.08
appian_q08/datafusion:vortex-compact	2148289532	1974816810	1.09

datafusion / parquet (1.050x ➖, 0↑ 1↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
appian_q01/datafusion:parquet	131457968	136519855	0.96
appian_q02/datafusion:parquet	614220158	609837732	1.01
appian_q03/datafusion:parquet	339761575	330204484	1.03
appian_q04/datafusion:parquet	45592538413	44294316897	1.03
appian_q05/datafusion:parquet	305371496	278242114	1.10
appian_q06/datafusion:parquet	384286971	356936219	1.08
appian_q07/datafusion:parquet 🚨	495820942	442823752	1.12
appian_q08/datafusion:parquet	2117424384	1948973718	1.09

duckdb / vortex-compact (1.078x ➖, 0↑ 2↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
appian_q01/duckdb:vortex-compact 🚨	209310134	187502669	1.12
appian_q02/duckdb:vortex-compact	663076772	615293416	1.08
appian_q03/duckdb:vortex-compact	316118903	289138791	1.09
appian_q04/duckdb:vortex-compact	1415064093	1353699599	1.05
appian_q05/duckdb:vortex-compact	318258081	289866542	1.10
appian_q06/duckdb:vortex-compact	824696469	805927133	1.02
appian_q07/duckdb:vortex-compact 🚨	414139519	370065795	1.12
appian_q08/duckdb:vortex-compact	1491048528	1415532689	1.05

duckdb / parquet (1.072x ➖, 0↑ 2↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
appian_q01/duckdb:parquet 🚨	221969334	200970646	1.10
appian_q02/duckdb:parquet	642341157	600661720	1.07
appian_q03/duckdb:parquet 🚨	321878821	289030535	1.11
appian_q04/duckdb:parquet	1381178759	1319874437	1.05
appian_q05/duckdb:parquet	333249008	313723016	1.06
appian_q06/duckdb:parquet	816261885	787276726	1.04
appian_q07/duckdb:parquet	392170108	361602327	1.08
appian_q08/duckdb:parquet	1344588309	1269980482	1.06

No file size changes detected.

connortsui20 · 2026-07-29T13:02:35Z

There's a lot of file size changes here but I'm pretty sure there is something wrong currently on develop with that anyways?

github-actions · 2026-07-29T13:11:04Z

Benchmarks: Random Access 📖

Verdict: No clear signal (low confidence)
Attributed non-control impact: -2.1%
Engines: Random access No clear signal (-2.1%, low confidence)
Vortex (geomean): 1.042x ➖
Parquet (geomean): 1.050x ➖
Shifts: Parquet (control) +5.0% · Median polish +3.5%

How to read Verdict and Engines

Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
Engines: Per-engine attribution using that engine's Parquet rows as controls. This answers whether each engine improved or regressed independently.
Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

random-access / vortex-file-compressed (1.042x ➖, 0↑ 3↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
random-access/feature-vectors/correlated/vortex-tokio-local-disk	3275482	3245429	1.01
random-access/feature-vectors/correlated/vortex-tokio-local-disk-footer 🚨	3046436	2707101	1.13
random-access/feature-vectors/uniform/vortex-tokio-local-disk	3762913	3784299	0.99
random-access/feature-vectors/uniform/vortex-tokio-local-disk-footer	6163489	6288274	0.98
random-access/nested-lists/correlated/vortex-tokio-local-disk	333615	338077	0.99
random-access/nested-lists/correlated/vortex-tokio-local-disk-footer	448302	428775	1.05
random-access/nested-lists/uniform/vortex-tokio-local-disk	2035145	1867546	1.09
random-access/nested-lists/uniform/vortex-tokio-local-disk-footer 🚨	2155846	1952611	1.10
random-access/nested-structs/correlated/vortex-tokio-local-disk	456084	428530	1.06
random-access/nested-structs/correlated/vortex-tokio-local-disk-footer	649486	653837	0.99
random-access/nested-structs/uniform/vortex-tokio-local-disk	1327808	1330766	1.00
random-access/nested-structs/uniform/vortex-tokio-local-disk-footer	1663781	1668385	1.00
random-access/taxi/correlated/vortex-tokio-local-disk 🚨	1339472	1217489	1.10
random-access/taxi/correlated/vortex-tokio-local-disk-footer	1950305	1775328	1.10
random-access/taxi/uniform/vortex-tokio-local-disk	5126867	4846605	1.06
random-access/taxi/uniform/vortex-tokio-local-disk-footer	5595761	5308911	1.05
random-access/vortex-tokio-local-disk	738399	727529	1.01
random-access/vortex-tokio-local-disk-footer	1262707	1185429	1.07

random-access / parquet (1.050x ➖, 0↑ 3↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
random-access/feature-vectors/correlated/parquet-tokio-local-disk	8648743153	8496745121	1.02
random-access/feature-vectors/correlated/parquet-tokio-local-disk-footer	8539380233	8306059405	1.03
random-access/feature-vectors/uniform/parquet-tokio-local-disk	8427481584	8350194257	1.01
random-access/feature-vectors/uniform/parquet-tokio-local-disk-footer	8419490086	8409487794	1.00
random-access/nested-lists/correlated/parquet-tokio-local-disk	134839468	131105156	1.03
random-access/nested-lists/correlated/parquet-tokio-local-disk-footer	131794733	128867416	1.02
random-access/nested-lists/uniform/parquet-tokio-local-disk	131082015	129584127	1.01
random-access/nested-lists/uniform/parquet-tokio-local-disk-footer	134925157	128014950	1.05
random-access/nested-structs/correlated/parquet-tokio-local-disk 🚨	22779675	20381871	1.12
random-access/nested-structs/correlated/parquet-tokio-local-disk-footer 🚨	23083227	20213427	1.14
random-access/nested-structs/uniform/parquet-tokio-local-disk	22831489	21468026	1.06
random-access/nested-structs/uniform/parquet-tokio-local-disk-footer 🚨	23204255	20288354	1.14
random-access/parquet-tokio-local-disk	175101261	168775041	1.04
random-access/parquet-tokio-local-disk-footer	174485642	167567327	1.04
random-access/taxi/correlated/parquet-tokio-local-disk	262858071	253930438	1.04
random-access/taxi/correlated/parquet-tokio-local-disk-footer	264376904	251625033	1.05
random-access/taxi/uniform/parquet-tokio-local-disk	277711967	263938609	1.05
random-access/taxi/uniform/parquet-tokio-local-disk-footer	280251830	265358197	1.06

random-access / lance (1.014x ➖, 0↑ 0↓)

name	PR `4d0fc64` (ns)	base `f3d6795` (ns)	ratio (PR/base)
random-access/feature-vectors/correlated/lance-tokio-local-disk	373792	360496	1.04
random-access/feature-vectors/correlated/lance-tokio-local-disk-footer	1060255	1028273	1.03
random-access/feature-vectors/uniform/lance-tokio-local-disk	1181447	1160049	1.02
random-access/feature-vectors/uniform/lance-tokio-local-disk-footer	1938786	1846622	1.05
random-access/lance-tokio-local-disk	680122	660627	1.03
random-access/lance-tokio-local-disk-footer	1394971	1435486	0.97
random-access/nested-lists/correlated/lance-tokio-local-disk	226550	225909	1.00
random-access/nested-lists/correlated/lance-tokio-local-disk-footer	585089	575549	1.02
random-access/nested-lists/uniform/lance-tokio-local-disk	986774	1031033	0.96
random-access/nested-lists/uniform/lance-tokio-local-disk-footer	1402186	1372437	1.02
random-access/nested-structs/correlated/lance-tokio-local-disk	366806	361390	1.01
random-access/nested-structs/correlated/lance-tokio-local-disk-footer	538964	540422	1.00
random-access/nested-structs/uniform/lance-tokio-local-disk	2650149	2495581	1.06
random-access/nested-structs/uniform/lance-tokio-local-disk-footer	2784299	2747059	1.01
random-access/taxi/correlated/lance-tokio-local-disk	1012167	984431	1.03
random-access/taxi/correlated/lance-tokio-local-disk-footer	2036524	2063456	0.99
random-access/taxi/uniform/lance-tokio-local-disk	10104378	9933825	1.02
random-access/taxi/uniform/lance-tokio-local-disk-footer	11037013	10936940	1.01

connortsui20 · 2026-07-29T15:17:40Z

I forgot that by separating everything, I need to update the comment comparisons because they used to all be unified

changes since then

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

connortsui20 force-pushed the ct/benchmark-labels branch from 89eebfa to 77a7e5d Compare July 27, 2026 16:42

connortsui20 requested review from AdamGS and myrrc July 27, 2026 17:39

connortsui20 force-pushed the ct/benchmark-labels branch from 77a7e5d to 9cbc6da Compare July 27, 2026 17:49

connortsui20 added the changelog/ci label Jul 27, 2026

Base automatically changed from ct/declarative-bench to develop July 27, 2026 18:00

connortsui20 force-pushed the ct/benchmark-labels branch 2 times, most recently from 2d3d02c to c5173aa Compare July 27, 2026 18:06

connortsui20 marked this pull request as ready for review July 27, 2026 18:36

connortsui20 requested a review from onursatici July 27, 2026 18:37

myrrc previously approved these changes Jul 28, 2026

View reviewed changes

connortsui20 force-pushed the ct/benchmark-labels branch from c5173aa to 0b4eb99 Compare July 28, 2026 14:01

connortsui20 requested a review from myrrc July 28, 2026 14:03

connortsui20 force-pushed the ct/benchmark-labels branch from 0b4eb99 to dac3fb7 Compare July 28, 2026 14:34

connortsui20 requested review from 0ax1 and removed request for onursatici July 28, 2026 14:34

0ax1 reviewed Jul 28, 2026

View reviewed changes

connortsui20 force-pushed the ct/benchmark-labels branch 2 times, most recently from 4bbe37b to fb5f8aa Compare July 28, 2026 15:29

connortsui20 commented Jul 28, 2026

View reviewed changes

connortsui20 requested a review from 0ax1 July 28, 2026 15:31

connortsui20 force-pushed the ct/benchmark-labels branch from fb5f8aa to e332145 Compare July 28, 2026 16:45

connortsui20 added action/bench-random-access Run only the random-access benchmark on this PR action/bench-sql-compact Run only Vortex Compact SQL benchmarks on this PR labels Jul 29, 2026

github-actions Bot removed action/bench-sql-compact Run only Vortex Compact SQL benchmarks on this PR action/bench-random-access Run only the random-access benchmark on this PR labels Jul 29, 2026

connortsui20 marked this pull request as draft July 29, 2026 14:27

connortsui20 added action/bench-random-access Run only the random-access benchmark on this PR action/bench-sql-compact Run only Vortex Compact SQL benchmarks on this PR labels Jul 29, 2026

github-actions Bot removed action/bench-random-access Run only the random-access benchmark on this PR action/bench-sql-compact Run only Vortex Compact SQL benchmarks on this PR labels Jul 29, 2026

connortsui20 requested a review from AdamGS July 29, 2026 15:22

connortsui20 force-pushed the ct/benchmark-labels branch from 91df6c0 to 875a4b5 Compare July 29, 2026 16:30

connortsui20 added action/bench-random-access Run only the random-access benchmark on this PR action/bench-sql-compact Run only Vortex Compact SQL benchmarks on this PR labels Jul 29, 2026

connortsui20 marked this pull request as ready for review July 29, 2026 17:13

github-actions Bot removed action/bench-random-access Run only the random-access benchmark on this PR action/bench-sql-compact Run only Vortex Compact SQL benchmarks on this PR labels Jul 29, 2026

connortsui20 requested a review from joseph-isaacs July 30, 2026 15:31

connortsui20 force-pushed the ct/benchmark-labels branch from 875a4b5 to 0cc5bf5 Compare July 30, 2026 16:52

myrrc removed their request for review July 31, 2026 15:00

connortsui20 added 5 commits July 31, 2026 13:19

ci: add focused PR benchmark labels

a0a5b39

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

ci: make benchmark comparisons reliable

4f974d3

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

ci: add all CPU benchmarks label

b79ebea

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

fix: ignore zero-byte benchmark files

cc653c4

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

ci: include string benchmark in PR labels

b1b5ca5

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

connortsui20 force-pushed the ct/benchmark-labels branch from 1247c00 to b1b5ca5 Compare July 31, 2026 17:24

Uh oh!

Conversation

connortsui20 commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

connortsui20 commented Jul 27, 2026

Uh oh!

connortsui20 commented Jul 27, 2026

Uh oh!

codspeed-hq Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

myrrc commented Jul 28, 2026

Uh oh!

0ax1 left a comment

Choose a reason for hiding this comment

Uh oh!

0ax1 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

connortsui20 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

0ax1 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

connortsui20 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

connortsui20 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks: TPC-H SF=10 on S3 📖

Uh oh!

github-actions Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks: Statistical and Population Genetics 📖

Uh oh!

github-actions Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks: Clickbench on NVME 📖

Uh oh!

github-actions Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks: Appian on NVME 📖

Uh oh!

connortsui20 commented Jul 29, 2026

Uh oh!

github-actions Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks: Random Access 📖

Uh oh!

connortsui20 commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

connortsui20 commented Jul 27, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 27, 2026 •

edited

Loading

github-actions Bot commented Jul 29, 2026 •

edited

Loading

github-actions Bot commented Jul 29, 2026 •

edited

Loading

github-actions Bot commented Jul 29, 2026 •

edited

Loading

github-actions Bot commented Jul 29, 2026 •

edited

Loading

github-actions Bot commented Jul 29, 2026 •

edited

Loading