Metrics points and tags/dyn filter enabled

.claude/skills/ec2-cluster-provision/SKILL.md

@@ -0,0 +1,62 @@
+---
+name: ec2-cluster-provision
+description: uses the code present in this repository for provision an EC2 cluster for benchmarking purposes
+---
+
+This project uses a remote benchmarks EC2 cluster constructed with AWS CDK located at `benchmarks/cdk`.
+
+There's a package.json file in `benchmarks/cdk/package.json` with relevant commands about deploying.
+
+All the commands in this skill need to be prefixed with whatever the user declared in `./claude/settings.local.json`
+in the `aws-commands-prefix` key, typically for providing the commands with the correct permissions.
+(e.g., `$aws-commands-prefix npm run cdk deploy` or `$aws-commands-prexfix aws ssm ...`)
+
+Running `npm run cdk deploy` will provision the cluster with the resources specified in `benchmarks/cdk/lib/`.
+This takes a while typically (~5 mins). If the user data of the EC2 machines was changed, and you want those changes
+to take effect you will need to prepend the deployment command with `USER_DATA_CAUSES_REPLACEMENT=true`.
+
+Once the deployment is complete, the list of instance IDs will be printed to stdout.
+
+It's usually necessary to verify that everything was deployed correctly, and it's running fine. For that
+it's necessary to perform the following steps for the following engines:
+
+## Distributed DataFusion
+
+1. Port forward the 9000 port in a background terminal:
+   `aws ssm start-session --target $INSTANCE_ID --document-name AWS-StartPortForwardingSession --parameters "portNumber=9000,localPortNumber=9000"`
+2. Issue a command to /info to see what was deployed:
+   `curl http://localhost:9000/info | jq .`
+3. If everything's fine, you should see the list of listening workers and the build time
+
+## Trino
+
+1. Port forward the 8080 port in a background terminal:
+   `aws ssm start-session --target $INSTANCE_ID --document-name AWS-StartPortForwardingSession --parameters "portNumber=8080,localPortNumber=8080"`
+2. Issue a command to /v1/node to see what nodes are available for listening:
+   `curl -s -H "X-Trino-User: admin" http://localhost:8080/v1/node | jq .`
+   `curl -s -H "X-Trino-User: admin" http://localhost:8080/v1/info | jq .`
+3. Make sure that the response of the above is consistent with what is supposed to be deployed by
+   `benchmarks/cdk/lib/trino.ts`
+
+## Spark
+
+1. Port forward the 9003 port in a background terminal (this is a custom Python server `benchmarks/cdk/bin/spark_http.py`):
+   `aws ssm start-session --target $INSTANCE_ID --document-name AWS-StartPortForwardingSession --parameters "portNumber=9003,localPortNumber=9003"`
+2. You can issue curl queries to `http://localhost:9003/health` and `http://localhost:9003/query` to double-check that
+   everything is consistent with what's expected from `benchmarks/cdk/lib/spark.ts`
+
+## Ballista
+
+1. Port forward the 9002 port in a background terminal:
+   `aws ssm start-session --target $INSTANCE_ID --document-name AWS-StartPortForwardingSession --parameters "portNumber=9002,localPortNumber=9002"`
+2. Verify that there are in fact the expected workers connected to the scheduler. For this, all workers need to
+   have the appropriate instance IP configured in their launch command, which might not have been done by default,
+   because this happens in a post cloud-init command that might not run because of timeout.
+3. Verify with systemctl in each remote machine that all workers are running successfully. One thing that happens
+   very often is that `--external-host` is incorrectly set to localhost rather than the actual scheduler EC2 private IP
+
+Remember that for running port forward commands in the background, they take like 5 secs until the
+"waiting for connections" message appears. Until then, the port is still not forwarded.
+
+If at some point you need to run a command in all machines and get its output, you can do it
+with `npm run send-command your custom command`

.claude/skills/remote-benchmark/SKILL.md

@@ -0,0 +1,85 @@
+---
+name: remote-benchmark
+description: deploys the code to a remote EC2 cluster with the commands available in the package.json, port-forwards
+ a machine port, and runs benchmarks against it.
+---
+
+This project uses a remote benchmarks EC2 cluster constructed with AWS CDK located at `benchmarks/cdk`.
+
+There's a package.json file in `benchmarks/cdk/package.json` with relevant commands about benchmarking.
+
+All the commands in this skill need to be prefixed with whatever the user declared in `./claude/settings.local.json`
+in the `aws-commands-prefix` key, typically for providing the commands with the correct permissions.
+(e.g., `$aws-commands-prefix npm run fast-deploy` or `$aws-commands-prexfix aws ssm ...`)
+
+You can assume that the cluster is already there, and the only thing necessary is to execute the `npm run fast-deploy`
+command for deploying the current code to the EC2 cluster. Remember that all npm commands need to be run from the
+`benchmarks/cdk` folder.
+
+The `npm run fast-deploy` command will compile the current code and deploy it to the EC2 machines. If it fails,
+prompt the user to fix it. It will output several EC2 instance IDs: pick the first one, that's the one we will port
+forward locally in order to issue queries to it.
+
+Once the deployment is completed, port forward the first instance id to the local port 9000:
+
+```shell
+aws ssm start-session --target i-0000000000000 --document-name AWS-StartPortForwardingSession --parameters "portNumber=9000,localPortNumber=9000"
+```
+
+Remember to run that in the background, as that will block in place.
+
+Once the port is correctly listening locally (you will see a "waiting for connections" message), it's fine to start
+the benchmarks.
+
+You can start the benchmarks with `npm run datafusion-bench`, you can learn how this command works by running:
+
+```shell
+$ npm run datafusion-bench -- --help
+
+Usage: datafusion-bench [options]
+
+Options:
+  --dataset <string>                        Dataset to run queries on
+  -i, --iterations <number>                 Number of iterations (default: "3")
+  --bytes-processed-per-partition <number>  How many bytes each partition is expected to process (default: "134217728")
+  --batch-size <number>                     Standard Batch coalescing size (number of rows) (default: "32768")
+  --shuffle-batch-size <number>             Shuffle batch coalescing size (number of rows) (default: "32768")
+  --children-isolator-unions <number>       Use children isolator unions (default: "true")
+  --broadcast-joins <boolean>               Use broadcast joins (default: "false")
+  --collect-metrics <boolean>               Propagates metric collection (default: "true")
+  --compression <string>                    Compression algo to use within workers (lz4, zstd, none) (default: "lz4")
+  --queries <string>                        Specific queries to run
+  --debug <boolean>                         Print the generated plans to stdout
+  -h, --help                                display help for command
+```
+
+The --dataset command is mandatory, and its value can be any of the folder names in `benchmarks/data`, for example:
+clickbench_0-100, tpcds_sf1, tpch_sf1, tpch_sf10 or tpch_sf100.
+
+Also, the --queries argument can be used for executing just a partial subset of queries, for example:
+```shell
+--queries q1,q2,q3
+```
+
+When benchmarking a very specific feature, it's convenient to choose wisely a relevant query and just execute that one.
+
+The user provided the following arguments: $ARGUMENTS
+
+parse those and make sure you parse them correctly, for example `tpch_sf100 q1,q2,q4` means 
+`--dataset tpch_sf100 --queries q1,q2,q4`. Note that the user might also give natural language instructions in the 
+arguments, be smart while parsing those.
+
+### analyzing results
+
+results for individual queries will be dumped in the respective dataset folders, for example:
+
+`benchmarks/data/tpch_sf10/.results-remote/datafusion-distributed-main/q1.json`
+or
+`benchmarks/data/tpch_sf1/.results-remote/datafusion-distributed-new-branch/q2.json`
+
+You can inspect the results and the plan by reading the JSONs. Tip: use jq for printing nice results.
+
+As the results of previous branches are already stored in disk, they usually can be analyzed without re-running them
+again, that can be done by either:
+- Just looking at the latencies and plans in the output folders.
+- Running the `npm run datafusion-bench -- compare --dataset tpch_sfX datafusion-distributed-<base branch> datafusion-distributed-<compare branch>`

.gitattributes

@@ -1 +1,2 @@
 *.parquet filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

.github/actions/setup/action.yml

@@ -12,6 +12,9 @@ inputs:
 runs:
   using: 'composite'
   steps:
+    - name: Free up disk space
+      shell: bash
+      run: ${{ github.action_path }}/free_disk_space.sh
     - shell: bash
       run: |
         echo $HOME
@@ -26,3 +29,4 @@ runs:
     - uses: Swatinem/rust-cache@v2
       with:
         key: ${{ runner.os }}-${{ inputs.targets }}-rust-cache
+        prefix-key: v1-rust

.github/actions/setup/free_disk_space.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+# Script to free up disk space on CI systems
+# Source: https://github.com/apache/flink/blob/02d30ace69dc18555a5085eccf70ee884e73a16e/tools/azure-pipelines/free_disk_space.sh
+
+echo "Freeing up disk space on CI system"
+
+echo "Listing 100 largest packages"
+dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n | tail -n 100
+df -h
+
+echo "Removing large packages"
+sudo apt-get remove -y '^ghc-8.*'
+sudo apt-get remove -y '^dotnet-.*'
+sudo apt-get remove -y '^llvm-.*'
+sudo apt-get remove -y 'php.*'
+sudo apt-get remove -y azure-cli google-cloud-sdk hhvm google-chrome-stable firefox powershell mono-devel
+sudo apt-get autoremove -y
+sudo apt-get clean
+df -h
+
+echo "Removing large directories"
+rm -rf /usr/share/dotnet/
+df -h

.github/workflows/ci.yml

@@ -9,20 +9,28 @@ on:
 env:
   CARGO_TERM_COLOR: always
   RUST_BACKTRACE: 1
+  TPCDS_SCALE_FACTOR: 0.5 # 0.5 scale factor produces a data dir which is 124MB
 
 concurrency:
   group: ${{ github.ref }}
   cancel-in-progress: true
 
 jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: ./.github/actions/setup
+      - run: cargo build
+
   clippy:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
       - uses: ./.github/actions/setup
         with:
           components: clippy
-      - run: cargo clippy --all-targets --features integration
+      - run: cargo clippy --all-targets --all-features -- -D warnings
 
   unit-test:
     runs-on: ubuntu-latest
@@ -38,7 +46,29 @@ jobs:
     steps:
       - uses: actions/checkout@v4
       - uses: ./.github/actions/setup
-      - run: cargo test --features tpch --test tpch_validation_test
+      - run: cargo test --features tpch --test 'tpch_*'
+
+  tpcds-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: ./.github/actions/setup
+      - uses: actions/cache@v4
+        with:
+          path: testdata/tpcds/main.zip
+          key: "main.zip"
+      - run: cargo test --features tpcds --test 'tpcds_*'
+
+  clickbench-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: ./.github/actions/setup
+      - uses: actions/cache@v4
+        with:
+          path: testdata/clickbench/
+          key: "data"
+      - run: cargo test --features clickbench --test 'clickbench_*'
 
   format-check:
     runs-on: ubuntu-latest

.github/workflows/docs.yml

@@ -0,0 +1,55 @@
+name: Deploy Documentation
+
+on:
+  push:
+    branches:
+      - main
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+concurrency:
+  group: "pages"
+  cancel-in-progress: false
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          lfs: true
+
+      - name: Setup Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: '3.x'
+
+      - name: Install dependencies
+        run: |
+          pip install -r docs/requirements.txt
+
+      - name: Build documentation
+        run: |
+          cd docs
+          make html
+
+      - name: Upload artifact
+        uses: actions/upload-pages-artifact@v4
+        with:
+          path: docs/build/html
+
+  deploy:
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    runs-on: ubuntu-latest
+    needs: build
+    steps:
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4

.gitignore

@@ -1,4 +1,12 @@
 /.idea
 /target
 /benchmarks/data/
-testdata/tpch/data/
+testdata/tpch/*
+!testdata/tpch/queries
+testdata/tpcds/*
+!testdata/tpcds/queries
+!testdata/tpcds/README.md
+testdata/clickbench/*
+!testdata/clickbench/queries
+src/observability/gen/target/*
+src/observability/gen/Cargo.lock