Skip to content

feat(perf): migrate python perf image config from test-plans #808#37

Merged
seetadev merged 13 commits into
masterfrom
migrate/pr808-python-perf
Jul 7, 2026
Merged

feat(perf): migrate python perf image config from test-plans #808#37
seetadev merged 13 commits into
masterfrom
migrate/pr808-python-perf

Conversation

@acul71

@acul71 acul71 commented Feb 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

  • Migrate the effective changes from libp2p/test-plans#808 into libp2p/unified-testing.
  • Add python-v0.x to perf (perf/images.yaml) pinned to py-libp2p@abb5b34d (QUIC fix #1346).
  • Harness: compose project isolation, --timeout, PY_YAMUX_* injection for python containers.
  • CI: python-aware timeout (900s), PY_YAMUX_* defaults, skip baseline builds on impl-only PRs.

Context

Test plan

  • Run perf with --test-select "~python".
  • Verify python image builds and participates in selected perf matrix entries.
  • Validate known failing combinations remain correctly represented by aliases/filters (quic-v1 excluded from python transports for now).

CI notes

  • Impl-only perf PRs skip ~baselines to avoid heavy Quinn rebuilds on self-hosted runners.
  • Python PRs use --timeout 900 and PY_YAMUX_DISABLE_HYSTERESIS=1 / PY_YAMUX_ASSUME_RTT_MS=1.

@acul71
wip: python perf
7356427 
(cherry picked from commit 285b6d3b7d3e4004e19975ae4426a07b27dd7d5f)
@acul71 acul71 requested a review from dhuseby as a code owner February 24, 2026 02:48
@acul71

acul71 commented Feb 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Hi @seetadev, this was the same on the old test-plans

I checked the failing run and pulled the artifacts/logs for run-tests in this job:
https://github.com/libp2p/unified-testing/actions/runs/22334473417/job/64623801343?pr=37

Result summary

  • Total perf tests: 9
  • Passed: 3
  • Failed: 6

Passing cases

  • python-v0.x x python-v0.x (tcp, noise, mplex)
  • python-v0.x x python-v0.x (tcp, tls, mplex)
  • python-v0.x x python-v0.x (ws, noise, mplex)

Failing cases

  • python-v0.x x python-v0.x (tcp, noise, yamux) → timeout
  • python-v0.x x python-v0.x (tcp, tls, yamux) → timeout
  • python-v0.x x python-v0.x (ws, noise, yamux) → timeout
  • python-v0.x x python-v0.x (ws, tls, yamux) → timeout
  • python-v0.x x python-v0.x (ws, tls, mplex) → timeout
  • python-v0.x x python-v0.x (quic-v1) → runtime exception

Direct failure reason

  • The test runner enforces a hard 300s timeout (run-single-test.sh), and 5 scenarios hit that timeout (other then mplex are too slow)
  • QUIC fails with:
    • ValueError: Expected to receive 1073741824 bytes, but received 0

So this is a genuine test-execution failure (timeouts + one QUIC runtime error), not an artifact upload/workflow issue.

@acul71 acul71 marked this pull request as draft February 25, 2026 03:12
@dhuseby

dhuseby commented Feb 27, 2026

Copy link
Copy Markdown
Contributor

The test runner enforces a hard 300s timeout (run-single-test.sh), and 5 scenarios hit that timeout (other then mplex are too slow)

It sounds like that is incorrect. We shouldn't fail if an impl is super slow and it times out. However 300s is 5 minutes...any idea what is taking so long? I'm ok with doubling this but if that's not long enough I think py-libp2p needs some optimization work.

@acul71

acul71 commented Feb 28, 2026

Copy link
Copy Markdown
Collaborator Author

any idea what is taking so long?

Yamux is awfully slow in python. Will look in details why is that.
Same for ws + tls
For quic-v1 it seems there is some protocol error during the transfer

@dhuseby

dhuseby commented Feb 28, 2026

Copy link
Copy Markdown
Contributor

@acul71 the issues are deep in python. I ran the perf test with ./run.sh --impl-ignore '!python-v0.x' --upload-bytes 1048576 --download-bytes 1048576 --yes so that the transfer is just 1 MB instead of 1 GB and a timeout of 600 seconds instead of 300 and I'm still seeing errors.

  → [3/9] python-v0.x x python-v0.x (tcp, tls, yamux)...[FAILED]
  → [1/9] python-v0.x x python-v0.x (tcp, noise, yamux)...[FAILED]
  → [2/9] python-v0.x x python-v0.x (tcp, noise, mplex)...[SUCCESS]
  → [3/9] python-v0.x x python-v0.x (tcp, tls, yamux)...[FAILED]
  → [4/9] python-v0.x x python-v0.x (tcp, tls, mplex)...[SUCCESS]
  → [5/9] python-v0.x x python-v0.x (ws, noise, yamux)...[FAILED]
  → [6/9] python-v0.x x python-v0.x (ws, noise, mplex)...[SUCCESS]
  → [7/9] python-v0.x x python-v0.x (ws, tls, yamux)...[FAILED]
  → [8/9] python-v0.x x python-v0.x (ws, tls, mplex)...[SUCCESS]
  → [9/9] python-v0.x x python-v0.x (quic-v1)...[FAILED]

Here is the (tcp, tls, yamux) log file:

[2026-02-27 13:34:36] INFO: [3] python-v0.x x python-v0.x (tcp, tls, yamux) (key: daae8327)
[2026-02-27 13:34:36] INFO: Running: python-v0.x x python-v0.x (tcp, tls, yamux)
time="2026-02-27T13:34:36-07:00" level=warning msg="No services to build"
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__listener Creating 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__listener Created 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer Creating 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer Created 
Attaching to python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer, python-v0_x_x_python-v0_x__tcp__tls__yamux__listener
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__listener Starting 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__listener Started 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer Starting 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer Started 
python-v0_x_x_python-v0_x__tcp__tls__yamux__listener  | Connecting to Redis...
python-v0_x_x_python-v0_x__tcp__tls__yamux__listener  | Connected to Redis on attempt 1
python-v0_x_x_python-v0_x__tcp__tls__yamux__listener  | Connecting to Redis...
python-v0_x_x_python-v0_x__tcp__tls__yamux__listener  | Connected to Redis on attempt 1
python-v0_x_x_python-v0_x__tcp__tls__yamux__listener  | Perf service started (protocol /perf/1.0.0)
python-v0_x_x_python-v0_x__tcp__tls__yamux__listener  | Publishing address: /ip4/10.8.0.3/tcp/35325/p2p/12D3KooWPKT6n54FpMsjvKRFzy3AnEWywYmk222wxtx8pPELXNqW
python-v0_x_x_python-v0_x__tcp__tls__yamux__listener  | Listener ready, waiting for dialer...
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Connecting to Redis...
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Connected to Redis on attempt 1
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Connecting to Redis...
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Connected to Redis on attempt 1
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Waiting for listener address...
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Got listener address: /ip4/10.8.0.3/tcp/35325/p2p/12D3KooWPKT6n54FpMsjvKRFzy3AnEWywYmk222wxtx8pPELXNqW
python-v0_x_x_python-v0_x__tcp__tls__yamux__listener  | TLS inbound: no peer cert (Python ssl limitation)
python-v0_x_x_python-v0_x__tcp__tls__yamux__listener  | TLS inbound: using placeholder remote peer ID
python-v0_x_x_python-v0_x__tcp__tls__yamux__listener  | TLS secure_inbound: temporary peer ID: 12D3KooWReweEZtn3HxxKxxsECwGqtFc8Ew4bjzCfWEEt1csgiKP
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Connected to listener
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Upload iteration 1/10: 0.16 Gbps
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Upload iteration 2/10: 0.16 Gbps
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Upload iteration 3/10: 0.16 Gbps
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Upload iteration 4/10: 0.16 Gbps
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Upload iteration 5/10: 0.16 Gbps
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Upload iteration 6/10: 0.16 Gbps
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Upload iteration 7/10: 0.16 Gbps
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Upload iteration 8/10: 0.16 Gbps
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Upload iteration 9/10: 0.16 Gbps
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Upload iteration 10/10: 0.16 Gbps
python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer    | Download iteration 1/10: 0.16 Gbps
 Compose Stopping Gracefully Stopping... press Ctrl+C again to force
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer Stopping 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer Stopped 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__listener Stopping 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__listener Stopped 

[2026-02-27 13:44:46] ERROR:   ✗ Test timed out after 600s

[2026-02-27 13:44:46] ERROR: Test timed out after 600 seconds
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer Stopping 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer Stopped 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer Removing 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__dialer Removed 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__listener Stopping 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__listener Stopped 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__listener Removing 
 Container python-v0_x_x_python-v0_x__tcp__tls__yamux__listener Removed

So you can see that with 1 MB transfers it completes all 10 iterations and then times out anyway. Something else is going on other than just being slow.

@acul71
Merge remote-tracking branch 'origin/master' into migrate/pr808-pytho…
5b73421 
…n-perf

Made-with: Cursor

# Conflicts:
#	perf/images.yaml
@acul71

acul71 commented Mar 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

@acul71 the issues are deep in python. I ran the perf test with ./run.sh --impl-ignore '!python-v0.x' --upload-bytes 1048576 --download-bytes 1048576 --yes so that the transfer is just 1 MB instead of 1 GB and a timeout of 600 seconds instead of 300 and I'm still seeing errors.

@seetadev

I've just discovered that increasing the Yamux DEFAULT_WINDOW_SIZE I was able to make the test pass, with similar mplex throughoutput.

# Larger default window reduces flow-control churn on high-throughput links.
DEFAULT_WINDOW_SIZE = 16 * 1024 * 1024

But I discovered that number with testing, I don't know what the full implications are of this mod.

For quic the test is failing for problems in py-libp2p quic protocol implementation, investigation in course.

I'm going to review the Yamux module more in depth, in next days, before submitting the fix. I can eventually leave the DEFAULT as it is, and pass a specific parameter to modify the DEFAULT for perf test

About quic I'm oriented in removing this protocol, for now and put it back later when it will work properly.

acul71 added 2 commits March 6, 2026 23:18
@acul71
Merge remote-tracking branch 'origin/master' into migrate/pr808-pytho…
cb6e6bc 
…n-perf
@acul71
fix perf python image pin and compose project isolation
34f3be7 
Pin perf Python tests to the intended py-libp2p snapshot and avoid docker compose project/container name collisions between reruns by using a per-test-pass project suffix.

Made-with: Cursor
@acul71

acul71 commented Mar 7, 2026

Copy link
Copy Markdown
Collaborator Author

perf/lib/run-single-test.sh was updated to make Docker Compose project/container names unique per test pass.

Why this change was needed:

  • The previous naming reused the same compose project/container names for identical test selections across reruns.
  • On reruns, Docker Compose could try to Recreate stale references from an older run and fail with errors like No such container.
  • This produced infra-style false negatives (test fails before real protocol/perf execution).

What the fix does:

  • Derives a per-run suffix from TEST_PASS_NAME + TEST_NAME.
  • Uses that suffix in compose project name and explicit container names.
  • Keeps test behavior/metrics unchanged; it only isolates compose runtime state between runs.

Result:

  • Re-running the same selected test no longer collides with stale compose state from prior runs.

@acul71

acul71 commented Mar 7, 2026

Copy link
Copy Markdown
Collaborator Author

In CI/CD this test is failing for timeout

python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Connected to listener
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Upload 1/10: 0.43 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Upload 2/10: 0.43 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Upload 3/10: 0.43 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Upload 4/10: 0.43 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Upload 5/10: 0.43 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Upload 6/10: 0.43 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Upload 7/10: 0.43 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Upload 8/10: 0.43 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Upload 9/10: 0.43 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Upload 10/10: 0.43 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Download 1/10: 0.49 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Download 2/10: 0.50 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Download 3/10: 0.49 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Download 4/10: 0.49 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__4a514541_dialer    | Download 5/10: 0.50 Gbps
Gracefully stopping... (press Ctrl+C again to force)

in my linux-box works (faster hardware I think)

python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Connected to listener
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Upload 1/10: 1.05 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Upload 2/10: 1.10 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Upload 3/10: 1.07 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Upload 4/10: 1.04 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Upload 5/10: 1.06 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Upload 6/10: 0.99 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Upload 7/10: 0.91 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Upload 8/10: 0.99 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Upload 9/10: 1.01 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Upload 10/10: 1.07 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Download 1/10: 1.23 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Download 2/10: 1.23 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Download 3/10: 1.26 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Download 4/10: 1.22 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Download 5/10: 1.28 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Download 6/10: 1.23 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Download 7/10: 1.21 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Download 8/10: 1.28 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Download 9/10: 1.23 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Download 10/10: 1.25 Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | Latency iterations done
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | upload:
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   iterations: 10
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   min: 0.91
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   q1: 1.00
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   median: 1.05
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   q3: 1.07
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   max: 1.10
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   outliers: []
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   samples: [0.91, 0.99, 0.99, 1.01, 1.04, 1.05, 1.06, 1.07, 1.07, 1.1]
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   unit: Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | download:
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   iterations: 10
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   min: 1.21
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   q1: 1.23
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   median: 1.23
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   q3: 1.26
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   max: 1.28
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   outliers: []
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   samples: [1.21, 1.22, 1.23, 1.23, 1.23, 1.23, 1.25, 1.26, 1.28, 1.28]
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   unit: Gbps
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    | latency:
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   iterations: 100
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   min: 1.310
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   q1: 1.804
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   median: 2.266
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   q3: 2.831
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   max: 4.360
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   outliers: [4.952]
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   samples: [1.31, 1.322, 1.34, 1.345, 1.365, 1.377, 1.378, 1.393, 1.468, 1.481, 1.514, 1.567, 1.625, 1.635, 1.635, 1.652, 1.653, 1.669, 1.697, 1.739, 1.739, 1.757, 1.77, 1.775, 1.782, 1.811, 1.831, 1.855, 1.873, 1.904, 1.923, 1.936, 1.937, 1.956, 1.96, 2.005, 2.005, 2.006, 2.062, 2.178, 2.189, 2.191, 2.217, 2.221, 2.229, 2.235, 2.249, 2.252, 2.262, 2.265, 2.268, 2.282, 2.283, 2.297, 2.322, 2.326, 2.346, 2.36, 2.366, 2.384, 2.4, 2.477, 2.486, 2.487, 2.51, 2.519, 2.549, 2.577, 2.578, 2.587, 2.66, 2.746, 2.759, 2.762, 2.83, 2.833, 2.847, 2.908, 2.952, 2.958, 2.996, 3.083, 3.099, 3.12, 3.121, 3.143, 3.212, 3.221, 3.247, 3.319, 3.402, 3.421, 3.66, 3.669, 3.735, 3.959, 4.103, 4.187, 4.36, 4.952]
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer    |   unit: ms
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer exited with code 0
 Compose Stopping Aborting on container exit...
 Container python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer Stopping 
 Container python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer Stopped 
 Container python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_listener Stopping 
 Container python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_listener Stopped 
python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_listener exited with code 143
[2026-03-07 04:40:13] INFO:   ✓ Test complete
 Container python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer Stopping 
 Container python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer Stopped 
 Container python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer Removing 
 Container python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_dialer Removed 
 Container python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_listener Stopping 
 Container python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_listener Stopped 
 Container python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_listener Removing 
 Container python-v0_x_x_python-v0_x__ws__tls__mplex__216ffd9a_listener Removed

@acul71 acul71 mentioned this pull request Mar 7, 2026
Draft
3 tasks
@dhuseby

dhuseby commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

@acul71 it seems like you and @sumanjeet0012 are trying to do the same thing. @sumanjeet0012 has this PR: #52

I think you two should combine efforts.

acul71 and others added 4 commits May 26, 2026 16:54
@acul71
new yamux window fix
3505c4f
@acul71 @cursoragent
Merge branch 'master' into migrate/pr808-python-perf
e11e8f4 
Co-authored-by: Cursor <cursoragent@cursor.com>
@acul71 @cursoragent
fix: normalize perf implementation change detection
1d6f75d 
Compare perf implementation definitions as normalized JSON so comment-only edits do not mark unrelated implementations as changed, while keeping the local Python example in place in perf/images.yaml.

Co-authored-by: Cursor <cursoragent@cursor.com>
@acul71 @cursoragent
fix: bump python perf image commit
64646bd 
Update the pinned py-libp2p perf source to the lighter-image commit so CI builds the smaller Python perf Docker image.

Co-authored-by: Cursor <cursoragent@cursor.com>
@acul71

acul71 commented May 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

@seetadev @sumanjeet0012
Locally on my linux box the python x python perf test pass
Here it can't build the python docker image because it runs out of space
Can someone help ?
Thanks ;)

@seetadev

seetadev commented May 28, 2026

Copy link
Copy Markdown
Collaborator

@seetadev @sumanjeet0012 Locally on my linux box the python x python perf test pass Here it can't build the python docker image because it runs out of space Can someone help ? Thanks ;)

@acul71 : Definitely, will help you in this effort. This is fantastic. Thank you so much for fixing the yamux issues.

@acul71 @cursoragent
feat(perf): add harness --timeout and python PY_YAMUX_* injection
7753572 
Expose PERF_TEST_TIMEOUT_SECS via --timeout so slow stacks can run longer
than the default 300s cap. Forward PY_YAMUX_* tuning env vars into
python-v0.x containers only, bump the py-libp2p perf image commit, and
clarify timeout vs legacy --duration in the README.

Co-authored-by: Cursor <cursoragent@cursor.com>
@acul71

acul71 commented Jun 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

@seetadev @dhuseby @sumanjeet0012

CI failure: runner disk / Docker buildkit (not Python perf)

The perf workflow failed during Docker image build, before any tests ran.

Failed run: https://github.com/libp2p/unified-testing/actions/runs/27034698883

Root cause (from logs):

ERROR: failed to copy: ... buildkit/content/ingest/...: no space left on device
✗ Docker build failed
✗ Local build failed for quinn

Sequence:

  • perf-baselines-https, quic-go, iperf built OK (cached)
  • perf-baselines-quinn started (Rust cargo-chef base image + compile)
  • Self-hosted runner ran out of disk under /var/lib/docker
  • Python image build and all perf tests were never reached → no results.yaml

This is infra / CI scope, not a regression in the Python perf harness changes on this branch.

Suggested fixes (maintainers / CI)

  1. Runner hygiene — prune Docker/buildkit before perf PR jobs on self-hosted runners, e.g.: 
    docker builder prune -af --filter "until=24h" || true
docker system prune -af --filter "until=24h" || true
  2. PR workflow scope — for impl-only PRs (e.g. python-v0.x), skip ~default_baselines when changed-baselines is empty, so Quinn/https/quic-go/iperf are not rebuilt unless baselines actually changed.
  3. Re-run — ephemeral runners may pass on retry if disk is free on the next VM.

Local validation

Python self-tests were run locally on this branch (8/8 stacks passing in perf-dd24bf4a-061426-05-06-2026).

Happy to add a small perf-pr.yml / run-bash-perf-test patch to skip baselines on impl-only PRs or add a pre-build prune step if that helps.

@acul71 @cursoragent
Merge master and implement PR #37 review changes
05e34cd 
- Merge origin/master into migrate/pr808-python-perf
- Pin python-v0.x to py-libp2p abb5b34d (QUIC fix #1346)
- Preserve eth-p2p-z and nim in perf images alias
- DRY PY_YAMUX env injection in run-single-test.sh
- Wire --timeout and PY_YAMUX_* into perf PR CI
- Skip baseline builds on impl-only perf PRs

Co-authored-by: Cursor <cursoragent@cursor.com>
@acul71 acul71 marked this pull request as ready for review July 2, 2026 15:25
acul71 and others added 2 commits July 2, 2026 18:04
@acul71 @cursoragent
fix(perf): mark flaky python tls/mplex stacks in failing alias
8f5c3b6 
Document known py-libp2p perf flakes so PR CI (~failing ignore) runs
the stable noise stacks while tls/mplex issues are tracked upstream.

Co-authored-by: Cursor <cursoragent@cursor.com>
@acul71 @cursoragent
perf(python): unmask TLS stacks after py-libp2p TLS fix
b98168d 
Pin py-libp2p to 34d7bcda (fix/tls-libp2p-extension-verify) and remove
python TLS entries from ~failing. Validated locally: 8/8 python×python
perf stacks pass with --force-image-rebuild.

Related: libp2p/py-libp2p#1367 libp2p/py-libp2p#1368
Co-authored-by: Cursor <cursoragent@cursor.com>
@acul71

acul71 commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

Validation update — python×python perf (8/8 pass)

Local (--force-image-rebuild, timeout 900s)

  • Result: 8/8 pass (perf-08c8be91-140205-07-07-2026)
  • Wall time: ~17 min
  • py-libp2p pin: 34d7bcda (py-libp2p#1368 — TLS PKIX skip + libp2p extension verify)
Stack Upload (median) Download (median) Latency (median)
tcp, noise, yamux 0.45 Gbps 0.44 Gbps 6.7 ms
tcp, noise, mplex 0.82 Gbps 0.83 Gbps 5.2 ms
tcp, tls, yamux 0.39 Gbps 0.39 Gbps 6.3 ms
tcp, tls, mplex 0.50 Gbps 0.50 Gbps 4.7 ms
ws, noise, yamux 0.21 Gbps 0.23 Gbps 8.5 ms
ws, noise, mplex 0.31 Gbps 0.37 Gbps 6.0 ms
ws, tls, yamux 0.19 Gbps 0.22 Gbps 8.5 ms
ws, tls, mplex 0.29 Gbps 0.34 Gbps 5.6 ms

All four TLS stacks pass without TLSV1_ALERT_UNKNOWN_CA after unmasking from ~failing.

CI (run #28865769767)

  • Result: 8/8 pass (perf-007bc62b-122302-07-07-2026)
  • Wall time: 1h 14m (self-hosted runner, single worker, cold image build)
  • Per-test harness timeout: 900s (python-specific CI override)

Per-test wall times on CI (well above the 300s default — only tcp,noise,mplex finished under 300s):

# Stack ~Duration
1 tcp, noise, yamux 404s
2 tcp, noise, mplex 218s
3 tcp, tls, yamux 448s
4 tcp, tls, mplex 356s
5 ws, noise, yamux 793s
6 ws, noise, mplex 519s
7 ws, tls, yamux 852s
8 ws, tls, mplex 564s

Throughput on CI matches local (same ballpark; WS stacks slower than TCP as expected).

Timeout design

  • Harness default: 300s (PERF_TEST_TIMEOUT_SECS in perf/run.sh)
  • CI: 900s when changed impls include python, 300s otherwise (.github/workflows/perf-pr.yml resolve step)
  • Local docs: perf/README.md documents --timeout (default 300, example 900 for slow stacks)

Follow-up

  • QUIC (quic-v1) for python perf is still excluded from the matrix — will be addressed in a follow-up PR once py-libp2p QUIC interop is ready for perf.

Related

@seetadev

seetadev commented Jul 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

@acul71 : This is a great piece of work, nice job migrating the Python perf configuration while also improving the overall test harness.

I especially like the attention to CI reliability with the Python-specific timeout handling, compose project isolation, and skipping unnecessary baseline builds for impl-only PRs. The extensive investigation into the Yamux performance issues and the follow-up fixes in py-libp2p show a thorough debugging process, and it's great to see the final validation with all 8/8 Python perf stacks passing locally and in CI.

The changes are well documented, the rationale behind the timeout adjustments is clear, and the remaining QUIC work is appropriately scoped as follow-up.

LGTM! Thanks for driving this migration forward and for sticking with the debugging until the Python perf suite was green.

@sumanjeet0012 : Can you please check the self hosted runners. We see two CI/CD tests being skipped.

@acul71 and @sumanjeet0012 : Lets ensure that all tests are CI/CD tests passing.

@acul71

acul71 commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

This is a great piece of work, nice job migrating the Python perf configuration while also improving the overall test harness.

Thanks @seetadev

The two skipped run-tests jobs are Transport Interop (PR) and Hole Punch Interop (PR) — not perf. This PR only changes perf/ (and related harness/CI wiring), so detect-changed-impls correctly sets should-run-tests=false for transport and hole-punch. Those workflows still run resolve-parameters, but skip the heavy test matrix when no impls in their scope changed.

The perf workflow ran to completion: 8/8 python×python stacks passed (run #28865769767).

@seetadev

seetadev commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

@acul71 : Awesome, this is fantastic to hear. Appreciate your efforts. This is indeed an important milestone on perf module. This PR is indeed ready for final review + merge :)

Very happy to see all the interop tests passing.

CCing @PoulavBhowmick03, who works on key Ethereum consensus clients including Lighthouse and has played a key role in so many awesome rust@Ethereum initiatives.

Had a discussion with him in details yesterday. We will utilize this as a base and extend this to discover performance measurements in rust-libp2p and understand the landscape of lean Ethereum consensus clients vs. regular Ethereum consensus clients in rust.

Also, CCing @johannamoran and @mishmosh. Keeping them in sync with this update.

@seetadev seetadev merged commit 0469a5b into master Jul 7, 2026
6 checks passed
@seetadev seetadev deleted the migrate/pr808-python-perf branch July 7, 2026 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dhuseby dhuseby Awaiting requested review from dhuseby dhuseby is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@acul71 @dhuseby @seetadev