From d6b4216f45556b34be63e37cdd64aad6ca11315a Mon Sep 17 00:00:00 2001 From: Sergei Ozeranskii Date: Wed, 24 Jun 2026 13:48:08 +0200 Subject: [PATCH 1/2] fix: run django case study via django's own CI steps (incl. libmemcached-dev) django 6.0.6 ships no uv.lock; its Linux CI (python_matrix.yml) runs the suite via pip + runtests.py and installs libmemcached-dev itself, because pylibmc has no wheels for Python 3.12+ and the runner images don't bundle the headers. Replicate those exact steps (identical on both runners) instead of the earlier uv/tox attempt (which was flask's pattern and fails on django for lack of a lockfile). --- .github/workflows/casestudy-django.yml | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/.github/workflows/casestudy-django.yml b/.github/workflows/casestudy-django.yml index ecfae88..5b0fe79 100644 --- a/.github/workflows/casestudy-django.yml +++ b/.github/workflows/casestudy-django.yml @@ -31,10 +31,16 @@ jobs: - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 with: python-version: "3.14" - # django's own test command at this pin. Uses pip (not uv) as django ships no uv lockfile. - # SQLite default — no DB service, no secrets. --parallel exercises all 4 vCPU. - # No action-level cache: this is the honest cold path (warm = a primed colocated proxy). + # django's own Linux CI steps (their .github/workflows/python_matrix.yml): install + # libmemcached-dev (pylibmc has no wheels for Python 3.12+ and the runner images don't bundle + # the headers, so django installs them itself), upgrade packaging tools, install the test + # dependencies, run runtests.py on SQLite. Identical on both runners; no action-level cache + # is enabled — the honest cold path (warm = a primed colocated proxy, set up by the operator). + - name: Install libmemcached-dev for pylibmc + run: sudo apt-get update && sudo apt-get install -y libmemcached-dev - name: Install django and test dependencies - run: python -m pip install -e . -r tests/requirements/py3.txt + run: | + python -m pip install --upgrade pip setuptools wheel + python -m pip install -r tests/requirements/py3.txt -e . - name: Run django's test suite - run: python tests/runtests.py --parallel + run: python -Wall tests/runtests.py -v2 From 167d087292e33129dd29b3f538cf044a12e37ebb Mon Sep 17 00:00:00 2001 From: Sergei Ozeranskii Date: Wed, 24 Jun 2026 15:06:57 +0200 Subject: [PATCH 2/2] =?UTF-8?q?docs:=20formalize=20cache=20policy=20?= =?UTF-8?q?=E2=80=94=20cold-cold=20canonical,=20warm=20separate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Define the headline comparison as cold-cold (cache-neutral workflow; the GitHub side is cold by construction, the self-hosted side's transparent dependency cache cleared before each cold run) — it isolates the runner with no caching asymmetry. Warm stays a separately-labelled steady-state row (each platform's native cache primed), never compared against cold. --- docs/methodology.md | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/docs/methodology.md b/docs/methodology.md index 1d8a2e1..9aeb6fa 100644 --- a/docs/methodology.md +++ b/docs/methodology.md @@ -53,14 +53,23 @@ recorded alongside every published number. ## Cold cache vs warm cache Dependency download and resolution dominates many CI runs, and caching changes the picture -substantially. Cold and warm runs are therefore reported **separately**, never blended: - -- **Cold cache** — no pre-populated dependency cache; everything is fetched fresh. This is the - worst case and the most honest baseline. The reusable workflow does not enable any action-level - cache, so the default path is cold. -- **Warm cache** — dependencies are served from a primed cache. This reflects steady-state CI on - a project that runs frequently. For tempus.build runners this means the colocated proxy has - already been populated; for the GitHub side an equivalent primed state is set up by the operator. +substantially. Cold and warm runs are reported **separately**, never blended. + +The reusable workflow is **cache-neutral** — it enables no action-level cache — so it stays +identical across runners and any caching is purely each platform's own. Cold and warm are +controlled outside the workflow: + +- **Cold — the canonical, headline comparison.** No caches primed on either side; everything is + fetched fresh. The GitHub-hosted side is cold by construction (no action-level cache, so packages + come from upstream every run); the self-hosted side has its transparent dependency cache cleared + before each cold run (otherwise a prior run leaves it warm). With neither side caching, the number + isolates the runner itself — CPU, I/O, network — which is the most defensible apples-to-apples + baseline, and the one published as the headline. +- **Warm — steady-state, reported separately and labelled.** Each platform uses its own native + caching, primed: the self-hosted side's transparent dependency cache populated, the GitHub side + with its action-level cache primed. This reflects repeat CI on a frequently-run project. The + caching mechanisms differ per platform, so a warm row is always labelled as warm and never + compared against a cold one. A "speedup" claim must state which cache state it refers to. Comparing tempus warm against GitHub cold (or vice versa) is not a valid comparison and is not published.