diff --git a/.github/workflows/casestudy-django.yml b/.github/workflows/casestudy-django.yml index ecfae88..5b0fe79 100644 --- a/.github/workflows/casestudy-django.yml +++ b/.github/workflows/casestudy-django.yml @@ -31,10 +31,16 @@ jobs: - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 with: python-version: "3.14" - # django's own test command at this pin. Uses pip (not uv) as django ships no uv lockfile. - # SQLite default — no DB service, no secrets. --parallel exercises all 4 vCPU. - # No action-level cache: this is the honest cold path (warm = a primed colocated proxy). + # django's own Linux CI steps (their .github/workflows/python_matrix.yml): install + # libmemcached-dev (pylibmc has no wheels for Python 3.12+ and the runner images don't bundle + # the headers, so django installs them itself), upgrade packaging tools, install the test + # dependencies, run runtests.py on SQLite. Identical on both runners; no action-level cache + # is enabled — the honest cold path (warm = a primed colocated proxy, set up by the operator). + - name: Install libmemcached-dev for pylibmc + run: sudo apt-get update && sudo apt-get install -y libmemcached-dev - name: Install django and test dependencies - run: python -m pip install -e . -r tests/requirements/py3.txt + run: | + python -m pip install --upgrade pip setuptools wheel + python -m pip install -r tests/requirements/py3.txt -e . - name: Run django's test suite - run: python tests/runtests.py --parallel + run: python -Wall tests/runtests.py -v2 diff --git a/docs/methodology.md b/docs/methodology.md index 1d8a2e1..9aeb6fa 100644 --- a/docs/methodology.md +++ b/docs/methodology.md @@ -53,14 +53,23 @@ recorded alongside every published number. ## Cold cache vs warm cache Dependency download and resolution dominates many CI runs, and caching changes the picture -substantially. Cold and warm runs are therefore reported **separately**, never blended: - -- **Cold cache** — no pre-populated dependency cache; everything is fetched fresh. This is the - worst case and the most honest baseline. The reusable workflow does not enable any action-level - cache, so the default path is cold. -- **Warm cache** — dependencies are served from a primed cache. This reflects steady-state CI on - a project that runs frequently. For tempus.build runners this means the colocated proxy has - already been populated; for the GitHub side an equivalent primed state is set up by the operator. +substantially. Cold and warm runs are reported **separately**, never blended. + +The reusable workflow is **cache-neutral** — it enables no action-level cache — so it stays +identical across runners and any caching is purely each platform's own. Cold and warm are +controlled outside the workflow: + +- **Cold — the canonical, headline comparison.** No caches primed on either side; everything is + fetched fresh. The GitHub-hosted side is cold by construction (no action-level cache, so packages + come from upstream every run); the self-hosted side has its transparent dependency cache cleared + before each cold run (otherwise a prior run leaves it warm). With neither side caching, the number + isolates the runner itself — CPU, I/O, network — which is the most defensible apples-to-apples + baseline, and the one published as the headline. +- **Warm — steady-state, reported separately and labelled.** Each platform uses its own native + caching, primed: the self-hosted side's transparent dependency cache populated, the GitHub side + with its action-level cache primed. This reflects repeat CI on a frequently-run project. The + caching mechanisms differ per platform, so a warm row is always labelled as warm and never + compared against a cold one. A "speedup" claim must state which cache state it refers to. Comparing tempus warm against GitHub cold (or vice versa) is not a valid comparison and is not published.