From c01046abf4c5662a0b99647c9240bea4d2f03208 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 30 Jul 2024 10:38:11 +0000 Subject: [PATCH 001/406] Bump the ruby-dependencies group with 2 updates (#1617) Bumps the ruby-dependencies group with 2 updates: [rubocop](https://github.com/rubocop/rubocop) and [rubocop-rspec](https://github.com/rubocop/rubocop-rspec). Updates `rubocop` from 1.64.1 to 1.65.0 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.64.1...v1.65.0) Updates `rubocop-rspec` from 3.0.1 to 3.0.3 - [Release notes](https://github.com/rubocop/rubocop-rspec/releases) - [Changelog](https://github.com/rubocop/rubocop-rspec/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop-rspec/compare/v3.0.1...v3.0.3) --- updated-dependencies: - dependency-name: rubocop dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies - dependency-name: rubocop-rspec dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 90b394edb..d57e2e1e9 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -27,18 +27,18 @@ GEM parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) - parser (3.3.3.0) + parser (3.3.4.0) ast (~> 2.4.1) racc platform-api (3.7.0) heroics (~> 0.1.1) moneta (~> 1.0.0) rate_throttle_client (~> 0.1.0) - racc (1.8.0) + racc (1.8.1) rainbow (3.1.1) rate_throttle_client (0.1.2) regexp_parser (2.9.2) - rexml (3.3.0) + rexml (3.3.2) strscan rrrretry (1.0.0) rspec-core (3.13.0) @@ -49,20 +49,20 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.1) - rubocop (1.64.1) + rubocop (1.65.0) json (~> 2.3) language_server-protocol (>= 3.17.0) parallel (~> 1.10) parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) - regexp_parser (>= 1.8, < 3.0) + regexp_parser (>= 2.4, < 3.0) rexml (>= 3.2.5, < 4.0) rubocop-ast (>= 1.31.1, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 3.0) rubocop-ast (1.31.3) parser (>= 3.3.1.0) - rubocop-rspec (3.0.1) + rubocop-rspec (3.0.3) rubocop (~> 1.61) ruby-progressbar (1.13.0) strscan (3.1.0) From e9835b8b379b30ea208cc5bb95086924fac380b1 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 2 Aug 2024 08:40:02 +0000 Subject: [PATCH 002/406] Bump rubocop from 1.65.0 to 1.65.1 in the ruby-dependencies group (#1620) Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.65.0 to 1.65.1 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.65.0...v1.65.1) --- updated-dependencies: - dependency-name: rubocop dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index d57e2e1e9..a397589e4 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -38,7 +38,7 @@ GEM rainbow (3.1.1) rate_throttle_client (0.1.2) regexp_parser (2.9.2) - rexml (3.3.2) + rexml (3.3.4) strscan rrrretry (1.0.0) rspec-core (3.13.0) @@ -49,7 +49,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.1) - rubocop (1.65.0) + rubocop (1.65.1) json (~> 2.3) language_server-protocol (>= 3.17.0) parallel (~> 1.10) From 05f25ccdc29cf5558cdced7dd36f5b63ba482af9 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 7 Aug 2024 17:10:05 +0100 Subject: [PATCH 003/406] Add support for Python 3.12.5 (#1622) Release announcement: https://blog.python.org/2024/08/python-3125-released.html Changelog: https://docs.python.org/release/3.12.5/whatsnew/changelog.html#python-3-12-5 Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/10285472077 GUS-W-14846788. --- CHANGELOG.md | 2 ++ README.md | 4 ++-- bin/default_pythons | 2 +- spec/fixtures/pipenv_and_runtime_txt/runtime.txt | 2 +- spec/fixtures/python_3.12/runtime.txt | 2 +- spec/fixtures/runtime_txt_only/runtime.txt | 2 +- spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt | 2 +- spec/hatchet/python_version_spec.rb | 2 +- spec/spec_helper.rb | 2 +- 9 files changed, 11 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 23f823115..e7237d6d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## [Unreleased] +- Added support for Python 3.12.5. ([#1622](https://github.com/heroku/heroku-buildpack-python/pull/1622)) +- Changed the default Python version for new apps from 3.12.4 to 3.12.5. ([#1622](https://github.com/heroku/heroku-buildpack-python/pull/1622)) ## [v254] - 2024-07-16 diff --git a/README.md b/README.md index 32dc35a58..e5196f197 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ A `requirements.txt` must be present at the root of your application's repositor To specify your python version, you also need a `runtime.txt` file - unless you are using the default Python runtime version. -Current default Python Runtime: Python 3.12.4 +Current default Python Runtime: Python 3.12.5 Alternatively, you can provide a `setup.py` file, or a `Pipfile`. Using `pipenv` will generate `runtime.txt` at build time if one of the field `python_version` or `python_full_version` is specified in the `requires` section of your `Pipfile`. @@ -60,7 +60,7 @@ Specify a Python Runtime Supported runtime options include: -- `python-3.12.4` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) +- `python-3.12.5` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) - `python-3.11.9` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) - `python-3.10.14` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) - `python-3.9.19` on Heroku-20 and Heroku-22 only diff --git a/bin/default_pythons b/bin/default_pythons index 04caa85d7..70e1dd730 100755 --- a/bin/default_pythons +++ b/bin/default_pythons @@ -5,7 +5,7 @@ # the env vars to subprocesses. # shellcheck disable=2034 -LATEST_312="python-3.12.4" +LATEST_312="python-3.12.5" LATEST_311="python-3.11.9" LATEST_310="python-3.10.14" LATEST_39="python-3.9.19" diff --git a/spec/fixtures/pipenv_and_runtime_txt/runtime.txt b/spec/fixtures/pipenv_and_runtime_txt/runtime.txt index 74d315a82..8a7cba521 100644 --- a/spec/fixtures/pipenv_and_runtime_txt/runtime.txt +++ b/spec/fixtures/pipenv_and_runtime_txt/runtime.txt @@ -1 +1 @@ -python-3.12.4 +python-3.12.5 diff --git a/spec/fixtures/python_3.12/runtime.txt b/spec/fixtures/python_3.12/runtime.txt index 74d315a82..8a7cba521 100644 --- a/spec/fixtures/python_3.12/runtime.txt +++ b/spec/fixtures/python_3.12/runtime.txt @@ -1 +1 @@ -python-3.12.4 +python-3.12.5 diff --git a/spec/fixtures/runtime_txt_only/runtime.txt b/spec/fixtures/runtime_txt_only/runtime.txt index 74d315a82..8a7cba521 100644 --- a/spec/fixtures/runtime_txt_only/runtime.txt +++ b/spec/fixtures/runtime_txt_only/runtime.txt @@ -1 +1 @@ -python-3.12.4 +python-3.12.5 diff --git a/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt b/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt index eb7c2eafb..8e4f9da34 100644 --- a/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt +++ b/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt @@ -1,3 +1,3 @@ - python-3.12.4 + python-3.12.5 \ No newline at end of file diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 7ae02287b..e93946c56 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -199,7 +199,7 @@ include_examples 'builds with the requested Python version', LATEST_PYTHON_3_11 end - context 'when runtime.txt contains python-3.12.4' do + context 'when runtime.txt contains python-3.12.5' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.12') } include_examples 'builds with the requested Python version', LATEST_PYTHON_3_12 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 3efda0c20..7518287af 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -11,7 +11,7 @@ LATEST_PYTHON_3_9 = '3.9.19' LATEST_PYTHON_3_10 = '3.10.14' LATEST_PYTHON_3_11 = '3.11.9' -LATEST_PYTHON_3_12 = '3.12.4' +LATEST_PYTHON_3_12 = '3.12.5' DEFAULT_PYTHON_VERSION = LATEST_PYTHON_3_12 # The requirement versions are effectively buildpack constants, however, we want From 27bfcfb4b5db0754ef1a80e0bf51bab8ec7e34b0 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 7 Aug 2024 16:20:16 +0000 Subject: [PATCH 004/406] Prepare release v255 (#1623) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e7237d6d6..bce369897 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v255] - 2024-08-07 + - Added support for Python 3.12.5. ([#1622](https://github.com/heroku/heroku-buildpack-python/pull/1622)) - Changed the default Python version for new apps from 3.12.4 to 3.12.5. ([#1622](https://github.com/heroku/heroku-buildpack-python/pull/1622)) @@ -1022,7 +1025,8 @@ Default Python is now latest 2.7.10. Updated Pip and Distribute. - Setuptools updated to v16.0 - Pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v254...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v255...main +[v255]: https://github.com/heroku/heroku-buildpack-python/compare/v254...v255 [v254]: https://github.com/heroku/heroku-buildpack-python/compare/v253...v254 [v253]: https://github.com/heroku/heroku-buildpack-python/compare/v252...v253 [v252]: https://github.com/heroku/heroku-buildpack-python/compare/v251...v252 From 20407e4267a3f991eab475baefbd57bcf1c25154 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 27 Aug 2024 10:18:42 +0100 Subject: [PATCH 005/406] Bump the ruby-dependencies group with 2 updates (#1627) Bumps the ruby-dependencies group with 2 updates: [rspec-expectations](https://github.com/rspec/rspec-expectations) and [rubocop-rspec](https://github.com/rubocop/rubocop-rspec). Updates `rspec-expectations` from 3.13.1 to 3.13.2 - [Release notes](https://github.com/rspec/rspec-expectations/releases) - [Changelog](https://github.com/rspec/rspec-expectations/blob/main/Changelog.md) - [Commits](https://github.com/rspec/rspec-expectations/compare/v3.13.1...v3.13.2) Updates `rubocop-rspec` from 3.0.3 to 3.0.4 - [Release notes](https://github.com/rubocop/rubocop-rspec/releases) - [Changelog](https://github.com/rubocop/rubocop-rspec/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop-rspec/compare/v3.0.3...v3.0.4) --- updated-dependencies: - dependency-name: rspec-expectations dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rubocop-rspec dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index a397589e4..408f9833a 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -23,11 +23,11 @@ GEM language_server-protocol (3.17.0.3) moneta (1.0.0) multi_json (1.15.0) - parallel (1.25.1) + parallel (1.26.3) parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) - parser (3.3.4.0) + parser (3.3.4.2) ast (~> 2.4.1) racc platform-api (3.7.0) @@ -38,12 +38,12 @@ GEM rainbow (3.1.1) rate_throttle_client (0.1.2) regexp_parser (2.9.2) - rexml (3.3.4) + rexml (3.3.6) strscan rrrretry (1.0.0) rspec-core (3.13.0) rspec-support (~> 3.13.0) - rspec-expectations (3.13.1) + rspec-expectations (3.13.2) diff-lcs (>= 1.2.0, < 2.0) rspec-support (~> 3.13.0) rspec-retry (0.6.2) @@ -60,9 +60,9 @@ GEM rubocop-ast (>= 1.31.1, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 3.0) - rubocop-ast (1.31.3) + rubocop-ast (1.32.1) parser (>= 3.3.1.0) - rubocop-rspec (3.0.3) + rubocop-rspec (3.0.4) rubocop (~> 1.61) ruby-progressbar (1.13.0) strscan (3.1.0) From f2372d252aeca563943af92c414304e8a3135057 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 2 Sep 2024 09:37:05 +0000 Subject: [PATCH 006/406] Bump wheel from 0.43.0 to 0.44.0 (#1629) * Bump wheel from 0.43.0 to 0.44.0 Bumps [wheel](https://github.com/pypa/wheel) from 0.43.0 to 0.44.0. - [Release notes](https://github.com/pypa/wheel/releases) - [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst) - [Commits](https://github.com/pypa/wheel/compare/0.43.0...0.44.0) --- updated-dependencies: - dependency-name: wheel dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/wheel.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bce369897..9f7963c0c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated wheel from 0.43.0 to 0.44.0. ([#1629](https://github.com/heroku/heroku-buildpack-python/pull/1629)) ## [v255] - 2024-08-07 diff --git a/requirements/wheel.txt b/requirements/wheel.txt index ee8fec0fb..e9f6da376 100644 --- a/requirements/wheel.txt +++ b/requirements/wheel.txt @@ -1 +1 @@ -wheel==0.43.0 +wheel==0.44.0 From fb955549d4e0dc0f0ee704abdf7813c348728807 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 2 Sep 2024 10:52:54 +0000 Subject: [PATCH 007/406] Add setuptools to the Dependabot ignore list (#1631) Since we can't update to setuptools v71+ due to its new approach to vendored dependencies: https://github.com/heroku/heroku-buildpack-python/pull/1630#issuecomment-2324236653 When this buildpack adds support for Python 3.13 we will also be dropping the global installs of setuptools and wheel. GUS-W-16637281. --- .github/dependabot.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 366df6035..942ea5fa5 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -25,6 +25,10 @@ updates: directory: "/" schedule: interval: "monthly" + ignore: + # We're not updating to setuptools v71+ due to its new approach to vendored dependencies: + # https://github.com/heroku/heroku-buildpack-python/pull/1630#issuecomment-2324236653 + - dependency-name: "setuptools" labels: - "dependencies" - "python" From 928d0c593f6b6def16d6ad8451f56a997760eace Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 2 Sep 2024 11:11:22 +0000 Subject: [PATCH 008/406] Bump rubocop from 1.65.1 to 1.66.0 in the ruby-dependencies group (#1628) Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.65.1 to 1.66.0 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.65.1...v1.66.0) --- updated-dependencies: - dependency-name: rubocop dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 408f9833a..6dc35eb1b 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -38,8 +38,6 @@ GEM rainbow (3.1.1) rate_throttle_client (0.1.2) regexp_parser (2.9.2) - rexml (3.3.6) - strscan rrrretry (1.0.0) rspec-core (3.13.0) rspec-support (~> 3.13.0) @@ -49,15 +47,14 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.1) - rubocop (1.65.1) + rubocop (1.66.0) json (~> 2.3) language_server-protocol (>= 3.17.0) parallel (~> 1.10) parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) regexp_parser (>= 2.4, < 3.0) - rexml (>= 3.2.5, < 4.0) - rubocop-ast (>= 1.31.1, < 2.0) + rubocop-ast (>= 1.32.1, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 3.0) rubocop-ast (1.32.1) @@ -65,7 +62,6 @@ GEM rubocop-rspec (3.0.4) rubocop (~> 1.61) ruby-progressbar (1.13.0) - strscan (3.1.0) thor (1.3.1) threaded (0.0.4) unicode-display_width (2.5.0) From cee5f8344019298f8fe77b1d29f5ee7aec3f28dc Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Sat, 7 Sep 2024 19:50:45 +0000 Subject: [PATCH 009/406] Add support for Python 3.8.20, 3.9.20, 3.10.15, 3.11.10 and 3.12.6 (#1632) Release notes: https://blog.python.org/2024/09/python-3130rc2-3126-31110-31015-3920.html Changelogs: https://docs.python.org/release/3.8.20/whatsnew/changelog.html#python-3-8-20-final https://docs.python.org/release/3.9.20/whatsnew/changelog.html#python-3-9-20-final https://docs.python.org/release/3.10.15/whatsnew/changelog.html#python-3-10-15-final https://docs.python.org/release/3.11.10/whatsnew/changelog.html#python-3-11-10-final https://docs.python.org/release/3.12.6/whatsnew/changelog.html#python-3-12-6-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/10750205053/job/29818038722 https://github.com/heroku/heroku-buildpack-python/actions/runs/10751132694/job/29818059983 https://github.com/heroku/heroku-buildpack-python/actions/runs/10750206118 https://github.com/heroku/heroku-buildpack-python/actions/runs/10750206743 https://github.com/heroku/heroku-buildpack-python/actions/runs/10750207733 GUS-W-16690151. --- CHANGELOG.md | 2 ++ README.md | 12 ++++++------ bin/default_pythons | 10 +++++----- spec/fixtures/pipenv_and_runtime_txt/runtime.txt | 2 +- spec/fixtures/python_3.10/runtime.txt | 2 +- spec/fixtures/python_3.11/runtime.txt | 2 +- spec/fixtures/python_3.12/runtime.txt | 2 +- spec/fixtures/python_3.8/runtime.txt | 2 +- spec/fixtures/python_3.9/runtime.txt | 2 +- spec/fixtures/runtime_txt_only/runtime.txt | 2 +- .../runtime_txt_with_stray_whitespace/runtime.txt | 2 +- spec/hatchet/python_version_spec.rb | 12 ++++++------ spec/spec_helper.rb | 10 +++++----- 13 files changed, 32 insertions(+), 30 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f7963c0c..223288c17 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## [Unreleased] +- Added support for Python 3.8.20, 3.9.20, 3.10.15, 3.11.10 and 3.12.6. ([#1632](https://github.com/heroku/heroku-buildpack-python/pull/1632)) +- Changed the default Python version for new apps from 3.12.5 to 3.12.6. ([#1632](https://github.com/heroku/heroku-buildpack-python/pull/1632)) - Updated wheel from 0.43.0 to 0.44.0. ([#1629](https://github.com/heroku/heroku-buildpack-python/pull/1629)) ## [v255] - 2024-08-07 diff --git a/README.md b/README.md index e5196f197..82b93df63 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ A `requirements.txt` must be present at the root of your application's repositor To specify your python version, you also need a `runtime.txt` file - unless you are using the default Python runtime version. -Current default Python Runtime: Python 3.12.5 +Current default Python Runtime: Python 3.12.6 Alternatively, you can provide a `setup.py` file, or a `Pipfile`. Using `pipenv` will generate `runtime.txt` at build time if one of the field `python_version` or `python_full_version` is specified in the `requires` section of your `Pipfile`. @@ -60,8 +60,8 @@ Specify a Python Runtime Supported runtime options include: -- `python-3.12.5` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) -- `python-3.11.9` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) -- `python-3.10.14` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) -- `python-3.9.19` on Heroku-20 and Heroku-22 only -- `python-3.8.19` on Heroku-20 only +- `python-3.12.6` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) +- `python-3.11.10` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) +- `python-3.10.15` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) +- `python-3.9.20` on Heroku-20 and Heroku-22 only +- `python-3.8.20` on Heroku-20 only diff --git a/bin/default_pythons b/bin/default_pythons index 70e1dd730..30bd0b34c 100755 --- a/bin/default_pythons +++ b/bin/default_pythons @@ -5,11 +5,11 @@ # the env vars to subprocesses. # shellcheck disable=2034 -LATEST_312="python-3.12.5" -LATEST_311="python-3.11.9" -LATEST_310="python-3.10.14" -LATEST_39="python-3.9.19" -LATEST_38="python-3.8.19" +LATEST_312="python-3.12.6" +LATEST_311="python-3.11.10" +LATEST_310="python-3.10.15" +LATEST_39="python-3.9.20" +LATEST_38="python-3.8.20" # TODO: Remove these EOL versions once pipenv-python-version is refactored. LATEST_37="python-3.7.17" LATEST_36="python-3.6.15" diff --git a/spec/fixtures/pipenv_and_runtime_txt/runtime.txt b/spec/fixtures/pipenv_and_runtime_txt/runtime.txt index 8a7cba521..32bcba665 100644 --- a/spec/fixtures/pipenv_and_runtime_txt/runtime.txt +++ b/spec/fixtures/pipenv_and_runtime_txt/runtime.txt @@ -1 +1 @@ -python-3.12.5 +python-3.12.6 diff --git a/spec/fixtures/python_3.10/runtime.txt b/spec/fixtures/python_3.10/runtime.txt index 2bccd9d9d..bb60b7f69 100644 --- a/spec/fixtures/python_3.10/runtime.txt +++ b/spec/fixtures/python_3.10/runtime.txt @@ -1 +1 @@ -python-3.10.14 +python-3.10.15 diff --git a/spec/fixtures/python_3.11/runtime.txt b/spec/fixtures/python_3.11/runtime.txt index 546f3c8de..e34519556 100644 --- a/spec/fixtures/python_3.11/runtime.txt +++ b/spec/fixtures/python_3.11/runtime.txt @@ -1 +1 @@ -python-3.11.9 +python-3.11.10 diff --git a/spec/fixtures/python_3.12/runtime.txt b/spec/fixtures/python_3.12/runtime.txt index 8a7cba521..32bcba665 100644 --- a/spec/fixtures/python_3.12/runtime.txt +++ b/spec/fixtures/python_3.12/runtime.txt @@ -1 +1 @@ -python-3.12.5 +python-3.12.6 diff --git a/spec/fixtures/python_3.8/runtime.txt b/spec/fixtures/python_3.8/runtime.txt index 96307d57b..7494875b9 100644 --- a/spec/fixtures/python_3.8/runtime.txt +++ b/spec/fixtures/python_3.8/runtime.txt @@ -1 +1 @@ -python-3.8.19 +python-3.8.20 diff --git a/spec/fixtures/python_3.9/runtime.txt b/spec/fixtures/python_3.9/runtime.txt index 4458b433c..57f558859 100644 --- a/spec/fixtures/python_3.9/runtime.txt +++ b/spec/fixtures/python_3.9/runtime.txt @@ -1 +1 @@ -python-3.9.19 +python-3.9.20 diff --git a/spec/fixtures/runtime_txt_only/runtime.txt b/spec/fixtures/runtime_txt_only/runtime.txt index 8a7cba521..32bcba665 100644 --- a/spec/fixtures/runtime_txt_only/runtime.txt +++ b/spec/fixtures/runtime_txt_only/runtime.txt @@ -1 +1 @@ -python-3.12.5 +python-3.12.6 diff --git a/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt b/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt index 8e4f9da34..b7e93bdc0 100644 --- a/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt +++ b/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt @@ -1,3 +1,3 @@ - python-3.12.5 + python-3.12.6 \ No newline at end of file diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index e93946c56..f4d96c376 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -132,12 +132,12 @@ end end - context 'when runtime.txt contains python-3.8.19' do + context 'when runtime.txt contains python-3.8.20' do let(:allow_failure) { false } let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.8', allow_failure:) } context 'when using Heroku-20', stacks: %w[heroku-20] do - it 'builds with Python 3.8.19 but shows a deprecation warning' do + it 'builds with Python 3.8.20 but shows a deprecation warning' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected @@ -171,7 +171,7 @@ end end - context 'when runtime.txt contains python-3.9.19' do + context 'when runtime.txt contains python-3.9.20' do let(:allow_failure) { false } let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.9', allow_failure:) } @@ -187,19 +187,19 @@ end end - context 'when runtime.txt contains python-3.10.14' do + context 'when runtime.txt contains python-3.10.15' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.10') } include_examples 'builds with the requested Python version', LATEST_PYTHON_3_10 end - context 'when runtime.txt contains python-3.11.9' do + context 'when runtime.txt contains python-3.11.10' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.11') } include_examples 'builds with the requested Python version', LATEST_PYTHON_3_11 end - context 'when runtime.txt contains python-3.12.5' do + context 'when runtime.txt contains python-3.12.6' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.12') } include_examples 'builds with the requested Python version', LATEST_PYTHON_3_12 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 7518287af..6d72bc8ee 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -7,11 +7,11 @@ require 'rspec/retry' require 'hatchet' -LATEST_PYTHON_3_8 = '3.8.19' -LATEST_PYTHON_3_9 = '3.9.19' -LATEST_PYTHON_3_10 = '3.10.14' -LATEST_PYTHON_3_11 = '3.11.9' -LATEST_PYTHON_3_12 = '3.12.5' +LATEST_PYTHON_3_8 = '3.8.20' +LATEST_PYTHON_3_9 = '3.9.20' +LATEST_PYTHON_3_10 = '3.10.15' +LATEST_PYTHON_3_11 = '3.11.10' +LATEST_PYTHON_3_12 = '3.12.6' DEFAULT_PYTHON_VERSION = LATEST_PYTHON_3_12 # The requirement versions are effectively buildpack constants, however, we want From d13fbbdea7a86471872241f811eaf08787757d44 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Sat, 7 Sep 2024 19:55:29 +0000 Subject: [PATCH 010/406] Prepare release v256 (#1633) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 223288c17..b084b5158 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v256] - 2024-09-07 + - Added support for Python 3.8.20, 3.9.20, 3.10.15, 3.11.10 and 3.12.6. ([#1632](https://github.com/heroku/heroku-buildpack-python/pull/1632)) - Changed the default Python version for new apps from 3.12.5 to 3.12.6. ([#1632](https://github.com/heroku/heroku-buildpack-python/pull/1632)) - Updated wheel from 0.43.0 to 0.44.0. ([#1629](https://github.com/heroku/heroku-buildpack-python/pull/1629)) @@ -1028,7 +1031,8 @@ Default Python is now latest 2.7.10. Updated Pip and Distribute. - Setuptools updated to v16.0 - Pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v255...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v256...main +[v256]: https://github.com/heroku/heroku-buildpack-python/compare/v255...v256 [v255]: https://github.com/heroku/heroku-buildpack-python/compare/v254...v255 [v254]: https://github.com/heroku/heroku-buildpack-python/compare/v253...v254 [v253]: https://github.com/heroku/heroku-buildpack-python/compare/v252...v253 From b5bbcb795be74bd047fb292915ddf94be8803567 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 23 Sep 2024 07:18:12 +0000 Subject: [PATCH 011/406] Fix duplicate word in binary build script error messages (#1635) Fixes `Error: Error: ...` being shown in the error messages shown by the internal binary build script (used by buildpack maintainers to build/upload new Python versions). Also renames `error()` to `abort()` to make it clearer it terminates the script. --- builds/build_python_runtime.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index b1e81c7b8..88817225c 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -14,7 +14,7 @@ INSTALL_DIR="/tmp/python" SRC_DIR="/tmp/src" UPLOAD_DIR="/tmp/upload" -function error() { +function abort() { echo "Error: ${1}" >&2 exit 1 } @@ -45,12 +45,12 @@ case "${STACK}" in ) ;; *) - error "Error: Unsupported stack '${STACK}'!" + abort "Unsupported stack '${STACK}'!" ;; esac if [[ ! " ${SUPPORTED_PYTHON_VERSIONS[*]} " == *" ${PYTHON_MAJOR_VERSION} "* ]]; then - error "Error: Python ${PYTHON_MAJOR_VERSION} is not supported on ${STACK}!" + abort "Python ${PYTHON_MAJOR_VERSION} is not supported on ${STACK}!" fi # The release keys can be found on https://www.python.org/downloads/ -> "OpenPGP Public Keys". @@ -68,7 +68,7 @@ case "${PYTHON_MAJOR_VERSION}" in GPG_KEY_FINGERPRINT='E3FF2839C048B25C084DEBE9B26995E310250568' ;; *) - error "Error: Unsupported Python version '${PYTHON_MAJOR_VERSION}'!" + abort "Unsupported Python version '${PYTHON_MAJOR_VERSION}'!" ;; esac @@ -184,7 +184,7 @@ if [[ "${PYTHON_MAJOR_VERSION}" == 3.[8-9] ]]; then # - `lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.a` find "${INSTALL_DIR}" -type f -name '*.a' -print -exec strip --strip-unneeded '{}' + elif ! find "${INSTALL_DIR}" -type f -name '*.a' -print -exec false '{}' +; then - error "Error: Unexpected static libraries found!" + abort "Unexpected static libraries found!" fi # Remove unneeded test directories, similar to the official Docker Python images: From 0dde52d26fb2e9ffc112949e5e03fb3f628d8442 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 23 Sep 2024 07:18:29 +0000 Subject: [PATCH 012/406] Improve `make run` (#1636) The `make run` target provides a quick way to execute the buildpack locally in a container when developing on the buildpack. Previously it executed only `bin/{detect,compile,report}`. Now it also runs `bin/release`. In addition, the arguments passed to `bin/detect` have been corrected, so that they now match the Buildpack API (where only `BUILD_DIR` is passed to the script): https://devcenter.heroku.com/articles/buildpack-api#bin-detect --- Makefile | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/Makefile b/Makefile index 3e36a4eb4..1b4741ed9 100644 --- a/Makefile +++ b/Makefile @@ -19,13 +19,14 @@ run: @echo "Running buildpack using: STACK=$(STACK) FIXTURE=$(FIXTURE)" @docker run --rm -it -v $(PWD):/src:ro --tmpfs /app -e "HOME=/app" -e "STACK=$(STACK)" "$(STACK_IMAGE_TAG)" \ bash -eo pipefail -c '\ - mkdir /tmp/buildpack /tmp/build /tmp/cache /tmp/env \ - && cp -r /src/{bin,lib,requirements,vendor} /tmp/buildpack \ - && cp -rT /src/$(FIXTURE) /tmp/build \ - && cd /tmp/buildpack \ - && echo -e "\n~ Detect:" && bash ./bin/detect /tmp/build /tmp/cache /tmp/env \ - && echo -e "\n~ Compile:" && { ./bin/compile /tmp/build /tmp/cache /tmp/env || true; } \ - && echo -e "\n~ Report:" && ./bin/report /tmp/build /tmp/cache /tmp/env \ + mkdir /tmp/buildpack /tmp/build /tmp/cache /tmp/env; \ + cp -r /src/{bin,lib,requirements,vendor} /tmp/buildpack; \ + cp -rT /src/$(FIXTURE) /tmp/build; \ + cd /tmp/buildpack; \ + echo -e "\n~ Detect:" && ./bin/detect /tmp/build; \ + echo -e "\n~ Compile:" && { ./bin/compile /tmp/build /tmp/cache /tmp/env || SKIP_RELEASE=1; }; \ + echo -e "\n~ Report:" && ./bin/report /tmp/build /tmp/cache /tmp/env; \ + [[ -n "$${SKIP_RELEASE}" ]] || { echo -e "\n~ Release:" && ./bin/release /tmp/build && echo -e "\nBuild successful!"; }; \ ' @echo From 80e77819a117f5d7e06c45c603f574f2e6c45a78 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 23 Sep 2024 08:01:14 +0000 Subject: [PATCH 013/406] Misc integration test improvements (#1634) As part of adding Poetry + `.python-version` file support, a fair amount of buildpack refactoring will be required. This backports some of the additional test scenarios we now have in the new Python CNB, along with some test general improvements. It's been split out of later PRs for easier review. GUS-W-16807487. --- .github/workflows/ci.yml | 6 +- Gemfile.lock | 24 +-- bin/steps/pipenv | 1 + spec/fixtures/ci_pipenv/Pipfile | 10 + spec/fixtures/ci_pipenv/Pipfile.lock | 62 ++++++ spec/fixtures/ci_pipenv/app.json | 9 + spec/fixtures/ci_pipenv/bin/compile | 12 ++ spec/fixtures/ci_pipenv/bin/detect | 7 + spec/fixtures/ci_pipenv/bin/post_compile | 5 + spec/fixtures/ci_pipenv/bin/print-env-vars.sh | 5 + spec/fixtures/ci_requirements/app.json | 2 +- spec/fixtures/ci_requirements/bin/compile | 12 ++ spec/fixtures/ci_requirements/bin/detect | 7 + .../fixtures/ci_requirements/bin/post_compile | 5 + .../ci_requirements/bin/print-env-vars.sh | 5 + .../ci_requirements/requirements-test.txt | 2 +- .../fixtures/ci_requirements/requirements.txt | 3 +- spec/fixtures/hooks/bin/post_compile | 5 +- spec/fixtures/hooks/bin/pre_compile | 5 +- spec/fixtures/hooks/bin/print-env-vars.sh | 10 + spec/fixtures/pipenv_editable/bin/compile | 2 +- .../fixtures/pipenv_editable/bin/post_compile | 2 +- .../{test-entrypoints => test-entrypoints.sh} | 0 .../pipenv_python_version_unspecified/Pipfile | 2 +- .../Pipfile.lock | 10 +- .../bin/compile | 19 ++ .../bin/detect | 7 + .../setup.py | 9 + .../fixtures/python_3.10_outdated/runtime.txt | 2 +- spec/fixtures/python_3.8_outdated/runtime.txt | 2 +- spec/fixtures/python_3.9_outdated/runtime.txt | 2 +- spec/fixtures/requirements_basic/bin/compile | 18 ++ spec/fixtures/requirements_basic/bin/detect | 7 + .../requirements_basic/requirements.txt | 2 + .../requirements_editable/bin/compile | 2 +- .../requirements_editable/bin/post_compile | 2 +- .../{test-entrypoints => test-entrypoints.sh} | 0 .../requirements_git/requirements.txt | 2 - .../requirements_invalid/requirements.txt | 1 + spec/hatchet/ci_spec.rb | 197 ++++++++++++++---- spec/hatchet/django_spec.rb | 11 +- spec/hatchet/hooks_spec.rb | 83 ++++---- spec/hatchet/package_manager_spec.rb | 1 + spec/hatchet/pip_spec.rb | 134 +++++++++--- spec/hatchet/pipenv_spec.rb | 75 ++++++- spec/hatchet/python_update_warning_spec.rb | 20 +- spec/hatchet/python_version_spec.rb | 3 + spec/spec_helper.rb | 2 + 48 files changed, 659 insertions(+), 155 deletions(-) create mode 100644 spec/fixtures/ci_pipenv/Pipfile create mode 100644 spec/fixtures/ci_pipenv/Pipfile.lock create mode 100644 spec/fixtures/ci_pipenv/app.json create mode 100755 spec/fixtures/ci_pipenv/bin/compile create mode 100755 spec/fixtures/ci_pipenv/bin/detect create mode 100644 spec/fixtures/ci_pipenv/bin/post_compile create mode 100755 spec/fixtures/ci_pipenv/bin/print-env-vars.sh create mode 100755 spec/fixtures/ci_requirements/bin/compile create mode 100755 spec/fixtures/ci_requirements/bin/detect create mode 100644 spec/fixtures/ci_requirements/bin/post_compile create mode 100755 spec/fixtures/ci_requirements/bin/print-env-vars.sh create mode 100755 spec/fixtures/hooks/bin/print-env-vars.sh rename spec/fixtures/pipenv_editable/bin/{test-entrypoints => test-entrypoints.sh} (100%) create mode 100755 spec/fixtures/pipenv_python_version_unspecified/bin/compile create mode 100755 spec/fixtures/pipenv_python_version_unspecified/bin/detect create mode 100644 spec/fixtures/pipenv_python_version_unspecified/setup.py create mode 100755 spec/fixtures/requirements_basic/bin/compile create mode 100755 spec/fixtures/requirements_basic/bin/detect create mode 100644 spec/fixtures/requirements_basic/requirements.txt rename spec/fixtures/requirements_editable/bin/{test-entrypoints => test-entrypoints.sh} (100%) delete mode 100644 spec/fixtures/requirements_git/requirements.txt create mode 100644 spec/fixtures/requirements_invalid/requirements.txt diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ed5588b7d..d7f97513f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -33,14 +33,14 @@ jobs: matrix: stack: ["heroku-20", "heroku-22", "heroku-24"] env: - HATCHET_APP_LIMIT: 200 + HATCHET_APP_LIMIT: 300 HATCHET_DEFAULT_STACK: ${{ matrix.stack }} HATCHET_EXPENSIVE_MODE: 1 HEROKU_API_KEY: ${{ secrets.HEROKU_API_KEY }} HEROKU_API_USER: ${{ secrets.HEROKU_API_USER }} HEROKU_DISABLE_AUTOUPDATE: 1 - PARALLEL_SPLIT_TEST_PROCESSES: 60 - RSPEC_RETRY_RETRY_COUNT: 3 + PARALLEL_SPLIT_TEST_PROCESSES: 70 + RSPEC_RETRY_RETRY_COUNT: 2 steps: - name: Checkout uses: actions/checkout@v4 diff --git a/Gemfile.lock b/Gemfile.lock index 6dc35eb1b..7b8aeedce 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -5,7 +5,7 @@ GEM base64 (0.2.0) diff-lcs (1.5.1) erubis (2.7.0) - excon (0.110.0) + excon (0.111.0) heroics (0.1.3) base64 erubis (~> 2.0) @@ -27,7 +27,7 @@ GEM parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) - parser (3.3.4.2) + parser (3.3.5.0) ast (~> 2.4.1) racc platform-api (3.7.0) @@ -39,32 +39,32 @@ GEM rate_throttle_client (0.1.2) regexp_parser (2.9.2) rrrretry (1.0.0) - rspec-core (3.13.0) + rspec-core (3.13.1) rspec-support (~> 3.13.0) - rspec-expectations (3.13.2) + rspec-expectations (3.13.3) diff-lcs (>= 1.2.0, < 2.0) rspec-support (~> 3.13.0) rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.1) - rubocop (1.66.0) + rubocop (1.66.1) json (~> 2.3) language_server-protocol (>= 3.17.0) parallel (~> 1.10) parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) regexp_parser (>= 2.4, < 3.0) - rubocop-ast (>= 1.32.1, < 2.0) + rubocop-ast (>= 1.32.2, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 3.0) - rubocop-ast (1.32.1) + rubocop-ast (1.32.3) parser (>= 3.3.1.0) - rubocop-rspec (3.0.4) + rubocop-rspec (3.0.5) rubocop (~> 1.61) ruby-progressbar (1.13.0) - thor (1.3.1) + thor (1.3.2) threaded (0.0.4) - unicode-display_width (2.5.0) + unicode-display_width (2.6.0) webrick (1.8.1) PLATFORMS @@ -80,7 +80,7 @@ DEPENDENCIES rubocop-rspec RUBY VERSION - ruby 3.3.2p78 + ruby 3.3.5p100 BUNDLED WITH - 2.5.11 + 2.5.18 diff --git a/bin/steps/pipenv b/bin/steps/pipenv index d7a190b4b..1d97941c7 100755 --- a/bin/steps/pipenv +++ b/bin/steps/pipenv @@ -15,6 +15,7 @@ if [[ -f Pipfile ]]; then meta_set "package_manager" "pipenv" # Skip installing dependencies using pip later. + # TODO: Stop leaking this env var into subshells such as post_compile hooks. export SKIP_PIP_INSTALL=1 # Set Pip env vars diff --git a/spec/fixtures/ci_pipenv/Pipfile b/spec/fixtures/ci_pipenv/Pipfile new file mode 100644 index 000000000..6d6473eb4 --- /dev/null +++ b/spec/fixtures/ci_pipenv/Pipfile @@ -0,0 +1,10 @@ +[[source]] +url = "https://pypi.org/simple" +verify_ssl = true +name = "pypi" + +[packages] +typing-extensions = "*" + +[dev-packages] +pytest = "*" diff --git a/spec/fixtures/ci_pipenv/Pipfile.lock b/spec/fixtures/ci_pipenv/Pipfile.lock new file mode 100644 index 000000000..aa11d8ecf --- /dev/null +++ b/spec/fixtures/ci_pipenv/Pipfile.lock @@ -0,0 +1,62 @@ +{ + "_meta": { + "hash": { + "sha256": "9ac30f761973e7bb9a0425635eb284370fede0e49a74b475c418f98bf13f3075" + }, + "pipfile-spec": 6, + "requires": {}, + "sources": [ + { + "name": "pypi", + "url": "https://pypi.org/simple", + "verify_ssl": true + } + ] + }, + "default": { + "typing-extensions": { + "hashes": [ + "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", + "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" + ], + "index": "pypi", + "markers": "python_version >= '3.8'", + "version": "==4.12.2" + } + }, + "develop": { + "iniconfig": { + "hashes": [ + "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3", + "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374" + ], + "markers": "python_version >= '3.7'", + "version": "==2.0.0" + }, + "packaging": { + "hashes": [ + "sha256:026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002", + "sha256:5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124" + ], + "markers": "python_version >= '3.8'", + "version": "==24.1" + }, + "pluggy": { + "hashes": [ + "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1", + "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669" + ], + "markers": "python_version >= '3.8'", + "version": "==1.5.0" + }, + "pytest": { + "hashes": [ + "sha256:70b98107bd648308a7952b06e6ca9a50bc660be218d53c257cc1fc94fda10181", + "sha256:a6853c7375b2663155079443d2e45de913a911a11d669df02a50814944db57b2" + ], + "index": "pypi", + "markers": "python_version >= '3.8'", + "version": "==8.3.3" + } + } +} diff --git a/spec/fixtures/ci_pipenv/app.json b/spec/fixtures/ci_pipenv/app.json new file mode 100644 index 000000000..59617c169 --- /dev/null +++ b/spec/fixtures/ci_pipenv/app.json @@ -0,0 +1,9 @@ +{ + "environments": { + "test": { + "scripts": { + "test": "./bin/print-env-vars.sh && pytest --version" + } + } + } +} diff --git a/spec/fixtures/ci_pipenv/bin/compile b/spec/fixtures/ci_pipenv/bin/compile new file mode 100755 index 000000000..fcb7055e3 --- /dev/null +++ b/spec/fixtures/ci_pipenv/bin/compile @@ -0,0 +1,12 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack, and tests that the environment is +# configured as expected for buildpacks that run after the Python buildpack. + +set -euo pipefail + +BUILD_DIR="${1}" + +cd "${BUILD_DIR}" + +exec bin/print-env-vars.sh diff --git a/spec/fixtures/ci_pipenv/bin/detect b/spec/fixtures/ci_pipenv/bin/detect new file mode 100755 index 000000000..68cdcc4a2 --- /dev/null +++ b/spec/fixtures/ci_pipenv/bin/detect @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack. + +set -euo pipefail + +echo "Inline" diff --git a/spec/fixtures/ci_pipenv/bin/post_compile b/spec/fixtures/ci_pipenv/bin/post_compile new file mode 100644 index 000000000..15362f6b1 --- /dev/null +++ b/spec/fixtures/ci_pipenv/bin/post_compile @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -euo pipefail + +exec bin/print-env-vars.sh diff --git a/spec/fixtures/ci_pipenv/bin/print-env-vars.sh b/spec/fixtures/ci_pipenv/bin/print-env-vars.sh new file mode 100755 index 000000000..9e0bebe6b --- /dev/null +++ b/spec/fixtures/ci_pipenv/bin/print-env-vars.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -euo pipefail + +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|BUILD_DIR|CACHE_DIR|CI_NODE_.+|DYNO|ENV_DIR|HEROKU_TEST_RUN_.+|HOME|OLDPWD|PORT|PWD|SHLVL|STACK|TERM)=' diff --git a/spec/fixtures/ci_requirements/app.json b/spec/fixtures/ci_requirements/app.json index 99d1d237f..59617c169 100644 --- a/spec/fixtures/ci_requirements/app.json +++ b/spec/fixtures/ci_requirements/app.json @@ -2,7 +2,7 @@ "environments": { "test": { "scripts": { - "test": "pytest --version" + "test": "./bin/print-env-vars.sh && pytest --version" } } } diff --git a/spec/fixtures/ci_requirements/bin/compile b/spec/fixtures/ci_requirements/bin/compile new file mode 100755 index 000000000..fcb7055e3 --- /dev/null +++ b/spec/fixtures/ci_requirements/bin/compile @@ -0,0 +1,12 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack, and tests that the environment is +# configured as expected for buildpacks that run after the Python buildpack. + +set -euo pipefail + +BUILD_DIR="${1}" + +cd "${BUILD_DIR}" + +exec bin/print-env-vars.sh diff --git a/spec/fixtures/ci_requirements/bin/detect b/spec/fixtures/ci_requirements/bin/detect new file mode 100755 index 000000000..68cdcc4a2 --- /dev/null +++ b/spec/fixtures/ci_requirements/bin/detect @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack. + +set -euo pipefail + +echo "Inline" diff --git a/spec/fixtures/ci_requirements/bin/post_compile b/spec/fixtures/ci_requirements/bin/post_compile new file mode 100644 index 000000000..15362f6b1 --- /dev/null +++ b/spec/fixtures/ci_requirements/bin/post_compile @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -euo pipefail + +exec bin/print-env-vars.sh diff --git a/spec/fixtures/ci_requirements/bin/print-env-vars.sh b/spec/fixtures/ci_requirements/bin/print-env-vars.sh new file mode 100755 index 000000000..9e0bebe6b --- /dev/null +++ b/spec/fixtures/ci_requirements/bin/print-env-vars.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -euo pipefail + +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|BUILD_DIR|CACHE_DIR|CI_NODE_.+|DYNO|ENV_DIR|HEROKU_TEST_RUN_.+|HOME|OLDPWD|PORT|PWD|SHLVL|STACK|TERM)=' diff --git a/spec/fixtures/ci_requirements/requirements-test.txt b/spec/fixtures/ci_requirements/requirements-test.txt index e079f8a60..40543aaba 100644 --- a/spec/fixtures/ci_requirements/requirements-test.txt +++ b/spec/fixtures/ci_requirements/requirements-test.txt @@ -1 +1 @@ -pytest +pytest==8.3.3 diff --git a/spec/fixtures/ci_requirements/requirements.txt b/spec/fixtures/ci_requirements/requirements.txt index a42590beb..eec3a2223 100644 --- a/spec/fixtures/ci_requirements/requirements.txt +++ b/spec/fixtures/ci_requirements/requirements.txt @@ -1 +1,2 @@ -urllib3 +# This package has been picked since it has no dependencies and is small/fast to install. +typing-extensions==4.12.2 diff --git a/spec/fixtures/hooks/bin/post_compile b/spec/fixtures/hooks/bin/post_compile index 373d33889..4bd8ed0e8 100644 --- a/spec/fixtures/hooks/bin/post_compile +++ b/spec/fixtures/hooks/bin/post_compile @@ -2,5 +2,6 @@ set -euo pipefail -echo 'post_compile ran with env vars:' -printenv | cut -d '=' -f 1 | sort +echo '~ post_compile ran with env vars:' +bin/print-env-vars.sh +echo '~ post_compile complete' diff --git a/spec/fixtures/hooks/bin/pre_compile b/spec/fixtures/hooks/bin/pre_compile index 40d1a402e..9f3737bef 100644 --- a/spec/fixtures/hooks/bin/pre_compile +++ b/spec/fixtures/hooks/bin/pre_compile @@ -2,5 +2,6 @@ set -euo pipefail -echo 'pre_compile ran with env vars:' -printenv | cut -d '=' -f 1 | sort +echo '~ pre_compile ran with env vars:' +bin/print-env-vars.sh +echo '~ pre_compile complete' diff --git a/spec/fixtures/hooks/bin/print-env-vars.sh b/spec/fixtures/hooks/bin/print-env-vars.sh new file mode 100755 index 000000000..dbd598a54 --- /dev/null +++ b/spec/fixtures/hooks/bin/print-env-vars.sh @@ -0,0 +1,10 @@ +#!/usr/bin/env bash + +set -euo pipefail + +printenv | sort \ + | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|OLDPWD|REQUEST_ID|SHLVL)=' \ + | sed --regexp-extended \ + --expression 's#(=/tmp/build_)[^:/]+#\1#' \ + --expression 's#^(ENV_DIR=/tmp/).*#\1...#' \ + --expression 's#^(SOURCE_VERSION=).*#\1...#' diff --git a/spec/fixtures/pipenv_editable/bin/compile b/spec/fixtures/pipenv_editable/bin/compile index c506ad6b7..df17e9401 100755 --- a/spec/fixtures/pipenv_editable/bin/compile +++ b/spec/fixtures/pipenv_editable/bin/compile @@ -9,4 +9,4 @@ BUILD_DIR="${1}" cd "${BUILD_DIR}" -exec bin/test-entrypoints +exec bin/test-entrypoints.sh diff --git a/spec/fixtures/pipenv_editable/bin/post_compile b/spec/fixtures/pipenv_editable/bin/post_compile index 460dc2de7..6e77d159a 100755 --- a/spec/fixtures/pipenv_editable/bin/post_compile +++ b/spec/fixtures/pipenv_editable/bin/post_compile @@ -2,4 +2,4 @@ set -euo pipefail -exec bin/test-entrypoints +exec bin/test-entrypoints.sh diff --git a/spec/fixtures/pipenv_editable/bin/test-entrypoints b/spec/fixtures/pipenv_editable/bin/test-entrypoints.sh similarity index 100% rename from spec/fixtures/pipenv_editable/bin/test-entrypoints rename to spec/fixtures/pipenv_editable/bin/test-entrypoints.sh diff --git a/spec/fixtures/pipenv_python_version_unspecified/Pipfile b/spec/fixtures/pipenv_python_version_unspecified/Pipfile index 67803f45b..496fa1558 100644 --- a/spec/fixtures/pipenv_python_version_unspecified/Pipfile +++ b/spec/fixtures/pipenv_python_version_unspecified/Pipfile @@ -4,6 +4,6 @@ verify_ssl = true name = "pypi" [packages] -urllib3 = "*" +typing-extensions = "*" [dev-packages] diff --git a/spec/fixtures/pipenv_python_version_unspecified/Pipfile.lock b/spec/fixtures/pipenv_python_version_unspecified/Pipfile.lock index 4c426c2ae..e744fd450 100644 --- a/spec/fixtures/pipenv_python_version_unspecified/Pipfile.lock +++ b/spec/fixtures/pipenv_python_version_unspecified/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "a95c318e4395270fbd1e0f52dd8f12185db4c3258d4e03dd80de54b7c7aad8b1" + "sha256": "beb76460a63ef2f29eec7b281a3c7114d442db105096d7472b4b72a7504df8fe" }, "pipfile-spec": 6, "requires": {}, @@ -14,14 +14,14 @@ ] }, "default": { - "urllib3": { + "typing-extensions": { "hashes": [ - "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", - "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" + "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", + "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==2.2.1" + "version": "==4.12.2" } }, "develop": {} diff --git a/spec/fixtures/pipenv_python_version_unspecified/bin/compile b/spec/fixtures/pipenv_python_version_unspecified/bin/compile new file mode 100755 index 000000000..db00975d6 --- /dev/null +++ b/spec/fixtures/pipenv_python_version_unspecified/bin/compile @@ -0,0 +1,19 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack, and tests that the environment is +# configured as expected for buildpacks that run after the Python buildpack. + +set -euo pipefail + +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK)=' +echo + +python -c 'import pprint, sys; pprint.pp(sys.path)' +echo + +# TODO: Investigate why 'pipenv graph' doesn't work here. +# TODO: Remove --disable-pip-version-check in favour of exporting PIP_DISABLE_PIP_VERSION_CHECK +pip list --disable-pip-version-check +echo + +python -c 'import typing_extensions; print(typing_extensions)' diff --git a/spec/fixtures/pipenv_python_version_unspecified/bin/detect b/spec/fixtures/pipenv_python_version_unspecified/bin/detect new file mode 100755 index 000000000..68cdcc4a2 --- /dev/null +++ b/spec/fixtures/pipenv_python_version_unspecified/bin/detect @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack. + +set -euo pipefail + +echo "Inline" diff --git a/spec/fixtures/pipenv_python_version_unspecified/setup.py b/spec/fixtures/pipenv_python_version_unspecified/setup.py new file mode 100644 index 000000000..de90e907e --- /dev/null +++ b/spec/fixtures/pipenv_python_version_unspecified/setup.py @@ -0,0 +1,9 @@ +# This file is here to confirm we don't try and create the fallback requirements +# file containing '-e .' when using Pipenv. + +from setuptools import setup + +setup( + name='test', + install_requires=['six'], +) diff --git a/spec/fixtures/python_3.10_outdated/runtime.txt b/spec/fixtures/python_3.10_outdated/runtime.txt index a5da7cc4d..fadb07024 100644 --- a/spec/fixtures/python_3.10_outdated/runtime.txt +++ b/spec/fixtures/python_3.10_outdated/runtime.txt @@ -1 +1 @@ -python-3.10.5 +python-3.10.0 diff --git a/spec/fixtures/python_3.8_outdated/runtime.txt b/spec/fixtures/python_3.8_outdated/runtime.txt index d9a16ae1e..73b1cf81d 100644 --- a/spec/fixtures/python_3.8_outdated/runtime.txt +++ b/spec/fixtures/python_3.8_outdated/runtime.txt @@ -1 +1 @@ -python-3.8.12 +python-3.8.0 diff --git a/spec/fixtures/python_3.9_outdated/runtime.txt b/spec/fixtures/python_3.9_outdated/runtime.txt index 540296197..f72c5111f 100644 --- a/spec/fixtures/python_3.9_outdated/runtime.txt +++ b/spec/fixtures/python_3.9_outdated/runtime.txt @@ -1 +1 @@ -python-3.9.12 +python-3.9.0 diff --git a/spec/fixtures/requirements_basic/bin/compile b/spec/fixtures/requirements_basic/bin/compile new file mode 100755 index 000000000..68d243e7b --- /dev/null +++ b/spec/fixtures/requirements_basic/bin/compile @@ -0,0 +1,18 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack, and tests that the environment is +# configured as expected for buildpacks that run after the Python buildpack. + +set -euo pipefail + +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK)=' +echo + +python -c 'import pprint, sys; pprint.pp(sys.path)' +echo + +# TODO: Remove --disable-pip-version-check in favour of exporting PIP_DISABLE_PIP_VERSION_CHECK +pip list --disable-pip-version-check +echo + +python -c 'import typing_extensions; print(typing_extensions)' diff --git a/spec/fixtures/requirements_basic/bin/detect b/spec/fixtures/requirements_basic/bin/detect new file mode 100755 index 000000000..68cdcc4a2 --- /dev/null +++ b/spec/fixtures/requirements_basic/bin/detect @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack. + +set -euo pipefail + +echo "Inline" diff --git a/spec/fixtures/requirements_basic/requirements.txt b/spec/fixtures/requirements_basic/requirements.txt new file mode 100644 index 000000000..eec3a2223 --- /dev/null +++ b/spec/fixtures/requirements_basic/requirements.txt @@ -0,0 +1,2 @@ +# This package has been picked since it has no dependencies and is small/fast to install. +typing-extensions==4.12.2 diff --git a/spec/fixtures/requirements_editable/bin/compile b/spec/fixtures/requirements_editable/bin/compile index c506ad6b7..df17e9401 100755 --- a/spec/fixtures/requirements_editable/bin/compile +++ b/spec/fixtures/requirements_editable/bin/compile @@ -9,4 +9,4 @@ BUILD_DIR="${1}" cd "${BUILD_DIR}" -exec bin/test-entrypoints +exec bin/test-entrypoints.sh diff --git a/spec/fixtures/requirements_editable/bin/post_compile b/spec/fixtures/requirements_editable/bin/post_compile index 460dc2de7..6e77d159a 100755 --- a/spec/fixtures/requirements_editable/bin/post_compile +++ b/spec/fixtures/requirements_editable/bin/post_compile @@ -2,4 +2,4 @@ set -euo pipefail -exec bin/test-entrypoints +exec bin/test-entrypoints.sh diff --git a/spec/fixtures/requirements_editable/bin/test-entrypoints b/spec/fixtures/requirements_editable/bin/test-entrypoints.sh similarity index 100% rename from spec/fixtures/requirements_editable/bin/test-entrypoints rename to spec/fixtures/requirements_editable/bin/test-entrypoints.sh diff --git a/spec/fixtures/requirements_git/requirements.txt b/spec/fixtures/requirements_git/requirements.txt deleted file mode 100644 index 81e9dbfe0..000000000 --- a/spec/fixtures/requirements_git/requirements.txt +++ /dev/null @@ -1,2 +0,0 @@ -# This relies upon the VCS binaries from the stack image. -git+https://github.com/certifi/python-certifi diff --git a/spec/fixtures/requirements_invalid/requirements.txt b/spec/fixtures/requirements_invalid/requirements.txt new file mode 100644 index 000000000..db42b7ee6 --- /dev/null +++ b/spec/fixtures/requirements_invalid/requirements.txt @@ -0,0 +1 @@ +an-invalid-requirement! diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index eb0b2c5db..ad8053abf 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -3,46 +3,163 @@ require_relative '../spec_helper' RSpec.describe 'Heroku CI' do - it 'installs both normal and test dependencies and uses cache on subsequent runs' do - Hatchet::Runner.new('spec/fixtures/ci_requirements', allow_failure: true).run_ci do |test_run| - expect(test_run.output).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) - -----> Python app detected - -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} - To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - -----> Installing python-#{DEFAULT_PYTHON_VERSION} - -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - -----> Installing SQLite3 - -----> Installing requirements with pip - .* - Successfully installed urllib3-.* - -----> Installing test dependencies... - .* - Successfully installed .* pytest-.* - -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. - -----> No test-setup command provided. Skipping. - -----> Running test command `pytest --version`... - pytest .* - -----> test command `pytest --version` completed successfully - REGEX - - test_run.run_again - - expect(test_run.output).to match(Regexp.new(<<~REGEX)) - -----> Python app detected - -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} - To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - -----> No change in requirements detected, installing from cache - -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} - -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - -----> Installing SQLite3 - -----> Installing requirements with pip - -----> Installing test dependencies... - -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. - -----> No test-setup command provided. Skipping. - -----> Running test command `pytest --version`... - pytest .* - -----> test command `pytest --version` completed successfully - REGEX + let(:buildpacks) { [:default, 'heroku-community/inline'] } + + context 'when using pip' do + let(:app) { Hatchet::Runner.new('spec/fixtures/ci_requirements', buildpacks:) } + + it 'installs both normal and test dependencies and uses cache on subsequent runs' do + app.run_ci do |test_run| + expect(test_run.output).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + -----> Python app detected + -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + -----> Installing python-#{DEFAULT_PYTHON_VERSION} + -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + -----> Installing SQLite3 + -----> Installing requirements with pip + .* + Successfully installed typing-extensions-4.12.2 + -----> Installing test dependencies... + .* + Successfully installed .* pytest-8.3.3 + -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. + -----> Running post-compile hook + CI=true + CPLUS_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: + C_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: + DISABLE_COLLECTSTATIC=1 + INSTALL_TEST=1 + LANG=en_US.UTF-8 + LC_ALL=C.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + PATH=/app/.heroku/python/bin:/app/.heroku/vendor/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PIP_NO_PYTHON_VERSION_WARNING=1 + PKG_CONFIG_PATH=/app/.heroku/vendor/lib/pkg-config:/app/.heroku/python/lib/pkg-config: + PYTHONUNBUFFERED=1 + -----> Inline app detected + LANG=en_US.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PYTHONHASHSEED=random + PYTHONHOME=/app/.heroku/python + PYTHONPATH=/app + PYTHONUNBUFFERED=true + -----> No test-setup command provided. Skipping. + -----> Running test command `./bin/print-env-vars.sh && pytest --version`... + CI=true + DYNO_RAM=2560 + FORWARDED_ALLOW_IPS=\\* + GUNICORN_CMD_ARGS=--access-logfile - + LANG=en_US.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ + PYTHONHASHSEED=random + PYTHONHOME=/app/.heroku/python + PYTHONPATH=/app + PYTHONUNBUFFERED=true + WEB_CONCURRENCY=5 + pytest 8.3.3 + -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully + REGEX + + test_run.run_again + + expect(test_run.output).to include(<<~OUTPUT) + -----> Python app detected + -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + -----> No change in requirements detected, installing from cache + -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + -----> Installing SQLite3 + -----> Installing requirements with pip + -----> Installing test dependencies... + -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. + -----> Running post-compile hook + OUTPUT + end + end + end + + context 'when using Pipenv' do + let(:app) { Hatchet::Runner.new('spec/fixtures/ci_pipenv', buildpacks:) } + + it 'installs both normal and test dependencies and uses cache on subsequent runs' do + app.run_ci do |test_run| + expect(test_run.output).to match(Regexp.new(<<~REGEX)) + -----> Python app detected + -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + -----> Installing python-#{DEFAULT_PYTHON_VERSION} + -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + -----> Installing test dependencies + Installing dependencies from Pipfile.lock \\(.+\\)... + Installing dependencies from Pipfile.lock \\(.+\\)... + -----> Installing SQLite3 + -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. + -----> Running post-compile hook + CI=true + CPLUS_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: + C_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: + DISABLE_COLLECTSTATIC=1 + INSTALL_TEST=1 + LANG=en_US.UTF-8 + LC_ALL=C.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + PATH=/app/.heroku/python/bin:/app/.heroku/vendor/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PIP_NO_PYTHON_VERSION_WARNING=1 + PKG_CONFIG_PATH=/app/.heroku/vendor/lib/pkg-config:/app/.heroku/python/lib/pkg-config: + PYTHONUNBUFFERED=1 + SKIP_PIP_INSTALL=1 + -----> Inline app detected + LANG=en_US.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PYTHONHASHSEED=random + PYTHONHOME=/app/.heroku/python + PYTHONPATH=/app + PYTHONUNBUFFERED=true + -----> No test-setup command provided. Skipping. + -----> Running test command `./bin/print-env-vars.sh && pytest --version`... + CI=true + DYNO_RAM=2560 + FORWARDED_ALLOW_IPS=\\* + GUNICORN_CMD_ARGS=--access-logfile - + LANG=en_US.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ + PYTHONHASHSEED=random + PYTHONHOME=/app/.heroku/python + PYTHONPATH=/app + PYTHONUNBUFFERED=true + WEB_CONCURRENCY=5 + pytest 8.3.3 + -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully + REGEX + + test_run.run_again + + expect(test_run.output).to match(Regexp.new(<<~REGEX)) + -----> Python app detected + -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + -----> Installing test dependencies + Installing dependencies from Pipfile.lock \\(.+\\)... + Installing dependencies from Pipfile.lock \\(.+\\)... + -----> Installing SQLite3 + -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. + -----> Running post-compile hook + REGEX + end end end end diff --git a/spec/hatchet/django_spec.rb b/spec/hatchet/django_spec.rb index 865d5b833..bd7feb1a9 100644 --- a/spec/hatchet/django_spec.rb +++ b/spec/hatchet/django_spec.rb @@ -97,10 +97,19 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> \\$ python manage.py collectstatic --noinput remote: Traceback \\(most recent call last\\): - remote: .* + remote: .+ remote: ModuleNotFoundError: No module named 'gettingstarted' remote: remote: ! Error while running '\\$ python manage.py collectstatic --noinput'. + remote: See traceback above for details. + remote: + remote: You may need to update application code to resolve this error. + remote: Or, you can disable collectstatic for this application: + remote: + remote: \\$ heroku config:set DISABLE_COLLECTSTATIC=1 + remote: + remote: https://devcenter.heroku.com/articles/django-assets + remote: ! Push rejected, failed to compile Python app. REGEX end end diff --git a/spec/hatchet/hooks_spec.rb b/spec/hatchet/hooks_spec.rb index dd114a76b..6d66adf54 100644 --- a/spec/hatchet/hooks_spec.rb +++ b/spec/hatchet/hooks_spec.rb @@ -3,49 +3,62 @@ require_relative '../spec_helper' RSpec.describe 'Compile hooks' do - context 'when an app has bin/pre_compile and bin/post_compile scripts' do + # TODO: Run this on Heroku-22 too, once it has also migrated to the new build infrastructure. + # (Currently the test fails on the old infrastructure due to subtle differences in system PATH elements.) + context 'when an app has bin/pre_compile and bin/post_compile scripts', stacks: %w[heroku-20 heroku-24] do let(:app) { Hatchet::Runner.new('spec/fixtures/hooks', config: { 'SOME_APP_CONFIG_VAR' => '1' }) } it 'runs the hooks with the correct environment' do - expected_env_vars = %w[ - _ - BUILD_DIR - BUILDPACK_LOG_FILE - CACHE_DIR - C_INCLUDE_PATH - CPLUS_INCLUDE_PATH - DYNO - ENV_DIR - HOME - LANG - LD_LIBRARY_PATH - LIBRARY_PATH - OLDPWD - PATH - PIP_NO_PYTHON_VERSION_WARNING - PKG_CONFIG_PATH - PWD - PYTHONUNBUFFERED - REQUEST_ID - SHLVL - SOME_APP_CONFIG_VAR - SOURCE_VERSION - STACK - ] - app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + output = clean_output(app.output) + + expect(output).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Running pre-compile hook - remote: pre_compile ran with env vars: - remote: #{expected_env_vars.join("\nremote: ")} - remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} - remote: .* + remote: ~ pre_compile ran with env vars: + remote: BUILD_DIR=/tmp/build_ + remote: CACHE_DIR=/tmp/codon/tmp/cache + remote: C_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: + remote: CPLUS_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: + remote: ENV_DIR=/tmp/... + remote: HOME=/app + remote: LANG=en_US.UTF-8 + remote: LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + remote: LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + remote: PATH=/app/.heroku/python/bin:/app/.heroku/vendor/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PIP_NO_PYTHON_VERSION_WARNING=1 + remote: PKG_CONFIG_PATH=/app/.heroku/vendor/lib/pkg-config:/app/.heroku/python/lib/pkg-config: + remote: PWD=/tmp/build_ + remote: PYTHONUNBUFFERED=1 + remote: SOME_APP_CONFIG_VAR=1 + remote: SOURCE_VERSION=... + remote: STACK=#{app.stack} + remote: ~ pre_compile complete + OUTPUT + + expect(output).to include(<<~OUTPUT) remote: -----> Installing requirements with pip remote: -----> Running post-compile hook - remote: post_compile ran with env vars: - remote: #{expected_env_vars.join("\nremote: ")} - REGEX + remote: ~ post_compile ran with env vars: + remote: BUILD_DIR=/tmp/build_ + remote: CACHE_DIR=/tmp/codon/tmp/cache + remote: C_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: + remote: CPLUS_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: + remote: ENV_DIR=/tmp/... + remote: HOME=/app + remote: LANG=en_US.UTF-8 + remote: LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + remote: LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + remote: PATH=/app/.heroku/python/bin:/app/.heroku/vendor/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PIP_NO_PYTHON_VERSION_WARNING=1 + remote: PKG_CONFIG_PATH=/app/.heroku/vendor/lib/pkg-config:/app/.heroku/python/lib/pkg-config: + remote: PWD=/tmp/build_ + remote: PYTHONUNBUFFERED=1 + remote: SOME_APP_CONFIG_VAR=1 + remote: SOURCE_VERSION=... + remote: STACK=#{app.stack} + remote: ~ post_compile complete + OUTPUT end end end diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index 10cff59b4..d247e1deb 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -35,6 +35,7 @@ remote: ! https://devcenter.heroku.com/articles/getting-started-with-python remote: ! https://devcenter.heroku.com/articles/python-support remote: ! + remote: ! Push rejected, failed to compile Python app. OUTPUT end end diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 704a65c4b..5351975e0 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -12,12 +12,15 @@ end RSpec.describe 'Pip support' do - context 'when requirements.txt is unchanged since the last build' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_unspecified') } + # TODO: Run this on Heroku-22 too, once it has also migrated to the new build infrastructure. + # (Currently the test fails on the old infrastructure due to subtle differences in system PATH elements.) + context 'when requirements.txt is unchanged since the last build', stacks: %w[heroku-20 heroku-24] do + let(:buildpacks) { [:default, 'heroku-community/inline'] } + let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_basic', buildpacks:) } it 're-uses packages from the cache' do app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes @@ -25,12 +28,37 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip - remote: Collecting urllib3 \\(from -r requirements.txt \\(line 1\\)\\) - remote: Downloading urllib3-.* - remote: Downloading urllib3-.* - remote: Installing collected packages: urllib3 - remote: Successfully installed urllib3-.* - REGEX + remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) + remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) + remote: Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) + remote: Installing collected packages: typing-extensions + remote: Successfully installed typing-extensions-4.12.2 + remote: -----> Inline app detected + remote: LANG=en_US.UTF-8 + remote: LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + remote: LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + remote: PATH=/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PYTHONHASHSEED=random + remote: PYTHONHOME=/app/.heroku/python + remote: PYTHONPATH=/app + remote: PYTHONUNBUFFERED=true + remote: + remote: ['', + remote: '/app', + remote: '/app/.heroku/python/lib/python312.zip', + remote: '/app/.heroku/python/lib/python3.12', + remote: '/app/.heroku/python/lib/python3.12/lib-dynload', + remote: '/app/.heroku/python/lib/python3.12/site-packages'] + remote: + remote: Package Version + remote: ----------------- ------- + remote: pip #{PIP_VERSION} + remote: setuptools #{SETUPTOOLS_VERSION} + remote: typing_extensions 4.12.2 + remote: wheel #{WHEEL_VERSION} + remote: + remote: + OUTPUT app.commit! app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) @@ -42,21 +70,21 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip - remote: -----> Discovering process types + remote: -----> Inline app detected OUTPUT end end end context 'when requirements.txt has changed since the last build' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_unspecified') } + let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_basic') } it 'clears the cache before installing the packages again' do app.deploy do |app| File.write('requirements.txt', 'six', mode: 'a') app.commit! app.push! - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes @@ -65,32 +93,50 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip - remote: Collecting urllib3 \\(from -r requirements.txt \\(line 1\\)\\) - remote: Downloading urllib3-.* - remote: Collecting six \\(from -r requirements.txt \\(line 2\\)\\) - remote: Downloading six-.* - remote: Downloading urllib3-.* - remote: Downloading six-.* - remote: Installing collected packages: urllib3, six - remote: Successfully installed six-.* urllib3-.* - REGEX + remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) + remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) + remote: Collecting six (from -r requirements.txt (line 3)) + remote: Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB) + remote: Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) + remote: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) + remote: Installing collected packages: typing-extensions, six + remote: Successfully installed six-1.16.0 typing-extensions-4.12.2 + OUTPUT end end end - context 'when requirements.txt contains popular compiled packages' do - let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_compiled') } + context 'when the package manager has changed from Pipenv to pip since the last build' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_version_unspecified') } - include_examples 'installs successfully using pip' + # TODO: Fix this case so the cache is actually cleared. + it 'clears the cache before installing with pip' do + app.deploy do |app| + FileUtils.rm(['Pipfile', 'Pipfile.lock']) + FileUtils.cp(FIXTURE_DIR.join('requirements_basic/requirements.txt'), '.') + app.commit! + app.push! + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Python app detected + remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing SQLite3 + remote: -----> Installing requirements with pip + remote: -----> Discovering process types + REGEX + end + end end - context 'when requirements.txt contains Git requirements URLs' do - let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_git') } + context 'when requirements.txt contains popular compiled packages' do + let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_compiled') } include_examples 'installs successfully using pip' end - context 'when requirements.txt contains editable requirements' do + context 'when requirements.txt contains editable requirements (both VCS and local package)' do let(:buildpacks) { [:default, 'heroku-community/inline'] } let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_editable', buildpacks:) } @@ -120,7 +166,7 @@ REGEX # Test rewritten paths work at runtime. - expect(app.run('bin/test-entrypoints')).to include(<<~OUTPUT) + expect(app.run('bin/test-entrypoints.sh')).to include(<<~OUTPUT) easy-install.pth:/app/.heroku/src/gunicorn easy-install.pth:/app/packages/local_package_setup_py __editable___local_package_pyproject_toml_0_0_1_finder.py:/app/packages/local_package_pyproject_toml/local_package_pyproject_toml'} @@ -166,8 +212,21 @@ it 'installs packages from setup.py' do app.deploy do |app| - expect(app.output).to include('Running setup.py develop for test') - expect(app.output).to include('Successfully installed six') + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + remote: -----> Python app detected + remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing SQLite3 + remote: -----> Installing requirements with pip + remote: Obtaining file:///tmp/build_.* \\(from -r requirements.txt \\(line 1\\)\\) + remote: Preparing metadata \\(setup.py\\): started + remote: Preparing metadata \\(setup.py\\): finished with status 'done' + remote: .+ + remote: Installing collected packages: six, test + remote: Running setup.py develop for test + REGEX end end end @@ -186,6 +245,20 @@ end end + context 'when requirements.txt contains an invalid requirement' do + let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_invalid', allow_failure: true) } + + it 'aborts the build and displays the pip error' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Installing requirements with pip + remote: ERROR: Invalid requirement: 'an-invalid-requirement!' (from line 1 of requirements.txt) + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + context 'when requirements.txt contains GDAL but the GDAL C++ library is missing' do let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_gdal', allow_failure: true) } @@ -195,6 +268,7 @@ remote: ! Hello! Package installation failed since the GDAL library was not found. remote: ! For GDAL, GEOS and PROJ support, use the Geo buildpack alongside the Python buildpack: remote: ! https://github.com/heroku/heroku-geo-buildpack + remote: ! Push rejected, failed to compile Python app. OUTPUT end end diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index ff5b5ba1c..6c0e2fa97 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -30,6 +30,7 @@ remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes remote: ! + remote: ! Push rejected, failed to compile Python app. OUTPUT end end @@ -63,10 +64,14 @@ end context 'with a Pipfile.lock but no Python version specified' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_version_unspecified') } + let(:buildpacks) { [:default, 'heroku-community/inline'] } + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_version_unspecified', buildpacks:) } - it 'builds with the default Python version' do + # TODO: Run this on Heroku-22 too, once it has also migrated to the new build infrastructure. + # (Currently the test fails on the old infrastructure due to subtle differences in system PATH elements.) + it 'builds with the default Python version', stacks: %w[heroku-20 heroku-24] do app.deploy do |app| + # TODO: We should not be leaking the Pipenv installation into the app environment. expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} @@ -76,6 +81,37 @@ remote: -----> Installing dependencies with Pipenv #{PIPENV_VERSION} remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Installing SQLite3 + remote: -----> Inline app detected + remote: LANG=en_US.UTF-8 + remote: LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + remote: LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + remote: PATH=/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PYTHONHASHSEED=random + remote: PYTHONHOME=/app/.heroku/python + remote: PYTHONPATH=/app + remote: PYTHONUNBUFFERED=true + remote: + remote: \\['', + remote: '/app', + remote: '/app/.heroku/python/lib/python312.zip', + remote: '/app/.heroku/python/lib/python3.12', + remote: '/app/.heroku/python/lib/python3.12/lib-dynload', + remote: '/app/.heroku/python/lib/python3.12/site-packages'\\] + remote: + remote: Package Version + remote: ----------------- --------- + remote: certifi .+ + remote: distlib .+ + remote: filelock .+ + remote: pip #{PIP_VERSION} + remote: pipenv #{PIPENV_VERSION} + remote: platformdirs .+ + remote: setuptools #{SETUPTOOLS_VERSION} + remote: typing_extensions 4.12.2 + remote: virtualenv .+ + remote: wheel #{WHEEL_VERSION} + remote: + remote: \\ REGEX end end @@ -101,6 +137,7 @@ remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes remote: ! + remote: ! Push rejected, failed to compile Python app. OUTPUT end end @@ -126,6 +163,7 @@ remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes remote: ! + remote: ! Push rejected, failed to compile Python app. OUTPUT end end @@ -207,7 +245,7 @@ let(:allow_failure) { false } let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_full_version', allow_failure:) } - it 'builds with the outdated Python version specified' do + it 'builds with the outdated Python version specified and displays a deprecation warning' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected @@ -241,6 +279,7 @@ remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes remote: ! + remote: ! Push rejected, failed to compile Python app. OUTPUT end end @@ -260,6 +299,7 @@ remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes remote: ! + remote: ! Push rejected, failed to compile Python app. OUTPUT end end @@ -305,6 +345,32 @@ end end + context 'when the package manager has changed from pip to Pipenv since the last build' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_unspecified') } + + # TODO: Fix this case so the cache is actually cleared. + it 'clears the cache before installing with Pipenv' do + app.deploy do |app| + FileUtils.rm('requirements.txt') + FileUtils.cp(FIXTURE_DIR.join('pipenv_python_version_unspecified/Pipfile'), '.') + FileUtils.cp(FIXTURE_DIR.join('pipenv_python_version_unspecified/Pipfile.lock'), '.') + app.commit! + app.push! + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Python app detected + remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing dependencies with Pipenv #{PIPENV_VERSION} + remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Installing SQLite3 + remote: -----> Discovering process types + REGEX + end + end + end + context 'when there is both a Pipfile.lock and a requirements.txt' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_and_requirements_txt') } @@ -338,6 +404,7 @@ remote: Your Pipfile.lock \\(.+\\) is out of date. Expected: \\(.+\\). remote: .+ remote: ERROR:: Aborting deploy + remote: ! Push rejected, failed to compile Python app. REGEX end end @@ -373,7 +440,7 @@ REGEX # Test rewritten paths work at runtime. - expect(app.run('bin/test-entrypoints')).to include(<<~OUTPUT) + expect(app.run('bin/test-entrypoints.sh')).to include(<<~OUTPUT) easy-install.pth:/app/.heroku/src/gunicorn easy-install.pth:/app/packages/local_package_setup_py __editable___local_package_pyproject_toml_0_0_1_finder.py:/app/packages/local_package_pyproject_toml/local_package_pyproject_toml'} diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index 6c7f66d62..df909ee73 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -30,13 +30,17 @@ remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes remote: ! + remote: ! Push rejected, failed to compile Python app. OUTPUT end end end +# NOTE: We use the oldest patch releases (ie the '.0' releases) since we also want to test against +# the oldest Python versions available to users. This is particularly important given that older +# patch releases will bundle older pip, and the buildpack uses that pip during bootstrapping. RSpec.describe 'Python update warnings' do - context 'with a runtime.txt containing python-3.8.12' do + context 'with a runtime.txt containing python-3.8.0' do let(:allow_failure) { false } let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.8_outdated', allow_failure:) } @@ -60,7 +64,7 @@ remote: ! A Python security update is available! Upgrade as soon as possible to: python-#{LATEST_PYTHON_3_8} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! - remote: -----> Installing python-3.8.12 + remote: -----> Installing python-3.8.0 REGEX end end @@ -70,31 +74,31 @@ let(:allow_failure) { true } # We only support Python 3.8 on Heroku-20 and older. - include_examples 'aborts the build without showing an update warning', '3.8.12' + include_examples 'aborts the build without showing an update warning', '3.8.0' end end - context 'with a runtime.txt containing python-3.9.12' do + context 'with a runtime.txt containing python-3.9.0' do let(:allow_failure) { false } let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.9_outdated', allow_failure:) } context 'when using Heroku-22 or older', stacks: %w[heroku-20 heroku-22] do - include_examples 'warns there is a Python update available', '3.9.12', LATEST_PYTHON_3_9 + include_examples 'warns there is a Python update available', '3.9.0', LATEST_PYTHON_3_9 end context 'when using Heroku-24', stacks: %w[heroku-24] do let(:allow_failure) { true } # We only support Python 3.9 on Heroku-22 and older. - include_examples 'aborts the build without showing an update warning', '3.9.12' + include_examples 'aborts the build without showing an update warning', '3.9.0' end end - context 'with a runtime.txt containing python-3.10.5' do + context 'with a runtime.txt containing python-3.10.0' do let(:allow_failure) { false } let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.10_outdated', allow_failure:) } - include_examples 'warns there is a Python update available', '3.10.5', LATEST_PYTHON_3_10 + include_examples 'warns there is a Python update available', '3.10.0', LATEST_PYTHON_3_10 end context 'with a runtime.txt containing python-3.11.0' do diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index f4d96c376..a324089ba 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -31,6 +31,7 @@ remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes remote: ! + remote: ! Push rejected, failed to compile Python app. OUTPUT end end @@ -102,6 +103,7 @@ remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes remote: ! + remote: ! Push rejected, failed to compile Python app. OUTPUT end end @@ -127,6 +129,7 @@ remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes remote: ! + remote: ! Push rejected, failed to compile Python app. OUTPUT end end diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 6d72bc8ee..93e3b2bdb 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -7,6 +7,8 @@ require 'rspec/retry' require 'hatchet' +FIXTURE_DIR = Pathname.new(__FILE__).parent.join('fixtures') + LATEST_PYTHON_3_8 = '3.8.20' LATEST_PYTHON_3_9 = '3.9.20' LATEST_PYTHON_3_10 = '3.10.15' From b9f7f3f034e101a3d9186750d89ad6dda3651c24 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 23 Sep 2024 08:41:20 +0000 Subject: [PATCH 014/406] Add more files to the buildpack publishing file exclusion list (#1637) Since these files aren't needed for the buildpack to run, so can be excluded from the archive published to the buildpack registry. --- buildpack.toml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/buildpack.toml b/buildpack.toml index 0464f13c9..07f83d4cf 100644 --- a/buildpack.toml +++ b/buildpack.toml @@ -5,9 +5,11 @@ name = "Python" files = [ ".github/", "builds/", + "etc/publish.sh", "spec/", ".gitignore", ".rubocop.yml", + ".shellcheckrc", "Gemfile", "Gemfile.lock", "hatchet.json", From 3da3b3adb8b33f0133e876b9830ef28909f9e229 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 23 Sep 2024 12:25:30 +0000 Subject: [PATCH 015/406] Format shell scripts using shfmt (#1638) Shell scripts are now formatted using shfmt: https://github.com/mvdan/sh/blob/master/cmd/shfmt/shfmt.1.scd This is enforced in CI, and formatting can be applied locally using a new `make format` target. Notably, this also reformats the scripts to use tabs instead of spaces, since tabs are sadly the only way to be able to properly indent bash here documents (which is something the scripts here will be using a lot of soon) without having to resort to mixed tabs and spaces indentation in the same file. (Plus tabs is also the shfmt default style.) GUS-W-16808198. --- .editorconfig | 35 ++++ .github/workflows/ci.yml | 8 + Makefile | 10 +- bin/compile | 128 ++++++------ bin/detect | 38 ++-- bin/release | 10 +- bin/report | 104 +++++----- bin/steps/collectstatic | 42 ++-- bin/steps/hooks/post_compile | 14 +- bin/steps/hooks/pre_compile | 14 +- bin/steps/nltk | 26 +-- bin/steps/pip-install | 80 ++++---- bin/steps/pipenv | 84 ++++---- bin/steps/pipenv-python-version | 107 +++++----- bin/steps/python | 228 +++++++++++----------- bin/steps/sqlite3 | 140 ++++++------- bin/utils | 49 +++-- bin/warnings | 14 +- builds/build_python_runtime.sh | 212 ++++++++++---------- builds/test_python_runtime.sh | 39 ++-- etc/publish.sh | 6 +- lib/kvstore.sh | 94 ++++----- lib/metadata.sh | 52 ++--- spec/fixtures/hooks/bin/print-env-vars.sh | 10 +- vendor/WEB_CONCURRENCY.sh | 66 +++---- vendor/buildpack-stdlib_v8.sh | 128 ++++++------ 26 files changed, 893 insertions(+), 845 deletions(-) create mode 100644 .editorconfig diff --git a/.editorconfig b/.editorconfig new file mode 100644 index 000000000..0fb218967 --- /dev/null +++ b/.editorconfig @@ -0,0 +1,35 @@ +# https://editorconfig.org +root = true + +[*] +charset = utf-8 +end_of_line = lf +indent_style = space +insert_final_newline = true +trim_trailing_whitespace = true + +[*.sh] +binary_next_line = true +# We sadly have to use tabs in shell scripts otherwise we can't indent here documents: +# https://www.gnu.org/software/bash/manual/html_node/Redirections.html#Here-Documents +indent_style = tab +shell_variant = bash +switch_case_indent = true + +# Catches scripts that we can't give a .sh file extension, such as the Buildpack API scripts. +[**/bin/**] +binary_next_line = true +indent_style = tab +shell_variant = bash +switch_case_indent = true + +[.hatchet/repos/**] +ignore = true + +# The setup-ruby GitHub Action creates this directory when caching is enabled, and if +# its not ignored will cause false positives when running shfmt in the CI lint job. +[vendor/bundle/**] +ignore = true + +[Makefile] +indent_style = tab diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d7f97513f..56417d095 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -10,6 +10,10 @@ on: permissions: contents: read +env: + # Used by shfmt and more. + FORCE_COLOR: 1 + jobs: lint: runs-on: ubuntu-24.04 @@ -23,6 +27,10 @@ jobs: ruby-version: "3.3" - name: Run ShellCheck run: make lint-scripts + - name: Run shfmt + uses: docker://mvdan/shfmt:latest + with: + args: "--diff ." - name: Run Rubocop run: bundle exec rubocop diff --git a/Makefile b/Makefile index 1b4741ed9..009bffae9 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,5 @@ # These targets are not files -.PHONY: lint lint-scripts lint-ruby run publish +.PHONY: lint lint-scripts lint-ruby check-format format run publish STACK ?= heroku-24 FIXTURE ?= spec/fixtures/python_version_unspecified @@ -7,7 +7,7 @@ FIXTURE ?= spec/fixtures/python_version_unspecified # Converts a stack name of `heroku-NN` to its build Docker image tag of `heroku/heroku:NN-build`. STACK_IMAGE_TAG := heroku/$(subst -,:,$(STACK))-build -lint: lint-scripts lint-ruby +lint: lint-scripts check-format lint-ruby lint-scripts: @git ls-files -z --cached --others --exclude-standard 'bin/*' '*/bin/*' '*.sh' | xargs -0 shellcheck --check-sourced --color=always @@ -15,6 +15,12 @@ lint-scripts: lint-ruby: @bundle exec rubocop +check-format: + @shfmt --diff . + +format: + @shfmt --write --list . + run: @echo "Running buildpack using: STACK=$(STACK) FIXTURE=$(FIXTURE)" @docker run --rm -it -v $(PWD):/src:ro --tmpfs /app -e "HOME=/app" -e "STACK=$(STACK)" "$(STACK_IMAGE_TAG)" \ diff --git a/bin/compile b/bin/compile index f9c219c0c..c89420201 100755 --- a/bin/compile +++ b/bin/compile @@ -103,16 +103,16 @@ mkdir -p "$CACHE_DIR/.heroku" mkdir -p .heroku # The Python installation. -cp -R "$CACHE_DIR/.heroku/python" .heroku/ &> /dev/null || true +cp -R "$CACHE_DIR/.heroku/python" .heroku/ &>/dev/null || true # A plain text file which contains the current stack being used (used for cache busting). -cp -R "$CACHE_DIR/.heroku/python-stack" .heroku/ &> /dev/null || true +cp -R "$CACHE_DIR/.heroku/python-stack" .heroku/ &>/dev/null || true # A plain text file which contains the current python version being used (used for cache busting). -cp -R "$CACHE_DIR/.heroku/python-version" .heroku/ &> /dev/null || true +cp -R "$CACHE_DIR/.heroku/python-version" .heroku/ &>/dev/null || true # A plain text file which contains the current sqlite3 version being used (used for cache busting). -cp -R "$CACHE_DIR/.heroku/python-sqlite3-version" .heroku/ &> /dev/null || true +cp -R "$CACHE_DIR/.heroku/python-sqlite3-version" .heroku/ &>/dev/null || true # "editable" installations of code repositories, via pip or pipenv. if [[ -d "$CACHE_DIR/.heroku/src" ]]; then - cp -R "$CACHE_DIR/.heroku/src" .heroku/ &> /dev/null || true + cp -R "$CACHE_DIR/.heroku/src" .heroku/ &>/dev/null || true fi # The pre_compile hook. Customers rely on this. Don't remove it. @@ -124,47 +124,47 @@ source "${BUILDPACK_DIR}/bin/steps/hooks/pre_compile" # Sticky runtimes. If there was a previous build, and it used a given version of Python, # continue to use that version of Python in perpetuity. if [[ -f "$CACHE_DIR/.heroku/python-version" ]]; then - CACHED_PYTHON_VERSION=$(cat "$CACHE_DIR/.heroku/python-version") + CACHED_PYTHON_VERSION=$(cat "$CACHE_DIR/.heroku/python-version") fi # We didn't always record the stack version. This code is in place because of that. if [[ -f "$CACHE_DIR/.heroku/python-stack" ]]; then - CACHED_PYTHON_STACK=$(cat "$CACHE_DIR/.heroku/python-stack") + CACHED_PYTHON_STACK=$(cat "$CACHE_DIR/.heroku/python-stack") else - CACHED_PYTHON_STACK=$STACK + CACHED_PYTHON_STACK=$STACK fi # TODO: Move this into a new package manager handling implementation when adding Poetry support. # We intentionally don't mention `setup.py` here since it's being removed soon. if [[ ! -f requirements.txt && ! -f Pipfile && ! -f setup.py ]]; then - puts-warn - puts-warn "Error: Couldn't find any supported Python package manager files." - puts-warn - puts-warn "A Python app on Heroku must have either a 'requirements.txt' or" - puts-warn "'Pipfile' package manager file in the root directory of its" - puts-warn "source code." - puts-warn - puts-warn "Currently the root directory of your app contains:" - puts-warn - # TODO: Overhaul logging helpers so they can handle prefixing multi-line strings, and switch to them. - # shellcheck disable=SC2012 # Using `ls` instead of `find` is absolutely fine for this use case. - ls -1 --indicator-style=slash "${BUILD_DIR}" | sed 's/^/ ! /' - puts-warn - puts-warn "If your app already has a package manager file, check that it:" - puts-warn - puts-warn "1. Is in the top level directory (not a subdirectory)." - puts-warn "2. Has the correct spelling (the filenames are case-sensitive)." - puts-warn "3. Isn't listed in '.gitignore' or '.slugignore'." - puts-warn - puts-warn "Otherwise, add a package manager file to your app. If your app has" - puts-warn "no dependencies, then create an empty 'requirements.txt' file." - puts-warn - puts-warn "For help with using Python on Heroku, see:" - puts-warn "https://devcenter.heroku.com/articles/getting-started-with-python" - puts-warn "https://devcenter.heroku.com/articles/python-support" - puts-warn - meta_set "failure_reason" "package-manager-not-found" - exit 1 + puts-warn + puts-warn "Error: Couldn't find any supported Python package manager files." + puts-warn + puts-warn "A Python app on Heroku must have either a 'requirements.txt' or" + puts-warn "'Pipfile' package manager file in the root directory of its" + puts-warn "source code." + puts-warn + puts-warn "Currently the root directory of your app contains:" + puts-warn + # TODO: Overhaul logging helpers so they can handle prefixing multi-line strings, and switch to them. + # shellcheck disable=SC2012 # Using `ls` instead of `find` is absolutely fine for this use case. + ls -1 --indicator-style=slash "${BUILD_DIR}" | sed 's/^/ ! /' + puts-warn + puts-warn "If your app already has a package manager file, check that it:" + puts-warn + puts-warn "1. Is in the top level directory (not a subdirectory)." + puts-warn "2. Has the correct spelling (the filenames are case-sensitive)." + puts-warn "3. Isn't listed in '.gitignore' or '.slugignore'." + puts-warn + puts-warn "Otherwise, add a package manager file to your app. If your app has" + puts-warn "no dependencies, then create an empty 'requirements.txt' file." + puts-warn + puts-warn "For help with using Python on Heroku, see:" + puts-warn "https://devcenter.heroku.com/articles/getting-started-with-python" + puts-warn "https://devcenter.heroku.com/articles/python-support" + puts-warn + meta_set "failure_reason" "package-manager-not-found" + exit 1 fi # Pipenv Python version support. @@ -173,21 +173,21 @@ fi source "${BUILDPACK_DIR}/bin/steps/pipenv-python-version" if [[ -f runtime.txt ]]; then - # PYTHON_VERSION_SOURCE may have already been set by the pipenv-python-version step. - # TODO: Refactor this and stop pipenv-python-version using runtime.txt as an API. - PYTHON_VERSION_SOURCE=${PYTHON_VERSION_SOURCE:-"runtime.txt"} - puts-step "Using Python version specified in ${PYTHON_VERSION_SOURCE}" - meta_set "python_version_reason" "specified" + # PYTHON_VERSION_SOURCE may have already been set by the pipenv-python-version step. + # TODO: Refactor this and stop pipenv-python-version using runtime.txt as an API. + PYTHON_VERSION_SOURCE=${PYTHON_VERSION_SOURCE:-"runtime.txt"} + puts-step "Using Python version specified in ${PYTHON_VERSION_SOURCE}" + meta_set "python_version_reason" "specified" elif [[ -n "${CACHED_PYTHON_VERSION:-}" ]]; then - puts-step "No Python version was specified. Using the same version as the last build: ${CACHED_PYTHON_VERSION}" - echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" - meta_set "python_version_reason" "cached" - echo "${CACHED_PYTHON_VERSION}" > runtime.txt + puts-step "No Python version was specified. Using the same version as the last build: ${CACHED_PYTHON_VERSION}" + echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" + meta_set "python_version_reason" "cached" + echo "${CACHED_PYTHON_VERSION}" >runtime.txt else - puts-step "No Python version was specified. Using the buildpack default: ${DEFAULT_PYTHON_VERSION}" - echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" - meta_set "python_version_reason" "default" - echo "${DEFAULT_PYTHON_VERSION}" > runtime.txt + puts-step "No Python version was specified. Using the buildpack default: ${DEFAULT_PYTHON_VERSION}" + echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" + meta_set "python_version_reason" "default" + echo "${DEFAULT_PYTHON_VERSION}" >runtime.txt fi # Create the directory for .profile.d, if it doesn't exist. @@ -201,11 +201,11 @@ mkdir -p /app/.heroku/src # This is (hopefully obviously) because apps end up running from `/app` in production. # Realpath is used to support use-cases where one of the locations is a symlink to the other. if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then - # python expects to reside in /app, so set up symlinks - # we will not remove these later so subsequent buildpacks can still invoke it - ln -nsf "$BUILD_DIR/.heroku/python" /app/.heroku/python - ln -nsf "$BUILD_DIR/.heroku/vendor" /app/.heroku/vendor - # Note: .heroku/src is copied in later. + # python expects to reside in /app, so set up symlinks + # we will not remove these later so subsequent buildpacks can still invoke it + ln -nsf "$BUILD_DIR/.heroku/python" /app/.heroku/python + ln -nsf "$BUILD_DIR/.heroku/vendor" /app/.heroku/vendor + # Note: .heroku/src is copied in later. fi # Download / Install Python, from pre-build binaries available on Amazon S3. @@ -221,10 +221,10 @@ source "${BUILDPACK_DIR}/bin/steps/pipenv" # This allows for people to ship a setup.py application to Heroku if [[ ! -f requirements.txt ]] && [[ ! -f Pipfile ]]; then - meta_set "setup_py_only" "true" - echo "-e ." > requirements.txt + meta_set "setup_py_only" "true" + echo "-e ." >requirements.txt else - meta_set "setup_py_only" "false" + meta_set "setup_py_only" "false" fi # SQLite3 support. @@ -251,11 +251,10 @@ meta_time "nltk_downloader_duration" "${nltk_downloader_start_time}" # In CI, $BUILD_DIR is /app. # Realpath is used to support use-cases where one of the locations is a symlink to the other. if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then - rm -rf "$BUILD_DIR/.heroku/src" - deep-cp /app/.heroku/src "$BUILD_DIR/.heroku/src" + rm -rf "$BUILD_DIR/.heroku/src" + deep-cp /app/.heroku/src "$BUILD_DIR/.heroku/src" fi - # Django collectstatic support. # The buildpack automatically runs collectstatic for Django applications. # This is the cause for the majority of build failures on the Python platform. @@ -265,7 +264,6 @@ collectstatic_start_time=$(nowms) sub_env "${BUILDPACK_DIR}/bin/steps/collectstatic" meta_time "django_collectstatic_duration" "${collectstatic_start_time}" - # Programmatically create .profile.d script for application runtime environment variables. # Set the PATH to include Python / pip / pipenv / etc. @@ -286,7 +284,7 @@ set_default_env PYTHONPATH "\$HOME" # Python expects to be in /app, if at runtime, it is not, set # up symlinks… this can occur when the subdir buildpack is used. -cat <> "$PROFILE_PATH" +cat <>"$PROFILE_PATH" if [[ \$HOME != "/app" ]]; then mkdir -p /app/.heroku ln -nsf "\$HOME/.heroku/python" /app/.heroku/python @@ -298,7 +296,7 @@ EOT # (such as `/tmp/build_`) back to `/app`. This is not done during the build itself, since later # buildpacks still need the build time paths. if [[ "${BUILD_DIR}" != "/app" ]]; then - cat <> "$PROFILE_PATH" + cat <>"$PROFILE_PATH" find .heroku/python/lib/python*/site-packages/ -type f -and \( -name '*.egg-link' -or -name '*.pth' -or -name '__editable___*_finder.py' \) -exec sed -i -e 's#${BUILD_DIR}#/app#' {} \+ EOT fi @@ -320,9 +318,9 @@ rm -rf "$CACHE_DIR/.heroku/src" mkdir -p "$CACHE_DIR/.heroku" cp -R .heroku/python "$CACHE_DIR/.heroku/" cp -R .heroku/python-version "$CACHE_DIR/.heroku/" -cp -R .heroku/python-stack "$CACHE_DIR/.heroku/" &> /dev/null || true +cp -R .heroku/python-stack "$CACHE_DIR/.heroku/" &>/dev/null || true if [[ -d .heroku/src ]]; then - cp -R .heroku/src "$CACHE_DIR/.heroku/" &> /dev/null || true + cp -R .heroku/src "$CACHE_DIR/.heroku/" &>/dev/null || true fi meta_time "total_duration" "${compile_start_time}" diff --git a/bin/detect b/bin/detect index d7b27069f..498792bab 100755 --- a/bin/detect +++ b/bin/detect @@ -13,27 +13,27 @@ BUILD_DIR="${1}" # so that Python projects that are missing some of the required files still pass detection, # allowing us to show a helpful error message during the build phase. KNOWN_PYTHON_PROJECT_FILES=( - .python-version - app.py - main.py - manage.py - pdm.lock - Pipfile - Pipfile.lock - poetry.lock - pyproject.toml - requirements.txt - runtime.txt - setup.cfg - setup.py - uv.lock + .python-version + app.py + main.py + manage.py + pdm.lock + Pipfile + Pipfile.lock + poetry.lock + pyproject.toml + requirements.txt + runtime.txt + setup.cfg + setup.py + uv.lock ) for filename in "${KNOWN_PYTHON_PROJECT_FILES[@]}"; do - if [[ -f "${BUILD_DIR}/${filename}" ]]; then - echo "Python" - exit 0 - fi + if [[ -f "${BUILD_DIR}/${filename}" ]]; then + echo "Python" + exit 0 + fi done # Cytokine incorrectly indents the first line, so we have to leave it empty. @@ -43,7 +43,7 @@ echo 1>&2 # since during compile the build will still require a package manager file, so it # makes sense to describe the stricter requirements up front. # TODO: Overhaul logging helpers so they can handle prefixing multi-line strings, and switch to them. -sed 's/^/ ! /' 1>&2 << EOF +sed 's/^/ ! /' 1>&2 </dev/null; then - # Determine pysqlite3 usage since it's the only package that requires the sqlite3 headers. - if pip show pysqlite3 &>/dev/null; then - kv_pair pysqlite3_installed true - else - kv_pair pysqlite3_installed false - fi + # Determine pysqlite3 usage since it's the only package that requires the sqlite3 headers. + if pip show pysqlite3 &>/dev/null; then + kv_pair pysqlite3_installed true + else + kv_pair pysqlite3_installed false + fi - if pip show pysqlite3-binary &>/dev/null; then - kv_pair pysqlite3_binary_installed true - else - kv_pair pysqlite3_binary_installed false - fi + if pip show pysqlite3-binary &>/dev/null; then + kv_pair pysqlite3_binary_installed true + else + kv_pair pysqlite3_binary_installed false + fi fi diff --git a/bin/steps/collectstatic b/bin/steps/collectstatic index 5731408a4..41b33fe02 100755 --- a/bin/steps/collectstatic +++ b/bin/steps/collectstatic @@ -22,21 +22,21 @@ source "${BUILDPACK_DIR}/lib/metadata.sh" meta_init "${CACHE_DIR}" "python" if [[ -f .heroku/collectstatic_disabled ]]; then - puts-step "Skipping Django collectstatic since the file '.heroku/collectstatic_disabled' exists." - puts-warn "This approach is deprecated, please set the env var DISABLE_COLLECTSTATIC=1 instead." - meta_set "django_collectstatic" "disabled-file" - exit 0 + puts-step "Skipping Django collectstatic since the file '.heroku/collectstatic_disabled' exists." + puts-warn "This approach is deprecated, please set the env var DISABLE_COLLECTSTATIC=1 instead." + meta_set "django_collectstatic" "disabled-file" + exit 0 fi if [[ "${DISABLE_COLLECTSTATIC:-0}" != "0" ]]; then - puts-step "Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set." - meta_set "django_collectstatic" "disabled-env-var" - exit 0 + puts-step "Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set." + meta_set "django_collectstatic" "disabled-env-var" + exit 0 fi # Ensure that Django is actually installed. if ! is_module_available 'django'; then - exit 0 + exit 0 fi # Location of 'manage.py', if it exists. @@ -44,9 +44,9 @@ MANAGE_FILE=$(find . -maxdepth 3 -type f -name 'manage.py' -printf '%d\t%P\n' | MANAGE_FILE=${MANAGE_FILE:-fakepath} if [[ ! -f "${MANAGE_FILE}" ]]; then - puts-step "Skipping Django collectstatic since no manage.py file found." - meta_set "django_collectstatic" "skipped-no-manage-py" - exit 0 + puts-step "Skipping Django collectstatic since no manage.py file found." + meta_set "django_collectstatic" "skipped-no-manage-py" + exit 0 fi meta_set "django_collectstatic" "enabled" @@ -65,20 +65,20 @@ set -e echo if [[ "${COLLECTSTATIC_STATUS}" == 0 ]]; then - exit 0 + exit 0 fi # Display a warning if collectstatic failed. if grep -q 'SyntaxError' "$COLLECTSTATIC_LOG"; then - meta_set "failure_reason" "collectstatic-syntax-error" + meta_set "failure_reason" "collectstatic-syntax-error" elif grep -q 'ImproperlyConfigured' "$COLLECTSTATIC_LOG"; then - meta_set "failure_reason" "collectstatic-improper-configuration" + meta_set "failure_reason" "collectstatic-improper-configuration" elif grep -q 'The CSS file' "$COLLECTSTATIC_LOG"; then - meta_set "failure_reason" "collectstatic-fancy-references" + meta_set "failure_reason" "collectstatic-fancy-references" elif grep -q 'OSError' "$COLLECTSTATIC_LOG"; then - meta_set "failure_reason" "collectstatic-missing-file" + meta_set "failure_reason" "collectstatic-missing-file" else - meta_set "failure_reason" "collectstatic-other" + meta_set "failure_reason" "collectstatic-other" fi echo " ! Error while running '$ python $MANAGE_FILE collectstatic --noinput'." @@ -93,10 +93,10 @@ echo " https://devcenter.heroku.com/articles/django-assets" # Additionally, dump out the environment, if debug mode is on. if [[ -n "$DEBUG_COLLECTSTATIC" ]]; then - echo - echo "****** Collectstatic environment variables:" - echo - env | indent + echo + echo "****** Collectstatic environment variables:" + echo + env | indent fi exit 1 diff --git a/bin/steps/hooks/post_compile b/bin/steps/hooks/post_compile index 111491ee7..7a20ed0a9 100755 --- a/bin/steps/hooks/post_compile +++ b/bin/steps/hooks/post_compile @@ -1,12 +1,12 @@ #!/usr/bin/env bash if [[ -f bin/post_compile ]]; then - post_compile_hook_start_time=$(nowms) - meta_set "post_compile_hook" "true" - echo "-----> Running post-compile hook" - chmod +x bin/post_compile - sub_env bin/post_compile - meta_time "post_compile_hook_duration" "${post_compile_hook_start_time}" + post_compile_hook_start_time=$(nowms) + meta_set "post_compile_hook" "true" + echo "-----> Running post-compile hook" + chmod +x bin/post_compile + sub_env bin/post_compile + meta_time "post_compile_hook_duration" "${post_compile_hook_start_time}" else - meta_set "post_compile_hook" "false" + meta_set "post_compile_hook" "false" fi diff --git a/bin/steps/hooks/pre_compile b/bin/steps/hooks/pre_compile index c22eb1102..c5be04ce3 100755 --- a/bin/steps/hooks/pre_compile +++ b/bin/steps/hooks/pre_compile @@ -1,12 +1,12 @@ #!/usr/bin/env bash if [[ -f bin/pre_compile ]]; then - pre_compile_hook_start_time=$(nowms) - meta_set "pre_compile_hook" "true" - echo "-----> Running pre-compile hook" - chmod +x bin/pre_compile - sub_env bin/pre_compile - meta_time "pre_compile_hook_duration" "${pre_compile_hook_start_time}" + pre_compile_hook_start_time=$(nowms) + meta_set "pre_compile_hook" "true" + echo "-----> Running pre-compile hook" + chmod +x bin/pre_compile + sub_env bin/pre_compile + meta_time "pre_compile_hook_duration" "${pre_compile_hook_start_time}" else - meta_set "pre_compile_hook" "false" + meta_set "pre_compile_hook" "false" fi diff --git a/bin/steps/nltk b/bin/steps/nltk index f04e60cde..f5f939efa 100755 --- a/bin/steps/nltk +++ b/bin/steps/nltk @@ -17,22 +17,22 @@ EXPORT_PATH="${BUILDPACK_DIR}/export" # Check that nltk was installed by pip, otherwise obviously not needed if is_module_available 'nltk'; then - puts-step "Downloading NLTK corpora..." + puts-step "Downloading NLTK corpora..." - nltk_packages_definition="$BUILD_DIR/nltk.txt" + nltk_packages_definition="$BUILD_DIR/nltk.txt" - if [[ -f "$nltk_packages_definition" ]]; then - meta_set "nltk_downloader" "enabled" + if [[ -f "$nltk_packages_definition" ]]; then + meta_set "nltk_downloader" "enabled" - readarray -t nltk_packages < "$nltk_packages_definition" - puts-step "Downloading NLTK packages: ${nltk_packages[*]}" + readarray -t nltk_packages <"$nltk_packages_definition" + puts-step "Downloading NLTK packages: ${nltk_packages[*]}" - python -m nltk.downloader -d "$BUILD_DIR/.heroku/python/nltk_data" "${nltk_packages[@]}" | indent - set_env NLTK_DATA "/app/.heroku/python/nltk_data" + python -m nltk.downloader -d "$BUILD_DIR/.heroku/python/nltk_data" "${nltk_packages[@]}" | indent + set_env NLTK_DATA "/app/.heroku/python/nltk_data" - else - meta_set "nltk_downloader" "skipped-no-nltk-file" - puts-warn "'nltk.txt' not found, not downloading any corpora" - puts-warn "Learn more: https://devcenter.heroku.com/articles/python-nltk" - fi + else + meta_set "nltk_downloader" "skipped-no-nltk-file" + puts-warn "'nltk.txt' not found, not downloading any corpora" + puts-warn "Learn more: https://devcenter.heroku.com/articles/python-nltk" + fi fi diff --git a/bin/steps/pip-install b/bin/steps/pip-install index e180e649d..86ecfd0d3 100755 --- a/bin/steps/pip-install +++ b/bin/steps/pip-install @@ -1,44 +1,44 @@ #!/usr/bin/env bash if [[ -z "$SKIP_PIP_INSTALL" ]]; then - pip_install_start_time=$(nowms) - meta_set "package_manager" "pip" - - puts-step "Installing requirements with pip" - - # Set Pip env vars - # This reads certain environment variables set on the Heroku app config - # and makes them accessible to the pip install process. - # - # PIP_EXTRA_INDEX_URL allows for an alternate pypi URL to be used. - if [[ -r "$ENV_DIR/PIP_EXTRA_INDEX_URL" ]]; then - PIP_EXTRA_INDEX_URL="$(cat "$ENV_DIR/PIP_EXTRA_INDEX_URL")" - export PIP_EXTRA_INDEX_URL - fi - - set +e - - /app/.heroku/python/bin/pip install -r requirements.txt --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir --progress-bar off 2>&1 | tee "$WARNINGS_LOG" | cleanup | indent - PIP_STATUS="${PIPESTATUS[0]}" - set -e - - show-warnings - - if [[ ! $PIP_STATUS -eq 0 ]]; then - meta_set "failure_reason" "pip-install" - exit 1 - fi - - cp requirements.txt .heroku/python/requirements-declared.txt - /app/.heroku/python/bin/pip freeze --disable-pip-version-check > .heroku/python/requirements-installed.txt - - # Install test dependencies, for CI. - if [[ -n "$INSTALL_TEST" ]]; then - if [[ -f requirements-test.txt ]]; then - puts-step "Installing test dependencies..." - /app/.heroku/python/bin/pip install -r requirements-test.txt --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir 2>&1 | cleanup | indent - fi - fi - - meta_time "dependencies_install_duration" "${pip_install_start_time}" + pip_install_start_time=$(nowms) + meta_set "package_manager" "pip" + + puts-step "Installing requirements with pip" + + # Set Pip env vars + # This reads certain environment variables set on the Heroku app config + # and makes them accessible to the pip install process. + # + # PIP_EXTRA_INDEX_URL allows for an alternate pypi URL to be used. + if [[ -r "$ENV_DIR/PIP_EXTRA_INDEX_URL" ]]; then + PIP_EXTRA_INDEX_URL="$(cat "$ENV_DIR/PIP_EXTRA_INDEX_URL")" + export PIP_EXTRA_INDEX_URL + fi + + set +e + + /app/.heroku/python/bin/pip install -r requirements.txt --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir --progress-bar off 2>&1 | tee "$WARNINGS_LOG" | cleanup | indent + PIP_STATUS="${PIPESTATUS[0]}" + set -e + + show-warnings + + if [[ ! $PIP_STATUS -eq 0 ]]; then + meta_set "failure_reason" "pip-install" + exit 1 + fi + + cp requirements.txt .heroku/python/requirements-declared.txt + /app/.heroku/python/bin/pip freeze --disable-pip-version-check >.heroku/python/requirements-installed.txt + + # Install test dependencies, for CI. + if [[ -n "$INSTALL_TEST" ]]; then + if [[ -f requirements-test.txt ]]; then + puts-step "Installing test dependencies..." + /app/.heroku/python/bin/pip install -r requirements-test.txt --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir 2>&1 | cleanup | indent + fi + fi + + meta_time "dependencies_install_duration" "${pip_install_start_time}" fi diff --git a/bin/steps/pipenv b/bin/steps/pipenv index 1d97941c7..6974ae2d5 100755 --- a/bin/steps/pipenv +++ b/bin/steps/pipenv @@ -11,46 +11,46 @@ rm -f .heroku/python/Pipfile.lock.sha256 if [[ -f Pipfile ]]; then - pipenv_install_start_time=$(nowms) - meta_set "package_manager" "pipenv" - - # Skip installing dependencies using pip later. - # TODO: Stop leaking this env var into subshells such as post_compile hooks. - export SKIP_PIP_INSTALL=1 - - # Set Pip env vars - # This reads certain environment variables set on the Heroku app config - # and makes them accessible to the pip install process. - # - # PIP_EXTRA_INDEX_URL allows for an alternate pypi URL to be used. - if [[ -r "$ENV_DIR/PIP_EXTRA_INDEX_URL" ]]; then - PIP_EXTRA_INDEX_URL="$(cat "$ENV_DIR/PIP_EXTRA_INDEX_URL")" - export PIP_EXTRA_INDEX_URL - fi - - PIPENV_VERSION=$(get_requirement_version 'pipenv') - meta_set "pipenv_version" "${PIPENV_VERSION}" - - /app/.heroku/python/bin/pip install --quiet --disable-pip-version-check --no-cache-dir "pipenv==${PIPENV_VERSION}" - - # Install the test dependencies, for CI. - # TODO: This is currently inconsistent with the non-test path, since it assumes (but doesn't check for) a lockfile. - if [[ -n "$INSTALL_TEST" ]]; then - meta_set "pipenv_has_lockfile" "true" - puts-step "Installing test dependencies" - /app/.heroku/python/bin/pipenv install --dev --system --deploy --extra-pip-args='--src=/app/.heroku/src' 2>&1 | cleanup | indent - - # Install the dependencies. - elif [[ ! -f Pipfile.lock ]]; then - meta_set "pipenv_has_lockfile" "false" - puts-step "Installing dependencies with Pipenv ${PIPENV_VERSION}" - /app/.heroku/python/bin/pipenv install --system --skip-lock --extra-pip-args='--src=/app/.heroku/src' 2>&1 | indent - - else - meta_set "pipenv_has_lockfile" "true" - puts-step "Installing dependencies with Pipenv ${PIPENV_VERSION}" - /app/.heroku/python/bin/pipenv install --system --deploy --extra-pip-args='--src=/app/.heroku/src' 2>&1 | indent - fi - - meta_time "dependencies_install_duration" "${pipenv_install_start_time}" + pipenv_install_start_time=$(nowms) + meta_set "package_manager" "pipenv" + + # Skip installing dependencies using pip later. + # TODO: Stop leaking this env var into subshells such as post_compile hooks. + export SKIP_PIP_INSTALL=1 + + # Set Pip env vars + # This reads certain environment variables set on the Heroku app config + # and makes them accessible to the pip install process. + # + # PIP_EXTRA_INDEX_URL allows for an alternate pypi URL to be used. + if [[ -r "$ENV_DIR/PIP_EXTRA_INDEX_URL" ]]; then + PIP_EXTRA_INDEX_URL="$(cat "$ENV_DIR/PIP_EXTRA_INDEX_URL")" + export PIP_EXTRA_INDEX_URL + fi + + PIPENV_VERSION=$(get_requirement_version 'pipenv') + meta_set "pipenv_version" "${PIPENV_VERSION}" + + /app/.heroku/python/bin/pip install --quiet --disable-pip-version-check --no-cache-dir "pipenv==${PIPENV_VERSION}" + + # Install the test dependencies, for CI. + # TODO: This is currently inconsistent with the non-test path, since it assumes (but doesn't check for) a lockfile. + if [[ -n "$INSTALL_TEST" ]]; then + meta_set "pipenv_has_lockfile" "true" + puts-step "Installing test dependencies" + /app/.heroku/python/bin/pipenv install --dev --system --deploy --extra-pip-args='--src=/app/.heroku/src' 2>&1 | cleanup | indent + + # Install the dependencies. + elif [[ ! -f Pipfile.lock ]]; then + meta_set "pipenv_has_lockfile" "false" + puts-step "Installing dependencies with Pipenv ${PIPENV_VERSION}" + /app/.heroku/python/bin/pipenv install --system --skip-lock --extra-pip-args='--src=/app/.heroku/src' 2>&1 | indent + + else + meta_set "pipenv_has_lockfile" "true" + puts-step "Installing dependencies with Pipenv ${PIPENV_VERSION}" + /app/.heroku/python/bin/pipenv install --system --deploy --extra-pip-args='--src=/app/.heroku/src' 2>&1 | indent + fi + + meta_time "dependencies_install_duration" "${pipenv_install_start_time}" fi diff --git a/bin/steps/pipenv-python-version b/bin/steps/pipenv-python-version index 9adba9002..b6614cf99 100755 --- a/bin/steps/pipenv-python-version +++ b/bin/steps/pipenv-python-version @@ -4,60 +4,59 @@ if [[ -f $BUILD_DIR/Pipfile ]]; then - if [[ ! -f $BUILD_DIR/runtime.txt ]]; then - if [[ ! -f $BUILD_DIR/Pipfile.lock ]]; then - puts-warn "No 'Pipfile.lock' found! We recommend you commit this into your repository." - fi - if [[ -f $BUILD_DIR/Pipfile.lock ]]; then - # Ignore unused env var warning since this is used by bin/compile. - # shellcheck disable=2034 - PYTHON_VERSION_SOURCE='Pipfile.lock' - set +e - PYTHON=$(jq -r '._meta.requires.python_full_version' "$BUILD_DIR/Pipfile.lock") - if [[ "$PYTHON" != "null" ]]; then - echo "python-$PYTHON" > "$BUILD_DIR/runtime.txt" - fi - set -e + if [[ ! -f $BUILD_DIR/runtime.txt ]]; then + if [[ ! -f $BUILD_DIR/Pipfile.lock ]]; then + puts-warn "No 'Pipfile.lock' found! We recommend you commit this into your repository." + fi + if [[ -f $BUILD_DIR/Pipfile.lock ]]; then + # Ignore unused env var warning since this is used by bin/compile. + # shellcheck disable=2034 + PYTHON_VERSION_SOURCE='Pipfile.lock' + set +e + PYTHON=$(jq -r '._meta.requires.python_full_version' "$BUILD_DIR/Pipfile.lock") + if [[ "$PYTHON" != "null" ]]; then + echo "python-$PYTHON" >"$BUILD_DIR/runtime.txt" + fi + set -e - if [[ "$PYTHON" == "null" ]]; then - PYTHON=$(jq -r '._meta.requires.python_version' "$BUILD_DIR/Pipfile.lock") - case "${PYTHON}" in - 2.7) - echo "${LATEST_27}" > "${BUILD_DIR}/runtime.txt" - ;; - 3.4) - echo "${LATEST_34}" > "${BUILD_DIR}/runtime.txt" - ;; - 3.5) - echo "${LATEST_35}" > "${BUILD_DIR}/runtime.txt" - ;; - 3.6) - echo "${LATEST_36}" > "${BUILD_DIR}/runtime.txt" - ;; - 3.7) - echo "${LATEST_37}" > "${BUILD_DIR}/runtime.txt" - ;; - 3.8) - echo "${LATEST_38}" > "${BUILD_DIR}/runtime.txt" - ;; - 3.9) - echo "${LATEST_39}" > "${BUILD_DIR}/runtime.txt" - ;; - 3.10) - echo "${LATEST_310}" > "${BUILD_DIR}/runtime.txt" - ;; - 3.11) - echo "${LATEST_311}" > "${BUILD_DIR}/runtime.txt" - ;; - 3.12) - echo "${LATEST_312}" > "${BUILD_DIR}/runtime.txt" - ;; - # TODO: Make this case an error - *) ;; - esac - fi + if [[ "$PYTHON" == "null" ]]; then + PYTHON=$(jq -r '._meta.requires.python_version' "$BUILD_DIR/Pipfile.lock") + case "${PYTHON}" in + 2.7) + echo "${LATEST_27}" >"${BUILD_DIR}/runtime.txt" + ;; + 3.4) + echo "${LATEST_34}" >"${BUILD_DIR}/runtime.txt" + ;; + 3.5) + echo "${LATEST_35}" >"${BUILD_DIR}/runtime.txt" + ;; + 3.6) + echo "${LATEST_36}" >"${BUILD_DIR}/runtime.txt" + ;; + 3.7) + echo "${LATEST_37}" >"${BUILD_DIR}/runtime.txt" + ;; + 3.8) + echo "${LATEST_38}" >"${BUILD_DIR}/runtime.txt" + ;; + 3.9) + echo "${LATEST_39}" >"${BUILD_DIR}/runtime.txt" + ;; + 3.10) + echo "${LATEST_310}" >"${BUILD_DIR}/runtime.txt" + ;; + 3.11) + echo "${LATEST_311}" >"${BUILD_DIR}/runtime.txt" + ;; + 3.12) + echo "${LATEST_312}" >"${BUILD_DIR}/runtime.txt" + ;; + # TODO: Make this case an error + *) ;; + esac + fi - - fi - fi + fi + fi fi diff --git a/bin/steps/python b/bin/steps/python index 333a9302a..a481f9b06 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -7,51 +7,51 @@ PYTHON_VERSION="${PYTHON_VERSION##+([[:space:]])}" PYTHON_VERSION="${PYTHON_VERSION%%+([[:space:]])}" function eol_python_version_error() { - local major_version="${1}" - local eol_date="${2}" - puts-warn - puts-warn "Python ${major_version} reached upstream end-of-life on ${eol_date}, and is" - puts-warn "therefore no longer receiving security updates:" - puts-warn "https://devguide.python.org/versions/#supported-versions" - puts-warn - puts-warn "As such, it is no longer supported by this buildpack." - puts-warn - puts-warn "Please upgrade to a newer Python version." - puts-warn - puts-warn "For a list of the supported Python versions, see:" - puts-warn "https://devcenter.heroku.com/articles/python-support#supported-runtimes" - puts-warn - meta_set "failure_reason" "python-version-eol" - exit 1 + local major_version="${1}" + local eol_date="${2}" + puts-warn + puts-warn "Python ${major_version} reached upstream end-of-life on ${eol_date}, and is" + puts-warn "therefore no longer receiving security updates:" + puts-warn "https://devguide.python.org/versions/#supported-versions" + puts-warn + puts-warn "As such, it is no longer supported by this buildpack." + puts-warn + puts-warn "Please upgrade to a newer Python version." + puts-warn + puts-warn "For a list of the supported Python versions, see:" + puts-warn "https://devcenter.heroku.com/articles/python-support#supported-runtimes" + puts-warn + meta_set "failure_reason" "python-version-eol" + exit 1 } # We check for EOL prior to checking if the archive exists on S3, to ensure the more specific EOL error # message is still shown for newer stacks where the EOL Python versions might not have been built. case "${PYTHON_VERSION}" in - python-3.7.+([0-9])) - eol_python_version_error "3.7" "June 27th, 2023" - ;; - python-3.6.+([0-9])) - eol_python_version_error "3.6" "December 23rd, 2021" - ;; - *) ;; + python-3.7.+([0-9])) + eol_python_version_error "3.7" "June 27th, 2023" + ;; + python-3.6.+([0-9])) + eol_python_version_error "3.6" "December 23rd, 2021" + ;; + *) ;; esac # The Python runtime archive filename is of form: 'python-X.Y.Z-ubuntu-22.04-amd64.tar.zst' # The Ubuntu version is calculated from `STACK` since it's faster than calling `lsb_release`. -UBUNTU_VERSION="${STACK/heroku-}.04" +UBUNTU_VERSION="${STACK/heroku-/}.04" ARCH=$(dpkg --print-architecture) PYTHON_URL="${S3_BASE_URL}/${PYTHON_VERSION}-ubuntu-${UBUNTU_VERSION}-${ARCH}.tar.zst" if ! curl --output /dev/null --silent --head --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}"; then - puts-warn - puts-warn "Requested runtime '${PYTHON_VERSION}' is not available for this stack (${STACK})." - puts-warn - puts-warn "For a list of the supported Python versions, see:" - puts-warn "https://devcenter.heroku.com/articles/python-support#supported-runtimes" - puts-warn - meta_set "failure_reason" "python-version-not-found" - exit 1 + puts-warn + puts-warn "Requested runtime '${PYTHON_VERSION}' is not available for this stack (${STACK})." + puts-warn + puts-warn "For a list of the supported Python versions, see:" + puts-warn "https://devcenter.heroku.com/articles/python-support#supported-runtimes" + puts-warn + meta_set "failure_reason" "python-version-not-found" + exit 1 fi # TODO: Refactor Python version usage to use the non-prefixed form everywhere. @@ -60,109 +60,109 @@ meta_set "python_version" "${python_version_without_prefix}" meta_set "python_version_major" "${python_version_without_prefix%.*}" function warn_if_patch_update_available() { - local requested_version="${1}" - local latest_patch_version="${2}" - # Extract the patch version component of the version strings (ie: the '5' in '3.10.5'). - local requested_patch_number="${requested_version##*.}" - local latest_patch_number="${latest_patch_version##*.}" - if (( requested_patch_number < latest_patch_number )); then - puts-warn - puts-warn "A Python security update is available! Upgrade as soon as possible to: ${latest_patch_version}" - puts-warn "See: https://devcenter.heroku.com/articles/python-runtimes" - puts-warn - fi + local requested_version="${1}" + local latest_patch_version="${2}" + # Extract the patch version component of the version strings (ie: the '5' in '3.10.5'). + local requested_patch_number="${requested_version##*.}" + local latest_patch_number="${latest_patch_version##*.}" + if ((requested_patch_number < latest_patch_number)); then + puts-warn + puts-warn "A Python security update is available! Upgrade as soon as possible to: ${latest_patch_version}" + puts-warn "See: https://devcenter.heroku.com/articles/python-runtimes" + puts-warn + fi } # We wait until now to display outdated Python version warnings, since we only want to show them # if there weren't any errors with the version to avoid adding noise to the error messages. case "${PYTHON_VERSION}" in - python-3.12.*) - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_312}" - ;; - python-3.11.*) - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_311}" - ;; - python-3.10.*) - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_310}" - ;; - python-3.9.*) - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_39}" - ;; - python-3.8.*) - puts-warn - puts-warn "Python 3.8 will reach its upstream end-of-life in October 2024, at which" - puts-warn "point it will no longer receive security updates:" - puts-warn "https://devguide.python.org/versions/#supported-versions" - puts-warn - puts-warn "Support for Python 3.8 will be removed from this buildpack on December 4th, 2024." - puts-warn - puts-warn "Upgrade to a newer Python version as soon as possible to keep your app secure." - puts-warn "See: https://devcenter.heroku.com/articles/python-runtimes" - puts-warn - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_38}" - ;; - # TODO: Make this case an error, since it should be unreachable. - *) ;; + python-3.12.*) + warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_312}" + ;; + python-3.11.*) + warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_311}" + ;; + python-3.10.*) + warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_310}" + ;; + python-3.9.*) + warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_39}" + ;; + python-3.8.*) + puts-warn + puts-warn "Python 3.8 will reach its upstream end-of-life in October 2024, at which" + puts-warn "point it will no longer receive security updates:" + puts-warn "https://devguide.python.org/versions/#supported-versions" + puts-warn + puts-warn "Support for Python 3.8 will be removed from this buildpack on December 4th, 2024." + puts-warn + puts-warn "Upgrade to a newer Python version as soon as possible to keep your app secure." + puts-warn "See: https://devcenter.heroku.com/articles/python-runtimes" + puts-warn + warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_38}" + ;; + # TODO: Make this case an error, since it should be unreachable. + *) ;; esac if [[ "$STACK" != "$CACHED_PYTHON_STACK" ]]; then - puts-step "Stack has changed from $CACHED_PYTHON_STACK to $STACK, clearing cache" - rm -rf .heroku/python-stack .heroku/python-version .heroku/python .heroku/vendor .heroku/python .heroku/python-sqlite3-version + puts-step "Stack has changed from $CACHED_PYTHON_STACK to $STACK, clearing cache" + rm -rf .heroku/python-stack .heroku/python-version .heroku/python .heroku/vendor .heroku/python .heroku/python-sqlite3-version fi if [[ -f .heroku/python-version ]]; then - if [[ ! "$(cat .heroku/python-version)" == "$PYTHON_VERSION" ]]; then - puts-step "Python version has changed from $(cat .heroku/python-version) to ${PYTHON_VERSION}, clearing cache" - rm -rf .heroku/python - else - SKIP_INSTALL=1 - fi + if [[ ! "$(cat .heroku/python-version)" == "$PYTHON_VERSION" ]]; then + puts-step "Python version has changed from $(cat .heroku/python-version) to ${PYTHON_VERSION}, clearing cache" + rm -rf .heroku/python + else + SKIP_INSTALL=1 + fi fi # If using Pip, check if we should reinstall python dependencies given that requirements.txt # is non-deterministic (not all packages pinned, doesn't handle uninstalls etc). We don't need # to do this when using Pipenv, since it has a lockfile and syncs the packages for us. if [[ -f "${BUILD_DIR}/requirements.txt" ]]; then - if [[ ! -f "$CACHE_DIR/.heroku/requirements.txt" ]]; then - # This is a the first build of an app (or the build cache was cleared). Since there - # are no cached packages, we only need to store the requirements file for next time. - cp -R "$BUILD_DIR/requirements.txt" "$CACHE_DIR/.heroku/requirements.txt" - else - # IF there IS a cached directory, check for differences with the new one - if ! diff "$BUILD_DIR/requirements.txt" "$CACHE_DIR/.heroku/requirements.txt" &> /dev/null; then - puts-step "Requirements file has been changed, clearing cached dependencies" - # if there are any differences, clear the Python cache - # Installing Python over again does not take noticably more time - cp -R "$BUILD_DIR/requirements.txt" "$CACHE_DIR/.heroku/requirements.txt" - rm -rf .heroku/python - unset SKIP_INSTALL - else - puts-step "No change in requirements detected, installing from cache" - fi - fi + if [[ ! -f "$CACHE_DIR/.heroku/requirements.txt" ]]; then + # This is a the first build of an app (or the build cache was cleared). Since there + # are no cached packages, we only need to store the requirements file for next time. + cp -R "$BUILD_DIR/requirements.txt" "$CACHE_DIR/.heroku/requirements.txt" + else + # IF there IS a cached directory, check for differences with the new one + if ! diff "$BUILD_DIR/requirements.txt" "$CACHE_DIR/.heroku/requirements.txt" &>/dev/null; then + puts-step "Requirements file has been changed, clearing cached dependencies" + # if there are any differences, clear the Python cache + # Installing Python over again does not take noticably more time + cp -R "$BUILD_DIR/requirements.txt" "$CACHE_DIR/.heroku/requirements.txt" + rm -rf .heroku/python + unset SKIP_INSTALL + else + puts-step "No change in requirements detected, installing from cache" + fi + fi fi if [[ -n "${SKIP_INSTALL}" ]]; then - puts-step "Using cached install of ${PYTHON_VERSION}" + puts-step "Using cached install of ${PYTHON_VERSION}" else - puts-step "Installing ${PYTHON_VERSION}" + puts-step "Installing ${PYTHON_VERSION}" - # Prepare destination directory. - mkdir -p .heroku/python + # Prepare destination directory. + mkdir -p .heroku/python - if ! curl --silent --show-error --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}" | tar --zstd --extract --directory .heroku/python; then - # The Python version was confirmed to exist previously, so any failure here is due to - # a networking issue or archive/buildpack bug rather than the runtime not existing. - puts-warn "Failed to download/install ${PYTHON_VERSION}" - meta_set "failure_reason" "python-download" - exit 1 - fi + if ! curl --silent --show-error --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}" | tar --zstd --extract --directory .heroku/python; then + # The Python version was confirmed to exist previously, so any failure here is due to + # a networking issue or archive/buildpack bug rather than the runtime not existing. + puts-warn "Failed to download/install ${PYTHON_VERSION}" + meta_set "failure_reason" "python-download" + exit 1 + fi - # Record for future reference. - echo "$PYTHON_VERSION" > .heroku/python-version - echo "$STACK" > .heroku/python-stack + # Record for future reference. + echo "$PYTHON_VERSION" >.heroku/python-version + echo "$STACK" >.heroku/python-stack - hash -r + hash -r fi PIP_VERSION=$(get_requirement_version 'pip') @@ -186,12 +186,12 @@ BUNDLED_PIP_WHEEL_LIST=(.heroku/python/lib/python*/ensurepip/_bundled/pip-*.whl) BUNDLED_PIP_WHEEL="${BUNDLED_PIP_WHEEL_LIST[0]}" if [[ -z "${BUNDLED_PIP_WHEEL}" ]]; then - puts-warn "Failed to locate the bundled pip wheel" - meta_set "failure_reason" "bundled-pip-not-found" - exit 1 + puts-warn "Failed to locate the bundled pip wheel" + meta_set "failure_reason" "bundled-pip-not-found" + exit 1 fi /app/.heroku/python/bin/python "${BUNDLED_PIP_WHEEL}/pip" install --quiet --disable-pip-version-check --no-cache-dir \ - "pip==${PIP_VERSION}" "setuptools==${SETUPTOOLS_VERSION}" "wheel==${WHEEL_VERSION}" + "pip==${PIP_VERSION}" "setuptools==${SETUPTOOLS_VERSION}" "wheel==${WHEEL_VERSION}" hash -r diff --git a/bin/steps/sqlite3 b/bin/steps/sqlite3 index f21d606f4..c7200cebc 100755 --- a/bin/steps/sqlite3 +++ b/bin/steps/sqlite3 @@ -4,77 +4,77 @@ # and the APT buildpack should be used if an app needs the sqlite CLI/headers. sqlite3_install() { - HEROKU_PYTHON_DIR="$1" - HEADERS_ONLY="$3" - - mkdir -p "$HEROKU_PYTHON_DIR" - - APT_CACHE_DIR="$HEROKU_PYTHON_DIR/apt/cache" - APT_STATE_DIR="$HEROKU_PYTHON_DIR/apt/state" - - mkdir -p "$APT_CACHE_DIR/archives/partial" - mkdir -p "$APT_STATE_DIR/lists/partial" - - APT_OPTIONS=( - "--option=debug::nolocking=true" - "--option=dir::cache=${APT_CACHE_DIR}" - "--option=dir::state=${APT_STATE_DIR}" - "--option=dir::etc::sourcelist=/etc/apt/sources.list" - ) - - apt-get "${APT_OPTIONS[@]}" update > /dev/null 2>&1 - if [[ -z "$HEADERS_ONLY" ]]; then - apt-get "${APT_OPTIONS[@]}" -y -d --reinstall install libsqlite3-dev sqlite3 > /dev/null 2>&1 - else - apt-get "${APT_OPTIONS[@]}" -y -d --reinstall install libsqlite3-dev - fi - - find "$APT_CACHE_DIR/archives/" -name "*.deb" -exec dpkg -x {} "$HEROKU_PYTHON_DIR/sqlite3/" \; - - mkdir -p "$HEROKU_PYTHON_DIR/include" - mkdir -p "$HEROKU_PYTHON_DIR/lib" - - # remove old sqlite3 libraries/binaries - find "$HEROKU_PYTHON_DIR/include/" -name "sqlite3*.h" -exec rm -f {} \; - find "$HEROKU_PYTHON_DIR/lib/" -name "libsqlite3.*" -exec rm -f {} \; - rm -f "$HEROKU_PYTHON_DIR/lib/pkgconfig/sqlite3.pc" - rm -f "$HEROKU_PYTHON_DIR/bin/sqlite3" - - # eg: `x86_64` or `aarch64` - GNU_ARCH=$(arch) - - # copy over sqlite3 headers & bins and setup linking against the stack image library - mv "$HEROKU_PYTHON_DIR/sqlite3/usr/include/"* "$HEROKU_PYTHON_DIR/include/" - mv "$HEROKU_PYTHON_DIR/sqlite3/usr/lib/${GNU_ARCH}-linux-gnu"/libsqlite3.*a "$HEROKU_PYTHON_DIR/lib/" - mkdir -p "$HEROKU_PYTHON_DIR/lib/pkgconfig" - # set the right prefix/lib directories - sed -e 's/prefix=\/usr/prefix=\/app\/.heroku\/python/' -e "s/\/${GNU_ARCH}-linux-gnu//" "$HEROKU_PYTHON_DIR/sqlite3/usr/lib/${GNU_ARCH}-linux-gnu/pkgconfig/sqlite3.pc" > "$HEROKU_PYTHON_DIR/lib/pkgconfig/sqlite3.pc" - # need to point the libsqlite3.so to the stack image library for /usr/bin/ld -lsqlite3 - SQLITE3_LIBFILE="/usr/lib/${GNU_ARCH}-linux-gnu/$(readlink -n "$HEROKU_PYTHON_DIR/sqlite3/usr/lib/${GNU_ARCH}-linux-gnu/libsqlite3.so")" - ln -s "$SQLITE3_LIBFILE" "$HEROKU_PYTHON_DIR/lib/libsqlite3.so" - if [[ -z "$HEADERS_ONLY" ]]; then - mv "$HEROKU_PYTHON_DIR/sqlite3/usr/bin"/* "$HEROKU_PYTHON_DIR/bin/" - fi - - # cleanup - rm -rf "$HEROKU_PYTHON_DIR/sqlite3/" - rm -rf "$HEROKU_PYTHON_DIR/apt/" + HEROKU_PYTHON_DIR="$1" + HEADERS_ONLY="$3" + + mkdir -p "$HEROKU_PYTHON_DIR" + + APT_CACHE_DIR="$HEROKU_PYTHON_DIR/apt/cache" + APT_STATE_DIR="$HEROKU_PYTHON_DIR/apt/state" + + mkdir -p "$APT_CACHE_DIR/archives/partial" + mkdir -p "$APT_STATE_DIR/lists/partial" + + APT_OPTIONS=( + "--option=debug::nolocking=true" + "--option=dir::cache=${APT_CACHE_DIR}" + "--option=dir::state=${APT_STATE_DIR}" + "--option=dir::etc::sourcelist=/etc/apt/sources.list" + ) + + apt-get "${APT_OPTIONS[@]}" update >/dev/null 2>&1 + if [[ -z "$HEADERS_ONLY" ]]; then + apt-get "${APT_OPTIONS[@]}" -y -d --reinstall install libsqlite3-dev sqlite3 >/dev/null 2>&1 + else + apt-get "${APT_OPTIONS[@]}" -y -d --reinstall install libsqlite3-dev + fi + + find "$APT_CACHE_DIR/archives/" -name "*.deb" -exec dpkg -x {} "$HEROKU_PYTHON_DIR/sqlite3/" \; + + mkdir -p "$HEROKU_PYTHON_DIR/include" + mkdir -p "$HEROKU_PYTHON_DIR/lib" + + # remove old sqlite3 libraries/binaries + find "$HEROKU_PYTHON_DIR/include/" -name "sqlite3*.h" -exec rm -f {} \; + find "$HEROKU_PYTHON_DIR/lib/" -name "libsqlite3.*" -exec rm -f {} \; + rm -f "$HEROKU_PYTHON_DIR/lib/pkgconfig/sqlite3.pc" + rm -f "$HEROKU_PYTHON_DIR/bin/sqlite3" + + # eg: `x86_64` or `aarch64` + GNU_ARCH=$(arch) + + # copy over sqlite3 headers & bins and setup linking against the stack image library + mv "$HEROKU_PYTHON_DIR/sqlite3/usr/include/"* "$HEROKU_PYTHON_DIR/include/" + mv "$HEROKU_PYTHON_DIR/sqlite3/usr/lib/${GNU_ARCH}-linux-gnu"/libsqlite3.*a "$HEROKU_PYTHON_DIR/lib/" + mkdir -p "$HEROKU_PYTHON_DIR/lib/pkgconfig" + # set the right prefix/lib directories + sed -e 's/prefix=\/usr/prefix=\/app\/.heroku\/python/' -e "s/\/${GNU_ARCH}-linux-gnu//" "$HEROKU_PYTHON_DIR/sqlite3/usr/lib/${GNU_ARCH}-linux-gnu/pkgconfig/sqlite3.pc" >"$HEROKU_PYTHON_DIR/lib/pkgconfig/sqlite3.pc" + # need to point the libsqlite3.so to the stack image library for /usr/bin/ld -lsqlite3 + SQLITE3_LIBFILE="/usr/lib/${GNU_ARCH}-linux-gnu/$(readlink -n "$HEROKU_PYTHON_DIR/sqlite3/usr/lib/${GNU_ARCH}-linux-gnu/libsqlite3.so")" + ln -s "$SQLITE3_LIBFILE" "$HEROKU_PYTHON_DIR/lib/libsqlite3.so" + if [[ -z "$HEADERS_ONLY" ]]; then + mv "$HEROKU_PYTHON_DIR/sqlite3/usr/bin"/* "$HEROKU_PYTHON_DIR/bin/" + fi + + # cleanup + rm -rf "$HEROKU_PYTHON_DIR/sqlite3/" + rm -rf "$HEROKU_PYTHON_DIR/apt/" } buildpack_sqlite3_install() { - puts-step "Installing SQLite3" - - # TODO: This never actual prints failure or even aborts the build, since - # the conditional disables `set -e` inside the called function: - # https://stackoverflow.com/q/19789102 - # ...plus whoever wrote this forgot the `exit 1` in the `else` anyway. - if sqlite3_install "$BUILD_DIR/.heroku/python" ; then - # mcount "success.python.sqlite3" - : - else - echo "Sqlite3 failed to install." - # mcount "failure.python.sqlite3" - fi - - mkdir -p "$CACHE_DIR/.heroku/" + puts-step "Installing SQLite3" + + # TODO: This never actual prints failure or even aborts the build, since + # the conditional disables `set -e` inside the called function: + # https://stackoverflow.com/q/19789102 + # ...plus whoever wrote this forgot the `exit 1` in the `else` anyway. + if sqlite3_install "$BUILD_DIR/.heroku/python"; then + # mcount "success.python.sqlite3" + : + else + echo "Sqlite3 failed to install." + # mcount "failure.python.sqlite3" + fi + + mkdir -p "$CACHE_DIR/.heroku/" } diff --git a/bin/utils b/bin/utils index 5ce22d866..09961257f 100755 --- a/bin/utils +++ b/bin/utils @@ -8,57 +8,56 @@ shopt -s nullglob source "${BUILDPACK_DIR}/vendor/buildpack-stdlib_v8.sh" if [[ "$(uname)" == "Darwin" ]]; then - sed() { command sed -l "$@"; } + sed() { command sed -l "$@"; } else - sed() { command sed -u "$@"; } + sed() { command sed -u "$@"; } fi # Syntax sugar. indent() { - sed "s/^/ /" + sed "s/^/ /" } - # Clean up pip output cleanup() { - sed -e 's/\.\.\.\+/.../g' | sed -e '/already satisfied/Id' | sed -e '/No files were found to uninstall/Id' | sed -e '/Overwriting/Id' | sed -e '/python executable/Id' | sed -e '/no previously-included files/Id' + sed -e 's/\.\.\.\+/.../g' | sed -e '/already satisfied/Id' | sed -e '/No files were found to uninstall/Id' | sed -e '/Overwriting/Id' | sed -e '/python executable/Id' | sed -e '/no previously-included files/Id' } # Buildpack Steps. puts-step() { - echo "-----> $*" + echo "-----> $*" } # Buildpack Warnings. puts-warn() { - echo " ! $*" + echo " ! $*" } # Does some serious copying. deep-cp() { - declare source="$1" target="$2" + declare source="$1" target="$2" - mkdir -p "$target" + mkdir -p "$target" - # cp doesn't like being called without source params, - # so make sure they expand to something first. - # subshell to avoid surprising caller with shopts. - ( - shopt -s nullglob dotglob - set -- "$source"/!(tmp|.|..) - [[ $# == 0 ]] || cp -a "$@" "$target" - ) + # cp doesn't like being called without source params, + # so make sure they expand to something first. + # subshell to avoid surprising caller with shopts. + ( + shopt -s nullglob dotglob + set -- "$source"/!(tmp|.|..) + [[ $# == 0 ]] || cp -a "$@" "$target" + ) } # Measure the size of the Python installation. measure-size() { - { du -s .heroku/python 2>/dev/null || echo 0; } | awk '{print $1}' + { du -s .heroku/python 2>/dev/null || echo 0; } | awk '{print $1}' } # Returns 0 if the specified module exists, otherwise returns 1. is_module_available() { - local module_name="${1}" - python -c "import sys, importlib.util; sys.exit(0 if importlib.util.find_spec('${module_name}') else 1)" + local module_name="${1}" + python -c "import sys, importlib.util; sys.exit(0 if importlib.util.find_spec('${module_name}') else 1)" } # The requirement versions are effectively buildpack constants, however, we want @@ -66,9 +65,9 @@ is_module_available() { # files. The requirements files contain contents like `package==1.2.3` (and not just # the package version) so we have to extract the version substring from it. get_requirement_version() { - local package_name="${1}" - local requirement - requirement=$(cat "${BUILDPACK_DIR}/requirements/${package_name}.txt") - local requirement_version="${requirement#"${package_name}=="}" - echo "${requirement_version}" + local package_name="${1}" + local requirement + requirement=$(cat "${BUILDPACK_DIR}/requirements/${package_name}.txt") + local requirement_version="${requirement#"${package_name}=="}" + echo "${requirement_version}" } diff --git a/bin/warnings b/bin/warnings index 0036f5ccc..e3f7fba9c 100755 --- a/bin/warnings +++ b/bin/warnings @@ -1,14 +1,14 @@ #!/usr/bin/env bash gdal-missing() { - if grep -qi 'Could not find gdal-config' "$WARNINGS_LOG"; then - echo - puts-warn "Hello! Package installation failed since the GDAL library was not found." - puts-warn "For GDAL, GEOS and PROJ support, use the Geo buildpack alongside the Python buildpack:" - puts-warn "https://github.com/heroku/heroku-geo-buildpack" - fi + if grep -qi 'Could not find gdal-config' "$WARNINGS_LOG"; then + echo + puts-warn "Hello! Package installation failed since the GDAL library was not found." + puts-warn "For GDAL, GEOS and PROJ support, use the Geo buildpack alongside the Python buildpack:" + puts-warn "https://github.com/heroku/heroku-geo-buildpack" + fi } show-warnings() { - gdal-missing + gdal-missing } diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index 88817225c..e1c35c644 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -15,61 +15,61 @@ SRC_DIR="/tmp/src" UPLOAD_DIR="/tmp/upload" function abort() { - echo "Error: ${1}" >&2 - exit 1 + echo "Error: ${1}" >&2 + exit 1 } case "${STACK}" in - heroku-24) - SUPPORTED_PYTHON_VERSIONS=( - "3.10" - "3.11" - "3.12" - ) - ;; - heroku-22) - SUPPORTED_PYTHON_VERSIONS=( - "3.9" - "3.10" - "3.11" - "3.12" - ) - ;; - heroku-20) - SUPPORTED_PYTHON_VERSIONS=( - "3.8" - "3.9" - "3.10" - "3.11" - "3.12" - ) - ;; - *) - abort "Unsupported stack '${STACK}'!" - ;; + heroku-24) + SUPPORTED_PYTHON_VERSIONS=( + "3.10" + "3.11" + "3.12" + ) + ;; + heroku-22) + SUPPORTED_PYTHON_VERSIONS=( + "3.9" + "3.10" + "3.11" + "3.12" + ) + ;; + heroku-20) + SUPPORTED_PYTHON_VERSIONS=( + "3.8" + "3.9" + "3.10" + "3.11" + "3.12" + ) + ;; + *) + abort "Unsupported stack '${STACK}'!" + ;; esac if [[ ! " ${SUPPORTED_PYTHON_VERSIONS[*]} " == *" ${PYTHON_MAJOR_VERSION} "* ]]; then - abort "Python ${PYTHON_MAJOR_VERSION} is not supported on ${STACK}!" + abort "Python ${PYTHON_MAJOR_VERSION} is not supported on ${STACK}!" fi # The release keys can be found on https://www.python.org/downloads/ -> "OpenPGP Public Keys". case "${PYTHON_MAJOR_VERSION}" in - 3.12) - # https://github.com/Yhg1s.gpg - GPG_KEY_FINGERPRINT='7169605F62C751356D054A26A821E680E5FA6305' - ;; - 3.10|3.11) - # https://keybase.io/pablogsal/ - GPG_KEY_FINGERPRINT='A035C8C19219BA821ECEA86B64E628F8D684696D' - ;; - 3.8|3.9) - # https://keybase.io/ambv/ - GPG_KEY_FINGERPRINT='E3FF2839C048B25C084DEBE9B26995E310250568' - ;; - *) - abort "Unsupported Python version '${PYTHON_MAJOR_VERSION}'!" - ;; + 3.12) + # https://github.com/Yhg1s.gpg + GPG_KEY_FINGERPRINT='7169605F62C751356D054A26A821E680E5FA6305' + ;; + 3.10 | 3.11) + # https://keybase.io/pablogsal/ + GPG_KEY_FINGERPRINT='A035C8C19219BA821ECEA86B64E628F8D684696D' + ;; + 3.8 | 3.9) + # https://keybase.io/ambv/ + GPG_KEY_FINGERPRINT='E3FF2839C048B25C084DEBE9B26995E310250568' + ;; + *) + abort "Unsupported Python version '${PYTHON_MAJOR_VERSION}'!" + ;; esac echo "Building Python ${PYTHON_VERSION} for ${STACK} (${ARCH})..." @@ -94,59 +94,59 @@ cd "${SRC_DIR}" # for maximum compatibility / most battle-tested build configuration: # https://github.com/docker-library/python CONFIGURE_OPTS=( - # Explicitly set the target architecture rather than auto-detecting based on the host CPU. - # This only affects targets like i386 (for which we don't build), but we pass it anyway for - # completeness and parity with the Python Docker image builds. - "--build=$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" - # Support loadable extensions in the `_sqlite` extension module. - "--enable-loadable-sqlite-extensions" - # Enable recommended release build performance optimisations such as PGO. - "--enable-optimizations" - # Make autoconf's configure option validation more strict. - "--enable-option-checking=fatal" - # Install Python into `/tmp/python` rather than the default of `/usr/local`. - "--prefix=${INSTALL_DIR}" - # Skip running `ensurepip` as part of install, since the buildpack installs a curated - # version of pip itself (which ensures it's consistent across Python patch releases). - "--with-ensurepip=no" - # Build the `pyexpat` module using the `expat` library in the base image (which will - # automatically receive security updates), rather than CPython's vendored version. - "--with-system-expat" + # Explicitly set the target architecture rather than auto-detecting based on the host CPU. + # This only affects targets like i386 (for which we don't build), but we pass it anyway for + # completeness and parity with the Python Docker image builds. + "--build=$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" + # Support loadable extensions in the `_sqlite` extension module. + "--enable-loadable-sqlite-extensions" + # Enable recommended release build performance optimisations such as PGO. + "--enable-optimizations" + # Make autoconf's configure option validation more strict. + "--enable-option-checking=fatal" + # Install Python into `/tmp/python` rather than the default of `/usr/local`. + "--prefix=${INSTALL_DIR}" + # Skip running `ensurepip` as part of install, since the buildpack installs a curated + # version of pip itself (which ensures it's consistent across Python patch releases). + "--with-ensurepip=no" + # Build the `pyexpat` module using the `expat` library in the base image (which will + # automatically receive security updates), rather than CPython's vendored version. + "--with-system-expat" ) if [[ "${PYTHON_MAJOR_VERSION}" != 3.[8-9] ]]; then - CONFIGURE_OPTS+=( - # Shared builds are beneficial for a number of reasons: - # - Reduces the size of the build, since it avoids the duplication between - # the Python binary and the static library. - # - Permits use-cases that only work with the shared Python library, - # and not the static library (such as `pycall.rb` or `PyO3`). - # - More consistent with the official Python Docker images and other distributions. - # - # However, shared builds are slower unless `no-semantic-interposition`and LTO is used: - # https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup - # https://github.com/python/cpython/issues/83161 - # - # It's only as of Python 3.10 that `no-semantic-interposition` is enabled by default, - # so we only use shared builds on Python 3.10+ to avoid needing to override the default - # compiler flags. - "--enable-shared" - "--with-lto" - # Counter-intuitively, the static library is still generated by default even when - # the shared library is enabled, so we disable it to reduce the build size. - # This option only exists for Python 3.10+. - "--without-static-libpython" - ) + CONFIGURE_OPTS+=( + # Shared builds are beneficial for a number of reasons: + # - Reduces the size of the build, since it avoids the duplication between + # the Python binary and the static library. + # - Permits use-cases that only work with the shared Python library, + # and not the static library (such as `pycall.rb` or `PyO3`). + # - More consistent with the official Python Docker images and other distributions. + # + # However, shared builds are slower unless `no-semantic-interposition`and LTO is used: + # https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup + # https://github.com/python/cpython/issues/83161 + # + # It's only as of Python 3.10 that `no-semantic-interposition` is enabled by default, + # so we only use shared builds on Python 3.10+ to avoid needing to override the default + # compiler flags. + "--enable-shared" + "--with-lto" + # Counter-intuitively, the static library is still generated by default even when + # the shared library is enabled, so we disable it to reduce the build size. + # This option only exists for Python 3.10+. + "--without-static-libpython" + ) fi if [[ "${PYTHON_MAJOR_VERSION}" == "3.11" || "${PYTHON_MAJOR_VERSION}" == "3.12" ]]; then - CONFIGURE_OPTS+=( - # Skip building the test modules, since we remove them after the build anyway. - # This feature was added in Python 3.10+, however it wasn't until Python 3.11 - # that compatibility issues between it and PGO were fixed: - # https://github.com/python/cpython/pull/29315 - "--disable-test-modules" - ) + CONFIGURE_OPTS+=( + # Skip building the test modules, since we remove them after the build anyway. + # This feature was added in Python 3.10+, however it wasn't until Python 3.11 + # that compatibility issues between it and PGO were fixed: + # https://github.com/python/cpython/pull/29315 + "--disable-test-modules" + ) fi ./configure "${CONFIGURE_OPTS[@]}" @@ -163,28 +163,28 @@ fi # We only use `dpkg-buildflags` for Python versions where we build in shared mode (Python 3.9+), # since some of the options it enables interferes with the stripping of static libraries. if [[ "${PYTHON_MAJOR_VERSION}" == 3.[8-9] ]]; then - EXTRA_CFLAGS='' - LDFLAGS='-Wl,--strip-all' + EXTRA_CFLAGS='' + LDFLAGS='-Wl,--strip-all' else - EXTRA_CFLAGS="$(dpkg-buildflags --get CFLAGS)" - LDFLAGS="$(dpkg-buildflags --get LDFLAGS) -Wl,--strip-all" + EXTRA_CFLAGS="$(dpkg-buildflags --get CFLAGS)" + LDFLAGS="$(dpkg-buildflags --get LDFLAGS) -Wl,--strip-all" fi make -j "$(nproc)" "EXTRA_CFLAGS=${EXTRA_CFLAGS}" "LDFLAGS=${LDFLAGS}" make install if [[ "${PYTHON_MAJOR_VERSION}" == 3.[8-9] ]]; then - # On older versions of Python we're still building the static library, which has to be - # manually stripped since the linker stripping enabled in LDFLAGS doesn't cover them. - # We're using `--strip-unneeded` since `--strip-all` would remove the `.symtab` section - # that is required for static libraries to be able to be linked. - # `find` is used since there are multiple copies of the static library in version-specific - # locations, eg: - # - `lib/libpython3.9.a` - # - `lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.a` - find "${INSTALL_DIR}" -type f -name '*.a' -print -exec strip --strip-unneeded '{}' + + # On older versions of Python we're still building the static library, which has to be + # manually stripped since the linker stripping enabled in LDFLAGS doesn't cover them. + # We're using `--strip-unneeded` since `--strip-all` would remove the `.symtab` section + # that is required for static libraries to be able to be linked. + # `find` is used since there are multiple copies of the static library in version-specific + # locations, eg: + # - `lib/libpython3.9.a` + # - `lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.a` + find "${INSTALL_DIR}" -type f -name '*.a' -print -exec strip --strip-unneeded '{}' + elif ! find "${INSTALL_DIR}" -type f -name '*.a' -print -exec false '{}' +; then - abort "Unexpected static libraries found!" + abort "Unexpected static libraries found!" fi # Remove unneeded test directories, similar to the official Docker Python images: diff --git a/builds/test_python_runtime.sh b/builds/test_python_runtime.sh index 258600a31..8dfce8d91 100755 --- a/builds/test_python_runtime.sh +++ b/builds/test_python_runtime.sh @@ -22,27 +22,30 @@ tar --zstd --extract --verbose --file "${ARCHIVE_FILEPATH}" --directory "${INSTA # Check that all dynamically linked libraries exist in the run image (since it has fewer packages than the build image). LDD_OUTPUT=$(find "${INSTALL_DIR}" -type f,l \( -name 'python3' -o -name '*.so*' \) -exec ldd '{}' +) if grep 'not found' <<<"${LDD_OUTPUT}" | sort --unique; then - echo "The above dynamically linked libraries were not found!" - exit 1 + echo "The above dynamically linked libraries were not found!" + exit 1 fi # Check that optional and/or system library dependent stdlib modules were built. optional_stdlib_modules=( - _uuid - bz2 - ctypes - curses - dbm.gnu - dbm.ndbm - decimal - lzma - readline - sqlite3 - ssl - xml.parsers.expat - zlib + _uuid + bz2 + ctypes + curses + dbm.gnu + dbm.ndbm + decimal + lzma + readline + sqlite3 + ssl + xml.parsers.expat + zlib ) -if ! "${INSTALL_DIR}/bin/python3" -c "import $(IFS=, ; echo "${optional_stdlib_modules[*]}")"; then - echo "The above optional stdlib module failed to import! Check the compile logs to see if it was skipped due to missing libraries/headers." - exit 1 +if ! "${INSTALL_DIR}/bin/python3" -c "import $( + IFS=, + echo "${optional_stdlib_modules[*]}" +)"; then + echo "The above optional stdlib module failed to import! Check the compile logs to see if it was skipped due to missing libraries/headers." + exit 1 fi diff --git a/etc/publish.sh b/etc/publish.sh index c7801efe2..ea07f296e 100755 --- a/etc/publish.sh +++ b/etc/publish.sh @@ -9,9 +9,9 @@ newVersion="v$((curVersion + 1))" read -r -p "Deploy as version: $newVersion [y/n]? " choice case "$choice" in - y|Y ) echo "";; - n|N ) exit 0;; - * ) exit 1;; + y | Y) echo "" ;; + n | N) exit 0 ;; + *) exit 1 ;; esac git fetch origin diff --git a/lib/kvstore.sh b/lib/kvstore.sh index bd5c6fad0..17e2dd0d0 100644 --- a/lib/kvstore.sh +++ b/lib/kvstore.sh @@ -3,73 +3,73 @@ # Taken from: https://github.com/heroku/heroku-buildpack-nodejs/blob/main/lib/kvstore.sh kv_create() { - local f="${1}" - mkdir -p "$(dirname "${f}")" - touch "${f}" + local f="${1}" + mkdir -p "$(dirname "${f}")" + touch "${f}" } kv_clear() { - local f="${1}" - echo "" >"${f}" + local f="${1}" + echo "" >"${f}" } kv_set() { - if [[ $# -eq 3 ]]; then - local f="${1}" - if [[ -f "${f}" ]]; then - echo "${2}=${3}" >>"${f}" - fi - fi + if [[ $# -eq 3 ]]; then + local f="${1}" + if [[ -f "${f}" ]]; then + echo "${2}=${3}" >>"${f}" + fi + fi } kv_get() { - if [[ $# -eq 2 ]]; then - local f="${1}" - if [[ -f "${f}" ]]; then - grep "^${2}=" "${f}" | sed -e "s/^${2}=//" | tail -n 1 - fi - fi + if [[ $# -eq 2 ]]; then + local f="${1}" + if [[ -f "${f}" ]]; then + grep "^${2}=" "${f}" | sed -e "s/^${2}=//" | tail -n 1 + fi + fi } # get the value, but wrap it in quotes if it contains a space kv_get_escaped() { - local value - value=$(kv_get "${1}" "${2}") - if [[ "${value}" =~ [[:space:]]+ ]]; then - echo "\"${value}\"" - else - echo "${value}" - fi + local value + value=$(kv_get "${1}" "${2}") + if [[ "${value}" =~ [[:space:]]+ ]]; then + echo "\"${value}\"" + else + echo "${value}" + fi } kv_keys() { - local f="${1}" - local keys=() + local f="${1}" + local keys=() - if [[ -f "${f}" ]]; then - # Iterate over each line, splitting on the '=' character - # - # The || [[ -n "${key}" ]] statement addresses an issue with reading the last line - # of a file when there is no newline at the end. This will not happen if the file - # is created with this module, but can happen if it is written by hand. - # See: https://stackoverflow.com/questions/12916352/shell-script-read-missing-last-line - while IFS="=" read -r key value || [[ -n "${key}" ]]; do - # if there are any empty lines in the store, skip them - if [[ -n "${key}" ]]; then - keys+=("${key}") - fi - done <"${f}" + if [[ -f "${f}" ]]; then + # Iterate over each line, splitting on the '=' character + # + # The || [[ -n "${key}" ]] statement addresses an issue with reading the last line + # of a file when there is no newline at the end. This will not happen if the file + # is created with this module, but can happen if it is written by hand. + # See: https://stackoverflow.com/questions/12916352/shell-script-read-missing-last-line + while IFS="=" read -r key value || [[ -n "${key}" ]]; do + # if there are any empty lines in the store, skip them + if [[ -n "${key}" ]]; then + keys+=("${key}") + fi + done <"${f}" - echo "${keys[@]}" | tr ' ' '\n' | sort -u - fi + echo "${keys[@]}" | tr ' ' '\n' | sort -u + fi } kv_list() { - local f="${1}" + local f="${1}" - kv_keys "${f}" | tr ' ' '\n' | while read -r key; do - if [[ -n "${key}" ]]; then - echo "${key}=$(kv_get_escaped "${f}" "${key}")" - fi - done + kv_keys "${f}" | tr ' ' '\n' | while read -r key; do + if [[ -n "${key}" ]]; then + echo "${key}=$(kv_get_escaped "${f}" "${key}")" + fi + done } diff --git a/lib/metadata.sh b/lib/metadata.sh index 20aab3da3..7d2a39b7a 100644 --- a/lib/metadata.sh +++ b/lib/metadata.sh @@ -10,47 +10,47 @@ PREVIOUS_BUILD_DATA_FILE="" # Must be called before you can use any other methods meta_init() { - local cache_dir="${1}" - local buildpack_name="${2}" - BUILD_DATA_FILE="${cache_dir}/build-data/${buildpack_name}" - PREVIOUS_BUILD_DATA_FILE="${cache_dir}/build-data/${buildpack_name}-prev" + local cache_dir="${1}" + local buildpack_name="${2}" + BUILD_DATA_FILE="${cache_dir}/build-data/${buildpack_name}" + PREVIOUS_BUILD_DATA_FILE="${cache_dir}/build-data/${buildpack_name}-prev" } # Moves the data from the last build into the correct place, and clears the store # This should be called after meta_init in bin/compile meta_setup() { - # if the file already exists because it's from the last build, save it - if [[ -f "${BUILD_DATA_FILE}" ]]; then - cp "${BUILD_DATA_FILE}" "${PREVIOUS_BUILD_DATA_FILE}" - fi + # if the file already exists because it's from the last build, save it + if [[ -f "${BUILD_DATA_FILE}" ]]; then + cp "${BUILD_DATA_FILE}" "${PREVIOUS_BUILD_DATA_FILE}" + fi - kv_create "${BUILD_DATA_FILE}" - kv_clear "${BUILD_DATA_FILE}" + kv_create "${BUILD_DATA_FILE}" + kv_clear "${BUILD_DATA_FILE}" } # Force removal of exiting data file state. This is mostly useful during testing and not # expected to be used during buildpack execution. meta_force_clear() { - [[ -f "${BUILD_DATA_FILE}" ]] && rm "${BUILD_DATA_FILE}" - [[ -f "${PREVIOUS_BUILD_DATA_FILE}" ]] && rm "${PREVIOUS_BUILD_DATA_FILE}" + [[ -f "${BUILD_DATA_FILE}" ]] && rm "${BUILD_DATA_FILE}" + [[ -f "${PREVIOUS_BUILD_DATA_FILE}" ]] && rm "${PREVIOUS_BUILD_DATA_FILE}" } meta_get() { - kv_get "${BUILD_DATA_FILE}" "${1}" + kv_get "${BUILD_DATA_FILE}" "${1}" } meta_set() { - kv_set "${BUILD_DATA_FILE}" "${1}" "${2}" + kv_set "${BUILD_DATA_FILE}" "${1}" "${2}" } # Similar to mtime from buildpack-stdlib meta_time() { - local key="${1}" - local start="${2}" - local end="${3:-$(nowms)}" - local time - time="$(echo "${start}" "${end}" | awk '{ printf "%.3f", ($2 - $1)/1000 }')" - kv_set "${BUILD_DATA_FILE}" "${key}" "${time}" + local key="${1}" + local start="${2}" + local end="${3:-$(nowms)}" + local time + time="$(echo "${start}" "${end}" | awk '{ printf "%.3f", ($2 - $1)/1000 }')" + kv_set "${BUILD_DATA_FILE}" "${key}" "${time}" } # Retrieve a value from a previous build if it exists @@ -60,13 +60,13 @@ meta_time() { # - deployed with a new major version of Node # - etc meta_prev_get() { - kv_get "${PREVIOUS_BUILD_DATA_FILE}" "${1}" + kv_get "${PREVIOUS_BUILD_DATA_FILE}" "${1}" } log_meta_data() { - # print all values on one line in logfmt format - # https://brandur.org/logfmt - # the echo call ensures that all values are printed on a single line - # shellcheck disable=SC2005 disable=SC2046 - echo $(kv_list "${BUILD_DATA_FILE}") + # print all values on one line in logfmt format + # https://brandur.org/logfmt + # the echo call ensures that all values are printed on a single line + # shellcheck disable=SC2005 disable=SC2046 + echo $(kv_list "${BUILD_DATA_FILE}") } diff --git a/spec/fixtures/hooks/bin/print-env-vars.sh b/spec/fixtures/hooks/bin/print-env-vars.sh index dbd598a54..af6afded8 100755 --- a/spec/fixtures/hooks/bin/print-env-vars.sh +++ b/spec/fixtures/hooks/bin/print-env-vars.sh @@ -3,8 +3,8 @@ set -euo pipefail printenv | sort \ - | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|OLDPWD|REQUEST_ID|SHLVL)=' \ - | sed --regexp-extended \ - --expression 's#(=/tmp/build_)[^:/]+#\1#' \ - --expression 's#^(ENV_DIR=/tmp/).*#\1...#' \ - --expression 's#^(SOURCE_VERSION=).*#\1...#' + | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|OLDPWD|REQUEST_ID|SHLVL)=' \ + | sed --regexp-extended \ + --expression 's#(=/tmp/build_)[^:/]+#\1#' \ + --expression 's#^(ENV_DIR=/tmp/).*#\1...#' \ + --expression 's#^(SOURCE_VERSION=).*#\1...#' diff --git a/vendor/WEB_CONCURRENCY.sh b/vendor/WEB_CONCURRENCY.sh index d6cc1a234..c4ea72096 100755 --- a/vendor/WEB_CONCURRENCY.sh +++ b/vendor/WEB_CONCURRENCY.sh @@ -25,43 +25,43 @@ # exit on error, have to use return not exit, and returning non-zero doesn't have an effect. function detect_memory_limit_in_mb() { - local memory_limit_file='/sys/fs/cgroup/memory/memory.limit_in_bytes' + local memory_limit_file='/sys/fs/cgroup/memory/memory.limit_in_bytes' - # This memory limits file only exists on Heroku, or when using cgroups v1 (Docker < 20.10). - if [[ -f "${memory_limit_file}" ]]; then - local memory_limit_in_mb=$(($(cat "${memory_limit_file}") / 1048576)) + # This memory limits file only exists on Heroku, or when using cgroups v1 (Docker < 20.10). + if [[ -f "${memory_limit_file}" ]]; then + local memory_limit_in_mb=$(($(cat "${memory_limit_file}") / 1048576)) - # Ignore values above 1TB RAM, since when using cgroups v1 the limits file reports a - # bogus value of thousands of TB RAM when there is no container memory limit set. - if ((memory_limit_in_mb <= 1048576)); then - echo "${memory_limit_in_mb}" - return 0 - fi - fi + # Ignore values above 1TB RAM, since when using cgroups v1 the limits file reports a + # bogus value of thousands of TB RAM when there is no container memory limit set. + if ((memory_limit_in_mb <= 1048576)); then + echo "${memory_limit_in_mb}" + return 0 + fi + fi - return 1 + return 1 } function output() { - # Only display log output for web dynos, to prevent breaking one-off dyno scripting use-cases, - # and to prevent confusion from messages about WEB_CONCURRENCY in the logs of non-web workers. - # (We still actually set the env vars for all dyno types for consistency and easier debugging.) - if [[ "${DYNO:-}" == web.* ]]; then - echo "Python buildpack: $*" >&2 - fi + # Only display log output for web dynos, to prevent breaking one-off dyno scripting use-cases, + # and to prevent confusion from messages about WEB_CONCURRENCY in the logs of non-web workers. + # (We still actually set the env vars for all dyno types for consistency and easier debugging.) + if [[ "${DYNO:-}" == web.* ]]; then + echo "Python buildpack: $*" >&2 + fi } if ! available_memory_in_mb=$(detect_memory_limit_in_mb); then - # This should never occur on Heroku, but will be common for non-Heroku environments such as Dokku. - output "Couldn't determine available memory. Skipping automatic configuration of WEB_CONCURRENCY." - return 0 + # This should never occur on Heroku, but will be common for non-Heroku environments such as Dokku. + output "Couldn't determine available memory. Skipping automatic configuration of WEB_CONCURRENCY." + return 0 fi if ! cpu_cores=$(nproc); then - # This should never occur in practice, since this buildpack only supports being run on our base - # images, and nproc is installed in all of them. - output "Couldn't determine number of CPU cores. Skipping automatic configuration of WEB_CONCURRENCY." - return 0 + # This should never occur in practice, since this buildpack only supports being run on our base + # images, and nproc is installed in all of them. + output "Couldn't determine number of CPU cores. Skipping automatic configuration of WEB_CONCURRENCY." + return 0 fi output "Detected ${available_memory_in_mb} MB available memory and ${cpu_cores} CPU cores." @@ -71,25 +71,25 @@ output "Detected ${available_memory_in_mb} MB available memory and ${cpu_cores} export DYNO_RAM="${available_memory_in_mb}" if [[ -v WEB_CONCURRENCY ]]; then - output "Skipping automatic configuration of WEB_CONCURRENCY since it's already set." - return 0 + output "Skipping automatic configuration of WEB_CONCURRENCY since it's already set." + return 0 fi minimum_memory_per_process_in_mb=256 # Prevents WEB_CONCURRENCY being set to zero if the environment is extremely memory constrained. if ((available_memory_in_mb < minimum_memory_per_process_in_mb)); then - max_concurrency_for_available_memory=1 + max_concurrency_for_available_memory=1 else - max_concurrency_for_available_memory=$((available_memory_in_mb / minimum_memory_per_process_in_mb)) + max_concurrency_for_available_memory=$((available_memory_in_mb / minimum_memory_per_process_in_mb)) fi max_concurrency_for_cpu_cores=$((cpu_cores * 2 + 1)) if ((max_concurrency_for_available_memory < max_concurrency_for_cpu_cores)); then - export WEB_CONCURRENCY="${max_concurrency_for_available_memory}" - output "Defaulting WEB_CONCURRENCY to ${WEB_CONCURRENCY} based on the available memory." + export WEB_CONCURRENCY="${max_concurrency_for_available_memory}" + output "Defaulting WEB_CONCURRENCY to ${WEB_CONCURRENCY} based on the available memory." else - export WEB_CONCURRENCY="${max_concurrency_for_cpu_cores}" - output "Defaulting WEB_CONCURRENCY to ${WEB_CONCURRENCY} based on the number of CPU cores." + export WEB_CONCURRENCY="${max_concurrency_for_cpu_cores}" + output "Defaulting WEB_CONCURRENCY to ${WEB_CONCURRENCY} based on the number of CPU cores." fi diff --git a/vendor/buildpack-stdlib_v8.sh b/vendor/buildpack-stdlib_v8.sh index 69e0e7036..b06e729c1 100755 --- a/vendor/buildpack-stdlib_v8.sh +++ b/vendor/buildpack-stdlib_v8.sh @@ -8,55 +8,55 @@ # Buildpack Steps. puts_step() { - if [[ "$*" == "-" ]]; then - read -r output - else - output=$* - fi - echo -e "\\e[1m\\e[36m=== $output\\e[0m" - unset output + if [[ "$*" == "-" ]]; then + read -r output + else + output=$* + fi + echo -e "\\e[1m\\e[36m=== $output\\e[0m" + unset output } # Buildpack Error. puts_error() { - if [[ "$*" == "-" ]]; then - read -r output - else - output=$* - fi - echo -e "\\e[1m\\e[31m=!= $output\\e[0m" + if [[ "$*" == "-" ]]; then + read -r output + else + output=$* + fi + echo -e "\\e[1m\\e[31m=!= $output\\e[0m" } # Buildpack Warning. puts_warn() { - if [[ "$*" == "-" ]]; then - read -r output - else - output=$* - fi - echo -e "\\e[1m\\e[33m=!= $output\\e[0m" + if [[ "$*" == "-" ]]; then + read -r output + else + output=$* + fi + echo -e "\\e[1m\\e[33m=!= $output\\e[0m" } # Is verbose set? is_verbose() { - if [[ -n $BUILDPACK_VERBOSE ]]; then - return 0 - else - return 1 - fi + if [[ -n $BUILDPACK_VERBOSE ]]; then + return 0 + else + return 1 + fi } # Buildpack Verbose. puts_verbose() { - if is_verbose; then - if [[ "$*" == "-" ]]; then - read -r output - else - output=$* - fi - echo "$output" - unset output - fi + if is_verbose; then + if [[ "$*" == "-" ]]; then + read -r output + else + output=$* + fi + echo "$output" + unset output + fi } # Buildpack Utilities @@ -65,50 +65,50 @@ puts_verbose() { # Usage: $ set-env key value # NOTICE: Expects PROFILE_PATH & EXPORT_PATH to be set! set_env() { - # TODO: automatically create profile path directory if it doesn't exist. - echo "export $1=$2" >> "$PROFILE_PATH" - echo "export $1=$2" >> "$EXPORT_PATH" + # TODO: automatically create profile path directory if it doesn't exist. + echo "export $1=$2" >>"$PROFILE_PATH" + echo "export $1=$2" >>"$EXPORT_PATH" } # Usage: $ set-default-env key value # NOTICE: Expects PROFILE_PATH & EXPORT_PATH to be set! set_default_env() { - echo "export $1=\${$1:-$2}" >> "$PROFILE_PATH" - echo "export $1=\${$1:-$2}" >> "$EXPORT_PATH" + echo "export $1=\${$1:-$2}" >>"$PROFILE_PATH" + echo "export $1=\${$1:-$2}" >>"$EXPORT_PATH" } # Usage: $ un-set-env key # NOTICE: Expects PROFILE_PATH to be set! un_set_env() { - echo "unset $1" >> "$PROFILE_PATH" + echo "unset $1" >>"$PROFILE_PATH" } # Usage: $ _env-blacklist pattern # Outputs a regex of default blacklist env vars. _env_blacklist() { - local regex=${1:-''} - if [[ -n "$regex" ]]; then - regex="|$regex" - fi - echo "^(PATH|CPATH|CPPATH|LD_PRELOAD|LIBRARY_PATH|LD_LIBRARY_PATH|PYTHONHOME$regex)$" + local regex=${1:-''} + if [[ -n "$regex" ]]; then + regex="|$regex" + fi + echo "^(PATH|CPATH|CPPATH|LD_PRELOAD|LIBRARY_PATH|LD_LIBRARY_PATH|PYTHONHOME$regex)$" } # Usage: $ export-env ENV_DIR WHITELIST BLACKLIST # Exports the environment variables defined in the given directory. export_env() { - local env_dir=${1:-$ENV_DIR} - local whitelist=${2:-''} - local blacklist - blacklist="$(_env_blacklist "$3")" - if [[ -d "$env_dir" ]]; then - # Environment variable names won't contain characters affected by: - # shellcheck disable=SC2045 - for e in $(ls "$env_dir"); do - echo "$e" | grep -E "$whitelist" | grep -qvE "$blacklist" && - export "$e=$(cat "$env_dir/$e")" - : - done - fi + local env_dir=${1:-$ENV_DIR} + local whitelist=${2:-''} + local blacklist + blacklist="$(_env_blacklist "$3")" + if [[ -d "$env_dir" ]]; then + # Environment variable names won't contain characters affected by: + # shellcheck disable=SC2045 + for e in $(ls "$env_dir"); do + echo "$e" | grep -E "$whitelist" | grep -qvE "$blacklist" \ + && export "$e=$(cat "$env_dir/$e")" + : + done + fi } # Usage: $ sub-env command @@ -118,16 +118,16 @@ export_env() { # WHITELIST=${2:-''} # BLACKLIST=${3:-'^(GIT_DIR|PYTHONHOME|LD_LIBRARY_PATH|LIBRARY_PATH|PATH)$'} sub_env() { - ( - # TODO: Fix https://github.com/heroku/buildpack-stdlib/issues/37 - # shellcheck disable=SC2153 - export_env "$ENV_DIR" "$WHITELIST" "$BLACKLIST" + ( + # TODO: Fix https://github.com/heroku/buildpack-stdlib/issues/37 + # shellcheck disable=SC2153 + export_env "$ENV_DIR" "$WHITELIST" "$BLACKLIST" - "$@" - ) + "$@" + ) } # Returns the current time, in milliseconds. nowms() { - date +%s%3N + date +%s%3N } From 6697fd5fe400a531ade7234b954cdb607f9069cd Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 23 Sep 2024 12:35:06 +0000 Subject: [PATCH 016/406] Switch to a new output helper for error messages (#1639) The existing logging functions in `bin/utils` don't support being passed multi-line output. Upcoming PRs are going to be adding a number of new error messages, so this adds a new output module under `lib/` (that supports multi-line output and also uses colour) and switches the existing buildpack error messages to it. Now that the buildpack error messages use ANSI colour codes, the Hatchet output has to have ANSI sequences stripped prior to the output assertions. GUS-W-16808943. --- CHANGELOG.md | 1 + bin/compile | 51 +++++++++++----------- bin/detect | 15 ++++--- bin/steps/python | 42 +++++++++--------- builds/test_python_runtime.sh | 11 +++-- lib/output.sh | 18 ++++++++ spec/hatchet/detect_spec.rb | 1 + spec/hatchet/package_manager_spec.rb | 4 +- spec/hatchet/pipenv_spec.rb | 26 ++++++----- spec/hatchet/python_update_warning_spec.rb | 6 +-- spec/hatchet/python_version_spec.rb | 14 +++--- spec/spec_helper.rb | 9 ++-- 12 files changed, 117 insertions(+), 81 deletions(-) create mode 100644 lib/output.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index b084b5158..dc1e65506 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Buildpack error messages are now more consistently formatted and use colour. ([#1639](https://github.com/heroku/heroku-buildpack-python/pull/1639)) ## [v256] - 2024-09-07 diff --git a/bin/compile b/bin/compile index c89420201..9bd6482b7 100755 --- a/bin/compile +++ b/bin/compile @@ -15,6 +15,7 @@ BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) source "${BUILDPACK_DIR}/bin/utils" source "${BUILDPACK_DIR}/lib/metadata.sh" +source "${BUILDPACK_DIR}/lib/output.sh" compile_start_time=$(nowms) @@ -137,32 +138,30 @@ fi # TODO: Move this into a new package manager handling implementation when adding Poetry support. # We intentionally don't mention `setup.py` here since it's being removed soon. if [[ ! -f requirements.txt && ! -f Pipfile && ! -f setup.py ]]; then - puts-warn - puts-warn "Error: Couldn't find any supported Python package manager files." - puts-warn - puts-warn "A Python app on Heroku must have either a 'requirements.txt' or" - puts-warn "'Pipfile' package manager file in the root directory of its" - puts-warn "source code." - puts-warn - puts-warn "Currently the root directory of your app contains:" - puts-warn - # TODO: Overhaul logging helpers so they can handle prefixing multi-line strings, and switch to them. - # shellcheck disable=SC2012 # Using `ls` instead of `find` is absolutely fine for this use case. - ls -1 --indicator-style=slash "${BUILD_DIR}" | sed 's/^/ ! /' - puts-warn - puts-warn "If your app already has a package manager file, check that it:" - puts-warn - puts-warn "1. Is in the top level directory (not a subdirectory)." - puts-warn "2. Has the correct spelling (the filenames are case-sensitive)." - puts-warn "3. Isn't listed in '.gitignore' or '.slugignore'." - puts-warn - puts-warn "Otherwise, add a package manager file to your app. If your app has" - puts-warn "no dependencies, then create an empty 'requirements.txt' file." - puts-warn - puts-warn "For help with using Python on Heroku, see:" - puts-warn "https://devcenter.heroku.com/articles/getting-started-with-python" - puts-warn "https://devcenter.heroku.com/articles/python-support" - puts-warn + display_error <<-EOF + Error: Couldn't find any supported Python package manager files. + + A Python app on Heroku must have either a 'requirements.txt' or + 'Pipfile' package manager file in the root directory of its + source code. + + Currently the root directory of your app contains: + + $(ls -1 --indicator-style=slash "${BUILD_DIR}") + + If your app already has a package manager file, check that it: + + 1. Is in the top level directory (not a subdirectory). + 2. Has the correct spelling (the filenames are case-sensitive). + 3. Isn't listed in '.gitignore' or '.slugignore'. + + Otherwise, add a package manager file to your app. If your app has + no dependencies, then create an empty 'requirements.txt' file. + + For help with using Python on Heroku, see: + https://devcenter.heroku.com/articles/getting-started-with-python + https://devcenter.heroku.com/articles/python-support + EOF meta_set "failure_reason" "package-manager-not-found" exit 1 fi diff --git a/bin/detect b/bin/detect index 498792bab..69345eb78 100755 --- a/bin/detect +++ b/bin/detect @@ -1,11 +1,16 @@ #!/usr/bin/env bash -# Usage: bin/compile +# Usage: bin/detect # See: https://devcenter.heroku.com/articles/buildpack-api set -euo pipefail BUILD_DIR="${1}" +# The absolute path to the root of the buildpack. +BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) + +source "${BUILDPACK_DIR}/lib/output.sh" + # Filenames that if found in a project mean it should be treated as a Python project, # and so pass this buildpack's detection phase. # @@ -36,14 +41,10 @@ for filename in "${KNOWN_PYTHON_PROJECT_FILES[@]}"; do fi done -# Cytokine incorrectly indents the first line, so we have to leave it empty. -echo 1>&2 - # Note: This error message intentionally doesn't list all of the filetypes above, # since during compile the build will still require a package manager file, so it -# makes sense to describe the stricter requirements up front. -# TODO: Overhaul logging helpers so they can handle prefixing multi-line strings, and switch to them. -sed 's/^/ ! /' 1>&2 <&2 + exit 1 +} + # We intentionally extract the Python runtime into a different directory to the one into which it # was originally installed before being packaged, to check that relocation works (since buildpacks # depend on it). Since the Python binary was built in shared mode, `LD_LIBRARY_PATH` must be set @@ -22,8 +27,7 @@ tar --zstd --extract --verbose --file "${ARCHIVE_FILEPATH}" --directory "${INSTA # Check that all dynamically linked libraries exist in the run image (since it has fewer packages than the build image). LDD_OUTPUT=$(find "${INSTALL_DIR}" -type f,l \( -name 'python3' -o -name '*.so*' \) -exec ldd '{}' +) if grep 'not found' <<<"${LDD_OUTPUT}" | sort --unique; then - echo "The above dynamically linked libraries were not found!" - exit 1 + abort "The above dynamically linked libraries were not found!" fi # Check that optional and/or system library dependent stdlib modules were built. @@ -46,6 +50,5 @@ if ! "${INSTALL_DIR}/bin/python3" -c "import $( IFS=, echo "${optional_stdlib_modules[*]}" )"; then - echo "The above optional stdlib module failed to import! Check the compile logs to see if it was skipped due to missing libraries/headers." - exit 1 + abort "The above optional stdlib module failed to import! Check the compile logs to see if it was skipped due to missing libraries/headers." fi diff --git a/lib/output.sh b/lib/output.sh new file mode 100644 index 000000000..e063ddd88 --- /dev/null +++ b/lib/output.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash + +ANSI_RED='\033[1;31m' +ANSI_RESET='\033[0m' + +# shellcheck disable=SC2120 # Prevent warnings about unused arguments due to the split args vs stdin API. +function display_error() { + # Send all output to stderr + exec 1>&2 + # If arguments are given, redirect them to stdin. This allows the function + # to be invoked with either a string argument or stdin (e.g. via <<-EOF). + (($#)) && exec <<<"${@}" + echo + while IFS= read -r line; do + echo -e "${ANSI_RED} ! ${line}${ANSI_RESET}" + done + echo +} diff --git a/spec/hatchet/detect_spec.rb b/spec/hatchet/detect_spec.rb index ceb25581f..dbe7f1e93 100644 --- a/spec/hatchet/detect_spec.rb +++ b/spec/hatchet/detect_spec.rb @@ -39,6 +39,7 @@ remote: ! https://devcenter.heroku.com/articles/getting-started-with-python remote: ! https://devcenter.heroku.com/articles/python-support remote: + remote: remote: More info: https://devcenter.heroku.com/articles/buildpacks#detection-failure OUTPUT end diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index d247e1deb..70473fc81 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -10,7 +10,7 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: ! + remote: remote: ! Error: Couldn't find any supported Python package manager files. remote: ! remote: ! A Python app on Heroku must have either a 'requirements.txt' or @@ -34,7 +34,7 @@ remote: ! For help with using Python on Heroku, see: remote: ! https://devcenter.heroku.com/articles/getting-started-with-python remote: ! https://devcenter.heroku.com/articles/python-support - remote: ! + remote: remote: ! Push rejected, failed to compile Python app. OUTPUT end diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 6c0e2fa97..ccd5dee93 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -24,12 +24,12 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python version specified in Pipfile.lock - remote: ! - remote: ! Requested runtime 'python-#{requested_version}' is not available for this stack (#{app.stack}). + remote: + remote: ! Error: Requested runtime 'python-#{requested_version}' is not available for this stack (#{app.stack}). remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: ! + remote: remote: ! Push rejected, failed to compile Python app. OUTPUT end @@ -125,6 +125,8 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~OUTPUT)) remote: -----> Python app detected remote: -----> Using Python version specified in Pipfile.lock + remote: + remote: ! Error: Python 3.6 is no longer supported. remote: ! remote: ! Python 3.6 reached upstream end-of-life on December 23rd, 2021, and is remote: ! therefore no longer receiving security updates: @@ -136,7 +138,7 @@ remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: ! + remote: remote: ! Push rejected, failed to compile Python app. OUTPUT end @@ -151,6 +153,8 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~OUTPUT)) remote: -----> Python app detected remote: -----> Using Python version specified in Pipfile.lock + remote: + remote: ! Error: Python 3.7 is no longer supported. remote: ! remote: ! Python 3.7 reached upstream end-of-life on June 27th, 2023, and is remote: ! therefore no longer receiving security updates: @@ -162,7 +166,7 @@ remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: ! + remote: remote: ! Push rejected, failed to compile Python app. OUTPUT end @@ -273,12 +277,12 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python version specified in Pipfile.lock - remote: ! - remote: ! Requested runtime '^3.12' is not available for this stack (#{app.stack}). + remote: + remote: ! Error: Requested runtime '^3.12' is not available for this stack (#{app.stack}). remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: ! + remote: remote: ! Push rejected, failed to compile Python app. OUTPUT end @@ -293,12 +297,12 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python version specified in Pipfile.lock - remote: ! - remote: ! Requested runtime 'python-X.Y.Z' is not available for this stack (#{app.stack}). + remote: + remote: ! Error: Requested runtime 'python-X.Y.Z' is not available for this stack (#{app.stack}). remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: ! + remote: remote: ! Push rejected, failed to compile Python app. OUTPUT end diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index df909ee73..d6d43c2f6 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -24,12 +24,12 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python version specified in runtime.txt - remote: ! - remote: ! Requested runtime 'python-#{requested_version}' is not available for this stack (#{app.stack}). + remote: + remote: ! Error: Requested runtime 'python-#{requested_version}' is not available for this stack (#{app.stack}). remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: ! + remote: remote: ! Push rejected, failed to compile Python app. OUTPUT end diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index a324089ba..f13c148ac 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -25,12 +25,12 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python version specified in runtime.txt - remote: ! - remote: ! Requested runtime '#{requested_runtime}' is not available for this stack (#{app.stack}). + remote: + remote: ! Error: Requested runtime '#{requested_runtime}' is not available for this stack (#{app.stack}). remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: ! + remote: remote: ! Push rejected, failed to compile Python app. OUTPUT end @@ -91,6 +91,8 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python version specified in runtime.txt + remote: + remote: ! Error: Python 3.6 is no longer supported. remote: ! remote: ! Python 3.6 reached upstream end-of-life on December 23rd, 2021, and is remote: ! therefore no longer receiving security updates: @@ -102,7 +104,7 @@ remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: ! + remote: remote: ! Push rejected, failed to compile Python app. OUTPUT end @@ -117,6 +119,8 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python version specified in runtime.txt + remote: + remote: ! Error: Python 3.7 is no longer supported. remote: ! remote: ! Python 3.7 reached upstream end-of-life on June 27th, 2023, and is remote: ! therefore no longer receiving security updates: @@ -128,7 +132,7 @@ remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: ! + remote: remote: ! Push rejected, failed to compile Python app. OUTPUT end diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 93e3b2bdb..5f761bf3b 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -51,9 +51,12 @@ def get_requirement_version(package_name) end def clean_output(output) - # Remove trailing whitespace characters added by Git: - # https://github.com/heroku/hatchet/issues/162 - output.gsub(/ {8}(?=\R)/, '') + output + # Remove trailing whitespace characters added by Git: + # https://github.com/heroku/hatchet/issues/162 + .gsub(/ {8}(?=\R)/, '') + # Remove ANSI colour codes used in buildpack output (e.g. error messages). + .gsub(/\e\[[0-9;]+m/, '') end def update_buildpacks(app, buildpacks) From 8b00114eeb91e7c767da5d64fec386a61cdde0b4 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 24 Sep 2024 15:42:08 +0000 Subject: [PATCH 017/406] Refactor package manager handling (#1640) Previously, the buildpack didn't track which package manager was being used, but instead had a number of fragile file presence checks and env vars like `SKIP_PIP_INSTALL` scattered throughout (which won't work when the package manager isn't a binary choice). As such, package manager handling has been refactored to allow for more easily adding Poetry support later. This also continues the gradual migration of code from sourced `bin/steps/` scripts, to functions declared in `lib/`. As part of this, I've started namespacing the lib functions using the `module::fn_name()` pattern described in: https://google.github.io/styleguide/shellguide.html#function-names On the most part I've tried to keep this refactoring a no-op (aside from fixes for a few quirks/bugs found whilst working on this). Longer term we will be deprecating/sunsetting several features, such as the `setup.py` fallback, using Pipenv without a lockfile, and support for apps that have ended up with the files for multiple package managers. GUS-W-16811078. --- CHANGELOG.md | 7 +- bin/compile | 112 ++++++++++++------------------ bin/report | 2 + bin/steps/pip-install | 44 ------------ bin/steps/pipenv-python-version | 4 +- bin/steps/python | 31 --------- buildpack.toml | 1 + lib/kvstore.sh | 2 + lib/metadata.sh | 2 + lib/output.sh | 2 + lib/package_manager.sh | 77 ++++++++++++++++++++ lib/pip.sh | 61 ++++++++++++++++ bin/steps/pipenv => lib/pipenv.sh | 52 ++++++-------- lib/utils.sh | 29 ++++++++ spec/hatchet/ci_spec.rb | 11 +-- spec/hatchet/pip_spec.rb | 2 +- spec/hatchet/pipenv_spec.rb | 49 ++++++++----- 17 files changed, 290 insertions(+), 198 deletions(-) delete mode 100755 bin/steps/pip-install create mode 100644 lib/package_manager.sh create mode 100644 lib/pip.sh rename bin/steps/pipenv => lib/pipenv.sh (60%) mode change 100755 => 100644 create mode 100644 lib/utils.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index dc1e65506..f0e0c71e1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,12 @@ ## [Unreleased] -- Buildpack error messages are now more consistently formatted and use colour. ([#1639](https://github.com/heroku/heroku-buildpack-python/pull/1639)) +- Moved the SQLite3 install step prior to installing dependencies when using Pipenv. This now matches the behaviour when using pip and allows dependencies to actually use the headers. ([#1640](https://github.com/heroku/heroku-buildpack-python/pull/1640)) +- Stopped exposing the `SKIP_PIP_INSTALL` env var to `bin/post_compile` and other subprocesses when using Pipenv. ([#1640](https://github.com/heroku/heroku-buildpack-python/pull/1640)) +- Stopped creating `.heroku/python/requirements-{declared,installed}.txt` files when using pip. ([#1640](https://github.com/heroku/heroku-buildpack-python/pull/1640)) +- Stopped creating a placeholder `requirements.txt` file when an app only has a `setup.py` file and no other package manager files. Instead pip is now invoked directly using `--editable .`. ([#1640](https://github.com/heroku/heroku-buildpack-python/pull/1640)) +- Improved buildpack metrics for package manager detection and duration of install steps. ([#1640](https://github.com/heroku/heroku-buildpack-python/pull/1640)) +- Updated buildpack-generated error messages to use colour and be more consistently formatted. ([#1639](https://github.com/heroku/heroku-buildpack-python/pull/1639)) ## [v256] - 2024-09-07 diff --git a/bin/compile b/bin/compile index 9bd6482b7..d598039e9 100755 --- a/bin/compile +++ b/bin/compile @@ -16,6 +16,10 @@ BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) source "${BUILDPACK_DIR}/bin/utils" source "${BUILDPACK_DIR}/lib/metadata.sh" source "${BUILDPACK_DIR}/lib/output.sh" +source "${BUILDPACK_DIR}/lib/package_manager.sh" +source "${BUILDPACK_DIR}/lib/pip.sh" +source "${BUILDPACK_DIR}/lib/pipenv.sh" +source "${BUILDPACK_DIR}/lib/utils.sh" compile_start_time=$(nowms) @@ -97,12 +101,10 @@ cd "$BUILD_DIR" # - The build is executed, modifying `~/.heroku/{known-paths}`. # - Once the build is complete, `~/.heroku/{known-paths}` is copied back into the cache. -# Create the cache directory, if it doesn't exist. mkdir -p "$CACHE_DIR/.heroku" # Restore old artifacts from the cache. mkdir -p .heroku - # The Python installation. cp -R "$CACHE_DIR/.heroku/python" .heroku/ &>/dev/null || true # A plain text file which contains the current stack being used (used for cache busting). @@ -116,10 +118,7 @@ if [[ -d "$CACHE_DIR/.heroku/src" ]]; then cp -R "$CACHE_DIR/.heroku/src" .heroku/ &>/dev/null || true fi -# The pre_compile hook. Customers rely on this. Don't remove it. -# This part of the code is used to allow users to customize their build experience -# without forking the buildpack by providing a `bin/pre_compile` script, which gets -# run inline with the buildpack automatically. +# Runs a `bin/pre_compile` script if found in the app source, allowing build customisation. source "${BUILDPACK_DIR}/bin/steps/hooks/pre_compile" # Sticky runtimes. If there was a previous build, and it used a given version of Python, @@ -128,43 +127,15 @@ if [[ -f "$CACHE_DIR/.heroku/python-version" ]]; then CACHED_PYTHON_VERSION=$(cat "$CACHE_DIR/.heroku/python-version") fi -# We didn't always record the stack version. This code is in place because of that. +# We didn't always record the stack version. if [[ -f "$CACHE_DIR/.heroku/python-stack" ]]; then CACHED_PYTHON_STACK=$(cat "$CACHE_DIR/.heroku/python-stack") else CACHED_PYTHON_STACK=$STACK fi -# TODO: Move this into a new package manager handling implementation when adding Poetry support. -# We intentionally don't mention `setup.py` here since it's being removed soon. -if [[ ! -f requirements.txt && ! -f Pipfile && ! -f setup.py ]]; then - display_error <<-EOF - Error: Couldn't find any supported Python package manager files. - - A Python app on Heroku must have either a 'requirements.txt' or - 'Pipfile' package manager file in the root directory of its - source code. - - Currently the root directory of your app contains: - - $(ls -1 --indicator-style=slash "${BUILD_DIR}") - - If your app already has a package manager file, check that it: - - 1. Is in the top level directory (not a subdirectory). - 2. Has the correct spelling (the filenames are case-sensitive). - 3. Isn't listed in '.gitignore' or '.slugignore'. - - Otherwise, add a package manager file to your app. If your app has - no dependencies, then create an empty 'requirements.txt' file. - - For help with using Python on Heroku, see: - https://devcenter.heroku.com/articles/getting-started-with-python - https://devcenter.heroku.com/articles/python-support - EOF - meta_set "failure_reason" "package-manager-not-found" - exit 1 -fi +PACKAGE_MANAGER=$(package_manager::determine_package_manager "${BUILD_DIR}") +meta_set "package_manager" "${PACKAGE_MANAGER}" # Pipenv Python version support. # Detect the version of Python requested from a Pipfile (e.g. python_version or python_full_version). @@ -189,9 +160,9 @@ else echo "${DEFAULT_PYTHON_VERSION}" >runtime.txt fi -# Create the directory for .profile.d, if it doesn't exist. +# The directory for the .profile.d scripts. mkdir -p "$(dirname "$PROFILE_PATH")" -# Create the directory for editable source code installation, if it doesn't exist. +# The directory for editable VCS dependencies. mkdir -p /app/.heroku/src # On Heroku CI, builds happen in `/app`. Otherwise, on the Heroku platform, @@ -207,46 +178,58 @@ if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then # Note: .heroku/src is copied in later. fi -# Download / Install Python, from pre-build binaries available on Amazon S3. -# This step also bootstraps pip / setuptools. +# Download and install Python using pre-built binaries from S3. install_python_start_time=$(nowms) source "${BUILDPACK_DIR}/bin/steps/python" meta_time "python_install_duration" "${install_python_start_time}" -# Install Pipenv dependencies, if a Pipfile was provided. -source "${BUILDPACK_DIR}/bin/steps/pipenv" - -# If no requirements.txt file given, assume `setup.py develop` is intended. -# This allows for people to ship a setup.py application to Heroku - -if [[ ! -f requirements.txt ]] && [[ ! -f Pipfile ]]; then - meta_set "setup_py_only" "true" - echo "-e ." >requirements.txt -else - meta_set "setup_py_only" "false" -fi +# Install the package manager and related tools. +package_manager_install_start_time=$(nowms) +bundled_pip_module_path="$(utils::bundled_pip_module_path "${BUILD_DIR}")" +case "${PACKAGE_MANAGER}" in + pip) + pip::install_pip_setuptools_wheel "${bundled_pip_module_path}" + ;; + pipenv) + # TODO: Stop installing pip when using Pipenv. + pip::install_pip_setuptools_wheel "${bundled_pip_module_path}" + pipenv::install_pipenv + ;; + *) + abort_internal_error "Unhandled package manager" + ;; +esac +meta_time "package_manager_install_duration" "${package_manager_install_start_time}" # SQLite3 support. -# This sets up and installs sqlite3 dev headers and the sqlite3 binary but not the -# libsqlite3-0 library since that exists on the stack image. +# Installs the sqlite3 dev headers and sqlite3 binary but not the +# libsqlite3-0 library since that exists in the base image. install_sqlite_start_time=$(nowms) source "${BUILDPACK_DIR}/bin/steps/sqlite3" buildpack_sqlite3_install meta_time "sqlite_install_duration" "${install_sqlite_start_time}" -# pip install -# ----------- - -# Install dependencies with pip (where the magic happens). -source "${BUILDPACK_DIR}/bin/steps/pip-install" +# Install app dependencies. +dependencies_install_start_time=$(nowms) +case "${PACKAGE_MANAGER}" in + pip) + pip::install_dependencies + ;; + pipenv) + pipenv::install_dependencies + ;; + *) + abort_internal_error "Unhandled package manager" + ;; +esac +meta_time "dependencies_install_duration" "${dependencies_install_start_time}" # Support for NLTK corpora. nltk_downloader_start_time=$(nowms) sub_env "${BUILDPACK_DIR}/bin/steps/nltk" meta_time "nltk_downloader_duration" "${nltk_downloader_start_time}" -# Support for editable installations. Here, we are copying pip–created src directory, -# and copying it into the proper place (the logical place to do this was early, but it must be done here). +# Support for editable installations. # In CI, $BUILD_DIR is /app. # Realpath is used to support use-cases where one of the locations is a symlink to the other. if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then @@ -256,9 +239,6 @@ fi # Django collectstatic support. # The buildpack automatically runs collectstatic for Django applications. -# This is the cause for the majority of build failures on the Python platform. -# These failures are intentional — if collectstatic (which can be tricky, at times) fails, -# your build fails. collectstatic_start_time=$(nowms) sub_env "${BUILDPACK_DIR}/bin/steps/collectstatic" meta_time "django_collectstatic_duration" "${collectstatic_start_time}" @@ -304,7 +284,7 @@ fi cp "${BUILDPACK_DIR}/vendor/WEB_CONCURRENCY.sh" "$WEB_CONCURRENCY_PROFILE_PATH" cp "${BUILDPACK_DIR}/vendor/python.gunicorn.sh" "$GUNICORN_PROFILE_PATH" -# Experimental post_compile hook. Don't remove this. +# Runs a `bin/post_compile` script if found in the app source, allowing build customisation. source "${BUILDPACK_DIR}/bin/steps/hooks/post_compile" # Store new artifacts in the cache. diff --git a/bin/report b/bin/report index 9210d7371..5697a7f57 100755 --- a/bin/report +++ b/bin/report @@ -67,6 +67,7 @@ STRING_FIELDS=( failure_reason nltk_downloader package_manager + package_manager_multiple_found pip_version pipenv_version python_version_major @@ -81,6 +82,7 @@ ALL_OTHER_FIELDS=( dependencies_install_duration django_collectstatic_duration nltk_downloader_duration + package_manager_install_duration pipenv_has_lockfile post_compile_hook post_compile_hook_duration diff --git a/bin/steps/pip-install b/bin/steps/pip-install deleted file mode 100755 index 86ecfd0d3..000000000 --- a/bin/steps/pip-install +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/env bash - -if [[ -z "$SKIP_PIP_INSTALL" ]]; then - pip_install_start_time=$(nowms) - meta_set "package_manager" "pip" - - puts-step "Installing requirements with pip" - - # Set Pip env vars - # This reads certain environment variables set on the Heroku app config - # and makes them accessible to the pip install process. - # - # PIP_EXTRA_INDEX_URL allows for an alternate pypi URL to be used. - if [[ -r "$ENV_DIR/PIP_EXTRA_INDEX_URL" ]]; then - PIP_EXTRA_INDEX_URL="$(cat "$ENV_DIR/PIP_EXTRA_INDEX_URL")" - export PIP_EXTRA_INDEX_URL - fi - - set +e - - /app/.heroku/python/bin/pip install -r requirements.txt --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir --progress-bar off 2>&1 | tee "$WARNINGS_LOG" | cleanup | indent - PIP_STATUS="${PIPESTATUS[0]}" - set -e - - show-warnings - - if [[ ! $PIP_STATUS -eq 0 ]]; then - meta_set "failure_reason" "pip-install" - exit 1 - fi - - cp requirements.txt .heroku/python/requirements-declared.txt - /app/.heroku/python/bin/pip freeze --disable-pip-version-check >.heroku/python/requirements-installed.txt - - # Install test dependencies, for CI. - if [[ -n "$INSTALL_TEST" ]]; then - if [[ -f requirements-test.txt ]]; then - puts-step "Installing test dependencies..." - /app/.heroku/python/bin/pip install -r requirements-test.txt --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir 2>&1 | cleanup | indent - fi - fi - - meta_time "dependencies_install_duration" "${pip_install_start_time}" -fi diff --git a/bin/steps/pipenv-python-version b/bin/steps/pipenv-python-version index b6614cf99..6b4bd46e5 100755 --- a/bin/steps/pipenv-python-version +++ b/bin/steps/pipenv-python-version @@ -1,8 +1,10 @@ #!/usr/bin/env bash +# TODO: Move this to lib/ as part of the refactoring for .python-version support. + # Detect Python-version with Pipenv. -if [[ -f $BUILD_DIR/Pipfile ]]; then +if [[ "${PACKAGE_MANAGER}" == "pipenv" ]]; then if [[ ! -f $BUILD_DIR/runtime.txt ]]; then if [[ ! -f $BUILD_DIR/Pipfile.lock ]]; then diff --git a/bin/steps/python b/bin/steps/python index 3c1e36e57..56af2778e 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -166,34 +166,3 @@ else hash -r fi - -PIP_VERSION=$(get_requirement_version 'pip') -SETUPTOOLS_VERSION=$(get_requirement_version 'setuptools') -WHEEL_VERSION=$(get_requirement_version 'wheel') - -puts-step "Installing pip ${PIP_VERSION}, setuptools ${SETUPTOOLS_VERSION} and wheel ${WHEEL_VERSION}" - -meta_set "pip_version" "${PIP_VERSION}" -meta_set "setuptools_version" "${SETUPTOOLS_VERSION}" -meta_set "wheel_version" "${WHEEL_VERSION}" - -# Python bundles Pip within its standard library, which we can use to install our chosen -# pip version from PyPI, saving us from having to download the usual pip bootstrap script. -# We have to use a glob since the bundled wheel filename contains the pip version, which -# differs between Python versions. We also have to handle the case where there are multiple -# matching pip wheels, since in some versions of Python (eg 3.9.0) multiple versions of pip -# were accidentally bundled upstream. Note: This implementation relies upon `nullglob` being -# set, which is the case thanks to the `bin/utils` that was run earlier. -BUNDLED_PIP_WHEEL_LIST=(.heroku/python/lib/python*/ensurepip/_bundled/pip-*.whl) -BUNDLED_PIP_WHEEL="${BUNDLED_PIP_WHEEL_LIST[0]}" - -if [[ -z "${BUNDLED_PIP_WHEEL}" ]]; then - display_error "Error: Failed to locate the bundled pip wheel." - meta_set "failure_reason" "bundled-pip-not-found" - exit 1 -fi - -/app/.heroku/python/bin/python "${BUNDLED_PIP_WHEEL}/pip" install --quiet --disable-pip-version-check --no-cache-dir \ - "pip==${PIP_VERSION}" "setuptools==${SETUPTOOLS_VERSION}" "wheel==${WHEEL_VERSION}" - -hash -r diff --git a/buildpack.toml b/buildpack.toml index 07f83d4cf..4ccd6f242 100644 --- a/buildpack.toml +++ b/buildpack.toml @@ -7,6 +7,7 @@ files = [ "builds/", "etc/publish.sh", "spec/", + ".editorconfig", ".gitignore", ".rubocop.yml", ".shellcheckrc", diff --git a/lib/kvstore.sh b/lib/kvstore.sh index 17e2dd0d0..4bfdc85ac 100644 --- a/lib/kvstore.sh +++ b/lib/kvstore.sh @@ -1,5 +1,7 @@ #!/usr/bin/env bash +# TODO: Switch this file to using namespaced functions like `kvstore::`. + # Taken from: https://github.com/heroku/heroku-buildpack-nodejs/blob/main/lib/kvstore.sh kv_create() { diff --git a/lib/metadata.sh b/lib/metadata.sh index 7d2a39b7a..3def5ded3 100644 --- a/lib/metadata.sh +++ b/lib/metadata.sh @@ -1,5 +1,7 @@ #!/usr/bin/env bash +# TODO: Switch this file to using namespaced functions like `metadata::`. + # Based on: https://github.com/heroku/heroku-buildpack-nodejs/blob/main/lib/metadata.sh source "${BUILDPACK_DIR}/lib/kvstore.sh" diff --git a/lib/output.sh b/lib/output.sh index e063ddd88..246e689bf 100644 --- a/lib/output.sh +++ b/lib/output.sh @@ -1,5 +1,7 @@ #!/usr/bin/env bash +# TODO: Switch this file to using namespaced functions like `output::`. + ANSI_RED='\033[1;31m' ANSI_RESET='\033[0m' diff --git a/lib/package_manager.sh b/lib/package_manager.sh new file mode 100644 index 000000000..d776c08a5 --- /dev/null +++ b/lib/package_manager.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash + +function package_manager::determine_package_manager() { + local build_dir="${1}" + local package_managers_found=() + + if [[ -f "${build_dir}/Pipfile.lock" ]]; then + package_managers_found+=(pipenv) + meta_set "pipenv_has_lockfile" "true" + elif [[ -f "${build_dir}/Pipfile" ]]; then + # TODO: Start requiring a Pipfile.lock and make this branch a "missing lockfile" error instead. + package_managers_found+=(pipenv) + meta_set "pipenv_has_lockfile" "false" + fi + + if [[ -f "${build_dir}/requirements.txt" ]]; then + package_managers_found+=(pip) + fi + + # TODO: Deprecate/sunset this fallback, since using setup.py declared dependencies is + # not a best practice, and we can only guess as to which package manager to use. + if ((${#package_managers_found[@]} == 0)) && [[ -f "${build_dir}/setup.py" ]]; then + package_managers_found+=(pip) + meta_set "setup_py_only" "true" + else + meta_set "setup_py_only" "false" + fi + + case "${#package_managers_found[@]}" in + 1) + echo "${package_managers_found[0]}" + return 0 + ;; + 0) + display_error <<-EOF + Error: Couldn't find any supported Python package manager files. + + A Python app on Heroku must have either a 'requirements.txt' or + 'Pipfile' package manager file in the root directory of its + source code. + + Currently the root directory of your app contains: + + $(ls -1 --indicator-style=slash "${build_dir}" || true) + + If your app already has a package manager file, check that it: + + 1. Is in the top level directory (not a subdirectory). + 2. Has the correct spelling (the filenames are case-sensitive). + 3. Isn't listed in '.gitignore' or '.slugignore'. + + Otherwise, add a package manager file to your app. If your app has + no dependencies, then create an empty 'requirements.txt' file. + + For help with using Python on Heroku, see: + https://devcenter.heroku.com/articles/getting-started-with-python + https://devcenter.heroku.com/articles/python-support + EOF + meta_set "failure_reason" "package-manager-not-found" + return 1 + ;; + *) + # If multiple package managers are found, use the first found above. + # TODO: Turn this case into an error since it results in support tickets from users + # who don't realise they have multiple package manager files and think their changes + # aren't taking effect. (We'll need to wait until after Poetry support has landed, + # and people have had a chance to migrate from the third-party Poetry buildpack, + # since using it results in both a requirements.txt and a poetry.lock.) + echo "${package_managers_found[0]}" + meta_set "package_manager_multiple_found" "$( + IFS=, + echo "${package_managers_found[*]}" + )" + return 0 + ;; + esac +} diff --git a/lib/pip.sh b/lib/pip.sh new file mode 100644 index 000000000..4ba2a6812 --- /dev/null +++ b/lib/pip.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash + +function pip::install_pip_setuptools_wheel() { + # We use the pip wheel bundled within Python's standard library to install our chosen + # pip version, since it's faster than `ensurepip` followed by an upgrade in place. + local bundled_pip_module_path="${1}" + + # TODO: Either make these `local` or move elsewhere as part of the cache invalidation refactoring. + PIP_VERSION=$(get_requirement_version 'pip') + SETUPTOOLS_VERSION=$(get_requirement_version 'setuptools') + WHEEL_VERSION=$(get_requirement_version 'wheel') + meta_set "pip_version" "${PIP_VERSION}" + meta_set "setuptools_version" "${SETUPTOOLS_VERSION}" + meta_set "wheel_version" "${WHEEL_VERSION}" + + puts-step "Installing pip ${PIP_VERSION}, setuptools ${SETUPTOOLS_VERSION} and wheel ${WHEEL_VERSION}" + + /app/.heroku/python/bin/python "${bundled_pip_module_path}" install --quiet --disable-pip-version-check --no-cache-dir \ + "pip==${PIP_VERSION}" "setuptools==${SETUPTOOLS_VERSION}" "wheel==${WHEEL_VERSION}" +} + +function pip::install_dependencies() { + puts-step "Installing requirements with pip" + + # Make select pip config vars set on the Heroku app available to pip. + # TODO: Expose all config vars (after suitable checks are added for unsafe env vars) + # to allow for the env var interpolation feature of requirements files to work. + # + # PIP_EXTRA_INDEX_URL allows for an alternate pypi URL to be used. + if [[ -r "${ENV_DIR}/PIP_EXTRA_INDEX_URL" ]]; then + PIP_EXTRA_INDEX_URL="$(cat "${ENV_DIR}/PIP_EXTRA_INDEX_URL")" + export PIP_EXTRA_INDEX_URL + fi + + # TODO: Deprecate/sunset this missing requirements file fallback. + if [[ -f setup.py && ! -f requirements.txt ]]; then + args=(--editable .) + else + args=(-r requirements.txt) + fi + + set +e + /app/.heroku/python/bin/pip install "${args[@]}" --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir --progress-bar off 2>&1 | tee "$WARNINGS_LOG" | cleanup | indent + local PIP_STATUS="${PIPESTATUS[0]}" + set -e + + show-warnings + + if [[ ! ${PIP_STATUS} -eq 0 ]]; then + meta_set "failure_reason" "pip-install" + return 1 + fi + + # Install test dependencies, for Heroku CI. + if [[ -n "${INSTALL_TEST}" ]]; then + if [[ -f requirements-test.txt ]]; then + puts-step "Installing test dependencies..." + /app/.heroku/python/bin/pip install -r requirements-test.txt --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir 2>&1 | cleanup | indent + fi + fi +} diff --git a/bin/steps/pipenv b/lib/pipenv.sh old mode 100755 new mode 100644 similarity index 60% rename from bin/steps/pipenv rename to lib/pipenv.sh index 6974ae2d5..56ee6b5c2 --- a/bin/steps/pipenv +++ b/lib/pipenv.sh @@ -3,54 +3,46 @@ # export CLINT_FORCE_COLOR=1 # export PIPENV_FORCE_COLOR=1 +function pipenv::install_pipenv() { + # TODO: Either make this `local` or move elsewhere as part of the cache invalidation refactoring. + PIPENV_VERSION=$(get_requirement_version 'pipenv') + meta_set "pipenv_version" "${PIPENV_VERSION}" + + puts-step "Installing Pipenv ${PIPENV_VERSION}" + + # TODO: Install Pipenv into a venv so it isn't leaked into the app environment. + # TODO: Explore viability of making Pipenv only be available during the build, to reduce slug size. + /app/.heroku/python/bin/pip install --quiet --disable-pip-version-check --no-cache-dir "pipenv==${PIPENV_VERSION}" +} + # Previous versions of the buildpack used to cache the checksum of the lockfile to allow # for skipping pipenv install if the lockfile was unchanged. However, this is not always safe # to do (the lockfile can refer to dependencies that can change independently of the lockfile, # for example, when using a local non-editable file dependency), so we no longer ever skip # install, and instead defer to pipenv to determine whether install is actually a no-op. -rm -f .heroku/python/Pipfile.lock.sha256 - -if [[ -f Pipfile ]]; then - pipenv_install_start_time=$(nowms) - meta_set "package_manager" "pipenv" - - # Skip installing dependencies using pip later. - # TODO: Stop leaking this env var into subshells such as post_compile hooks. - export SKIP_PIP_INSTALL=1 - - # Set Pip env vars - # This reads certain environment variables set on the Heroku app config - # and makes them accessible to the pip install process. +function pipenv::install_dependencies() { + # Make select pip config vars set on the Heroku app available to the pip used by Pipenv. + # TODO: Expose all config vars (after suitable checks are added for unsafe env vars). # # PIP_EXTRA_INDEX_URL allows for an alternate pypi URL to be used. - if [[ -r "$ENV_DIR/PIP_EXTRA_INDEX_URL" ]]; then - PIP_EXTRA_INDEX_URL="$(cat "$ENV_DIR/PIP_EXTRA_INDEX_URL")" + if [[ -r "${ENV_DIR}/PIP_EXTRA_INDEX_URL" ]]; then + PIP_EXTRA_INDEX_URL="$(cat "${ENV_DIR}/PIP_EXTRA_INDEX_URL")" export PIP_EXTRA_INDEX_URL fi - PIPENV_VERSION=$(get_requirement_version 'pipenv') - meta_set "pipenv_version" "${PIPENV_VERSION}" - - /app/.heroku/python/bin/pip install --quiet --disable-pip-version-check --no-cache-dir "pipenv==${PIPENV_VERSION}" - # Install the test dependencies, for CI. # TODO: This is currently inconsistent with the non-test path, since it assumes (but doesn't check for) a lockfile. - if [[ -n "$INSTALL_TEST" ]]; then - meta_set "pipenv_has_lockfile" "true" - puts-step "Installing test dependencies" + if [[ -n "${INSTALL_TEST}" ]]; then + puts-step "Installing test dependencies with Pipenv" /app/.heroku/python/bin/pipenv install --dev --system --deploy --extra-pip-args='--src=/app/.heroku/src' 2>&1 | cleanup | indent # Install the dependencies. elif [[ ! -f Pipfile.lock ]]; then - meta_set "pipenv_has_lockfile" "false" - puts-step "Installing dependencies with Pipenv ${PIPENV_VERSION}" + puts-step "Installing dependencies with Pipenv" /app/.heroku/python/bin/pipenv install --system --skip-lock --extra-pip-args='--src=/app/.heroku/src' 2>&1 | indent else - meta_set "pipenv_has_lockfile" "true" - puts-step "Installing dependencies with Pipenv ${PIPENV_VERSION}" + puts-step "Installing dependencies with Pipenv" /app/.heroku/python/bin/pipenv install --system --deploy --extra-pip-args='--src=/app/.heroku/src' 2>&1 | indent fi - - meta_time "dependencies_install_duration" "${pipenv_install_start_time}" -fi +} diff --git a/lib/utils.sh b/lib/utils.sh new file mode 100644 index 000000000..fb7da23a6 --- /dev/null +++ b/lib/utils.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash + +# Python bundles Pip within its standard library, which we can use to install our chosen +# pip version from PyPI, saving us from having to download the usual pip bootstrap script. +function utils::bundled_pip_module_path() { + local build_dir="${1}" + + # We have to use a glob since the bundled wheel filename contains the pip version, which + # differs between Python versions. We also have to handle the case where there are multiple + # matching pip wheels, since in some versions of Python (eg 3.9.0) multiple versions of pip + # were accidentally bundled upstream. Note: This implementation relies upon `nullglob` being + # set, which is the case thanks to the `bin/utils` that was run earlier. + local bundled_pip_wheel_list=("${build_dir}"/.heroku/python/lib/python*/ensurepip/_bundled/pip-*.whl) + local bundled_pip_wheel="${bundled_pip_wheel_list[0]}" + + if [[ -z "${bundled_pip_wheel}" ]]; then + display_error "Error: Failed to locate the bundled pip wheel." + meta_set "failure_reason" "bundled-pip-not-found" + return 1 + fi + + echo "${bundled_pip_wheel}/pip" +} + +function utils::abort_internal_error() { + local message="${1}" + display_error "Internal error: ${message} (line $(caller || true))." + exit 1 +} diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index ad8053abf..f6403c1ac 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -96,10 +96,11 @@ To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-#{DEFAULT_PYTHON_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - -----> Installing test dependencies + -----> Installing Pipenv #{PIPENV_VERSION} + -----> Installing SQLite3 + -----> Installing test dependencies with Pipenv Installing dependencies from Pipfile.lock \\(.+\\)... Installing dependencies from Pipfile.lock \\(.+\\)... - -----> Installing SQLite3 -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running post-compile hook CI=true @@ -115,7 +116,6 @@ PIP_NO_PYTHON_VERSION_WARNING=1 PKG_CONFIG_PATH=/app/.heroku/vendor/lib/pkg-config:/app/.heroku/python/lib/pkg-config: PYTHONUNBUFFERED=1 - SKIP_PIP_INSTALL=1 -----> Inline app detected LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: @@ -152,10 +152,11 @@ To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - -----> Installing test dependencies + -----> Installing Pipenv #{PIPENV_VERSION} + -----> Installing SQLite3 + -----> Installing test dependencies with Pipenv Installing dependencies from Pipfile.lock \\(.+\\)... Installing dependencies from Pipfile.lock \\(.+\\)... - -----> Installing SQLite3 -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running post-compile hook REGEX diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 5351975e0..b57910e60 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -220,7 +220,7 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip - remote: Obtaining file:///tmp/build_.* \\(from -r requirements.txt \\(line 1\\)\\) + remote: Obtaining file:///tmp/build_.* remote: Preparing metadata \\(setup.py\\): started remote: Preparing metadata \\(setup.py\\): finished with status 'done' remote: .+ diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index ccd5dee93..f8b810b3f 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -10,9 +10,10 @@ remote: -----> Using Python version specified in Pipfile.lock remote: -----> Installing python-#{python_version} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing dependencies with Pipenv #{PIPENV_VERSION} - remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 + remote: -----> Installing dependencies with Pipenv + remote: Installing dependencies from Pipfile.lock \\(.+\\)... REGEX end end @@ -49,7 +50,9 @@ remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing dependencies with Pipenv #{PIPENV_VERSION} + remote: -----> Installing Pipenv #{PIPENV_VERSION} + remote: -----> Installing SQLite3 + remote: -----> Installing dependencies with Pipenv remote: The flag --skip-lock has been reintroduced \\(but is not recommended\\). Without remote: the lock resolver it is difficult to manage multiple package indexes, and hash remote: checking is not provided. However it can help manage installs with current @@ -57,7 +60,6 @@ remote: Pipfile.lock not found, creating... .+ remote: Installing dependencies from Pipfile... - remote: -----> Installing SQLite3 REGEX end end @@ -78,9 +80,10 @@ remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing dependencies with Pipenv #{PIPENV_VERSION} - remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 + remote: -----> Installing dependencies with Pipenv + remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Inline app detected remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: @@ -195,9 +198,10 @@ remote: ! remote: -----> Installing python-#{LATEST_PYTHON_3_8} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing dependencies with Pipenv #{PIPENV_VERSION} - remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 + remote: -----> Installing dependencies with Pipenv + remote: Installing dependencies from Pipfile.lock \\(.+\\)... REGEX end end @@ -260,9 +264,10 @@ remote: ! remote: -----> Installing python-3.12.2 remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing dependencies with Pipenv #{PIPENV_VERSION} - remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 + remote: -----> Installing dependencies with Pipenv + remote: Installing dependencies from Pipfile.lock \\(.+\\)... REGEX end end @@ -319,9 +324,10 @@ remote: -----> Using Python version specified in runtime.txt remote: -----> Installing python-#{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing dependencies with Pipenv #{PIPENV_VERSION} - remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 + remote: -----> Installing dependencies with Pipenv + remote: Installing dependencies from Pipfile.lock \\(.+\\)... REGEX end end @@ -340,9 +346,10 @@ remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing dependencies with Pipenv #{PIPENV_VERSION} - remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 + remote: -----> Installing dependencies with Pipenv + remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Discovering process types REGEX end @@ -366,9 +373,10 @@ remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing dependencies with Pipenv #{PIPENV_VERSION} - remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 + remote: -----> Installing dependencies with Pipenv + remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Discovering process types REGEX end @@ -385,9 +393,10 @@ remote: -----> Using Python version specified in Pipfile.lock remote: -----> Installing python-#{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing dependencies with Pipenv #{PIPENV_VERSION} - remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 + remote: -----> Installing dependencies with Pipenv + remote: Installing dependencies from Pipfile.lock \\(.+\\)... REGEX end end @@ -404,7 +413,9 @@ remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing dependencies with Pipenv #{PIPENV_VERSION} + remote: -----> Installing Pipenv #{PIPENV_VERSION} + remote: -----> Installing SQLite3 + remote: -----> Installing dependencies with Pipenv remote: Your Pipfile.lock \\(.+\\) is out of date. Expected: \\(.+\\). remote: .+ remote: ERROR:: Aborting deploy From 54940fd175434907abb7cf35fa9d5f6eca061852 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 24 Sep 2024 17:52:22 +0000 Subject: [PATCH 018/406] Prepare release v257 (#1641) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f0e0c71e1..052e84eb6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v257] - 2024-09-24 + - Moved the SQLite3 install step prior to installing dependencies when using Pipenv. This now matches the behaviour when using pip and allows dependencies to actually use the headers. ([#1640](https://github.com/heroku/heroku-buildpack-python/pull/1640)) - Stopped exposing the `SKIP_PIP_INSTALL` env var to `bin/post_compile` and other subprocesses when using Pipenv. ([#1640](https://github.com/heroku/heroku-buildpack-python/pull/1640)) - Stopped creating `.heroku/python/requirements-{declared,installed}.txt` files when using pip. ([#1640](https://github.com/heroku/heroku-buildpack-python/pull/1640)) @@ -1037,7 +1040,8 @@ Default Python is now latest 2.7.10. Updated Pip and Distribute. - Setuptools updated to v16.0 - Pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v256...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v257...main +[v257]: https://github.com/heroku/heroku-buildpack-python/compare/v256...v257 [v256]: https://github.com/heroku/heroku-buildpack-python/compare/v255...v256 [v255]: https://github.com/heroku/heroku-buildpack-python/compare/v254...v255 [v254]: https://github.com/heroku/heroku-buildpack-python/compare/v253...v254 From 4750e23348fcbba193b3e297f0ba72afbbbe87e1 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 30 Sep 2024 08:39:55 +0000 Subject: [PATCH 019/406] Consistently use lowercase "pip" (#1643) Since the upstream name of the project is all lowercase. See: https://pip.pypa.io https://github.com/pypa/pip/issues/7793 --- CHANGELOG.md | 20 ++++++++++---------- bin/report | 2 +- bin/steps/python | 2 +- lib/utils.sh | 2 +- spec/hatchet/pip_spec.rb | 2 +- 5 files changed, 14 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 052e84eb6..a681a8707 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -538,7 +538,7 @@ ## v164 - 2020-02-20 -- Update requirements.txt builds to use Pip 20.0.2 +- Update requirements.txt builds to use pip 20.0.2 - Download get-pip.py to tmpdir instead of root dir ## v163 - 2019-12-23 @@ -667,7 +667,7 @@ Default to 3.6.5, bugfixes. ## v133 -Fixes for Pip 10 release. +Fixes for pip 10 release. ## v132 @@ -1013,32 +1013,32 @@ Improved compatibility with multi and node.js buildpacks. ## v63 - 2015-10-08 -Updated Pip and Setuptools. +Updated pip and Setuptools. - Setuptools updated to v18.3.2 -- Pip updated to v7.1.2 +- pip updated to v7.1.2 ## v62 - 2015-08-07 -Updated Pip and Setuptools. +Updated pip and Setuptools. - Setuptools updated to v18.1 -- Pip updated to v7.1.0 +- pip updated to v7.1.0 ## v61 - 2015-06-30 -Updated Pip and Setuptools. +Updated pip and Setuptools. - Setuptools updated to v18.0.1 -- Pip updated to v7.0.3 +- pip updated to v7.0.3 ## v60 - 2015-05-27 -Default Python is now latest 2.7.10. Updated Pip and Distribute. +Default Python is now latest 2.7.10. Updated pip and Distribute. - Default Python version is v2.7.10 - Setuptools updated to v16.0 -- Pip updated to v7.0.1 +- pip updated to v7.0.1 [unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v257...main [v257]: https://github.com/heroku/heroku-buildpack-python/compare/v256...v257 diff --git a/bin/report b/bin/report index 5697a7f57..94bfbaae1 100755 --- a/bin/report +++ b/bin/report @@ -25,7 +25,7 @@ CACHE_DIR="${2}" BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) # The build system doesn't source the `export` script before running this script, so we have to do -# so manually (if it exists) so that buildpack Python/Pip can be found (if the build succeeded). +# so manually (if it exists) so that buildpack Python/pip can be found (if the build succeeded). # We have to disable Bash error on undefined variables for now, since not all env vars used in the # export script will be set by default (eg `LIBRARY_PATH`). EXPORT_FILE="${BUILDPACK_DIR}/export" diff --git a/bin/steps/python b/bin/steps/python index 56af2778e..cd1ec3b8f 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -121,7 +121,7 @@ if [[ -f .heroku/python-version ]]; then fi fi -# If using Pip, check if we should reinstall python dependencies given that requirements.txt +# If using pip, check if we should reinstall python dependencies given that requirements.txt # is non-deterministic (not all packages pinned, doesn't handle uninstalls etc). We don't need # to do this when using Pipenv, since it has a lockfile and syncs the packages for us. if [[ -f "${BUILD_DIR}/requirements.txt" ]]; then diff --git a/lib/utils.sh b/lib/utils.sh index fb7da23a6..af0aabb14 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -# Python bundles Pip within its standard library, which we can use to install our chosen +# Python bundles pip within its standard library, which we can use to install our chosen # pip version from PyPI, saving us from having to download the usual pip bootstrap script. function utils::bundled_pip_module_path() { local build_dir="${1}" diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index b57910e60..e79546cf1 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -11,7 +11,7 @@ end end -RSpec.describe 'Pip support' do +RSpec.describe 'pip support' do # TODO: Run this on Heroku-22 too, once it has also migrated to the new build infrastructure. # (Currently the test fails on the old infrastructure due to subtle differences in system PATH elements.) context 'when requirements.txt is unchanged since the last build', stacks: %w[heroku-20 heroku-24] do From 8656b217ee728716c6de16890771a4eb2196982b Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 30 Sep 2024 08:41:26 +0000 Subject: [PATCH 020/406] Remove remnants of the `.heroku/vendor/` directory (#1644) Since the buildpack hasn't vendored anything in it since 2020: https://github.com/heroku/heroku-buildpack-python/pull/1113 --- CHANGELOG.md | 1 + bin/compile | 34 +++++++++------------- spec/hatchet/ci_spec.rb | 40 +++++++++++++------------- spec/hatchet/hooks_spec.rb | 24 ++++++++-------- spec/hatchet/pip_spec.rb | 4 +-- spec/hatchet/pipenv_spec.rb | 4 +-- spec/hatchet/profile_d_scripts_spec.rb | 8 +++--- 7 files changed, 55 insertions(+), 60 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a681a8707..cfe960e83 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Removed remnants of the unused `.heroku/vendor/` directory. ([#1644](https://github.com/heroku/heroku-buildpack-python/pull/1644)) ## [v257] - 2024-09-24 diff --git a/bin/compile b/bin/compile index d598039e9..b850249a1 100755 --- a/bin/compile +++ b/bin/compile @@ -71,20 +71,17 @@ EXPORT_PATH="${BUILDPACK_DIR}/export" GUNICORN_PROFILE_PATH="$BUILD_DIR/.profile.d/python.gunicorn.sh" WEB_CONCURRENCY_PROFILE_PATH="$BUILD_DIR/.profile.d/WEB_CONCURRENCY.sh" -export PATH=/app/.heroku/python/bin:/app/.heroku/vendor/bin:$PATH +export PATH="/app/.heroku/python/bin:${PATH}" # Tell Python to not buffer it's stdin/stdout. export PYTHONUNBUFFERED=1 -# Set the locale to a well-known and expected standard. -export LANG=en_US.UTF-8 -# `~/.heroku/vendor` is an place where the buildpack may stick pre-build binaries for known -# C dependencies. This section configures Python (GCC, more specifically) -# and pip to automatically include these paths when building binaries. -# TODO: Stop adding .heroku/vendor here now that the buildpack no longer vendors anything. -export C_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include:$C_INCLUDE_PATH -export CPLUS_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include:$CPLUS_INCLUDE_PATH -export LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib:$LIBRARY_PATH -export LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib:$LD_LIBRARY_PATH -export PKG_CONFIG_PATH=/app/.heroku/vendor/lib/pkg-config:/app/.heroku/python/lib/pkg-config:$PKG_CONFIG_PATH +# Ensure Python uses a Unicode locale, to prevent the issues described in: +# https://github.com/docker-library/python/pull/570 +export LANG="en_US.UTF-8" +export C_INCLUDE_PATH="/app/.heroku/python/include:${C_INCLUDE_PATH}" +export CPLUS_INCLUDE_PATH="/app/.heroku/python/include:${CPLUS_INCLUDE_PATH}" +export LIBRARY_PATH="/app/.heroku/python/lib:${LIBRARY_PATH}" +export LD_LIBRARY_PATH="/app/.heroku/python/lib:${LD_LIBRARY_PATH}" +export PKG_CONFIG_PATH="/app/.heroku/python/lib/pkg-config:${PKG_CONFIG_PATH}" # Global pip options (https://pip.pypa.io/en/stable/user_guide/#environment-variables). # Disable pip's warnings about EOL Python since we show our own. @@ -174,7 +171,6 @@ if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then # python expects to reside in /app, so set up symlinks # we will not remove these later so subsequent buildpacks can still invoke it ln -nsf "$BUILD_DIR/.heroku/python" /app/.heroku/python - ln -nsf "$BUILD_DIR/.heroku/vendor" /app/.heroku/vendor # Note: .heroku/src is copied in later. fi @@ -246,20 +242,20 @@ meta_time "django_collectstatic_duration" "${collectstatic_start_time}" # Programmatically create .profile.d script for application runtime environment variables. # Set the PATH to include Python / pip / pipenv / etc. -set_env PATH "\$HOME/.heroku/python/bin:\$PATH" +set_env PATH "\${HOME}/.heroku/python/bin:\${PATH}" # Tell Python to run in unbuffered mode. set_env PYTHONUNBUFFERED true # Tell Python where it lives. -set_env PYTHONHOME "\$HOME/.heroku/python" +set_env PYTHONHOME "\${HOME}/.heroku/python" # Set variables for C libraries. -set_env LIBRARY_PATH "\$HOME/.heroku/vendor/lib:\$HOME/.heroku/python/lib:\$LIBRARY_PATH" -set_env LD_LIBRARY_PATH "\$HOME/.heroku/vendor/lib:\$HOME/.heroku/python/lib:\$LD_LIBRARY_PATH" +set_env LIBRARY_PATH "\${HOME}/.heroku/python/lib:\${LIBRARY_PATH}" +set_env LD_LIBRARY_PATH "\${HOME}/.heroku/python/lib:\${LD_LIBRARY_PATH}" # Locale. set_default_env LANG en_US.UTF-8 # The Python hash seed is set to random. set_default_env PYTHONHASHSEED random # Tell Python to look for Python modules in the /app dir. Don't change this. -set_default_env PYTHONPATH "\$HOME" +set_default_env PYTHONPATH "\${HOME}" # Python expects to be in /app, if at runtime, it is not, set # up symlinks… this can occur when the subdir buildpack is used. @@ -267,7 +263,6 @@ cat <>"$PROFILE_PATH" if [[ \$HOME != "/app" ]]; then mkdir -p /app/.heroku ln -nsf "\$HOME/.heroku/python" /app/.heroku/python - ln -nsf "\$HOME/.heroku/vendor" /app/.heroku/vendor fi EOT @@ -291,7 +286,6 @@ source "${BUILDPACK_DIR}/bin/steps/hooks/post_compile" rm -rf "$CACHE_DIR/.heroku/python" rm -rf "$CACHE_DIR/.heroku/python-version" rm -rf "$CACHE_DIR/.heroku/python-stack" -rm -rf "$CACHE_DIR/.heroku/vendor" rm -rf "$CACHE_DIR/.heroku/src" mkdir -p "$CACHE_DIR/.heroku" diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index f6403c1ac..dd2f726b2 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -26,22 +26,22 @@ -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running post-compile hook CI=true - CPLUS_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: - C_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: + CPLUS_INCLUDE_PATH=/app/.heroku/python/include: + C_INCLUDE_PATH=/app/.heroku/python/include: DISABLE_COLLECTSTATIC=1 INSTALL_TEST=1 LANG=en_US.UTF-8 LC_ALL=C.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - PATH=/app/.heroku/python/bin:/app/.heroku/vendor/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + LD_LIBRARY_PATH=/app/.heroku/python/lib: + LIBRARY_PATH=/app/.heroku/python/lib: + PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PIP_NO_PYTHON_VERSION_WARNING=1 - PKG_CONFIG_PATH=/app/.heroku/vendor/lib/pkg-config:/app/.heroku/python/lib/pkg-config: + PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config: PYTHONUNBUFFERED=1 -----> Inline app detected LANG=en_US.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + LD_LIBRARY_PATH=/app/.heroku/python/lib: + LIBRARY_PATH=/app/.heroku/python/lib: PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python @@ -54,8 +54,8 @@ FORWARDED_ALLOW_IPS=\\* GUNICORN_CMD_ARGS=--access-logfile - LANG=en_US.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + LD_LIBRARY_PATH=/app/.heroku/python/lib: + LIBRARY_PATH=/app/.heroku/python/lib: PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python @@ -104,22 +104,22 @@ -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running post-compile hook CI=true - CPLUS_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: - C_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: + CPLUS_INCLUDE_PATH=/app/.heroku/python/include: + C_INCLUDE_PATH=/app/.heroku/python/include: DISABLE_COLLECTSTATIC=1 INSTALL_TEST=1 LANG=en_US.UTF-8 LC_ALL=C.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - PATH=/app/.heroku/python/bin:/app/.heroku/vendor/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + LD_LIBRARY_PATH=/app/.heroku/python/lib: + LIBRARY_PATH=/app/.heroku/python/lib: + PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PIP_NO_PYTHON_VERSION_WARNING=1 - PKG_CONFIG_PATH=/app/.heroku/vendor/lib/pkg-config:/app/.heroku/python/lib/pkg-config: + PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config: PYTHONUNBUFFERED=1 -----> Inline app detected LANG=en_US.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + LD_LIBRARY_PATH=/app/.heroku/python/lib: + LIBRARY_PATH=/app/.heroku/python/lib: PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python @@ -132,8 +132,8 @@ FORWARDED_ALLOW_IPS=\\* GUNICORN_CMD_ARGS=--access-logfile - LANG=en_US.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + LD_LIBRARY_PATH=/app/.heroku/python/lib: + LIBRARY_PATH=/app/.heroku/python/lib: PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python diff --git a/spec/hatchet/hooks_spec.rb b/spec/hatchet/hooks_spec.rb index 6d66adf54..bab91ea03 100644 --- a/spec/hatchet/hooks_spec.rb +++ b/spec/hatchet/hooks_spec.rb @@ -18,16 +18,16 @@ remote: ~ pre_compile ran with env vars: remote: BUILD_DIR=/tmp/build_ remote: CACHE_DIR=/tmp/codon/tmp/cache - remote: C_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: - remote: CPLUS_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: + remote: C_INCLUDE_PATH=/app/.heroku/python/include: + remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include: remote: ENV_DIR=/tmp/... remote: HOME=/app remote: LANG=en_US.UTF-8 - remote: LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - remote: LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - remote: PATH=/app/.heroku/python/bin:/app/.heroku/vendor/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib: + remote: LIBRARY_PATH=/app/.heroku/python/lib: + remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PIP_NO_PYTHON_VERSION_WARNING=1 - remote: PKG_CONFIG_PATH=/app/.heroku/vendor/lib/pkg-config:/app/.heroku/python/lib/pkg-config: + remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config: remote: PWD=/tmp/build_ remote: PYTHONUNBUFFERED=1 remote: SOME_APP_CONFIG_VAR=1 @@ -42,16 +42,16 @@ remote: ~ post_compile ran with env vars: remote: BUILD_DIR=/tmp/build_ remote: CACHE_DIR=/tmp/codon/tmp/cache - remote: C_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: - remote: CPLUS_INCLUDE_PATH=/app/.heroku/vendor/include:/app/.heroku/python/include: + remote: C_INCLUDE_PATH=/app/.heroku/python/include: + remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include: remote: ENV_DIR=/tmp/... remote: HOME=/app remote: LANG=en_US.UTF-8 - remote: LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - remote: LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - remote: PATH=/app/.heroku/python/bin:/app/.heroku/vendor/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib: + remote: LIBRARY_PATH=/app/.heroku/python/lib: + remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PIP_NO_PYTHON_VERSION_WARNING=1 - remote: PKG_CONFIG_PATH=/app/.heroku/vendor/lib/pkg-config:/app/.heroku/python/lib/pkg-config: + remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config: remote: PWD=/tmp/build_ remote: PYTHONUNBUFFERED=1 remote: SOME_APP_CONFIG_VAR=1 diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index e79546cf1..97d17f03d 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -35,8 +35,8 @@ remote: Successfully installed typing-extensions-4.12.2 remote: -----> Inline app detected remote: LANG=en_US.UTF-8 - remote: LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - remote: LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib: + remote: LIBRARY_PATH=/app/.heroku/python/lib: remote: PATH=/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PYTHONHASHSEED=random remote: PYTHONHOME=/app/.heroku/python diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index f8b810b3f..71de067d4 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -86,8 +86,8 @@ remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Inline app detected remote: LANG=en_US.UTF-8 - remote: LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - remote: LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib: + remote: LIBRARY_PATH=/app/.heroku/python/lib: remote: PATH=/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PYTHONHASHSEED=random remote: PYTHONHOME=/app/.heroku/python diff --git a/spec/hatchet/profile_d_scripts_spec.rb b/spec/hatchet/profile_d_scripts_spec.rb index 434d6ccad..d2c49f89b 100644 --- a/spec/hatchet/profile_d_scripts_spec.rb +++ b/spec/hatchet/profile_d_scripts_spec.rb @@ -19,8 +19,8 @@ GUNICORN_CMD_ARGS=--access-logfile - HOME=/app LANG=en_US.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: - LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib: + LD_LIBRARY_PATH=/app/.heroku/python/lib: + LIBRARY_PATH=/app/.heroku/python/lib: PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin PWD=/app PYTHONHASHSEED=random @@ -55,8 +55,8 @@ GUNICORN_CMD_ARGS=this-should-be-preserved HOME=/app LANG=C.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib:/this-should-be-preserved - LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib:/this-should-be-preserved + LD_LIBRARY_PATH=/app/.heroku/python/lib:/this-should-be-preserved + LIBRARY_PATH=/app/.heroku/python/lib:/this-should-be-preserved PATH=/app/.heroku/python/bin:/this-should-be-preserved:/usr/local/bin:/usr/bin:/bin PWD=/app PYTHONHASHSEED=this-should-be-preserved From e3dbc780b3c0c251b2477208722d3b10fc008ede Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 30 Sep 2024 09:11:13 +0000 Subject: [PATCH 021/406] Avoid trailing colon when prepending to optional env vars (#1645) When prepending a value to an env var that uses a delimiter, the delimiter should not be added when the env var wasn't previously set. To achieve this we use the ${var:+VALUE_IF_SET} parameter expansion feature. See: https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html --- CHANGELOG.md | 1 + bin/compile | 14 +++++----- spec/hatchet/ci_spec.rb | 36 +++++++++++++------------- spec/hatchet/hooks_spec.rb | 20 +++++++------- spec/hatchet/pip_spec.rb | 4 +-- spec/hatchet/pipenv_spec.rb | 4 +-- spec/hatchet/profile_d_scripts_spec.rb | 4 +-- 7 files changed, 42 insertions(+), 41 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cfe960e83..1f1610f64 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Stopped adding a trailing `:` to `C_INCLUDE_PATH`, `CPLUS_INCLUDE_PATH`, `LIBRARY_PATH`, `LD_LIBRARY_PATH` and `PKG_CONFIG_PATH`. ([#1645](https://github.com/heroku/heroku-buildpack-python/pull/1645)) - Removed remnants of the unused `.heroku/vendor/` directory. ([#1644](https://github.com/heroku/heroku-buildpack-python/pull/1644)) ## [v257] - 2024-09-24 diff --git a/bin/compile b/bin/compile index b850249a1..243a749c5 100755 --- a/bin/compile +++ b/bin/compile @@ -77,11 +77,11 @@ export PYTHONUNBUFFERED=1 # Ensure Python uses a Unicode locale, to prevent the issues described in: # https://github.com/docker-library/python/pull/570 export LANG="en_US.UTF-8" -export C_INCLUDE_PATH="/app/.heroku/python/include:${C_INCLUDE_PATH}" -export CPLUS_INCLUDE_PATH="/app/.heroku/python/include:${CPLUS_INCLUDE_PATH}" -export LIBRARY_PATH="/app/.heroku/python/lib:${LIBRARY_PATH}" -export LD_LIBRARY_PATH="/app/.heroku/python/lib:${LD_LIBRARY_PATH}" -export PKG_CONFIG_PATH="/app/.heroku/python/lib/pkg-config:${PKG_CONFIG_PATH}" +export C_INCLUDE_PATH="/app/.heroku/python/include${C_INCLUDE_PATH:+:${C_INCLUDE_PATH}}" +export CPLUS_INCLUDE_PATH="/app/.heroku/python/include${CPLUS_INCLUDE_PATH:+:${CPLUS_INCLUDE_PATH}}" +export LIBRARY_PATH="/app/.heroku/python/lib${LIBRARY_PATH:+:${LIBRARY_PATH}}" +export LD_LIBRARY_PATH="/app/.heroku/python/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" +export PKG_CONFIG_PATH="/app/.heroku/python/lib/pkg-config${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}" # Global pip options (https://pip.pypa.io/en/stable/user_guide/#environment-variables). # Disable pip's warnings about EOL Python since we show our own. @@ -248,8 +248,8 @@ set_env PYTHONUNBUFFERED true # Tell Python where it lives. set_env PYTHONHOME "\${HOME}/.heroku/python" # Set variables for C libraries. -set_env LIBRARY_PATH "\${HOME}/.heroku/python/lib:\${LIBRARY_PATH}" -set_env LD_LIBRARY_PATH "\${HOME}/.heroku/python/lib:\${LD_LIBRARY_PATH}" +set_env LIBRARY_PATH "\${HOME}/.heroku/python/lib\${LIBRARY_PATH:+:\${LIBRARY_PATH}}" +set_env LD_LIBRARY_PATH "\${HOME}/.heroku/python/lib\${LD_LIBRARY_PATH:+:\${LD_LIBRARY_PATH}}" # Locale. set_default_env LANG en_US.UTF-8 # The Python hash seed is set to random. diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index dd2f726b2..4a29a6911 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -26,22 +26,22 @@ -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running post-compile hook CI=true - CPLUS_INCLUDE_PATH=/app/.heroku/python/include: - C_INCLUDE_PATH=/app/.heroku/python/include: + CPLUS_INCLUDE_PATH=/app/.heroku/python/include + C_INCLUDE_PATH=/app/.heroku/python/include DISABLE_COLLECTSTATIC=1 INSTALL_TEST=1 LANG=en_US.UTF-8 LC_ALL=C.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/python/lib: - LIBRARY_PATH=/app/.heroku/python/lib: + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PIP_NO_PYTHON_VERSION_WARNING=1 - PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config: + PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 -----> Inline app detected LANG=en_US.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/python/lib: - LIBRARY_PATH=/app/.heroku/python/lib: + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python @@ -54,8 +54,8 @@ FORWARDED_ALLOW_IPS=\\* GUNICORN_CMD_ARGS=--access-logfile - LANG=en_US.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/python/lib: - LIBRARY_PATH=/app/.heroku/python/lib: + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python @@ -104,22 +104,22 @@ -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running post-compile hook CI=true - CPLUS_INCLUDE_PATH=/app/.heroku/python/include: - C_INCLUDE_PATH=/app/.heroku/python/include: + CPLUS_INCLUDE_PATH=/app/.heroku/python/include + C_INCLUDE_PATH=/app/.heroku/python/include DISABLE_COLLECTSTATIC=1 INSTALL_TEST=1 LANG=en_US.UTF-8 LC_ALL=C.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/python/lib: - LIBRARY_PATH=/app/.heroku/python/lib: + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PIP_NO_PYTHON_VERSION_WARNING=1 - PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config: + PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 -----> Inline app detected LANG=en_US.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/python/lib: - LIBRARY_PATH=/app/.heroku/python/lib: + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python @@ -132,8 +132,8 @@ FORWARDED_ALLOW_IPS=\\* GUNICORN_CMD_ARGS=--access-logfile - LANG=en_US.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/python/lib: - LIBRARY_PATH=/app/.heroku/python/lib: + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python diff --git a/spec/hatchet/hooks_spec.rb b/spec/hatchet/hooks_spec.rb index bab91ea03..05f0f927c 100644 --- a/spec/hatchet/hooks_spec.rb +++ b/spec/hatchet/hooks_spec.rb @@ -18,16 +18,16 @@ remote: ~ pre_compile ran with env vars: remote: BUILD_DIR=/tmp/build_ remote: CACHE_DIR=/tmp/codon/tmp/cache - remote: C_INCLUDE_PATH=/app/.heroku/python/include: - remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include: + remote: C_INCLUDE_PATH=/app/.heroku/python/include + remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include remote: ENV_DIR=/tmp/... remote: HOME=/app remote: LANG=en_US.UTF-8 - remote: LD_LIBRARY_PATH=/app/.heroku/python/lib: - remote: LIBRARY_PATH=/app/.heroku/python/lib: + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib + remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PIP_NO_PYTHON_VERSION_WARNING=1 - remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config: + remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config remote: PWD=/tmp/build_ remote: PYTHONUNBUFFERED=1 remote: SOME_APP_CONFIG_VAR=1 @@ -42,16 +42,16 @@ remote: ~ post_compile ran with env vars: remote: BUILD_DIR=/tmp/build_ remote: CACHE_DIR=/tmp/codon/tmp/cache - remote: C_INCLUDE_PATH=/app/.heroku/python/include: - remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include: + remote: C_INCLUDE_PATH=/app/.heroku/python/include + remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include remote: ENV_DIR=/tmp/... remote: HOME=/app remote: LANG=en_US.UTF-8 - remote: LD_LIBRARY_PATH=/app/.heroku/python/lib: - remote: LIBRARY_PATH=/app/.heroku/python/lib: + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib + remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PIP_NO_PYTHON_VERSION_WARNING=1 - remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config: + remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config remote: PWD=/tmp/build_ remote: PYTHONUNBUFFERED=1 remote: SOME_APP_CONFIG_VAR=1 diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 97d17f03d..34f37fae6 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -35,8 +35,8 @@ remote: Successfully installed typing-extensions-4.12.2 remote: -----> Inline app detected remote: LANG=en_US.UTF-8 - remote: LD_LIBRARY_PATH=/app/.heroku/python/lib: - remote: LIBRARY_PATH=/app/.heroku/python/lib: + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib + remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PYTHONHASHSEED=random remote: PYTHONHOME=/app/.heroku/python diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 71de067d4..72342994e 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -86,8 +86,8 @@ remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Inline app detected remote: LANG=en_US.UTF-8 - remote: LD_LIBRARY_PATH=/app/.heroku/python/lib: - remote: LIBRARY_PATH=/app/.heroku/python/lib: + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib + remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PYTHONHASHSEED=random remote: PYTHONHOME=/app/.heroku/python diff --git a/spec/hatchet/profile_d_scripts_spec.rb b/spec/hatchet/profile_d_scripts_spec.rb index d2c49f89b..0e36a1efb 100644 --- a/spec/hatchet/profile_d_scripts_spec.rb +++ b/spec/hatchet/profile_d_scripts_spec.rb @@ -19,8 +19,8 @@ GUNICORN_CMD_ARGS=--access-logfile - HOME=/app LANG=en_US.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/python/lib: - LIBRARY_PATH=/app/.heroku/python/lib: + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin PWD=/app PYTHONHASHSEED=random From 2a23f41f5bf835fe7279c66c12ca53b441f7ea3c Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 30 Sep 2024 15:00:29 +0000 Subject: [PATCH 022/406] Remove deadcode from vendored buildpack-stdlib (#1647) The buildpack isn't using these logging helpers (they don't match the existing logging style, and also don't properly handling multi-line logs. GUS-W-16864551. --- vendor/buildpack-stdlib_v8.sh | 56 ----------------------------------- 1 file changed, 56 deletions(-) diff --git a/vendor/buildpack-stdlib_v8.sh b/vendor/buildpack-stdlib_v8.sh index b06e729c1..be0e93d15 100755 --- a/vendor/buildpack-stdlib_v8.sh +++ b/vendor/buildpack-stdlib_v8.sh @@ -3,62 +3,6 @@ # Based on: # https://raw.githubusercontent.com/heroku/buildpack-stdlib/v8/stdlib.sh -# Standard Output -# --------------- - -# Buildpack Steps. -puts_step() { - if [[ "$*" == "-" ]]; then - read -r output - else - output=$* - fi - echo -e "\\e[1m\\e[36m=== $output\\e[0m" - unset output -} - -# Buildpack Error. -puts_error() { - if [[ "$*" == "-" ]]; then - read -r output - else - output=$* - fi - echo -e "\\e[1m\\e[31m=!= $output\\e[0m" -} - -# Buildpack Warning. -puts_warn() { - if [[ "$*" == "-" ]]; then - read -r output - else - output=$* - fi - echo -e "\\e[1m\\e[33m=!= $output\\e[0m" -} - -# Is verbose set? -is_verbose() { - if [[ -n $BUILDPACK_VERBOSE ]]; then - return 0 - else - return 1 - fi -} - -# Buildpack Verbose. -puts_verbose() { - if is_verbose; then - if [[ "$*" == "-" ]]; then - read -r output - else - output=$* - fi - echo "$output" - unset output - fi -} - # Buildpack Utilities # ------------------- From 92c9328f3128246394b9552f109f8706b78130ee Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 30 Sep 2024 15:01:32 +0000 Subject: [PATCH 023/406] Fix calls to `utils::abort_internal_error` (#1648) Since the function was renamed as part of namespacing the functions under `lib/`. These codepaths are only ever hit in the case of a buildpack ~enum inconsistency bug (they are the equivalent of Rust's `unreachable!`), so it's not possible to have tests for them. --- bin/compile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/bin/compile b/bin/compile index 243a749c5..82e470c72 100755 --- a/bin/compile +++ b/bin/compile @@ -192,7 +192,7 @@ case "${PACKAGE_MANAGER}" in pipenv::install_pipenv ;; *) - abort_internal_error "Unhandled package manager" + utils::abort_internal_error "Unhandled package manager" ;; esac meta_time "package_manager_install_duration" "${package_manager_install_start_time}" @@ -215,7 +215,7 @@ case "${PACKAGE_MANAGER}" in pipenv::install_dependencies ;; *) - abort_internal_error "Unhandled package manager" + utils::abort_internal_error "Unhandled package manager" ;; esac meta_time "dependencies_install_duration" "${dependencies_install_start_time}" From 4a264fc30f6b119fcff17b56b8bcc6e59d90249d Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 30 Sep 2024 15:11:05 +0000 Subject: [PATCH 024/406] Enable Bash exit on undefined variables (`set -u`) (#1646) Since it's a best practice that helps make bugs more visible. Aside from the `DEBUG_COLLECTSTATIC` change, all of the others only affect buildpack internals. GUS-W-16864420. --- CHANGELOG.md | 1 + bin/compile | 7 +++++-- bin/release | 4 ---- bin/report | 4 ---- bin/steps/collectstatic | 4 ++-- bin/steps/nltk | 2 +- bin/steps/python | 2 +- bin/steps/sqlite3 | 2 +- lib/pip.sh | 2 +- lib/pipenv.sh | 2 +- vendor/buildpack-stdlib_v8.sh | 2 +- 11 files changed, 14 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f1610f64..7ce2f8e02 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Fixed Django collectstatic debug output being shown if `DEBUG_COLLECTSTATIC` was set to `0` or the empty string. ([#1646](https://github.com/heroku/heroku-buildpack-python/pull/1646)) - Stopped adding a trailing `:` to `C_INCLUDE_PATH`, `CPLUS_INCLUDE_PATH`, `LIBRARY_PATH`, `LD_LIBRARY_PATH` and `PKG_CONFIG_PATH`. ([#1645](https://github.com/heroku/heroku-buildpack-python/pull/1645)) - Removed remnants of the unused `.heroku/vendor/` directory. ([#1644](https://github.com/heroku/heroku-buildpack-python/pull/1644)) diff --git a/bin/compile b/bin/compile index 82e470c72..f4d1d61e4 100755 --- a/bin/compile +++ b/bin/compile @@ -2,9 +2,12 @@ # Usage: bin/compile # See: https://devcenter.heroku.com/articles/buildpack-api -set -eo pipefail +set -euo pipefail -[[ -n "$BUILDPACK_XTRACE" ]] && set -o xtrace +# Note: This can't be enabled via app config vars, since at this point they haven't been loaded from ENV_DIR. +if [[ "${BUILDPACK_XTRACE:-0}" == "1" ]]; then + set -o xtrace +fi BUILD_DIR="${1}" CACHE_DIR="${2}" diff --git a/bin/release b/bin/release index 30930b41f..f3200610b 100755 --- a/bin/release +++ b/bin/release @@ -12,12 +12,8 @@ BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) # Unfortunately the build system doesn't source the `export` script before # running `bin/release`, so we have to do so manually to ensure the buildpack # Python is used by `is_module_available` instead of system Python. -# We also have to disable Bash error on undefined variables, since not all env -# vars used in the export script will be set by default (eg `LIBRARY_PATH`). -set +u # shellcheck source=/dev/null source "${BUILDPACK_DIR}/export" -set -u source "${BUILDPACK_DIR}/bin/utils" diff --git a/bin/report b/bin/report index 94bfbaae1..80a1d7c7e 100755 --- a/bin/report +++ b/bin/report @@ -26,14 +26,10 @@ BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) # The build system doesn't source the `export` script before running this script, so we have to do # so manually (if it exists) so that buildpack Python/pip can be found (if the build succeeded). -# We have to disable Bash error on undefined variables for now, since not all env vars used in the -# export script will be set by default (eg `LIBRARY_PATH`). EXPORT_FILE="${BUILDPACK_DIR}/export" if [[ -f "${EXPORT_FILE}" ]]; then - set +u # shellcheck source=/dev/null source "${EXPORT_FILE}" - set -u fi source "${BUILDPACK_DIR}/lib/metadata.sh" diff --git a/bin/steps/collectstatic b/bin/steps/collectstatic index 41b33fe02..555eff1e8 100755 --- a/bin/steps/collectstatic +++ b/bin/steps/collectstatic @@ -12,7 +12,7 @@ # This script is run in a subshell via sub_env so doesn't inherit the options/vars/utils from `bin/compile`. # TODO: Stop running this script in a subshell. -set -eo pipefail +set -euo pipefail BUILDPACK_DIR=$(cd "$(dirname "$(dirname "$(dirname "${BASH_SOURCE[0]}")")")" && pwd) source "${BUILDPACK_DIR}/bin/utils" @@ -92,7 +92,7 @@ echo echo " https://devcenter.heroku.com/articles/django-assets" # Additionally, dump out the environment, if debug mode is on. -if [[ -n "$DEBUG_COLLECTSTATIC" ]]; then +if [[ "${DEBUG_COLLECTSTATIC:-0}" == "1" ]]; then echo echo "****** Collectstatic environment variables:" echo diff --git a/bin/steps/nltk b/bin/steps/nltk index f5f939efa..680bd2102 100755 --- a/bin/steps/nltk +++ b/bin/steps/nltk @@ -2,7 +2,7 @@ # This script is run in a subshell via sub_env so doesn't inherit the options/vars/utils from `bin/compile`. # TODO: Stop running this script in a subshell. -set -eo pipefail +set -euo pipefail BUILDPACK_DIR=$(cd "$(dirname "$(dirname "$(dirname "${BASH_SOURCE[0]}")")")" && pwd) source "${BUILDPACK_DIR}/bin/utils" diff --git a/bin/steps/python b/bin/steps/python index cd1ec3b8f..05c4f302d 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -144,7 +144,7 @@ if [[ -f "${BUILD_DIR}/requirements.txt" ]]; then fi fi -if [[ -n "${SKIP_INSTALL}" ]]; then +if [[ "${SKIP_INSTALL:-0}" == "1" ]]; then puts-step "Using cached install of ${PYTHON_VERSION}" else puts-step "Installing ${PYTHON_VERSION}" diff --git a/bin/steps/sqlite3 b/bin/steps/sqlite3 index c7200cebc..826e45f74 100755 --- a/bin/steps/sqlite3 +++ b/bin/steps/sqlite3 @@ -5,7 +5,7 @@ sqlite3_install() { HEROKU_PYTHON_DIR="$1" - HEADERS_ONLY="$3" + HEADERS_ONLY="${3:-}" mkdir -p "$HEROKU_PYTHON_DIR" diff --git a/lib/pip.sh b/lib/pip.sh index 4ba2a6812..70839364e 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -52,7 +52,7 @@ function pip::install_dependencies() { fi # Install test dependencies, for Heroku CI. - if [[ -n "${INSTALL_TEST}" ]]; then + if [[ "${INSTALL_TEST:-0}" == "1" ]]; then if [[ -f requirements-test.txt ]]; then puts-step "Installing test dependencies..." /app/.heroku/python/bin/pip install -r requirements-test.txt --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir 2>&1 | cleanup | indent diff --git a/lib/pipenv.sh b/lib/pipenv.sh index 56ee6b5c2..8d6578430 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -32,7 +32,7 @@ function pipenv::install_dependencies() { # Install the test dependencies, for CI. # TODO: This is currently inconsistent with the non-test path, since it assumes (but doesn't check for) a lockfile. - if [[ -n "${INSTALL_TEST}" ]]; then + if [[ "${INSTALL_TEST:-0}" == "1" ]]; then puts-step "Installing test dependencies with Pipenv" /app/.heroku/python/bin/pipenv install --dev --system --deploy --extra-pip-args='--src=/app/.heroku/src' 2>&1 | cleanup | indent diff --git a/vendor/buildpack-stdlib_v8.sh b/vendor/buildpack-stdlib_v8.sh index be0e93d15..824f9341c 100755 --- a/vendor/buildpack-stdlib_v8.sh +++ b/vendor/buildpack-stdlib_v8.sh @@ -65,7 +65,7 @@ sub_env() { ( # TODO: Fix https://github.com/heroku/buildpack-stdlib/issues/37 # shellcheck disable=SC2153 - export_env "$ENV_DIR" "$WHITELIST" "$BLACKLIST" + export_env "$ENV_DIR" "${WHITELIST:-}" "${BLACKLIST:-}" "$@" ) From 92661b8312db03ef7268267d3156ae8ae4d46605 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 30 Sep 2024 15:37:03 +0000 Subject: [PATCH 025/406] Further `make run` improvements (#1649) * Stops using the `-it` options, since whilst interactive mode allows for instant termination of the container using Ctrl/CMD+C during development, it interferes with running the command via `watchexec`. Similarly, enabling the TTY leads to tools not behaving the same as they will on the actual Heroku build system (eg using colours in auto mode), so that has been disabled too, to make this development workflow more representative. Doing so also means the GitHub Actions workaround for TTY is no longer required. * Sanitises the env vars to more closely match those allow-listed in Cytokine. This prevents vars like `CNB_USER_ID` and `CNB_STACK_ID` from leaking through from the new combined Heroku+CNB base images. * If an inline buildpack is detected in the test fixture, it is now run too - matching what the equivalent Hatchet test would be configured to do. GUS-W-16866418. --- .github/workflows/ci.yml | 5 ----- Makefile | 12 ++++++++---- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 56417d095..a319a1b86 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -65,11 +65,6 @@ jobs: container-test: runs-on: ubuntu-24.04 - defaults: - run: - # Work around lack of TTY causing errors when using `docker run -it`: - # https://github.com/actions/runner/issues/241 - shell: 'script -q -e -c "bash -eo pipefail {0}"' steps: - name: Checkout uses: actions/checkout@v4 diff --git a/Makefile b/Makefile index 009bffae9..43a6113b3 100644 --- a/Makefile +++ b/Makefile @@ -23,16 +23,20 @@ format: run: @echo "Running buildpack using: STACK=$(STACK) FIXTURE=$(FIXTURE)" - @docker run --rm -it -v $(PWD):/src:ro --tmpfs /app -e "HOME=/app" -e "STACK=$(STACK)" "$(STACK_IMAGE_TAG)" \ - bash -eo pipefail -c '\ + @docker run --rm -v $(PWD):/src:ro --tmpfs /app -e "HOME=/app" -e "STACK=$(STACK)" "$(STACK_IMAGE_TAG)" \ + bash -euo pipefail -c '\ mkdir /tmp/buildpack /tmp/build /tmp/cache /tmp/env; \ cp -r /src/{bin,lib,requirements,vendor} /tmp/buildpack; \ cp -rT /src/$(FIXTURE) /tmp/build; \ cd /tmp/buildpack; \ + unset $$(printenv | cut -d '=' -f 1 | grep -vE "^(HOME|LANG|PATH|STACK)$$"); \ echo -e "\n~ Detect:" && ./bin/detect /tmp/build; \ - echo -e "\n~ Compile:" && { ./bin/compile /tmp/build /tmp/cache /tmp/env || SKIP_RELEASE=1; }; \ + echo -e "\n~ Compile:" && { ./bin/compile /tmp/build /tmp/cache /tmp/env || COMPILE_FAILED=1; }; \ echo -e "\n~ Report:" && ./bin/report /tmp/build /tmp/cache /tmp/env; \ - [[ -n "$${SKIP_RELEASE}" ]] || { echo -e "\n~ Release:" && ./bin/release /tmp/build && echo -e "\nBuild successful!"; }; \ + [[ "$${COMPILE_FAILED:-}" == "1" ]] && exit 0; \ + [[ -f /tmp/build/bin/compile ]] && { echo -e "\n~ Compile (Inline Buildpack):" && (source ./export && /tmp/build/bin/compile /tmp/build /tmp/cache /tmp/env); }; \ + echo -e "\n~ Release:" && ./bin/release /tmp/build; \ + echo -e "\nBuild successful!"; \ ' @echo From b0d0cb596dd6cd1f9d909220fd462d15977946c1 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 1 Oct 2024 07:43:43 +0000 Subject: [PATCH 026/406] Add support for Python 3.12.7 (#1650) Release announcement: https://www.python.org/downloads/release/python-3127/ Changelog: https://docs.python.org/3.12/whatsnew/changelog.html#python-3-12-7-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/11119534362 GUS-W-14846794. --- CHANGELOG.md | 2 ++ README.md | 4 ++-- bin/default_pythons | 2 +- spec/fixtures/pipenv_and_runtime_txt/runtime.txt | 2 +- spec/fixtures/python_3.12/runtime.txt | 2 +- spec/fixtures/runtime_txt_only/runtime.txt | 2 +- spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt | 2 +- spec/hatchet/python_version_spec.rb | 2 +- spec/spec_helper.rb | 2 +- 9 files changed, 11 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ce2f8e02..7b97e3651 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## [Unreleased] +- Added support for Python 3.12.7. ([#1650](https://github.com/heroku/heroku-buildpack-python/pull/1650)) +- Changed the default Python version for new apps from 3.12.6 to 3.12.7. ([#1650](https://github.com/heroku/heroku-buildpack-python/pull/1650)) - Fixed Django collectstatic debug output being shown if `DEBUG_COLLECTSTATIC` was set to `0` or the empty string. ([#1646](https://github.com/heroku/heroku-buildpack-python/pull/1646)) - Stopped adding a trailing `:` to `C_INCLUDE_PATH`, `CPLUS_INCLUDE_PATH`, `LIBRARY_PATH`, `LD_LIBRARY_PATH` and `PKG_CONFIG_PATH`. ([#1645](https://github.com/heroku/heroku-buildpack-python/pull/1645)) - Removed remnants of the unused `.heroku/vendor/` directory. ([#1644](https://github.com/heroku/heroku-buildpack-python/pull/1644)) diff --git a/README.md b/README.md index 82b93df63..d811f09eb 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ A `requirements.txt` must be present at the root of your application's repositor To specify your python version, you also need a `runtime.txt` file - unless you are using the default Python runtime version. -Current default Python Runtime: Python 3.12.6 +Current default Python Runtime: Python 3.12.7 Alternatively, you can provide a `setup.py` file, or a `Pipfile`. Using `pipenv` will generate `runtime.txt` at build time if one of the field `python_version` or `python_full_version` is specified in the `requires` section of your `Pipfile`. @@ -60,7 +60,7 @@ Specify a Python Runtime Supported runtime options include: -- `python-3.12.6` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) +- `python-3.12.7` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) - `python-3.11.10` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) - `python-3.10.15` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) - `python-3.9.20` on Heroku-20 and Heroku-22 only diff --git a/bin/default_pythons b/bin/default_pythons index 30bd0b34c..51e9cc247 100755 --- a/bin/default_pythons +++ b/bin/default_pythons @@ -5,7 +5,7 @@ # the env vars to subprocesses. # shellcheck disable=2034 -LATEST_312="python-3.12.6" +LATEST_312="python-3.12.7" LATEST_311="python-3.11.10" LATEST_310="python-3.10.15" LATEST_39="python-3.9.20" diff --git a/spec/fixtures/pipenv_and_runtime_txt/runtime.txt b/spec/fixtures/pipenv_and_runtime_txt/runtime.txt index 32bcba665..32905d6e0 100644 --- a/spec/fixtures/pipenv_and_runtime_txt/runtime.txt +++ b/spec/fixtures/pipenv_and_runtime_txt/runtime.txt @@ -1 +1 @@ -python-3.12.6 +python-3.12.7 diff --git a/spec/fixtures/python_3.12/runtime.txt b/spec/fixtures/python_3.12/runtime.txt index 32bcba665..32905d6e0 100644 --- a/spec/fixtures/python_3.12/runtime.txt +++ b/spec/fixtures/python_3.12/runtime.txt @@ -1 +1 @@ -python-3.12.6 +python-3.12.7 diff --git a/spec/fixtures/runtime_txt_only/runtime.txt b/spec/fixtures/runtime_txt_only/runtime.txt index 32bcba665..32905d6e0 100644 --- a/spec/fixtures/runtime_txt_only/runtime.txt +++ b/spec/fixtures/runtime_txt_only/runtime.txt @@ -1 +1 @@ -python-3.12.6 +python-3.12.7 diff --git a/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt b/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt index b7e93bdc0..05a24a304 100644 --- a/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt +++ b/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt @@ -1,3 +1,3 @@ - python-3.12.6 + python-3.12.7 \ No newline at end of file diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index f13c148ac..14d1ffdc6 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -206,7 +206,7 @@ include_examples 'builds with the requested Python version', LATEST_PYTHON_3_11 end - context 'when runtime.txt contains python-3.12.6' do + context 'when runtime.txt contains python-3.12.7' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.12') } include_examples 'builds with the requested Python version', LATEST_PYTHON_3_12 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 5f761bf3b..cfbb95da1 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -13,7 +13,7 @@ LATEST_PYTHON_3_9 = '3.9.20' LATEST_PYTHON_3_10 = '3.10.15' LATEST_PYTHON_3_11 = '3.11.10' -LATEST_PYTHON_3_12 = '3.12.6' +LATEST_PYTHON_3_12 = '3.12.7' DEFAULT_PYTHON_VERSION = LATEST_PYTHON_3_12 # The requirement versions are effectively buildpack constants, however, we want From 39abf4fe22f009fcb8cb1435bc76287564bb6698 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 1 Oct 2024 08:22:15 +0000 Subject: [PATCH 027/406] Prepare release v258 (#1651) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b97e3651..4a20a17f6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v258] - 2024-10-01 + - Added support for Python 3.12.7. ([#1650](https://github.com/heroku/heroku-buildpack-python/pull/1650)) - Changed the default Python version for new apps from 3.12.6 to 3.12.7. ([#1650](https://github.com/heroku/heroku-buildpack-python/pull/1650)) - Fixed Django collectstatic debug output being shown if `DEBUG_COLLECTSTATIC` was set to `0` or the empty string. ([#1646](https://github.com/heroku/heroku-buildpack-python/pull/1646)) @@ -1045,7 +1048,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v257...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v258...main +[v258]: https://github.com/heroku/heroku-buildpack-python/compare/v257...v258 [v257]: https://github.com/heroku/heroku-buildpack-python/compare/v256...v257 [v256]: https://github.com/heroku/heroku-buildpack-python/compare/v255...v256 [v255]: https://github.com/heroku/heroku-buildpack-python/compare/v254...v255 From 2b04d805ddbcfcd7e35745d9f6b73496eff3dcc8 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 1 Oct 2024 15:25:41 +0000 Subject: [PATCH 028/406] Bump rubocop-rspec from 3.0.5 to 3.1.0 in the ruby-dependencies group (#1653) Bumps the ruby-dependencies group with 1 update: [rubocop-rspec](https://github.com/rubocop/rubocop-rspec). Updates `rubocop-rspec` from 3.0.5 to 3.1.0 - [Release notes](https://github.com/rubocop/rubocop-rspec/releases) - [Changelog](https://github.com/rubocop/rubocop-rspec/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop-rspec/compare/v3.0.5...v3.1.0) --- updated-dependencies: - dependency-name: rubocop-rspec dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Gemfile.lock b/Gemfile.lock index 7b8aeedce..126b0f21a 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -59,7 +59,7 @@ GEM unicode-display_width (>= 2.4.0, < 3.0) rubocop-ast (1.32.3) parser (>= 3.3.1.0) - rubocop-rspec (3.0.5) + rubocop-rspec (3.1.0) rubocop (~> 1.61) ruby-progressbar (1.13.0) thor (1.3.2) From 02f8ec0f5827275416911af03e949387a3c3ee02 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 3 Oct 2024 14:44:05 +0000 Subject: [PATCH 029/406] Enable remaining optional Shellcheck rules (#1654) In #1596 a number of optional Shellcheck rules were enabled. However, there were some which were deferred due to the number of fixes required, and so were left disabled via the `.shellcheckrc` file. In order that we can get the benefit of these rules for new code, I've removed the global disabling in favour of per file or per line `disable` directives (and in some cases, fixing outright). These can then be fixed piecemeal as refactorings occur later. Of note, one of these optional Shellcheck rules (SC2311) would have saved me a fair amount of debugging time earlier today in the new Python version handling implementation. GUS-W-16898648. --- .shellcheckrc | 13 ------------- bin/compile | 4 ++++ bin/detect | 2 +- bin/release | 1 + bin/report | 2 ++ bin/steps/collectstatic | 5 +++-- bin/steps/nltk | 7 ++++--- bin/steps/pipenv-python-version | 2 ++ bin/steps/python | 5 +++++ bin/steps/sqlite3 | 4 ++++ bin/utils | 9 +++------ bin/warnings | 3 ++- builds/build_python_runtime.sh | 5 +++-- etc/publish.sh | 1 + lib/kvstore.sh | 5 +++++ lib/metadata.sh | 8 ++++++-- lib/output.sh | 4 ++++ lib/package_manager.sh | 4 ++++ lib/pip.sh | 8 +++++++- lib/pipenv.sh | 5 +++++ lib/utils.sh | 4 ++++ vendor/buildpack-stdlib_v8.sh | 5 ++++- 22 files changed, 74 insertions(+), 32 deletions(-) diff --git a/.shellcheckrc b/.shellcheckrc index 44e54caa5..da9f6a163 100644 --- a/.shellcheckrc +++ b/.shellcheckrc @@ -1,15 +1,2 @@ # Enable all checks, including the optional ones that are off by default. enable=all - -# TODO: Triage and potentially enable more of these optional checks. - -# SC2154 (warning): var is referenced but not assigned. -disable=SC2154 -# SC2250 (style): Prefer putting braces around variable references even when not strictly required. -disable=SC2250 -# SC2310 (info): This function is invoked in an 'if' condition so set -e will be disabled. -disable=SC2310 -# SC2311 (info): Bash implicitly disabled set -e for this function invocation because it's inside a command substitution. -disable=SC2311 -# SC2312 (info): Consider invoking this command separately to avoid masking its return value -disable=SC2312 diff --git a/bin/compile b/bin/compile index f4d1d61e4..e9b0eb3eb 100755 --- a/bin/compile +++ b/bin/compile @@ -1,6 +1,7 @@ #!/usr/bin/env bash # Usage: bin/compile # See: https://devcenter.heroku.com/articles/buildpack-api +# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. set -euo pipefail @@ -131,6 +132,7 @@ fi if [[ -f "$CACHE_DIR/.heroku/python-stack" ]]; then CACHED_PYTHON_STACK=$(cat "$CACHE_DIR/.heroku/python-stack") else + # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. CACHED_PYTHON_STACK=$STACK fi @@ -170,6 +172,7 @@ mkdir -p /app/.heroku/src # symlinks to emulate that we are operating in `/app` during the build process. # This is (hopefully obviously) because apps end up running from `/app` in production. # Realpath is used to support use-cases where one of the locations is a symlink to the other. +# shellcheck disable=SC2312 # TODO: Invoke this command separately to avoid masking its return value. if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then # python expects to reside in /app, so set up symlinks # we will not remove these later so subsequent buildpacks can still invoke it @@ -231,6 +234,7 @@ meta_time "nltk_downloader_duration" "${nltk_downloader_start_time}" # Support for editable installations. # In CI, $BUILD_DIR is /app. # Realpath is used to support use-cases where one of the locations is a symlink to the other. +# shellcheck disable=SC2312 # TODO: Invoke this command separately to avoid masking its return value. if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then rm -rf "$BUILD_DIR/.heroku/src" deep-cp /app/.heroku/src "$BUILD_DIR/.heroku/src" diff --git a/bin/detect b/bin/detect index 69345eb78..949c24dca 100755 --- a/bin/detect +++ b/bin/detect @@ -54,7 +54,7 @@ source code. Currently the root directory of your app contains: -$(ls -1 --indicator-style=slash "${BUILD_DIR}") +$(ls -1 --indicator-style=slash "${BUILD_DIR}" || true) If your app already has a package manager file, check that it: diff --git a/bin/release b/bin/release index f3200610b..0e8058d1f 100755 --- a/bin/release +++ b/bin/release @@ -17,6 +17,7 @@ source "${BUILDPACK_DIR}/export" source "${BUILDPACK_DIR}/bin/utils" +# shellcheck disable=SC2310 # TODO: This function is invoked in an '&&' condition so set -e will be disabled. if [[ -f "${BUILD_DIR}/manage.py" ]] && is_module_available 'django' && is_module_available 'psycopg2'; then cat <<-'EOF' --- diff --git a/bin/report b/bin/report index 80a1d7c7e..b45378672 100755 --- a/bin/report +++ b/bin/report @@ -91,10 +91,12 @@ ALL_OTHER_FIELDS=( ) for field in "${STRING_FIELDS[@]}"; do + # shellcheck disable=SC2312 # TODO: Invoke this command separately to avoid masking its return value. kv_pair_string "${field}" "$(meta_get "${field}")" done for field in "${ALL_OTHER_FIELDS[@]}"; do + # shellcheck disable=SC2312 # TODO: Invoke this command separately to avoid masking its return value. kv_pair "${field}" "$(meta_get "${field}")" done diff --git a/bin/steps/collectstatic b/bin/steps/collectstatic index 555eff1e8..751582c19 100755 --- a/bin/steps/collectstatic +++ b/bin/steps/collectstatic @@ -1,4 +1,5 @@ #!/usr/bin/env bash +# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. # Django Collectstatic runner. If you have Django installed, collectstatic will # automatically be executed as part of the build process. If collectstatic @@ -18,8 +19,7 @@ source "${BUILDPACK_DIR}/bin/utils" # Required for `meta_set`. source "${BUILDPACK_DIR}/lib/metadata.sh" -# shellcheck disable=SC2153 -meta_init "${CACHE_DIR}" "python" +meta_init "${CACHE_DIR:?}" "python" if [[ -f .heroku/collectstatic_disabled ]]; then puts-step "Skipping Django collectstatic since the file '.heroku/collectstatic_disabled' exists." @@ -35,6 +35,7 @@ if [[ "${DISABLE_COLLECTSTATIC:-0}" != "0" ]]; then fi # Ensure that Django is actually installed. +# shellcheck disable=SC2310 # TODO: This function is invoked in an 'if' condition so set -e will be disabled. if ! is_module_available 'django'; then exit 0 fi diff --git a/bin/steps/nltk b/bin/steps/nltk index 680bd2102..c849285ac 100755 --- a/bin/steps/nltk +++ b/bin/steps/nltk @@ -1,4 +1,5 @@ #!/usr/bin/env bash +# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. # This script is run in a subshell via sub_env so doesn't inherit the options/vars/utils from `bin/compile`. # TODO: Stop running this script in a subshell. @@ -8,14 +9,14 @@ source "${BUILDPACK_DIR}/bin/utils" # Required for `meta_set`. source "${BUILDPACK_DIR}/lib/metadata.sh" -# shellcheck disable=SC2153 -meta_init "${CACHE_DIR}" "python" +meta_init "${CACHE_DIR:?}" "python" # These are required by `set_env`. -PROFILE_PATH="${BUILD_DIR}/.profile.d/python.sh" +PROFILE_PATH="${BUILD_DIR:?}/.profile.d/python.sh" EXPORT_PATH="${BUILDPACK_DIR}/export" # Check that nltk was installed by pip, otherwise obviously not needed +# shellcheck disable=SC2310 # TODO: This function is invoked in an 'if' condition so set -e will be disabled. if is_module_available 'nltk'; then puts-step "Downloading NLTK corpora..." diff --git a/bin/steps/pipenv-python-version b/bin/steps/pipenv-python-version index 6b4bd46e5..2a76c56a2 100755 --- a/bin/steps/pipenv-python-version +++ b/bin/steps/pipenv-python-version @@ -1,4 +1,6 @@ #!/usr/bin/env bash +# shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. +# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. # TODO: Move this to lib/ as part of the refactoring for .python-version support. diff --git a/bin/steps/python b/bin/steps/python index 05c4f302d..362e9e42e 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -1,4 +1,8 @@ #!/usr/bin/env bash +# shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. +# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. + +set -euo pipefail PYTHON_VERSION=$(cat runtime.txt) # Remove leading and trailing whitespace. Note: This implementation relies upon @@ -113,6 +117,7 @@ if [[ "$STACK" != "$CACHED_PYTHON_STACK" ]]; then fi if [[ -f .heroku/python-version ]]; then + # shellcheck disable=SC2312 # TODO: Invoke this command separately to avoid masking its return value. if [[ ! "$(cat .heroku/python-version)" == "$PYTHON_VERSION" ]]; then puts-step "Python version has changed from $(cat .heroku/python-version) to ${PYTHON_VERSION}, clearing cache" rm -rf .heroku/python diff --git a/bin/steps/sqlite3 b/bin/steps/sqlite3 index 826e45f74..a174fa2b2 100755 --- a/bin/steps/sqlite3 +++ b/bin/steps/sqlite3 @@ -1,4 +1,5 @@ #!/usr/bin/env bash +# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. # TODO: Remove this entirely since the Python stdlib now includes modern sqlite support, # and the APT buildpack should be used if an app needs the sqlite CLI/headers. @@ -68,6 +69,8 @@ buildpack_sqlite3_install() { # the conditional disables `set -e` inside the called function: # https://stackoverflow.com/q/19789102 # ...plus whoever wrote this forgot the `exit 1` in the `else` anyway. + # shellcheck disable=SC2310 # TODO: This function is invoked in an 'if' condition so set -e will be disabled. + # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. if sqlite3_install "$BUILD_DIR/.heroku/python"; then # mcount "success.python.sqlite3" : @@ -76,5 +79,6 @@ buildpack_sqlite3_install() { # mcount "failure.python.sqlite3" fi + # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. mkdir -p "$CACHE_DIR/.heroku/" } diff --git a/bin/utils b/bin/utils index 09961257f..45ab374d3 100755 --- a/bin/utils +++ b/bin/utils @@ -1,17 +1,14 @@ #!/usr/bin/env bash +# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. # Be careful about moving these to bin/compile, since this utils script is sourced # directly by other scripts run via subshells, and not only from bin/compile. shopt -s extglob shopt -s nullglob -source "${BUILDPACK_DIR}/vendor/buildpack-stdlib_v8.sh" +source "${BUILDPACK_DIR:?}/vendor/buildpack-stdlib_v8.sh" -if [[ "$(uname)" == "Darwin" ]]; then - sed() { command sed -l "$@"; } -else - sed() { command sed -u "$@"; } -fi +sed() { command sed -u "$@"; } # Syntax sugar. indent() { diff --git a/bin/warnings b/bin/warnings index e3f7fba9c..e24d41ea9 100755 --- a/bin/warnings +++ b/bin/warnings @@ -1,7 +1,8 @@ #!/usr/bin/env bash gdal-missing() { - if grep -qi 'Could not find gdal-config' "$WARNINGS_LOG"; then + # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. + if grep -qi 'Could not find gdal-config' "${WARNINGS_LOG}"; then echo puts-warn "Hello! Package installation failed since the GDAL library was not found." puts-warn "For GDAL, GEOS and PROJ support, use the Geo buildpack alongside the Python buildpack:" diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index e1c35c644..194670ea8 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -19,7 +19,7 @@ function abort() { exit 1 } -case "${STACK}" in +case "${STACK:?}" in heroku-24) SUPPORTED_PYTHON_VERSIONS=( "3.10" @@ -170,7 +170,8 @@ else LDFLAGS="$(dpkg-buildflags --get LDFLAGS) -Wl,--strip-all" fi -make -j "$(nproc)" "EXTRA_CFLAGS=${EXTRA_CFLAGS}" "LDFLAGS=${LDFLAGS}" +CPU_COUNT="$(nproc)" +make -j "${CPU_COUNT}" "EXTRA_CFLAGS=${EXTRA_CFLAGS}" "LDFLAGS=${LDFLAGS}" make install if [[ "${PYTHON_MAJOR_VERSION}" == 3.[8-9] ]]; then diff --git a/etc/publish.sh b/etc/publish.sh index ea07f296e..77499f50b 100755 --- a/etc/publish.sh +++ b/etc/publish.sh @@ -1,4 +1,5 @@ #!/usr/bin/env bash +# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. set -euo pipefail diff --git a/lib/kvstore.sh b/lib/kvstore.sh index 4bfdc85ac..ba3809383 100644 --- a/lib/kvstore.sh +++ b/lib/kvstore.sh @@ -1,5 +1,9 @@ #!/usr/bin/env bash +# This is technically redundant, since all consumers of this lib will have enabled these, +# however, it helps Shellcheck realise the options under which these functions will run. +set -euo pipefail + # TODO: Switch this file to using namespaced functions like `kvstore::`. # Taken from: https://github.com/heroku/heroku-buildpack-nodejs/blob/main/lib/kvstore.sh @@ -71,6 +75,7 @@ kv_list() { kv_keys "${f}" | tr ' ' '\n' | while read -r key; do if [[ -n "${key}" ]]; then + # shellcheck disable=SC2312 # TODO: Invoke this command separately to avoid masking its return value. echo "${key}=$(kv_get_escaped "${f}" "${key}")" fi done diff --git a/lib/metadata.sh b/lib/metadata.sh index 3def5ded3..99f1919a3 100644 --- a/lib/metadata.sh +++ b/lib/metadata.sh @@ -1,10 +1,14 @@ #!/usr/bin/env bash +# This is technically redundant, since all consumers of this lib will have enabled these, +# however, it helps Shellcheck realise the options under which these functions will run. +set -euo pipefail + # TODO: Switch this file to using namespaced functions like `metadata::`. # Based on: https://github.com/heroku/heroku-buildpack-nodejs/blob/main/lib/metadata.sh -source "${BUILDPACK_DIR}/lib/kvstore.sh" +source "${BUILDPACK_DIR:?}/lib/kvstore.sh" # Variables shared by this whole module BUILD_DATA_FILE="" @@ -69,6 +73,6 @@ log_meta_data() { # print all values on one line in logfmt format # https://brandur.org/logfmt # the echo call ensures that all values are printed on a single line - # shellcheck disable=SC2005 disable=SC2046 + # shellcheck disable=SC2005,SC2046,SC2312 echo $(kv_list "${BUILD_DATA_FILE}") } diff --git a/lib/output.sh b/lib/output.sh index 246e689bf..4141e1a20 100644 --- a/lib/output.sh +++ b/lib/output.sh @@ -1,5 +1,9 @@ #!/usr/bin/env bash +# This is technically redundant, since all consumers of this lib will have enabled these, +# however, it helps Shellcheck realise the options under which these functions will run. +set -euo pipefail + # TODO: Switch this file to using namespaced functions like `output::`. ANSI_RED='\033[1;31m' diff --git a/lib/package_manager.sh b/lib/package_manager.sh index d776c08a5..544ea238a 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -1,5 +1,9 @@ #!/usr/bin/env bash +# This is technically redundant, since all consumers of this lib will have enabled these, +# however, it helps Shellcheck realise the options under which these functions will run. +set -euo pipefail + function package_manager::determine_package_manager() { local build_dir="${1}" local package_managers_found=() diff --git a/lib/pip.sh b/lib/pip.sh index 70839364e..2fa7e14b7 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -1,5 +1,9 @@ #!/usr/bin/env bash +# This is technically redundant, since all consumers of this lib will have enabled these, +# however, it helps Shellcheck realise the options under which these functions will run. +set -euo pipefail + function pip::install_pip_setuptools_wheel() { # We use the pip wheel bundled within Python's standard library to install our chosen # pip version, since it's faster than `ensurepip` followed by an upgrade in place. @@ -27,6 +31,7 @@ function pip::install_dependencies() { # to allow for the env var interpolation feature of requirements files to work. # # PIP_EXTRA_INDEX_URL allows for an alternate pypi URL to be used. + # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. if [[ -r "${ENV_DIR}/PIP_EXTRA_INDEX_URL" ]]; then PIP_EXTRA_INDEX_URL="$(cat "${ENV_DIR}/PIP_EXTRA_INDEX_URL")" export PIP_EXTRA_INDEX_URL @@ -40,7 +45,8 @@ function pip::install_dependencies() { fi set +e - /app/.heroku/python/bin/pip install "${args[@]}" --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir --progress-bar off 2>&1 | tee "$WARNINGS_LOG" | cleanup | indent + # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. + /app/.heroku/python/bin/pip install "${args[@]}" --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir --progress-bar off 2>&1 | tee "${WARNINGS_LOG}" | cleanup | indent local PIP_STATUS="${PIPESTATUS[0]}" set -e diff --git a/lib/pipenv.sh b/lib/pipenv.sh index 8d6578430..f718a7d2b 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -1,5 +1,9 @@ #!/usr/bin/env bash +# This is technically redundant, since all consumers of this lib will have enabled these, +# however, it helps Shellcheck realise the options under which these functions will run. +set -euo pipefail + # export CLINT_FORCE_COLOR=1 # export PIPENV_FORCE_COLOR=1 @@ -25,6 +29,7 @@ function pipenv::install_dependencies() { # TODO: Expose all config vars (after suitable checks are added for unsafe env vars). # # PIP_EXTRA_INDEX_URL allows for an alternate pypi URL to be used. + # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. if [[ -r "${ENV_DIR}/PIP_EXTRA_INDEX_URL" ]]; then PIP_EXTRA_INDEX_URL="$(cat "${ENV_DIR}/PIP_EXTRA_INDEX_URL")" export PIP_EXTRA_INDEX_URL diff --git a/lib/utils.sh b/lib/utils.sh index af0aabb14..e3071e249 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -1,5 +1,9 @@ #!/usr/bin/env bash +# This is technically redundant, since all consumers of this lib will have enabled these, +# however, it helps Shellcheck realise the options under which these functions will run. +set -euo pipefail + # Python bundles pip within its standard library, which we can use to install our chosen # pip version from PyPI, saving us from having to download the usual pip bootstrap script. function utils::bundled_pip_module_path() { diff --git a/vendor/buildpack-stdlib_v8.sh b/vendor/buildpack-stdlib_v8.sh index 824f9341c..343f9b28d 100755 --- a/vendor/buildpack-stdlib_v8.sh +++ b/vendor/buildpack-stdlib_v8.sh @@ -1,4 +1,8 @@ #!/usr/bin/env bash +# shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. +# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. + +set -euo pipefail # Based on: # https://raw.githubusercontent.com/heroku/buildpack-stdlib/v8/stdlib.sh @@ -64,7 +68,6 @@ export_env() { sub_env() { ( # TODO: Fix https://github.com/heroku/buildpack-stdlib/issues/37 - # shellcheck disable=SC2153 export_env "$ENV_DIR" "${WHITELIST:-}" "${BLACKLIST:-}" "$@" From 92633fcc677bd3b680ba8166a903b136a3f93785 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 3 Oct 2024 16:04:46 +0000 Subject: [PATCH 030/406] Enable Bash's `inherit_errexit` (#1655) By default Bash subshells (including command substitutions) do not inherit exit on error from the caller. This can cause surprises when functions are called from within command substitutions whose error handling is relying upon exit on error. For example, this cause error handling not to work as expected in the upcoming Python version handling reimplementation. However, as of Bash 4.4 this can be changed by enabling `inherit_errexit`. (The oldest Bash on our supported stacks is 5.0.) See: https://www.shellcheck.net/wiki/SC2311 https://www.gnu.org/software/bash/manual/html_node/The-Shopt-Builtin.html GUS-W-16899232. --- CHANGELOG.md | 1 + bin/compile | 1 + bin/detect | 1 + bin/release | 1 + bin/report | 1 + bin/test-compile | 1 + builds/build_python_runtime.sh | 1 + builds/test_python_runtime.sh | 1 + etc/publish.sh | 1 + 9 files changed, 9 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4a20a17f6..22d62395c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Improved the robustness of buildpack error handling by enabling `inherit_errexit`. ([#1655](https://github.com/heroku/heroku-buildpack-python/pull/1655)) ## [v258] - 2024-10-01 diff --git a/bin/compile b/bin/compile index e9b0eb3eb..706a61a43 100755 --- a/bin/compile +++ b/bin/compile @@ -4,6 +4,7 @@ # shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. set -euo pipefail +shopt -s inherit_errexit # Note: This can't be enabled via app config vars, since at this point they haven't been loaded from ENV_DIR. if [[ "${BUILDPACK_XTRACE:-0}" == "1" ]]; then diff --git a/bin/detect b/bin/detect index 949c24dca..37d7ccdb1 100755 --- a/bin/detect +++ b/bin/detect @@ -3,6 +3,7 @@ # See: https://devcenter.heroku.com/articles/buildpack-api set -euo pipefail +shopt -s inherit_errexit BUILD_DIR="${1}" diff --git a/bin/release b/bin/release index 0e8058d1f..af46b035d 100755 --- a/bin/release +++ b/bin/release @@ -3,6 +3,7 @@ # See: https://devcenter.heroku.com/articles/buildpack-api set -euo pipefail +shopt -s inherit_errexit BUILD_DIR="${1}" diff --git a/bin/report b/bin/report index b45378672..44079d6d0 100755 --- a/bin/report +++ b/bin/report @@ -18,6 +18,7 @@ # when developing run `make compile` in this repo locally, which runs `bin/report` too. set -euo pipefail +shopt -s inherit_errexit CACHE_DIR="${2}" diff --git a/bin/test-compile b/bin/test-compile index 03fd6fadb..1fef46275 100755 --- a/bin/test-compile +++ b/bin/test-compile @@ -3,6 +3,7 @@ # See: https://devcenter.heroku.com/articles/testpack-api set -euo pipefail +shopt -s inherit_errexit # The absolute path to the root of the buildpack. BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index 194670ea8..5aad9222b 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -1,6 +1,7 @@ #!/usr/bin/env bash set -euo pipefail +shopt -s inherit_errexit PYTHON_VERSION="${1:?"Error: The Python version to build must be specified as the first argument."}" PYTHON_MAJOR_VERSION="${PYTHON_VERSION%.*}" diff --git a/builds/test_python_runtime.sh b/builds/test_python_runtime.sh index 4a21750d4..9eb41a590 100755 --- a/builds/test_python_runtime.sh +++ b/builds/test_python_runtime.sh @@ -1,6 +1,7 @@ #!/usr/bin/env bash set -euo pipefail +shopt -s inherit_errexit ARCHIVE_FILEPATH="${1:?"Error: The filepath of the Python runtime archive must be specified as the first argument."}" diff --git a/etc/publish.sh b/etc/publish.sh index 77499f50b..1a445e6d7 100755 --- a/etc/publish.sh +++ b/etc/publish.sh @@ -2,6 +2,7 @@ # shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. set -euo pipefail +shopt -s inherit_errexit BP_NAME=${1:-"heroku/python"} From f29d3dcf5186862a2bc78effcf750a0650ae530e Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 7 Oct 2024 08:06:50 +0000 Subject: [PATCH 031/406] Add support for Python 3.9 on Heroku-24 (#1656) Previously Python 3.9 was only supported on the Heroku-22 stack and older, since it reaches EOL in Oct 2025. However, adding support for Python 3.9 to Heroku-24 will make it easier for users upgrading stacks, and also give more Python version options to CNB users that need ARM64 support, given that multi-arch is only available for Heroku-24+. It also simplifies the version error checks we need to perform in the upcoming Python version handling refactor. GUS-W-16917997. --- .github/workflows/build_python_runtime.yml | 4 ++-- CHANGELOG.md | 1 + README.md | 2 +- builds/build_python_runtime.sh | 9 +-------- spec/hatchet/pipenv_spec.rb | 14 ++----------- spec/hatchet/python_update_warning_spec.rb | 23 +++++----------------- spec/hatchet/python_version_spec.rb | 14 ++----------- 7 files changed, 14 insertions(+), 53 deletions(-) diff --git a/.github/workflows/build_python_runtime.yml b/.github/workflows/build_python_runtime.yml index 76812cb75..724d4045a 100644 --- a/.github/workflows/build_python_runtime.yml +++ b/.github/workflows/build_python_runtime.yml @@ -80,8 +80,8 @@ jobs: run: aws s3 sync ./upload "s3://${S3_BUCKET}" heroku-24: - # On Heroku-24 we only support Python 3.10+. - if: inputs.stack == 'heroku-24' || (inputs.stack == 'auto' && !startsWith(inputs.python_version,'3.8.') && !startsWith(inputs.python_version,'3.9.')) + # On Heroku-24 we only support Python 3.9+. + if: inputs.stack == 'heroku-24' || (inputs.stack == 'auto' && !startsWith(inputs.python_version,'3.8.')) strategy: fail-fast: false matrix: diff --git a/CHANGELOG.md b/CHANGELOG.md index 22d62395c..fc87c0cd3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Added support for Python 3.9 on Heroku-24. ([#1656](https://github.com/heroku/heroku-buildpack-python/pull/1656)) - Improved the robustness of buildpack error handling by enabling `inherit_errexit`. ([#1655](https://github.com/heroku/heroku-buildpack-python/pull/1655)) ## [v258] - 2024-10-01 diff --git a/README.md b/README.md index d811f09eb..3c78ac632 100644 --- a/README.md +++ b/README.md @@ -63,5 +63,5 @@ Supported runtime options include: - `python-3.12.7` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) - `python-3.11.10` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) - `python-3.10.15` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) -- `python-3.9.20` on Heroku-20 and Heroku-22 only +- `python-3.9.20` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) - `python-3.8.20` on Heroku-20 only diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index 5aad9222b..ea5c0eb29 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -21,14 +21,7 @@ function abort() { } case "${STACK:?}" in - heroku-24) - SUPPORTED_PYTHON_VERSIONS=( - "3.10" - "3.11" - "3.12" - ) - ;; - heroku-22) + heroku-22 | heroku-24) SUPPORTED_PYTHON_VERSIONS=( "3.9" "3.10" diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 72342994e..c90092dbf 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -216,19 +216,9 @@ end context 'with a Pipfile.lock containing python_version 3.9' do - let(:allow_failure) { false } - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.9', allow_failure:) } - - context 'when using Heroku-22 or older', stacks: %w[heroku-20 heroku-22] do - include_examples 'builds using Pipenv with the requested Python version', LATEST_PYTHON_3_9 - end + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.9') } - context 'when using Heroku-24', stacks: %w[heroku-24] do - let(:allow_failure) { true } - - # We only support Python 3.9 on Heroku-22 and older. - include_examples 'aborts the build with a runtime not available message (Pipenv)', LATEST_PYTHON_3_9 - end + include_examples 'builds using Pipenv with the requested Python version', LATEST_PYTHON_3_9 end context 'with a Pipfile.lock containing python_version 3.10' do diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index d6d43c2f6..284544c91 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -79,38 +79,25 @@ end context 'with a runtime.txt containing python-3.9.0' do - let(:allow_failure) { false } - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.9_outdated', allow_failure:) } - - context 'when using Heroku-22 or older', stacks: %w[heroku-20 heroku-22] do - include_examples 'warns there is a Python update available', '3.9.0', LATEST_PYTHON_3_9 - end - - context 'when using Heroku-24', stacks: %w[heroku-24] do - let(:allow_failure) { true } + let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.9_outdated') } - # We only support Python 3.9 on Heroku-22 and older. - include_examples 'aborts the build without showing an update warning', '3.9.0' - end + include_examples 'warns there is a Python update available', '3.9.0', LATEST_PYTHON_3_9 end context 'with a runtime.txt containing python-3.10.0' do - let(:allow_failure) { false } - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.10_outdated', allow_failure:) } + let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.10_outdated') } include_examples 'warns there is a Python update available', '3.10.0', LATEST_PYTHON_3_10 end context 'with a runtime.txt containing python-3.11.0' do - let(:allow_failure) { false } - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.11_outdated', allow_failure:) } + let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.11_outdated') } include_examples 'warns there is a Python update available', '3.11.0', LATEST_PYTHON_3_11 end context 'with a runtime.txt containing python-3.12.0' do - let(:allow_failure) { false } - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.12_outdated', allow_failure:) } + let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.12_outdated') } include_examples 'warns there is a Python update available', '3.12.0', LATEST_PYTHON_3_12 end diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 14d1ffdc6..44038a080 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -179,19 +179,9 @@ end context 'when runtime.txt contains python-3.9.20' do - let(:allow_failure) { false } - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.9', allow_failure:) } - - context 'when using Heroku-22 or older', stacks: %w[heroku-20 heroku-22] do - include_examples 'builds with the requested Python version', LATEST_PYTHON_3_9 - end + let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.9') } - context 'when using Heroku-24', stacks: %w[heroku-24] do - let(:allow_failure) { true } - - # We only support Python 3.9 on Heroku-22 and older. - include_examples 'aborts the build with a runtime not available message', "python-#{LATEST_PYTHON_3_9}" - end + include_examples 'builds with the requested Python version', LATEST_PYTHON_3_9 end context 'when runtime.txt contains python-3.10.15' do From a5dfa07b96a802d26033cf4a9972fc71fcbfc924 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 7 Oct 2024 20:42:42 +0000 Subject: [PATCH 032/406] Add additional buildpack metrics (#1657) Covering: - use of outdated Python patch versions - any occurrences of internal errors (Split out of the upcoming Python version refactor for easier review) --- CHANGELOG.md | 1 + bin/compile | 12 ++++++------ bin/report | 1 + bin/steps/pipenv-python-version | 2 +- bin/steps/python | 3 +++ lib/utils.sh | 1 + 6 files changed, 13 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fc87c0cd3..9345e2bc7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Added support for Python 3.9 on Heroku-24. ([#1656](https://github.com/heroku/heroku-buildpack-python/pull/1656)) +- Added buildpack metrics for use of outdated Python patch versions and occurrences of internal errors. ([#1657](https://github.com/heroku/heroku-buildpack-python/pull/1657)) - Improved the robustness of buildpack error handling by enabling `inherit_errexit`. ([#1655](https://github.com/heroku/heroku-buildpack-python/pull/1655)) ## [v258] - 2024-10-01 diff --git a/bin/compile b/bin/compile index 706a61a43..4c9c0a5f0 100755 --- a/bin/compile +++ b/bin/compile @@ -137,8 +137,8 @@ else CACHED_PYTHON_STACK=$STACK fi -PACKAGE_MANAGER=$(package_manager::determine_package_manager "${BUILD_DIR}") -meta_set "package_manager" "${PACKAGE_MANAGER}" +package_manager="$(package_manager::determine_package_manager "${BUILD_DIR}")" +meta_set "package_manager" "${package_manager}" # Pipenv Python version support. # Detect the version of Python requested from a Pipfile (e.g. python_version or python_full_version). @@ -189,7 +189,7 @@ meta_time "python_install_duration" "${install_python_start_time}" # Install the package manager and related tools. package_manager_install_start_time=$(nowms) bundled_pip_module_path="$(utils::bundled_pip_module_path "${BUILD_DIR}")" -case "${PACKAGE_MANAGER}" in +case "${package_manager}" in pip) pip::install_pip_setuptools_wheel "${bundled_pip_module_path}" ;; @@ -199,7 +199,7 @@ case "${PACKAGE_MANAGER}" in pipenv::install_pipenv ;; *) - utils::abort_internal_error "Unhandled package manager" + utils::abort_internal_error "Unhandled package manager: ${package_manager}" ;; esac meta_time "package_manager_install_duration" "${package_manager_install_start_time}" @@ -214,7 +214,7 @@ meta_time "sqlite_install_duration" "${install_sqlite_start_time}" # Install app dependencies. dependencies_install_start_time=$(nowms) -case "${PACKAGE_MANAGER}" in +case "${package_manager}" in pip) pip::install_dependencies ;; @@ -222,7 +222,7 @@ case "${PACKAGE_MANAGER}" in pipenv::install_dependencies ;; *) - utils::abort_internal_error "Unhandled package manager" + utils::abort_internal_error "Unhandled package manager: ${package_manager}" ;; esac meta_time "dependencies_install_duration" "${dependencies_install_start_time}" diff --git a/bin/report b/bin/report index 44079d6d0..71bd20384 100755 --- a/bin/report +++ b/bin/report @@ -86,6 +86,7 @@ ALL_OTHER_FIELDS=( pre_compile_hook pre_compile_hook_duration python_install_duration + python_version_outdated setup_py_only sqlite_install_duration total_duration diff --git a/bin/steps/pipenv-python-version b/bin/steps/pipenv-python-version index 2a76c56a2..de2aad798 100755 --- a/bin/steps/pipenv-python-version +++ b/bin/steps/pipenv-python-version @@ -6,7 +6,7 @@ # Detect Python-version with Pipenv. -if [[ "${PACKAGE_MANAGER}" == "pipenv" ]]; then +if [[ "${package_manager}" == "pipenv" ]]; then if [[ ! -f $BUILD_DIR/runtime.txt ]]; then if [[ ! -f $BUILD_DIR/Pipfile.lock ]]; then diff --git a/bin/steps/python b/bin/steps/python index 362e9e42e..a2bdb0f12 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -76,6 +76,9 @@ function warn_if_patch_update_available() { puts-warn "A Python security update is available! Upgrade as soon as possible to: ${latest_patch_version}" puts-warn "See: https://devcenter.heroku.com/articles/python-runtimes" puts-warn + meta_set "python_version_outdated" "true" + else + meta_set "python_version_outdated" "false" fi } diff --git a/lib/utils.sh b/lib/utils.sh index e3071e249..7d702c54f 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -29,5 +29,6 @@ function utils::bundled_pip_module_path() { function utils::abort_internal_error() { local message="${1}" display_error "Internal error: ${message} (line $(caller || true))." + meta_set "failure_reason" "internal-error" exit 1 } From 42499ff4dd7101e9e10c4bba7a3b8d39ca120a65 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 9 Oct 2024 08:40:40 +0000 Subject: [PATCH 033/406] Refactor Python version handling (#1658) The previous implementation of Python version detection/resolution revolved heavily around the `runtime.txt` file, even though the version could originate from other sources. For example, a fake `runtime.txt` would be written out into the build directory containing the desired Python version, even if the version originated from `Pipfile.lock`. This meant all later Python version handling in the buildpack would see a `runtime.txt` file, along with versions specified in the syntax of that file (e.g. `python-N.N.N` strings), even though that wasn't the format in which the user had specified the version. Now, the buildpack explicitly tracks the requested version and its origin (rather than using the `runtime.txt` file as an API), along with the resolved Python major and full versions (which makes later Python version conditionals less fragile). In addition, the Python version specifiers are validated upfront at the point of parsing the relevant data source, so that clearer error messages can be shown. Lastly, the Python version resolution (the mapping of major Python versions to the latest patch release) has been decoupled from the Pipenv version implementation and made more robust, so it can also be used by the upcoming `.python-version` file support. GUS-W-16821309. GUS-W-7924371. GUS-W-8104668. --- CHANGELOG.md | 4 + bin/compile | 64 ++-- bin/default_pythons | 19 -- bin/steps/pipenv-python-version | 66 ---- bin/steps/python | 124 +++----- lib/python_version.sh | 286 ++++++++++++++++++ .../Pipfile | 2 +- .../pipenv_lockfile_invalid_json/Pipfile.lock | 1 + spec/fixtures/pipenv_python_3.6/Pipfile.lock | 29 -- .../pipenv_python_full_version/Pipfile | 5 +- .../pipenv_python_full_version/Pipfile.lock | 11 +- .../Pipfile | 2 +- .../Pipfile.lock | 10 +- .../requirements.txt | 0 spec/fixtures/python_2.7/runtime.txt | 1 + .../fixtures/python_3.10_outdated/runtime.txt | 1 - .../fixtures/python_3.11_outdated/runtime.txt | 1 - .../fixtures/python_3.12_outdated/runtime.txt | 1 - spec/fixtures/python_3.6/requirements.txt | 1 - spec/fixtures/python_3.6/runtime.txt | 1 - spec/fixtures/python_3.7/requirements.txt | 1 - .../requirements.txt | 0 .../runtime.txt | 1 + .../requirements.txt | 0 .../runtime.txt | 1 + .../requirements.txt | 0 .../runtime.txt | 0 .../requirements.txt | 0 .../runtime.txt | 0 spec/hatchet/ci_spec.rb | 16 +- spec/hatchet/pip_spec.rb | 20 +- spec/hatchet/pipenv_spec.rb | 200 ++++++------ spec/hatchet/python_update_warning_spec.rb | 100 +++--- spec/hatchet/python_version_spec.rb | 170 ++++++++--- spec/hatchet/stack_spec.rb | 10 +- spec/spec_helper.rb | 3 +- 36 files changed, 680 insertions(+), 471 deletions(-) delete mode 100755 bin/default_pythons delete mode 100755 bin/steps/pipenv-python-version create mode 100644 lib/python_version.sh rename spec/fixtures/{pipenv_python_3.6 => pipenv_lockfile_invalid_json}/Pipfile (84%) create mode 100644 spec/fixtures/pipenv_lockfile_invalid_json/Pipfile.lock delete mode 100644 spec/fixtures/pipenv_python_3.6/Pipfile.lock rename spec/fixtures/{python_3.10_outdated => python_2.7}/requirements.txt (100%) create mode 100644 spec/fixtures/python_2.7/runtime.txt delete mode 100644 spec/fixtures/python_3.10_outdated/runtime.txt delete mode 100644 spec/fixtures/python_3.11_outdated/runtime.txt delete mode 100644 spec/fixtures/python_3.12_outdated/runtime.txt delete mode 100644 spec/fixtures/python_3.6/requirements.txt delete mode 100644 spec/fixtures/python_3.6/runtime.txt rename spec/fixtures/{python_3.11_outdated => python_version_non_existent_major}/requirements.txt (100%) create mode 100644 spec/fixtures/python_version_non_existent_major/runtime.txt rename spec/fixtures/{python_3.12_outdated => python_version_non_existent_patch}/requirements.txt (100%) create mode 100644 spec/fixtures/python_version_non_existent_patch/runtime.txt rename spec/fixtures/{python_3.8_outdated => python_version_outdated}/requirements.txt (100%) rename spec/fixtures/{python_3.9_outdated => python_version_outdated}/runtime.txt (100%) rename spec/fixtures/{python_3.9_outdated => python_version_outdated_and_deprecated}/requirements.txt (100%) rename spec/fixtures/{python_3.8_outdated => python_version_outdated_and_deprecated}/runtime.txt (100%) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9345e2bc7..d0fd0fd39 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## [Unreleased] +- Improved build log output about the detected Python version. ([#1658](https://github.com/heroku/heroku-buildpack-python/pull/1658)) +- Improved error messages shown when the requested Python version is not a valid version string or is for an unknown/non-existent major Python version. ([#1658](https://github.com/heroku/heroku-buildpack-python/pull/1658)) +- Improved error messages shown when `Pipfile.lock` is not valid JSON. ([#1658](https://github.com/heroku/heroku-buildpack-python/pull/1658)) +- Fixed invalid Python versions being silently ignored when they were specified via the `python_version` field in `Pipfile.lock`. ([#1658](https://github.com/heroku/heroku-buildpack-python/pull/1658)) - Added support for Python 3.9 on Heroku-24. ([#1656](https://github.com/heroku/heroku-buildpack-python/pull/1656)) - Added buildpack metrics for use of outdated Python patch versions and occurrences of internal errors. ([#1657](https://github.com/heroku/heroku-buildpack-python/pull/1657)) - Improved the robustness of buildpack error handling by enabling `inherit_errexit`. ([#1655](https://github.com/heroku/heroku-buildpack-python/pull/1655)) diff --git a/bin/compile b/bin/compile index 4c9c0a5f0..12c74f3b6 100755 --- a/bin/compile +++ b/bin/compile @@ -24,6 +24,7 @@ source "${BUILDPACK_DIR}/lib/output.sh" source "${BUILDPACK_DIR}/lib/package_manager.sh" source "${BUILDPACK_DIR}/lib/pip.sh" source "${BUILDPACK_DIR}/lib/pipenv.sh" +source "${BUILDPACK_DIR}/lib/python_version.sh" source "${BUILDPACK_DIR}/lib/utils.sh" compile_start_time=$(nowms) @@ -46,9 +47,6 @@ export BUILD_DIR CACHE_DIR ENV_DIR S3_BASE_URL="${BUILDPACK_S3_BASE_URL:-"https://heroku-buildpack-python.s3.us-east-1.amazonaws.com"}" # This has to be exported since it's used by the geo-libs step which is run in a subshell. -# Default Python Versions -source "${BUILDPACK_DIR}/bin/default_pythons" - # Common Problem Warnings: # This section creates a temporary file in which to stick the output of `pip install`. # The `warnings` subscript then greps through this for common problems and guides @@ -123,10 +121,14 @@ fi # Runs a `bin/pre_compile` script if found in the app source, allowing build customisation. source "${BUILDPACK_DIR}/bin/steps/hooks/pre_compile" -# Sticky runtimes. If there was a previous build, and it used a given version of Python, -# continue to use that version of Python in perpetuity. +# TODO: Clear the cache if this isn't a valid version, as part of the cache refactor. +# (Currently the version is instead validated in `read_requested_python_version()`) if [[ -f "$CACHE_DIR/.heroku/python-version" ]]; then - CACHED_PYTHON_VERSION=$(cat "$CACHE_DIR/.heroku/python-version") + cached_python_version="$(cat "${CACHE_DIR}/.heroku/python-version")" + # `python-X.Y.Z` -> `X.Y` + cached_python_version="${cached_python_version#python-}" +else + cached_python_version= fi # We didn't always record the stack version. @@ -140,29 +142,37 @@ fi package_manager="$(package_manager::determine_package_manager "${BUILD_DIR}")" meta_set "package_manager" "${package_manager}" -# Pipenv Python version support. -# Detect the version of Python requested from a Pipfile (e.g. python_version or python_full_version). -# Convert it to a runtime.txt file. -source "${BUILDPACK_DIR}/bin/steps/pipenv-python-version" - -if [[ -f runtime.txt ]]; then - # PYTHON_VERSION_SOURCE may have already been set by the pipenv-python-version step. - # TODO: Refactor this and stop pipenv-python-version using runtime.txt as an API. - PYTHON_VERSION_SOURCE=${PYTHON_VERSION_SOURCE:-"runtime.txt"} - puts-step "Using Python version specified in ${PYTHON_VERSION_SOURCE}" - meta_set "python_version_reason" "specified" -elif [[ -n "${CACHED_PYTHON_VERSION:-}" ]]; then - puts-step "No Python version was specified. Using the same version as the last build: ${CACHED_PYTHON_VERSION}" - echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" - meta_set "python_version_reason" "cached" - echo "${CACHED_PYTHON_VERSION}" >runtime.txt -else - puts-step "No Python version was specified. Using the buildpack default: ${DEFAULT_PYTHON_VERSION}" - echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" - meta_set "python_version_reason" "default" - echo "${DEFAULT_PYTHON_VERSION}" >runtime.txt +# TODO: Move this warning to lib/package_manager.sh once `output::warning()` exists +# (puts-warn outputs to stdout, which would break `determine_package_manager()` as is). +# TODO: Adjust this warning to mention support for missing Pipfile.lock will be removed soon. +if [[ "${package_manager}" == "pipenv" && ! -f "${BUILD_DIR}/Pipfile.lock" ]]; then + puts-warn "No 'Pipfile.lock' found! We recommend you commit this into your repository." fi +# We use the Bash 4.3+ `nameref` feature to pass back multiple values from this function +# without having to hardcode globals. See: https://stackoverflow.com/a/38997681 +python_version::read_requested_python_version "${BUILD_DIR}" "${package_manager}" "${cached_python_version}" requested_python_version python_version_origin +meta_set "python_version_reason" "${python_version_origin}" + +case "${python_version_origin}" in + default) + puts-step "No Python version was specified. Using the buildpack default: Python ${requested_python_version}" + echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" + ;; + cached) + puts-step "No Python version was specified. Using the same version as the last build: Python ${requested_python_version}" + echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" + ;; + *) + puts-step "Using Python ${requested_python_version} specified in ${python_version_origin}" + ;; +esac + +python_full_version="$(python_version::resolve_python_version "${requested_python_version}" "${python_version_origin}")" +python_major_version="${python_full_version%.*}" +meta_set "python_version" "${python_full_version}" +meta_set "python_version_major" "${python_major_version}" + # The directory for the .profile.d scripts. mkdir -p "$(dirname "$PROFILE_PATH")" # The directory for editable VCS dependencies. diff --git a/bin/default_pythons b/bin/default_pythons deleted file mode 100755 index 51e9cc247..000000000 --- a/bin/default_pythons +++ /dev/null @@ -1,19 +0,0 @@ -#!/usr/bin/env bash - -# Disable unused env var warning, since shellcheck doesn't take into account -# that this file is sourced. We don't want to use export since it exposes -# the env vars to subprocesses. -# shellcheck disable=2034 - -LATEST_312="python-3.12.7" -LATEST_311="python-3.11.10" -LATEST_310="python-3.10.15" -LATEST_39="python-3.9.20" -LATEST_38="python-3.8.20" -# TODO: Remove these EOL versions once pipenv-python-version is refactored. -LATEST_37="python-3.7.17" -LATEST_36="python-3.6.15" -LATEST_35="python-3.5.10" -LATEST_34="python-3.4.10" -LATEST_27="python-2.7.18" -DEFAULT_PYTHON_VERSION="${LATEST_312}" diff --git a/bin/steps/pipenv-python-version b/bin/steps/pipenv-python-version deleted file mode 100755 index de2aad798..000000000 --- a/bin/steps/pipenv-python-version +++ /dev/null @@ -1,66 +0,0 @@ -#!/usr/bin/env bash -# shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. -# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. - -# TODO: Move this to lib/ as part of the refactoring for .python-version support. - -# Detect Python-version with Pipenv. - -if [[ "${package_manager}" == "pipenv" ]]; then - - if [[ ! -f $BUILD_DIR/runtime.txt ]]; then - if [[ ! -f $BUILD_DIR/Pipfile.lock ]]; then - puts-warn "No 'Pipfile.lock' found! We recommend you commit this into your repository." - fi - if [[ -f $BUILD_DIR/Pipfile.lock ]]; then - # Ignore unused env var warning since this is used by bin/compile. - # shellcheck disable=2034 - PYTHON_VERSION_SOURCE='Pipfile.lock' - set +e - PYTHON=$(jq -r '._meta.requires.python_full_version' "$BUILD_DIR/Pipfile.lock") - if [[ "$PYTHON" != "null" ]]; then - echo "python-$PYTHON" >"$BUILD_DIR/runtime.txt" - fi - set -e - - if [[ "$PYTHON" == "null" ]]; then - PYTHON=$(jq -r '._meta.requires.python_version' "$BUILD_DIR/Pipfile.lock") - case "${PYTHON}" in - 2.7) - echo "${LATEST_27}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.4) - echo "${LATEST_34}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.5) - echo "${LATEST_35}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.6) - echo "${LATEST_36}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.7) - echo "${LATEST_37}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.8) - echo "${LATEST_38}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.9) - echo "${LATEST_39}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.10) - echo "${LATEST_310}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.11) - echo "${LATEST_311}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.12) - echo "${LATEST_312}" >"${BUILD_DIR}/runtime.txt" - ;; - # TODO: Make this case an error - *) ;; - esac - fi - - fi - fi -fi diff --git a/bin/steps/python b/bin/steps/python index a2bdb0f12..725846d15 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -4,54 +4,21 @@ set -euo pipefail -PYTHON_VERSION=$(cat runtime.txt) -# Remove leading and trailing whitespace. Note: This implementation relies upon -# `extglob` being set, which is the case thanks to `bin/utils` being run earlier. -PYTHON_VERSION="${PYTHON_VERSION##+([[:space:]])}" -PYTHON_VERSION="${PYTHON_VERSION%%+([[:space:]])}" - -function eol_python_version_error() { - local major_version="${1}" - local eol_date="${2}" - display_error <<-EOF - Error: Python ${major_version} is no longer supported. - - Python ${major_version} reached upstream end-of-life on ${eol_date}, and is - therefore no longer receiving security updates: - https://devguide.python.org/versions/#supported-versions - - As such, it is no longer supported by this buildpack. - - Please upgrade to a newer Python version. - - For a list of the supported Python versions, see: - https://devcenter.heroku.com/articles/python-support#supported-runtimes - EOF - meta_set "failure_reason" "python-version-eol" - exit 1 -} - -# We check for EOL prior to checking if the archive exists on S3, to ensure the more specific EOL error -# message is still shown for newer stacks where the EOL Python versions might not have been built. -case "${PYTHON_VERSION}" in - python-3.7.+([0-9])) - eol_python_version_error "3.7" "June 27th, 2023" - ;; - python-3.6.+([0-9])) - eol_python_version_error "3.6" "December 23rd, 2021" - ;; - *) ;; -esac - # The Python runtime archive filename is of form: 'python-X.Y.Z-ubuntu-22.04-amd64.tar.zst' # The Ubuntu version is calculated from `STACK` since it's faster than calling `lsb_release`. UBUNTU_VERSION="${STACK/heroku-/}.04" ARCH=$(dpkg --print-architecture) -PYTHON_URL="${S3_BASE_URL}/${PYTHON_VERSION}-ubuntu-${UBUNTU_VERSION}-${ARCH}.tar.zst" - +PYTHON_URL="${S3_BASE_URL}/python-${python_full_version}-ubuntu-${UBUNTU_VERSION}-${ARCH}.tar.zst" + +# The Python version validation earlier will have filtered out most unsupported versions. +# However, the version might still not be found if either: +# 1. It's a Python major version we've deprecated and so is only available on older stacks (i.e: Python 3.8). +# 2. If an exact Python version was requested and the patch version doesn't exist (e.g. 3.12.999). +# 3. The user has pinned to an older buildpack version and the S3 bucket location or layout has changed since. +# TODO: Update this message to be more specific once Python 3.8 support is dropped. if ! curl --output /dev/null --silent --head --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}"; then display_error <<-EOF - Error: Requested runtime '${PYTHON_VERSION}' is not available for this stack (${STACK}). + Error: Python ${python_full_version} is not available for this stack (${STACK}). For a list of the supported Python versions, see: https://devcenter.heroku.com/articles/python-support#supported-runtimes @@ -60,20 +27,17 @@ if ! curl --output /dev/null --silent --head --fail --retry 3 --retry-connrefuse exit 1 fi -# TODO: Refactor Python version usage to use the non-prefixed form everywhere. -python_version_without_prefix="${PYTHON_VERSION#python-}" -meta_set "python_version" "${python_version_without_prefix}" -meta_set "python_version_major" "${python_version_without_prefix%.*}" - function warn_if_patch_update_available() { - local requested_version="${1}" - local latest_patch_version="${2}" + local requested_full_version="${1}" + local requested_major_version="${2}" + local latest_patch_version + latest_patch_version="$(python_version::resolve_python_version "${requested_major_version}" "${python_version_origin}")" # Extract the patch version component of the version strings (ie: the '5' in '3.10.5'). - local requested_patch_number="${requested_version##*.}" + local requested_patch_number="${requested_full_version##*.}" local latest_patch_number="${latest_patch_version##*.}" if ((requested_patch_number < latest_patch_number)); then puts-warn - puts-warn "A Python security update is available! Upgrade as soon as possible to: ${latest_patch_version}" + puts-warn "A Python security update is available! Upgrade as soon as possible to: Python ${latest_patch_version}" puts-warn "See: https://devcenter.heroku.com/articles/python-runtimes" puts-warn meta_set "python_version_outdated" "true" @@ -84,45 +48,31 @@ function warn_if_patch_update_available() { # We wait until now to display outdated Python version warnings, since we only want to show them # if there weren't any errors with the version to avoid adding noise to the error messages. -case "${PYTHON_VERSION}" in - python-3.12.*) - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_312}" - ;; - python-3.11.*) - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_311}" - ;; - python-3.10.*) - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_310}" - ;; - python-3.9.*) - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_39}" - ;; - python-3.8.*) - puts-warn - puts-warn "Python 3.8 will reach its upstream end-of-life in October 2024, at which" - puts-warn "point it will no longer receive security updates:" - puts-warn "https://devguide.python.org/versions/#supported-versions" - puts-warn - puts-warn "Support for Python 3.8 will be removed from this buildpack on December 4th, 2024." - puts-warn - puts-warn "Upgrade to a newer Python version as soon as possible to keep your app secure." - puts-warn "See: https://devcenter.heroku.com/articles/python-runtimes" - puts-warn - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_38}" - ;; - # TODO: Make this case an error, since it should be unreachable. - *) ;; -esac +# TODO: Move this into lib/ as part of the warnings refactor. +if [[ "${python_major_version}" == "3.8" ]]; then + puts-warn + puts-warn "Python 3.8 will reach its upstream end-of-life in October 2024, at which" + puts-warn "point it will no longer receive security updates:" + puts-warn "https://devguide.python.org/versions/#supported-versions" + puts-warn + puts-warn "Support for Python 3.8 will be removed from this buildpack on December 4th, 2024." + puts-warn + puts-warn "Upgrade to a newer Python version as soon as possible to keep your app secure." + puts-warn "See: https://devcenter.heroku.com/articles/python-runtimes" + puts-warn +fi + +warn_if_patch_update_available "${python_full_version}" "${python_major_version}" if [[ "$STACK" != "$CACHED_PYTHON_STACK" ]]; then puts-step "Stack has changed from $CACHED_PYTHON_STACK to $STACK, clearing cache" rm -rf .heroku/python-stack .heroku/python-version .heroku/python .heroku/vendor .heroku/python .heroku/python-sqlite3-version fi +# TODO: Clean this up as part of the cache refactor. if [[ -f .heroku/python-version ]]; then - # shellcheck disable=SC2312 # TODO: Invoke this command separately to avoid masking its return value. - if [[ ! "$(cat .heroku/python-version)" == "$PYTHON_VERSION" ]]; then - puts-step "Python version has changed from $(cat .heroku/python-version) to ${PYTHON_VERSION}, clearing cache" + if [[ "${cached_python_version}" != "${python_full_version}" ]]; then + puts-step "Python version has changed from ${cached_python_version} to ${python_full_version}, clearing cache" rm -rf .heroku/python else SKIP_INSTALL=1 @@ -153,9 +103,9 @@ if [[ -f "${BUILD_DIR}/requirements.txt" ]]; then fi if [[ "${SKIP_INSTALL:-0}" == "1" ]]; then - puts-step "Using cached install of ${PYTHON_VERSION}" + puts-step "Using cached install of Python ${python_full_version}" else - puts-step "Installing ${PYTHON_VERSION}" + puts-step "Installing Python ${python_full_version}" # Prepare destination directory. mkdir -p .heroku/python @@ -163,13 +113,13 @@ else if ! curl --silent --show-error --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}" | tar --zstd --extract --directory .heroku/python; then # The Python version was confirmed to exist previously, so any failure here is due to # a networking issue or archive/buildpack bug rather than the runtime not existing. - display_error "Error: Failed to download/install ${PYTHON_VERSION}." + display_error "Error: Failed to download/install Python ${python_full_version}." meta_set "failure_reason" "python-download" exit 1 fi # Record for future reference. - echo "$PYTHON_VERSION" >.heroku/python-version + echo "python-${python_full_version}" >.heroku/python-version echo "$STACK" >.heroku/python-stack hash -r diff --git a/lib/python_version.sh b/lib/python_version.sh new file mode 100644 index 000000000..8b88e6e37 --- /dev/null +++ b/lib/python_version.sh @@ -0,0 +1,286 @@ +#!/usr/bin/env bash + +# This is technically redundant, since all consumers of this lib will have enabled these, +# however, it helps Shellcheck realise the options under which these functions will run. +set -euo pipefail + +LATEST_PYTHON_3_8="3.8.20" +LATEST_PYTHON_3_9="3.9.20" +LATEST_PYTHON_3_10="3.10.15" +LATEST_PYTHON_3_11="3.11.10" +LATEST_PYTHON_3_12="3.12.7" + +DEFAULT_PYTHON_FULL_VERSION="${LATEST_PYTHON_3_12}" +DEFAULT_PYTHON_MAJOR_VERSION="${DEFAULT_PYTHON_FULL_VERSION%.*}" + +# Integer with no redundant leading zeros. +INT_REGEX="(0|[1-9][0-9]*)" +# Versions of form N.N or N.N.N. +PYTHON_VERSION_REGEX="${INT_REGEX}\.${INT_REGEX}(\.${INT_REGEX})?" +# Versions of form N.N.N only. +PYTHON_FULL_VERSION_REGEX="${INT_REGEX}\.${INT_REGEX}\.${INT_REGEX}" + +# Determine what Python version has been requested for the project. +# +# Returns a version request of form N.N or N.N.N, with basic validation checks that the version +# matches those forms. EOL version checks will be performed later, when this version request is +# resolved to an exact Python version. +# +# If an app specifies the Python version via multiple means, then the order of precedence is: +# 1. runtime.txt +# 2. Pipfile.lock (`python_full_version` field) +# 3. Pipfile.lock (`python_version` field) +# +# If a version wasn't specified by the app, then new apps/those with an empty cache will use +# a buildpack default version for the first build, and then subsequent cached builds will use +# the same Python full version in perpetuity (aka sticky versions). Sticky versioning leads to +# confusing UX so is something we want to deprecate/sunset in the future (and have already done +# so in the Python CNB). +# TODO: Change the sticky versioning implementation so it's only sticky to the major version +# rather than the full version, so apps that don't specify a Python version at least get +# security patch updates. +function python_version::read_requested_python_version() { + local build_dir="${1}" + local package_manager="${2}" + local cached_python_version="${3}" + # We use the Bash 4.3+ `nameref` feature to pass back multiple values from this function + # without having to hardcode globals. See: https://stackoverflow.com/a/38997681 + declare -n version="${4}" + declare -n origin="${5}" + local contents + + local runtime_txt_path="${build_dir}/runtime.txt" + if [[ -f "${runtime_txt_path}" ]]; then + contents="$(cat "${runtime_txt_path}")" + version="$(python_version::parse_runtime_txt "${contents}")" + origin="runtime.txt" + return 0 + fi + + if [[ "${package_manager}" == "pipenv" ]]; then + version="$(python_version::read_pipenv_python_version "${build_dir}")" + # The Python version fields in a Pipfile.lock are optional. + if [[ -n "${version}" ]]; then + origin="Pipfile.lock" + return 0 + fi + fi + + # Protect against invalid versions somehow having been written into the cache. + # TODO: Move this validation into the cache handling as part of the cache refactor? + if [[ "${cached_python_version}" =~ ^${PYTHON_VERSION_REGEX}$ ]]; then + version="${cached_python_version}" + origin="cached" + else + version="${DEFAULT_PYTHON_MAJOR_VERSION}" + # shellcheck disable=SC2034 # This isn't unused, Shellcheck doesn't follow namerefs. + origin="default" + fi +} + +# Parse the contents of a runtime.txt file and return the Python version substring (e.g. `3.12.0`). +function python_version::parse_runtime_txt() { + local contents="${1}" + + # The file must contain a string of form `python-N.N.N` (leading/trailing whitespace is permitted). + if [[ "${contents}" =~ ^[[:space:]]*python-(${PYTHON_FULL_VERSION_REGEX})[[:space:]]*$ ]]; then + local version="${BASH_REMATCH[1]}" + echo "${version}" + else + display_error <<-EOF + Error: Invalid Python version in runtime.txt. + + The Python version specified in 'runtime.txt' is not in + the correct format. + + The following file contents were found: + ${contents} + + However, the version string must begin with a 'python-' prefix, + followed by the version specified as '..'. + Comments are not supported. + + For example, to request Python ${DEFAULT_PYTHON_FULL_VERSION}, use: + python-${DEFAULT_PYTHON_FULL_VERSION} + + Please update 'runtime.txt' to use a valid version string, or + else remove the file to instead use the default version + (currently Python ${DEFAULT_PYTHON_FULL_VERSION}). + EOF + meta_set "failure_reason" "python-version-invalid" + return 1 + fi +} + +# Read the Python version from a Pipfile.lock, which can exist in one of two optional fields, +# `python_full_version` (as N.N.N) and `python_version` (as N.N). If both fields are +# defined, we will use the value set in `python_full_version`. See: +# https://pipenv.pypa.io/en/latest/specifiers.html#specifying-versions-of-python +function python_version::read_pipenv_python_version() { + local build_dir="${1}" + local pipfile_lock_path="${build_dir}/Pipfile.lock" + local version + + # We currently permit using Pipenv without a `Pipfile.lock`, however, in the future we will + # require a lockfile, at which point this conditional can be removed. + if [[ ! -f "${pipfile_lock_path}" ]]; then + return 0 + fi + + if ! version=$(jq --raw-output '._meta.requires.python_full_version // ._meta.requires.python_version' "${pipfile_lock_path}" 2>&1); then + display_error <<-EOF + Error: Cannot parse Pipfile.lock. + + A Pipfile.lock file was found, however, it could not be parsed: + ${version} + + This is likely due to it not being valid JSON. + + Run 'pipenv lock' to regenerate/fix the lockfile. + EOF + meta_set "failure_reason" "pipfile-lock-invalid" + return 1 + fi + + # Neither of the optional fields were found. + if [[ "${version}" == "null" ]]; then + return 0 + fi + + # We don't use separate version validation rules for both fields (e.g. ensuring a patch version + # only exists for `python_full_version`) since Pipenv doesn't distinguish between them either: + # https://github.com/pypa/pipenv/blob/v2024.1.0/pipenv/project.py#L392-L398 + if [[ "${version}" =~ ^${PYTHON_VERSION_REGEX}$ ]]; then + echo "${version}" + else + display_error <<-EOF + Error: Invalid Python version in Pipfile / Pipfile.lock. + + The Python version specified in Pipfile / Pipfile.lock by the + 'python_version' or 'python_full_version' field is not valid. + + The following version was found: + ${version} + + However, the version must be specified as either: + 1. '.' (recommended, for automatic security updates) + 2. '..' (to pin to an exact Python version) + + Please update your 'Pipfile' to use a valid Python version and + then run 'pipenv lock' to regenerate the lockfile. + + For more information, see: + https://pipenv.pypa.io/en/latest/specifiers.html#specifying-versions-of-python + EOF + meta_set "failure_reason" "python-version-invalid" + return 1 + fi +} + +# Resolve a requested Python version (which can be of form N.N or N.N.N) to a specific +# Python version of form N.N.N. Rejects Python major versions that are not supported. +function python_version::resolve_python_version() { + local requested_python_version="${1}" + local python_version_origin="${2}" + + if [[ ! "${requested_python_version}" =~ ^${PYTHON_VERSION_REGEX}$ ]]; then + # The Python version was previously validated, so this should never occur. + utils::abort_internal_error "Invalid Python version: ${requested_python_version}" + fi + + local major="${BASH_REMATCH[1]}" + local minor="${BASH_REMATCH[2]}" + + if ((major < 3 || (major == 3 && minor < 8))); then + if [[ "${python_version_origin}" == "cached" ]]; then + display_error <<-EOF + Error: The cached Python version has reached end-of-life. + + Your app does not specify a Python version, and so normally + would use the version cached from the last build (${requested_python_version}). + + However, Python ${major}.${minor} has reached its upstream end-of-life, + and is therefore no longer receiving security updates: + https://devguide.python.org/versions/#supported-versions + + As such, it is no longer supported by this buildpack. + + Please upgrade to a newer Python version by creating a + 'runtime.txt' file that contains a Python version like: + python-${DEFAULT_PYTHON_FULL_VERSION} + + For a list of the supported Python versions, see: + https://devcenter.heroku.com/articles/python-support#supported-runtimes + EOF + else + display_error <<-EOF + Error: The requested Python version has reached end-of-life. + + Python ${major}.${minor} has reached its upstream end-of-life, and is + therefore no longer receiving security updates: + https://devguide.python.org/versions/#supported-versions + + As such, it is no longer supported by this buildpack. + + Please upgrade to a newer Python version by updating the + version configured via the '${python_version_origin}' file. + + For a list of the supported Python versions, see: + https://devcenter.heroku.com/articles/python-support#supported-runtimes + EOF + fi + meta_set "failure_reason" "python-version-eol" + return 1 + fi + + if (((major == 3 && minor > 12) || major >= 4)); then + if [[ "${python_version_origin}" == "cached" ]]; then + display_error <<-EOF + Error: The cached Python version is not recognised. + + Your app does not specify a Python version, and so normally + would use the version cached from the last build (${requested_python_version}). + + However, Python ${major}.${minor} is not recognised by this version + of the buildpack. + + This can occur if you have downgraded the version of the + buildpack to an older version. + + Please switch back to a newer version of this buildpack. + EOF + else + display_error <<-EOF + Error: The requested Python version is not recognised. + + The requested Python version ${major}.${minor} is not recognised. + + Check that this Python version has been officially released, + and that the Python buildpack has added support for it: + https://devguide.python.org/versions/#supported-versions + https://devcenter.heroku.com/articles/python-support#supported-runtimes + + If it has, make sure that you are using the latest version + of this buildpack: + https://devcenter.heroku.com/articles/python-support#checking-the-python-buildpack-version + + Otherwise, switch to a supported version (such as Python ${DEFAULT_PYTHON_MAJOR_VERSION}) + by updating the version configured via the '${python_version_origin}' file. + EOF + fi + meta_set "failure_reason" "python-version-unknown" + return 1 + fi + + # If an exact Python version was requested, there's nothing to resolve. + # Otherwise map major version specifiers to the latest patch release. + case "${requested_python_version}" in + *.*.*) echo "${requested_python_version}" ;; + 3.8) echo "${LATEST_PYTHON_3_8}" ;; + 3.9) echo "${LATEST_PYTHON_3_9}" ;; + 3.10) echo "${LATEST_PYTHON_3_10}" ;; + 3.11) echo "${LATEST_PYTHON_3_11}" ;; + 3.12) echo "${LATEST_PYTHON_3_12}" ;; + *) utils::abort_internal_error "Unhandled Python major version: ${requested_python_version}" ;; + esac +} diff --git a/spec/fixtures/pipenv_python_3.6/Pipfile b/spec/fixtures/pipenv_lockfile_invalid_json/Pipfile similarity index 84% rename from spec/fixtures/pipenv_python_3.6/Pipfile rename to spec/fixtures/pipenv_lockfile_invalid_json/Pipfile index 62a39bb91..9bf60112c 100644 --- a/spec/fixtures/pipenv_python_3.6/Pipfile +++ b/spec/fixtures/pipenv_lockfile_invalid_json/Pipfile @@ -9,4 +9,4 @@ urllib3 = "*" [dev-packages] [requires] -python_version = "3.6" +python_version = "3.12" diff --git a/spec/fixtures/pipenv_lockfile_invalid_json/Pipfile.lock b/spec/fixtures/pipenv_lockfile_invalid_json/Pipfile.lock new file mode 100644 index 000000000..0e5491864 --- /dev/null +++ b/spec/fixtures/pipenv_lockfile_invalid_json/Pipfile.lock @@ -0,0 +1 @@ +INVALID JSON diff --git a/spec/fixtures/pipenv_python_3.6/Pipfile.lock b/spec/fixtures/pipenv_python_3.6/Pipfile.lock deleted file mode 100644 index 738941ddd..000000000 --- a/spec/fixtures/pipenv_python_3.6/Pipfile.lock +++ /dev/null @@ -1,29 +0,0 @@ -{ - "_meta": { - "hash": { - "sha256": "fe045e48581ffc6cb50af922ec04120b504f96ff6d6917fc535c8a9b76b8b71b" - }, - "pipfile-spec": 6, - "requires": { - "python_version": "3.6" - }, - "sources": [ - { - "name": "pypi", - "url": "https://pypi.org/simple", - "verify_ssl": true - } - ] - }, - "default": { - "urllib3": { - "hashes": [ - "sha256:076907bf8fd355cde77728471316625a4d2f7e713c125f51953bb5b3eecf4f72", - "sha256:75edcdc2f7d85b137124a6c3c9fc3933cdeaa12ecb9a6a959f22797a0feca7e1" - ], - "index": "pypi", - "version": "==1.26.14" - } - }, - "develop": {} -} diff --git a/spec/fixtures/pipenv_python_full_version/Pipfile b/spec/fixtures/pipenv_python_full_version/Pipfile index 88c5c59bc..aa0852315 100644 --- a/spec/fixtures/pipenv_python_full_version/Pipfile +++ b/spec/fixtures/pipenv_python_full_version/Pipfile @@ -9,4 +9,7 @@ urllib3 = "*" [dev-packages] [requires] -python_full_version = "3.12.2" +# Uses the oldest Python version supported by all stacks, to validate Pipenv works with it. +python_full_version = "3.9.0" +# Tests that `python_full_version` takes precedence. +python_version = "3.12" diff --git a/spec/fixtures/pipenv_python_full_version/Pipfile.lock b/spec/fixtures/pipenv_python_full_version/Pipfile.lock index 2346c8a33..888c9982e 100644 --- a/spec/fixtures/pipenv_python_full_version/Pipfile.lock +++ b/spec/fixtures/pipenv_python_full_version/Pipfile.lock @@ -1,11 +1,12 @@ { "_meta": { "hash": { - "sha256": "5f4af5700c245f61e22d09479627ca936d458a6aa693452337170f369c642f40" + "sha256": "068e8668ba7ae121efca9caf66d4796267713afe33c711e01a0852fe5e9a1848" }, "pipfile-spec": 6, "requires": { - "python_full_version": "3.12.2" + "python_full_version": "3.9.0", + "python_version": "3.12" }, "sources": [ { @@ -18,12 +19,12 @@ "default": { "urllib3": { "hashes": [ - "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", - "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" + "sha256:ca899ca043dcb1bafa3e262d73aa25c465bfb49e0bd9dd5d59f1d0acba2f8fac", + "sha256:e7d814a81dad81e6caf2ec9fdedb284ecc9c73076b62654547cc64ccdcae26e9" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==2.2.1" + "version": "==2.2.3" } }, "develop": {} diff --git a/spec/fixtures/pipenv_python_full_version_invalid/Pipfile b/spec/fixtures/pipenv_python_full_version_invalid/Pipfile index 683918cbd..1782422d3 100644 --- a/spec/fixtures/pipenv_python_full_version_invalid/Pipfile +++ b/spec/fixtures/pipenv_python_full_version_invalid/Pipfile @@ -9,4 +9,4 @@ urllib3 = "*" [dev-packages] [requires] -python_full_version = "X.Y.Z" +python_full_version = "3.9.*" diff --git a/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock b/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock index 3b2d178ab..107e6f348 100644 --- a/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock +++ b/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock @@ -1,11 +1,11 @@ { "_meta": { "hash": { - "sha256": "7efd722f93856d46f698123972037594c6b8621e54b5a2be8ce71f5707a13c33" + "sha256": "c629159445d228792003fe74526738c9155072a74a29f2d4920fc8aab878db0a" }, "pipfile-spec": 6, "requires": { - "python_full_version": "X.Y.Z" + "python_full_version": "3.9.*" }, "sources": [ { @@ -18,12 +18,12 @@ "default": { "urllib3": { "hashes": [ - "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", - "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" + "sha256:ca899ca043dcb1bafa3e262d73aa25c465bfb49e0bd9dd5d59f1d0acba2f8fac", + "sha256:e7d814a81dad81e6caf2ec9fdedb284ecc9c73076b62654547cc64ccdcae26e9" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==2.2.1" + "version": "==2.2.3" } }, "develop": {} diff --git a/spec/fixtures/python_3.10_outdated/requirements.txt b/spec/fixtures/python_2.7/requirements.txt similarity index 100% rename from spec/fixtures/python_3.10_outdated/requirements.txt rename to spec/fixtures/python_2.7/requirements.txt diff --git a/spec/fixtures/python_2.7/runtime.txt b/spec/fixtures/python_2.7/runtime.txt new file mode 100644 index 000000000..586b67310 --- /dev/null +++ b/spec/fixtures/python_2.7/runtime.txt @@ -0,0 +1 @@ +python-2.7.18 diff --git a/spec/fixtures/python_3.10_outdated/runtime.txt b/spec/fixtures/python_3.10_outdated/runtime.txt deleted file mode 100644 index fadb07024..000000000 --- a/spec/fixtures/python_3.10_outdated/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.10.0 diff --git a/spec/fixtures/python_3.11_outdated/runtime.txt b/spec/fixtures/python_3.11_outdated/runtime.txt deleted file mode 100644 index 335156c09..000000000 --- a/spec/fixtures/python_3.11_outdated/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.11.0 diff --git a/spec/fixtures/python_3.12_outdated/runtime.txt b/spec/fixtures/python_3.12_outdated/runtime.txt deleted file mode 100644 index 44f8fbe3a..000000000 --- a/spec/fixtures/python_3.12_outdated/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.12.0 diff --git a/spec/fixtures/python_3.6/requirements.txt b/spec/fixtures/python_3.6/requirements.txt deleted file mode 100644 index a42590beb..000000000 --- a/spec/fixtures/python_3.6/requirements.txt +++ /dev/null @@ -1 +0,0 @@ -urllib3 diff --git a/spec/fixtures/python_3.6/runtime.txt b/spec/fixtures/python_3.6/runtime.txt deleted file mode 100644 index 6aaad49f7..000000000 --- a/spec/fixtures/python_3.6/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.6.15 diff --git a/spec/fixtures/python_3.7/requirements.txt b/spec/fixtures/python_3.7/requirements.txt index a42590beb..e69de29bb 100644 --- a/spec/fixtures/python_3.7/requirements.txt +++ b/spec/fixtures/python_3.7/requirements.txt @@ -1 +0,0 @@ -urllib3 diff --git a/spec/fixtures/python_3.11_outdated/requirements.txt b/spec/fixtures/python_version_non_existent_major/requirements.txt similarity index 100% rename from spec/fixtures/python_3.11_outdated/requirements.txt rename to spec/fixtures/python_version_non_existent_major/requirements.txt diff --git a/spec/fixtures/python_version_non_existent_major/runtime.txt b/spec/fixtures/python_version_non_existent_major/runtime.txt new file mode 100644 index 000000000..b6885d9e1 --- /dev/null +++ b/spec/fixtures/python_version_non_existent_major/runtime.txt @@ -0,0 +1 @@ +python-3.999.0 diff --git a/spec/fixtures/python_3.12_outdated/requirements.txt b/spec/fixtures/python_version_non_existent_patch/requirements.txt similarity index 100% rename from spec/fixtures/python_3.12_outdated/requirements.txt rename to spec/fixtures/python_version_non_existent_patch/requirements.txt diff --git a/spec/fixtures/python_version_non_existent_patch/runtime.txt b/spec/fixtures/python_version_non_existent_patch/runtime.txt new file mode 100644 index 000000000..f9907fe4a --- /dev/null +++ b/spec/fixtures/python_version_non_existent_patch/runtime.txt @@ -0,0 +1 @@ +python-3.12.999 diff --git a/spec/fixtures/python_3.8_outdated/requirements.txt b/spec/fixtures/python_version_outdated/requirements.txt similarity index 100% rename from spec/fixtures/python_3.8_outdated/requirements.txt rename to spec/fixtures/python_version_outdated/requirements.txt diff --git a/spec/fixtures/python_3.9_outdated/runtime.txt b/spec/fixtures/python_version_outdated/runtime.txt similarity index 100% rename from spec/fixtures/python_3.9_outdated/runtime.txt rename to spec/fixtures/python_version_outdated/runtime.txt diff --git a/spec/fixtures/python_3.9_outdated/requirements.txt b/spec/fixtures/python_version_outdated_and_deprecated/requirements.txt similarity index 100% rename from spec/fixtures/python_3.9_outdated/requirements.txt rename to spec/fixtures/python_version_outdated_and_deprecated/requirements.txt diff --git a/spec/fixtures/python_3.8_outdated/runtime.txt b/spec/fixtures/python_version_outdated_and_deprecated/runtime.txt similarity index 100% rename from spec/fixtures/python_3.8_outdated/runtime.txt rename to spec/fixtures/python_version_outdated_and_deprecated/runtime.txt diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 4a29a6911..444b65e55 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -12,9 +12,9 @@ app.run_ci do |test_run| expect(test_run.output).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) -----> Python app detected - -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - -----> Installing python-#{DEFAULT_PYTHON_VERSION} + -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing SQLite3 -----> Installing requirements with pip @@ -70,10 +70,10 @@ expect(test_run.output).to include(<<~OUTPUT) -----> Python app detected - -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes -----> No change in requirements detected, installing from cache - -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing SQLite3 -----> Installing requirements with pip @@ -92,9 +92,9 @@ app.run_ci do |test_run| expect(test_run.output).to match(Regexp.new(<<~REGEX)) -----> Python app detected - -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - -----> Installing python-#{DEFAULT_PYTHON_VERSION} + -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing Pipenv #{PIPENV_VERSION} -----> Installing SQLite3 @@ -148,9 +148,9 @@ expect(test_run.output).to match(Regexp.new(<<~REGEX)) -----> Python app detected - -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing Pipenv #{PIPENV_VERSION} -----> Installing SQLite3 diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 34f37fae6..178be593d 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -22,9 +22,9 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip @@ -63,10 +63,10 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> No change in requirements detected, installing from cache - remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip @@ -86,10 +86,10 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Requirements file has been changed, clearing cached dependencies - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip @@ -118,9 +118,9 @@ app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip @@ -214,9 +214,9 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index c90092dbf..942bdb808 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -2,13 +2,13 @@ require_relative '../spec_helper' -RSpec.shared_examples 'builds using Pipenv with the requested Python version' do |python_version| - it "builds with Python #{python_version}" do +RSpec.shared_examples 'builds using Pipenv with the requested Python version' do |requested_version, resolved_version| + it "builds with Python #{requested_version}" do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock - remote: -----> Installing python-#{python_version} + remote: -----> Using Python #{requested_version} specified in Pipfile.lock + remote: -----> Installing Python #{resolved_version} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -19,24 +19,6 @@ end end -RSpec.shared_examples 'aborts the build with a runtime not available message (Pipenv)' do |requested_version| - it 'aborts the build with a runtime not available message' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock - remote: - remote: ! Error: Requested runtime 'python-#{requested_version}' is not available for this stack (#{app.stack}). - remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: - remote: ! Push rejected, failed to compile Python app. - OUTPUT - end - end -end - RSpec.describe 'Pipenv support' do context 'without a Pipfile.lock' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_no_lockfile') } @@ -46,9 +28,9 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected remote: ! No 'Pipfile.lock' found! We recommend you commit this into your repository. - remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -76,9 +58,9 @@ # TODO: We should not be leaking the Pipenv installation into the app environment. expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -120,52 +102,25 @@ end end - context 'with a Pipfile.lock containing python_version 3.6' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.6', allow_failure: true) } - - it 'aborts the build with an EOL message' do - app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~OUTPUT)) - remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock - remote: - remote: ! Error: Python 3.6 is no longer supported. - remote: ! - remote: ! Python 3.6 reached upstream end-of-life on December 23rd, 2021, and is - remote: ! therefore no longer receiving security updates: - remote: ! https://devguide.python.org/versions/#supported-versions - remote: ! - remote: ! As such, it is no longer supported by this buildpack. - remote: ! - remote: ! Please upgrade to a newer Python version. - remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: - remote: ! Push rejected, failed to compile Python app. - OUTPUT - end - end - end - - context 'with a Pipfile.lock containing python_version 3.7' do + context 'with a Pipfile.lock containing an EOL python_version' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.7', allow_failure: true) } it 'aborts the build with an EOL message' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~OUTPUT)) remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock + remote: -----> Using Python 3.7 specified in Pipfile.lock remote: - remote: ! Error: Python 3.7 is no longer supported. + remote: ! Error: The requested Python version has reached end-of-life. remote: ! - remote: ! Python 3.7 reached upstream end-of-life on June 27th, 2023, and is + remote: ! Python 3.7 has reached its upstream end-of-life, and is remote: ! therefore no longer receiving security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! remote: ! As such, it is no longer supported by this buildpack. remote: ! - remote: ! Please upgrade to a newer Python version. + remote: ! Please upgrade to a newer Python version by updating the + remote: ! version configured via the 'Pipfile.lock' file. remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes @@ -185,7 +140,7 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock + remote: -----> Using Python 3.8 specified in Pipfile.lock remote: ! remote: ! Python 3.8 will reach its upstream end-of-life in October 2024, at which remote: ! point it will no longer receive security updates: @@ -196,7 +151,7 @@ remote: ! Upgrade to a newer Python version as soon as possible to keep your app secure. remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! - remote: -----> Installing python-#{LATEST_PYTHON_3_8} + remote: -----> Installing Python #{LATEST_PYTHON_3_8} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -211,35 +166,53 @@ let(:allow_failure) { true } # We only support Python 3.8 on Heroku-20 and older. - include_examples 'aborts the build with a runtime not available message (Pipenv)', LATEST_PYTHON_3_8 + it 'aborts the build with a version not available message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.8 specified in Pipfile.lock + remote: + remote: ! Error: Python #{LATEST_PYTHON_3_8} is not available for this stack (#{app.stack}). + remote: ! + remote: ! For a list of the supported Python versions, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end end end context 'with a Pipfile.lock containing python_version 3.9' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.9') } - include_examples 'builds using Pipenv with the requested Python version', LATEST_PYTHON_3_9 + include_examples 'builds using Pipenv with the requested Python version', '3.9', LATEST_PYTHON_3_9 end context 'with a Pipfile.lock containing python_version 3.10' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.10') } - include_examples 'builds using Pipenv with the requested Python version', LATEST_PYTHON_3_10 + include_examples 'builds using Pipenv with the requested Python version', '3.10', LATEST_PYTHON_3_10 end context 'with a Pipfile.lock containing python_version 3.11' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.11') } - include_examples 'builds using Pipenv with the requested Python version', LATEST_PYTHON_3_11 + include_examples 'builds using Pipenv with the requested Python version', '3.11', LATEST_PYTHON_3_11 end context 'with a Pipfile.lock containing python_version 3.12' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.12') } - include_examples 'builds using Pipenv with the requested Python version', LATEST_PYTHON_3_12 + include_examples 'builds using Pipenv with the requested Python version', '3.12', LATEST_PYTHON_3_12 end - context 'with a Pipfile.lock containing python_full_version 3.12.2' do + # As well as testing `python_full_version`, this also tests: + # 1. That `python_full_version` takes precedence over `python_version`. + # 2. That Pipenv works on the oldest Python version supported by all stacks. + # 3. That the security update available message works for Pipenv too. + context 'with a Pipfile.lock containing python_full_version 3.9.0' do let(:allow_failure) { false } let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_full_version', allow_failure:) } @@ -247,12 +220,12 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock + remote: -----> Using Python 3.9.0 specified in Pipfile.lock remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: python-#{LATEST_PYTHON_3_12} + remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! - remote: -----> Installing python-3.12.2 + remote: -----> Installing Python 3.9.0 remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -263,20 +236,55 @@ end end - context 'with a Pipfile.lock containing an invalid python_version', - skip: 'unknown python_version values are currently ignored (W-8104668)' do + context 'with a Pipfile.lock containing invalid JSON' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_lockfile_invalid_json', allow_failure: true) } + + it 'fails the build' do + app.deploy do |app| + # The exact JQ error message varies between JQ versions, and thus across stacks. + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Python app detected + remote: + remote: ! Error: Cannot parse Pipfile.lock. + remote: ! + remote: ! A Pipfile.lock file was found, however, it could not be parsed: + remote: ! (jq: )?parse error: Invalid numeric literal at line 1, column 8 + remote: ! + remote: ! This is likely due to it not being valid JSON. + remote: ! + remote: ! Run 'pipenv lock' to regenerate/fix the lockfile. + remote: + remote: ! Push rejected, failed to compile Python app. + REGEX + end + end + end + + context 'with a Pipfile.lock containing an invalid python_version' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_version_invalid', allow_failure: true) } it 'fails the build' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock remote: - remote: ! Error: Requested runtime '^3.12' is not available for this stack (#{app.stack}). + remote: ! Error: Invalid Python version in Pipfile / Pipfile.lock. remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: ! The Python version specified in Pipfile / Pipfile.lock by the + remote: ! 'python_version' or 'python_full_version' field is not valid. + remote: ! + remote: ! The following version was found: + remote: ! ^3.12 + remote: ! + remote: ! However, the version must be specified as either: + remote: ! 1. '.' (recommended, for automatic security updates) + remote: ! 2. '..' (to pin to an exact Python version) + remote: ! + remote: ! Please update your 'Pipfile' to use a valid Python version and + remote: ! then run 'pipenv lock' to regenerate the lockfile. + remote: ! + remote: ! For more information, see: + remote: ! https://pipenv.pypa.io/en/latest/specifiers.html#specifying-versions-of-python remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -291,12 +299,24 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock remote: - remote: ! Error: Requested runtime 'python-X.Y.Z' is not available for this stack (#{app.stack}). + remote: ! Error: Invalid Python version in Pipfile / Pipfile.lock. remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: ! The Python version specified in Pipfile / Pipfile.lock by the + remote: ! 'python_version' or 'python_full_version' field is not valid. + remote: ! + remote: ! The following version was found: + remote: ! 3.9.* + remote: ! + remote: ! However, the version must be specified as either: + remote: ! 1. '.' (recommended, for automatic security updates) + remote: ! 2. '..' (to pin to an exact Python version) + remote: ! + remote: ! Please update your 'Pipfile' to use a valid Python version and + remote: ! then run 'pipenv lock' to regenerate the lockfile. + remote: ! + remote: ! For more information, see: + remote: ! https://pipenv.pypa.io/en/latest/specifiers.html#specifying-versions-of-python remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -311,8 +331,8 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt - remote: -----> Installing python-#{LATEST_PYTHON_3_12} + remote: -----> Using Python #{LATEST_PYTHON_3_12} specified in runtime.txt + remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -332,9 +352,9 @@ app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -359,9 +379,9 @@ app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -380,8 +400,8 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock - remote: -----> Installing python-#{LATEST_PYTHON_3_12} + remote: -----> Using Python 3.12 specified in Pipfile.lock + remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -399,9 +419,9 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index 284544c91..787f24a18 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -2,54 +2,20 @@ require_relative '../spec_helper' -RSpec.shared_examples 'warns there is a Python update available' do |requested_version, latest_version| - it 'warns there is a Python update available' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt - remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: python-#{latest_version} - remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: ! - remote: -----> Installing python-#{requested_version} - OUTPUT - end - end -end - -RSpec.shared_examples 'aborts the build without showing an update warning' do |requested_version| - it 'aborts the build without showing an update warning' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt - remote: - remote: ! Error: Requested runtime 'python-#{requested_version}' is not available for this stack (#{app.stack}). - remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: - remote: ! Push rejected, failed to compile Python app. - OUTPUT - end - end -end - # NOTE: We use the oldest patch releases (ie the '.0' releases) since we also want to test against # the oldest Python versions available to users. This is particularly important given that older # patch releases will bundle older pip, and the buildpack uses that pip during bootstrapping. RSpec.describe 'Python update warnings' do - context 'with a runtime.txt containing python-3.8.0' do + context 'with a runtime.txt containing an outdated patch version that is also a deprecated major version' do let(:allow_failure) { false } - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.8_outdated', allow_failure:) } + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_outdated_and_deprecated', allow_failure:) } context 'when using Heroku-20', stacks: %w[heroku-20] do it 'warns about both the deprecated major version and the patch update' do app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt + remote: -----> Using Python 3.8.0 specified in runtime.txt remote: ! remote: ! Python 3.8 will reach its upstream end-of-life in October 2024, at which remote: ! point it will no longer receive security updates: @@ -61,11 +27,11 @@ remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: python-#{LATEST_PYTHON_3_8} + remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_8} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! - remote: -----> Installing python-3.8.0 - REGEX + remote: -----> Installing Python 3.8.0 + OUTPUT end end end @@ -74,31 +40,39 @@ let(:allow_failure) { true } # We only support Python 3.8 on Heroku-20 and older. - include_examples 'aborts the build without showing an update warning', '3.8.0' + it 'aborts the build without showing an update warning' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.8.0 specified in runtime.txt + remote: + remote: ! Error: Python 3.8.0 is not available for this stack (#{app.stack}). + remote: ! + remote: ! For a list of the supported Python versions, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end end end - context 'with a runtime.txt containing python-3.9.0' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.9_outdated') } - - include_examples 'warns there is a Python update available', '3.9.0', LATEST_PYTHON_3_9 - end - - context 'with a runtime.txt containing python-3.10.0' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.10_outdated') } - - include_examples 'warns there is a Python update available', '3.10.0', LATEST_PYTHON_3_10 - end - - context 'with a runtime.txt containing python-3.11.0' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.11_outdated') } - - include_examples 'warns there is a Python update available', '3.11.0', LATEST_PYTHON_3_11 - end - - context 'with a runtime.txt containing python-3.12.0' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.12_outdated') } + context 'with a runtime.txt containing an outdated patch version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_outdated') } - include_examples 'warns there is a Python update available', '3.12.0', LATEST_PYTHON_3_12 + it 'warns there is a Python update available' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.9.0 specified in runtime.txt + remote: ! + remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} + remote: ! See: https://devcenter.heroku.com/articles/python-runtimes + remote: ! + remote: -----> Installing Python 3.9.0 + OUTPUT + end + end end end diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 44038a080..e0d01049d 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -2,37 +2,19 @@ require_relative '../spec_helper' -RSpec.shared_examples 'builds with the requested Python version' do |python_version| - it "builds with Python #{python_version}" do +RSpec.shared_examples 'builds with the requested Python version' do |requested_version| + it "builds with Python #{requested_version}" do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt - remote: -----> Installing python-#{python_version} + remote: -----> Using Python #{requested_version} specified in runtime.txt + remote: -----> Installing Python #{requested_version} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip remote: Collecting urllib3 (from -r requirements.txt (line 1)) OUTPUT - expect(app.run('python -V')).to include("Python #{python_version}") - end - end -end - -RSpec.shared_examples 'aborts the build with a runtime not available message' do |requested_runtime| - it 'aborts the build with a runtime not available message' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt - remote: - remote: ! Error: Requested runtime '#{requested_runtime}' is not available for this stack (#{app.stack}). - remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: - remote: ! Push rejected, failed to compile Python app. - OUTPUT + expect(app.run('python -V')).to include("Python #{requested_version}") end end end @@ -47,9 +29,9 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} OUTPUT end end @@ -68,14 +50,14 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-3.12.3 + remote: -----> No Python version was specified. Using the same version as the last build: Python 3.12.3 remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: python-#{LATEST_PYTHON_3_12} + remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_12} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! remote: -----> No change in requirements detected, installing from cache - remote: -----> Using cached install of python-3.12.3 + remote: -----> Using cached install of Python 3.12.3 OUTPUT expect(app.run('python -V')).to include('Python 3.12.3') end @@ -83,24 +65,25 @@ end end - context 'when runtime.txt contains python-3.6.15' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.6', allow_failure: true) } + context 'when runtime.txt contains python-2.7.18' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_2.7', allow_failure: true) } it 'aborts the build with an EOL message' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt + remote: -----> Using Python 2.7.18 specified in runtime.txt remote: - remote: ! Error: Python 3.6 is no longer supported. + remote: ! Error: The requested Python version has reached end-of-life. remote: ! - remote: ! Python 3.6 reached upstream end-of-life on December 23rd, 2021, and is + remote: ! Python 2.7 has reached its upstream end-of-life, and is remote: ! therefore no longer receiving security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! remote: ! As such, it is no longer supported by this buildpack. remote: ! - remote: ! Please upgrade to a newer Python version. + remote: ! Please upgrade to a newer Python version by updating the + remote: ! version configured via the 'runtime.txt' file. remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes @@ -118,17 +101,18 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt + remote: -----> Using Python 3.7.17 specified in runtime.txt remote: - remote: ! Error: Python 3.7 is no longer supported. + remote: ! Error: The requested Python version has reached end-of-life. remote: ! - remote: ! Python 3.7 reached upstream end-of-life on June 27th, 2023, and is + remote: ! Python 3.7 has reached its upstream end-of-life, and is remote: ! therefore no longer receiving security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! remote: ! As such, it is no longer supported by this buildpack. remote: ! - remote: ! Please upgrade to a newer Python version. + remote: ! Please upgrade to a newer Python version by updating the + remote: ! version configured via the 'runtime.txt' file. remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes @@ -148,7 +132,7 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt + remote: -----> Using Python #{LATEST_PYTHON_3_8} specified in runtime.txt remote: ! remote: ! Python 3.8 will reach its upstream end-of-life in October 2024, at which remote: ! point it will no longer receive security updates: @@ -159,7 +143,7 @@ remote: ! Upgrade to a newer Python version as soon as possible to keep your app secure. remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! - remote: -----> Installing python-#{LATEST_PYTHON_3_8} + remote: -----> Installing Python #{LATEST_PYTHON_3_8} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip @@ -174,7 +158,21 @@ let(:allow_failure) { true } # We only support Python 3.8 on Heroku-20 and older. - include_examples 'aborts the build with a runtime not available message', "python-#{LATEST_PYTHON_3_8}" + it 'aborts the build with a version not available message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python #{LATEST_PYTHON_3_8} specified in runtime.txt + remote: + remote: ! Error: Python #{LATEST_PYTHON_3_8} is not available for this stack (#{app.stack}). + remote: ! + remote: ! For a list of the supported Python versions, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end end end @@ -202,10 +200,88 @@ include_examples 'builds with the requested Python version', LATEST_PYTHON_3_12 end - context 'when runtime.txt contains an invalid python version string' do + context 'when runtime.txt contains an invalid Python version string' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_invalid', allow_failure: true) } - include_examples 'aborts the build with a runtime not available message', 'python-3.12.0invalid' + it 'aborts the build with an invalid runtime.txt message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: + remote: ! Error: Invalid Python version in runtime.txt. + remote: ! + remote: ! The Python version specified in 'runtime.txt' is not in + remote: ! the correct format. + remote: ! + remote: ! The following file contents were found: + remote: ! python-3.12.0invalid + remote: ! + remote: ! However, the version string must begin with a 'python-' prefix, + remote: ! followed by the version specified as '..'. + remote: ! Comments are not supported. + remote: ! + remote: ! For example, to request Python 3.12.7, use: + remote: ! python-3.12.7 + remote: ! + remote: ! Please update 'runtime.txt' to use a valid version string, or + remote: ! else remove the file to instead use the default version + remote: ! (currently Python 3.12.7). + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + context 'when runtime.txt contains an non-existent Python major version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_non_existent_major', allow_failure: true) } + + it 'aborts the build with an invalid runtime.txt message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.999.0 specified in runtime.txt + remote: + remote: ! Error: The requested Python version is not recognised. + remote: ! + remote: ! The requested Python version 3.999 is not recognised. + remote: ! + remote: ! Check that this Python version has been officially released, + remote: ! and that the Python buildpack has added support for it: + remote: ! https://devguide.python.org/versions/#supported-versions + remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: ! + remote: ! If it has, make sure that you are using the latest version + remote: ! of this buildpack: + remote: ! https://devcenter.heroku.com/articles/python-support#checking-the-python-buildpack-version + remote: ! + remote: ! Otherwise, switch to a supported version (such as Python 3.12) + remote: ! by updating the version configured via the 'runtime.txt' file. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + context 'when runtime.txt contains a non-existent Python patch version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_non_existent_patch', allow_failure: true) } + + it 'aborts the build with a version not available message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.12.999 specified in runtime.txt + remote: + remote: ! Error: Python 3.12.999 is not available for this stack (#{app.stack}). + remote: ! + remote: ! For a list of the supported Python versions, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end end context 'when runtime.txt contains stray whitespace' do @@ -215,7 +291,7 @@ end context 'when the requested Python version has changed since the last build' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.11') } + let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.9') } it 'builds with the new Python version after removing the old install' do app.deploy do |app| @@ -225,10 +301,10 @@ # TODO: The output shouldn't say "installing from cache", since it's not. expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt - remote: -----> Python version has changed from python-#{LATEST_PYTHON_3_11} to python-#{LATEST_PYTHON_3_12}, clearing cache + remote: -----> Using Python #{LATEST_PYTHON_3_12} specified in runtime.txt + remote: -----> Python version has changed from #{LATEST_PYTHON_3_9} to #{LATEST_PYTHON_3_12}, clearing cache remote: -----> No change in requirements detected, installing from cache - remote: -----> Installing python-#{LATEST_PYTHON_3_12} + remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index 35f05dfbf..0135e0628 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -22,15 +22,15 @@ # TODO: The requirements output shouldn't say "installing from cache", since it's not. expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-3.12.3 + remote: -----> No Python version was specified. Using the same version as the last build: Python 3.12.3 remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: python-#{LATEST_PYTHON_3_12} + remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_12} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! remote: -----> Stack has changed from heroku-22 to heroku-24, clearing cache remote: -----> No change in requirements detected, installing from cache - remote: -----> Installing python-3.12.3 + remote: -----> Installing Python 3.12.3 remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip @@ -51,11 +51,11 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Stack has changed from heroku-24 to heroku-22, clearing cache remote: -----> No change in requirements detected, installing from cache - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index cfbb95da1..3c0632a64 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -14,7 +14,8 @@ LATEST_PYTHON_3_10 = '3.10.15' LATEST_PYTHON_3_11 = '3.11.10' LATEST_PYTHON_3_12 = '3.12.7' -DEFAULT_PYTHON_VERSION = LATEST_PYTHON_3_12 +DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_12 +DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') # The requirement versions are effectively buildpack constants, however, we want # Dependabot to be able to update them, which requires that they be in requirements From 5a6ec71813bc26fb8bb4614f3ac26db11c5785e0 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 9 Oct 2024 08:47:44 +0000 Subject: [PATCH 034/406] Prepare release v259 (#1659) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d0fd0fd39..51ee3d60d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v259] - 2024-10-09 + - Improved build log output about the detected Python version. ([#1658](https://github.com/heroku/heroku-buildpack-python/pull/1658)) - Improved error messages shown when the requested Python version is not a valid version string or is for an unknown/non-existent major Python version. ([#1658](https://github.com/heroku/heroku-buildpack-python/pull/1658)) - Improved error messages shown when `Pipfile.lock` is not valid JSON. ([#1658](https://github.com/heroku/heroku-buildpack-python/pull/1658)) @@ -1055,7 +1058,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v258...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v259...main +[v259]: https://github.com/heroku/heroku-buildpack-python/compare/v258...v259 [v258]: https://github.com/heroku/heroku-buildpack-python/compare/v257...v258 [v257]: https://github.com/heroku/heroku-buildpack-python/compare/v256...v257 [v256]: https://github.com/heroku/heroku-buildpack-python/compare/v255...v256 From dd4d7591265e38965a3ee4dbb17c8fcca1e3bd7f Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 10 Oct 2024 20:07:57 +0000 Subject: [PATCH 035/406] Add support for Python 3.13 (#1661) Add support for Python 3.13, and release 3.13.0. The default Python version remains unchanged (at 3.12.x) for now. Notably for Python 3.13 we now: - No longer install setuptools and wheel - matching what the wider ecosystem has already done for Python 3.12+. (See the Python CNB's removal PR for more details: https://github.com/heroku/buildpacks-python/pull/243) - No longer install the SQLite headers and CLI, as the first step towards dropping that rarely used feature. - Have to disable some of the tests that run during the PGO profiling phase when on Heroku-22 (see PR comments for more details). In addition, for all Python versions we now also remove the `idle3` and `pydoc3` scripts, since they do not work with relocated Python and so have been broken for some time. Their functionality continues to be available by invoking them via their modules instead (e.g. `python -m pydoc`). Release announcement: https://blog.python.org/2024/10/python-3130-final-released.html https://www.python.org/downloads/release/python-3130/ Details on what's new in Python 3.13: https://docs.python.org/3.13/whatsnew/3.13.html Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/11280580537 Python 3.13 readiness status of the top 360 packages on PyPI: https://pyreadiness.org/3.13/ GUS-W-14846826. GUS-W-14846839. GUS-W-16944574. --- CHANGELOG.md | 2 ++ README.md | 1 + bin/compile | 15 ++++---- builds/Dockerfile | 2 +- builds/build_python_runtime.sh | 32 ++++++++++++++--- ....13-ubuntu-22.04-libexpat-workaround.patch | 24 +++++++++++++ builds/test_python_runtime.sh | 22 +++++++++++- lib/pip.sh | 34 ++++++++++++++---- lib/python_version.sh | 4 ++- spec/fixtures/pipenv_python_3.13/Pipfile | 12 +++++++ spec/fixtures/pipenv_python_3.13/Pipfile.lock | 30 ++++++++++++++++ spec/fixtures/python_3.13/requirements.txt | 1 + spec/fixtures/python_3.13/runtime.txt | 1 + spec/hatchet/pipenv_spec.rb | 18 ++++++++++ spec/hatchet/python_version_spec.rb | 35 ++++++++++++++----- spec/spec_helper.rb | 1 + 16 files changed, 206 insertions(+), 28 deletions(-) create mode 100644 builds/python-3.13-ubuntu-22.04-libexpat-workaround.patch create mode 100644 spec/fixtures/pipenv_python_3.13/Pipfile create mode 100644 spec/fixtures/pipenv_python_3.13/Pipfile.lock create mode 100644 spec/fixtures/python_3.13/requirements.txt create mode 100644 spec/fixtures/python_3.13/runtime.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index 51ee3d60d..67e7d1e01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## [Unreleased] +- Added support for Python 3.13. ([#1661](https://github.com/heroku/heroku-buildpack-python/pull/1661)) +- Removed the `idle3` and `pydoc3` scripts since they do not work with relocated Python and so have been broken for some time. Invoke them via their modules instead (e.g. `python -m pydoc`). ([#1661](https://github.com/heroku/heroku-buildpack-python/pull/1661)) ## [v259] - 2024-10-09 diff --git a/README.md b/README.md index 3c78ac632..f590addf9 100644 --- a/README.md +++ b/README.md @@ -60,6 +60,7 @@ Specify a Python Runtime Supported runtime options include: +- `python-3.13.0` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) - `python-3.12.7` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) - `python-3.11.10` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) - `python-3.10.15` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) diff --git a/bin/compile b/bin/compile index 12c74f3b6..bc6621c15 100755 --- a/bin/compile +++ b/bin/compile @@ -201,11 +201,11 @@ package_manager_install_start_time=$(nowms) bundled_pip_module_path="$(utils::bundled_pip_module_path "${BUILD_DIR}")" case "${package_manager}" in pip) - pip::install_pip_setuptools_wheel "${bundled_pip_module_path}" + pip::install_pip_setuptools_wheel "${bundled_pip_module_path}" "${python_major_version}" ;; pipenv) # TODO: Stop installing pip when using Pipenv. - pip::install_pip_setuptools_wheel "${bundled_pip_module_path}" + pip::install_pip_setuptools_wheel "${bundled_pip_module_path}" "${python_major_version}" pipenv::install_pipenv ;; *) @@ -217,10 +217,13 @@ meta_time "package_manager_install_duration" "${package_manager_install_start_ti # SQLite3 support. # Installs the sqlite3 dev headers and sqlite3 binary but not the # libsqlite3-0 library since that exists in the base image. -install_sqlite_start_time=$(nowms) -source "${BUILDPACK_DIR}/bin/steps/sqlite3" -buildpack_sqlite3_install -meta_time "sqlite_install_duration" "${install_sqlite_start_time}" +# We skip this step on Python 3.13, as a first step towards removing this feature. +if [[ "${python_major_version}" == +(3.8|3.9|3.10|3.11|3.12) ]]; then + install_sqlite_start_time=$(nowms) + source "${BUILDPACK_DIR}/bin/steps/sqlite3" + buildpack_sqlite3_install + meta_time "sqlite_install_duration" "${install_sqlite_start_time}" +fi # Install app dependencies. dependencies_install_start_time=$(nowms) diff --git a/builds/Dockerfile b/builds/Dockerfile index f3fcfffb7..cc16e9b25 100644 --- a/builds/Dockerfile +++ b/builds/Dockerfile @@ -15,4 +15,4 @@ RUN apt-get update --error-on=any \ && rm -rf /var/lib/apt/lists/* WORKDIR /tmp -COPY build_python_runtime.sh . +COPY build_python_runtime.sh python-3.13-ubuntu-22.04-libexpat-workaround.patch . diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index ea5c0eb29..3da58a09c 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -27,6 +27,7 @@ case "${STACK:?}" in "3.10" "3.11" "3.12" + "3.13" ) ;; heroku-20) @@ -36,6 +37,7 @@ case "${STACK:?}" in "3.10" "3.11" "3.12" + "3.13" ) ;; *) @@ -49,6 +51,10 @@ fi # The release keys can be found on https://www.python.org/downloads/ -> "OpenPGP Public Keys". case "${PYTHON_MAJOR_VERSION}" in + 3.13) + # https://github.com/Yhg1s.gpg + GPG_KEY_FINGERPRINT='7169605F62C751356D054A26A821E680E5FA6305' + ;; 3.12) # https://github.com/Yhg1s.gpg GPG_KEY_FINGERPRINT='7169605F62C751356D054A26A821E680E5FA6305' @@ -84,6 +90,14 @@ gpg --batch --verify python.tgz.asc python.tgz tar --extract --file python.tgz --strip-components=1 --directory "${SRC_DIR}" cd "${SRC_DIR}" +# Work around PGO profile test failures with Python 3.13 on Ubuntu 22.04, due to the tests +# checking the raw libexpat version which doesn't account for Ubuntu backports: +# https://github.com/heroku/heroku-buildpack-python/pull/1661#issuecomment-2405259352 +# https://github.com/python/cpython/issues/125067 +if [[ "${PYTHON_MAJOR_VERSION}" == "3.13" && "${STACK}" == "heroku-22" ]]; then + patch -p1 12) || major >= 4)); then + if (((major == 3 && minor > 13) || major >= 4)); then if [[ "${python_version_origin}" == "cached" ]]; then display_error <<-EOF Error: The cached Python version is not recognised. @@ -281,6 +282,7 @@ function python_version::resolve_python_version() { 3.10) echo "${LATEST_PYTHON_3_10}" ;; 3.11) echo "${LATEST_PYTHON_3_11}" ;; 3.12) echo "${LATEST_PYTHON_3_12}" ;; + 3.13) echo "${LATEST_PYTHON_3_13}" ;; *) utils::abort_internal_error "Unhandled Python major version: ${requested_python_version}" ;; esac } diff --git a/spec/fixtures/pipenv_python_3.13/Pipfile b/spec/fixtures/pipenv_python_3.13/Pipfile new file mode 100644 index 000000000..73f50fc86 --- /dev/null +++ b/spec/fixtures/pipenv_python_3.13/Pipfile @@ -0,0 +1,12 @@ +[[source]] +url = "https://pypi.org/simple" +verify_ssl = true +name = "pypi" + +[packages] +urllib3 = "*" + +[dev-packages] + +[requires] +python_version = "3.13" diff --git a/spec/fixtures/pipenv_python_3.13/Pipfile.lock b/spec/fixtures/pipenv_python_3.13/Pipfile.lock new file mode 100644 index 000000000..dcd0b3ca7 --- /dev/null +++ b/spec/fixtures/pipenv_python_3.13/Pipfile.lock @@ -0,0 +1,30 @@ +{ + "_meta": { + "hash": { + "sha256": "60dc67ee4223a391c5e0ae2b4d7ea54a7b245773d76b6ff82156dda97a3e4fb2" + }, + "pipfile-spec": 6, + "requires": { + "python_version": "3.13" + }, + "sources": [ + { + "name": "pypi", + "url": "https://pypi.org/simple", + "verify_ssl": true + } + ] + }, + "default": { + "urllib3": { + "hashes": [ + "sha256:ca899ca043dcb1bafa3e262d73aa25c465bfb49e0bd9dd5d59f1d0acba2f8fac", + "sha256:e7d814a81dad81e6caf2ec9fdedb284ecc9c73076b62654547cc64ccdcae26e9" + ], + "index": "pypi", + "markers": "python_version >= '3.8'", + "version": "==2.2.3" + } + }, + "develop": {} +} diff --git a/spec/fixtures/python_3.13/requirements.txt b/spec/fixtures/python_3.13/requirements.txt new file mode 100644 index 000000000..a42590beb --- /dev/null +++ b/spec/fixtures/python_3.13/requirements.txt @@ -0,0 +1 @@ +urllib3 diff --git a/spec/fixtures/python_3.13/runtime.txt b/spec/fixtures/python_3.13/runtime.txt new file mode 100644 index 000000000..dae7fec92 --- /dev/null +++ b/spec/fixtures/python_3.13/runtime.txt @@ -0,0 +1 @@ +python-3.13.0 diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 942bdb808..91d4fbb1a 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -208,6 +208,24 @@ include_examples 'builds using Pipenv with the requested Python version', '3.12', LATEST_PYTHON_3_12 end + context 'with a Pipfile.lock containing python_version 3.13' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.13') } + + it 'builds with latest Python 3.13' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Python app detected + remote: -----> Using Python 3.13 specified in Pipfile.lock + remote: -----> Installing Python #{LATEST_PYTHON_3_13} + remote: -----> Installing pip #{PIP_VERSION} + remote: -----> Installing Pipenv #{PIPENV_VERSION} + remote: -----> Installing dependencies with Pipenv + remote: Installing dependencies from Pipfile.lock \\(.+\\)... + REGEX + end + end + end + # As well as testing `python_full_version`, this also tests: # 1. That `python_full_version` takes precedence over `python_version`. # 2. That Pipenv works on the oldest Python version supported by all stacks. diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index e0d01049d..e9751a920 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -5,15 +5,26 @@ RSpec.shared_examples 'builds with the requested Python version' do |requested_version| it "builds with Python #{requested_version}" do app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python #{requested_version} specified in runtime.txt - remote: -----> Installing Python #{requested_version} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing SQLite3 - remote: -----> Installing requirements with pip - remote: Collecting urllib3 (from -r requirements.txt (line 1)) - OUTPUT + if requested_version.start_with?('3.13.') + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python #{requested_version} specified in runtime.txt + remote: -----> Installing Python #{requested_version} + remote: -----> Installing pip #{PIP_VERSION} + remote: -----> Installing requirements with pip + remote: Collecting urllib3 (from -r requirements.txt (line 1)) + OUTPUT + else + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python #{requested_version} specified in runtime.txt + remote: -----> Installing Python #{requested_version} + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing SQLite3 + remote: -----> Installing requirements with pip + remote: Collecting urllib3 (from -r requirements.txt (line 1)) + OUTPUT + end expect(app.run('python -V')).to include("Python #{requested_version}") end end @@ -200,6 +211,12 @@ include_examples 'builds with the requested Python version', LATEST_PYTHON_3_12 end + context 'when runtime.txt contains python-3.13.0' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.13') } + + include_examples 'builds with the requested Python version', LATEST_PYTHON_3_13 + end + context 'when runtime.txt contains an invalid Python version string' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_invalid', allow_failure: true) } diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 3c0632a64..4fddb3764 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -14,6 +14,7 @@ LATEST_PYTHON_3_10 = '3.10.15' LATEST_PYTHON_3_11 = '3.11.10' LATEST_PYTHON_3_12 = '3.12.7' +LATEST_PYTHON_3_13 = '3.13.0' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_12 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From a2cf258ef18ce0ffce2b9a8f8ae33d91c645ddee Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 10 Oct 2024 20:17:50 +0000 Subject: [PATCH 036/406] Prepare release v260 (#1662) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 67e7d1e01..aada1653b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v260] - 2024-10-10 + - Added support for Python 3.13. ([#1661](https://github.com/heroku/heroku-buildpack-python/pull/1661)) - Removed the `idle3` and `pydoc3` scripts since they do not work with relocated Python and so have been broken for some time. Invoke them via their modules instead (e.g. `python -m pydoc`). ([#1661](https://github.com/heroku/heroku-buildpack-python/pull/1661)) @@ -1060,7 +1063,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v259...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v260...main +[v260]: https://github.com/heroku/heroku-buildpack-python/compare/v259...v260 [v259]: https://github.com/heroku/heroku-buildpack-python/compare/v258...v259 [v258]: https://github.com/heroku/heroku-buildpack-python/compare/v257...v258 [v257]: https://github.com/heroku/heroku-buildpack-python/compare/v256...v257 From 737bba13baa110e9f6dcc79b0bdea3fa2875ccd5 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 14 Oct 2024 13:53:23 +0000 Subject: [PATCH 037/406] Create a GitHub Release when publishing a new buildpack version (#1663) Previously when a new buildpack version was published only a Git tag was created. Now a GitHub release will be too, which contains the release's changelog entry extracted from `CHANGELOG.md`. This change makes it possible for people to subscribe to just the repository release event notifications, rather than needing to subscribe to all pull requests events and manually spot release PRs being opened/merged. The GitHub CLI is used to create the release, which also creates the Git tag - meaning the manual Git revision/tag/push steps are no longer required. In addition, some environment/release validation checks have been added, which: - Improve the onboarding UX if required tools aren't installed. - Prevent accidentally running the publish step before having triggered/merged the prepare release PR workflow. - Prevent version mismatch issues if the buildpack were ever rolled back on the registry. See: https://cli.github.com/manual/gh_release_view https://cli.github.com/manual/gh_release_create https://github.com/heroku/libcnb.rs/blob/ba5e8231b88f6f72533e8a9ba01d9745b3f40301/.github/workflows/release.yml#L52-L60 GUS-W-16959313. --- etc/publish.sh | 67 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 52 insertions(+), 15 deletions(-) diff --git a/etc/publish.sh b/etc/publish.sh index 1a445e6d7..d40b12b85 100755 --- a/etc/publish.sh +++ b/etc/publish.sh @@ -1,28 +1,65 @@ #!/usr/bin/env bash -# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. set -euo pipefail shopt -s inherit_errexit -BP_NAME=${1:-"heroku/python"} +buildpack_registry_name="heroku/python" -curVersion=$(heroku buildpacks:versions "$BP_NAME" | awk 'FNR == 3 { print $1 }') -newVersion="v$((curVersion + 1))" +function abort() { + echo + echo "Error: ${1}" >&2 + exit 1 +} -read -r -p "Deploy as version: $newVersion [y/n]? " choice -case "$choice" in - y | Y) echo "" ;; +echo "Checking environment..." + +if ! command -v gh >/dev/null; then + abort "Install the GitHub CLI first: https://cli.github.com" +fi + +if ! heroku buildpacks:publish --help >/dev/null; then + abort "Install the Buildpack Registry plugin first: https://github.com/heroku/plugin-buildpack-registry" +fi + +# Explicitly check the CLI is logged in, since the Buildpack Registry plugin doesn't handle re-authing +# expired logins properly, which can otherwise lead to the release aborting partway through. +if ! heroku whoami >/dev/null; then + abort "Log into the Heroku CLI first: heroku login" +fi + +echo "Checking buildpack versions in sync..." +current_github_release_version=$(gh release view --json tagName --jq '.tagName' | tr --delete 'v') +current_registry_version="$(heroku buildpacks:versions "${buildpack_registry_name}" | awk 'FNR == 3 { print $1 }')" + +if [[ "${current_github_release_version}" != "${current_registry_version}" ]]; then + abort "The current GitHub release version (v${current_github_release_version}) does not match the registry version (v${current_registry_version}), likely due to a registry rollback. Fix this first!" +fi + +new_version="$((current_github_release_version + 1))" +new_git_tag="v${new_version}" + +echo "Extracting changelog entry for this release..." +git fetch origin +# Using `git show` to avoid having to disrupt the current branch/working directory. +changelog_entry="$(git show origin/main:CHANGELOG.md | awk "/^## \[v${new_version}\]/{flag=1; next} /^## /{flag=0} flag")" + +if [[ -n "${changelog_entry}" ]]; then + echo -e "${changelog_entry}\n" +else + abort "Unable to find changelog entry for v${new_version}. Has the prepare release PR been triggered/merged?" +fi + +read -r -p "Deploy as ${new_git_tag} [y/n]? " choice +case "${choice}" in + y | Y) ;; n | N) exit 0 ;; *) exit 1 ;; esac -git fetch origin -originMain=$(git rev-parse origin/main) -echo "Tagging commit $originMain with $newVersion... " -git tag "$newVersion" "${originMain:?}" -git push origin "refs/tags/${newVersion}" +echo -e "\nCreating GitHub release..." +gh release create "${new_git_tag}" --title "${new_git_tag}" --notes "${changelog_entry}" -echo -e "\nPublishing to the buildpack registry..." -heroku buildpacks:publish "$BP_NAME" "$newVersion" +echo -e "\nPublishing to the Heroku Buildpack Registry..." +heroku buildpacks:publish "${buildpack_registry_name}" "${new_git_tag}" echo -heroku buildpacks:versions "${BP_NAME}" | head -n 3 +heroku buildpacks:versions "${buildpack_registry_name}" | head -n 3 From d9bdeabd7bd03c5d59abafbbdb85e836a754844a Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 14 Oct 2024 14:04:36 +0000 Subject: [PATCH 038/406] Add support for the `.python-version` file (#1664) Following on from the recent Python version refactoring, this now adds support for configuring the app's Python version using a `.python-version` file. This file is used by several tools in the Python ecosystem (such as pyenv, `actions/setup-python`, uv), whereas the existing `runtime.txt` file is proprietary to Heroku. This change is the classic Python buildpack equivalent of the Python CNB change here: https://github.com/heroku/buildpacks-python/pull/272 If both a `runtime.txt` file and a `.python-version` file are present, then the `runtime.txt` file will take precedence. However, use of the `.python-version` file is now recommended, since `runtime.txt` will be deprecated in the future. Both the `runtime.txt` file and `.python-version` file take precedence over any Python version specified in a `Pipfile.lock` for Pipenv users. We support the following `.python-version` syntax: - Major Python version (e.g. `3.13`, which will then be resolved to the latest Python 3.13). (This form is recommended, since it allows for Python security updates to be pulled in without having to manually bump the version.) - Exact Python version (e.g. `3.13.0`) - Comments (lines starting with `#`) - Blank lines We don't support the following `.python-version` features: - Specifying multiple Python versions - Prefixing versions with `python-` (since this form is undocumented and will likely be deprecated by pyenv in the future) In addition, the existing `runtime.txt` support has been updated to allow specifying just the major Python version, in order to increase feature parity between the files, and avoid confusion if users try to use the major version only syntax from `.python-version` in their `runtime.txt`. Refs #913. Refs #932. GUS-W-7671453. GUS-W-16821357. --- .github/workflows/ci.yml | 2 +- CHANGELOG.md | 2 + README.md | 107 +++--- bin/compile | 3 + bin/steps/python | 4 +- lib/python_version.sh | 159 +++++++-- spec/fixtures/ci_pipenv/.python-version | 1 + spec/fixtures/ci_requirements/.python-version | 1 + .../pipenv_and_runtime_txt/runtime.txt | 1 - .../Pipfile | 2 +- .../Pipfile.lock | 10 +- .../bin/compile | 0 .../bin/detect | 0 .../setup.py | 0 spec/fixtures/pipenv_no_lockfile/Pipfile | 5 +- spec/fixtures/pipenv_python_3.10/Pipfile | 12 - spec/fixtures/pipenv_python_3.11/Pipfile | 12 - spec/fixtures/pipenv_python_3.11/Pipfile.lock | 30 -- spec/fixtures/pipenv_python_3.13/Pipfile | 12 - spec/fixtures/pipenv_python_3.13/Pipfile.lock | 30 -- spec/fixtures/pipenv_python_3.7/Pipfile.lock | 30 -- spec/fixtures/pipenv_python_3.8/Pipfile | 12 - spec/fixtures/pipenv_python_3.8/Pipfile.lock | 30 -- spec/fixtures/pipenv_python_3.9/Pipfile | 12 - spec/fixtures/pipenv_python_3.9/Pipfile.lock | 30 -- .../pipenv_python_full_version/Pipfile | 5 +- .../pipenv_python_full_version/Pipfile.lock | 10 +- .../Pipfile | 2 +- .../Pipfile.lock | 10 +- .../.python-version | 2 + .../Pipfile | 3 +- .../Pipfile.lock | 10 +- .../Pipfile | 2 +- .../Pipfile.lock | 12 +- .../pipenv_python_version_invalid/Pipfile | 2 +- .../Pipfile.lock | 10 +- .../pyproject.toml} | 0 .../subdir/some-file | 0 spec/fixtures/python_3.10/.python-version | 2 + spec/fixtures/python_3.10/requirements.txt | 3 +- spec/fixtures/python_3.10/runtime.txt | 1 - spec/fixtures/python_3.11/.python-version | 2 + spec/fixtures/python_3.11/requirements.txt | 3 +- spec/fixtures/python_3.11/runtime.txt | 1 - spec/fixtures/python_3.12/.python-version | 2 + spec/fixtures/python_3.12/requirements.txt | 3 +- spec/fixtures/python_3.12/runtime.txt | 1 - spec/fixtures/python_3.13/.python-version | 2 + spec/fixtures/python_3.13/requirements.txt | 3 +- spec/fixtures/python_3.13/runtime.txt | 1 - spec/fixtures/python_3.7/runtime.txt | 1 - spec/fixtures/python_3.8/.python-version | 2 + spec/fixtures/python_3.8/requirements.txt | 3 +- spec/fixtures/python_3.8/runtime.txt | 1 - spec/fixtures/python_3.9/.python-version | 2 + spec/fixtures/python_3.9/requirements.txt | 3 +- spec/fixtures/python_3.9/runtime.txt | 1 - .../python_version_eol/.python-version | 8 + .../requirements.txt | 0 .../.python-version | 10 + .../requirements.txt | 0 .../.python-version | 4 + .../requirements.txt | 0 .../.python-version | 0 .../requirements.txt | 0 .../.python-version | 1 + .../runtime.txt | 1 - .../.python-version | 1 + .../runtime.txt | 1 - .../python_version_outdated/.python-version | 1 + .../python_version_outdated/runtime.txt | 1 - .../requirements.txt | 3 +- .../requirements_basic/.python-version | 1 + .../.python-version | 2 + .../requirements.txt | 0 .../runtime.txt | 3 + .../runtime_txt_eol_version/requirements.txt | 0 .../runtime.txt | 0 .../requirements.txt | 0 .../runtime.txt | 0 spec/fixtures/runtime_txt_only/runtime.txt | 1 - .../requirements.txt | 1 - .../runtime.txt | 3 - spec/fixtures/setup_py_only/.python-version | 1 + spec/hatchet/ci_spec.rb | 12 +- spec/hatchet/package_manager_spec.rb | 4 +- spec/hatchet/pip_spec.rb | 22 +- spec/hatchet/pipenv_spec.rb | 322 ++++++----------- spec/hatchet/python_update_warning_spec.rb | 6 +- spec/hatchet/python_version_spec.rb | 331 +++++++++++------- spec/hatchet/stack_spec.rb | 4 +- 91 files changed, 630 insertions(+), 726 deletions(-) create mode 100644 spec/fixtures/ci_pipenv/.python-version create mode 100644 spec/fixtures/ci_requirements/.python-version delete mode 100644 spec/fixtures/pipenv_and_runtime_txt/runtime.txt rename spec/fixtures/{pipenv_python_3.12 => pipenv_basic}/Pipfile (85%) rename spec/fixtures/{pipenv_python_3.12 => pipenv_basic}/Pipfile.lock (59%) rename spec/fixtures/{pipenv_python_version_unspecified => pipenv_basic}/bin/compile (100%) rename spec/fixtures/{pipenv_python_version_unspecified => pipenv_basic}/bin/detect (100%) rename spec/fixtures/{pipenv_python_version_unspecified => pipenv_basic}/setup.py (100%) delete mode 100644 spec/fixtures/pipenv_python_3.10/Pipfile delete mode 100644 spec/fixtures/pipenv_python_3.11/Pipfile delete mode 100644 spec/fixtures/pipenv_python_3.11/Pipfile.lock delete mode 100644 spec/fixtures/pipenv_python_3.13/Pipfile delete mode 100644 spec/fixtures/pipenv_python_3.13/Pipfile.lock delete mode 100644 spec/fixtures/pipenv_python_3.7/Pipfile.lock delete mode 100644 spec/fixtures/pipenv_python_3.8/Pipfile delete mode 100644 spec/fixtures/pipenv_python_3.8/Pipfile.lock delete mode 100644 spec/fixtures/pipenv_python_3.9/Pipfile delete mode 100644 spec/fixtures/pipenv_python_3.9/Pipfile.lock create mode 100644 spec/fixtures/pipenv_python_version_and_python_version_file/.python-version rename spec/fixtures/{pipenv_and_runtime_txt => pipenv_python_version_and_python_version_file}/Pipfile (60%) rename spec/fixtures/{pipenv_and_runtime_txt => pipenv_python_version_and_python_version_file}/Pipfile.lock (59%) rename spec/fixtures/{pipenv_python_3.7 => pipenv_python_version_eol}/Pipfile (85%) rename spec/fixtures/{pipenv_python_3.10 => pipenv_python_version_eol}/Pipfile.lock (55%) rename spec/fixtures/{python_2.7/requirements.txt => pyproject_toml_only/pyproject.toml} (100%) rename spec/fixtures/{runtime_txt_only => pyproject_toml_only}/subdir/some-file (100%) create mode 100644 spec/fixtures/python_3.10/.python-version delete mode 100644 spec/fixtures/python_3.10/runtime.txt create mode 100644 spec/fixtures/python_3.11/.python-version delete mode 100644 spec/fixtures/python_3.11/runtime.txt create mode 100644 spec/fixtures/python_3.12/.python-version delete mode 100644 spec/fixtures/python_3.12/runtime.txt create mode 100644 spec/fixtures/python_3.13/.python-version delete mode 100644 spec/fixtures/python_3.13/runtime.txt delete mode 100644 spec/fixtures/python_3.7/runtime.txt create mode 100644 spec/fixtures/python_3.8/.python-version delete mode 100644 spec/fixtures/python_3.8/runtime.txt create mode 100644 spec/fixtures/python_3.9/.python-version delete mode 100644 spec/fixtures/python_3.9/runtime.txt create mode 100644 spec/fixtures/python_version_eol/.python-version rename spec/fixtures/{python_3.7 => python_version_eol}/requirements.txt (100%) create mode 100644 spec/fixtures/python_version_file_invalid_version/.python-version rename spec/fixtures/{python_version_invalid => python_version_file_invalid_version}/requirements.txt (100%) create mode 100644 spec/fixtures/python_version_file_multiple_versions/.python-version create mode 100644 spec/fixtures/python_version_file_multiple_versions/requirements.txt create mode 100644 spec/fixtures/python_version_file_no_version/.python-version create mode 100644 spec/fixtures/python_version_file_no_version/requirements.txt create mode 100644 spec/fixtures/python_version_non_existent_major/.python-version delete mode 100644 spec/fixtures/python_version_non_existent_major/runtime.txt create mode 100644 spec/fixtures/python_version_non_existent_patch/.python-version delete mode 100644 spec/fixtures/python_version_non_existent_patch/runtime.txt create mode 100644 spec/fixtures/python_version_outdated/.python-version delete mode 100644 spec/fixtures/python_version_outdated/runtime.txt create mode 100644 spec/fixtures/requirements_basic/.python-version create mode 100644 spec/fixtures/runtime_txt_and_python_version_file/.python-version create mode 100644 spec/fixtures/runtime_txt_and_python_version_file/requirements.txt create mode 100644 spec/fixtures/runtime_txt_and_python_version_file/runtime.txt create mode 100644 spec/fixtures/runtime_txt_eol_version/requirements.txt rename spec/fixtures/{python_2.7 => runtime_txt_eol_version}/runtime.txt (100%) create mode 100644 spec/fixtures/runtime_txt_invalid_version/requirements.txt rename spec/fixtures/{python_version_invalid => runtime_txt_invalid_version}/runtime.txt (100%) delete mode 100644 spec/fixtures/runtime_txt_only/runtime.txt delete mode 100644 spec/fixtures/runtime_txt_with_stray_whitespace/requirements.txt delete mode 100644 spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt create mode 100644 spec/fixtures/setup_py_only/.python-version diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a319a1b86..d754e02ef 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -73,4 +73,4 @@ jobs: - name: Run buildpack using default app fixture run: make run - name: Run buildpack using an app fixture that's expected to fail - run: make run FIXTURE=spec/fixtures/python_version_invalid/ + run: make run FIXTURE=spec/fixtures/python_version_file_invalid_version/ diff --git a/CHANGELOG.md b/CHANGELOG.md index aada1653b..7563bc94c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## [Unreleased] +- Added support for configuring the Python version using a `.python-version` file. Both the `3.N` and `3.N.N` version forms are supported (the former is recommended). The existing `runtime.txt` file will take precedence if both files are found, however, we recommend switching to `.python-version` since it is more commonly supported in the Python ecosystem. ([#1664](https://github.com/heroku/heroku-buildpack-python/pull/1664)) +- Added support for specifying only the Python major version in `runtime.txt` instead of requiring the full Python version (for example `python-3.N` instead of `python-3.N.N`). ([#1664](https://github.com/heroku/heroku-buildpack-python/pull/1664)) ## [v260] - 2024-10-10 diff --git a/README.md b/README.md index f590addf9..235dffcd4 100644 --- a/README.md +++ b/README.md @@ -8,61 +8,52 @@ This is the official [Heroku buildpack](https://devcenter.heroku.com/articles/bu Recommended web frameworks include **Django** and **Flask**, among others. The recommended webserver is **Gunicorn**. There are no restrictions around what software can be used (as long as it's pip-installable). Web processes must bind to `$PORT`, and only the HTTP protocol is permitted for incoming connections. -See it in Action ----------------- -``` -$ ls -my-application requirements.txt runtime.txt - -$ git push heroku main -Counting objects: 4, done. -Delta compression using up to 8 threads. -Compressing objects: 100% (2/2), done. -Writing objects: 100% (4/4), 276 bytes | 276.00 KiB/s, done. -Total 4 (delta 0), reused 0 (delta 0) -remote: Compressing source files... done. -remote: Building source: -remote: -remote: -----> Python app detected -remote: -----> Installing python -remote: -----> Installing pip -remote: -----> Installing SQLite3 -remote: -----> Installing requirements with pip -remote: Collecting flask (from -r /tmp/build_c2c067ef79ff14c9bf1aed6796f9ed1f/requirements.txt (line 1)) -remote: Downloading ... -remote: Installing collected packages: Werkzeug, click, MarkupSafe, Jinja2, itsdangerous, flask -remote: Successfully installed Jinja2-2.10 MarkupSafe-1.1.0 Werkzeug-0.14.1 click-7.0 flask-1.0.2 itsdangerous-1.1.0 -remote: -remote: -----> Discovering process types -remote: Procfile declares types -> (none) -remote: -``` - -A `requirements.txt` must be present at the root of your application's repository to deploy. - -To specify your python version, you also need a `runtime.txt` file - unless you are using the default Python runtime version. - -Current default Python Runtime: Python 3.12.7 - -Alternatively, you can provide a `setup.py` file, or a `Pipfile`. -Using `pipenv` will generate `runtime.txt` at build time if one of the field `python_version` or `python_full_version` is specified in the `requires` section of your `Pipfile`. - -Specify a Buildpack Version ---------------------------- - -You can specify the latest production release of this buildpack for upcoming builds of an existing application: - - $ heroku buildpacks:set heroku/python - - -Specify a Python Runtime ------------------------- - -Supported runtime options include: - -- `python-3.13.0` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) -- `python-3.12.7` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) -- `python-3.11.10` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) -- `python-3.10.15` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) -- `python-3.9.20` on all [supported stacks](https://devcenter.heroku.com/articles/stack#stack-support-details) -- `python-3.8.20` on Heroku-20 only +## Getting Started + +See the [Getting Started on Heroku with Python](https://devcenter.heroku.com/articles/getting-started-with-python) tutorial. + +## Application Requirements + +A `requirements.txt` or `Pipfile` file must be present in the root (top-level) directory of your app's source code. + +## Configuration + +### Python Version + +We recommend that you specify a Python version for your app rather than relying on the buildpack's default Python version. + +For example, to request the latest patch release of Python 3.13, create a `.python-version` file in +the root directory of your app containing: +`3.13` + +The buildpack will look for a Python version in the following places (in descending order of precedence): + +1. `runtime.txt` file (deprecated) +2. `.python-version` file (recommended) +3. The `python_full_version` field in the `Pipfile.lock` file +4. The `python_version` field in the `Pipfile.lock` file + +If none of those are found, the buildpack will use a default Python version for the first +build of an app, and then subsequent builds of that app will be pinned to that version +unless the build cache is cleared or you request a different version. + +The current default Python version is: 3.12 + +The supported Python versions are: + +- Python 3.13 +- Python 3.12 +- Python 3.11 +- Python 3.10 + +These Python versions are deprecated on Heroku: + +- Python 3.9 +- Python 3.8 (only available on the [Heroku-20](https://devcenter.heroku.com/articles/heroku-20-stack) stack) + +Python versions older than those listed above are no longer supported, since they have reached +end-of-life [upstream](https://devguide.python.org/versions/#supported-versions). + +## Documentation + +For more information about using Python on Heroku, see [Dev Center](https://devcenter.heroku.com/categories/python-support). diff --git a/bin/compile b/bin/compile index bc6621c15..f751965fc 100755 --- a/bin/compile +++ b/bin/compile @@ -154,6 +154,9 @@ fi python_version::read_requested_python_version "${BUILD_DIR}" "${package_manager}" "${cached_python_version}" requested_python_version python_version_origin meta_set "python_version_reason" "${python_version_origin}" +# TODO: More strongly recommend specifying a Python version (eg switch the messaging to +# be a warning instead, after version resolution, and mention .python-version inline) +# TODO: Add runtime.txt deprecation warning. case "${python_version_origin}" in default) puts-step "No Python version was specified. Using the buildpack default: Python ${requested_python_version}" diff --git a/bin/steps/python b/bin/steps/python index 725846d15..e4b605df2 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -18,7 +18,7 @@ PYTHON_URL="${S3_BASE_URL}/python-${python_full_version}-ubuntu-${UBUNTU_VERSION # TODO: Update this message to be more specific once Python 3.8 support is dropped. if ! curl --output /dev/null --silent --head --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}"; then display_error <<-EOF - Error: Python ${python_full_version} is not available for this stack (${STACK}). + Error: Python ${python_full_version} isn't available for this stack (${STACK}). For a list of the supported Python versions, see: https://devcenter.heroku.com/articles/python-support#supported-runtimes @@ -35,6 +35,8 @@ function warn_if_patch_update_available() { # Extract the patch version component of the version strings (ie: the '5' in '3.10.5'). local requested_patch_number="${requested_full_version##*.}" local latest_patch_number="${latest_patch_version##*.}" + # TODO: Update this message to suggest using the .python-version major version syntax to stay up to date, + # once runtime.txt is deprecated and sticky-versioning only pins to the major version. if ((requested_patch_number < latest_patch_number)); then puts-warn puts-warn "A Python security update is available! Upgrade as soon as possible to: Python ${latest_patch_version}" diff --git a/lib/python_version.sh b/lib/python_version.sh index 820daf3a9..aa4df3539 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -18,8 +18,6 @@ DEFAULT_PYTHON_MAJOR_VERSION="${DEFAULT_PYTHON_FULL_VERSION%.*}" INT_REGEX="(0|[1-9][0-9]*)" # Versions of form N.N or N.N.N. PYTHON_VERSION_REGEX="${INT_REGEX}\.${INT_REGEX}(\.${INT_REGEX})?" -# Versions of form N.N.N only. -PYTHON_FULL_VERSION_REGEX="${INT_REGEX}\.${INT_REGEX}\.${INT_REGEX}" # Determine what Python version has been requested for the project. # @@ -29,8 +27,9 @@ PYTHON_FULL_VERSION_REGEX="${INT_REGEX}\.${INT_REGEX}\.${INT_REGEX}" # # If an app specifies the Python version via multiple means, then the order of precedence is: # 1. runtime.txt -# 2. Pipfile.lock (`python_full_version` field) -# 3. Pipfile.lock (`python_version` field) +# 2. .python-version +# 3. Pipfile.lock (`python_full_version` field) +# 4. Pipfile.lock (`python_version` field) # # If a version wasn't specified by the app, then new apps/those with an empty cache will use # a buildpack default version for the first build, and then subsequent cached builds will use @@ -58,6 +57,14 @@ function python_version::read_requested_python_version() { return 0 fi + local python_version_file_path="${build_dir}/.python-version" + if [[ -f "${python_version_file_path}" ]]; then + contents="$(cat "${python_version_file_path}")" + version="$(python_version::parse_python_version_file "${contents}")" + origin=".python-version" + return 0 + fi + if [[ "${package_manager}" == "pipenv" ]]; then version="$(python_version::read_pipenv_python_version "${build_dir}")" # The Python version fields in a Pipfile.lock are optional. @@ -79,40 +86,122 @@ function python_version::read_requested_python_version() { fi } -# Parse the contents of a runtime.txt file and return the Python version substring (e.g. `3.12.0`). +# Parse the contents of a runtime.txt file and return the Python version substring (e.g. `3.12` or `3.12.0`). function python_version::parse_runtime_txt() { local contents="${1}" - # The file must contain a string of form `python-N.N.N` (leading/trailing whitespace is permitted). - if [[ "${contents}" =~ ^[[:space:]]*python-(${PYTHON_FULL_VERSION_REGEX})[[:space:]]*$ ]]; then + # The file must contain a string of form `python-N.N` or `python-N.N.N`. + # Leading/trailing whitespace is permitted. + if [[ "${contents}" =~ ^[[:space:]]*python-(${PYTHON_VERSION_REGEX})[[:space:]]*$ ]]; then local version="${BASH_REMATCH[1]}" echo "${version}" else display_error <<-EOF Error: Invalid Python version in runtime.txt. - The Python version specified in 'runtime.txt' is not in + The Python version specified in 'runtime.txt' isn't in the correct format. The following file contents were found: ${contents} - However, the version string must begin with a 'python-' prefix, - followed by the version specified as '..'. - Comments are not supported. + However, the version must be specified as either: + 1. 'python-.' (recommended, for automatic patch updates) + 2. 'python-..' (to pin to an exact patch version) - For example, to request Python ${DEFAULT_PYTHON_FULL_VERSION}, use: - python-${DEFAULT_PYTHON_FULL_VERSION} + Remember to include the 'python-' prefix. Comments aren't supported. - Please update 'runtime.txt' to use a valid version string, or - else remove the file to instead use the default version - (currently Python ${DEFAULT_PYTHON_FULL_VERSION}). + For example, to request the latest version of Python ${DEFAULT_PYTHON_MAJOR_VERSION}, + update the 'runtime.txt' file so it contains: + python-${DEFAULT_PYTHON_MAJOR_VERSION} EOF - meta_set "failure_reason" "python-version-invalid" + meta_set "failure_reason" "runtime-txt::invalid-version" return 1 fi } +# Parse the contents of a .python-version file and return the Python version substring (e.g. `3.12` or `3.12.0`). +function python_version::parse_python_version_file() { + local contents="${1}" + local version_lines=() + + while IFS= read -r line; do + # Ignore lines that only contain whitespace and/or comments. + if [[ ! "${line}" =~ ^[[:space:]]*(#.*)?$ ]]; then + version_lines+=("${line}") + fi + done <<<"${contents}" + + case "${#version_lines[@]}" in + 1) + local line="${version_lines[0]}" + if [[ "${line}" =~ ^[[:space:]]*(${PYTHON_VERSION_REGEX})[[:space:]]*$ ]]; then + local version="${BASH_REMATCH[1]}" + echo "${version}" + return 0 + else + display_error <<-EOF + Error: Invalid Python version in .python-version. + + The Python version specified in '.python-version' isn't in + the correct format. + + The following version was found: + ${line} + + However, the version must be specified as either: + 1. '.' (recommended, for automatic patch updates) + 2. '..' (to pin to an exact patch version) + + Don't include quotes or a 'python-' prefix. To include + comments, add them on their own line, prefixed with '#'. + + For example, to request the latest version of Python ${DEFAULT_PYTHON_MAJOR_VERSION}, + update the '.python-version' file so it contains: + ${DEFAULT_PYTHON_MAJOR_VERSION} + EOF + meta_set "failure_reason" "python-version-file::invalid-version" + return 1 + fi + ;; + 0) + display_error <<-EOF + Error: Invalid Python version in .python-version. + + No Python version was found in the '.python-version' file. + + Update the file so that it contains a valid Python version + such as '${DEFAULT_PYTHON_MAJOR_VERSION}'. + + If the file already contains a version, check the line doesn't + begin with a '#', otherwise it will be treated as a comment. + EOF + meta_set "failure_reason" "python-version-file::no-version" + return 1 + ;; + *) + display_error <<-EOF + Error: Invalid Python version in .python-version. + + Multiple Python versions were found in the '.python-version' + file: + + $( + IFS=$'\n' + echo "${version_lines[*]}" + ) + + Update the file so it contains only one Python version. + + If the additional versions are actually comments, prefix + those lines with '#'. + EOF + meta_set "failure_reason" "python-version-file::multiple-versions" + return 1 + ;; + esac +} + # Read the Python version from a Pipfile.lock, which can exist in one of two optional fields, # `python_full_version` (as N.N.N) and `python_version` (as N.N). If both fields are # defined, we will use the value set in `python_full_version`. See: @@ -130,16 +219,16 @@ function python_version::read_pipenv_python_version() { if ! version=$(jq --raw-output '._meta.requires.python_full_version // ._meta.requires.python_version' "${pipfile_lock_path}" 2>&1); then display_error <<-EOF - Error: Cannot parse Pipfile.lock. + Error: Can't parse Pipfile.lock. - A Pipfile.lock file was found, however, it could not be parsed: + A Pipfile.lock file was found, however, it couldn't be parsed: ${version} This is likely due to it not being valid JSON. Run 'pipenv lock' to regenerate/fix the lockfile. EOF - meta_set "failure_reason" "pipfile-lock-invalid" + meta_set "failure_reason" "pipfile-lock::invalid-json" return 1 fi @@ -158,14 +247,14 @@ function python_version::read_pipenv_python_version() { Error: Invalid Python version in Pipfile / Pipfile.lock. The Python version specified in Pipfile / Pipfile.lock by the - 'python_version' or 'python_full_version' field is not valid. + 'python_version' or 'python_full_version' field isn't valid. The following version was found: ${version} However, the version must be specified as either: - 1. '.' (recommended, for automatic security updates) - 2. '..' (to pin to an exact Python version) + 1. '.' (recommended, for automatic patch updates) + 2. '..' (to pin to an exact patch version) Please update your 'Pipfile' to use a valid Python version and then run 'pipenv lock' to regenerate the lockfile. @@ -173,13 +262,13 @@ function python_version::read_pipenv_python_version() { For more information, see: https://pipenv.pypa.io/en/latest/specifiers.html#specifying-versions-of-python EOF - meta_set "failure_reason" "python-version-invalid" + meta_set "failure_reason" "pipfile-lock::invalid-version" return 1 fi } # Resolve a requested Python version (which can be of form N.N or N.N.N) to a specific -# Python version of form N.N.N. Rejects Python major versions that are not supported. +# Python version of form N.N.N. Rejects Python major versions that aren't supported. function python_version::resolve_python_version() { local requested_python_version="${1}" local python_version_origin="${2}" @@ -197,7 +286,7 @@ function python_version::resolve_python_version() { display_error <<-EOF Error: The cached Python version has reached end-of-life. - Your app does not specify a Python version, and so normally + Your app doesn't specify a Python version, and so normally would use the version cached from the last build (${requested_python_version}). However, Python ${major}.${minor} has reached its upstream end-of-life, @@ -207,8 +296,8 @@ function python_version::resolve_python_version() { As such, it is no longer supported by this buildpack. Please upgrade to a newer Python version by creating a - 'runtime.txt' file that contains a Python version like: - python-${DEFAULT_PYTHON_FULL_VERSION} + '.python-version' file that contains a Python version like: + ${DEFAULT_PYTHON_MAJOR_VERSION} For a list of the supported Python versions, see: https://devcenter.heroku.com/articles/python-support#supported-runtimes @@ -230,19 +319,19 @@ function python_version::resolve_python_version() { https://devcenter.heroku.com/articles/python-support#supported-runtimes EOF fi - meta_set "failure_reason" "python-version-eol" + meta_set "failure_reason" "python-version::eol" return 1 fi if (((major == 3 && minor > 13) || major >= 4)); then if [[ "${python_version_origin}" == "cached" ]]; then display_error <<-EOF - Error: The cached Python version is not recognised. + Error: The cached Python version isn't recognised. - Your app does not specify a Python version, and so normally + Your app doesn't specify a Python version, and so normally would use the version cached from the last build (${requested_python_version}). - However, Python ${major}.${minor} is not recognised by this version + However, Python ${major}.${minor} isn't recognised by this version of the buildpack. This can occur if you have downgraded the version of the @@ -252,9 +341,9 @@ function python_version::resolve_python_version() { EOF else display_error <<-EOF - Error: The requested Python version is not recognised. + Error: The requested Python version isn't recognised. - The requested Python version ${major}.${minor} is not recognised. + The requested Python version ${major}.${minor} isn't recognised. Check that this Python version has been officially released, and that the Python buildpack has added support for it: @@ -269,7 +358,7 @@ function python_version::resolve_python_version() { by updating the version configured via the '${python_version_origin}' file. EOF fi - meta_set "failure_reason" "python-version-unknown" + meta_set "failure_reason" "python-version::unknown-major" return 1 fi diff --git a/spec/fixtures/ci_pipenv/.python-version b/spec/fixtures/ci_pipenv/.python-version new file mode 100644 index 000000000..e4fba2183 --- /dev/null +++ b/spec/fixtures/ci_pipenv/.python-version @@ -0,0 +1 @@ +3.12 diff --git a/spec/fixtures/ci_requirements/.python-version b/spec/fixtures/ci_requirements/.python-version new file mode 100644 index 000000000..e4fba2183 --- /dev/null +++ b/spec/fixtures/ci_requirements/.python-version @@ -0,0 +1 @@ +3.12 diff --git a/spec/fixtures/pipenv_and_runtime_txt/runtime.txt b/spec/fixtures/pipenv_and_runtime_txt/runtime.txt deleted file mode 100644 index 32905d6e0..000000000 --- a/spec/fixtures/pipenv_and_runtime_txt/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.12.7 diff --git a/spec/fixtures/pipenv_python_3.12/Pipfile b/spec/fixtures/pipenv_basic/Pipfile similarity index 85% rename from spec/fixtures/pipenv_python_3.12/Pipfile rename to spec/fixtures/pipenv_basic/Pipfile index 9bf60112c..883fd6c01 100644 --- a/spec/fixtures/pipenv_python_3.12/Pipfile +++ b/spec/fixtures/pipenv_basic/Pipfile @@ -4,7 +4,7 @@ verify_ssl = true name = "pypi" [packages] -urllib3 = "*" +typing-extensions = "*" [dev-packages] diff --git a/spec/fixtures/pipenv_python_3.12/Pipfile.lock b/spec/fixtures/pipenv_basic/Pipfile.lock similarity index 59% rename from spec/fixtures/pipenv_python_3.12/Pipfile.lock rename to spec/fixtures/pipenv_basic/Pipfile.lock index cc5ce05bc..30824a47d 100644 --- a/spec/fixtures/pipenv_python_3.12/Pipfile.lock +++ b/spec/fixtures/pipenv_basic/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "13de6076b4dc8bbad334b7e4ae8e8ae63f7f8b4aad794c7f0173871b6927c8d0" + "sha256": "9661ed313a79ccb68c7dc4e639068f86ddd91e307ec2ed60498858d002e9b547" }, "pipfile-spec": 6, "requires": { @@ -16,14 +16,14 @@ ] }, "default": { - "urllib3": { + "typing-extensions": { "hashes": [ - "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", - "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" + "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", + "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==2.2.1" + "version": "==4.12.2" } }, "develop": {} diff --git a/spec/fixtures/pipenv_python_version_unspecified/bin/compile b/spec/fixtures/pipenv_basic/bin/compile similarity index 100% rename from spec/fixtures/pipenv_python_version_unspecified/bin/compile rename to spec/fixtures/pipenv_basic/bin/compile diff --git a/spec/fixtures/pipenv_python_version_unspecified/bin/detect b/spec/fixtures/pipenv_basic/bin/detect similarity index 100% rename from spec/fixtures/pipenv_python_version_unspecified/bin/detect rename to spec/fixtures/pipenv_basic/bin/detect diff --git a/spec/fixtures/pipenv_python_version_unspecified/setup.py b/spec/fixtures/pipenv_basic/setup.py similarity index 100% rename from spec/fixtures/pipenv_python_version_unspecified/setup.py rename to spec/fixtures/pipenv_basic/setup.py diff --git a/spec/fixtures/pipenv_no_lockfile/Pipfile b/spec/fixtures/pipenv_no_lockfile/Pipfile index 67803f45b..883fd6c01 100644 --- a/spec/fixtures/pipenv_no_lockfile/Pipfile +++ b/spec/fixtures/pipenv_no_lockfile/Pipfile @@ -4,6 +4,9 @@ verify_ssl = true name = "pypi" [packages] -urllib3 = "*" +typing-extensions = "*" [dev-packages] + +[requires] +python_version = "3.12" diff --git a/spec/fixtures/pipenv_python_3.10/Pipfile b/spec/fixtures/pipenv_python_3.10/Pipfile deleted file mode 100644 index 37291dc4f..000000000 --- a/spec/fixtures/pipenv_python_3.10/Pipfile +++ /dev/null @@ -1,12 +0,0 @@ -[[source]] -url = "https://pypi.org/simple" -verify_ssl = true -name = "pypi" - -[packages] -urllib3 = "*" - -[dev-packages] - -[requires] -python_version = "3.10" diff --git a/spec/fixtures/pipenv_python_3.11/Pipfile b/spec/fixtures/pipenv_python_3.11/Pipfile deleted file mode 100644 index 1d28b2380..000000000 --- a/spec/fixtures/pipenv_python_3.11/Pipfile +++ /dev/null @@ -1,12 +0,0 @@ -[[source]] -url = "https://pypi.org/simple" -verify_ssl = true -name = "pypi" - -[packages] -urllib3 = "*" - -[dev-packages] - -[requires] -python_version = "3.11" diff --git a/spec/fixtures/pipenv_python_3.11/Pipfile.lock b/spec/fixtures/pipenv_python_3.11/Pipfile.lock deleted file mode 100644 index 5b0ccff7d..000000000 --- a/spec/fixtures/pipenv_python_3.11/Pipfile.lock +++ /dev/null @@ -1,30 +0,0 @@ -{ - "_meta": { - "hash": { - "sha256": "c23e4774efa7153ba8addfbe118e9cc0c526a063275824cf25cb2198e0ff1b33" - }, - "pipfile-spec": 6, - "requires": { - "python_version": "3.11" - }, - "sources": [ - { - "name": "pypi", - "url": "https://pypi.org/simple", - "verify_ssl": true - } - ] - }, - "default": { - "urllib3": { - "hashes": [ - "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", - "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" - ], - "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==2.2.1" - } - }, - "develop": {} -} diff --git a/spec/fixtures/pipenv_python_3.13/Pipfile b/spec/fixtures/pipenv_python_3.13/Pipfile deleted file mode 100644 index 73f50fc86..000000000 --- a/spec/fixtures/pipenv_python_3.13/Pipfile +++ /dev/null @@ -1,12 +0,0 @@ -[[source]] -url = "https://pypi.org/simple" -verify_ssl = true -name = "pypi" - -[packages] -urllib3 = "*" - -[dev-packages] - -[requires] -python_version = "3.13" diff --git a/spec/fixtures/pipenv_python_3.13/Pipfile.lock b/spec/fixtures/pipenv_python_3.13/Pipfile.lock deleted file mode 100644 index dcd0b3ca7..000000000 --- a/spec/fixtures/pipenv_python_3.13/Pipfile.lock +++ /dev/null @@ -1,30 +0,0 @@ -{ - "_meta": { - "hash": { - "sha256": "60dc67ee4223a391c5e0ae2b4d7ea54a7b245773d76b6ff82156dda97a3e4fb2" - }, - "pipfile-spec": 6, - "requires": { - "python_version": "3.13" - }, - "sources": [ - { - "name": "pypi", - "url": "https://pypi.org/simple", - "verify_ssl": true - } - ] - }, - "default": { - "urllib3": { - "hashes": [ - "sha256:ca899ca043dcb1bafa3e262d73aa25c465bfb49e0bd9dd5d59f1d0acba2f8fac", - "sha256:e7d814a81dad81e6caf2ec9fdedb284ecc9c73076b62654547cc64ccdcae26e9" - ], - "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==2.2.3" - } - }, - "develop": {} -} diff --git a/spec/fixtures/pipenv_python_3.7/Pipfile.lock b/spec/fixtures/pipenv_python_3.7/Pipfile.lock deleted file mode 100644 index 90c8735f4..000000000 --- a/spec/fixtures/pipenv_python_3.7/Pipfile.lock +++ /dev/null @@ -1,30 +0,0 @@ -{ - "_meta": { - "hash": { - "sha256": "b4a533c276cc9b89b506de9007fbe2210b33f8322ea81f62989117e5ec3adc54" - }, - "pipfile-spec": 6, - "requires": { - "python_version": "3.7" - }, - "sources": [ - { - "name": "pypi", - "url": "https://pypi.org/simple", - "verify_ssl": true - } - ] - }, - "default": { - "urllib3": { - "hashes": [ - "sha256:13abf37382ea2ce6fb744d4dad67838eec857c9f4f57009891805e0b5e123594", - "sha256:ef16afa8ba34a1f989db38e1dbbe0c302e4289a47856990d0682e374563ce35e" - ], - "index": "pypi", - "markers": "python_version >= '3.7'", - "version": "==2.0.5" - } - }, - "develop": {} -} diff --git a/spec/fixtures/pipenv_python_3.8/Pipfile b/spec/fixtures/pipenv_python_3.8/Pipfile deleted file mode 100644 index 5aea20cfa..000000000 --- a/spec/fixtures/pipenv_python_3.8/Pipfile +++ /dev/null @@ -1,12 +0,0 @@ -[[source]] -url = "https://pypi.org/simple" -verify_ssl = true -name = "pypi" - -[packages] -urllib3 = "*" - -[dev-packages] - -[requires] -python_version = "3.8" diff --git a/spec/fixtures/pipenv_python_3.8/Pipfile.lock b/spec/fixtures/pipenv_python_3.8/Pipfile.lock deleted file mode 100644 index f1618e6a9..000000000 --- a/spec/fixtures/pipenv_python_3.8/Pipfile.lock +++ /dev/null @@ -1,30 +0,0 @@ -{ - "_meta": { - "hash": { - "sha256": "e9eb0bba6cb0a97533d25cd3d68db92d4afd6b1dc88cd7a51607d8c34475eae0" - }, - "pipfile-spec": 6, - "requires": { - "python_version": "3.8" - }, - "sources": [ - { - "name": "pypi", - "url": "https://pypi.org/simple", - "verify_ssl": true - } - ] - }, - "default": { - "urllib3": { - "hashes": [ - "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", - "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" - ], - "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==2.2.1" - } - }, - "develop": {} -} diff --git a/spec/fixtures/pipenv_python_3.9/Pipfile b/spec/fixtures/pipenv_python_3.9/Pipfile deleted file mode 100644 index 7e9cf996d..000000000 --- a/spec/fixtures/pipenv_python_3.9/Pipfile +++ /dev/null @@ -1,12 +0,0 @@ -[[source]] -url = "https://pypi.org/simple" -verify_ssl = true -name = "pypi" - -[packages] -urllib3 = "*" - -[dev-packages] - -[requires] -python_version = "3.9" diff --git a/spec/fixtures/pipenv_python_3.9/Pipfile.lock b/spec/fixtures/pipenv_python_3.9/Pipfile.lock deleted file mode 100644 index 527944fb6..000000000 --- a/spec/fixtures/pipenv_python_3.9/Pipfile.lock +++ /dev/null @@ -1,30 +0,0 @@ -{ - "_meta": { - "hash": { - "sha256": "7776f372e3411c742b3e8a4209e67f0832f9e249d4c86ce28ece146afaef68d1" - }, - "pipfile-spec": 6, - "requires": { - "python_version": "3.9" - }, - "sources": [ - { - "name": "pypi", - "url": "https://pypi.org/simple", - "verify_ssl": true - } - ] - }, - "default": { - "urllib3": { - "hashes": [ - "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", - "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" - ], - "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==2.2.1" - } - }, - "develop": {} -} diff --git a/spec/fixtures/pipenv_python_full_version/Pipfile b/spec/fixtures/pipenv_python_full_version/Pipfile index aa0852315..f3e9b3e9e 100644 --- a/spec/fixtures/pipenv_python_full_version/Pipfile +++ b/spec/fixtures/pipenv_python_full_version/Pipfile @@ -4,12 +4,13 @@ verify_ssl = true name = "pypi" [packages] -urllib3 = "*" +typing-extensions = "*" [dev-packages] [requires] # Uses the oldest Python version supported by all stacks, to validate Pipenv works with it. python_full_version = "3.9.0" -# Tests that `python_full_version` takes precedence. + +# `python_full_version` should take precedence over this. python_version = "3.12" diff --git a/spec/fixtures/pipenv_python_full_version/Pipfile.lock b/spec/fixtures/pipenv_python_full_version/Pipfile.lock index 888c9982e..3269fa9ce 100644 --- a/spec/fixtures/pipenv_python_full_version/Pipfile.lock +++ b/spec/fixtures/pipenv_python_full_version/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "068e8668ba7ae121efca9caf66d4796267713afe33c711e01a0852fe5e9a1848" + "sha256": "20aa9cfaef42c18684f934bc2c4e062167465b9a062fa284441acf1fe712dc60" }, "pipfile-spec": 6, "requires": { @@ -17,14 +17,14 @@ ] }, "default": { - "urllib3": { + "typing-extensions": { "hashes": [ - "sha256:ca899ca043dcb1bafa3e262d73aa25c465bfb49e0bd9dd5d59f1d0acba2f8fac", - "sha256:e7d814a81dad81e6caf2ec9fdedb284ecc9c73076b62654547cc64ccdcae26e9" + "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", + "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==2.2.3" + "version": "==4.12.2" } }, "develop": {} diff --git a/spec/fixtures/pipenv_python_full_version_invalid/Pipfile b/spec/fixtures/pipenv_python_full_version_invalid/Pipfile index 1782422d3..4016fd876 100644 --- a/spec/fixtures/pipenv_python_full_version_invalid/Pipfile +++ b/spec/fixtures/pipenv_python_full_version_invalid/Pipfile @@ -4,7 +4,7 @@ verify_ssl = true name = "pypi" [packages] -urllib3 = "*" +typing-extensions = "*" [dev-packages] diff --git a/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock b/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock index 107e6f348..a9d6269b5 100644 --- a/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock +++ b/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "c629159445d228792003fe74526738c9155072a74a29f2d4920fc8aab878db0a" + "sha256": "254efba85c5858a7dfe232cb38f9ead91bf6e50bbf82f51a8a5a01904ca8712e" }, "pipfile-spec": 6, "requires": { @@ -16,14 +16,14 @@ ] }, "default": { - "urllib3": { + "typing-extensions": { "hashes": [ - "sha256:ca899ca043dcb1bafa3e262d73aa25c465bfb49e0bd9dd5d59f1d0acba2f8fac", - "sha256:e7d814a81dad81e6caf2ec9fdedb284ecc9c73076b62654547cc64ccdcae26e9" + "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", + "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==2.2.3" + "version": "==4.12.2" } }, "develop": {} diff --git a/spec/fixtures/pipenv_python_version_and_python_version_file/.python-version b/spec/fixtures/pipenv_python_version_and_python_version_file/.python-version new file mode 100644 index 000000000..6d977fdcf --- /dev/null +++ b/spec/fixtures/pipenv_python_version_and_python_version_file/.python-version @@ -0,0 +1,2 @@ +# This should take precedence over the Python version in Pipfile.lock. +3.13 diff --git a/spec/fixtures/pipenv_and_runtime_txt/Pipfile b/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile similarity index 60% rename from spec/fixtures/pipenv_and_runtime_txt/Pipfile rename to spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile index 1d28b2380..5afe25b60 100644 --- a/spec/fixtures/pipenv_and_runtime_txt/Pipfile +++ b/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile @@ -4,9 +4,10 @@ verify_ssl = true name = "pypi" [packages] -urllib3 = "*" +typing-extensions = "*" [dev-packages] [requires] +# The version in .python-version should take precedence over this. python_version = "3.11" diff --git a/spec/fixtures/pipenv_and_runtime_txt/Pipfile.lock b/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile.lock similarity index 59% rename from spec/fixtures/pipenv_and_runtime_txt/Pipfile.lock rename to spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile.lock index 5b0ccff7d..c505f1d3c 100644 --- a/spec/fixtures/pipenv_and_runtime_txt/Pipfile.lock +++ b/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "c23e4774efa7153ba8addfbe118e9cc0c526a063275824cf25cb2198e0ff1b33" + "sha256": "5a5846099fd5ceb0291e704a94f49cf0b8a226109fdae915a61253b56eaf3ed6" }, "pipfile-spec": 6, "requires": { @@ -16,14 +16,14 @@ ] }, "default": { - "urllib3": { + "typing-extensions": { "hashes": [ - "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", - "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" + "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", + "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==2.2.1" + "version": "==4.12.2" } }, "develop": {} diff --git a/spec/fixtures/pipenv_python_3.7/Pipfile b/spec/fixtures/pipenv_python_version_eol/Pipfile similarity index 85% rename from spec/fixtures/pipenv_python_3.7/Pipfile rename to spec/fixtures/pipenv_python_version_eol/Pipfile index c973bafe3..afe33e1b7 100644 --- a/spec/fixtures/pipenv_python_3.7/Pipfile +++ b/spec/fixtures/pipenv_python_version_eol/Pipfile @@ -4,7 +4,7 @@ verify_ssl = true name = "pypi" [packages] -urllib3 = "*" +typing-extensions = "*" [dev-packages] diff --git a/spec/fixtures/pipenv_python_3.10/Pipfile.lock b/spec/fixtures/pipenv_python_version_eol/Pipfile.lock similarity index 55% rename from spec/fixtures/pipenv_python_3.10/Pipfile.lock rename to spec/fixtures/pipenv_python_version_eol/Pipfile.lock index 7b07be0dc..cf0460577 100644 --- a/spec/fixtures/pipenv_python_3.10/Pipfile.lock +++ b/spec/fixtures/pipenv_python_version_eol/Pipfile.lock @@ -1,11 +1,11 @@ { "_meta": { "hash": { - "sha256": "0975b9caa75f76ce36c7cf6ee6fc4b63510c0970ff8e322e0b5e6ee7af2d32e8" + "sha256": "319488142f6df2cfe3244f5ebb99c88eaa0ef808d4560570175577ce045dd02e" }, "pipfile-spec": 6, "requires": { - "python_version": "3.10" + "python_version": "3.7" }, "sources": [ { @@ -16,14 +16,14 @@ ] }, "default": { - "urllib3": { + "typing-extensions": { "hashes": [ - "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", - "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" + "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", + "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==2.2.1" + "version": "==4.12.2" } }, "develop": {} diff --git a/spec/fixtures/pipenv_python_version_invalid/Pipfile b/spec/fixtures/pipenv_python_version_invalid/Pipfile index 326889f7a..43fe5b889 100644 --- a/spec/fixtures/pipenv_python_version_invalid/Pipfile +++ b/spec/fixtures/pipenv_python_version_invalid/Pipfile @@ -4,7 +4,7 @@ verify_ssl = true name = "pypi" [packages] -urllib3 = "*" +typing-extensions = "*" [dev-packages] diff --git a/spec/fixtures/pipenv_python_version_invalid/Pipfile.lock b/spec/fixtures/pipenv_python_version_invalid/Pipfile.lock index 47d197cdd..054e8d6cc 100644 --- a/spec/fixtures/pipenv_python_version_invalid/Pipfile.lock +++ b/spec/fixtures/pipenv_python_version_invalid/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "584096cd3f6a5c68b5f2e21993fc104773efb507ed1945ca26c789d86b22d883" + "sha256": "c86b61d10a1742dfbc75378ee86a81c24420506b154b9c19d69aa6e6c02c61b2" }, "pipfile-spec": 6, "requires": { @@ -16,14 +16,14 @@ ] }, "default": { - "urllib3": { + "typing-extensions": { "hashes": [ - "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", - "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" + "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", + "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==2.2.1" + "version": "==4.12.2" } }, "develop": {} diff --git a/spec/fixtures/python_2.7/requirements.txt b/spec/fixtures/pyproject_toml_only/pyproject.toml similarity index 100% rename from spec/fixtures/python_2.7/requirements.txt rename to spec/fixtures/pyproject_toml_only/pyproject.toml diff --git a/spec/fixtures/runtime_txt_only/subdir/some-file b/spec/fixtures/pyproject_toml_only/subdir/some-file similarity index 100% rename from spec/fixtures/runtime_txt_only/subdir/some-file rename to spec/fixtures/pyproject_toml_only/subdir/some-file diff --git a/spec/fixtures/python_3.10/.python-version b/spec/fixtures/python_3.10/.python-version new file mode 100644 index 000000000..3aa278d1a --- /dev/null +++ b/spec/fixtures/python_3.10/.python-version @@ -0,0 +1,2 @@ +# Comments are supported +3.10 diff --git a/spec/fixtures/python_3.10/requirements.txt b/spec/fixtures/python_3.10/requirements.txt index a42590beb..eec3a2223 100644 --- a/spec/fixtures/python_3.10/requirements.txt +++ b/spec/fixtures/python_3.10/requirements.txt @@ -1 +1,2 @@ -urllib3 +# This package has been picked since it has no dependencies and is small/fast to install. +typing-extensions==4.12.2 diff --git a/spec/fixtures/python_3.10/runtime.txt b/spec/fixtures/python_3.10/runtime.txt deleted file mode 100644 index bb60b7f69..000000000 --- a/spec/fixtures/python_3.10/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.10.15 diff --git a/spec/fixtures/python_3.11/.python-version b/spec/fixtures/python_3.11/.python-version new file mode 100644 index 000000000..576fc2bfe --- /dev/null +++ b/spec/fixtures/python_3.11/.python-version @@ -0,0 +1,2 @@ +# Comments are supported +3.11 diff --git a/spec/fixtures/python_3.11/requirements.txt b/spec/fixtures/python_3.11/requirements.txt index a42590beb..eec3a2223 100644 --- a/spec/fixtures/python_3.11/requirements.txt +++ b/spec/fixtures/python_3.11/requirements.txt @@ -1 +1,2 @@ -urllib3 +# This package has been picked since it has no dependencies and is small/fast to install. +typing-extensions==4.12.2 diff --git a/spec/fixtures/python_3.11/runtime.txt b/spec/fixtures/python_3.11/runtime.txt deleted file mode 100644 index e34519556..000000000 --- a/spec/fixtures/python_3.11/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.11.10 diff --git a/spec/fixtures/python_3.12/.python-version b/spec/fixtures/python_3.12/.python-version new file mode 100644 index 000000000..936a75516 --- /dev/null +++ b/spec/fixtures/python_3.12/.python-version @@ -0,0 +1,2 @@ +# Comments are supported +3.12 diff --git a/spec/fixtures/python_3.12/requirements.txt b/spec/fixtures/python_3.12/requirements.txt index a42590beb..eec3a2223 100644 --- a/spec/fixtures/python_3.12/requirements.txt +++ b/spec/fixtures/python_3.12/requirements.txt @@ -1 +1,2 @@ -urllib3 +# This package has been picked since it has no dependencies and is small/fast to install. +typing-extensions==4.12.2 diff --git a/spec/fixtures/python_3.12/runtime.txt b/spec/fixtures/python_3.12/runtime.txt deleted file mode 100644 index 32905d6e0..000000000 --- a/spec/fixtures/python_3.12/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.12.7 diff --git a/spec/fixtures/python_3.13/.python-version b/spec/fixtures/python_3.13/.python-version new file mode 100644 index 000000000..d23c34390 --- /dev/null +++ b/spec/fixtures/python_3.13/.python-version @@ -0,0 +1,2 @@ +# Comments are supported +3.13 diff --git a/spec/fixtures/python_3.13/requirements.txt b/spec/fixtures/python_3.13/requirements.txt index a42590beb..eec3a2223 100644 --- a/spec/fixtures/python_3.13/requirements.txt +++ b/spec/fixtures/python_3.13/requirements.txt @@ -1 +1,2 @@ -urllib3 +# This package has been picked since it has no dependencies and is small/fast to install. +typing-extensions==4.12.2 diff --git a/spec/fixtures/python_3.13/runtime.txt b/spec/fixtures/python_3.13/runtime.txt deleted file mode 100644 index dae7fec92..000000000 --- a/spec/fixtures/python_3.13/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.13.0 diff --git a/spec/fixtures/python_3.7/runtime.txt b/spec/fixtures/python_3.7/runtime.txt deleted file mode 100644 index 113b0bcca..000000000 --- a/spec/fixtures/python_3.7/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.7.17 diff --git a/spec/fixtures/python_3.8/.python-version b/spec/fixtures/python_3.8/.python-version new file mode 100644 index 000000000..6a8981ea9 --- /dev/null +++ b/spec/fixtures/python_3.8/.python-version @@ -0,0 +1,2 @@ +# Comments are supported +3.8 diff --git a/spec/fixtures/python_3.8/requirements.txt b/spec/fixtures/python_3.8/requirements.txt index a42590beb..eec3a2223 100644 --- a/spec/fixtures/python_3.8/requirements.txt +++ b/spec/fixtures/python_3.8/requirements.txt @@ -1 +1,2 @@ -urllib3 +# This package has been picked since it has no dependencies and is small/fast to install. +typing-extensions==4.12.2 diff --git a/spec/fixtures/python_3.8/runtime.txt b/spec/fixtures/python_3.8/runtime.txt deleted file mode 100644 index 7494875b9..000000000 --- a/spec/fixtures/python_3.8/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.8.20 diff --git a/spec/fixtures/python_3.9/.python-version b/spec/fixtures/python_3.9/.python-version new file mode 100644 index 000000000..54f2f0246 --- /dev/null +++ b/spec/fixtures/python_3.9/.python-version @@ -0,0 +1,2 @@ +# Comments are supported +3.9 diff --git a/spec/fixtures/python_3.9/requirements.txt b/spec/fixtures/python_3.9/requirements.txt index a42590beb..eec3a2223 100644 --- a/spec/fixtures/python_3.9/requirements.txt +++ b/spec/fixtures/python_3.9/requirements.txt @@ -1 +1,2 @@ -urllib3 +# This package has been picked since it has no dependencies and is small/fast to install. +typing-extensions==4.12.2 diff --git a/spec/fixtures/python_3.9/runtime.txt b/spec/fixtures/python_3.9/runtime.txt deleted file mode 100644 index 57f558859..000000000 --- a/spec/fixtures/python_3.9/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.9.20 diff --git a/spec/fixtures/python_version_eol/.python-version b/spec/fixtures/python_version_eol/.python-version new file mode 100644 index 000000000..81cd8ab6c --- /dev/null +++ b/spec/fixtures/python_version_eol/.python-version @@ -0,0 +1,8 @@ +# Comments are supported. + # Even when indented +# +# So are empty lines, and leading/trailing whitespace. + + + 3.7 + diff --git a/spec/fixtures/python_3.7/requirements.txt b/spec/fixtures/python_version_eol/requirements.txt similarity index 100% rename from spec/fixtures/python_3.7/requirements.txt rename to spec/fixtures/python_version_eol/requirements.txt diff --git a/spec/fixtures/python_version_file_invalid_version/.python-version b/spec/fixtures/python_version_file_invalid_version/.python-version new file mode 100644 index 000000000..4ffb299eb --- /dev/null +++ b/spec/fixtures/python_version_file_invalid_version/.python-version @@ -0,0 +1,10 @@ +# Comments are supported. + # Even when indented +# +# So are empty lines, and leading/trailing whitespace. + + + 3.12.0invalid + + +# 2.7.18 diff --git a/spec/fixtures/python_version_invalid/requirements.txt b/spec/fixtures/python_version_file_invalid_version/requirements.txt similarity index 100% rename from spec/fixtures/python_version_invalid/requirements.txt rename to spec/fixtures/python_version_file_invalid_version/requirements.txt diff --git a/spec/fixtures/python_version_file_multiple_versions/.python-version b/spec/fixtures/python_version_file_multiple_versions/.python-version new file mode 100644 index 000000000..99b9a4e4c --- /dev/null +++ b/spec/fixtures/python_version_file_multiple_versions/.python-version @@ -0,0 +1,4 @@ +// invalid comment +# Valid comment +3.12 +2.7 diff --git a/spec/fixtures/python_version_file_multiple_versions/requirements.txt b/spec/fixtures/python_version_file_multiple_versions/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/python_version_file_no_version/.python-version b/spec/fixtures/python_version_file_no_version/.python-version new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/python_version_file_no_version/requirements.txt b/spec/fixtures/python_version_file_no_version/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/python_version_non_existent_major/.python-version b/spec/fixtures/python_version_non_existent_major/.python-version new file mode 100644 index 000000000..f1f7e4608 --- /dev/null +++ b/spec/fixtures/python_version_non_existent_major/.python-version @@ -0,0 +1 @@ +3.999 diff --git a/spec/fixtures/python_version_non_existent_major/runtime.txt b/spec/fixtures/python_version_non_existent_major/runtime.txt deleted file mode 100644 index b6885d9e1..000000000 --- a/spec/fixtures/python_version_non_existent_major/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.999.0 diff --git a/spec/fixtures/python_version_non_existent_patch/.python-version b/spec/fixtures/python_version_non_existent_patch/.python-version new file mode 100644 index 000000000..bb40ea005 --- /dev/null +++ b/spec/fixtures/python_version_non_existent_patch/.python-version @@ -0,0 +1 @@ +3.12.999 diff --git a/spec/fixtures/python_version_non_existent_patch/runtime.txt b/spec/fixtures/python_version_non_existent_patch/runtime.txt deleted file mode 100644 index f9907fe4a..000000000 --- a/spec/fixtures/python_version_non_existent_patch/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.12.999 diff --git a/spec/fixtures/python_version_outdated/.python-version b/spec/fixtures/python_version_outdated/.python-version new file mode 100644 index 000000000..a5c4c7633 --- /dev/null +++ b/spec/fixtures/python_version_outdated/.python-version @@ -0,0 +1 @@ +3.9.0 diff --git a/spec/fixtures/python_version_outdated/runtime.txt b/spec/fixtures/python_version_outdated/runtime.txt deleted file mode 100644 index f72c5111f..000000000 --- a/spec/fixtures/python_version_outdated/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.9.0 diff --git a/spec/fixtures/python_version_unspecified/requirements.txt b/spec/fixtures/python_version_unspecified/requirements.txt index a42590beb..eec3a2223 100644 --- a/spec/fixtures/python_version_unspecified/requirements.txt +++ b/spec/fixtures/python_version_unspecified/requirements.txt @@ -1 +1,2 @@ -urllib3 +# This package has been picked since it has no dependencies and is small/fast to install. +typing-extensions==4.12.2 diff --git a/spec/fixtures/requirements_basic/.python-version b/spec/fixtures/requirements_basic/.python-version new file mode 100644 index 000000000..e4fba2183 --- /dev/null +++ b/spec/fixtures/requirements_basic/.python-version @@ -0,0 +1 @@ +3.12 diff --git a/spec/fixtures/runtime_txt_and_python_version_file/.python-version b/spec/fixtures/runtime_txt_and_python_version_file/.python-version new file mode 100644 index 000000000..154e97ed3 --- /dev/null +++ b/spec/fixtures/runtime_txt_and_python_version_file/.python-version @@ -0,0 +1,2 @@ +# The version in runtime.txt should take precedence over this. +3.9 diff --git a/spec/fixtures/runtime_txt_and_python_version_file/requirements.txt b/spec/fixtures/runtime_txt_and_python_version_file/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/runtime_txt_and_python_version_file/runtime.txt b/spec/fixtures/runtime_txt_and_python_version_file/runtime.txt new file mode 100644 index 000000000..406dcdde4 --- /dev/null +++ b/spec/fixtures/runtime_txt_and_python_version_file/runtime.txt @@ -0,0 +1,3 @@ + + python-3.13 + \ No newline at end of file diff --git a/spec/fixtures/runtime_txt_eol_version/requirements.txt b/spec/fixtures/runtime_txt_eol_version/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/python_2.7/runtime.txt b/spec/fixtures/runtime_txt_eol_version/runtime.txt similarity index 100% rename from spec/fixtures/python_2.7/runtime.txt rename to spec/fixtures/runtime_txt_eol_version/runtime.txt diff --git a/spec/fixtures/runtime_txt_invalid_version/requirements.txt b/spec/fixtures/runtime_txt_invalid_version/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/python_version_invalid/runtime.txt b/spec/fixtures/runtime_txt_invalid_version/runtime.txt similarity index 100% rename from spec/fixtures/python_version_invalid/runtime.txt rename to spec/fixtures/runtime_txt_invalid_version/runtime.txt diff --git a/spec/fixtures/runtime_txt_only/runtime.txt b/spec/fixtures/runtime_txt_only/runtime.txt deleted file mode 100644 index 32905d6e0..000000000 --- a/spec/fixtures/runtime_txt_only/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.12.7 diff --git a/spec/fixtures/runtime_txt_with_stray_whitespace/requirements.txt b/spec/fixtures/runtime_txt_with_stray_whitespace/requirements.txt deleted file mode 100644 index a42590beb..000000000 --- a/spec/fixtures/runtime_txt_with_stray_whitespace/requirements.txt +++ /dev/null @@ -1 +0,0 @@ -urllib3 diff --git a/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt b/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt deleted file mode 100644 index 05a24a304..000000000 --- a/spec/fixtures/runtime_txt_with_stray_whitespace/runtime.txt +++ /dev/null @@ -1,3 +0,0 @@ - - python-3.12.7 - \ No newline at end of file diff --git a/spec/fixtures/setup_py_only/.python-version b/spec/fixtures/setup_py_only/.python-version new file mode 100644 index 000000000..e4fba2183 --- /dev/null +++ b/spec/fixtures/setup_py_only/.python-version @@ -0,0 +1 @@ +3.12 diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 444b65e55..80a97706c 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -12,8 +12,7 @@ app.run_ci do |test_run| expect(test_run.output).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) -----> Python app detected - -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} - To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing SQLite3 @@ -70,8 +69,7 @@ expect(test_run.output).to include(<<~OUTPUT) -----> Python app detected - -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} - To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> No change in requirements detected, installing from cache -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} @@ -92,8 +90,7 @@ app.run_ci do |test_run| expect(test_run.output).to match(Regexp.new(<<~REGEX)) -----> Python app detected - -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} - To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing Pipenv #{PIPENV_VERSION} @@ -148,8 +145,7 @@ expect(test_run.output).to match(Regexp.new(<<~REGEX)) -----> Python app detected - -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} - To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing Pipenv #{PIPENV_VERSION} diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index 70473fc81..cde1f154b 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -4,7 +4,7 @@ RSpec.describe 'Package manager support' do context 'when there are no supported package manager files' do - let(:app) { Hatchet::Runner.new('spec/fixtures/runtime_txt_only', allow_failure: true) } + let(:app) { Hatchet::Runner.new('spec/fixtures/pyproject_toml_only', allow_failure: true) } it 'fails the build with an informative error message' do app.deploy do |app| @@ -19,7 +19,7 @@ remote: ! remote: ! Currently the root directory of your app contains: remote: ! - remote: ! runtime.txt + remote: ! pyproject.toml remote: ! subdir/ remote: ! remote: ! If your app already has a package manager file, check that it: diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 178be593d..59b2ba0cc 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -22,8 +22,7 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 @@ -63,8 +62,7 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> No change in requirements detected, installing from cache remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} @@ -86,8 +84,7 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Requirements file has been changed, clearing cached dependencies remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} @@ -107,19 +104,19 @@ end context 'when the package manager has changed from Pipenv to pip since the last build' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_version_unspecified') } + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_basic') } # TODO: Fix this case so the cache is actually cleared. it 'clears the cache before installing with pip' do app.deploy do |app| FileUtils.rm(['Pipfile', 'Pipfile.lock']) + FileUtils.cp(FIXTURE_DIR.join('requirements_basic/.python-version'), '.') FileUtils.cp(FIXTURE_DIR.join('requirements_basic/requirements.txt'), '.') app.commit! app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 @@ -142,7 +139,7 @@ it 'rewrites .pth, .egg-link and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Running post-compile hook remote: easy-install.pth:/app/.heroku/src/gunicorn remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py @@ -181,7 +178,7 @@ # Test that the cached .pth files work correctly. app.commit! app.push! - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Running post-compile hook remote: easy-install.pth:/app/.heroku/src/gunicorn remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py @@ -214,8 +211,7 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 91d4fbb1a..8abe6fb6d 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -2,64 +2,20 @@ require_relative '../spec_helper' -RSpec.shared_examples 'builds using Pipenv with the requested Python version' do |requested_version, resolved_version| - it "builds with Python #{requested_version}" do - app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) - remote: -----> Python app detected - remote: -----> Using Python #{requested_version} specified in Pipfile.lock - remote: -----> Installing Python #{resolved_version} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing SQLite3 - remote: -----> Installing dependencies with Pipenv - remote: Installing dependencies from Pipfile.lock \\(.+\\)... - REGEX - end - end -end - RSpec.describe 'Pipenv support' do - context 'without a Pipfile.lock' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_no_lockfile') } - - it 'builds with the default Python version using just the Pipfile' do - app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) - remote: -----> Python app detected - remote: ! No 'Pipfile.lock' found! We recommend you commit this into your repository. - remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing SQLite3 - remote: -----> Installing dependencies with Pipenv - remote: The flag --skip-lock has been reintroduced \\(but is not recommended\\). Without - remote: the lock resolver it is difficult to manage multiple package indexes, and hash - remote: checking is not provided. However it can help manage installs with current - remote: deficiencies in locking across platforms. - remote: Pipfile.lock not found, creating... - .+ - remote: Installing dependencies from Pipfile... - REGEX - end - end - end - - context 'with a Pipfile.lock but no Python version specified' do + context 'with a Pipfile.lock that is unchanged since the last build' do let(:buildpacks) { [:default, 'heroku-community/inline'] } - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_version_unspecified', buildpacks:) } + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_basic', buildpacks:) } # TODO: Run this on Heroku-22 too, once it has also migrated to the new build infrastructure. # (Currently the test fails on the old infrastructure due to subtle differences in system PATH elements.) - it 'builds with the default Python version', stacks: %w[heroku-20 heroku-24] do + it 'builds with the specified python_version and re-uses packages from the cache', + stacks: %w[heroku-20 heroku-24] do app.deploy do |app| # TODO: We should not be leaking the Pipenv installation into the app environment. expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} @@ -98,127 +54,81 @@ remote: remote: \\ REGEX + app.commit! + app.push! + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Python app detected + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock + remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing Pipenv #{PIPENV_VERSION} + remote: -----> Installing SQLite3 + remote: -----> Installing dependencies with Pipenv + remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Inline app detected + REGEX end end end - context 'with a Pipfile.lock containing an EOL python_version' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.7', allow_failure: true) } + # As well as testing the Pipfile.lock `python_full_version` field, this also tests: + # 1. That `python_full_version` takes precedence over the `python_version` field. + # 2. That Pipenv works on the oldest Python version supported by all stacks. + # 3. That the security update available message works for Pipenv too. + context 'with a Pipfile.lock containing python_full_version 3.9.0' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_full_version') } - it 'aborts the build with an EOL message' do + it 'builds with the outdated Python version specified and displays a deprecation warning' do app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~OUTPUT)) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python 3.7 specified in Pipfile.lock - remote: - remote: ! Error: The requested Python version has reached end-of-life. - remote: ! - remote: ! Python 3.7 has reached its upstream end-of-life, and is - remote: ! therefore no longer receiving security updates: - remote: ! https://devguide.python.org/versions/#supported-versions - remote: ! - remote: ! As such, it is no longer supported by this buildpack. + remote: -----> Using Python 3.9.0 specified in Pipfile.lock remote: ! - remote: ! Please upgrade to a newer Python version by updating the - remote: ! version configured via the 'Pipfile.lock' file. + remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} + remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: - remote: ! Push rejected, failed to compile Python app. - OUTPUT + remote: -----> Installing Python 3.9.0 + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing Pipenv #{PIPENV_VERSION} + remote: -----> Installing SQLite3 + remote: -----> Installing dependencies with Pipenv + remote: Installing dependencies from Pipfile.lock \\(.+\\)... + REGEX end end end - context 'with a Pipfile.lock containing python_version 3.8' do - let(:allow_failure) { false } - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.8', allow_failure:) } + context 'when there is a both a Pipfile.lock python_version and a .python-version file' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_version_and_python_version_file') } - context 'when using Heroku-20', stacks: %w[heroku-20] do - it 'builds with the latest Python 3.8 but shows a deprecation warning' do - app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) - remote: -----> Python app detected - remote: -----> Using Python 3.8 specified in Pipfile.lock - remote: ! - remote: ! Python 3.8 will reach its upstream end-of-life in October 2024, at which - remote: ! point it will no longer receive security updates: - remote: ! https://devguide.python.org/versions/#supported-versions - remote: ! - remote: ! Support for Python 3.8 will be removed from this buildpack on December 4th, 2024. - remote: ! - remote: ! Upgrade to a newer Python version as soon as possible to keep your app secure. - remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: ! - remote: -----> Installing Python #{LATEST_PYTHON_3_8} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing SQLite3 - remote: -----> Installing dependencies with Pipenv - remote: Installing dependencies from Pipfile.lock \\(.+\\)... - REGEX - end - end - end - - context 'when using Heroku-22 or newer', stacks: %w[heroku-22 heroku-24] do - let(:allow_failure) { true } - - # We only support Python 3.8 on Heroku-20 and older. - it 'aborts the build with a version not available message' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python 3.8 specified in Pipfile.lock - remote: - remote: ! Error: Python #{LATEST_PYTHON_3_8} is not available for this stack (#{app.stack}). - remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: - remote: ! Push rejected, failed to compile Python app. - OUTPUT - end + it 'builds with the Python version from the .python-version file' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Python app detected + remote: -----> Using Python 3.13 specified in .python-version + remote: -----> Installing Python #{LATEST_PYTHON_3_13} + remote: -----> Installing pip #{PIP_VERSION} + remote: -----> Installing Pipenv #{PIPENV_VERSION} + remote: -----> Installing dependencies with Pipenv + remote: Installing dependencies from Pipfile.lock \\(.+\\)... + REGEX end end end - context 'with a Pipfile.lock containing python_version 3.9' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.9') } - - include_examples 'builds using Pipenv with the requested Python version', '3.9', LATEST_PYTHON_3_9 - end - - context 'with a Pipfile.lock containing python_version 3.10' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.10') } - - include_examples 'builds using Pipenv with the requested Python version', '3.10', LATEST_PYTHON_3_10 - end - - context 'with a Pipfile.lock containing python_version 3.11' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.11') } - - include_examples 'builds using Pipenv with the requested Python version', '3.11', LATEST_PYTHON_3_11 - end - - context 'with a Pipfile.lock containing python_version 3.12' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.12') } - - include_examples 'builds using Pipenv with the requested Python version', '3.12', LATEST_PYTHON_3_12 - end - - context 'with a Pipfile.lock containing python_version 3.13' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.13') } + context 'with a Pipfile.lock but no Python version specified' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_version_unspecified') } - it 'builds with latest Python 3.13' do + it 'builds with the default Python version' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python 3.13 specified in Pipfile.lock - remote: -----> Installing Python #{LATEST_PYTHON_3_13} - remote: -----> Installing pip #{PIP_VERSION} + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} + remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} + remote: -----> Installing SQLite3 remote: -----> Installing dependencies with Pipenv remote: Installing dependencies from Pipfile.lock \\(.+\\)... REGEX @@ -226,29 +136,28 @@ end end - # As well as testing `python_full_version`, this also tests: - # 1. That `python_full_version` takes precedence over `python_version`. - # 2. That Pipenv works on the oldest Python version supported by all stacks. - # 3. That the security update available message works for Pipenv too. - context 'with a Pipfile.lock containing python_full_version 3.9.0' do - let(:allow_failure) { false } - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_full_version', allow_failure:) } + context 'without a Pipfile.lock' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_no_lockfile') } - it 'builds with the outdated Python version specified and displays a deprecation warning' do + it 'builds with the default Python version using just the Pipfile' do app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected - remote: -----> Using Python 3.9.0 specified in Pipfile.lock - remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} - remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: ! - remote: -----> Installing Python 3.9.0 + remote: ! No 'Pipfile.lock' found! We recommend you commit this into your repository. + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} + remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing dependencies with Pipenv - remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: The flag --skip-lock has been reintroduced \\(but is not recommended\\). Without + remote: the lock resolver it is difficult to manage multiple package indexes, and hash + remote: checking is not provided. However it can help manage installs with current + remote: deficiencies in locking across platforms. + remote: Pipfile.lock not found, creating... + .+ + remote: Installing dependencies from Pipfile... REGEX end end @@ -263,9 +172,9 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected remote: - remote: ! Error: Cannot parse Pipfile.lock. + remote: ! Error: Can't parse Pipfile.lock. remote: ! - remote: ! A Pipfile.lock file was found, however, it could not be parsed: + remote: ! A Pipfile.lock file was found, however, it couldn't be parsed: remote: ! (jq: )?parse error: Invalid numeric literal at line 1, column 8 remote: ! remote: ! This is likely due to it not being valid JSON. @@ -289,14 +198,14 @@ remote: ! Error: Invalid Python version in Pipfile / Pipfile.lock. remote: ! remote: ! The Python version specified in Pipfile / Pipfile.lock by the - remote: ! 'python_version' or 'python_full_version' field is not valid. + remote: ! 'python_version' or 'python_full_version' field isn't valid. remote: ! remote: ! The following version was found: remote: ! ^3.12 remote: ! remote: ! However, the version must be specified as either: - remote: ! 1. '.' (recommended, for automatic security updates) - remote: ! 2. '..' (to pin to an exact Python version) + remote: ! 1. '.' (recommended, for automatic patch updates) + remote: ! 2. '..' (to pin to an exact patch version) remote: ! remote: ! Please update your 'Pipfile' to use a valid Python version and remote: ! then run 'pipenv lock' to regenerate the lockfile. @@ -321,14 +230,14 @@ remote: ! Error: Invalid Python version in Pipfile / Pipfile.lock. remote: ! remote: ! The Python version specified in Pipfile / Pipfile.lock by the - remote: ! 'python_version' or 'python_full_version' field is not valid. + remote: ! 'python_version' or 'python_full_version' field isn't valid. remote: ! remote: ! The following version was found: remote: ! 3.9.* remote: ! remote: ! However, the version must be specified as either: - remote: ! 1. '.' (recommended, for automatic security updates) - remote: ! 2. '..' (to pin to an exact Python version) + remote: ! 1. '.' (recommended, for automatic patch updates) + remote: ! 2. '..' (to pin to an exact patch version) remote: ! remote: ! Please update your 'Pipfile' to use a valid Python version and remote: ! then run 'pipenv lock' to regenerate the lockfile. @@ -342,63 +251,50 @@ end end - context 'when there is a both a Pipfile.lock python_version and a runtime.txt' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_and_runtime_txt') } - - it 'builds with the Python version from runtime.txt' do - app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) - remote: -----> Python app detected - remote: -----> Using Python #{LATEST_PYTHON_3_12} specified in runtime.txt - remote: -----> Installing Python #{LATEST_PYTHON_3_12} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing SQLite3 - remote: -----> Installing dependencies with Pipenv - remote: Installing dependencies from Pipfile.lock \\(.+\\)... - REGEX - end - end - end - - context 'when Pipfile.lock is unchanged since the last build' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_version_unspecified') } + context 'with a Pipfile.lock containing an EOL python_version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_version_eol', allow_failure: true) } - it 're-uses packages from the cache' do + it 'aborts the build with an EOL message' do app.deploy do |app| - app.commit! - app.push! - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to match(Regexp.new(<<~OUTPUT)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing SQLite3 - remote: -----> Installing dependencies with Pipenv - remote: Installing dependencies from Pipfile.lock \\(.+\\)... - remote: -----> Discovering process types - REGEX + remote: -----> Using Python 3.7 specified in Pipfile.lock + remote: + remote: ! Error: The requested Python version has reached end-of-life. + remote: ! + remote: ! Python 3.7 has reached its upstream end-of-life, and is + remote: ! therefore no longer receiving security updates: + remote: ! https://devguide.python.org/versions/#supported-versions + remote: ! + remote: ! As such, it is no longer supported by this buildpack. + remote: ! + remote: ! Please upgrade to a newer Python version by updating the + remote: ! version configured via the 'Pipfile.lock' file. + remote: ! + remote: ! For a list of the supported Python versions, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT end end end context 'when the package manager has changed from pip to Pipenv since the last build' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_unspecified') } + let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_basic') } # TODO: Fix this case so the cache is actually cleared. it 'clears the cache before installing with Pipenv' do app.deploy do |app| + FileUtils.rm('.python-version') FileUtils.rm('requirements.txt') - FileUtils.cp(FIXTURE_DIR.join('pipenv_python_version_unspecified/Pipfile'), '.') - FileUtils.cp(FIXTURE_DIR.join('pipenv_python_version_unspecified/Pipfile.lock'), '.') + FileUtils.cp(FIXTURE_DIR.join('pipenv_basic/Pipfile'), '.') + FileUtils.cp(FIXTURE_DIR.join('pipenv_basic/Pipfile.lock'), '.') app.commit! app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} @@ -459,7 +355,7 @@ it 'rewrites .pth, .egg-link and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Running post-compile hook remote: easy-install.pth:/app/.heroku/src/gunicorn remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py @@ -498,7 +394,7 @@ # Test that the cached .pth files work correctly. app.commit! app.push! - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Running post-compile hook remote: easy-install.pth:/app/.heroku/src/gunicorn remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index 787f24a18..664c15bc1 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -46,7 +46,7 @@ remote: -----> Python app detected remote: -----> Using Python 3.8.0 specified in runtime.txt remote: - remote: ! Error: Python 3.8.0 is not available for this stack (#{app.stack}). + remote: ! Error: Python 3.8.0 isn't available for this stack (#{app.stack}). remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes @@ -58,14 +58,14 @@ end end - context 'with a runtime.txt containing an outdated patch version' do + context 'with a .python-version file containing an outdated patch version' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_outdated') } it 'warns there is a Python update available' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python 3.9.0 specified in runtime.txt + remote: -----> Using Python 3.9.0 specified in .python-version remote: ! remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index e9751a920..ec6313d30 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -2,30 +2,30 @@ require_relative '../spec_helper' -RSpec.shared_examples 'builds with the requested Python version' do |requested_version| +RSpec.shared_examples 'builds with the requested Python version' do |requested_version, resolved_version| it "builds with Python #{requested_version}" do app.deploy do |app| - if requested_version.start_with?('3.13.') + if requested_version == '3.13' expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python #{requested_version} specified in runtime.txt - remote: -----> Installing Python #{requested_version} + remote: -----> Using Python #{requested_version} specified in .python-version + remote: -----> Installing Python #{resolved_version} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing requirements with pip - remote: Collecting urllib3 (from -r requirements.txt (line 1)) + remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT else expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python #{requested_version} specified in runtime.txt - remote: -----> Installing Python #{requested_version} + remote: -----> Using Python #{requested_version} specified in .python-version + remote: -----> Installing Python #{resolved_version} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip - remote: Collecting urllib3 (from -r requirements.txt (line 1)) + remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT end - expect(app.run('python -V')).to include("Python #{requested_version}") + expect(app.run('python -V')).to include("Python #{resolved_version}") end end end @@ -76,74 +76,16 @@ end end - context 'when runtime.txt contains python-2.7.18' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_2.7', allow_failure: true) } - - it 'aborts the build with an EOL message' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python 2.7.18 specified in runtime.txt - remote: - remote: ! Error: The requested Python version has reached end-of-life. - remote: ! - remote: ! Python 2.7 has reached its upstream end-of-life, and is - remote: ! therefore no longer receiving security updates: - remote: ! https://devguide.python.org/versions/#supported-versions - remote: ! - remote: ! As such, it is no longer supported by this buildpack. - remote: ! - remote: ! Please upgrade to a newer Python version by updating the - remote: ! version configured via the 'runtime.txt' file. - remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: - remote: ! Push rejected, failed to compile Python app. - OUTPUT - end - end - end - - context 'when runtime.txt contains python-3.7.17' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.7', allow_failure: true) } - - it 'aborts the build with an EOL message' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python 3.7.17 specified in runtime.txt - remote: - remote: ! Error: The requested Python version has reached end-of-life. - remote: ! - remote: ! Python 3.7 has reached its upstream end-of-life, and is - remote: ! therefore no longer receiving security updates: - remote: ! https://devguide.python.org/versions/#supported-versions - remote: ! - remote: ! As such, it is no longer supported by this buildpack. - remote: ! - remote: ! Please upgrade to a newer Python version by updating the - remote: ! version configured via the 'runtime.txt' file. - remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: - remote: ! Push rejected, failed to compile Python app. - OUTPUT - end - end - end - - context 'when runtime.txt contains python-3.8.20' do + context 'when .python-version contains Python 3.8' do let(:allow_failure) { false } let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.8', allow_failure:) } context 'when using Heroku-20', stacks: %w[heroku-20] do - it 'builds with Python 3.8.20 but shows a deprecation warning' do + it 'builds with latest Python 3.8 but shows a deprecation warning' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python #{LATEST_PYTHON_3_8} specified in runtime.txt + remote: -----> Using Python 3.8 specified in .python-version remote: ! remote: ! Python 3.8 will reach its upstream end-of-life in October 2024, at which remote: ! point it will no longer receive security updates: @@ -158,7 +100,7 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip - remote: Collecting urllib3 (from -r requirements.txt (line 1)) + remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT expect(app.run('python -V')).to include("Python #{LATEST_PYTHON_3_8}") end @@ -173,9 +115,9 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python #{LATEST_PYTHON_3_8} specified in runtime.txt + remote: -----> Using Python 3.8 specified in .python-version remote: - remote: ! Error: Python #{LATEST_PYTHON_3_8} is not available for this stack (#{app.stack}). + remote: ! Error: Python #{LATEST_PYTHON_3_8} isn't available for this stack (#{app.stack}). remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes @@ -187,62 +129,143 @@ end end - context 'when runtime.txt contains python-3.9.20' do + context 'when .python-version contains Python 3.9' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.9') } - include_examples 'builds with the requested Python version', LATEST_PYTHON_3_9 + include_examples 'builds with the requested Python version', '3.9', LATEST_PYTHON_3_9 end - context 'when runtime.txt contains python-3.10.15' do + context 'when .python-version contains Python 3.10' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.10') } - include_examples 'builds with the requested Python version', LATEST_PYTHON_3_10 + include_examples 'builds with the requested Python version', '3.10', LATEST_PYTHON_3_10 end - context 'when runtime.txt contains python-3.11.10' do + context 'when .python-version contains Python 3.11' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.11') } - include_examples 'builds with the requested Python version', LATEST_PYTHON_3_11 + include_examples 'builds with the requested Python version', '3.11', LATEST_PYTHON_3_11 end - context 'when runtime.txt contains python-3.12.7' do + context 'when .python-version contains Python 3.12' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.12') } - include_examples 'builds with the requested Python version', LATEST_PYTHON_3_12 + include_examples 'builds with the requested Python version', '3.12', LATEST_PYTHON_3_12 end - context 'when runtime.txt contains python-3.13.0' do + context 'when .python-version contains Python 3.13' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.13') } - include_examples 'builds with the requested Python version', LATEST_PYTHON_3_13 + include_examples 'builds with the requested Python version', '3.13', LATEST_PYTHON_3_13 end - context 'when runtime.txt contains an invalid Python version string' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_invalid', allow_failure: true) } + context 'when .python-version contains an invalid Python version string' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_file_invalid_version', allow_failure: true) } - it 'aborts the build with an invalid runtime.txt message' do + it 'aborts the build with an invalid .python-version message' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: - remote: ! Error: Invalid Python version in runtime.txt. + remote: ! Error: Invalid Python version in .python-version. remote: ! - remote: ! The Python version specified in 'runtime.txt' is not in + remote: ! The Python version specified in '.python-version' isn't in remote: ! the correct format. remote: ! - remote: ! The following file contents were found: - remote: ! python-3.12.0invalid + remote: ! The following version was found: + remote: ! 3.12.0invalid + remote: ! + remote: ! However, the version must be specified as either: + remote: ! 1. '.' (recommended, for automatic patch updates) + remote: ! 2. '..' (to pin to an exact patch version) + remote: ! + remote: ! Don't include quotes or a 'python-' prefix. To include + remote: ! comments, add them on their own line, prefixed with '#'. + remote: ! + remote: ! For example, to request the latest version of Python #{DEFAULT_PYTHON_MAJOR_VERSION}, + remote: ! update the '.python-version' file so it contains: + remote: ! #{DEFAULT_PYTHON_MAJOR_VERSION} + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + context 'when .python-version does not contain a Python version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_file_no_version', allow_failure: true) } + + it 'aborts the build with a no version string found message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: + remote: ! Error: Invalid Python version in .python-version. + remote: ! + remote: ! No Python version was found in the '.python-version' file. + remote: ! + remote: ! Update the file so that it contains a valid Python version + remote: ! such as '3.12'. + remote: ! + remote: ! If the file already contains a version, check the line doesn't + remote: ! begin with a '#', otherwise it will be treated as a comment. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + context 'when .python-version contains multiple Python versions' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_file_multiple_versions', allow_failure: true) } + + it 'aborts the build with a multiple versions not supported message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: + remote: ! Error: Invalid Python version in .python-version. remote: ! - remote: ! However, the version string must begin with a 'python-' prefix, - remote: ! followed by the version specified as '..'. - remote: ! Comments are not supported. + remote: ! Multiple Python versions were found in the '.python-version' + remote: ! file: remote: ! - remote: ! For example, to request Python 3.12.7, use: - remote: ! python-3.12.7 + remote: ! // invalid comment + remote: ! 3.12 + remote: ! 2.7 remote: ! - remote: ! Please update 'runtime.txt' to use a valid version string, or - remote: ! else remove the file to instead use the default version - remote: ! (currently Python 3.12.7). + remote: ! Update the file so it contains only one Python version. + remote: ! + remote: ! If the additional versions are actually comments, prefix + remote: ! those lines with '#'. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + context 'when .python-version contains an EOL Python 3.x version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_eol', allow_failure: true) } + + it 'aborts the build with an EOL message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.7 specified in .python-version + remote: + remote: ! Error: The requested Python version has reached end-of-life. + remote: ! + remote: ! Python 3.7 has reached its upstream end-of-life, and is + remote: ! therefore no longer receiving security updates: + remote: ! https://devguide.python.org/versions/#supported-versions + remote: ! + remote: ! As such, it is no longer supported by this buildpack. + remote: ! + remote: ! Please upgrade to a newer Python version by updating the + remote: ! version configured via the '.python-version' file. + remote: ! + remote: ! For a list of the supported Python versions, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -250,18 +273,18 @@ end end - context 'when runtime.txt contains an non-existent Python major version' do + context 'when .python-version contains an non-existent Python major version' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_non_existent_major', allow_failure: true) } - it 'aborts the build with an invalid runtime.txt message' do + it 'aborts the build with an invalid .python-version message' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python 3.999.0 specified in runtime.txt + remote: -----> Using Python 3.999 specified in .python-version remote: - remote: ! Error: The requested Python version is not recognised. + remote: ! Error: The requested Python version isn't recognised. remote: ! - remote: ! The requested Python version 3.999 is not recognised. + remote: ! The requested Python version 3.999 isn't recognised. remote: ! remote: ! Check that this Python version has been officially released, remote: ! and that the Python buildpack has added support for it: @@ -272,8 +295,8 @@ remote: ! of this buildpack: remote: ! https://devcenter.heroku.com/articles/python-support#checking-the-python-buildpack-version remote: ! - remote: ! Otherwise, switch to a supported version (such as Python 3.12) - remote: ! by updating the version configured via the 'runtime.txt' file. + remote: ! Otherwise, switch to a supported version (such as Python #{DEFAULT_PYTHON_MAJOR_VERSION}) + remote: ! by updating the version configured via the '.python-version' file. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -281,16 +304,77 @@ end end - context 'when runtime.txt contains a non-existent Python patch version' do + context 'when .python-version contains a non-existent Python patch version' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_non_existent_patch', allow_failure: true) } it 'aborts the build with a version not available message' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python 3.12.999 specified in runtime.txt + remote: -----> Using Python 3.12.999 specified in .python-version + remote: + remote: ! Error: Python 3.12.999 isn't available for this stack (#{app.stack}). + remote: ! + remote: ! For a list of the supported Python versions, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + context 'when runtime.txt contains an invalid Python version string' do + let(:app) { Hatchet::Runner.new('spec/fixtures/runtime_txt_invalid_version', allow_failure: true) } + + it 'aborts the build with an invalid runtime.txt message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected remote: - remote: ! Error: Python 3.12.999 is not available for this stack (#{app.stack}). + remote: ! Error: Invalid Python version in runtime.txt. + remote: ! + remote: ! The Python version specified in 'runtime.txt' isn't in + remote: ! the correct format. + remote: ! + remote: ! The following file contents were found: + remote: ! python-3.12.0invalid + remote: ! + remote: ! However, the version must be specified as either: + remote: ! 1. 'python-.' (recommended, for automatic patch updates) + remote: ! 2. 'python-..' (to pin to an exact patch version) + remote: ! + remote: ! Remember to include the 'python-' prefix. Comments aren't supported. + remote: ! + remote: ! For example, to request the latest version of Python #{DEFAULT_PYTHON_MAJOR_VERSION}, + remote: ! update the 'runtime.txt' file so it contains: + remote: ! python-#{DEFAULT_PYTHON_MAJOR_VERSION} + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + context 'when runtime.txt contains an EOL Python 2.x version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/runtime_txt_eol_version', allow_failure: true) } + + it 'aborts the build with an EOL message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 2.7.18 specified in runtime.txt + remote: + remote: ! Error: The requested Python version has reached end-of-life. + remote: ! + remote: ! Python 2.7 has reached its upstream end-of-life, and is + remote: ! therefore no longer receiving security updates: + remote: ! https://devguide.python.org/versions/#supported-versions + remote: ! + remote: ! As such, it is no longer supported by this buildpack. + remote: ! + remote: ! Please upgrade to a newer Python version by updating the + remote: ! version configured via the 'runtime.txt' file. remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes @@ -301,10 +385,22 @@ end end - context 'when runtime.txt contains stray whitespace' do - let(:app) { Hatchet::Runner.new('spec/fixtures/runtime_txt_with_stray_whitespace') } + # This also tests runtime.txt support for the major version only syntax, as well as the handling + # of runtime.txt files that contain stray whitespace. + context 'when there is both a runtime.txt and .python-version file' do + let(:app) { Hatchet::Runner.new('spec/fixtures/runtime_txt_and_python_version_file') } - include_examples 'builds with the requested Python version', LATEST_PYTHON_3_12 + it 'builds with the version from runtime.txt' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.13 specified in runtime.txt + remote: -----> Installing Python #{LATEST_PYTHON_3_13} + remote: -----> Installing pip #{PIP_VERSION} + remote: -----> Installing requirements with pip + OUTPUT + end + end end context 'when the requested Python version has changed since the last build' do @@ -312,20 +408,19 @@ it 'builds with the new Python version after removing the old install' do app.deploy do |app| - File.write('runtime.txt', "python-#{LATEST_PYTHON_3_12}") + File.write('.python-version', '3.13') app.commit! app.push! # TODO: The output shouldn't say "installing from cache", since it's not. expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python #{LATEST_PYTHON_3_12} specified in runtime.txt - remote: -----> Python version has changed from #{LATEST_PYTHON_3_9} to #{LATEST_PYTHON_3_12}, clearing cache + remote: -----> Using Python 3.13 specified in .python-version + remote: -----> Python version has changed from #{LATEST_PYTHON_3_9} to #{LATEST_PYTHON_3_13}, clearing cache remote: -----> No change in requirements detected, installing from cache - remote: -----> Installing Python #{LATEST_PYTHON_3_12} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing SQLite3 + remote: -----> Installing Python #{LATEST_PYTHON_3_13} + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing requirements with pip - remote: Collecting urllib3 (from -r requirements.txt (line 1)) + remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT end end diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index 0135e0628..3f55f5337 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -34,7 +34,7 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip - remote: Collecting urllib3 (from -r requirements.txt (line 1)) + remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT end end @@ -59,7 +59,7 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip - remote: Collecting urllib3 (from -r requirements.txt (line 1)) + remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT end end From 99684a6ffdb23e50a9a2ee0b59ddf3c5e865ea5d Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Mon, 14 Oct 2024 14:30:22 +0000 Subject: [PATCH 039/406] Prepare release v261 (#1665) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7563bc94c..8223b4187 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v261] - 2024-10-14 + - Added support for configuring the Python version using a `.python-version` file. Both the `3.N` and `3.N.N` version forms are supported (the former is recommended). The existing `runtime.txt` file will take precedence if both files are found, however, we recommend switching to `.python-version` since it is more commonly supported in the Python ecosystem. ([#1664](https://github.com/heroku/heroku-buildpack-python/pull/1664)) - Added support for specifying only the Python major version in `runtime.txt` instead of requiring the full Python version (for example `python-3.N` instead of `python-3.N.N`). ([#1664](https://github.com/heroku/heroku-buildpack-python/pull/1664)) @@ -1065,7 +1068,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v260...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v261...main +[v261]: https://github.com/heroku/heroku-buildpack-python/compare/v260...v261 [v260]: https://github.com/heroku/heroku-buildpack-python/compare/v259...v260 [v259]: https://github.com/heroku/heroku-buildpack-python/compare/v258...v259 [v258]: https://github.com/heroku/heroku-buildpack-python/compare/v257...v258 From f03ed3f3e71cbfa373f5efad250181791fdb8319 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 23 Oct 2024 14:10:01 +0000 Subject: [PATCH 040/406] Improve output utils for warnings, steps and indented output (#1666) * Switches warnings to the same approach used for errors (which also adds colour) * Switches to the new style utils for steps + indentation * Removes usage of the `exec 1>&2` pattern since it breaks the output returned by Bash functions, amongst other issues. GUS-W-17048951. --- CHANGELOG.md | 1 + bin/compile | 13 +---- bin/detect | 2 +- bin/steps/collectstatic | 43 ++++++++------ bin/steps/nltk | 11 ++-- bin/steps/python | 53 ++++++++++-------- bin/steps/sqlite3 | 2 +- bin/utils | 15 ----- bin/warnings | 10 ++-- lib/output.sh | 65 ++++++++++++++++++---- lib/package_manager.sh | 8 ++- lib/pip.sh | 10 ++-- lib/pipenv.sh | 14 ++--- lib/python_version.sh | 20 +++---- lib/utils.sh | 4 +- spec/hatchet/django_spec.rb | 26 ++++++--- spec/hatchet/nltk_spec.rb | 4 +- spec/hatchet/pip_spec.rb | 6 +- spec/hatchet/pipenv_spec.rb | 12 +++- spec/hatchet/python_update_warning_spec.rb | 16 ++++-- spec/hatchet/python_version_spec.rb | 10 +++- spec/hatchet/stack_spec.rb | 6 +- 22 files changed, 216 insertions(+), 135 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8223b4187..4379617db 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated buildpack-generated warning messages to use colour and be more consistently formatted. ([#1666](https://github.com/heroku/heroku-buildpack-python/pull/1666)) ## [v261] - 2024-10-14 diff --git a/bin/compile b/bin/compile index f751965fc..f318f0fa3 100755 --- a/bin/compile +++ b/bin/compile @@ -142,13 +142,6 @@ fi package_manager="$(package_manager::determine_package_manager "${BUILD_DIR}")" meta_set "package_manager" "${package_manager}" -# TODO: Move this warning to lib/package_manager.sh once `output::warning()` exists -# (puts-warn outputs to stdout, which would break `determine_package_manager()` as is). -# TODO: Adjust this warning to mention support for missing Pipfile.lock will be removed soon. -if [[ "${package_manager}" == "pipenv" && ! -f "${BUILD_DIR}/Pipfile.lock" ]]; then - puts-warn "No 'Pipfile.lock' found! We recommend you commit this into your repository." -fi - # We use the Bash 4.3+ `nameref` feature to pass back multiple values from this function # without having to hardcode globals. See: https://stackoverflow.com/a/38997681 python_version::read_requested_python_version "${BUILD_DIR}" "${package_manager}" "${cached_python_version}" requested_python_version python_version_origin @@ -159,15 +152,15 @@ meta_set "python_version_reason" "${python_version_origin}" # TODO: Add runtime.txt deprecation warning. case "${python_version_origin}" in default) - puts-step "No Python version was specified. Using the buildpack default: Python ${requested_python_version}" + output::step "No Python version was specified. Using the buildpack default: Python ${requested_python_version}" echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" ;; cached) - puts-step "No Python version was specified. Using the same version as the last build: Python ${requested_python_version}" + output::step "No Python version was specified. Using the same version as the last build: Python ${requested_python_version}" echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" ;; *) - puts-step "Using Python ${requested_python_version} specified in ${python_version_origin}" + output::step "Using Python ${requested_python_version} specified in ${python_version_origin}" ;; esac diff --git a/bin/detect b/bin/detect index 37d7ccdb1..9f6303298 100755 --- a/bin/detect +++ b/bin/detect @@ -45,7 +45,7 @@ done # Note: This error message intentionally doesn't list all of the filetypes above, # since during compile the build will still require a package manager file, so it # makes sense to describe the stricter requirements upfront. -display_error <&1 | tee "$COLLECTSTATIC_LOG" | sed '/^Post-processed/d;/^Copying/d;/^$/d' | indent +python "$MANAGE_FILE" collectstatic --noinput --traceback 2>&1 | tee "$COLLECTSTATIC_LOG" | sed '/^Post-processed/d;/^Copying/d;/^$/d' | output::indent COLLECTSTATIC_STATUS="${PIPESTATUS[0]}" set -e @@ -82,22 +87,28 @@ else meta_set "failure_reason" "collectstatic-other" fi -echo " ! Error while running '$ python $MANAGE_FILE collectstatic --noinput'." -echo " See traceback above for details." -echo -echo " You may need to update application code to resolve this error." -echo " Or, you can disable collectstatic for this application:" -echo -echo " $ heroku config:set DISABLE_COLLECTSTATIC=1" -echo -echo " https://devcenter.heroku.com/articles/django-assets" +output::error <<-EOF + Error: Unable to generate Django static files. + + The 'python ${MANAGE_FILE} collectstatic --noinput' Django + management command to generate static files failed. + + See the traceback above for details. + + You may need to update application code to resolve this error. + Or, you can disable collectstatic for this application: + + $ heroku config:set DISABLE_COLLECTSTATIC=1 + + https://devcenter.heroku.com/articles/django-assets +EOF # Additionally, dump out the environment, if debug mode is on. if [[ "${DEBUG_COLLECTSTATIC:-0}" == "1" ]]; then echo echo "****** Collectstatic environment variables:" echo - env | indent + env | output::indent fi exit 1 diff --git a/bin/steps/nltk b/bin/steps/nltk index c849285ac..804e7a41d 100755 --- a/bin/steps/nltk +++ b/bin/steps/nltk @@ -6,6 +6,7 @@ set -euo pipefail BUILDPACK_DIR=$(cd "$(dirname "$(dirname "$(dirname "${BASH_SOURCE[0]}")")")" && pwd) source "${BUILDPACK_DIR}/bin/utils" +source "${BUILDPACK_DIR}/lib/output.sh" # Required for `meta_set`. source "${BUILDPACK_DIR}/lib/metadata.sh" @@ -18,7 +19,7 @@ EXPORT_PATH="${BUILDPACK_DIR}/export" # Check that nltk was installed by pip, otherwise obviously not needed # shellcheck disable=SC2310 # TODO: This function is invoked in an 'if' condition so set -e will be disabled. if is_module_available 'nltk'; then - puts-step "Downloading NLTK corpora..." + output::step "Downloading NLTK corpora..." nltk_packages_definition="$BUILD_DIR/nltk.txt" @@ -26,14 +27,14 @@ if is_module_available 'nltk'; then meta_set "nltk_downloader" "enabled" readarray -t nltk_packages <"$nltk_packages_definition" - puts-step "Downloading NLTK packages: ${nltk_packages[*]}" + output::step "Downloading NLTK packages: ${nltk_packages[*]}" - python -m nltk.downloader -d "$BUILD_DIR/.heroku/python/nltk_data" "${nltk_packages[@]}" | indent + python -m nltk.downloader -d "$BUILD_DIR/.heroku/python/nltk_data" "${nltk_packages[@]}" | output::indent set_env NLTK_DATA "/app/.heroku/python/nltk_data" else meta_set "nltk_downloader" "skipped-no-nltk-file" - puts-warn "'nltk.txt' not found, not downloading any corpora" - puts-warn "Learn more: https://devcenter.heroku.com/articles/python-nltk" + echo " 'nltk.txt' not found, not downloading any corpora" + echo " Learn more: https://devcenter.heroku.com/articles/python-nltk" fi fi diff --git a/bin/steps/python b/bin/steps/python index e4b605df2..37fbf6a34 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -17,7 +17,7 @@ PYTHON_URL="${S3_BASE_URL}/python-${python_full_version}-ubuntu-${UBUNTU_VERSION # 3. The user has pinned to an older buildpack version and the S3 bucket location or layout has changed since. # TODO: Update this message to be more specific once Python 3.8 support is dropped. if ! curl --output /dev/null --silent --head --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}"; then - display_error <<-EOF + output::error <<-EOF Error: Python ${python_full_version} isn't available for this stack (${STACK}). For a list of the supported Python versions, see: @@ -38,10 +38,12 @@ function warn_if_patch_update_available() { # TODO: Update this message to suggest using the .python-version major version syntax to stay up to date, # once runtime.txt is deprecated and sticky-versioning only pins to the major version. if ((requested_patch_number < latest_patch_number)); then - puts-warn - puts-warn "A Python security update is available! Upgrade as soon as possible to: Python ${latest_patch_version}" - puts-warn "See: https://devcenter.heroku.com/articles/python-runtimes" - puts-warn + output::warning <<-EOF + Warning: A Python security update is available! + + Upgrade as soon as possible to: Python ${latest_patch_version} + See: https://devcenter.heroku.com/articles/python-runtimes + EOF meta_set "python_version_outdated" "true" else meta_set "python_version_outdated" "false" @@ -52,29 +54,31 @@ function warn_if_patch_update_available() { # if there weren't any errors with the version to avoid adding noise to the error messages. # TODO: Move this into lib/ as part of the warnings refactor. if [[ "${python_major_version}" == "3.8" ]]; then - puts-warn - puts-warn "Python 3.8 will reach its upstream end-of-life in October 2024, at which" - puts-warn "point it will no longer receive security updates:" - puts-warn "https://devguide.python.org/versions/#supported-versions" - puts-warn - puts-warn "Support for Python 3.8 will be removed from this buildpack on December 4th, 2024." - puts-warn - puts-warn "Upgrade to a newer Python version as soon as possible to keep your app secure." - puts-warn "See: https://devcenter.heroku.com/articles/python-runtimes" - puts-warn + output::warning <<-'EOF' + Warning: Support for Python 3.8 is ending soon! + + Python 3.8 will reach its upstream end-of-life in October 2024, at which + point it will no longer receive security updates: + https://devguide.python.org/versions/#supported-versions + + Support for Python 3.8 will be removed from this buildpack on December 4th, 2024. + + Upgrade to a newer Python version as soon as possible to keep your app secure. + See: https://devcenter.heroku.com/articles/python-runtimes + EOF fi warn_if_patch_update_available "${python_full_version}" "${python_major_version}" if [[ "$STACK" != "$CACHED_PYTHON_STACK" ]]; then - puts-step "Stack has changed from $CACHED_PYTHON_STACK to $STACK, clearing cache" + output::step "Stack has changed from $CACHED_PYTHON_STACK to $STACK, clearing cache" rm -rf .heroku/python-stack .heroku/python-version .heroku/python .heroku/vendor .heroku/python .heroku/python-sqlite3-version fi # TODO: Clean this up as part of the cache refactor. if [[ -f .heroku/python-version ]]; then if [[ "${cached_python_version}" != "${python_full_version}" ]]; then - puts-step "Python version has changed from ${cached_python_version} to ${python_full_version}, clearing cache" + output::step "Python version has changed from ${cached_python_version} to ${python_full_version}, clearing cache" rm -rf .heroku/python else SKIP_INSTALL=1 @@ -92,22 +96,22 @@ if [[ -f "${BUILD_DIR}/requirements.txt" ]]; then else # IF there IS a cached directory, check for differences with the new one if ! diff "$BUILD_DIR/requirements.txt" "$CACHE_DIR/.heroku/requirements.txt" &>/dev/null; then - puts-step "Requirements file has been changed, clearing cached dependencies" + output::step "Requirements file has been changed, clearing cached dependencies" # if there are any differences, clear the Python cache # Installing Python over again does not take noticably more time cp -R "$BUILD_DIR/requirements.txt" "$CACHE_DIR/.heroku/requirements.txt" rm -rf .heroku/python unset SKIP_INSTALL else - puts-step "No change in requirements detected, installing from cache" + output::step "No change in requirements detected, installing from cache" fi fi fi if [[ "${SKIP_INSTALL:-0}" == "1" ]]; then - puts-step "Using cached install of Python ${python_full_version}" + output::step "Using cached install of Python ${python_full_version}" else - puts-step "Installing Python ${python_full_version}" + output::step "Installing Python ${python_full_version}" # Prepare destination directory. mkdir -p .heroku/python @@ -115,7 +119,12 @@ else if ! curl --silent --show-error --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}" | tar --zstd --extract --directory .heroku/python; then # The Python version was confirmed to exist previously, so any failure here is due to # a networking issue or archive/buildpack bug rather than the runtime not existing. - display_error "Error: Failed to download/install Python ${python_full_version}." + output::error <<-EOF + Error: Failed to download/install Python ${python_full_version}. + + In some cases, this happens due to an unstable network connection. + Please try again and to see if the error resolves itself. + EOF meta_set "failure_reason" "python-download" exit 1 fi diff --git a/bin/steps/sqlite3 b/bin/steps/sqlite3 index a174fa2b2..3ef119a35 100755 --- a/bin/steps/sqlite3 +++ b/bin/steps/sqlite3 @@ -63,7 +63,7 @@ sqlite3_install() { } buildpack_sqlite3_install() { - puts-step "Installing SQLite3" + output::step "Installing SQLite3" # TODO: This never actual prints failure or even aborts the build, since # the conditional disables `set -e` inside the called function: diff --git a/bin/utils b/bin/utils index 45ab374d3..c2b25254b 100755 --- a/bin/utils +++ b/bin/utils @@ -10,26 +10,11 @@ source "${BUILDPACK_DIR:?}/vendor/buildpack-stdlib_v8.sh" sed() { command sed -u "$@"; } -# Syntax sugar. -indent() { - sed "s/^/ /" -} - # Clean up pip output cleanup() { sed -e 's/\.\.\.\+/.../g' | sed -e '/already satisfied/Id' | sed -e '/No files were found to uninstall/Id' | sed -e '/Overwriting/Id' | sed -e '/python executable/Id' | sed -e '/no previously-included files/Id' } -# Buildpack Steps. -puts-step() { - echo "-----> $*" -} - -# Buildpack Warnings. -puts-warn() { - echo " ! $*" -} - # Does some serious copying. deep-cp() { declare source="$1" target="$2" diff --git a/bin/warnings b/bin/warnings index e24d41ea9..55cb9b6fb 100755 --- a/bin/warnings +++ b/bin/warnings @@ -3,10 +3,12 @@ gdal-missing() { # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. if grep -qi 'Could not find gdal-config' "${WARNINGS_LOG}"; then - echo - puts-warn "Hello! Package installation failed since the GDAL library was not found." - puts-warn "For GDAL, GEOS and PROJ support, use the Geo buildpack alongside the Python buildpack:" - puts-warn "https://github.com/heroku/heroku-geo-buildpack" + output::error <<-'EOF' + Error: Package installation failed since the GDAL library was not found. + + For GDAL, GEOS and PROJ support, use the Geo buildpack alongside the Python buildpack: + https://github.com/heroku/heroku-geo-buildpack + EOF fi } diff --git a/lib/output.sh b/lib/output.sh index 4141e1a20..a94074a1c 100644 --- a/lib/output.sh +++ b/lib/output.sh @@ -4,21 +4,62 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail -# TODO: Switch this file to using namespaced functions like `output::`. - ANSI_RED='\033[1;31m' +ANSI_YELLOW='\033[1;33m' ANSI_RESET='\033[0m' -# shellcheck disable=SC2120 # Prevent warnings about unused arguments due to the split args vs stdin API. -function display_error() { - # Send all output to stderr - exec 1>&2 - # If arguments are given, redirect them to stdin. This allows the function - # to be invoked with either a string argument or stdin (e.g. via <<-EOF). - (($#)) && exec <<<"${@}" - echo +# Output a single line step message to stdout. +# +# Usage: +# ``` +# output::step "Installing Python ..." +# ``` +function output::step() { + echo "-----> ${1}" +} + +# Indent passed stdout. Typically used to indent command output within a step. +# +# Usage: +# ``` +# pip install ... | output::indent +# ``` +function output::indent() { + sed --unbuffered "s/^/ /" +} + +# Output a styled multi-line warning message to stderr. +# +# Usage: +# ``` +# output::warning <<-EOF +# Warning: The warning summary. +# +# Detailed description. +# EOF +# ``` +function output::warning() { + echo >&2 + while IFS= read -r line; do + echo -e "${ANSI_YELLOW} ! ${line}${ANSI_RESET}" >&2 + done + echo >&2 +} + +# Output a styled multi-line error message to stderr. +# +# Usage: +# ``` +# output::error <<-EOF +# Error: The error summary. +# +# Detailed description. +# EOF +# ``` +function output::error() { + echo >&2 while IFS= read -r line; do - echo -e "${ANSI_RED} ! ${line}${ANSI_RESET}" + echo -e "${ANSI_RED} ! ${line}${ANSI_RESET}" >&2 done - echo + echo >&2 } diff --git a/lib/package_manager.sh b/lib/package_manager.sh index 544ea238a..4c9fdd7f6 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -13,6 +13,12 @@ function package_manager::determine_package_manager() { meta_set "pipenv_has_lockfile" "true" elif [[ -f "${build_dir}/Pipfile" ]]; then # TODO: Start requiring a Pipfile.lock and make this branch a "missing lockfile" error instead. + # TODO: Adjust this warning to mention support for missing Pipfile.lock will be removed soon. + output::warning <<-'EOF' + Warning: No 'Pipfile.lock' found! + + We recommend you commit this into your repository. + EOF package_managers_found+=(pipenv) meta_set "pipenv_has_lockfile" "false" fi @@ -36,7 +42,7 @@ function package_manager::determine_package_manager() { return 0 ;; 0) - display_error <<-EOF + output::error <<-EOF Error: Couldn't find any supported Python package manager files. A Python app on Heroku must have either a 'requirements.txt' or diff --git a/lib/pip.sh b/lib/pip.sh index c8f554730..eb4535a2d 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -39,14 +39,14 @@ function pip::install_pip_setuptools_wheel() { packages_display_text+=", setuptools ${SETUPTOOLS_VERSION} and wheel ${WHEEL_VERSION}" fi - puts-step "Installing ${packages_display_text}" + output::step "Installing ${packages_display_text}" /app/.heroku/python/bin/python "${bundled_pip_module_path}" install --quiet --disable-pip-version-check --no-cache-dir \ "${packages_to_install[@]}" } function pip::install_dependencies() { - puts-step "Installing requirements with pip" + output::step "Installing requirements with pip" # Make select pip config vars set on the Heroku app available to pip. # TODO: Expose all config vars (after suitable checks are added for unsafe env vars) @@ -68,7 +68,7 @@ function pip::install_dependencies() { set +e # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. - /app/.heroku/python/bin/pip install "${args[@]}" --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir --progress-bar off 2>&1 | tee "${WARNINGS_LOG}" | cleanup | indent + /app/.heroku/python/bin/pip install "${args[@]}" --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir --progress-bar off 2>&1 | tee "${WARNINGS_LOG}" | cleanup | output::indent local PIP_STATUS="${PIPESTATUS[0]}" set -e @@ -82,8 +82,8 @@ function pip::install_dependencies() { # Install test dependencies, for Heroku CI. if [[ "${INSTALL_TEST:-0}" == "1" ]]; then if [[ -f requirements-test.txt ]]; then - puts-step "Installing test dependencies..." - /app/.heroku/python/bin/pip install -r requirements-test.txt --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir 2>&1 | cleanup | indent + output::step "Installing test dependencies..." + /app/.heroku/python/bin/pip install -r requirements-test.txt --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir 2>&1 | cleanup | output::indent fi fi } diff --git a/lib/pipenv.sh b/lib/pipenv.sh index f718a7d2b..bfe0032d6 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -12,7 +12,7 @@ function pipenv::install_pipenv() { PIPENV_VERSION=$(get_requirement_version 'pipenv') meta_set "pipenv_version" "${PIPENV_VERSION}" - puts-step "Installing Pipenv ${PIPENV_VERSION}" + output::step "Installing Pipenv ${PIPENV_VERSION}" # TODO: Install Pipenv into a venv so it isn't leaked into the app environment. # TODO: Explore viability of making Pipenv only be available during the build, to reduce slug size. @@ -38,16 +38,16 @@ function pipenv::install_dependencies() { # Install the test dependencies, for CI. # TODO: This is currently inconsistent with the non-test path, since it assumes (but doesn't check for) a lockfile. if [[ "${INSTALL_TEST:-0}" == "1" ]]; then - puts-step "Installing test dependencies with Pipenv" - /app/.heroku/python/bin/pipenv install --dev --system --deploy --extra-pip-args='--src=/app/.heroku/src' 2>&1 | cleanup | indent + output::step "Installing test dependencies with Pipenv" + /app/.heroku/python/bin/pipenv install --dev --system --deploy --extra-pip-args='--src=/app/.heroku/src' 2>&1 | cleanup | output::indent # Install the dependencies. elif [[ ! -f Pipfile.lock ]]; then - puts-step "Installing dependencies with Pipenv" - /app/.heroku/python/bin/pipenv install --system --skip-lock --extra-pip-args='--src=/app/.heroku/src' 2>&1 | indent + output::step "Installing dependencies with Pipenv" + /app/.heroku/python/bin/pipenv install --system --skip-lock --extra-pip-args='--src=/app/.heroku/src' 2>&1 | output::indent else - puts-step "Installing dependencies with Pipenv" - /app/.heroku/python/bin/pipenv install --system --deploy --extra-pip-args='--src=/app/.heroku/src' 2>&1 | indent + output::step "Installing dependencies with Pipenv" + /app/.heroku/python/bin/pipenv install --system --deploy --extra-pip-args='--src=/app/.heroku/src' 2>&1 | output::indent fi } diff --git a/lib/python_version.sh b/lib/python_version.sh index aa4df3539..bd52223fa 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -96,7 +96,7 @@ function python_version::parse_runtime_txt() { local version="${BASH_REMATCH[1]}" echo "${version}" else - display_error <<-EOF + output::error <<-EOF Error: Invalid Python version in runtime.txt. The Python version specified in 'runtime.txt' isn't in @@ -140,7 +140,7 @@ function python_version::parse_python_version_file() { echo "${version}" return 0 else - display_error <<-EOF + output::error <<-EOF Error: Invalid Python version in .python-version. The Python version specified in '.python-version' isn't in @@ -165,7 +165,7 @@ function python_version::parse_python_version_file() { fi ;; 0) - display_error <<-EOF + output::error <<-EOF Error: Invalid Python version in .python-version. No Python version was found in the '.python-version' file. @@ -180,7 +180,7 @@ function python_version::parse_python_version_file() { return 1 ;; *) - display_error <<-EOF + output::error <<-EOF Error: Invalid Python version in .python-version. Multiple Python versions were found in the '.python-version' @@ -218,7 +218,7 @@ function python_version::read_pipenv_python_version() { fi if ! version=$(jq --raw-output '._meta.requires.python_full_version // ._meta.requires.python_version' "${pipfile_lock_path}" 2>&1); then - display_error <<-EOF + output::error <<-EOF Error: Can't parse Pipfile.lock. A Pipfile.lock file was found, however, it couldn't be parsed: @@ -243,7 +243,7 @@ function python_version::read_pipenv_python_version() { if [[ "${version}" =~ ^${PYTHON_VERSION_REGEX}$ ]]; then echo "${version}" else - display_error <<-EOF + output::error <<-EOF Error: Invalid Python version in Pipfile / Pipfile.lock. The Python version specified in Pipfile / Pipfile.lock by the @@ -283,7 +283,7 @@ function python_version::resolve_python_version() { if ((major < 3 || (major == 3 && minor < 8))); then if [[ "${python_version_origin}" == "cached" ]]; then - display_error <<-EOF + output::error <<-EOF Error: The cached Python version has reached end-of-life. Your app doesn't specify a Python version, and so normally @@ -303,7 +303,7 @@ function python_version::resolve_python_version() { https://devcenter.heroku.com/articles/python-support#supported-runtimes EOF else - display_error <<-EOF + output::error <<-EOF Error: The requested Python version has reached end-of-life. Python ${major}.${minor} has reached its upstream end-of-life, and is @@ -325,7 +325,7 @@ function python_version::resolve_python_version() { if (((major == 3 && minor > 13) || major >= 4)); then if [[ "${python_version_origin}" == "cached" ]]; then - display_error <<-EOF + output::error <<-EOF Error: The cached Python version isn't recognised. Your app doesn't specify a Python version, and so normally @@ -340,7 +340,7 @@ function python_version::resolve_python_version() { Please switch back to a newer version of this buildpack. EOF else - display_error <<-EOF + output::error <<-EOF Error: The requested Python version isn't recognised. The requested Python version ${major}.${minor} isn't recognised. diff --git a/lib/utils.sh b/lib/utils.sh index 7d702c54f..c64bbfc79 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -18,7 +18,7 @@ function utils::bundled_pip_module_path() { local bundled_pip_wheel="${bundled_pip_wheel_list[0]}" if [[ -z "${bundled_pip_wheel}" ]]; then - display_error "Error: Failed to locate the bundled pip wheel." + output::error "Error: Failed to locate the bundled pip wheel." meta_set "failure_reason" "bundled-pip-not-found" return 1 fi @@ -28,7 +28,7 @@ function utils::bundled_pip_module_path() { function utils::abort_internal_error() { local message="${1}" - display_error "Internal error: ${message} (line $(caller || true))." + output::error "Internal error: ${message} (line $(caller || true))." meta_set "failure_reason" "internal-error" exit 1 } diff --git a/spec/hatchet/django_spec.rb b/spec/hatchet/django_spec.rb index bd7feb1a9..38f7e3e0a 100644 --- a/spec/hatchet/django_spec.rb +++ b/spec/hatchet/django_spec.rb @@ -82,7 +82,11 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Skipping Django collectstatic since the file '.heroku/collectstatic_disabled' exists. - remote: ! This approach is deprecated, please set the env var DISABLE_COLLECTSTATIC=1 instead. + remote: + remote: ! Warning: The .heroku/collectstatic_disabled file is deprecated. + remote: ! + remote: ! Please remove the file and set the env var DISABLE_COLLECTSTATIC=1 instead. + remote: OUTPUT expect(app.output).not_to include('manage.py collectstatic') end @@ -100,15 +104,21 @@ remote: .+ remote: ModuleNotFoundError: No module named 'gettingstarted' remote: - remote: ! Error while running '\\$ python manage.py collectstatic --noinput'. - remote: See traceback above for details. - remote: - remote: You may need to update application code to resolve this error. - remote: Or, you can disable collectstatic for this application: remote: - remote: \\$ heroku config:set DISABLE_COLLECTSTATIC=1 + remote: ! Error: Unable to generate Django static files. + remote: ! + remote: ! The 'python manage.py collectstatic --noinput' Django + remote: ! management command to generate static files failed. + remote: ! + remote: ! See the traceback above for details. + remote: ! + remote: ! You may need to update application code to resolve this error. + remote: ! Or, you can disable collectstatic for this application: + remote: ! + remote: ! \\$ heroku config:set DISABLE_COLLECTSTATIC=1 + remote: ! + remote: ! https://devcenter.heroku.com/articles/django-assets remote: - remote: https://devcenter.heroku.com/articles/django-assets remote: ! Push rejected, failed to compile Python app. REGEX end diff --git a/spec/hatchet/nltk_spec.rb b/spec/hatchet/nltk_spec.rb index 0ed2a167f..28b1ab660 100644 --- a/spec/hatchet/nltk_spec.rb +++ b/spec/hatchet/nltk_spec.rb @@ -32,8 +32,8 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Downloading NLTK corpora... - remote: ! 'nltk.txt' not found, not downloading any corpora - remote: ! Learn more: https://devcenter.heroku.com/articles/python-nltk + remote: 'nltk.txt' not found, not downloading any corpora + remote: Learn more: https://devcenter.heroku.com/articles/python-nltk REGEX end end diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 59b2ba0cc..0a4cbec57 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -261,9 +261,13 @@ it 'outputs instructions for how to resolve the build failure' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: ! Hello! Package installation failed since the GDAL library was not found. + remote: note: This error originates from a subprocess, and is likely not a problem with pip. + remote: + remote: ! Error: Package installation failed since the GDAL library was not found. + remote: ! remote: ! For GDAL, GEOS and PROJ support, use the Geo buildpack alongside the Python buildpack: remote: ! https://github.com/heroku/heroku-geo-buildpack + remote: remote: ! Push rejected, failed to compile Python app. OUTPUT end diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 8abe6fb6d..02686accf 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -83,10 +83,12 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected remote: -----> Using Python 3.9.0 specified in Pipfile.lock + remote: + remote: ! Warning: A Python security update is available! remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} + remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: ! + remote: remote: -----> Installing Python 3.9.0 remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} @@ -143,7 +145,11 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected - remote: ! No 'Pipfile.lock' found! We recommend you commit this into your repository. + remote: + remote: ! Warning: No 'Pipfile.lock' found! + remote: ! + remote: ! We recommend you commit this into your repository. + remote: remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index 664c15bc1..c10e5c288 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -16,6 +16,8 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python 3.8.0 specified in runtime.txt + remote: + remote: ! Warning: Support for Python 3.8 is ending soon! remote: ! remote: ! Python 3.8 will reach its upstream end-of-life in October 2024, at which remote: ! point it will no longer receive security updates: @@ -25,11 +27,13 @@ remote: ! remote: ! Upgrade to a newer Python version as soon as possible to keep your app secure. remote: ! See: https://devcenter.heroku.com/articles/python-runtimes + remote: + remote: + remote: ! Warning: A Python security update is available! remote: ! - remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_8} + remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_8} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: ! + remote: remote: -----> Installing Python 3.8.0 OUTPUT end @@ -66,10 +70,12 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python 3.9.0 specified in .python-version + remote: + remote: ! Warning: A Python security update is available! remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} + remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: ! + remote: remote: -----> Installing Python 3.9.0 OUTPUT end diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index ec6313d30..215148806 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -63,10 +63,12 @@ remote: -----> Python app detected remote: -----> No Python version was specified. Using the same version as the last build: Python 3.12.3 remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: + remote: ! Warning: A Python security update is available! remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_12} + remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_12} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: ! + remote: remote: -----> No change in requirements detected, installing from cache remote: -----> Using cached install of Python 3.12.3 OUTPUT @@ -86,6 +88,8 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python 3.8 specified in .python-version + remote: + remote: ! Warning: Support for Python 3.8 is ending soon! remote: ! remote: ! Python 3.8 will reach its upstream end-of-life in October 2024, at which remote: ! point it will no longer receive security updates: @@ -95,7 +99,7 @@ remote: ! remote: ! Upgrade to a newer Python version as soon as possible to keep your app secure. remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: ! + remote: remote: -----> Installing Python #{LATEST_PYTHON_3_8} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index 3f55f5337..9c84e5d16 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -24,10 +24,12 @@ remote: -----> Python app detected remote: -----> No Python version was specified. Using the same version as the last build: Python 3.12.3 remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: + remote: ! Warning: A Python security update is available! remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_12} + remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_12} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: ! + remote: remote: -----> Stack has changed from heroku-22 to heroku-24, clearing cache remote: -----> No change in requirements detected, installing from cache remote: -----> Installing Python 3.12.3 From cbe99d6712681c264ceb92eabba9d05292352a1d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 23 Oct 2024 21:03:34 +0000 Subject: [PATCH 041/406] Bump the ruby-dependencies group with 2 updates (#1669) * Bump the ruby-dependencies group with 2 updates Bumps the ruby-dependencies group with 2 updates: [rspec-core](https://github.com/rspec/rspec-core) and [rubocop](https://github.com/rubocop/rubocop). Updates `rspec-core` from 3.13.1 to 3.13.2 - [Release notes](https://github.com/rspec/rspec-core/releases) - [Changelog](https://github.com/rspec/rspec-core/blob/main/Changelog.md) - [Commits](https://github.com/rspec/rspec-core/compare/v3.13.1...v3.13.2) Updates `rubocop` from 1.66.1 to 1.67.0 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.66.1...v1.67.0) --- updated-dependencies: - dependency-name: rspec-core dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rubocop dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] * Bump the minimum Ruby version to 3.2 Since 3.1 is due to EOL soon. * Add explicit dependency on `logger` To work around: https://github.com/excon/excon/issues/860 And refresh the lockfile. --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- Gemfile | 4 +++- Gemfile.lock | 10 ++++++---- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/Gemfile b/Gemfile index 6bd480a38..ddaf2f60c 100644 --- a/Gemfile +++ b/Gemfile @@ -2,10 +2,12 @@ source 'https://rubygems.org' -ruby '>= 3.1', '< 3.4' +ruby '>= 3.2', '< 3.4' group :test, :development do gem 'heroku_hatchet' + # Work around https://github.com/excon/excon/issues/860 + gem 'logger' gem 'parallel_split_test' gem 'rspec-core' gem 'rspec-expectations' diff --git a/Gemfile.lock b/Gemfile.lock index 126b0f21a..0d99cf2df 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -5,7 +5,7 @@ GEM base64 (0.2.0) diff-lcs (1.5.1) erubis (2.7.0) - excon (0.111.0) + excon (0.112.0) heroics (0.1.3) base64 erubis (~> 2.0) @@ -21,6 +21,7 @@ GEM threaded (~> 0) json (2.7.2) language_server-protocol (3.17.0.3) + logger (1.6.1) moneta (1.0.0) multi_json (1.15.0) parallel (1.26.3) @@ -39,7 +40,7 @@ GEM rate_throttle_client (0.1.2) regexp_parser (2.9.2) rrrretry (1.0.0) - rspec-core (3.13.1) + rspec-core (3.13.2) rspec-support (~> 3.13.0) rspec-expectations (3.13.3) diff-lcs (>= 1.2.0, < 2.0) @@ -47,7 +48,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.1) - rubocop (1.66.1) + rubocop (1.67.0) json (~> 2.3) language_server-protocol (>= 3.17.0) parallel (~> 1.10) @@ -65,13 +66,14 @@ GEM thor (1.3.2) threaded (0.0.4) unicode-display_width (2.6.0) - webrick (1.8.1) + webrick (1.8.2) PLATFORMS ruby DEPENDENCIES heroku_hatchet + logger parallel_split_test rspec-core rspec-expectations From bf2ce13ce72774efdc5bd02ca257b347394ce70d Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 24 Oct 2024 08:59:51 +0000 Subject: [PATCH 042/406] Improve error handling and log output of `{pre,post}_compile` hooks (#1667) * If the hooks fail, there is now an explicit error message (plus metrics event). * The build log step now references the full script location to aid. with understanding of what's happening. * Any output of the hooks is now indented to aid readability. * The hooks implementation has been refactored and moved to `lib/` as part of the overall buildpack reorganisation/refactoring. * Test coverage has been added for the failing hooks cases (previously only the successful hooks scenario was tested). (This is one step towards ensuring all buildpack failure cases have an explicit error message and a metrics `failure_reason`.) GUS-W-8059892. --- CHANGELOG.md | 1 + bin/compile | 5 +- bin/steps/hooks/post_compile | 12 -- bin/steps/hooks/pre_compile | 12 -- lib/hooks.sh | 41 ++++++ .../bin/post_compile | 6 + .../requirements.txt | 0 .../hooks_failing_pre_compile/bin/pre_compile | 6 + .../requirements.txt | 0 spec/hatchet/ci_spec.rb | 60 ++++---- spec/hatchet/hooks_spec.rb | 130 ++++++++++++------ spec/hatchet/pip_spec.rb | 40 +++--- spec/hatchet/pipenv_spec.rb | 40 +++--- 13 files changed, 217 insertions(+), 136 deletions(-) delete mode 100755 bin/steps/hooks/post_compile delete mode 100755 bin/steps/hooks/pre_compile create mode 100644 lib/hooks.sh create mode 100644 spec/fixtures/hooks_failing_post_compile/bin/post_compile create mode 100644 spec/fixtures/hooks_failing_post_compile/requirements.txt create mode 100644 spec/fixtures/hooks_failing_pre_compile/bin/pre_compile create mode 100644 spec/fixtures/hooks_failing_pre_compile/requirements.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index 4379617db..077f921c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Updated buildpack-generated warning messages to use colour and be more consistently formatted. ([#1666](https://github.com/heroku/heroku-buildpack-python/pull/1666)) +- Improved build log output and error messages for the `bin/pre_compile` and `bin/post_compile` customisation hooks. ([#1667](https://github.com/heroku/heroku-buildpack-python/pull/1667)) ## [v261] - 2024-10-14 diff --git a/bin/compile b/bin/compile index f318f0fa3..7e40754d6 100755 --- a/bin/compile +++ b/bin/compile @@ -19,6 +19,7 @@ ENV_DIR="${3}" BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) source "${BUILDPACK_DIR}/bin/utils" +source "${BUILDPACK_DIR}/lib/hooks.sh" source "${BUILDPACK_DIR}/lib/metadata.sh" source "${BUILDPACK_DIR}/lib/output.sh" source "${BUILDPACK_DIR}/lib/package_manager.sh" @@ -119,7 +120,7 @@ if [[ -d "$CACHE_DIR/.heroku/src" ]]; then fi # Runs a `bin/pre_compile` script if found in the app source, allowing build customisation. -source "${BUILDPACK_DIR}/bin/steps/hooks/pre_compile" +hooks::run_hook "pre_compile" # TODO: Clear the cache if this isn't a valid version, as part of the cache refactor. # (Currently the version is instead validated in `read_requested_python_version()`) @@ -297,7 +298,7 @@ cp "${BUILDPACK_DIR}/vendor/WEB_CONCURRENCY.sh" "$WEB_CONCURRENCY_PROFILE_PATH" cp "${BUILDPACK_DIR}/vendor/python.gunicorn.sh" "$GUNICORN_PROFILE_PATH" # Runs a `bin/post_compile` script if found in the app source, allowing build customisation. -source "${BUILDPACK_DIR}/bin/steps/hooks/post_compile" +hooks::run_hook "post_compile" # Store new artifacts in the cache. rm -rf "$CACHE_DIR/.heroku/python" diff --git a/bin/steps/hooks/post_compile b/bin/steps/hooks/post_compile deleted file mode 100755 index 7a20ed0a9..000000000 --- a/bin/steps/hooks/post_compile +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env bash - -if [[ -f bin/post_compile ]]; then - post_compile_hook_start_time=$(nowms) - meta_set "post_compile_hook" "true" - echo "-----> Running post-compile hook" - chmod +x bin/post_compile - sub_env bin/post_compile - meta_time "post_compile_hook_duration" "${post_compile_hook_start_time}" -else - meta_set "post_compile_hook" "false" -fi diff --git a/bin/steps/hooks/pre_compile b/bin/steps/hooks/pre_compile deleted file mode 100755 index c5be04ce3..000000000 --- a/bin/steps/hooks/pre_compile +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env bash - -if [[ -f bin/pre_compile ]]; then - pre_compile_hook_start_time=$(nowms) - meta_set "pre_compile_hook" "true" - echo "-----> Running pre-compile hook" - chmod +x bin/pre_compile - sub_env bin/pre_compile - meta_time "pre_compile_hook_duration" "${pre_compile_hook_start_time}" -else - meta_set "pre_compile_hook" "false" -fi diff --git a/lib/hooks.sh b/lib/hooks.sh new file mode 100644 index 000000000..19eb89839 --- /dev/null +++ b/lib/hooks.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash + +# This is technically redundant, since all consumers of this lib will have enabled these, +# however, it helps Shellcheck realise the options under which these functions will run. +set -euo pipefail + +# Used to run the `bin/pre_compile` and `bin/post_compile`s scripts if found in the app source, +# allowing for build customisation. +function hooks::run_hook() { + local hook_name="${1}" + local script_path="bin/${hook_name}" + + if [[ -f "${script_path}" ]]; then + local hook_start_time + hook_start_time=$(nowms) + output::step "Running ${script_path} hook" + meta_set "${hook_name}_hook" "true" + chmod +x "${script_path}" + + # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. + if ! sub_env "${script_path}" |& output::indent; then + output::error <<-EOF + Error: Failed to run the ${script_path} script. + + We found a '${script_path}' script in your app source, so ran + it to allow for customisation of the build process. + + However, this script exited with a non-zero exit status. + + Fix any errors output by your script above, or remove/rename + the script to prevent it from being run during the build. + EOF + meta_set "failure_reason" "hooks::${hook_name}" + exit 1 + fi + + meta_time "${hook_name}_hook_duration" "${hook_start_time}" + else + meta_set "${hook_name}_hook" "false" + fi +} diff --git a/spec/fixtures/hooks_failing_post_compile/bin/post_compile b/spec/fixtures/hooks_failing_post_compile/bin/post_compile new file mode 100644 index 000000000..9cdc2b9a1 --- /dev/null +++ b/spec/fixtures/hooks_failing_post_compile/bin/post_compile @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +set -euo pipefail + +echo 'Some post_compile error!' >&2 +exit 1 diff --git a/spec/fixtures/hooks_failing_post_compile/requirements.txt b/spec/fixtures/hooks_failing_post_compile/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/hooks_failing_pre_compile/bin/pre_compile b/spec/fixtures/hooks_failing_pre_compile/bin/pre_compile new file mode 100644 index 000000000..07e0236cf --- /dev/null +++ b/spec/fixtures/hooks_failing_pre_compile/bin/pre_compile @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +set -euo pipefail + +echo 'Some pre_compile error!' >&2 +exit 1 diff --git a/spec/fixtures/hooks_failing_pre_compile/requirements.txt b/spec/fixtures/hooks_failing_pre_compile/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 80a97706c..125c2d8d0 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -23,20 +23,20 @@ .* Successfully installed .* pytest-8.3.3 -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. - -----> Running post-compile hook - CI=true - CPLUS_INCLUDE_PATH=/app/.heroku/python/include - C_INCLUDE_PATH=/app/.heroku/python/include - DISABLE_COLLECTSTATIC=1 - INSTALL_TEST=1 - LANG=en_US.UTF-8 - LC_ALL=C.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/python/lib - LIBRARY_PATH=/app/.heroku/python/lib - PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ - PIP_NO_PYTHON_VERSION_WARNING=1 - PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config - PYTHONUNBUFFERED=1 + -----> Running bin/post_compile hook + CI=true + CPLUS_INCLUDE_PATH=/app/.heroku/python/include + C_INCLUDE_PATH=/app/.heroku/python/include + DISABLE_COLLECTSTATIC=1 + INSTALL_TEST=1 + LANG=en_US.UTF-8 + LC_ALL=C.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib + PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PIP_NO_PYTHON_VERSION_WARNING=1 + PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config + PYTHONUNBUFFERED=1 -----> Inline app detected LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib @@ -77,7 +77,7 @@ -----> Installing requirements with pip -----> Installing test dependencies... -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. - -----> Running post-compile hook + -----> Running bin/post_compile hook OUTPUT end end @@ -99,20 +99,20 @@ Installing dependencies from Pipfile.lock \\(.+\\)... Installing dependencies from Pipfile.lock \\(.+\\)... -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. - -----> Running post-compile hook - CI=true - CPLUS_INCLUDE_PATH=/app/.heroku/python/include - C_INCLUDE_PATH=/app/.heroku/python/include - DISABLE_COLLECTSTATIC=1 - INSTALL_TEST=1 - LANG=en_US.UTF-8 - LC_ALL=C.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/python/lib - LIBRARY_PATH=/app/.heroku/python/lib - PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ - PIP_NO_PYTHON_VERSION_WARNING=1 - PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config - PYTHONUNBUFFERED=1 + -----> Running bin/post_compile hook + CI=true + CPLUS_INCLUDE_PATH=/app/.heroku/python/include + C_INCLUDE_PATH=/app/.heroku/python/include + DISABLE_COLLECTSTATIC=1 + INSTALL_TEST=1 + LANG=en_US.UTF-8 + LC_ALL=C.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib + PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PIP_NO_PYTHON_VERSION_WARNING=1 + PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config + PYTHONUNBUFFERED=1 -----> Inline app detected LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib @@ -154,7 +154,7 @@ Installing dependencies from Pipfile.lock \\(.+\\)... Installing dependencies from Pipfile.lock \\(.+\\)... -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. - -----> Running post-compile hook + -----> Running bin/post_compile hook REGEX end end diff --git a/spec/hatchet/hooks_spec.rb b/spec/hatchet/hooks_spec.rb index 05f0f927c..599c788bf 100644 --- a/spec/hatchet/hooks_spec.rb +++ b/spec/hatchet/hooks_spec.rb @@ -14,50 +14,100 @@ expect(output).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Running pre-compile hook - remote: ~ pre_compile ran with env vars: - remote: BUILD_DIR=/tmp/build_ - remote: CACHE_DIR=/tmp/codon/tmp/cache - remote: C_INCLUDE_PATH=/app/.heroku/python/include - remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include - remote: ENV_DIR=/tmp/... - remote: HOME=/app - remote: LANG=en_US.UTF-8 - remote: LD_LIBRARY_PATH=/app/.heroku/python/lib - remote: LIBRARY_PATH=/app/.heroku/python/lib - remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - remote: PIP_NO_PYTHON_VERSION_WARNING=1 - remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config - remote: PWD=/tmp/build_ - remote: PYTHONUNBUFFERED=1 - remote: SOME_APP_CONFIG_VAR=1 - remote: SOURCE_VERSION=... - remote: STACK=#{app.stack} - remote: ~ pre_compile complete + remote: -----> Running bin/pre_compile hook + remote: ~ pre_compile ran with env vars: + remote: BUILD_DIR=/tmp/build_ + remote: CACHE_DIR=/tmp/codon/tmp/cache + remote: C_INCLUDE_PATH=/app/.heroku/python/include + remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include + remote: ENV_DIR=/tmp/... + remote: HOME=/app + remote: LANG=en_US.UTF-8 + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib + remote: LIBRARY_PATH=/app/.heroku/python/lib + remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PIP_NO_PYTHON_VERSION_WARNING=1 + remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config + remote: PWD=/tmp/build_ + remote: PYTHONUNBUFFERED=1 + remote: SOME_APP_CONFIG_VAR=1 + remote: SOURCE_VERSION=... + remote: STACK=#{app.stack} + remote: ~ pre_compile complete OUTPUT expect(output).to include(<<~OUTPUT) remote: -----> Installing requirements with pip - remote: -----> Running post-compile hook - remote: ~ post_compile ran with env vars: - remote: BUILD_DIR=/tmp/build_ - remote: CACHE_DIR=/tmp/codon/tmp/cache - remote: C_INCLUDE_PATH=/app/.heroku/python/include - remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include - remote: ENV_DIR=/tmp/... - remote: HOME=/app - remote: LANG=en_US.UTF-8 - remote: LD_LIBRARY_PATH=/app/.heroku/python/lib - remote: LIBRARY_PATH=/app/.heroku/python/lib - remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - remote: PIP_NO_PYTHON_VERSION_WARNING=1 - remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config - remote: PWD=/tmp/build_ - remote: PYTHONUNBUFFERED=1 - remote: SOME_APP_CONFIG_VAR=1 - remote: SOURCE_VERSION=... - remote: STACK=#{app.stack} - remote: ~ post_compile complete + remote: -----> Running bin/post_compile hook + remote: ~ post_compile ran with env vars: + remote: BUILD_DIR=/tmp/build_ + remote: CACHE_DIR=/tmp/codon/tmp/cache + remote: C_INCLUDE_PATH=/app/.heroku/python/include + remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include + remote: ENV_DIR=/tmp/... + remote: HOME=/app + remote: LANG=en_US.UTF-8 + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib + remote: LIBRARY_PATH=/app/.heroku/python/lib + remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PIP_NO_PYTHON_VERSION_WARNING=1 + remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config + remote: PWD=/tmp/build_ + remote: PYTHONUNBUFFERED=1 + remote: SOME_APP_CONFIG_VAR=1 + remote: SOURCE_VERSION=... + remote: STACK=#{app.stack} + remote: ~ post_compile complete + OUTPUT + end + end + end + + context 'when an app has a failing bin/pre_compile script' do + let(:app) { Hatchet::Runner.new('spec/fixtures/hooks_failing_pre_compile', allow_failure: true) } + + it 'aborts the build with a suitable error message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Running bin/pre_compile hook + remote: Some pre_compile error! + remote: + remote: ! Error: Failed to run the bin/pre_compile script. + remote: ! + remote: ! We found a 'bin/pre_compile' script in your app source, so ran + remote: ! it to allow for customisation of the build process. + remote: ! + remote: ! However, this script exited with a non-zero exit status. + remote: ! + remote: ! Fix any errors output by your script above, or remove/rename + remote: ! the script to prevent it from being run during the build. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + context 'when an app has a failing bin/post_compile script' do + let(:app) { Hatchet::Runner.new('spec/fixtures/hooks_failing_post_compile', allow_failure: true) } + + it 'aborts the build with a suitable error message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Running bin/post_compile hook + remote: Some post_compile error! + remote: + remote: ! Error: Failed to run the bin/post_compile script. + remote: ! + remote: ! We found a 'bin/post_compile' script in your app source, so ran + remote: ! it to allow for customisation of the build process. + remote: ! + remote: ! However, this script exited with a non-zero exit status. + remote: ! + remote: ! Fix any errors output by your script above, or remove/rename + remote: ! the script to prevent it from being run during the build. + remote: + remote: ! Push rejected, failed to compile Python app. OUTPUT end end diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 0a4cbec57..2da252cb3 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -140,16 +140,16 @@ it 'rewrites .pth, .egg-link and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) - remote: -----> Running post-compile hook - remote: easy-install.pth:/app/.heroku/src/gunicorn - remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py - remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py - remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: -----> Running bin/post_compile hook + remote: easy-install.pth:/app/.heroku/src/gunicorn + remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: gunicorn.egg-link:/app/.heroku/src/gunicorn + remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py + remote: + remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) remote: -----> Inline app detected remote: easy-install.pth:/app/.heroku/src/gunicorn remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py @@ -179,16 +179,16 @@ app.commit! app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) - remote: -----> Running post-compile hook - remote: easy-install.pth:/app/.heroku/src/gunicorn - remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py - remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py - remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: -----> Running bin/post_compile hook + remote: easy-install.pth:/app/.heroku/src/gunicorn + remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: gunicorn.egg-link:/app/.heroku/src/gunicorn + remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py + remote: + remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) remote: -----> Inline app detected remote: easy-install.pth:/app/.heroku/src/gunicorn remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 02686accf..b4ffa7f4b 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -362,16 +362,16 @@ it 'rewrites .pth, .egg-link and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) - remote: -----> Running post-compile hook - remote: easy-install.pth:/app/.heroku/src/gunicorn - remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py - remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py - remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: -----> Running bin/post_compile hook + remote: easy-install.pth:/app/.heroku/src/gunicorn + remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: gunicorn.egg-link:/app/.heroku/src/gunicorn + remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py + remote: + remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) remote: -----> Inline app detected remote: easy-install.pth:/app/.heroku/src/gunicorn remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py @@ -401,16 +401,16 @@ app.commit! app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) - remote: -----> Running post-compile hook - remote: easy-install.pth:/app/.heroku/src/gunicorn - remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py - remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py - remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: -----> Running bin/post_compile hook + remote: easy-install.pth:/app/.heroku/src/gunicorn + remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: gunicorn.egg-link:/app/.heroku/src/gunicorn + remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py + remote: + remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) remote: -----> Inline app detected remote: easy-install.pth:/app/.heroku/src/gunicorn remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py From d92180d4e8c347b680d512129c927a775ed7333c Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 24 Oct 2024 09:03:05 +0000 Subject: [PATCH 043/406] Unskip env var tests on Heroku-22 (#1668) Since Heroku-22 has now been migrated to the new build infrastructure, so no longer has the small system env var differences that were causing these tests to fail on that stack (which is why they had been skipped as part of #1634). --- spec/hatchet/hooks_spec.rb | 4 +--- spec/hatchet/pip_spec.rb | 4 +--- spec/hatchet/pipenv_spec.rb | 5 +---- 3 files changed, 3 insertions(+), 10 deletions(-) diff --git a/spec/hatchet/hooks_spec.rb b/spec/hatchet/hooks_spec.rb index 599c788bf..8ea91859a 100644 --- a/spec/hatchet/hooks_spec.rb +++ b/spec/hatchet/hooks_spec.rb @@ -3,9 +3,7 @@ require_relative '../spec_helper' RSpec.describe 'Compile hooks' do - # TODO: Run this on Heroku-22 too, once it has also migrated to the new build infrastructure. - # (Currently the test fails on the old infrastructure due to subtle differences in system PATH elements.) - context 'when an app has bin/pre_compile and bin/post_compile scripts', stacks: %w[heroku-20 heroku-24] do + context 'when an app has bin/pre_compile and bin/post_compile scripts' do let(:app) { Hatchet::Runner.new('spec/fixtures/hooks', config: { 'SOME_APP_CONFIG_VAR' => '1' }) } it 'runs the hooks with the correct environment' do diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 2da252cb3..199f234cb 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -12,9 +12,7 @@ end RSpec.describe 'pip support' do - # TODO: Run this on Heroku-22 too, once it has also migrated to the new build infrastructure. - # (Currently the test fails on the old infrastructure due to subtle differences in system PATH elements.) - context 'when requirements.txt is unchanged since the last build', stacks: %w[heroku-20 heroku-24] do + context 'when requirements.txt is unchanged since the last build' do let(:buildpacks) { [:default, 'heroku-community/inline'] } let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_basic', buildpacks:) } diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index b4ffa7f4b..4be306e46 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -7,10 +7,7 @@ let(:buildpacks) { [:default, 'heroku-community/inline'] } let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_basic', buildpacks:) } - # TODO: Run this on Heroku-22 too, once it has also migrated to the new build infrastructure. - # (Currently the test fails on the old infrastructure due to subtle differences in system PATH elements.) - it 'builds with the specified python_version and re-uses packages from the cache', - stacks: %w[heroku-20 heroku-24] do + it 'builds with the specified python_version and re-uses packages from the cache' do app.deploy do |app| # TODO: We should not be leaking the Pipenv installation into the app environment. expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) From 99bd9f9a6f4f51824419c475fbb72f3c6904320f Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 25 Oct 2024 11:38:46 +0000 Subject: [PATCH 044/406] Fix leftover output::error args usages (#1671) In #1666 the API for `output::error` was simplified, such that it only supports the message being passed via stdin (since that's the pattern used when passing a multi-line HERE-doc, and we want to encourage detailed error messages). However, there were a couple of remaining instances not switched over to the stdin API. These usages are in codepaths that cannot be tested via integration tests since they are unreachable unless there was an internal buildpack bug. (And Shellcheck also didn't spot these instances of unconsumed passed args.) GUS-W-17048951. --- lib/utils.sh | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/utils.sh b/lib/utils.sh index c64bbfc79..8a0dd4a16 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -18,7 +18,9 @@ function utils::bundled_pip_module_path() { local bundled_pip_wheel="${bundled_pip_wheel_list[0]}" if [[ -z "${bundled_pip_wheel}" ]]; then - output::error "Error: Failed to locate the bundled pip wheel." + output::error <<-'EOF' + Error: Failed to locate the bundled pip wheel. + EOF meta_set "failure_reason" "bundled-pip-not-found" return 1 fi @@ -28,7 +30,9 @@ function utils::bundled_pip_module_path() { function utils::abort_internal_error() { local message="${1}" - output::error "Internal error: ${message} (line $(caller || true))." + output::error <<-EOF + Internal error: ${message} (line $(caller || true)). + EOF meta_set "failure_reason" "internal-error" exit 1 } From dba9b862dea3ad0277e7d3876489c2a90e383b05 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Fri, 25 Oct 2024 12:05:58 +0000 Subject: [PATCH 045/406] Prepare release v262 (#1672) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 077f921c2..2c5a1fb8e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v262] - 2024-10-25 + - Updated buildpack-generated warning messages to use colour and be more consistently formatted. ([#1666](https://github.com/heroku/heroku-buildpack-python/pull/1666)) - Improved build log output and error messages for the `bin/pre_compile` and `bin/post_compile` customisation hooks. ([#1667](https://github.com/heroku/heroku-buildpack-python/pull/1667)) @@ -1070,7 +1073,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v261...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v262...main +[v262]: https://github.com/heroku/heroku-buildpack-python/compare/v261...v262 [v261]: https://github.com/heroku/heroku-buildpack-python/compare/v260...v261 [v260]: https://github.com/heroku/heroku-buildpack-python/compare/v259...v260 [v259]: https://github.com/heroku/heroku-buildpack-python/compare/v258...v259 From f00f258dcf7a739e0475c0917736b2ce3578a725 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 31 Oct 2024 14:56:40 +0000 Subject: [PATCH 046/406] Improve cache handling (#1679) Refactors cache handling to dedicated functions under `lib/cache.sh` (rather than being scattered around the buildpack), and makes the following improvements: - Ensures the cache is now also discards the cache when the package manager (or its version) changes. - Improves the build log output shown when restoring or discarding the cache. For example, if the cache was invalidated all reasons are now shown. - Stops performing unnecessary cache file copies when the cache is due to be invalidated. This required moving the cache restoration step to after the `bin/pre_compile` hook runs. - Fixes cache restoration in the case where an app's `requirements.txt` was formerly a symlink. - Adds buildpack metrics for the status of the cache and duration of cache restoration/saving. Fixes #1673. Fixes #1674. Fixes #1675. Fixes #1676. Fixes #1677. Fixes #1678. Prep for #796. Unblocks upgrading pip (since #1674 prevents https://github.com/pypa/pip/issues/12950). GUS-W-16811131. --- CHANGELOG.md | 5 + bin/compile | 63 +----- bin/report | 3 + bin/steps/python | 89 ++------ bin/steps/sqlite3 | 3 - lib/cache.sh | 192 ++++++++++++++++++ lib/kvstore.sh | 7 +- lib/pip.sh | 12 +- lib/pipenv.sh | 6 +- lib/python_version.sh | 3 +- .../requirements_basic/requirements.txt | 3 +- .../requirements_basic/requirements/prod.txt | 5 + spec/hatchet/ci_spec.rb | 3 +- spec/hatchet/pip_spec.rb | 30 ++- spec/hatchet/pipenv_spec.rb | 35 +++- spec/hatchet/python_update_warning_spec.rb | 6 +- spec/hatchet/python_version_spec.rb | 13 +- spec/hatchet/stack_spec.rb | 12 +- 18 files changed, 322 insertions(+), 168 deletions(-) create mode 100644 lib/cache.sh mode change 100644 => 120000 spec/fixtures/requirements_basic/requirements.txt create mode 100644 spec/fixtures/requirements_basic/requirements/prod.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c5a1fb8e..575845c60 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ ## [Unreleased] +- Fixed cache handling so that it now also discards the cache when the package manager (or its version) changes. ([#1679](https://github.com/heroku/heroku-buildpack-python/pull/1679)) +- Improved the build log output shown when restoring or discarding the cache. For example, if the cache was invalidated all reasons are now shown. ([#1679](https://github.com/heroku/heroku-buildpack-python/pull/1679)) +- Stopped performing unnecessary cache file copies when the cache is due to be invalidated. This required moving the cache restoration step to after the `bin/pre_compile` hook runs. ([#1679](https://github.com/heroku/heroku-buildpack-python/pull/1679)) +- Fixed cache restoration in the case where an app's `requirements.txt` was formerly a symlink. ([#1679](https://github.com/heroku/heroku-buildpack-python/pull/1679)) +- Added buildpack metrics for the status of the cache and duration of cache restoration/saving. ([#1679](https://github.com/heroku/heroku-buildpack-python/pull/1679)) ## [v262] - 2024-10-25 diff --git a/bin/compile b/bin/compile index 7e40754d6..4c2d271c3 100755 --- a/bin/compile +++ b/bin/compile @@ -19,6 +19,7 @@ ENV_DIR="${3}" BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) source "${BUILDPACK_DIR}/bin/utils" +source "${BUILDPACK_DIR}/lib/cache.sh" source "${BUILDPACK_DIR}/lib/hooks.sh" source "${BUILDPACK_DIR}/lib/metadata.sh" source "${BUILDPACK_DIR}/lib/output.sh" @@ -93,56 +94,14 @@ export PIP_NO_PYTHON_VERSION_WARNING=1 cd "$BUILD_DIR" -# The Cache -# --------- - -# The workflow for the Python Buildpack's cache is as follows: -# -# - `~/.heroku/{known-paths}` are copied from the cache into the slug. -# - The build is executed, modifying `~/.heroku/{known-paths}`. -# - Once the build is complete, `~/.heroku/{known-paths}` is copied back into the cache. - -mkdir -p "$CACHE_DIR/.heroku" - -# Restore old artifacts from the cache. -mkdir -p .heroku -# The Python installation. -cp -R "$CACHE_DIR/.heroku/python" .heroku/ &>/dev/null || true -# A plain text file which contains the current stack being used (used for cache busting). -cp -R "$CACHE_DIR/.heroku/python-stack" .heroku/ &>/dev/null || true -# A plain text file which contains the current python version being used (used for cache busting). -cp -R "$CACHE_DIR/.heroku/python-version" .heroku/ &>/dev/null || true -# A plain text file which contains the current sqlite3 version being used (used for cache busting). -cp -R "$CACHE_DIR/.heroku/python-sqlite3-version" .heroku/ &>/dev/null || true -# "editable" installations of code repositories, via pip or pipenv. -if [[ -d "$CACHE_DIR/.heroku/src" ]]; then - cp -R "$CACHE_DIR/.heroku/src" .heroku/ &>/dev/null || true -fi - # Runs a `bin/pre_compile` script if found in the app source, allowing build customisation. hooks::run_hook "pre_compile" -# TODO: Clear the cache if this isn't a valid version, as part of the cache refactor. -# (Currently the version is instead validated in `read_requested_python_version()`) -if [[ -f "$CACHE_DIR/.heroku/python-version" ]]; then - cached_python_version="$(cat "${CACHE_DIR}/.heroku/python-version")" - # `python-X.Y.Z` -> `X.Y` - cached_python_version="${cached_python_version#python-}" -else - cached_python_version= -fi - -# We didn't always record the stack version. -if [[ -f "$CACHE_DIR/.heroku/python-stack" ]]; then - CACHED_PYTHON_STACK=$(cat "$CACHE_DIR/.heroku/python-stack") -else - # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. - CACHED_PYTHON_STACK=$STACK -fi - package_manager="$(package_manager::determine_package_manager "${BUILD_DIR}")" meta_set "package_manager" "${package_manager}" +cached_python_version="$(cache::cached_python_version "${CACHE_DIR}")" + # We use the Bash 4.3+ `nameref` feature to pass back multiple values from this function # without having to hardcode globals. See: https://stackoverflow.com/a/38997681 python_version::read_requested_python_version "${BUILD_DIR}" "${package_manager}" "${cached_python_version}" requested_python_version python_version_origin @@ -170,6 +129,8 @@ python_major_version="${python_full_version%.*}" meta_set "python_version" "${python_full_version}" meta_set "python_version_major" "${python_major_version}" +cache::restore "${BUILD_DIR}" "${CACHE_DIR}" "${STACK:?}" "${cached_python_version}" "${python_full_version}" "${package_manager}" + # The directory for the .profile.d scripts. mkdir -p "$(dirname "$PROFILE_PATH")" # The directory for editable VCS dependencies. @@ -300,18 +261,6 @@ cp "${BUILDPACK_DIR}/vendor/python.gunicorn.sh" "$GUNICORN_PROFILE_PATH" # Runs a `bin/post_compile` script if found in the app source, allowing build customisation. hooks::run_hook "post_compile" -# Store new artifacts in the cache. -rm -rf "$CACHE_DIR/.heroku/python" -rm -rf "$CACHE_DIR/.heroku/python-version" -rm -rf "$CACHE_DIR/.heroku/python-stack" -rm -rf "$CACHE_DIR/.heroku/src" - -mkdir -p "$CACHE_DIR/.heroku" -cp -R .heroku/python "$CACHE_DIR/.heroku/" -cp -R .heroku/python-version "$CACHE_DIR/.heroku/" -cp -R .heroku/python-stack "$CACHE_DIR/.heroku/" &>/dev/null || true -if [[ -d .heroku/src ]]; then - cp -R .heroku/src "$CACHE_DIR/.heroku/" &>/dev/null || true -fi +cache::save "${BUILD_DIR}" "${CACHE_DIR}" "${STACK}" "${python_full_version}" "${package_manager}" meta_time "total_duration" "${compile_start_time}" diff --git a/bin/report b/bin/report index 71bd20384..ecc08f0fa 100755 --- a/bin/report +++ b/bin/report @@ -60,6 +60,7 @@ kv_pair_string() { } STRING_FIELDS=( + cache_status django_collectstatic failure_reason nltk_downloader @@ -76,6 +77,8 @@ STRING_FIELDS=( # We don't want to quote numeric or boolean fields. ALL_OTHER_FIELDS=( + cache_restore_duration + cache_save_duration dependencies_install_duration django_collectstatic_duration nltk_downloader_duration diff --git a/bin/steps/python b/bin/steps/python index 37fbf6a34..316bb19ab 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -1,6 +1,5 @@ #!/usr/bin/env bash # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. -# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. set -euo pipefail @@ -27,6 +26,28 @@ if ! curl --output /dev/null --silent --head --fail --retry 3 --retry-connrefuse exit 1 fi +if [[ -f "${BUILD_DIR}/.heroku/python/bin/python" ]]; then + output::step "Using cached install of Python ${python_full_version}" +else + output::step "Installing Python ${python_full_version}" + mkdir -p "${BUILD_DIR}/.heroku/python" + + if ! curl --silent --show-error --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}" | tar --zstd --extract --directory "${BUILD_DIR}/.heroku/python"; then + # The Python version was confirmed to exist previously, so any failure here is due to + # a networking issue or archive/buildpack bug rather than the runtime not existing. + output::error <<-EOF + Error: Failed to download/install Python ${python_full_version}. + + In some cases, this happens due to an unstable network connection. + Please try again and to see if the error resolves itself. + EOF + meta_set "failure_reason" "python-download" + exit 1 + fi + + hash -r +fi + function warn_if_patch_update_available() { local requested_full_version="${1}" local requested_major_version="${2}" @@ -69,69 +90,3 @@ if [[ "${python_major_version}" == "3.8" ]]; then fi warn_if_patch_update_available "${python_full_version}" "${python_major_version}" - -if [[ "$STACK" != "$CACHED_PYTHON_STACK" ]]; then - output::step "Stack has changed from $CACHED_PYTHON_STACK to $STACK, clearing cache" - rm -rf .heroku/python-stack .heroku/python-version .heroku/python .heroku/vendor .heroku/python .heroku/python-sqlite3-version -fi - -# TODO: Clean this up as part of the cache refactor. -if [[ -f .heroku/python-version ]]; then - if [[ "${cached_python_version}" != "${python_full_version}" ]]; then - output::step "Python version has changed from ${cached_python_version} to ${python_full_version}, clearing cache" - rm -rf .heroku/python - else - SKIP_INSTALL=1 - fi -fi - -# If using pip, check if we should reinstall python dependencies given that requirements.txt -# is non-deterministic (not all packages pinned, doesn't handle uninstalls etc). We don't need -# to do this when using Pipenv, since it has a lockfile and syncs the packages for us. -if [[ -f "${BUILD_DIR}/requirements.txt" ]]; then - if [[ ! -f "$CACHE_DIR/.heroku/requirements.txt" ]]; then - # This is a the first build of an app (or the build cache was cleared). Since there - # are no cached packages, we only need to store the requirements file for next time. - cp -R "$BUILD_DIR/requirements.txt" "$CACHE_DIR/.heroku/requirements.txt" - else - # IF there IS a cached directory, check for differences with the new one - if ! diff "$BUILD_DIR/requirements.txt" "$CACHE_DIR/.heroku/requirements.txt" &>/dev/null; then - output::step "Requirements file has been changed, clearing cached dependencies" - # if there are any differences, clear the Python cache - # Installing Python over again does not take noticably more time - cp -R "$BUILD_DIR/requirements.txt" "$CACHE_DIR/.heroku/requirements.txt" - rm -rf .heroku/python - unset SKIP_INSTALL - else - output::step "No change in requirements detected, installing from cache" - fi - fi -fi - -if [[ "${SKIP_INSTALL:-0}" == "1" ]]; then - output::step "Using cached install of Python ${python_full_version}" -else - output::step "Installing Python ${python_full_version}" - - # Prepare destination directory. - mkdir -p .heroku/python - - if ! curl --silent --show-error --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}" | tar --zstd --extract --directory .heroku/python; then - # The Python version was confirmed to exist previously, so any failure here is due to - # a networking issue or archive/buildpack bug rather than the runtime not existing. - output::error <<-EOF - Error: Failed to download/install Python ${python_full_version}. - - In some cases, this happens due to an unstable network connection. - Please try again and to see if the error resolves itself. - EOF - meta_set "failure_reason" "python-download" - exit 1 - fi - - # Record for future reference. - echo "python-${python_full_version}" >.heroku/python-version - echo "$STACK" >.heroku/python-stack - - hash -r -fi diff --git a/bin/steps/sqlite3 b/bin/steps/sqlite3 index 3ef119a35..1a94d5965 100755 --- a/bin/steps/sqlite3 +++ b/bin/steps/sqlite3 @@ -78,7 +78,4 @@ buildpack_sqlite3_install() { echo "Sqlite3 failed to install." # mcount "failure.python.sqlite3" fi - - # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. - mkdir -p "$CACHE_DIR/.heroku/" } diff --git a/lib/cache.sh b/lib/cache.sh new file mode 100644 index 000000000..3e814f56e --- /dev/null +++ b/lib/cache.sh @@ -0,0 +1,192 @@ +#!/usr/bin/env bash + +# This is technically redundant, since all consumers of this lib will have enabled these, +# however, it helps Shellcheck realise the options under which these functions will run. +set -euo pipefail + +# Read the full Python version of the Python install in the cache, or the empty string +# if the cache is empty or doesn't contain a Python version metadata file. +function cache::cached_python_version() { + local cache_dir="${1}" + + if [[ -f "${cache_dir}/.heroku/python-version" ]]; then + local version + version="$(cat "${cache_dir}/.heroku/python-version")" + # For historical reasons the version has always been stored as `python-X.Y.Z`, + # so we have to remove the `python-` prefix. + echo "${version#python-}" + fi +} + +# Validates and restores the contents of the build cache if appropriate. +# The cache is discarded if any of the following have changed: +# - Stack +# - Python version +# - Package manager +# - Package manager version +# - requirements.txt contents (pip only) +function cache::restore() { + local build_dir="${1}" + local cache_dir="${2}" + local stack="${3}" + local cached_python_version="${4}" + local python_full_version="${5}" + local package_manager="${6}" + + if [[ ! -e "${cache_dir}/.heroku/python" ]]; then + meta_set "cache_status" "empty" + return 0 + fi + + local cache_restore_start_time + cache_restore_start_time=$(nowms) + local cache_invalidation_reasons=() + + local cached_stack + cached_stack="$(cat "${cache_dir}/.heroku/python-stack" || true)" + if [[ "${cached_stack}" != "${stack}" ]]; then + cache_invalidation_reasons+=("The stack has changed from ${cached_stack:-"unknown"} to ${stack}") + fi + + if [[ "${cached_python_version}" != "${python_full_version}" ]]; then + cache_invalidation_reasons+=("The Python version has changed from ${cached_python_version:-"unknown"} to ${python_full_version}") + fi + + # The metadata store only exists in caches created in v252+ of the buildpack (released 2024-06-17), + # so here and below we have to handle the case where `meta_prev_get` returns the empty string. + local cached_package_manager + cached_package_manager="$(meta_prev_get "package_manager")" + if [[ -z "${cached_package_manager}" ]]; then + # Using `compgen` since `[[ -d ... ]]` doesn't work with globs. + if compgen -G "${cache_dir}/.heroku/python/lib/python*/site-packages/pipenv" >/dev/null; then + cached_package_manager="pipenv" + elif compgen -G "${cache_dir}/.heroku/python/lib/python*/site-packages/pip" >/dev/null; then + cached_package_manager="pip" + fi + fi + + if [[ "${cached_package_manager}" != "${package_manager}" ]]; then + cache_invalidation_reasons+=("The package manager has changed from ${cached_package_manager:-"unknown"} to ${package_manager}") + else + case "${package_manager}" in + pip) + local cached_pip_version + cached_pip_version="$(meta_prev_get "pip_version")" + # Handle caches written by buildpack versions older than v252 (see above). + if [[ -z "${cached_pip_version}" ]]; then + # Whilst we don't know the old version, we know the pip version has likely + # changed since the last build, and would rather err on the side of caution. + # (The pip version was last updated in v246, but will be updated again soon.) + cache_invalidation_reasons+=("The pip version has changed") + elif [[ "${cached_pip_version}" != "${PIP_VERSION:?}" ]]; then + cache_invalidation_reasons+=("The pip version has changed from ${cached_pip_version} to ${PIP_VERSION}") + fi + + # We invalidate the cache if requirements.txt changes since pip is a package installer + # rather than a project/environment manager, and so does not deterministically manage + # installed Python packages. For example, if a package entry in a requirements file is + # later removed, pip will not uninstall the package. This check can be removed if we + # ever switch to only caching pip's HTTP/wheel cache rather than site-packages. + # TODO: Remove the `-f` check once the setup.py fallback feature is removed. + if [[ -f "${build_dir}/requirements.txt" ]] && ! cmp --silent "${cache_dir}/.heroku/requirements.txt" "${build_dir}/requirements.txt"; then + cache_invalidation_reasons+=("The contents of requirements.txt changed") + fi + ;; + pipenv) + local cached_pipenv_version + cached_pipenv_version="$(meta_prev_get "pipenv_version")" + # Handle caches written by buildpack versions older than v252 (see above). + if [[ -z "${cached_pipenv_version}" ]]; then + # Whilst we don't know the old version, we know the Pipenv version has definitely + # changed since buildpack v251. + cache_invalidation_reasons+=("The Pipenv version has changed") + elif [[ "${cached_pipenv_version}" != "${PIPENV_VERSION:?}" ]]; then + cache_invalidation_reasons+=("The Pipenv version has changed from ${cached_pipenv_version} to ${PIPENV_VERSION}") + fi + ;; + *) + utils::abort_internal_error "Unhandled package manager: ${package_manager}" + ;; + esac + fi + + if [[ -n "${cache_invalidation_reasons[*]}" ]]; then + output::step "Discarding cache since:" + local reason + for reason in "${cache_invalidation_reasons[@]}"; do + echo " - ${reason}" + done + + rm -rf \ + "${cache_dir}/.heroku/python" \ + "${cache_dir}/.heroku/python-stack" \ + "${cache_dir}/.heroku/python-version" \ + "${cache_dir}/.heroku/src" \ + "${cache_dir}/.heroku/requirements.txt" + + meta_set "cache_status" "discarded" + else + output::step "Restoring cache" + mkdir -p "${build_dir}/.heroku" + + # NB: For now this has to handle files already existing in build_dir since some apps accidentally + # run the Python buildpack twice. TODO: Add an explicit check/error for duplicate buildpacks. + # TODO: Investigate why errors are ignored and ideally stop doing so. + # TODO: Compare the performance of moving the directory vs copying files. + cp -R "${cache_dir}/.heroku/python" "${build_dir}/.heroku/" &>/dev/null || true + + # Editable VCS code repositories, used by pip/pipenv. + if [[ -d "${cache_dir}/.heroku/src" ]]; then + cp -R "${cache_dir}/.heroku/src" "${build_dir}/.heroku/" &>/dev/null || true + fi + + meta_set "cache_status" "reused" + fi + + # Remove any legacy cache contents written by older buildpack versions. + rm -rf \ + "${cache_dir}/.heroku/python-sqlite3-version" \ + "${cache_dir}/.heroku/vendor" + + meta_time "cache_restore_duration" "${cache_restore_start_time}" +} + +# Copies Python and dependencies from the build directory to the cache, for use by subsequent builds. +function cache::save() { + local build_dir="${1}" + local cache_dir="${2}" + local stack="${3}" + local python_full_version="${4}" + local package_manager="${5}" + + local cache_save_start_time + cache_save_start_time=$(nowms) + + mkdir -p "${cache_dir}/.heroku" + + rm -rf "${cache_dir}/.heroku/python" + cp -R "${build_dir}/.heroku/python" "${cache_dir}/.heroku/" + + # Editable VCS code repositories, used by pip/pipenv. + rm -rf "${cache_dir}/.heroku/src" + if [[ -d "${build_dir}/.heroku/src" ]]; then + # TODO: Investigate why errors are ignored and ideally stop doing so. + cp -R "${build_dir}/.heroku/src" "${cache_dir}/.heroku/" &>/dev/null || true + fi + + # Metadata used by subsequent builds to determine whether the cache can be reused. + # These are written/consumed via separate files and not the metadata store for compatibility + # with buildpack versions prior to the metadata store existing (which was only added in v252). + echo "${stack}" >"${cache_dir}/.heroku/python-stack" + # For historical reasons the Python version was always stored with a `python-` prefix. + # We continue to use that format so that the file can be read by older buildpack versions. + echo "python-${python_full_version}" >"${cache_dir}/.heroku/python-version" + + # TODO: Simplify this once multiple package manager files being found is turned into an + # error and the setup.py fallback feature is removed. + if [[ "${package_manager}" == "pip" && -f "${build_dir}/requirements.txt" ]]; then + cp "${build_dir}/requirements.txt" "${cache_dir}/.heroku/" + fi + + meta_time "cache_save_duration" "${cache_save_start_time}" +} diff --git a/lib/kvstore.sh b/lib/kvstore.sh index ba3809383..c4a684fea 100644 --- a/lib/kvstore.sh +++ b/lib/kvstore.sh @@ -20,6 +20,7 @@ kv_clear() { } kv_set() { + # TODO: Stop ignoring an incorrect number of passed arguments. if [[ $# -eq 3 ]]; then local f="${1}" if [[ -f "${f}" ]]; then @@ -28,11 +29,15 @@ kv_set() { fi } +# Returns a key from the key-value store file, or else the empty string +# if the file doesn't exist or the key wasn't found. kv_get() { + # TODO: Stop ignoring an incorrect number of passed arguments. if [[ $# -eq 2 ]]; then local f="${1}" if [[ -f "${f}" ]]; then - grep "^${2}=" "${f}" | sed -e "s/^${2}=//" | tail -n 1 + # shellcheck disable=SC2310 # This function is invoked in an || condition so set -e will be disabled. + grep "^${2}=" "${f}" | sed -e "s/^${2}=//" | tail -n 1 || true fi fi } diff --git a/lib/pip.sh b/lib/pip.sh index eb4535a2d..e58cea5af 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -4,14 +4,16 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail +PIP_VERSION=$(get_requirement_version 'pip') +SETUPTOOLS_VERSION=$(get_requirement_version 'setuptools') +WHEEL_VERSION=$(get_requirement_version 'wheel') + function pip::install_pip_setuptools_wheel() { # We use the pip wheel bundled within Python's standard library to install our chosen # pip version, since it's faster than `ensurepip` followed by an upgrade in place. local bundled_pip_module_path="${1}" local python_major_version="${2}" - # TODO: Either make these `local` or move elsewhere as part of the cache invalidation refactoring. - PIP_VERSION=$(get_requirement_version 'pip') meta_set "pip_version" "${PIP_VERSION}" local packages_to_install=( @@ -27,11 +29,8 @@ function pip::install_pip_setuptools_wheel() { # - Most of the Python ecosystem has stopped installing them for Python 3.12+ already. # See the Python CNB's removal for more details: https://github.com/heroku/buildpacks-python/pull/243 if [[ "${python_major_version}" == +(3.8|3.9|3.10|3.11|3.12) ]]; then - SETUPTOOLS_VERSION=$(get_requirement_version 'setuptools') - WHEEL_VERSION=$(get_requirement_version 'wheel') meta_set "setuptools_version" "${SETUPTOOLS_VERSION}" meta_set "wheel_version" "${WHEEL_VERSION}" - packages_to_install+=( "setuptools==${SETUPTOOLS_VERSION}" "wheel==${WHEEL_VERSION}" @@ -39,6 +38,9 @@ function pip::install_pip_setuptools_wheel() { packages_display_text+=", setuptools ${SETUPTOOLS_VERSION} and wheel ${WHEEL_VERSION}" fi + # Note: We still perform this install step even if the cache was reused, since we have no guarantee + # that the cached package versions are correct (different versions could have been specified in the + # app's requirements.txt in the last build). The install will be a no-op if the versions match. output::step "Installing ${packages_display_text}" /app/.heroku/python/bin/python "${bundled_pip_module_path}" install --quiet --disable-pip-version-check --no-cache-dir \ diff --git a/lib/pipenv.sh b/lib/pipenv.sh index bfe0032d6..cbd524a61 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -4,17 +4,19 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail +PIPENV_VERSION=$(get_requirement_version 'pipenv') + +# TODO: Either enable or remove these. # export CLINT_FORCE_COLOR=1 # export PIPENV_FORCE_COLOR=1 function pipenv::install_pipenv() { - # TODO: Either make this `local` or move elsewhere as part of the cache invalidation refactoring. - PIPENV_VERSION=$(get_requirement_version 'pipenv') meta_set "pipenv_version" "${PIPENV_VERSION}" output::step "Installing Pipenv ${PIPENV_VERSION}" # TODO: Install Pipenv into a venv so it isn't leaked into the app environment. + # TODO: Skip installing Pipenv if its version hasn't changed (once it's installed into a venv). # TODO: Explore viability of making Pipenv only be available during the build, to reduce slug size. /app/.heroku/python/bin/pip install --quiet --disable-pip-version-check --no-cache-dir "pipenv==${PIPENV_VERSION}" } diff --git a/lib/python_version.sh b/lib/python_version.sh index bd52223fa..d08fed160 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -74,8 +74,7 @@ function python_version::read_requested_python_version() { fi fi - # Protect against invalid versions somehow having been written into the cache. - # TODO: Move this validation into the cache handling as part of the cache refactor? + # Protect against unsupported (eg PyPy) or invalid versions being found in the cache metadata. if [[ "${cached_python_version}" =~ ^${PYTHON_VERSION_REGEX}$ ]]; then version="${cached_python_version}" origin="cached" diff --git a/spec/fixtures/requirements_basic/requirements.txt b/spec/fixtures/requirements_basic/requirements.txt deleted file mode 100644 index eec3a2223..000000000 --- a/spec/fixtures/requirements_basic/requirements.txt +++ /dev/null @@ -1,2 +0,0 @@ -# This package has been picked since it has no dependencies and is small/fast to install. -typing-extensions==4.12.2 diff --git a/spec/fixtures/requirements_basic/requirements.txt b/spec/fixtures/requirements_basic/requirements.txt new file mode 120000 index 000000000..eb513c3cd --- /dev/null +++ b/spec/fixtures/requirements_basic/requirements.txt @@ -0,0 +1 @@ +requirements/prod.txt \ No newline at end of file diff --git a/spec/fixtures/requirements_basic/requirements/prod.txt b/spec/fixtures/requirements_basic/requirements/prod.txt new file mode 100644 index 000000000..8a7dafe83 --- /dev/null +++ b/spec/fixtures/requirements_basic/requirements/prod.txt @@ -0,0 +1,5 @@ +# This requirements file is symlinked from the repo root's requirements.txt +# in order to test that symlinked requirements files work. + +# This package has been picked since it has no dependencies and is small/fast to install. +typing-extensions==4.12.2 diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 125c2d8d0..098fcdfca 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -70,7 +70,7 @@ expect(test_run.output).to include(<<~OUTPUT) -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version - -----> No change in requirements detected, installing from cache + -----> Restoring cache -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing SQLite3 @@ -146,6 +146,7 @@ expect(test_run.output).to match(Regexp.new(<<~REGEX)) -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + -----> Restoring cache -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing Pipenv #{PIPENV_VERSION} diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 199f234cb..dcf9bb762 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -25,7 +25,7 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip - remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) + remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 5)) remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) remote: Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) remote: Installing collected packages: typing-extensions @@ -61,7 +61,7 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version - remote: -----> No change in requirements detected, installing from cache + remote: -----> Restoring cache remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 @@ -77,20 +77,23 @@ it 'clears the cache before installing the packages again' do app.deploy do |app| - File.write('requirements.txt', 'six', mode: 'a') + # The test fixture's requirements.txt is a symlink to a requirements file in a subdirectory in + # order to test that symlinked requirements files work in general and with cache invalidation. + File.write('requirements/prod.txt', 'six', mode: 'a') app.commit! app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version - remote: -----> Requirements file has been changed, clearing cached dependencies + remote: -----> Discarding cache since: + remote: - The contents of requirements.txt changed remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip - remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) + remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 5)) remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) - remote: Collecting six (from -r requirements.txt (line 3)) + remote: Collecting six (from -r requirements.txt (line 6)) remote: Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB) remote: Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) remote: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) @@ -104,7 +107,6 @@ context 'when the package manager has changed from Pipenv to pip since the last build' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_basic') } - # TODO: Fix this case so the cache is actually cleared. it 'clears the cache before installing with pip' do app.deploy do |app| FileUtils.rm(['Pipfile', 'Pipfile.lock']) @@ -112,15 +114,21 @@ FileUtils.cp(FIXTURE_DIR.join('requirements_basic/requirements.txt'), '.') app.commit! app.push! - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version - remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Discarding cache since: + remote: - The package manager has changed from pipenv to pip + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip - remote: -----> Discovering process types - REGEX + remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 5)) + remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) + remote: Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) + remote: Installing collected packages: typing-extensions + remote: Successfully installed typing-extensions-4.12.2 + OUTPUT end end end diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 4be306e46..57c01614d 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -56,6 +56,7 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock + remote: -----> Restoring cache remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} @@ -80,13 +81,13 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected remote: -----> Using Python 3.9.0 specified in Pipfile.lock + remote: -----> Installing Python 3.9.0 remote: remote: ! Warning: A Python security update is available! remote: ! remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: - remote: -----> Installing Python 3.9.0 remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -283,10 +284,36 @@ end end + context 'when the Pipenv version has changed since the last build' do + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v253'] } + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_basic', buildpacks:) } + + it 'clears the cache before installing' do + app.deploy do |app| + update_buildpacks(app, [:default]) + app.commit! + app.push! + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Python app detected + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock + remote: -----> Discarding cache since: + remote: - The Python version has changed from 3.12.4 to #{DEFAULT_PYTHON_FULL_VERSION} + remote: - The Pipenv version has changed from 2023.12.1 to #{PIPENV_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing Pipenv #{PIPENV_VERSION} + remote: -----> Installing SQLite3 + remote: -----> Installing dependencies with Pipenv + remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Discovering process types + REGEX + end + end + end + context 'when the package manager has changed from pip to Pipenv since the last build' do let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_basic') } - # TODO: Fix this case so the cache is actually cleared. it 'clears the cache before installing with Pipenv' do app.deploy do |app| FileUtils.rm('.python-version') @@ -298,7 +325,9 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock - remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Discarding cache since: + remote: - The package manager has changed from pip to pipenv + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index c10e5c288..993a07454 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -16,6 +16,7 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python 3.8.0 specified in runtime.txt + remote: -----> Installing Python 3.8.0 remote: remote: ! Warning: Support for Python 3.8 is ending soon! remote: ! @@ -34,7 +35,7 @@ remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_8} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: - remote: -----> Installing Python 3.8.0 + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} OUTPUT end end @@ -70,13 +71,14 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python 3.9.0 specified in .python-version + remote: -----> Installing Python 3.9.0 remote: remote: ! Warning: A Python security update is available! remote: ! remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: - remote: -----> Installing Python 3.9.0 + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} OUTPUT end end diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 215148806..26318061f 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -63,14 +63,16 @@ remote: -----> Python app detected remote: -----> No Python version was specified. Using the same version as the last build: Python 3.12.3 remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Discarding cache since: + remote: - The pip version has changed + remote: -----> Installing Python 3.12.3 remote: remote: ! Warning: A Python security update is available! remote: ! remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_12} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: - remote: -----> No change in requirements detected, installing from cache - remote: -----> Using cached install of Python 3.12.3 + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} OUTPUT expect(app.run('python -V')).to include('Python 3.12.3') end @@ -88,6 +90,7 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python 3.8 specified in .python-version + remote: -----> Installing Python #{LATEST_PYTHON_3_8} remote: remote: ! Warning: Support for Python 3.8 is ending soon! remote: ! @@ -100,7 +103,6 @@ remote: ! Upgrade to a newer Python version as soon as possible to keep your app secure. remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: - remote: -----> Installing Python #{LATEST_PYTHON_3_8} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip @@ -415,12 +417,11 @@ File.write('.python-version', '3.13') app.commit! app.push! - # TODO: The output shouldn't say "installing from cache", since it's not. expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python 3.13 specified in .python-version - remote: -----> Python version has changed from #{LATEST_PYTHON_3_9} to #{LATEST_PYTHON_3_13}, clearing cache - remote: -----> No change in requirements detected, installing from cache + remote: -----> Discarding cache since: + remote: - The Python version has changed from #{LATEST_PYTHON_3_9} to #{LATEST_PYTHON_3_13} remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing requirements with pip diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index 9c84e5d16..097eb92dc 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -19,20 +19,20 @@ update_buildpacks(app, [:default]) app.commit! app.push! - # TODO: The requirements output shouldn't say "installing from cache", since it's not. expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> No Python version was specified. Using the same version as the last build: Python 3.12.3 remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Discarding cache since: + remote: - The stack has changed from heroku-22 to heroku-24 + remote: - The pip version has changed + remote: -----> Installing Python 3.12.3 remote: remote: ! Warning: A Python security update is available! remote: ! remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_12} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: - remote: -----> Stack has changed from heroku-22 to heroku-24, clearing cache - remote: -----> No change in requirements detected, installing from cache - remote: -----> Installing Python 3.12.3 remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip @@ -55,8 +55,8 @@ remote: -----> Python app detected remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Stack has changed from heroku-24 to heroku-22, clearing cache - remote: -----> No change in requirements detected, installing from cache + remote: -----> Discarding cache since: + remote: - The stack has changed from heroku-24 to heroku-22 remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 From d8e2ed993499188b73dec85a976cc2b8bc0d1861 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 31 Oct 2024 16:29:46 +0000 Subject: [PATCH 047/406] Misc package manager handling refactor (#1680) Split out of the Poetry PR to keep that one smaller. --- bin/compile | 8 ++++---- bin/utils | 12 ------------ lib/package_manager.sh | 4 +++- lib/pip.sh | 13 ++++++++----- lib/pipenv.sh | 2 +- lib/utils.sh | 16 ++++++++++++++-- 6 files changed, 30 insertions(+), 25 deletions(-) diff --git a/bin/compile b/bin/compile index 4c2d271c3..ce78b794c 100755 --- a/bin/compile +++ b/bin/compile @@ -19,6 +19,7 @@ ENV_DIR="${3}" BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) source "${BUILDPACK_DIR}/bin/utils" +source "${BUILDPACK_DIR}/lib/utils.sh" source "${BUILDPACK_DIR}/lib/cache.sh" source "${BUILDPACK_DIR}/lib/hooks.sh" source "${BUILDPACK_DIR}/lib/metadata.sh" @@ -27,7 +28,6 @@ source "${BUILDPACK_DIR}/lib/package_manager.sh" source "${BUILDPACK_DIR}/lib/pip.sh" source "${BUILDPACK_DIR}/lib/pipenv.sh" source "${BUILDPACK_DIR}/lib/python_version.sh" -source "${BUILDPACK_DIR}/lib/utils.sh" compile_start_time=$(nowms) @@ -75,6 +75,7 @@ PROFILE_PATH="$BUILD_DIR/.profile.d/python.sh" EXPORT_PATH="${BUILDPACK_DIR}/export" GUNICORN_PROFILE_PATH="$BUILD_DIR/.profile.d/python.gunicorn.sh" WEB_CONCURRENCY_PROFILE_PATH="$BUILD_DIR/.profile.d/WEB_CONCURRENCY.sh" +python_home='/app/.heroku/python' export PATH="/app/.heroku/python/bin:${PATH}" # Tell Python to not buffer it's stdin/stdout. @@ -156,14 +157,13 @@ meta_time "python_install_duration" "${install_python_start_time}" # Install the package manager and related tools. package_manager_install_start_time=$(nowms) -bundled_pip_module_path="$(utils::bundled_pip_module_path "${BUILD_DIR}")" case "${package_manager}" in pip) - pip::install_pip_setuptools_wheel "${bundled_pip_module_path}" "${python_major_version}" + pip::install_pip_setuptools_wheel "${python_home}" "${python_major_version}" ;; pipenv) # TODO: Stop installing pip when using Pipenv. - pip::install_pip_setuptools_wheel "${bundled_pip_module_path}" "${python_major_version}" + pip::install_pip_setuptools_wheel "${python_home}" "${python_major_version}" pipenv::install_pipenv ;; *) diff --git a/bin/utils b/bin/utils index c2b25254b..40f16a29b 100755 --- a/bin/utils +++ b/bin/utils @@ -41,15 +41,3 @@ is_module_available() { local module_name="${1}" python -c "import sys, importlib.util; sys.exit(0 if importlib.util.find_spec('${module_name}') else 1)" } - -# The requirement versions are effectively buildpack constants, however, we want -# Dependabot to be able to update them, which requires that they be in requirements -# files. The requirements files contain contents like `package==1.2.3` (and not just -# the package version) so we have to extract the version substring from it. -get_requirement_version() { - local package_name="${1}" - local requirement - requirement=$(cat "${BUILDPACK_DIR}/requirements/${package_name}.txt") - local requirement_version="${requirement#"${package_name}=="}" - echo "${requirement_version}" -} diff --git a/lib/package_manager.sh b/lib/package_manager.sh index 4c9fdd7f6..a06d212e0 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -36,7 +36,9 @@ function package_manager::determine_package_manager() { meta_set "setup_py_only" "false" fi - case "${#package_managers_found[@]}" in + local num_package_managers_found=${#package_managers_found[@]} + + case "${num_package_managers_found}" in 1) echo "${package_managers_found[0]}" return 0 diff --git a/lib/pip.sh b/lib/pip.sh index e58cea5af..1882f7f75 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -4,15 +4,18 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail -PIP_VERSION=$(get_requirement_version 'pip') -SETUPTOOLS_VERSION=$(get_requirement_version 'setuptools') -WHEEL_VERSION=$(get_requirement_version 'wheel') +PIP_VERSION=$(utils::get_requirement_version 'pip') +SETUPTOOLS_VERSION=$(utils::get_requirement_version 'setuptools') +WHEEL_VERSION=$(utils::get_requirement_version 'wheel') function pip::install_pip_setuptools_wheel() { + local python_home="${1}" + local python_major_version="${2}" + # We use the pip wheel bundled within Python's standard library to install our chosen # pip version, since it's faster than `ensurepip` followed by an upgrade in place. - local bundled_pip_module_path="${1}" - local python_major_version="${2}" + local bundled_pip_module_path + bundled_pip_module_path="$(utils::bundled_pip_module_path "${python_home}")" meta_set "pip_version" "${PIP_VERSION}" diff --git a/lib/pipenv.sh b/lib/pipenv.sh index cbd524a61..321b0a483 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -4,7 +4,7 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail -PIPENV_VERSION=$(get_requirement_version 'pipenv') +PIPENV_VERSION=$(utils::get_requirement_version 'pipenv') # TODO: Either enable or remove these. # export CLINT_FORCE_COLOR=1 diff --git a/lib/utils.sh b/lib/utils.sh index 8a0dd4a16..c855ade3e 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -4,17 +4,29 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail +# The requirement versions are effectively buildpack constants, however, we want +# Dependabot to be able to update them, which requires that they be in requirements +# files. The requirements files contain contents like `package==1.2.3` (and not just +# the package version) so we have to extract the version substring from it. +function utils::get_requirement_version() { + local package_name="${1}" + local requirement + requirement=$(cat "${BUILDPACK_DIR:?}/requirements/${package_name}.txt") + local requirement_version="${requirement#"${package_name}=="}" + echo "${requirement_version}" +} + # Python bundles pip within its standard library, which we can use to install our chosen # pip version from PyPI, saving us from having to download the usual pip bootstrap script. function utils::bundled_pip_module_path() { - local build_dir="${1}" + local python_home="${1}" # We have to use a glob since the bundled wheel filename contains the pip version, which # differs between Python versions. We also have to handle the case where there are multiple # matching pip wheels, since in some versions of Python (eg 3.9.0) multiple versions of pip # were accidentally bundled upstream. Note: This implementation relies upon `nullglob` being # set, which is the case thanks to the `bin/utils` that was run earlier. - local bundled_pip_wheel_list=("${build_dir}"/.heroku/python/lib/python*/ensurepip/_bundled/pip-*.whl) + local bundled_pip_wheel_list=("${python_home}"/lib/python*/ensurepip/_bundled/pip-*.whl) local bundled_pip_wheel="${bundled_pip_wheel_list[0]}" if [[ -z "${bundled_pip_wheel}" ]]; then From 4df96c285ed1aeed4ecf6909887165d4fc712381 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 31 Oct 2024 16:38:01 +0000 Subject: [PATCH 048/406] Prepare release v263 (#1681) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 575845c60..5cef03271 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v263] - 2024-10-31 + - Fixed cache handling so that it now also discards the cache when the package manager (or its version) changes. ([#1679](https://github.com/heroku/heroku-buildpack-python/pull/1679)) - Improved the build log output shown when restoring or discarding the cache. For example, if the cache was invalidated all reasons are now shown. ([#1679](https://github.com/heroku/heroku-buildpack-python/pull/1679)) - Stopped performing unnecessary cache file copies when the cache is due to be invalidated. This required moving the cache restoration step to after the `bin/pre_compile` hook runs. ([#1679](https://github.com/heroku/heroku-buildpack-python/pull/1679)) @@ -1078,7 +1081,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v262...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v263...main +[v263]: https://github.com/heroku/heroku-buildpack-python/compare/v262...v263 [v262]: https://github.com/heroku/heroku-buildpack-python/compare/v261...v262 [v261]: https://github.com/heroku/heroku-buildpack-python/compare/v260...v261 [v260]: https://github.com/heroku/heroku-buildpack-python/compare/v259...v260 From 6dda58abb53b3c25599e30c3e6f98dabc59e2f19 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 1 Nov 2024 11:34:08 +0000 Subject: [PATCH 049/406] Bump the ruby-dependencies group with 2 updates (#1683) Bumps the ruby-dependencies group with 2 updates: [rubocop](https://github.com/rubocop/rubocop) and [rubocop-rspec](https://github.com/rubocop/rubocop-rspec). Updates `rubocop` from 1.67.0 to 1.68.0 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.67.0...v1.68.0) Updates `rubocop-rspec` from 3.1.0 to 3.2.0 - [Release notes](https://github.com/rubocop/rubocop-rspec/releases) - [Changelog](https://github.com/rubocop/rubocop-rspec/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop-rspec/compare/v3.1.0...v3.2.0) --- updated-dependencies: - dependency-name: rubocop dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies - dependency-name: rubocop-rspec dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 0d99cf2df..41f0139f6 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -19,7 +19,7 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.7.2) + json (2.7.5) language_server-protocol (3.17.0.3) logger (1.6.1) moneta (1.0.0) @@ -28,7 +28,7 @@ GEM parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) - parser (3.3.5.0) + parser (3.3.5.1) ast (~> 2.4.1) racc platform-api (3.7.0) @@ -48,7 +48,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.1) - rubocop (1.67.0) + rubocop (1.68.0) json (~> 2.3) language_server-protocol (>= 3.17.0) parallel (~> 1.10) @@ -58,9 +58,9 @@ GEM rubocop-ast (>= 1.32.2, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 3.0) - rubocop-ast (1.32.3) + rubocop-ast (1.33.0) parser (>= 3.3.1.0) - rubocop-rspec (3.1.0) + rubocop-rspec (3.2.0) rubocop (~> 1.61) ruby-progressbar (1.13.0) thor (1.3.2) From 895a004fc022a30502bd21d60e5a0ff225c7b122 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 6 Nov 2024 09:12:33 +0000 Subject: [PATCH 050/406] Add support for Poetry (#1682) After many refactoring/preparation PRs, we're now ready to add support for the package manager Poetry: https://python-poetry.org To use Poetry, apps must have a `poetry.lock` lockfile, which can be created by running `poetry lock` locally, after adding Poetry config to `pyproject.toml` (which can be done either manually or by using `poetry init`). For now, if a `requirements.txt` or `Pipfile` is found it will take precedence over `poetry.lock` for backwards compatibility (in the future this will become a warning then an error). This means users of the third-party `python-poetry-buildpack` will need to remove that buildpack in order to use the new native Poetry support, since it exports a `requirements.txt` file during the build. Poetry is installed into the build cache rather than the slug, so is not available at run-time (since it's not typically needed at run-time and doing so reduces the slug size). The entrypoints of installed dependencies are available on `PATH`, so use of `poetry run` or `poetry shell` is not required at run-time to use dependencies in the environment. When using Poetry, pip is not installed since Poetry includes its own internal vendored copy that it will use instead (for the small number of Poetry operations for which it still calls out to pip, such as package uninstalls). During normal (non-CI) builds, the `poetry install --sync` command is run using `--only main` so as to only install the main `[tool.poetry.dependencies]` dependencies group from `pyproject.toml` and not any of the app's other dependency groups (such as test/dev/... groups, eg `[tool.poetry.group.test.dependencies]`). On Heroku CI, all default Poetry dependency groups are installed (i.e. all groups minus those marked as `optional = true`). Relevant Poetry docs: - https://python-poetry.org/docs/cli/#install - https://python-poetry.org/docs/configuration/ - https://python-poetry.org/docs/managing-dependencies/#dependency-groups See also the Python CNB equivalent of this PR: - https://github.com/heroku/buildpacks-python/pull/261 Note: We don't support controlling the Python version via Poetry's `tool.poetry.dependencies.python` field, since that field typically contains a version range, which is not safe to use. Use the newly added `.python-version` file support instead. For more on this, see the longer explanation over in the Python CNB repo: https://github.com/heroku/buildpacks-python/issues/260 Closes #796. Closes #835. GUS-W-16810914. --- CHANGELOG.md | 1 + README.md | 2 +- bin/compile | 11 +- bin/detect | 6 +- bin/report | 1 + bin/steps/collectstatic | 2 +- lib/cache.sh | 9 + lib/package_manager.sh | 18 +- lib/poetry.sh | 133 +++++++++++ requirements/poetry.txt | 1 + spec/fixtures/ci_poetry/.python-version | 1 + spec/fixtures/ci_poetry/app.json | 9 + spec/fixtures/ci_poetry/bin/compile | 12 + spec/fixtures/ci_poetry/bin/detect | 7 + spec/fixtures/ci_poetry/bin/post_compile | 5 + spec/fixtures/ci_poetry/bin/print-env-vars.sh | 5 + spec/fixtures/ci_poetry/poetry.lock | 85 +++++++ spec/fixtures/ci_poetry/pyproject.toml | 9 + spec/fixtures/pipenv_basic/bin/compile | 6 +- spec/fixtures/poetry_basic/.python-version | 1 + spec/fixtures/poetry_basic/bin/compile | 23 ++ spec/fixtures/poetry_basic/bin/detect | 7 + spec/fixtures/poetry_basic/poetry.lock | 85 +++++++ spec/fixtures/poetry_basic/pyproject.toml | 10 + spec/fixtures/poetry_editable/bin/compile | 12 + spec/fixtures/poetry_editable/bin/detect | 7 + .../fixtures/poetry_editable/bin/post_compile | 5 + .../poetry_editable/bin/test-entrypoints.sh | 18 ++ .../local_package_pyproject_toml/__init__.py | 2 + .../pyproject.toml | 6 + .../local_package_setup_py/__init__.py | 2 + .../packages/local_package_setup_py/setup.cfg | 10 + .../packages/local_package_setup_py/setup.py | 3 + spec/fixtures/poetry_editable/poetry.lock | 76 ++++++ .../poetry_editable/__init__.py | 0 spec/fixtures/poetry_editable/pyproject.toml | 15 ++ .../poetry_lockfile_out_of_sync/poetry.lock | 7 + .../pyproject.toml | 8 + spec/fixtures/requirements_basic/bin/compile | 6 +- .../.python-version | 1 + .../poetry.lock | 17 ++ .../pyproject.toml | 6 + .../requirements.txt | 2 + spec/hatchet/ci_spec.rb | 85 ++++++- spec/hatchet/detect_spec.rb | 6 +- spec/hatchet/package_manager_spec.rb | 6 +- spec/hatchet/poetry_spec.rb | 226 ++++++++++++++++++ spec/spec_helper.rb | 1 + 48 files changed, 954 insertions(+), 22 deletions(-) create mode 100644 lib/poetry.sh create mode 100644 requirements/poetry.txt create mode 100644 spec/fixtures/ci_poetry/.python-version create mode 100644 spec/fixtures/ci_poetry/app.json create mode 100755 spec/fixtures/ci_poetry/bin/compile create mode 100755 spec/fixtures/ci_poetry/bin/detect create mode 100644 spec/fixtures/ci_poetry/bin/post_compile create mode 100755 spec/fixtures/ci_poetry/bin/print-env-vars.sh create mode 100644 spec/fixtures/ci_poetry/poetry.lock create mode 100644 spec/fixtures/ci_poetry/pyproject.toml create mode 100644 spec/fixtures/poetry_basic/.python-version create mode 100755 spec/fixtures/poetry_basic/bin/compile create mode 100755 spec/fixtures/poetry_basic/bin/detect create mode 100644 spec/fixtures/poetry_basic/poetry.lock create mode 100644 spec/fixtures/poetry_basic/pyproject.toml create mode 100755 spec/fixtures/poetry_editable/bin/compile create mode 100755 spec/fixtures/poetry_editable/bin/detect create mode 100755 spec/fixtures/poetry_editable/bin/post_compile create mode 100755 spec/fixtures/poetry_editable/bin/test-entrypoints.sh create mode 100644 spec/fixtures/poetry_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py create mode 100644 spec/fixtures/poetry_editable/packages/local_package_pyproject_toml/pyproject.toml create mode 100644 spec/fixtures/poetry_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py create mode 100644 spec/fixtures/poetry_editable/packages/local_package_setup_py/setup.cfg create mode 100644 spec/fixtures/poetry_editable/packages/local_package_setup_py/setup.py create mode 100644 spec/fixtures/poetry_editable/poetry.lock create mode 100644 spec/fixtures/poetry_editable/poetry_editable/__init__.py create mode 100644 spec/fixtures/poetry_editable/pyproject.toml create mode 100644 spec/fixtures/poetry_lockfile_out_of_sync/poetry.lock create mode 100644 spec/fixtures/poetry_lockfile_out_of_sync/pyproject.toml create mode 100644 spec/fixtures/requirements_txt_and_poetry_lock/.python-version create mode 100644 spec/fixtures/requirements_txt_and_poetry_lock/poetry.lock create mode 100644 spec/fixtures/requirements_txt_and_poetry_lock/pyproject.toml create mode 100644 spec/fixtures/requirements_txt_and_poetry_lock/requirements.txt create mode 100644 spec/hatchet/poetry_spec.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index 5cef03271..012a9cc52 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Added support for the package manager Poetry. Apps must have a `pyproject.toml` + `poetry.lock` and no other package manager files (otherwise pip/Pipenv will take precedence for backwards compatibility). ([#1682](https://github.com/heroku/heroku-buildpack-python/pull/1682)) ## [v263] - 2024-10-31 diff --git a/README.md b/README.md index 235dffcd4..f2767f7db 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ See the [Getting Started on Heroku with Python](https://devcenter.heroku.com/art ## Application Requirements -A `requirements.txt` or `Pipfile` file must be present in the root (top-level) directory of your app's source code. +A `requirements.txt`, `Pipfile` or `poetry.lock` file must be present in the root (top-level) directory of your app's source code. ## Configuration diff --git a/bin/compile b/bin/compile index ce78b794c..9ce9cb60f 100755 --- a/bin/compile +++ b/bin/compile @@ -28,6 +28,7 @@ source "${BUILDPACK_DIR}/lib/package_manager.sh" source "${BUILDPACK_DIR}/lib/pip.sh" source "${BUILDPACK_DIR}/lib/pipenv.sh" source "${BUILDPACK_DIR}/lib/python_version.sh" +source "${BUILDPACK_DIR}/lib/poetry.sh" compile_start_time=$(nowms) @@ -166,6 +167,9 @@ case "${package_manager}" in pip::install_pip_setuptools_wheel "${python_home}" "${python_major_version}" pipenv::install_pipenv ;; + poetry) + poetry::install_poetry "${python_home}" "${CACHE_DIR}" "${EXPORT_PATH}" + ;; *) utils::abort_internal_error "Unhandled package manager: ${package_manager}" ;; @@ -175,8 +179,8 @@ meta_time "package_manager_install_duration" "${package_manager_install_start_ti # SQLite3 support. # Installs the sqlite3 dev headers and sqlite3 binary but not the # libsqlite3-0 library since that exists in the base image. -# We skip this step on Python 3.13, as a first step towards removing this feature. -if [[ "${python_major_version}" == +(3.8|3.9|3.10|3.11|3.12) ]]; then +# We skip this step on Python 3.13 or when using Poetry, as a first step towards removing this feature. +if [[ "${python_major_version}" == +(3.8|3.9|3.10|3.11|3.12) && "${package_manager}" != "poetry" ]]; then install_sqlite_start_time=$(nowms) source "${BUILDPACK_DIR}/bin/steps/sqlite3" buildpack_sqlite3_install @@ -192,6 +196,9 @@ case "${package_manager}" in pipenv) pipenv::install_dependencies ;; + poetry) + poetry::install_dependencies + ;; *) utils::abort_internal_error "Unhandled package manager: ${package_manager}" ;; diff --git a/bin/detect b/bin/detect index 9f6303298..4b5a535eb 100755 --- a/bin/detect +++ b/bin/detect @@ -49,9 +49,9 @@ output::error <>"${export_file}" + # Force Poetry to manage the system Python site-packages instead of using venvs. + export POETRY_VIRTUALENVS_CREATE="false" + echo 'export POETRY_VIRTUALENVS_CREATE="false"' >>"${export_file}" +} + +# Note: We cache site-packages since: +# - It results in faster builds than only caching Poetry's download/wheel cache. +# - It's safe to do so, since `poetry install --sync` fully manages the environment +# (including e.g. uninstalling packages when they are removed from the lockfile). +# +# With site-packages cached there is no need to persist Poetry's download/wheel cache in the build +# cache, so we let Poetry write it to the home directory where it will be discarded at the end of +# the build. We don't use `--no-cache` since the cache still offers benefits (such as avoiding +# repeat downloads of PEP-517/518 build requirements). +function poetry::install_dependencies() { + local poetry_install_command=( + poetry + install + --sync + ) + + # On Heroku CI, all default Poetry dependency groups are installed (i.e. all groups minus those + # marked as `optional = true`). Otherwise, only the 'main' Poetry dependency group is installed. + if [[ ! -v INSTALL_TEST ]]; then + poetry_install_command+=(--only main) + fi + + # We only display the most relevant command args here, to improve the signal to noise ratio. + output::step "Installing dependencies using '${poetry_install_command[*]}'" + + # `--compile`: Compiles Python bytecode, to improve app boot times (pip does this by default). + # `--no-ansi`: Whilst we'd prefer to enable colour if possible, Poetry also emits ANSI escape + # codes for redrawing lines, which renders badly in persisted build logs. + # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. + if ! { + "${poetry_install_command[@]}" --compile --no-ansi --no-interaction \ + |& tee "${WARNINGS_LOG:?}" \ + |& grep --invert-match 'Skipping virtualenv creation' \ + |& output::indent + }; then + show-warnings + + output::error <<-EOF + Error: Unable to install dependencies using Poetry. + + See the log output above for more information. + EOF + meta_set "failure_reason" "install-dependencies::poetry" + return 1 + fi +} diff --git a/requirements/poetry.txt b/requirements/poetry.txt new file mode 100644 index 000000000..8e44691cb --- /dev/null +++ b/requirements/poetry.txt @@ -0,0 +1 @@ +poetry==1.8.4 diff --git a/spec/fixtures/ci_poetry/.python-version b/spec/fixtures/ci_poetry/.python-version new file mode 100644 index 000000000..e4fba2183 --- /dev/null +++ b/spec/fixtures/ci_poetry/.python-version @@ -0,0 +1 @@ +3.12 diff --git a/spec/fixtures/ci_poetry/app.json b/spec/fixtures/ci_poetry/app.json new file mode 100644 index 000000000..59617c169 --- /dev/null +++ b/spec/fixtures/ci_poetry/app.json @@ -0,0 +1,9 @@ +{ + "environments": { + "test": { + "scripts": { + "test": "./bin/print-env-vars.sh && pytest --version" + } + } + } +} diff --git a/spec/fixtures/ci_poetry/bin/compile b/spec/fixtures/ci_poetry/bin/compile new file mode 100755 index 000000000..fcb7055e3 --- /dev/null +++ b/spec/fixtures/ci_poetry/bin/compile @@ -0,0 +1,12 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack, and tests that the environment is +# configured as expected for buildpacks that run after the Python buildpack. + +set -euo pipefail + +BUILD_DIR="${1}" + +cd "${BUILD_DIR}" + +exec bin/print-env-vars.sh diff --git a/spec/fixtures/ci_poetry/bin/detect b/spec/fixtures/ci_poetry/bin/detect new file mode 100755 index 000000000..68cdcc4a2 --- /dev/null +++ b/spec/fixtures/ci_poetry/bin/detect @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack. + +set -euo pipefail + +echo "Inline" diff --git a/spec/fixtures/ci_poetry/bin/post_compile b/spec/fixtures/ci_poetry/bin/post_compile new file mode 100644 index 000000000..15362f6b1 --- /dev/null +++ b/spec/fixtures/ci_poetry/bin/post_compile @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -euo pipefail + +exec bin/print-env-vars.sh diff --git a/spec/fixtures/ci_poetry/bin/print-env-vars.sh b/spec/fixtures/ci_poetry/bin/print-env-vars.sh new file mode 100755 index 000000000..9e0bebe6b --- /dev/null +++ b/spec/fixtures/ci_poetry/bin/print-env-vars.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -euo pipefail + +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|BUILD_DIR|CACHE_DIR|CI_NODE_.+|DYNO|ENV_DIR|HEROKU_TEST_RUN_.+|HOME|OLDPWD|PORT|PWD|SHLVL|STACK|TERM)=' diff --git a/spec/fixtures/ci_poetry/poetry.lock b/spec/fixtures/ci_poetry/poetry.lock new file mode 100644 index 000000000..a727a186c --- /dev/null +++ b/spec/fixtures/ci_poetry/poetry.lock @@ -0,0 +1,85 @@ +# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand. + +[[package]] +name = "colorama" +version = "0.4.6" +description = "Cross-platform colored terminal text." +optional = false +python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,>=2.7" +files = [ + {file = "colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"}, + {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"}, +] + +[[package]] +name = "iniconfig" +version = "2.0.0" +description = "brain-dead simple config-ini parsing" +optional = false +python-versions = ">=3.7" +files = [ + {file = "iniconfig-2.0.0-py3-none-any.whl", hash = "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374"}, + {file = "iniconfig-2.0.0.tar.gz", hash = "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3"}, +] + +[[package]] +name = "packaging" +version = "24.1" +description = "Core utilities for Python packages" +optional = false +python-versions = ">=3.8" +files = [ + {file = "packaging-24.1-py3-none-any.whl", hash = "sha256:5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124"}, + {file = "packaging-24.1.tar.gz", hash = "sha256:026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002"}, +] + +[[package]] +name = "pluggy" +version = "1.5.0" +description = "plugin and hook calling mechanisms for python" +optional = false +python-versions = ">=3.8" +files = [ + {file = "pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669"}, + {file = "pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1"}, +] + +[package.extras] +dev = ["pre-commit", "tox"] +testing = ["pytest", "pytest-benchmark"] + +[[package]] +name = "pytest" +version = "8.3.3" +description = "pytest: simple powerful testing with Python" +optional = false +python-versions = ">=3.8" +files = [ + {file = "pytest-8.3.3-py3-none-any.whl", hash = "sha256:a6853c7375b2663155079443d2e45de913a911a11d669df02a50814944db57b2"}, + {file = "pytest-8.3.3.tar.gz", hash = "sha256:70b98107bd648308a7952b06e6ca9a50bc660be218d53c257cc1fc94fda10181"}, +] + +[package.dependencies] +colorama = {version = "*", markers = "sys_platform == \"win32\""} +iniconfig = "*" +packaging = "*" +pluggy = ">=1.5,<2" + +[package.extras] +dev = ["argcomplete", "attrs (>=19.2)", "hypothesis (>=3.56)", "mock", "pygments (>=2.7.2)", "requests", "setuptools", "xmlschema"] + +[[package]] +name = "typing-extensions" +version = "4.12.2" +description = "Backported and Experimental Type Hints for Python 3.8+" +optional = false +python-versions = ">=3.8" +files = [ + {file = "typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d"}, + {file = "typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"}, +] + +[metadata] +lock-version = "2.0" +python-versions = "^3.12" +content-hash = "2e71a7976f439ce69fc771708b83dcfc6f795072ea73a7c2de0241878cbd378a" diff --git a/spec/fixtures/ci_poetry/pyproject.toml b/spec/fixtures/ci_poetry/pyproject.toml new file mode 100644 index 000000000..99aa593b8 --- /dev/null +++ b/spec/fixtures/ci_poetry/pyproject.toml @@ -0,0 +1,9 @@ +[tool.poetry] +package-mode = false + +[tool.poetry.dependencies] +python = "^3.12" +typing-extensions = "*" + +[tool.poetry.group.test.dependencies] +pytest = "*" diff --git a/spec/fixtures/pipenv_basic/bin/compile b/spec/fixtures/pipenv_basic/bin/compile index db00975d6..f73ce9e2f 100755 --- a/spec/fixtures/pipenv_basic/bin/compile +++ b/spec/fixtures/pipenv_basic/bin/compile @@ -5,7 +5,11 @@ set -euo pipefail -printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK)=' +BUILD_DIR="${1}" + +cd "${BUILD_DIR}" + +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|OLDPWD|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK)=' echo python -c 'import pprint, sys; pprint.pp(sys.path)' diff --git a/spec/fixtures/poetry_basic/.python-version b/spec/fixtures/poetry_basic/.python-version new file mode 100644 index 000000000..e4fba2183 --- /dev/null +++ b/spec/fixtures/poetry_basic/.python-version @@ -0,0 +1 @@ +3.12 diff --git a/spec/fixtures/poetry_basic/bin/compile b/spec/fixtures/poetry_basic/bin/compile new file mode 100755 index 000000000..55423bd97 --- /dev/null +++ b/spec/fixtures/poetry_basic/bin/compile @@ -0,0 +1,23 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack, and tests that the environment is +# configured as expected for buildpacks that run after the Python buildpack. + +set -euo pipefail + +BUILD_DIR="${1}" + +cd "${BUILD_DIR}" + +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|OLDPWD|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK)=' +echo + +python -c 'import pprint, sys; pprint.pp(sys.path)' +echo + +# The show command also lists dependencies that are in optional groups in pyproject.toml +# but that aren't actually installed, for which the only option is to filter out by hand. +poetry show | grep -v ' (!) ' +echo + +python -c 'import typing_extensions; print(typing_extensions)' diff --git a/spec/fixtures/poetry_basic/bin/detect b/spec/fixtures/poetry_basic/bin/detect new file mode 100755 index 000000000..68cdcc4a2 --- /dev/null +++ b/spec/fixtures/poetry_basic/bin/detect @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack. + +set -euo pipefail + +echo "Inline" diff --git a/spec/fixtures/poetry_basic/poetry.lock b/spec/fixtures/poetry_basic/poetry.lock new file mode 100644 index 000000000..a727a186c --- /dev/null +++ b/spec/fixtures/poetry_basic/poetry.lock @@ -0,0 +1,85 @@ +# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand. + +[[package]] +name = "colorama" +version = "0.4.6" +description = "Cross-platform colored terminal text." +optional = false +python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,>=2.7" +files = [ + {file = "colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"}, + {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"}, +] + +[[package]] +name = "iniconfig" +version = "2.0.0" +description = "brain-dead simple config-ini parsing" +optional = false +python-versions = ">=3.7" +files = [ + {file = "iniconfig-2.0.0-py3-none-any.whl", hash = "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374"}, + {file = "iniconfig-2.0.0.tar.gz", hash = "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3"}, +] + +[[package]] +name = "packaging" +version = "24.1" +description = "Core utilities for Python packages" +optional = false +python-versions = ">=3.8" +files = [ + {file = "packaging-24.1-py3-none-any.whl", hash = "sha256:5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124"}, + {file = "packaging-24.1.tar.gz", hash = "sha256:026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002"}, +] + +[[package]] +name = "pluggy" +version = "1.5.0" +description = "plugin and hook calling mechanisms for python" +optional = false +python-versions = ">=3.8" +files = [ + {file = "pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669"}, + {file = "pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1"}, +] + +[package.extras] +dev = ["pre-commit", "tox"] +testing = ["pytest", "pytest-benchmark"] + +[[package]] +name = "pytest" +version = "8.3.3" +description = "pytest: simple powerful testing with Python" +optional = false +python-versions = ">=3.8" +files = [ + {file = "pytest-8.3.3-py3-none-any.whl", hash = "sha256:a6853c7375b2663155079443d2e45de913a911a11d669df02a50814944db57b2"}, + {file = "pytest-8.3.3.tar.gz", hash = "sha256:70b98107bd648308a7952b06e6ca9a50bc660be218d53c257cc1fc94fda10181"}, +] + +[package.dependencies] +colorama = {version = "*", markers = "sys_platform == \"win32\""} +iniconfig = "*" +packaging = "*" +pluggy = ">=1.5,<2" + +[package.extras] +dev = ["argcomplete", "attrs (>=19.2)", "hypothesis (>=3.56)", "mock", "pygments (>=2.7.2)", "requests", "setuptools", "xmlschema"] + +[[package]] +name = "typing-extensions" +version = "4.12.2" +description = "Backported and Experimental Type Hints for Python 3.8+" +optional = false +python-versions = ">=3.8" +files = [ + {file = "typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d"}, + {file = "typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"}, +] + +[metadata] +lock-version = "2.0" +python-versions = "^3.12" +content-hash = "2e71a7976f439ce69fc771708b83dcfc6f795072ea73a7c2de0241878cbd378a" diff --git a/spec/fixtures/poetry_basic/pyproject.toml b/spec/fixtures/poetry_basic/pyproject.toml new file mode 100644 index 000000000..1b21050ab --- /dev/null +++ b/spec/fixtures/poetry_basic/pyproject.toml @@ -0,0 +1,10 @@ +[tool.poetry] +package-mode = false + +[tool.poetry.dependencies] +python = "^3.12" +typing-extensions = "*" + +# This group shouldn't be installed due to us passing `--only main`. +[tool.poetry.group.test.dependencies] +pytest = "*" diff --git a/spec/fixtures/poetry_editable/bin/compile b/spec/fixtures/poetry_editable/bin/compile new file mode 100755 index 000000000..df17e9401 --- /dev/null +++ b/spec/fixtures/poetry_editable/bin/compile @@ -0,0 +1,12 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack, and tests that editable requirements are +# usable by buildpacks that run after the Python buildpack during the build. + +set -euo pipefail + +BUILD_DIR="${1}" + +cd "${BUILD_DIR}" + +exec bin/test-entrypoints.sh diff --git a/spec/fixtures/poetry_editable/bin/detect b/spec/fixtures/poetry_editable/bin/detect new file mode 100755 index 000000000..68cdcc4a2 --- /dev/null +++ b/spec/fixtures/poetry_editable/bin/detect @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack. + +set -euo pipefail + +echo "Inline" diff --git a/spec/fixtures/poetry_editable/bin/post_compile b/spec/fixtures/poetry_editable/bin/post_compile new file mode 100755 index 000000000..6e77d159a --- /dev/null +++ b/spec/fixtures/poetry_editable/bin/post_compile @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -euo pipefail + +exec bin/test-entrypoints.sh diff --git a/spec/fixtures/poetry_editable/bin/test-entrypoints.sh b/spec/fixtures/poetry_editable/bin/test-entrypoints.sh new file mode 100755 index 000000000..fc941ed3f --- /dev/null +++ b/spec/fixtures/poetry_editable/bin/test-entrypoints.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash + +set -euo pipefail + +cd .heroku/python/lib/python*/site-packages/ + +# List any path like strings in .pth, and finder files in site-packages. +grep --extended-regexp --only-matching -- '/\S+' *.pth __editable___*_finder.py | sort +echo + +echo -n "Running entrypoint for the pyproject.toml-based local package: " +local_package_pyproject_toml + +echo -n "Running entrypoint for the setup.py-based local package: " +local_package_setup_py + +echo -n "Running entrypoint for the VCS package: " +gunicorn --version diff --git a/spec/fixtures/poetry_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py b/spec/fixtures/poetry_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py new file mode 100644 index 000000000..b86d791d2 --- /dev/null +++ b/spec/fixtures/poetry_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py @@ -0,0 +1,2 @@ +def hello(): + print("Hello pyproject.toml!") diff --git a/spec/fixtures/poetry_editable/packages/local_package_pyproject_toml/pyproject.toml b/spec/fixtures/poetry_editable/packages/local_package_pyproject_toml/pyproject.toml new file mode 100644 index 000000000..f567c3777 --- /dev/null +++ b/spec/fixtures/poetry_editable/packages/local_package_pyproject_toml/pyproject.toml @@ -0,0 +1,6 @@ +[project] +name = "local_package_pyproject_toml" +version = "0.0.1" + +[project.scripts] +local_package_pyproject_toml = "local_package_pyproject_toml:hello" diff --git a/spec/fixtures/poetry_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py b/spec/fixtures/poetry_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py new file mode 100644 index 000000000..e98d42373 --- /dev/null +++ b/spec/fixtures/poetry_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py @@ -0,0 +1,2 @@ +def hello(): + print("Hello setup.py!") diff --git a/spec/fixtures/poetry_editable/packages/local_package_setup_py/setup.cfg b/spec/fixtures/poetry_editable/packages/local_package_setup_py/setup.cfg new file mode 100644 index 000000000..eff513964 --- /dev/null +++ b/spec/fixtures/poetry_editable/packages/local_package_setup_py/setup.cfg @@ -0,0 +1,10 @@ +[metadata] +name = local_package_setup_py +version = 0.0.1 + +[options] +packages = local_package_setup_py + +[options.entry_points] +console_scripts = + local_package_setup_py = local_package_setup_py:hello diff --git a/spec/fixtures/poetry_editable/packages/local_package_setup_py/setup.py b/spec/fixtures/poetry_editable/packages/local_package_setup_py/setup.py new file mode 100644 index 000000000..606849326 --- /dev/null +++ b/spec/fixtures/poetry_editable/packages/local_package_setup_py/setup.py @@ -0,0 +1,3 @@ +from setuptools import setup + +setup() diff --git a/spec/fixtures/poetry_editable/poetry.lock b/spec/fixtures/poetry_editable/poetry.lock new file mode 100644 index 000000000..4047b0d15 --- /dev/null +++ b/spec/fixtures/poetry_editable/poetry.lock @@ -0,0 +1,76 @@ +# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand. + +[[package]] +name = "gunicorn" +version = "20.1.0" +description = "WSGI HTTP Server for UNIX" +optional = false +python-versions = ">=3.5" +files = [] +develop = true + +[package.dependencies] +setuptools = ">=3.0" + +[package.extras] +eventlet = ["eventlet (>=0.24.1)"] +gevent = ["gevent (>=1.4.0)"] +setproctitle = ["setproctitle"] +tornado = ["tornado (>=0.2)"] + +[package.source] +type = "git" +url = "https://github.com/benoitc/gunicorn.git" +reference = "20.1.0" +resolved_reference = "61ccfd6c38d477a908e0f376757bbb884438053a" + +[[package]] +name = "local_package_pyproject_toml" +version = "0.0.1" +description = "" +optional = false +python-versions = "*" +files = [] +develop = true + +[package.source] +type = "directory" +url = "packages/local_package_pyproject_toml" + +[[package]] +name = "local_package_setup_py" +version = "0.0.1" +description = "" +optional = false +python-versions = "*" +files = [] +develop = true + +[package.source] +type = "directory" +url = "packages/local_package_setup_py" + +[[package]] +name = "setuptools" +version = "75.2.0" +description = "Easily download, build, install, upgrade, and uninstall Python packages" +optional = false +python-versions = ">=3.8" +files = [ + {file = "setuptools-75.2.0-py3-none-any.whl", hash = "sha256:a7fcb66f68b4d9e8e66b42f9876150a3371558f98fa32222ffaa5bced76406f8"}, + {file = "setuptools-75.2.0.tar.gz", hash = "sha256:753bb6ebf1f465a1912e19ed1d41f403a79173a9acf66a42e7e6aec45c3c16ec"}, +] + +[package.extras] +check = ["pytest-checkdocs (>=2.4)", "pytest-ruff (>=0.2.1)", "ruff (>=0.5.2)"] +core = ["importlib-metadata (>=6)", "importlib-resources (>=5.10.2)", "jaraco.collections", "jaraco.functools", "jaraco.text (>=3.7)", "more-itertools", "more-itertools (>=8.8)", "packaging", "packaging (>=24)", "platformdirs (>=2.6.2)", "tomli (>=2.0.1)", "wheel (>=0.43.0)"] +cover = ["pytest-cov"] +doc = ["furo", "jaraco.packaging (>=9.3)", "jaraco.tidelift (>=1.4)", "pygments-github-lexers (==0.0.5)", "pyproject-hooks (!=1.1)", "rst.linker (>=1.9)", "sphinx (>=3.5)", "sphinx-favicon", "sphinx-inline-tabs", "sphinx-lint", "sphinx-notfound-page (>=1,<2)", "sphinx-reredirects", "sphinxcontrib-towncrier", "towncrier (<24.7)"] +enabler = ["pytest-enabler (>=2.2)"] +test = ["build[virtualenv] (>=1.0.3)", "filelock (>=3.4.0)", "ini2toml[lite] (>=0.14)", "jaraco.develop (>=7.21)", "jaraco.envs (>=2.2)", "jaraco.path (>=3.2.0)", "jaraco.test", "packaging (>=23.2)", "pip (>=19.1)", "pyproject-hooks (!=1.1)", "pytest (>=6,!=8.1.*)", "pytest-home (>=0.5)", "pytest-perf", "pytest-subprocess", "pytest-timeout", "pytest-xdist (>=3)", "tomli-w (>=1.0.0)", "virtualenv (>=13.0.0)", "wheel (>=0.44.0)"] +type = ["importlib-metadata (>=7.0.2)", "jaraco.develop (>=7.21)", "mypy (==1.11.*)", "pytest-mypy"] + +[metadata] +lock-version = "2.0" +python-versions = "^3.12" +content-hash = "f362eb3b600ebb153055e8c4ed85f1cdeb18ffd8528710d5e1631c448a2f8adc" diff --git a/spec/fixtures/poetry_editable/poetry_editable/__init__.py b/spec/fixtures/poetry_editable/poetry_editable/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/poetry_editable/pyproject.toml b/spec/fixtures/poetry_editable/pyproject.toml new file mode 100644 index 000000000..7bcc6ab80 --- /dev/null +++ b/spec/fixtures/poetry_editable/pyproject.toml @@ -0,0 +1,15 @@ +[tool.poetry] +name = "poetry-editable" +version = "0.0.1" +description = "" +authors = [] + +[tool.poetry.dependencies] +python = "^3.12" +gunicorn = { git = "https://github.com/benoitc/gunicorn.git", tag = "20.1.0", develop = true } +local-package-pyproject-toml = { path = "packages/local_package_pyproject_toml", develop = true } +local-package-setup-py = { path = "packages/local_package_setup_py", develop = true } + +[build-system] +requires = ["poetry-core"] +build-backend = "poetry.core.masonry.api" diff --git a/spec/fixtures/poetry_lockfile_out_of_sync/poetry.lock b/spec/fixtures/poetry_lockfile_out_of_sync/poetry.lock new file mode 100644 index 000000000..8738e5fc8 --- /dev/null +++ b/spec/fixtures/poetry_lockfile_out_of_sync/poetry.lock @@ -0,0 +1,7 @@ +# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand. +package = [] + +[metadata] +lock-version = "2.0" +python-versions = "^3.12" +content-hash = "34e39677d8527182346093002688d17a5d2fc204b9eb3e094b2e6ac519028228" diff --git a/spec/fixtures/poetry_lockfile_out_of_sync/pyproject.toml b/spec/fixtures/poetry_lockfile_out_of_sync/pyproject.toml new file mode 100644 index 000000000..f0367e35e --- /dev/null +++ b/spec/fixtures/poetry_lockfile_out_of_sync/pyproject.toml @@ -0,0 +1,8 @@ +[tool.poetry] +package-mode = false + +[tool.poetry.dependencies] +python = "^3.12" + +# This dependency isn't in the lockfile. +typing-extensions = "*" diff --git a/spec/fixtures/requirements_basic/bin/compile b/spec/fixtures/requirements_basic/bin/compile index 68d243e7b..29cce648e 100755 --- a/spec/fixtures/requirements_basic/bin/compile +++ b/spec/fixtures/requirements_basic/bin/compile @@ -5,7 +5,11 @@ set -euo pipefail -printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK)=' +BUILD_DIR="${1}" + +cd "${BUILD_DIR}" + +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|OLDPWD|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK)=' echo python -c 'import pprint, sys; pprint.pp(sys.path)' diff --git a/spec/fixtures/requirements_txt_and_poetry_lock/.python-version b/spec/fixtures/requirements_txt_and_poetry_lock/.python-version new file mode 100644 index 000000000..e4fba2183 --- /dev/null +++ b/spec/fixtures/requirements_txt_and_poetry_lock/.python-version @@ -0,0 +1 @@ +3.12 diff --git a/spec/fixtures/requirements_txt_and_poetry_lock/poetry.lock b/spec/fixtures/requirements_txt_and_poetry_lock/poetry.lock new file mode 100644 index 000000000..7022f9054 --- /dev/null +++ b/spec/fixtures/requirements_txt_and_poetry_lock/poetry.lock @@ -0,0 +1,17 @@ +# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand. + +[[package]] +name = "typing-extensions" +version = "4.12.2" +description = "Backported and Experimental Type Hints for Python 3.8+" +optional = false +python-versions = ">=3.8" +files = [ + {file = "typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d"}, + {file = "typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"}, +] + +[metadata] +lock-version = "2.0" +python-versions = "^3.12" +content-hash = "2600e9ec45a8acb482a004ff092cb1453c2550aeec843430255a623cad8f7f86" diff --git a/spec/fixtures/requirements_txt_and_poetry_lock/pyproject.toml b/spec/fixtures/requirements_txt_and_poetry_lock/pyproject.toml new file mode 100644 index 000000000..b6077bf2c --- /dev/null +++ b/spec/fixtures/requirements_txt_and_poetry_lock/pyproject.toml @@ -0,0 +1,6 @@ +[tool.poetry] +package-mode = false + +[tool.poetry.dependencies] +python = "^3.12" +typing-extensions = "*" diff --git a/spec/fixtures/requirements_txt_and_poetry_lock/requirements.txt b/spec/fixtures/requirements_txt_and_poetry_lock/requirements.txt new file mode 100644 index 000000000..eec3a2223 --- /dev/null +++ b/spec/fixtures/requirements_txt_and_poetry_lock/requirements.txt @@ -0,0 +1,2 @@ +# This package has been picked since it has no dependencies and is small/fast to install. +typing-extensions==4.12.2 diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 098fcdfca..06594b0a1 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -66,7 +66,6 @@ REGEX test_run.run_again - expect(test_run.output).to include(<<~OUTPUT) -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version @@ -142,7 +141,6 @@ REGEX test_run.run_again - expect(test_run.output).to match(Regexp.new(<<~REGEX)) -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version @@ -160,4 +158,87 @@ end end end + + context 'when using Poetry' do + let(:app) { Hatchet::Runner.new('spec/fixtures/ci_poetry', buildpacks:) } + + it 'installs both normal and test dependencies and uses cache on subsequent runs' do + app.run_ci do |test_run| + expect(test_run.output).to match(Regexp.new(<<~REGEX)) + -----> Python app detected + -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + -----> Installing Poetry #{POETRY_VERSION} + -----> Installing dependencies using 'poetry install --sync' + Installing dependencies from lock file + + Package operations: 5 installs, 0 updates, 0 removals + + - Installing iniconfig .+ + - Installing packaging .+ + - Installing pluggy .+ + - Installing pytest .+ + - Installing typing-extensions .+ + -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. + -----> Running bin/post_compile hook + CI=true + CPLUS_INCLUDE_PATH=/app/.heroku/python/include + C_INCLUDE_PATH=/app/.heroku/python/include + DISABLE_COLLECTSTATIC=1 + INSTALL_TEST=1 + LANG=en_US.UTF-8 + LC_ALL=C.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib + PATH=/tmp/cache.+/.heroku/python-poetry/bin:/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PIP_NO_PYTHON_VERSION_WARNING=1 + PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config + POETRY_VIRTUALENVS_CREATE=false + PYTHONUNBUFFERED=1 + -----> Inline app detected + LANG=en_US.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib + PATH=/app/.heroku/python/bin:/tmp/cache.+/.heroku/python-poetry/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + POETRY_VIRTUALENVS_CREATE=false + PYTHONHASHSEED=random + PYTHONHOME=/app/.heroku/python + PYTHONPATH=/app + PYTHONUNBUFFERED=true + -----> No test-setup command provided. Skipping. + -----> Running test command `./bin/print-env-vars.sh && pytest --version`... + CI=true + DYNO_RAM=2560 + FORWARDED_ALLOW_IPS=\\* + GUNICORN_CMD_ARGS=--access-logfile - + LANG=en_US.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib + PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ + PYTHONHASHSEED=random + PYTHONHOME=/app/.heroku/python + PYTHONPATH=/app + PYTHONUNBUFFERED=true + WEB_CONCURRENCY=5 + pytest 8.3.3 + -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully + REGEX + + test_run.run_again + expect(test_run.output).to include(<<~OUTPUT) + -----> Python app detected + -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + -----> Restoring cache + -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} + -----> Installing Poetry #{POETRY_VERSION} + -----> Installing dependencies using 'poetry install --sync' + Installing dependencies from lock file + + No dependencies to install or update + -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. + -----> Running bin/post_compile hook + OUTPUT + end + end + end end diff --git a/spec/hatchet/detect_spec.rb b/spec/hatchet/detect_spec.rb index dbe7f1e93..044b0010a 100644 --- a/spec/hatchet/detect_spec.rb +++ b/spec/hatchet/detect_spec.rb @@ -17,9 +17,9 @@ remote: ! Error: Your app is configured to use the Python buildpack, remote: ! but we couldn't find any supported Python project files. remote: ! - remote: ! A Python app on Heroku must have either a 'requirements.txt' or - remote: ! 'Pipfile' package manager file in the root directory of its - remote: ! source code. + remote: ! A Python app on Heroku must have either a 'requirements.txt', + remote: ! 'Pipfile' or 'poetry.lock' package manager file in the root + remote: ! directory of its source code. remote: ! remote: ! Currently the root directory of your app contains: remote: ! diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index cde1f154b..e390d62e8 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -13,9 +13,9 @@ remote: remote: ! Error: Couldn't find any supported Python package manager files. remote: ! - remote: ! A Python app on Heroku must have either a 'requirements.txt' or - remote: ! 'Pipfile' package manager file in the root directory of its - remote: ! source code. + remote: ! A Python app on Heroku must have either a 'requirements.txt', + remote: ! 'Pipfile' or 'poetry.lock' package manager file in the root + remote: ! directory of its source code. remote: ! remote: ! Currently the root directory of your app contains: remote: ! diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb new file mode 100644 index 000000000..5aa092068 --- /dev/null +++ b/spec/hatchet/poetry_spec.rb @@ -0,0 +1,226 @@ +# frozen_string_literal: true + +require_relative '../spec_helper' + +RSpec.describe 'Poetry support' do + context 'with a poetry.lock' do + let(:buildpacks) { [:default, 'heroku-community/inline'] } + let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_basic', buildpacks:) } + + it 'installs successfully using Poetry and on rebuilds uses the cache' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Installing Poetry #{POETRY_VERSION} + remote: -----> Installing dependencies using 'poetry install --sync --only main' + remote: Installing dependencies from lock file + remote: + remote: Package operations: 1 install, 0 updates, 0 removals + remote: + remote: - Installing typing-extensions (4.12.2) + remote: -----> Inline app detected + remote: LANG=en_US.UTF-8 + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib + remote: LIBRARY_PATH=/app/.heroku/python/lib + remote: PATH=/app/.heroku/python/bin:/tmp/codon/tmp/cache/.heroku/python-poetry/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: POETRY_VIRTUALENVS_CREATE=false + remote: PYTHONHASHSEED=random + remote: PYTHONHOME=/app/.heroku/python + remote: PYTHONPATH=/app + remote: PYTHONUNBUFFERED=true + remote: + remote: ['', + remote: '/app', + remote: '/app/.heroku/python/lib/python312.zip', + remote: '/app/.heroku/python/lib/python3.12', + remote: '/app/.heroku/python/lib/python3.12/lib-dynload', + remote: '/app/.heroku/python/lib/python3.12/site-packages'] + remote: + remote: Skipping virtualenv creation, as specified in config file. + remote: typing-extensions 4.12.2 Backported and Experimental Type Hints for Python ... + remote: + remote: + OUTPUT + app.commit! + app.push! + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + remote: -----> Restoring cache + remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Using cached Poetry #{POETRY_VERSION} + remote: -----> Installing dependencies using 'poetry install --sync --only main' + remote: Installing dependencies from lock file + remote: + remote: No dependencies to install or update + remote: -----> Inline app detected + OUTPUT + end + end + end + + # TODO: Make this also test the Poetry version changing, the next (first) time we update Poetry, + # by using an older buildpack version for the initial build. + context 'when the requested Python version has changed since the last build' do + let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_basic') } + + it 'clears the cache before installing' do + app.deploy do |app| + File.write('.python-version', '3.13') + app.commit! + app.push! + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.13 specified in .python-version + remote: -----> Discarding cache since: + remote: - The Python version has changed from #{LATEST_PYTHON_3_12} to #{LATEST_PYTHON_3_13} + remote: -----> Installing Python #{LATEST_PYTHON_3_13} + remote: -----> Installing Poetry #{POETRY_VERSION} + remote: -----> Installing dependencies using 'poetry install --sync --only main' + remote: Installing dependencies from lock file + remote: + remote: Package operations: 1 install, 0 updates, 0 removals + remote: + remote: - Installing typing-extensions (4.12.2) + remote: -----> Discovering process types + OUTPUT + end + end + end + + context 'when the package manager has changed from pip to Poetry since the last build' do + let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_basic') } + + it 'clears the cache before installing with Poetry' do + app.deploy do |app| + FileUtils.rm('requirements.txt') + FileUtils.cp(FIXTURE_DIR.join('poetry_basic/pyproject.toml'), '.') + FileUtils.cp(FIXTURE_DIR.join('poetry_basic/poetry.lock'), '.') + app.commit! + app.push! + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + remote: -----> Discarding cache since: + remote: - The package manager has changed from pip to poetry + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Installing Poetry #{POETRY_VERSION} + remote: -----> Installing dependencies using 'poetry install --sync --only main' + remote: Installing dependencies from lock file + remote: + remote: Package operations: 1 install, 0 updates, 0 removals + remote: + remote: - Installing typing-extensions (4.12.2) + remote: -----> Discovering process types + OUTPUT + end + end + end + + context 'when poetry.lock contains editable requirements (both VCS and local package)' do + let(:buildpacks) { [:default, 'heroku-community/inline'] } + let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_editable', buildpacks:) } + + it 'rewrites .pth, .egg-link and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Running bin/post_compile hook + remote: __editable___gunicorn_20_1_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} + remote: poetry_editable.pth:/tmp/build_.+ + remote: + remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: -----> Inline app detected + remote: __editable___gunicorn_20_1_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} + remote: poetry_editable.pth:/tmp/build_.+ + remote: + remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + REGEX + + # Test rewritten paths work at runtime. + expect(app.run('bin/test-entrypoints.sh')).to include(<<~OUTPUT) + __editable___gunicorn_20_1_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} + __editable___local_package_pyproject_toml_0_0_1_finder.py:/app/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + __editable___local_package_setup_py_0_0_1_finder.py:/app/packages/local_package_setup_py/local_package_setup_py'} + poetry_editable.pth:/app + + Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! + Running entrypoint for the setup.py-based local package: Hello setup.py! + Running entrypoint for the VCS package: gunicorn (version 20.1.0) + OUTPUT + + # Test that the cached .pth files work correctly. + app.commit! + app.push! + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Running bin/post_compile hook + remote: __editable___gunicorn_20_1_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} + remote: poetry_editable.pth:/tmp/build_.+ + remote: + remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: -----> Inline app detected + remote: __editable___gunicorn_20_1_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} + remote: poetry_editable.pth:/tmp/build_.+ + remote: + remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + REGEX + end + end + end + + context 'when poetry.lock is out of sync with pyproject.toml' do + let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_lockfile_out_of_sync', allow_failure: true) } + + it 'fails the build' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Installing dependencies using 'poetry install --sync --only main' + remote: Installing dependencies from lock file + remote: + remote: pyproject.toml changed significantly since poetry.lock was last generated. Run `poetry lock [--no-update]` to fix the lock file. + remote: + remote: ! Error: Unable to install dependencies using Poetry. + remote: ! + remote: ! See the log output above for more information. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + # This case will be turned into an error in the future, but for now is required for backwards compatibility. + context 'when there is both a poetry.lock and a requirements.txt' do + let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_txt_and_poetry_lock') } + + it 'build using pip rather than Poetry' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + remote: -----> Installing Python #{LATEST_PYTHON_3_12} + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing SQLite3 + remote: -----> Installing requirements with pip + OUTPUT + end + end + end +end diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 4fddb3764..52860db92 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -31,6 +31,7 @@ def get_requirement_version(package_name) SETUPTOOLS_VERSION = get_requirement_version('setuptools') WHEEL_VERSION = get_requirement_version('wheel') PIPENV_VERSION = get_requirement_version('pipenv') +POETRY_VERSION = get_requirement_version('poetry') # Work around the return value for `default_buildpack` changing after deploy: # https://github.com/heroku/hatchet/issues/180 From 158def02c5b6e83b0c676e9997f781859b553ad9 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 6 Nov 2024 09:17:23 +0000 Subject: [PATCH 051/406] Prepare release v264 (#1686) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 012a9cc52..7c2f84b3f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v264] - 2024-11-06 + - Added support for the package manager Poetry. Apps must have a `pyproject.toml` + `poetry.lock` and no other package manager files (otherwise pip/Pipenv will take precedence for backwards compatibility). ([#1682](https://github.com/heroku/heroku-buildpack-python/pull/1682)) ## [v263] - 2024-10-31 @@ -1082,7 +1085,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v263...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v264...main +[v264]: https://github.com/heroku/heroku-buildpack-python/compare/v263...v264 [v263]: https://github.com/heroku/heroku-buildpack-python/compare/v262...v263 [v262]: https://github.com/heroku/heroku-buildpack-python/compare/v261...v262 [v261]: https://github.com/heroku/heroku-buildpack-python/compare/v260...v261 From b576e23d1e7e53d9c1996d77ffde756abfc0981a Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 6 Nov 2024 13:03:51 +0000 Subject: [PATCH 052/406] Fix using Poetry with outdated Python patch versions (#1687) The existing Poetry bootstrap process added in #1682 used the pip wheel bundled within the Python stdlib. This required use of pip's `--python` option, which was added to pip in v22.3 in 2022. All of the major Python versions we support have been updated to that pip version or newer, however, the older patch releases of some of those major Python versions can contain pip versions that are older (for example, whilst latest Python 3.9.x bundles pip v23.0.1, Python 3.9.0 bundles pip v20.2.1). Previously, using those older patch versions would result in: ``` -----> Installing Poetry 1.8.4 Usage: pip [options] no such option: --python ! Error: Unable to install Poetry. ``` Whilst we strongly recommend users upgrade to newer patch releases (since older versions are missing security updates and so likely insecure), we still want to support using Poetry on these versions, so the I've adjusted the bootstrap process to no longer use `--python` to prevent that error. --- CHANGELOG.md | 1 + bin/compile | 2 +- lib/poetry.sh | 25 +++++++--------- .../poetry_oldest_python/.python-version | 1 + .../fixtures/poetry_oldest_python/poetry.lock | 17 +++++++++++ .../poetry_oldest_python/pyproject.toml | 6 ++++ spec/hatchet/poetry_spec.rb | 29 +++++++++++++++++++ 7 files changed, 66 insertions(+), 15 deletions(-) create mode 100644 spec/fixtures/poetry_oldest_python/.python-version create mode 100644 spec/fixtures/poetry_oldest_python/poetry.lock create mode 100644 spec/fixtures/poetry_oldest_python/pyproject.toml diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c2f84b3f..4d4630981 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Fixed Poetry installation when using outdated patch versions of Python 3.8, 3.9 and 3.10, whose bundled pip doesn't support the `--python` option. ([#1687](https://github.com/heroku/heroku-buildpack-python/pull/1687)) ## [v264] - 2024-11-06 diff --git a/bin/compile b/bin/compile index 9ce9cb60f..da98a59a5 100755 --- a/bin/compile +++ b/bin/compile @@ -168,7 +168,7 @@ case "${package_manager}" in pipenv::install_pipenv ;; poetry) - poetry::install_poetry "${python_home}" "${CACHE_DIR}" "${EXPORT_PATH}" + poetry::install_poetry "${CACHE_DIR}" "${EXPORT_PATH}" ;; *) utils::abort_internal_error "Unhandled package manager: ${package_manager}" diff --git a/lib/poetry.sh b/lib/poetry.sh index bcceccd59..b21d7e583 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -7,9 +7,8 @@ set -euo pipefail POETRY_VERSION=$(utils::get_requirement_version 'poetry') function poetry::install_poetry() { - local python_home="${1}" - local cache_dir="${2}" - local export_file="${3}" + local cache_dir="${1}" + local export_file="${2}" # We store Poetry in the build cache, since we only need it during the build. local poetry_root="${cache_dir}/.heroku/python-poetry" @@ -39,19 +38,17 @@ function poetry::install_poetry() { # The Poetry directory will already exist in the relocated cache case mentioned above. rm -rf "${poetry_root}" - python -m venv --without-pip "${poetry_venv_dir}" - - # We use the pip wheel bundled within Python's standard library to install Poetry. - # Whilst Poetry does still require pip for some tasks (such as package uninstalls), - # it bundles its own copy for use as a fallback. As such we don't need to install pip - # into the Poetry venv (and in fact, Poetry wouldn't use this install anyway, since - # it only finds an external pip if it exists in the target venv). - local bundled_pip_module_path - bundled_pip_module_path="$(utils::bundled_pip_module_path "${python_home}")" + # We can't use the pip wheel bundled within Python's standard library to install Poetry + # (which would allow us to use `--without-pip` here to skip the pip install), since it + # requires using the `--python` option, which was only added in pip v22.3. And whilst + # all major Python versions we support now bundled a newer pip than that, some apps + # are still using outdated patch releases of those Python versions, whose bundled pip + # can be older (for example Python 3.9.0 ships with pip v20.2.1). Once Python 3.10 EOLs + # we can switch back to the previous approach since Python 3.11.0 ships with pip v22.3. + python -m venv "${poetry_venv_dir}" if ! { - python "${bundled_pip_module_path}" \ - --python "${poetry_venv_dir}" \ + "${poetry_venv_dir}/bin/pip" \ install \ --disable-pip-version-check \ --no-cache-dir \ diff --git a/spec/fixtures/poetry_oldest_python/.python-version b/spec/fixtures/poetry_oldest_python/.python-version new file mode 100644 index 000000000..a5c4c7633 --- /dev/null +++ b/spec/fixtures/poetry_oldest_python/.python-version @@ -0,0 +1 @@ +3.9.0 diff --git a/spec/fixtures/poetry_oldest_python/poetry.lock b/spec/fixtures/poetry_oldest_python/poetry.lock new file mode 100644 index 000000000..5f96ba1c8 --- /dev/null +++ b/spec/fixtures/poetry_oldest_python/poetry.lock @@ -0,0 +1,17 @@ +# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand. + +[[package]] +name = "typing-extensions" +version = "4.12.2" +description = "Backported and Experimental Type Hints for Python 3.8+" +optional = false +python-versions = ">=3.8" +files = [ + {file = "typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d"}, + {file = "typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"}, +] + +[metadata] +lock-version = "2.0" +python-versions = "^3.9" +content-hash = "3b8ea94b618b2c3590bcbb224f2e2c723e6c14c8da90ba6917ca4a4fdcb716cf" diff --git a/spec/fixtures/poetry_oldest_python/pyproject.toml b/spec/fixtures/poetry_oldest_python/pyproject.toml new file mode 100644 index 000000000..c69fd9272 --- /dev/null +++ b/spec/fixtures/poetry_oldest_python/pyproject.toml @@ -0,0 +1,6 @@ +[tool.poetry] +package-mode = false + +[tool.poetry.dependencies] +python = "^3.9" +typing-extensions = "*" diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 5aa092068..0476ae811 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -185,6 +185,35 @@ end end + # This checks that the Poetry bootstrap works even with older bundled pip, and that + # our chosen Poetry version also supports our oldest supported Python version. + context 'when using the oldest supported Python version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_oldest_python') } + + it 'installs successfully' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.9.0 specified in .python-version + remote: -----> Installing Python 3.9.0 + remote: + remote: ! Warning: A Python security update is available! + remote: ! + remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} + remote: ! See: https://devcenter.heroku.com/articles/python-runtimes + remote: + remote: -----> Installing Poetry #{POETRY_VERSION} + remote: -----> Installing dependencies using 'poetry install --sync --only main' + remote: Installing dependencies from lock file + remote: + remote: Package operations: 1 install, 0 updates, 0 removals + remote: + remote: - Installing typing-extensions (4.12.2) + OUTPUT + end + end + end + context 'when poetry.lock is out of sync with pyproject.toml' do let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_lockfile_out_of_sync', allow_failure: true) } From eed2dc7ddbe131602da750617983826ad6b22974 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 6 Nov 2024 13:14:25 +0000 Subject: [PATCH 053/406] Prepare release v265 (#1688) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d4630981..9fdb97b82 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v265] - 2024-11-06 + - Fixed Poetry installation when using outdated patch versions of Python 3.8, 3.9 and 3.10, whose bundled pip doesn't support the `--python` option. ([#1687](https://github.com/heroku/heroku-buildpack-python/pull/1687)) ## [v264] - 2024-11-06 @@ -1086,7 +1089,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v264...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v265...main +[v265]: https://github.com/heroku/heroku-buildpack-python/compare/v264...v265 [v264]: https://github.com/heroku/heroku-buildpack-python/compare/v263...v264 [v263]: https://github.com/heroku/heroku-buildpack-python/compare/v262...v263 [v262]: https://github.com/heroku/heroku-buildpack-python/compare/v261...v262 From 1f6857ebddf87d1a26124cc39de448546dcb973c Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 8 Nov 2024 13:33:40 +0000 Subject: [PATCH 054/406] Log output and error handling improvements (#1689) This backports a number of the package manager log output and error handling improvements from the new Poetry implementation to the existing pip and Pipenv implementations, along with a few other misc improvements to metrics and error handling. GUS-W-8059892. GUS-W-17166168. --- CHANGELOG.md | 3 ++ bin/compile | 4 ++ bin/steps/collectstatic | 2 +- bin/utils | 7 --- lib/package_manager.sh | 4 +- lib/pip.sh | 81 +++++++++++++++++++++-------- lib/pipenv.sh | 73 +++++++++++++++++++++----- lib/poetry.sh | 27 +++++++--- lib/python_version.sh | 16 +++--- lib/utils.sh | 4 +- spec/hatchet/ci_spec.rb | 16 +++--- spec/hatchet/hooks_spec.rb | 2 +- spec/hatchet/pip_spec.rb | 26 ++++++--- spec/hatchet/pipenv_spec.rb | 27 ++++++---- spec/hatchet/poetry_spec.rb | 4 +- spec/hatchet/python_version_spec.rb | 10 ++-- spec/hatchet/stack_spec.rb | 4 +- 17 files changed, 210 insertions(+), 100 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9fdb97b82..05734c0b1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] +- Updated the build log message shown when installing dependencies to include the package manager command being run. ([#1689](https://github.com/heroku/heroku-buildpack-python/pull/1689)) +- Improved the error messages and buildpack metrics for package manager related failures. ([#1689](https://github.com/heroku/heroku-buildpack-python/pull/1689)) +- Changed test dependency installation on Heroku CI to now install `requirements.txt` and `requirements-test.txt` in a single `pip install` invocation rather than separately. This allows pip's resolver to resolve any version conflicts between the two files. ([#1689](https://github.com/heroku/heroku-buildpack-python/pull/1689)) ## [v265] - 2024-11-06 diff --git a/bin/compile b/bin/compile index da98a59a5..2b64a4aa1 100755 --- a/bin/compile +++ b/bin/compile @@ -78,6 +78,10 @@ GUNICORN_PROFILE_PATH="$BUILD_DIR/.profile.d/python.gunicorn.sh" WEB_CONCURRENCY_PROFILE_PATH="$BUILD_DIR/.profile.d/WEB_CONCURRENCY.sh" python_home='/app/.heroku/python' +# NB: Python must be added to PATH using the symlinked `/app` location and not its actual location +# in BUILD_DIR, so that Python reports its location (via `sys.prefix`, `sys.executable` and others) +# using `/app` paths which will still work at run-time after relocation. Amongst other things, this +# ensures that the shebang lines in the entrypoint scripts of installed packages are correct. export PATH="/app/.heroku/python/bin:${PATH}" # Tell Python to not buffer it's stdin/stdout. export PYTHONUNBUFFERED=1 diff --git a/bin/steps/collectstatic b/bin/steps/collectstatic index f4b621af5..bcd0ef4aa 100755 --- a/bin/steps/collectstatic +++ b/bin/steps/collectstatic @@ -64,7 +64,7 @@ export PYTHONPATH COLLECTSTATIC_LOG=$(mktemp) set +e -python "$MANAGE_FILE" collectstatic --noinput --traceback 2>&1 | tee "$COLLECTSTATIC_LOG" | sed '/^Post-processed/d;/^Copying/d;/^$/d' | output::indent +python "$MANAGE_FILE" collectstatic --noinput --traceback 2>&1 | tee "$COLLECTSTATIC_LOG" | sed --unbuffered '/^Post-processed/d;/^Copying/d;/^$/d' | output::indent COLLECTSTATIC_STATUS="${PIPESTATUS[0]}" set -e diff --git a/bin/utils b/bin/utils index 40f16a29b..9e662191a 100755 --- a/bin/utils +++ b/bin/utils @@ -8,13 +8,6 @@ shopt -s nullglob source "${BUILDPACK_DIR:?}/vendor/buildpack-stdlib_v8.sh" -sed() { command sed -u "$@"; } - -# Clean up pip output -cleanup() { - sed -e 's/\.\.\.\+/.../g' | sed -e '/already satisfied/Id' | sed -e '/No files were found to uninstall/Id' | sed -e '/Overwriting/Id' | sed -e '/python executable/Id' | sed -e '/no previously-included files/Id' -} - # Does some serious copying. deep-cp() { declare source="$1" target="$2" diff --git a/lib/package_manager.sh b/lib/package_manager.sh index 48fa73d1c..c4c9350ff 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -77,8 +77,8 @@ function package_manager::determine_package_manager() { https://devcenter.heroku.com/articles/getting-started-with-python https://devcenter.heroku.com/articles/python-support EOF - meta_set "failure_reason" "package-manager-not-found" - return 1 + meta_set "failure_reason" "package-manager::none-found" + exit 1 ;; *) # If multiple package managers are found, use the first found above. diff --git a/lib/pip.sh b/lib/pip.sh index 1882f7f75..3b26250f6 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -46,13 +46,30 @@ function pip::install_pip_setuptools_wheel() { # app's requirements.txt in the last build). The install will be a no-op if the versions match. output::step "Installing ${packages_display_text}" - /app/.heroku/python/bin/python "${bundled_pip_module_path}" install --quiet --disable-pip-version-check --no-cache-dir \ - "${packages_to_install[@]}" + if ! { + python "${bundled_pip_module_path}" \ + install \ + --disable-pip-version-check \ + --no-cache-dir \ + --no-input \ + --quiet \ + "${packages_to_install[@]}" + }; then + output::error <<-EOF + Error: Unable to install pip. + + Try building again to see if the error resolves itself. + + If that does not help, check the status of PyPI (the Python + package repository service), here: + https://status.python.org + EOF + meta_set "failure_reason" "install-package-manager::pip" + exit 1 + fi } function pip::install_dependencies() { - output::step "Installing requirements with pip" - # Make select pip config vars set on the Heroku app available to pip. # TODO: Expose all config vars (after suitable checks are added for unsafe env vars) # to allow for the env var interpolation feature of requirements files to work. @@ -64,31 +81,49 @@ function pip::install_dependencies() { export PIP_EXTRA_INDEX_URL fi + local pip_install_command=( + pip + install + ) + # TODO: Deprecate/sunset this missing requirements file fallback. if [[ -f setup.py && ! -f requirements.txt ]]; then - args=(--editable .) + pip_install_command+=(--editable .) else - args=(-r requirements.txt) + pip_install_command+=(-r requirements.txt) fi - set +e - # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. - /app/.heroku/python/bin/pip install "${args[@]}" --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir --progress-bar off 2>&1 | tee "${WARNINGS_LOG}" | cleanup | output::indent - local PIP_STATUS="${PIPESTATUS[0]}" - set -e - - show-warnings - - if [[ ! ${PIP_STATUS} -eq 0 ]]; then - meta_set "failure_reason" "pip-install" - return 1 + # Install test dependencies too when the buildpack is invoked via `bin/test-compile` on Heroku CI. + # We install both requirements files at the same time to allow pip to resolve version conflicts. + if [[ -v INSTALL_TEST && -f requirements-test.txt ]]; then + pip_install_command+=(-r requirements-test.txt) fi - # Install test dependencies, for Heroku CI. - if [[ "${INSTALL_TEST:-0}" == "1" ]]; then - if [[ -f requirements-test.txt ]]; then - output::step "Installing test dependencies..." - /app/.heroku/python/bin/pip install -r requirements-test.txt --exists-action=w --src='/app/.heroku/src' --disable-pip-version-check --no-cache-dir 2>&1 | cleanup | output::indent - fi + # We only display the most relevant command args here, to improve the signal to noise ratio. + output::step "Installing dependencies using '${pip_install_command[*]}'" + + # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. + if ! { + "${pip_install_command[@]}" \ + --disable-pip-version-check \ + --exists-action=w \ + --no-cache-dir \ + --no-input \ + --progress-bar off \ + --src='/app/.heroku/src' \ + |& tee "${WARNINGS_LOG:?}" \ + |& sed --unbuffered --expression '/Requirement already satisfied/d' \ + |& output::indent + }; then + # TODO: Overhaul warnings and combine them with error handling. + show-warnings + + output::error <<-EOF + Error: Unable to install dependencies using pip. + + See the log output above for more information. + EOF + meta_set "failure_reason" "install-dependencies::pip" + exit 1 fi } diff --git a/lib/pipenv.sh b/lib/pipenv.sh index 321b0a483..34519eb1b 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -18,14 +18,34 @@ function pipenv::install_pipenv() { # TODO: Install Pipenv into a venv so it isn't leaked into the app environment. # TODO: Skip installing Pipenv if its version hasn't changed (once it's installed into a venv). # TODO: Explore viability of making Pipenv only be available during the build, to reduce slug size. - /app/.heroku/python/bin/pip install --quiet --disable-pip-version-check --no-cache-dir "pipenv==${PIPENV_VERSION}" + if ! { + pip \ + install \ + --disable-pip-version-check \ + --no-cache-dir \ + --no-input \ + --quiet \ + "pipenv==${PIPENV_VERSION}" + }; then + output::error <<-EOF + Error: Unable to install Pipenv. + + Try building again to see if the error resolves itself. + + If that does not help, check the status of PyPI (the Python + package repository service), here: + https://status.python.org + EOF + meta_set "failure_reason" "install-package-manager::pipenv" + exit 1 + fi } # Previous versions of the buildpack used to cache the checksum of the lockfile to allow # for skipping pipenv install if the lockfile was unchanged. However, this is not always safe # to do (the lockfile can refer to dependencies that can change independently of the lockfile, # for example, when using a local non-editable file dependency), so we no longer ever skip -# install, and instead defer to pipenv to determine whether install is actually a no-op. +# install, and instead defer to Pipenv to determine whether install is actually a no-op. function pipenv::install_dependencies() { # Make select pip config vars set on the Heroku app available to the pip used by Pipenv. # TODO: Expose all config vars (after suitable checks are added for unsafe env vars). @@ -37,19 +57,44 @@ function pipenv::install_dependencies() { export PIP_EXTRA_INDEX_URL fi - # Install the test dependencies, for CI. - # TODO: This is currently inconsistent with the non-test path, since it assumes (but doesn't check for) a lockfile. - if [[ "${INSTALL_TEST:-0}" == "1" ]]; then - output::step "Installing test dependencies with Pipenv" - /app/.heroku/python/bin/pipenv install --dev --system --deploy --extra-pip-args='--src=/app/.heroku/src' 2>&1 | cleanup | output::indent - - # Install the dependencies. - elif [[ ! -f Pipfile.lock ]]; then - output::step "Installing dependencies with Pipenv" - /app/.heroku/python/bin/pipenv install --system --skip-lock --extra-pip-args='--src=/app/.heroku/src' 2>&1 | output::indent + # Note: We can't use `pipenv sync` since it doesn't validate that the lockfile is up to date. + local pipenv_install_command=( + pipenv + install + ) + # TODO: Make Pipfile.lock mandatory during package manager selection. + if [[ ! -f Pipfile.lock ]]; then + pipenv_install_command+=(--skip-lock) else - output::step "Installing dependencies with Pipenv" - /app/.heroku/python/bin/pipenv install --system --deploy --extra-pip-args='--src=/app/.heroku/src' 2>&1 | output::indent + pipenv_install_command+=(--deploy) + fi + + # Install test dependencies too when the buildpack is invoked via `bin/test-compile` on Heroku CI. + if [[ -v INSTALL_TEST ]]; then + pipenv_install_command+=(--dev) + fi + + # We only display the most relevant command args here, to improve the signal to noise ratio. + output::step "Installing dependencies using '${pipenv_install_command[*]}'" + + # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. + if ! { + "${pipenv_install_command[@]}" \ + --extra-pip-args='--src=/app/.heroku/src' \ + --system \ + |& tee "${WARNINGS_LOG:?}" \ + |& output::indent + }; then + # TODO: Overhaul warnings and combine them with error handling. + show-warnings + + output::error <<-EOF + Error: Unable to install dependencies using Pipenv. + + See the log output above for more information. + EOF + meta_set "failure_reason" "install-dependencies::pipenv" + exit 1 fi } diff --git a/lib/poetry.sh b/lib/poetry.sh index b21d7e583..95b54ff66 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -45,7 +45,18 @@ function poetry::install_poetry() { # are still using outdated patch releases of those Python versions, whose bundled pip # can be older (for example Python 3.9.0 ships with pip v20.2.1). Once Python 3.10 EOLs # we can switch back to the previous approach since Python 3.11.0 ships with pip v22.3. - python -m venv "${poetry_venv_dir}" + if ! python -m venv "${poetry_venv_dir}"; then + output::error <<-EOF + Internal Error: Unable to create virtual environment for Poetry. + + The 'python -m venv' command to create a virtual environment did + not exit successfully. + + See the log output above for more information. + EOF + meta_set "failure_reason" "create-venv::poetry" + exit 1 + fi if ! { "${poetry_venv_dir}/bin/pip" \ @@ -65,8 +76,8 @@ function poetry::install_poetry() { package repository service), here: https://status.python.org EOF - meta_set "failure_reason" "install-poetry" - return 1 + meta_set "failure_reason" "install-package-manager::poetry" + exit 1 fi mkdir -p "${poetry_bin_dir}" @@ -112,11 +123,15 @@ function poetry::install_dependencies() { # codes for redrawing lines, which renders badly in persisted build logs. # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { - "${poetry_install_command[@]}" --compile --no-ansi --no-interaction \ + "${poetry_install_command[@]}" \ + --compile \ + --no-ansi \ + --no-interaction \ |& tee "${WARNINGS_LOG:?}" \ - |& grep --invert-match 'Skipping virtualenv creation' \ + |& sed --unbuffered --expression '/Skipping virtualenv creation/d' \ |& output::indent }; then + # TODO: Overhaul warnings and combine them with error handling. show-warnings output::error <<-EOF @@ -125,6 +140,6 @@ function poetry::install_dependencies() { See the log output above for more information. EOF meta_set "failure_reason" "install-dependencies::poetry" - return 1 + exit 1 fi } diff --git a/lib/python_version.sh b/lib/python_version.sh index d08fed160..de0dbca2a 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -115,7 +115,7 @@ function python_version::parse_runtime_txt() { python-${DEFAULT_PYTHON_MAJOR_VERSION} EOF meta_set "failure_reason" "runtime-txt::invalid-version" - return 1 + exit 1 fi } @@ -160,7 +160,7 @@ function python_version::parse_python_version_file() { ${DEFAULT_PYTHON_MAJOR_VERSION} EOF meta_set "failure_reason" "python-version-file::invalid-version" - return 1 + exit 1 fi ;; 0) @@ -176,7 +176,7 @@ function python_version::parse_python_version_file() { begin with a '#', otherwise it will be treated as a comment. EOF meta_set "failure_reason" "python-version-file::no-version" - return 1 + exit 1 ;; *) output::error <<-EOF @@ -196,7 +196,7 @@ function python_version::parse_python_version_file() { those lines with '#'. EOF meta_set "failure_reason" "python-version-file::multiple-versions" - return 1 + exit 1 ;; esac } @@ -228,7 +228,7 @@ function python_version::read_pipenv_python_version() { Run 'pipenv lock' to regenerate/fix the lockfile. EOF meta_set "failure_reason" "pipfile-lock::invalid-json" - return 1 + exit 1 fi # Neither of the optional fields were found. @@ -262,7 +262,7 @@ function python_version::read_pipenv_python_version() { https://pipenv.pypa.io/en/latest/specifiers.html#specifying-versions-of-python EOF meta_set "failure_reason" "pipfile-lock::invalid-version" - return 1 + exit 1 fi } @@ -319,7 +319,7 @@ function python_version::resolve_python_version() { EOF fi meta_set "failure_reason" "python-version::eol" - return 1 + exit 1 fi if (((major == 3 && minor > 13) || major >= 4)); then @@ -358,7 +358,7 @@ function python_version::resolve_python_version() { EOF fi meta_set "failure_reason" "python-version::unknown-major" - return 1 + exit 1 fi # If an exact Python version was requested, there's nothing to resolve. diff --git a/lib/utils.sh b/lib/utils.sh index c855ade3e..61254de5c 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -31,10 +31,10 @@ function utils::bundled_pip_module_path() { if [[ -z "${bundled_pip_wheel}" ]]; then output::error <<-'EOF' - Error: Failed to locate the bundled pip wheel. + Internal Error: Unable to locate the ensurepip pip wheel file. EOF meta_set "failure_reason" "bundled-pip-not-found" - return 1 + exit 1 fi echo "${bundled_pip_wheel}/pip" diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 06594b0a1..661cd36b0 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -16,12 +16,9 @@ -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing SQLite3 - -----> Installing requirements with pip - .* - Successfully installed typing-extensions-4.12.2 - -----> Installing test dependencies... - .* - Successfully installed .* pytest-8.3.3 + -----> Installing dependencies using 'pip install -r requirements.txt -r requirements-test.txt' + .+ + Successfully installed .+ pytest-8.3.3 typing-extensions-4.12.2 -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running bin/post_compile hook CI=true @@ -73,8 +70,7 @@ -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing SQLite3 - -----> Installing requirements with pip - -----> Installing test dependencies... + -----> Installing dependencies using 'pip install -r requirements.txt -r requirements-test.txt' -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running bin/post_compile hook OUTPUT @@ -94,7 +90,7 @@ -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing Pipenv #{PIPENV_VERSION} -----> Installing SQLite3 - -----> Installing test dependencies with Pipenv + -----> Installing dependencies using 'pipenv install --deploy --dev' Installing dependencies from Pipfile.lock \\(.+\\)... Installing dependencies from Pipfile.lock \\(.+\\)... -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. @@ -149,7 +145,7 @@ -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing Pipenv #{PIPENV_VERSION} -----> Installing SQLite3 - -----> Installing test dependencies with Pipenv + -----> Installing dependencies using 'pipenv install --deploy --dev' Installing dependencies from Pipfile.lock \\(.+\\)... Installing dependencies from Pipfile.lock \\(.+\\)... -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. diff --git a/spec/hatchet/hooks_spec.rb b/spec/hatchet/hooks_spec.rb index 8ea91859a..1f6af134e 100644 --- a/spec/hatchet/hooks_spec.rb +++ b/spec/hatchet/hooks_spec.rb @@ -35,7 +35,7 @@ OUTPUT expect(output).to include(<<~OUTPUT) - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: -----> Running bin/post_compile hook remote: ~ post_compile ran with env vars: remote: BUILD_DIR=/tmp/build_ diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index dcf9bb762..3f3305366 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -5,7 +5,7 @@ RSpec.shared_examples 'installs successfully using pip' do it 'installs successfully using pip' do app.deploy do |app| - expect(app.output).to include('Installing requirements with pip') + expect(app.output).to include("Installing dependencies using 'pip install -r requirements.txt'") expect(app.output).to include('Successfully installed') end end @@ -24,7 +24,7 @@ remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 5)) remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) remote: Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) @@ -65,7 +65,7 @@ remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: -----> Inline app detected OUTPUT end @@ -90,7 +90,7 @@ remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 5)) remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) remote: Collecting six (from -r requirements.txt (line 6)) @@ -122,7 +122,7 @@ remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 5)) remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) remote: Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) @@ -221,7 +221,7 @@ remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install --editable .' remote: Obtaining file:///tmp/build_.* remote: Preparing metadata \\(setup.py\\): started remote: Preparing metadata \\(setup.py\\): finished with status 'done' @@ -239,7 +239,7 @@ it 'installs packages only from requirements.txt' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting urllib3 (from -r requirements.txt (line 1)) OUTPUT expect(app.output).not_to include('Running setup.py develop') @@ -253,8 +253,13 @@ it 'aborts the build and displays the pip error' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: ERROR: Invalid requirement: 'an-invalid-requirement!' (from line 1 of requirements.txt) + remote: + remote: ! Error: Unable to install dependencies using pip. + remote: ! + remote: ! See the log output above for more information. + remote: remote: ! Push rejected, failed to compile Python app. OUTPUT end @@ -274,6 +279,11 @@ remote: ! For GDAL, GEOS and PROJ support, use the Geo buildpack alongside the Python buildpack: remote: ! https://github.com/heroku/heroku-geo-buildpack remote: + remote: + remote: ! Error: Unable to install dependencies using pip. + remote: ! + remote: ! See the log output above for more information. + remote: remote: ! Push rejected, failed to compile Python app. OUTPUT end diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 57c01614d..befe00b72 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -17,7 +17,7 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing dependencies with Pipenv + remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Inline app detected remote: LANG=en_US.UTF-8 @@ -61,7 +61,7 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing dependencies with Pipenv + remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Inline app detected REGEX @@ -91,7 +91,7 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing dependencies with Pipenv + remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... REGEX end @@ -109,7 +109,7 @@ remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing dependencies with Pipenv + remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... REGEX end @@ -129,7 +129,7 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing dependencies with Pipenv + remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... REGEX end @@ -154,7 +154,7 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing dependencies with Pipenv + remote: -----> Installing dependencies using 'pipenv install --skip-lock' remote: The flag --skip-lock has been reintroduced \\(but is not recommended\\). Without remote: the lock resolver it is difficult to manage multiple package indexes, and hash remote: checking is not provided. However it can help manage installs with current @@ -303,7 +303,7 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing dependencies with Pipenv + remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Discovering process types REGEX @@ -331,7 +331,7 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing dependencies with Pipenv + remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Discovering process types REGEX @@ -351,13 +351,15 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing dependencies with Pipenv + remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... REGEX end end end + # This tests not only our handling of failing dependency installation, but also that we're running + # Pipenv in such a way that it errors if the lockfile is out of sync, rather than simply updating it. context 'when the Pipfile.lock is out of sync with Pipfile' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_lockfile_out_of_sync', allow_failure: true) } @@ -371,10 +373,15 @@ remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing dependencies with Pipenv + remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Your Pipfile.lock \\(.+\\) is out of date. Expected: \\(.+\\). remote: .+ remote: ERROR:: Aborting deploy + remote: + remote: ! Error: Unable to install dependencies using Pipenv. + remote: ! + remote: ! See the log output above for more information. + remote: remote: ! Push rejected, failed to compile Python app. REGEX end diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 0476ae811..69ddfc2e1 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -214,6 +214,8 @@ end end + # This tests not only our handling of failing dependency installation, but also that we're running + # Poetry in such a way that it errors if the lockfile is out of sync, rather than simply updating it. context 'when poetry.lock is out of sync with pyproject.toml' do let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_lockfile_out_of_sync', allow_failure: true) } @@ -247,7 +249,7 @@ remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' OUTPUT end end diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 26318061f..d566bc752 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -11,7 +11,7 @@ remote: -----> Using Python #{requested_version} specified in .python-version remote: -----> Installing Python #{resolved_version} remote: -----> Installing pip #{PIP_VERSION} - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT else @@ -21,7 +21,7 @@ remote: -----> Installing Python #{resolved_version} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT end @@ -105,7 +105,7 @@ remote: remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT expect(app.run('python -V')).to include("Python #{LATEST_PYTHON_3_8}") @@ -403,7 +403,7 @@ remote: -----> Using Python 3.13 specified in runtime.txt remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing pip #{PIP_VERSION} - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' OUTPUT end end @@ -424,7 +424,7 @@ remote: - The Python version has changed from #{LATEST_PYTHON_3_9} to #{LATEST_PYTHON_3_13} remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing pip #{PIP_VERSION} - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT end diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index 097eb92dc..c30a09b88 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -35,7 +35,7 @@ remote: remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT end @@ -60,7 +60,7 @@ remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 - remote: -----> Installing requirements with pip + remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT end From 0cd8efc50ef603c5c6e83fc8a17d9d447c40fea1 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 8 Nov 2024 17:58:46 +0000 Subject: [PATCH 055/406] Add a warning when multiple package managers are found (#1692) Currently the buildpack will use whichever package manager it finds first, if the files of multiple package managers are found. This occasionally results in support tickets where the user believes the build to not be installing dependencies correctly, when in fact they are adding dependencies to the wrong package manager file. As such, we want to make the presence of multiple package manager files an error, but first we should add a warning so it's not a surprise. (Plus with the recent Poetry support addition, there will be apps using the third party Poetry buildpack that have both a `poetry.lock` and the generated `requirements.txt` until they remove the third-party buildpack.) Towards #1691. GUS-W-17167927. --- CHANGELOG.md | 1 + lib/output.sh | 19 +++++++++++++++++++ lib/package_manager.sh | 32 ++++++++++++++++++++++++++++++++ spec/hatchet/pipenv_spec.rb | 16 ++++++++++++++++ spec/hatchet/poetry_spec.rb | 22 ++++++++++++++++++++++ 5 files changed, 90 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 05734c0b1..ffccb62c6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Added a warning when the files for multiple package managers are found. In the future this warning will become an error. ([#1692](https://github.com/heroku/heroku-buildpack-python/pull/1692)) - Updated the build log message shown when installing dependencies to include the package manager command being run. ([#1689](https://github.com/heroku/heroku-buildpack-python/pull/1689)) - Improved the error messages and buildpack metrics for package manager related failures. ([#1689](https://github.com/heroku/heroku-buildpack-python/pull/1689)) - Changed test dependency installation on Heroku CI to now install `requirements.txt` and `requirements-test.txt` in a single `pip install` invocation rather than separately. This allows pip's resolver to resolve any version conflicts between the two files. ([#1689](https://github.com/heroku/heroku-buildpack-python/pull/1689)) diff --git a/lib/output.sh b/lib/output.sh index a94074a1c..6893a5319 100644 --- a/lib/output.sh +++ b/lib/output.sh @@ -4,6 +4,7 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail +ANSI_BLUE='\033[1;34m' ANSI_RED='\033[1;31m' ANSI_YELLOW='\033[1;33m' ANSI_RESET='\033[0m' @@ -28,6 +29,24 @@ function output::indent() { sed --unbuffered "s/^/ /" } +# Output a styled multi-line notice message to stderr. +# +# Usage: +# ``` +# output::notice <<-EOF +# Note: The note summary. +# +# Detailed description. +# EOF +# ``` +function output::notice() { + echo >&2 + while IFS= read -r line; do + echo -e "${ANSI_BLUE} ! ${line}${ANSI_RESET}" >&2 + done + echo >&2 +} + # Output a styled multi-line warning message to stderr. # # Usage: diff --git a/lib/package_manager.sh b/lib/package_manager.sh index c4c9350ff..c140542af 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -7,9 +7,11 @@ set -euo pipefail function package_manager::determine_package_manager() { local build_dir="${1}" local package_managers_found=() + local package_managers_found_display_text=() if [[ -f "${build_dir}/Pipfile.lock" ]]; then package_managers_found+=(pipenv) + package_managers_found_display_text+=("Pipfile.lock (Pipenv)") meta_set "pipenv_has_lockfile" "true" elif [[ -f "${build_dir}/Pipfile" ]]; then # TODO: Start requiring a Pipfile.lock and make this branch a "missing lockfile" error instead. @@ -20,11 +22,13 @@ function package_manager::determine_package_manager() { We recommend you commit this into your repository. EOF package_managers_found+=(pipenv) + package_managers_found_display_text+=("Pipfile (Pipenv)") meta_set "pipenv_has_lockfile" "false" fi if [[ -f "${build_dir}/requirements.txt" ]]; then package_managers_found+=(pip) + package_managers_found_display_text+=("requirements.txt (pip)") fi # This must be after the requirements.txt check, so that the requirements.txt exported by @@ -34,12 +38,14 @@ function package_manager::determine_package_manager() { # ordering here won't matter. if [[ -f "${build_dir}/poetry.lock" ]]; then package_managers_found+=(poetry) + package_managers_found_display_text+=("poetry.lock (Poetry)") fi # TODO: Deprecate/sunset this fallback, since using setup.py declared dependencies is # not a best practice, and we can only guess as to which package manager to use. if ((${#package_managers_found[@]} == 0)) && [[ -f "${build_dir}/setup.py" ]]; then package_managers_found+=(pip) + package_managers_found_display_text+=("setup.py (pip)") meta_set "setup_py_only" "true" else meta_set "setup_py_only" "false" @@ -87,6 +93,32 @@ function package_manager::determine_package_manager() { # aren't taking effect. (We'll need to wait until after Poetry support has landed, # and people have had a chance to migrate from the Poetry buildpack mentioned above.) echo "${package_managers_found[0]}" + + output::warning <<-EOF + Warning: Multiple Python package manager files were found. + + Exactly one package manager file should be present in your app's + source code, however, several were found: + + $(printf -- "%s\n" "${package_managers_found_display_text[@]}") + + For now, we will build your app using the first package manager + listed above, however, in the future this warning will become + an error. + + Decide which package manager you want to use with your app, and + then delete the file(s) and any config from the others. + EOF + + if [[ "${package_managers_found[*]}" == *"poetry"* ]]; then + output::notice <<-EOF + Note: We recently added support for the package manager Poetry. + If you are using a third-party Poetry buildpack you must remove + it, otherwise the requirements.txt file it generates will cause + the warning above. + EOF + fi + meta_set "package_manager_multiple_found" "$( IFS=, echo "${package_managers_found[*]}" diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index befe00b72..494add751 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -346,6 +346,22 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected + remote: + remote: ! Warning: Multiple Python package manager files were found. + remote: ! + remote: ! Exactly one package manager file should be present in your app's + remote: ! source code, however, several were found: + remote: ! + remote: ! Pipfile.lock \\(Pipenv\\) + remote: ! requirements.txt \\(pip\\) + remote: ! + remote: ! For now, we will build your app using the first package manager + remote: ! listed above, however, in the future this warning will become + remote: ! an error. + remote: ! + remote: ! Decide which package manager you want to use with your app, and + remote: ! then delete the file\\(s\\) and any config from the others. + remote: remote: -----> Using Python 3.12 specified in Pipfile.lock remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 69ddfc2e1..ae875027c 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -245,6 +245,28 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected + remote: + remote: ! Warning: Multiple Python package manager files were found. + remote: ! + remote: ! Exactly one package manager file should be present in your app's + remote: ! source code, however, several were found: + remote: ! + remote: ! requirements.txt (pip) + remote: ! poetry.lock (Poetry) + remote: ! + remote: ! For now, we will build your app using the first package manager + remote: ! listed above, however, in the future this warning will become + remote: ! an error. + remote: ! + remote: ! Decide which package manager you want to use with your app, and + remote: ! then delete the file(s) and any config from the others. + remote: + remote: + remote: ! Note: We recently added support for the package manager Poetry. + remote: ! If you are using a third-party Poetry buildpack you must remove + remote: ! it, otherwise the requirements.txt file it generates will cause + remote: ! the warning above. + remote: remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} From c496baa74e14103ba72c7ac2d35121e1a72e01d7 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 8 Nov 2024 19:15:28 +0000 Subject: [PATCH 056/406] Improve NLTK downloader log output and error messages (#1690) Improves the build log output, error messages and buildpack failure metrics for the NLTK downloader feature, and adds a test for the app having an invalid `nltk.txt` (since previously only the successful cases were tested). This should be the last failure type seen in the wild that doesn't emit a `failure_reason` metric. GUS-W-8059892. --- CHANGELOG.md | 3 +- bin/steps/nltk | 24 +++++++--- spec/fixtures/nltk_txt_invalid/nltk.txt | 1 + .../nltk_txt_invalid/requirements.txt | 1 + spec/hatchet/nltk_spec.rb | 46 +++++++++++++++---- 5 files changed, 60 insertions(+), 15 deletions(-) create mode 100644 spec/fixtures/nltk_txt_invalid/nltk.txt create mode 100644 spec/fixtures/nltk_txt_invalid/requirements.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index ffccb62c6..6ad5059c4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,8 +4,9 @@ - Added a warning when the files for multiple package managers are found. In the future this warning will become an error. ([#1692](https://github.com/heroku/heroku-buildpack-python/pull/1692)) - Updated the build log message shown when installing dependencies to include the package manager command being run. ([#1689](https://github.com/heroku/heroku-buildpack-python/pull/1689)) -- Improved the error messages and buildpack metrics for package manager related failures. ([#1689](https://github.com/heroku/heroku-buildpack-python/pull/1689)) - Changed test dependency installation on Heroku CI to now install `requirements.txt` and `requirements-test.txt` in a single `pip install` invocation rather than separately. This allows pip's resolver to resolve any version conflicts between the two files. ([#1689](https://github.com/heroku/heroku-buildpack-python/pull/1689)) +- Improved the error messages and buildpack metrics for package manager related failures. ([#1689](https://github.com/heroku/heroku-buildpack-python/pull/1689)) +- Improved the build log output, error messages and buildpack failure metrics for the NLTK downloader feature. ([#1690](https://github.com/heroku/heroku-buildpack-python/pull/1690)) ## [v265] - 2024-11-06 diff --git a/bin/steps/nltk b/bin/steps/nltk index 804e7a41d..ad99493fe 100755 --- a/bin/steps/nltk +++ b/bin/steps/nltk @@ -1,5 +1,4 @@ #!/usr/bin/env bash -# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. # This script is run in a subshell via sub_env so doesn't inherit the options/vars/utils from `bin/compile`. # TODO: Stop running this script in a subshell. @@ -21,17 +20,30 @@ EXPORT_PATH="${BUILDPACK_DIR}/export" if is_module_available 'nltk'; then output::step "Downloading NLTK corpora..." - nltk_packages_definition="$BUILD_DIR/nltk.txt" + nltk_packages_definition="${BUILD_DIR}/nltk.txt" - if [[ -f "$nltk_packages_definition" ]]; then + if [[ -f "${nltk_packages_definition}" ]]; then meta_set "nltk_downloader" "enabled" - readarray -t nltk_packages <"$nltk_packages_definition" + readarray -t nltk_packages <"${nltk_packages_definition}" output::step "Downloading NLTK packages: ${nltk_packages[*]}" - python -m nltk.downloader -d "$BUILD_DIR/.heroku/python/nltk_data" "${nltk_packages[@]}" | output::indent - set_env NLTK_DATA "/app/.heroku/python/nltk_data" + nltk_data_dir="/app/.heroku/python/nltk_data" + if ! python -m nltk.downloader -d "${nltk_data_dir}" "${nltk_packages[@]}" |& output::indent; then + output::error <<-EOF + Error: Unable to download NLTK data. + + The 'python -m nltk.downloader' command to download NLTK + data did not exit successfully. + + See the log output above for more information. + EOF + meta_set "failure_reason" "nltk-downloader" + exit 1 + fi + + set_env NLTK_DATA "${nltk_data_dir}" else meta_set "nltk_downloader" "skipped-no-nltk-file" echo " 'nltk.txt' not found, not downloading any corpora" diff --git a/spec/fixtures/nltk_txt_invalid/nltk.txt b/spec/fixtures/nltk_txt_invalid/nltk.txt new file mode 100644 index 000000000..719ce91ff --- /dev/null +++ b/spec/fixtures/nltk_txt_invalid/nltk.txt @@ -0,0 +1 @@ +invalid! diff --git a/spec/fixtures/nltk_txt_invalid/requirements.txt b/spec/fixtures/nltk_txt_invalid/requirements.txt new file mode 100644 index 000000000..846929621 --- /dev/null +++ b/spec/fixtures/nltk_txt_invalid/requirements.txt @@ -0,0 +1 @@ +nltk diff --git a/spec/hatchet/nltk_spec.rb b/spec/hatchet/nltk_spec.rb index 28b1ab660..d61451278 100644 --- a/spec/hatchet/nltk_spec.rb +++ b/spec/hatchet/nltk_spec.rb @@ -11,13 +11,13 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Downloading NLTK corpora... remote: -----> Downloading NLTK packages: city_database stopwords - remote: .*: RuntimeWarning: 'nltk.downloader' found in sys.modules after import of package 'nltk', but prior to execution of 'nltk.downloader'; this may result in unpredictable behaviour - remote: \\[nltk_data\\] Downloading package city_database to - remote: \\[nltk_data\\] /tmp/build_.*/.heroku/python/nltk_data... - remote: \\[nltk_data\\] Unzipping corpora/city_database.zip. - remote: \\[nltk_data\\] Downloading package stopwords to - remote: \\[nltk_data\\] /tmp/build_.*/.heroku/python/nltk_data... - remote: \\[nltk_data\\] Unzipping corpora/stopwords.zip. + remote: .*: RuntimeWarning: 'nltk.downloader' found in sys.modules after import of package 'nltk', but prior to execution of 'nltk.downloader'; this may result in unpredictable behaviour + remote: \\[nltk_data\\] Downloading package city_database to + remote: \\[nltk_data\\] /app/.heroku/python/nltk_data... + remote: \\[nltk_data\\] Unzipping corpora/city_database.zip. + remote: \\[nltk_data\\] Downloading package stopwords to + remote: \\[nltk_data\\] /app/.heroku/python/nltk_data... + remote: \\[nltk_data\\] Unzipping corpora/stopwords.zip. REGEX # TODO: Add a test that the downloaded corpora can be found at runtime. @@ -44,7 +44,37 @@ it 'does not try to install the specified NLTK corpora' do app.deploy do |app| - expect(app.output.downcase).not_to include('nltk') + expect(app.output).not_to include('NLTK') + expect(app.output).not_to include('nltk_data') + end + end + end + + context 'when nltk.txt contains invalid entries' do + let(:app) { Hatchet::Runner.new('spec/fixtures/nltk_txt_invalid', allow_failure: true) } + + it 'fails the build' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + remote: -----> Downloading NLTK corpora... + remote: -----> Downloading NLTK packages: invalid! + remote: .+: RuntimeWarning: 'nltk.downloader' found in sys.modules after import of package 'nltk', but prior to execution of 'nltk.downloader'; this may result in unpredictable behaviour + remote: \\[nltk_data\\] Error loading invalid!: Package 'invalid!' not found in + remote: \\[nltk_data\\] index + remote: Error installing package. Retry\\? \\[n/y/e\\] + remote: Traceback \\(most recent call last\\): + remote: .+ + remote: EOFError: EOF when reading a line + remote: + remote: ! Error: Unable to download NLTK data. + remote: ! + remote: ! The 'python -m nltk.downloader' command to download NLTK + remote: ! data did not exit successfully. + remote: ! + remote: ! See the log output above for more information. + remote: + remote: ! Push rejected, failed to compile Python app. + REGEX end end end From bfe0d096238f6502c93c75797cda1e3c632178ba Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 8 Nov 2024 19:17:32 +0000 Subject: [PATCH 057/406] Link to the stable release version of Pipenv's docs (#1693) Since the `/latest/` URLs are for the unreleased development version, rather than the latest stable release. --- lib/python_version.sh | 4 ++-- spec/hatchet/pipenv_spec.rb | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/python_version.sh b/lib/python_version.sh index de0dbca2a..20b268db4 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -204,7 +204,7 @@ function python_version::parse_python_version_file() { # Read the Python version from a Pipfile.lock, which can exist in one of two optional fields, # `python_full_version` (as N.N.N) and `python_version` (as N.N). If both fields are # defined, we will use the value set in `python_full_version`. See: -# https://pipenv.pypa.io/en/latest/specifiers.html#specifying-versions-of-python +# https://pipenv.pypa.io/en/stable/specifiers.html#specifying-versions-of-python function python_version::read_pipenv_python_version() { local build_dir="${1}" local pipfile_lock_path="${build_dir}/Pipfile.lock" @@ -259,7 +259,7 @@ function python_version::read_pipenv_python_version() { then run 'pipenv lock' to regenerate the lockfile. For more information, see: - https://pipenv.pypa.io/en/latest/specifiers.html#specifying-versions-of-python + https://pipenv.pypa.io/en/stable/specifiers.html#specifying-versions-of-python EOF meta_set "failure_reason" "pipfile-lock::invalid-version" exit 1 diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 494add751..283020384 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -215,7 +215,7 @@ remote: ! then run 'pipenv lock' to regenerate the lockfile. remote: ! remote: ! For more information, see: - remote: ! https://pipenv.pypa.io/en/latest/specifiers.html#specifying-versions-of-python + remote: ! https://pipenv.pypa.io/en/stable/specifiers.html#specifying-versions-of-python remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -247,7 +247,7 @@ remote: ! then run 'pipenv lock' to regenerate the lockfile. remote: ! remote: ! For more information, see: - remote: ! https://pipenv.pypa.io/en/latest/specifiers.html#specifying-versions-of-python + remote: ! https://pipenv.pypa.io/en/stable/specifiers.html#specifying-versions-of-python remote: remote: ! Push rejected, failed to compile Python app. OUTPUT From 1416814a17252e1d656d3f60ee862ae1fa0495bb Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Fri, 8 Nov 2024 19:40:41 +0000 Subject: [PATCH 058/406] Prepare release v266 (#1694) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6ad5059c4..28939e61c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v266] - 2024-11-08 + - Added a warning when the files for multiple package managers are found. In the future this warning will become an error. ([#1692](https://github.com/heroku/heroku-buildpack-python/pull/1692)) - Updated the build log message shown when installing dependencies to include the package manager command being run. ([#1689](https://github.com/heroku/heroku-buildpack-python/pull/1689)) - Changed test dependency installation on Heroku CI to now install `requirements.txt` and `requirements-test.txt` in a single `pip install` invocation rather than separately. This allows pip's resolver to resolve any version conflicts between the two files. ([#1689](https://github.com/heroku/heroku-buildpack-python/pull/1689)) @@ -1094,7 +1097,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v265...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v266...main +[v266]: https://github.com/heroku/heroku-buildpack-python/compare/v265...v266 [v265]: https://github.com/heroku/heroku-buildpack-python/compare/v264...v265 [v264]: https://github.com/heroku/heroku-buildpack-python/compare/v263...v264 [v263]: https://github.com/heroku/heroku-buildpack-python/compare/v262...v263 From ddf6aeab1be85d8e9282053d433d6e6780ac08f2 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 12 Nov 2024 10:29:03 +0000 Subject: [PATCH 059/406] Deprecate using Pipenv without a lockfile (#1695) Using Pipenv without a lockfile can result in nondeterministic package installs, and so for some years if `Pipfile.lock` is missing we've emitted a recommendation in the build log to add it. In addition, Pipenv has also since deprecated the `--skip-lockfile` feature upstream. As such, we'd like to sunset support for using Pipenv without a lockfile, but first we should make the existing warning more explicit and include instructions on how to generate a lockfile. GUS-W-17201692. --- CHANGELOG.md | 1 + lib/package_manager.sh | 12 ++++++++++-- spec/hatchet/pipenv_spec.rb | 11 ++++++++++- 3 files changed, 21 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 28939e61c..3862a88f3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Deprecated using Pipenv without a lockfile (`Pipfile.lock`). ([#1695](https://github.com/heroku/heroku-buildpack-python/pull/1695)) ## [v266] - 2024-11-08 diff --git a/lib/package_manager.sh b/lib/package_manager.sh index c140542af..eee49ab42 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -15,11 +15,19 @@ function package_manager::determine_package_manager() { meta_set "pipenv_has_lockfile" "true" elif [[ -f "${build_dir}/Pipfile" ]]; then # TODO: Start requiring a Pipfile.lock and make this branch a "missing lockfile" error instead. - # TODO: Adjust this warning to mention support for missing Pipfile.lock will be removed soon. output::warning <<-'EOF' Warning: No 'Pipfile.lock' found! - We recommend you commit this into your repository. + A 'Pipfile' file was found, however, the associated 'Pipfile.lock' + Pipenv lockfile was not. This means your app dependency versions + are not pinned, which means the package versions used on Heroku + might not match those installed in other environments. + + For now, we will install your dependencies without a lockfile, + however, in the future this warning will become an error. + + Run 'pipenv lock' locally to generate the lockfile, and make sure + that 'Pipfile.lock' is not listed in '.gitignore' or '.slugignore'. EOF package_managers_found+=(pipenv) package_managers_found_display_text+=("Pipfile (Pipenv)") diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 283020384..8aeeb697f 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -146,7 +146,16 @@ remote: remote: ! Warning: No 'Pipfile.lock' found! remote: ! - remote: ! We recommend you commit this into your repository. + remote: ! A 'Pipfile' file was found, however, the associated 'Pipfile.lock' + remote: ! Pipenv lockfile was not. This means your app dependency versions + remote: ! are not pinned, which means the package versions used on Heroku + remote: ! might not match those installed in other environments. + remote: ! + remote: ! For now, we will install your dependencies without a lockfile, + remote: ! however, in the future this warning will become an error. + remote: ! + remote: ! Run 'pipenv lock' locally to generate the lockfile, and make sure + remote: ! that 'Pipfile.lock' is not listed in '.gitignore' or '.slugignore'. remote: remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes From 498e2d2e1c77a1cd196d9d6b7754ad1fe0dc7be1 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 12 Nov 2024 18:14:36 +0000 Subject: [PATCH 060/406] Invoke `python -m venv` from a different working directory (#1698) When `python -m venv` is run, after creating the virtual environment it invokes `ensurepip` to install pip. The `ensurepip` module in older Python versions didn't correctly run its Python subprocesses in isolated mode, meaning that the working directory is added to `sys.path`. This can cause issues if the app's build directory contains files/directories that shadow expected package names (such as a `brotli` directory). For example: ``` -----> Installing Poetry 1.8.4 Error: Command '['/tmp/codon/tmp/cache/.heroku/python-poetry/venv/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1. ! Internal Error: Unable to create virtual environment for Poetry. ! ! The 'python -m venv' command to create a virtual environment did ! not exit successfully. ! ! See the log output above for more information. ``` The best fix is for apps to upgrade to newer Python patch versions (3.8.14+, 3.9.14+, 3.10.6+), since they include the upstream `ensurepip` fix as well as many other bug and security fixes. However, to ensure venv creation still works on these older Python versions, as a workaround we can run the `python -m venv` command from a different working directory (as an alternative to isolated mode). Fixes #1697. GUS-W-17215816. --- CHANGELOG.md | 1 + lib/poetry.sh | 6 +++++- spec/fixtures/poetry_oldest_python/brotli/.gitkeep | 2 ++ spec/hatchet/poetry_spec.rb | 6 ++++-- 4 files changed, 12 insertions(+), 3 deletions(-) create mode 100644 spec/fixtures/poetry_oldest_python/brotli/.gitkeep diff --git a/CHANGELOG.md b/CHANGELOG.md index 3862a88f3..914616f2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Deprecated using Pipenv without a lockfile (`Pipfile.lock`). ([#1695](https://github.com/heroku/heroku-buildpack-python/pull/1695)) +- Fixed Poetry venv creation when using an outdated Python version whose `ensurepip` module doesn't enable isolated mode, and the app's build directory contents shadows a package imported by pip (such as `brotli`). ([#1698](https://github.com/heroku/heroku-buildpack-python/pull/1698)) ## [v266] - 2024-11-08 diff --git a/lib/poetry.sh b/lib/poetry.sh index 95b54ff66..062639b3a 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -37,6 +37,7 @@ function poetry::install_poetry() { # The Poetry directory will already exist in the relocated cache case mentioned above. rm -rf "${poetry_root}" + mkdir -p "${poetry_root}" # We can't use the pip wheel bundled within Python's standard library to install Poetry # (which would allow us to use `--without-pip` here to skip the pip install), since it @@ -45,7 +46,10 @@ function poetry::install_poetry() { # are still using outdated patch releases of those Python versions, whose bundled pip # can be older (for example Python 3.9.0 ships with pip v20.2.1). Once Python 3.10 EOLs # we can switch back to the previous approach since Python 3.11.0 ships with pip v22.3. - if ! python -m venv "${poetry_venv_dir}"; then + # Changing the working directory away from the build dir is required to work around an + # `ensurepip` bug in older Python versions, where it doesn't run Python in isolated mode: + # https://github.com/heroku/heroku-buildpack-python/issues/1697 + if ! (cd "${poetry_root}" && python -m venv "${poetry_venv_dir}"); then output::error <<-EOF Internal Error: Unable to create virtual environment for Poetry. diff --git a/spec/fixtures/poetry_oldest_python/brotli/.gitkeep b/spec/fixtures/poetry_oldest_python/brotli/.gitkeep new file mode 100644 index 000000000..98d7394a0 --- /dev/null +++ b/spec/fixtures/poetry_oldest_python/brotli/.gitkeep @@ -0,0 +1,2 @@ +The brotli directory tests the workaround for an `ensurepip` bug in older Python versions: +https://github.com/heroku/heroku-buildpack-python/issues/1697 diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index ae875027c..ca2caf1d9 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -185,8 +185,10 @@ end end - # This checks that the Poetry bootstrap works even with older bundled pip, and that - # our chosen Poetry version also supports our oldest supported Python version. + # This checks that the Poetry bootstrap works even with older bundled pip, and that our + # chosen Poetry version also supports our oldest supported Python version. The fixture + # also includes a `brotli` directory to test the workaround for an `ensurepip` bug in + # older Python versions: https://github.com/heroku/heroku-buildpack-python/issues/1697 context 'when using the oldest supported Python version' do let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_oldest_python') } From ca99c39f9e3627fff727788206f7fcc4ff064b7f Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 12 Nov 2024 18:22:48 +0000 Subject: [PATCH 061/406] Prepare release v267 (#1696) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 914616f2a..a9b933927 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v267] - 2024-11-12 + - Deprecated using Pipenv without a lockfile (`Pipfile.lock`). ([#1695](https://github.com/heroku/heroku-buildpack-python/pull/1695)) - Fixed Poetry venv creation when using an outdated Python version whose `ensurepip` module doesn't enable isolated mode, and the app's build directory contents shadows a package imported by pip (such as `brotli`). ([#1698](https://github.com/heroku/heroku-buildpack-python/pull/1698)) @@ -1099,7 +1102,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v266...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v267...main +[v267]: https://github.com/heroku/heroku-buildpack-python/compare/v266...v267 [v266]: https://github.com/heroku/heroku-buildpack-python/compare/v265...v266 [v265]: https://github.com/heroku/heroku-buildpack-python/compare/v264...v265 [v264]: https://github.com/heroku/heroku-buildpack-python/compare/v263...v264 From 5409e5e31ee135f5872de7366000ca8d45d8b775 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 3 Dec 2024 12:02:32 +0000 Subject: [PATCH 062/406] Bump the ruby-dependencies group across 1 directory with 2 updates (#1709) * Bump the ruby-dependencies group across 1 directory with 2 updates Bumps the ruby-dependencies group with 2 updates in the / directory: [logger](https://github.com/ruby/logger) and [rubocop](https://github.com/rubocop/rubocop). Updates `logger` from 1.6.1 to 1.6.2 - [Release notes](https://github.com/ruby/logger/releases) - [Commits](https://github.com/ruby/logger/compare/v1.6.1...v1.6.2) Updates `rubocop` from 1.68.0 to 1.69.1 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.68.0...v1.69.1) --- updated-dependencies: - dependency-name: logger dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rubocop dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] * bundle update --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- Gemfile.lock | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 41f0139f6..4162ad6ba 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -19,16 +19,16 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.7.5) + json (2.9.0) language_server-protocol (3.17.0.3) - logger (1.6.1) + logger (1.6.2) moneta (1.0.0) multi_json (1.15.0) parallel (1.26.3) parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) - parser (3.3.5.1) + parser (3.3.6.0) ast (~> 2.4.1) racc platform-api (3.7.0) @@ -38,7 +38,7 @@ GEM racc (1.8.1) rainbow (3.1.1) rate_throttle_client (0.1.2) - regexp_parser (2.9.2) + regexp_parser (2.9.3) rrrretry (1.0.0) rspec-core (3.13.2) rspec-support (~> 3.13.0) @@ -47,26 +47,28 @@ GEM rspec-support (~> 3.13.0) rspec-retry (0.6.2) rspec-core (> 3.3) - rspec-support (3.13.1) - rubocop (1.68.0) + rspec-support (3.13.2) + rubocop (1.69.1) json (~> 2.3) language_server-protocol (>= 3.17.0) parallel (~> 1.10) parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) - regexp_parser (>= 2.4, < 3.0) - rubocop-ast (>= 1.32.2, < 2.0) + regexp_parser (>= 2.9.3, < 3.0) + rubocop-ast (>= 1.36.2, < 2.0) ruby-progressbar (~> 1.7) - unicode-display_width (>= 2.4.0, < 3.0) - rubocop-ast (1.33.0) + unicode-display_width (>= 2.4.0, < 4.0) + rubocop-ast (1.36.2) parser (>= 3.3.1.0) rubocop-rspec (3.2.0) rubocop (~> 1.61) ruby-progressbar (1.13.0) thor (1.3.2) threaded (0.0.4) - unicode-display_width (2.6.0) - webrick (1.8.2) + unicode-display_width (3.1.2) + unicode-emoji (~> 4.0, >= 4.0.4) + unicode-emoji (4.0.4) + webrick (1.9.1) PLATFORMS ruby @@ -85,4 +87,4 @@ RUBY VERSION ruby 3.3.5p100 BUNDLED WITH - 2.5.18 + 2.5.23 From 6be689b7d8fed0f5d4fef55d61bb0aa3c874cc65 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 3 Dec 2024 13:13:36 +0000 Subject: [PATCH 063/406] Bump pip from 24.0 to 24.3.1 (#1685) * Bump pip from 24.0 to 24.3.1 Bumps [pip](https://github.com/pypa/pip) from 24.0 to 24.3.1. - [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst) - [Commits](https://github.com/pypa/pip/compare/24.0...24.3.1) --- updated-dependencies: - dependency-name: pip dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Update tests + comments * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + lib/cache.sh | 5 ++--- requirements/pip.txt | 2 +- spec/hatchet/pip_spec.rb | 5 ++++- spec/hatchet/python_version_spec.rb | 10 +++++----- 5 files changed, 13 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a9b933927..592c1d13e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated pip from 24.0 to 24.3.1. ([#1685](https://github.com/heroku/heroku-buildpack-python/pull/1685)) ## [v267] - 2024-11-12 diff --git a/lib/cache.sh b/lib/cache.sh index bacebc999..551b8a87a 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -74,9 +74,8 @@ function cache::restore() { cached_pip_version="$(meta_prev_get "pip_version")" # Handle caches written by buildpack versions older than v252 (see above). if [[ -z "${cached_pip_version}" ]]; then - # Whilst we don't know the old version, we know the pip version has likely - # changed since the last build, and would rather err on the side of caution. - # (The pip version was last updated in v246, but will be updated again soon.) + # Whilst we don't know the old version, we know the pip version has definitely + # changed since buildpack v251. cache_invalidation_reasons+=("The pip version has changed") elif [[ "${cached_pip_version}" != "${PIP_VERSION:?}" ]]; then cache_invalidation_reasons+=("The pip version has changed from ${cached_pip_version} to ${PIP_VERSION}") diff --git a/requirements/pip.txt b/requirements/pip.txt index f293eef6f..662f25f21 100644 --- a/requirements/pip.txt +++ b/requirements/pip.txt @@ -1 +1 @@ -pip==24.0 +pip==24.3.1 diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 3f3305366..19fc6d6e0 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -227,6 +227,7 @@ remote: Preparing metadata \\(setup.py\\): finished with status 'done' remote: .+ remote: Installing collected packages: six, test + remote: DEPRECATION: Legacy editable install of test==0.0.0 from file:///tmp/build_.* \\(setup.py develop\\) is deprecated. pip 25.0 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at https://github.com/pypa/pip/issues/11457 remote: Running setup.py develop for test REGEX end @@ -254,7 +255,9 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: ERROR: Invalid requirement: 'an-invalid-requirement!' (from line 1 of requirements.txt) + remote: ERROR: Invalid requirement: 'an-invalid-requirement!': Expected end or semicolon (after name and no valid version specifier) + remote: an-invalid-requirement! + remote: ^ (from line 1 of requirements.txt) remote: remote: ! Error: Unable to install dependencies using pip. remote: ! diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index d566bc752..bf0503806 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -52,7 +52,7 @@ # This test performs an initial build using an older buildpack version, followed # by a build using the current version. This ensures that the current buildpack # can successfully read the version metadata written to the build cache in the past. - let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v250'] } + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v257'] } it 'builds with the same Python version as the last build' do app.deploy do |app| @@ -61,11 +61,11 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: Python 3.12.3 + remote: -----> No Python version was specified. Using the same version as the last build: Python 3.12.6 remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Discarding cache since: - remote: - The pip version has changed - remote: -----> Installing Python 3.12.3 + remote: - The pip version has changed from 24.0 to #{PIP_VERSION} + remote: -----> Installing Python 3.12.6 remote: remote: ! Warning: A Python security update is available! remote: ! @@ -74,7 +74,7 @@ remote: remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} OUTPUT - expect(app.run('python -V')).to include('Python 3.12.3') + expect(app.run('python -V')).to include('Python 3.12.6') end end end From a0dbcf77cc3a8e45ac0d428406903e45ea9f3f3f Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 3 Dec 2024 13:56:53 +0000 Subject: [PATCH 064/406] Bump wheel from 0.44.0 to 0.45.1 (#1707) * Bump wheel from 0.44.0 to 0.45.1 Bumps [wheel](https://github.com/pypa/wheel) from 0.44.0 to 0.45.1. - [Release notes](https://github.com/pypa/wheel/releases) - [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst) - [Commits](https://github.com/pypa/wheel/compare/0.44.0...0.45.1) --- updated-dependencies: - dependency-name: wheel dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/wheel.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 592c1d13e..15d9b343d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Updated pip from 24.0 to 24.3.1. ([#1685](https://github.com/heroku/heroku-buildpack-python/pull/1685)) +- Updated wheel from 0.44.0 to 0.45.1. ([#1707](https://github.com/heroku/heroku-buildpack-python/pull/1707)) ## [v267] - 2024-11-12 diff --git a/requirements/wheel.txt b/requirements/wheel.txt index e9f6da376..0a0864cb4 100644 --- a/requirements/wheel.txt +++ b/requirements/wheel.txt @@ -1 +1 @@ -wheel==0.44.0 +wheel==0.45.1 From 1b44202e0bc76aa79d42e5fcc051e49bf1f89c7b Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 4 Dec 2024 09:35:35 +0000 Subject: [PATCH 065/406] Prepare release v268 (#1711) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 15d9b343d..c9d758a3d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v268] - 2024-12-04 + - Updated pip from 24.0 to 24.3.1. ([#1685](https://github.com/heroku/heroku-buildpack-python/pull/1685)) - Updated wheel from 0.44.0 to 0.45.1. ([#1707](https://github.com/heroku/heroku-buildpack-python/pull/1707)) @@ -1104,7 +1107,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v267...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v268...main +[v268]: https://github.com/heroku/heroku-buildpack-python/compare/v267...v268 [v267]: https://github.com/heroku/heroku-buildpack-python/compare/v266...v267 [v266]: https://github.com/heroku/heroku-buildpack-python/compare/v265...v266 [v265]: https://github.com/heroku/heroku-buildpack-python/compare/v264...v265 From 7ca1718b3555579e166e056c5b437c1b0344ccf7 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 4 Dec 2024 15:29:05 +0000 Subject: [PATCH 066/406] Add support for Python 3.13.1, 3.12.8, 3.11.11, 3.10.16 and 3.9.21 (#1712) Release announcement: https://blog.python.org/2024/12/python-3131-3128-31111-31016-and-3921.html Changelog: https://docs.python.org/3.13/whatsnew/changelog.html#python-3-13-1-final https://docs.python.org/3.12/whatsnew/changelog.html#python-3-12-8-final https://docs.python.org/release/3.11.11/whatsnew/changelog.html#python-3-11-11-final https://docs.python.org/release/3.10.16/whatsnew/changelog.html#python-3-10-16-final https://docs.python.org/release/3.9.21/whatsnew/changelog.html#python-3-9-21-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/12159159015 https://github.com/heroku/heroku-buildpack-python/actions/runs/12159160169 https://github.com/heroku/heroku-buildpack-python/actions/runs/12159161685 https://github.com/heroku/heroku-buildpack-python/actions/runs/12162379421 https://github.com/heroku/heroku-buildpack-python/actions/runs/12159164423 GUS-W-14846803. GUS-W-17350155. GUS-W-17367730. --- CHANGELOG.md | 5 +++++ lib/python_version.sh | 10 +++++----- spec/spec_helper.rb | 10 +++++----- 3 files changed, 15 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c9d758a3d..1562be38e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ ## [Unreleased] +- The Python 3.13 version alias now resolves to Python 3.13.1. ([#1712](https://github.com/heroku/heroku-buildpack-python/pull/1712)) +- The Python 3.12 version alias now resolves to Python 3.12.8. ([#1712](https://github.com/heroku/heroku-buildpack-python/pull/1712)) +- The Python 3.11 version alias now resolves to Python 3.11.11. ([#1712](https://github.com/heroku/heroku-buildpack-python/pull/1712)) +- The Python 3.10 version alias now resolves to Python 3.10.16. ([#1712](https://github.com/heroku/heroku-buildpack-python/pull/1712)) +- The Python 3.9 version alias now resolves to Python 3.9.21. ([#1712](https://github.com/heroku/heroku-buildpack-python/pull/1712)) ## [v268] - 2024-12-04 diff --git a/lib/python_version.sh b/lib/python_version.sh index 20b268db4..169fcf139 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -5,11 +5,11 @@ set -euo pipefail LATEST_PYTHON_3_8="3.8.20" -LATEST_PYTHON_3_9="3.9.20" -LATEST_PYTHON_3_10="3.10.15" -LATEST_PYTHON_3_11="3.11.10" -LATEST_PYTHON_3_12="3.12.7" -LATEST_PYTHON_3_13="3.13.0" +LATEST_PYTHON_3_9="3.9.21" +LATEST_PYTHON_3_10="3.10.16" +LATEST_PYTHON_3_11="3.11.11" +LATEST_PYTHON_3_12="3.12.8" +LATEST_PYTHON_3_13="3.13.1" DEFAULT_PYTHON_FULL_VERSION="${LATEST_PYTHON_3_12}" DEFAULT_PYTHON_MAJOR_VERSION="${DEFAULT_PYTHON_FULL_VERSION%.*}" diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 52860db92..9668d6aa3 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -10,11 +10,11 @@ FIXTURE_DIR = Pathname.new(__FILE__).parent.join('fixtures') LATEST_PYTHON_3_8 = '3.8.20' -LATEST_PYTHON_3_9 = '3.9.20' -LATEST_PYTHON_3_10 = '3.10.15' -LATEST_PYTHON_3_11 = '3.11.10' -LATEST_PYTHON_3_12 = '3.12.7' -LATEST_PYTHON_3_13 = '3.13.0' +LATEST_PYTHON_3_9 = '3.9.21' +LATEST_PYTHON_3_10 = '3.10.16' +LATEST_PYTHON_3_11 = '3.11.11' +LATEST_PYTHON_3_12 = '3.12.8' +LATEST_PYTHON_3_13 = '3.13.1' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_12 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From dc79a48ef12f1d0f64e23c9fec57b45566fe59f2 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 4 Dec 2024 16:14:17 +0000 Subject: [PATCH 067/406] Prepare release v269 (#1713) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1562be38e..5d309be16 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v269] - 2024-12-04 + - The Python 3.13 version alias now resolves to Python 3.13.1. ([#1712](https://github.com/heroku/heroku-buildpack-python/pull/1712)) - The Python 3.12 version alias now resolves to Python 3.12.8. ([#1712](https://github.com/heroku/heroku-buildpack-python/pull/1712)) - The Python 3.11 version alias now resolves to Python 3.11.11. ([#1712](https://github.com/heroku/heroku-buildpack-python/pull/1712)) @@ -1112,7 +1115,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v268...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v269...main +[v269]: https://github.com/heroku/heroku-buildpack-python/compare/v268...v269 [v268]: https://github.com/heroku/heroku-buildpack-python/compare/v267...v268 [v267]: https://github.com/heroku/heroku-buildpack-python/compare/v266...v267 [v266]: https://github.com/heroku/heroku-buildpack-python/compare/v265...v266 From b77dd09b8b1fa03e93eeade66cd48f76e4553791 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 6 Dec 2024 13:43:34 +0000 Subject: [PATCH 068/406] Make Python version pinning pin to the major version only (#1714) For apps that do not specify an explicit Python version (e.g.: via a `.python-version` or `runtime.txt` file), the buildpack uses a curated default version for the first build of the app. Then for subsequent builds of the app, the buildpack selects a Python version based on the version found in the build cache, so that the version used for the app doesn't change in a breaking way over time as the buildpack's own default version changes. This feature is referred to as "version pinning" and/or "sticky versions". The existing implementation of this feature pinned the version to the full Python version (e.g. `3.13.0`), meaning that the app would always use that exact Python version, even when newer backwards-compatible patch releases (such as `3.13.1`) became available over time. Now that we have Python major version -> latest patch version resolution support (as of #1658) and improved build output around cache invalidation reasons (as of #1679), we can switch to instead only pinning to the major Python version (e.g. `3.13`). This allows apps that do not specify a Python version to pick up any bug and security fixes for their major Python version the next time the app is built, whilst still keeping the compatibility properties of version pinning. Longer term, the plan is to deprecate/sunset version pinning entirely (since it leads to confusing UX / lack of parity between multiple apps deployed from the same codebase at different times, e.g. review apps), and the Python CNB has already dropped support for it. However, that will be a breaking change for the classic buildpack, so out of scope for now. GUS-W-17384879. --- CHANGELOG.md | 1 + bin/compile | 8 ++++---- lib/cache.sh | 8 ++++---- lib/python_version.sh | 14 +++++++------- spec/hatchet/pip_spec.rb | 10 +++++----- spec/hatchet/python_version_spec.rb | 13 ++++--------- spec/hatchet/stack_spec.rb | 13 ++++--------- 7 files changed, 29 insertions(+), 38 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5d309be16..7584f3c9d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Changed Python version pinning behaviour for apps that do not specify a Python version. Repeat builds are now pinned to the major Python version only (`3.X`) instead of the full Python version (`3.X.Y`), so that they always use the latest patch version. ([#1714](https://github.com/heroku/heroku-buildpack-python/pull/1714)) ## [v269] - 2024-12-04 diff --git a/bin/compile b/bin/compile index 2b64a4aa1..495d8c20d 100755 --- a/bin/compile +++ b/bin/compile @@ -106,11 +106,11 @@ hooks::run_hook "pre_compile" package_manager="$(package_manager::determine_package_manager "${BUILD_DIR}")" meta_set "package_manager" "${package_manager}" -cached_python_version="$(cache::cached_python_version "${CACHE_DIR}")" +cached_python_full_version="$(cache::cached_python_full_version "${CACHE_DIR}")" # We use the Bash 4.3+ `nameref` feature to pass back multiple values from this function # without having to hardcode globals. See: https://stackoverflow.com/a/38997681 -python_version::read_requested_python_version "${BUILD_DIR}" "${package_manager}" "${cached_python_version}" requested_python_version python_version_origin +python_version::read_requested_python_version "${BUILD_DIR}" "${package_manager}" "${cached_python_full_version}" requested_python_version python_version_origin meta_set "python_version_reason" "${python_version_origin}" # TODO: More strongly recommend specifying a Python version (eg switch the messaging to @@ -122,7 +122,7 @@ case "${python_version_origin}" in echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" ;; cached) - output::step "No Python version was specified. Using the same version as the last build: Python ${requested_python_version}" + output::step "No Python version was specified. Using the same major version as the last build: Python ${requested_python_version}" echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" ;; *) @@ -135,7 +135,7 @@ python_major_version="${python_full_version%.*}" meta_set "python_version" "${python_full_version}" meta_set "python_version_major" "${python_major_version}" -cache::restore "${BUILD_DIR}" "${CACHE_DIR}" "${STACK:?}" "${cached_python_version}" "${python_full_version}" "${package_manager}" +cache::restore "${BUILD_DIR}" "${CACHE_DIR}" "${STACK:?}" "${cached_python_full_version}" "${python_full_version}" "${package_manager}" # The directory for the .profile.d scripts. mkdir -p "$(dirname "$PROFILE_PATH")" diff --git a/lib/cache.sh b/lib/cache.sh index 551b8a87a..9d9415ba3 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -6,7 +6,7 @@ set -euo pipefail # Read the full Python version of the Python install in the cache, or the empty string # if the cache is empty or doesn't contain a Python version metadata file. -function cache::cached_python_version() { +function cache::cached_python_full_version() { local cache_dir="${1}" if [[ -f "${cache_dir}/.heroku/python-version" ]]; then @@ -29,7 +29,7 @@ function cache::restore() { local build_dir="${1}" local cache_dir="${2}" local stack="${3}" - local cached_python_version="${4}" + local cached_python_full_version="${4}" local python_full_version="${5}" local package_manager="${6}" @@ -48,8 +48,8 @@ function cache::restore() { cache_invalidation_reasons+=("The stack has changed from ${cached_stack:-"unknown"} to ${stack}") fi - if [[ "${cached_python_version}" != "${python_full_version}" ]]; then - cache_invalidation_reasons+=("The Python version has changed from ${cached_python_version:-"unknown"} to ${python_full_version}") + if [[ "${cached_python_full_version}" != "${python_full_version}" ]]; then + cache_invalidation_reasons+=("The Python version has changed from ${cached_python_full_version:-"unknown"} to ${python_full_version}") fi # The metadata store only exists in caches created in v252+ of the buildpack (released 2024-06-17), diff --git a/lib/python_version.sh b/lib/python_version.sh index 169fcf139..9513d7470 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -18,6 +18,8 @@ DEFAULT_PYTHON_MAJOR_VERSION="${DEFAULT_PYTHON_FULL_VERSION%.*}" INT_REGEX="(0|[1-9][0-9]*)" # Versions of form N.N or N.N.N. PYTHON_VERSION_REGEX="${INT_REGEX}\.${INT_REGEX}(\.${INT_REGEX})?" +# Versions of form N.N.N only. +PYTHON_FULL_VERSION_REGEX="${INT_REGEX}\.${INT_REGEX}\.${INT_REGEX}" # Determine what Python version has been requested for the project. # @@ -33,16 +35,13 @@ PYTHON_VERSION_REGEX="${INT_REGEX}\.${INT_REGEX}(\.${INT_REGEX})?" # # If a version wasn't specified by the app, then new apps/those with an empty cache will use # a buildpack default version for the first build, and then subsequent cached builds will use -# the same Python full version in perpetuity (aka sticky versions). Sticky versioning leads to +# the same Python major version in perpetuity (aka sticky versions). Sticky versioning leads to # confusing UX so is something we want to deprecate/sunset in the future (and have already done # so in the Python CNB). -# TODO: Change the sticky versioning implementation so it's only sticky to the major version -# rather than the full version, so apps that don't specify a Python version at least get -# security patch updates. function python_version::read_requested_python_version() { local build_dir="${1}" local package_manager="${2}" - local cached_python_version="${3}" + local cached_python_full_version="${3}" # We use the Bash 4.3+ `nameref` feature to pass back multiple values from this function # without having to hardcode globals. See: https://stackoverflow.com/a/38997681 declare -n version="${4}" @@ -75,8 +74,9 @@ function python_version::read_requested_python_version() { fi # Protect against unsupported (eg PyPy) or invalid versions being found in the cache metadata. - if [[ "${cached_python_version}" =~ ^${PYTHON_VERSION_REGEX}$ ]]; then - version="${cached_python_version}" + if [[ "${cached_python_full_version}" =~ ^${PYTHON_FULL_VERSION_REGEX}$ ]]; then + local cached_python_major_version="${cached_python_full_version%.*}" + version="${cached_python_major_version}" origin="cached" else version="${DEFAULT_PYTHON_MAJOR_VERSION}" diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 19fc6d6e0..e2af8d585 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -79,7 +79,7 @@ app.deploy do |app| # The test fixture's requirements.txt is a symlink to a requirements file in a subdirectory in # order to test that symlinked requirements files work in general and with cache invalidation. - File.write('requirements/prod.txt', 'six', mode: 'a') + File.write('requirements/prod.txt', 'six==1.17.0', mode: 'a') app.commit! app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) @@ -93,12 +93,12 @@ remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 5)) remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) - remote: Collecting six (from -r requirements.txt (line 6)) - remote: Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB) + remote: Collecting six==1.17.0 (from -r requirements.txt (line 6)) + remote: Downloading six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB) remote: Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) - remote: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) + remote: Downloading six-1.17.0-py2.py3-none-any.whl (11 kB) remote: Installing collected packages: typing-extensions, six - remote: Successfully installed six-1.16.0 typing-extensions-4.12.2 + remote: Successfully installed six-1.17.0 typing-extensions-4.12.2 OUTPUT end end diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index bf0503806..5c2227a33 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -61,20 +61,15 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: Python 3.12.6 + remote: -----> No Python version was specified. Using the same major version as the last build: Python 3.12 remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Discarding cache since: + remote: - The Python version has changed from 3.12.6 to #{LATEST_PYTHON_3_12} remote: - The pip version has changed from 24.0 to #{PIP_VERSION} - remote: -----> Installing Python 3.12.6 - remote: - remote: ! Warning: A Python security update is available! - remote: ! - remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_12} - remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: + remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} OUTPUT - expect(app.run('python -V')).to include('Python 3.12.6') + expect(app.run('python -V')).to include("Python #{LATEST_PYTHON_3_12}") end end end diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index c30a09b88..b9cff5c9f 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -21,18 +21,13 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: Python 3.12.3 + remote: -----> No Python version was specified. Using the same major version as the last build: Python 3.12 remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Discarding cache since: remote: - The stack has changed from heroku-22 to heroku-24 + remote: - The Python version has changed from 3.12.3 to #{LATEST_PYTHON_3_12} remote: - The pip version has changed - remote: -----> Installing Python 3.12.3 - remote: - remote: ! Warning: A Python security update is available! - remote: ! - remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_12} - remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: + remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pip install -r requirements.txt' @@ -53,7 +48,7 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> No Python version was specified. Using the same major version as the last build: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Discarding cache since: remote: - The stack has changed from heroku-24 to heroku-22 From 29f1606d0fc638a89e8214233e76a68ca2c0f833 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 6 Dec 2024 19:28:36 +0000 Subject: [PATCH 069/406] Use Python 3.13 as the default Python version for new apps (#1715) Previously if a new app did not specify a Python version (e.g. via a `.python-version` file), then the buildpack would default to the latest Python 3.12 release. Now the buildpack will use the latest Python 3.13 release. This does not affect: * apps that specify an explicit Python version * existing apps that have already had a build using the previous default version (since that version is remembered from one build to the next, via the "version pinning" / "sticky versions" feature). Use-cases involving new apps that aren't compatible with Python 3.13 will now need to specify Python 3.12 explicitly: https://devcenter.heroku.com/articles/python-support#specify-a-python-version However most apps should be compatible, since Python 3.13 was released a couple of months ago (2024-10-07), does not contain significant breaking changes over Python 3.12, and has been happily used by the python-getting-started project since 2024-10-10. See: https://docs.python.org/3.13/whatsnew/3.13.html GUS-W-14846972. --- .github/workflows/build_python_runtime.yml | 2 +- CHANGELOG.md | 1 + README.md | 2 +- lib/python_version.sh | 2 +- spec/fixtures/ci_pipenv/.python-version | 2 +- spec/fixtures/ci_pipenv/Pipfile.lock | 12 +++--- spec/fixtures/ci_poetry/.python-version | 2 +- spec/fixtures/ci_poetry/poetry.lock | 16 ++++---- spec/fixtures/ci_poetry/pyproject.toml | 2 +- spec/fixtures/ci_requirements/.python-version | 2 +- .../ci_requirements/requirements-test.txt | 2 +- .../pipenv_and_requirements_txt/Pipfile | 2 +- .../pipenv_and_requirements_txt/Pipfile.lock | 10 ++--- spec/fixtures/pipenv_basic/Pipfile | 2 +- spec/fixtures/pipenv_basic/Pipfile.lock | 4 +- spec/fixtures/pipenv_editable/.python-version | 1 + spec/fixtures/pipenv_editable/Pipfile.lock | 8 ++-- .../pipenv_lockfile_out_of_sync/Pipfile | 2 +- spec/fixtures/pipenv_no_lockfile/Pipfile | 2 +- spec/fixtures/poetry_basic/.python-version | 2 +- spec/fixtures/poetry_basic/poetry.lock | 16 ++++---- spec/fixtures/poetry_basic/pyproject.toml | 2 +- spec/fixtures/poetry_editable/poetry.lock | 20 ++++----- spec/fixtures/poetry_editable/pyproject.toml | 2 +- .../poetry_lockfile_out_of_sync/poetry.lock | 4 +- .../pyproject.toml | 2 +- .../requirements_basic/.python-version | 2 +- .../requirements_compiled/.python-version | 2 + .../requirements_editable/.python-version | 3 ++ .../.python-version | 2 +- .../poetry.lock | 4 +- .../pyproject.toml | 2 +- spec/fixtures/setup_py_only/.python-version | 2 +- spec/hatchet/ci_spec.rb | 20 ++++----- spec/hatchet/pip_spec.rb | 35 +++++++--------- spec/hatchet/pipenv_spec.rb | 41 ++++++++----------- spec/hatchet/poetry_spec.rb | 22 +++++----- spec/hatchet/python_version_spec.rb | 2 +- spec/hatchet/stack_spec.rb | 3 +- spec/spec_helper.rb | 2 +- 40 files changed, 126 insertions(+), 140 deletions(-) create mode 100644 spec/fixtures/pipenv_editable/.python-version create mode 100644 spec/fixtures/requirements_compiled/.python-version create mode 100644 spec/fixtures/requirements_editable/.python-version diff --git a/.github/workflows/build_python_runtime.yml b/.github/workflows/build_python_runtime.yml index 724d4045a..9b90fc052 100644 --- a/.github/workflows/build_python_runtime.yml +++ b/.github/workflows/build_python_runtime.yml @@ -5,7 +5,7 @@ on: workflow_dispatch: inputs: python_version: - description: "Python version (eg: 3.12.0)" + description: "Python version (eg: 3.13.0)" type: string required: true stack: diff --git a/CHANGELOG.md b/CHANGELOG.md index 7584f3c9d..9ce314759 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Changed the default Python version for new apps from 3.12 to 3.13. ([#1715](https://github.com/heroku/heroku-buildpack-python/pull/1715)) - Changed Python version pinning behaviour for apps that do not specify a Python version. Repeat builds are now pinned to the major Python version only (`3.X`) instead of the full Python version (`3.X.Y`), so that they always use the latest patch version. ([#1714](https://github.com/heroku/heroku-buildpack-python/pull/1714)) ## [v269] - 2024-12-04 diff --git a/README.md b/README.md index f2767f7db..34684120f 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ If none of those are found, the buildpack will use a default Python version for build of an app, and then subsequent builds of that app will be pinned to that version unless the build cache is cleared or you request a different version. -The current default Python version is: 3.12 +The current default Python version is: 3.13 The supported Python versions are: diff --git a/lib/python_version.sh b/lib/python_version.sh index 9513d7470..161118294 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -11,7 +11,7 @@ LATEST_PYTHON_3_11="3.11.11" LATEST_PYTHON_3_12="3.12.8" LATEST_PYTHON_3_13="3.13.1" -DEFAULT_PYTHON_FULL_VERSION="${LATEST_PYTHON_3_12}" +DEFAULT_PYTHON_FULL_VERSION="${LATEST_PYTHON_3_13}" DEFAULT_PYTHON_MAJOR_VERSION="${DEFAULT_PYTHON_FULL_VERSION%.*}" # Integer with no redundant leading zeros. diff --git a/spec/fixtures/ci_pipenv/.python-version b/spec/fixtures/ci_pipenv/.python-version index e4fba2183..24ee5b1be 100644 --- a/spec/fixtures/ci_pipenv/.python-version +++ b/spec/fixtures/ci_pipenv/.python-version @@ -1 +1 @@ -3.12 +3.13 diff --git a/spec/fixtures/ci_pipenv/Pipfile.lock b/spec/fixtures/ci_pipenv/Pipfile.lock index aa11d8ecf..a7d99ba86 100644 --- a/spec/fixtures/ci_pipenv/Pipfile.lock +++ b/spec/fixtures/ci_pipenv/Pipfile.lock @@ -35,11 +35,11 @@ }, "packaging": { "hashes": [ - "sha256:026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002", - "sha256:5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124" + "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759", + "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f" ], "markers": "python_version >= '3.8'", - "version": "==24.1" + "version": "==24.2" }, "pluggy": { "hashes": [ @@ -51,12 +51,12 @@ }, "pytest": { "hashes": [ - "sha256:70b98107bd648308a7952b06e6ca9a50bc660be218d53c257cc1fc94fda10181", - "sha256:a6853c7375b2663155079443d2e45de913a911a11d669df02a50814944db57b2" + "sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6", + "sha256:965370d062bce11e73868e0335abac31b4d3de0e82f4007408d242b4f8610761" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==8.3.3" + "version": "==8.3.4" } } } diff --git a/spec/fixtures/ci_poetry/.python-version b/spec/fixtures/ci_poetry/.python-version index e4fba2183..24ee5b1be 100644 --- a/spec/fixtures/ci_poetry/.python-version +++ b/spec/fixtures/ci_poetry/.python-version @@ -1 +1 @@ -3.12 +3.13 diff --git a/spec/fixtures/ci_poetry/poetry.lock b/spec/fixtures/ci_poetry/poetry.lock index a727a186c..7fe5d0794 100644 --- a/spec/fixtures/ci_poetry/poetry.lock +++ b/spec/fixtures/ci_poetry/poetry.lock @@ -24,13 +24,13 @@ files = [ [[package]] name = "packaging" -version = "24.1" +version = "24.2" description = "Core utilities for Python packages" optional = false python-versions = ">=3.8" files = [ - {file = "packaging-24.1-py3-none-any.whl", hash = "sha256:5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124"}, - {file = "packaging-24.1.tar.gz", hash = "sha256:026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002"}, + {file = "packaging-24.2-py3-none-any.whl", hash = "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759"}, + {file = "packaging-24.2.tar.gz", hash = "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f"}, ] [[package]] @@ -50,13 +50,13 @@ testing = ["pytest", "pytest-benchmark"] [[package]] name = "pytest" -version = "8.3.3" +version = "8.3.4" description = "pytest: simple powerful testing with Python" optional = false python-versions = ">=3.8" files = [ - {file = "pytest-8.3.3-py3-none-any.whl", hash = "sha256:a6853c7375b2663155079443d2e45de913a911a11d669df02a50814944db57b2"}, - {file = "pytest-8.3.3.tar.gz", hash = "sha256:70b98107bd648308a7952b06e6ca9a50bc660be218d53c257cc1fc94fda10181"}, + {file = "pytest-8.3.4-py3-none-any.whl", hash = "sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6"}, + {file = "pytest-8.3.4.tar.gz", hash = "sha256:965370d062bce11e73868e0335abac31b4d3de0e82f4007408d242b4f8610761"}, ] [package.dependencies] @@ -81,5 +81,5 @@ files = [ [metadata] lock-version = "2.0" -python-versions = "^3.12" -content-hash = "2e71a7976f439ce69fc771708b83dcfc6f795072ea73a7c2de0241878cbd378a" +python-versions = "^3.13" +content-hash = "56ec6342f4a39b402d2b65bd08cfb3af4136a02bb58674df49eac8d317376a7f" diff --git a/spec/fixtures/ci_poetry/pyproject.toml b/spec/fixtures/ci_poetry/pyproject.toml index 99aa593b8..ad7398c6e 100644 --- a/spec/fixtures/ci_poetry/pyproject.toml +++ b/spec/fixtures/ci_poetry/pyproject.toml @@ -2,7 +2,7 @@ package-mode = false [tool.poetry.dependencies] -python = "^3.12" +python = "^3.13" typing-extensions = "*" [tool.poetry.group.test.dependencies] diff --git a/spec/fixtures/ci_requirements/.python-version b/spec/fixtures/ci_requirements/.python-version index e4fba2183..24ee5b1be 100644 --- a/spec/fixtures/ci_requirements/.python-version +++ b/spec/fixtures/ci_requirements/.python-version @@ -1 +1 @@ -3.12 +3.13 diff --git a/spec/fixtures/ci_requirements/requirements-test.txt b/spec/fixtures/ci_requirements/requirements-test.txt index 40543aaba..d197ada2f 100644 --- a/spec/fixtures/ci_requirements/requirements-test.txt +++ b/spec/fixtures/ci_requirements/requirements-test.txt @@ -1 +1 @@ -pytest==8.3.3 +pytest==8.3.4 diff --git a/spec/fixtures/pipenv_and_requirements_txt/Pipfile b/spec/fixtures/pipenv_and_requirements_txt/Pipfile index 9bf60112c..73f50fc86 100644 --- a/spec/fixtures/pipenv_and_requirements_txt/Pipfile +++ b/spec/fixtures/pipenv_and_requirements_txt/Pipfile @@ -9,4 +9,4 @@ urllib3 = "*" [dev-packages] [requires] -python_version = "3.12" +python_version = "3.13" diff --git a/spec/fixtures/pipenv_and_requirements_txt/Pipfile.lock b/spec/fixtures/pipenv_and_requirements_txt/Pipfile.lock index cc5ce05bc..dcd0b3ca7 100644 --- a/spec/fixtures/pipenv_and_requirements_txt/Pipfile.lock +++ b/spec/fixtures/pipenv_and_requirements_txt/Pipfile.lock @@ -1,11 +1,11 @@ { "_meta": { "hash": { - "sha256": "13de6076b4dc8bbad334b7e4ae8e8ae63f7f8b4aad794c7f0173871b6927c8d0" + "sha256": "60dc67ee4223a391c5e0ae2b4d7ea54a7b245773d76b6ff82156dda97a3e4fb2" }, "pipfile-spec": 6, "requires": { - "python_version": "3.12" + "python_version": "3.13" }, "sources": [ { @@ -18,12 +18,12 @@ "default": { "urllib3": { "hashes": [ - "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", - "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" + "sha256:ca899ca043dcb1bafa3e262d73aa25c465bfb49e0bd9dd5d59f1d0acba2f8fac", + "sha256:e7d814a81dad81e6caf2ec9fdedb284ecc9c73076b62654547cc64ccdcae26e9" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==2.2.1" + "version": "==2.2.3" } }, "develop": {} diff --git a/spec/fixtures/pipenv_basic/Pipfile b/spec/fixtures/pipenv_basic/Pipfile index 883fd6c01..7a763f509 100644 --- a/spec/fixtures/pipenv_basic/Pipfile +++ b/spec/fixtures/pipenv_basic/Pipfile @@ -9,4 +9,4 @@ typing-extensions = "*" [dev-packages] [requires] -python_version = "3.12" +python_version = "3.13" diff --git a/spec/fixtures/pipenv_basic/Pipfile.lock b/spec/fixtures/pipenv_basic/Pipfile.lock index 30824a47d..44a737c1a 100644 --- a/spec/fixtures/pipenv_basic/Pipfile.lock +++ b/spec/fixtures/pipenv_basic/Pipfile.lock @@ -1,11 +1,11 @@ { "_meta": { "hash": { - "sha256": "9661ed313a79ccb68c7dc4e639068f86ddd91e307ec2ed60498858d002e9b547" + "sha256": "83d7242edaa31ec24731c102c5debc217f3e5f5fa5b4e992de07d4d25805715e" }, "pipfile-spec": 6, "requires": { - "python_version": "3.12" + "python_version": "3.13" }, "sources": [ { diff --git a/spec/fixtures/pipenv_editable/.python-version b/spec/fixtures/pipenv_editable/.python-version new file mode 100644 index 000000000..e4fba2183 --- /dev/null +++ b/spec/fixtures/pipenv_editable/.python-version @@ -0,0 +1 @@ +3.12 diff --git a/spec/fixtures/pipenv_editable/Pipfile.lock b/spec/fixtures/pipenv_editable/Pipfile.lock index 9ed2169b9..ae096d558 100644 --- a/spec/fixtures/pipenv_editable/Pipfile.lock +++ b/spec/fixtures/pipenv_editable/Pipfile.lock @@ -30,11 +30,11 @@ }, "setuptools": { "hashes": [ - "sha256:6c1fccdac05a97e598fb0ae3bbed5904ccb317337a51139dcd51453611bbb987", - "sha256:c636ac361bc47580504644275c9ad802c50415c7522212252c033bd15f301f32" + "sha256:8199222558df7c86216af4f84c30e9b34a61d8ba19366cc914424cdbd28252f6", + "sha256:ce74b49e8f7110f9bf04883b730f4765b774ef3ef28f722cce7c273d253aaf7d" ], - "markers": "python_version >= '3.8'", - "version": "==69.5.1" + "markers": "python_version >= '3.9'", + "version": "==75.6.0" } }, "develop": {} diff --git a/spec/fixtures/pipenv_lockfile_out_of_sync/Pipfile b/spec/fixtures/pipenv_lockfile_out_of_sync/Pipfile index 9bf60112c..73f50fc86 100644 --- a/spec/fixtures/pipenv_lockfile_out_of_sync/Pipfile +++ b/spec/fixtures/pipenv_lockfile_out_of_sync/Pipfile @@ -9,4 +9,4 @@ urllib3 = "*" [dev-packages] [requires] -python_version = "3.12" +python_version = "3.13" diff --git a/spec/fixtures/pipenv_no_lockfile/Pipfile b/spec/fixtures/pipenv_no_lockfile/Pipfile index 883fd6c01..7a763f509 100644 --- a/spec/fixtures/pipenv_no_lockfile/Pipfile +++ b/spec/fixtures/pipenv_no_lockfile/Pipfile @@ -9,4 +9,4 @@ typing-extensions = "*" [dev-packages] [requires] -python_version = "3.12" +python_version = "3.13" diff --git a/spec/fixtures/poetry_basic/.python-version b/spec/fixtures/poetry_basic/.python-version index e4fba2183..24ee5b1be 100644 --- a/spec/fixtures/poetry_basic/.python-version +++ b/spec/fixtures/poetry_basic/.python-version @@ -1 +1 @@ -3.12 +3.13 diff --git a/spec/fixtures/poetry_basic/poetry.lock b/spec/fixtures/poetry_basic/poetry.lock index a727a186c..7fe5d0794 100644 --- a/spec/fixtures/poetry_basic/poetry.lock +++ b/spec/fixtures/poetry_basic/poetry.lock @@ -24,13 +24,13 @@ files = [ [[package]] name = "packaging" -version = "24.1" +version = "24.2" description = "Core utilities for Python packages" optional = false python-versions = ">=3.8" files = [ - {file = "packaging-24.1-py3-none-any.whl", hash = "sha256:5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124"}, - {file = "packaging-24.1.tar.gz", hash = "sha256:026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002"}, + {file = "packaging-24.2-py3-none-any.whl", hash = "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759"}, + {file = "packaging-24.2.tar.gz", hash = "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f"}, ] [[package]] @@ -50,13 +50,13 @@ testing = ["pytest", "pytest-benchmark"] [[package]] name = "pytest" -version = "8.3.3" +version = "8.3.4" description = "pytest: simple powerful testing with Python" optional = false python-versions = ">=3.8" files = [ - {file = "pytest-8.3.3-py3-none-any.whl", hash = "sha256:a6853c7375b2663155079443d2e45de913a911a11d669df02a50814944db57b2"}, - {file = "pytest-8.3.3.tar.gz", hash = "sha256:70b98107bd648308a7952b06e6ca9a50bc660be218d53c257cc1fc94fda10181"}, + {file = "pytest-8.3.4-py3-none-any.whl", hash = "sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6"}, + {file = "pytest-8.3.4.tar.gz", hash = "sha256:965370d062bce11e73868e0335abac31b4d3de0e82f4007408d242b4f8610761"}, ] [package.dependencies] @@ -81,5 +81,5 @@ files = [ [metadata] lock-version = "2.0" -python-versions = "^3.12" -content-hash = "2e71a7976f439ce69fc771708b83dcfc6f795072ea73a7c2de0241878cbd378a" +python-versions = "^3.13" +content-hash = "56ec6342f4a39b402d2b65bd08cfb3af4136a02bb58674df49eac8d317376a7f" diff --git a/spec/fixtures/poetry_basic/pyproject.toml b/spec/fixtures/poetry_basic/pyproject.toml index 1b21050ab..9fc0e8356 100644 --- a/spec/fixtures/poetry_basic/pyproject.toml +++ b/spec/fixtures/poetry_basic/pyproject.toml @@ -2,7 +2,7 @@ package-mode = false [tool.poetry.dependencies] -python = "^3.12" +python = "^3.13" typing-extensions = "*" # This group shouldn't be installed due to us passing `--only main`. diff --git a/spec/fixtures/poetry_editable/poetry.lock b/spec/fixtures/poetry_editable/poetry.lock index 4047b0d15..a26fb9272 100644 --- a/spec/fixtures/poetry_editable/poetry.lock +++ b/spec/fixtures/poetry_editable/poetry.lock @@ -52,25 +52,25 @@ url = "packages/local_package_setup_py" [[package]] name = "setuptools" -version = "75.2.0" +version = "75.6.0" description = "Easily download, build, install, upgrade, and uninstall Python packages" optional = false -python-versions = ">=3.8" +python-versions = ">=3.9" files = [ - {file = "setuptools-75.2.0-py3-none-any.whl", hash = "sha256:a7fcb66f68b4d9e8e66b42f9876150a3371558f98fa32222ffaa5bced76406f8"}, - {file = "setuptools-75.2.0.tar.gz", hash = "sha256:753bb6ebf1f465a1912e19ed1d41f403a79173a9acf66a42e7e6aec45c3c16ec"}, + {file = "setuptools-75.6.0-py3-none-any.whl", hash = "sha256:ce74b49e8f7110f9bf04883b730f4765b774ef3ef28f722cce7c273d253aaf7d"}, + {file = "setuptools-75.6.0.tar.gz", hash = "sha256:8199222558df7c86216af4f84c30e9b34a61d8ba19366cc914424cdbd28252f6"}, ] [package.extras] -check = ["pytest-checkdocs (>=2.4)", "pytest-ruff (>=0.2.1)", "ruff (>=0.5.2)"] -core = ["importlib-metadata (>=6)", "importlib-resources (>=5.10.2)", "jaraco.collections", "jaraco.functools", "jaraco.text (>=3.7)", "more-itertools", "more-itertools (>=8.8)", "packaging", "packaging (>=24)", "platformdirs (>=2.6.2)", "tomli (>=2.0.1)", "wheel (>=0.43.0)"] +check = ["pytest-checkdocs (>=2.4)", "pytest-ruff (>=0.2.1)", "ruff (>=0.7.0)"] +core = ["importlib_metadata (>=6)", "jaraco.collections", "jaraco.functools (>=4)", "jaraco.text (>=3.7)", "more_itertools", "more_itertools (>=8.8)", "packaging", "packaging (>=24.2)", "platformdirs (>=4.2.2)", "tomli (>=2.0.1)", "wheel (>=0.43.0)"] cover = ["pytest-cov"] doc = ["furo", "jaraco.packaging (>=9.3)", "jaraco.tidelift (>=1.4)", "pygments-github-lexers (==0.0.5)", "pyproject-hooks (!=1.1)", "rst.linker (>=1.9)", "sphinx (>=3.5)", "sphinx-favicon", "sphinx-inline-tabs", "sphinx-lint", "sphinx-notfound-page (>=1,<2)", "sphinx-reredirects", "sphinxcontrib-towncrier", "towncrier (<24.7)"] enabler = ["pytest-enabler (>=2.2)"] -test = ["build[virtualenv] (>=1.0.3)", "filelock (>=3.4.0)", "ini2toml[lite] (>=0.14)", "jaraco.develop (>=7.21)", "jaraco.envs (>=2.2)", "jaraco.path (>=3.2.0)", "jaraco.test", "packaging (>=23.2)", "pip (>=19.1)", "pyproject-hooks (!=1.1)", "pytest (>=6,!=8.1.*)", "pytest-home (>=0.5)", "pytest-perf", "pytest-subprocess", "pytest-timeout", "pytest-xdist (>=3)", "tomli-w (>=1.0.0)", "virtualenv (>=13.0.0)", "wheel (>=0.44.0)"] -type = ["importlib-metadata (>=7.0.2)", "jaraco.develop (>=7.21)", "mypy (==1.11.*)", "pytest-mypy"] +test = ["build[virtualenv] (>=1.0.3)", "filelock (>=3.4.0)", "ini2toml[lite] (>=0.14)", "jaraco.develop (>=7.21)", "jaraco.envs (>=2.2)", "jaraco.path (>=3.2.0)", "jaraco.test (>=5.5)", "packaging (>=24.2)", "pip (>=19.1)", "pyproject-hooks (!=1.1)", "pytest (>=6,!=8.1.*)", "pytest-home (>=0.5)", "pytest-perf", "pytest-subprocess", "pytest-timeout", "pytest-xdist (>=3)", "tomli-w (>=1.0.0)", "virtualenv (>=13.0.0)", "wheel (>=0.44.0)"] +type = ["importlib_metadata (>=7.0.2)", "jaraco.develop (>=7.21)", "mypy (>=1.12,<1.14)", "pytest-mypy"] [metadata] lock-version = "2.0" -python-versions = "^3.12" -content-hash = "f362eb3b600ebb153055e8c4ed85f1cdeb18ffd8528710d5e1631c448a2f8adc" +python-versions = "^3.13" +content-hash = "622d4971a16f64f64de12d8208024658e6d7b9e30e1315e6970110ec5e70975c" diff --git a/spec/fixtures/poetry_editable/pyproject.toml b/spec/fixtures/poetry_editable/pyproject.toml index 7bcc6ab80..b7659537b 100644 --- a/spec/fixtures/poetry_editable/pyproject.toml +++ b/spec/fixtures/poetry_editable/pyproject.toml @@ -5,7 +5,7 @@ description = "" authors = [] [tool.poetry.dependencies] -python = "^3.12" +python = "^3.13" gunicorn = { git = "https://github.com/benoitc/gunicorn.git", tag = "20.1.0", develop = true } local-package-pyproject-toml = { path = "packages/local_package_pyproject_toml", develop = true } local-package-setup-py = { path = "packages/local_package_setup_py", develop = true } diff --git a/spec/fixtures/poetry_lockfile_out_of_sync/poetry.lock b/spec/fixtures/poetry_lockfile_out_of_sync/poetry.lock index 8738e5fc8..e9d42af7d 100644 --- a/spec/fixtures/poetry_lockfile_out_of_sync/poetry.lock +++ b/spec/fixtures/poetry_lockfile_out_of_sync/poetry.lock @@ -3,5 +3,5 @@ package = [] [metadata] lock-version = "2.0" -python-versions = "^3.12" -content-hash = "34e39677d8527182346093002688d17a5d2fc204b9eb3e094b2e6ac519028228" +python-versions = "^3.13" +content-hash = "f01b553f3895e558c34b4f10542e05acdef39bf0527c8090bd136d914dc73f94" diff --git a/spec/fixtures/poetry_lockfile_out_of_sync/pyproject.toml b/spec/fixtures/poetry_lockfile_out_of_sync/pyproject.toml index f0367e35e..032945f60 100644 --- a/spec/fixtures/poetry_lockfile_out_of_sync/pyproject.toml +++ b/spec/fixtures/poetry_lockfile_out_of_sync/pyproject.toml @@ -2,7 +2,7 @@ package-mode = false [tool.poetry.dependencies] -python = "^3.12" +python = "^3.13" # This dependency isn't in the lockfile. typing-extensions = "*" diff --git a/spec/fixtures/requirements_basic/.python-version b/spec/fixtures/requirements_basic/.python-version index e4fba2183..24ee5b1be 100644 --- a/spec/fixtures/requirements_basic/.python-version +++ b/spec/fixtures/requirements_basic/.python-version @@ -1 +1 @@ -3.12 +3.13 diff --git a/spec/fixtures/requirements_compiled/.python-version b/spec/fixtures/requirements_compiled/.python-version new file mode 100644 index 000000000..56ada5c99 --- /dev/null +++ b/spec/fixtures/requirements_compiled/.python-version @@ -0,0 +1,2 @@ +# TODO: Switch this to using Python 3.13 as part of the sqlite removal. +3.12 diff --git a/spec/fixtures/requirements_editable/.python-version b/spec/fixtures/requirements_editable/.python-version new file mode 100644 index 000000000..9b9fc4643 --- /dev/null +++ b/spec/fixtures/requirements_editable/.python-version @@ -0,0 +1,3 @@ +# Note: This test intentionally uses Python 3.12, so that we test *.egg-link +# path rewriting using older globally installed setuptools. +3.12 diff --git a/spec/fixtures/requirements_txt_and_poetry_lock/.python-version b/spec/fixtures/requirements_txt_and_poetry_lock/.python-version index e4fba2183..24ee5b1be 100644 --- a/spec/fixtures/requirements_txt_and_poetry_lock/.python-version +++ b/spec/fixtures/requirements_txt_and_poetry_lock/.python-version @@ -1 +1 @@ -3.12 +3.13 diff --git a/spec/fixtures/requirements_txt_and_poetry_lock/poetry.lock b/spec/fixtures/requirements_txt_and_poetry_lock/poetry.lock index 7022f9054..f80c1b5c7 100644 --- a/spec/fixtures/requirements_txt_and_poetry_lock/poetry.lock +++ b/spec/fixtures/requirements_txt_and_poetry_lock/poetry.lock @@ -13,5 +13,5 @@ files = [ [metadata] lock-version = "2.0" -python-versions = "^3.12" -content-hash = "2600e9ec45a8acb482a004ff092cb1453c2550aeec843430255a623cad8f7f86" +python-versions = "^3.13" +content-hash = "247bc9eddaeea191a758d051d2eae56b572964d566c6ab0c81822ee7535083dd" diff --git a/spec/fixtures/requirements_txt_and_poetry_lock/pyproject.toml b/spec/fixtures/requirements_txt_and_poetry_lock/pyproject.toml index b6077bf2c..ebfd436a7 100644 --- a/spec/fixtures/requirements_txt_and_poetry_lock/pyproject.toml +++ b/spec/fixtures/requirements_txt_and_poetry_lock/pyproject.toml @@ -2,5 +2,5 @@ package-mode = false [tool.poetry.dependencies] -python = "^3.12" +python = "^3.13" typing-extensions = "*" diff --git a/spec/fixtures/setup_py_only/.python-version b/spec/fixtures/setup_py_only/.python-version index e4fba2183..24ee5b1be 100644 --- a/spec/fixtures/setup_py_only/.python-version +++ b/spec/fixtures/setup_py_only/.python-version @@ -1 +1 @@ -3.12 +3.13 diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 661cd36b0..824564447 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -14,11 +14,10 @@ -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - -----> Installing SQLite3 + -----> Installing pip #{PIP_VERSION} -----> Installing dependencies using 'pip install -r requirements.txt -r requirements-test.txt' .+ - Successfully installed .+ pytest-8.3.3 typing-extensions-4.12.2 + Successfully installed .+ pytest-.+ typing-extensions-.+ -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running bin/post_compile hook CI=true @@ -58,7 +57,7 @@ PYTHONPATH=/app PYTHONUNBUFFERED=true WEB_CONCURRENCY=5 - pytest 8.3.3 + pytest .+ -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully REGEX @@ -68,8 +67,7 @@ -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Restoring cache -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} - -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - -----> Installing SQLite3 + -----> Installing pip #{PIP_VERSION} -----> Installing dependencies using 'pip install -r requirements.txt -r requirements-test.txt' -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running bin/post_compile hook @@ -87,9 +85,8 @@ -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + -----> Installing pip #{PIP_VERSION} -----> Installing Pipenv #{PIPENV_VERSION} - -----> Installing SQLite3 -----> Installing dependencies using 'pipenv install --deploy --dev' Installing dependencies from Pipfile.lock \\(.+\\)... Installing dependencies from Pipfile.lock \\(.+\\)... @@ -132,7 +129,7 @@ PYTHONPATH=/app PYTHONUNBUFFERED=true WEB_CONCURRENCY=5 - pytest 8.3.3 + pytest .+ -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully REGEX @@ -142,9 +139,8 @@ -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Restoring cache -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} - -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + -----> Installing pip #{PIP_VERSION} -----> Installing Pipenv #{PIPENV_VERSION} - -----> Installing SQLite3 -----> Installing dependencies using 'pipenv install --deploy --dev' Installing dependencies from Pipfile.lock \\(.+\\)... Installing dependencies from Pipfile.lock \\(.+\\)... @@ -216,7 +212,7 @@ PYTHONPATH=/app PYTHONUNBUFFERED=true WEB_CONCURRENCY=5 - pytest 8.3.3 + pytest .+ -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully REGEX diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index e2af8d585..1468c0e0b 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -22,8 +22,7 @@ remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing SQLite3 + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 5)) remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) @@ -42,19 +41,17 @@ remote: remote: ['', remote: '/app', - remote: '/app/.heroku/python/lib/python312.zip', - remote: '/app/.heroku/python/lib/python3.12', - remote: '/app/.heroku/python/lib/python3.12/lib-dynload', - remote: '/app/.heroku/python/lib/python3.12/site-packages'] + remote: '/app/.heroku/python/lib/python313.zip', + remote: '/app/.heroku/python/lib/python3.13', + remote: '/app/.heroku/python/lib/python3.13/lib-dynload', + remote: '/app/.heroku/python/lib/python3.13/site-packages'] remote: remote: Package Version remote: ----------------- ------- remote: pip #{PIP_VERSION} - remote: setuptools #{SETUPTOOLS_VERSION} remote: typing_extensions 4.12.2 - remote: wheel #{WHEEL_VERSION} remote: - remote: + remote: OUTPUT app.commit! app.push! @@ -63,8 +60,7 @@ remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Restoring cache remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing SQLite3 + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: -----> Inline app detected OUTPUT @@ -88,8 +84,7 @@ remote: -----> Discarding cache since: remote: - The contents of requirements.txt changed remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing SQLite3 + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 5)) remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) @@ -120,8 +115,7 @@ remote: -----> Discarding cache since: remote: - The package manager has changed from pipenv to pip remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing SQLite3 + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 5)) remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) @@ -133,12 +127,15 @@ end end + # TODO: Switch this to using Python 3.13 as part of the sqlite removal. context 'when requirements.txt contains popular compiled packages' do let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_compiled') } include_examples 'installs successfully using pip' end + # This test intentionally uses Python 3.12, so that we test rewriting using older globally installed + # setuptools. The Poetry equivalent of this test covers the PEP-517/518 setuptools case. context 'when requirements.txt contains editable requirements (both VCS and local package)' do let(:buildpacks) { [:default, 'heroku-community/inline'] } let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_editable', buildpacks:) } @@ -219,16 +216,12 @@ remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing SQLite3 + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install --editable .' remote: Obtaining file:///tmp/build_.* - remote: Preparing metadata \\(setup.py\\): started - remote: Preparing metadata \\(setup.py\\): finished with status 'done' remote: .+ remote: Installing collected packages: six, test - remote: DEPRECATION: Legacy editable install of test==0.0.0 from file:///tmp/build_.* \\(setup.py develop\\) is deprecated. pip 25.0 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at https://github.com/pypa/pip/issues/11457 - remote: Running setup.py develop for test + remote: Successfully installed six-.+ test-0.0.0 REGEX end end diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 8aeeb697f..7683592fd 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -14,9 +14,8 @@ remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Inline app detected @@ -31,10 +30,10 @@ remote: remote: \\['', remote: '/app', - remote: '/app/.heroku/python/lib/python312.zip', - remote: '/app/.heroku/python/lib/python3.12', - remote: '/app/.heroku/python/lib/python3.12/lib-dynload', - remote: '/app/.heroku/python/lib/python3.12/site-packages'\\] + remote: '/app/.heroku/python/lib/python313.zip', + remote: '/app/.heroku/python/lib/python3.13', + remote: '/app/.heroku/python/lib/python3.13/lib-dynload', + remote: '/app/.heroku/python/lib/python3.13/site-packages'\\] remote: remote: Package Version remote: ----------------- --------- @@ -44,12 +43,11 @@ remote: pip #{PIP_VERSION} remote: pipenv #{PIPENV_VERSION} remote: platformdirs .+ - remote: setuptools #{SETUPTOOLS_VERSION} + remote: setuptools .+ remote: typing_extensions 4.12.2 remote: virtualenv .+ - remote: wheel #{WHEEL_VERSION} remote: - remote: \\ + remote: \\ REGEX app.commit! app.push! @@ -58,9 +56,8 @@ remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock remote: -----> Restoring cache remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Inline app detected @@ -126,9 +123,8 @@ remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... REGEX @@ -160,9 +156,8 @@ remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pipenv install --skip-lock' remote: The flag --skip-lock has been reintroduced \\(but is not recommended\\). Without remote: the lock resolver it is difficult to manage multiple package indexes, and hash @@ -309,9 +304,8 @@ remote: - The Python version has changed from 3.12.4 to #{DEFAULT_PYTHON_FULL_VERSION} remote: - The Pipenv version has changed from 2023.12.1 to #{PIPENV_VERSION} remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Discovering process types @@ -337,9 +331,8 @@ remote: -----> Discarding cache since: remote: - The package manager has changed from pip to pipenv remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Discovering process types @@ -371,11 +364,10 @@ remote: ! Decide which package manager you want to use with your app, and remote: ! then delete the file\\(s\\) and any config from the others. remote: - remote: -----> Using Python 3.12 specified in Pipfile.lock - remote: -----> Installing Python #{LATEST_PYTHON_3_12} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... REGEX @@ -395,9 +387,8 @@ remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Your Pipfile.lock \\(.+\\) is out of date. Expected: \\(.+\\). remote: .+ diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index ca2caf1d9..c4514019d 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -33,15 +33,15 @@ remote: remote: ['', remote: '/app', - remote: '/app/.heroku/python/lib/python312.zip', - remote: '/app/.heroku/python/lib/python3.12', - remote: '/app/.heroku/python/lib/python3.12/lib-dynload', - remote: '/app/.heroku/python/lib/python3.12/site-packages'] + remote: '/app/.heroku/python/lib/python313.zip', + remote: '/app/.heroku/python/lib/python3.13', + remote: '/app/.heroku/python/lib/python3.13/lib-dynload', + remote: '/app/.heroku/python/lib/python3.13/site-packages'] remote: remote: Skipping virtualenv creation, as specified in config file. remote: typing-extensions 4.12.2 Backported and Experimental Type Hints for Python ... remote: - remote: + remote: OUTPUT app.commit! app.push! @@ -64,18 +64,19 @@ # TODO: Make this also test the Poetry version changing, the next (first) time we update Poetry, # by using an older buildpack version for the initial build. context 'when the requested Python version has changed since the last build' do - let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_basic') } + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v268'] } + let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_basic', buildpacks:) } it 'clears the cache before installing' do app.deploy do |app| - File.write('.python-version', '3.13') + update_buildpacks(app, [:default]) app.commit! app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python 3.13 specified in .python-version remote: -----> Discarding cache since: - remote: - The Python version has changed from #{LATEST_PYTHON_3_12} to #{LATEST_PYTHON_3_13} + remote: - The Python version has changed from 3.13.0 to #{LATEST_PYTHON_3_13} remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing Poetry #{POETRY_VERSION} remote: -----> Installing dependencies using 'poetry install --sync --only main' @@ -270,9 +271,8 @@ remote: ! the warning above. remote: remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version - remote: -----> Installing Python #{LATEST_PYTHON_3_12} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing SQLite3 + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' OUTPUT end diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 5c2227a33..28ee14641 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -206,7 +206,7 @@ remote: ! No Python version was found in the '.python-version' file. remote: ! remote: ! Update the file so that it contains a valid Python version - remote: ! such as '3.12'. + remote: ! such as '#{DEFAULT_PYTHON_MAJOR_VERSION}'. remote: ! remote: ! If the file already contains a version, check the line doesn't remote: ! begin with a '#', otherwise it will be treated as a comment. diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index b9cff5c9f..46817cad6 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -53,8 +53,7 @@ remote: -----> Discarding cache since: remote: - The stack has changed from heroku-24 to heroku-22 remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing SQLite3 + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 9668d6aa3..2405d9303 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -15,7 +15,7 @@ LATEST_PYTHON_3_11 = '3.11.11' LATEST_PYTHON_3_12 = '3.12.8' LATEST_PYTHON_3_13 = '3.13.1' -DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_12 +DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_13 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') # The requirement versions are effectively buildpack constants, however, we want From 23bc5279ce943db49de8387c08e94a77ce3dbb9f Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 9 Dec 2024 11:00:55 +0000 Subject: [PATCH 070/406] Bump poetry from 1.8.4 to 1.8.5 (#1716) * Bump poetry from 1.8.4 to 1.8.5 Bumps [poetry](https://github.com/python-poetry/poetry) from 1.8.4 to 1.8.5. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/1.8.5/CHANGELOG.md) - [Commits](https://github.com/python-poetry/poetry/compare/1.8.4...1.8.5) --- updated-dependencies: - dependency-name: poetry dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Update test * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/poetry.txt | 2 +- spec/hatchet/poetry_spec.rb | 3 +-- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ce314759..01bd83d90 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ - Changed the default Python version for new apps from 3.12 to 3.13. ([#1715](https://github.com/heroku/heroku-buildpack-python/pull/1715)) - Changed Python version pinning behaviour for apps that do not specify a Python version. Repeat builds are now pinned to the major Python version only (`3.X`) instead of the full Python version (`3.X.Y`), so that they always use the latest patch version. ([#1714](https://github.com/heroku/heroku-buildpack-python/pull/1714)) +- Updated Poetry from 1.8.4 to 1.8.5. ([#1716](https://github.com/heroku/heroku-buildpack-python/pull/1716)) ## [v269] - 2024-12-04 diff --git a/requirements/poetry.txt b/requirements/poetry.txt index 8e44691cb..b7c67421a 100644 --- a/requirements/poetry.txt +++ b/requirements/poetry.txt @@ -1 +1 @@ -poetry==1.8.4 +poetry==1.8.5 diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index c4514019d..6d3786bd8 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -61,8 +61,6 @@ end end - # TODO: Make this also test the Poetry version changing, the next (first) time we update Poetry, - # by using an older buildpack version for the initial build. context 'when the requested Python version has changed since the last build' do let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v268'] } let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_basic', buildpacks:) } @@ -77,6 +75,7 @@ remote: -----> Using Python 3.13 specified in .python-version remote: -----> Discarding cache since: remote: - The Python version has changed from 3.13.0 to #{LATEST_PYTHON_3_13} + remote: - The Poetry version has changed from 1.8.4 to #{POETRY_VERSION} remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing Poetry #{POETRY_VERSION} remote: -----> Installing dependencies using 'poetry install --sync --only main' From 07005f2a3f56c78d9ae51398378e7ae629aa1f54 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 10 Dec 2024 10:08:36 +0000 Subject: [PATCH 071/406] Prepare release v270 (#1717) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 01bd83d90..9cbd0987a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v270] - 2024-12-10 + - Changed the default Python version for new apps from 3.12 to 3.13. ([#1715](https://github.com/heroku/heroku-buildpack-python/pull/1715)) - Changed Python version pinning behaviour for apps that do not specify a Python version. Repeat builds are now pinned to the major Python version only (`3.X`) instead of the full Python version (`3.X.Y`), so that they always use the latest patch version. ([#1714](https://github.com/heroku/heroku-buildpack-python/pull/1714)) - Updated Poetry from 1.8.4 to 1.8.5. ([#1716](https://github.com/heroku/heroku-buildpack-python/pull/1716)) @@ -1118,7 +1121,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v269...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v270...main +[v270]: https://github.com/heroku/heroku-buildpack-python/compare/v269...v270 [v269]: https://github.com/heroku/heroku-buildpack-python/compare/v268...v269 [v268]: https://github.com/heroku/heroku-buildpack-python/compare/v267...v268 [v267]: https://github.com/heroku/heroku-buildpack-python/compare/v266...v267 From c4e4a4f390012d8d006847b63c1048341dca99cf Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 12 Dec 2024 16:06:20 +0000 Subject: [PATCH 072/406] Improve the error message shown for legacy Celery (#1718) pip 24.1+ rejects packages that use invalid versions in their metadata (such as Celery older than v5.2.1). We now annotate such failures with additional help text, mentioning the package needs to be upgraded, and link to the changelog entry which has additional advice: https://devcenter.heroku.com/changelog-items/3073 GUS-W-17415804. --- .github/workflows/ci.yml | 2 +- CHANGELOG.md | 1 + bin/warnings | 19 +++++++++--- .../requirements.txt | 1 + spec/hatchet/pip_spec.rb | 29 +++++++++++++++++++ 5 files changed, 47 insertions(+), 5 deletions(-) create mode 100644 spec/fixtures/requirements_legacy_celery/requirements.txt diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d754e02ef..a93559ca7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -47,7 +47,7 @@ jobs: HEROKU_API_KEY: ${{ secrets.HEROKU_API_KEY }} HEROKU_API_USER: ${{ secrets.HEROKU_API_USER }} HEROKU_DISABLE_AUTOUPDATE: 1 - PARALLEL_SPLIT_TEST_PROCESSES: 70 + PARALLEL_SPLIT_TEST_PROCESSES: 75 RSPEC_RETRY_RETRY_COUNT: 2 steps: - name: Checkout diff --git a/CHANGELOG.md b/CHANGELOG.md index 9cbd0987a..3672a05b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Improved the error message shown when pip install fails due to pip rejecting a package with invalid version metadata. ([#1718](https://github.com/heroku/heroku-buildpack-python/pull/1718)) ## [v270] - 2024-12-10 diff --git a/bin/warnings b/bin/warnings index 55cb9b6fb..0c1d861fd 100755 --- a/bin/warnings +++ b/bin/warnings @@ -1,6 +1,6 @@ #!/usr/bin/env bash -gdal-missing() { +show-warnings() { # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. if grep -qi 'Could not find gdal-config' "${WARNINGS_LOG}"; then output::error <<-'EOF' @@ -10,8 +10,19 @@ gdal-missing() { https://github.com/heroku/heroku-geo-buildpack EOF fi -} -show-warnings() { - gdal-missing + # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. + if grep -qi 'Please use pip<24.1 if you need to use this version' "${WARNINGS_LOG}"; then + output::error <<-'EOF' + Error: One of your dependencies contains broken metadata. + + Newer versions of pip reject packages that use invalid versions + in their metadata (such as Celery older than v5.2.1). + + Try upgrading to a newer version of the affected package. + + For more help, see: + https://devcenter.heroku.com/changelog-items/3073 + EOF + fi } diff --git a/spec/fixtures/requirements_legacy_celery/requirements.txt b/spec/fixtures/requirements_legacy_celery/requirements.txt new file mode 100644 index 000000000..2abca0701 --- /dev/null +++ b/spec/fixtures/requirements_legacy_celery/requirements.txt @@ -0,0 +1 @@ +celery==5.2.0 diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 1468c0e0b..a9cb7d595 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -285,4 +285,33 @@ end end end + + context 'when requirements.txt contains an old version of Celery with invalid metadata' do + let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_legacy_celery', allow_failure: true) } + + it 'outputs instructions for how to resolve the build failure' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: ERROR: No matching distribution found for celery==5.2.0 + remote: + remote: ! Error: One of your dependencies contains broken metadata. + remote: ! + remote: ! Newer versions of pip reject packages that use invalid versions + remote: ! in their metadata (such as Celery older than v5.2.1). + remote: ! + remote: ! Try upgrading to a newer version of the affected package. + remote: ! + remote: ! For more help, see: + remote: ! https://devcenter.heroku.com/changelog-items/3073 + remote: + remote: + remote: ! Error: Unable to install dependencies using pip. + remote: ! + remote: ! See the log output above for more information. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end end From e74c46eefd2d12428e40d5925d71d773cc650f96 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 12 Dec 2024 16:08:57 +0000 Subject: [PATCH 073/406] Use local scope for variables in output helpers (#1719) Since `read` defines the `line variable as global by default, meaning `line` was visible outside these functions. --- lib/output.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/output.sh b/lib/output.sh index 6893a5319..84aff374b 100644 --- a/lib/output.sh +++ b/lib/output.sh @@ -40,6 +40,7 @@ function output::indent() { # EOF # ``` function output::notice() { + local line echo >&2 while IFS= read -r line; do echo -e "${ANSI_BLUE} ! ${line}${ANSI_RESET}" >&2 @@ -58,6 +59,7 @@ function output::notice() { # EOF # ``` function output::warning() { + local line echo >&2 while IFS= read -r line; do echo -e "${ANSI_YELLOW} ! ${line}${ANSI_RESET}" >&2 @@ -76,6 +78,7 @@ function output::warning() { # EOF # ``` function output::error() { + local line echo >&2 while IFS= read -r line; do echo -e "${ANSI_RED} ! ${line}${ANSI_RESET}" >&2 From 8a2749978e0f2162dd8167b20842ececd38095ac Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 12 Dec 2024 16:16:46 +0000 Subject: [PATCH 074/406] Improve the error message when bundled pip cannot be found (#1720) The Python stdlib bundles a copy of pip, which the buildpack uses to bootstrap the real pip installation. This copy of pip should always exist, and thus not finding it is treated as an internal error. However, via Honeycomb I discovered one app that manages to hit this case regardless, and when it does so, the error message is not the "internal error" error message, but instead a Bash unbound variable error like: ``` -----> Using Python 3.9.20 specified in .python-version -----> Using cached install of Python 3.9.20 /tmp/buildpack/lib/utils.sh: line 33: bundled_pip_wheel_list[0]: unbound variable ``` My guess for why pip is not being found in this case is that the app's Git repo may have a broken/EOL Python install committed to it (which doesn't include the `ensurepip` module), which tricks the cache restoration into thinking it can re-use the cache. I will be adding an explicit warning/error for finding an existing `.heroku/python/` directory in the app in a later PR, however, for now this change: (a) fixes the display of the internal error message, (b) adds more debugging output to the error message so that I can confirm my theory as to the root cause for this app. Towards #1710. GUS-W-17386432. --- CHANGELOG.md | 1 + lib/pip.sh | 2 +- lib/utils.sh | 33 +++++++++++-------- .../.heroku/python/bin/python | 8 +++++ .../python_in_app_source/.python-version | 1 + .../python_in_app_source/requirements.txt | 0 spec/hatchet/checks_spec.rb | 30 +++++++++++++++++ 7 files changed, 61 insertions(+), 14 deletions(-) create mode 100644 spec/fixtures/python_in_app_source/.heroku/python/bin/python create mode 100644 spec/fixtures/python_in_app_source/.python-version create mode 100644 spec/fixtures/python_in_app_source/requirements.txt create mode 100644 spec/hatchet/checks_spec.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index 3672a05b4..e49de62b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Improved the error message shown when pip install fails due to pip rejecting a package with invalid version metadata. ([#1718](https://github.com/heroku/heroku-buildpack-python/pull/1718)) +- Improved the error message shown when the copy of pip bundled in the `ensurepip` module cannot be found. ([#1720](https://github.com/heroku/heroku-buildpack-python/pull/1720)) ## [v270] - 2024-12-10 diff --git a/lib/pip.sh b/lib/pip.sh index 3b26250f6..c6bedb92a 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -15,7 +15,7 @@ function pip::install_pip_setuptools_wheel() { # We use the pip wheel bundled within Python's standard library to install our chosen # pip version, since it's faster than `ensurepip` followed by an upgrade in place. local bundled_pip_module_path - bundled_pip_module_path="$(utils::bundled_pip_module_path "${python_home}")" + bundled_pip_module_path="$(utils::bundled_pip_module_path "${python_home}" "${python_major_version}")" meta_set "pip_version" "${PIP_VERSION}" diff --git a/lib/utils.sh b/lib/utils.sh index 61254de5c..2bcae44c2 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -20,24 +20,31 @@ function utils::get_requirement_version() { # pip version from PyPI, saving us from having to download the usual pip bootstrap script. function utils::bundled_pip_module_path() { local python_home="${1}" + local python_major_version="${2}" - # We have to use a glob since the bundled wheel filename contains the pip version, which - # differs between Python versions. We also have to handle the case where there are multiple - # matching pip wheels, since in some versions of Python (eg 3.9.0) multiple versions of pip - # were accidentally bundled upstream. Note: This implementation relies upon `nullglob` being - # set, which is the case thanks to the `bin/utils` that was run earlier. - local bundled_pip_wheel_list=("${python_home}"/lib/python*/ensurepip/_bundled/pip-*.whl) - local bundled_pip_wheel="${bundled_pip_wheel_list[0]}" - - if [[ -z "${bundled_pip_wheel}" ]]; then - output::error <<-'EOF' - Internal Error: Unable to locate the ensurepip pip wheel file. + local bundled_wheels_dir="${python_home}/lib/python${python_major_version}/ensurepip/_bundled" + + # We have to use a glob since the bundled wheel filename contains the pip version, which differs + # between Python versions. We use compgen to avoid having to set nullglob, since there may be no + # matches in the case of a broken Python install. We also have to handle the case where there are + # multiple matching pip wheels, since in some versions of Python (eg 3.9.0) multiple versions of + # pip were accidentally bundled upstream (we use tail since we want the newest pip version). + if bundled_pip_wheel="$(compgen -G "${bundled_wheels_dir}/pip-*.whl" | tail --lines=1)"; then + # The pip module exists inside the pip wheel (which is a zip file), however, Python can load + # it directly by appending the module name to the zip filename, as though it were a path. + echo "${bundled_pip_wheel}/pip" + else + output::error <<-EOF + Internal Error: Unable to locate the bundled copy of pip. + + The Python buildpack could not locate the copy of pip bundled + inside Python's 'ensurepip' module: + + $(find "${bundled_wheels_dir}/" 2>&1 || find "${python_home}/" -type d 2>&1 || true) EOF meta_set "failure_reason" "bundled-pip-not-found" exit 1 fi - - echo "${bundled_pip_wheel}/pip" } function utils::abort_internal_error() { diff --git a/spec/fixtures/python_in_app_source/.heroku/python/bin/python b/spec/fixtures/python_in_app_source/.heroku/python/bin/python new file mode 100644 index 000000000..04b03ed28 --- /dev/null +++ b/spec/fixtures/python_in_app_source/.heroku/python/bin/python @@ -0,0 +1,8 @@ +#!/usr/bin/env bash + +# This file emulates a Python install having been committed to the app's Git repo. +# For example, by downloading a slug, extracting it, and committing the results. + +set -euo pipefail + +exit 0 diff --git a/spec/fixtures/python_in_app_source/.python-version b/spec/fixtures/python_in_app_source/.python-version new file mode 100644 index 000000000..24ee5b1be --- /dev/null +++ b/spec/fixtures/python_in_app_source/.python-version @@ -0,0 +1 @@ +3.13 diff --git a/spec/fixtures/python_in_app_source/requirements.txt b/spec/fixtures/python_in_app_source/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/hatchet/checks_spec.rb b/spec/hatchet/checks_spec.rb new file mode 100644 index 000000000..d29554821 --- /dev/null +++ b/spec/hatchet/checks_spec.rb @@ -0,0 +1,30 @@ +# frozen_string_literal: true + +require_relative '../spec_helper' + +RSpec.describe 'Buildpack validation checks' do + context 'when the app source contains a broken Python install' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_in_app_source', allow_failure: true) } + + it 'fails detection' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: + remote: ! Internal Error: Unable to locate the bundled copy of pip. + remote: ! + remote: ! The Python buildpack could not locate the copy of pip bundled + remote: ! inside Python's 'ensurepip' module: + remote: ! + remote: ! find: ‘/app/.heroku/python/lib/python3.13/ensurepip/_bundled/’: No such file or directory + remote: ! /app/.heroku/python/ + remote: ! /app/.heroku/python/bin + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end +end From eb29986ac9bf8e9274762cbf321414ba857d1d47 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 12 Dec 2024 16:51:58 +0000 Subject: [PATCH 075/406] Update the Python 3.8 EOL warning sunset date (#1721) Since I've decided to defer Python 3.8 sunset by a month given the upcoming holidays and imminent change moratorium. This will also allow for adding an equivalent warning to the Python CNB, so both can be sunset at the same time. Xref: https://devcenter.heroku.com/changelog-items/2768 https://devcenter.heroku.com/articles/python-support#supported-python-versions GUS-W-17422347. --- CHANGELOG.md | 1 + bin/steps/python | 6 +++--- spec/hatchet/python_update_warning_spec.rb | 6 +++--- spec/hatchet/python_version_spec.rb | 6 +++--- 4 files changed, 10 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e49de62b0..bc7870e0c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated the Python 3.8 EOL warning message with the new sunset date. ([#1721](https://github.com/heroku/heroku-buildpack-python/pull/1721)) - Improved the error message shown when pip install fails due to pip rejecting a package with invalid version metadata. ([#1718](https://github.com/heroku/heroku-buildpack-python/pull/1718)) - Improved the error message shown when the copy of pip bundled in the `ensurepip` module cannot be found. ([#1720](https://github.com/heroku/heroku-buildpack-python/pull/1720)) diff --git a/bin/steps/python b/bin/steps/python index 316bb19ab..7ef72339c 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -78,11 +78,11 @@ if [[ "${python_major_version}" == "3.8" ]]; then output::warning <<-'EOF' Warning: Support for Python 3.8 is ending soon! - Python 3.8 will reach its upstream end-of-life in October 2024, at which - point it will no longer receive security updates: + Python 3.8 reached its upstream end-of-life on 7th October 2024, and so + no longer receives security updates: https://devguide.python.org/versions/#supported-versions - Support for Python 3.8 will be removed from this buildpack on December 4th, 2024. + Support for Python 3.8 will be removed from this buildpack on 7th January 2025. Upgrade to a newer Python version as soon as possible to keep your app secure. See: https://devcenter.heroku.com/articles/python-runtimes diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index 993a07454..2cae6ecae 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -20,11 +20,11 @@ remote: remote: ! Warning: Support for Python 3.8 is ending soon! remote: ! - remote: ! Python 3.8 will reach its upstream end-of-life in October 2024, at which - remote: ! point it will no longer receive security updates: + remote: ! Python 3.8 reached its upstream end-of-life on 7th October 2024, and so + remote: ! no longer receives security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! - remote: ! Support for Python 3.8 will be removed from this buildpack on December 4th, 2024. + remote: ! Support for Python 3.8 will be removed from this buildpack on 7th January 2025. remote: ! remote: ! Upgrade to a newer Python version as soon as possible to keep your app secure. remote: ! See: https://devcenter.heroku.com/articles/python-runtimes diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 28ee14641..e5689bc08 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -89,11 +89,11 @@ remote: remote: ! Warning: Support for Python 3.8 is ending soon! remote: ! - remote: ! Python 3.8 will reach its upstream end-of-life in October 2024, at which - remote: ! point it will no longer receive security updates: + remote: ! Python 3.8 reached its upstream end-of-life on 7th October 2024, and so + remote: ! no longer receives security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! - remote: ! Support for Python 3.8 will be removed from this buildpack on December 4th, 2024. + remote: ! Support for Python 3.8 will be removed from this buildpack on 7th January 2025. remote: ! remote: ! Upgrade to a newer Python version as soon as possible to keep your app secure. remote: ! See: https://devcenter.heroku.com/articles/python-runtimes From e831d7e33db095471f5808c6e9ac8184ac887bd6 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 12 Dec 2024 17:01:02 +0000 Subject: [PATCH 076/406] Prepare release v271 (#1722) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bc7870e0c..fa23c30f6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v271] - 2024-12-12 + - Updated the Python 3.8 EOL warning message with the new sunset date. ([#1721](https://github.com/heroku/heroku-buildpack-python/pull/1721)) - Improved the error message shown when pip install fails due to pip rejecting a package with invalid version metadata. ([#1718](https://github.com/heroku/heroku-buildpack-python/pull/1718)) - Improved the error message shown when the copy of pip bundled in the `ensurepip` module cannot be found. ([#1720](https://github.com/heroku/heroku-buildpack-python/pull/1720)) @@ -1124,7 +1127,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v270...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v271...main +[v271]: https://github.com/heroku/heroku-buildpack-python/compare/v270...v271 [v270]: https://github.com/heroku/heroku-buildpack-python/compare/v269...v270 [v269]: https://github.com/heroku/heroku-buildpack-python/compare/v268...v269 [v268]: https://github.com/heroku/heroku-buildpack-python/compare/v267...v268 From dfd025bab45d274b8e54cf0af0fb74dc79ffdf06 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 13 Dec 2024 12:57:42 +0000 Subject: [PATCH 077/406] Fix Dev Center links to reflect recent article URL changes (#1723) After the recent Dev Center re-org, some of the articles URLs and/or heading anchor link names have changed. GUS-W-17428557. --- CHANGELOG.md | 1 + bin/steps/nltk | 1 - bin/steps/python | 2 +- lib/python_version.sh | 11 ++++++----- spec/hatchet/nltk_spec.rb | 1 - spec/hatchet/pipenv_spec.rb | 2 +- spec/hatchet/python_update_warning_spec.rb | 2 +- spec/hatchet/python_version_spec.rb | 15 ++++++++------- 8 files changed, 18 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fa23c30f6..eae0bf86e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Fixed Dev Center links to reflect recent article URL changes. ([#1723](https://github.com/heroku/heroku-buildpack-python/pull/1723)) ## [v271] - 2024-12-12 diff --git a/bin/steps/nltk b/bin/steps/nltk index ad99493fe..6fc60c7ec 100755 --- a/bin/steps/nltk +++ b/bin/steps/nltk @@ -47,6 +47,5 @@ if is_module_available 'nltk'; then else meta_set "nltk_downloader" "skipped-no-nltk-file" echo " 'nltk.txt' not found, not downloading any corpora" - echo " Learn more: https://devcenter.heroku.com/articles/python-nltk" fi fi diff --git a/bin/steps/python b/bin/steps/python index 7ef72339c..797bc038d 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -20,7 +20,7 @@ if ! curl --output /dev/null --silent --head --fail --retry 3 --retry-connrefuse Error: Python ${python_full_version} isn't available for this stack (${STACK}). For a list of the supported Python versions, see: - https://devcenter.heroku.com/articles/python-support#supported-runtimes + https://devcenter.heroku.com/articles/python-support#supported-python-versions EOF meta_set "failure_reason" "python-version-not-found" exit 1 diff --git a/lib/python_version.sh b/lib/python_version.sh index 161118294..6d008c849 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -299,7 +299,7 @@ function python_version::resolve_python_version() { ${DEFAULT_PYTHON_MAJOR_VERSION} For a list of the supported Python versions, see: - https://devcenter.heroku.com/articles/python-support#supported-runtimes + https://devcenter.heroku.com/articles/python-support#supported-python-versions EOF else output::error <<-EOF @@ -315,7 +315,7 @@ function python_version::resolve_python_version() { version configured via the '${python_version_origin}' file. For a list of the supported Python versions, see: - https://devcenter.heroku.com/articles/python-support#supported-runtimes + https://devcenter.heroku.com/articles/python-support#supported-python-versions EOF fi meta_set "failure_reason" "python-version::eol" @@ -347,11 +347,12 @@ function python_version::resolve_python_version() { Check that this Python version has been officially released, and that the Python buildpack has added support for it: https://devguide.python.org/versions/#supported-versions - https://devcenter.heroku.com/articles/python-support#supported-runtimes + https://devcenter.heroku.com/articles/python-support#supported-python-versions If it has, make sure that you are using the latest version - of this buildpack: - https://devcenter.heroku.com/articles/python-support#checking-the-python-buildpack-version + of this buildpack, and have not pinned to an older release: + https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks + https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references Otherwise, switch to a supported version (such as Python ${DEFAULT_PYTHON_MAJOR_VERSION}) by updating the version configured via the '${python_version_origin}' file. diff --git a/spec/hatchet/nltk_spec.rb b/spec/hatchet/nltk_spec.rb index d61451278..80c2265b1 100644 --- a/spec/hatchet/nltk_spec.rb +++ b/spec/hatchet/nltk_spec.rb @@ -33,7 +33,6 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Downloading NLTK corpora... remote: 'nltk.txt' not found, not downloading any corpora - remote: Learn more: https://devcenter.heroku.com/articles/python-nltk REGEX end end diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 7683592fd..b763e8a7a 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -280,7 +280,7 @@ remote: ! version configured via the 'Pipfile.lock' file. remote: ! remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: remote: ! Push rejected, failed to compile Python app. OUTPUT diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index 2cae6ecae..ede46f2f0 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -54,7 +54,7 @@ remote: ! Error: Python 3.8.0 isn't available for this stack (#{app.stack}). remote: ! remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: remote: ! Push rejected, failed to compile Python app. OUTPUT diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index e5689bc08..43b098231 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -121,7 +121,7 @@ remote: ! Error: Python #{LATEST_PYTHON_3_8} isn't available for this stack (#{app.stack}). remote: ! remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -266,7 +266,7 @@ remote: ! version configured via the '.python-version' file. remote: ! remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -290,11 +290,12 @@ remote: ! Check that this Python version has been officially released, remote: ! and that the Python buildpack has added support for it: remote: ! https://devguide.python.org/versions/#supported-versions - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: ! remote: ! If it has, make sure that you are using the latest version - remote: ! of this buildpack: - remote: ! https://devcenter.heroku.com/articles/python-support#checking-the-python-buildpack-version + remote: ! of this buildpack, and have not pinned to an older release: + remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks + remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references remote: ! remote: ! Otherwise, switch to a supported version (such as Python #{DEFAULT_PYTHON_MAJOR_VERSION}) remote: ! by updating the version configured via the '.python-version' file. @@ -317,7 +318,7 @@ remote: ! Error: Python 3.12.999 isn't available for this stack (#{app.stack}). remote: ! remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -378,7 +379,7 @@ remote: ! version configured via the 'runtime.txt' file. remote: ! remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: remote: ! Push rejected, failed to compile Python app. OUTPUT From f2251775a89cb6ea3348b610275a12184d4d0d73 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 13 Dec 2024 20:56:44 +0000 Subject: [PATCH 078/406] Add metrics for the existence of a uv lockfile (#1725) To help gauge how often a uv lockfile exists. (Either from someone trying to use uv and the build failing, or when a third-party buildpack is used to export the uv lockfile to a requirements file etc). Towards #1616. GUS-W-17431743. --- CHANGELOG.md | 1 + bin/report | 1 + lib/package_manager.sh | 9 +++++++++ spec/hatchet/package_manager_spec.rb | 5 +++++ 4 files changed, 16 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index eae0bf86e..be0f589cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Fixed Dev Center links to reflect recent article URL changes. ([#1723](https://github.com/heroku/heroku-buildpack-python/pull/1723)) +- Added metrics for the existence of a uv lockfile. ([#1725](https://github.com/heroku/heroku-buildpack-python/pull/1725)) ## [v271] - 2024-12-12 diff --git a/bin/report b/bin/report index dcba72ef2..eb354bdbc 100755 --- a/bin/report +++ b/bin/report @@ -94,6 +94,7 @@ ALL_OTHER_FIELDS=( setup_py_only sqlite_install_duration total_duration + uv_lockfile ) for field in "${STRING_FIELDS[@]}"; do diff --git a/lib/package_manager.sh b/lib/package_manager.sh index eee49ab42..cc0bb6afc 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -59,6 +59,10 @@ function package_manager::determine_package_manager() { meta_set "setup_py_only" "false" fi + if [[ -f "${build_dir}/uv.lock" ]]; then + meta_set "uv_lockfile" "true" + fi + local num_package_managers_found=${#package_managers_found[@]} case "${num_package_managers_found}" in @@ -87,6 +91,11 @@ function package_manager::determine_package_manager() { Otherwise, add a package manager file to your app. If your app has no dependencies, then create an empty 'requirements.txt' file. + If you would like to see support for the package manager uv, + please vote and comment on these GitHub issues: + https://github.com/heroku/heroku-buildpack-python/issues/1616 + https://github.com/heroku/roadmap/issues/323 + For help with using Python on Heroku, see: https://devcenter.heroku.com/articles/getting-started-with-python https://devcenter.heroku.com/articles/python-support diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index e390d62e8..fb07813f2 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -31,6 +31,11 @@ remote: ! Otherwise, add a package manager file to your app. If your app has remote: ! no dependencies, then create an empty 'requirements.txt' file. remote: ! + remote: ! If you would like to see support for the package manager uv, + remote: ! please vote and comment on these GitHub issues: + remote: ! https://github.com/heroku/heroku-buildpack-python/issues/1616 + remote: ! https://github.com/heroku/roadmap/issues/323 + remote: ! remote: ! For help with using Python on Heroku, see: remote: ! https://devcenter.heroku.com/articles/getting-started-with-python remote: ! https://devcenter.heroku.com/articles/python-support From 0d4221847b72879529afbd8affa33bf4ebd7e7e5 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 13 Dec 2024 21:11:22 +0000 Subject: [PATCH 079/406] Add build environment validation checks (#1724) - Adds a warning if the Python buildpack has been run multiple times in the same build. - Adds a warning if an existing `.heroku/python/` directory is found in the app source. - Improves the error message shown if the buildpack is used on an unsupported stack. Previously the build would fail with a curl download error depending on the availability of assets on S3. This scenario can only happen outside of Heroku, or in an old buildpack is used with a brand new stack (eg for Heroku-26). The two warnings are the first step towards #1704 and #1710, and will be turned into errors in January 2025. Towards #1704 and #1710. GUS-W-17386432. GUS-W-17309709. --- CHANGELOG.md | 3 + bin/compile | 14 ++++- bin/report | 2 + lib/cache.sh | 2 +- lib/checks.sh | 116 ++++++++++++++++++++++++++++++++++++ spec/hatchet/checks_spec.rb | 63 ++++++++++++++++++++ 6 files changed, 197 insertions(+), 3 deletions(-) create mode 100644 lib/checks.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index be0f589cd..2cb5c1599 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] +- Added a warning if the Python buildpack has been run multiple times in the same build. In January 2025 this warning will be made an error. ([#1724](https://github.com/heroku/heroku-buildpack-python/pull/1724)) +- Added a warning if an existing `.heroku/python/` directory is found in the app source. In January 2025 this warning will be made an error. ([#1724](https://github.com/heroku/heroku-buildpack-python/pull/1724)) +- Improved the error message shown if the buildpack is used on an unsupported stack. ([#1724](https://github.com/heroku/heroku-buildpack-python/pull/1724)) - Fixed Dev Center links to reflect recent article URL changes. ([#1723](https://github.com/heroku/heroku-buildpack-python/pull/1723)) - Added metrics for the existence of a uv lockfile. ([#1725](https://github.com/heroku/heroku-buildpack-python/pull/1725)) diff --git a/bin/compile b/bin/compile index 495d8c20d..7edf8a44c 100755 --- a/bin/compile +++ b/bin/compile @@ -21,6 +21,7 @@ BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) source "${BUILDPACK_DIR}/bin/utils" source "${BUILDPACK_DIR}/lib/utils.sh" source "${BUILDPACK_DIR}/lib/cache.sh" +source "${BUILDPACK_DIR}/lib/checks.sh" source "${BUILDPACK_DIR}/lib/hooks.sh" source "${BUILDPACK_DIR}/lib/metadata.sh" source "${BUILDPACK_DIR}/lib/output.sh" @@ -32,10 +33,15 @@ source "${BUILDPACK_DIR}/lib/poetry.sh" compile_start_time=$(nowms) -# Initialise metadata store. +# Initialise the buildpack metadata store. +# This is used to track state across builds (for cache invalidation and messaging when build +# configuration changes) and also so that `bin/report` can generate the build report. meta_init "${CACHE_DIR}" "python" meta_setup +checks::ensure_supported_stack "${STACK:?Required env var STACK is not set}" +checks::warn_if_duplicate_python_buildpack "${BUILD_DIR}" + # Prepend proper path for old-school virtualenv hackery. # This may not be necessary. export PATH=:/usr/local/bin:$PATH @@ -103,6 +109,10 @@ cd "$BUILD_DIR" # Runs a `bin/pre_compile` script if found in the app source, allowing build customisation. hooks::run_hook "pre_compile" +# This check must be after the pre_compile hook, so that we can check not only the original +# app source, but also that the hook hasn't written to '.heroku/python/' either. +checks::warn_if_existing_python_dir_present "${BUILD_DIR}" + package_manager="$(package_manager::determine_package_manager "${BUILD_DIR}")" meta_set "package_manager" "${package_manager}" @@ -135,7 +145,7 @@ python_major_version="${python_full_version%.*}" meta_set "python_version" "${python_full_version}" meta_set "python_version_major" "${python_major_version}" -cache::restore "${BUILD_DIR}" "${CACHE_DIR}" "${STACK:?}" "${cached_python_full_version}" "${python_full_version}" "${package_manager}" +cache::restore "${BUILD_DIR}" "${CACHE_DIR}" "${STACK}" "${cached_python_full_version}" "${python_full_version}" "${package_manager}" # The directory for the .profile.d scripts. mkdir -p "$(dirname "$PROFILE_PATH")" diff --git a/bin/report b/bin/report index eb354bdbc..54325611d 100755 --- a/bin/report +++ b/bin/report @@ -82,6 +82,8 @@ ALL_OTHER_FIELDS=( cache_save_duration dependencies_install_duration django_collectstatic_duration + duplicate_python_buildpack + existing_python_dir nltk_downloader_duration package_manager_install_duration pipenv_has_lockfile diff --git a/lib/cache.sh b/lib/cache.sh index 9d9415ba3..ef62d582f 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -138,7 +138,7 @@ function cache::restore() { mkdir -p "${build_dir}/.heroku" # NB: For now this has to handle files already existing in build_dir since some apps accidentally - # run the Python buildpack twice. TODO: Add an explicit check/error for duplicate buildpacks. + # run the Python buildpack twice. TODO: Refactor this once duplicate buildpacks become an error. # TODO: Investigate why errors are ignored and ideally stop doing so. # TODO: Compare the performance of moving the directory vs copying files. cp -R "${cache_dir}/.heroku/python" "${build_dir}/.heroku/" &>/dev/null || true diff --git a/lib/checks.sh b/lib/checks.sh new file mode 100644 index 000000000..9129244e7 --- /dev/null +++ b/lib/checks.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash + +function checks::ensure_supported_stack() { + local stack="${1}" + + case "${stack}" in + heroku-20 | heroku-22 | heroku-24) + return 0 + ;; + cedar* | heroku-16 | heroku-18) + # This error will only ever be seen on non-Heroku environments, since the + # Heroku build system rejects builds using EOL stacks. + output::error <<-EOF + Error: The '${stack}' stack is no longer supported. + + This buildpack no longer supports the '${stack}' stack since it has + reached its end-of-life: + https://devcenter.heroku.com/articles/stack#stack-support-details-for-apps-using-classic-buildpacks + + Upgrade to a newer stack to continue using this buildpack. + EOF + meta_set "failure_reason" "stack::eol" + exit 1 + ;; + *) + output::error <<-EOF + Error: The '${stack}' stack is not recognised. + + This buildpack does not recognise or support the '${stack}' stack. + + If '${stack}' is a valid stack, make sure that you are using the latest + version of this buildpack and have not pinned to an older release: + https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks + https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references + EOF + meta_set "failure_reason" "stack::unknown" + exit 1 + ;; + esac +} + +# TODO: Turn this into an error in January 2025. +function checks::warn_if_duplicate_python_buildpack() { + local build_dir="${1}" + + # The check for the `PYTHONHOME` env var prevents this warning triggering in the case + # where the Python install was committed to the Git repo (which will be handled later). + # (The env var can only have come from the `export` file of an earlier buildpack, + # since app provided config vars haven't been exported to the environment here.) + if [[ -f "${build_dir}/.heroku/python/bin/python" && -v PYTHONHOME ]]; then + output::warning <<-EOF + Warning: The Python buildpack has already been run this build. + + An existing Python installation was found in the build directory + from a buildpack run earlier in the build. + + This normally means there are duplicate Python buildpacks set + on your app, which is not supported, can cause errors and + slow down builds. + + Check the buildpacks set on your app and remove any duplicate + Python buildpack entries: + https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks + https://devcenter.heroku.com/articles/managing-buildpacks#remove-classic-buildpacks + + If you have a use-case that requires duplicate buildpacks, + please comment on: + https://github.com/heroku/heroku-buildpack-python/issues/1704 + + In January 2025 this warning will be made an error. + EOF + meta_set "duplicate_python_buildpack" "true" + # shellcheck disable=SC2034 # This is used below until we make this check an error. + DUPLICATE_PYTHON_BUILDPACK=1 + fi +} + +# TODO: Turn this into an error in January 2025. +function checks::warn_if_existing_python_dir_present() { + local build_dir="${1}" + + # Avoid warning twice in the case of duplicate buildpacks. + # TODO: Remove this once `warn_if_duplicate_python_buildpack` becomes an error. + if [[ -v DUPLICATE_PYTHON_BUILDPACK ]]; then + return 0 + fi + + # We use `-e` here to catch the case where `python` is a file rather than a directory. + if [[ -e "${build_dir}/.heroku/python" ]]; then + output::warning <<-EOF + Warning: Existing '.heroku/python/' directory found. + + Your app's source code contains an existing directory named + '.heroku/python/', which is where the Python buildpack needs + to install its files. This existing directory contains: + + $(find .heroku/python/ -maxdepth 2 || true) + + Writing to internal locations used by the Python buildpack + is not supported and can cause unexpected errors. + + If you have committed a '.heroku/python/' directory to your + Git repo, you must delete it or use a different location. + + Otherwise, check that an earlier buildpack or 'bin/pre_compile' + hook has not created this directory. + + If you have a use-case that requires writing to this location, + please comment on: + https://github.com/heroku/heroku-buildpack-python/issues/1704 + + In January 2025 this warning will be made an error. + EOF + meta_set "existing_python_dir" "true" + fi +} diff --git a/spec/hatchet/checks_spec.rb b/spec/hatchet/checks_spec.rb index d29554821..ed55fc139 100644 --- a/spec/hatchet/checks_spec.rb +++ b/spec/hatchet/checks_spec.rb @@ -3,6 +3,43 @@ require_relative '../spec_helper' RSpec.describe 'Buildpack validation checks' do + context 'when there are duplicate Python buildpacks set on the app' do + let(:buildpacks) { %i[default default] } + let(:app) { Hatchet::Runner.new("spec/fixtures/python_#{DEFAULT_PYTHON_MAJOR_VERSION}", buildpacks:) } + + it 'fails detection' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: + remote: ! Warning: The Python buildpack has already been run this build. + remote: ! + remote: ! An existing Python installation was found in the build directory + remote: ! from a buildpack run earlier in the build. + remote: ! + remote: ! This normally means there are duplicate Python buildpacks set + remote: ! on your app, which is not supported, can cause errors and + remote: ! slow down builds. + remote: ! + remote: ! Check the buildpacks set on your app and remove any duplicate + remote: ! Python buildpack entries: + remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks + remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#remove-classic-buildpacks + remote: ! + remote: ! If you have a use-case that requires duplicate buildpacks, + remote: ! please comment on: + remote: ! https://github.com/heroku/heroku-buildpack-python/issues/1704 + remote: ! + remote: ! In January 2025 this warning will be made an error. + remote: + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + remote: -----> Restoring cache + remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} + OUTPUT + end + end + end + context 'when the app source contains a broken Python install' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_in_app_source', allow_failure: true) } @@ -10,6 +47,32 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected + remote: + remote: ! Warning: Existing '.heroku/python/' directory found. + remote: ! + remote: ! Your app's source code contains an existing directory named + remote: ! '.heroku/python/', which is where the Python buildpack needs + remote: ! to install its files. This existing directory contains: + remote: ! + remote: ! .heroku/python/ + remote: ! .heroku/python/bin + remote: ! .heroku/python/bin/python + remote: ! + remote: ! Writing to internal locations used by the Python buildpack + remote: ! is not supported and can cause unexpected errors. + remote: ! + remote: ! If you have committed a '.heroku/python/' directory to your + remote: ! Git repo, you must delete it or use a different location. + remote: ! + remote: ! Otherwise, check that an earlier buildpack or 'bin/pre_compile' + remote: ! hook has not created this directory. + remote: ! + remote: ! If you have a use-case that requires writing to this location, + remote: ! please comment on: + remote: ! https://github.com/heroku/heroku-buildpack-python/issues/1704 + remote: ! + remote: ! In January 2025 this warning will be made an error. + remote: remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: From 873238369226da4ac02d7d68e89df8c92ad618dd Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Fri, 13 Dec 2024 21:20:40 +0000 Subject: [PATCH 080/406] Prepare release v272 (#1726) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2cb5c1599..aed33bbe0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v272] - 2024-12-13 + - Added a warning if the Python buildpack has been run multiple times in the same build. In January 2025 this warning will be made an error. ([#1724](https://github.com/heroku/heroku-buildpack-python/pull/1724)) - Added a warning if an existing `.heroku/python/` directory is found in the app source. In January 2025 this warning will be made an error. ([#1724](https://github.com/heroku/heroku-buildpack-python/pull/1724)) - Improved the error message shown if the buildpack is used on an unsupported stack. ([#1724](https://github.com/heroku/heroku-buildpack-python/pull/1724)) @@ -1132,7 +1135,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v271...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v272...main +[v272]: https://github.com/heroku/heroku-buildpack-python/compare/v271...v272 [v271]: https://github.com/heroku/heroku-buildpack-python/compare/v270...v271 [v270]: https://github.com/heroku/heroku-buildpack-python/compare/v269...v270 [v269]: https://github.com/heroku/heroku-buildpack-python/compare/v268...v269 From 4f4c3c9dd6958dd0d8df37526b6377ca8ef3f70d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 1 Jan 2025 13:06:55 +0000 Subject: [PATCH 081/406] Bump the ruby-dependencies group with 3 updates (#1727) * Bump the ruby-dependencies group with 3 updates Bumps the ruby-dependencies group with 3 updates: [logger](https://github.com/ruby/logger), [rubocop](https://github.com/rubocop/rubocop) and [rubocop-rspec](https://github.com/rubocop/rubocop-rspec). Updates `logger` from 1.6.2 to 1.6.4 - [Release notes](https://github.com/ruby/logger/releases) - [Commits](https://github.com/ruby/logger/compare/v1.6.2...v1.6.4) Updates `rubocop` from 1.69.1 to 1.69.2 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.69.1...v1.69.2) Updates `rubocop-rspec` from 3.2.0 to 3.3.0 - [Release notes](https://github.com/rubocop/rubocop-rspec/releases) - [Changelog](https://github.com/rubocop/rubocop-rspec/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop-rspec/compare/v3.2.0...v3.3.0) --- updated-dependencies: - dependency-name: logger dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rubocop dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rubocop-rspec dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] * Fix pipenv tests after new certifi release Since the new version number is wider, which caused the pip table rendering to change. --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- Gemfile.lock | 14 +++++++------- spec/hatchet/pipenv_spec.rb | 2 +- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 4162ad6ba..9c610e2f0 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -19,9 +19,9 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.9.0) + json (2.9.1) language_server-protocol (3.17.0.3) - logger (1.6.2) + logger (1.6.4) moneta (1.0.0) multi_json (1.15.0) parallel (1.26.3) @@ -38,7 +38,7 @@ GEM racc (1.8.1) rainbow (3.1.1) rate_throttle_client (0.1.2) - regexp_parser (2.9.3) + regexp_parser (2.10.0) rrrretry (1.0.0) rspec-core (3.13.2) rspec-support (~> 3.13.0) @@ -48,7 +48,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.2) - rubocop (1.69.1) + rubocop (1.69.2) json (~> 2.3) language_server-protocol (>= 3.17.0) parallel (~> 1.10) @@ -58,14 +58,14 @@ GEM rubocop-ast (>= 1.36.2, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 4.0) - rubocop-ast (1.36.2) + rubocop-ast (1.37.0) parser (>= 3.3.1.0) - rubocop-rspec (3.2.0) + rubocop-rspec (3.3.0) rubocop (~> 1.61) ruby-progressbar (1.13.0) thor (1.3.2) threaded (0.0.4) - unicode-display_width (3.1.2) + unicode-display_width (3.1.3) unicode-emoji (~> 4.0, >= 4.0.4) unicode-emoji (4.0.4) webrick (1.9.1) diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index b763e8a7a..f6d3ba1ab 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -36,7 +36,7 @@ remote: '/app/.heroku/python/lib/python3.13/site-packages'\\] remote: remote: Package Version - remote: ----------------- --------- + remote: ----------------- -+ remote: certifi .+ remote: distlib .+ remote: filelock .+ From f686af68753614d95c738921bc858a91d303ca36 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 2 Jan 2025 16:46:10 +0000 Subject: [PATCH 082/406] Include dotfiles in error message file listings (#1728) The error messages shown when buildpack detection fails (or when detection passes, but no required package manager files are found later) include a file listing of the project to help with debugging. (e.g. to make misspellings or having forgotten to commit files more obvious.) Previously these file listings didn't include dotfiles, which meant not all project files would be shown. This is particularly relevant given that the the newly supported `.python-version` file is a dotfile. GUS-W-17524155. --- CHANGELOG.md | 1 + bin/detect | 2 +- lib/package_manager.sh | 2 +- spec/fixtures/no_python_project_files/.example-dotfile | 0 spec/fixtures/pyproject_toml_only/.example-dotfile | 0 spec/hatchet/detect_spec.rb | 1 + spec/hatchet/package_manager_spec.rb | 1 + 7 files changed, 5 insertions(+), 2 deletions(-) create mode 100644 spec/fixtures/no_python_project_files/.example-dotfile create mode 100644 spec/fixtures/pyproject_toml_only/.example-dotfile diff --git a/CHANGELOG.md b/CHANGELOG.md index aed33bbe0..a9dddf0f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Improved the file listing in the error messages shown when buildpack detection fails or when no Python package manager files are found. ([#1728](https://github.com/heroku/heroku-buildpack-python/pull/1728)) ## [v272] - 2024-12-13 diff --git a/bin/detect b/bin/detect index 4b5a535eb..35c325e42 100755 --- a/bin/detect +++ b/bin/detect @@ -55,7 +55,7 @@ directory of its source code. Currently the root directory of your app contains: -$(ls -1 --indicator-style=slash "${BUILD_DIR}" || true) +$(ls -1A --indicator-style=slash "${BUILD_DIR}" || true) If your app already has a package manager file, check that it: diff --git a/lib/package_manager.sh b/lib/package_manager.sh index cc0bb6afc..97a883136 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -80,7 +80,7 @@ function package_manager::determine_package_manager() { Currently the root directory of your app contains: - $(ls -1 --indicator-style=slash "${build_dir}" || true) + $(ls -1A --indicator-style=slash "${build_dir}" || true) If your app already has a package manager file, check that it: diff --git a/spec/fixtures/no_python_project_files/.example-dotfile b/spec/fixtures/no_python_project_files/.example-dotfile new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/pyproject_toml_only/.example-dotfile b/spec/fixtures/pyproject_toml_only/.example-dotfile new file mode 100644 index 000000000..e69de29bb diff --git a/spec/hatchet/detect_spec.rb b/spec/hatchet/detect_spec.rb index 044b0010a..dd99d8358 100644 --- a/spec/hatchet/detect_spec.rb +++ b/spec/hatchet/detect_spec.rb @@ -23,6 +23,7 @@ remote: ! remote: ! Currently the root directory of your app contains: remote: ! + remote: ! .example-dotfile remote: ! README.md remote: ! subdir/ remote: ! diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index fb07813f2..1b85b2bb1 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -19,6 +19,7 @@ remote: ! remote: ! Currently the root directory of your app contains: remote: ! + remote: ! .example-dotfile remote: ! pyproject.toml remote: ! subdir/ remote: ! From 23479e682c7a4667c3cb961e23b99c9340f5a972 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 3 Jan 2025 15:26:33 +0000 Subject: [PATCH 083/406] Expand buildpack detection known file list (#1729) This adds more Python project related file and directory names to the list recognised by buildpack detection. Such apps will still fail to build successfully (since they are missing a package manager file), but will now be shown a more helpful error message during compile, rather than the generic: "No default language could be detected for this app" The list is based on builds logs analysis of builds that filed to pass detection, plus builds that passed detection but didn't have a valid package manager file. (Possible since the build logs error message includes a file listing of the root directory of the app.) GUS-W-17530056. --- CHANGELOG.md | 1 + bin/detect | 19 +++++++++++++++++-- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a9dddf0f7..7de187e13 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Added more Python project related file and directory names to the list recognised by buildpack detection. ([#1729](https://github.com/heroku/heroku-buildpack-python/pull/1729)) - Improved the file listing in the error messages shown when buildpack detection fails or when no Python package manager files are found. ([#1728](https://github.com/heroku/heroku-buildpack-python/pull/1728)) ## [v272] - 2024-12-13 diff --git a/bin/detect b/bin/detect index 35c325e42..1317d6b47 100755 --- a/bin/detect +++ b/bin/detect @@ -20,6 +20,7 @@ source "${BUILDPACK_DIR}/lib/output.sh" # allowing us to show a helpful error message during the build phase. KNOWN_PYTHON_PROJECT_FILES=( .python-version + __init__.py app.py main.py manage.py @@ -30,13 +31,27 @@ KNOWN_PYTHON_PROJECT_FILES=( pyproject.toml requirements.txt runtime.txt + server.py setup.cfg setup.py uv.lock + # Commonly seen misspellings of requirements.txt. (Which occur since pip doesn't + # create/manage requirements files itself, so the filenames are manually typed.) + requirement.txt + Requirements.txt + requirements.text + requirements.txt.txt + requirments.txt + # Whilst virtual environments shouldn't be committed to Git (and so shouldn't + # normally be present during the build), they are often present for beginner + # Python apps that are missing all of the other Python related files above. + .venv/ + venv/ ) -for filename in "${KNOWN_PYTHON_PROJECT_FILES[@]}"; do - if [[ -f "${BUILD_DIR}/${filename}" ]]; then +for filepath in "${KNOWN_PYTHON_PROJECT_FILES[@]}"; do + # Using -e since we need to check for both files and directories. + if [[ -e "${BUILD_DIR}/${filepath}" ]]; then echo "Python" exit 0 fi From 9a649bc0cb47673b18d3296817b3f5a45d1e9cb3 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Fri, 3 Jan 2025 16:05:09 +0000 Subject: [PATCH 084/406] Prepare release v273 (#1730) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7de187e13..76c5e5834 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v273] - 2025-01-03 + - Added more Python project related file and directory names to the list recognised by buildpack detection. ([#1729](https://github.com/heroku/heroku-buildpack-python/pull/1729)) - Improved the file listing in the error messages shown when buildpack detection fails or when no Python package manager files are found. ([#1728](https://github.com/heroku/heroku-buildpack-python/pull/1728)) @@ -1137,7 +1140,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v272...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v273...main +[v273]: https://github.com/heroku/heroku-buildpack-python/compare/v272...v273 [v272]: https://github.com/heroku/heroku-buildpack-python/compare/v271...v272 [v271]: https://github.com/heroku/heroku-buildpack-python/compare/v270...v271 [v270]: https://github.com/heroku/heroku-buildpack-python/compare/v269...v270 From 009880b815adc2332c4aadbdfd3e8895112be19b Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 8 Jan 2025 12:07:41 +0000 Subject: [PATCH 085/406] Remove Python 3.8 support + deprecate Python 3.9 (#1732) Python 3.8 reached upstream EOL on 7th October 2024: https://devguide.python.org/versions/#supported-versions The Python version support policy is that supported versions follows the upstream EOL lifecycle: https://devcenter.heroku.com/articles/python-support#python-version-support-policy As such, the buildpack has been showing a deprecation warning with a scheduled removal date since December 2023: - https://github.com/heroku/heroku-buildpack-python/pull/1515 - https://github.com/heroku/heroku-buildpack-python/pull/1721 - https://devcenter.heroku.com/changelog-items/2768 (Plus Python 3.8 binaries have never been available for Heroku-22 or Heroku-24, meaning it was only ever supported on Heroku-20 and older.) Dropping support for Python 3.8 also unblocks upgrading to Poetry v2 (which has already dropped 3.8 support). Apps using Python 3.8 that aren't able to upgrade immediately will need to pin to an older buildpack version temporarily (which will work until Heroku-20 EOLs). A deprecation warning has now also been added for Python 3.9. GUS-W-17472311. GUS-W-14846945. --- .github/workflows/build_python_runtime.yml | 6 +- CHANGELOG.md | 3 + README.md | 1 - bin/compile | 2 +- bin/steps/python | 22 ++-- builds/build_python_runtime.sh | 24 ++-- lib/pip.sh | 2 +- lib/python_version.sh | 47 ++++---- .../pipenv_python_version_eol/Pipfile | 2 +- .../pipenv_python_version_eol/Pipfile.lock | 4 +- spec/fixtures/python_3.8/.python-version | 2 - spec/fixtures/python_3.8/requirements.txt | 2 - .../python_version_eol/.python-version | 2 +- .../python_version_outdated/.python-version | 2 +- .../runtime.txt | 2 +- spec/hatchet/pipenv_spec.rb | 30 +++-- spec/hatchet/poetry_spec.rb | 16 +++ spec/hatchet/python_update_warning_spec.rb | 88 ++++++-------- spec/hatchet/python_version_spec.rb | 114 +++++++----------- spec/spec_helper.rb | 1 - 20 files changed, 173 insertions(+), 199 deletions(-) delete mode 100644 spec/fixtures/python_3.8/.python-version delete mode 100644 spec/fixtures/python_3.8/requirements.txt diff --git a/.github/workflows/build_python_runtime.yml b/.github/workflows/build_python_runtime.yml index 9b90fc052..3e8d5dc73 100644 --- a/.github/workflows/build_python_runtime.yml +++ b/.github/workflows/build_python_runtime.yml @@ -58,8 +58,7 @@ jobs: run: aws s3 sync ./upload "s3://${S3_BUCKET}" heroku-22: - # On Heroku-22 we only support Python 3.9+. - if: inputs.stack == 'heroku-22' || (inputs.stack == 'auto' && !startsWith(inputs.python_version,'3.8.')) + if: inputs.stack == 'heroku-22' || inputs.stack == 'auto' runs-on: pub-hk-ubuntu-24.04-xlarge env: STACK_VERSION: "22" @@ -80,8 +79,7 @@ jobs: run: aws s3 sync ./upload "s3://${S3_BUCKET}" heroku-24: - # On Heroku-24 we only support Python 3.9+. - if: inputs.stack == 'heroku-24' || (inputs.stack == 'auto' && !startsWith(inputs.python_version,'3.8.')) + if: inputs.stack == 'heroku-24' || inputs.stack == 'auto' strategy: fail-fast: false matrix: diff --git a/CHANGELOG.md b/CHANGELOG.md index 76c5e5834..0c5e5c2f9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] +- Added a deprecation warning for Python 3.9. ([#1732](https://github.com/heroku/heroku-buildpack-python/pull/1732)) +- Removed support for Python 3.8. ([#1732](https://github.com/heroku/heroku-buildpack-python/pull/1732)) +- Improved the error messages shown for EOL or unrecognised major Python versions. ([#1732](https://github.com/heroku/heroku-buildpack-python/pull/1732)) ## [v273] - 2025-01-03 diff --git a/README.md b/README.md index 34684120f..3c76ddc14 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,6 @@ The supported Python versions are: These Python versions are deprecated on Heroku: - Python 3.9 -- Python 3.8 (only available on the [Heroku-20](https://devcenter.heroku.com/articles/heroku-20-stack) stack) Python versions older than those listed above are no longer supported, since they have reached end-of-life [upstream](https://devguide.python.org/versions/#supported-versions). diff --git a/bin/compile b/bin/compile index 7edf8a44c..08232b5e7 100755 --- a/bin/compile +++ b/bin/compile @@ -194,7 +194,7 @@ meta_time "package_manager_install_duration" "${package_manager_install_start_ti # Installs the sqlite3 dev headers and sqlite3 binary but not the # libsqlite3-0 library since that exists in the base image. # We skip this step on Python 3.13 or when using Poetry, as a first step towards removing this feature. -if [[ "${python_major_version}" == +(3.8|3.9|3.10|3.11|3.12) && "${package_manager}" != "poetry" ]]; then +if [[ "${python_major_version}" == +(3.9|3.10|3.11|3.12) && "${package_manager}" != "poetry" ]]; then install_sqlite_start_time=$(nowms) source "${BUILDPACK_DIR}/bin/steps/sqlite3" buildpack_sqlite3_install diff --git a/bin/steps/python b/bin/steps/python index 797bc038d..b345d5cd8 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -14,7 +14,7 @@ PYTHON_URL="${S3_BASE_URL}/python-${python_full_version}-ubuntu-${UBUNTU_VERSION # 1. It's a Python major version we've deprecated and so is only available on older stacks (i.e: Python 3.8). # 2. If an exact Python version was requested and the patch version doesn't exist (e.g. 3.12.999). # 3. The user has pinned to an older buildpack version and the S3 bucket location or layout has changed since. -# TODO: Update this message to be more specific once Python 3.8 support is dropped. +# TODO: Update this message to be more specific now that Python 3.8 support has been removed, and so (1) can no longer occur. if ! curl --output /dev/null --silent --head --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}"; then output::error <<-EOF Error: Python ${python_full_version} isn't available for this stack (${STACK}). @@ -74,18 +74,22 @@ function warn_if_patch_update_available() { # We wait until now to display outdated Python version warnings, since we only want to show them # if there weren't any errors with the version to avoid adding noise to the error messages. # TODO: Move this into lib/ as part of the warnings refactor. -if [[ "${python_major_version}" == "3.8" ]]; then - output::warning <<-'EOF' - Warning: Support for Python 3.8 is ending soon! +if [[ "${python_major_version}" == "3.9" ]]; then + output::warning <<-EOF + Warning: Support for Python 3.9 is ending soon! - Python 3.8 reached its upstream end-of-life on 7th October 2024, and so - no longer receives security updates: + Python 3.9 will reach its upstream end-of-life in October 2025, + at which point it will no longer receive security updates: https://devguide.python.org/versions/#supported-versions - Support for Python 3.8 will be removed from this buildpack on 7th January 2025. + As such, support for Python 3.9 will be removed from this + buildpack on 7th January 2026. - Upgrade to a newer Python version as soon as possible to keep your app secure. - See: https://devcenter.heroku.com/articles/python-runtimes + Upgrade to a newer Python version as soon as possible, by + changing the version in your ${python_version_origin} file. + + For more information, see: + https://devcenter.heroku.com/articles/python-support#supported-python-versions EOF fi diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index 3da58a09c..0f442c6ff 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -21,7 +21,7 @@ function abort() { } case "${STACK:?}" in - heroku-22 | heroku-24) + heroku-20 | heroku-22 | heroku-24) SUPPORTED_PYTHON_VERSIONS=( "3.9" "3.10" @@ -30,16 +30,6 @@ case "${STACK:?}" in "3.13" ) ;; - heroku-20) - SUPPORTED_PYTHON_VERSIONS=( - "3.8" - "3.9" - "3.10" - "3.11" - "3.12" - "3.13" - ) - ;; *) abort "Unsupported stack '${STACK}'!" ;; @@ -63,7 +53,7 @@ case "${PYTHON_MAJOR_VERSION}" in # https://keybase.io/pablogsal/ GPG_KEY_FINGERPRINT='A035C8C19219BA821ECEA86B64E628F8D684696D' ;; - 3.8 | 3.9) + 3.9) # https://keybase.io/ambv/ GPG_KEY_FINGERPRINT='E3FF2839C048B25C084DEBE9B26995E310250568' ;; @@ -122,7 +112,7 @@ CONFIGURE_OPTS=( "--with-system-expat" ) -if [[ "${PYTHON_MAJOR_VERSION}" != +(3.8|3.9) ]]; then +if [[ "${PYTHON_MAJOR_VERSION}" != +(3.9) ]]; then CONFIGURE_OPTS+=( # Shared builds are beneficial for a number of reasons: # - Reduces the size of the build, since it avoids the duplication between @@ -147,7 +137,7 @@ if [[ "${PYTHON_MAJOR_VERSION}" != +(3.8|3.9) ]]; then ) fi -if [[ "${PYTHON_MAJOR_VERSION}" != +(3.8|3.9|3.10) ]]; then +if [[ "${PYTHON_MAJOR_VERSION}" != +(3.9|3.10) ]]; then CONFIGURE_OPTS+=( # Skip building the test modules, since we remove them after the build anyway. # This feature was added in Python 3.10+, however it wasn't until Python 3.11 @@ -170,7 +160,7 @@ fi # - https://github.com/docker-library/python/issues/810 # We only use `dpkg-buildflags` for Python versions where we build in shared mode (Python 3.9+), # since some of the options it enables interferes with the stripping of static libraries. -if [[ "${PYTHON_MAJOR_VERSION}" == +(3.8|3.9) ]]; then +if [[ "${PYTHON_MAJOR_VERSION}" == +(3.9) ]]; then EXTRA_CFLAGS='' LDFLAGS='-Wl,--strip-all' else @@ -182,7 +172,7 @@ CPU_COUNT="$(nproc)" make -j "${CPU_COUNT}" "EXTRA_CFLAGS=${EXTRA_CFLAGS}" "LDFLAGS=${LDFLAGS}" make install -if [[ "${PYTHON_MAJOR_VERSION}" == +(3.8|3.9) ]]; then +if [[ "${PYTHON_MAJOR_VERSION}" == +(3.9) ]]; then # On older versions of Python we're still building the static library, which has to be # manually stripped since the linker stripping enabled in LDFLAGS doesn't cover them. # We're using `--strip-unneeded` since `--strip-all` would remove the `.symtab` section @@ -233,7 +223,7 @@ LD_LIBRARY_PATH="${SRC_DIR}" "${SRC_DIR}/python" -m compileall -f --invalidation # (e.g. `python -m pydoc`) if needed. rm "${INSTALL_DIR}"/bin/{idle,pydoc}* # The 2to3 module and entrypoint was removed from the stdlib in Python 3.13. -if [[ "${PYTHON_MAJOR_VERSION}" == +(3.8|3.9|3.10|3.11|3.12) ]]; then +if [[ "${PYTHON_MAJOR_VERSION}" == +(3.9|3.10|3.11|3.12) ]]; then rm "${INSTALL_DIR}"/bin/2to3* fi diff --git a/lib/pip.sh b/lib/pip.sh index c6bedb92a..929063fb4 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -31,7 +31,7 @@ function pip::install_pip_setuptools_wheel() { # them installed. # - Most of the Python ecosystem has stopped installing them for Python 3.12+ already. # See the Python CNB's removal for more details: https://github.com/heroku/buildpacks-python/pull/243 - if [[ "${python_major_version}" == +(3.8|3.9|3.10|3.11|3.12) ]]; then + if [[ "${python_major_version}" == +(3.9|3.10|3.11|3.12) ]]; then meta_set "setuptools_version" "${SETUPTOOLS_VERSION}" meta_set "wheel_version" "${WHEEL_VERSION}" packages_to_install+=( diff --git a/lib/python_version.sh b/lib/python_version.sh index 6d008c849..f7bacb6ad 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -4,13 +4,15 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail -LATEST_PYTHON_3_8="3.8.20" LATEST_PYTHON_3_9="3.9.21" LATEST_PYTHON_3_10="3.10.16" LATEST_PYTHON_3_11="3.11.11" LATEST_PYTHON_3_12="3.12.8" LATEST_PYTHON_3_13="3.13.1" +OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=9 +NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=13 + DEFAULT_PYTHON_FULL_VERSION="${LATEST_PYTHON_3_13}" DEFAULT_PYTHON_MAJOR_VERSION="${DEFAULT_PYTHON_FULL_VERSION%.*}" @@ -280,7 +282,7 @@ function python_version::resolve_python_version() { local major="${BASH_REMATCH[1]}" local minor="${BASH_REMATCH[2]}" - if ((major < 3 || (major == 3 && minor < 8))); then + if ((major < 3 || (major == 3 && minor < OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION))); then if [[ "${python_version_origin}" == "cached" ]]; then output::error <<-EOF Error: The cached Python version has reached end-of-life. @@ -292,14 +294,13 @@ function python_version::resolve_python_version() { and is therefore no longer receiving security updates: https://devguide.python.org/versions/#supported-versions - As such, it is no longer supported by this buildpack. - - Please upgrade to a newer Python version by creating a - '.python-version' file that contains a Python version like: - ${DEFAULT_PYTHON_MAJOR_VERSION} - - For a list of the supported Python versions, see: + As such, it's no longer supported by this buildpack: https://devcenter.heroku.com/articles/python-support#supported-python-versions + + Please upgrade to at least Python 3.${OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION} by creating a + .python-version file in the root directory of your app, + that contains a Python version like: + 3.${OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION} EOF else output::error <<-EOF @@ -309,20 +310,18 @@ function python_version::resolve_python_version() { therefore no longer receiving security updates: https://devguide.python.org/versions/#supported-versions - As such, it is no longer supported by this buildpack. - - Please upgrade to a newer Python version by updating the - version configured via the '${python_version_origin}' file. - - For a list of the supported Python versions, see: + As such, it's no longer supported by this buildpack: https://devcenter.heroku.com/articles/python-support#supported-python-versions + + Please upgrade to at least Python 3.${OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION} by changing the + version in your ${python_version_origin} file. EOF fi meta_set "failure_reason" "python-version::eol" exit 1 fi - if (((major == 3 && minor > 13) || major >= 4)); then + if (((major == 3 && minor > NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION) || major >= 4)); then if [[ "${python_version_origin}" == "cached" ]]; then output::error <<-EOF Error: The cached Python version isn't recognised. @@ -336,7 +335,14 @@ function python_version::resolve_python_version() { This can occur if you have downgraded the version of the buildpack to an older version. - Please switch back to a newer version of this buildpack. + Please switch back to a newer version of this buildpack: + https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks + https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references + + Alternatively, request an older Python version by creating + a .python-version file in the root directory of your app, + that contains a Python version like: + 3.${NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION} EOF else output::error <<-EOF @@ -350,12 +356,12 @@ function python_version::resolve_python_version() { https://devcenter.heroku.com/articles/python-support#supported-python-versions If it has, make sure that you are using the latest version - of this buildpack, and have not pinned to an older release: + of this buildpack, and haven't pinned to an older release: https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references - Otherwise, switch to a supported version (such as Python ${DEFAULT_PYTHON_MAJOR_VERSION}) - by updating the version configured via the '${python_version_origin}' file. + Otherwise, switch to a supported version (such as Python 3.${NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION}) + by changing the version in your ${python_version_origin} file. EOF fi meta_set "failure_reason" "python-version::unknown-major" @@ -366,7 +372,6 @@ function python_version::resolve_python_version() { # Otherwise map major version specifiers to the latest patch release. case "${requested_python_version}" in *.*.*) echo "${requested_python_version}" ;; - 3.8) echo "${LATEST_PYTHON_3_8}" ;; 3.9) echo "${LATEST_PYTHON_3_9}" ;; 3.10) echo "${LATEST_PYTHON_3_10}" ;; 3.11) echo "${LATEST_PYTHON_3_11}" ;; diff --git a/spec/fixtures/pipenv_python_version_eol/Pipfile b/spec/fixtures/pipenv_python_version_eol/Pipfile index afe33e1b7..40c03e309 100644 --- a/spec/fixtures/pipenv_python_version_eol/Pipfile +++ b/spec/fixtures/pipenv_python_version_eol/Pipfile @@ -9,4 +9,4 @@ typing-extensions = "*" [dev-packages] [requires] -python_version = "3.7" +python_version = "3.8" diff --git a/spec/fixtures/pipenv_python_version_eol/Pipfile.lock b/spec/fixtures/pipenv_python_version_eol/Pipfile.lock index cf0460577..ef45647a8 100644 --- a/spec/fixtures/pipenv_python_version_eol/Pipfile.lock +++ b/spec/fixtures/pipenv_python_version_eol/Pipfile.lock @@ -1,11 +1,11 @@ { "_meta": { "hash": { - "sha256": "319488142f6df2cfe3244f5ebb99c88eaa0ef808d4560570175577ce045dd02e" + "sha256": "feaf6b91bd5a191f779b2c2199b3e328369c87e6a598f02a3ba6c76d877e176f" }, "pipfile-spec": 6, "requires": { - "python_version": "3.7" + "python_version": "3.8" }, "sources": [ { diff --git a/spec/fixtures/python_3.8/.python-version b/spec/fixtures/python_3.8/.python-version deleted file mode 100644 index 6a8981ea9..000000000 --- a/spec/fixtures/python_3.8/.python-version +++ /dev/null @@ -1,2 +0,0 @@ -# Comments are supported -3.8 diff --git a/spec/fixtures/python_3.8/requirements.txt b/spec/fixtures/python_3.8/requirements.txt deleted file mode 100644 index eec3a2223..000000000 --- a/spec/fixtures/python_3.8/requirements.txt +++ /dev/null @@ -1,2 +0,0 @@ -# This package has been picked since it has no dependencies and is small/fast to install. -typing-extensions==4.12.2 diff --git a/spec/fixtures/python_version_eol/.python-version b/spec/fixtures/python_version_eol/.python-version index 81cd8ab6c..2237ff9e6 100644 --- a/spec/fixtures/python_version_eol/.python-version +++ b/spec/fixtures/python_version_eol/.python-version @@ -4,5 +4,5 @@ # So are empty lines, and leading/trailing whitespace. - 3.7 + 3.8 diff --git a/spec/fixtures/python_version_outdated/.python-version b/spec/fixtures/python_version_outdated/.python-version index a5c4c7633..30291cba2 100644 --- a/spec/fixtures/python_version_outdated/.python-version +++ b/spec/fixtures/python_version_outdated/.python-version @@ -1 +1 @@ -3.9.0 +3.10.0 diff --git a/spec/fixtures/python_version_outdated_and_deprecated/runtime.txt b/spec/fixtures/python_version_outdated_and_deprecated/runtime.txt index 73b1cf81d..f72c5111f 100644 --- a/spec/fixtures/python_version_outdated_and_deprecated/runtime.txt +++ b/spec/fixtures/python_version_outdated_and_deprecated/runtime.txt @@ -1 +1 @@ -python-3.8.0 +python-3.9.0 diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index f6d3ba1ab..4a44a7825 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -80,6 +80,22 @@ remote: -----> Using Python 3.9.0 specified in Pipfile.lock remote: -----> Installing Python 3.9.0 remote: + remote: ! Warning: Support for Python 3.9 is ending soon! + remote: ! + remote: ! Python 3.9 will reach its upstream end-of-life in October 2025, + remote: ! at which point it will no longer receive security updates: + remote: ! https://devguide.python.org/versions/#supported-versions + remote: ! + remote: ! As such, support for Python 3.9 will be removed from this + remote: ! buildpack on 7th January 2026. + remote: ! + remote: ! Upgrade to a newer Python version as soon as possible, by + remote: ! changing the version in your Pipfile.lock file. + remote: ! + remote: ! For more information, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions + remote: + remote: remote: ! Warning: A Python security update is available! remote: ! remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} @@ -266,21 +282,19 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~OUTPUT)) remote: -----> Python app detected - remote: -----> Using Python 3.7 specified in Pipfile.lock + remote: -----> Using Python 3.8 specified in Pipfile.lock remote: remote: ! Error: The requested Python version has reached end-of-life. remote: ! - remote: ! Python 3.7 has reached its upstream end-of-life, and is + remote: ! Python 3.8 has reached its upstream end-of-life, and is remote: ! therefore no longer receiving security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! - remote: ! As such, it is no longer supported by this buildpack. - remote: ! - remote: ! Please upgrade to a newer Python version by updating the - remote: ! version configured via the 'Pipfile.lock' file. - remote: ! - remote: ! For a list of the supported Python versions, see: + remote: ! As such, it's no longer supported by this buildpack: remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions + remote: ! + remote: ! Please upgrade to at least Python 3.9 by changing the + remote: ! version in your Pipfile.lock file. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 6d3786bd8..0429c0310 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -199,6 +199,22 @@ remote: -----> Using Python 3.9.0 specified in .python-version remote: -----> Installing Python 3.9.0 remote: + remote: ! Warning: Support for Python 3.9 is ending soon! + remote: ! + remote: ! Python 3.9 will reach its upstream end-of-life in October 2025, + remote: ! at which point it will no longer receive security updates: + remote: ! https://devguide.python.org/versions/#supported-versions + remote: ! + remote: ! As such, support for Python 3.9 will be removed from this + remote: ! buildpack on 7th January 2026. + remote: ! + remote: ! Upgrade to a newer Python version as soon as possible, by + remote: ! changing the version in your .python-version file. + remote: ! + remote: ! For more information, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions + remote: + remote: remote: ! Warning: A Python security update is available! remote: ! remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index ede46f2f0..09123a5b6 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -7,58 +7,38 @@ # patch releases will bundle older pip, and the buildpack uses that pip during bootstrapping. RSpec.describe 'Python update warnings' do context 'with a runtime.txt containing an outdated patch version that is also a deprecated major version' do - let(:allow_failure) { false } - let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_outdated_and_deprecated', allow_failure:) } + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_outdated_and_deprecated') } - context 'when using Heroku-20', stacks: %w[heroku-20] do - it 'warns about both the deprecated major version and the patch update' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python 3.8.0 specified in runtime.txt - remote: -----> Installing Python 3.8.0 - remote: - remote: ! Warning: Support for Python 3.8 is ending soon! - remote: ! - remote: ! Python 3.8 reached its upstream end-of-life on 7th October 2024, and so - remote: ! no longer receives security updates: - remote: ! https://devguide.python.org/versions/#supported-versions - remote: ! - remote: ! Support for Python 3.8 will be removed from this buildpack on 7th January 2025. - remote: ! - remote: ! Upgrade to a newer Python version as soon as possible to keep your app secure. - remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: - remote: - remote: ! Warning: A Python security update is available! - remote: ! - remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_8} - remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - OUTPUT - end - end - end - - context 'when using Heroku-22 or newer', stacks: %w[heroku-22 heroku-24] do - let(:allow_failure) { true } - - # We only support Python 3.8 on Heroku-20 and older. - it 'aborts the build without showing an update warning' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python 3.8.0 specified in runtime.txt - remote: - remote: ! Error: Python 3.8.0 isn't available for this stack (#{app.stack}). - remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions - remote: - remote: ! Push rejected, failed to compile Python app. - OUTPUT - end + it 'warns about both the deprecated major version and the patch update' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.9.0 specified in runtime.txt + remote: -----> Installing Python 3.9.0 + remote: + remote: ! Warning: Support for Python 3.9 is ending soon! + remote: ! + remote: ! Python 3.9 will reach its upstream end-of-life in October 2025, + remote: ! at which point it will no longer receive security updates: + remote: ! https://devguide.python.org/versions/#supported-versions + remote: ! + remote: ! As such, support for Python 3.9 will be removed from this + remote: ! buildpack on 7th January 2026. + remote: ! + remote: ! Upgrade to a newer Python version as soon as possible, by + remote: ! changing the version in your runtime.txt file. + remote: ! + remote: ! For more information, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions + remote: + remote: + remote: ! Warning: A Python security update is available! + remote: ! + remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} + remote: ! See: https://devcenter.heroku.com/articles/python-runtimes + remote: + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + OUTPUT end end end @@ -70,12 +50,12 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python 3.9.0 specified in .python-version - remote: -----> Installing Python 3.9.0 + remote: -----> Using Python 3.10.0 specified in .python-version + remote: -----> Installing Python 3.10.0 remote: remote: ! Warning: A Python security update is available! remote: ! - remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} + remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_10} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 43b098231..3b547f7ee 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -75,65 +75,39 @@ end end - context 'when .python-version contains Python 3.8' do - let(:allow_failure) { false } - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.8', allow_failure:) } - - context 'when using Heroku-20', stacks: %w[heroku-20] do - it 'builds with latest Python 3.8 but shows a deprecation warning' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python 3.8 specified in .python-version - remote: -----> Installing Python #{LATEST_PYTHON_3_8} - remote: - remote: ! Warning: Support for Python 3.8 is ending soon! - remote: ! - remote: ! Python 3.8 reached its upstream end-of-life on 7th October 2024, and so - remote: ! no longer receives security updates: - remote: ! https://devguide.python.org/versions/#supported-versions - remote: ! - remote: ! Support for Python 3.8 will be removed from this buildpack on 7th January 2025. - remote: ! - remote: ! Upgrade to a newer Python version as soon as possible to keep your app secure. - remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing SQLite3 - remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) - OUTPUT - expect(app.run('python -V')).to include("Python #{LATEST_PYTHON_3_8}") - end - end - end - - context 'when using Heroku-22 or newer', stacks: %w[heroku-22 heroku-24] do - let(:allow_failure) { true } - - # We only support Python 3.8 on Heroku-20 and older. - it 'aborts the build with a version not available message' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python 3.8 specified in .python-version - remote: - remote: ! Error: Python #{LATEST_PYTHON_3_8} isn't available for this stack (#{app.stack}). - remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions - remote: - remote: ! Push rejected, failed to compile Python app. - OUTPUT - end - end - end - end - context 'when .python-version contains Python 3.9' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.9') } - include_examples 'builds with the requested Python version', '3.9', LATEST_PYTHON_3_9 + it 'builds with Python 3.9 but shows a deprecation warning' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.9 specified in .python-version + remote: -----> Installing Python #{LATEST_PYTHON_3_9} + remote: + remote: ! Warning: Support for Python 3.9 is ending soon! + remote: ! + remote: ! Python 3.9 will reach its upstream end-of-life in October 2025, + remote: ! at which point it will no longer receive security updates: + remote: ! https://devguide.python.org/versions/#supported-versions + remote: ! + remote: ! As such, support for Python 3.9 will be removed from this + remote: ! buildpack on 7th January 2026. + remote: ! + remote: ! Upgrade to a newer Python version as soon as possible, by + remote: ! changing the version in your .python-version file. + remote: ! + remote: ! For more information, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions + remote: + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing SQLite3 + remote: -----> Installing dependencies using 'pip install -r requirements.txt' + remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) + OUTPUT + expect(app.run('python -V')).to include("Python #{LATEST_PYTHON_3_9}") + end + end end context 'when .python-version contains Python 3.10' do @@ -252,21 +226,19 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python 3.7 specified in .python-version + remote: -----> Using Python 3.8 specified in .python-version remote: remote: ! Error: The requested Python version has reached end-of-life. remote: ! - remote: ! Python 3.7 has reached its upstream end-of-life, and is + remote: ! Python 3.8 has reached its upstream end-of-life, and is remote: ! therefore no longer receiving security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! - remote: ! As such, it is no longer supported by this buildpack. - remote: ! - remote: ! Please upgrade to a newer Python version by updating the - remote: ! version configured via the '.python-version' file. - remote: ! - remote: ! For a list of the supported Python versions, see: + remote: ! As such, it's no longer supported by this buildpack: remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions + remote: ! + remote: ! Please upgrade to at least Python 3.9 by changing the + remote: ! version in your .python-version file. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -293,12 +265,12 @@ remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: ! remote: ! If it has, make sure that you are using the latest version - remote: ! of this buildpack, and have not pinned to an older release: + remote: ! of this buildpack, and haven't pinned to an older release: remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references remote: ! remote: ! Otherwise, switch to a supported version (such as Python #{DEFAULT_PYTHON_MAJOR_VERSION}) - remote: ! by updating the version configured via the '.python-version' file. + remote: ! by changing the version in your .python-version file. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -373,13 +345,11 @@ remote: ! therefore no longer receiving security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! - remote: ! As such, it is no longer supported by this buildpack. - remote: ! - remote: ! Please upgrade to a newer Python version by updating the - remote: ! version configured via the 'runtime.txt' file. - remote: ! - remote: ! For a list of the supported Python versions, see: + remote: ! As such, it's no longer supported by this buildpack: remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions + remote: ! + remote: ! Please upgrade to at least Python 3.9 by changing the + remote: ! version in your runtime.txt file. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 2405d9303..af1ca38cf 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -9,7 +9,6 @@ FIXTURE_DIR = Pathname.new(__FILE__).parent.join('fixtures') -LATEST_PYTHON_3_8 = '3.8.20' LATEST_PYTHON_3_9 = '3.9.21' LATEST_PYTHON_3_10 = '3.10.16' LATEST_PYTHON_3_11 = '3.11.11' From 3fede9b5bd4075ac05f07f9ea47ffa515d76a195 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 8 Jan 2025 12:25:13 +0000 Subject: [PATCH 086/406] Prepare release v274 (#1733) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0c5e5c2f9..4e3f7fab7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v274] - 2025-01-08 + - Added a deprecation warning for Python 3.9. ([#1732](https://github.com/heroku/heroku-buildpack-python/pull/1732)) - Removed support for Python 3.8. ([#1732](https://github.com/heroku/heroku-buildpack-python/pull/1732)) - Improved the error messages shown for EOL or unrecognised major Python versions. ([#1732](https://github.com/heroku/heroku-buildpack-python/pull/1732)) @@ -1143,7 +1146,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v273...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v274...main +[v274]: https://github.com/heroku/heroku-buildpack-python/compare/v273...v274 [v273]: https://github.com/heroku/heroku-buildpack-python/compare/v272...v273 [v272]: https://github.com/heroku/heroku-buildpack-python/compare/v271...v272 [v271]: https://github.com/heroku/heroku-buildpack-python/compare/v270...v271 From 7de568e2944363df2af4d7ded633c31f31be9978 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:43:43 +0000 Subject: [PATCH 087/406] Bump poetry from 1.8.5 to 2.0.1 (#1734) * Bump poetry from 1.8.5 to 2.0.1 Bumps [poetry](https://github.com/python-poetry/poetry) from 1.8.5 to 2.0.1. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](https://github.com/python-poetry/poetry/compare/1.8.5...2.0.1) --- updated-dependencies: - dependency-name: poetry dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] * Switch to `poetry sync` Since `poetry install --sync` has been deprecated in favour of the new `poetry sync` command. * Switch one fixture to using the `[project]` table So we have coverage of both the old and new style dependency table syntax. * Revert "Switch one fixture to using the `[project]` table" This reverts commit a0bb95163811b20252409aec077d5fbf7c97bf06. Since the `poetry_basic` fixture is also used for the old buildpack version test, and so fails when the Poetry 1.x version in the old buildpack can't read the Poetry v2 lockfile. We can land this later, once a historic buildpack version exists that uses a Poetry 2.x release. * Add changelog entry * Update test log output for new version --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + lib/poetry.sh | 7 +++---- requirements/poetry.txt | 2 +- spec/hatchet/ci_spec.rb | 4 ++-- spec/hatchet/poetry_spec.rb | 14 +++++++------- 5 files changed, 14 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e3f7fab7..709e3cf87 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated Poetry from 1.8.5 to 2.0.1. ([#1734](https://github.com/heroku/heroku-buildpack-python/pull/1734)) ## [v274] - 2025-01-08 diff --git a/lib/poetry.sh b/lib/poetry.sh index 062639b3a..a227ae1cd 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -99,8 +99,8 @@ function poetry::install_poetry() { # Note: We cache site-packages since: # - It results in faster builds than only caching Poetry's download/wheel cache. -# - It's safe to do so, since `poetry install --sync` fully manages the environment -# (including e.g. uninstalling packages when they are removed from the lockfile). +# - It's safe to do so, since `poetry sync` fully manages the environment (including +# e.g. uninstalling packages when they are removed from the lockfile). # # With site-packages cached there is no need to persist Poetry's download/wheel cache in the build # cache, so we let Poetry write it to the home directory where it will be discarded at the end of @@ -109,8 +109,7 @@ function poetry::install_poetry() { function poetry::install_dependencies() { local poetry_install_command=( poetry - install - --sync + sync ) # On Heroku CI, all default Poetry dependency groups are installed (i.e. all groups minus those diff --git a/requirements/poetry.txt b/requirements/poetry.txt index b7c67421a..d1c8013e2 100644 --- a/requirements/poetry.txt +++ b/requirements/poetry.txt @@ -1 +1 @@ -poetry==1.8.5 +poetry==2.0.1 diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 824564447..fcd7198ba 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -161,7 +161,7 @@ -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing Poetry #{POETRY_VERSION} - -----> Installing dependencies using 'poetry install --sync' + -----> Installing dependencies using 'poetry sync' Installing dependencies from lock file Package operations: 5 installs, 0 updates, 0 removals @@ -223,7 +223,7 @@ -----> Restoring cache -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing Poetry #{POETRY_VERSION} - -----> Installing dependencies using 'poetry install --sync' + -----> Installing dependencies using 'poetry sync' Installing dependencies from lock file No dependencies to install or update diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 0429c0310..24e23c236 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -14,7 +14,7 @@ remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing Poetry #{POETRY_VERSION} - remote: -----> Installing dependencies using 'poetry install --sync --only main' + remote: -----> Installing dependencies using 'poetry sync --only main' remote: Installing dependencies from lock file remote: remote: Package operations: 1 install, 0 updates, 0 removals @@ -51,7 +51,7 @@ remote: -----> Restoring cache remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Using cached Poetry #{POETRY_VERSION} - remote: -----> Installing dependencies using 'poetry install --sync --only main' + remote: -----> Installing dependencies using 'poetry sync --only main' remote: Installing dependencies from lock file remote: remote: No dependencies to install or update @@ -78,7 +78,7 @@ remote: - The Poetry version has changed from 1.8.4 to #{POETRY_VERSION} remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing Poetry #{POETRY_VERSION} - remote: -----> Installing dependencies using 'poetry install --sync --only main' + remote: -----> Installing dependencies using 'poetry sync --only main' remote: Installing dependencies from lock file remote: remote: Package operations: 1 install, 0 updates, 0 removals @@ -107,7 +107,7 @@ remote: - The package manager has changed from pip to poetry remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing Poetry #{POETRY_VERSION} - remote: -----> Installing dependencies using 'poetry install --sync --only main' + remote: -----> Installing dependencies using 'poetry sync --only main' remote: Installing dependencies from lock file remote: remote: Package operations: 1 install, 0 updates, 0 removals @@ -221,7 +221,7 @@ remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: remote: -----> Installing Poetry #{POETRY_VERSION} - remote: -----> Installing dependencies using 'poetry install --sync --only main' + remote: -----> Installing dependencies using 'poetry sync --only main' remote: Installing dependencies from lock file remote: remote: Package operations: 1 install, 0 updates, 0 removals @@ -240,10 +240,10 @@ it 'fails the build' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Installing dependencies using 'poetry install --sync --only main' + remote: -----> Installing dependencies using 'poetry sync --only main' remote: Installing dependencies from lock file remote: - remote: pyproject.toml changed significantly since poetry.lock was last generated. Run `poetry lock [--no-update]` to fix the lock file. + remote: pyproject.toml changed significantly since poetry.lock was last generated. Run `poetry lock` to fix the lock file. remote: remote: ! Error: Unable to install dependencies using Poetry. remote: ! From 68913bf0edfda9332e0a6170d3acf9f2737da452 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:51:33 +0000 Subject: [PATCH 088/406] Bump the ruby-dependencies group with 2 updates (#1735) * Bump the ruby-dependencies group with 2 updates Bumps the ruby-dependencies group with 2 updates: [logger](https://github.com/ruby/logger) and [rubocop](https://github.com/rubocop/rubocop). Updates `logger` from 1.6.4 to 1.6.5 - [Release notes](https://github.com/ruby/logger/releases) - [Commits](https://github.com/ruby/logger/compare/v1.6.4...v1.6.5) Updates `rubocop` from 1.69.2 to 1.70.0 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.69.2...v1.70.0) --- updated-dependencies: - dependency-name: logger dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rubocop dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] * Update Ruby/bundler versions --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- .github/workflows/ci.yml | 4 ++-- .github/workflows/hatchet_app_cleaner.yml | 2 +- Gemfile | 2 +- Gemfile.lock | 10 +++++----- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a93559ca7..57f26122d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -24,7 +24,7 @@ jobs: uses: ruby/setup-ruby@v1 with: bundler-cache: true - ruby-version: "3.3" + ruby-version: "3.4" - name: Run ShellCheck run: make lint-scripts - name: Run shfmt @@ -56,7 +56,7 @@ jobs: uses: ruby/setup-ruby@v1 with: bundler-cache: true - ruby-version: "3.3" + ruby-version: "3.4" - name: Hatchet setup run: bundle exec hatchet ci:setup - name: Run Hatchet integration tests diff --git a/.github/workflows/hatchet_app_cleaner.yml b/.github/workflows/hatchet_app_cleaner.yml index b9eb06cdf..aa7a88512 100644 --- a/.github/workflows/hatchet_app_cleaner.yml +++ b/.github/workflows/hatchet_app_cleaner.yml @@ -24,7 +24,7 @@ jobs: uses: ruby/setup-ruby@v1 with: bundler-cache: true - ruby-version: "3.3" + ruby-version: "3.4" - name: Run Hatchet destroy # Only apps older than 10 minutes are destroyed, to ensure that any # in progress CI runs are not interrupted. diff --git a/Gemfile b/Gemfile index ddaf2f60c..215de3b19 100644 --- a/Gemfile +++ b/Gemfile @@ -2,7 +2,7 @@ source 'https://rubygems.org' -ruby '>= 3.2', '< 3.4' +ruby '>= 3.2', '< 3.5' group :test, :development do gem 'heroku_hatchet' diff --git a/Gemfile.lock b/Gemfile.lock index 9c610e2f0..d73682f90 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -21,7 +21,7 @@ GEM threaded (~> 0) json (2.9.1) language_server-protocol (3.17.0.3) - logger (1.6.4) + logger (1.6.5) moneta (1.0.0) multi_json (1.15.0) parallel (1.26.3) @@ -48,7 +48,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.2) - rubocop (1.69.2) + rubocop (1.70.0) json (~> 2.3) language_server-protocol (>= 3.17.0) parallel (~> 1.10) @@ -65,7 +65,7 @@ GEM ruby-progressbar (1.13.0) thor (1.3.2) threaded (0.0.4) - unicode-display_width (3.1.3) + unicode-display_width (3.1.4) unicode-emoji (~> 4.0, >= 4.0.4) unicode-emoji (4.0.4) webrick (1.9.1) @@ -84,7 +84,7 @@ DEPENDENCIES rubocop-rspec RUBY VERSION - ruby 3.3.5p100 + ruby 3.4.1p0 BUNDLED WITH - 2.5.23 + 2.6.2 From b90a4755ae2008a5cdae2b32632a9fb4da49db0d Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Mon, 13 Jan 2025 18:28:14 +0000 Subject: [PATCH 089/406] Prepare release v275 (#1736) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 709e3cf87..caeaa579b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v275] - 2025-01-13 + - Updated Poetry from 1.8.5 to 2.0.1. ([#1734](https://github.com/heroku/heroku-buildpack-python/pull/1734)) ## [v274] - 2025-01-08 @@ -1147,7 +1150,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v274...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v275...main +[v275]: https://github.com/heroku/heroku-buildpack-python/compare/v274...v275 [v274]: https://github.com/heroku/heroku-buildpack-python/compare/v273...v274 [v273]: https://github.com/heroku/heroku-buildpack-python/compare/v272...v273 [v272]: https://github.com/heroku/heroku-buildpack-python/compare/v271...v272 From 4c4cb7a604cbc86a947858e625cccd86c2a098e7 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 27 Jan 2025 10:50:41 +0000 Subject: [PATCH 090/406] Tests: Misc package manager test improvements (#1738) Backports some test naming/strategy improvements from the CNB, along with some general classic-specific clean-ups. Towards #1616. GUS-W-17679633. --- .../.python-version | 0 .../{ci_requirements => ci_pip}/app.json | 0 .../{ci_requirements => ci_pip}/bin/compile | 0 .../{ci_requirements => ci_pip}/bin/detect | 0 .../bin/post_compile | 0 .../bin/print-env-vars.sh | 0 .../requirements-test.txt | 0 .../requirements.txt | 0 .../requirements.txt | 0 .../.python-version | 0 .../Pipfile | 6 - .../multiple_package_managers/Pipfile.lock | 18 +++ .../multiple_package_managers/poetry.lock | 0 .../multiple_package_managers/pyproject.toml | 0 .../requirements.txt | 0 .../multiple_package_managers/setup.py | 0 .../.python-version | 0 .../bin/compile | 0 .../bin/detect | 0 spec/fixtures/pip_basic/requirements-test.txt | 2 + .../requirements.txt | 0 .../requirements/prod.txt | 0 spec/fixtures/pip_basic/setup.py | 2 + spec/fixtures/pip_compiled/requirements.txt | 2 + .../.python-version | 0 .../bin/compile | 0 .../bin/detect | 0 .../bin/post_compile | 0 .../bin/test-entrypoints.sh | 0 .../local_package_pyproject_toml/__init__.py | 0 .../pyproject.toml | 0 .../local_package_setup_py/__init__.py | 0 .../packages/local_package_setup_py/setup.cfg | 0 .../packages/local_package_setup_py/setup.py | 0 .../requirements.txt | 0 .../requirements.txt | 0 .../requirements.txt | 0 .../requirements.txt | 0 .../.python-version | 0 spec/fixtures/pip_pysqlite3/requirements.txt | 1 + .../pipenv_and_requirements_txt/Pipfile.lock | 30 ----- .../requirements.txt | 1 - spec/fixtures/pipenv_basic/setup.py | 11 +- spec/fixtures/poetry_basic/setup.py | 2 + .../requirements_compiled/requirements.txt | 6 - .../poetry.lock | 17 --- .../pyproject.toml | 6 - .../requirements.txt | 2 - .../requirements.txt | 1 - .../requirements_txt_and_setup_py/setup.py | 6 - spec/hatchet/ci_spec.rb | 2 +- spec/hatchet/django_spec.rb | 4 +- spec/hatchet/package_manager_spec.rb | 92 +++++++++++++ spec/hatchet/pip_spec.rb | 99 +++----------- spec/hatchet/pipenv_spec.rb | 121 +++++------------- spec/hatchet/poetry_spec.rb | 70 +--------- spec/hatchet/python_version_spec.rb | 8 +- 57 files changed, 182 insertions(+), 327 deletions(-) rename spec/fixtures/{ci_requirements => ci_pip}/.python-version (100%) rename spec/fixtures/{ci_requirements => ci_pip}/app.json (100%) rename spec/fixtures/{ci_requirements => ci_pip}/bin/compile (100%) rename spec/fixtures/{ci_requirements => ci_pip}/bin/detect (100%) rename spec/fixtures/{ci_requirements => ci_pip}/bin/post_compile (100%) rename spec/fixtures/{ci_requirements => ci_pip}/bin/print-env-vars.sh (100%) rename spec/fixtures/{ci_requirements => ci_pip}/requirements-test.txt (100%) rename spec/fixtures/{ci_requirements => ci_pip}/requirements.txt (100%) rename spec/fixtures/{requirements_django_latest => django_latest}/requirements.txt (100%) rename spec/fixtures/{requirements_basic => multiple_package_managers}/.python-version (100%) rename spec/fixtures/{pipenv_and_requirements_txt => multiple_package_managers}/Pipfile (56%) create mode 100644 spec/fixtures/multiple_package_managers/Pipfile.lock create mode 100644 spec/fixtures/multiple_package_managers/poetry.lock create mode 100644 spec/fixtures/multiple_package_managers/pyproject.toml create mode 100644 spec/fixtures/multiple_package_managers/requirements.txt create mode 100644 spec/fixtures/multiple_package_managers/setup.py rename spec/fixtures/{requirements_txt_and_poetry_lock => pip_basic}/.python-version (100%) rename spec/fixtures/{requirements_basic => pip_basic}/bin/compile (100%) rename spec/fixtures/{requirements_basic => pip_basic}/bin/detect (100%) create mode 100644 spec/fixtures/pip_basic/requirements-test.txt rename spec/fixtures/{requirements_basic => pip_basic}/requirements.txt (100%) rename spec/fixtures/{requirements_basic => pip_basic}/requirements/prod.txt (100%) create mode 100644 spec/fixtures/pip_basic/setup.py create mode 100644 spec/fixtures/pip_compiled/requirements.txt rename spec/fixtures/{requirements_editable => pip_editable}/.python-version (100%) rename spec/fixtures/{requirements_editable => pip_editable}/bin/compile (100%) rename spec/fixtures/{requirements_editable => pip_editable}/bin/detect (100%) rename spec/fixtures/{requirements_editable => pip_editable}/bin/post_compile (100%) rename spec/fixtures/{requirements_editable => pip_editable}/bin/test-entrypoints.sh (100%) rename spec/fixtures/{requirements_editable => pip_editable}/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py (100%) rename spec/fixtures/{requirements_editable => pip_editable}/packages/local_package_pyproject_toml/pyproject.toml (100%) rename spec/fixtures/{requirements_editable => pip_editable}/packages/local_package_setup_py/local_package_setup_py/__init__.py (100%) rename spec/fixtures/{requirements_editable => pip_editable}/packages/local_package_setup_py/setup.cfg (100%) rename spec/fixtures/{requirements_editable => pip_editable}/packages/local_package_setup_py/setup.py (100%) rename spec/fixtures/{requirements_editable => pip_editable}/requirements.txt (100%) rename spec/fixtures/{requirements_gdal => pip_gdal}/requirements.txt (100%) rename spec/fixtures/{requirements_invalid => pip_invalid_requirement}/requirements.txt (100%) rename spec/fixtures/{requirements_legacy_celery => pip_legacy_celery}/requirements.txt (100%) rename spec/fixtures/{requirements_compiled => pip_pysqlite3}/.python-version (100%) create mode 100644 spec/fixtures/pip_pysqlite3/requirements.txt delete mode 100644 spec/fixtures/pipenv_and_requirements_txt/Pipfile.lock delete mode 100644 spec/fixtures/pipenv_and_requirements_txt/requirements.txt create mode 100644 spec/fixtures/poetry_basic/setup.py delete mode 100644 spec/fixtures/requirements_compiled/requirements.txt delete mode 100644 spec/fixtures/requirements_txt_and_poetry_lock/poetry.lock delete mode 100644 spec/fixtures/requirements_txt_and_poetry_lock/pyproject.toml delete mode 100644 spec/fixtures/requirements_txt_and_poetry_lock/requirements.txt delete mode 100644 spec/fixtures/requirements_txt_and_setup_py/requirements.txt delete mode 100644 spec/fixtures/requirements_txt_and_setup_py/setup.py diff --git a/spec/fixtures/ci_requirements/.python-version b/spec/fixtures/ci_pip/.python-version similarity index 100% rename from spec/fixtures/ci_requirements/.python-version rename to spec/fixtures/ci_pip/.python-version diff --git a/spec/fixtures/ci_requirements/app.json b/spec/fixtures/ci_pip/app.json similarity index 100% rename from spec/fixtures/ci_requirements/app.json rename to spec/fixtures/ci_pip/app.json diff --git a/spec/fixtures/ci_requirements/bin/compile b/spec/fixtures/ci_pip/bin/compile similarity index 100% rename from spec/fixtures/ci_requirements/bin/compile rename to spec/fixtures/ci_pip/bin/compile diff --git a/spec/fixtures/ci_requirements/bin/detect b/spec/fixtures/ci_pip/bin/detect similarity index 100% rename from spec/fixtures/ci_requirements/bin/detect rename to spec/fixtures/ci_pip/bin/detect diff --git a/spec/fixtures/ci_requirements/bin/post_compile b/spec/fixtures/ci_pip/bin/post_compile similarity index 100% rename from spec/fixtures/ci_requirements/bin/post_compile rename to spec/fixtures/ci_pip/bin/post_compile diff --git a/spec/fixtures/ci_requirements/bin/print-env-vars.sh b/spec/fixtures/ci_pip/bin/print-env-vars.sh similarity index 100% rename from spec/fixtures/ci_requirements/bin/print-env-vars.sh rename to spec/fixtures/ci_pip/bin/print-env-vars.sh diff --git a/spec/fixtures/ci_requirements/requirements-test.txt b/spec/fixtures/ci_pip/requirements-test.txt similarity index 100% rename from spec/fixtures/ci_requirements/requirements-test.txt rename to spec/fixtures/ci_pip/requirements-test.txt diff --git a/spec/fixtures/ci_requirements/requirements.txt b/spec/fixtures/ci_pip/requirements.txt similarity index 100% rename from spec/fixtures/ci_requirements/requirements.txt rename to spec/fixtures/ci_pip/requirements.txt diff --git a/spec/fixtures/requirements_django_latest/requirements.txt b/spec/fixtures/django_latest/requirements.txt similarity index 100% rename from spec/fixtures/requirements_django_latest/requirements.txt rename to spec/fixtures/django_latest/requirements.txt diff --git a/spec/fixtures/requirements_basic/.python-version b/spec/fixtures/multiple_package_managers/.python-version similarity index 100% rename from spec/fixtures/requirements_basic/.python-version rename to spec/fixtures/multiple_package_managers/.python-version diff --git a/spec/fixtures/pipenv_and_requirements_txt/Pipfile b/spec/fixtures/multiple_package_managers/Pipfile similarity index 56% rename from spec/fixtures/pipenv_and_requirements_txt/Pipfile rename to spec/fixtures/multiple_package_managers/Pipfile index 73f50fc86..6a061ae6a 100644 --- a/spec/fixtures/pipenv_and_requirements_txt/Pipfile +++ b/spec/fixtures/multiple_package_managers/Pipfile @@ -4,9 +4,3 @@ verify_ssl = true name = "pypi" [packages] -urllib3 = "*" - -[dev-packages] - -[requires] -python_version = "3.13" diff --git a/spec/fixtures/multiple_package_managers/Pipfile.lock b/spec/fixtures/multiple_package_managers/Pipfile.lock new file mode 100644 index 000000000..d2c2cba4c --- /dev/null +++ b/spec/fixtures/multiple_package_managers/Pipfile.lock @@ -0,0 +1,18 @@ +{ + "_meta": { + "hash": { + "sha256": "ebffa69a1fa192d1cef7cb42ad79231ca976565c5ce371a70160b3048d3cbc06" + }, + "pipfile-spec": 6, + "requires": {}, + "sources": [ + { + "name": "pypi", + "url": "https://pypi.org/simple", + "verify_ssl": true + } + ] + }, + "default": {}, + "develop": {} +} diff --git a/spec/fixtures/multiple_package_managers/poetry.lock b/spec/fixtures/multiple_package_managers/poetry.lock new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/multiple_package_managers/pyproject.toml b/spec/fixtures/multiple_package_managers/pyproject.toml new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/multiple_package_managers/requirements.txt b/spec/fixtures/multiple_package_managers/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/multiple_package_managers/setup.py b/spec/fixtures/multiple_package_managers/setup.py new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/requirements_txt_and_poetry_lock/.python-version b/spec/fixtures/pip_basic/.python-version similarity index 100% rename from spec/fixtures/requirements_txt_and_poetry_lock/.python-version rename to spec/fixtures/pip_basic/.python-version diff --git a/spec/fixtures/requirements_basic/bin/compile b/spec/fixtures/pip_basic/bin/compile similarity index 100% rename from spec/fixtures/requirements_basic/bin/compile rename to spec/fixtures/pip_basic/bin/compile diff --git a/spec/fixtures/requirements_basic/bin/detect b/spec/fixtures/pip_basic/bin/detect similarity index 100% rename from spec/fixtures/requirements_basic/bin/detect rename to spec/fixtures/pip_basic/bin/detect diff --git a/spec/fixtures/pip_basic/requirements-test.txt b/spec/fixtures/pip_basic/requirements-test.txt new file mode 100644 index 000000000..abe3e9819 --- /dev/null +++ b/spec/fixtures/pip_basic/requirements-test.txt @@ -0,0 +1,2 @@ +# This shouldn't be installed, since requirements-test.txt should only be used on Heroku CI. +pytest diff --git a/spec/fixtures/requirements_basic/requirements.txt b/spec/fixtures/pip_basic/requirements.txt similarity index 100% rename from spec/fixtures/requirements_basic/requirements.txt rename to spec/fixtures/pip_basic/requirements.txt diff --git a/spec/fixtures/requirements_basic/requirements/prod.txt b/spec/fixtures/pip_basic/requirements/prod.txt similarity index 100% rename from spec/fixtures/requirements_basic/requirements/prod.txt rename to spec/fixtures/pip_basic/requirements/prod.txt diff --git a/spec/fixtures/pip_basic/setup.py b/spec/fixtures/pip_basic/setup.py new file mode 100644 index 000000000..b5dc486bd --- /dev/null +++ b/spec/fixtures/pip_basic/setup.py @@ -0,0 +1,2 @@ +# This tests that the setup.py fallback is not used when other package manager files exist. +raise RuntimeError("setup.py should not be run!") diff --git a/spec/fixtures/pip_compiled/requirements.txt b/spec/fixtures/pip_compiled/requirements.txt new file mode 100644 index 000000000..ce7cf1ab7 --- /dev/null +++ b/spec/fixtures/pip_compiled/requirements.txt @@ -0,0 +1,2 @@ +# A quick to install package that relies on the Python headers from the base image. +git+https://github.com/pypa/wheel.git@0.44.0#egg=extension.dist&subdirectory=tests/testdata/extension.dist diff --git a/spec/fixtures/requirements_editable/.python-version b/spec/fixtures/pip_editable/.python-version similarity index 100% rename from spec/fixtures/requirements_editable/.python-version rename to spec/fixtures/pip_editable/.python-version diff --git a/spec/fixtures/requirements_editable/bin/compile b/spec/fixtures/pip_editable/bin/compile similarity index 100% rename from spec/fixtures/requirements_editable/bin/compile rename to spec/fixtures/pip_editable/bin/compile diff --git a/spec/fixtures/requirements_editable/bin/detect b/spec/fixtures/pip_editable/bin/detect similarity index 100% rename from spec/fixtures/requirements_editable/bin/detect rename to spec/fixtures/pip_editable/bin/detect diff --git a/spec/fixtures/requirements_editable/bin/post_compile b/spec/fixtures/pip_editable/bin/post_compile similarity index 100% rename from spec/fixtures/requirements_editable/bin/post_compile rename to spec/fixtures/pip_editable/bin/post_compile diff --git a/spec/fixtures/requirements_editable/bin/test-entrypoints.sh b/spec/fixtures/pip_editable/bin/test-entrypoints.sh similarity index 100% rename from spec/fixtures/requirements_editable/bin/test-entrypoints.sh rename to spec/fixtures/pip_editable/bin/test-entrypoints.sh diff --git a/spec/fixtures/requirements_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py b/spec/fixtures/pip_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py similarity index 100% rename from spec/fixtures/requirements_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py rename to spec/fixtures/pip_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py diff --git a/spec/fixtures/requirements_editable/packages/local_package_pyproject_toml/pyproject.toml b/spec/fixtures/pip_editable/packages/local_package_pyproject_toml/pyproject.toml similarity index 100% rename from spec/fixtures/requirements_editable/packages/local_package_pyproject_toml/pyproject.toml rename to spec/fixtures/pip_editable/packages/local_package_pyproject_toml/pyproject.toml diff --git a/spec/fixtures/requirements_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py b/spec/fixtures/pip_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py similarity index 100% rename from spec/fixtures/requirements_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py rename to spec/fixtures/pip_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py diff --git a/spec/fixtures/requirements_editable/packages/local_package_setup_py/setup.cfg b/spec/fixtures/pip_editable/packages/local_package_setup_py/setup.cfg similarity index 100% rename from spec/fixtures/requirements_editable/packages/local_package_setup_py/setup.cfg rename to spec/fixtures/pip_editable/packages/local_package_setup_py/setup.cfg diff --git a/spec/fixtures/requirements_editable/packages/local_package_setup_py/setup.py b/spec/fixtures/pip_editable/packages/local_package_setup_py/setup.py similarity index 100% rename from spec/fixtures/requirements_editable/packages/local_package_setup_py/setup.py rename to spec/fixtures/pip_editable/packages/local_package_setup_py/setup.py diff --git a/spec/fixtures/requirements_editable/requirements.txt b/spec/fixtures/pip_editable/requirements.txt similarity index 100% rename from spec/fixtures/requirements_editable/requirements.txt rename to spec/fixtures/pip_editable/requirements.txt diff --git a/spec/fixtures/requirements_gdal/requirements.txt b/spec/fixtures/pip_gdal/requirements.txt similarity index 100% rename from spec/fixtures/requirements_gdal/requirements.txt rename to spec/fixtures/pip_gdal/requirements.txt diff --git a/spec/fixtures/requirements_invalid/requirements.txt b/spec/fixtures/pip_invalid_requirement/requirements.txt similarity index 100% rename from spec/fixtures/requirements_invalid/requirements.txt rename to spec/fixtures/pip_invalid_requirement/requirements.txt diff --git a/spec/fixtures/requirements_legacy_celery/requirements.txt b/spec/fixtures/pip_legacy_celery/requirements.txt similarity index 100% rename from spec/fixtures/requirements_legacy_celery/requirements.txt rename to spec/fixtures/pip_legacy_celery/requirements.txt diff --git a/spec/fixtures/requirements_compiled/.python-version b/spec/fixtures/pip_pysqlite3/.python-version similarity index 100% rename from spec/fixtures/requirements_compiled/.python-version rename to spec/fixtures/pip_pysqlite3/.python-version diff --git a/spec/fixtures/pip_pysqlite3/requirements.txt b/spec/fixtures/pip_pysqlite3/requirements.txt new file mode 100644 index 000000000..864b10382 --- /dev/null +++ b/spec/fixtures/pip_pysqlite3/requirements.txt @@ -0,0 +1 @@ +pysqlite3 diff --git a/spec/fixtures/pipenv_and_requirements_txt/Pipfile.lock b/spec/fixtures/pipenv_and_requirements_txt/Pipfile.lock deleted file mode 100644 index dcd0b3ca7..000000000 --- a/spec/fixtures/pipenv_and_requirements_txt/Pipfile.lock +++ /dev/null @@ -1,30 +0,0 @@ -{ - "_meta": { - "hash": { - "sha256": "60dc67ee4223a391c5e0ae2b4d7ea54a7b245773d76b6ff82156dda97a3e4fb2" - }, - "pipfile-spec": 6, - "requires": { - "python_version": "3.13" - }, - "sources": [ - { - "name": "pypi", - "url": "https://pypi.org/simple", - "verify_ssl": true - } - ] - }, - "default": { - "urllib3": { - "hashes": [ - "sha256:ca899ca043dcb1bafa3e262d73aa25c465bfb49e0bd9dd5d59f1d0acba2f8fac", - "sha256:e7d814a81dad81e6caf2ec9fdedb284ecc9c73076b62654547cc64ccdcae26e9" - ], - "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==2.2.3" - } - }, - "develop": {} -} diff --git a/spec/fixtures/pipenv_and_requirements_txt/requirements.txt b/spec/fixtures/pipenv_and_requirements_txt/requirements.txt deleted file mode 100644 index a42590beb..000000000 --- a/spec/fixtures/pipenv_and_requirements_txt/requirements.txt +++ /dev/null @@ -1 +0,0 @@ -urllib3 diff --git a/spec/fixtures/pipenv_basic/setup.py b/spec/fixtures/pipenv_basic/setup.py index de90e907e..b5dc486bd 100644 --- a/spec/fixtures/pipenv_basic/setup.py +++ b/spec/fixtures/pipenv_basic/setup.py @@ -1,9 +1,2 @@ -# This file is here to confirm we don't try and create the fallback requirements -# file containing '-e .' when using Pipenv. - -from setuptools import setup - -setup( - name='test', - install_requires=['six'], -) +# This tests that the setup.py fallback is not used when other package manager files exist. +raise RuntimeError("setup.py should not be run!") diff --git a/spec/fixtures/poetry_basic/setup.py b/spec/fixtures/poetry_basic/setup.py new file mode 100644 index 000000000..b5dc486bd --- /dev/null +++ b/spec/fixtures/poetry_basic/setup.py @@ -0,0 +1,2 @@ +# This tests that the setup.py fallback is not used when other package manager files exist. +raise RuntimeError("setup.py should not be run!") diff --git a/spec/fixtures/requirements_compiled/requirements.txt b/spec/fixtures/requirements_compiled/requirements.txt deleted file mode 100644 index 2331d3f79..000000000 --- a/spec/fixtures/requirements_compiled/requirements.txt +++ /dev/null @@ -1,6 +0,0 @@ -# Packages that rely on headers from the stack image (depending on wheel availability). -cffi -mysqlclient -psycopg2 -pylibmc -pysqlite3 diff --git a/spec/fixtures/requirements_txt_and_poetry_lock/poetry.lock b/spec/fixtures/requirements_txt_and_poetry_lock/poetry.lock deleted file mode 100644 index f80c1b5c7..000000000 --- a/spec/fixtures/requirements_txt_and_poetry_lock/poetry.lock +++ /dev/null @@ -1,17 +0,0 @@ -# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand. - -[[package]] -name = "typing-extensions" -version = "4.12.2" -description = "Backported and Experimental Type Hints for Python 3.8+" -optional = false -python-versions = ">=3.8" -files = [ - {file = "typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d"}, - {file = "typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"}, -] - -[metadata] -lock-version = "2.0" -python-versions = "^3.13" -content-hash = "247bc9eddaeea191a758d051d2eae56b572964d566c6ab0c81822ee7535083dd" diff --git a/spec/fixtures/requirements_txt_and_poetry_lock/pyproject.toml b/spec/fixtures/requirements_txt_and_poetry_lock/pyproject.toml deleted file mode 100644 index ebfd436a7..000000000 --- a/spec/fixtures/requirements_txt_and_poetry_lock/pyproject.toml +++ /dev/null @@ -1,6 +0,0 @@ -[tool.poetry] -package-mode = false - -[tool.poetry.dependencies] -python = "^3.13" -typing-extensions = "*" diff --git a/spec/fixtures/requirements_txt_and_poetry_lock/requirements.txt b/spec/fixtures/requirements_txt_and_poetry_lock/requirements.txt deleted file mode 100644 index eec3a2223..000000000 --- a/spec/fixtures/requirements_txt_and_poetry_lock/requirements.txt +++ /dev/null @@ -1,2 +0,0 @@ -# This package has been picked since it has no dependencies and is small/fast to install. -typing-extensions==4.12.2 diff --git a/spec/fixtures/requirements_txt_and_setup_py/requirements.txt b/spec/fixtures/requirements_txt_and_setup_py/requirements.txt deleted file mode 100644 index a42590beb..000000000 --- a/spec/fixtures/requirements_txt_and_setup_py/requirements.txt +++ /dev/null @@ -1 +0,0 @@ -urllib3 diff --git a/spec/fixtures/requirements_txt_and_setup_py/setup.py b/spec/fixtures/requirements_txt_and_setup_py/setup.py deleted file mode 100644 index 8d636d69a..000000000 --- a/spec/fixtures/requirements_txt_and_setup_py/setup.py +++ /dev/null @@ -1,6 +0,0 @@ -from setuptools import setup - -setup( - name='test', - install_requires=['six'], -) diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index fcd7198ba..30126af91 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -6,7 +6,7 @@ let(:buildpacks) { [:default, 'heroku-community/inline'] } context 'when using pip' do - let(:app) { Hatchet::Runner.new('spec/fixtures/ci_requirements', buildpacks:) } + let(:app) { Hatchet::Runner.new('spec/fixtures/ci_pip', buildpacks:) } it 'installs both normal and test dependencies and uses cache on subsequent runs' do app.run_ci do |test_run| diff --git a/spec/hatchet/django_spec.rb b/spec/hatchet/django_spec.rb index 38f7e3e0a..6372ea097 100644 --- a/spec/hatchet/django_spec.rb +++ b/spec/hatchet/django_spec.rb @@ -32,7 +32,7 @@ end context 'when Django is installed but manage.py does not exist' do - let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_django_latest') } + let(:app) { Hatchet::Runner.new('spec/fixtures/django_latest') } it 'skips collectstatic' do app.deploy do |app| @@ -44,7 +44,7 @@ context 'when DISABLE_COLLECTSTATIC=1' do let(:app) do - Hatchet::Runner.new('spec/fixtures/requirements_django_latest', config: { 'DISABLE_COLLECTSTATIC' => '1' }) + Hatchet::Runner.new('spec/fixtures/django_latest', config: { 'DISABLE_COLLECTSTATIC' => '1' }) end it 'skips collectstatic' do diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index 1b85b2bb1..631eb8df3 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -46,4 +46,96 @@ end end end + + # TODO: Deprecate/sunset the setup.py file fallback. + context 'when there is only a setup.py' do + let(:app) { Hatchet::Runner.new('spec/fixtures/setup_py_only') } + + it 'installs packages from setup.py using pip' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + remote: -----> Python app detected + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} + remote: -----> Installing dependencies using 'pip install --editable .' + remote: Obtaining file:///tmp/build_.* + remote: .+ + remote: Installing collected packages: six, test + remote: Successfully installed six-.+ test-0.0.0 + REGEX + end + end + end + + # This case will be turned into an error in the future. + context 'when there are multiple package manager files' do + let(:app) { Hatchet::Runner.new('spec/fixtures/multiple_package_managers') } + + it 'outputs a warning and builds with the first listed' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Python app detected + remote: + remote: ! Warning: Multiple Python package manager files were found. + remote: ! + remote: ! Exactly one package manager file should be present in your app's + remote: ! source code, however, several were found: + remote: ! + remote: ! Pipfile.lock \\(Pipenv\\) + remote: ! requirements.txt \\(pip\\) + remote: ! poetry.lock \\(Poetry\\) + remote: ! + remote: ! For now, we will build your app using the first package manager + remote: ! listed above, however, in the future this warning will become + remote: ! an error. + remote: ! + remote: ! Decide which package manager you want to use with your app, and + remote: ! then delete the file\\(s\\) and any config from the others. + remote: + remote: + remote: ! Note: We recently added support for the package manager Poetry. + remote: ! If you are using a third-party Poetry buildpack you must remove + remote: ! it, otherwise the requirements.txt file it generates will cause + remote: ! the warning above. + remote: + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} + remote: -----> Installing Pipenv #{PIPENV_VERSION} + remote: -----> Installing dependencies using 'pipenv install --deploy' + remote: Installing dependencies from Pipfile.lock \\(.+\\)... + REGEX + end + end + end + + context 'when the package manager has changed since the last build' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pip_basic') } + + it 'clears the cache before installing with the new package manager' do + app.deploy do |app| + FileUtils.rm('requirements.txt') + FileUtils.cp(FIXTURE_DIR.join('poetry_basic/pyproject.toml'), '.') + FileUtils.cp(FIXTURE_DIR.join('poetry_basic/poetry.lock'), '.') + app.commit! + app.push! + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + remote: -----> Discarding cache since: + remote: - The package manager has changed from pip to poetry + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Installing Poetry #{POETRY_VERSION} + remote: -----> Installing dependencies using 'poetry sync --only main' + remote: Installing dependencies from lock file + remote: + remote: Package operations: 1 install, 0 updates, 0 removals + remote: + remote: - Installing typing-extensions (4.12.2) + remote: -----> Discovering process types + OUTPUT + end + end + end end diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index a9cb7d595..d4990ae33 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -2,19 +2,10 @@ require_relative '../spec_helper' -RSpec.shared_examples 'installs successfully using pip' do - it 'installs successfully using pip' do - app.deploy do |app| - expect(app.output).to include("Installing dependencies using 'pip install -r requirements.txt'") - expect(app.output).to include('Successfully installed') - end - end -end - RSpec.describe 'pip support' do context 'when requirements.txt is unchanged since the last build' do let(:buildpacks) { [:default, 'heroku-community/inline'] } - let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_basic', buildpacks:) } + let(:app) { Hatchet::Runner.new('spec/fixtures/pip_basic', buildpacks:) } it 're-uses packages from the cache' do app.deploy do |app| @@ -69,7 +60,7 @@ end context 'when requirements.txt has changed since the last build' do - let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_basic') } + let(:app) { Hatchet::Runner.new('spec/fixtures/pip_basic') } it 'clears the cache before installing the packages again' do app.deploy do |app| @@ -99,46 +90,11 @@ end end - context 'when the package manager has changed from Pipenv to pip since the last build' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_basic') } - - it 'clears the cache before installing with pip' do - app.deploy do |app| - FileUtils.rm(['Pipfile', 'Pipfile.lock']) - FileUtils.cp(FIXTURE_DIR.join('requirements_basic/.python-version'), '.') - FileUtils.cp(FIXTURE_DIR.join('requirements_basic/requirements.txt'), '.') - app.commit! - app.push! - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version - remote: -----> Discarding cache since: - remote: - The package manager has changed from pipenv to pip - remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION} - remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 5)) - remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) - remote: Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) - remote: Installing collected packages: typing-extensions - remote: Successfully installed typing-extensions-4.12.2 - OUTPUT - end - end - end - - # TODO: Switch this to using Python 3.13 as part of the sqlite removal. - context 'when requirements.txt contains popular compiled packages' do - let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_compiled') } - - include_examples 'installs successfully using pip' - end - # This test intentionally uses Python 3.12, so that we test rewriting using older globally installed # setuptools. The Poetry equivalent of this test covers the PEP-517/518 setuptools case. context 'when requirements.txt contains editable requirements (both VCS and local package)' do let(:buildpacks) { [:default, 'heroku-community/inline'] } - let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_editable', buildpacks:) } + let(:app) { Hatchet::Runner.new('spec/fixtures/pip_editable', buildpacks:) } it 'rewrites .pth, .egg-link and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do app.deploy do |app| @@ -207,42 +163,18 @@ end end - context 'when there is only a setup.py' do - let(:app) { Hatchet::Runner.new('spec/fixtures/setup_py_only') } + context 'when requirements.txt contains a package that needs compiling against the Python headers' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pip_compiled') } - it 'installs packages from setup.py' do + it 'installs successfully using pip' do app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) - remote: -----> Python app detected - remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version - remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION} - remote: -----> Installing dependencies using 'pip install --editable .' - remote: Obtaining file:///tmp/build_.* - remote: .+ - remote: Installing collected packages: six, test - remote: Successfully installed six-.+ test-0.0.0 - REGEX - end - end - end - - context 'when there is both a requirements.txt and setup.py' do - let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_txt_and_setup_py') } - - it 'installs packages only from requirements.txt' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting urllib3 (from -r requirements.txt (line 1)) - OUTPUT - expect(app.output).not_to include('Running setup.py develop') + expect(app.output).to include('Building wheel for extension.dist') end end end context 'when requirements.txt contains an invalid requirement' do - let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_invalid', allow_failure: true) } + let(:app) { Hatchet::Runner.new('spec/fixtures/pip_invalid_requirement', allow_failure: true) } it 'aborts the build and displays the pip error' do app.deploy do |app| @@ -263,7 +195,7 @@ end context 'when requirements.txt contains GDAL but the GDAL C++ library is missing' do - let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_gdal', allow_failure: true) } + let(:app) { Hatchet::Runner.new('spec/fixtures/pip_gdal', allow_failure: true) } it 'outputs instructions for how to resolve the build failure' do app.deploy do |app| @@ -287,7 +219,7 @@ end context 'when requirements.txt contains an old version of Celery with invalid metadata' do - let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_legacy_celery', allow_failure: true) } + let(:app) { Hatchet::Runner.new('spec/fixtures/pip_legacy_celery', allow_failure: true) } it 'outputs instructions for how to resolve the build failure' do app.deploy do |app| @@ -314,4 +246,15 @@ end end end + + # TODO: Switch this to using Python 3.13 as part of the sqlite removal. + context 'when requirements.txt contains pysqlite3' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pip_pysqlite3') } + + it 'installs successfully using pip' do + app.deploy do |app| + expect(app.output).to include('Building wheel for pysqlite3') + end + end + end end diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 4a44a7825..b97bfe7b9 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -302,7 +302,7 @@ end end - context 'when the Pipenv version has changed since the last build' do + context 'when the Pipenv and Python versions have changed since the last build' do let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v253'] } let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_basic', buildpacks:) } @@ -328,96 +328,6 @@ end end - context 'when the package manager has changed from pip to Pipenv since the last build' do - let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_basic') } - - it 'clears the cache before installing with Pipenv' do - app.deploy do |app| - FileUtils.rm('.python-version') - FileUtils.rm('requirements.txt') - FileUtils.cp(FIXTURE_DIR.join('pipenv_basic/Pipfile'), '.') - FileUtils.cp(FIXTURE_DIR.join('pipenv_basic/Pipfile.lock'), '.') - app.commit! - app.push! - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) - remote: -----> Python app detected - remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock - remote: -----> Discarding cache since: - remote: - The package manager has changed from pip to pipenv - remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION} - remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: Installing dependencies from Pipfile.lock \\(.+\\)... - remote: -----> Discovering process types - REGEX - end - end - end - - context 'when there is both a Pipfile.lock and a requirements.txt' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_and_requirements_txt') } - - it 'builds with Pipenv rather than pip' do - app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) - remote: -----> Python app detected - remote: - remote: ! Warning: Multiple Python package manager files were found. - remote: ! - remote: ! Exactly one package manager file should be present in your app's - remote: ! source code, however, several were found: - remote: ! - remote: ! Pipfile.lock \\(Pipenv\\) - remote: ! requirements.txt \\(pip\\) - remote: ! - remote: ! For now, we will build your app using the first package manager - remote: ! listed above, however, in the future this warning will become - remote: ! an error. - remote: ! - remote: ! Decide which package manager you want to use with your app, and - remote: ! then delete the file\\(s\\) and any config from the others. - remote: - remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock - remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION} - remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: Installing dependencies from Pipfile.lock \\(.+\\)... - REGEX - end - end - end - - # This tests not only our handling of failing dependency installation, but also that we're running - # Pipenv in such a way that it errors if the lockfile is out of sync, rather than simply updating it. - context 'when the Pipfile.lock is out of sync with Pipfile' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_lockfile_out_of_sync', allow_failure: true) } - - it 'fails the build' do - app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) - remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION} - remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: Your Pipfile.lock \\(.+\\) is out of date. Expected: \\(.+\\). - remote: .+ - remote: ERROR:: Aborting deploy - remote: - remote: ! Error: Unable to install dependencies using Pipenv. - remote: ! - remote: ! See the log output above for more information. - remote: - remote: ! Push rejected, failed to compile Python app. - REGEX - end - end - end - context 'when Pipfile contains editable requirements' do let(:buildpacks) { [:default, 'heroku-community/inline'] } let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_editable', buildpacks:) } @@ -488,4 +398,33 @@ end end end + + # This tests not only our handling of failing dependency installation, but also that we're running + # Pipenv in such a way that it errors if the lockfile is out of sync, rather than simply updating it. + context 'when the Pipfile.lock is out of sync with Pipfile' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_lockfile_out_of_sync', allow_failure: true) } + + it 'fails the build' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + remote: -----> Python app detected + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} + remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} + remote: -----> Installing Pipenv #{PIPENV_VERSION} + remote: -----> Installing dependencies using 'pipenv install --deploy' + remote: Your Pipfile.lock \\(.+\\) is out of date. Expected: \\(.+\\). + remote: .+ + remote: ERROR:: Aborting deploy + remote: + remote: ! Error: Unable to install dependencies using Pipenv. + remote: ! + remote: ! See the log output above for more information. + remote: + remote: ! Push rejected, failed to compile Python app. + REGEX + end + end + end end diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 24e23c236..e7e85d446 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -61,7 +61,7 @@ end end - context 'when the requested Python version has changed since the last build' do + context 'when the Poetry and Python versions have changed since the last build' do let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v268'] } let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_basic', buildpacks:) } @@ -90,35 +90,6 @@ end end - context 'when the package manager has changed from pip to Poetry since the last build' do - let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_basic') } - - it 'clears the cache before installing with Poetry' do - app.deploy do |app| - FileUtils.rm('requirements.txt') - FileUtils.cp(FIXTURE_DIR.join('poetry_basic/pyproject.toml'), '.') - FileUtils.cp(FIXTURE_DIR.join('poetry_basic/poetry.lock'), '.') - app.commit! - app.push! - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version - remote: -----> Discarding cache since: - remote: - The package manager has changed from pip to poetry - remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing Poetry #{POETRY_VERSION} - remote: -----> Installing dependencies using 'poetry sync --only main' - remote: Installing dependencies from lock file - remote: - remote: Package operations: 1 install, 0 updates, 0 removals - remote: - remote: - Installing typing-extensions (4.12.2) - remote: -----> Discovering process types - OUTPUT - end - end - end - context 'when poetry.lock contains editable requirements (both VCS and local package)' do let(:buildpacks) { [:default, 'heroku-community/inline'] } let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_editable', buildpacks:) } @@ -254,43 +225,4 @@ end end end - - # This case will be turned into an error in the future, but for now is required for backwards compatibility. - context 'when there is both a poetry.lock and a requirements.txt' do - let(:app) { Hatchet::Runner.new('spec/fixtures/requirements_txt_and_poetry_lock') } - - it 'build using pip rather than Poetry' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: - remote: ! Warning: Multiple Python package manager files were found. - remote: ! - remote: ! Exactly one package manager file should be present in your app's - remote: ! source code, however, several were found: - remote: ! - remote: ! requirements.txt (pip) - remote: ! poetry.lock (Poetry) - remote: ! - remote: ! For now, we will build your app using the first package manager - remote: ! listed above, however, in the future this warning will become - remote: ! an error. - remote: ! - remote: ! Decide which package manager you want to use with your app, and - remote: ! then delete the file(s) and any config from the others. - remote: - remote: - remote: ! Note: We recently added support for the package manager Poetry. - remote: ! If you are using a third-party Poetry buildpack you must remove - remote: ! it, otherwise the requirements.txt file it generates will cause - remote: ! the warning above. - remote: - remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version - remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION} - remote: -----> Installing dependencies using 'pip install -r requirements.txt' - OUTPUT - end - end - end end diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 3b547f7ee..635bca936 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -50,8 +50,12 @@ context 'with an app last built using an older default Python version' do # This test performs an initial build using an older buildpack version, followed - # by a build using the current version. This ensures that the current buildpack - # can successfully read the version metadata written to the build cache in the past. + # by a build using the current version. This ensures that: + # - The current buildpack can successfully read the version metadata + # written to the build cache by older buildpack versions. + # - If no Python version is specified, the same major version as the + # last build is used (sticky versioning). + # - Changes in the pip version are handled correctly. let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v257'] } it 'builds with the same Python version as the last build' do From 17b87c0205e3879343c52159947139c238c25a6d Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 27 Jan 2025 11:45:56 +0000 Subject: [PATCH 091/406] Update make target in `bin/report` debug instructions (#1739) Since the command is now named `make run` after #1597. --- bin/report | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/report b/bin/report index 54325611d..f05d67b6c 100755 --- a/bin/report +++ b/bin/report @@ -15,7 +15,7 @@ # Failures in this script don't cause the overall build to fail (and won't appear in user # facing build logs) to avoid breaking builds unnecessarily / causing confusion. To debug # issues check the internal build system logs for `buildpack.report.failed` events, or -# when developing run `make compile` in this repo locally, which runs `bin/report` too. +# when developing run `make run` in this repo locally, which runs `bin/report` too. set -euo pipefail shopt -s inherit_errexit From b3e0b29a867ed6a9f8cb53341f713791a4fbab0d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 3 Feb 2025 08:34:11 +0000 Subject: [PATCH 092/406] Bump the ruby-dependencies group with 2 updates (#1742) Bumps the ruby-dependencies group with 2 updates: [rubocop](https://github.com/rubocop/rubocop) and [rubocop-rspec](https://github.com/rubocop/rubocop-rspec). Updates `rubocop` from 1.70.0 to 1.71.1 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.70.0...v1.71.1) Updates `rubocop-rspec` from 3.3.0 to 3.4.0 - [Release notes](https://github.com/rubocop/rubocop-rspec/releases) - [Changelog](https://github.com/rubocop/rubocop-rspec/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop-rspec/compare/v3.3.0...v3.4.0) --- updated-dependencies: - dependency-name: rubocop dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies - dependency-name: rubocop-rspec dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index d73682f90..b708a7e83 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -20,7 +20,7 @@ GEM thor (~> 1) threaded (~> 0) json (2.9.1) - language_server-protocol (3.17.0.3) + language_server-protocol (3.17.0.4) logger (1.6.5) moneta (1.0.0) multi_json (1.15.0) @@ -28,7 +28,7 @@ GEM parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) - parser (3.3.6.0) + parser (3.3.7.0) ast (~> 2.4.1) racc platform-api (3.7.0) @@ -48,19 +48,19 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.2) - rubocop (1.70.0) + rubocop (1.71.1) json (~> 2.3) language_server-protocol (>= 3.17.0) parallel (~> 1.10) parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) regexp_parser (>= 2.9.3, < 3.0) - rubocop-ast (>= 1.36.2, < 2.0) + rubocop-ast (>= 1.38.0, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 4.0) - rubocop-ast (1.37.0) + rubocop-ast (1.38.0) parser (>= 3.3.1.0) - rubocop-rspec (3.3.0) + rubocop-rspec (3.4.0) rubocop (~> 1.61) ruby-progressbar (1.13.0) thor (1.3.2) From a61ce1c6c20fabff3a31418bcc65ada255c8b8a6 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 5 Feb 2025 00:24:00 +0000 Subject: [PATCH 093/406] Deprecate support for the `runtime.txt` file (#1743) The `runtime.txt` file is a classic Heroku Python buildpack invention that's not widely supported in the Python ecosystem. Instead, most other tooling (pyenv, package managers, GitHub Actions, dependency update bots etc) support/use the `.python-version` file. As such, we recently added `.python-version` support to both the Python CNB and the classic Python buildpack, and updated all documentation and guides to use it instead of the `runtime.txt` file. eg: https://devcenter.heroku.com/articles/python-runtimes We would prefer apps use the new file, since it helps ensure their deployed app is using the same Python version used locally (via eg pyenv or uv) or in CI. As such this adds a deprecation warning for apps using `runtime.txt`. Closes #1642. GUS-W-16878260. --- CHANGELOG.md | 2 + README.md | 3 + bin/compile | 26 +++++- lib/python_version.sh | 101 +++++++++++++-------- spec/hatchet/pipenv_spec.rb | 40 ++++---- spec/hatchet/python_update_warning_spec.rb | 22 +++++ spec/hatchet/python_version_spec.rb | 82 ++++++++++++----- 7 files changed, 197 insertions(+), 79 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index caeaa579b..ca8032023 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## [Unreleased] +- Deprecated support for the `runtime.txt` file. ([#1743](https://github.com/heroku/heroku-buildpack-python/pull/1743)) +- Improved the error messages shown when `.python-version`, `runtime.txt` or `Pipfile.lock` contain an invalid Python version. ([#1743](https://github.com/heroku/heroku-buildpack-python/pull/1743)) ## [v275] - 2025-01-13 diff --git a/README.md b/README.md index 3c76ddc14..557d77e6d 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,9 @@ For example, to request the latest patch release of Python 3.13, create a `.pyth the root directory of your app containing: `3.13` +We strongly recommend that you use the major version form instead of pinning to an exact version, +since it will allow your app to receive Python security updates. + The buildpack will look for a Python version in the following places (in descending order of precedence): 1. `runtime.txt` file (deprecated) diff --git a/bin/compile b/bin/compile index 08232b5e7..10a2187ce 100755 --- a/bin/compile +++ b/bin/compile @@ -125,7 +125,6 @@ meta_set "python_version_reason" "${python_version_origin}" # TODO: More strongly recommend specifying a Python version (eg switch the messaging to # be a warning instead, after version resolution, and mention .python-version inline) -# TODO: Add runtime.txt deprecation warning. case "${python_version_origin}" in default) output::step "No Python version was specified. Using the buildpack default: Python ${requested_python_version}" @@ -145,6 +144,31 @@ python_major_version="${python_full_version%.*}" meta_set "python_version" "${python_full_version}" meta_set "python_version_major" "${python_major_version}" +if [[ "${python_version_origin}" == "runtime.txt" ]]; then + output::warning <<-EOF + Warning: The runtime.txt file is deprecated. + + The runtime.txt file is deprecated since it has been replaced + by the more widely supported .python-version file. + + Please delete your runtime.txt file and create a new file named: + .python-version + + Make sure to include the '.' at the start of the filename. + + In the new file, specify your app's Python version without + quotes or a 'python-' prefix. For example: + ${python_major_version} + + We strongly recommend that you use the major version form + instead of pinning to an exact version, since it will allow + your app to receive Python security updates. + + In the future support for runtime.txt will be removed and + this warning will be made an error. + EOF +fi + cache::restore "${BUILD_DIR}" "${CACHE_DIR}" "${STACK}" "${cached_python_full_version}" "${python_full_version}" "${package_manager}" # The directory for the .profile.d scripts. diff --git a/lib/python_version.sh b/lib/python_version.sh index f7bacb6ad..eaf201e9a 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -30,10 +30,10 @@ PYTHON_FULL_VERSION_REGEX="${INT_REGEX}\.${INT_REGEX}\.${INT_REGEX}" # resolved to an exact Python version. # # If an app specifies the Python version via multiple means, then the order of precedence is: -# 1. runtime.txt -# 2. .python-version -# 3. Pipfile.lock (`python_full_version` field) -# 4. Pipfile.lock (`python_version` field) +# 1. `runtime.txt` file (deprecated) +# 2. `.python-version` file (recommended) +# 3. The `python_full_version` field in the `Pipfile.lock` file +# 4. The `python_version` field in the `Pipfile.lock` file # # If a version wasn't specified by the app, then new apps/those with an empty cache will use # a buildpack default version for the first build, and then subsequent cached builds will use @@ -100,21 +100,29 @@ function python_version::parse_runtime_txt() { output::error <<-EOF Error: Invalid Python version in runtime.txt. - The Python version specified in 'runtime.txt' isn't in - the correct format. + The Python version specified in your runtime.txt file isn't + in the correct format. - The following file contents were found: + The following file contents were found, which aren't valid: ${contents} - However, the version must be specified as either: - 1. 'python-.' (recommended, for automatic patch updates) - 2. 'python-..' (to pin to an exact patch version) + However, the runtime.txt file is deprecated since it has + been replaced by the .python-version file. As such, we + recommend that you switch to using a .python-version file + instead of fixing your runtime.txt file. - Remember to include the 'python-' prefix. Comments aren't supported. + Please delete your runtime.txt file and create a new file named: + .python-version - For example, to request the latest version of Python ${DEFAULT_PYTHON_MAJOR_VERSION}, - update the 'runtime.txt' file so it contains: - python-${DEFAULT_PYTHON_MAJOR_VERSION} + Make sure to include the '.' at the start of the filename. + + In the new file, specify your app's Python version without + quotes or a 'python-' prefix. For example: + ${DEFAULT_PYTHON_MAJOR_VERSION} + + We strongly recommend that you use the major version form + instead of pinning to an exact version, since it will allow + your app to receive Python security updates. EOF meta_set "failure_reason" "runtime-txt::invalid-version" exit 1 @@ -144,22 +152,26 @@ function python_version::parse_python_version_file() { output::error <<-EOF Error: Invalid Python version in .python-version. - The Python version specified in '.python-version' isn't in - the correct format. + The Python version specified in your .python-version file + isn't in the correct format. The following version was found: ${line} - However, the version must be specified as either: - 1. '.' (recommended, for automatic patch updates) - 2. '..' (to pin to an exact patch version) + However, the Python version must be specified as either: + 1. The major version only: 3.X (recommended) + 2. An exact patch version: 3.X.Y Don't include quotes or a 'python-' prefix. To include comments, add them on their own line, prefixed with '#'. For example, to request the latest version of Python ${DEFAULT_PYTHON_MAJOR_VERSION}, - update the '.python-version' file so it contains: + update your .python-version file so it contains: ${DEFAULT_PYTHON_MAJOR_VERSION} + + We strongly recommend that you use the major version form + instead of pinning to an exact version, since it will allow + your app to receive Python security updates. EOF meta_set "failure_reason" "python-version-file::invalid-version" exit 1 @@ -169,10 +181,13 @@ function python_version::parse_python_version_file() { output::error <<-EOF Error: Invalid Python version in .python-version. - No Python version was found in the '.python-version' file. + No Python version was found in your .python-version file. - Update the file so that it contains a valid Python version - such as '${DEFAULT_PYTHON_MAJOR_VERSION}'. + Update the file so that it contains a valid Python version. + + For example, to request the latest version of Python ${DEFAULT_PYTHON_MAJOR_VERSION}, + update your .python-version file so it contains: + ${DEFAULT_PYTHON_MAJOR_VERSION} If the file already contains a version, check the line doesn't begin with a '#', otherwise it will be treated as a comment. @@ -184,8 +199,7 @@ function python_version::parse_python_version_file() { output::error <<-EOF Error: Invalid Python version in .python-version. - Multiple Python versions were found in the '.python-version' - file: + Multiple versions were found in your .python-version file: $( IFS=$'\n' @@ -194,8 +208,8 @@ function python_version::parse_python_version_file() { Update the file so it contains only one Python version. - If the additional versions are actually comments, prefix - those lines with '#'. + If you have added comments to the file, make sure that those + lines begin with a '#', so that they are ignored. EOF meta_set "failure_reason" "python-version-file::multiple-versions" exit 1 @@ -245,20 +259,24 @@ function python_version::read_pipenv_python_version() { echo "${version}" else output::error <<-EOF - Error: Invalid Python version in Pipfile / Pipfile.lock. + Error: Invalid Python version in Pipfile.lock. - The Python version specified in Pipfile / Pipfile.lock by the - 'python_version' or 'python_full_version' field isn't valid. + The Python version specified in your Pipfile.lock file by the + 'python_version' or 'python_full_version' fields isn't valid. The following version was found: ${version} - However, the version must be specified as either: - 1. '.' (recommended, for automatic patch updates) - 2. '..' (to pin to an exact patch version) + However, the Python version must be specified as either: + 1. The major version only: 3.X (recommended) + 2. An exact patch version: 3.X.Y + + Please update your Pipfile to use a valid Python version and + then run 'pipenv lock' to regenerate Pipfile.lock. - Please update your 'Pipfile' to use a valid Python version and - then run 'pipenv lock' to regenerate the lockfile. + We strongly recommend that you use the major version form + instead of pinning to an exact version, since it will allow + your app to receive Python security updates. For more information, see: https://pipenv.pypa.io/en/stable/specifiers.html#specifying-versions-of-python @@ -297,10 +315,15 @@ function python_version::resolve_python_version() { As such, it's no longer supported by this buildpack: https://devcenter.heroku.com/articles/python-support#supported-python-versions - Please upgrade to at least Python 3.${OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION} by creating a - .python-version file in the root directory of your app, - that contains a Python version like: - 3.${OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION} + Please upgrade to at least Python 3.${OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION} by configuring an + explicit Python version for your app. + + Create a .python-version file in the root directory of your + app, that contains a Python version like: + 3.${NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION} + + When creating this file make sure to include the '.' at the + start of the filename. EOF else output::error <<-EOF diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index b97bfe7b9..6949e3934 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -219,20 +219,24 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: - remote: ! Error: Invalid Python version in Pipfile / Pipfile.lock. + remote: ! Error: Invalid Python version in Pipfile.lock. remote: ! - remote: ! The Python version specified in Pipfile / Pipfile.lock by the - remote: ! 'python_version' or 'python_full_version' field isn't valid. + remote: ! The Python version specified in your Pipfile.lock file by the + remote: ! 'python_version' or 'python_full_version' fields isn't valid. remote: ! remote: ! The following version was found: remote: ! ^3.12 remote: ! - remote: ! However, the version must be specified as either: - remote: ! 1. '.' (recommended, for automatic patch updates) - remote: ! 2. '..' (to pin to an exact patch version) + remote: ! However, the Python version must be specified as either: + remote: ! 1. The major version only: 3.X (recommended) + remote: ! 2. An exact patch version: 3.X.Y remote: ! - remote: ! Please update your 'Pipfile' to use a valid Python version and - remote: ! then run 'pipenv lock' to regenerate the lockfile. + remote: ! Please update your Pipfile to use a valid Python version and + remote: ! then run 'pipenv lock' to regenerate Pipfile.lock. + remote: ! + remote: ! We strongly recommend that you use the major version form + remote: ! instead of pinning to an exact version, since it will allow + remote: ! your app to receive Python security updates. remote: ! remote: ! For more information, see: remote: ! https://pipenv.pypa.io/en/stable/specifiers.html#specifying-versions-of-python @@ -251,20 +255,24 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: - remote: ! Error: Invalid Python version in Pipfile / Pipfile.lock. + remote: ! Error: Invalid Python version in Pipfile.lock. remote: ! - remote: ! The Python version specified in Pipfile / Pipfile.lock by the - remote: ! 'python_version' or 'python_full_version' field isn't valid. + remote: ! The Python version specified in your Pipfile.lock file by the + remote: ! 'python_version' or 'python_full_version' fields isn't valid. remote: ! remote: ! The following version was found: remote: ! 3.9.* remote: ! - remote: ! However, the version must be specified as either: - remote: ! 1. '.' (recommended, for automatic patch updates) - remote: ! 2. '..' (to pin to an exact patch version) + remote: ! However, the Python version must be specified as either: + remote: ! 1. The major version only: 3.X (recommended) + remote: ! 2. An exact patch version: 3.X.Y + remote: ! + remote: ! Please update your Pipfile to use a valid Python version and + remote: ! then run 'pipenv lock' to regenerate Pipfile.lock. remote: ! - remote: ! Please update your 'Pipfile' to use a valid Python version and - remote: ! then run 'pipenv lock' to regenerate the lockfile. + remote: ! We strongly recommend that you use the major version form + remote: ! instead of pinning to an exact version, since it will allow + remote: ! your app to receive Python security updates. remote: ! remote: ! For more information, see: remote: ! https://pipenv.pypa.io/en/stable/specifiers.html#specifying-versions-of-python diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index 09123a5b6..23365eaba 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -14,6 +14,28 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python 3.9.0 specified in runtime.txt + remote: + remote: ! Warning: The runtime.txt file is deprecated. + remote: ! + remote: ! The runtime.txt file is deprecated since it has been replaced + remote: ! by the more widely supported .python-version file. + remote: ! + remote: ! Please delete your runtime.txt file and create a new file named: + remote: ! .python-version + remote: ! + remote: ! Make sure to include the '.' at the start of the filename. + remote: ! + remote: ! In the new file, specify your app's Python version without + remote: ! quotes or a 'python-' prefix. For example: + remote: ! 3.9 + remote: ! + remote: ! We strongly recommend that you use the major version form + remote: ! instead of pinning to an exact version, since it will allow + remote: ! your app to receive Python security updates. + remote: ! + remote: ! In the future support for runtime.txt will be removed and + remote: ! this warning will be made an error. + remote: remote: -----> Installing Python 3.9.0 remote: remote: ! Warning: Support for Python 3.9 is ending soon! diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 635bca936..e01771661 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -148,22 +148,26 @@ remote: remote: ! Error: Invalid Python version in .python-version. remote: ! - remote: ! The Python version specified in '.python-version' isn't in - remote: ! the correct format. + remote: ! The Python version specified in your .python-version file + remote: ! isn't in the correct format. remote: ! remote: ! The following version was found: remote: ! 3.12.0invalid remote: ! - remote: ! However, the version must be specified as either: - remote: ! 1. '.' (recommended, for automatic patch updates) - remote: ! 2. '..' (to pin to an exact patch version) + remote: ! However, the Python version must be specified as either: + remote: ! 1. The major version only: 3.X (recommended) + remote: ! 2. An exact patch version: 3.X.Y remote: ! remote: ! Don't include quotes or a 'python-' prefix. To include remote: ! comments, add them on their own line, prefixed with '#'. remote: ! remote: ! For example, to request the latest version of Python #{DEFAULT_PYTHON_MAJOR_VERSION}, - remote: ! update the '.python-version' file so it contains: + remote: ! update your .python-version file so it contains: remote: ! #{DEFAULT_PYTHON_MAJOR_VERSION} + remote: ! + remote: ! We strongly recommend that you use the major version form + remote: ! instead of pinning to an exact version, since it will allow + remote: ! your app to receive Python security updates. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -181,10 +185,13 @@ remote: remote: ! Error: Invalid Python version in .python-version. remote: ! - remote: ! No Python version was found in the '.python-version' file. + remote: ! No Python version was found in your .python-version file. + remote: ! + remote: ! Update the file so that it contains a valid Python version. remote: ! - remote: ! Update the file so that it contains a valid Python version - remote: ! such as '#{DEFAULT_PYTHON_MAJOR_VERSION}'. + remote: ! For example, to request the latest version of Python #{DEFAULT_PYTHON_MAJOR_VERSION}, + remote: ! update your .python-version file so it contains: + remote: ! #{DEFAULT_PYTHON_MAJOR_VERSION} remote: ! remote: ! If the file already contains a version, check the line doesn't remote: ! begin with a '#', otherwise it will be treated as a comment. @@ -205,8 +212,7 @@ remote: remote: ! Error: Invalid Python version in .python-version. remote: ! - remote: ! Multiple Python versions were found in the '.python-version' - remote: ! file: + remote: ! Multiple versions were found in your .python-version file: remote: ! remote: ! // invalid comment remote: ! 3.12 @@ -214,8 +220,8 @@ remote: ! remote: ! Update the file so it contains only one Python version. remote: ! - remote: ! If the additional versions are actually comments, prefix - remote: ! those lines with '#'. + remote: ! If you have added comments to the file, make sure that those + remote: ! lines begin with a '#', so that they are ignored. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -312,21 +318,29 @@ remote: remote: ! Error: Invalid Python version in runtime.txt. remote: ! - remote: ! The Python version specified in 'runtime.txt' isn't in - remote: ! the correct format. + remote: ! The Python version specified in your runtime.txt file isn't + remote: ! in the correct format. remote: ! - remote: ! The following file contents were found: + remote: ! The following file contents were found, which aren't valid: remote: ! python-3.12.0invalid remote: ! - remote: ! However, the version must be specified as either: - remote: ! 1. 'python-.' (recommended, for automatic patch updates) - remote: ! 2. 'python-..' (to pin to an exact patch version) + remote: ! However, the runtime.txt file is deprecated since it has + remote: ! been replaced by the .python-version file. As such, we + remote: ! recommend that you switch to using a .python-version file + remote: ! instead of fixing your runtime.txt file. remote: ! - remote: ! Remember to include the 'python-' prefix. Comments aren't supported. + remote: ! Please delete your runtime.txt file and create a new file named: + remote: ! .python-version remote: ! - remote: ! For example, to request the latest version of Python #{DEFAULT_PYTHON_MAJOR_VERSION}, - remote: ! update the 'runtime.txt' file so it contains: - remote: ! python-#{DEFAULT_PYTHON_MAJOR_VERSION} + remote: ! Make sure to include the '.' at the start of the filename. + remote: ! + remote: ! In the new file, specify your app's Python version without + remote: ! quotes or a 'python-' prefix. For example: + remote: ! #{DEFAULT_PYTHON_MAJOR_VERSION} + remote: ! + remote: ! We strongly recommend that you use the major version form + remote: ! instead of pinning to an exact version, since it will allow + remote: ! your app to receive Python security updates. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -371,6 +385,28 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python 3.13 specified in runtime.txt + remote: + remote: ! Warning: The runtime.txt file is deprecated. + remote: ! + remote: ! The runtime.txt file is deprecated since it has been replaced + remote: ! by the more widely supported .python-version file. + remote: ! + remote: ! Please delete your runtime.txt file and create a new file named: + remote: ! .python-version + remote: ! + remote: ! Make sure to include the '.' at the start of the filename. + remote: ! + remote: ! In the new file, specify your app's Python version without + remote: ! quotes or a 'python-' prefix. For example: + remote: ! 3.13 + remote: ! + remote: ! We strongly recommend that you use the major version form + remote: ! instead of pinning to an exact version, since it will allow + remote: ! your app to receive Python security updates. + remote: ! + remote: ! In the future support for runtime.txt will be removed and + remote: ! this warning will be made an error. + remote: remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' From 7531824c10b1e99a1b39101dd132725b1ebb0534 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 5 Feb 2025 00:33:09 +0000 Subject: [PATCH 094/406] Add support for Python 3.13.2 and 3.12.9 (#1744) Release announcement: https://blog.python.org/2025/02/python-3132-and-3129-now-available.html Changelog: https://docs.python.org/3.13/whatsnew/changelog.html#python-3-13-2-final https://docs.python.org/3.12/whatsnew/changelog.html#python-3-12-9-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/13146826968 https://github.com/heroku/heroku-buildpack-python/actions/runs/13146829096 GUS-W-17476919. GUS-W-17476929. --- CHANGELOG.md | 2 ++ lib/python_version.sh | 4 ++-- spec/spec_helper.rb | 4 ++-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ca8032023..541890adb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## [Unreleased] +- The Python 3.13 version alias now resolves to Python 3.13.2. ([#1744](https://github.com/heroku/heroku-buildpack-python/pull/1744)) +- The Python 3.12 version alias now resolves to Python 3.12.9. ([#1744](https://github.com/heroku/heroku-buildpack-python/pull/1744)) - Deprecated support for the `runtime.txt` file. ([#1743](https://github.com/heroku/heroku-buildpack-python/pull/1743)) - Improved the error messages shown when `.python-version`, `runtime.txt` or `Pipfile.lock` contain an invalid Python version. ([#1743](https://github.com/heroku/heroku-buildpack-python/pull/1743)) diff --git a/lib/python_version.sh b/lib/python_version.sh index eaf201e9a..eec397496 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -7,8 +7,8 @@ set -euo pipefail LATEST_PYTHON_3_9="3.9.21" LATEST_PYTHON_3_10="3.10.16" LATEST_PYTHON_3_11="3.11.11" -LATEST_PYTHON_3_12="3.12.8" -LATEST_PYTHON_3_13="3.13.1" +LATEST_PYTHON_3_12="3.12.9" +LATEST_PYTHON_3_13="3.13.2" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=9 NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=13 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index af1ca38cf..b8e49eb22 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -12,8 +12,8 @@ LATEST_PYTHON_3_9 = '3.9.21' LATEST_PYTHON_3_10 = '3.10.16' LATEST_PYTHON_3_11 = '3.11.11' -LATEST_PYTHON_3_12 = '3.12.8' -LATEST_PYTHON_3_13 = '3.13.1' +LATEST_PYTHON_3_12 = '3.12.9' +LATEST_PYTHON_3_13 = '3.13.2' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_13 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From c097dfec4d80c99b1948006d517568365c6c06ac Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 5 Feb 2025 11:32:19 +0000 Subject: [PATCH 095/406] Prepare release v276 (#1745) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 541890adb..c7817d81e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v276] - 2025-02-05 + - The Python 3.13 version alias now resolves to Python 3.13.2. ([#1744](https://github.com/heroku/heroku-buildpack-python/pull/1744)) - The Python 3.12 version alias now resolves to Python 3.12.9. ([#1744](https://github.com/heroku/heroku-buildpack-python/pull/1744)) - Deprecated support for the `runtime.txt` file. ([#1743](https://github.com/heroku/heroku-buildpack-python/pull/1743)) @@ -1154,7 +1157,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v275...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v276...main +[v276]: https://github.com/heroku/heroku-buildpack-python/compare/v275...v276 [v275]: https://github.com/heroku/heroku-buildpack-python/compare/v274...v275 [v274]: https://github.com/heroku/heroku-buildpack-python/compare/v273...v274 [v273]: https://github.com/heroku/heroku-buildpack-python/compare/v272...v273 From 20a782981498143afe584247f0eacd16fb793de7 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 12 Feb 2025 11:16:08 +0000 Subject: [PATCH 096/406] Improve metrics for builds that fail (#1746) Adds a `failure_detail` field in addition to the existing `failure_reason`, which contains additional context relevant to the failure where available. This will make it easier to find trends in the most frequent user-caused failure modes (eg invalid Python version specifier) so I can then adjust error messages/docs/implementation to improve UX. This context sometimes contain user input, so the values saved to the metadata store now also have additional escaping and validation performed before writing the value (in addition to the existing YAML escaping performed in `bin/report`). GUS-W-17800067. --- CHANGELOG.md | 1 + bin/report | 1 + bin/steps/python | 2 ++ lib/checks.sh | 2 ++ lib/kvstore.sh | 14 +++++++++++++- lib/python_version.sh | 19 ++++++++++++++++--- lib/utils.sh | 6 ++++-- 7 files changed, 39 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c7817d81e..7be1eea19 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Improved buildpack metrics for builds that fail. ([#1746](https://github.com/heroku/heroku-buildpack-python/pull/1746)) ## [v276] - 2025-02-05 diff --git a/bin/report b/bin/report index f05d67b6c..cc0d1a116 100755 --- a/bin/report +++ b/bin/report @@ -62,6 +62,7 @@ kv_pair_string() { STRING_FIELDS=( cache_status django_collectstatic + failure_detail failure_reason nltk_downloader package_manager diff --git a/bin/steps/python b/bin/steps/python index b345d5cd8..6855448de 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -23,6 +23,7 @@ if ! curl --output /dev/null --silent --head --fail --retry 3 --retry-connrefuse https://devcenter.heroku.com/articles/python-support#supported-python-versions EOF meta_set "failure_reason" "python-version-not-found" + meta_set "failure_detail" "${python_full_version}" exit 1 fi @@ -42,6 +43,7 @@ else Please try again and to see if the error resolves itself. EOF meta_set "failure_reason" "python-download" + # TODO: Set failure_detail here once refactored. exit 1 fi diff --git a/lib/checks.sh b/lib/checks.sh index 9129244e7..3a43cdad7 100644 --- a/lib/checks.sh +++ b/lib/checks.sh @@ -20,6 +20,7 @@ function checks::ensure_supported_stack() { Upgrade to a newer stack to continue using this buildpack. EOF meta_set "failure_reason" "stack::eol" + meta_set "failure_detail" "${stack}" exit 1 ;; *) @@ -34,6 +35,7 @@ function checks::ensure_supported_stack() { https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references EOF meta_set "failure_reason" "stack::unknown" + meta_set "failure_detail" "${stack}" exit 1 ;; esac diff --git a/lib/kvstore.sh b/lib/kvstore.sh index c4a684fea..b5c15aa60 100644 --- a/lib/kvstore.sh +++ b/lib/kvstore.sh @@ -23,8 +23,20 @@ kv_set() { # TODO: Stop ignoring an incorrect number of passed arguments. if [[ $# -eq 3 ]]; then local f="${1}" + local key="${2}" + + # Truncate the value to an arbitrary 100 characters since it will sometimes contain user-provided + # inputs which may be unbounded in size. Ideally individual call sites will perform more aggressive + # truncation themselves based on the expected value size, however this is here as a fallback. + # (Honeycomb supports string fields up to 64KB in size, however, it's not worth filling up the + # metadata store or bloating the payload passed back to Vacuole/submitted to Honeycomb given the + # extra content in those cases is not normally useful.) + local value="${3:0:100}" + # Replace newlines since the data store file format requires that keys don't span multiple lines. + value="${value//$'\n'/ }" + if [[ -f "${f}" ]]; then - echo "${2}=${3}" >>"${f}" + echo "${key}=${value}" >>"${f}" fi fi } diff --git a/lib/python_version.sh b/lib/python_version.sh index eec397496..652c4e478 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -104,7 +104,7 @@ function python_version::parse_runtime_txt() { in the correct format. The following file contents were found, which aren't valid: - ${contents} + ${contents:0:100} However, the runtime.txt file is deprecated since it has been replaced by the .python-version file. As such, we @@ -125,6 +125,7 @@ function python_version::parse_runtime_txt() { your app to receive Python security updates. EOF meta_set "failure_reason" "runtime-txt::invalid-version" + meta_set "failure_detail" "${contents:0:50}" exit 1 fi } @@ -174,6 +175,7 @@ function python_version::parse_python_version_file() { your app to receive Python security updates. EOF meta_set "failure_reason" "python-version-file::invalid-version" + meta_set "failure_detail" "${line:0:50}" exit 1 fi ;; @@ -193,9 +195,11 @@ function python_version::parse_python_version_file() { begin with a '#', otherwise it will be treated as a comment. EOF meta_set "failure_reason" "python-version-file::no-version" + meta_set "failure_detail" "${contents:0:50}" exit 1 ;; *) + local first_five_version_lines=("${version_lines[@]:0:5}") output::error <<-EOF Error: Invalid Python version in .python-version. @@ -203,7 +207,7 @@ function python_version::parse_python_version_file() { $( IFS=$'\n' - echo "${version_lines[*]}" + echo "${first_five_version_lines[*]}" ) Update the file so it contains only one Python version. @@ -212,6 +216,10 @@ function python_version::parse_python_version_file() { lines begin with a '#', so that they are ignored. EOF meta_set "failure_reason" "python-version-file::multiple-versions" + meta_set "failure_detail" "$( + IFS=, + echo "${first_five_version_lines[*]}" + )" exit 1 ;; esac @@ -233,17 +241,19 @@ function python_version::read_pipenv_python_version() { fi if ! version=$(jq --raw-output '._meta.requires.python_full_version // ._meta.requires.python_version' "${pipfile_lock_path}" 2>&1); then + local jq_error_message="${version}" output::error <<-EOF Error: Can't parse Pipfile.lock. A Pipfile.lock file was found, however, it couldn't be parsed: - ${version} + ${jq_error_message} This is likely due to it not being valid JSON. Run 'pipenv lock' to regenerate/fix the lockfile. EOF meta_set "failure_reason" "pipfile-lock::invalid-json" + meta_set "failure_detail" "${jq_error_message:0:100}" exit 1 fi @@ -282,6 +292,7 @@ function python_version::read_pipenv_python_version() { https://pipenv.pypa.io/en/stable/specifiers.html#specifying-versions-of-python EOF meta_set "failure_reason" "pipfile-lock::invalid-version" + meta_set "failure_detail" "${version:0:50}" exit 1 fi } @@ -341,6 +352,7 @@ function python_version::resolve_python_version() { EOF fi meta_set "failure_reason" "python-version::eol" + meta_set "failure_detail" "${major}.${minor}" exit 1 fi @@ -388,6 +400,7 @@ function python_version::resolve_python_version() { EOF fi meta_set "failure_reason" "python-version::unknown-major" + meta_set "failure_detail" "${major}.${minor}" exit 1 fi diff --git a/lib/utils.sh b/lib/utils.sh index 2bcae44c2..916886431 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -48,10 +48,12 @@ function utils::bundled_pip_module_path() { } function utils::abort_internal_error() { - local message="${1}" + local message + message="${1} (line $(caller || true))" output::error <<-EOF - Internal error: ${message} (line $(caller || true)). + Internal error: ${message}. EOF meta_set "failure_reason" "internal-error" + meta_set "failure_detail" "${message}" exit 1 } From a840ce3862fa1289ace40965a5f3adbaf17dfa33 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 12 Feb 2025 21:39:16 +0000 Subject: [PATCH 097/406] Link to changelog from runtime.txt deprecation message (#1747) Now that the changelog post exists: https://devcenter.heroku.com/changelog-items/3141 --- bin/compile | 3 ++- spec/hatchet/python_update_warning_spec.rb | 3 ++- spec/hatchet/python_version_spec.rb | 3 ++- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/bin/compile b/bin/compile index 10a2187ce..7fbeda2c1 100755 --- a/bin/compile +++ b/bin/compile @@ -149,7 +149,8 @@ if [[ "${python_version_origin}" == "runtime.txt" ]]; then Warning: The runtime.txt file is deprecated. The runtime.txt file is deprecated since it has been replaced - by the more widely supported .python-version file. + by the more widely supported .python-version file: + https://devcenter.heroku.com/changelog-items/3141 Please delete your runtime.txt file and create a new file named: .python-version diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index 23365eaba..4f7991dac 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -18,7 +18,8 @@ remote: ! Warning: The runtime.txt file is deprecated. remote: ! remote: ! The runtime.txt file is deprecated since it has been replaced - remote: ! by the more widely supported .python-version file. + remote: ! by the more widely supported .python-version file: + remote: ! https://devcenter.heroku.com/changelog-items/3141 remote: ! remote: ! Please delete your runtime.txt file and create a new file named: remote: ! .python-version diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index e01771661..4f8742fae 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -389,7 +389,8 @@ remote: ! Warning: The runtime.txt file is deprecated. remote: ! remote: ! The runtime.txt file is deprecated since it has been replaced - remote: ! by the more widely supported .python-version file. + remote: ! by the more widely supported .python-version file: + remote: ! https://devcenter.heroku.com/changelog-items/3141 remote: ! remote: ! Please delete your runtime.txt file and create a new file named: remote: ! .python-version From 4e8ae51d635dcdfe6e9232bb6d945fcd21ba956b Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 17 Feb 2025 18:01:40 +0000 Subject: [PATCH 098/406] Improve Python download/installation (#1749) Refactors the Python download/install and outdated version warning steps, improves error/warning messages and improves buildpack metrics. See the changelog entries for more details. Fixes #1701. Closes #1708. GUS-W-8059919. GUS-W-17844538. GUS-W-17844985. GUS-W-17845321. --- CHANGELOG.md | 8 +- bin/compile | 25 ++-- bin/report | 3 + bin/steps/python | 98 --------------- lib/python.sh | 139 +++++++++++++++++++++ lib/python_version.sh | 59 +++++++++ spec/hatchet/pipenv_spec.rb | 19 ++- spec/hatchet/poetry_spec.rb | 19 ++- spec/hatchet/python_update_warning_spec.rb | 38 ++++-- spec/hatchet/python_version_spec.rb | 33 ++++- 10 files changed, 308 insertions(+), 133 deletions(-) delete mode 100755 bin/steps/python create mode 100644 lib/python.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 7be1eea19..2abd8137e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,13 @@ ## [Unreleased] -- Improved buildpack metrics for builds that fail. ([#1746](https://github.com/heroku/heroku-buildpack-python/pull/1746)) +- Improved the warning message shown when the requested Python version is not the latest patch version. ([#1749](https://github.com/heroku/heroku-buildpack-python/pull/1749)) +- Improved the error message shown when the requested Python patch version isn't available. ([#1749](https://github.com/heroku/heroku-buildpack-python/pull/1749)) +- Improved the error message shown if there was a networking or server related error downloading Python. ([#1749](https://github.com/heroku/heroku-buildpack-python/pull/1749)) +- Adjusted the curl options used when downloading Python to set a maximum download time of 120s to prevent hanging builds in the case of network issues. ([#1749](https://github.com/heroku/heroku-buildpack-python/pull/1749)) +- Refactored the Python download step to avoid an unnecessary version check `HEAD` request to S3 prior to downloading Python or reusing a cached install. ([#1749](https://github.com/heroku/heroku-buildpack-python/pull/1749)) +- Improved buildpack metrics for Python version selection. ([#1749](https://github.com/heroku/heroku-buildpack-python/pull/1749)) +- Improved buildpack metrics for builds that fail. ([#1746](https://github.com/heroku/heroku-buildpack-python/pull/1746) and [#1749](https://github.com/heroku/heroku-buildpack-python/pull/1749)) ## [v276] - 2025-02-05 diff --git a/bin/compile b/bin/compile index 7fbeda2c1..727966b06 100755 --- a/bin/compile +++ b/bin/compile @@ -28,8 +28,9 @@ source "${BUILDPACK_DIR}/lib/output.sh" source "${BUILDPACK_DIR}/lib/package_manager.sh" source "${BUILDPACK_DIR}/lib/pip.sh" source "${BUILDPACK_DIR}/lib/pipenv.sh" -source "${BUILDPACK_DIR}/lib/python_version.sh" source "${BUILDPACK_DIR}/lib/poetry.sh" +source "${BUILDPACK_DIR}/lib/python_version.sh" +source "${BUILDPACK_DIR}/lib/python.sh" compile_start_time=$(nowms) @@ -49,13 +50,6 @@ export PATH=:/usr/local/bin:$PATH # Exported for use in subshells, such as the steps run via sub_env. export BUILD_DIR CACHE_DIR ENV_DIR -# Set the base URL for downloading buildpack assets like Python runtimes. -# The user can provide BUILDPACK_S3_BASE_URL to specify a custom target. -# Note: this is designed for non-Heroku use, as it does not use the user-provided -# environment variable mechanism (the ENV_DIR). -S3_BASE_URL="${BUILDPACK_S3_BASE_URL:-"https://heroku-buildpack-python.s3.us-east-1.amazonaws.com"}" -# This has to be exported since it's used by the geo-libs step which is run in a subshell. - # Common Problem Warnings: # This section creates a temporary file in which to stick the output of `pip install`. # The `warnings` subscript then greps through this for common problems and guides @@ -121,6 +115,7 @@ cached_python_full_version="$(cache::cached_python_full_version "${CACHE_DIR}")" # We use the Bash 4.3+ `nameref` feature to pass back multiple values from this function # without having to hardcode globals. See: https://stackoverflow.com/a/38997681 python_version::read_requested_python_version "${BUILD_DIR}" "${package_manager}" "${cached_python_full_version}" requested_python_version python_version_origin +meta_set "python_version_requested" "${requested_python_version}" meta_set "python_version_reason" "${python_version_origin}" # TODO: More strongly recommend specifying a Python version (eg switch the messaging to @@ -144,6 +139,12 @@ python_major_version="${python_full_version%.*}" meta_set "python_version" "${python_full_version}" meta_set "python_version_major" "${python_major_version}" +if [[ "${requested_python_version}" == "${python_full_version}" ]]; then + meta_set "python_version_pinned" "true" +else + meta_set "python_version_pinned" "false" +fi + if [[ "${python_version_origin}" == "runtime.txt" ]]; then output::warning <<-EOF Warning: The runtime.txt file is deprecated. @@ -170,6 +171,9 @@ if [[ "${python_version_origin}" == "runtime.txt" ]]; then EOF fi +python_version::warn_if_deprecated_major_version "${python_major_version}" "${python_version_origin}" +python_version::warn_if_patch_update_available "${python_full_version}" "${python_major_version}" "${python_version_origin}" + cache::restore "${BUILD_DIR}" "${CACHE_DIR}" "${STACK}" "${cached_python_full_version}" "${python_full_version}" "${package_manager}" # The directory for the .profile.d scripts. @@ -190,10 +194,7 @@ if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then # Note: .heroku/src is copied in later. fi -# Download and install Python using pre-built binaries from S3. -install_python_start_time=$(nowms) -source "${BUILDPACK_DIR}/bin/steps/python" -meta_time "python_install_duration" "${install_python_start_time}" +python::install "${BUILD_DIR}" "${STACK}" "${python_full_version}" "${python_major_version}" "${python_version_origin}" # Install the package manager and related tools. package_manager_install_start_time=$(nowms) diff --git a/bin/report b/bin/report index cc0d1a116..9a3eca067 100755 --- a/bin/report +++ b/bin/report @@ -72,6 +72,7 @@ STRING_FIELDS=( poetry_version python_version_major python_version_reason + python_version_requested python_version setuptools_version wheel_version @@ -81,6 +82,7 @@ STRING_FIELDS=( ALL_OTHER_FIELDS=( cache_restore_duration cache_save_duration + custom_s3_base_url dependencies_install_duration django_collectstatic_duration duplicate_python_buildpack @@ -94,6 +96,7 @@ ALL_OTHER_FIELDS=( pre_compile_hook_duration python_install_duration python_version_outdated + python_version_pinned setup_py_only sqlite_install_duration total_duration diff --git a/bin/steps/python b/bin/steps/python deleted file mode 100755 index 6855448de..000000000 --- a/bin/steps/python +++ /dev/null @@ -1,98 +0,0 @@ -#!/usr/bin/env bash -# shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. - -set -euo pipefail - -# The Python runtime archive filename is of form: 'python-X.Y.Z-ubuntu-22.04-amd64.tar.zst' -# The Ubuntu version is calculated from `STACK` since it's faster than calling `lsb_release`. -UBUNTU_VERSION="${STACK/heroku-/}.04" -ARCH=$(dpkg --print-architecture) -PYTHON_URL="${S3_BASE_URL}/python-${python_full_version}-ubuntu-${UBUNTU_VERSION}-${ARCH}.tar.zst" - -# The Python version validation earlier will have filtered out most unsupported versions. -# However, the version might still not be found if either: -# 1. It's a Python major version we've deprecated and so is only available on older stacks (i.e: Python 3.8). -# 2. If an exact Python version was requested and the patch version doesn't exist (e.g. 3.12.999). -# 3. The user has pinned to an older buildpack version and the S3 bucket location or layout has changed since. -# TODO: Update this message to be more specific now that Python 3.8 support has been removed, and so (1) can no longer occur. -if ! curl --output /dev/null --silent --head --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}"; then - output::error <<-EOF - Error: Python ${python_full_version} isn't available for this stack (${STACK}). - - For a list of the supported Python versions, see: - https://devcenter.heroku.com/articles/python-support#supported-python-versions - EOF - meta_set "failure_reason" "python-version-not-found" - meta_set "failure_detail" "${python_full_version}" - exit 1 -fi - -if [[ -f "${BUILD_DIR}/.heroku/python/bin/python" ]]; then - output::step "Using cached install of Python ${python_full_version}" -else - output::step "Installing Python ${python_full_version}" - mkdir -p "${BUILD_DIR}/.heroku/python" - - if ! curl --silent --show-error --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}" | tar --zstd --extract --directory "${BUILD_DIR}/.heroku/python"; then - # The Python version was confirmed to exist previously, so any failure here is due to - # a networking issue or archive/buildpack bug rather than the runtime not existing. - output::error <<-EOF - Error: Failed to download/install Python ${python_full_version}. - - In some cases, this happens due to an unstable network connection. - Please try again and to see if the error resolves itself. - EOF - meta_set "failure_reason" "python-download" - # TODO: Set failure_detail here once refactored. - exit 1 - fi - - hash -r -fi - -function warn_if_patch_update_available() { - local requested_full_version="${1}" - local requested_major_version="${2}" - local latest_patch_version - latest_patch_version="$(python_version::resolve_python_version "${requested_major_version}" "${python_version_origin}")" - # Extract the patch version component of the version strings (ie: the '5' in '3.10.5'). - local requested_patch_number="${requested_full_version##*.}" - local latest_patch_number="${latest_patch_version##*.}" - # TODO: Update this message to suggest using the .python-version major version syntax to stay up to date, - # once runtime.txt is deprecated and sticky-versioning only pins to the major version. - if ((requested_patch_number < latest_patch_number)); then - output::warning <<-EOF - Warning: A Python security update is available! - - Upgrade as soon as possible to: Python ${latest_patch_version} - See: https://devcenter.heroku.com/articles/python-runtimes - EOF - meta_set "python_version_outdated" "true" - else - meta_set "python_version_outdated" "false" - fi -} - -# We wait until now to display outdated Python version warnings, since we only want to show them -# if there weren't any errors with the version to avoid adding noise to the error messages. -# TODO: Move this into lib/ as part of the warnings refactor. -if [[ "${python_major_version}" == "3.9" ]]; then - output::warning <<-EOF - Warning: Support for Python 3.9 is ending soon! - - Python 3.9 will reach its upstream end-of-life in October 2025, - at which point it will no longer receive security updates: - https://devguide.python.org/versions/#supported-versions - - As such, support for Python 3.9 will be removed from this - buildpack on 7th January 2026. - - Upgrade to a newer Python version as soon as possible, by - changing the version in your ${python_version_origin} file. - - For more information, see: - https://devcenter.heroku.com/articles/python-support#supported-python-versions - EOF -fi - -warn_if_patch_update_available "${python_full_version}" "${python_major_version}" diff --git a/lib/python.sh b/lib/python.sh new file mode 100644 index 000000000..4bbeb344a --- /dev/null +++ b/lib/python.sh @@ -0,0 +1,139 @@ +#!/usr/bin/env bash + +# This is technically redundant, since all consumers of this lib will have enabled these, +# however, it helps Shellcheck realise the options under which these functions will run. +set -euo pipefail + +DEFAULT_S3_BASE_URL="https://heroku-buildpack-python.s3.us-east-1.amazonaws.com" + +function python::install() { + local build_dir="${1}" + local stack="${2}" + local python_full_version="${3}" + local python_major_version="${4}" + local python_version_origin="${5}" + + local install_python_start_time + install_python_start_time=$(nowms) + local install_dir="${build_dir}/.heroku/python" + + if [[ -f "${install_dir}/bin/python" ]]; then + output::step "Using cached install of Python ${python_full_version}" + else + output::step "Installing Python ${python_full_version}" + + mkdir -p "${install_dir}" + + # Note: This can't be used via app config vars, since it doesn't reference the value from ENV_DIR. + # TODO: Remove this for parity with the Python CNB, if metrics show it to be unused on Heroku. + if [[ -v BUILDPACK_S3_BASE_URL ]]; then + local s3_base_url="${BUILDPACK_S3_BASE_URL}" + meta_set "custom_s3_base_url" "true" + else + local s3_base_url="${DEFAULT_S3_BASE_URL}" + fi + + # Calculating the Ubuntu version from the stack name saves having to shell out to `lsb_release`. + local ubuntu_version="${stack/heroku-/}.04" + local arch + arch=$(dpkg --print-architecture) + # e.g.: https://heroku-buildpack-python.s3.us-east-1.amazonaws.com/python-3.13.0-ubuntu-24.04-amd64.tar.zst + local python_url="${s3_base_url}/python-${python_full_version}-ubuntu-${ubuntu_version}-${arch}.tar.zst" + + local error_log + error_log=$(mktemp) + + # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. + if ! { + { + # We set max-time for improved UX/metrics for hanging downloads compared to relying + # on the build system timeout. The Python archives are only ~10 MB so take < 1s to + # download on Heroku's build system, however, we use much higher timeouts so that + # the buildpack works in non-Heroku environments that may be far from `us-east-1` + # or have a slower connection. We don't use `--speed-limit` since it gives worse + # error messages when used with retries and piping to tar. + curl \ + --connect-timeout 10 \ + --fail \ + --max-time 120 \ + --retry-max-time 120 \ + --retry 3 \ + --retry-connrefused \ + --show-error \ + --silent \ + "${python_url}" \ + | tar \ + --directory "${install_dir}" \ + --extract \ + --zstd + } \ + |& tee "${error_log}" \ + |& output::indent + }; then + local latest_known_patch_version + latest_known_patch_version="$(python_version::resolve_python_version "${python_major_version}" "${python_version_origin}")" + # Ideally we would inspect the HTTP status code directly instead of grepping, however: + # 1. We want to pipe to tar (since it's faster than performing the download and + # decompression/extraction as separate steps), so can't write to stdout. + # 2. We want to display the original stderr to the user, so can't write to stderr. + # 3. Curl's `--write-out` feature only supports outputting to a file (as opposed to + # stdout/stderr) as of curl v8.3.0, which is newer than the curl on Heroku-20/22. + # This has an integration test run against all stacks, which will mean we will know + # if future versions of curl change the error message string. + # + # We have to check for HTTP 403s too, since S3 will return a 403 instead of a 404 for + # missing files, if the S3 bucket does not have public list permissions enabled. + if [[ "${python_full_version}" != "${latest_known_patch_version}" ]] && grep --quiet "The requested URL returned error: 40[34]" "${error_log}"; then + output::error <<-EOF + Error: The requested Python version isn't available. + + Your app's ${python_version_origin} file specifies a Python version + of ${python_full_version}, however, we couldn't find that version on S3. + + Check that this Python version has been released upstream, + and that the Python buildpack has added support for it: + https://www.python.org/downloads/ + https://github.com/heroku/heroku-buildpack-python/blob/main/CHANGELOG.md + + If it has, make sure that you are using the latest version + of this buildpack, and haven't pinned to an older release: + https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks + https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references + + We also strongly recommend that you do not pin your app to an + exact Python version such as ${python_full_version}, and instead only specify + the major Python version of ${python_major_version} in your ${python_version_origin} file. + This will allow your app to receive the latest available Python + patch version automatically, and prevent this type of error. + EOF + meta_set "failure_reason" "python-version::unknown-patch" + meta_set "failure_detail" "${python_full_version}" + else + output::error <<-EOF + Error: Unable to download/install Python. + + An error occurred while downloading/installing the Python + runtime archive from: + ${python_url} + + In some cases, this happens due to a temporary issue with + the network connection or server. + + First, make sure that you are using the latest version + of this buildpack, and haven't pinned to an older release: + https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks + https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references + + Then try building again to see if the error resolves itself. + EOF + meta_set "failure_reason" "install-python" + # e.g.: 'curl: (6) Could not resolve host: heroku-buildpack-python.s3.us-east-1.amazonaws.com' + meta_set "failure_detail" "$(head --lines=1 "${error_log}" || true)" + fi + + exit 1 + fi + fi + + meta_time "python_install_duration" "${install_python_start_time}" +} diff --git a/lib/python_version.sh b/lib/python_version.sh index 652c4e478..4405b7650 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -416,3 +416,62 @@ function python_version::resolve_python_version() { *) utils::abort_internal_error "Unhandled Python major version: ${requested_python_version}" ;; esac } + +function python_version::warn_if_deprecated_major_version() { + local requested_major_version="${1}" + local version_origin="${2}" + + if [[ "${requested_major_version}" == "3.9" ]]; then + output::warning <<-EOF + Warning: Support for Python 3.9 is ending soon! + + Python 3.9 will reach its upstream end-of-life in October 2025, + at which point it will no longer receive security updates: + https://devguide.python.org/versions/#supported-versions + + As such, support for Python 3.9 will be removed from this + buildpack on 7th January 2026. + + Upgrade to a newer Python version as soon as possible, by + changing the version in your ${version_origin} file. + + For more information, see: + https://devcenter.heroku.com/articles/python-support#supported-python-versions + EOF + fi +} + +function python_version::warn_if_patch_update_available() { + local python_full_version="${1}" + local python_major_version="${2}" + local python_version_origin="${3}" + + local latest_known_patch_version + latest_known_patch_version="$(python_version::resolve_python_version "${python_major_version}" "${python_version_origin}")" + # Extract the patch version component of the version strings (ie: the '2' in '3.13.2'). + local requested_patch_number="${python_full_version##*.}" + local latest_patch_number="${latest_known_patch_version##*.}" + + if ((requested_patch_number < latest_patch_number)); then + output::warning <<-EOF + Warning: A Python patch update is available! + + Your app is using Python ${python_full_version}, however, there is a newer + patch release of Python ${python_major_version} available: ${latest_known_patch_version} + + It is important to always use the latest patch version of + Python to keep your app secure. + + Update your ${python_version_origin} file to use the new version. + + We strongly recommend that you do not pin your app to an + exact Python version such as ${python_full_version}, and instead only specify + the major Python version of ${python_major_version} in your ${python_version_origin} file. + This will allow your app to receive the latest available Python + patch version automatically and prevent this warning. + EOF + meta_set "python_version_outdated" "true" + else + meta_set "python_version_outdated" "false" + fi +} diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 6949e3934..7a657ae82 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -78,7 +78,6 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected remote: -----> Using Python 3.9.0 specified in Pipfile.lock - remote: -----> Installing Python 3.9.0 remote: remote: ! Warning: Support for Python 3.9 is ending soon! remote: ! @@ -96,11 +95,23 @@ remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: remote: - remote: ! Warning: A Python security update is available! + remote: ! Warning: A Python patch update is available! + remote: ! + remote: ! Your app is using Python 3.9.0, however, there is a newer + remote: ! patch release of Python 3.9 available: #{LATEST_PYTHON_3_9} + remote: ! + remote: ! It is important to always use the latest patch version of + remote: ! Python to keep your app secure. remote: ! - remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} - remote: ! See: https://devcenter.heroku.com/articles/python-runtimes + remote: ! Update your Pipfile.lock file to use the new version. + remote: ! + remote: ! We strongly recommend that you do not pin your app to an + remote: ! exact Python version such as 3.9.0, and instead only specify + remote: ! the major Python version of 3.9 in your Pipfile.lock file. + remote: ! This will allow your app to receive the latest available Python + remote: ! patch version automatically and prevent this warning. remote: + remote: -----> Installing Python 3.9.0 remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index e7e85d446..bad760d30 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -168,7 +168,6 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python 3.9.0 specified in .python-version - remote: -----> Installing Python 3.9.0 remote: remote: ! Warning: Support for Python 3.9 is ending soon! remote: ! @@ -186,11 +185,23 @@ remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: remote: - remote: ! Warning: A Python security update is available! + remote: ! Warning: A Python patch update is available! + remote: ! + remote: ! Your app is using Python 3.9.0, however, there is a newer + remote: ! patch release of Python 3.9 available: #{LATEST_PYTHON_3_9} + remote: ! + remote: ! It is important to always use the latest patch version of + remote: ! Python to keep your app secure. remote: ! - remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} - remote: ! See: https://devcenter.heroku.com/articles/python-runtimes + remote: ! Update your .python-version file to use the new version. + remote: ! + remote: ! We strongly recommend that you do not pin your app to an + remote: ! exact Python version such as 3.9.0, and instead only specify + remote: ! the major Python version of 3.9 in your .python-version file. + remote: ! This will allow your app to receive the latest available Python + remote: ! patch version automatically and prevent this warning. remote: + remote: -----> Installing Python 3.9.0 remote: -----> Installing Poetry #{POETRY_VERSION} remote: -----> Installing dependencies using 'poetry sync --only main' remote: Installing dependencies from lock file diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index 4f7991dac..c1025f415 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -37,7 +37,6 @@ remote: ! In the future support for runtime.txt will be removed and remote: ! this warning will be made an error. remote: - remote: -----> Installing Python 3.9.0 remote: remote: ! Warning: Support for Python 3.9 is ending soon! remote: ! @@ -55,11 +54,23 @@ remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: remote: - remote: ! Warning: A Python security update is available! + remote: ! Warning: A Python patch update is available! + remote: ! + remote: ! Your app is using Python 3.9.0, however, there is a newer + remote: ! patch release of Python 3.9 available: #{LATEST_PYTHON_3_9} + remote: ! + remote: ! It is important to always use the latest patch version of + remote: ! Python to keep your app secure. remote: ! - remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} - remote: ! See: https://devcenter.heroku.com/articles/python-runtimes + remote: ! Update your runtime.txt file to use the new version. + remote: ! + remote: ! We strongly recommend that you do not pin your app to an + remote: ! exact Python version such as 3.9.0, and instead only specify + remote: ! the major Python version of 3.9 in your runtime.txt file. + remote: ! This will allow your app to receive the latest available Python + remote: ! patch version automatically and prevent this warning. remote: + remote: -----> Installing Python 3.9.0 remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} OUTPUT end @@ -74,13 +85,24 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python 3.10.0 specified in .python-version - remote: -----> Installing Python 3.10.0 remote: - remote: ! Warning: A Python security update is available! + remote: ! Warning: A Python patch update is available! + remote: ! + remote: ! Your app is using Python 3.10.0, however, there is a newer + remote: ! patch release of Python 3.10 available: #{LATEST_PYTHON_3_10} + remote: ! + remote: ! It is important to always use the latest patch version of + remote: ! Python to keep your app secure. remote: ! - remote: ! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_10} - remote: ! See: https://devcenter.heroku.com/articles/python-runtimes + remote: ! Update your .python-version file to use the new version. + remote: ! + remote: ! We strongly recommend that you do not pin your app to an + remote: ! exact Python version such as 3.10.0, and instead only specify + remote: ! the major Python version of 3.10 in your .python-version file. + remote: ! This will allow your app to receive the latest available Python + remote: ! patch version automatically and prevent this warning. remote: + remote: -----> Installing Python 3.10.0 remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} OUTPUT end diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 4f8742fae..639b67d1c 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -87,7 +87,6 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python 3.9 specified in .python-version - remote: -----> Installing Python #{LATEST_PYTHON_3_9} remote: remote: ! Warning: Support for Python 3.9 is ending soon! remote: ! @@ -104,6 +103,7 @@ remote: ! For more information, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: + remote: -----> Installing Python #{LATEST_PYTHON_3_9} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pip install -r requirements.txt' @@ -293,17 +293,38 @@ it 'aborts the build with a version not available message' do app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected remote: -----> Using Python 3.12.999 specified in .python-version + remote: -----> Installing Python 3.12.999 + remote: curl: \\(22\\) The requested URL returned error: 404.* + remote: zstd: /\\*stdin\\*\\\\: unexpected end of file + remote: tar: Child returned status 1 + remote: tar: Error is not recoverable: exiting now remote: - remote: ! Error: Python 3.12.999 isn't available for this stack (#{app.stack}). + remote: ! Error: The requested Python version isn't available. remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions + remote: ! Your app's .python-version file specifies a Python version + remote: ! of 3.12.999, however, we couldn't find that version on S3. + remote: ! + remote: ! Check that this Python version has been released upstream, + remote: ! and that the Python buildpack has added support for it: + remote: ! https://www.python.org/downloads/ + remote: ! https://github.com/heroku/heroku-buildpack-python/blob/main/CHANGELOG.md + remote: ! + remote: ! If it has, make sure that you are using the latest version + remote: ! of this buildpack, and haven't pinned to an older release: + remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks + remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references + remote: ! + remote: ! We also strongly recommend that you do not pin your app to an + remote: ! exact Python version such as 3.12.999, and instead only specify + remote: ! the major Python version of 3.12 in your .python-version file. + remote: ! This will allow your app to receive the latest available Python + remote: ! patch version automatically, and prevent this type of error. remote: remote: ! Push rejected, failed to compile Python app. - OUTPUT + REGEX end end end From fc441449edad0a631352ad51cd10f54677a1c01b Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Mon, 17 Feb 2025 18:16:24 +0000 Subject: [PATCH 099/406] Prepare release v277 (#1750) * Prepare release v277 * Update changelog --------- Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2abd8137e..0820beac7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,11 +2,15 @@ ## [Unreleased] + +## [v277] - 2025-02-17 + - Improved the warning message shown when the requested Python version is not the latest patch version. ([#1749](https://github.com/heroku/heroku-buildpack-python/pull/1749)) - Improved the error message shown when the requested Python patch version isn't available. ([#1749](https://github.com/heroku/heroku-buildpack-python/pull/1749)) - Improved the error message shown if there was a networking or server related error downloading Python. ([#1749](https://github.com/heroku/heroku-buildpack-python/pull/1749)) - Adjusted the curl options used when downloading Python to set a maximum download time of 120s to prevent hanging builds in the case of network issues. ([#1749](https://github.com/heroku/heroku-buildpack-python/pull/1749)) - Refactored the Python download step to avoid an unnecessary version check `HEAD` request to S3 prior to downloading Python or reusing a cached install. ([#1749](https://github.com/heroku/heroku-buildpack-python/pull/1749)) +- Updated the `runtime.txt` deprecation warning to include a link to the deprecation changelog post. ([#1747](https://github.com/heroku/heroku-buildpack-python/pull/1747)) - Improved buildpack metrics for Python version selection. ([#1749](https://github.com/heroku/heroku-buildpack-python/pull/1749)) - Improved buildpack metrics for builds that fail. ([#1746](https://github.com/heroku/heroku-buildpack-python/pull/1746) and [#1749](https://github.com/heroku/heroku-buildpack-python/pull/1749)) @@ -1164,7 +1168,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v276...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v277...main +[v277]: https://github.com/heroku/heroku-buildpack-python/compare/v276...v277 [v276]: https://github.com/heroku/heroku-buildpack-python/compare/v275...v276 [v275]: https://github.com/heroku/heroku-buildpack-python/compare/v274...v275 [v274]: https://github.com/heroku/heroku-buildpack-python/compare/v273...v274 From 51193fa1c429a3b0899d1e7c7e08d22382083a31 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 19 Feb 2025 12:29:26 +0000 Subject: [PATCH 100/406] Update changelog links and tests after Git tag archival (#1752) The Git tags for buildpack releases v208 to v265 have been archived (renamed from `vNNN` to `archive/vNNN`) for the reasons in #1699 (similar to the process performed in the past for release v207 and older). As such, the changelog compare URLs need updating, as do the test fixtures that test building an app whose last (cached) build was performed using an older version of the buildpack. Closes #1699. GUS-W-17308840. --- CHANGELOG.md | 118 ++++++++++++++-------------- spec/hatchet/pipenv_spec.rb | 3 +- spec/hatchet/poetry_spec.rb | 6 +- spec/hatchet/python_version_spec.rb | 4 +- spec/hatchet/stack_spec.rb | 12 +-- 5 files changed, 72 insertions(+), 71 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0820beac7..e51af3572 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1180,65 +1180,65 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. [v269]: https://github.com/heroku/heroku-buildpack-python/compare/v268...v269 [v268]: https://github.com/heroku/heroku-buildpack-python/compare/v267...v268 [v267]: https://github.com/heroku/heroku-buildpack-python/compare/v266...v267 -[v266]: https://github.com/heroku/heroku-buildpack-python/compare/v265...v266 -[v265]: https://github.com/heroku/heroku-buildpack-python/compare/v264...v265 -[v264]: https://github.com/heroku/heroku-buildpack-python/compare/v263...v264 -[v263]: https://github.com/heroku/heroku-buildpack-python/compare/v262...v263 -[v262]: https://github.com/heroku/heroku-buildpack-python/compare/v261...v262 -[v261]: https://github.com/heroku/heroku-buildpack-python/compare/v260...v261 -[v260]: https://github.com/heroku/heroku-buildpack-python/compare/v259...v260 -[v259]: https://github.com/heroku/heroku-buildpack-python/compare/v258...v259 -[v258]: https://github.com/heroku/heroku-buildpack-python/compare/v257...v258 -[v257]: https://github.com/heroku/heroku-buildpack-python/compare/v256...v257 -[v256]: https://github.com/heroku/heroku-buildpack-python/compare/v255...v256 -[v255]: https://github.com/heroku/heroku-buildpack-python/compare/v254...v255 -[v254]: https://github.com/heroku/heroku-buildpack-python/compare/v253...v254 -[v253]: https://github.com/heroku/heroku-buildpack-python/compare/v252...v253 -[v252]: https://github.com/heroku/heroku-buildpack-python/compare/v251...v252 -[v251]: https://github.com/heroku/heroku-buildpack-python/compare/v250...v251 -[v250]: https://github.com/heroku/heroku-buildpack-python/compare/v249...v250 -[v249]: https://github.com/heroku/heroku-buildpack-python/compare/v248...v249 -[v248]: https://github.com/heroku/heroku-buildpack-python/compare/v247...v248 -[v247]: https://github.com/heroku/heroku-buildpack-python/compare/v246...v247 -[v246]: https://github.com/heroku/heroku-buildpack-python/compare/v245...v246 -[v245]: https://github.com/heroku/heroku-buildpack-python/compare/v244...v245 -[v244]: https://github.com/heroku/heroku-buildpack-python/compare/v243...v244 -[v243]: https://github.com/heroku/heroku-buildpack-python/compare/v242...v243 -[v242]: https://github.com/heroku/heroku-buildpack-python/compare/v241...v242 -[v241]: https://github.com/heroku/heroku-buildpack-python/compare/v240...v241 -[v240]: https://github.com/heroku/heroku-buildpack-python/compare/v239...v240 -[v239]: https://github.com/heroku/heroku-buildpack-python/compare/v238...v239 -[v238]: https://github.com/heroku/heroku-buildpack-python/compare/v237...v238 -[v237]: https://github.com/heroku/heroku-buildpack-python/compare/v236...v237 -[v236]: https://github.com/heroku/heroku-buildpack-python/compare/v235...v236 -[v235]: https://github.com/heroku/heroku-buildpack-python/compare/v234...v235 -[v234]: https://github.com/heroku/heroku-buildpack-python/compare/v233...v234 -[v233]: https://github.com/heroku/heroku-buildpack-python/compare/v232...v233 -[v232]: https://github.com/heroku/heroku-buildpack-python/compare/v231...v232 -[v231]: https://github.com/heroku/heroku-buildpack-python/compare/v230...v231 -[v230]: https://github.com/heroku/heroku-buildpack-python/compare/v229...v230 -[v229]: https://github.com/heroku/heroku-buildpack-python/compare/v228...v229 -[v228]: https://github.com/heroku/heroku-buildpack-python/compare/v227...v228 -[v227]: https://github.com/heroku/heroku-buildpack-python/compare/v226...v227 -[v226]: https://github.com/heroku/heroku-buildpack-python/compare/v225...v226 -[v225]: https://github.com/heroku/heroku-buildpack-python/compare/v224...v225 -[v224]: https://github.com/heroku/heroku-buildpack-python/compare/v223...v224 -[v223]: https://github.com/heroku/heroku-buildpack-python/compare/v222...v223 -[v222]: https://github.com/heroku/heroku-buildpack-python/compare/v221...v222 -[v221]: https://github.com/heroku/heroku-buildpack-python/compare/v220...v221 -[v220]: https://github.com/heroku/heroku-buildpack-python/compare/v219...v220 -[v219]: https://github.com/heroku/heroku-buildpack-python/compare/v218...v219 -[v218]: https://github.com/heroku/heroku-buildpack-python/compare/v217...v218 -[v217]: https://github.com/heroku/heroku-buildpack-python/compare/v216...v217 -[v216]: https://github.com/heroku/heroku-buildpack-python/compare/v215...v216 -[v215]: https://github.com/heroku/heroku-buildpack-python/compare/v214...v215 -[v214]: https://github.com/heroku/heroku-buildpack-python/compare/v213...v214 -[v213]: https://github.com/heroku/heroku-buildpack-python/compare/v212...v213 -[v212]: https://github.com/heroku/heroku-buildpack-python/compare/v211...v212 -[v211]: https://github.com/heroku/heroku-buildpack-python/compare/v210...v211 -[v210]: https://github.com/heroku/heroku-buildpack-python/compare/v209...v210 -[v209]: https://github.com/heroku/heroku-buildpack-python/compare/v208...v209 -[v208]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v207...v208 +[v266]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v265...v266 +[v265]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v264...archive/v265 +[v264]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v263...archive/v264 +[v263]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v262...archive/v263 +[v262]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v261...archive/v262 +[v261]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v260...archive/v261 +[v260]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v259...archive/v260 +[v259]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v258...archive/v259 +[v258]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v257...archive/v258 +[v257]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v256...archive/v257 +[v256]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v255...archive/v256 +[v255]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v254...archive/v255 +[v254]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v253...archive/v254 +[v253]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v252...archive/v253 +[v252]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v251...archive/v252 +[v251]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v250...archive/v251 +[v250]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v249...archive/v250 +[v249]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v248...archive/v249 +[v248]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v247...archive/v248 +[v247]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v246...archive/v247 +[v246]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v245...archive/v246 +[v245]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v244...archive/v245 +[v244]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v243...archive/v244 +[v243]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v242...archive/v243 +[v242]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v241...archive/v242 +[v241]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v240...archive/v241 +[v240]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v239...archive/v240 +[v239]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v238...archive/v239 +[v238]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v237...archive/v238 +[v237]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v236...archive/v237 +[v236]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v235...archive/v236 +[v235]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v234...archive/v235 +[v234]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v233...archive/v234 +[v233]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v232...archive/v233 +[v232]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v231...archive/v232 +[v231]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v230...archive/v231 +[v230]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v229...archive/v230 +[v229]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v228...archive/v229 +[v228]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v227...archive/v228 +[v227]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v226...archive/v227 +[v226]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v225...archive/v226 +[v225]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v224...archive/v225 +[v224]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v223...archive/v224 +[v223]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v222...archive/v223 +[v222]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v221...archive/v222 +[v221]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v220...archive/v221 +[v220]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v219...archive/v220 +[v219]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v218...archive/v219 +[v218]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v217...archive/v218 +[v217]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v216...archive/v217 +[v216]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v215...archive/v216 +[v215]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v214...archive/v215 +[v214]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v213...archive/v214 +[v213]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v212...archive/v213 +[v212]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v211...archive/v212 +[v211]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v210...archive/v211 +[v210]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v209...archive/v210 +[v209]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v208...archive/v209 +[v208]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v207...archive/v208 [v207]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v206...archive/v207 [v206]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v205...archive/v206 [v205]: https://github.com/heroku/heroku-buildpack-python/compare/archive/v204...archive/v205 diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 7a657ae82..8d154cfec 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -322,7 +322,8 @@ end context 'when the Pipenv and Python versions have changed since the last build' do - let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v253'] } + # TODO: Bump this buildpack version the next time we update the Pipenv version. + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#archive/v253'] } let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_basic', buildpacks:) } it 'clears the cache before installing' do diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index bad760d30..1fa8ae836 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -62,7 +62,7 @@ end context 'when the Poetry and Python versions have changed since the last build' do - let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v268'] } + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v274'] } let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_basic', buildpacks:) } it 'clears the cache before installing' do @@ -74,8 +74,8 @@ remote: -----> Python app detected remote: -----> Using Python 3.13 specified in .python-version remote: -----> Discarding cache since: - remote: - The Python version has changed from 3.13.0 to #{LATEST_PYTHON_3_13} - remote: - The Poetry version has changed from 1.8.4 to #{POETRY_VERSION} + remote: - The Python version has changed from 3.13.1 to #{LATEST_PYTHON_3_13} + remote: - The Poetry version has changed from 1.8.5 to #{POETRY_VERSION} remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing Poetry #{POETRY_VERSION} remote: -----> Installing dependencies using 'poetry sync --only main' diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 639b67d1c..9e2452e59 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -56,7 +56,7 @@ # - If no Python version is specified, the same major version as the # last build is used (sticky versioning). # - Changes in the pip version are handled correctly. - let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v257'] } + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v267'] } it 'builds with the same Python version as the last build' do app.deploy do |app| @@ -68,7 +68,7 @@ remote: -----> No Python version was specified. Using the same major version as the last build: Python 3.12 remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Discarding cache since: - remote: - The Python version has changed from 3.12.6 to #{LATEST_PYTHON_3_12} + remote: - The Python version has changed from 3.12.7 to #{LATEST_PYTHON_3_12} remote: - The pip version has changed from 24.0 to #{PIP_VERSION} remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index 46817cad6..974e6d607 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -4,12 +4,12 @@ RSpec.describe 'Stack changes' do context 'when the stack is upgraded from Heroku-22 to Heroku-24', stacks: %w[heroku-22] do - # This test performs an initial build using an older buildpack version, followed - # by a build using the current version. This ensures that the current buildpack - # can successfully read the stack metadata written to the build cache in the past. - # The buildpack version chosen is one which had an older default Python version, so - # we can also prove that clearing the cache didn't lose the Python version metadata. - let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v250'] } + # This test performs an initial build using an older buildpack version, followed by a build + # using the current version. This ensures that the current buildpack can successfully read + # the stack metadata written to the build cache in the past. The buildpack version chosen is + # the oldest to support Heroku-24, and which had an older default Python version so we can + # also prove that clearing the cache didn't lose the sticky Python version metadata. + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#archive/v250'] } let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_unspecified', buildpacks:) } it 'clears the cache before installing again whilst preserving the sticky Python version' do From ee0a9eb4d21f3d14c7328dda7718e62c0a15c473 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 19 Feb 2025 16:30:16 +0000 Subject: [PATCH 101/406] Change the editable VCS directory location for pip and Pipenv (#1753) When a dependency from a version control system (eg Git) is installed in editable mode, the package manager has to clone the repository somewhere long-lived, that is then referenced by the `.pth` file added to `site-packages`. (When installed in normal non-editable mode, the repo checkouts are instead saved to a temporary directory and deleted after the package is installed.) Until now, the buildpack configured pip and Pipenv to store these repos at `/app/.heroku/src/`, then later copied those files into the build directory and build cache. However, this approach isn't needed with the `.pth` rewriting we have now. In addition, the existing implementation didn't actually restore the cached `src/` directory, so the repos stored in the cache were never re-used on subsequent builds anyway. Now, pip and pipenv are configured to store the repositories at `/.heroku/python/src/`, which means: - The behaviour now matches that when using Poetry. - The repos get cached/restored/invalidated for free, as part of the existing handling of the `.heroku/python/` directory, and we avoid the additional directory copy from `/app` to `/tmp`, both of which help reduce build times. GUS-W-17863838. --- CHANGELOG.md | 1 + bin/compile | 12 ---------- bin/utils | 16 -------------- lib/cache.sh | 14 +----------- lib/pip.sh | 2 +- lib/pipenv.sh | 2 +- spec/hatchet/pip_spec.rb | 44 ++++++++++++++++++------------------- spec/hatchet/pipenv_spec.rb | 44 ++++++++++++++++++------------------- 8 files changed, 48 insertions(+), 87 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e51af3572..bf4cba079 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Changed the location of repositories for editable VCS dependencies when using pip and Pipenv, to improve build performance and match the behaviour when using Poetry. ([#1753](https://github.com/heroku/heroku-buildpack-python/pull/1753)) ## [v277] - 2025-02-17 diff --git a/bin/compile b/bin/compile index 727966b06..9f2869fc6 100755 --- a/bin/compile +++ b/bin/compile @@ -178,8 +178,6 @@ cache::restore "${BUILD_DIR}" "${CACHE_DIR}" "${STACK}" "${cached_python_full_ve # The directory for the .profile.d scripts. mkdir -p "$(dirname "$PROFILE_PATH")" -# The directory for editable VCS dependencies. -mkdir -p /app/.heroku/src # On Heroku CI, builds happen in `/app`. Otherwise, on the Heroku platform, # they occur in a temp directory. Because Python is not portable, we must create @@ -191,7 +189,6 @@ if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then # python expects to reside in /app, so set up symlinks # we will not remove these later so subsequent buildpacks can still invoke it ln -nsf "$BUILD_DIR/.heroku/python" /app/.heroku/python - # Note: .heroku/src is copied in later. fi python::install "${BUILD_DIR}" "${STACK}" "${python_full_version}" "${python_major_version}" "${python_version_origin}" @@ -250,15 +247,6 @@ nltk_downloader_start_time=$(nowms) sub_env "${BUILDPACK_DIR}/bin/steps/nltk" meta_time "nltk_downloader_duration" "${nltk_downloader_start_time}" -# Support for editable installations. -# In CI, $BUILD_DIR is /app. -# Realpath is used to support use-cases where one of the locations is a symlink to the other. -# shellcheck disable=SC2312 # TODO: Invoke this command separately to avoid masking its return value. -if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then - rm -rf "$BUILD_DIR/.heroku/src" - deep-cp /app/.heroku/src "$BUILD_DIR/.heroku/src" -fi - # Django collectstatic support. # The buildpack automatically runs collectstatic for Django applications. collectstatic_start_time=$(nowms) diff --git a/bin/utils b/bin/utils index 9e662191a..195172a7a 100755 --- a/bin/utils +++ b/bin/utils @@ -8,22 +8,6 @@ shopt -s nullglob source "${BUILDPACK_DIR:?}/vendor/buildpack-stdlib_v8.sh" -# Does some serious copying. -deep-cp() { - declare source="$1" target="$2" - - mkdir -p "$target" - - # cp doesn't like being called without source params, - # so make sure they expand to something first. - # subshell to avoid surprising caller with shopts. - ( - shopt -s nullglob dotglob - set -- "$source"/!(tmp|.|..) - [[ $# == 0 ]] || cp -a "$@" "$target" - ) -} - # Measure the size of the Python installation. measure-size() { { du -s .heroku/python 2>/dev/null || echo 0; } | awk '{print $1}' diff --git a/lib/cache.sh b/lib/cache.sh index ef62d582f..8353a9093 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -129,7 +129,6 @@ function cache::restore() { "${cache_dir}/.heroku/python-poetry" \ "${cache_dir}/.heroku/python-stack" \ "${cache_dir}/.heroku/python-version" \ - "${cache_dir}/.heroku/src" \ "${cache_dir}/.heroku/requirements.txt" meta_set "cache_status" "discarded" @@ -143,17 +142,13 @@ function cache::restore() { # TODO: Compare the performance of moving the directory vs copying files. cp -R "${cache_dir}/.heroku/python" "${build_dir}/.heroku/" &>/dev/null || true - # Editable VCS code repositories, used by pip/pipenv. - if [[ -d "${cache_dir}/.heroku/src" ]]; then - cp -R "${cache_dir}/.heroku/src" "${build_dir}/.heroku/" &>/dev/null || true - fi - meta_set "cache_status" "reused" fi # Remove any legacy cache contents written by older buildpack versions. rm -rf \ "${cache_dir}/.heroku/python-sqlite3-version" \ + "${cache_dir}/.heroku/src" \ "${cache_dir}/.heroku/vendor" meta_time "cache_restore_duration" "${cache_restore_start_time}" @@ -175,13 +170,6 @@ function cache::save() { rm -rf "${cache_dir}/.heroku/python" cp -R "${build_dir}/.heroku/python" "${cache_dir}/.heroku/" - # Editable VCS code repositories, used by pip/pipenv. - rm -rf "${cache_dir}/.heroku/src" - if [[ -d "${build_dir}/.heroku/src" ]]; then - # TODO: Investigate why errors are ignored and ideally stop doing so. - cp -R "${build_dir}/.heroku/src" "${cache_dir}/.heroku/" &>/dev/null || true - fi - # Metadata used by subsequent builds to determine whether the cache can be reused. # These are written/consumed via separate files and not the metadata store for compatibility # with buildpack versions prior to the metadata store existing (which was only added in v252). diff --git a/lib/pip.sh b/lib/pip.sh index 929063fb4..62ebc43ee 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -110,7 +110,7 @@ function pip::install_dependencies() { --no-cache-dir \ --no-input \ --progress-bar off \ - --src='/app/.heroku/src' \ + --src='/app/.heroku/python/src' \ |& tee "${WARNINGS_LOG:?}" \ |& sed --unbuffered --expression '/Requirement already satisfied/d' \ |& output::indent diff --git a/lib/pipenv.sh b/lib/pipenv.sh index 34519eb1b..9e8eb4594 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -81,7 +81,7 @@ function pipenv::install_dependencies() { # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { "${pipenv_install_command[@]}" \ - --extra-pip-args='--src=/app/.heroku/src' \ + --extra-pip-args='--src=/app/.heroku/python/src' \ --system \ |& tee "${WARNINGS_LOG:?}" \ |& output::indent diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index d4990ae33..64ddb274f 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -100,21 +100,21 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Running bin/post_compile hook - remote: easy-install.pth:/app/.heroku/src/gunicorn - remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py - remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py + remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn + remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn + remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) remote: -----> Inline app detected - remote: easy-install.pth:/app/.heroku/src/gunicorn - remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py - remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py + remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn + remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn + remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! @@ -123,10 +123,10 @@ # Test rewritten paths work at runtime. expect(app.run('bin/test-entrypoints.sh')).to include(<<~OUTPUT) - easy-install.pth:/app/.heroku/src/gunicorn + easy-install.pth:/app/.heroku/python/src/gunicorn easy-install.pth:/app/packages/local_package_setup_py __editable___local_package_pyproject_toml_0_0_1_finder.py:/app/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - gunicorn.egg-link:/app/.heroku/src/gunicorn + gunicorn.egg-link:/app/.heroku/python/src/gunicorn local-package-setup-py.egg-link:/app/packages/local_package_setup_py Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! @@ -139,21 +139,21 @@ app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Running bin/post_compile hook - remote: easy-install.pth:/app/.heroku/src/gunicorn - remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py - remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py + remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn + remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn + remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) remote: -----> Inline app detected - remote: easy-install.pth:/app/.heroku/src/gunicorn - remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py - remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py + remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn + remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn + remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 8d154cfec..18addeaec 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -356,21 +356,21 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Running bin/post_compile hook - remote: easy-install.pth:/app/.heroku/src/gunicorn - remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py - remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py + remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn + remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn + remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) remote: -----> Inline app detected - remote: easy-install.pth:/app/.heroku/src/gunicorn - remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py - remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py + remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn + remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn + remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! @@ -379,10 +379,10 @@ # Test rewritten paths work at runtime. expect(app.run('bin/test-entrypoints.sh')).to include(<<~OUTPUT) - easy-install.pth:/app/.heroku/src/gunicorn + easy-install.pth:/app/.heroku/python/src/gunicorn easy-install.pth:/app/packages/local_package_setup_py __editable___local_package_pyproject_toml_0_0_1_finder.py:/app/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - gunicorn.egg-link:/app/.heroku/src/gunicorn + gunicorn.egg-link:/app/.heroku/python/src/gunicorn local-package-setup-py.egg-link:/app/packages/local_package_setup_py Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! @@ -395,21 +395,21 @@ app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Running bin/post_compile hook - remote: easy-install.pth:/app/.heroku/src/gunicorn - remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py - remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py + remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn + remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn + remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) remote: -----> Inline app detected - remote: easy-install.pth:/app/.heroku/src/gunicorn - remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py - remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py + remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn + remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn + remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! From f2d2bc7c4ad4b9745f441f7bd7a01b002270a73a Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 20 Feb 2025 12:13:35 +0000 Subject: [PATCH 102/406] Improve package manager tests (#1754) Backports a number of updates/improvements to the pip/Poetry/Pipenv package manager integration tests found while working on the tests for uv. --- spec/fixtures/pipenv_editable/.python-version | 2 + spec/fixtures/pipenv_editable/Pipfile | 3 +- spec/fixtures/pipenv_editable/Pipfile.lock | 12 +++-- .../pipenv_editable/__init__.py | 0 spec/fixtures/pipenv_editable/pyproject.toml | 8 ++++ .../poetry_editable/bin/test-entrypoints.sh | 2 +- spec/fixtures/poetry_editable/poetry.lock | 46 +++++++++---------- spec/fixtures/poetry_editable/pyproject.toml | 2 +- spec/hatchet/ci_spec.rb | 12 ++--- spec/hatchet/pip_spec.rb | 5 +- spec/hatchet/pipenv_spec.rb | 13 +++++- spec/hatchet/poetry_spec.rb | 45 +++++++++++++----- 12 files changed, 98 insertions(+), 52 deletions(-) create mode 100644 spec/fixtures/pipenv_editable/pipenv_editable/__init__.py create mode 100644 spec/fixtures/pipenv_editable/pyproject.toml diff --git a/spec/fixtures/pipenv_editable/.python-version b/spec/fixtures/pipenv_editable/.python-version index e4fba2183..b81161743 100644 --- a/spec/fixtures/pipenv_editable/.python-version +++ b/spec/fixtures/pipenv_editable/.python-version @@ -1 +1,3 @@ +# Note: This test has to use Python 3.12 until we work around the +# Pipenv editable VCS dependency cache invalidation bug. 3.12 diff --git a/spec/fixtures/pipenv_editable/Pipfile b/spec/fixtures/pipenv_editable/Pipfile index fcbd9c1a0..910d90713 100644 --- a/spec/fixtures/pipenv_editable/Pipfile +++ b/spec/fixtures/pipenv_editable/Pipfile @@ -4,6 +4,7 @@ verify_ssl = true name = "pypi" [packages] +gunicorn = {git = "git+https://github.com/benoitc/gunicorn", ref = "20.1.0", editable = true} local-package-pyproject-toml = {file = "packages/local_package_pyproject_toml", editable = true} local-package-setup-py = {file = "packages/local_package_setup_py", editable = true} -gunicorn = {git = "git+https://github.com/benoitc/gunicorn", ref = "20.1.0", editable = true} +pipenv-editable = {file = ".", editable = true} diff --git a/spec/fixtures/pipenv_editable/Pipfile.lock b/spec/fixtures/pipenv_editable/Pipfile.lock index ae096d558..f6f2acc1b 100644 --- a/spec/fixtures/pipenv_editable/Pipfile.lock +++ b/spec/fixtures/pipenv_editable/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "5de61064f835cc6b28000b3ac1b7b5c38a40829ccc38ff46088e36bb3bd628fa" + "sha256": "ed18332549853701536926a2bae44d9fc142a08c105d4fbf108dbde04b3fedf2" }, "pipfile-spec": 6, "requires": {}, @@ -28,13 +28,17 @@ "editable": true, "file": "packages/local_package_setup_py" }, + "pipenv-editable": { + "editable": true, + "file": "." + }, "setuptools": { "hashes": [ - "sha256:8199222558df7c86216af4f84c30e9b34a61d8ba19366cc914424cdbd28252f6", - "sha256:ce74b49e8f7110f9bf04883b730f4765b774ef3ef28f722cce7c273d253aaf7d" + "sha256:c5afc8f407c626b8313a86e10311dd3f661c6cd9c09d4bf8c15c0e11f9f2b0e6", + "sha256:e3982f444617239225d675215d51f6ba05f845d4eec313da4418fdbb56fb27e3" ], "markers": "python_version >= '3.9'", - "version": "==75.6.0" + "version": "==75.8.0" } }, "develop": {} diff --git a/spec/fixtures/pipenv_editable/pipenv_editable/__init__.py b/spec/fixtures/pipenv_editable/pipenv_editable/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/pipenv_editable/pyproject.toml b/spec/fixtures/pipenv_editable/pyproject.toml new file mode 100644 index 000000000..cf7b46f69 --- /dev/null +++ b/spec/fixtures/pipenv_editable/pyproject.toml @@ -0,0 +1,8 @@ +[project] +name = "pipenv-editable" +version = "0.0.0" +requires-python = ">=3.12" + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" diff --git a/spec/fixtures/poetry_editable/bin/test-entrypoints.sh b/spec/fixtures/poetry_editable/bin/test-entrypoints.sh index fc941ed3f..0e66f44bb 100755 --- a/spec/fixtures/poetry_editable/bin/test-entrypoints.sh +++ b/spec/fixtures/poetry_editable/bin/test-entrypoints.sh @@ -4,7 +4,7 @@ set -euo pipefail cd .heroku/python/lib/python*/site-packages/ -# List any path like strings in .pth, and finder files in site-packages. +# List any path like strings in the .pth and finder files in site-packages. grep --extended-regexp --only-matching -- '/\S+' *.pth __editable___*_finder.py | sort echo diff --git a/spec/fixtures/poetry_editable/poetry.lock b/spec/fixtures/poetry_editable/poetry.lock index a26fb9272..30df026b9 100644 --- a/spec/fixtures/poetry_editable/poetry.lock +++ b/spec/fixtures/poetry_editable/poetry.lock @@ -1,35 +1,38 @@ -# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand. +# This file is automatically @generated by Poetry 2.1.1 and should not be changed by hand. [[package]] name = "gunicorn" -version = "20.1.0" +version = "23.0.0" description = "WSGI HTTP Server for UNIX" optional = false -python-versions = ">=3.5" +python-versions = ">=3.7" +groups = ["main"] files = [] develop = true [package.dependencies] -setuptools = ">=3.0" +packaging = "*" [package.extras] -eventlet = ["eventlet (>=0.24.1)"] +eventlet = ["eventlet (>=0.24.1,!=0.36.0)"] gevent = ["gevent (>=1.4.0)"] setproctitle = ["setproctitle"] +testing = ["coverage", "eventlet", "gevent", "pytest", "pytest-cov"] tornado = ["tornado (>=0.2)"] [package.source] type = "git" url = "https://github.com/benoitc/gunicorn.git" -reference = "20.1.0" -resolved_reference = "61ccfd6c38d477a908e0f376757bbb884438053a" +reference = "HEAD" +resolved_reference = "bacbf8aa5152b94e44aa5d2a94aeaf0318a85248" [[package]] -name = "local_package_pyproject_toml" +name = "local-package-pyproject-toml" version = "0.0.1" description = "" optional = false python-versions = "*" +groups = ["main"] files = [] develop = true @@ -43,6 +46,7 @@ version = "0.0.1" description = "" optional = false python-versions = "*" +groups = ["main"] files = [] develop = true @@ -51,26 +55,18 @@ type = "directory" url = "packages/local_package_setup_py" [[package]] -name = "setuptools" -version = "75.6.0" -description = "Easily download, build, install, upgrade, and uninstall Python packages" +name = "packaging" +version = "24.2" +description = "Core utilities for Python packages" optional = false -python-versions = ">=3.9" +python-versions = ">=3.8" +groups = ["main"] files = [ - {file = "setuptools-75.6.0-py3-none-any.whl", hash = "sha256:ce74b49e8f7110f9bf04883b730f4765b774ef3ef28f722cce7c273d253aaf7d"}, - {file = "setuptools-75.6.0.tar.gz", hash = "sha256:8199222558df7c86216af4f84c30e9b34a61d8ba19366cc914424cdbd28252f6"}, + {file = "packaging-24.2-py3-none-any.whl", hash = "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759"}, + {file = "packaging-24.2.tar.gz", hash = "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f"}, ] -[package.extras] -check = ["pytest-checkdocs (>=2.4)", "pytest-ruff (>=0.2.1)", "ruff (>=0.7.0)"] -core = ["importlib_metadata (>=6)", "jaraco.collections", "jaraco.functools (>=4)", "jaraco.text (>=3.7)", "more_itertools", "more_itertools (>=8.8)", "packaging", "packaging (>=24.2)", "platformdirs (>=4.2.2)", "tomli (>=2.0.1)", "wheel (>=0.43.0)"] -cover = ["pytest-cov"] -doc = ["furo", "jaraco.packaging (>=9.3)", "jaraco.tidelift (>=1.4)", "pygments-github-lexers (==0.0.5)", "pyproject-hooks (!=1.1)", "rst.linker (>=1.9)", "sphinx (>=3.5)", "sphinx-favicon", "sphinx-inline-tabs", "sphinx-lint", "sphinx-notfound-page (>=1,<2)", "sphinx-reredirects", "sphinxcontrib-towncrier", "towncrier (<24.7)"] -enabler = ["pytest-enabler (>=2.2)"] -test = ["build[virtualenv] (>=1.0.3)", "filelock (>=3.4.0)", "ini2toml[lite] (>=0.14)", "jaraco.develop (>=7.21)", "jaraco.envs (>=2.2)", "jaraco.path (>=3.2.0)", "jaraco.test (>=5.5)", "packaging (>=24.2)", "pip (>=19.1)", "pyproject-hooks (!=1.1)", "pytest (>=6,!=8.1.*)", "pytest-home (>=0.5)", "pytest-perf", "pytest-subprocess", "pytest-timeout", "pytest-xdist (>=3)", "tomli-w (>=1.0.0)", "virtualenv (>=13.0.0)", "wheel (>=0.44.0)"] -type = ["importlib_metadata (>=7.0.2)", "jaraco.develop (>=7.21)", "mypy (>=1.12,<1.14)", "pytest-mypy"] - [metadata] -lock-version = "2.0" +lock-version = "2.1" python-versions = "^3.13" -content-hash = "622d4971a16f64f64de12d8208024658e6d7b9e30e1315e6970110ec5e70975c" +content-hash = "f3fb8b00f781abfc5c963f9a266d16757b9b92cc415866eb28f156b4083fbf48" diff --git a/spec/fixtures/poetry_editable/pyproject.toml b/spec/fixtures/poetry_editable/pyproject.toml index b7659537b..256e2d984 100644 --- a/spec/fixtures/poetry_editable/pyproject.toml +++ b/spec/fixtures/poetry_editable/pyproject.toml @@ -6,7 +6,7 @@ authors = [] [tool.poetry.dependencies] python = "^3.13" -gunicorn = { git = "https://github.com/benoitc/gunicorn.git", tag = "20.1.0", develop = true } +gunicorn = { git = "https://github.com/benoitc/gunicorn.git", develop = true } local-package-pyproject-toml = { path = "packages/local_package_pyproject_toml", develop = true } local-package-setup-py = { path = "packages/local_package_setup_py", develop = true } diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 30126af91..aa4f58dd2 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -10,7 +10,7 @@ it 'installs both normal and test dependencies and uses cache on subsequent runs' do app.run_ci do |test_run| - expect(test_run.output).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + expect(clean_output(test_run.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} @@ -62,7 +62,7 @@ REGEX test_run.run_again - expect(test_run.output).to include(<<~OUTPUT) + expect(clean_output(test_run.output)).to include(<<~OUTPUT) -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Restoring cache @@ -81,7 +81,7 @@ it 'installs both normal and test dependencies and uses cache on subsequent runs' do app.run_ci do |test_run| - expect(test_run.output).to match(Regexp.new(<<~REGEX)) + expect(clean_output(test_run.output)).to match(Regexp.new(<<~REGEX)) -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} @@ -134,7 +134,7 @@ REGEX test_run.run_again - expect(test_run.output).to match(Regexp.new(<<~REGEX)) + expect(clean_output(test_run.output)).to match(Regexp.new(<<~REGEX)) -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Restoring cache @@ -156,7 +156,7 @@ it 'installs both normal and test dependencies and uses cache on subsequent runs' do app.run_ci do |test_run| - expect(test_run.output).to match(Regexp.new(<<~REGEX)) + expect(clean_output(test_run.output)).to match(Regexp.new(<<~REGEX)) -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} @@ -217,7 +217,7 @@ REGEX test_run.run_again - expect(test_run.output).to include(<<~OUTPUT) + expect(clean_output(test_run.output)).to include(<<~OUTPUT) -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Restoring cache diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 64ddb274f..f67a55aa1 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -91,7 +91,8 @@ end # This test intentionally uses Python 3.12, so that we test rewriting using older globally installed - # setuptools. The Poetry equivalent of this test covers the PEP-517/518 setuptools case. + # setuptools (which causes .egg-link files to be created too). The Pipenv and Poetry equivalents of + # this test covers the PEP-517/518 setuptools case. context 'when requirements.txt contains editable requirements (both VCS and local package)' do let(:buildpacks) { [:default, 'heroku-community/inline'] } let(:app) { Hatchet::Runner.new('spec/fixtures/pip_editable', buildpacks:) } @@ -159,6 +160,8 @@ remote: Running entrypoint for the setup.py-based local package: Hello setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) REGEX + # Test that the VCS repo checkout was cached correctly. + expect(app.output).to include('Updating /app/.heroku/python/src/gunicorn clone (to revision 20.1.0)') end end end diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 18addeaec..0ff606900 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -47,7 +47,7 @@ remote: typing_extensions 4.12.2 remote: virtualenv .+ remote: - remote: \\ + remote: REGEX app.commit! app.push! @@ -348,6 +348,8 @@ end end + # This test has to use Python 3.12 until we work around the Pipenv editable VCS dependency + # cache invalidation bug when using pyproject.toml / PEP517 based installs. context 'when Pipfile contains editable requirements' do let(:buildpacks) { [:default, 'heroku-community/inline'] } let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_editable', buildpacks:) } @@ -355,12 +357,15 @@ it 'rewrites .pth, .egg-link and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Installing dependencies using 'pipenv install --deploy' + remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Running bin/post_compile hook remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py + remote: _pipenv_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! @@ -371,6 +376,7 @@ remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py + remote: _pipenv_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! @@ -384,6 +390,7 @@ __editable___local_package_pyproject_toml_0_0_1_finder.py:/app/packages/local_package_pyproject_toml/local_package_pyproject_toml'} gunicorn.egg-link:/app/.heroku/python/src/gunicorn local-package-setup-py.egg-link:/app/packages/local_package_setup_py + _pipenv_editable.pth:/app Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! Running entrypoint for the setup.py-based local package: Hello setup.py! @@ -394,12 +401,15 @@ app.commit! app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Installing dependencies using 'pipenv install --deploy' + remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Running bin/post_compile hook remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py + remote: _pipenv_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! @@ -410,6 +420,7 @@ remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py + remote: _pipenv_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 1fa8ae836..456a35996 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -94,63 +94,84 @@ let(:buildpacks) { [:default, 'heroku-community/inline'] } let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_editable', buildpacks:) } - it 'rewrites .pth, .egg-link and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do + it 'rewrites .pth and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Installing dependencies using 'poetry sync --only main' + remote: Installing dependencies from lock file + remote: + remote: Package operations: 4 installs, 0 updates, 0 removals + remote: + remote: - Installing packaging \\(24.2\\) + remote: - Installing gunicorn \\(23.0.0 bacbf8a\\) + remote: - Installing local-package-pyproject-toml \\(0.0.1 /tmp/build_.+/packages/local_package_pyproject_toml\\) + remote: - Installing local-package-setup-py \\(0.0.1 /tmp/build_.+/packages/local_package_setup_py\\) + remote: + remote: Installing the current project: poetry-editable \\(0.0.1\\) remote: -----> Running bin/post_compile hook - remote: __editable___gunicorn_20_1_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} + remote: __editable___gunicorn_23_0_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: poetry_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Inline app detected - remote: __editable___gunicorn_20_1_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} + remote: __editable___gunicorn_23_0_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: poetry_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) REGEX # Test rewritten paths work at runtime. expect(app.run('bin/test-entrypoints.sh')).to include(<<~OUTPUT) - __editable___gunicorn_20_1_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} + __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} __editable___local_package_pyproject_toml_0_0_1_finder.py:/app/packages/local_package_pyproject_toml/local_package_pyproject_toml'} __editable___local_package_setup_py_0_0_1_finder.py:/app/packages/local_package_setup_py/local_package_setup_py'} poetry_editable.pth:/app Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! Running entrypoint for the setup.py-based local package: Hello setup.py! - Running entrypoint for the VCS package: gunicorn (version 20.1.0) + Running entrypoint for the VCS package: gunicorn (version 23.0.0) OUTPUT # Test that the cached .pth files work correctly. app.commit! app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Installing dependencies using 'poetry sync --only main' + remote: Installing dependencies from lock file + remote: + remote: Package operations: 0 installs, 3 updates, 0 removals + remote: + remote: - Updating gunicorn \\(23.0.0 /app/.heroku/python/src/gunicorn -> 23.0.0 bacbf8a\\) + remote: - Updating local-package-pyproject-toml \\(0.0.1 /tmp/build_.+/packages/local_package_pyproject_toml -> 0.0.1 /tmp/build_.+/packages/local_package_pyproject_toml\\) + remote: - Updating local-package-setup-py \\(0.0.1 /tmp/build_.+/packages/local_package_setup_py -> 0.0.1 /tmp/build_.+/packages/local_package_setup_py\\) + remote: + remote: Installing the current project: poetry-editable \\(0.0.1\\) remote: -----> Running bin/post_compile hook - remote: __editable___gunicorn_20_1_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} + remote: __editable___gunicorn_23_0_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: poetry_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Inline app detected - remote: __editable___gunicorn_20_1_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} + remote: __editable___gunicorn_23_0_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: poetry_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) REGEX end end @@ -160,7 +181,7 @@ # chosen Poetry version also supports our oldest supported Python version. The fixture # also includes a `brotli` directory to test the workaround for an `ensurepip` bug in # older Python versions: https://github.com/heroku/heroku-buildpack-python/issues/1697 - context 'when using the oldest supported Python version' do + context 'when using our oldest supported Python version' do let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_oldest_python') } it 'installs successfully' do From bb282273a9c0fdcf952408fe6a668247427cf009 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 21 Feb 2025 08:30:15 +0000 Subject: [PATCH 103/406] `make run` workflow improvements (#1755) Updates `make run` (used locally during development) to: - Run a second build after the first, which allows for easy testing of cached workflows. This second build uses a different build directory path to the first, to match the Heroku Cedar/classic build system and allow for testing that relocation/path rewriting works as expected. - Exit non-zero if the compile failed (it previously didn't, so we could test `bin/report` for failing builds - but that's now handled via a customisable exit code). GUS-W-17879839. --- .github/workflows/ci.yml | 2 +- Makefile | 23 ++++++++++++++--------- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 57f26122d..56224eeb1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -73,4 +73,4 @@ jobs: - name: Run buildpack using default app fixture run: make run - name: Run buildpack using an app fixture that's expected to fail - run: make run FIXTURE=spec/fixtures/python_version_file_invalid_version/ + run: make run FIXTURE=spec/fixtures/python_version_file_invalid_version/ COMPILE_FAILURE_EXIT_CODE=0 diff --git a/Makefile b/Makefile index 43a6113b3..6ba94fef7 100644 --- a/Makefile +++ b/Makefile @@ -3,6 +3,8 @@ STACK ?= heroku-24 FIXTURE ?= spec/fixtures/python_version_unspecified +# Allow overriding the exit code in CI, so we can test bin/report works for failing builds. +COMPILE_FAILURE_EXIT_CODE ?= 1 # Converts a stack name of `heroku-NN` to its build Docker image tag of `heroku/heroku:NN-build`. STACK_IMAGE_TAG := heroku/$(subst -,:,$(STACK))-build @@ -24,18 +26,21 @@ format: run: @echo "Running buildpack using: STACK=$(STACK) FIXTURE=$(FIXTURE)" @docker run --rm -v $(PWD):/src:ro --tmpfs /app -e "HOME=/app" -e "STACK=$(STACK)" "$(STACK_IMAGE_TAG)" \ - bash -euo pipefail -c '\ - mkdir /tmp/buildpack /tmp/build /tmp/cache /tmp/env; \ + bash -euo pipefail -O dotglob -c '\ + mkdir /tmp/buildpack /tmp/cache /tmp/env; \ cp -r /src/{bin,lib,requirements,vendor} /tmp/buildpack; \ - cp -rT /src/$(FIXTURE) /tmp/build; \ + cp -r /src/$(FIXTURE) /tmp/build_1; \ cd /tmp/buildpack; \ unset $$(printenv | cut -d '=' -f 1 | grep -vE "^(HOME|LANG|PATH|STACK)$$"); \ - echo -e "\n~ Detect:" && ./bin/detect /tmp/build; \ - echo -e "\n~ Compile:" && { ./bin/compile /tmp/build /tmp/cache /tmp/env || COMPILE_FAILED=1; }; \ - echo -e "\n~ Report:" && ./bin/report /tmp/build /tmp/cache /tmp/env; \ - [[ "$${COMPILE_FAILED:-}" == "1" ]] && exit 0; \ - [[ -f /tmp/build/bin/compile ]] && { echo -e "\n~ Compile (Inline Buildpack):" && (source ./export && /tmp/build/bin/compile /tmp/build /tmp/cache /tmp/env); }; \ - echo -e "\n~ Release:" && ./bin/release /tmp/build; \ + echo -en "\n~ Detect: " && ./bin/detect /tmp/build_1; \ + echo -e "\n~ Compile:" && { ./bin/compile /tmp/build_1 /tmp/cache /tmp/env || COMPILE_FAILED=1; }; \ + echo -e "\n~ Report:" && ./bin/report /tmp/build_1 /tmp/cache /tmp/env; \ + [[ "$${COMPILE_FAILED:-}" == "1" ]] && exit $(COMPILE_FAILURE_EXIT_CODE); \ + [[ -f /tmp/build_1/bin/compile ]] && { echo -e "\n~ Compile (Inline Buildpack):" && (source ./export && /tmp/build_1/bin/compile /tmp/build_1 /tmp/cache /tmp/env); }; \ + echo -e "\n~ Release:" && ./bin/release /tmp/build_1; \ + rm -rf /app/* /tmp/buildpack/export /tmp/build_1; \ + cp -r /src/$(FIXTURE) /tmp/build_2; \ + echo -e "\n~ Recompile:" && ./bin/compile /tmp/build_2 /tmp/cache /tmp/env; \ echo -e "\nBuild successful!"; \ ' @echo From b38d4620c382d9645d3cca449f24748acee5e5bd Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Sat, 22 Feb 2025 08:21:47 +0000 Subject: [PATCH 104/406] Rewrite editable VCS paths at build time too (#1756) Heroku builds occur at a different path to which the app will be run at run-time. As such, we have to perform path rewriting for editable dependencies, so that they work after relocation. The existing rewriting is performed at app boot (see code comment for more details), and works fine with pip and Poetry. However, I discovered that Pipenv doesn't correctly reinstall editable VCS dependencies if they use the new PEP660 style editable interface, which I've reported upstream here: https://github.com/pypa/pipenv/issues/6348 This issue has affected apps using editable VCS dependencies with Pipenv for some time, but until now only at build-time for cached builds. However, after #1753 (which thankfully isn't yet released, due to me catching this as part of updating the tests to exercise the new PEP660 style editable interface) would otherwise affect apps at run-time too. As a workaround, we can perform build time rewriting of paths too, but must do so only for VCS dependencies (see code comment for why). Lastly, the Pipenv bug also requires that we perform explicit cache invalidation for Pipenv apps after the src dir move in #1753. GUS-W-17884520. --- CHANGELOG.md | 1 + bin/compile | 42 ++++++++++++++--- lib/cache.sh | 4 ++ .../pip_editable/bin/test-entrypoints.sh | 2 +- spec/fixtures/pipenv_editable/.python-version | 3 -- spec/fixtures/pipenv_editable/Pipfile | 2 +- spec/fixtures/pipenv_editable/Pipfile.lock | 20 ++++----- .../pipenv_editable/bin/test-entrypoints.sh | 4 +- spec/fixtures/pipenv_editable/pyproject.toml | 2 +- spec/hatchet/pip_spec.rb | 16 +++---- spec/hatchet/pipenv_spec.rb | 45 +++++++------------ spec/hatchet/poetry_spec.rb | 8 ++-- 12 files changed, 85 insertions(+), 64 deletions(-) delete mode 100644 spec/fixtures/pipenv_editable/.python-version diff --git a/CHANGELOG.md b/CHANGELOG.md index bf4cba079..73ffabe25 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Added build-time rewriting of editable VCS dependency paths (in addition to the existing run-time rewriting), to work around an upstream Pipenv bug with editable VCS dependencies not being reinstalled correctly for cached builds. ([#1756](https://github.com/heroku/heroku-buildpack-python/pull/1756)) - Changed the location of repositories for editable VCS dependencies when using pip and Pipenv, to improve build performance and match the behaviour when using Poetry. ([#1753](https://github.com/heroku/heroku-buildpack-python/pull/1753)) ## [v277] - 2025-02-17 diff --git a/bin/compile b/bin/compile index 9f2869fc6..7a20a8faf 100755 --- a/bin/compile +++ b/bin/compile @@ -280,13 +280,43 @@ if [[ \$HOME != "/app" ]]; then fi EOT -# At runtime, rewrite paths in editable package .egg-link, .pth and finder files from the build time paths -# (such as `/tmp/build_`) back to `/app`. This is not done during the build itself, since later -# buildpacks still need the build time paths. +# When dependencies are installed in editable mode, the package manager/build backend creates `.pth` +# (and related) files in site-packages, which contain absolute path references to the actual location +# of the packages. By default the Heroku build runs from a directory like `/tmp/build_`, which +# changes every build and also differs from the app location at runtime (`/app`). This means any build +# directory paths referenced in .pth and related files will no longer exist at runtime or during cached +# rebuilds, unless we rewrite the paths. +# +# Ideally, we would be able to rewrite all paths to use the `/app/.heroku/python/` symlink trick we use +# when invoking Python, since then the same path would work across the current build, runtime and cached +# rebuilds. However, this trick only works for paths under that directory (since it's not possible to +# symlink `/app` or other directories we don't own), and when apps use path-based editable dependencies +# the paths will be outside of that (such as a subdirectory of the app source, or even the root of the +# build directory). We also can't just rewrite all paths now ready for runtime, since other buildpacks +# might run after this one that make use of the editable dependencies. As such, we have to perform path +# rewriting for path-based editable dependencies at app boot instead. +# +# For VCS editable dependencies, we can use the symlink trick and so configure the repo checkout location +# as `/app/.heroku/python/src/`, which in theory should mean the `.pth` files use that path. However, +# some build backends (such as setuptools' PEP660 implementation) call realpath on it causing the +# `/tmp/build_*` location to be written instead, meaning VCS src paths need to be rewritten regardless. +# +# In addition to ensuring dependencies work for subsequent buildpacks and at runtime, they must also +# work for cached rebuilds. Most package managers will reinstall editable dependencies regardless on +# next install, which means we can avoid having to rewrite paths on cache restore from the old build +# directory to the new location (`/tmp/build_`). However, Pipenv has a bug when using +# PEP660 style editable VCS dependencies where it won't reinstall if it's missing (or in our case, the +# path has changed), which means we must make sure that VCS src paths stored in the cache do use the +# symlink path. See: https://github.com/pypa/pipenv/issues/6348 +# +# As such, we have to perform two rewrites: +# 1. At build time, of just the VCS editable paths (which we can safely change to /app paths early). +# 2. At runtime, to rewrite the remaining path-based editable dependency paths. if [[ "${BUILD_DIR}" != "/app" ]]; then - cat <>"$PROFILE_PATH" -find .heroku/python/lib/python*/site-packages/ -type f -and \( -name '*.egg-link' -or -name '*.pth' -or -name '__editable___*_finder.py' \) -exec sed -i -e 's#${BUILD_DIR}#/app#' {} \+ -EOT + find .heroku/python/lib/python*/site-packages/ -type f -and \( -name '*.egg-link' -or -name '*.pth' -or -name '__editable___*_finder.py' \) -exec sed -i -e "s#${BUILD_DIR}/.heroku/python#/app/.heroku/python#" {} \+ + cat <<-EOT >>"${PROFILE_PATH}" + find .heroku/python/lib/python*/site-packages/ -type f -and \( -name '*.egg-link' -or -name '*.pth' -or -name '__editable___*_finder.py' \) -exec sed -i -e 's#${BUILD_DIR}#/app#' {} \+ + EOT fi # Install sane-default script for $WEB_CONCURRENCY and $FORWARDED_ALLOW_IPS. diff --git a/lib/cache.sh b/lib/cache.sh index 8353a9093..f5055b982 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -102,6 +102,10 @@ function cache::restore() { elif [[ "${cached_pipenv_version}" != "${PIPENV_VERSION:?}" ]]; then cache_invalidation_reasons+=("The Pipenv version has changed from ${cached_pipenv_version} to ${PIPENV_VERSION}") fi + # TODO: Remove this next time the Pipenv version is bumped (since it will trigger cache invalidation of its own) + if [[ -d "${cache_dir}/.heroku/src" ]]; then + cache_invalidation_reasons+=("The editable VCS repository location has changed (and Pipenv doesn't handle this correctly)") + fi ;; poetry) local cached_poetry_version diff --git a/spec/fixtures/pip_editable/bin/test-entrypoints.sh b/spec/fixtures/pip_editable/bin/test-entrypoints.sh index 4c27ba0f4..43afcb553 100755 --- a/spec/fixtures/pip_editable/bin/test-entrypoints.sh +++ b/spec/fixtures/pip_editable/bin/test-entrypoints.sh @@ -4,7 +4,7 @@ set -euo pipefail cd .heroku/python/lib/python*/site-packages/ -# List any path like strings in .egg-link, .pth, and finder files in site-packages. +# List any path like strings in the .egg-link, .pth, and finder files in site-packages. grep --extended-regexp --only-matching -- '/\S+' *.egg-link *.pth __editable___*_finder.py | sort echo diff --git a/spec/fixtures/pipenv_editable/.python-version b/spec/fixtures/pipenv_editable/.python-version deleted file mode 100644 index b81161743..000000000 --- a/spec/fixtures/pipenv_editable/.python-version +++ /dev/null @@ -1,3 +0,0 @@ -# Note: This test has to use Python 3.12 until we work around the -# Pipenv editable VCS dependency cache invalidation bug. -3.12 diff --git a/spec/fixtures/pipenv_editable/Pipfile b/spec/fixtures/pipenv_editable/Pipfile index 910d90713..8bb51ee30 100644 --- a/spec/fixtures/pipenv_editable/Pipfile +++ b/spec/fixtures/pipenv_editable/Pipfile @@ -4,7 +4,7 @@ verify_ssl = true name = "pypi" [packages] -gunicorn = {git = "git+https://github.com/benoitc/gunicorn", ref = "20.1.0", editable = true} +gunicorn = {git = "git+https://github.com/benoitc/gunicorn", editable = true} local-package-pyproject-toml = {file = "packages/local_package_pyproject_toml", editable = true} local-package-setup-py = {file = "packages/local_package_setup_py", editable = true} pipenv-editable = {file = ".", editable = true} diff --git a/spec/fixtures/pipenv_editable/Pipfile.lock b/spec/fixtures/pipenv_editable/Pipfile.lock index f6f2acc1b..8c4b604a3 100644 --- a/spec/fixtures/pipenv_editable/Pipfile.lock +++ b/spec/fixtures/pipenv_editable/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "ed18332549853701536926a2bae44d9fc142a08c105d4fbf108dbde04b3fedf2" + "sha256": "bedd8ea507283c5458c9c2cb1fd55a6e5e69fecba7814ffc96bf25e03feaeabf" }, "pipfile-spec": 6, "requires": {}, @@ -18,7 +18,7 @@ "editable": true, "git": "git+https://github.com/benoitc/gunicorn", "markers": "python_version >= '3.7'", - "ref": "61ccfd6c38d477a908e0f376757bbb884438053a" + "ref": "bacbf8aa5152b94e44aa5d2a94aeaf0318a85248" }, "local-package-pyproject-toml": { "editable": true, @@ -28,17 +28,17 @@ "editable": true, "file": "packages/local_package_setup_py" }, + "packaging": { + "hashes": [ + "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759", + "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f" + ], + "markers": "python_version >= '3.8'", + "version": "==24.2" + }, "pipenv-editable": { "editable": true, "file": "." - }, - "setuptools": { - "hashes": [ - "sha256:c5afc8f407c626b8313a86e10311dd3f661c6cd9c09d4bf8c15c0e11f9f2b0e6", - "sha256:e3982f444617239225d675215d51f6ba05f845d4eec313da4418fdbb56fb27e3" - ], - "markers": "python_version >= '3.9'", - "version": "==75.8.0" } }, "develop": {} diff --git a/spec/fixtures/pipenv_editable/bin/test-entrypoints.sh b/spec/fixtures/pipenv_editable/bin/test-entrypoints.sh index 4c27ba0f4..0e66f44bb 100755 --- a/spec/fixtures/pipenv_editable/bin/test-entrypoints.sh +++ b/spec/fixtures/pipenv_editable/bin/test-entrypoints.sh @@ -4,8 +4,8 @@ set -euo pipefail cd .heroku/python/lib/python*/site-packages/ -# List any path like strings in .egg-link, .pth, and finder files in site-packages. -grep --extended-regexp --only-matching -- '/\S+' *.egg-link *.pth __editable___*_finder.py | sort +# List any path like strings in the .pth and finder files in site-packages. +grep --extended-regexp --only-matching -- '/\S+' *.pth __editable___*_finder.py | sort echo echo -n "Running entrypoint for the pyproject.toml-based local package: " diff --git a/spec/fixtures/pipenv_editable/pyproject.toml b/spec/fixtures/pipenv_editable/pyproject.toml index cf7b46f69..2ca4481db 100644 --- a/spec/fixtures/pipenv_editable/pyproject.toml +++ b/spec/fixtures/pipenv_editable/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "pipenv-editable" version = "0.0.0" -requires-python = ">=3.12" +requires-python = ">=3.13" [build-system] requires = ["hatchling"] diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index f67a55aa1..f27f713c4 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -101,20 +101,20 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Running bin/post_compile hook - remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn + remote: easy-install.pth:/app/.heroku/python/src/gunicorn remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn + remote: gunicorn.egg-link:/app/.heroku/python/src/gunicorn remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) remote: -----> Inline app detected - remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn + remote: easy-install.pth:/app/.heroku/python/src/gunicorn remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn + remote: gunicorn.egg-link:/app/.heroku/python/src/gunicorn remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! @@ -140,20 +140,20 @@ app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Running bin/post_compile hook - remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn + remote: easy-install.pth:/app/.heroku/python/src/gunicorn remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn + remote: gunicorn.egg-link:/app/.heroku/python/src/gunicorn remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) remote: -----> Inline app detected - remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn + remote: easy-install.pth:/app/.heroku/python/src/gunicorn remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn + remote: gunicorn.egg-link:/app/.heroku/python/src/gunicorn remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 0ff606900..02f9007fa 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -337,6 +337,7 @@ remote: -----> Discarding cache since: remote: - The Python version has changed from 3.12.4 to #{DEFAULT_PYTHON_FULL_VERSION} remote: - The Pipenv version has changed from 2023.12.1 to #{PIPENV_VERSION} + remote: - The editable VCS repository location has changed \\(and Pipenv doesn't handle this correctly\\) remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} @@ -348,53 +349,45 @@ end end - # This test has to use Python 3.12 until we work around the Pipenv editable VCS dependency - # cache invalidation bug when using pyproject.toml / PEP517 based installs. context 'when Pipfile contains editable requirements' do let(:buildpacks) { [:default, 'heroku-community/inline'] } let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_editable', buildpacks:) } - it 'rewrites .pth, .egg-link and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do + it 'rewrites .pth and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Running bin/post_compile hook - remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn - remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: _pipenv_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Inline app detected - remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn - remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: _pipenv_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) REGEX # Test rewritten paths work at runtime. expect(app.run('bin/test-entrypoints.sh')).to include(<<~OUTPUT) - easy-install.pth:/app/.heroku/python/src/gunicorn - easy-install.pth:/app/packages/local_package_setup_py + __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} __editable___local_package_pyproject_toml_0_0_1_finder.py:/app/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - gunicorn.egg-link:/app/.heroku/python/src/gunicorn - local-package-setup-py.egg-link:/app/packages/local_package_setup_py + __editable___local_package_setup_py_0_0_1_finder.py:/app/packages/local_package_setup_py/local_package_setup_py'} _pipenv_editable.pth:/app Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! Running entrypoint for the setup.py-based local package: Hello setup.py! - Running entrypoint for the VCS package: gunicorn (version 20.1.0) + Running entrypoint for the VCS package: gunicorn (version 23.0.0) OUTPUT # Test that the cached .pth files work correctly. @@ -404,27 +397,23 @@ remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Running bin/post_compile hook - remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn - remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: _pipenv_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Inline app detected - remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn - remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: _pipenv_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) REGEX end end diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 456a35996..874715f00 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -109,7 +109,7 @@ remote: remote: Installing the current project: poetry-editable \\(0.0.1\\) remote: -----> Running bin/post_compile hook - remote: __editable___gunicorn_23_0_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} + remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: poetry_editable.pth:/tmp/build_.+ @@ -118,7 +118,7 @@ remote: Running entrypoint for the setup.py-based local package: Hello setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Inline app detected - remote: __editable___gunicorn_23_0_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} + remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: poetry_editable.pth:/tmp/build_.+ @@ -155,7 +155,7 @@ remote: remote: Installing the current project: poetry-editable \\(0.0.1\\) remote: -----> Running bin/post_compile hook - remote: __editable___gunicorn_23_0_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} + remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: poetry_editable.pth:/tmp/build_.+ @@ -164,7 +164,7 @@ remote: Running entrypoint for the setup.py-based local package: Hello setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Inline app detected - remote: __editable___gunicorn_23_0_0_finder.py:/tmp/build_.+/.heroku/python/src/gunicorn/gunicorn'} + remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: poetry_editable.pth:/tmp/build_.+ From bf280f4524e53f48aaa031d25c9e789a2bc7160d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 22 Feb 2025 08:39:32 +0000 Subject: [PATCH 105/406] Bump the ruby-dependencies group with 4 updates (#1757) Bumps the ruby-dependencies group with 4 updates: [logger](https://github.com/ruby/logger), [rspec-core](https://github.com/rspec/rspec-core), [rubocop](https://github.com/rubocop/rubocop) and [rubocop-rspec](https://github.com/rubocop/rubocop-rspec). Updates `logger` from 1.6.5 to 1.6.6 - [Release notes](https://github.com/ruby/logger/releases) - [Commits](https://github.com/ruby/logger/compare/v1.6.5...v1.6.6) Updates `rspec-core` from 3.13.2 to 3.13.3 - [Release notes](https://github.com/rspec/rspec-core/releases) - [Changelog](https://github.com/rspec/rspec-core/blob/main/Changelog.md) - [Commits](https://github.com/rspec/rspec-core/commits) Updates `rubocop` from 1.71.1 to 1.72.2 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.71.1...v1.72.2) Updates `rubocop-rspec` from 3.4.0 to 3.5.0 - [Release notes](https://github.com/rubocop/rubocop-rspec/releases) - [Changelog](https://github.com/rubocop/rubocop-rspec/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop-rspec/compare/v3.4.0...v3.5.0) --- updated-dependencies: - dependency-name: logger dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rspec-core dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rubocop dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies - dependency-name: rubocop-rspec dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index b708a7e83..efa4e8d6b 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -19,16 +19,17 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.9.1) + json (2.10.1) language_server-protocol (3.17.0.4) - logger (1.6.5) + lint_roller (1.1.0) + logger (1.6.6) moneta (1.0.0) multi_json (1.15.0) parallel (1.26.3) parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) - parser (3.3.7.0) + parser (3.3.7.1) ast (~> 2.4.1) racc platform-api (3.7.0) @@ -40,7 +41,7 @@ GEM rate_throttle_client (0.1.2) regexp_parser (2.10.0) rrrretry (1.0.0) - rspec-core (3.13.2) + rspec-core (3.13.3) rspec-support (~> 3.13.0) rspec-expectations (3.13.3) diff-lcs (>= 1.2.0, < 2.0) @@ -48,9 +49,10 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.2) - rubocop (1.71.1) + rubocop (1.72.2) json (~> 2.3) - language_server-protocol (>= 3.17.0) + language_server-protocol (~> 3.17.0.2) + lint_roller (~> 1.1.0) parallel (~> 1.10) parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) @@ -60,8 +62,9 @@ GEM unicode-display_width (>= 2.4.0, < 4.0) rubocop-ast (1.38.0) parser (>= 3.3.1.0) - rubocop-rspec (3.4.0) - rubocop (~> 1.61) + rubocop-rspec (3.5.0) + lint_roller (~> 1.1) + rubocop (~> 1.72, >= 1.72.1) ruby-progressbar (1.13.0) thor (1.3.2) threaded (0.0.4) From 53a59ae3cb7f0bd7b55d5bdcc2005989b2d3825b Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Mon, 24 Feb 2025 09:22:12 +0000 Subject: [PATCH 106/406] Prepare release v278 (#1760) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 73ffabe25..3b17a1d9a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v278] - 2025-02-24 + - Added build-time rewriting of editable VCS dependency paths (in addition to the existing run-time rewriting), to work around an upstream Pipenv bug with editable VCS dependencies not being reinstalled correctly for cached builds. ([#1756](https://github.com/heroku/heroku-buildpack-python/pull/1756)) - Changed the location of repositories for editable VCS dependencies when using pip and Pipenv, to improve build performance and match the behaviour when using Poetry. ([#1753](https://github.com/heroku/heroku-buildpack-python/pull/1753)) @@ -1170,7 +1173,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v277...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v278...main +[v278]: https://github.com/heroku/heroku-buildpack-python/compare/v277...v278 [v277]: https://github.com/heroku/heroku-buildpack-python/compare/v276...v277 [v276]: https://github.com/heroku/heroku-buildpack-python/compare/v275...v276 [v275]: https://github.com/heroku/heroku-buildpack-python/compare/v274...v275 From 992f51d5c94bdf0de25d651d7bc9ca5e20b65eb3 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 24 Feb 2025 14:35:10 +0000 Subject: [PATCH 107/406] Stop installing pip into Poetry's virtual environment (#1761) This is an alternative approach to installing Poetry that means we can skip installing pip into its virtual environment, but still support the outdated Python versions which bundle older pip (that don't support the `--python` option; see #1687) or that don't correctly isolate the environment when running `ensurepip` (see #1698). Skipping installing pip speeds up the cold cache build for Poetry slightly, and also reduces the build cache size (which will help with the cache save and restore times for warm builds too). The pip installed in the Poetry venv wasn't exposed to apps (since it wasn't on `PATH`) so is safe to remove. GUS-W-17895154. --- CHANGELOG.md | 1 + bin/compile | 2 +- lib/poetry.sh | 31 +++++++++++++++++-------------- 3 files changed, 19 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b17a1d9a..78d572d74 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Stopped installing pip into Poetry's virtual environment. ([#1761](https://github.com/heroku/heroku-buildpack-python/pull/1761)) ## [v278] - 2025-02-24 diff --git a/bin/compile b/bin/compile index 7a20a8faf..cda662380 100755 --- a/bin/compile +++ b/bin/compile @@ -205,7 +205,7 @@ case "${package_manager}" in pipenv::install_pipenv ;; poetry) - poetry::install_poetry "${CACHE_DIR}" "${EXPORT_PATH}" + poetry::install_poetry "${python_home}" "${python_major_version}" "${CACHE_DIR}" "${EXPORT_PATH}" ;; *) utils::abort_internal_error "Unhandled package manager: ${package_manager}" diff --git a/lib/poetry.sh b/lib/poetry.sh index a227ae1cd..5bdfdf9c5 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -7,8 +7,10 @@ set -euo pipefail POETRY_VERSION=$(utils::get_requirement_version 'poetry') function poetry::install_poetry() { - local cache_dir="${1}" - local export_file="${2}" + local python_home="${1}" + local python_major_version="${2}" + local cache_dir="${3}" + local export_file="${4}" # We store Poetry in the build cache, since we only need it during the build. local poetry_root="${cache_dir}/.heroku/python-poetry" @@ -39,17 +41,12 @@ function poetry::install_poetry() { rm -rf "${poetry_root}" mkdir -p "${poetry_root}" - # We can't use the pip wheel bundled within Python's standard library to install Poetry - # (which would allow us to use `--without-pip` here to skip the pip install), since it - # requires using the `--python` option, which was only added in pip v22.3. And whilst - # all major Python versions we support now bundled a newer pip than that, some apps - # are still using outdated patch releases of those Python versions, whose bundled pip - # can be older (for example Python 3.9.0 ships with pip v20.2.1). Once Python 3.10 EOLs - # we can switch back to the previous approach since Python 3.11.0 ships with pip v22.3. - # Changing the working directory away from the build dir is required to work around an - # `ensurepip` bug in older Python versions, where it doesn't run Python in isolated mode: - # https://github.com/heroku/heroku-buildpack-python/issues/1697 - if ! (cd "${poetry_root}" && python -m venv "${poetry_venv_dir}"); then + # We use the pip wheel bundled within Python's standard library to install Poetry. + # Whilst Poetry does still require pip for some tasks (such as package uninstalls), + # it bundles its own copy for use as a fallback. As such we don't need to install pip + # into the Poetry venv (and in fact, Poetry wouldn't use this install anyway, since + # it only finds an external pip if it exists in the target venv). + if ! python -m venv --without-pip "${poetry_venv_dir}"; then output::error <<-EOF Internal Error: Unable to create virtual environment for Poetry. @@ -62,8 +59,14 @@ function poetry::install_poetry() { exit 1 fi + local bundled_pip_module_path + bundled_pip_module_path="$(utils::bundled_pip_module_path "${python_home}" "${python_major_version}")" + + # We must call the venv Python directly here, rather than relying on pip's `--python` + # option, since `--python` was only added in pip v22.3, so isn't supported by the older + # pip versions bundled with Python 3.9/3.10. if ! { - "${poetry_venv_dir}/bin/pip" \ + "${poetry_venv_dir}/bin/python" "${bundled_pip_module_path}" \ install \ --disable-pip-version-check \ --no-cache-dir \ From ff523c256984f4bb7b5dc694919f65837db3bb3a Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 24 Feb 2025 14:38:05 +0000 Subject: [PATCH 108/406] Misc Poetry cleanups (#1762) Ported from the experimental uv branch, since the uv implementation was based on the Poetry one. --- lib/poetry.sh | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/lib/poetry.sh b/lib/poetry.sh index 5bdfdf9c5..695fe78a3 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -94,14 +94,20 @@ function poetry::install_poetry() { fi export PATH="${poetry_bin_dir}:${PATH}" - echo "export PATH=\"${poetry_bin_dir}:\${PATH}\"" >>"${export_file}" # Force Poetry to manage the system Python site-packages instead of using venvs. export POETRY_VIRTUALENVS_CREATE="false" - echo 'export POETRY_VIRTUALENVS_CREATE="false"' >>"${export_file}" + + # Set the same env vars in the environment used by later buildpacks. + cat >>"${export_file}" <<-EOF + export PATH="${poetry_bin_dir}:\${PATH}" + export POETRY_VIRTUALENVS_CREATE="false" + EOF } # Note: We cache site-packages since: # - It results in faster builds than only caching Poetry's download/wheel cache. +# - It improves the UX of the build log, since Poetry will display which packages were +# added/removed since the last successful build. # - It's safe to do so, since `poetry sync` fully manages the environment (including # e.g. uninstalling packages when they are removed from the lockfile). # From 25172f45c5747fa3248e7a69c06062e79cba79df Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 24 Feb 2025 14:40:42 +0000 Subject: [PATCH 109/406] Migrate to Rubocop's new plugin syntax (#1763) To suppress this Rubocop warning: ``` $ make lint rubocop-rspec extension supports plugin, specify `plugins: rubocop-rspec` instead of `require: rubocop-rspec` in /Users/emorley/src/heroku-buildpack-python/.rubocop.yml. For more information, see https://docs.rubocop.org/rubocop/plugin_migration_guide.html. ``` --- .rubocop.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.rubocop.yml b/.rubocop.yml index 44a322aef..2edf1e039 100644 --- a/.rubocop.yml +++ b/.rubocop.yml @@ -1,4 +1,4 @@ -require: +plugins: - rubocop-rspec AllCops: From 3033b54deff927e3ecc41cbf91469cefb45b26f7 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 24 Feb 2025 15:53:48 +0000 Subject: [PATCH 110/406] Improve package manager bootstrap error messages (#1764) * Any errors are now indented using the same indent helper used for other buildpack subprocesses. * Adds "temporary network issue" verbiage based on that now used for the Python runtime download step. GUS-W-17895970. --- CHANGELOG.md | 1 + lib/pip.sh | 10 +++++++--- lib/pipenv.sh | 11 ++++++++--- lib/poetry.sh | 13 +++++++++---- 4 files changed, 25 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 78d572d74..aa1bcd342 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Improved the error messages shown if installing pip/Poetry/Pipenv fails. ([#1764](https://github.com/heroku/heroku-buildpack-python/pull/1764)) - Stopped installing pip into Poetry's virtual environment. ([#1761](https://github.com/heroku/heroku-buildpack-python/pull/1761)) ## [v278] - 2025-02-24 diff --git a/lib/pip.sh b/lib/pip.sh index 62ebc43ee..4d05462f5 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -46,6 +46,7 @@ function pip::install_pip_setuptools_wheel() { # app's requirements.txt in the last build). The install will be a no-op if the versions match. output::step "Installing ${packages_display_text}" + # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { python "${bundled_pip_module_path}" \ install \ @@ -53,15 +54,18 @@ function pip::install_pip_setuptools_wheel() { --no-cache-dir \ --no-input \ --quiet \ - "${packages_to_install[@]}" + "${packages_to_install[@]}" \ + |& output::indent }; then output::error <<-EOF Error: Unable to install pip. + In some cases, this happens due to a temporary issue with + the network connection or Python Package Index (PyPI). + Try building again to see if the error resolves itself. - If that does not help, check the status of PyPI (the Python - package repository service), here: + If that does not help, check the status of PyPI here: https://status.python.org EOF meta_set "failure_reason" "install-package-manager::pip" diff --git a/lib/pipenv.sh b/lib/pipenv.sh index 9e8eb4594..c5c50beb4 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -18,6 +18,8 @@ function pipenv::install_pipenv() { # TODO: Install Pipenv into a venv so it isn't leaked into the app environment. # TODO: Skip installing Pipenv if its version hasn't changed (once it's installed into a venv). # TODO: Explore viability of making Pipenv only be available during the build, to reduce slug size. + + # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { pip \ install \ @@ -25,15 +27,18 @@ function pipenv::install_pipenv() { --no-cache-dir \ --no-input \ --quiet \ - "pipenv==${PIPENV_VERSION}" + "pipenv==${PIPENV_VERSION}" \ + |& output::indent }; then output::error <<-EOF Error: Unable to install Pipenv. + In some cases, this happens due to a temporary issue with + the network connection or Python Package Index (PyPI). + Try building again to see if the error resolves itself. - If that does not help, check the status of PyPI (the Python - package repository service), here: + If that does not help, check the status of PyPI here: https://status.python.org EOF meta_set "failure_reason" "install-package-manager::pipenv" diff --git a/lib/poetry.sh b/lib/poetry.sh index 695fe78a3..6b4888eea 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -46,7 +46,8 @@ function poetry::install_poetry() { # it bundles its own copy for use as a fallback. As such we don't need to install pip # into the Poetry venv (and in fact, Poetry wouldn't use this install anyway, since # it only finds an external pip if it exists in the target venv). - if ! python -m venv --without-pip "${poetry_venv_dir}"; then + # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. + if ! python -m venv --without-pip "${poetry_venv_dir}" |& output::indent; then output::error <<-EOF Internal Error: Unable to create virtual environment for Poetry. @@ -65,6 +66,7 @@ function poetry::install_poetry() { # We must call the venv Python directly here, rather than relying on pip's `--python` # option, since `--python` was only added in pip v22.3, so isn't supported by the older # pip versions bundled with Python 3.9/3.10. + # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { "${poetry_venv_dir}/bin/python" "${bundled_pip_module_path}" \ install \ @@ -72,15 +74,18 @@ function poetry::install_poetry() { --no-cache-dir \ --no-input \ --quiet \ - "poetry==${POETRY_VERSION}" + "poetry==${POETRY_VERSION}" \ + |& output::indent }; then output::error <<-EOF Error: Unable to install Poetry. + In some cases, this happens due to a temporary issue with + the network connection or Python Package Index (PyPI). + Try building again to see if the error resolves itself. - If that does not help, check the status of PyPI (the Python - package repository service), here: + If that does not help, check the status of PyPI here: https://status.python.org EOF meta_set "failure_reason" "install-package-manager::poetry" From caa1a19a8b47486eb20e169f0362cd8594110e15 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 24 Feb 2025 17:40:36 +0000 Subject: [PATCH 111/406] Stop filtering out pip's "Requirement already satisfied" log lines (#1765) Since while removing them makes the install logs slightly shorter, it hides what is happening and makes it harder to see what package version a historic build may have been using from the logs alone. For example, when comparing a last successful build to a newly failing build. This now matches the behaviour for our other supported package managers, where we don't filter out install lines relating to already installed/cached packages. As a compromise, we still edit the lines slightly, to remove the redundant site-packages path information, which would otherwise cause each package message to span multiple lines. GUS-W-17897541. --- CHANGELOG.md | 1 + lib/pip.sh | 4 +++- spec/hatchet/ci_spec.rb | 7 +++++-- spec/hatchet/pip_spec.rb | 1 + 4 files changed, 10 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index aa1bcd342..5b2b73c3e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Stopped filtering out pip's `Requirement already satisfied:` log lines when installing dependencies. ([#1765](https://github.com/heroku/heroku-buildpack-python/pull/1765)) - Improved the error messages shown if installing pip/Poetry/Pipenv fails. ([#1764](https://github.com/heroku/heroku-buildpack-python/pull/1764)) - Stopped installing pip into Poetry's virtual environment. ([#1761](https://github.com/heroku/heroku-buildpack-python/pull/1761)) diff --git a/lib/pip.sh b/lib/pip.sh index 4d05462f5..e56e6d536 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -106,6 +106,8 @@ function pip::install_dependencies() { # We only display the most relevant command args here, to improve the signal to noise ratio. output::step "Installing dependencies using '${pip_install_command[*]}'" + # The sed usage is to reduce the verbosity of output lines like: + # "Requirement already satisfied: typing-extensions==4.12.2 in /app/.heroku/python/lib/python3.13/site-packages (from -r requirements.txt (line 5)) (4.12.2)" # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { "${pip_install_command[@]}" \ @@ -116,7 +118,7 @@ function pip::install_dependencies() { --progress-bar off \ --src='/app/.heroku/python/src' \ |& tee "${WARNINGS_LOG:?}" \ - |& sed --unbuffered --expression '/Requirement already satisfied/d' \ + |& sed --unbuffered --expression 's# in /app/.heroku/python/lib/python.*/site-packages##' \ |& output::indent }; then # TODO: Overhaul warnings and combine them with error handling. diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index aa4f58dd2..5ff6041be 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -62,16 +62,19 @@ REGEX test_run.run_again - expect(clean_output(test_run.output)).to include(<<~OUTPUT) + expect(clean_output(test_run.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Restoring cache -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION} -----> Installing dependencies using 'pip install -r requirements.txt -r requirements-test.txt' + Requirement already satisfied: typing-extensions==.+ + Requirement already satisfied: pytest==.+ + .+ -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running bin/post_compile hook - OUTPUT + REGEX end end end diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index f27f713c4..5cc9f8b8e 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -53,6 +53,7 @@ remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' + remote: Requirement already satisfied: typing-extensions==4.12.2 (from -r requirements.txt (line 5)) (4.12.2) remote: -----> Inline app detected OUTPUT end From 20cd0b69e571d2a720ba2fd59d383294a5236435 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 25 Feb 2025 22:37:39 +0000 Subject: [PATCH 112/406] Bump poetry from 2.0.1 to 2.1.1 (#1758) * Bump poetry from 2.0.1 to 2.1.1 Bumps [poetry](https://github.com/python-poetry/poetry) from 2.0.1 to 2.1.1. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](https://github.com/python-poetry/poetry/compare/2.0.1...2.1.1) --- updated-dependencies: - dependency-name: poetry dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Set `POETRY_VIRTUALENVS_USE_POETRY_PYTHON` Since in theory it should force Poetry to not search for other Python versions if the project's `requires-python` (or `tool.poetry` equivalent) doesn't match the Python version we installed. * Add changelog entry * Update one fixture to PEP-621 style `pyproject.toml` So we have fixtures testing both the new and old style config. --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + lib/poetry.sh | 4 ++++ requirements/poetry.txt | 2 +- spec/fixtures/poetry_basic/poetry.lock | 15 +++++++++++---- spec/fixtures/poetry_basic/pyproject.toml | 14 +++++++++----- spec/hatchet/ci_spec.rb | 2 ++ spec/hatchet/poetry_spec.rb | 5 +++-- 7 files changed, 31 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b2b73c3e..29c88e5c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated Poetry from 2.0.1 to 2.1.1. ([#1758](https://github.com/heroku/heroku-buildpack-python/pull/1758)) - Stopped filtering out pip's `Requirement already satisfied:` log lines when installing dependencies. ([#1765](https://github.com/heroku/heroku-buildpack-python/pull/1765)) - Improved the error messages shown if installing pip/Poetry/Pipenv fails. ([#1764](https://github.com/heroku/heroku-buildpack-python/pull/1764)) - Stopped installing pip into Poetry's virtual environment. ([#1761](https://github.com/heroku/heroku-buildpack-python/pull/1761)) diff --git a/lib/poetry.sh b/lib/poetry.sh index 6b4888eea..a453356a1 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -101,11 +101,15 @@ function poetry::install_poetry() { export PATH="${poetry_bin_dir}:${PATH}" # Force Poetry to manage the system Python site-packages instead of using venvs. export POETRY_VIRTUALENVS_CREATE="false" + # Force Poetry to use our Python rather than scanning PATH (which might pick system Python). + # Though this currently doesn't work as documented: https://github.com/python-poetry/poetry/issues/10226 + export POETRY_VIRTUALENVS_USE_POETRY_PYTHON="true" # Set the same env vars in the environment used by later buildpacks. cat >>"${export_file}" <<-EOF export PATH="${poetry_bin_dir}:\${PATH}" export POETRY_VIRTUALENVS_CREATE="false" + export POETRY_VIRTUALENVS_USE_POETRY_PYTHON="true" EOF } diff --git a/requirements/poetry.txt b/requirements/poetry.txt index d1c8013e2..f71727796 100644 --- a/requirements/poetry.txt +++ b/requirements/poetry.txt @@ -1 +1 @@ -poetry==2.0.1 +poetry==2.1.1 diff --git a/spec/fixtures/poetry_basic/poetry.lock b/spec/fixtures/poetry_basic/poetry.lock index 7fe5d0794..166f8fea8 100644 --- a/spec/fixtures/poetry_basic/poetry.lock +++ b/spec/fixtures/poetry_basic/poetry.lock @@ -1,4 +1,4 @@ -# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand. +# This file is automatically @generated by Poetry 2.1.1 and should not be changed by hand. [[package]] name = "colorama" @@ -6,6 +6,8 @@ version = "0.4.6" description = "Cross-platform colored terminal text." optional = false python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,>=2.7" +groups = ["dev"] +markers = "sys_platform == \"win32\"" files = [ {file = "colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"}, {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"}, @@ -17,6 +19,7 @@ version = "2.0.0" description = "brain-dead simple config-ini parsing" optional = false python-versions = ">=3.7" +groups = ["dev"] files = [ {file = "iniconfig-2.0.0-py3-none-any.whl", hash = "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374"}, {file = "iniconfig-2.0.0.tar.gz", hash = "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3"}, @@ -28,6 +31,7 @@ version = "24.2" description = "Core utilities for Python packages" optional = false python-versions = ">=3.8" +groups = ["dev"] files = [ {file = "packaging-24.2-py3-none-any.whl", hash = "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759"}, {file = "packaging-24.2.tar.gz", hash = "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f"}, @@ -39,6 +43,7 @@ version = "1.5.0" description = "plugin and hook calling mechanisms for python" optional = false python-versions = ">=3.8" +groups = ["dev"] files = [ {file = "pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669"}, {file = "pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1"}, @@ -54,6 +59,7 @@ version = "8.3.4" description = "pytest: simple powerful testing with Python" optional = false python-versions = ">=3.8" +groups = ["dev"] files = [ {file = "pytest-8.3.4-py3-none-any.whl", hash = "sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6"}, {file = "pytest-8.3.4.tar.gz", hash = "sha256:965370d062bce11e73868e0335abac31b4d3de0e82f4007408d242b4f8610761"}, @@ -74,12 +80,13 @@ version = "4.12.2" description = "Backported and Experimental Type Hints for Python 3.8+" optional = false python-versions = ">=3.8" +groups = ["main"] files = [ {file = "typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d"}, {file = "typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"}, ] [metadata] -lock-version = "2.0" -python-versions = "^3.13" -content-hash = "56ec6342f4a39b402d2b65bd08cfb3af4136a02bb58674df49eac8d317376a7f" +lock-version = "2.1" +python-versions = ">=3.13" +content-hash = "8e82be2391208aa1b8599893f6abec30c0aa5702a846842fd442f50de45481e2" diff --git a/spec/fixtures/poetry_basic/pyproject.toml b/spec/fixtures/poetry_basic/pyproject.toml index 9fc0e8356..85bd48b9d 100644 --- a/spec/fixtures/poetry_basic/pyproject.toml +++ b/spec/fixtures/poetry_basic/pyproject.toml @@ -1,10 +1,14 @@ +[project] +name = "poetry-basic" +version = "0.0.0" +requires-python = ">=3.13" +dependencies = [ + "typing-extensions" +] + [tool.poetry] package-mode = false -[tool.poetry.dependencies] -python = "^3.13" -typing-extensions = "*" - # This group shouldn't be installed due to us passing `--only main`. -[tool.poetry.group.test.dependencies] +[tool.poetry.group.dev.dependencies] pytest = "*" diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 5ff6041be..f11e37a23 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -189,6 +189,7 @@ PIP_NO_PYTHON_VERSION_WARNING=1 PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config POETRY_VIRTUALENVS_CREATE=false + POETRY_VIRTUALENVS_USE_POETRY_PYTHON=true PYTHONUNBUFFERED=1 -----> Inline app detected LANG=en_US.UTF-8 @@ -196,6 +197,7 @@ LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/tmp/cache.+/.heroku/python-poetry/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ POETRY_VIRTUALENVS_CREATE=false + POETRY_VIRTUALENVS_USE_POETRY_PYTHON=true PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python PYTHONPATH=/app diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 874715f00..9d8d6c4ac 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -26,6 +26,7 @@ remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin:/tmp/codon/tmp/cache/.heroku/python-poetry/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: POETRY_VIRTUALENVS_CREATE=false + remote: POETRY_VIRTUALENVS_USE_POETRY_PYTHON=true remote: PYTHONHASHSEED=random remote: PYTHONHOME=/app/.heroku/python remote: PYTHONPATH=/app @@ -62,7 +63,7 @@ end context 'when the Poetry and Python versions have changed since the last build' do - let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v274'] } + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v275'] } let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_basic', buildpacks:) } it 'clears the cache before installing' do @@ -75,7 +76,7 @@ remote: -----> Using Python 3.13 specified in .python-version remote: -----> Discarding cache since: remote: - The Python version has changed from 3.13.1 to #{LATEST_PYTHON_3_13} - remote: - The Poetry version has changed from 1.8.5 to #{POETRY_VERSION} + remote: - The Poetry version has changed from 2.0.1 to #{POETRY_VERSION} remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing Poetry #{POETRY_VERSION} remote: -----> Installing dependencies using 'poetry sync --only main' From a85e52c7f1cae4262d51f0769e6abc7139df6082 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 26 Feb 2025 08:18:29 +0000 Subject: [PATCH 113/406] Prepare release v279 (#1766) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 29c88e5c7..108387e76 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v279] - 2025-02-26 + - Updated Poetry from 2.0.1 to 2.1.1. ([#1758](https://github.com/heroku/heroku-buildpack-python/pull/1758)) - Stopped filtering out pip's `Requirement already satisfied:` log lines when installing dependencies. ([#1765](https://github.com/heroku/heroku-buildpack-python/pull/1765)) - Improved the error messages shown if installing pip/Poetry/Pipenv fails. ([#1764](https://github.com/heroku/heroku-buildpack-python/pull/1764)) @@ -1177,7 +1180,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v278...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v279...main +[v279]: https://github.com/heroku/heroku-buildpack-python/compare/v278...v279 [v278]: https://github.com/heroku/heroku-buildpack-python/compare/v277...v278 [v277]: https://github.com/heroku/heroku-buildpack-python/compare/v276...v277 [v276]: https://github.com/heroku/heroku-buildpack-python/compare/v275...v276 From 7c5d82043b6926b8d675f1e9aa04c29edeb1f1ab Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 13 Mar 2025 10:27:04 +0000 Subject: [PATCH 114/406] Bump the ruby-dependencies group across 1 directory with 2 updates (#1770) * Bump the ruby-dependencies group across 1 directory with 2 updates Bumps the ruby-dependencies group with 2 updates in the / directory: [heroku_hatchet](https://github.com/heroku/hatchet) and [rubocop](https://github.com/rubocop/rubocop). Updates `heroku_hatchet` from 8.0.4 to 8.0.5 - [Changelog](https://github.com/heroku/hatchet/blob/main/CHANGELOG.md) - [Commits](https://github.com/heroku/hatchet/compare/v8.0.4...v8.0.5) Updates `rubocop` from 1.72.2 to 1.74.0 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.72.2...v1.74.0) --- updated-dependencies: - dependency-name: heroku_hatchet dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rubocop dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] * Remove logger workaround + refresh lockfile --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- Gemfile | 2 -- Gemfile.lock | 22 +++++++++++----------- 2 files changed, 11 insertions(+), 13 deletions(-) diff --git a/Gemfile b/Gemfile index 215de3b19..2abf2b7cb 100644 --- a/Gemfile +++ b/Gemfile @@ -6,8 +6,6 @@ ruby '>= 3.2', '< 3.5' group :test, :development do gem 'heroku_hatchet' - # Work around https://github.com/excon/excon/issues/860 - gem 'logger' gem 'parallel_split_test' gem 'rspec-core' gem 'rspec-expectations' diff --git a/Gemfile.lock b/Gemfile.lock index efa4e8d6b..22b12ea29 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -3,9 +3,10 @@ GEM specs: ast (2.4.2) base64 (0.2.0) - diff-lcs (1.5.1) + diff-lcs (1.6.0) erubis (2.7.0) - excon (0.112.0) + excon (1.2.5) + logger heroics (0.1.3) base64 erubis (~> 2.0) @@ -13,13 +14,13 @@ GEM moneta multi_json (>= 1.9.2) webrick - heroku_hatchet (8.0.4) - excon (~> 0) + heroku_hatchet (8.0.5) + excon (< 2) platform-api (~> 3) rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.10.1) + json (2.10.2) language_server-protocol (3.17.0.4) lint_roller (1.1.0) logger (1.6.6) @@ -32,7 +33,7 @@ GEM parser (3.3.7.1) ast (~> 2.4.1) racc - platform-api (3.7.0) + platform-api (3.8.0) heroics (~> 0.1.1) moneta (~> 1.0.0) rate_throttle_client (~> 0.1.0) @@ -49,7 +50,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.2) - rubocop (1.72.2) + rubocop (1.74.0) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) @@ -60,7 +61,7 @@ GEM rubocop-ast (>= 1.38.0, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 4.0) - rubocop-ast (1.38.0) + rubocop-ast (1.38.1) parser (>= 3.3.1.0) rubocop-rspec (3.5.0) lint_roller (~> 1.1) @@ -78,7 +79,6 @@ PLATFORMS DEPENDENCIES heroku_hatchet - logger parallel_split_test rspec-core rspec-expectations @@ -87,7 +87,7 @@ DEPENDENCIES rubocop-rspec RUBY VERSION - ruby 3.4.1p0 + ruby 3.4.2p28 BUNDLED WITH - 2.6.2 + 2.6.3 From 49b1639b7be8acc9ca6d171db2d77b0d7659028d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 2 Apr 2025 13:59:14 +0100 Subject: [PATCH 115/406] Bump rubocop from 1.74.0 to 1.75.1 in the ruby-dependencies group (#1771) Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.74.0 to 1.75.1 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.74.0...v1.75.1) --- updated-dependencies: - dependency-name: rubocop dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 22b12ea29..d633d680f 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,7 +1,7 @@ GEM remote: https://rubygems.org/ specs: - ast (2.4.2) + ast (2.4.3) base64 (0.2.0) diff-lcs (1.6.0) erubis (2.7.0) @@ -30,13 +30,14 @@ GEM parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) - parser (3.3.7.1) + parser (3.3.7.4) ast (~> 2.4.1) racc platform-api (3.8.0) heroics (~> 0.1.1) moneta (~> 1.0.0) rate_throttle_client (~> 0.1.0) + prism (1.4.0) racc (1.8.1) rainbow (3.1.1) rate_throttle_client (0.1.2) @@ -50,7 +51,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.2) - rubocop (1.74.0) + rubocop (1.75.1) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) @@ -58,11 +59,12 @@ GEM parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) regexp_parser (>= 2.9.3, < 3.0) - rubocop-ast (>= 1.38.0, < 2.0) + rubocop-ast (>= 1.43.0, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 4.0) - rubocop-ast (1.38.1) - parser (>= 3.3.1.0) + rubocop-ast (1.43.0) + parser (>= 3.3.7.2) + prism (~> 1.4) rubocop-rspec (3.5.0) lint_roller (~> 1.1) rubocop (~> 1.72, >= 1.72.1) From 41dcf88044d20ff5a7dd68955d740c925ddb8dc9 Mon Sep 17 00:00:00 2001 From: Pablo Temporini Date: Thu, 3 Apr 2025 11:05:11 -0300 Subject: [PATCH 116/406] Update repo metadata (#1773) Signed-off-by: Pablo Temporini --- .github/CODEOWNERS | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 52082f8a2..33be3e324 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,4 +1,6 @@ # Default to requesting pull request reviews from the Heroku Languages team. +#ECCN:Open Source +#GUSINFO:Languages,Heroku Python Platform * @heroku/languages # However, request review from the language owner instead for files that are updated From 51ec7f4b810b7af1aef86cedc884a0e281d408f1 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 8 Apr 2025 09:37:41 +0000 Subject: [PATCH 117/406] Bump pip from 24.3.1 to 25.0.1 (#1759) * Bump pip from 24.3.1 to 25.0.1 Bumps [pip](https://github.com/pypa/pip) from 24.3.1 to 25.0.1. - [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst) - [Commits](https://github.com/pypa/pip/compare/24.3.1...25.0.1) --- updated-dependencies: - dependency-name: pip dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] * Stop setting `PIP_NO_PYTHON_VERSION_WARNING=1` Since that option is a no-op, and now causes a deprecation warning: ``` DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. ``` * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + bin/compile | 4 ---- lib/pip.sh | 1 + requirements/pip.txt | 2 +- spec/hatchet/ci_spec.rb | 3 --- spec/hatchet/hooks_spec.rb | 2 -- 6 files changed, 3 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 108387e76..f5f4b496d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated pip from 24.3.1 to 25.0.1. ([#1759](https://github.com/heroku/heroku-buildpack-python/pull/1759)) ## [v279] - 2025-02-26 diff --git a/bin/compile b/bin/compile index cda662380..b75752b91 100755 --- a/bin/compile +++ b/bin/compile @@ -94,10 +94,6 @@ export LIBRARY_PATH="/app/.heroku/python/lib${LIBRARY_PATH:+:${LIBRARY_PATH}}" export LD_LIBRARY_PATH="/app/.heroku/python/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" export PKG_CONFIG_PATH="/app/.heroku/python/lib/pkg-config${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}" -# Global pip options (https://pip.pypa.io/en/stable/user_guide/#environment-variables). -# Disable pip's warnings about EOL Python since we show our own. -export PIP_NO_PYTHON_VERSION_WARNING=1 - cd "$BUILD_DIR" # Runs a `bin/pre_compile` script if found in the app source, allowing build customisation. diff --git a/lib/pip.sh b/lib/pip.sh index e56e6d536..ca95abfd6 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -106,6 +106,7 @@ function pip::install_dependencies() { # We only display the most relevant command args here, to improve the signal to noise ratio. output::step "Installing dependencies using '${pip_install_command[*]}'" + # TODO: Remove --disable-pip-version-check in favour of exporting PIP_DISABLE_PIP_VERSION_CHECK. # The sed usage is to reduce the verbosity of output lines like: # "Requirement already satisfied: typing-extensions==4.12.2 in /app/.heroku/python/lib/python3.13/site-packages (from -r requirements.txt (line 5)) (4.12.2)" # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. diff --git a/requirements/pip.txt b/requirements/pip.txt index 662f25f21..059862d50 100644 --- a/requirements/pip.txt +++ b/requirements/pip.txt @@ -1 +1 @@ -pip==24.3.1 +pip==25.0.1 diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index f11e37a23..50d722c51 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -30,7 +30,6 @@ LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ - PIP_NO_PYTHON_VERSION_WARNING=1 PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 -----> Inline app detected @@ -105,7 +104,6 @@ LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ - PIP_NO_PYTHON_VERSION_WARNING=1 PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 -----> Inline app detected @@ -186,7 +184,6 @@ LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib PATH=/tmp/cache.+/.heroku/python-poetry/bin:/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ - PIP_NO_PYTHON_VERSION_WARNING=1 PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config POETRY_VIRTUALENVS_CREATE=false POETRY_VIRTUALENVS_USE_POETRY_PYTHON=true diff --git a/spec/hatchet/hooks_spec.rb b/spec/hatchet/hooks_spec.rb index 1f6af134e..eb20a7e83 100644 --- a/spec/hatchet/hooks_spec.rb +++ b/spec/hatchet/hooks_spec.rb @@ -24,7 +24,6 @@ remote: LD_LIBRARY_PATH=/app/.heroku/python/lib remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - remote: PIP_NO_PYTHON_VERSION_WARNING=1 remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config remote: PWD=/tmp/build_ remote: PYTHONUNBUFFERED=1 @@ -48,7 +47,6 @@ remote: LD_LIBRARY_PATH=/app/.heroku/python/lib remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - remote: PIP_NO_PYTHON_VERSION_WARNING=1 remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config remote: PWD=/tmp/build_ remote: PYTHONUNBUFFERED=1 From 627847a165c70df114ac733f25a2bf33acf44473 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 8 Apr 2025 11:01:45 +0100 Subject: [PATCH 118/406] Bump poetry from 2.1.1 to 2.1.2 (#1772) * Bump poetry from 2.1.1 to 2.1.2 Bumps [poetry](https://github.com/python-poetry/poetry) from 2.1.1 to 2.1.2. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](https://github.com/python-poetry/poetry/compare/2.1.1...2.1.2) --- updated-dependencies: - dependency-name: poetry dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/poetry.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f5f4b496d..c7e341e82 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Updated pip from 24.3.1 to 25.0.1. ([#1759](https://github.com/heroku/heroku-buildpack-python/pull/1759)) +- Updated Poetry from 2.1.1 to 2.1.2. ([#1772](https://github.com/heroku/heroku-buildpack-python/pull/1772)) ## [v279] - 2025-02-26 diff --git a/requirements/poetry.txt b/requirements/poetry.txt index f71727796..7fc861c0f 100644 --- a/requirements/poetry.txt +++ b/requirements/poetry.txt @@ -1 +1 @@ -poetry==2.1.1 +poetry==2.1.2 From 76a8152a700f23bf6bda713bc6f6b2be4658819c Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 8 Apr 2025 10:05:55 +0000 Subject: [PATCH 119/406] Prepare release v280 (#1774) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c7e341e82..f35671877 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v280] - 2025-04-08 + - Updated pip from 24.3.1 to 25.0.1. ([#1759](https://github.com/heroku/heroku-buildpack-python/pull/1759)) - Updated Poetry from 2.1.1 to 2.1.2. ([#1772](https://github.com/heroku/heroku-buildpack-python/pull/1772)) @@ -1182,7 +1185,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v279...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v280...main +[v280]: https://github.com/heroku/heroku-buildpack-python/compare/v279...v280 [v279]: https://github.com/heroku/heroku-buildpack-python/compare/v278...v279 [v278]: https://github.com/heroku/heroku-buildpack-python/compare/v277...v278 [v277]: https://github.com/heroku/heroku-buildpack-python/compare/v276...v277 From 3ee5020fb061b3375785db12ddae60c1a187c613 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 9 Apr 2025 00:12:18 +0100 Subject: [PATCH 120/406] Add support for Python 3.13.3, 3.12.10, 3.11.12, 3.10.17 and 3.9.22 (#1775) Release announcement: https://blog.python.org/2025/04/python-3140a7-3133-31210-31112-31017.html Changelogs: https://docs.python.org/release/3.13.3/whatsnew/changelog.html#python-3-13-3-final https://docs.python.org/release/3.12.10/whatsnew/changelog.html#python-3-12-10-final https://docs.python.org/release/3.11.12/whatsnew/changelog.html#python-3-11-12-final https://docs.python.org/release/3.10.17/whatsnew/changelog.html#python-3-10-17-final https://docs.python.org/release/3.9.22/whatsnew/changelog.html#python-3-9-22-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/14341087204 https://github.com/heroku/heroku-buildpack-python/actions/runs/14341085641 https://github.com/heroku/heroku-buildpack-python/actions/runs/14341083031 https://github.com/heroku/heroku-buildpack-python/actions/runs/14341080059 https://github.com/heroku/heroku-buildpack-python/actions/runs/14344846015 GUS-W-17604324. GUS-W-17604386. GUS-W-18229136. --- CHANGELOG.md | 5 +++++ lib/python_version.sh | 10 +++++----- spec/spec_helper.rb | 10 +++++----- 3 files changed, 15 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f35671877..946a97a9c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ ## [Unreleased] +- The Python 3.13 version alias now resolves to Python 3.13.3. ([#1775](https://github.com/heroku/heroku-buildpack-python/pull/1775)) +- The Python 3.12 version alias now resolves to Python 3.12.10. ([#1775](https://github.com/heroku/heroku-buildpack-python/pull/1775)) +- The Python 3.11 version alias now resolves to Python 3.11.12. ([#1775](https://github.com/heroku/heroku-buildpack-python/pull/1775)) +- The Python 3.10 version alias now resolves to Python 3.10.17. ([#1775](https://github.com/heroku/heroku-buildpack-python/pull/1775)) +- The Python 3.9 version alias now resolves to Python 3.9.22. ([#1775](https://github.com/heroku/heroku-buildpack-python/pull/1775)) ## [v280] - 2025-04-08 diff --git a/lib/python_version.sh b/lib/python_version.sh index 4405b7650..02735b3b5 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -4,11 +4,11 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail -LATEST_PYTHON_3_9="3.9.21" -LATEST_PYTHON_3_10="3.10.16" -LATEST_PYTHON_3_11="3.11.11" -LATEST_PYTHON_3_12="3.12.9" -LATEST_PYTHON_3_13="3.13.2" +LATEST_PYTHON_3_9="3.9.22" +LATEST_PYTHON_3_10="3.10.17" +LATEST_PYTHON_3_11="3.11.12" +LATEST_PYTHON_3_12="3.12.10" +LATEST_PYTHON_3_13="3.13.3" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=9 NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=13 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index b8e49eb22..2f561389c 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -9,11 +9,11 @@ FIXTURE_DIR = Pathname.new(__FILE__).parent.join('fixtures') -LATEST_PYTHON_3_9 = '3.9.21' -LATEST_PYTHON_3_10 = '3.10.16' -LATEST_PYTHON_3_11 = '3.11.11' -LATEST_PYTHON_3_12 = '3.12.9' -LATEST_PYTHON_3_13 = '3.13.2' +LATEST_PYTHON_3_9 = '3.9.22' +LATEST_PYTHON_3_10 = '3.10.17' +LATEST_PYTHON_3_11 = '3.11.12' +LATEST_PYTHON_3_12 = '3.12.10' +LATEST_PYTHON_3_13 = '3.13.3' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_13 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From b2966193afbc86d88b0ee694ffbb33cba74fc23c Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 8 Apr 2025 23:20:42 +0000 Subject: [PATCH 121/406] Prepare release v281 (#1776) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 946a97a9c..d45887f0e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v281] - 2025-04-08 + - The Python 3.13 version alias now resolves to Python 3.13.3. ([#1775](https://github.com/heroku/heroku-buildpack-python/pull/1775)) - The Python 3.12 version alias now resolves to Python 3.12.10. ([#1775](https://github.com/heroku/heroku-buildpack-python/pull/1775)) - The Python 3.11 version alias now resolves to Python 3.11.12. ([#1775](https://github.com/heroku/heroku-buildpack-python/pull/1775)) @@ -1190,7 +1193,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v280...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v281...main +[v281]: https://github.com/heroku/heroku-buildpack-python/compare/v280...v281 [v280]: https://github.com/heroku/heroku-buildpack-python/compare/v279...v280 [v279]: https://github.com/heroku/heroku-buildpack-python/compare/v278...v279 [v278]: https://github.com/heroku/heroku-buildpack-python/compare/v277...v278 From ba7a7e3ee96ad46eaa622742a80e4f92217715db Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 1 May 2025 17:07:48 +0100 Subject: [PATCH 122/406] Remove support for Heroku-20 (#1778) Since the Heroku-20 stack has reached end-of-life, and as such builds using it are no longer supported by the Heroku build system: https://devcenter.heroku.com/changelog-items/3230 This fixes the integration tests failing in CI for the Heroku-20 stack, due to the build system now (as expected) rejecting the jobs. Any non-Heroku consumers of this buildpack that wish to continue using the Heroku-20 stack should pin to the previous version of this buildpack. GUS-W-14707057. --- .github/workflows/build_python_runtime.yml | 22 ---------------------- .github/workflows/ci.yml | 2 +- CHANGELOG.md | 1 + builds/build_python_runtime.sh | 2 +- lib/checks.sh | 4 ++-- lib/python.sh | 2 +- 6 files changed, 6 insertions(+), 27 deletions(-) diff --git a/.github/workflows/build_python_runtime.yml b/.github/workflows/build_python_runtime.yml index 3e8d5dc73..ad6b05a98 100644 --- a/.github/workflows/build_python_runtime.yml +++ b/.github/workflows/build_python_runtime.yml @@ -13,7 +13,6 @@ on: type: choice options: - auto - - heroku-20 - heroku-22 - heroku-24 default: auto @@ -36,27 +35,6 @@ env: # Unfortunately these jobs cannot be easily written as a matrix since `matrix.exclude` does not # support expression syntax, and the `matrix` context is not available inside the job `if` key. jobs: - heroku-20: - if: inputs.stack == 'heroku-20' || inputs.stack == 'auto' - runs-on: pub-hk-ubuntu-24.04-xlarge - env: - STACK_VERSION: "20" - steps: - - name: Checkout - uses: actions/checkout@v4 - - name: Build Docker image - run: docker build --platform="linux/amd64" --pull --tag buildenv --build-arg=STACK_VERSION builds/ - - name: Compile and package Python runtime - run: docker run --rm --volume="${PWD}/upload:/tmp/upload" buildenv ./build_python_runtime.sh "${{ inputs.python_version }}" - - name: Test Python runtime - run: | - RUN_IMAGE='heroku/heroku:${{ env.STACK_VERSION }}' - ARCHIVE_FILENAME='python-${{ inputs.python_version }}-ubuntu-${{ env.STACK_VERSION }}.04-amd64.tar.zst' - docker run --rm --volume="${PWD}/upload:/upload:ro" --volume="${PWD}/builds:/builds:ro" "${RUN_IMAGE}" /builds/test_python_runtime.sh "/upload/${ARCHIVE_FILENAME}" - - name: Upload Python runtime archive to S3 - if: (!inputs.dry_run) - run: aws s3 sync ./upload "s3://${S3_BUCKET}" - heroku-22: if: inputs.stack == 'heroku-22' || inputs.stack == 'auto' runs-on: pub-hk-ubuntu-24.04-xlarge diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 56224eeb1..c14d400f6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -39,7 +39,7 @@ jobs: strategy: fail-fast: false matrix: - stack: ["heroku-20", "heroku-22", "heroku-24"] + stack: ["heroku-22", "heroku-24"] env: HATCHET_APP_LIMIT: 300 HATCHET_DEFAULT_STACK: ${{ matrix.stack }} diff --git a/CHANGELOG.md b/CHANGELOG.md index d45887f0e..abc9f8620 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Removed support for Heroku-20. ([#1778](https://github.com/heroku/heroku-buildpack-python/pull/1778)) ## [v281] - 2025-04-08 diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index 0f442c6ff..197738295 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -21,7 +21,7 @@ function abort() { } case "${STACK:?}" in - heroku-20 | heroku-22 | heroku-24) + heroku-22 | heroku-24) SUPPORTED_PYTHON_VERSIONS=( "3.9" "3.10" diff --git a/lib/checks.sh b/lib/checks.sh index 3a43cdad7..42f8831d2 100644 --- a/lib/checks.sh +++ b/lib/checks.sh @@ -4,7 +4,7 @@ function checks::ensure_supported_stack() { local stack="${1}" case "${stack}" in - heroku-20 | heroku-22 | heroku-24) + heroku-22 | heroku-24) return 0 ;; cedar* | heroku-16 | heroku-18) @@ -15,7 +15,7 @@ function checks::ensure_supported_stack() { This buildpack no longer supports the '${stack}' stack since it has reached its end-of-life: - https://devcenter.heroku.com/articles/stack#stack-support-details-for-apps-using-classic-buildpacks + https://devcenter.heroku.com/articles/stack#stack-support-details Upgrade to a newer stack to continue using this buildpack. EOF diff --git a/lib/python.sh b/lib/python.sh index 4bbeb344a..32733730e 100644 --- a/lib/python.sh +++ b/lib/python.sh @@ -77,7 +77,7 @@ function python::install() { # decompression/extraction as separate steps), so can't write to stdout. # 2. We want to display the original stderr to the user, so can't write to stderr. # 3. Curl's `--write-out` feature only supports outputting to a file (as opposed to - # stdout/stderr) as of curl v8.3.0, which is newer than the curl on Heroku-20/22. + # stdout/stderr) as of curl v8.3.0, which is newer than the curl on Heroku-22. # This has an integration test run against all stacks, which will mean we will know # if future versions of curl change the error message string. # From ce2d1e09dc4eada1a283e7e637ed70bb3c8a2cd0 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 1 May 2025 17:42:12 +0100 Subject: [PATCH 123/406] Bump the ruby-dependencies group with 3 updates (#1781) * Bump the ruby-dependencies group with 3 updates Bumps the ruby-dependencies group with 3 updates: [rspec-expectations](https://github.com/rspec/rspec-expectations), [rubocop](https://github.com/rubocop/rubocop) and [rubocop-rspec](https://github.com/rubocop/rubocop-rspec). Updates `rspec-expectations` from 3.13.3 to 3.13.4 - [Release notes](https://github.com/rspec/rspec-expectations/releases) - [Changelog](https://github.com/rspec/rspec-expectations/blob/main/Changelog.md) - [Commits](https://github.com/rspec/rspec-expectations/commits) Updates `rubocop` from 1.75.1 to 1.75.4 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.75.1...v1.75.4) Updates `rubocop-rspec` from 3.5.0 to 3.6.0 - [Release notes](https://github.com/rubocop/rubocop-rspec/releases) - [Changelog](https://github.com/rubocop/rubocop-rspec/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop-rspec/compare/v3.5.0...v3.6.0) --- updated-dependencies: - dependency-name: rspec-expectations dependency-version: 3.13.4 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rubocop dependency-version: 1.75.4 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rubocop-rspec dependency-version: 3.6.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] * Refresh lockfile * Fix RSpec lint errors --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- Gemfile.lock | 26 +++++++++++++------------- spec/hatchet/python_version_spec.rb | 8 ++++---- 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index d633d680f..e7bef30a0 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -3,7 +3,7 @@ GEM specs: ast (2.4.3) base64 (0.2.0) - diff-lcs (1.6.0) + diff-lcs (1.6.1) erubis (2.7.0) excon (1.2.5) logger @@ -20,17 +20,17 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.10.2) + json (2.11.3) language_server-protocol (3.17.0.4) lint_roller (1.1.0) - logger (1.6.6) + logger (1.7.0) moneta (1.0.0) multi_json (1.15.0) - parallel (1.26.3) + parallel (1.27.0) parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) - parser (3.3.7.4) + parser (3.3.8.0) ast (~> 2.4.1) racc platform-api (3.8.0) @@ -45,13 +45,13 @@ GEM rrrretry (1.0.0) rspec-core (3.13.3) rspec-support (~> 3.13.0) - rspec-expectations (3.13.3) + rspec-expectations (3.13.4) diff-lcs (>= 1.2.0, < 2.0) rspec-support (~> 3.13.0) rspec-retry (0.6.2) rspec-core (> 3.3) - rspec-support (3.13.2) - rubocop (1.75.1) + rspec-support (3.13.3) + rubocop (1.75.4) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) @@ -59,13 +59,13 @@ GEM parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) regexp_parser (>= 2.9.3, < 3.0) - rubocop-ast (>= 1.43.0, < 2.0) + rubocop-ast (>= 1.44.0, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 4.0) - rubocop-ast (1.43.0) + rubocop-ast (1.44.1) parser (>= 3.3.7.2) prism (~> 1.4) - rubocop-rspec (3.5.0) + rubocop-rspec (3.6.0) lint_roller (~> 1.1) rubocop (~> 1.72, >= 1.72.1) ruby-progressbar (1.13.0) @@ -89,7 +89,7 @@ DEPENDENCIES rubocop-rspec RUBY VERSION - ruby 3.4.2p28 + ruby 3.4.3p32 BUNDLED WITH - 2.6.3 + 2.6.8 diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 9e2452e59..175ab3e6f 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -117,25 +117,25 @@ context 'when .python-version contains Python 3.10' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.10') } - include_examples 'builds with the requested Python version', '3.10', LATEST_PYTHON_3_10 + it_behaves_like 'builds with the requested Python version', '3.10', LATEST_PYTHON_3_10 end context 'when .python-version contains Python 3.11' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.11') } - include_examples 'builds with the requested Python version', '3.11', LATEST_PYTHON_3_11 + it_behaves_like 'builds with the requested Python version', '3.11', LATEST_PYTHON_3_11 end context 'when .python-version contains Python 3.12' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.12') } - include_examples 'builds with the requested Python version', '3.12', LATEST_PYTHON_3_12 + it_behaves_like 'builds with the requested Python version', '3.12', LATEST_PYTHON_3_12 end context 'when .python-version contains Python 3.13' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.13') } - include_examples 'builds with the requested Python version', '3.13', LATEST_PYTHON_3_13 + it_behaves_like 'builds with the requested Python version', '3.13', LATEST_PYTHON_3_13 end context 'when .python-version contains an invalid Python version string' do From 3e99fc6ba795dc557d5e6a4f7e9ac08a139b620f Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Fri, 2 May 2025 10:33:33 +0000 Subject: [PATCH 124/406] Prepare release v282 (#1782) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index abc9f8620..e4ee913fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v282] - 2025-05-02 + - Removed support for Heroku-20. ([#1778](https://github.com/heroku/heroku-buildpack-python/pull/1778)) ## [v281] - 2025-04-08 @@ -1194,7 +1197,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v281...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v282...main +[v282]: https://github.com/heroku/heroku-buildpack-python/compare/v281...v282 [v281]: https://github.com/heroku/heroku-buildpack-python/compare/v280...v281 [v280]: https://github.com/heroku/heroku-buildpack-python/compare/v279...v280 [v279]: https://github.com/heroku/heroku-buildpack-python/compare/v278...v279 From 6fd807973a5bbf3830c6b758f501f5a0f3c8c1ee Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 2 May 2025 16:36:17 +0100 Subject: [PATCH 125/406] Backport error message/build output improvements from CNB (#1783) Backports the error message/build output improvements made as part of: - https://github.com/heroku/buildpacks-python/pull/352 - https://github.com/heroku/buildpacks-python/pull/353 - https://github.com/heroku/buildpacks-python/pull/354 - https://github.com/heroku/buildpacks-python/pull/355 Plus: - applies similar changes to equivalent error messages that only exist in the classic buildpack (eg the Pipenv errors) - switches to use of contractions as per the CX team's style guidelines GUS-W-18225347. GUS-W-18421778. --- CHANGELOG.md | 4 + bin/compile | 17 ++-- bin/steps/nltk | 2 +- lib/checks.sh | 12 +-- lib/package_manager.sh | 4 +- lib/pip.sh | 2 +- lib/pipenv.sh | 2 +- lib/poetry.sh | 2 +- lib/python.sh | 2 +- lib/python_version.sh | 90 ++++++++++++------- lib/utils.sh | 6 +- .../.python-version | 3 +- .../runtime_txt_invalid_version/runtime.txt | 2 +- spec/hatchet/checks_spec.rb | 12 +-- spec/hatchet/nltk_spec.rb | 2 +- spec/hatchet/pipenv_spec.rb | 35 +++++--- spec/hatchet/poetry_spec.rb | 2 +- spec/hatchet/python_update_warning_spec.rb | 21 +++-- spec/hatchet/python_version_spec.rb | 78 ++++++++++------ 19 files changed, 185 insertions(+), 113 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e4ee913fa..679794547 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## [Unreleased] +- Improved the instructions for migrating from `runtime.txt` to `.python-version`. ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783)) +- Improved the error message instructions shown when `.python-version`, `runtime.txt` or `Pipfile.lock` contain an invalid Python version. ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783)) +- Improved the rendering of the error message shown if `.python-version` or `runtime.txt` contain stray invisible characters (such as ASCII control codes). ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783)) +- Improved the upgrade instructions shown for EOL and unsupported Python versions. ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783)) ## [v282] - 2025-05-02 diff --git a/bin/compile b/bin/compile index b75752b91..d4e67729c 100755 --- a/bin/compile +++ b/bin/compile @@ -152,15 +152,20 @@ if [[ "${python_version_origin}" == "runtime.txt" ]]; then Please delete your runtime.txt file and create a new file named: .python-version - Make sure to include the '.' at the start of the filename. + Make sure to include the '.' character at the start of the + filename. Don't add a file extension such as '.txt'. - In the new file, specify your app's Python version without - quotes or a 'python-' prefix. For example: + In the new file, specify your app's major Python version number + only. Don't include quotes or a 'python-' prefix. + + For example, to request the latest version of Python ${python_major_version}, + update your .python-version file so it contains exactly: ${python_major_version} - We strongly recommend that you use the major version form - instead of pinning to an exact version, since it will allow - your app to receive Python security updates. + We strongly recommend that you don't specify the Python patch + version number, since it will pin your app to an exact Python + version and so stop your app from receiving security updates + each time it builds. In the future support for runtime.txt will be removed and this warning will be made an error. diff --git a/bin/steps/nltk b/bin/steps/nltk index 6fc60c7ec..202d71433 100755 --- a/bin/steps/nltk +++ b/bin/steps/nltk @@ -35,7 +35,7 @@ if is_module_available 'nltk'; then Error: Unable to download NLTK data. The 'python -m nltk.downloader' command to download NLTK - data did not exit successfully. + data didn't exit successfully. See the log output above for more information. EOF diff --git a/lib/checks.sh b/lib/checks.sh index 42f8831d2..38a4ac457 100644 --- a/lib/checks.sh +++ b/lib/checks.sh @@ -25,12 +25,12 @@ function checks::ensure_supported_stack() { ;; *) output::error <<-EOF - Error: The '${stack}' stack is not recognised. + Error: The '${stack}' stack isn't recognised. - This buildpack does not recognise or support the '${stack}' stack. + This buildpack doesn't recognise or support the '${stack}' stack. If '${stack}' is a valid stack, make sure that you are using the latest - version of this buildpack and have not pinned to an older release: + version of this buildpack and haven't pinned to an older release: https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references EOF @@ -57,7 +57,7 @@ function checks::warn_if_duplicate_python_buildpack() { from a buildpack run earlier in the build. This normally means there are duplicate Python buildpacks set - on your app, which is not supported, can cause errors and + on your app, which isn't supported, can cause errors and slow down builds. Check the buildpacks set on your app and remove any duplicate @@ -99,13 +99,13 @@ function checks::warn_if_existing_python_dir_present() { $(find .heroku/python/ -maxdepth 2 || true) Writing to internal locations used by the Python buildpack - is not supported and can cause unexpected errors. + isn't supported and can cause unexpected errors. If you have committed a '.heroku/python/' directory to your Git repo, you must delete it or use a different location. Otherwise, check that an earlier buildpack or 'bin/pre_compile' - hook has not created this directory. + hook hasn't created this directory. If you have a use-case that requires writing to this location, please comment on: diff --git a/lib/package_manager.sh b/lib/package_manager.sh index 97a883136..a688586d7 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -20,14 +20,14 @@ function package_manager::determine_package_manager() { A 'Pipfile' file was found, however, the associated 'Pipfile.lock' Pipenv lockfile was not. This means your app dependency versions - are not pinned, which means the package versions used on Heroku + aren't pinned, which means the package versions used on Heroku might not match those installed in other environments. For now, we will install your dependencies without a lockfile, however, in the future this warning will become an error. Run 'pipenv lock' locally to generate the lockfile, and make sure - that 'Pipfile.lock' is not listed in '.gitignore' or '.slugignore'. + that 'Pipfile.lock' isn't listed in '.gitignore' or '.slugignore'. EOF package_managers_found+=(pipenv) package_managers_found_display_text+=("Pipfile (Pipenv)") diff --git a/lib/pip.sh b/lib/pip.sh index ca95abfd6..b4f208bb7 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -65,7 +65,7 @@ function pip::install_pip_setuptools_wheel() { Try building again to see if the error resolves itself. - If that does not help, check the status of PyPI here: + If that doesn't help, check the status of PyPI here: https://status.python.org EOF meta_set "failure_reason" "install-package-manager::pip" diff --git a/lib/pipenv.sh b/lib/pipenv.sh index c5c50beb4..a43192a37 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -38,7 +38,7 @@ function pipenv::install_pipenv() { Try building again to see if the error resolves itself. - If that does not help, check the status of PyPI here: + If that doesn't help, check the status of PyPI here: https://status.python.org EOF meta_set "failure_reason" "install-package-manager::pipenv" diff --git a/lib/poetry.sh b/lib/poetry.sh index a453356a1..a50294422 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -85,7 +85,7 @@ function poetry::install_poetry() { Try building again to see if the error resolves itself. - If that does not help, check the status of PyPI here: + If that doesn't help, check the status of PyPI here: https://status.python.org EOF meta_set "failure_reason" "install-package-manager::poetry" diff --git a/lib/python.sh b/lib/python.sh index 32733730e..bbf38878d 100644 --- a/lib/python.sh +++ b/lib/python.sh @@ -100,7 +100,7 @@ function python::install() { https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references - We also strongly recommend that you do not pin your app to an + We also strongly recommend that you don't pin your app to an exact Python version such as ${python_full_version}, and instead only specify the major Python version of ${python_major_version} in your ${python_version_origin} file. This will allow your app to receive the latest available Python diff --git a/lib/python_version.sh b/lib/python_version.sh index 02735b3b5..fab68a4b8 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -52,7 +52,7 @@ function python_version::read_requested_python_version() { local runtime_txt_path="${build_dir}/runtime.txt" if [[ -f "${runtime_txt_path}" ]]; then - contents="$(cat "${runtime_txt_path}")" + contents="$(cat --show-nonprinting "${runtime_txt_path}")" version="$(python_version::parse_runtime_txt "${contents}")" origin="runtime.txt" return 0 @@ -60,7 +60,7 @@ function python_version::read_requested_python_version() { local python_version_file_path="${build_dir}/.python-version" if [[ -f "${python_version_file_path}" ]]; then - contents="$(cat "${python_version_file_path}")" + contents="$(cat --show-nonprinting "${python_version_file_path}")" version="$(python_version::parse_python_version_file "${contents}")" origin=".python-version" return 0 @@ -106,23 +106,28 @@ function python_version::parse_runtime_txt() { The following file contents were found, which aren't valid: ${contents:0:100} - However, the runtime.txt file is deprecated since it has - been replaced by the .python-version file. As such, we - recommend that you switch to using a .python-version file + However, the runtime.txt file is deprecated since it has been + replaced by the more widely supported .python-version file. + As such, we recommend that you switch to using .python-version instead of fixing your runtime.txt file. Please delete your runtime.txt file and create a new file named: .python-version - Make sure to include the '.' at the start of the filename. + Make sure to include the '.' character at the start of the + filename. Don't add a file extension such as '.txt'. - In the new file, specify your app's Python version without - quotes or a 'python-' prefix. For example: + In the new file, specify your app's major Python version number + only. Don't include quotes or a 'python-' prefix. + + For example, to request the latest version of Python ${DEFAULT_PYTHON_MAJOR_VERSION}, + update your .python-version file so it contains exactly: ${DEFAULT_PYTHON_MAJOR_VERSION} - We strongly recommend that you use the major version form - instead of pinning to an exact version, since it will allow - your app to receive Python security updates. + We strongly recommend that you don't specify the Python patch + version number, since it will pin your app to an exact Python + version and so stop your app from receiving security updates + each time it builds. EOF meta_set "failure_reason" "runtime-txt::invalid-version" meta_set "failure_detail" "${contents:0:50}" @@ -160,19 +165,20 @@ function python_version::parse_python_version_file() { ${line} However, the Python version must be specified as either: - 1. The major version only: 3.X (recommended) - 2. An exact patch version: 3.X.Y + 1. The major version only, for example: ${DEFAULT_PYTHON_MAJOR_VERSION} (recommended) + 2. An exact patch version, for example: ${DEFAULT_PYTHON_MAJOR_VERSION}.999 - Don't include quotes or a 'python-' prefix. To include - comments, add them on their own line, prefixed with '#'. + Don't include quotes, a 'python-' prefix or wildcards. Any + code comments must be on a separate line prefixed with '#'. For example, to request the latest version of Python ${DEFAULT_PYTHON_MAJOR_VERSION}, - update your .python-version file so it contains: + update your .python-version file so it contains exactly: ${DEFAULT_PYTHON_MAJOR_VERSION} - We strongly recommend that you use the major version form - instead of pinning to an exact version, since it will allow - your app to receive Python security updates. + We strongly recommend that you don't specify the Python patch + version number, since it will pin your app to an exact Python + version and so stop your app from receiving security updates + each time it builds. EOF meta_set "failure_reason" "python-version-file::invalid-version" meta_set "failure_detail" "${line:0:50}" @@ -185,10 +191,11 @@ function python_version::parse_python_version_file() { No Python version was found in your .python-version file. - Update the file so that it contains a valid Python version. + Update the file so that it contains your app's major Python + version number. Don't include quotes or a 'python-' prefix. For example, to request the latest version of Python ${DEFAULT_PYTHON_MAJOR_VERSION}, - update your .python-version file so it contains: + update your .python-version file so it contains exactly: ${DEFAULT_PYTHON_MAJOR_VERSION} If the file already contains a version, check the line doesn't @@ -212,6 +219,10 @@ function python_version::parse_python_version_file() { Update the file so it contains only one Python version. + For example, to request the latest version of Python ${DEFAULT_PYTHON_MAJOR_VERSION}, + update your .python-version file so it contains exactly: + ${DEFAULT_PYTHON_MAJOR_VERSION} + If you have added comments to the file, make sure that those lines begin with a '#', so that they are ignored. EOF @@ -278,15 +289,18 @@ function python_version::read_pipenv_python_version() { ${version} However, the Python version must be specified as either: - 1. The major version only: 3.X (recommended) - 2. An exact patch version: 3.X.Y + 1. The major version only, for example: ${DEFAULT_PYTHON_MAJOR_VERSION} (recommended) + 2. An exact patch version, for example: ${DEFAULT_PYTHON_MAJOR_VERSION}.999 + + Wildcards aren't supported. Please update your Pipfile to use a valid Python version and then run 'pipenv lock' to regenerate Pipfile.lock. - We strongly recommend that you use the major version form - instead of pinning to an exact version, since it will allow - your app to receive Python security updates. + We strongly recommend that you don't specify the Python patch + version number, since it will pin your app to an exact Python + version and so stop your app from receiving security updates + each time it builds. For more information, see: https://pipenv.pypa.io/en/stable/specifiers.html#specifying-versions-of-python @@ -329,12 +343,21 @@ function python_version::resolve_python_version() { Please upgrade to at least Python 3.${OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION} by configuring an explicit Python version for your app. - Create a .python-version file in the root directory of your - app, that contains a Python version like: - 3.${NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION} + Create a new file in the root directory of your app named: + .python-version + + Make sure to include the '.' character at the start of the + filename. Don't add a file extension such as '.txt'. + + In the new file, specify the new major Python version number + only. Don't include quotes or a 'python-' prefix. - When creating this file make sure to include the '.' at the - start of the filename. + For example, to request the latest version of Python 3.${OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION}, + update your .python-version file so it contains exactly: + 3.${OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION} + + If possible, we recommend upgrading all the way to Python ${DEFAULT_PYTHON_MAJOR_VERSION}, + since it contains many performance and usability improvements. EOF else output::error <<-EOF @@ -349,6 +372,9 @@ function python_version::resolve_python_version() { Please upgrade to at least Python 3.${OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION} by changing the version in your ${python_version_origin} file. + + If possible, we recommend upgrading all the way to Python ${DEFAULT_PYTHON_MAJOR_VERSION}, + since it contains many performance and usability improvements. EOF fi meta_set "failure_reason" "python-version::eol" @@ -464,7 +490,7 @@ function python_version::warn_if_patch_update_available() { Update your ${python_version_origin} file to use the new version. - We strongly recommend that you do not pin your app to an + We strongly recommend that you don't pin your app to an exact Python version such as ${python_full_version}, and instead only specify the major Python version of ${python_major_version} in your ${python_version_origin} file. This will allow your app to receive the latest available Python diff --git a/lib/utils.sh b/lib/utils.sh index 916886431..d35cba11d 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -35,10 +35,10 @@ function utils::bundled_pip_module_path() { echo "${bundled_pip_wheel}/pip" else output::error <<-EOF - Internal Error: Unable to locate the bundled copy of pip. + Internal Error: Unable to locate the Python stdlib's bundled pip. - The Python buildpack could not locate the copy of pip bundled - inside Python's 'ensurepip' module: + Couldn't find the pip wheel file bundled inside the Python + stdlib's 'ensurepip' module: $(find "${bundled_wheels_dir}/" 2>&1 || find "${python_home}/" -type d 2>&1 || true) EOF diff --git a/spec/fixtures/python_version_file_invalid_version/.python-version b/spec/fixtures/python_version_file_invalid_version/.python-version index 4ffb299eb..3877ea217 100644 --- a/spec/fixtures/python_version_file_invalid_version/.python-version +++ b/spec/fixtures/python_version_file_invalid_version/.python-version @@ -4,7 +4,8 @@ # So are empty lines, and leading/trailing whitespace. - 3.12.0invalid + # The version number has a trailing control character. + 3.12.0 # 2.7.18 diff --git a/spec/fixtures/runtime_txt_invalid_version/runtime.txt b/spec/fixtures/runtime_txt_invalid_version/runtime.txt index 504402610..4a52c21f2 100644 --- a/spec/fixtures/runtime_txt_invalid_version/runtime.txt +++ b/spec/fixtures/runtime_txt_invalid_version/runtime.txt @@ -1 +1 @@ -python-3.12.0invalid +python-3.12.0 diff --git a/spec/hatchet/checks_spec.rb b/spec/hatchet/checks_spec.rb index ed55fc139..e67c8aea5 100644 --- a/spec/hatchet/checks_spec.rb +++ b/spec/hatchet/checks_spec.rb @@ -18,7 +18,7 @@ remote: ! from a buildpack run earlier in the build. remote: ! remote: ! This normally means there are duplicate Python buildpacks set - remote: ! on your app, which is not supported, can cause errors and + remote: ! on your app, which isn't supported, can cause errors and remote: ! slow down builds. remote: ! remote: ! Check the buildpacks set on your app and remove any duplicate @@ -59,13 +59,13 @@ remote: ! .heroku/python/bin/python remote: ! remote: ! Writing to internal locations used by the Python buildpack - remote: ! is not supported and can cause unexpected errors. + remote: ! isn't supported and can cause unexpected errors. remote: ! remote: ! If you have committed a '.heroku/python/' directory to your remote: ! Git repo, you must delete it or use a different location. remote: ! remote: ! Otherwise, check that an earlier buildpack or 'bin/pre_compile' - remote: ! hook has not created this directory. + remote: ! hook hasn't created this directory. remote: ! remote: ! If you have a use-case that requires writing to this location, remote: ! please comment on: @@ -76,10 +76,10 @@ remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: - remote: ! Internal Error: Unable to locate the bundled copy of pip. + remote: ! Internal Error: Unable to locate the Python stdlib's bundled pip. remote: ! - remote: ! The Python buildpack could not locate the copy of pip bundled - remote: ! inside Python's 'ensurepip' module: + remote: ! Couldn't find the pip wheel file bundled inside the Python + remote: ! stdlib's 'ensurepip' module: remote: ! remote: ! find: ‘/app/.heroku/python/lib/python3.13/ensurepip/_bundled/’: No such file or directory remote: ! /app/.heroku/python/ diff --git a/spec/hatchet/nltk_spec.rb b/spec/hatchet/nltk_spec.rb index 80c2265b1..56ed95ce4 100644 --- a/spec/hatchet/nltk_spec.rb +++ b/spec/hatchet/nltk_spec.rb @@ -68,7 +68,7 @@ remote: ! Error: Unable to download NLTK data. remote: ! remote: ! The 'python -m nltk.downloader' command to download NLTK - remote: ! data did not exit successfully. + remote: ! data didn't exit successfully. remote: ! remote: ! See the log output above for more information. remote: diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 02f9007fa..fff5c72c0 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -105,7 +105,7 @@ remote: ! remote: ! Update your Pipfile.lock file to use the new version. remote: ! - remote: ! We strongly recommend that you do not pin your app to an + remote: ! We strongly recommend that you don't pin your app to an remote: ! exact Python version such as 3.9.0, and instead only specify remote: ! the major Python version of 3.9 in your Pipfile.lock file. remote: ! This will allow your app to receive the latest available Python @@ -171,14 +171,14 @@ remote: ! remote: ! A 'Pipfile' file was found, however, the associated 'Pipfile.lock' remote: ! Pipenv lockfile was not. This means your app dependency versions - remote: ! are not pinned, which means the package versions used on Heroku + remote: ! aren't pinned, which means the package versions used on Heroku remote: ! might not match those installed in other environments. remote: ! remote: ! For now, we will install your dependencies without a lockfile, remote: ! however, in the future this warning will become an error. remote: ! remote: ! Run 'pipenv lock' locally to generate the lockfile, and make sure - remote: ! that 'Pipfile.lock' is not listed in '.gitignore' or '.slugignore'. + remote: ! that 'Pipfile.lock' isn't listed in '.gitignore' or '.slugignore'. remote: remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes @@ -239,15 +239,18 @@ remote: ! ^3.12 remote: ! remote: ! However, the Python version must be specified as either: - remote: ! 1. The major version only: 3.X (recommended) - remote: ! 2. An exact patch version: 3.X.Y + remote: ! 1. The major version only, for example: #{DEFAULT_PYTHON_MAJOR_VERSION} (recommended) + remote: ! 2. An exact patch version, for example: #{DEFAULT_PYTHON_MAJOR_VERSION}.999 + remote: ! + remote: ! Wildcards aren't supported. remote: ! remote: ! Please update your Pipfile to use a valid Python version and remote: ! then run 'pipenv lock' to regenerate Pipfile.lock. remote: ! - remote: ! We strongly recommend that you use the major version form - remote: ! instead of pinning to an exact version, since it will allow - remote: ! your app to receive Python security updates. + remote: ! We strongly recommend that you don't specify the Python patch + remote: ! version number, since it will pin your app to an exact Python + remote: ! version and so stop your app from receiving security updates + remote: ! each time it builds. remote: ! remote: ! For more information, see: remote: ! https://pipenv.pypa.io/en/stable/specifiers.html#specifying-versions-of-python @@ -275,15 +278,18 @@ remote: ! 3.9.* remote: ! remote: ! However, the Python version must be specified as either: - remote: ! 1. The major version only: 3.X (recommended) - remote: ! 2. An exact patch version: 3.X.Y + remote: ! 1. The major version only, for example: #{DEFAULT_PYTHON_MAJOR_VERSION} (recommended) + remote: ! 2. An exact patch version, for example: #{DEFAULT_PYTHON_MAJOR_VERSION}.999 + remote: ! + remote: ! Wildcards aren't supported. remote: ! remote: ! Please update your Pipfile to use a valid Python version and remote: ! then run 'pipenv lock' to regenerate Pipfile.lock. remote: ! - remote: ! We strongly recommend that you use the major version form - remote: ! instead of pinning to an exact version, since it will allow - remote: ! your app to receive Python security updates. + remote: ! We strongly recommend that you don't specify the Python patch + remote: ! version number, since it will pin your app to an exact Python + remote: ! version and so stop your app from receiving security updates + remote: ! each time it builds. remote: ! remote: ! For more information, see: remote: ! https://pipenv.pypa.io/en/stable/specifiers.html#specifying-versions-of-python @@ -314,6 +320,9 @@ remote: ! remote: ! Please upgrade to at least Python 3.9 by changing the remote: ! version in your Pipfile.lock file. + remote: ! + remote: ! If possible, we recommend upgrading all the way to Python #{DEFAULT_PYTHON_MAJOR_VERSION}, + remote: ! since it contains many performance and usability improvements. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 9d8d6c4ac..242718664 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -217,7 +217,7 @@ remote: ! remote: ! Update your .python-version file to use the new version. remote: ! - remote: ! We strongly recommend that you do not pin your app to an + remote: ! We strongly recommend that you don't pin your app to an remote: ! exact Python version such as 3.9.0, and instead only specify remote: ! the major Python version of 3.9 in your .python-version file. remote: ! This will allow your app to receive the latest available Python diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index c1025f415..983c92afb 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -24,15 +24,20 @@ remote: ! Please delete your runtime.txt file and create a new file named: remote: ! .python-version remote: ! - remote: ! Make sure to include the '.' at the start of the filename. + remote: ! Make sure to include the '.' character at the start of the + remote: ! filename. Don't add a file extension such as '.txt'. remote: ! - remote: ! In the new file, specify your app's Python version without - remote: ! quotes or a 'python-' prefix. For example: + remote: ! In the new file, specify your app's major Python version number + remote: ! only. Don't include quotes or a 'python-' prefix. + remote: ! + remote: ! For example, to request the latest version of Python 3.9, + remote: ! update your .python-version file so it contains exactly: remote: ! 3.9 remote: ! - remote: ! We strongly recommend that you use the major version form - remote: ! instead of pinning to an exact version, since it will allow - remote: ! your app to receive Python security updates. + remote: ! We strongly recommend that you don't specify the Python patch + remote: ! version number, since it will pin your app to an exact Python + remote: ! version and so stop your app from receiving security updates + remote: ! each time it builds. remote: ! remote: ! In the future support for runtime.txt will be removed and remote: ! this warning will be made an error. @@ -64,7 +69,7 @@ remote: ! remote: ! Update your runtime.txt file to use the new version. remote: ! - remote: ! We strongly recommend that you do not pin your app to an + remote: ! We strongly recommend that you don't pin your app to an remote: ! exact Python version such as 3.9.0, and instead only specify remote: ! the major Python version of 3.9 in your runtime.txt file. remote: ! This will allow your app to receive the latest available Python @@ -96,7 +101,7 @@ remote: ! remote: ! Update your .python-version file to use the new version. remote: ! - remote: ! We strongly recommend that you do not pin your app to an + remote: ! We strongly recommend that you don't pin your app to an remote: ! exact Python version such as 3.10.0, and instead only specify remote: ! the major Python version of 3.10 in your .python-version file. remote: ! This will allow your app to receive the latest available Python diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 175ab3e6f..f3c44df90 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -152,22 +152,23 @@ remote: ! isn't in the correct format. remote: ! remote: ! The following version was found: - remote: ! 3.12.0invalid + remote: ! 3.12.0^[ remote: ! remote: ! However, the Python version must be specified as either: - remote: ! 1. The major version only: 3.X (recommended) - remote: ! 2. An exact patch version: 3.X.Y + remote: ! 1. The major version only, for example: #{DEFAULT_PYTHON_MAJOR_VERSION} (recommended) + remote: ! 2. An exact patch version, for example: #{DEFAULT_PYTHON_MAJOR_VERSION}.999 remote: ! - remote: ! Don't include quotes or a 'python-' prefix. To include - remote: ! comments, add them on their own line, prefixed with '#'. + remote: ! Don't include quotes, a 'python-' prefix or wildcards. Any + remote: ! code comments must be on a separate line prefixed with '#'. remote: ! remote: ! For example, to request the latest version of Python #{DEFAULT_PYTHON_MAJOR_VERSION}, - remote: ! update your .python-version file so it contains: + remote: ! update your .python-version file so it contains exactly: remote: ! #{DEFAULT_PYTHON_MAJOR_VERSION} remote: ! - remote: ! We strongly recommend that you use the major version form - remote: ! instead of pinning to an exact version, since it will allow - remote: ! your app to receive Python security updates. + remote: ! We strongly recommend that you don't specify the Python patch + remote: ! version number, since it will pin your app to an exact Python + remote: ! version and so stop your app from receiving security updates + remote: ! each time it builds. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -187,10 +188,11 @@ remote: ! remote: ! No Python version was found in your .python-version file. remote: ! - remote: ! Update the file so that it contains a valid Python version. + remote: ! Update the file so that it contains your app's major Python + remote: ! version number. Don't include quotes or a 'python-' prefix. remote: ! remote: ! For example, to request the latest version of Python #{DEFAULT_PYTHON_MAJOR_VERSION}, - remote: ! update your .python-version file so it contains: + remote: ! update your .python-version file so it contains exactly: remote: ! #{DEFAULT_PYTHON_MAJOR_VERSION} remote: ! remote: ! If the file already contains a version, check the line doesn't @@ -220,6 +222,10 @@ remote: ! remote: ! Update the file so it contains only one Python version. remote: ! + remote: ! For example, to request the latest version of Python #{DEFAULT_PYTHON_MAJOR_VERSION}, + remote: ! update your .python-version file so it contains exactly: + remote: ! #{DEFAULT_PYTHON_MAJOR_VERSION} + remote: ! remote: ! If you have added comments to the file, make sure that those remote: ! lines begin with a '#', so that they are ignored. remote: @@ -249,6 +255,9 @@ remote: ! remote: ! Please upgrade to at least Python 3.9 by changing the remote: ! version in your .python-version file. + remote: ! + remote: ! If possible, we recommend upgrading all the way to Python #{DEFAULT_PYTHON_MAJOR_VERSION}, + remote: ! since it contains many performance and usability improvements. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -317,7 +326,7 @@ remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references remote: ! - remote: ! We also strongly recommend that you do not pin your app to an + remote: ! We also strongly recommend that you don't pin your app to an remote: ! exact Python version such as 3.12.999, and instead only specify remote: ! the major Python version of 3.12 in your .python-version file. remote: ! This will allow your app to receive the latest available Python @@ -343,25 +352,30 @@ remote: ! in the correct format. remote: ! remote: ! The following file contents were found, which aren't valid: - remote: ! python-3.12.0invalid + remote: ! python-3.12.0^[ remote: ! - remote: ! However, the runtime.txt file is deprecated since it has - remote: ! been replaced by the .python-version file. As such, we - remote: ! recommend that you switch to using a .python-version file + remote: ! However, the runtime.txt file is deprecated since it has been + remote: ! replaced by the more widely supported .python-version file. + remote: ! As such, we recommend that you switch to using .python-version remote: ! instead of fixing your runtime.txt file. remote: ! remote: ! Please delete your runtime.txt file and create a new file named: remote: ! .python-version remote: ! - remote: ! Make sure to include the '.' at the start of the filename. + remote: ! Make sure to include the '.' character at the start of the + remote: ! filename. Don't add a file extension such as '.txt'. remote: ! - remote: ! In the new file, specify your app's Python version without - remote: ! quotes or a 'python-' prefix. For example: + remote: ! In the new file, specify your app's major Python version number + remote: ! only. Don't include quotes or a 'python-' prefix. + remote: ! + remote: ! For example, to request the latest version of Python #{DEFAULT_PYTHON_MAJOR_VERSION}, + remote: ! update your .python-version file so it contains exactly: remote: ! #{DEFAULT_PYTHON_MAJOR_VERSION} remote: ! - remote: ! We strongly recommend that you use the major version form - remote: ! instead of pinning to an exact version, since it will allow - remote: ! your app to receive Python security updates. + remote: ! We strongly recommend that you don't specify the Python patch + remote: ! version number, since it will pin your app to an exact Python + remote: ! version and so stop your app from receiving security updates + remote: ! each time it builds. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -389,6 +403,9 @@ remote: ! remote: ! Please upgrade to at least Python 3.9 by changing the remote: ! version in your runtime.txt file. + remote: ! + remote: ! If possible, we recommend upgrading all the way to Python #{DEFAULT_PYTHON_MAJOR_VERSION}, + remote: ! since it contains many performance and usability improvements. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -416,15 +433,20 @@ remote: ! Please delete your runtime.txt file and create a new file named: remote: ! .python-version remote: ! - remote: ! Make sure to include the '.' at the start of the filename. + remote: ! Make sure to include the '.' character at the start of the + remote: ! filename. Don't add a file extension such as '.txt'. + remote: ! + remote: ! In the new file, specify your app's major Python version number + remote: ! only. Don't include quotes or a 'python-' prefix. remote: ! - remote: ! In the new file, specify your app's Python version without - remote: ! quotes or a 'python-' prefix. For example: + remote: ! For example, to request the latest version of Python 3.13, + remote: ! update your .python-version file so it contains exactly: remote: ! 3.13 remote: ! - remote: ! We strongly recommend that you use the major version form - remote: ! instead of pinning to an exact version, since it will allow - remote: ! your app to receive Python security updates. + remote: ! We strongly recommend that you don't specify the Python patch + remote: ! version number, since it will pin your app to an exact Python + remote: ! version and so stop your app from receiving security updates + remote: ! each time it builds. remote: ! remote: ! In the future support for runtime.txt will be removed and remote: ! this warning will be made an error. From 359af928f39a2693a035bdba0736894d4bf8471c Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 5 May 2025 10:58:09 +0100 Subject: [PATCH 126/406] Add more tests for cached Python version related errors (#1784) We already had tests for apps that explicitly request an EOL or unrecognised Python version, however, didn't have any for the case where that invalid version was inferred from the cached Python version (because the app didn't specify an explicit Python version). Since we display slightly different error messages for both cases, it's worth having explicit tests for the inferred from cache scenario too - particularly since we'll be refactoring these error messages as part of the upcoming uv work. GUS-W-18436832. --- .../python_version_eol_cached/bin/pre_compile | 9 ++ .../requirements.txt | 0 .../bin/pre_compile | 8 ++ .../requirements.txt | 0 spec/hatchet/python_version_spec.rb | 90 +++++++++++++++++++ 5 files changed, 107 insertions(+) create mode 100755 spec/fixtures/python_version_eol_cached/bin/pre_compile create mode 100644 spec/fixtures/python_version_eol_cached/requirements.txt create mode 100755 spec/fixtures/python_version_non_existent_major_cached/bin/pre_compile create mode 100644 spec/fixtures/python_version_non_existent_major_cached/requirements.txt diff --git a/spec/fixtures/python_version_eol_cached/bin/pre_compile b/spec/fixtures/python_version_eol_cached/bin/pre_compile new file mode 100755 index 000000000..5bce49644 --- /dev/null +++ b/spec/fixtures/python_version_eol_cached/bin/pre_compile @@ -0,0 +1,9 @@ +#!/usr/bin/env bash + +set -euo pipefail + +# Emulate the cache from a previous build that used a now-EOL Python version, since we can't +# actually perform a build using Python 3.8, since it was only supported until Heroku-20. +# TODO: Switch to doing a real build once Python 3.9 EOLs. +mkdir -p "${CACHE_DIR:?}/.heroku" +echo "python-3.8.20" >"${CACHE_DIR}/.heroku/python-version" diff --git a/spec/fixtures/python_version_eol_cached/requirements.txt b/spec/fixtures/python_version_eol_cached/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/python_version_non_existent_major_cached/bin/pre_compile b/spec/fixtures/python_version_non_existent_major_cached/bin/pre_compile new file mode 100755 index 000000000..1075c4857 --- /dev/null +++ b/spec/fixtures/python_version_non_existent_major_cached/bin/pre_compile @@ -0,0 +1,8 @@ +#!/usr/bin/env bash + +set -euo pipefail + +# Emulate the cache from a previous build which used a newer default Python version +# than the version supported by this buildpack version. +mkdir -p "${CACHE_DIR:?}/.heroku" +echo "python-3.99.0" >"${CACHE_DIR}/.heroku/python-version" diff --git a/spec/fixtures/python_version_non_existent_major_cached/requirements.txt b/spec/fixtures/python_version_non_existent_major_cached/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index f3c44df90..8bca310f2 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -77,6 +77,96 @@ end end end + + context 'with an app last built using an EOL Python version' do + # This fixture emulates the cache from a previous build that used a now-EOL Python version, + # since we can't actually perform a build using Python 3.8, since it was only supported + # until Heroku-20. TODO: Switch to doing a real EOL cached rebuild once Python 3.9 EOLs. + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_eol_cached', allow_failure: true) } + + it 'aborts the build with an EOL message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Running bin/pre_compile hook + remote: -----> No Python version was specified. Using the same major version as the last build: Python 3.8 + remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: + remote: ! Error: The cached Python version has reached end-of-life. + remote: ! + remote: ! Your app doesn't specify a Python version, and so normally + remote: ! would use the version cached from the last build (3.8). + remote: ! + remote: ! However, Python 3.8 has reached its upstream end-of-life, + remote: ! and is therefore no longer receiving security updates: + remote: ! https://devguide.python.org/versions/#supported-versions + remote: ! + remote: ! As such, it's no longer supported by this buildpack: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions + remote: ! + remote: ! Please upgrade to at least Python 3.9 by configuring an + remote: ! explicit Python version for your app. + remote: ! + remote: ! Create a new file in the root directory of your app named: + remote: ! .python-version + remote: ! + remote: ! Make sure to include the '.' character at the start of the + remote: ! filename. Don't add a file extension such as '.txt'. + remote: ! + remote: ! In the new file, specify the new major Python version number + remote: ! only. Don't include quotes or a 'python-' prefix. + remote: ! + remote: ! For example, to request the latest version of Python 3.9, + remote: ! update your .python-version file so it contains exactly: + remote: ! 3.9 + remote: ! + remote: ! If possible, we recommend upgrading all the way to Python #{DEFAULT_PYTHON_MAJOR_VERSION}, + remote: ! since it contains many performance and usability improvements. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + context 'with an app last built using an unrecognised Python version' do + # This fixture emulates the cache from a previous build which used a newer default + # Python version than the version supported by this buildpack version. + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_non_existent_major_cached', allow_failure: true) } + + it 'aborts the build with an EOL message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Running bin/pre_compile hook + remote: -----> No Python version was specified. Using the same major version as the last build: Python 3.99 + remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: + remote: ! Error: The cached Python version isn't recognised. + remote: ! + remote: ! Your app doesn't specify a Python version, and so normally + remote: ! would use the version cached from the last build (3.99). + remote: ! + remote: ! However, Python 3.99 isn't recognised by this version + remote: ! of the buildpack. + remote: ! + remote: ! This can occur if you have downgraded the version of the + remote: ! buildpack to an older version. + remote: ! + remote: ! Please switch back to a newer version of this buildpack: + remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks + remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references + remote: ! + remote: ! Alternatively, request an older Python version by creating + remote: ! a .python-version file in the root directory of your app, + remote: ! that contains a Python version like: + remote: ! 3.13 + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end end context 'when .python-version contains Python 3.9' do From 2de9c05420d94316aba282ae3e0558820632afd8 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 6 May 2025 10:25:50 +0100 Subject: [PATCH 127/406] Start rename of `python_version_reason` metric to `python_version_origin` (#1785) So that the metric name reflects the name used within the codebase. I've left the old metric name in for now, so that existing queries continue to work. Once all old data has aged out that doesn't have the new metric name, I can migrate the queries over to the new name. GUS-W-18437360. --- bin/compile | 2 ++ bin/report | 1 + 2 files changed, 3 insertions(+) diff --git a/bin/compile b/bin/compile index d4e67729c..65c820148 100755 --- a/bin/compile +++ b/bin/compile @@ -112,6 +112,8 @@ cached_python_full_version="$(cache::cached_python_full_version "${CACHE_DIR}")" # without having to hardcode globals. See: https://stackoverflow.com/a/38997681 python_version::read_requested_python_version "${BUILD_DIR}" "${package_manager}" "${cached_python_full_version}" requested_python_version python_version_origin meta_set "python_version_requested" "${requested_python_version}" +meta_set "python_version_origin" "${python_version_origin}" +# TODO: Remove this once the old Honeycomb events that don't have `python_version_origin` have aged out. meta_set "python_version_reason" "${python_version_origin}" # TODO: More strongly recommend specifying a Python version (eg switch the messaging to diff --git a/bin/report b/bin/report index 9a3eca067..62d4a9183 100755 --- a/bin/report +++ b/bin/report @@ -71,6 +71,7 @@ STRING_FIELDS=( pipenv_version poetry_version python_version_major + python_version_origin python_version_reason python_version_requested python_version From 0d459189b26d50f16cff7e747644e8429a566e28 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 6 May 2025 11:10:39 +0100 Subject: [PATCH 128/406] Refactor Python version file related warnings and errors (#1786) To reduce duplication of content, particularly since there are more error/warning message variants that need to be added for uv support (due to the `.python-version` file needing to be made mandatory when using uv). GUS-W-18439072. --- CHANGELOG.md | 4 +- bin/compile | 36 +------ lib/python_version.sh | 115 +++++++++++++++------ spec/hatchet/python_update_warning_spec.rb | 5 +- spec/hatchet/python_version_spec.rb | 22 ++-- 5 files changed, 108 insertions(+), 74 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 679794547..98d420b01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,9 +3,9 @@ ## [Unreleased] - Improved the instructions for migrating from `runtime.txt` to `.python-version`. ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783)) -- Improved the error message instructions shown when `.python-version`, `runtime.txt` or `Pipfile.lock` contain an invalid Python version. ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783)) +- Improved the error message instructions shown when `.python-version`, `runtime.txt` or `Pipfile.lock` contain an invalid Python version. ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783) and [#1786](https://github.com/heroku/heroku-buildpack-python/pull/1786)) - Improved the rendering of the error message shown if `.python-version` or `runtime.txt` contain stray invisible characters (such as ASCII control codes). ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783)) -- Improved the upgrade instructions shown for EOL and unsupported Python versions. ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783)) +- Improved the upgrade instructions shown for EOL and unsupported Python versions. ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783) and [#1786](https://github.com/heroku/heroku-buildpack-python/pull/1786)) ## [v282] - 2025-05-02 diff --git a/bin/compile b/bin/compile index 65c820148..b732d6c3b 100755 --- a/bin/compile +++ b/bin/compile @@ -143,37 +143,11 @@ else meta_set "python_version_pinned" "false" fi -if [[ "${python_version_origin}" == "runtime.txt" ]]; then - output::warning <<-EOF - Warning: The runtime.txt file is deprecated. - - The runtime.txt file is deprecated since it has been replaced - by the more widely supported .python-version file: - https://devcenter.heroku.com/changelog-items/3141 - - Please delete your runtime.txt file and create a new file named: - .python-version - - Make sure to include the '.' character at the start of the - filename. Don't add a file extension such as '.txt'. - - In the new file, specify your app's major Python version number - only. Don't include quotes or a 'python-' prefix. - - For example, to request the latest version of Python ${python_major_version}, - update your .python-version file so it contains exactly: - ${python_major_version} - - We strongly recommend that you don't specify the Python patch - version number, since it will pin your app to an exact Python - version and so stop your app from receiving security updates - each time it builds. - - In the future support for runtime.txt will be removed and - this warning will be made an error. - EOF -fi - +# We wait until after Python version resolution to show these warnings, to avoid causing confusion +# as to what was a warning vs an error. In addition, several of the error messages contain similar +# content to the warnings (such as encouraging use of .python-version and major version syntax), +# which would mean duplicate content if we showed both. +python_version::warn_if_python_version_file_missing "${python_version_origin}" "${python_major_version}" python_version::warn_if_deprecated_major_version "${python_major_version}" "${python_version_origin}" python_version::warn_if_patch_update_available "${python_full_version}" "${python_major_version}" "${python_version_origin}" diff --git a/lib/python_version.sh b/lib/python_version.sh index fab68a4b8..8965cbed1 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -97,6 +97,8 @@ function python_version::parse_runtime_txt() { local version="${BASH_REMATCH[1]}" echo "${version}" else + local instructions + instructions="$(python_version::python_version_file_instructions "runtime.txt" "${DEFAULT_PYTHON_MAJOR_VERSION}")" output::error <<-EOF Error: Invalid Python version in runtime.txt. @@ -107,27 +109,13 @@ function python_version::parse_runtime_txt() { ${contents:0:100} However, the runtime.txt file is deprecated since it has been - replaced by the more widely supported .python-version file. + replaced by the more widely supported .python-version file: + https://devcenter.heroku.com/changelog-items/3141 + As such, we recommend that you switch to using .python-version instead of fixing your runtime.txt file. - Please delete your runtime.txt file and create a new file named: - .python-version - - Make sure to include the '.' character at the start of the - filename. Don't add a file extension such as '.txt'. - - In the new file, specify your app's major Python version number - only. Don't include quotes or a 'python-' prefix. - - For example, to request the latest version of Python ${DEFAULT_PYTHON_MAJOR_VERSION}, - update your .python-version file so it contains exactly: - ${DEFAULT_PYTHON_MAJOR_VERSION} - - We strongly recommend that you don't specify the Python patch - version number, since it will pin your app to an exact Python - version and so stop your app from receiving security updates - each time it builds. + ${instructions} EOF meta_set "failure_reason" "runtime-txt::invalid-version" meta_set "failure_detail" "${contents:0:50}" @@ -327,6 +315,8 @@ function python_version::resolve_python_version() { if ((major < 3 || (major == 3 && minor < OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION))); then if [[ "${python_version_origin}" == "cached" ]]; then + local instructions + instructions="$(python_version::python_version_file_instructions "${python_version_origin}" "3.${OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION}")" output::error <<-EOF Error: The cached Python version has reached end-of-life. @@ -343,21 +333,7 @@ function python_version::resolve_python_version() { Please upgrade to at least Python 3.${OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION} by configuring an explicit Python version for your app. - Create a new file in the root directory of your app named: - .python-version - - Make sure to include the '.' character at the start of the - filename. Don't add a file extension such as '.txt'. - - In the new file, specify the new major Python version number - only. Don't include quotes or a 'python-' prefix. - - For example, to request the latest version of Python 3.${OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION}, - update your .python-version file so it contains exactly: - 3.${OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION} - - If possible, we recommend upgrading all the way to Python ${DEFAULT_PYTHON_MAJOR_VERSION}, - since it contains many performance and usability improvements. + ${instructions} EOF else output::error <<-EOF @@ -443,6 +419,79 @@ function python_version::resolve_python_version() { esac } +function python_version::warn_if_python_version_file_missing() { + local python_version_origin="${1}" + local python_major_version="${2}" + + if [[ "${python_version_origin}" == ".python-version" ]]; then + return 0 + fi + + local instructions + instructions="$(python_version::python_version_file_instructions "${python_version_origin}" "${python_major_version}")" + + case "${python_version_origin}" in + default | cached) + # TODO: Add a warning for this case. + ;; + runtime.txt) + output::warning <<-EOF + Warning: The runtime.txt file is deprecated. + + The runtime.txt file is deprecated since it has been replaced + by the more widely supported .python-version file: + https://devcenter.heroku.com/changelog-items/3141 + + Please switch to using a .python-version file instead. + + ${instructions} + + In the future support for runtime.txt will be removed and + this warning will be made an error. + EOF + ;; + Pipfile.lock) + # Decide if we want to warn for this case. + ;; + *) utils::abort_internal_error "Unhandled Python version origin: ${python_version_origin}" ;; + esac +} + +function python_version::python_version_file_instructions() { + local python_version_origin="${1}" + local python_major_version="${2}" + + if [[ "${python_version_origin}" == "runtime.txt" ]]; then + cat <<-EOF + Delete your runtime.txt file and create a new file in the + root directory of your app named: + EOF + else + cat <<-EOF + Create a new file in the root directory of your app named: + EOF + fi + + cat <<-EOF + .python-version + + Make sure to include the '.' character at the start of the + filename. Don't add a file extension such as '.txt'. + + In the new file, specify your app's major Python version number + only. Don't include quotes or a 'python-' prefix. + + For example, to request the latest version of Python ${python_major_version}, + update your .python-version file so it contains exactly: + ${python_major_version} + + We strongly recommend that you don't specify the Python patch + version number, since it will pin your app to an exact Python + version and so stop your app from receiving security updates + each time it builds. + EOF +} + function python_version::warn_if_deprecated_major_version() { local requested_major_version="${1}" local version_origin="${2}" diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index 983c92afb..1cc094200 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -21,7 +21,10 @@ remote: ! by the more widely supported .python-version file: remote: ! https://devcenter.heroku.com/changelog-items/3141 remote: ! - remote: ! Please delete your runtime.txt file and create a new file named: + remote: ! Please switch to using a .python-version file instead. + remote: ! + remote: ! Delete your runtime.txt file and create a new file in the + remote: ! root directory of your app named: remote: ! .python-version remote: ! remote: ! Make sure to include the '.' character at the start of the diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 8bca310f2..1d294ad29 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -113,15 +113,17 @@ remote: ! Make sure to include the '.' character at the start of the remote: ! filename. Don't add a file extension such as '.txt'. remote: ! - remote: ! In the new file, specify the new major Python version number + remote: ! In the new file, specify your app's major Python version number remote: ! only. Don't include quotes or a 'python-' prefix. remote: ! remote: ! For example, to request the latest version of Python 3.9, remote: ! update your .python-version file so it contains exactly: remote: ! 3.9 remote: ! - remote: ! If possible, we recommend upgrading all the way to Python #{DEFAULT_PYTHON_MAJOR_VERSION}, - remote: ! since it contains many performance and usability improvements. + remote: ! We strongly recommend that you don't specify the Python patch + remote: ! version number, since it will pin your app to an exact Python + remote: ! version and so stop your app from receiving security updates + remote: ! each time it builds. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -378,7 +380,7 @@ remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references remote: ! - remote: ! Otherwise, switch to a supported version (such as Python #{DEFAULT_PYTHON_MAJOR_VERSION}) + remote: ! Otherwise, switch to a supported version (such as Python 3.13) remote: ! by changing the version in your .python-version file. remote: remote: ! Push rejected, failed to compile Python app. @@ -445,11 +447,14 @@ remote: ! python-3.12.0^[ remote: ! remote: ! However, the runtime.txt file is deprecated since it has been - remote: ! replaced by the more widely supported .python-version file. + remote: ! replaced by the more widely supported .python-version file: + remote: ! https://devcenter.heroku.com/changelog-items/3141 + remote: ! remote: ! As such, we recommend that you switch to using .python-version remote: ! instead of fixing your runtime.txt file. remote: ! - remote: ! Please delete your runtime.txt file and create a new file named: + remote: ! Delete your runtime.txt file and create a new file in the + remote: ! root directory of your app named: remote: ! .python-version remote: ! remote: ! Make sure to include the '.' character at the start of the @@ -520,7 +525,10 @@ remote: ! by the more widely supported .python-version file: remote: ! https://devcenter.heroku.com/changelog-items/3141 remote: ! - remote: ! Please delete your runtime.txt file and create a new file named: + remote: ! Please switch to using a .python-version file instead. + remote: ! + remote: ! Delete your runtime.txt file and create a new file in the + remote: ! root directory of your app named: remote: ! .python-version remote: ! remote: ! Make sure to include the '.' character at the start of the From c6fc772bf5b2635694caa335a4d443101552977a Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 6 May 2025 12:02:44 +0100 Subject: [PATCH 129/406] Warn if an app doesn't specify a Python version (#1787) Adds a warning when an app doesn't specify a Python version and instead relies upon the buildpack's default/cached version. Relying on the default Python version isn't recommended, since: 1. The default version may not match the version used locally, in CI, or in other instances of the app (including eg review apps). 2. The default version changes over time, and while the classic buildpack does reuse the major version cached from the last build, this is lost is the cache is manually cleared by the user, or when migrating from one app to another (including when migrating to a new Fir app). In addition, using a .python-version file will be mandatory for apps that use the upcoming uv support (since uv has native .python-version file support, and without that file, the UX around conflicting versions would be worse) - so it makes sense that we should at least warn when not using one for other package managers too. GUS-W-18439666. --- CHANGELOG.md | 6 +- bin/compile | 4 - bin/detect | 2 + lib/package_manager.sh | 2 + lib/python_version.sh | 26 +++++- .../.python-version | 1 + spec/hatchet/detect_spec.rb | 2 + spec/hatchet/package_manager_spec.rb | 2 + spec/hatchet/pipenv_spec.rb | 87 ++++++++++++++++++- spec/hatchet/python_version_spec.rb | 86 +++++++++++++++++- spec/hatchet/stack_spec.rb | 49 +++++++++-- 11 files changed, 247 insertions(+), 20 deletions(-) create mode 100644 spec/fixtures/pipenv_lockfile_out_of_sync/.python-version diff --git a/CHANGELOG.md b/CHANGELOG.md index 98d420b01..99a433e72 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,10 +2,12 @@ ## [Unreleased] +- Added a warning when an app doesn't specify a Python version and instead relies upon the default/cached version. ([#1787](https://github.com/heroku/heroku-buildpack-python/pull/1787)) - Improved the instructions for migrating from `runtime.txt` to `.python-version`. ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783)) -- Improved the error message instructions shown when `.python-version`, `runtime.txt` or `Pipfile.lock` contain an invalid Python version. ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783) and [#1786](https://github.com/heroku/heroku-buildpack-python/pull/1786)) -- Improved the rendering of the error message shown if `.python-version` or `runtime.txt` contain stray invisible characters (such as ASCII control codes). ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783)) +- Improved the error messages shown when `.python-version`, `runtime.txt` or `Pipfile.lock` contain an invalid Python version. ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783) and [#1786](https://github.com/heroku/heroku-buildpack-python/pull/1786)) +- Improved the rendering of the error message shown when `.python-version` or `runtime.txt` contain stray invisible characters (such as ASCII control codes). ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783)) - Improved the upgrade instructions shown for EOL and unsupported Python versions. ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783) and [#1786](https://github.com/heroku/heroku-buildpack-python/pull/1786)) +- Improved the error messages shown when no Python package manager files are found. ([#1787](https://github.com/heroku/heroku-buildpack-python/pull/1787)) ## [v282] - 2025-05-02 diff --git a/bin/compile b/bin/compile index b732d6c3b..b052052db 100755 --- a/bin/compile +++ b/bin/compile @@ -116,16 +116,12 @@ meta_set "python_version_origin" "${python_version_origin}" # TODO: Remove this once the old Honeycomb events that don't have `python_version_origin` have aged out. meta_set "python_version_reason" "${python_version_origin}" -# TODO: More strongly recommend specifying a Python version (eg switch the messaging to -# be a warning instead, after version resolution, and mention .python-version inline) case "${python_version_origin}" in default) output::step "No Python version was specified. Using the buildpack default: Python ${requested_python_version}" - echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" ;; cached) output::step "No Python version was specified. Using the same major version as the last build: Python ${requested_python_version}" - echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" ;; *) output::step "Using Python ${requested_python_version} specified in ${python_version_origin}" diff --git a/bin/detect b/bin/detect index 1317d6b47..c532ee044 100755 --- a/bin/detect +++ b/bin/detect @@ -77,6 +77,8 @@ If your app already has a package manager file, check that it: 1. Is in the top level directory (not a subdirectory). 2. Has the correct spelling (the filenames are case-sensitive). 3. Isn't listed in '.gitignore' or '.slugignore'. +4. Has been added to the Git repository using 'git add --all' + and then committed using 'git commit'. Otherwise, add a package manager file to your app. If your app has no dependencies, then create an empty 'requirements.txt' file. diff --git a/lib/package_manager.sh b/lib/package_manager.sh index a688586d7..c1b8d8e76 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -87,6 +87,8 @@ function package_manager::determine_package_manager() { 1. Is in the top level directory (not a subdirectory). 2. Has the correct spelling (the filenames are case-sensitive). 3. Isn't listed in '.gitignore' or '.slugignore'. + 4. Has been added to the Git repository using 'git add --all' + and then committed using 'git commit'. Otherwise, add a package manager file to your app. If your app has no dependencies, then create an empty 'requirements.txt' file. diff --git a/lib/python_version.sh b/lib/python_version.sh index 8965cbed1..65bd8770f 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -432,7 +432,31 @@ function python_version::warn_if_python_version_file_missing() { case "${python_version_origin}" in default | cached) - # TODO: Add a warning for this case. + output::warning <<-EOF + Warning: No Python version was specified. + + Your app doesn't specify a Python version and so the buildpack + picked a default version for you. + + Relying on this default version isn't recommended, since it + can change over time and may not be consistent with your local + development environment, CI or other instances of your app. + + Please configure an explicit Python version for your app. + + ${instructions} + + If your app already has a .python-version file, check that it: + + 1. Is in the top level directory (not a subdirectory). + 2. Is named exactly '.python-version' in all lowercase. + 3. Isn't listed in '.gitignore' or '.slugignore'. + 4. Has been added to the Git repository using 'git add --all' + and then committed using 'git commit'. + + In the future we will require the use of a .python-version + file and this warning will be made an error. + EOF ;; runtime.txt) output::warning <<-EOF diff --git a/spec/fixtures/pipenv_lockfile_out_of_sync/.python-version b/spec/fixtures/pipenv_lockfile_out_of_sync/.python-version new file mode 100644 index 000000000..24ee5b1be --- /dev/null +++ b/spec/fixtures/pipenv_lockfile_out_of_sync/.python-version @@ -0,0 +1 @@ +3.13 diff --git a/spec/hatchet/detect_spec.rb b/spec/hatchet/detect_spec.rb index dd99d8358..edbe242df 100644 --- a/spec/hatchet/detect_spec.rb +++ b/spec/hatchet/detect_spec.rb @@ -32,6 +32,8 @@ remote: ! 1. Is in the top level directory (not a subdirectory). remote: ! 2. Has the correct spelling (the filenames are case-sensitive). remote: ! 3. Isn't listed in '.gitignore' or '.slugignore'. + remote: ! 4. Has been added to the Git repository using 'git add --all' + remote: ! and then committed using 'git commit'. remote: ! remote: ! Otherwise, add a package manager file to your app. If your app has remote: ! no dependencies, then create an empty 'requirements.txt' file. diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index 631eb8df3..eccdf03d1 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -28,6 +28,8 @@ remote: ! 1. Is in the top level directory (not a subdirectory). remote: ! 2. Has the correct spelling (the filenames are case-sensitive). remote: ! 3. Isn't listed in '.gitignore' or '.slugignore'. + remote: ! 4. Has been added to the Git repository using 'git add --all' + remote: ! and then committed using 'git commit'. remote: ! remote: ! Otherwise, add a package manager file to your app. If your app has remote: ! no dependencies, then create an empty 'requirements.txt' file. diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index fff5c72c0..771338cf9 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -148,7 +148,47 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: + remote: ! Warning: No Python version was specified. + remote: ! + remote: ! Your app doesn't specify a Python version and so the buildpack + remote: ! picked a default version for you. + remote: ! + remote: ! Relying on this default version isn't recommended, since it + remote: ! can change over time and may not be consistent with your local + remote: ! development environment, CI or other instances of your app. + remote: ! + remote: ! Please configure an explicit Python version for your app. + remote: ! + remote: ! Create a new file in the root directory of your app named: + remote: ! .python-version + remote: ! + remote: ! Make sure to include the '.' character at the start of the + remote: ! filename. Don't add a file extension such as '.txt'. + remote: ! + remote: ! In the new file, specify your app's major Python version number + remote: ! only. Don't include quotes or a 'python-' prefix. + remote: ! + remote: ! For example, to request the latest version of Python #{DEFAULT_PYTHON_MAJOR_VERSION}, + remote: ! update your .python-version file so it contains exactly: + remote: ! #{DEFAULT_PYTHON_MAJOR_VERSION} + remote: ! + remote: ! We strongly recommend that you don't specify the Python patch + remote: ! version number, since it will pin your app to an exact Python + remote: ! version and so stop your app from receiving security updates + remote: ! each time it builds. + remote: ! + remote: ! If your app already has a .python-version file, check that it: + remote: ! + remote: ! 1. Is in the top level directory \\(not a subdirectory\\). + remote: ! 2. Is named exactly '.python-version' in all lowercase. + remote: ! 3. Isn't listed in '.gitignore' or '.slugignore'. + remote: ! 4. Has been added to the Git repository using 'git add --all' + remote: ! and then committed using 'git commit'. + remote: ! + remote: ! In the future we will require the use of a .python-version + remote: ! file and this warning will be made an error. + remote: remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} @@ -181,7 +221,47 @@ remote: ! that 'Pipfile.lock' isn't listed in '.gitignore' or '.slugignore'. remote: remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: + remote: ! Warning: No Python version was specified. + remote: ! + remote: ! Your app doesn't specify a Python version and so the buildpack + remote: ! picked a default version for you. + remote: ! + remote: ! Relying on this default version isn't recommended, since it + remote: ! can change over time and may not be consistent with your local + remote: ! development environment, CI or other instances of your app. + remote: ! + remote: ! Please configure an explicit Python version for your app. + remote: ! + remote: ! Create a new file in the root directory of your app named: + remote: ! .python-version + remote: ! + remote: ! Make sure to include the '.' character at the start of the + remote: ! filename. Don't add a file extension such as '.txt'. + remote: ! + remote: ! In the new file, specify your app's major Python version number + remote: ! only. Don't include quotes or a 'python-' prefix. + remote: ! + remote: ! For example, to request the latest version of Python #{DEFAULT_PYTHON_MAJOR_VERSION}, + remote: ! update your .python-version file so it contains exactly: + remote: ! #{DEFAULT_PYTHON_MAJOR_VERSION} + remote: ! + remote: ! We strongly recommend that you don't specify the Python patch + remote: ! version number, since it will pin your app to an exact Python + remote: ! version and so stop your app from receiving security updates + remote: ! each time it builds. + remote: ! + remote: ! If your app already has a .python-version file, check that it: + remote: ! + remote: ! 1. Is in the top level directory \\(not a subdirectory\\). + remote: ! 2. Is named exactly '.python-version' in all lowercase. + remote: ! 3. Isn't listed in '.gitignore' or '.slugignore'. + remote: ! 4. Has been added to the Git repository using 'git add --all' + remote: ! and then committed using 'git commit'. + remote: ! + remote: ! In the future we will require the use of a .python-version + remote: ! file and this warning will be made an error. + remote: remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} @@ -437,8 +517,7 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 1d294ad29..5085414fe 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -41,7 +41,47 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: + remote: ! Warning: No Python version was specified. + remote: ! + remote: ! Your app doesn't specify a Python version and so the buildpack + remote: ! picked a default version for you. + remote: ! + remote: ! Relying on this default version isn't recommended, since it + remote: ! can change over time and may not be consistent with your local + remote: ! development environment, CI or other instances of your app. + remote: ! + remote: ! Please configure an explicit Python version for your app. + remote: ! + remote: ! Create a new file in the root directory of your app named: + remote: ! .python-version + remote: ! + remote: ! Make sure to include the '.' character at the start of the + remote: ! filename. Don't add a file extension such as '.txt'. + remote: ! + remote: ! In the new file, specify your app's major Python version number + remote: ! only. Don't include quotes or a 'python-' prefix. + remote: ! + remote: ! For example, to request the latest version of Python #{DEFAULT_PYTHON_MAJOR_VERSION}, + remote: ! update your .python-version file so it contains exactly: + remote: ! #{DEFAULT_PYTHON_MAJOR_VERSION} + remote: ! + remote: ! We strongly recommend that you don't specify the Python patch + remote: ! version number, since it will pin your app to an exact Python + remote: ! version and so stop your app from receiving security updates + remote: ! each time it builds. + remote: ! + remote: ! If your app already has a .python-version file, check that it: + remote: ! + remote: ! 1. Is in the top level directory (not a subdirectory). + remote: ! 2. Is named exactly '.python-version' in all lowercase. + remote: ! 3. Isn't listed in '.gitignore' or '.slugignore'. + remote: ! 4. Has been added to the Git repository using 'git add --all' + remote: ! and then committed using 'git commit'. + remote: ! + remote: ! In the future we will require the use of a .python-version + remote: ! file and this warning will be made an error. + remote: remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} OUTPUT end @@ -66,7 +106,47 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> No Python version was specified. Using the same major version as the last build: Python 3.12 - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: + remote: ! Warning: No Python version was specified. + remote: ! + remote: ! Your app doesn't specify a Python version and so the buildpack + remote: ! picked a default version for you. + remote: ! + remote: ! Relying on this default version isn't recommended, since it + remote: ! can change over time and may not be consistent with your local + remote: ! development environment, CI or other instances of your app. + remote: ! + remote: ! Please configure an explicit Python version for your app. + remote: ! + remote: ! Create a new file in the root directory of your app named: + remote: ! .python-version + remote: ! + remote: ! Make sure to include the '.' character at the start of the + remote: ! filename. Don't add a file extension such as '.txt'. + remote: ! + remote: ! In the new file, specify your app's major Python version number + remote: ! only. Don't include quotes or a 'python-' prefix. + remote: ! + remote: ! For example, to request the latest version of Python 3.12, + remote: ! update your .python-version file so it contains exactly: + remote: ! 3.12 + remote: ! + remote: ! We strongly recommend that you don't specify the Python patch + remote: ! version number, since it will pin your app to an exact Python + remote: ! version and so stop your app from receiving security updates + remote: ! each time it builds. + remote: ! + remote: ! If your app already has a .python-version file, check that it: + remote: ! + remote: ! 1. Is in the top level directory (not a subdirectory). + remote: ! 2. Is named exactly '.python-version' in all lowercase. + remote: ! 3. Isn't listed in '.gitignore' or '.slugignore'. + remote: ! 4. Has been added to the Git repository using 'git add --all' + remote: ! and then committed using 'git commit'. + remote: ! + remote: ! In the future we will require the use of a .python-version + remote: ! file and this warning will be made an error. + remote: remote: -----> Discarding cache since: remote: - The Python version has changed from 3.12.7 to #{LATEST_PYTHON_3_12} remote: - The pip version has changed from 24.0 to #{PIP_VERSION} @@ -90,7 +170,6 @@ remote: -----> Python app detected remote: -----> Running bin/pre_compile hook remote: -----> No Python version was specified. Using the same major version as the last build: Python 3.8 - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: remote: ! Error: The cached Python version has reached end-of-life. remote: ! @@ -142,7 +221,6 @@ remote: -----> Python app detected remote: -----> Running bin/pre_compile hook remote: -----> No Python version was specified. Using the same major version as the last build: Python 3.99 - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: remote: ! Error: The cached Python version isn't recognised. remote: ! diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index 974e6d607..cc9975386 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -22,7 +22,47 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> No Python version was specified. Using the same major version as the last build: Python 3.12 - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: + remote: ! Warning: No Python version was specified. + remote: ! + remote: ! Your app doesn't specify a Python version and so the buildpack + remote: ! picked a default version for you. + remote: ! + remote: ! Relying on this default version isn't recommended, since it + remote: ! can change over time and may not be consistent with your local + remote: ! development environment, CI or other instances of your app. + remote: ! + remote: ! Please configure an explicit Python version for your app. + remote: ! + remote: ! Create a new file in the root directory of your app named: + remote: ! .python-version + remote: ! + remote: ! Make sure to include the '.' character at the start of the + remote: ! filename. Don't add a file extension such as '.txt'. + remote: ! + remote: ! In the new file, specify your app's major Python version number + remote: ! only. Don't include quotes or a 'python-' prefix. + remote: ! + remote: ! For example, to request the latest version of Python 3.12, + remote: ! update your .python-version file so it contains exactly: + remote: ! 3.12 + remote: ! + remote: ! We strongly recommend that you don't specify the Python patch + remote: ! version number, since it will pin your app to an exact Python + remote: ! version and so stop your app from receiving security updates + remote: ! each time it builds. + remote: ! + remote: ! If your app already has a .python-version file, check that it: + remote: ! + remote: ! 1. Is in the top level directory (not a subdirectory). + remote: ! 2. Is named exactly '.python-version' in all lowercase. + remote: ! 3. Isn't listed in '.gitignore' or '.slugignore'. + remote: ! 4. Has been added to the Git repository using 'git add --all' + remote: ! and then committed using 'git commit'. + remote: ! + remote: ! In the future we will require the use of a .python-version + remote: ! file and this warning will be made an error. + remote: remote: -----> Discarding cache since: remote: - The stack has changed from heroku-22 to heroku-24 remote: - The Python version has changed from 3.12.3 to #{LATEST_PYTHON_3_12} @@ -38,7 +78,7 @@ end context 'when the stack is downgraded from Heroku-24 to Heroku-22', stacks: %w[heroku-24] do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_unspecified') } + let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.13') } it 'clears the cache before installing again' do app.deploy do |app| @@ -48,11 +88,10 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same major version as the last build: Python #{DEFAULT_PYTHON_MAJOR_VERSION} - remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes + remote: -----> Using Python 3.13 specified in .python-version remote: -----> Discarding cache since: remote: - The stack has changed from heroku-24 to heroku-22 - remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) From 7f983c36c241d806c6c354620ac10c724765ad57 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 6 May 2025 11:13:39 +0000 Subject: [PATCH 130/406] Prepare release v283 (#1788) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 99a433e72..03c4405df 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v283] - 2025-05-06 + - Added a warning when an app doesn't specify a Python version and instead relies upon the default/cached version. ([#1787](https://github.com/heroku/heroku-buildpack-python/pull/1787)) - Improved the instructions for migrating from `runtime.txt` to `.python-version`. ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783)) - Improved the error messages shown when `.python-version`, `runtime.txt` or `Pipfile.lock` contain an invalid Python version. ([#1783](https://github.com/heroku/heroku-buildpack-python/pull/1783) and [#1786](https://github.com/heroku/heroku-buildpack-python/pull/1786)) @@ -1203,7 +1206,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v282...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v283...main +[v283]: https://github.com/heroku/heroku-buildpack-python/compare/v282...v283 [v282]: https://github.com/heroku/heroku-buildpack-python/compare/v281...v282 [v281]: https://github.com/heroku/heroku-buildpack-python/compare/v280...v281 [v280]: https://github.com/heroku/heroku-buildpack-python/compare/v279...v280 From c1042d22205cb0c5393ca4526a969c25252c4d81 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 6 May 2025 21:42:09 +0100 Subject: [PATCH 131/406] Fix handling of version files that contain CRLFs (#1789) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From looking at metrics, I noticed that the `cat --show-nonprinting` approach added in #1783 to aid the visualisation of invisible characters (such as ASCII control characters) also escapes carriage return characters (to `^M`), which we don't want. Instead, `sed` is now used which replaces anything not matching the `:print:` and `:space:` groups with the Unicode substitution character (`�`) - which means the classic buildpack now also matches the CNB's behaviour: https://github.com/heroku/buildpacks-python/blob/f2fdd00edf0f63b298cf88c82377885c88666440/src/python_version_file.rs#L16-L19 For a mapping of what `:print:` and `:space:` cover, see the chart at the bottom of this page: https://en.cppreference.com/w/cpp/string/byte/isprint GUS-W-18225347. --- CHANGELOG.md | 1 + lib/python_version.sh | 4 ++-- lib/utils.sh | 9 +++++++++ .../.python-version | Bin 194 -> 379 bytes spec/hatchet/python_version_spec.rb | 4 ++-- 5 files changed, 14 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 03c4405df..8b69a4a8a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Fix parsing of `runtime.txt` and `.python-version` files that contain CRLF characters. ([#1789](https://github.com/heroku/heroku-buildpack-python/pull/1789)) ## [v283] - 2025-05-06 diff --git a/lib/python_version.sh b/lib/python_version.sh index 65bd8770f..8b99ab7ad 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -52,7 +52,7 @@ function python_version::read_requested_python_version() { local runtime_txt_path="${build_dir}/runtime.txt" if [[ -f "${runtime_txt_path}" ]]; then - contents="$(cat --show-nonprinting "${runtime_txt_path}")" + contents="$(utils::read_file_with_special_chars_substituted "${runtime_txt_path}")" version="$(python_version::parse_runtime_txt "${contents}")" origin="runtime.txt" return 0 @@ -60,7 +60,7 @@ function python_version::read_requested_python_version() { local python_version_file_path="${build_dir}/.python-version" if [[ -f "${python_version_file_path}" ]]; then - contents="$(cat --show-nonprinting "${python_version_file_path}")" + contents="$(utils::read_file_with_special_chars_substituted "${python_version_file_path}")" version="$(python_version::parse_python_version_file "${contents}")" origin=".python-version" return 0 diff --git a/lib/utils.sh b/lib/utils.sh index d35cba11d..2345961e4 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -16,6 +16,15 @@ function utils::get_requirement_version() { echo "${requirement_version}" } +# Replaces any invisible unwanted characters (such as ASCII control codes) with the Unicode +# replacement character, so they are visible in error messages. Also removes any carriage +# return characters, to prevent them interfering with the rendering of error messages that +# include the raw file contents. +function utils::read_file_with_special_chars_substituted() { + local file="${1}" + sed --regexp-extended --expression 's/[^[:print:][:space:]]/�/g' --expression 's/\r$//' "${file}" +} + # Python bundles pip within its standard library, which we can use to install our chosen # pip version from PyPI, saving us from having to download the usual pip bootstrap script. function utils::bundled_pip_module_path() { diff --git a/spec/fixtures/python_version_file_invalid_version/.python-version b/spec/fixtures/python_version_file_invalid_version/.python-version index 3877ea21724840691429ce42fd059dd8cd8ecdcf..3a618f7a5803e3afdb1121474ed4a3150573950a 100644 GIT binary patch literal 379 zcmZvXy-ou$5QJ%S(mesqNY{V_2MGm;_Bg62MFJ1toR`>=e@BNdorp^l%i3ag zzMb6$UG$zpW~fBq{W!*yNw;+cc@6pl1$-MRHfZsYuB%2qHt|T1AM+2K4a9d)LBo;K zHtZg9QdZ=hz%~<)Y9JX}SF_o~X0_!a6nPT6_41u@lZ|CTqa%D-DMm3$>lg(JC4+AZ zS+@KzgaakEF(B;ymQn#;PtAL%|E=)WFEPZBQ*;=NN@~cI&bHO(g|=LdECJqX?dWWt eF6=km<)Ob%?bUqIK3)R_JKecxpWFFMUHt*%m2YPN delta 125 zcmey(bck_+q%@ZT5GX6SmZjz?lxF~m%)FG;ypq%uF6D{l=HXmGc_30&2+2rQC`&CW z&dkqK$SciFN-a{zNGw)JR46G*%*@HmOIJwF&nqd)&rwLuNGwWBE=euY18OwZGc?jO TkOmvV#iguZq-U;YXu$;lK Date: Tue, 6 May 2025 20:45:54 +0000 Subject: [PATCH 132/406] Prepare release v284 (#1790) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b69a4a8a..7d66a275f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v284] - 2025-05-06 + - Fix parsing of `runtime.txt` and `.python-version` files that contain CRLF characters. ([#1789](https://github.com/heroku/heroku-buildpack-python/pull/1789)) ## [v283] - 2025-05-06 @@ -1207,7 +1210,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v283...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v284...main +[v284]: https://github.com/heroku/heroku-buildpack-python/compare/v283...v284 [v283]: https://github.com/heroku/heroku-buildpack-python/compare/v282...v283 [v282]: https://github.com/heroku/heroku-buildpack-python/compare/v281...v282 [v281]: https://github.com/heroku/heroku-buildpack-python/compare/v280...v281 From 68c76bad0c4cf5c3d60f48622fdf424f7d85c6f3 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 8 May 2025 14:13:47 +0100 Subject: [PATCH 133/406] Improve escaping of newlines in metrics key-value store (#1792) Since otherwise any field that contains a carriage return character could span multiple lines in the internal metrics data store, which then wouldn't be read back in its entirety by `bin/report`, which would lead to a truncated field in Honeycomb. Such characters are much less likely now after #1789, however, they can still be present in the user-provided input in some cases (that ends up in metrics fields like `failure_detail`), plus from a general correctness point of view, the key-value store's attribute saving functions should be escaping all forms of newline characters. I've also changed the escaping strategy to use literal `\n` and `\r` characters so it's possible to distinguish between multi-line and single line (but space delimited) values more easily in Honeycomb. GUS-W-18471014. --- CHANGELOG.md | 3 ++- lib/kvstore.sh | 6 ++++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d66a275f..6e38d9bbe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,10 +2,11 @@ ## [Unreleased] +- Improved internal buildpack metrics handling of attributes that contain newline characters. ([#1792](https://github.com/heroku/heroku-buildpack-python/pull/1792)) ## [v284] - 2025-05-06 -- Fix parsing of `runtime.txt` and `.python-version` files that contain CRLF characters. ([#1789](https://github.com/heroku/heroku-buildpack-python/pull/1789)) +- Fixed parsing of `runtime.txt` and `.python-version` files that contain CRLF characters. ([#1789](https://github.com/heroku/heroku-buildpack-python/pull/1789)) ## [v283] - 2025-05-06 diff --git a/lib/kvstore.sh b/lib/kvstore.sh index b5c15aa60..a6377f77f 100644 --- a/lib/kvstore.sh +++ b/lib/kvstore.sh @@ -32,8 +32,10 @@ kv_set() { # metadata store or bloating the payload passed back to Vacuole/submitted to Honeycomb given the # extra content in those cases is not normally useful.) local value="${3:0:100}" - # Replace newlines since the data store file format requires that keys don't span multiple lines. - value="${value//$'\n'/ }" + # Replace newlines and carriage returns since the data store file format requires that keys don't + # span multiple lines. + value="${value//$'\n'/\\n}" + value="${value//$'\r'/\\r}" if [[ -f "${f}" ]]; then echo "${key}=${value}" >>"${f}" From ba264cdd021953e1cab51463af1cc87ce9d2bb5d Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 8 May 2025 13:27:10 +0000 Subject: [PATCH 134/406] Prepare release v285 (#1793) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6e38d9bbe..a917f9628 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v285] - 2025-05-08 + - Improved internal buildpack metrics handling of attributes that contain newline characters. ([#1792](https://github.com/heroku/heroku-buildpack-python/pull/1792)) ## [v284] - 2025-05-06 @@ -1211,7 +1214,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v284...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v285...main +[v285]: https://github.com/heroku/heroku-buildpack-python/compare/v284...v285 [v284]: https://github.com/heroku/heroku-buildpack-python/compare/v283...v284 [v283]: https://github.com/heroku/heroku-buildpack-python/compare/v282...v283 [v282]: https://github.com/heroku/heroku-buildpack-python/compare/v281...v282 From a02f99aaaa7c32a6380e97ec6f45582b025edea9 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 13 May 2025 20:12:43 +0000 Subject: [PATCH 135/406] Bump rubocop from 1.75.4 to 1.75.5 in the ruby-dependencies group (#1794) Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.75.4 to 1.75.5 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.75.4...v1.75.5) --- updated-dependencies: - dependency-name: rubocop dependency-version: 1.75.5 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index e7bef30a0..204dea856 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -20,8 +20,8 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.11.3) - language_server-protocol (3.17.0.4) + json (2.12.0) + language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) moneta (1.0.0) @@ -51,7 +51,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.3) - rubocop (1.75.4) + rubocop (1.75.5) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) From 9a5a3000258b4b06969036b87547e1122cdd5f88 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 13 May 2025 21:21:59 +0100 Subject: [PATCH 136/406] Add support for the package manager uv (#1791) Adds support for the package manager uv: https://docs.astral.sh/uv/ And specifically uv's project mode (`uv sync`), which uses a lockfile rather then a `requirements.txt` file: https://docs.astral.sh/uv/guides/projects/ (While uv does have a pip compatible mode that supports requirements files, using a lockfile is a best practice for deploying an app, and not using one means losing most of the benefits of using uv.) Apps must have a `pyproject.toml`, `uv.lock` and `.python-version` file (all of which are created by default by `uv init` followed by either `uv lock` or any of the other commands that sync/update the lockfile). We don't support `runtime.txt` (or omitting the `.python-version` file) when using uv, since otherwise it leads to a much worse UX in several cases. (For example, when the `runtime.txt` version or buildpack default version differs from `requires-python` in the `pyproject.toml` file. Or worse, when `requires-python` happens to match the distro Python in the underlying base image.) For now, if a `requirements.txt`, `Pipfile` or `poetry.lock` are found, they will take precedence over `uv.lock` for backwards compatibility (currently a warning is shown, but in the future this will become an error). This means users of the third-party `heroku-buildpack-uv` will need to remove that buildpack in order to use the new native uv support, since it exports a `requirements.txt` file during the build which will take precedence. For standard (non-Heroku CI) builds, the buildpack installs dependencies using `uv sync --locked --no-default-groups`, to ensure development/test dependencies are not installed. We don't use `--no-dev` instead, since that only excludes the `dev` group, and some apps may be using other group names and have added them to uv's `default-groups` in `pyproject.toml` so that local development workflows work with a plain `uv sync` with no additional args. (PEP 735 doesn't actually prescribe recommended group names, and in fact uses `test` in its example rather than `dev`, so non-standard names may be common.) For Heroku CI builds, the buildpack installs dependencies using `uv sync --locked`, which will install all default groups. See: https://docs.astral.sh/uv/concepts/projects/dependencies/#dependency-groups For now, we don't expose user provided env vars to `uv sync` (similar to the behaviour for other package managers), however, will start doing so as part of addressing #1451 and #1700. uv is installed into the build cache rather than the slug, so is not available at run-time (since it's not typically needed at run-time and doing so reduces the slug size). Both Python and the entrypoints of installed dependencies are available on `PATH`, so use of `uv run` is not required at run-time to use dependencies in the environment. Closes #1616. GUS-W-17310796. --- .github/dependabot.yml | 4 +- .github/workflows/ci.yml | 4 +- CHANGELOG.md | 1 + README.md | 4 +- bin/compile | 13 +- bin/report | 2 +- lib/cache.sh | 9 + lib/package_manager.sh | 35 +- lib/python_version.sh | 54 ++- lib/uv.sh | 165 +++++++ requirements/uv.txt | 1 + spec/fixtures/ci_uv/.python-version | 1 + spec/fixtures/ci_uv/app.json | 9 + spec/fixtures/ci_uv/bin/compile | 12 + spec/fixtures/ci_uv/bin/detect | 7 + spec/fixtures/ci_uv/bin/post_compile | 5 + spec/fixtures/ci_uv/bin/print-env-vars.sh | 5 + spec/fixtures/ci_uv/pyproject.toml | 12 + spec/fixtures/ci_uv/uv.lock | 82 ++++ .../multiple_package_managers/uv.lock | 0 spec/fixtures/uv_basic/.python-version | 1 + spec/fixtures/uv_basic/bin/compile | 21 + spec/fixtures/uv_basic/bin/detect | 7 + spec/fixtures/uv_basic/pyproject.toml | 13 + spec/fixtures/uv_basic/uv.lock | 82 ++++ spec/fixtures/uv_editable/.python-version | 1 + spec/fixtures/uv_editable/bin/compile | 12 + spec/fixtures/uv_editable/bin/detect | 7 + spec/fixtures/uv_editable/bin/post_compile | 5 + .../uv_editable/bin/test-entrypoints.sh | 21 + .../local_package_pyproject_toml/__init__.py | 2 + .../pyproject.toml | 6 + .../local_package_setup_py/__init__.py | 2 + .../packages/local_package_setup_py/setup.cfg | 10 + .../packages/local_package_setup_py/setup.py | 3 + spec/fixtures/uv_editable/pyproject.toml | 25 ++ .../uv_editable/src/uv_editable/__init__.py | 2 + spec/fixtures/uv_editable/uv.lock | 47 ++ .../uv_lockfile_out_of_sync/.python-version | 1 + .../uv_lockfile_out_of_sync/pyproject.toml | 9 + spec/fixtures/uv_lockfile_out_of_sync/uv.lock | 8 + .../.python-version | 2 + .../pyproject.toml | 9 + .../uv_mismatched_python_version/uv.lock | 8 + .../uv_no_python_version_file/pyproject.toml | 9 + .../uv_no_python_version_file/uv.lock | 8 + .../fixtures/uv_oldest_python/.python-version | 1 + spec/fixtures/uv_oldest_python/pyproject.toml | 7 + spec/fixtures/uv_oldest_python/uv.lock | 23 + spec/fixtures/uv_runtime_txt/.python-version | 5 + spec/fixtures/uv_runtime_txt/pyproject.toml | 9 + spec/fixtures/uv_runtime_txt/runtime.txt | 1 + spec/fixtures/uv_runtime_txt/uv.lock | 8 + spec/hatchet/ci_spec.rb | 85 ++++ spec/hatchet/package_manager_spec.rb | 48 ++- spec/hatchet/uv_spec.rb | 406 ++++++++++++++++++ spec/spec_helper.rb | 1 + 57 files changed, 1303 insertions(+), 37 deletions(-) create mode 100644 lib/uv.sh create mode 100644 requirements/uv.txt create mode 100644 spec/fixtures/ci_uv/.python-version create mode 100644 spec/fixtures/ci_uv/app.json create mode 100755 spec/fixtures/ci_uv/bin/compile create mode 100755 spec/fixtures/ci_uv/bin/detect create mode 100644 spec/fixtures/ci_uv/bin/post_compile create mode 100755 spec/fixtures/ci_uv/bin/print-env-vars.sh create mode 100644 spec/fixtures/ci_uv/pyproject.toml create mode 100644 spec/fixtures/ci_uv/uv.lock create mode 100644 spec/fixtures/multiple_package_managers/uv.lock create mode 100644 spec/fixtures/uv_basic/.python-version create mode 100755 spec/fixtures/uv_basic/bin/compile create mode 100755 spec/fixtures/uv_basic/bin/detect create mode 100644 spec/fixtures/uv_basic/pyproject.toml create mode 100644 spec/fixtures/uv_basic/uv.lock create mode 100644 spec/fixtures/uv_editable/.python-version create mode 100755 spec/fixtures/uv_editable/bin/compile create mode 100755 spec/fixtures/uv_editable/bin/detect create mode 100755 spec/fixtures/uv_editable/bin/post_compile create mode 100755 spec/fixtures/uv_editable/bin/test-entrypoints.sh create mode 100644 spec/fixtures/uv_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py create mode 100644 spec/fixtures/uv_editable/packages/local_package_pyproject_toml/pyproject.toml create mode 100644 spec/fixtures/uv_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py create mode 100644 spec/fixtures/uv_editable/packages/local_package_setup_py/setup.cfg create mode 100644 spec/fixtures/uv_editable/packages/local_package_setup_py/setup.py create mode 100644 spec/fixtures/uv_editable/pyproject.toml create mode 100644 spec/fixtures/uv_editable/src/uv_editable/__init__.py create mode 100644 spec/fixtures/uv_editable/uv.lock create mode 100644 spec/fixtures/uv_lockfile_out_of_sync/.python-version create mode 100644 spec/fixtures/uv_lockfile_out_of_sync/pyproject.toml create mode 100644 spec/fixtures/uv_lockfile_out_of_sync/uv.lock create mode 100644 spec/fixtures/uv_mismatched_python_version/.python-version create mode 100644 spec/fixtures/uv_mismatched_python_version/pyproject.toml create mode 100644 spec/fixtures/uv_mismatched_python_version/uv.lock create mode 100644 spec/fixtures/uv_no_python_version_file/pyproject.toml create mode 100644 spec/fixtures/uv_no_python_version_file/uv.lock create mode 100644 spec/fixtures/uv_oldest_python/.python-version create mode 100644 spec/fixtures/uv_oldest_python/pyproject.toml create mode 100644 spec/fixtures/uv_oldest_python/uv.lock create mode 100644 spec/fixtures/uv_runtime_txt/.python-version create mode 100644 spec/fixtures/uv_runtime_txt/pyproject.toml create mode 100644 spec/fixtures/uv_runtime_txt/runtime.txt create mode 100644 spec/fixtures/uv_runtime_txt/uv.lock create mode 100644 spec/hatchet/uv_spec.rb diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 942ea5fa5..ee3475f73 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -24,7 +24,9 @@ updates: - package-ecosystem: "pip" directory: "/" schedule: - interval: "monthly" + # We set this to a more frequent interval than the above since uv is still under rapid + # iteration, and we'll generally want to be on a recent (if not the latest) version. + interval: "weekly" ignore: # We're not updating to setuptools v71+ due to its new approach to vendored dependencies: # https://github.com/heroku/heroku-buildpack-python/pull/1630#issuecomment-2324236653 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c14d400f6..90310deb8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -47,8 +47,8 @@ jobs: HEROKU_API_KEY: ${{ secrets.HEROKU_API_KEY }} HEROKU_API_USER: ${{ secrets.HEROKU_API_USER }} HEROKU_DISABLE_AUTOUPDATE: 1 - PARALLEL_SPLIT_TEST_PROCESSES: 75 - RSPEC_RETRY_RETRY_COUNT: 2 + PARALLEL_SPLIT_TEST_PROCESSES: 85 + RSPEC_RETRY_RETRY_COUNT: 1 steps: - name: Checkout uses: actions/checkout@v4 diff --git a/CHANGELOG.md b/CHANGELOG.md index a917f9628..8569fc0e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Added support for the package manager uv. ([#1791](https://github.com/heroku/heroku-buildpack-python/pull/1791)) ## [v285] - 2025-05-08 diff --git a/README.md b/README.md index 557d77e6d..7c2733023 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,9 @@ See the [Getting Started on Heroku with Python](https://devcenter.heroku.com/art ## Application Requirements -A `requirements.txt`, `Pipfile` or `poetry.lock` file must be present in the root (top-level) directory of your app's source code. +A `requirements.txt`, `Pipfile`, `poetry.lock`, or `uv.lock` file must be present in the root (top-level) directory of your app's source code. + +When using the package manager [uv](https://docs.astral.sh/uv/) a `.python-version` file is also required. ## Configuration diff --git a/bin/compile b/bin/compile index b052052db..f938acef9 100755 --- a/bin/compile +++ b/bin/compile @@ -31,6 +31,7 @@ source "${BUILDPACK_DIR}/lib/pipenv.sh" source "${BUILDPACK_DIR}/lib/poetry.sh" source "${BUILDPACK_DIR}/lib/python_version.sh" source "${BUILDPACK_DIR}/lib/python.sh" +source "${BUILDPACK_DIR}/lib/uv.sh" compile_start_time=$(nowms) @@ -143,7 +144,7 @@ fi # as to what was a warning vs an error. In addition, several of the error messages contain similar # content to the warnings (such as encouraging use of .python-version and major version syntax), # which would mean duplicate content if we showed both. -python_version::warn_if_python_version_file_missing "${python_version_origin}" "${python_major_version}" +python_version::warn_or_error_if_python_version_file_missing "${python_version_origin}" "${python_major_version}" python_version::warn_if_deprecated_major_version "${python_major_version}" "${python_version_origin}" python_version::warn_if_patch_update_available "${python_full_version}" "${python_major_version}" "${python_version_origin}" @@ -180,6 +181,9 @@ case "${package_manager}" in poetry) poetry::install_poetry "${python_home}" "${python_major_version}" "${CACHE_DIR}" "${EXPORT_PATH}" ;; + uv) + uv::install_uv "${CACHE_DIR}" "${EXPORT_PATH}" "${python_home}" + ;; *) utils::abort_internal_error "Unhandled package manager: ${package_manager}" ;; @@ -189,8 +193,8 @@ meta_time "package_manager_install_duration" "${package_manager_install_start_ti # SQLite3 support. # Installs the sqlite3 dev headers and sqlite3 binary but not the # libsqlite3-0 library since that exists in the base image. -# We skip this step on Python 3.13 or when using Poetry, as a first step towards removing this feature. -if [[ "${python_major_version}" == +(3.9|3.10|3.11|3.12) && "${package_manager}" != "poetry" ]]; then +# We skip this step on Python 3.13 or when using Poetry/uv, as a first step towards removing this feature. +if [[ "${python_major_version}" == +(3.9|3.10|3.11|3.12) && "${package_manager}" != +(poetry|uv) ]]; then install_sqlite_start_time=$(nowms) source "${BUILDPACK_DIR}/bin/steps/sqlite3" buildpack_sqlite3_install @@ -209,6 +213,9 @@ case "${package_manager}" in poetry) poetry::install_dependencies ;; + uv) + uv::install_dependencies + ;; *) utils::abort_internal_error "Unhandled package manager: ${package_manager}" ;; diff --git a/bin/report b/bin/report index 62d4a9183..85ab72f94 100755 --- a/bin/report +++ b/bin/report @@ -76,6 +76,7 @@ STRING_FIELDS=( python_version_requested python_version setuptools_version + uv_version wheel_version ) @@ -101,7 +102,6 @@ ALL_OTHER_FIELDS=( setup_py_only sqlite_install_duration total_duration - uv_lockfile ) for field in "${STRING_FIELDS[@]}"; do diff --git a/lib/cache.sh b/lib/cache.sh index f5055b982..bdd6c543d 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -115,6 +115,14 @@ function cache::restore() { cache_invalidation_reasons+=("The Poetry version has changed from ${cached_poetry_version:-"unknown"} to ${POETRY_VERSION}") fi ;; + uv) + local cached_uv_version + cached_uv_version="$(meta_prev_get "uv_version")" + # uv support was added after the metadata store, so we'll always have the version here. + if [[ "${cached_uv_version}" != "${UV_VERSION:?}" ]]; then + cache_invalidation_reasons+=("The uv version has changed from ${cached_uv_version:-"unknown"} to ${UV_VERSION}") + fi + ;; *) utils::abort_internal_error "Unhandled package manager: ${package_manager}" ;; @@ -132,6 +140,7 @@ function cache::restore() { "${cache_dir}/.heroku/python" \ "${cache_dir}/.heroku/python-poetry" \ "${cache_dir}/.heroku/python-stack" \ + "${cache_dir}/.heroku/python-uv" \ "${cache_dir}/.heroku/python-version" \ "${cache_dir}/.heroku/requirements.txt" diff --git a/lib/package_manager.sh b/lib/package_manager.sh index c1b8d8e76..091600e16 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -49,6 +49,11 @@ function package_manager::determine_package_manager() { package_managers_found_display_text+=("poetry.lock (Poetry)") fi + if [[ -f "${build_dir}/uv.lock" ]]; then + package_managers_found+=(uv) + package_managers_found_display_text+=("uv.lock (uv)") + fi + # TODO: Deprecate/sunset this fallback, since using setup.py declared dependencies is # not a best practice, and we can only guess as to which package manager to use. if ((${#package_managers_found[@]} == 0)) && [[ -f "${build_dir}/setup.py" ]]; then @@ -59,10 +64,6 @@ function package_manager::determine_package_manager() { meta_set "setup_py_only" "false" fi - if [[ -f "${build_dir}/uv.lock" ]]; then - meta_set "uv_lockfile" "true" - fi - local num_package_managers_found=${#package_managers_found[@]} case "${num_package_managers_found}" in @@ -75,8 +76,8 @@ function package_manager::determine_package_manager() { Error: Couldn't find any supported Python package manager files. A Python app on Heroku must have either a 'requirements.txt', - 'Pipfile' or 'poetry.lock' package manager file in the root - directory of its source code. + 'Pipfile', 'poetry.lock' or 'uv.lock' package manager file in + the root directory of its source code. Currently the root directory of your app contains: @@ -93,10 +94,10 @@ function package_manager::determine_package_manager() { Otherwise, add a package manager file to your app. If your app has no dependencies, then create an empty 'requirements.txt' file. - If you would like to see support for the package manager uv, - please vote and comment on these GitHub issues: - https://github.com/heroku/heroku-buildpack-python/issues/1616 - https://github.com/heroku/roadmap/issues/323 + If you aren't sure which package manager to use, we recommend + trying uv, since it supports lockfiles, is extremely fast, and + is actively maintained by a full-time team: + https://docs.astral.sh/uv/ For help with using Python on Heroku, see: https://devcenter.heroku.com/articles/getting-started-with-python @@ -127,6 +128,11 @@ function package_manager::determine_package_manager() { Decide which package manager you want to use with your app, and then delete the file(s) and any config from the others. + + If you aren't sure which package manager to use, we recommend + trying uv, since it supports lockfiles, is extremely fast, and + is actively maintained by a full-time team: + https://docs.astral.sh/uv/ EOF if [[ "${package_managers_found[*]}" == *"poetry"* ]]; then @@ -138,6 +144,15 @@ function package_manager::determine_package_manager() { EOF fi + if [[ "${package_managers_found[*]}" == *"uv"* ]]; then + output::notice <<-EOF + Note: We recently added support for the package manager uv. + If you are using a third-party uv buildpack you must remove + it, otherwise the requirements.txt file it generates will cause + the warning above. + EOF + fi + meta_set "package_manager_multiple_found" "$( IFS=, echo "${package_managers_found[*]}" diff --git a/lib/python_version.sh b/lib/python_version.sh index 8b99ab7ad..41af8a816 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -419,7 +419,7 @@ function python_version::resolve_python_version() { esac } -function python_version::warn_if_python_version_file_missing() { +function python_version::warn_or_error_if_python_version_file_missing() { local python_version_origin="${1}" local python_major_version="${2}" @@ -432,6 +432,31 @@ function python_version::warn_if_python_version_file_missing() { case "${python_version_origin}" in default | cached) + if [[ "${package_manager}" == "uv" ]]; then + output::error <<-EOF + Error: No Python version was specified. + + When using the package manager uv on Heroku, you must specify + your app's Python version with a .python-version file. + + To add a .python-version file: + + 1. Make sure you are in the root directory of your app + and not a subdirectory. + 2. Run 'uv python pin ${python_major_version}' + (adjust to match your app's major Python version). + 3. Commit the changes to your Git repository using + 'git add --all' and then 'git commit'. + + Note: We strongly recommend that you don't specify the Python + patch version number in your .python-version file, since it will + pin your app to an exact Python version and so stop your app from + receiving security updates each time it builds. + EOF + meta_set "failure_reason" "python-version-file::not-found" + exit 1 + fi + output::warning <<-EOF Warning: No Python version was specified. @@ -459,6 +484,33 @@ function python_version::warn_if_python_version_file_missing() { EOF ;; runtime.txt) + if [[ "${package_manager}" == "uv" ]]; then + output::error <<-EOF + Error: The runtime.txt file isn't supported when using uv. + + When using the package manager uv on Heroku, you must specify + your app's Python version with a .python-version file and not + a runtime.txt file. + + To switch to a .python-version file: + + 1. Make sure you are in the root directory of your app + and not a subdirectory. + 2. Delete your runtime.txt file. + 3. Run 'uv python pin ${python_major_version}' + (adjust to match your app's major Python version). + 4. Commit the changes to your Git repository using + 'git add --all' and then 'git commit'. + + Note: We strongly recommend that you don't specify the Python + patch version number in your .python-version file, since it will + pin your app to an exact Python version and so stop your app from + receiving security updates each time it builds. + EOF + meta_set "failure_reason" "runtime-txt::not-supported" + exit 1 + fi + output::warning <<-EOF Warning: The runtime.txt file is deprecated. diff --git a/lib/uv.sh b/lib/uv.sh new file mode 100644 index 000000000..c8e48a72a --- /dev/null +++ b/lib/uv.sh @@ -0,0 +1,165 @@ +#!/usr/bin/env bash + +# This is technically redundant, since all consumers of this lib will have enabled these, +# however, it helps Shellcheck realise the options under which these functions will run. +set -euo pipefail + +UV_VERSION=$(utils::get_requirement_version 'uv') + +function uv::install_uv() { + local cache_dir="${1}" + local export_file="${2}" + local python_home="${3}" + + # We store uv in the build cache, since we only need it during the build. + local uv_dir="${cache_dir}/.heroku/python-uv" + + meta_set "uv_version" "${UV_VERSION}" + + # The earlier buildpack cache invalidation step will have already handled the case where + # the uv version has changed, so here we only need to check whether the uv binary exists. + if [[ -f "${uv_dir}/uv" ]]; then + output::step "Using cached uv ${UV_VERSION}" + else + output::step "Installing uv ${UV_VERSION}" + mkdir -p "${uv_dir}" + + local gnu_arch + # eg: `x86_64` or `aarch64`. + gnu_arch=$(arch) + local uv_url="https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-${gnu_arch}-unknown-linux-gnu.tar.gz" + + if ! { + # We set max-time for improved UX for hanging downloads compared to relying on the build system + # timeout. The uv archive is only ~15 MB so takes < 1s to download on Heroku's build system, + # however, we use much higher timeouts so that the buildpack works in non-Heroku or local + # environments that may have a slower connection. We don't use `--speed-limit` since it gives + # worse error messages when used with retries and piping to tar. + # We have to use `--strip-components` since the archive contents are nested under a subdirectory. + curl \ + --connect-timeout 10 \ + --fail \ + --location \ + --max-time 120 \ + --retry-max-time 120 \ + --retry 3 \ + --retry-connrefused \ + --show-error \ + --silent \ + "${uv_url}" \ + | tar \ + --directory "${uv_dir}" \ + --extract \ + --gzip \ + --no-anchored \ + --strip-components 1 + }; then + output::error <<-EOF + Error: Unable to install uv. + + Failed to download/install uv from GitHub: + ${uv_url} + + In some cases, this happens due to a temporary issue with + the network connection or GitHub's API/CDN. + + Try building again to see if the error resolves itself. + + If that doesn't help, check the status of GitHub here: + https://www.githubstatus.com + EOF + meta_set "failure_reason" "install-package-manager::uv" + exit 1 + fi + fi + + export PATH="${uv_dir}:${PATH}" + # Make uv manage the system site-packages of our Python install instead of creating a venv. + export UV_PROJECT_ENVIRONMENT="${python_home}" + # Prevent uv from downloading/using its own Python installation. + export UV_NO_MANAGED_PYTHON="1" + export UV_PYTHON_DOWNLOADS="never" + + # Set the same env vars in the environment used by later buildpacks. + cat >>"${export_file}" <<-EOF + export PATH="${uv_dir}:\${PATH}" + export UV_PROJECT_ENVIRONMENT="${python_home}" + export UV_NO_MANAGED_PYTHON="1" + export UV_PYTHON_DOWNLOADS="never" + EOF + + # As a performance optimisation, uv attempts to use hardlinks instead of copying files from its + # download cache into site-packages, and will emit a warning if it has to fall back to copying. + # By default uv stores its cache under `$HOME/.cache`, and for standard Heroku builds `$HOME` is + # `/app`, which is on a different filesystem mount to the build directory (which is under `/tmp`), + # meaning hardlinks can't be used. To avoid this we tell uv to store its cache in `/tmp`. + # However, we have to do so conditionally, since for Heroku CI both the home directory and + # the build directory are `/app`, where hardlinks already work and changing the cache to `/tmp` + # would instead break them. + # + # There's also a third case, a non-CI build where the app has the undocumented `build-in-app-dir` + # labs enabled, however, for that scenario the build directory is `/app` and the home directory + # is `/tmp`, so we can't use hardlinks unless we wrote the cache to the build directory and + # manually deleted it after. For now we ignore this case since not many apps use that labs, + # and uv will still work after falling back to copying files (with a warning). + # + # Longer term, ideally uv's `--no-cache` option would co-locate the temporary cache it writes + # alongside site-packages (see https://github.com/astral-sh/uv/issues/11385), and we could use + # that everywhere since we don't actually need to persist uv's cache (see below). + # shellcheck disable=SC2312 # Invoke this command separately to avoid masking its return value. + if [[ "$(realpath "${python_home}")" =~ ^/tmp/ ]]; then + export UV_CACHE_DIR='/tmp/uv-cache' + echo 'export UV_CACHE_DIR="/tmp/uv-cache"' >>"${export_file}" + fi +} + +# Note: We cache site-packages since: +# - It results in faster builds than only caching uv's download/wheel cache. +# - It improves the UX of the build log, since uv will display which packages were +# added/removed since the last successful build. +# - It's safe to do so, since `uv sync` fully manages the environment (including +# e.g. uninstalling packages when they are removed from the lockfile). +# +# With site-packages cached there is no need to persist uv's cache in the build cache, so we let +# uv write it to the home directory (or `UV_CACHE_DIR`, see above) where it will be discarded at +# the end of the build. We don't use `--no-cache` since all it does is make uv switch to a temporary +# cache directory under `/tmp`, which would mean hardlinks can't be used for Heroku CI (see above). +function uv::install_dependencies() { + local uv_install_command=( + uv + sync + --locked + ) + + # Unless we're building on Heroku CI, we omit the default dependency groups (such as `dev`): + # https://docs.astral.sh/uv/concepts/projects/dependencies/#dependency-groups + if [[ ! -v INSTALL_TEST ]]; then + uv_install_command+=(--no-default-groups) + fi + + # We only display the most relevant command args here, to improve the signal to noise ratio. + output::step "Installing dependencies using '${uv_install_command[*]}'" + + # TODO: Expose app config vars to the install command as part of doing so for all package managers. + # `--compile-bytecode`: Improves app boot times (pip does this by default). + # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. + if ! { + "${uv_install_command[@]}" \ + --color always \ + --compile-bytecode \ + --no-progress \ + |& tee "${WARNINGS_LOG:?}" \ + |& output::indent + }; then + # TODO: Overhaul warnings and combine them with error handling (for all package managers). + show-warnings + + output::error <<-EOF + Error: Unable to install dependencies using uv. + + See the log output above for more information. + EOF + meta_set "failure_reason" "install-dependencies::uv" + exit 1 + fi +} diff --git a/requirements/uv.txt b/requirements/uv.txt new file mode 100644 index 000000000..48ea67dbd --- /dev/null +++ b/requirements/uv.txt @@ -0,0 +1 @@ +uv==0.7.3 diff --git a/spec/fixtures/ci_uv/.python-version b/spec/fixtures/ci_uv/.python-version new file mode 100644 index 000000000..24ee5b1be --- /dev/null +++ b/spec/fixtures/ci_uv/.python-version @@ -0,0 +1 @@ +3.13 diff --git a/spec/fixtures/ci_uv/app.json b/spec/fixtures/ci_uv/app.json new file mode 100644 index 000000000..59617c169 --- /dev/null +++ b/spec/fixtures/ci_uv/app.json @@ -0,0 +1,9 @@ +{ + "environments": { + "test": { + "scripts": { + "test": "./bin/print-env-vars.sh && pytest --version" + } + } + } +} diff --git a/spec/fixtures/ci_uv/bin/compile b/spec/fixtures/ci_uv/bin/compile new file mode 100755 index 000000000..fcb7055e3 --- /dev/null +++ b/spec/fixtures/ci_uv/bin/compile @@ -0,0 +1,12 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack, and tests that the environment is +# configured as expected for buildpacks that run after the Python buildpack. + +set -euo pipefail + +BUILD_DIR="${1}" + +cd "${BUILD_DIR}" + +exec bin/print-env-vars.sh diff --git a/spec/fixtures/ci_uv/bin/detect b/spec/fixtures/ci_uv/bin/detect new file mode 100755 index 000000000..68cdcc4a2 --- /dev/null +++ b/spec/fixtures/ci_uv/bin/detect @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack. + +set -euo pipefail + +echo "Inline" diff --git a/spec/fixtures/ci_uv/bin/post_compile b/spec/fixtures/ci_uv/bin/post_compile new file mode 100644 index 000000000..15362f6b1 --- /dev/null +++ b/spec/fixtures/ci_uv/bin/post_compile @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -euo pipefail + +exec bin/print-env-vars.sh diff --git a/spec/fixtures/ci_uv/bin/print-env-vars.sh b/spec/fixtures/ci_uv/bin/print-env-vars.sh new file mode 100755 index 000000000..9e0bebe6b --- /dev/null +++ b/spec/fixtures/ci_uv/bin/print-env-vars.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -euo pipefail + +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|BUILD_DIR|CACHE_DIR|CI_NODE_.+|DYNO|ENV_DIR|HEROKU_TEST_RUN_.+|HOME|OLDPWD|PORT|PWD|SHLVL|STACK|TERM)=' diff --git a/spec/fixtures/ci_uv/pyproject.toml b/spec/fixtures/ci_uv/pyproject.toml new file mode 100644 index 000000000..9be1eb818 --- /dev/null +++ b/spec/fixtures/ci_uv/pyproject.toml @@ -0,0 +1,12 @@ +[project] +name = "ci-uv" +version = "0.0.0" +requires-python = ">=3.13" +dependencies = [ + "typing-extensions", +] + +[dependency-groups] +dev = [ + "pytest", +] diff --git a/spec/fixtures/ci_uv/uv.lock b/spec/fixtures/ci_uv/uv.lock new file mode 100644 index 000000000..ffd69a888 --- /dev/null +++ b/spec/fixtures/ci_uv/uv.lock @@ -0,0 +1,82 @@ +version = 1 +revision = 2 +requires-python = ">=3.13" + +[[package]] +name = "ci-uv" +version = "0.0.0" +source = { virtual = "." } +dependencies = [ + { name = "typing-extensions" }, +] + +[package.dev-dependencies] +dev = [ + { name = "pytest" }, +] + +[package.metadata] +requires-dist = [{ name = "typing-extensions" }] + +[package.metadata.requires-dev] +dev = [{ name = "pytest" }] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "iniconfig" +version = "2.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f2/97/ebf4da567aa6827c909642694d71c9fcf53e5b504f2d96afea02718862f3/iniconfig-2.1.0.tar.gz", hash = "sha256:3abbd2e30b36733fee78f9c7f7308f2d0050e88f0087fd25c2645f63c773e1c7", size = 4793, upload-time = "2025-03-19T20:09:59.721Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2c/e1/e6716421ea10d38022b952c159d5161ca1193197fb744506875fbb87ea7b/iniconfig-2.1.0-py3-none-any.whl", hash = "sha256:9deba5723312380e77435581c6bf4935c94cbfab9b1ed33ef8d238ea168eb760", size = 6050, upload-time = "2025-03-19T20:10:01.071Z" }, +] + +[[package]] +name = "packaging" +version = "25.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a1/d4/1fc4078c65507b51b96ca8f8c3ba19e6a61c8253c72794544580a7b6c24d/packaging-25.0.tar.gz", hash = "sha256:d443872c98d677bf60f6a1f2f8c1cb748e8fe762d2bf9d3148b5599295b0fc4f", size = 165727, upload-time = "2025-04-19T11:48:59.673Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/20/12/38679034af332785aac8774540895e234f4d07f7545804097de4b666afd8/packaging-25.0-py3-none-any.whl", hash = "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484", size = 66469, upload-time = "2025-04-19T11:48:57.875Z" }, +] + +[[package]] +name = "pluggy" +version = "1.5.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/96/2d/02d4312c973c6050a18b314a5ad0b3210edb65a906f868e31c111dede4a6/pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1", size = 67955, upload-time = "2024-04-20T21:34:42.531Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/88/5f/e351af9a41f866ac3f1fac4ca0613908d9a41741cfcf2228f4ad853b697d/pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669", size = 20556, upload-time = "2024-04-20T21:34:40.434Z" }, +] + +[[package]] +name = "pytest" +version = "8.3.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ae/3c/c9d525a414d506893f0cd8a8d0de7706446213181570cdbd766691164e40/pytest-8.3.5.tar.gz", hash = "sha256:f4efe70cc14e511565ac476b57c279e12a855b11f48f212af1080ef2263d3845", size = 1450891, upload-time = "2025-03-02T12:54:54.503Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/30/3d/64ad57c803f1fa1e963a7946b6e0fea4a70df53c1a7fed304586539c2bac/pytest-8.3.5-py3-none-any.whl", hash = "sha256:c69214aa47deac29fad6c2a4f590b9c4a9fdb16a403176fe154b79c0b4d4d820", size = 343634, upload-time = "2025-03-02T12:54:52.069Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.13.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/37/23083fcd6e35492953e8d2aaaa68b860eb422b34627b13f2ce3eb6106061/typing_extensions-4.13.2.tar.gz", hash = "sha256:e6c81219bd689f51865d9e372991c540bda33a0379d5573cddb9a3a23f7caaef", size = 106967, upload-time = "2025-04-10T14:19:05.416Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8b/54/b1ae86c0973cc6f0210b53d508ca3641fb6d0c56823f288d108bc7ab3cc8/typing_extensions-4.13.2-py3-none-any.whl", hash = "sha256:a439e7c04b49fec3e5d3e2beaa21755cadbbdc391694e28ccdd36ca4a1408f8c", size = 45806, upload-time = "2025-04-10T14:19:03.967Z" }, +] diff --git a/spec/fixtures/multiple_package_managers/uv.lock b/spec/fixtures/multiple_package_managers/uv.lock new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/uv_basic/.python-version b/spec/fixtures/uv_basic/.python-version new file mode 100644 index 000000000..24ee5b1be --- /dev/null +++ b/spec/fixtures/uv_basic/.python-version @@ -0,0 +1 @@ +3.13 diff --git a/spec/fixtures/uv_basic/bin/compile b/spec/fixtures/uv_basic/bin/compile new file mode 100755 index 000000000..8b5578776 --- /dev/null +++ b/spec/fixtures/uv_basic/bin/compile @@ -0,0 +1,21 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack, and tests that the environment is +# configured as expected for buildpacks that run after the Python buildpack. + +set -euo pipefail + +BUILD_DIR="${1}" + +cd "${BUILD_DIR}" + +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|OLDPWD|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK)=' +echo + +python -c 'import pprint, sys; pprint.pp(sys.path)' +echo + +uv pip list +echo + +python -c 'import typing_extensions; print(typing_extensions)' diff --git a/spec/fixtures/uv_basic/bin/detect b/spec/fixtures/uv_basic/bin/detect new file mode 100755 index 000000000..68cdcc4a2 --- /dev/null +++ b/spec/fixtures/uv_basic/bin/detect @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack. + +set -euo pipefail + +echo "Inline" diff --git a/spec/fixtures/uv_basic/pyproject.toml b/spec/fixtures/uv_basic/pyproject.toml new file mode 100644 index 000000000..ed1942882 --- /dev/null +++ b/spec/fixtures/uv_basic/pyproject.toml @@ -0,0 +1,13 @@ +[project] +name = "uv-basic" +version = "0.0.0" +requires-python = ">=3.13" +dependencies = [ + "typing-extensions", +] + +[dependency-groups] +# This group shouldn't be installed due to us passing `--no-default-groups`. +dev = [ + "pytest", +] diff --git a/spec/fixtures/uv_basic/uv.lock b/spec/fixtures/uv_basic/uv.lock new file mode 100644 index 000000000..87eb42270 --- /dev/null +++ b/spec/fixtures/uv_basic/uv.lock @@ -0,0 +1,82 @@ +version = 1 +revision = 2 +requires-python = ">=3.13" + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "iniconfig" +version = "2.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f2/97/ebf4da567aa6827c909642694d71c9fcf53e5b504f2d96afea02718862f3/iniconfig-2.1.0.tar.gz", hash = "sha256:3abbd2e30b36733fee78f9c7f7308f2d0050e88f0087fd25c2645f63c773e1c7", size = 4793, upload-time = "2025-03-19T20:09:59.721Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2c/e1/e6716421ea10d38022b952c159d5161ca1193197fb744506875fbb87ea7b/iniconfig-2.1.0-py3-none-any.whl", hash = "sha256:9deba5723312380e77435581c6bf4935c94cbfab9b1ed33ef8d238ea168eb760", size = 6050, upload-time = "2025-03-19T20:10:01.071Z" }, +] + +[[package]] +name = "packaging" +version = "25.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a1/d4/1fc4078c65507b51b96ca8f8c3ba19e6a61c8253c72794544580a7b6c24d/packaging-25.0.tar.gz", hash = "sha256:d443872c98d677bf60f6a1f2f8c1cb748e8fe762d2bf9d3148b5599295b0fc4f", size = 165727, upload-time = "2025-04-19T11:48:59.673Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/20/12/38679034af332785aac8774540895e234f4d07f7545804097de4b666afd8/packaging-25.0-py3-none-any.whl", hash = "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484", size = 66469, upload-time = "2025-04-19T11:48:57.875Z" }, +] + +[[package]] +name = "pluggy" +version = "1.5.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/96/2d/02d4312c973c6050a18b314a5ad0b3210edb65a906f868e31c111dede4a6/pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1", size = 67955, upload-time = "2024-04-20T21:34:42.531Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/88/5f/e351af9a41f866ac3f1fac4ca0613908d9a41741cfcf2228f4ad853b697d/pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669", size = 20556, upload-time = "2024-04-20T21:34:40.434Z" }, +] + +[[package]] +name = "pytest" +version = "8.3.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ae/3c/c9d525a414d506893f0cd8a8d0de7706446213181570cdbd766691164e40/pytest-8.3.5.tar.gz", hash = "sha256:f4efe70cc14e511565ac476b57c279e12a855b11f48f212af1080ef2263d3845", size = 1450891, upload-time = "2025-03-02T12:54:54.503Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/30/3d/64ad57c803f1fa1e963a7946b6e0fea4a70df53c1a7fed304586539c2bac/pytest-8.3.5-py3-none-any.whl", hash = "sha256:c69214aa47deac29fad6c2a4f590b9c4a9fdb16a403176fe154b79c0b4d4d820", size = 343634, upload-time = "2025-03-02T12:54:52.069Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.13.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/37/23083fcd6e35492953e8d2aaaa68b860eb422b34627b13f2ce3eb6106061/typing_extensions-4.13.2.tar.gz", hash = "sha256:e6c81219bd689f51865d9e372991c540bda33a0379d5573cddb9a3a23f7caaef", size = 106967, upload-time = "2025-04-10T14:19:05.416Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8b/54/b1ae86c0973cc6f0210b53d508ca3641fb6d0c56823f288d108bc7ab3cc8/typing_extensions-4.13.2-py3-none-any.whl", hash = "sha256:a439e7c04b49fec3e5d3e2beaa21755cadbbdc391694e28ccdd36ca4a1408f8c", size = 45806, upload-time = "2025-04-10T14:19:03.967Z" }, +] + +[[package]] +name = "uv-basic" +version = "0.0.0" +source = { virtual = "." } +dependencies = [ + { name = "typing-extensions" }, +] + +[package.dev-dependencies] +dev = [ + { name = "pytest" }, +] + +[package.metadata] +requires-dist = [{ name = "typing-extensions" }] + +[package.metadata.requires-dev] +dev = [{ name = "pytest" }] diff --git a/spec/fixtures/uv_editable/.python-version b/spec/fixtures/uv_editable/.python-version new file mode 100644 index 000000000..24ee5b1be --- /dev/null +++ b/spec/fixtures/uv_editable/.python-version @@ -0,0 +1 @@ +3.13 diff --git a/spec/fixtures/uv_editable/bin/compile b/spec/fixtures/uv_editable/bin/compile new file mode 100755 index 000000000..df17e9401 --- /dev/null +++ b/spec/fixtures/uv_editable/bin/compile @@ -0,0 +1,12 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack, and tests that editable requirements are +# usable by buildpacks that run after the Python buildpack during the build. + +set -euo pipefail + +BUILD_DIR="${1}" + +cd "${BUILD_DIR}" + +exec bin/test-entrypoints.sh diff --git a/spec/fixtures/uv_editable/bin/detect b/spec/fixtures/uv_editable/bin/detect new file mode 100755 index 000000000..68cdcc4a2 --- /dev/null +++ b/spec/fixtures/uv_editable/bin/detect @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +# This file is run by the inline buildpack. + +set -euo pipefail + +echo "Inline" diff --git a/spec/fixtures/uv_editable/bin/post_compile b/spec/fixtures/uv_editable/bin/post_compile new file mode 100755 index 000000000..6e77d159a --- /dev/null +++ b/spec/fixtures/uv_editable/bin/post_compile @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -euo pipefail + +exec bin/test-entrypoints.sh diff --git a/spec/fixtures/uv_editable/bin/test-entrypoints.sh b/spec/fixtures/uv_editable/bin/test-entrypoints.sh new file mode 100755 index 000000000..20085b398 --- /dev/null +++ b/spec/fixtures/uv_editable/bin/test-entrypoints.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash + +set -euo pipefail + +cd .heroku/python/lib/python*/site-packages/ + +# List any path like strings in the .pth, and finder files in site-packages. +grep --extended-regexp --only-matching -- '/\S+' *.pth __editable___*_finder.py | sort +echo + +echo -n "Running entrypoint for the current package: " +uv-editable + +echo -n "Running entrypoint for the pyproject.toml-based local package: " +local_package_pyproject_toml + +echo -n "Running entrypoint for the setup.py-based local package: " +local_package_setup_py + +echo -n "Running entrypoint for the VCS package: " +gunicorn --version diff --git a/spec/fixtures/uv_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py b/spec/fixtures/uv_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py new file mode 100644 index 000000000..6ce02c341 --- /dev/null +++ b/spec/fixtures/uv_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py @@ -0,0 +1,2 @@ +def hello(): + print("Hello from pyproject.toml!") diff --git a/spec/fixtures/uv_editable/packages/local_package_pyproject_toml/pyproject.toml b/spec/fixtures/uv_editable/packages/local_package_pyproject_toml/pyproject.toml new file mode 100644 index 000000000..f567c3777 --- /dev/null +++ b/spec/fixtures/uv_editable/packages/local_package_pyproject_toml/pyproject.toml @@ -0,0 +1,6 @@ +[project] +name = "local_package_pyproject_toml" +version = "0.0.1" + +[project.scripts] +local_package_pyproject_toml = "local_package_pyproject_toml:hello" diff --git a/spec/fixtures/uv_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py b/spec/fixtures/uv_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py new file mode 100644 index 000000000..8dccb8a0c --- /dev/null +++ b/spec/fixtures/uv_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py @@ -0,0 +1,2 @@ +def hello(): + print("Hello from setup.py!") diff --git a/spec/fixtures/uv_editable/packages/local_package_setup_py/setup.cfg b/spec/fixtures/uv_editable/packages/local_package_setup_py/setup.cfg new file mode 100644 index 000000000..eff513964 --- /dev/null +++ b/spec/fixtures/uv_editable/packages/local_package_setup_py/setup.cfg @@ -0,0 +1,10 @@ +[metadata] +name = local_package_setup_py +version = 0.0.1 + +[options] +packages = local_package_setup_py + +[options.entry_points] +console_scripts = + local_package_setup_py = local_package_setup_py:hello diff --git a/spec/fixtures/uv_editable/packages/local_package_setup_py/setup.py b/spec/fixtures/uv_editable/packages/local_package_setup_py/setup.py new file mode 100644 index 000000000..606849326 --- /dev/null +++ b/spec/fixtures/uv_editable/packages/local_package_setup_py/setup.py @@ -0,0 +1,3 @@ +from setuptools import setup + +setup() diff --git a/spec/fixtures/uv_editable/pyproject.toml b/spec/fixtures/uv_editable/pyproject.toml new file mode 100644 index 000000000..74aec087e --- /dev/null +++ b/spec/fixtures/uv_editable/pyproject.toml @@ -0,0 +1,25 @@ +[project] +name = "uv-editable" +version = "0.0.0" +requires-python = ">=3.13" +dependencies = [ + "gunicorn", + "local-package-pyproject-toml", + "local-package-setup-py", +] + +[project.scripts] +uv-editable = "uv_editable:main" + +[build-system] +requires = ["uv_build"] +build-backend = "uv_build" + +[tool.uv.sources] +# uv doesn't support editable mode with VCS dependencies: +# https://github.com/astral-sh/uv/issues/5442 +# If that ever changes, we should switch to `editable = true` for gunicorn too. +# We still include the Git gunicorn dependency here to ensure we have VCS coverage. +gunicorn = { git = "https://github.com/benoitc/gunicorn" } +local-package-pyproject-toml = { path = "packages/local_package_pyproject_toml", editable = true } +local-package-setup-py = { path = "packages/local_package_setup_py", editable = true } diff --git a/spec/fixtures/uv_editable/src/uv_editable/__init__.py b/spec/fixtures/uv_editable/src/uv_editable/__init__.py new file mode 100644 index 000000000..ae2b5f267 --- /dev/null +++ b/spec/fixtures/uv_editable/src/uv_editable/__init__.py @@ -0,0 +1,2 @@ +def main() -> None: + print("Hello from uv-editable!") diff --git a/spec/fixtures/uv_editable/uv.lock b/spec/fixtures/uv_editable/uv.lock new file mode 100644 index 000000000..f44782cdb --- /dev/null +++ b/spec/fixtures/uv_editable/uv.lock @@ -0,0 +1,47 @@ +version = 1 +revision = 2 +requires-python = ">=3.13" + +[[package]] +name = "gunicorn" +version = "23.0.0" +source = { git = "https://github.com/benoitc/gunicorn#a86ea1e4e6c271d1cd1823c7e14490123f9238fe" } +dependencies = [ + { name = "packaging" }, +] + +[[package]] +name = "local-package-pyproject-toml" +version = "0.0.1" +source = { editable = "packages/local_package_pyproject_toml" } + +[[package]] +name = "local-package-setup-py" +version = "0.0.1" +source = { editable = "packages/local_package_setup_py" } + +[[package]] +name = "packaging" +version = "25.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a1/d4/1fc4078c65507b51b96ca8f8c3ba19e6a61c8253c72794544580a7b6c24d/packaging-25.0.tar.gz", hash = "sha256:d443872c98d677bf60f6a1f2f8c1cb748e8fe762d2bf9d3148b5599295b0fc4f", size = 165727, upload-time = "2025-04-19T11:48:59.673Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/20/12/38679034af332785aac8774540895e234f4d07f7545804097de4b666afd8/packaging-25.0-py3-none-any.whl", hash = "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484", size = 66469, upload-time = "2025-04-19T11:48:57.875Z" }, +] + +[[package]] +name = "uv-editable" +version = "0.0.0" +source = { editable = "." } +dependencies = [ + { name = "gunicorn" }, + { name = "local-package-pyproject-toml" }, + { name = "local-package-setup-py" }, +] + +[package.metadata] +requires-dist = [ + { name = "gunicorn", git = "https://github.com/benoitc/gunicorn" }, + { name = "local-package-pyproject-toml", editable = "packages/local_package_pyproject_toml" }, + { name = "local-package-setup-py", editable = "packages/local_package_setup_py" }, +] diff --git a/spec/fixtures/uv_lockfile_out_of_sync/.python-version b/spec/fixtures/uv_lockfile_out_of_sync/.python-version new file mode 100644 index 000000000..24ee5b1be --- /dev/null +++ b/spec/fixtures/uv_lockfile_out_of_sync/.python-version @@ -0,0 +1 @@ +3.13 diff --git a/spec/fixtures/uv_lockfile_out_of_sync/pyproject.toml b/spec/fixtures/uv_lockfile_out_of_sync/pyproject.toml new file mode 100644 index 000000000..aca766082 --- /dev/null +++ b/spec/fixtures/uv_lockfile_out_of_sync/pyproject.toml @@ -0,0 +1,9 @@ +[project] +name = "uv-lockfile-out-of-sync" +version = "0.0.0" +requires-python = ">=3.13" + +# This dependency isn't in the lockfile. +dependencies = [ + "typing-extensions", +] diff --git a/spec/fixtures/uv_lockfile_out_of_sync/uv.lock b/spec/fixtures/uv_lockfile_out_of_sync/uv.lock new file mode 100644 index 000000000..475542093 --- /dev/null +++ b/spec/fixtures/uv_lockfile_out_of_sync/uv.lock @@ -0,0 +1,8 @@ +version = 1 +revision = 2 +requires-python = ">=3.13" + +[[package]] +name = "uv-lockfile-out-of-sync" +version = "0.0.0" +source = { virtual = "." } diff --git a/spec/fixtures/uv_mismatched_python_version/.python-version b/spec/fixtures/uv_mismatched_python_version/.python-version new file mode 100644 index 000000000..88c9ebc82 --- /dev/null +++ b/spec/fixtures/uv_mismatched_python_version/.python-version @@ -0,0 +1,2 @@ +# The version here intentionally doesn't match requires-python in pyproject.toml +3.13 diff --git a/spec/fixtures/uv_mismatched_python_version/pyproject.toml b/spec/fixtures/uv_mismatched_python_version/pyproject.toml new file mode 100644 index 000000000..b93af5953 --- /dev/null +++ b/spec/fixtures/uv_mismatched_python_version/pyproject.toml @@ -0,0 +1,9 @@ +[project] +name = "uv-mismatched-python-version" +version = "0.0.0" +# The Python version here intentionally: +# - Doesn't match the version in .python-version +# - Matches the distro Python version in Ubuntu 24.04 +# ...so we can test both the error message and that uv doesn't fall back to system Python. +requires-python = "==3.12.*" +dependencies = [] diff --git a/spec/fixtures/uv_mismatched_python_version/uv.lock b/spec/fixtures/uv_mismatched_python_version/uv.lock new file mode 100644 index 000000000..604350e24 --- /dev/null +++ b/spec/fixtures/uv_mismatched_python_version/uv.lock @@ -0,0 +1,8 @@ +version = 1 +revision = 2 +requires-python = "==3.12.*" + +[[package]] +name = "uv-mismatched-python-version" +version = "0.0.0" +source = { virtual = "." } diff --git a/spec/fixtures/uv_no_python_version_file/pyproject.toml b/spec/fixtures/uv_no_python_version_file/pyproject.toml new file mode 100644 index 000000000..dce78d6ba --- /dev/null +++ b/spec/fixtures/uv_no_python_version_file/pyproject.toml @@ -0,0 +1,9 @@ +[project] +name = "uv-no-python-version-file" +version = "0.0.0" +# The Python version here intentionally: +# - Doesn't match the buildpack's default Python version +# - Matches the distro Python version in Ubuntu 24.04 +# ...so we can test both the error message and that uv doesn't fall back to system Python. +requires-python = "==3.12.*" +dependencies = [] diff --git a/spec/fixtures/uv_no_python_version_file/uv.lock b/spec/fixtures/uv_no_python_version_file/uv.lock new file mode 100644 index 000000000..2c9c3cf8d --- /dev/null +++ b/spec/fixtures/uv_no_python_version_file/uv.lock @@ -0,0 +1,8 @@ +version = 1 +revision = 2 +requires-python = "==3.12.*" + +[[package]] +name = "uv-no-python-version-file" +version = "0.0.0" +source = { virtual = "." } diff --git a/spec/fixtures/uv_oldest_python/.python-version b/spec/fixtures/uv_oldest_python/.python-version new file mode 100644 index 000000000..a5c4c7633 --- /dev/null +++ b/spec/fixtures/uv_oldest_python/.python-version @@ -0,0 +1 @@ +3.9.0 diff --git a/spec/fixtures/uv_oldest_python/pyproject.toml b/spec/fixtures/uv_oldest_python/pyproject.toml new file mode 100644 index 000000000..08b9a31bc --- /dev/null +++ b/spec/fixtures/uv_oldest_python/pyproject.toml @@ -0,0 +1,7 @@ +[project] +name = "uv-oldest-python" +version = "0.0.0" +requires-python = "==3.9.*" +dependencies = [ + "typing-extensions", +] diff --git a/spec/fixtures/uv_oldest_python/uv.lock b/spec/fixtures/uv_oldest_python/uv.lock new file mode 100644 index 000000000..27cbd3836 --- /dev/null +++ b/spec/fixtures/uv_oldest_python/uv.lock @@ -0,0 +1,23 @@ +version = 1 +revision = 2 +requires-python = "==3.9.*" + +[[package]] +name = "typing-extensions" +version = "4.13.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/37/23083fcd6e35492953e8d2aaaa68b860eb422b34627b13f2ce3eb6106061/typing_extensions-4.13.2.tar.gz", hash = "sha256:e6c81219bd689f51865d9e372991c540bda33a0379d5573cddb9a3a23f7caaef", size = 106967, upload-time = "2025-04-10T14:19:05.416Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8b/54/b1ae86c0973cc6f0210b53d508ca3641fb6d0c56823f288d108bc7ab3cc8/typing_extensions-4.13.2-py3-none-any.whl", hash = "sha256:a439e7c04b49fec3e5d3e2beaa21755cadbbdc391694e28ccdd36ca4a1408f8c", size = 45806, upload-time = "2025-04-10T14:19:03.967Z" }, +] + +[[package]] +name = "uv-oldest-python" +version = "0.0.0" +source = { virtual = "." } +dependencies = [ + { name = "typing-extensions" }, +] + +[package.metadata] +requires-dist = [{ name = "typing-extensions" }] diff --git a/spec/fixtures/uv_runtime_txt/.python-version b/spec/fixtures/uv_runtime_txt/.python-version new file mode 100644 index 000000000..1be10d256 --- /dev/null +++ b/spec/fixtures/uv_runtime_txt/.python-version @@ -0,0 +1,5 @@ +# The Python version here intentionally: +# - Doesn't match that in runtime.txt +# - Matches the distro Python version in Ubuntu 24.04 +# ...so we can test both the error message and that uv doesn't fall back to system Python. +3.12 diff --git a/spec/fixtures/uv_runtime_txt/pyproject.toml b/spec/fixtures/uv_runtime_txt/pyproject.toml new file mode 100644 index 000000000..c8f057a16 --- /dev/null +++ b/spec/fixtures/uv_runtime_txt/pyproject.toml @@ -0,0 +1,9 @@ +[project] +name = "uv-runtime-txt" +version = "0.0.0" +# The Python version here intentionally: +# - Doesn't match the version in runtime.txt +# - Matches the distro Python version in Ubuntu 24.04 +# ...so we can test both the error message and that uv doesn't fall back to system Python. +requires-python = "==3.12.*" +dependencies = [] diff --git a/spec/fixtures/uv_runtime_txt/runtime.txt b/spec/fixtures/uv_runtime_txt/runtime.txt new file mode 100644 index 000000000..67ebc4e9a --- /dev/null +++ b/spec/fixtures/uv_runtime_txt/runtime.txt @@ -0,0 +1 @@ +python-3.11 diff --git a/spec/fixtures/uv_runtime_txt/uv.lock b/spec/fixtures/uv_runtime_txt/uv.lock new file mode 100644 index 000000000..f5d9d9025 --- /dev/null +++ b/spec/fixtures/uv_runtime_txt/uv.lock @@ -0,0 +1,8 @@ +version = 1 +revision = 2 +requires-python = ">=3.13" + +[[package]] +name = "uv-runtime-txt" +version = "0.0.0" +source = { virtual = "." } diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 50d722c51..724dc43b0 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -235,4 +235,89 @@ end end end + + context 'when using uv' do + let(:app) { Hatchet::Runner.new('spec/fixtures/ci_uv', buildpacks:) } + + it 'installs both normal and test dependencies and uses cache on subsequent runs' do + app.run_ci do |test_run| + expect(clean_output(test_run.output)).to match(Regexp.new(<<~REGEX)) + -----> Python app detected + -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + -----> Installing uv #{UV_VERSION} + -----> Installing dependencies using 'uv sync --locked' + Resolved 7 packages in .+s + Prepared 5 packages in .+s + Installed 5 packages in .+s + Bytecode compiled .+ files in .+s + \\+ iniconfig==.+ + \\+ packaging==.+ + \\+ pluggy==.+ + \\+ pytest==.+ + \\+ typing-extensions==.+ + -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. + -----> Running bin/post_compile hook + CI=true + CPLUS_INCLUDE_PATH=/app/.heroku/python/include + C_INCLUDE_PATH=/app/.heroku/python/include + DISABLE_COLLECTSTATIC=1 + INSTALL_TEST=1 + LANG=en_US.UTF-8 + LC_ALL=C.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib + PATH=/tmp/cache.+/.heroku/python-uv:/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config + PYTHONUNBUFFERED=1 + UV_NO_MANAGED_PYTHON=1 + UV_PROJECT_ENVIRONMENT=/app/.heroku/python + UV_PYTHON_DOWNLOADS=never + -----> Inline app detected + LANG=en_US.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib + PATH=/app/.heroku/python/bin:/tmp/cache.+/.heroku/python-uv:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PYTHONHASHSEED=random + PYTHONHOME=/app/.heroku/python + PYTHONPATH=/app + PYTHONUNBUFFERED=true + UV_NO_MANAGED_PYTHON=1 + UV_PROJECT_ENVIRONMENT=/app/.heroku/python + UV_PYTHON_DOWNLOADS=never + -----> No test-setup command provided. Skipping. + -----> Running test command `./bin/print-env-vars.sh && pytest --version`... + CI=true + DYNO_RAM=2560 + FORWARDED_ALLOW_IPS=\\* + GUNICORN_CMD_ARGS=--access-logfile - + LANG=en_US.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib + PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ + PYTHONHASHSEED=random + PYTHONHOME=/app/.heroku/python + PYTHONPATH=/app + PYTHONUNBUFFERED=true + WEB_CONCURRENCY=5 + pytest .+ + -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully + REGEX + + test_run.run_again + expect(clean_output(test_run.output)).to match(Regexp.new(<<~REGEX)) + -----> Python app detected + -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + -----> Restoring cache + -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} + -----> Using cached uv #{UV_VERSION} + -----> Installing dependencies using 'uv sync --locked' + Resolved 7 packages in .+s + Bytecode compiled .+ files in .+s + -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. + -----> Running bin/post_compile hook + REGEX + end + end + end end diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index eccdf03d1..3e3f4c49d 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -14,8 +14,8 @@ remote: ! Error: Couldn't find any supported Python package manager files. remote: ! remote: ! A Python app on Heroku must have either a 'requirements.txt', - remote: ! 'Pipfile' or 'poetry.lock' package manager file in the root - remote: ! directory of its source code. + remote: ! 'Pipfile', 'poetry.lock' or 'uv.lock' package manager file in + remote: ! the root directory of its source code. remote: ! remote: ! Currently the root directory of your app contains: remote: ! @@ -34,10 +34,10 @@ remote: ! Otherwise, add a package manager file to your app. If your app has remote: ! no dependencies, then create an empty 'requirements.txt' file. remote: ! - remote: ! If you would like to see support for the package manager uv, - remote: ! please vote and comment on these GitHub issues: - remote: ! https://github.com/heroku/heroku-buildpack-python/issues/1616 - remote: ! https://github.com/heroku/roadmap/issues/323 + remote: ! If you aren't sure which package manager to use, we recommend + remote: ! trying uv, since it supports lockfiles, is extremely fast, and + remote: ! is actively maintained by a full-time team: + remote: ! https://docs.astral.sh/uv/ remote: ! remote: ! For help with using Python on Heroku, see: remote: ! https://devcenter.heroku.com/articles/getting-started-with-python @@ -87,6 +87,7 @@ remote: ! Pipfile.lock \\(Pipenv\\) remote: ! requirements.txt \\(pip\\) remote: ! poetry.lock \\(Poetry\\) + remote: ! uv.lock \\(uv\\) remote: ! remote: ! For now, we will build your app using the first package manager remote: ! listed above, however, in the future this warning will become @@ -94,6 +95,11 @@ remote: ! remote: ! Decide which package manager you want to use with your app, and remote: ! then delete the file\\(s\\) and any config from the others. + remote: ! + remote: ! If you aren't sure which package manager to use, we recommend + remote: ! trying uv, since it supports lockfiles, is extremely fast, and + remote: ! is actively maintained by a full-time team: + remote: ! https://docs.astral.sh/uv/ remote: remote: remote: ! Note: We recently added support for the package manager Poetry. @@ -101,6 +107,12 @@ remote: ! it, otherwise the requirements.txt file it generates will cause remote: ! the warning above. remote: + remote: + remote: ! Note: We recently added support for the package manager uv. + remote: ! If you are using a third-party uv buildpack you must remove + remote: ! it, otherwise the requirements.txt file it generates will cause + remote: ! the warning above. + remote: remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION} @@ -118,25 +130,25 @@ it 'clears the cache before installing with the new package manager' do app.deploy do |app| FileUtils.rm('requirements.txt') - FileUtils.cp(FIXTURE_DIR.join('poetry_basic/pyproject.toml'), '.') - FileUtils.cp(FIXTURE_DIR.join('poetry_basic/poetry.lock'), '.') + FileUtils.cp(FIXTURE_DIR.join('uv_basic/pyproject.toml'), '.') + FileUtils.cp(FIXTURE_DIR.join('uv_basic/uv.lock'), '.') app.commit! app.push! - expect(clean_output(app.output)).to include(<<~OUTPUT) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Discarding cache since: - remote: - The package manager has changed from pip to poetry + remote: - The package manager has changed from pip to uv remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing Poetry #{POETRY_VERSION} - remote: -----> Installing dependencies using 'poetry sync --only main' - remote: Installing dependencies from lock file - remote: - remote: Package operations: 1 install, 0 updates, 0 removals - remote: - remote: - Installing typing-extensions (4.12.2) + remote: -----> Installing uv #{UV_VERSION} + remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' + remote: Resolved 7 packages in .+s + remote: Prepared 1 package in .+s + remote: Installed 1 package in .+s + remote: Bytecode compiled 1 file in .+s + remote: \\+ typing-extensions==4.13.2 remote: -----> Discovering process types - OUTPUT + REGEX end end end diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb new file mode 100644 index 000000000..7931c8865 --- /dev/null +++ b/spec/hatchet/uv_spec.rb @@ -0,0 +1,406 @@ +# frozen_string_literal: true + +require_relative '../spec_helper' + +RSpec.describe 'uv support' do + context 'with a uv.lock' do + let(:buildpacks) { [:default, 'heroku-community/inline'] } + let(:app) { Hatchet::Runner.new('spec/fixtures/uv_basic', buildpacks:) } + + it 'installs successfully using uv and on rebuilds uses the cache' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Python app detected + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Installing uv #{UV_VERSION} + remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' + remote: Resolved 7 packages in .+s + remote: Prepared 1 package in .+s + remote: Installed 1 package in .+s + remote: Bytecode compiled 1 file in .+s + remote: \\+ typing-extensions==4.13.2 + remote: -----> Inline app detected + remote: LANG=en_US.UTF-8 + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib + remote: LIBRARY_PATH=/app/.heroku/python/lib + remote: PATH=/app/.heroku/python/bin:/tmp/codon/tmp/cache/.heroku/python-uv:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PYTHONHASHSEED=random + remote: PYTHONHOME=/app/.heroku/python + remote: PYTHONPATH=/app + remote: PYTHONUNBUFFERED=true + remote: UV_CACHE_DIR=/tmp/uv-cache + remote: UV_NO_MANAGED_PYTHON=1 + remote: UV_PROJECT_ENVIRONMENT=/app/.heroku/python + remote: UV_PYTHON_DOWNLOADS=never + remote: + remote: \\['', + remote: '/app', + remote: '/app/.heroku/python/lib/python313.zip', + remote: '/app/.heroku/python/lib/python3.13', + remote: '/app/.heroku/python/lib/python3.13/lib-dynload', + remote: '/app/.heroku/python/lib/python3.13/site-packages'\\] + remote: + remote: Using Python #{DEFAULT_PYTHON_FULL_VERSION} environment at: /app/.heroku/python + remote: Package Version + remote: ----------------- ------- + remote: typing-extensions 4.13.2 + remote: + remote: + REGEX + app.commit! + app.push! + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Python app detected + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version + remote: -----> Restoring cache + remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Using cached uv #{UV_VERSION} + remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' + remote: Resolved 7 packages in .+s + remote: Bytecode compiled 1 file in .+s + remote: -----> Inline app detected + REGEX + end + end + end + + # TODO: Enable this once a previous buildpack release exists that uses an older uv version. + context 'when the uv version has changed since the last build', + skip: 'requires prior buildpack release that uses older uv' do + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#vTODO'] } + let(:app) { Hatchet::Runner.new('spec/fixtures/uv_basic', buildpacks:) } + + it 'clears the cache before installing' do + app.deploy do |app| + update_buildpacks(app, [:default]) + app.commit! + app.push! + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Python app detected + remote: -----> Using Python 3.13 specified in .python-version + remote: -----> Discarding cache since: + remote: - The uv version has changed from TODO to #{UV_VERSION} + remote: -----> Installing Python #{LATEST_PYTHON_3_13} + remote: -----> Installing uv #{UV_VERSION} + remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' + remote: Resolved 7 packages in .+s + remote: Prepared 1 package in .+s + remote: Installed 1 package in .+s + remote: Bytecode compiled 1 file in .+s + remote: \\+ typing-extensions==4.13.2 + remote: -----> Discovering process types + REGEX + end + end + end + + # uv doesn't support editable mode with VCS dependencies, so unlike the editable tests for the other + # package managers the gunicorn dependency isn't editable. However, we still include it to ensure we + # have VCS coverage. See: https://github.com/astral-sh/uv/issues/5442 + context 'when uv.lock contains editable requirements and a VCS dependency' do + let(:buildpacks) { [:default, 'heroku-community/inline'] } + let(:app) { Hatchet::Runner.new('spec/fixtures/uv_editable', buildpacks:) } + + it 'rewrites .pth and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' + remote: Resolved 5 packages in .+s + remote: .+ + remote: Prepared 5 packages in .+s + remote: Installed 5 packages in .+s + remote: Bytecode compiled .+ files in .+s + remote: \\+ gunicorn==23.0.0 \\(from git\\+https://github.com/benoitc/gunicorn@a86ea1e4e6c271d1cd1823c7e14490123f9238fe\\) + remote: \\+ local-package-pyproject-toml==0.0.1 \\(from file:///tmp/build_.+/packages/local_package_pyproject_toml\\) + remote: \\+ local-package-setup-py==0.0.1 \\(from file:///tmp/build_.+/packages/local_package_setup_py\\) + remote: \\+ packaging==25.0 + remote: \\+ uv-editable==0.0.0 \\(from file:///tmp/build_.+\\) + remote: -----> Running bin/post_compile hook + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} + remote: uv_editable.pth:/tmp/build_.+/src + remote: + remote: Running entrypoint for the current package: Hello from uv-editable! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) + remote: -----> Inline app detected + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} + remote: uv_editable.pth:/tmp/build_.+/src + remote: + remote: Running entrypoint for the current package: Hello from uv-editable! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) + REGEX + + # Test rewritten paths work at runtime. + expect(app.run('bin/test-entrypoints.sh')).to include(<<~OUTPUT) + __editable___local_package_pyproject_toml_0_0_1_finder.py:/app/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + __editable___local_package_setup_py_0_0_1_finder.py:/app/packages/local_package_setup_py/local_package_setup_py'} + uv_editable.pth:/app/src + + Running entrypoint for the current package: Hello from uv-editable! + Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + Running entrypoint for the setup.py-based local package: Hello from setup.py! + Running entrypoint for the VCS package: gunicorn (version 23.0.0) + OUTPUT + + # Test that the cached .pth files work correctly. + app.commit! + app.push! + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' + remote: Resolved 5 packages in .+ + remote: .+ + remote: Prepared 3 packages in .+s + remote: Uninstalled 3 packages in .+s + remote: Installed 3 packages in .+s + remote: Bytecode compiled .+ files in .+s + remote: - local-package-pyproject-toml==0.0.1 \\(from file:///tmp/build_.+/packages/local_package_pyproject_toml\\) + remote: \\+ local-package-pyproject-toml==0.0.1 \\(from file:///tmp/build_.+/packages/local_package_pyproject_toml\\) + remote: - local-package-setup-py==0.0.1 \\(from file:///tmp/build_.+/packages/local_package_setup_py\\) + remote: \\+ local-package-setup-py==0.0.1 \\(from file:///tmp/build_.+/packages/local_package_setup_py\\) + remote: - uv-editable==0.0.0 \\(from file:///tmp/build_.+\\) + remote: \\+ uv-editable==0.0.0 \\(from file:///tmp/build_.+\\) + remote: -----> Running bin/post_compile hook + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} + remote: uv_editable.pth:/tmp/build_.+/src + remote: + remote: Running entrypoint for the current package: Hello from uv-editable! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) + remote: -----> Inline app detected + remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} + remote: uv_editable.pth:/tmp/build_.+/src + remote: + remote: Running entrypoint for the current package: Hello from uv-editable! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) + REGEX + end + end + end + + context 'when using our oldest supported Python version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/uv_oldest_python') } + + it 'installs successfully' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Python app detected + remote: -----> Using Python 3.9.0 specified in .python-version + remote: + remote: ! Warning: Support for Python 3.9 is ending soon! + remote: ! + remote: ! Python 3.9 will reach its upstream end-of-life in October 2025, + remote: ! at which point it will no longer receive security updates: + remote: ! https://devguide.python.org/versions/#supported-versions + remote: ! + remote: ! As such, support for Python 3.9 will be removed from this + remote: ! buildpack on 7th January 2026. + remote: ! + remote: ! Upgrade to a newer Python version as soon as possible, by + remote: ! changing the version in your .python-version file. + remote: ! + remote: ! For more information, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions + remote: + remote: + remote: ! Warning: A Python patch update is available! + remote: ! + remote: ! Your app is using Python 3.9.0, however, there is a newer + remote: ! patch release of Python 3.9 available: #{LATEST_PYTHON_3_9} + remote: ! + remote: ! It is important to always use the latest patch version of + remote: ! Python to keep your app secure. + remote: ! + remote: ! Update your .python-version file to use the new version. + remote: ! + remote: ! We strongly recommend that you don't pin your app to an + remote: ! exact Python version such as 3.9.0, and instead only specify + remote: ! the major Python version of 3.9 in your .python-version file. + remote: ! This will allow your app to receive the latest available Python + remote: ! patch version automatically and prevent this warning. + remote: + remote: -----> Installing Python 3.9.0 + remote: -----> Installing uv #{UV_VERSION} + remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' + remote: Resolved 2 packages in .+s + remote: Prepared 1 package in .+s + remote: Installed 1 package in .+s + remote: Bytecode compiled 1 file in .+s + remote: \\+ typing-extensions==4.13.2 + REGEX + end + end + end + + # This tests the error message when there is no .python-version file, and in particular the case where + # the buildpack's default Python version is not compatible with `requires-python` in pyproject.toml. + # (Since we must prevent uv from downloading its own Python or using system Python, and also + # want a clearer error message than using `--python` or `UV_PYTHON` would give us). + context 'when there is no .python-version file' do + context 'when there is no cached Python version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/uv_no_python_version_file', allow_failure: true) } + + it 'fails the build with .python-version instructions' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} + remote: + remote: ! Error: No Python version was specified. + remote: ! + remote: ! When using the package manager uv on Heroku, you must specify + remote: ! your app's Python version with a .python-version file. + remote: ! + remote: ! To add a .python-version file: + remote: ! + remote: ! 1. Make sure you are in the root directory of your app + remote: ! and not a subdirectory. + remote: ! 2. Run 'uv python pin #{DEFAULT_PYTHON_MAJOR_VERSION}' + remote: ! (adjust to match your app's major Python version). + remote: ! 3. Commit the changes to your Git repository using + remote: ! 'git add --all' and then 'git commit'. + remote: ! + remote: ! Note: We strongly recommend that you don't specify the Python + remote: ! patch version number in your .python-version file, since it will + remote: ! pin your app to an exact Python version and so stop your app from + remote: ! receiving security updates each time it builds. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + context 'when there is a cached Python version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_unspecified', allow_failure: true) } + + it 'fails the build with .python-version instructions' do + app.deploy do |app| + FileUtils.rm('requirements.txt') + FileUtils.cp(FIXTURE_DIR.join('uv_no_python_version_file/pyproject.toml'), '.') + FileUtils.cp(FIXTURE_DIR.join('uv_no_python_version_file/uv.lock'), '.') + app.commit! + app.push! + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> No Python version was specified. Using the same major version as the last build: Python #{DEFAULT_PYTHON_MAJOR_VERSION} + remote: + remote: ! Error: No Python version was specified. + remote: ! + remote: ! When using the package manager uv on Heroku, you must specify + remote: ! your app's Python version with a .python-version file. + remote: ! + remote: ! To add a .python-version file: + remote: ! + remote: ! 1. Make sure you are in the root directory of your app + remote: ! and not a subdirectory. + remote: ! 2. Run 'uv python pin #{DEFAULT_PYTHON_MAJOR_VERSION}' + remote: ! (adjust to match your app's major Python version). + remote: ! 3. Commit the changes to your Git repository using + remote: ! 'git add --all' and then 'git commit'. + remote: ! + remote: ! Note: We strongly recommend that you don't specify the Python + remote: ! patch version number in your .python-version file, since it will + remote: ! pin your app to an exact Python version and so stop your app from + remote: ! receiving security updates each time it builds. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + end + + # This tests the error message when a runtime.txt is present, and in particular the case where + # the runtime.txt version is not compatible with `requires-python` in pyproject.toml. + # (Since we must prevent uv from downloading its own Python or using system Python, and also + # want a clearer error message than using `--python` or `UV_PYTHON` would give us). + context 'when there is a runtime.txt file' do + let(:app) { Hatchet::Runner.new('spec/fixtures/uv_runtime_txt', allow_failure: true) } + + it 'fails the build with runtime.txt migration instructions' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Using Python 3.11 specified in runtime.txt + remote: + remote: ! Error: The runtime.txt file isn't supported when using uv. + remote: ! + remote: ! When using the package manager uv on Heroku, you must specify + remote: ! your app's Python version with a .python-version file and not + remote: ! a runtime.txt file. + remote: ! + remote: ! To switch to a .python-version file: + remote: ! + remote: ! 1. Make sure you are in the root directory of your app + remote: ! and not a subdirectory. + remote: ! 2. Delete your runtime.txt file. + remote: ! 3. Run 'uv python pin 3.11' + remote: ! (adjust to match your app's major Python version). + remote: ! 4. Commit the changes to your Git repository using + remote: ! 'git add --all' and then 'git commit'. + remote: ! + remote: ! Note: We strongly recommend that you don't specify the Python + remote: ! patch version number in your .python-version file, since it will + remote: ! pin your app to an exact Python version and so stop your app from + remote: ! receiving security updates each time it builds. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + # This tests the error message when `requires-python` in pyproject.toml isn't compatible with + # the version in .python-version. This might seem unnecessary since it's testing something uv + # validates itself, however, the quality of the error message here depends on what uv options + # we use (for example, using `--python` or `UV_PYTHON` results in a worse error message). + context 'when requires-python in pyproject.toml is incompatible with .python-version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/uv_mismatched_python_version', allow_failure: true) } + + it 'fails the build' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' + remote: Using CPython #{DEFAULT_PYTHON_FULL_VERSION} interpreter at: /app/.heroku/python/bin/python#{DEFAULT_PYTHON_MAJOR_VERSION} + remote: error: The Python request from `.python-version` resolved to Python #{DEFAULT_PYTHON_FULL_VERSION}, which is incompatible with the project's Python requirement: `==3.12.*`. Use `uv python pin` to update the `.python-version` file to a compatible version. + remote: + remote: ! Error: Unable to install dependencies using uv. + remote: ! + remote: ! See the log output above for more information. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + # This tests not only our handling of failing dependency installation, but also that we're running + # uv in such a way that it errors if the lockfile is out of sync, rather than simply updating it. + context 'when uv.lock is out of sync with pyproject.toml' do + let(:app) { Hatchet::Runner.new('spec/fixtures/uv_lockfile_out_of_sync', allow_failure: true) } + + it 'fails the build' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' + remote: Resolved 2 packages in .+s + remote: error: The lockfile at `uv.lock` needs to be updated, but `--locked` was provided. To update the lockfile, run `uv lock`. + remote: + remote: ! Error: Unable to install dependencies using uv. + remote: ! + remote: ! See the log output above for more information. + remote: + remote: ! Push rejected, failed to compile Python app. + REGEX + end + end + end +end diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 2f561389c..b6db10188 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -31,6 +31,7 @@ def get_requirement_version(package_name) WHEEL_VERSION = get_requirement_version('wheel') PIPENV_VERSION = get_requirement_version('pipenv') POETRY_VERSION = get_requirement_version('poetry') +UV_VERSION = get_requirement_version('uv') # Work around the return value for `default_buildpack` changing after deploy: # https://github.com/heroku/hatchet/issues/180 From 28d12f228c3ffaadb7f531b3316e6726a455734d Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 13 May 2025 20:39:55 +0000 Subject: [PATCH 137/406] Prepare release v286 (#1798) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8569fc0e0..4ee716341 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v286] - 2025-05-13 + - Added support for the package manager uv. ([#1791](https://github.com/heroku/heroku-buildpack-python/pull/1791)) ## [v285] - 2025-05-08 @@ -1215,7 +1218,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v285...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v286...main +[v286]: https://github.com/heroku/heroku-buildpack-python/compare/v285...v286 [v285]: https://github.com/heroku/heroku-buildpack-python/compare/v284...v285 [v284]: https://github.com/heroku/heroku-buildpack-python/compare/v283...v284 [v283]: https://github.com/heroku/heroku-buildpack-python/compare/v282...v283 From c9942a6829fe2e064db7a26c9d95fbbd1bd9ab72 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 17 May 2025 13:30:18 +0100 Subject: [PATCH 138/406] Bump uv from 0.7.3 to 0.7.5 (#1800) * Bump uv from 0.7.3 to 0.7.5 Bumps [uv](https://github.com/astral-sh/uv) from 0.7.3 to 0.7.5. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.7.3...0.7.5) --- updated-dependencies: - dependency-name: uv dependency-version: 0.7.5 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry * Enable uv version change test --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- spec/hatchet/uv_spec.rb | 8 +++----- 3 files changed, 5 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4ee716341..05284f405 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.7.3 to 0.7.5. ([#1800](https://github.com/heroku/heroku-buildpack-python/pull/1800)) ## [v286] - 2025-05-13 diff --git a/requirements/uv.txt b/requirements/uv.txt index 48ea67dbd..9aa9d13fc 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.7.3 +uv==0.7.5 diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 7931c8865..f123fdf70 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -65,10 +65,8 @@ end end - # TODO: Enable this once a previous buildpack release exists that uses an older uv version. - context 'when the uv version has changed since the last build', - skip: 'requires prior buildpack release that uses older uv' do - let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#vTODO'] } + context 'when the uv version has changed since the last build' do + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v286'] } let(:app) { Hatchet::Runner.new('spec/fixtures/uv_basic', buildpacks:) } it 'clears the cache before installing' do @@ -80,7 +78,7 @@ remote: -----> Python app detected remote: -----> Using Python 3.13 specified in .python-version remote: -----> Discarding cache since: - remote: - The uv version has changed from TODO to #{UV_VERSION} + remote: - The uv version has changed from 0.7.3 to #{UV_VERSION} remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing uv #{UV_VERSION} remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' From 305025c8c53f1d8387b09ce3c0ebd4b85162e0d2 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 20 May 2025 14:56:18 +0100 Subject: [PATCH 139/406] Bump rubocop from 1.75.5 to 1.75.6 in the ruby-dependencies group (#1804) Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.75.5 to 1.75.6 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.75.5...v1.75.6) --- updated-dependencies: - dependency-name: rubocop dependency-version: 1.75.6 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Gemfile.lock b/Gemfile.lock index 204dea856..b57fef2fe 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -51,7 +51,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.3) - rubocop (1.75.5) + rubocop (1.75.6) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) From 18a9865b3703c78634b05208149217acf29a2b6e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 20 May 2025 14:25:54 +0000 Subject: [PATCH 140/406] Bump uv from 0.7.5 to 0.7.6 (#1803) * Bump uv from 0.7.5 to 0.7.6 Bumps [uv](https://github.com/astral-sh/uv) from 0.7.5 to 0.7.6. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.7.5...0.7.6) --- updated-dependencies: - dependency-name: uv dependency-version: 0.7.6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 2 +- requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 05284f405..f0a4f12a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## [Unreleased] -- Updated uv from 0.7.3 to 0.7.5. ([#1800](https://github.com/heroku/heroku-buildpack-python/pull/1800)) +- Updated uv from 0.7.3 to 0.7.6. ([#1800](https://github.com/heroku/heroku-buildpack-python/pull/1800) and [#1803](https://github.com/heroku/heroku-buildpack-python/pull/1803)) ## [v286] - 2025-05-13 diff --git a/requirements/uv.txt b/requirements/uv.txt index 9aa9d13fc..8d7271dfd 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.7.5 +uv==0.7.6 From f97a9538453ae5040823a893024e9af8213349b3 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 20 May 2025 14:35:45 +0000 Subject: [PATCH 141/406] Bump poetry from 2.1.2 to 2.1.3 (#1797) * Bump poetry from 2.1.2 to 2.1.3 Bumps [poetry](https://github.com/python-poetry/poetry) from 2.1.2 to 2.1.3. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](https://github.com/python-poetry/poetry/compare/2.1.2...2.1.3) --- updated-dependencies: - dependency-name: poetry dependency-version: 2.1.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/poetry.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f0a4f12a4..35e405e8f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Updated uv from 0.7.3 to 0.7.6. ([#1800](https://github.com/heroku/heroku-buildpack-python/pull/1800) and [#1803](https://github.com/heroku/heroku-buildpack-python/pull/1803)) +- Updated Poetry from 2.1.2 to 2.1.3. ([#1797](https://github.com/heroku/heroku-buildpack-python/pull/1797)) ## [v286] - 2025-05-13 diff --git a/requirements/poetry.txt b/requirements/poetry.txt index 7fc861c0f..3a370aa53 100644 --- a/requirements/poetry.txt +++ b/requirements/poetry.txt @@ -1 +1 @@ -poetry==2.1.2 +poetry==2.1.3 From 0fca821e84320fe67017e85927d7ca7423da4adb Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 20 May 2025 14:52:56 +0000 Subject: [PATCH 142/406] Bump pip from 25.0.1 to 25.1.1 (#1795) * Bump pip from 25.0.1 to 25.1.1 Bumps [pip](https://github.com/pypa/pip) from 25.0.1 to 25.1.1. - [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst) - [Commits](https://github.com/pypa/pip/compare/25.0.1...25.1.1) --- updated-dependencies: - dependency-name: pip dependency-version: 25.1.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 3 ++- requirements/pip.txt | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 35e405e8f..3945fc24e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,8 +2,9 @@ ## [Unreleased] -- Updated uv from 0.7.3 to 0.7.6. ([#1800](https://github.com/heroku/heroku-buildpack-python/pull/1800) and [#1803](https://github.com/heroku/heroku-buildpack-python/pull/1803)) +- Updated pip from 25.0.1 to 25.1.1. ([#1795](https://github.com/heroku/heroku-buildpack-python/pull/1795)) - Updated Poetry from 2.1.2 to 2.1.3. ([#1797](https://github.com/heroku/heroku-buildpack-python/pull/1797)) +- Updated uv from 0.7.3 to 0.7.6. ([#1800](https://github.com/heroku/heroku-buildpack-python/pull/1800) and [#1803](https://github.com/heroku/heroku-buildpack-python/pull/1803)) ## [v286] - 2025-05-13 diff --git a/requirements/pip.txt b/requirements/pip.txt index 059862d50..e85a84b2e 100644 --- a/requirements/pip.txt +++ b/requirements/pip.txt @@ -1 +1 @@ -pip==25.0.1 +pip==25.1.1 From d5361ca970d09f595cbdd88792acaf5658b83327 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 20 May 2025 17:11:35 +0000 Subject: [PATCH 143/406] Prepare release v287 (#1805) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3945fc24e..6290127a1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v287] - 2025-05-20 + - Updated pip from 25.0.1 to 25.1.1. ([#1795](https://github.com/heroku/heroku-buildpack-python/pull/1795)) - Updated Poetry from 2.1.2 to 2.1.3. ([#1797](https://github.com/heroku/heroku-buildpack-python/pull/1797)) - Updated uv from 0.7.3 to 0.7.6. ([#1800](https://github.com/heroku/heroku-buildpack-python/pull/1800) and [#1803](https://github.com/heroku/heroku-buildpack-python/pull/1803)) @@ -1221,7 +1224,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v286...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v287...main +[v287]: https://github.com/heroku/heroku-buildpack-python/compare/v286...v287 [v286]: https://github.com/heroku/heroku-buildpack-python/compare/v285...v286 [v285]: https://github.com/heroku/heroku-buildpack-python/compare/v284...v285 [v284]: https://github.com/heroku/heroku-buildpack-python/compare/v283...v284 From 1e32b775d70e9f5070745e9fc216fe28c17b583e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 2 Jun 2025 12:15:19 +0000 Subject: [PATCH 144/406] Bump the ruby-dependencies group with 3 updates (#1807) Bumps the ruby-dependencies group with 3 updates: [rspec-core](https://github.com/rspec/rspec), [rspec-expectations](https://github.com/rspec/rspec) and [rubocop](https://github.com/rubocop/rubocop). Updates `rspec-core` from 3.13.3 to 3.13.4 - [Changelog](https://github.com/rspec/rspec/blob/rspec-core-v3.13.4/rspec-core/Changelog.md) - [Commits](https://github.com/rspec/rspec/compare/rspec-core-v3.13.3...rspec-core-v3.13.4) Updates `rspec-expectations` from 3.13.4 to 3.13.5 - [Changelog](https://github.com/rspec/rspec/blob/rspec-expectations-v3.13.5/rspec-expectations/Changelog.md) - [Commits](https://github.com/rspec/rspec/compare/rspec-expectations-v3.13.4...rspec-expectations-v3.13.5) Updates `rubocop` from 1.75.6 to 1.75.8 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.75.6...v1.75.8) --- updated-dependencies: - dependency-name: rspec-core dependency-version: 3.13.4 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rspec-expectations dependency-version: 3.13.5 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rubocop dependency-version: 1.75.8 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index b57fef2fe..9a1ac9e33 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -3,7 +3,7 @@ GEM specs: ast (2.4.3) base64 (0.2.0) - diff-lcs (1.6.1) + diff-lcs (1.6.2) erubis (2.7.0) excon (1.2.5) logger @@ -20,7 +20,7 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.12.0) + json (2.12.2) language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) @@ -43,15 +43,15 @@ GEM rate_throttle_client (0.1.2) regexp_parser (2.10.0) rrrretry (1.0.0) - rspec-core (3.13.3) + rspec-core (3.13.4) rspec-support (~> 3.13.0) - rspec-expectations (3.13.4) + rspec-expectations (3.13.5) diff-lcs (>= 1.2.0, < 2.0) rspec-support (~> 3.13.0) rspec-retry (0.6.2) rspec-core (> 3.3) - rspec-support (3.13.3) - rubocop (1.75.6) + rspec-support (3.13.4) + rubocop (1.75.8) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) From 59acfddcea9da5bdd1a1dbbb9e45d0ec5ccff98d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 3 Jun 2025 23:04:24 +0100 Subject: [PATCH 145/406] Bump uv from 0.7.6 to 0.7.10 (#1811) * Bump uv from 0.7.6 to 0.7.10 Bumps [uv](https://github.com/astral-sh/uv) from 0.7.6 to 0.7.10. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.7.6...0.7.10) --- updated-dependencies: - dependency-name: uv dependency-version: 0.7.10 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6290127a1..103a1678d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.7.6 to 0.7.10. ([#1811](https://github.com/heroku/heroku-buildpack-python/pull/1811)) ## [v287] - 2025-05-20 diff --git a/requirements/uv.txt b/requirements/uv.txt index 8d7271dfd..8035a0dc1 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.7.6 +uv==0.7.10 From dde549edf86850b6ab22512630f02374a41082ef Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 3 Jun 2025 23:17:18 +0100 Subject: [PATCH 146/406] Add support for Python 3.13.4, 3.12.11, 3.11.13, 3.10.18 and 3.9.23 (#1810) Release announcement: https://blog.python.org/2025/06/python-3134-31211-31113-31018-and-3923.html Changelogs: https://docs.python.org/release/3.13.4/whatsnew/changelog.html#python-3-13-4-final https://docs.python.org/release/3.12.11/whatsnew/changelog.html#python-3-12-11-final https://docs.python.org/release/3.11.12/whatsnew/changelog.html#python-3-11-12-final https://docs.python.org/release/3.10.18/whatsnew/changelog.html#python-3-10-18-final https://docs.python.org/release/3.9.23/whatsnew/changelog.html#python-3-9-23-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/15427510896 https://github.com/heroku/heroku-buildpack-python/actions/runs/15427507963 https://github.com/heroku/heroku-buildpack-python/actions/runs/15427504571 https://github.com/heroku/heroku-buildpack-python/actions/runs/15427485959 https://github.com/heroku/heroku-buildpack-python/actions/runs/15427448283 GUS-W-17604340. GUS-W-18699915. --- CHANGELOG.md | 5 +++++ lib/python_version.sh | 10 +++++----- spec/hatchet/uv_spec.rb | 3 ++- spec/spec_helper.rb | 10 +++++----- 4 files changed, 17 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 103a1678d..dc4201bf1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ ## [Unreleased] +- The Python 3.13 version alias now resolves to Python 3.13.4. ([#1810](https://github.com/heroku/heroku-buildpack-python/pull/1810)) +- The Python 3.12 version alias now resolves to Python 3.12.11. ([#1810](https://github.com/heroku/heroku-buildpack-python/pull/1810)) +- The Python 3.11 version alias now resolves to Python 3.11.13. ([#1810](https://github.com/heroku/heroku-buildpack-python/pull/1810)) +- The Python 3.10 version alias now resolves to Python 3.10.18. ([#1810](https://github.com/heroku/heroku-buildpack-python/pull/1810)) +- The Python 3.9 version alias now resolves to Python 3.9.23. ([#1810](https://github.com/heroku/heroku-buildpack-python/pull/1810)) - Updated uv from 0.7.6 to 0.7.10. ([#1811](https://github.com/heroku/heroku-buildpack-python/pull/1811)) ## [v287] - 2025-05-20 diff --git a/lib/python_version.sh b/lib/python_version.sh index 41af8a816..ea1985290 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -4,11 +4,11 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail -LATEST_PYTHON_3_9="3.9.22" -LATEST_PYTHON_3_10="3.10.17" -LATEST_PYTHON_3_11="3.11.12" -LATEST_PYTHON_3_12="3.12.10" -LATEST_PYTHON_3_13="3.13.3" +LATEST_PYTHON_3_9="3.9.23" +LATEST_PYTHON_3_10="3.10.18" +LATEST_PYTHON_3_11="3.11.13" +LATEST_PYTHON_3_12="3.12.11" +LATEST_PYTHON_3_13="3.13.4" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=9 NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=13 diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index f123fdf70..86bb211b1 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -65,7 +65,7 @@ end end - context 'when the uv version has changed since the last build' do + context 'when the uv and Python versions have changed since the last build' do let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v286'] } let(:app) { Hatchet::Runner.new('spec/fixtures/uv_basic', buildpacks:) } @@ -78,6 +78,7 @@ remote: -----> Python app detected remote: -----> Using Python 3.13 specified in .python-version remote: -----> Discarding cache since: + remote: - The Python version has changed from 3.13.3 to #{LATEST_PYTHON_3_13} remote: - The uv version has changed from 0.7.3 to #{UV_VERSION} remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing uv #{UV_VERSION} diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index b6db10188..381c501b2 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -9,11 +9,11 @@ FIXTURE_DIR = Pathname.new(__FILE__).parent.join('fixtures') -LATEST_PYTHON_3_9 = '3.9.22' -LATEST_PYTHON_3_10 = '3.10.17' -LATEST_PYTHON_3_11 = '3.11.12' -LATEST_PYTHON_3_12 = '3.12.10' -LATEST_PYTHON_3_13 = '3.13.3' +LATEST_PYTHON_3_9 = '3.9.23' +LATEST_PYTHON_3_10 = '3.10.18' +LATEST_PYTHON_3_11 = '3.11.13' +LATEST_PYTHON_3_12 = '3.12.11' +LATEST_PYTHON_3_13 = '3.13.4' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_13 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From 508474e2310d063f0a31c2f97b5e5afb881a702a Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 3 Jun 2025 22:32:34 +0000 Subject: [PATCH 147/406] Prepare release v288 (#1812) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dc4201bf1..aeba3c9f3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v288] - 2025-06-03 + - The Python 3.13 version alias now resolves to Python 3.13.4. ([#1810](https://github.com/heroku/heroku-buildpack-python/pull/1810)) - The Python 3.12 version alias now resolves to Python 3.12.11. ([#1810](https://github.com/heroku/heroku-buildpack-python/pull/1810)) - The Python 3.11 version alias now resolves to Python 3.11.13. ([#1810](https://github.com/heroku/heroku-buildpack-python/pull/1810)) @@ -1230,7 +1233,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v287...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v288...main +[v288]: https://github.com/heroku/heroku-buildpack-python/compare/v287...v288 [v287]: https://github.com/heroku/heroku-buildpack-python/compare/v286...v287 [v286]: https://github.com/heroku/heroku-buildpack-python/compare/v285...v286 [v285]: https://github.com/heroku/heroku-buildpack-python/compare/v284...v285 From 8989db154e9e7b2410eee5197154aa5f730e46a9 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 12 Jun 2025 11:43:00 +0100 Subject: [PATCH 148/406] Add support for Python 3.13.5 (#1814) Release announcement: https://blog.python.org/2025/06/python-3135-is-now-available.html Changelog: https://docs.python.org/release/3.13.5/whatsnew/changelog.html#python-3-13-5-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/15606346533 GUS-W-18767577. --- CHANGELOG.md | 1 + lib/python_version.sh | 2 +- spec/spec_helper.rb | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index aeba3c9f3..b7f69dc1b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- The Python 3.13 version alias now resolves to Python 3.13.5. ([#1814](https://github.com/heroku/heroku-buildpack-python/pull/1814)) ## [v288] - 2025-06-03 diff --git a/lib/python_version.sh b/lib/python_version.sh index ea1985290..c4a264f79 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -8,7 +8,7 @@ LATEST_PYTHON_3_9="3.9.23" LATEST_PYTHON_3_10="3.10.18" LATEST_PYTHON_3_11="3.11.13" LATEST_PYTHON_3_12="3.12.11" -LATEST_PYTHON_3_13="3.13.4" +LATEST_PYTHON_3_13="3.13.5" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=9 NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=13 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 381c501b2..ff1a39195 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -13,7 +13,7 @@ LATEST_PYTHON_3_10 = '3.10.18' LATEST_PYTHON_3_11 = '3.11.13' LATEST_PYTHON_3_12 = '3.12.11' -LATEST_PYTHON_3_13 = '3.13.4' +LATEST_PYTHON_3_13 = '3.13.5' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_13 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From 770aaed8b20dd166cd1bdafe125fc436693812d1 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 12 Jun 2025 10:49:12 +0000 Subject: [PATCH 149/406] Prepare release v289 (#1815) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b7f69dc1b..6fa870cb5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v289] - 2025-06-12 + - The Python 3.13 version alias now resolves to Python 3.13.5. ([#1814](https://github.com/heroku/heroku-buildpack-python/pull/1814)) ## [v288] - 2025-06-03 @@ -1234,7 +1237,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v288...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v289...main +[v289]: https://github.com/heroku/heroku-buildpack-python/compare/v288...v289 [v288]: https://github.com/heroku/heroku-buildpack-python/compare/v287...v288 [v287]: https://github.com/heroku/heroku-buildpack-python/compare/v286...v287 [v286]: https://github.com/heroku/heroku-buildpack-python/compare/v285...v286 From 44bbad49309b29266f43abea480c6e1963b5ba63 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 16 Jun 2025 12:13:55 +0100 Subject: [PATCH 150/406] Fix intermittent test failure in uv CI test (#1820) uv's package install log output is non-deterministic, since uv uses concurrency and the output depends on which package install finishes first. As such, the existing test would intermittently fail, eg: https://github.com/heroku/heroku-buildpack-python/actions/runs/15675903805/job/44159360614?pr=1818#step:5:703 To prevent this, the test assertion now uses a more relaxed regex that also permits either ordering of the final two packages (the top level packages that we care about, that will always be installed last). GUS-W-18792940. --- spec/hatchet/ci_spec.rb | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 724dc43b0..f57988977 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -241,7 +241,8 @@ it 'installs both normal and test dependencies and uses cache on subsequent runs' do app.run_ci do |test_run| - expect(clean_output(test_run.output)).to match(Regexp.new(<<~REGEX)) + # The uv install log output order is non-deterministic, hence the regex. + expect(clean_output(test_run.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} @@ -251,11 +252,9 @@ Prepared 5 packages in .+s Installed 5 packages in .+s Bytecode compiled .+ files in .+s - \\+ iniconfig==.+ - \\+ packaging==.+ - \\+ pluggy==.+ - \\+ pytest==.+ - \\+ typing-extensions==.+ + .+ + \\+ (pytest|typing-extensions)==.+ + \\+ (pytest|typing-extensions)==.+ -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running bin/post_compile hook CI=true From d70e0f3dc70863e8b3fba01feb877c1079ca2620 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 16 Jun 2025 11:47:24 +0000 Subject: [PATCH 151/406] Bump rubocop from 1.75.8 to 1.76.1 in the ruby-dependencies group (#1818) * Bump rubocop from 1.75.8 to 1.76.1 in the ruby-dependencies group Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.75.8 to 1.76.1 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.75.8...v1.76.1) --- updated-dependencies: - dependency-name: rubocop dependency-version: 1.76.1 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] * Refresh lockfile --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- Gemfile.lock | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 9a1ac9e33..7fdc2255d 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -2,10 +2,10 @@ GEM remote: https://rubygems.org/ specs: ast (2.4.3) - base64 (0.2.0) + base64 (0.3.0) diff-lcs (1.6.2) erubis (2.7.0) - excon (1.2.5) + excon (1.2.7) logger heroics (0.1.3) base64 @@ -51,7 +51,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.4) - rubocop (1.75.8) + rubocop (1.76.1) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) @@ -59,10 +59,10 @@ GEM parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) regexp_parser (>= 2.9.3, < 3.0) - rubocop-ast (>= 1.44.0, < 2.0) + rubocop-ast (>= 1.45.0, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 4.0) - rubocop-ast (1.44.1) + rubocop-ast (1.45.1) parser (>= 3.3.7.2) prism (~> 1.4) rubocop-rspec (3.6.0) @@ -89,7 +89,7 @@ DEPENDENCIES rubocop-rspec RUBY VERSION - ruby 3.4.3p32 + ruby 3.4.4p34 BUNDLED WITH - 2.6.8 + 2.6.9 From 8d2f7d3ff5f7eac7d1a91dae9f21599765fa9a90 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 16 Jun 2025 11:59:30 +0000 Subject: [PATCH 152/406] Bump uv from 0.7.10 to 0.7.13 (#1819) * Bump uv from 0.7.10 to 0.7.13 Bumps [uv](https://github.com/astral-sh/uv) from 0.7.10 to 0.7.13. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.7.10...0.7.13) --- updated-dependencies: - dependency-name: uv dependency-version: 0.7.13 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6fa870cb5..70a5d43c4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.7.10 to 0.7.13. ([#1819](https://github.com/heroku/heroku-buildpack-python/pull/1819)) ## [v289] - 2025-06-12 diff --git a/requirements/uv.txt b/requirements/uv.txt index 8035a0dc1..73efe59b8 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.7.10 +uv==0.7.13 From cfe59dca653e732440502d142fb282a0f10475b6 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 17 Jun 2025 14:49:57 +0100 Subject: [PATCH 153/406] Fix intermittent test failure in Poetry CI test (#1823) Similar to #1820, but this time for Poetry, whose test has been passing fine for months, but today has now started acting up too: https://github.com/heroku/heroku-buildpack-python/actions/runs/15703896757/job/44245421673?pr=1821#step:5:731 GUS-W-18807444. --- spec/hatchet/ci_spec.rb | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index f57988977..48953c89a 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -157,7 +157,8 @@ it 'installs both normal and test dependencies and uses cache on subsequent runs' do app.run_ci do |test_run| - expect(clean_output(test_run.output)).to match(Regexp.new(<<~REGEX)) + # The Poetry install log output order is non-deterministic, hence the regex. + expect(clean_output(test_run.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} @@ -167,11 +168,9 @@ Package operations: 5 installs, 0 updates, 0 removals - - Installing iniconfig .+ - - Installing packaging .+ - - Installing pluggy .+ - - Installing pytest .+ - - Installing typing-extensions .+ + .+ + - Installing (pytest|typing-extensions) .+ + - Installing (pytest|typing-extensions) .+ -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running bin/post_compile hook CI=true From 14ddc6e779e76d9aa8849ac9c718230c8f4114de Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 17 Jun 2025 14:04:23 +0000 Subject: [PATCH 154/406] Prepare release v290 (#1821) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 70a5d43c4..01ca0d9d4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v290] - 2025-06-17 + - Updated uv from 0.7.10 to 0.7.13. ([#1819](https://github.com/heroku/heroku-buildpack-python/pull/1819)) ## [v289] - 2025-06-12 @@ -1238,7 +1241,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v289...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v290...main +[v290]: https://github.com/heroku/heroku-buildpack-python/compare/v289...v290 [v289]: https://github.com/heroku/heroku-buildpack-python/compare/v288...v289 [v288]: https://github.com/heroku/heroku-buildpack-python/compare/v287...v288 [v287]: https://github.com/heroku/heroku-buildpack-python/compare/v286...v287 From 925331d90229a23b23adf944f9848b2e583bb93a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 1 Jul 2025 16:29:11 +0100 Subject: [PATCH 155/406] Bump the ruby-dependencies group with 2 updates (#1825) Bumps the ruby-dependencies group with 2 updates: [rspec-core](https://github.com/rspec/rspec) and [rubocop](https://github.com/rubocop/rubocop). Updates `rspec-core` from 3.13.4 to 3.13.5 - [Changelog](https://github.com/rspec/rspec/blob/rspec-core-v3.13.5/rspec-core/Changelog.md) - [Commits](https://github.com/rspec/rspec/compare/rspec-core-v3.13.4...rspec-core-v3.13.5) Updates `rubocop` from 1.76.1 to 1.77.0 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.76.1...v1.77.0) --- updated-dependencies: - dependency-name: rspec-core dependency-version: 3.13.5 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rubocop dependency-version: 1.77.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 7fdc2255d..5698e3d6f 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -43,7 +43,7 @@ GEM rate_throttle_client (0.1.2) regexp_parser (2.10.0) rrrretry (1.0.0) - rspec-core (3.13.4) + rspec-core (3.13.5) rspec-support (~> 3.13.0) rspec-expectations (3.13.5) diff-lcs (>= 1.2.0, < 2.0) @@ -51,7 +51,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.4) - rubocop (1.76.1) + rubocop (1.77.0) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) @@ -59,7 +59,7 @@ GEM parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) regexp_parser (>= 2.9.3, < 3.0) - rubocop-ast (>= 1.45.0, < 2.0) + rubocop-ast (>= 1.45.1, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 4.0) rubocop-ast (1.45.1) From 637af542a7d3ffaad6e4f28a454fee8bbd8858f9 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 3 Jul 2025 13:23:31 +0100 Subject: [PATCH 156/406] Further reduce team review request noise from CODEOWNERS (#1826) We already request language owner review for various automation and Dependabot related PRs, to reduce the amount of review request notification noise sent to the language owner team alias. This adds the GitHub Actions workflow directory to this list too, so that Dependabot PRs that bump the Actions versions are treated the same as the language-specific Dependabot PRs. GUS-W-18951033. --- .github/CODEOWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 33be3e324..fa79aa89b 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -7,4 +7,5 @@ # by Dependabot or release automation, to reduce team review request noise. CHANGELOG.md @edmorley Gemfile.lock @edmorley +/.github/workflows/ @edmorley /requirements/ @edmorley From 710260f6d0bf249ae6c8397193bcc50e06771f94 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 8 Jul 2025 10:24:13 +0100 Subject: [PATCH 157/406] Bump uv from 0.7.13 to 0.7.19 (#1827) * Bump uv from 0.7.13 to 0.7.19 Bumps [uv](https://github.com/astral-sh/uv) from 0.7.13 to 0.7.19. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.7.13...0.7.19) --- updated-dependencies: - dependency-name: uv dependency-version: 0.7.19 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Update test fixture assertion * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- spec/hatchet/uv_spec.rb | 3 ++- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 01ca0d9d4..3d1499d11 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.7.13 to 0.7.19. ([#1827](https://github.com/heroku/heroku-buildpack-python/pull/1827)) ## [v290] - 2025-06-17 diff --git a/requirements/uv.txt b/requirements/uv.txt index 73efe59b8..19a5be563 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.7.13 +uv==0.7.19 diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 86bb211b1..3fe994e6d 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -369,7 +369,8 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' remote: Using CPython #{DEFAULT_PYTHON_FULL_VERSION} interpreter at: /app/.heroku/python/bin/python#{DEFAULT_PYTHON_MAJOR_VERSION} - remote: error: The Python request from `.python-version` resolved to Python #{DEFAULT_PYTHON_FULL_VERSION}, which is incompatible with the project's Python requirement: `==3.12.*`. Use `uv python pin` to update the `.python-version` file to a compatible version. + remote: error: The Python request from `.python-version` resolved to Python #{DEFAULT_PYTHON_FULL_VERSION}, which is incompatible with the project's Python requirement: `==3.12.*` (from `project.requires-python`) + remote: Use `uv python pin` to update the `.python-version` file to a compatible version remote: remote: ! Error: Unable to install dependencies using uv. remote: ! From c8c134537fc89039e1f65f91cfb08718d7e85796 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 10 Jul 2025 07:14:53 +0000 Subject: [PATCH 158/406] Bump uv from 0.7.19 to 0.7.20 (#1829) * Bump uv from 0.7.19 to 0.7.20 Bumps [uv](https://github.com/astral-sh/uv) from 0.7.19 to 0.7.20. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.7.19...0.7.20) --- updated-dependencies: - dependency-name: uv dependency-version: 0.7.20 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 2 +- requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d1499d11..a4b5ebdba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## [Unreleased] -- Updated uv from 0.7.13 to 0.7.19. ([#1827](https://github.com/heroku/heroku-buildpack-python/pull/1827)) +- Updated uv from 0.7.13 to 0.7.20. ([#1827](https://github.com/heroku/heroku-buildpack-python/pull/1827) and [#1829](https://github.com/heroku/heroku-buildpack-python/pull/1829)) ## [v290] - 2025-06-17 diff --git a/requirements/uv.txt b/requirements/uv.txt index 19a5be563..d9584f673 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.7.19 +uv==0.7.20 From d12491205d858e68d47d676ba2c74b1622bea117 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 10 Jul 2025 15:11:06 +0100 Subject: [PATCH 159/406] Change the `python/` directory related warnings to errors (#1830) Builds will now error if either: - The Python buildpack has been run multiple times in the same build. - An existing `.heroku/python/` directory is found in the app source for any other reason (e.g. accidentally committed to Git) This upgrades the warnings added in #1724 in December 2024 to errors. No feedback/concerns were raised in the GitHub issue linked from the warning: https://github.com/heroku/heroku-buildpack-python/issues/1704 And Honeycomb data shows: - the number of apps hitting the warnings was minimal - in most cases, the builds hitting the warnings were failing already (eg due to broken `python/` directories having been committed to Git, and so turning the warning into an error will actually give them a better error message than before). Closes #1704. GUS-W-17309720. --- CHANGELOG.md | 6 ++++-- lib/checks.sh | 38 ++++++++++++------------------------- spec/hatchet/checks_spec.rb | 36 ++++++++--------------------------- 3 files changed, 24 insertions(+), 56 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a4b5ebdba..11598d58c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,8 @@ ## [Unreleased] - Updated uv from 0.7.13 to 0.7.20. ([#1827](https://github.com/heroku/heroku-buildpack-python/pull/1827) and [#1829](https://github.com/heroku/heroku-buildpack-python/pull/1829)) +- The build now errors if the Python buildpack has been run multiple times in the same build. This replaces the warning displayed since December 2024. ([#1830](https://github.com/heroku/heroku-buildpack-python/pull/1830)) +- The build now errors if an existing `.heroku/python/` directory is found in the app source. This replaces the warning displayed since December 2024. ([#1830](https://github.com/heroku/heroku-buildpack-python/pull/1830)) ## [v290] - 2025-06-17 @@ -112,8 +114,8 @@ ## [v272] - 2024-12-13 -- Added a warning if the Python buildpack has been run multiple times in the same build. In January 2025 this warning will be made an error. ([#1724](https://github.com/heroku/heroku-buildpack-python/pull/1724)) -- Added a warning if an existing `.heroku/python/` directory is found in the app source. In January 2025 this warning will be made an error. ([#1724](https://github.com/heroku/heroku-buildpack-python/pull/1724)) +- Added a warning if the Python buildpack has been run multiple times in the same build. In the future this warning will be made an error. ([#1724](https://github.com/heroku/heroku-buildpack-python/pull/1724)) +- Added a warning if an existing `.heroku/python/` directory is found in the app source. In the future this warning will be made an error. ([#1724](https://github.com/heroku/heroku-buildpack-python/pull/1724)) - Improved the error message shown if the buildpack is used on an unsupported stack. ([#1724](https://github.com/heroku/heroku-buildpack-python/pull/1724)) - Fixed Dev Center links to reflect recent article URL changes. ([#1723](https://github.com/heroku/heroku-buildpack-python/pull/1723)) - Added metrics for the existence of a uv lockfile. ([#1725](https://github.com/heroku/heroku-buildpack-python/pull/1725)) diff --git a/lib/checks.sh b/lib/checks.sh index 38a4ac457..d25f29c6a 100644 --- a/lib/checks.sh +++ b/lib/checks.sh @@ -41,7 +41,6 @@ function checks::ensure_supported_stack() { esac } -# TODO: Turn this into an error in January 2025. function checks::warn_if_duplicate_python_buildpack() { local build_dir="${1}" @@ -50,8 +49,8 @@ function checks::warn_if_duplicate_python_buildpack() { # (The env var can only have come from the `export` file of an earlier buildpack, # since app provided config vars haven't been exported to the environment here.) if [[ -f "${build_dir}/.heroku/python/bin/python" && -v PYTHONHOME ]]; then - output::warning <<-EOF - Warning: The Python buildpack has already been run this build. + output::error <<-EOF + Error: The Python buildpack has already been run this build. An existing Python installation was found in the build directory from a buildpack run earlier in the build. @@ -65,32 +64,21 @@ function checks::warn_if_duplicate_python_buildpack() { https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks https://devcenter.heroku.com/articles/managing-buildpacks#remove-classic-buildpacks - If you have a use-case that requires duplicate buildpacks, - please comment on: - https://github.com/heroku/heroku-buildpack-python/issues/1704 - - In January 2025 this warning will be made an error. + Note: This error replaces the deprecation warning which was + displayed in build logs starting 13th December 2024. EOF - meta_set "duplicate_python_buildpack" "true" - # shellcheck disable=SC2034 # This is used below until we make this check an error. - DUPLICATE_PYTHON_BUILDPACK=1 + meta_set "failure_reason" "checks::duplicate-python-buildpack" + exit 1 fi } -# TODO: Turn this into an error in January 2025. function checks::warn_if_existing_python_dir_present() { local build_dir="${1}" - # Avoid warning twice in the case of duplicate buildpacks. - # TODO: Remove this once `warn_if_duplicate_python_buildpack` becomes an error. - if [[ -v DUPLICATE_PYTHON_BUILDPACK ]]; then - return 0 - fi - # We use `-e` here to catch the case where `python` is a file rather than a directory. if [[ -e "${build_dir}/.heroku/python" ]]; then - output::warning <<-EOF - Warning: Existing '.heroku/python/' directory found. + output::error <<-EOF + Error: Existing '.heroku/python/' directory found. Your app's source code contains an existing directory named '.heroku/python/', which is where the Python buildpack needs @@ -107,12 +95,10 @@ function checks::warn_if_existing_python_dir_present() { Otherwise, check that an earlier buildpack or 'bin/pre_compile' hook hasn't created this directory. - If you have a use-case that requires writing to this location, - please comment on: - https://github.com/heroku/heroku-buildpack-python/issues/1704 - - In January 2025 this warning will be made an error. + Note: This error replaces the deprecation warning which was + displayed in build logs starting 13th December 2024. EOF - meta_set "existing_python_dir" "true" + meta_set "failure_reason" "checks::existing-python-dir" + exit 1 fi } diff --git a/spec/hatchet/checks_spec.rb b/spec/hatchet/checks_spec.rb index e67c8aea5..c83f3c14f 100644 --- a/spec/hatchet/checks_spec.rb +++ b/spec/hatchet/checks_spec.rb @@ -5,14 +5,14 @@ RSpec.describe 'Buildpack validation checks' do context 'when there are duplicate Python buildpacks set on the app' do let(:buildpacks) { %i[default default] } - let(:app) { Hatchet::Runner.new("spec/fixtures/python_#{DEFAULT_PYTHON_MAJOR_VERSION}", buildpacks:) } + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_unspecified', buildpacks:, allow_failure: true) } it 'fails detection' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: - remote: ! Warning: The Python buildpack has already been run this build. + remote: ! Error: The Python buildpack has already been run this build. remote: ! remote: ! An existing Python installation was found in the build directory remote: ! from a buildpack run earlier in the build. @@ -26,15 +26,10 @@ remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#remove-classic-buildpacks remote: ! - remote: ! If you have a use-case that requires duplicate buildpacks, - remote: ! please comment on: - remote: ! https://github.com/heroku/heroku-buildpack-python/issues/1704 - remote: ! - remote: ! In January 2025 this warning will be made an error. + remote: ! Note: This error replaces the deprecation warning which was + remote: ! displayed in build logs starting 13th December 2024. remote: - remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version - remote: -----> Restoring cache - remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: ! Push rejected, failed to compile Python app. OUTPUT end end @@ -48,7 +43,7 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: - remote: ! Warning: Existing '.heroku/python/' directory found. + remote: ! Error: Existing '.heroku/python/' directory found. remote: ! remote: ! Your app's source code contains an existing directory named remote: ! '.heroku/python/', which is where the Python buildpack needs @@ -67,23 +62,8 @@ remote: ! Otherwise, check that an earlier buildpack or 'bin/pre_compile' remote: ! hook hasn't created this directory. remote: ! - remote: ! If you have a use-case that requires writing to this location, - remote: ! please comment on: - remote: ! https://github.com/heroku/heroku-buildpack-python/issues/1704 - remote: ! - remote: ! In January 2025 this warning will be made an error. - remote: - remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version - remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: - remote: ! Internal Error: Unable to locate the Python stdlib's bundled pip. - remote: ! - remote: ! Couldn't find the pip wheel file bundled inside the Python - remote: ! stdlib's 'ensurepip' module: - remote: ! - remote: ! find: ‘/app/.heroku/python/lib/python3.13/ensurepip/_bundled/’: No such file or directory - remote: ! /app/.heroku/python/ - remote: ! /app/.heroku/python/bin + remote: ! Note: This error replaces the deprecation warning which was + remote: ! displayed in build logs starting 13th December 2024. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT From 3a0dde760d03622fb14373b6d565eb3820e9ef2f Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 10 Jul 2025 14:24:21 +0000 Subject: [PATCH 160/406] Prepare release v291 (#1831) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 11598d58c..391022f32 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v291] - 2025-07-10 + - Updated uv from 0.7.13 to 0.7.20. ([#1827](https://github.com/heroku/heroku-buildpack-python/pull/1827) and [#1829](https://github.com/heroku/heroku-buildpack-python/pull/1829)) - The build now errors if the Python buildpack has been run multiple times in the same build. This replaces the warning displayed since December 2024. ([#1830](https://github.com/heroku/heroku-buildpack-python/pull/1830)) - The build now errors if an existing `.heroku/python/` directory is found in the app source. This replaces the warning displayed since December 2024. ([#1830](https://github.com/heroku/heroku-buildpack-python/pull/1830)) @@ -1244,7 +1247,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v290...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v291...main +[v291]: https://github.com/heroku/heroku-buildpack-python/compare/v290...v291 [v290]: https://github.com/heroku/heroku-buildpack-python/compare/v289...v290 [v289]: https://github.com/heroku/heroku-buildpack-python/compare/v288...v289 [v288]: https://github.com/heroku/heroku-buildpack-python/compare/v287...v288 From 608a912bab78f20f872e7ede87a0a89694b3f1c7 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 15 Jul 2025 07:45:54 +0100 Subject: [PATCH 161/406] Remove unused metrics (#1832) These are not set/used since #1830. --- bin/report | 2 -- 1 file changed, 2 deletions(-) diff --git a/bin/report b/bin/report index 85ab72f94..77af44eaa 100755 --- a/bin/report +++ b/bin/report @@ -87,8 +87,6 @@ ALL_OTHER_FIELDS=( custom_s3_base_url dependencies_install_duration django_collectstatic_duration - duplicate_python_buildpack - existing_python_dir nltk_downloader_duration package_manager_install_duration pipenv_has_lockfile From cf11b63a40050c2eb76fa7eed1e23985037ab8fa Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 15 Jul 2025 13:04:44 +0100 Subject: [PATCH 162/406] Require a `Pipfile.lock` when using Pipenv (#1833) For historical reasons, the buildpack has supported using the package manager Pipenv without a lockfile. However, doing so results in non-deterministic installs (package versions can vary across environments) and so is unsafe / not recommended. As such, since #1695 in November 2024, we've shown a deprecation warning if apps have only a `Pipfile` and no `Pipfile.lock` file. Plenty of time has now passed, and metrics show usage very few apps hitting this warning. In addition, upstream Pipenv has also deprecated support for `--skip-lock`: https://github.com/pypa/pipenv/commit/8775d5998e389ac43ddddb8db752fc222d6c7c0f As such, it's time to change this warning to an error. Affected apps will need to run `pipenv lock` and commit the resultant `Pipfile.lock` to Git. Closes #1702. GUS-W-17308738. --- CHANGELOG.md | 1 + README.md | 3 +- bin/detect | 4 +- bin/report | 1 - lib/package_manager.sh | 25 ++++++----- lib/pipenv.sh | 8 +--- lib/python_version.sh | 6 --- spec/hatchet/detect_spec.rb | 4 +- spec/hatchet/package_manager_spec.rb | 4 +- spec/hatchet/pipenv_spec.rb | 64 ++++------------------------ 10 files changed, 33 insertions(+), 87 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 391022f32..3b3a56c4b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- The build now errors when using Pipenv without its lockfile (`Pipfile.lock`). This replaces the warning displayed since November 2024. ([#1833](https://github.com/heroku/heroku-buildpack-python/pull/1833)) ## [v291] - 2025-07-10 diff --git a/README.md b/README.md index 7c2733023..506c28646 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,8 @@ See the [Getting Started on Heroku with Python](https://devcenter.heroku.com/art ## Application Requirements -A `requirements.txt`, `Pipfile`, `poetry.lock`, or `uv.lock` file must be present in the root (top-level) directory of your app's source code. +A `requirements.txt`, `Pipfile.lock`, `poetry.lock`, or `uv.lock` file must be present in the root (top-level) +directory of your app's source code. When using the package manager [uv](https://docs.astral.sh/uv/) a `.python-version` file is also required. diff --git a/bin/detect b/bin/detect index c532ee044..a45389ffc 100755 --- a/bin/detect +++ b/bin/detect @@ -65,8 +65,8 @@ Error: Your app is configured to use the Python buildpack, but we couldn't find any supported Python project files. A Python app on Heroku must have either a 'requirements.txt', -'Pipfile' or 'poetry.lock' package manager file in the root -directory of its source code. +'Pipfile.lock', 'poetry.lock' or 'uv.lock' package manager file +in the root directory of its source code. Currently the root directory of your app contains: diff --git a/bin/report b/bin/report index 77af44eaa..54065955b 100755 --- a/bin/report +++ b/bin/report @@ -89,7 +89,6 @@ ALL_OTHER_FIELDS=( django_collectstatic_duration nltk_downloader_duration package_manager_install_duration - pipenv_has_lockfile post_compile_hook post_compile_hook_duration pre_compile_hook diff --git a/lib/package_manager.sh b/lib/package_manager.sh index 091600e16..9c2fd9bfb 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -12,26 +12,29 @@ function package_manager::determine_package_manager() { if [[ -f "${build_dir}/Pipfile.lock" ]]; then package_managers_found+=(pipenv) package_managers_found_display_text+=("Pipfile.lock (Pipenv)") - meta_set "pipenv_has_lockfile" "true" elif [[ -f "${build_dir}/Pipfile" ]]; then - # TODO: Start requiring a Pipfile.lock and make this branch a "missing lockfile" error instead. - output::warning <<-'EOF' - Warning: No 'Pipfile.lock' found! + output::error <<-'EOF' + Error: No 'Pipfile.lock' found! A 'Pipfile' file was found, however, the associated 'Pipfile.lock' Pipenv lockfile was not. This means your app dependency versions aren't pinned, which means the package versions used on Heroku might not match those installed in other environments. - For now, we will install your dependencies without a lockfile, - however, in the future this warning will become an error. + Using Pipenv in this way is unsafe and no longer supported. Run 'pipenv lock' locally to generate the lockfile, and make sure that 'Pipfile.lock' isn't listed in '.gitignore' or '.slugignore'. + + Alternatively, if you wish to switch to another package manager, + delete your 'Pipfile' and then add either a 'requirements.txt', + 'poetry.lock' or 'uv.lock' file. + + Note: This error replaces the warning which was displayed in + build logs starting 12th November 2024. EOF - package_managers_found+=(pipenv) - package_managers_found_display_text+=("Pipfile (Pipenv)") - meta_set "pipenv_has_lockfile" "false" + meta_set "failure_reason" "package-manager::pipenv-missing-lockfile" + exit 1 fi if [[ -f "${build_dir}/requirements.txt" ]]; then @@ -76,8 +79,8 @@ function package_manager::determine_package_manager() { Error: Couldn't find any supported Python package manager files. A Python app on Heroku must have either a 'requirements.txt', - 'Pipfile', 'poetry.lock' or 'uv.lock' package manager file in - the root directory of its source code. + 'Pipfile.lock', 'poetry.lock' or 'uv.lock' package manager file + in the root directory of its source code. Currently the root directory of your app contains: diff --git a/lib/pipenv.sh b/lib/pipenv.sh index a43192a37..715279eff 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -66,15 +66,9 @@ function pipenv::install_dependencies() { local pipenv_install_command=( pipenv install + --deploy ) - # TODO: Make Pipfile.lock mandatory during package manager selection. - if [[ ! -f Pipfile.lock ]]; then - pipenv_install_command+=(--skip-lock) - else - pipenv_install_command+=(--deploy) - fi - # Install test dependencies too when the buildpack is invoked via `bin/test-compile` on Heroku CI. if [[ -v INSTALL_TEST ]]; then pipenv_install_command+=(--dev) diff --git a/lib/python_version.sh b/lib/python_version.sh index c4a264f79..3fcb7afcc 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -233,12 +233,6 @@ function python_version::read_pipenv_python_version() { local pipfile_lock_path="${build_dir}/Pipfile.lock" local version - # We currently permit using Pipenv without a `Pipfile.lock`, however, in the future we will - # require a lockfile, at which point this conditional can be removed. - if [[ ! -f "${pipfile_lock_path}" ]]; then - return 0 - fi - if ! version=$(jq --raw-output '._meta.requires.python_full_version // ._meta.requires.python_version' "${pipfile_lock_path}" 2>&1); then local jq_error_message="${version}" output::error <<-EOF diff --git a/spec/hatchet/detect_spec.rb b/spec/hatchet/detect_spec.rb index edbe242df..04b83ba49 100644 --- a/spec/hatchet/detect_spec.rb +++ b/spec/hatchet/detect_spec.rb @@ -18,8 +18,8 @@ remote: ! but we couldn't find any supported Python project files. remote: ! remote: ! A Python app on Heroku must have either a 'requirements.txt', - remote: ! 'Pipfile' or 'poetry.lock' package manager file in the root - remote: ! directory of its source code. + remote: ! 'Pipfile.lock', 'poetry.lock' or 'uv.lock' package manager file + remote: ! in the root directory of its source code. remote: ! remote: ! Currently the root directory of your app contains: remote: ! diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index 3e3f4c49d..fd086e69b 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -14,8 +14,8 @@ remote: ! Error: Couldn't find any supported Python package manager files. remote: ! remote: ! A Python app on Heroku must have either a 'requirements.txt', - remote: ! 'Pipfile', 'poetry.lock' or 'uv.lock' package manager file in - remote: ! the root directory of its source code. + remote: ! 'Pipfile.lock', 'poetry.lock' or 'uv.lock' package manager file + remote: ! in the root directory of its source code. remote: ! remote: ! Currently the root directory of your app contains: remote: ! diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 771338cf9..2c1457fd6 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -200,79 +200,33 @@ end context 'without a Pipfile.lock' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_no_lockfile') } + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_no_lockfile', allow_failure: true) } it 'builds with the default Python version using just the Pipfile' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected remote: - remote: ! Warning: No 'Pipfile.lock' found! + remote: ! Error: No 'Pipfile.lock' found! remote: ! remote: ! A 'Pipfile' file was found, however, the associated 'Pipfile.lock' remote: ! Pipenv lockfile was not. This means your app dependency versions remote: ! aren't pinned, which means the package versions used on Heroku remote: ! might not match those installed in other environments. remote: ! - remote: ! For now, we will install your dependencies without a lockfile, - remote: ! however, in the future this warning will become an error. + remote: ! Using Pipenv in this way is unsafe and no longer supported. remote: ! remote: ! Run 'pipenv lock' locally to generate the lockfile, and make sure remote: ! that 'Pipfile.lock' isn't listed in '.gitignore' or '.slugignore'. - remote: - remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} - remote: - remote: ! Warning: No Python version was specified. - remote: ! - remote: ! Your app doesn't specify a Python version and so the buildpack - remote: ! picked a default version for you. - remote: ! - remote: ! Relying on this default version isn't recommended, since it - remote: ! can change over time and may not be consistent with your local - remote: ! development environment, CI or other instances of your app. - remote: ! - remote: ! Please configure an explicit Python version for your app. - remote: ! - remote: ! Create a new file in the root directory of your app named: - remote: ! .python-version - remote: ! - remote: ! Make sure to include the '.' character at the start of the - remote: ! filename. Don't add a file extension such as '.txt'. - remote: ! - remote: ! In the new file, specify your app's major Python version number - remote: ! only. Don't include quotes or a 'python-' prefix. remote: ! - remote: ! For example, to request the latest version of Python #{DEFAULT_PYTHON_MAJOR_VERSION}, - remote: ! update your .python-version file so it contains exactly: - remote: ! #{DEFAULT_PYTHON_MAJOR_VERSION} - remote: ! - remote: ! We strongly recommend that you don't specify the Python patch - remote: ! version number, since it will pin your app to an exact Python - remote: ! version and so stop your app from receiving security updates - remote: ! each time it builds. - remote: ! - remote: ! If your app already has a .python-version file, check that it: - remote: ! - remote: ! 1. Is in the top level directory \\(not a subdirectory\\). - remote: ! 2. Is named exactly '.python-version' in all lowercase. - remote: ! 3. Isn't listed in '.gitignore' or '.slugignore'. - remote: ! 4. Has been added to the Git repository using 'git add --all' - remote: ! and then committed using 'git commit'. + remote: ! Alternatively, if you wish to switch to another package manager, + remote: ! delete your 'Pipfile' and then add either a 'requirements.txt', + remote: ! 'poetry.lock' or 'uv.lock' file. remote: ! - remote: ! In the future we will require the use of a .python-version - remote: ! file and this warning will be made an error. + remote: ! Note: This error replaces the warning which was displayed in + remote: ! build logs starting 12th November 2024. remote: - remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION} - remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing dependencies using 'pipenv install --skip-lock' - remote: The flag --skip-lock has been reintroduced \\(but is not recommended\\). Without - remote: the lock resolver it is difficult to manage multiple package indexes, and hash - remote: checking is not provided. However it can help manage installs with current - remote: deficiencies in locking across platforms. - remote: Pipfile.lock not found, creating... - .+ - remote: Installing dependencies from Pipfile... + remote: ! Push rejected, failed to compile Python app. REGEX end end From c519e5a84bb396668ae21e5bc5aac7ece37ab30f Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 15 Jul 2025 15:06:24 +0100 Subject: [PATCH 163/406] Port uv test improvements to other package manager tests (#1834) Several improvements were made to uv's package manager related tests when adding support for uv. This ports those improvements to the tests for the other package managers too. A couple of the new tests have to be marked as skipped, due to bugs that they exposed upstream in Poetry and Pipenv. GUS-W-18319828. GUS-W-18319833. --- .../local_package_pyproject_toml/__init__.py | 2 +- .../local_package_setup_py/__init__.py | 2 +- .../pipenv_editable/bin/test-entrypoints.sh | 3 ++ .../local_package_pyproject_toml/__init__.py | 2 +- .../local_package_setup_py/__init__.py | 2 +- .../pipenv_editable/__init__.py | 2 + spec/fixtures/pipenv_editable/pyproject.toml | 3 ++ .../.python-version | 2 + .../pipenv_mismatched_python_version/Pipfile | 10 +++++ .../Pipfile.lock | 30 ++++++++++++++ .../poetry_editable/bin/test-entrypoints.sh | 3 ++ .../local_package_pyproject_toml/__init__.py | 2 +- .../local_package_setup_py/__init__.py | 2 +- .../poetry_editable/__init__.py | 2 + spec/fixtures/poetry_editable/pyproject.toml | 3 ++ .../.python-version | 2 + .../poetry.lock | 18 ++++++++ .../pyproject.toml | 13 ++++++ spec/hatchet/pip_spec.rb | 20 ++++----- spec/hatchet/pipenv_spec.rb | 41 ++++++++++++++----- spec/hatchet/poetry_spec.rb | 41 ++++++++++++++----- 21 files changed, 169 insertions(+), 36 deletions(-) create mode 100644 spec/fixtures/pipenv_mismatched_python_version/.python-version create mode 100644 spec/fixtures/pipenv_mismatched_python_version/Pipfile create mode 100644 spec/fixtures/pipenv_mismatched_python_version/Pipfile.lock create mode 100644 spec/fixtures/poetry_mismatched_python_version/.python-version create mode 100644 spec/fixtures/poetry_mismatched_python_version/poetry.lock create mode 100644 spec/fixtures/poetry_mismatched_python_version/pyproject.toml diff --git a/spec/fixtures/pip_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py b/spec/fixtures/pip_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py index b86d791d2..6ce02c341 100644 --- a/spec/fixtures/pip_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py +++ b/spec/fixtures/pip_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py @@ -1,2 +1,2 @@ def hello(): - print("Hello pyproject.toml!") + print("Hello from pyproject.toml!") diff --git a/spec/fixtures/pip_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py b/spec/fixtures/pip_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py index e98d42373..8dccb8a0c 100644 --- a/spec/fixtures/pip_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py +++ b/spec/fixtures/pip_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py @@ -1,2 +1,2 @@ def hello(): - print("Hello setup.py!") + print("Hello from setup.py!") diff --git a/spec/fixtures/pipenv_editable/bin/test-entrypoints.sh b/spec/fixtures/pipenv_editable/bin/test-entrypoints.sh index 0e66f44bb..8057cbdcf 100755 --- a/spec/fixtures/pipenv_editable/bin/test-entrypoints.sh +++ b/spec/fixtures/pipenv_editable/bin/test-entrypoints.sh @@ -8,6 +8,9 @@ cd .heroku/python/lib/python*/site-packages/ grep --extended-regexp --only-matching -- '/\S+' *.pth __editable___*_finder.py | sort echo +echo -n "Running entrypoint for the current package: " +pipenv-editable + echo -n "Running entrypoint for the pyproject.toml-based local package: " local_package_pyproject_toml diff --git a/spec/fixtures/pipenv_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py b/spec/fixtures/pipenv_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py index b86d791d2..6ce02c341 100644 --- a/spec/fixtures/pipenv_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py +++ b/spec/fixtures/pipenv_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py @@ -1,2 +1,2 @@ def hello(): - print("Hello pyproject.toml!") + print("Hello from pyproject.toml!") diff --git a/spec/fixtures/pipenv_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py b/spec/fixtures/pipenv_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py index e98d42373..8dccb8a0c 100644 --- a/spec/fixtures/pipenv_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py +++ b/spec/fixtures/pipenv_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py @@ -1,2 +1,2 @@ def hello(): - print("Hello setup.py!") + print("Hello from setup.py!") diff --git a/spec/fixtures/pipenv_editable/pipenv_editable/__init__.py b/spec/fixtures/pipenv_editable/pipenv_editable/__init__.py index e69de29bb..0f03b9343 100644 --- a/spec/fixtures/pipenv_editable/pipenv_editable/__init__.py +++ b/spec/fixtures/pipenv_editable/pipenv_editable/__init__.py @@ -0,0 +1,2 @@ +def main() -> None: + print("Hello from pipenv-editable!") diff --git a/spec/fixtures/pipenv_editable/pyproject.toml b/spec/fixtures/pipenv_editable/pyproject.toml index 2ca4481db..f6dd79f06 100644 --- a/spec/fixtures/pipenv_editable/pyproject.toml +++ b/spec/fixtures/pipenv_editable/pyproject.toml @@ -3,6 +3,9 @@ name = "pipenv-editable" version = "0.0.0" requires-python = ">=3.13" +[project.scripts] +pipenv-editable = "pipenv_editable:main" + [build-system] requires = ["hatchling"] build-backend = "hatchling.build" diff --git a/spec/fixtures/pipenv_mismatched_python_version/.python-version b/spec/fixtures/pipenv_mismatched_python_version/.python-version new file mode 100644 index 000000000..d0c1d45b8 --- /dev/null +++ b/spec/fixtures/pipenv_mismatched_python_version/.python-version @@ -0,0 +1,2 @@ +# The version here intentionally doesn't match `python_version` in Pipfile.lock +3.13 diff --git a/spec/fixtures/pipenv_mismatched_python_version/Pipfile b/spec/fixtures/pipenv_mismatched_python_version/Pipfile new file mode 100644 index 000000000..f438ee65c --- /dev/null +++ b/spec/fixtures/pipenv_mismatched_python_version/Pipfile @@ -0,0 +1,10 @@ +[[source]] +url = "https://pypi.org/simple" +verify_ssl = true +name = "pypi" + +[packages] +typing-extensions = "*" + +[requires] +python_version = "3.12" diff --git a/spec/fixtures/pipenv_mismatched_python_version/Pipfile.lock b/spec/fixtures/pipenv_mismatched_python_version/Pipfile.lock new file mode 100644 index 000000000..6fdc90f5c --- /dev/null +++ b/spec/fixtures/pipenv_mismatched_python_version/Pipfile.lock @@ -0,0 +1,30 @@ +{ + "_meta": { + "hash": { + "sha256": "9661ed313a79ccb68c7dc4e639068f86ddd91e307ec2ed60498858d002e9b547" + }, + "pipfile-spec": 6, + "requires": { + "python_version": "3.12" + }, + "sources": [ + { + "name": "pypi", + "url": "https://pypi.org/simple", + "verify_ssl": true + } + ] + }, + "default": { + "typing-extensions": { + "hashes": [ + "sha256:a439e7c04b49fec3e5d3e2beaa21755cadbbdc391694e28ccdd36ca4a1408f8c", + "sha256:e6c81219bd689f51865d9e372991c540bda33a0379d5573cddb9a3a23f7caaef" + ], + "index": "pypi", + "markers": "python_version >= '3.8'", + "version": "==4.13.2" + } + }, + "develop": {} +} diff --git a/spec/fixtures/poetry_editable/bin/test-entrypoints.sh b/spec/fixtures/poetry_editable/bin/test-entrypoints.sh index 0e66f44bb..dbdf50fee 100755 --- a/spec/fixtures/poetry_editable/bin/test-entrypoints.sh +++ b/spec/fixtures/poetry_editable/bin/test-entrypoints.sh @@ -8,6 +8,9 @@ cd .heroku/python/lib/python*/site-packages/ grep --extended-regexp --only-matching -- '/\S+' *.pth __editable___*_finder.py | sort echo +echo -n "Running entrypoint for the current package: " +poetry-editable + echo -n "Running entrypoint for the pyproject.toml-based local package: " local_package_pyproject_toml diff --git a/spec/fixtures/poetry_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py b/spec/fixtures/poetry_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py index b86d791d2..6ce02c341 100644 --- a/spec/fixtures/poetry_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py +++ b/spec/fixtures/poetry_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py @@ -1,2 +1,2 @@ def hello(): - print("Hello pyproject.toml!") + print("Hello from pyproject.toml!") diff --git a/spec/fixtures/poetry_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py b/spec/fixtures/poetry_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py index e98d42373..8dccb8a0c 100644 --- a/spec/fixtures/poetry_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py +++ b/spec/fixtures/poetry_editable/packages/local_package_setup_py/local_package_setup_py/__init__.py @@ -1,2 +1,2 @@ def hello(): - print("Hello setup.py!") + print("Hello from setup.py!") diff --git a/spec/fixtures/poetry_editable/poetry_editable/__init__.py b/spec/fixtures/poetry_editable/poetry_editable/__init__.py index e69de29bb..e00661f98 100644 --- a/spec/fixtures/poetry_editable/poetry_editable/__init__.py +++ b/spec/fixtures/poetry_editable/poetry_editable/__init__.py @@ -0,0 +1,2 @@ +def main() -> None: + print("Hello from poetry-editable!") diff --git a/spec/fixtures/poetry_editable/pyproject.toml b/spec/fixtures/poetry_editable/pyproject.toml index 256e2d984..8df4cea76 100644 --- a/spec/fixtures/poetry_editable/pyproject.toml +++ b/spec/fixtures/poetry_editable/pyproject.toml @@ -10,6 +10,9 @@ gunicorn = { git = "https://github.com/benoitc/gunicorn.git", develop = true } local-package-pyproject-toml = { path = "packages/local_package_pyproject_toml", develop = true } local-package-setup-py = { path = "packages/local_package_setup_py", develop = true } +[tool.poetry.scripts] +poetry-editable = 'poetry_editable:main' + [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api" diff --git a/spec/fixtures/poetry_mismatched_python_version/.python-version b/spec/fixtures/poetry_mismatched_python_version/.python-version new file mode 100644 index 000000000..88c9ebc82 --- /dev/null +++ b/spec/fixtures/poetry_mismatched_python_version/.python-version @@ -0,0 +1,2 @@ +# The version here intentionally doesn't match requires-python in pyproject.toml +3.13 diff --git a/spec/fixtures/poetry_mismatched_python_version/poetry.lock b/spec/fixtures/poetry_mismatched_python_version/poetry.lock new file mode 100644 index 000000000..5da81ae0f --- /dev/null +++ b/spec/fixtures/poetry_mismatched_python_version/poetry.lock @@ -0,0 +1,18 @@ +# This file is automatically @generated by Poetry 2.1.3 and should not be changed by hand. + +[[package]] +name = "typing-extensions" +version = "4.13.2" +description = "Backported and Experimental Type Hints for Python 3.8+" +optional = false +python-versions = ">=3.8" +groups = ["main"] +files = [ + {file = "typing_extensions-4.13.2-py3-none-any.whl", hash = "sha256:a439e7c04b49fec3e5d3e2beaa21755cadbbdc391694e28ccdd36ca4a1408f8c"}, + {file = "typing_extensions-4.13.2.tar.gz", hash = "sha256:e6c81219bd689f51865d9e372991c540bda33a0379d5573cddb9a3a23f7caaef"}, +] + +[metadata] +lock-version = "2.1" +python-versions = "3.12.*" +content-hash = "30427e182c131408fcdf15a6838a771f7e7c1e67fa6e0c33ccc82194052de8df" diff --git a/spec/fixtures/poetry_mismatched_python_version/pyproject.toml b/spec/fixtures/poetry_mismatched_python_version/pyproject.toml new file mode 100644 index 000000000..ed26e3dc4 --- /dev/null +++ b/spec/fixtures/poetry_mismatched_python_version/pyproject.toml @@ -0,0 +1,13 @@ +[project] +name = "poetry-mismatched-python-version" +version = "0.0.0" +# This Python version intentionally doesn't match that in .python-version. +# The version here also matches the distro Python version in Ubuntu 24.04, +# so we can also test Poetry doesn't fall back to system Python. +requires-python = "3.12.*" +dependencies = [ + "typing-extensions" +] + +[tool.poetry] +package-mode = false diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 5cc9f8b8e..0db1f7693 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -108,8 +108,8 @@ remote: gunicorn.egg-link:/app/.heroku/python/src/gunicorn remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) remote: -----> Inline app detected remote: easy-install.pth:/app/.heroku/python/src/gunicorn @@ -118,8 +118,8 @@ remote: gunicorn.egg-link:/app/.heroku/python/src/gunicorn remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) REGEX @@ -131,8 +131,8 @@ gunicorn.egg-link:/app/.heroku/python/src/gunicorn local-package-setup-py.egg-link:/app/packages/local_package_setup_py - Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - Running entrypoint for the setup.py-based local package: Hello setup.py! + Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + Running entrypoint for the setup.py-based local package: Hello from setup.py! Running entrypoint for the VCS package: gunicorn (version 20.1.0) OUTPUT @@ -147,8 +147,8 @@ remote: gunicorn.egg-link:/app/.heroku/python/src/gunicorn remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) remote: -----> Inline app detected remote: easy-install.pth:/app/.heroku/python/src/gunicorn @@ -157,8 +157,8 @@ remote: gunicorn.egg-link:/app/.heroku/python/src/gunicorn remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) REGEX # Test that the VCS repo checkout was cached correctly. diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 2c1457fd6..78346f03c 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -407,8 +407,9 @@ remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: _pipenv_editable.pth:/tmp/build_.+ remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the current package: Hello from pipenv-editable! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Inline app detected remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} @@ -416,8 +417,9 @@ remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: _pipenv_editable.pth:/tmp/build_.+ remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the current package: Hello from pipenv-editable! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) REGEX @@ -428,8 +430,9 @@ __editable___local_package_setup_py_0_0_1_finder.py:/app/packages/local_package_setup_py/local_package_setup_py'} _pipenv_editable.pth:/app - Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - Running entrypoint for the setup.py-based local package: Hello setup.py! + Running entrypoint for the current package: Hello from pipenv-editable! + Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + Running entrypoint for the setup.py-based local package: Hello from setup.py! Running entrypoint for the VCS package: gunicorn (version 23.0.0) OUTPUT @@ -445,8 +448,9 @@ remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: _pipenv_editable.pth:/tmp/build_.+ remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the current package: Hello from pipenv-editable! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Inline app detected remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} @@ -454,14 +458,31 @@ remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: _pipenv_editable.pth:/tmp/build_.+ remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the current package: Hello from pipenv-editable! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) REGEX end end end + # This tests that Pipenv doesn't fall back to system Python if the Python version in + # pyproject.toml doesn't match that in Pipfile / Pipfile.lock. + context 'when requires-python in pyproject.toml is incompatible with .python-version', + skip: 'this is currently broken upstream: https://github.com/pypa/pipenv/issues/6403' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_mismatched_python_version', allow_failure: true) } + + it 'fails the build' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Installing dependencies using 'pipenv install --deploy' + remote: + OUTPUT + end + end + end + # This tests not only our handling of failing dependency installation, but also that we're running # Pipenv in such a way that it errors if the lockfile is out of sync, rather than simply updating it. context 'when the Pipfile.lock is out of sync with Pipfile' do diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 242718664..b2b5a782f 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -115,8 +115,9 @@ remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: poetry_editable.pth:/tmp/build_.+ remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the current package: Hello from poetry-editable! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Inline app detected remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} @@ -124,8 +125,9 @@ remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: poetry_editable.pth:/tmp/build_.+ remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the current package: Hello from poetry-editable! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) REGEX @@ -136,8 +138,9 @@ __editable___local_package_setup_py_0_0_1_finder.py:/app/packages/local_package_setup_py/local_package_setup_py'} poetry_editable.pth:/app - Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - Running entrypoint for the setup.py-based local package: Hello setup.py! + Running entrypoint for the current package: Hello from poetry-editable! + Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + Running entrypoint for the setup.py-based local package: Hello from setup.py! Running entrypoint for the VCS package: gunicorn (version 23.0.0) OUTPUT @@ -161,8 +164,9 @@ remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: poetry_editable.pth:/tmp/build_.+ remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the current package: Hello from poetry-editable! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Inline app detected remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} @@ -170,8 +174,9 @@ remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: poetry_editable.pth:/tmp/build_.+ remote: - remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml! - remote: Running entrypoint for the setup.py-based local package: Hello setup.py! + remote: Running entrypoint for the current package: Hello from poetry-editable! + remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! + remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) REGEX end @@ -236,6 +241,22 @@ end end + # This tests that Poetry doesn't download its own Python or fall back to system Python + # if the Python version in pyproject.toml doesn't match that in .python-version. + context 'when requires-python in pyproject.toml is incompatible with .python-version', + skip: 'this is currently broken upstream: https://github.com/python-poetry/poetry/issues/10226' do + let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_mismatched_python_version', allow_failure: true) } + + it 'fails the build' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Installing dependencies using 'poetry sync --only main' + remote: + OUTPUT + end + end + end + # This tests not only our handling of failing dependency installation, but also that we're running # Poetry in such a way that it errors if the lockfile is out of sync, rather than simply updating it. context 'when poetry.lock is out of sync with pyproject.toml' do From e8652f390f0ba6bc5cab2d2f0e4e078d28e93841 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 18 Jul 2025 12:43:21 +0100 Subject: [PATCH 164/406] Update checks function names now that they error (#1836) After #1830, the two `checks::warn_if_*` functions now error rather than warn, so the function names have been renamed so they don't mislead about their behaviour. --- bin/compile | 4 ++-- lib/checks.sh | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/bin/compile b/bin/compile index f938acef9..8ab418cc6 100755 --- a/bin/compile +++ b/bin/compile @@ -42,7 +42,7 @@ meta_init "${CACHE_DIR}" "python" meta_setup checks::ensure_supported_stack "${STACK:?Required env var STACK is not set}" -checks::warn_if_duplicate_python_buildpack "${BUILD_DIR}" +checks::duplicate_python_buildpack "${BUILD_DIR}" # Prepend proper path for old-school virtualenv hackery. # This may not be necessary. @@ -102,7 +102,7 @@ hooks::run_hook "pre_compile" # This check must be after the pre_compile hook, so that we can check not only the original # app source, but also that the hook hasn't written to '.heroku/python/' either. -checks::warn_if_existing_python_dir_present "${BUILD_DIR}" +checks::existing_python_dir_present "${BUILD_DIR}" package_manager="$(package_manager::determine_package_manager "${BUILD_DIR}")" meta_set "package_manager" "${package_manager}" diff --git a/lib/checks.sh b/lib/checks.sh index d25f29c6a..e5f0bc478 100644 --- a/lib/checks.sh +++ b/lib/checks.sh @@ -41,7 +41,7 @@ function checks::ensure_supported_stack() { esac } -function checks::warn_if_duplicate_python_buildpack() { +function checks::duplicate_python_buildpack() { local build_dir="${1}" # The check for the `PYTHONHOME` env var prevents this warning triggering in the case @@ -72,7 +72,7 @@ function checks::warn_if_duplicate_python_buildpack() { fi } -function checks::warn_if_existing_python_dir_present() { +function checks::existing_python_dir_present() { local build_dir="${1}" # We use `-e` here to catch the case where `python` is a file rather than a directory. From 5cfd16cd7b58086a850c3852294836e3139b5784 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 18 Jul 2025 12:47:35 +0100 Subject: [PATCH 165/406] Port package manager test improvements from CNB (#1835) Misc test improvements noticed while working on the Pipenv refactor + version update. GUS-W-17482289. --- spec/fixtures/pip_basic/bin/compile | 1 + spec/fixtures/pipenv_basic/bin/compile | 4 ++-- spec/fixtures/poetry_basic/bin/compile | 1 + spec/fixtures/uv_basic/bin/compile | 1 + spec/hatchet/pip_spec.rb | 4 ++++ spec/hatchet/pipenv_spec.rb | 4 ++++ spec/hatchet/poetry_spec.rb | 1 + spec/hatchet/uv_spec.rb | 1 + 8 files changed, 15 insertions(+), 2 deletions(-) diff --git a/spec/fixtures/pip_basic/bin/compile b/spec/fixtures/pip_basic/bin/compile index 29cce648e..cb66e67da 100755 --- a/spec/fixtures/pip_basic/bin/compile +++ b/spec/fixtures/pip_basic/bin/compile @@ -16,6 +16,7 @@ python -c 'import pprint, sys; pprint.pp(sys.path)' echo # TODO: Remove --disable-pip-version-check in favour of exporting PIP_DISABLE_PIP_VERSION_CHECK +pip --version --disable-pip-version-check pip list --disable-pip-version-check echo diff --git a/spec/fixtures/pipenv_basic/bin/compile b/spec/fixtures/pipenv_basic/bin/compile index f73ce9e2f..83cb72a51 100755 --- a/spec/fixtures/pipenv_basic/bin/compile +++ b/spec/fixtures/pipenv_basic/bin/compile @@ -15,8 +15,8 @@ echo python -c 'import pprint, sys; pprint.pp(sys.path)' echo -# TODO: Investigate why 'pipenv graph' doesn't work here. -# TODO: Remove --disable-pip-version-check in favour of exporting PIP_DISABLE_PIP_VERSION_CHECK +pipenv --version +# We have to resort to using pip to list installed packages, since `pipenv graph` doesn't support `--system`. pip list --disable-pip-version-check echo diff --git a/spec/fixtures/poetry_basic/bin/compile b/spec/fixtures/poetry_basic/bin/compile index 55423bd97..9491a868e 100755 --- a/spec/fixtures/poetry_basic/bin/compile +++ b/spec/fixtures/poetry_basic/bin/compile @@ -15,6 +15,7 @@ echo python -c 'import pprint, sys; pprint.pp(sys.path)' echo +poetry --version # The show command also lists dependencies that are in optional groups in pyproject.toml # but that aren't actually installed, for which the only option is to filter out by hand. poetry show | grep -v ' (!) ' diff --git a/spec/fixtures/uv_basic/bin/compile b/spec/fixtures/uv_basic/bin/compile index 8b5578776..0e3ad460c 100755 --- a/spec/fixtures/uv_basic/bin/compile +++ b/spec/fixtures/uv_basic/bin/compile @@ -15,6 +15,7 @@ echo python -c 'import pprint, sys; pprint.pp(sys.path)' echo +uv --version uv pip list echo diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 0db1f7693..1fd1ed721 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -37,6 +37,7 @@ remote: '/app/.heroku/python/lib/python3.13/lib-dynload', remote: '/app/.heroku/python/lib/python3.13/site-packages'] remote: + remote: pip #{PIP_VERSION} from /app/.heroku/python/lib/python3.13/site-packages/pip (python 3.13) remote: Package Version remote: ----------------- ------- remote: pip #{PIP_VERSION} @@ -56,6 +57,9 @@ remote: Requirement already satisfied: typing-extensions==4.12.2 (from -r requirements.txt (line 5)) (4.12.2) remote: -----> Inline app detected OUTPUT + # Test that pip is available at run-time too (since for historical reasons it's been + # made available after the build, and users now rely on this). + expect(app.run('pip --version')).to include("pip #{PIP_VERSION}") end end end diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 78346f03c..7d6f9d098 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -35,6 +35,7 @@ remote: '/app/.heroku/python/lib/python3.13/lib-dynload', remote: '/app/.heroku/python/lib/python3.13/site-packages'\\] remote: + remote: pipenv, version #{PIPENV_VERSION} remote: Package Version remote: ----------------- -+ remote: certifi .+ @@ -62,6 +63,9 @@ remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Inline app detected REGEX + # Test that Pipenv is available at run-time too (since for historical reasons it's been + # made available after the build, and users now rely on this). + expect(app.run('pipenv --version')).to include("version #{PIPENV_VERSION}") end end end diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index b2b5a782f..44af81982 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -39,6 +39,7 @@ remote: '/app/.heroku/python/lib/python3.13/lib-dynload', remote: '/app/.heroku/python/lib/python3.13/site-packages'] remote: + remote: Poetry (version #{POETRY_VERSION}) remote: Skipping virtualenv creation, as specified in config file. remote: typing-extensions 4.12.2 Backported and Experimental Type Hints for Python ... remote: diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 3fe994e6d..f8c2b2fa7 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -41,6 +41,7 @@ remote: '/app/.heroku/python/lib/python3.13/lib-dynload', remote: '/app/.heroku/python/lib/python3.13/site-packages'\\] remote: + remote: uv #{UV_VERSION} remote: Using Python #{DEFAULT_PYTHON_FULL_VERSION} environment at: /app/.heroku/python remote: Package Version remote: ----------------- ------- From aac92f4eb8b6f5abfaa99c727ea26d764e24f1e1 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 22 Jul 2025 07:08:02 +0000 Subject: [PATCH 166/406] Bump the ruby-dependencies group with 2 updates (#1838) Bumps the ruby-dependencies group with 2 updates: [heroku_hatchet](https://github.com/heroku/hatchet) and [rubocop](https://github.com/rubocop/rubocop). Updates `heroku_hatchet` from 8.0.5 to 8.0.6 - [Changelog](https://github.com/heroku/hatchet/blob/main/CHANGELOG.md) - [Commits](https://github.com/heroku/hatchet/compare/v8.0.5...v8.0.6) Updates `rubocop` from 1.77.0 to 1.78.0 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.77.0...v1.78.0) --- updated-dependencies: - dependency-name: heroku_hatchet dependency-version: 8.0.6 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rubocop dependency-version: 1.78.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 5698e3d6f..406ccda39 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -5,7 +5,7 @@ GEM base64 (0.3.0) diff-lcs (1.6.2) erubis (2.7.0) - excon (1.2.7) + excon (1.2.8) logger heroics (0.1.3) base64 @@ -14,18 +14,18 @@ GEM moneta multi_json (>= 1.9.2) webrick - heroku_hatchet (8.0.5) + heroku_hatchet (8.0.6) excon (< 2) platform-api (~> 3) rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.12.2) + json (2.13.0) language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) moneta (1.0.0) - multi_json (1.15.0) + multi_json (1.17.0) parallel (1.27.0) parallel_split_test (0.10.0) parallel (>= 0.5.13) @@ -51,7 +51,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.4) - rubocop (1.77.0) + rubocop (1.78.0) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) @@ -62,14 +62,14 @@ GEM rubocop-ast (>= 1.45.1, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 4.0) - rubocop-ast (1.45.1) + rubocop-ast (1.46.0) parser (>= 3.3.7.2) prism (~> 1.4) rubocop-rspec (3.6.0) lint_roller (~> 1.1) rubocop (~> 1.72, >= 1.72.1) ruby-progressbar (1.13.0) - thor (1.3.2) + thor (1.4.0) threaded (0.0.4) unicode-display_width (3.1.4) unicode-emoji (~> 4.0, >= 4.0.4) From e5206cbdb2f5d56424195bded72274b5730c4672 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 23 Jul 2025 14:12:29 +0100 Subject: [PATCH 167/406] Remove superseded `python_version_reason` metric (#1839) Since more than 60 days have passed since the replacement metric (`python_version_origin`) was added in #1785, so all build events within the Honeycomb retention time span now have the new metric name. GUS-W-19116066. --- bin/compile | 2 -- bin/report | 1 - 2 files changed, 3 deletions(-) diff --git a/bin/compile b/bin/compile index 8ab418cc6..413967b19 100755 --- a/bin/compile +++ b/bin/compile @@ -114,8 +114,6 @@ cached_python_full_version="$(cache::cached_python_full_version "${CACHE_DIR}")" python_version::read_requested_python_version "${BUILD_DIR}" "${package_manager}" "${cached_python_full_version}" requested_python_version python_version_origin meta_set "python_version_requested" "${requested_python_version}" meta_set "python_version_origin" "${python_version_origin}" -# TODO: Remove this once the old Honeycomb events that don't have `python_version_origin` have aged out. -meta_set "python_version_reason" "${python_version_origin}" case "${python_version_origin}" in default) diff --git a/bin/report b/bin/report index 54065955b..5ae6839dd 100755 --- a/bin/report +++ b/bin/report @@ -72,7 +72,6 @@ STRING_FIELDS=( poetry_version python_version_major python_version_origin - python_version_reason python_version_requested python_version setuptools_version From f6bbc993c68dd298e8da728d1b45a2ae8981d1fc Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 23 Jul 2025 14:22:19 +0100 Subject: [PATCH 168/406] Update Pipenv to 2025.0.4 and improve installation/caching (#1840) * Updates Pipenv from 2024.0.1 to 2025.0.4. Changelog: https://github.com/pypa/pipenv/blob/main/CHANGELOG.md#202504-2025-07-07 * Switches to installing Pipenv into its own virtual environment, so that it and its dependencies don't leak into the app environment. * Stops installing pip (in addition to Pipenv) when the chosen package manager is Pipenv, to match the behaviour when using Poetry or uv. * Forces the build cache to be cleared if the contents of `Pipfile.lock` has changed since the last build. This is required to work around `pipenv {install,sync}` by design not uninstalling packages if they are removed from the lockfile (?!). We also can't use `pipenv clean` since it doesn't work with `--system` / `PIPENV_SYSTEM`. Apps that have been inadvertently depending on Pipenv's own dependencies implicitly, will need to add those dependencies explicitly to their `Pipfile` and regenerate `Pipfile.lock`. Similarly, any apps that for some reason use pip later in the build in addition to Pipenv, will need to either add pip as an explicit dependency in their `Pipfile`, or else for adhoc use-cases (such as in one-off dynos), can temporarily install pip using `python -m ensurepip --default-pip`. Supersedes #1828. GUS-W-17482289. GUS-W-17482412. GUS-W-19116903. --- CHANGELOG.md | 4 + bin/compile | 4 +- lib/cache.sh | 17 ++- lib/pipenv.sh | 141 ++++++++++++++++++------- requirements/pipenv.txt | 2 +- spec/fixtures/pipenv_basic/bin/compile | 3 +- spec/hatchet/ci_spec.rb | 13 +-- spec/hatchet/package_manager_spec.rb | 1 - spec/hatchet/pipenv_spec.rb | 57 +++++----- 9 files changed, 164 insertions(+), 78 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b3a56c4b..699fac6a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## [Unreleased] +- Updated Pipenv from 2024.0.1 to 2025.0.4. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840)) +- Fixed the way Pipenv is installed, so that it and its dependencies are installed into a separate virtual environment rather than same environment as the app. If your app inadvertently depended on Pipenv's internal dependencies, you will need to add those dependencies explicitly to your `Pipfile`. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840)) +- Stopped installing pip when Pipenv is the chosen package manager. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840)) +- The build cache is now cleared when using Pipenv if the contents of `Pipfile.lock` has changed since the last build. This is required to work around Pipenv not uninstalling packages when they are removed from the lockfile. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840)) - The build now errors when using Pipenv without its lockfile (`Pipfile.lock`). This replaces the warning displayed since November 2024. ([#1833](https://github.com/heroku/heroku-buildpack-python/pull/1833)) ## [v291] - 2025-07-10 diff --git a/bin/compile b/bin/compile index 413967b19..71efc7a9a 100755 --- a/bin/compile +++ b/bin/compile @@ -172,9 +172,7 @@ case "${package_manager}" in pip::install_pip_setuptools_wheel "${python_home}" "${python_major_version}" ;; pipenv) - # TODO: Stop installing pip when using Pipenv. - pip::install_pip_setuptools_wheel "${python_home}" "${python_major_version}" - pipenv::install_pipenv + pipenv::install_pipenv "${python_home}" "${python_major_version}" "${EXPORT_PATH}" "${PROFILE_PATH}" ;; poetry) poetry::install_poetry "${python_home}" "${python_major_version}" "${CACHE_DIR}" "${EXPORT_PATH}" diff --git a/lib/cache.sh b/lib/cache.sh index bdd6c543d..4237dee15 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -87,6 +87,7 @@ function cache::restore() { # later removed, pip will not uninstall the package. This check can be removed if we # ever switch to only caching pip's HTTP/wheel cache rather than site-packages. # TODO: Remove the `-f` check once the setup.py fallback feature is removed. + # TODO: Switch this to using sha256sum like the Pipenv implementation. if [[ -f "${build_dir}/requirements.txt" ]] && ! cmp --silent "${cache_dir}/.heroku/requirements.txt" "${build_dir}/requirements.txt"; then cache_invalidation_reasons+=("The contents of requirements.txt changed") fi @@ -102,9 +103,14 @@ function cache::restore() { elif [[ "${cached_pipenv_version}" != "${PIPENV_VERSION:?}" ]]; then cache_invalidation_reasons+=("The Pipenv version has changed from ${cached_pipenv_version} to ${PIPENV_VERSION}") fi - # TODO: Remove this next time the Pipenv version is bumped (since it will trigger cache invalidation of its own) - if [[ -d "${cache_dir}/.heroku/src" ]]; then - cache_invalidation_reasons+=("The editable VCS repository location has changed (and Pipenv doesn't handle this correctly)") + # `pipenv {install,sync}` by design don't actually uninstall packages on their own (!!): + # and we can't use `pipenv clean` since it isn't compatible with `--system`. + # https://github.com/pypa/pipenv/issues/3365 + # We have to explicitly check for the presence of the Pipfile.lock.sha256 file, + # since we only started writing it to the build cache as of buildpack v292+. + local pipfile_lock_checksum_file="${cache_dir}/.heroku/python/Pipfile.lock.sha256" + if [[ -f "${pipfile_lock_checksum_file}" ]] && ! sha256sum --check --strict --status "${pipfile_lock_checksum_file}"; then + cache_invalidation_reasons+=("The contents of Pipfile.lock changed") fi ;; poetry) @@ -194,7 +200,12 @@ function cache::save() { # TODO: Simplify this once multiple package manager files being found is turned into an # error and the setup.py fallback feature is removed. if [[ "${package_manager}" == "pip" && -f "${build_dir}/requirements.txt" ]]; then + # TODO: Switch this to using sha256sum like the Pipenv implementation. cp "${build_dir}/requirements.txt" "${cache_dir}/.heroku/" + elif [[ "${package_manager}" == "pipenv" ]]; then + # This must use a relative path for the lockfile, since the output file will contain + # the path specified, and the build directory path changes every build. + sha256sum Pipfile.lock >"${cache_dir}/.heroku/python/Pipfile.lock.sha256" fi meta_time "cache_save_duration" "${cache_save_start_time}" diff --git a/lib/pipenv.sh b/lib/pipenv.sh index 715279eff..980042d66 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -6,44 +6,105 @@ set -euo pipefail PIPENV_VERSION=$(utils::get_requirement_version 'pipenv') -# TODO: Either enable or remove these. -# export CLINT_FORCE_COLOR=1 -# export PIPENV_FORCE_COLOR=1 - function pipenv::install_pipenv() { - meta_set "pipenv_version" "${PIPENV_VERSION}" - - output::step "Installing Pipenv ${PIPENV_VERSION}" - - # TODO: Install Pipenv into a venv so it isn't leaked into the app environment. - # TODO: Skip installing Pipenv if its version hasn't changed (once it's installed into a venv). - # TODO: Explore viability of making Pipenv only be available during the build, to reduce slug size. + local python_home="${1}" + local python_major_version="${2}" + local export_file="${3}" + local profile_d_file="${4}" + + # Ideally we would only make Pipenv available during the build to reduce slug size, however, + # the buildpack has historically not done that and so some apps are relying on it at run-time + # (for example via `pipenv run` commands in their Procfile). As such, we have to store it in + # the build directory, but must do so via the symlinked `/app/.heroku/python` path so the + # venv doesn't break when the build directory is relocated to /app at run-time. + local pipenv_root="${python_home}/pipenv" + + # We nest the venv and then symlink the `pipenv` script to prevent the rest of `venv/bin/` + # (such as entrypoint scripts from Pipenv's dependencies, or the venv's activation scripts) + # from being added to PATH and exposed to the app. + local pipenv_bin_dir="${pipenv_root}/bin" + local pipenv_venv_dir="${pipenv_root}/venv" - # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. - if ! { - pip \ - install \ - --disable-pip-version-check \ - --no-cache-dir \ - --no-input \ - --quiet \ - "pipenv==${PIPENV_VERSION}" \ - |& output::indent - }; then - output::error <<-EOF - Error: Unable to install Pipenv. - - In some cases, this happens due to a temporary issue with - the network connection or Python Package Index (PyPI). - - Try building again to see if the error resolves itself. + meta_set "pipenv_version" "${PIPENV_VERSION}" - If that doesn't help, check the status of PyPI here: - https://status.python.org - EOF - meta_set "failure_reason" "install-package-manager::pipenv" - exit 1 + # The earlier buildpack cache invalidation step will have already handled the case where the + # Pipenv version has changed, so here we only need to check that a Pipenv install exists. + # Note: We don't need to use the `-e` trick of `install_poetry()` since we're installing into + # a constant path, rather than the cache directory (which can change location). + if [[ -f "${pipenv_bin_dir}/pipenv" ]]; then + output::step "Using cached Pipenv ${PIPENV_VERSION}" + else + output::step "Installing Pipenv ${PIPENV_VERSION}" + + mkdir -p "${pipenv_root}" + + # We use the pip wheel bundled within Python's standard library to install Pipenv, + # since Pipenv vendors its own pip, so doesn't need an install in the venv. + # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. + if ! python -m venv --without-pip "${pipenv_venv_dir}" |& output::indent; then + output::error <<-EOF + Internal Error: Unable to create virtual environment for Pipenv. + + The 'python -m venv' command to create a virtual environment did + not exit successfully. + + See the log output above for more information. + EOF + meta_set "failure_reason" "create-venv::pipenv" + exit 1 + fi + + local bundled_pip_module_path + bundled_pip_module_path="$(utils::bundled_pip_module_path "${python_home}" "${python_major_version}")" + + # We must call the venv Python directly here, rather than relying on pip's `--python` + # option, since `--python` was only added in pip v22.3, so isn't supported by the older + # pip versions bundled with Python 3.9/3.10. + # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. + if ! { + "${pipenv_venv_dir}/bin/python" "${bundled_pip_module_path}" \ + install \ + --disable-pip-version-check \ + --no-cache-dir \ + --no-input \ + --quiet \ + "pipenv==${PIPENV_VERSION}" \ + |& output::indent + }; then + output::error <<-EOF + Error: Unable to install Pipenv. + + In some cases, this happens due to a temporary issue with + the network connection or Python Package Index (PyPI). + + Try building again to see if the error resolves itself. + + If that doesn't help, check the status of PyPI here: + https://status.python.org + EOF + meta_set "failure_reason" "install-package-manager::pipenv" + exit 1 + fi + + mkdir -p "${pipenv_bin_dir}" + ln --symbolic --no-target-directory "${pipenv_venv_dir}/bin/pipenv" "${pipenv_bin_dir}/pipenv" fi + + export PATH="${pipenv_bin_dir}:${PATH}" + # Force Pipenv to manage the system Python site-packages instead of using venvs. + export PIPENV_SYSTEM="1" + + # Set the same env vars in the environment used by later buildpacks. + cat >>"${export_file}" <<-EOF + export PATH="${pipenv_bin_dir}:\${PATH}" + export PIPENV_SYSTEM="1" + EOF + + # And the environment used at app run-time. + cat >>"${profile_d_file}" <<-EOF + export PATH="${pipenv_bin_dir}:\${PATH}" + export PIPENV_SYSTEM="1" + EOF } # Previous versions of the buildpack used to cache the checksum of the lockfile to allow @@ -62,7 +123,8 @@ function pipenv::install_dependencies() { export PIP_EXTRA_INDEX_URL fi - # Note: We can't use `pipenv sync` since it doesn't validate that the lockfile is up to date. + # Note: We can't use `pipenv sync` since it doesn't support `--deploy` and so won't abort + # if the lockfile is out of date. local pipenv_install_command=( pipenv install @@ -77,11 +139,14 @@ function pipenv::install_dependencies() { # We only display the most relevant command args here, to improve the signal to noise ratio. output::step "Installing dependencies using '${pipenv_install_command[*]}'" + # TODO: Expose app config vars to the install command as part of doing so for all package managers. + # `PIPENV_NOSPIN`: Disable progress spinners. + # `PIP_SRC`: Override the editable VCS repo location from its default of inside the build directory + # (Pipenv uses pip internally, and doesn't offer its own config option for this). # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { - "${pipenv_install_command[@]}" \ - --extra-pip-args='--src=/app/.heroku/python/src' \ - --system \ + PIPENV_NOSPIN="1" PIP_SRC="/app/.heroku/python/src" \ + "${pipenv_install_command[@]}" \ |& tee "${WARNINGS_LOG:?}" \ |& output::indent }; then diff --git a/requirements/pipenv.txt b/requirements/pipenv.txt index 34b9e5c44..b239da87f 100644 --- a/requirements/pipenv.txt +++ b/requirements/pipenv.txt @@ -1 +1 @@ -pipenv==2024.0.1 +pipenv==2025.0.4 diff --git a/spec/fixtures/pipenv_basic/bin/compile b/spec/fixtures/pipenv_basic/bin/compile index 83cb72a51..aa11dd877 100755 --- a/spec/fixtures/pipenv_basic/bin/compile +++ b/spec/fixtures/pipenv_basic/bin/compile @@ -17,7 +17,8 @@ echo pipenv --version # We have to resort to using pip to list installed packages, since `pipenv graph` doesn't support `--system`. -pip list --disable-pip-version-check +python -m ensurepip --default-pip >/dev/null +pip list --disable-pip-version-check --exclude pip echo python -c 'import typing_extensions; print(typing_extensions)' diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 48953c89a..cbda7c339 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -87,7 +87,6 @@ -----> Python app detected -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - -----> Installing pip #{PIP_VERSION} -----> Installing Pipenv #{PIPENV_VERSION} -----> Installing dependencies using 'pipenv install --deploy --dev' Installing dependencies from Pipfile.lock \\(.+\\)... @@ -103,14 +102,16 @@ LC_ALL=C.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib - PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PATH=/app/.heroku/python/pipenv/bin:/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PIPENV_SYSTEM=1 PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 -----> Inline app detected LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib - PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PIPENV_SYSTEM=1 PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python PYTHONPATH=/app @@ -124,7 +125,8 @@ LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib - PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ + PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ + PIPENV_SYSTEM=1 PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python PYTHONPATH=/app @@ -140,8 +142,7 @@ -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version -----> Restoring cache -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} - -----> Installing pip #{PIP_VERSION} - -----> Installing Pipenv #{PIPENV_VERSION} + -----> Using cached Pipenv #{PIPENV_VERSION} -----> Installing dependencies using 'pipenv install --deploy --dev' Installing dependencies from Pipfile.lock \\(.+\\)... Installing dependencies from Pipfile.lock \\(.+\\)... diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index fd086e69b..989fffd23 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -115,7 +115,6 @@ remote: remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 7d6f9d098..c425abae4 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -9,12 +9,10 @@ it 'builds with the specified python_version and re-uses packages from the cache' do app.deploy do |app| - # TODO: We should not be leaking the Pipenv installation into the app environment. expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... @@ -22,7 +20,8 @@ remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/python/lib remote: LIBRARY_PATH=/app/.heroku/python/lib - remote: PATH=/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PIPENV_SYSTEM=1 remote: PYTHONHASHSEED=random remote: PYTHONHOME=/app/.heroku/python remote: PYTHONPATH=/app @@ -38,15 +37,7 @@ remote: pipenv, version #{PIPENV_VERSION} remote: Package Version remote: ----------------- -+ - remote: certifi .+ - remote: distlib .+ - remote: filelock .+ - remote: pip #{PIP_VERSION} - remote: pipenv #{PIPENV_VERSION} - remote: platformdirs .+ - remote: setuptools .+ remote: typing_extensions 4.12.2 - remote: virtualenv .+ remote: remote: REGEX @@ -57,8 +48,7 @@ remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock remote: -----> Restoring cache remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION} - remote: -----> Installing Pipenv #{PIPENV_VERSION} + remote: -----> Using cached Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Inline app detected @@ -70,6 +60,28 @@ end end + context 'when Pipfile.lock has changed since the last build' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_basic') } + + it 'clears the cache before installing the packages again' do + app.deploy do |app| + File.write('Pipfile.lock', "\n", mode: 'a') + app.commit! + app.push! + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Python app detected + remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock + remote: -----> Discarding cache since: + remote: - The contents of Pipfile.lock changed + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: -----> Installing Pipenv #{PIPENV_VERSION} + remote: -----> Installing dependencies using 'pipenv install --deploy' + remote: Installing dependencies from Pipfile.lock \\(.+\\)... + REGEX + end + end + end + # As well as testing the Pipfile.lock `python_full_version` field, this also tests: # 1. That `python_full_version` takes precedence over the `python_version` field. # 2. That Pipenv works on the oldest Python version supported by all stacks. @@ -116,7 +128,6 @@ remote: ! patch version automatically and prevent this warning. remote: remote: -----> Installing Python 3.9.0 - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pipenv install --deploy' @@ -135,7 +146,6 @@ remote: -----> Python app detected remote: -----> Using Python 3.13 specified in .python-version remote: -----> Installing Python #{LATEST_PYTHON_3_13} - remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... @@ -194,7 +204,6 @@ remote: ! file and this warning will be made an error. remote: remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... @@ -368,9 +377,8 @@ end end - context 'when the Pipenv and Python versions have changed since the last build' do - # TODO: Bump this buildpack version the next time we update the Pipenv version. - let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#archive/v253'] } + context 'when the Pipenv version has changed since the last build' do + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v291'] } let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_basic', buildpacks:) } it 'clears the cache before installing' do @@ -382,11 +390,8 @@ remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock remote: -----> Discarding cache since: - remote: - The Python version has changed from 3.12.4 to #{DEFAULT_PYTHON_FULL_VERSION} - remote: - The Pipenv version has changed from 2023.12.1 to #{PIPENV_VERSION} - remote: - The editable VCS repository location has changed \\(and Pipenv doesn't handle this correctly\\) + remote: - The Pipenv version has changed from 2024.0.1 to #{PIPENV_VERSION} remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... @@ -498,10 +503,12 @@ remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: Your Pipfile.lock \\(.+\\) is out of date. Expected: \\(.+\\). + remote: Your Pipfile.lock + remote: \\(.+\\) is out of + remote: date. Expected: + remote: \\(.+\\). remote: .+ remote: ERROR:: Aborting deploy remote: From 14f2d55138bbde8955888c4409fa888fde0b1881 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 23 Jul 2025 13:38:16 +0000 Subject: [PATCH 169/406] Prepare release v292 (#1841) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 699fac6a8..4f06d2bad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v292] - 2025-07-23 + - Updated Pipenv from 2024.0.1 to 2025.0.4. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840)) - Fixed the way Pipenv is installed, so that it and its dependencies are installed into a separate virtual environment rather than same environment as the app. If your app inadvertently depended on Pipenv's internal dependencies, you will need to add those dependencies explicitly to your `Pipfile`. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840)) - Stopped installing pip when Pipenv is the chosen package manager. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840)) @@ -1252,7 +1255,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v291...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v292...main +[v292]: https://github.com/heroku/heroku-buildpack-python/compare/v291...v292 [v291]: https://github.com/heroku/heroku-buildpack-python/compare/v290...v291 [v290]: https://github.com/heroku/heroku-buildpack-python/compare/v289...v290 [v289]: https://github.com/heroku/heroku-buildpack-python/compare/v288...v289 From 0bb90f7f2888f2a0caf28e951170aab960b02740 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 23 Jul 2025 18:03:33 +0100 Subject: [PATCH 170/406] Work around Pipenv bug when using `--system` (#1842) After moving Pipenv to its own virtual environment in #1840 (to fix leaking Pipenv's dependencies into the app environment), apps that explicitly depend on a package that happen to also be a Pipenv dependency were seeing `ModuleNotFoundError` failures. For example: ``` ModuleNotFoundError: No module named 'certifi' ``` Or: ``` ModuleNotFoundError: No module named 'packaging' ``` It appears that Pipenv's `--system` feature has a bug whereby it sees its own dependencies (installed in its own environment) as already installed (even though they aren't actually installed in the app environment), so incorrectly skips installing them. In general, it seems Pipenv's support for `--system` is pretty buggy (for example, it doesn't work with `pipenv run` either): https://github.com/pypa/pipenv/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22--system%20flag%22 The classic Python buildpack doesn't currently use virtual environments (since it installs a dedicated Python install for the app, that's already separate from system Python), and switching to them would be a significant change that may cause other types of breakage (for example, to apps that hardcode the path to Python). As such for now, we need to continue to use `pipenv install --system` (as the buildpack has done for some time), but I have found the bug can be worked around if we set `VIRTUAL_ENV` to the Python install location, to trick Pipenv into thinking it's operating in a virtual environment. The Pipenv tests have been updated to confirm that an explicit `certifi` install now works. GUS-W-19118553. --- CHANGELOG.md | 1 + lib/pipenv.sh | 12 ++++++++++++ spec/fixtures/pipenv_basic/Pipfile | 1 + spec/fixtures/pipenv_basic/Pipfile.lock | 11 ++++++++++- spec/hatchet/ci_spec.rb | 6 ++++++ spec/hatchet/pipenv_spec.rb | 3 +++ 6 files changed, 33 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f06d2bad..d2df42298 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Work around a Pipenv bug when using `--system`, that causes packages to not be installed correctly if they are also a dependency of Pipenv (such as `certifi` ). ([#1842](https://github.com/heroku/heroku-buildpack-python/pull/1842)) ## [v292] - 2025-07-23 diff --git a/lib/pipenv.sh b/lib/pipenv.sh index 980042d66..88d272e47 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -93,17 +93,29 @@ function pipenv::install_pipenv() { export PATH="${pipenv_bin_dir}:${PATH}" # Force Pipenv to manage the system Python site-packages instead of using venvs. export PIPENV_SYSTEM="1" + # Hide Pipenv's notice about finding/using an existing virtual environment. + export PIPENV_VERBOSITY="-1" + # Work around a Pipenv bug when using `--system`, whereby it doesn't correctly install + # dependencies that happen to also be a dependency of Pipenv (such as `certifi`). + # In general Pipenv's support for its `--system` mode seems very buggy. Longer term we + # should explore moving to venvs, however, that will need to be coordinated across all + # package managers and also change paths for Python which could break other use cases. + export VIRTUAL_ENV="${python_home}" # Set the same env vars in the environment used by later buildpacks. cat >>"${export_file}" <<-EOF export PATH="${pipenv_bin_dir}:\${PATH}" export PIPENV_SYSTEM="1" + export PIPENV_VERBOSITY="-1" + export VIRTUAL_ENV="${python_home}" EOF # And the environment used at app run-time. cat >>"${profile_d_file}" <<-EOF export PATH="${pipenv_bin_dir}:\${PATH}" export PIPENV_SYSTEM="1" + export PIPENV_VERBOSITY="-1" + export VIRTUAL_ENV="${python_home}" EOF } diff --git a/spec/fixtures/pipenv_basic/Pipfile b/spec/fixtures/pipenv_basic/Pipfile index 7a763f509..0aee61f1c 100644 --- a/spec/fixtures/pipenv_basic/Pipfile +++ b/spec/fixtures/pipenv_basic/Pipfile @@ -4,6 +4,7 @@ verify_ssl = true name = "pypi" [packages] +certifi = "*" typing-extensions = "*" [dev-packages] diff --git a/spec/fixtures/pipenv_basic/Pipfile.lock b/spec/fixtures/pipenv_basic/Pipfile.lock index 44a737c1a..1236083b0 100644 --- a/spec/fixtures/pipenv_basic/Pipfile.lock +++ b/spec/fixtures/pipenv_basic/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "83d7242edaa31ec24731c102c5debc217f3e5f5fa5b4e992de07d4d25805715e" + "sha256": "64d6057c60457235901a48fab172d14e37f8cde498b38d7b4b643882e65c215a" }, "pipfile-spec": 6, "requires": { @@ -16,6 +16,15 @@ ] }, "default": { + "certifi": { + "hashes": [ + "sha256:6b31f564a415d79ee77df69d757bb49a5bb53bd9f756cbbe24394ffd6fc1f4b2", + "sha256:8ea99dbdfaaf2ba2f9bac77b9249ef62ec5218e7c2b2e903378ed5fccf765995" + ], + "index": "pypi", + "markers": "python_version >= '3.7'", + "version": "==2025.7.14" + }, "typing-extensions": { "hashes": [ "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index cbda7c339..5f8a5c276 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -104,18 +104,22 @@ LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/pipenv/bin:/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PIPENV_SYSTEM=1 + PIPENV_VERBOSITY=-1 PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 + VIRTUAL_ENV=/app/.heroku/python -----> Inline app detected LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PIPENV_SYSTEM=1 + PIPENV_VERBOSITY=-1 PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true + VIRTUAL_ENV=/app/.heroku/python -----> No test-setup command provided. Skipping. -----> Running test command `./bin/print-env-vars.sh && pytest --version`... CI=true @@ -127,10 +131,12 @@ LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ PIPENV_SYSTEM=1 + PIPENV_VERBOSITY=-1 PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true + VIRTUAL_ENV=/app/.heroku/python WEB_CONCURRENCY=5 pytest .+ -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index c425abae4..b8fc50c64 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -22,10 +22,12 @@ remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PIPENV_SYSTEM=1 + remote: PIPENV_VERBOSITY=-1 remote: PYTHONHASHSEED=random remote: PYTHONHOME=/app/.heroku/python remote: PYTHONPATH=/app remote: PYTHONUNBUFFERED=true + remote: VIRTUAL_ENV=/app/.heroku/python remote: remote: \\['', remote: '/app', @@ -37,6 +39,7 @@ remote: pipenv, version #{PIPENV_VERSION} remote: Package Version remote: ----------------- -+ + remote: certifi 2025.7.14 remote: typing_extensions 4.12.2 remote: remote: From f6f2a29f0c0efe1182dfb0987518f6268ae00a98 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 23 Jul 2025 17:09:46 +0000 Subject: [PATCH 171/406] Prepare release v293 (#1843) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d2df42298..e3f9e0d84 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v293] - 2025-07-23 + - Work around a Pipenv bug when using `--system`, that causes packages to not be installed correctly if they are also a dependency of Pipenv (such as `certifi` ). ([#1842](https://github.com/heroku/heroku-buildpack-python/pull/1842)) ## [v292] - 2025-07-23 @@ -1256,7 +1259,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v292...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v293...main +[v293]: https://github.com/heroku/heroku-buildpack-python/compare/v292...v293 [v292]: https://github.com/heroku/heroku-buildpack-python/compare/v291...v292 [v291]: https://github.com/heroku/heroku-buildpack-python/compare/v290...v291 [v290]: https://github.com/heroku/heroku-buildpack-python/compare/v289...v290 From 314c6358b6bc0614a150ef75d1a067685b44f706 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 28 Jul 2025 14:49:36 +0100 Subject: [PATCH 172/406] Add a build log message when the cache is being saved (#1844) To distinguish it from the installation dependencies step, particularly since copying files into the cache directory can take a while for apps with a large number of dependencies. GUS-W-19144883. --- CHANGELOG.md | 1 + lib/cache.sh | 2 ++ spec/hatchet/ci_spec.rb | 4 ++++ spec/hatchet/package_manager_spec.rb | 3 +++ spec/hatchet/pip_spec.rb | 5 +++++ spec/hatchet/pipenv_spec.rb | 9 +++++++++ spec/hatchet/poetry_spec.rb | 6 ++++++ spec/hatchet/uv_spec.rb | 6 ++++++ 8 files changed, 36 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e3f9e0d84..f17ad8ffc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Added a build log message for the build cache saving step. ([#1844](https://github.com/heroku/heroku-buildpack-python/pull/1844)) ## [v293] - 2025-07-23 diff --git a/lib/cache.sh b/lib/cache.sh index 4237dee15..2f595d30e 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -184,6 +184,8 @@ function cache::save() { local cache_save_start_time cache_save_start_time=$(nowms) + output::step "Saving cache" + mkdir -p "${cache_dir}/.heroku" rm -rf "${cache_dir}/.heroku/python" diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 5f8a5c276..576fee12e 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -32,6 +32,7 @@ PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 + -----> Saving cache -----> Inline app detected LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib @@ -108,6 +109,7 @@ PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 VIRTUAL_ENV=/app/.heroku/python + -----> Saving cache -----> Inline app detected LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib @@ -194,6 +196,7 @@ POETRY_VIRTUALENVS_CREATE=false POETRY_VIRTUALENVS_USE_POETRY_PYTHON=true PYTHONUNBUFFERED=1 + -----> Saving cache -----> Inline app detected LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib @@ -278,6 +281,7 @@ UV_NO_MANAGED_PYTHON=1 UV_PROJECT_ENVIRONMENT=/app/.heroku/python UV_PYTHON_DOWNLOADS=never + -----> Saving cache -----> Inline app detected LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index 989fffd23..b689946b8 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -65,6 +65,7 @@ remote: .+ remote: Installing collected packages: six, test remote: Successfully installed six-.+ test-0.0.0 + remote: -----> Saving cache REGEX end end @@ -118,6 +119,7 @@ remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Saving cache REGEX end end @@ -146,6 +148,7 @@ remote: Installed 1 package in .+s remote: Bytecode compiled 1 file in .+s remote: \\+ typing-extensions==4.13.2 + remote: -----> Saving cache remote: -----> Discovering process types REGEX end diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 1fd1ed721..8aa1ae959 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -20,6 +20,7 @@ remote: Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) remote: Installing collected packages: typing-extensions remote: Successfully installed typing-extensions-4.12.2 + remote: -----> Saving cache remote: -----> Inline app detected remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/python/lib @@ -55,6 +56,7 @@ remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Requirement already satisfied: typing-extensions==4.12.2 (from -r requirements.txt (line 5)) (4.12.2) + remote: -----> Saving cache remote: -----> Inline app detected OUTPUT # Test that pip is available at run-time too (since for historical reasons it's been @@ -90,6 +92,7 @@ remote: Downloading six-1.17.0-py2.py3-none-any.whl (11 kB) remote: Installing collected packages: typing-extensions, six remote: Successfully installed six-1.17.0 typing-extensions-4.12.2 + remote: -----> Saving cache OUTPUT end end @@ -115,6 +118,7 @@ remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: -----> Saving cache remote: -----> Inline app detected remote: easy-install.pth:/app/.heroku/python/src/gunicorn remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py @@ -154,6 +158,7 @@ remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: -----> Saving cache remote: -----> Inline app detected remote: easy-install.pth:/app/.heroku/python/src/gunicorn remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index b8fc50c64..deb0ab449 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -16,6 +16,7 @@ remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Saving cache remote: -----> Inline app detected remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/python/lib @@ -54,6 +55,7 @@ remote: -----> Using cached Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Saving cache remote: -----> Inline app detected REGEX # Test that Pipenv is available at run-time too (since for historical reasons it's been @@ -80,6 +82,7 @@ remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Saving cache REGEX end end @@ -135,6 +138,7 @@ remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Saving cache REGEX end end @@ -152,6 +156,7 @@ remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Saving cache REGEX end end @@ -210,6 +215,7 @@ remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Saving cache REGEX end end @@ -398,6 +404,7 @@ remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Saving cache remote: -----> Discovering process types REGEX end @@ -423,6 +430,7 @@ remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) + remote: -----> Saving cache remote: -----> Inline app detected remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} @@ -464,6 +472,7 @@ remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) + remote: -----> Saving cache remote: -----> Inline app detected remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 44af81982..5497e8cab 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -20,6 +20,7 @@ remote: Package operations: 1 install, 0 updates, 0 removals remote: remote: - Installing typing-extensions (4.12.2) + remote: -----> Saving cache remote: -----> Inline app detected remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/python/lib @@ -57,6 +58,7 @@ remote: Installing dependencies from lock file remote: remote: No dependencies to install or update + remote: -----> Saving cache remote: -----> Inline app detected OUTPUT end @@ -86,6 +88,7 @@ remote: Package operations: 1 install, 0 updates, 0 removals remote: remote: - Installing typing-extensions (4.12.2) + remote: -----> Saving cache remote: -----> Discovering process types OUTPUT end @@ -120,6 +123,7 @@ remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) + remote: -----> Saving cache remote: -----> Inline app detected remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} @@ -169,6 +173,7 @@ remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) + remote: -----> Saving cache remote: -----> Inline app detected remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} @@ -237,6 +242,7 @@ remote: Package operations: 1 install, 0 updates, 0 removals remote: remote: - Installing typing-extensions (4.12.2) + remote: -----> Saving cache OUTPUT end end diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index f8c2b2fa7..7496d3df8 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -20,6 +20,7 @@ remote: Installed 1 package in .+s remote: Bytecode compiled 1 file in .+s remote: \\+ typing-extensions==4.13.2 + remote: -----> Saving cache remote: -----> Inline app detected remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/python/lib @@ -60,6 +61,7 @@ remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' remote: Resolved 7 packages in .+s remote: Bytecode compiled 1 file in .+s + remote: -----> Saving cache remote: -----> Inline app detected REGEX end @@ -89,6 +91,7 @@ remote: Installed 1 package in .+s remote: Bytecode compiled 1 file in .+s remote: \\+ typing-extensions==4.13.2 + remote: -----> Saving cache remote: -----> Discovering process types REGEX end @@ -125,6 +128,7 @@ remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) + remote: -----> Saving cache remote: -----> Inline app detected remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} @@ -174,6 +178,7 @@ remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) + remote: -----> Saving cache remote: -----> Inline app detected remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} @@ -237,6 +242,7 @@ remote: Installed 1 package in .+s remote: Bytecode compiled 1 file in .+s remote: \\+ typing-extensions==4.13.2 + remote: -----> Saving cache REGEX end end From 9494cc66ca829fdf52f14472099004070677f4fc Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 28 Jul 2025 15:03:34 +0100 Subject: [PATCH 173/406] Speed up cache restoration (#1845) By moving the Python directory instead of copying, which internally performs a rename instead of a copy if the cache directory is on the same filesystem mount as the build directory (which is the case for standard builds). This should be particularly beneficial for apps with a large number of Python dependencies (and so a large site-packages) - where we see cache restoration taking up to 30 seconds in Honeycomb (P99.9). GUS-W-19144883. --- CHANGELOG.md | 1 + lib/cache.sh | 20 +++++++++++--------- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f17ad8ffc..6e8685869 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Improved performance of Python build cache restoration. ([#1845](https://github.com/heroku/heroku-buildpack-python/pull/1845)) - Added a build log message for the build cache saving step. ([#1844](https://github.com/heroku/heroku-buildpack-python/pull/1844)) ## [v293] - 2025-07-23 diff --git a/lib/cache.sh b/lib/cache.sh index 2f595d30e..421470790 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -33,7 +33,7 @@ function cache::restore() { local python_full_version="${5}" local package_manager="${6}" - if [[ ! -e "${cache_dir}/.heroku/python" ]]; then + if [[ ! -d "${cache_dir}/.heroku/python" ]]; then meta_set "cache_status" "empty" return 0 fi @@ -154,13 +154,10 @@ function cache::restore() { else output::step "Restoring cache" mkdir -p "${build_dir}/.heroku" - - # NB: For now this has to handle files already existing in build_dir since some apps accidentally - # run the Python buildpack twice. TODO: Refactor this once duplicate buildpacks become an error. - # TODO: Investigate why errors are ignored and ideally stop doing so. - # TODO: Compare the performance of moving the directory vs copying files. - cp -R "${cache_dir}/.heroku/python" "${build_dir}/.heroku/" &>/dev/null || true - + # Moving the files directly in place is much faster than copying when both the cache and + # build directory are on the same filesystem mount. The Python directory is guaranteed + # to not already exist thanks to the earlier `checks::existing_python_dir_present()`. + mv "${cache_dir}/.heroku/python" "${build_dir}/.heroku/" meta_set "cache_status" "reused" fi @@ -189,7 +186,12 @@ function cache::save() { mkdir -p "${cache_dir}/.heroku" rm -rf "${cache_dir}/.heroku/python" - cp -R "${build_dir}/.heroku/python" "${cache_dir}/.heroku/" + # In theory we should be able to use `--reflink=auto` here for improved performance, however, + # initial benchmarking showed it to be slower with the file system type / mounts used by the + # Heroku build system for some reason. (Copying was faster using `--link`, however, that fails + # when copying cross-mount such as for Heroku CI and build-in-app-dir, plus hardlinks could + # result in unintended cache mutation if later buildpacks add/remove packages etc.) + cp --recursive "${build_dir}/.heroku/python" "${cache_dir}/.heroku/" # Metadata used by subsequent builds to determine whether the cache can be reused. # These are written/consumed via separate files and not the metadata store for compatibility From e094206515982668fab6e33896651279254b27fc Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Mon, 28 Jul 2025 14:24:43 +0000 Subject: [PATCH 174/406] Prepare release v294 (#1846) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6e8685869..2719a8b66 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v294] - 2025-07-28 + - Improved performance of Python build cache restoration. ([#1845](https://github.com/heroku/heroku-buildpack-python/pull/1845)) - Added a build log message for the build cache saving step. ([#1844](https://github.com/heroku/heroku-buildpack-python/pull/1844)) @@ -1261,7 +1264,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v293...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v294...main +[v294]: https://github.com/heroku/heroku-buildpack-python/compare/v293...v294 [v293]: https://github.com/heroku/heroku-buildpack-python/compare/v292...v293 [v292]: https://github.com/heroku/heroku-buildpack-python/compare/v291...v292 [v291]: https://github.com/heroku/heroku-buildpack-python/compare/v290...v291 From 2997328b48bc23f07a696163ae259229dd9a4f8c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 31 Jul 2025 11:28:46 +0000 Subject: [PATCH 175/406] Bump uv from 0.7.20 to 0.8.4 (#1847) * Bump uv from 0.7.20 to 0.8.4 Bumps [uv](https://github.com/astral-sh/uv) from 0.7.20 to 0.8.4. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.7.20...0.8.4) --- updated-dependencies: - dependency-name: uv dependency-version: 0.8.4 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Update test fixture for new output * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- spec/hatchet/uv_spec.rb | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2719a8b66..f2b2c1fd0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.7.20 to 0.8.4. ([#1847](https://github.com/heroku/heroku-buildpack-python/pull/1847)) ## [v294] - 2025-07-28 diff --git a/requirements/uv.txt b/requirements/uv.txt index d9584f673..04d297a45 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.7.20 +uv==0.8.4 diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 7496d3df8..821ae0d20 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -399,7 +399,7 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' remote: Resolved 2 packages in .+s - remote: error: The lockfile at `uv.lock` needs to be updated, but `--locked` was provided. To update the lockfile, run `uv lock`. + remote: The lockfile at `uv.lock` needs to be updated, but `--locked` was provided. To update the lockfile, run `uv lock`. remote: remote: ! Error: Unable to install dependencies using uv. remote: ! From 0e85c3e11da5c73e1c73b8f26aa1934d80b1d0a5 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 1 Aug 2025 15:04:44 +0000 Subject: [PATCH 176/406] Bump rubocop from 1.78.0 to 1.79.1 in the ruby-dependencies group (#1850) Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.78.0 to 1.79.1 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.78.0...v1.79.1) --- updated-dependencies: - dependency-name: rubocop dependency-version: 1.79.1 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 406ccda39..3f5119fbd 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -20,7 +20,7 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.13.0) + json (2.13.2) language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) @@ -30,7 +30,7 @@ GEM parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) - parser (3.3.8.0) + parser (3.3.9.0) ast (~> 2.4.1) racc platform-api (3.8.0) @@ -51,7 +51,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.4) - rubocop (1.78.0) + rubocop (1.79.1) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) @@ -59,7 +59,7 @@ GEM parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) regexp_parser (>= 2.9.3, < 3.0) - rubocop-ast (>= 1.45.1, < 2.0) + rubocop-ast (>= 1.46.0, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 4.0) rubocop-ast (1.46.0) @@ -89,7 +89,7 @@ DEPENDENCIES rubocop-rspec RUBY VERSION - ruby 3.4.4p34 + ruby 3.4.5p51 BUNDLED WITH - 2.6.9 + 2.7.1 From 593a05c7c4d108928711413d6280db126f8f5985 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 1 Aug 2025 16:28:41 +0100 Subject: [PATCH 177/406] Improve tests for old Python versions (#1849) * Updates the test for Python 3.9 with pip to perform a second build that tests the cached rebuild, ready for a fix to the log output cleanup regex in a later PR (pip outputs a different path style with older Python). * Moves the outdated version tests around to be more consistent with the equivalent tests for other package managers. GUS-W-19183260. --- .../pip_oldest_python/requirements.txt | 2 + .../runtime.txt | 0 .../python_version_outdated/.python-version | 2 +- .../requirements.txt | 0 spec/hatchet/pip_spec.rb | 99 +++++++++++++++ spec/hatchet/python_update_warning_spec.rb | 119 ------------------ spec/hatchet/python_version_spec.rb | 32 +++++ 7 files changed, 134 insertions(+), 120 deletions(-) create mode 100644 spec/fixtures/pip_oldest_python/requirements.txt rename spec/fixtures/{python_version_outdated_and_deprecated => pip_oldest_python}/runtime.txt (100%) delete mode 100644 spec/fixtures/python_version_outdated_and_deprecated/requirements.txt delete mode 100644 spec/hatchet/python_update_warning_spec.rb diff --git a/spec/fixtures/pip_oldest_python/requirements.txt b/spec/fixtures/pip_oldest_python/requirements.txt new file mode 100644 index 000000000..1d83d0de9 --- /dev/null +++ b/spec/fixtures/pip_oldest_python/requirements.txt @@ -0,0 +1,2 @@ +# This package has been picked since it has no dependencies and is small/fast to install. +typing-extensions==4.14.1 diff --git a/spec/fixtures/python_version_outdated_and_deprecated/runtime.txt b/spec/fixtures/pip_oldest_python/runtime.txt similarity index 100% rename from spec/fixtures/python_version_outdated_and_deprecated/runtime.txt rename to spec/fixtures/pip_oldest_python/runtime.txt diff --git a/spec/fixtures/python_version_outdated/.python-version b/spec/fixtures/python_version_outdated/.python-version index 30291cba2..3e388a4ac 100644 --- a/spec/fixtures/python_version_outdated/.python-version +++ b/spec/fixtures/python_version_outdated/.python-version @@ -1 +1 @@ -3.10.0 +3.13.2 diff --git a/spec/fixtures/python_version_outdated_and_deprecated/requirements.txt b/spec/fixtures/python_version_outdated_and_deprecated/requirements.txt deleted file mode 100644 index e69de29bb..000000000 diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 8aa1ae959..33a151c29 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -176,6 +176,105 @@ end end + # This checks that the pip bootstrap works even with older bundled pip, and that our + # chosen Pip version also supports our oldest supported Python version. + context 'when using our oldest supported Python version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pip_oldest_python') } + + it 'installs successfully' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.9.0 specified in runtime.txt + remote: + remote: ! Warning: The runtime.txt file is deprecated. + remote: ! + remote: ! The runtime.txt file is deprecated since it has been replaced + remote: ! by the more widely supported .python-version file: + remote: ! https://devcenter.heroku.com/changelog-items/3141 + remote: ! + remote: ! Please switch to using a .python-version file instead. + remote: ! + remote: ! Delete your runtime.txt file and create a new file in the + remote: ! root directory of your app named: + remote: ! .python-version + remote: ! + remote: ! Make sure to include the '.' character at the start of the + remote: ! filename. Don't add a file extension such as '.txt'. + remote: ! + remote: ! In the new file, specify your app's major Python version number + remote: ! only. Don't include quotes or a 'python-' prefix. + remote: ! + remote: ! For example, to request the latest version of Python 3.9, + remote: ! update your .python-version file so it contains exactly: + remote: ! 3.9 + remote: ! + remote: ! We strongly recommend that you don't specify the Python patch + remote: ! version number, since it will pin your app to an exact Python + remote: ! version and so stop your app from receiving security updates + remote: ! each time it builds. + remote: ! + remote: ! In the future support for runtime.txt will be removed and + remote: ! this warning will be made an error. + remote: + remote: + remote: ! Warning: Support for Python 3.9 is ending soon! + remote: ! + remote: ! Python 3.9 will reach its upstream end-of-life in October 2025, + remote: ! at which point it will no longer receive security updates: + remote: ! https://devguide.python.org/versions/#supported-versions + remote: ! + remote: ! As such, support for Python 3.9 will be removed from this + remote: ! buildpack on 7th January 2026. + remote: ! + remote: ! Upgrade to a newer Python version as soon as possible, by + remote: ! changing the version in your runtime.txt file. + remote: ! + remote: ! For more information, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions + remote: + remote: + remote: ! Warning: A Python patch update is available! + remote: ! + remote: ! Your app is using Python 3.9.0, however, there is a newer + remote: ! patch release of Python 3.9 available: #{LATEST_PYTHON_3_9} + remote: ! + remote: ! It is important to always use the latest patch version of + remote: ! Python to keep your app secure. + remote: ! + remote: ! Update your runtime.txt file to use the new version. + remote: ! + remote: ! We strongly recommend that you don't pin your app to an + remote: ! exact Python version such as 3.9.0, and instead only specify + remote: ! the major Python version of 3.9 in your runtime.txt file. + remote: ! This will allow your app to receive the latest available Python + remote: ! patch version automatically and prevent this warning. + remote: + remote: -----> Installing Python 3.9.0 + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing SQLite3 + remote: -----> Installing dependencies using 'pip install -r requirements.txt' + remote: Collecting typing-extensions==4.14.1 (from -r requirements.txt (line 2)) + remote: Downloading typing_extensions-4.14.1-py3-none-any.whl.metadata (3.0 kB) + remote: Downloading typing_extensions-4.14.1-py3-none-any.whl (43 kB) + remote: Installing collected packages: typing-extensions + remote: Successfully installed typing-extensions-4.14.1 + remote: -----> Saving cache + OUTPUT + app.commit! + app.push! + # Test that our regex for cleaning up the "Requirement already satisfied" lines also works + # with the relative paths output when pip is run with Python 3.10 and older. This and the + # regex variant can be removed once support for Python 3.10 and older is dropped. + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Installing dependencies using 'pip install -r requirements.txt' + remote: Requirement already satisfied: typing-extensions==4.14.1 in ./.heroku/python/lib/python3.9/site-packages (from -r requirements.txt (line 2)) (4.14.1) + remote: -----> Saving cache + OUTPUT + end + end + end + context 'when requirements.txt contains a package that needs compiling against the Python headers' do let(:app) { Hatchet::Runner.new('spec/fixtures/pip_compiled') } diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb deleted file mode 100644 index 1cc094200..000000000 --- a/spec/hatchet/python_update_warning_spec.rb +++ /dev/null @@ -1,119 +0,0 @@ -# frozen_string_literal: true - -require_relative '../spec_helper' - -# NOTE: We use the oldest patch releases (ie the '.0' releases) since we also want to test against -# the oldest Python versions available to users. This is particularly important given that older -# patch releases will bundle older pip, and the buildpack uses that pip during bootstrapping. -RSpec.describe 'Python update warnings' do - context 'with a runtime.txt containing an outdated patch version that is also a deprecated major version' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_outdated_and_deprecated') } - - it 'warns about both the deprecated major version and the patch update' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python 3.9.0 specified in runtime.txt - remote: - remote: ! Warning: The runtime.txt file is deprecated. - remote: ! - remote: ! The runtime.txt file is deprecated since it has been replaced - remote: ! by the more widely supported .python-version file: - remote: ! https://devcenter.heroku.com/changelog-items/3141 - remote: ! - remote: ! Please switch to using a .python-version file instead. - remote: ! - remote: ! Delete your runtime.txt file and create a new file in the - remote: ! root directory of your app named: - remote: ! .python-version - remote: ! - remote: ! Make sure to include the '.' character at the start of the - remote: ! filename. Don't add a file extension such as '.txt'. - remote: ! - remote: ! In the new file, specify your app's major Python version number - remote: ! only. Don't include quotes or a 'python-' prefix. - remote: ! - remote: ! For example, to request the latest version of Python 3.9, - remote: ! update your .python-version file so it contains exactly: - remote: ! 3.9 - remote: ! - remote: ! We strongly recommend that you don't specify the Python patch - remote: ! version number, since it will pin your app to an exact Python - remote: ! version and so stop your app from receiving security updates - remote: ! each time it builds. - remote: ! - remote: ! In the future support for runtime.txt will be removed and - remote: ! this warning will be made an error. - remote: - remote: - remote: ! Warning: Support for Python 3.9 is ending soon! - remote: ! - remote: ! Python 3.9 will reach its upstream end-of-life in October 2025, - remote: ! at which point it will no longer receive security updates: - remote: ! https://devguide.python.org/versions/#supported-versions - remote: ! - remote: ! As such, support for Python 3.9 will be removed from this - remote: ! buildpack on 7th January 2026. - remote: ! - remote: ! Upgrade to a newer Python version as soon as possible, by - remote: ! changing the version in your runtime.txt file. - remote: ! - remote: ! For more information, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions - remote: - remote: - remote: ! Warning: A Python patch update is available! - remote: ! - remote: ! Your app is using Python 3.9.0, however, there is a newer - remote: ! patch release of Python 3.9 available: #{LATEST_PYTHON_3_9} - remote: ! - remote: ! It is important to always use the latest patch version of - remote: ! Python to keep your app secure. - remote: ! - remote: ! Update your runtime.txt file to use the new version. - remote: ! - remote: ! We strongly recommend that you don't pin your app to an - remote: ! exact Python version such as 3.9.0, and instead only specify - remote: ! the major Python version of 3.9 in your runtime.txt file. - remote: ! This will allow your app to receive the latest available Python - remote: ! patch version automatically and prevent this warning. - remote: - remote: -----> Installing Python 3.9.0 - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - OUTPUT - end - end - end - - context 'with a .python-version file containing an outdated patch version' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_outdated') } - - it 'warns there is a Python update available' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python 3.10.0 specified in .python-version - remote: - remote: ! Warning: A Python patch update is available! - remote: ! - remote: ! Your app is using Python 3.10.0, however, there is a newer - remote: ! patch release of Python 3.10 available: #{LATEST_PYTHON_3_10} - remote: ! - remote: ! It is important to always use the latest patch version of - remote: ! Python to keep your app secure. - remote: ! - remote: ! Update your .python-version file to use the new version. - remote: ! - remote: ! We strongly recommend that you don't pin your app to an - remote: ! exact Python version such as 3.10.0, and instead only specify - remote: ! the major Python version of 3.10 in your .python-version file. - remote: ! This will allow your app to receive the latest available Python - remote: ! patch version automatically and prevent this warning. - remote: - remote: -----> Installing Python 3.10.0 - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - OUTPUT - end - end - end -end diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 28f2bf4dd..0793353a4 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -508,6 +508,38 @@ end end + context 'when .python-version contains an outdated Python patch version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_outdated') } + + it 'warns there is a Python update available' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.13.2 specified in .python-version + remote: + remote: ! Warning: A Python patch update is available! + remote: ! + remote: ! Your app is using Python 3.13.2, however, there is a newer + remote: ! patch release of Python 3.13 available: #{LATEST_PYTHON_3_13} + remote: ! + remote: ! It is important to always use the latest patch version of + remote: ! Python to keep your app secure. + remote: ! + remote: ! Update your .python-version file to use the new version. + remote: ! + remote: ! We strongly recommend that you don't pin your app to an + remote: ! exact Python version such as 3.13.2, and instead only specify + remote: ! the major Python version of 3.13 in your .python-version file. + remote: ! This will allow your app to receive the latest available Python + remote: ! patch version automatically and prevent this warning. + remote: + remote: -----> Installing Python 3.13.2 + remote: -----> Installing pip #{PIP_VERSION} + OUTPUT + end + end + end + context 'when runtime.txt contains an invalid Python version string' do let(:app) { Hatchet::Runner.new('spec/fixtures/runtime_txt_invalid_version', allow_failure: true) } From 26b8f863cb90a339bf57e4e819ea542effee4370 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 1 Aug 2025 16:52:52 +0100 Subject: [PATCH 178/406] Cleanup pip's `Requirement already satisfied` output on older Python (#1851) On older Python pip outputs paths in a different format. The log output cleanup regex has now been updated to match against that output too (in addition to that of newer Python), so the log lines are cleaned up on Python 3.10 and older too. GUS-W-19183260. --- CHANGELOG.md | 1 + lib/pip.sh | 6 +++++- spec/hatchet/pip_spec.rb | 2 +- 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f2b2c1fd0..4c2701c62 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Updated uv from 0.7.20 to 0.8.4. ([#1847](https://github.com/heroku/heroku-buildpack-python/pull/1847)) +- Reduced the verbosity of pip's `Requirement already satisfied:` log lines when using Python 3.10 and older. ([#1851](https://github.com/heroku/heroku-buildpack-python/pull/1851)) ## [v294] - 2025-07-28 diff --git a/lib/pip.sh b/lib/pip.sh index b4f208bb7..4a7b3cde8 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -108,6 +108,9 @@ function pip::install_dependencies() { # TODO: Remove --disable-pip-version-check in favour of exporting PIP_DISABLE_PIP_VERSION_CHECK. # The sed usage is to reduce the verbosity of output lines like: + # ...when using Python 3.10 and older: + # "Requirement already satisfied: typing-extensions==4.12.2 in ./.heroku/python/lib/python3.10/site-packages (from -r requirements.txt (line 2)) (4.12.2)" + # ...when using Python 3.11+: # "Requirement already satisfied: typing-extensions==4.12.2 in /app/.heroku/python/lib/python3.13/site-packages (from -r requirements.txt (line 5)) (4.12.2)" # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { @@ -119,7 +122,8 @@ function pip::install_dependencies() { --progress-bar off \ --src='/app/.heroku/python/src' \ |& tee "${WARNINGS_LOG:?}" \ - |& sed --unbuffered --expression 's# in /app/.heroku/python/lib/python.*/site-packages##' \ + |& sed --unbuffered --regexp-extended \ + --expression 's# in (/app|\.)/\.heroku/python/lib/python[0-9.]+/site-packages##' \ |& output::indent }; then # TODO: Overhaul warnings and combine them with error handling. diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 33a151c29..b4982ec16 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -268,7 +268,7 @@ # regex variant can be removed once support for Python 3.10 and older is dropped. expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Requirement already satisfied: typing-extensions==4.14.1 in ./.heroku/python/lib/python3.9/site-packages (from -r requirements.txt (line 2)) (4.14.1) + remote: Requirement already satisfied: typing-extensions==4.14.1 (from -r requirements.txt (line 2)) (4.14.1) remote: -----> Saving cache OUTPUT end From a0b06f6c2503534384faacc69c58cbebf699e49f Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Fri, 1 Aug 2025 19:04:00 +0000 Subject: [PATCH 179/406] Prepare release v295 (#1852) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c2701c62..0893a5526 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v295] - 2025-08-01 + - Updated uv from 0.7.20 to 0.8.4. ([#1847](https://github.com/heroku/heroku-buildpack-python/pull/1847)) - Reduced the verbosity of pip's `Requirement already satisfied:` log lines when using Python 3.10 and older. ([#1851](https://github.com/heroku/heroku-buildpack-python/pull/1851)) @@ -1266,7 +1269,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v294...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v295...main +[v295]: https://github.com/heroku/heroku-buildpack-python/compare/v294...v295 [v294]: https://github.com/heroku/heroku-buildpack-python/compare/v293...v294 [v293]: https://github.com/heroku/heroku-buildpack-python/compare/v292...v293 [v292]: https://github.com/heroku/heroku-buildpack-python/compare/v291...v292 From 4a11b01b52120894789fc4ba5e073112c5abe725 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 4 Aug 2025 14:39:41 +0100 Subject: [PATCH 180/406] Fix lint errors with Shellcheck v0.11.0 (#1853) A new Shellcheck version was just released: https://github.com/koalaman/shellcheck/releases/tag/v0.11.0 ...which reports some new lint errors: ``` $ make lint In builds/build_python_runtime.sh line 38: if [[ ! " ${SUPPORTED_PYTHON_VERSIONS[*]} " == *" ${PYTHON_MAJOR_VERSION} "* ]]; then ^-- SC2335 (style): Use a != b instead of ! a == b. For more information: https://www.shellcheck.net/wiki/SC2335 -- Use a != b instead of ! a == b. make: *** [lint-scripts] Error 1 ``` GUS-W-19245644. --- builds/build_python_runtime.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index 197738295..1be5a1fad 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -35,7 +35,7 @@ case "${STACK:?}" in ;; esac -if [[ ! " ${SUPPORTED_PYTHON_VERSIONS[*]} " == *" ${PYTHON_MAJOR_VERSION} "* ]]; then +if [[ " ${SUPPORTED_PYTHON_VERSIONS[*]} " != *" ${PYTHON_MAJOR_VERSION} "* ]]; then abort "Python ${PYTHON_MAJOR_VERSION} is not supported on ${STACK}!" fi From 3e5f13b470a26fc5d29f3979438b17c4f9129db5 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 6 Aug 2025 09:00:48 +0100 Subject: [PATCH 181/406] Stop installing SQLite headers and CLI (#1854) For historical reasons, the classic Python buildpack has installed the `libsqlite3-dev`, `libsqlite3-0` and `sqlite3` packages as part of the build. However, these are not actually required for the Python `sqlite3` stdlib module to work (since the headers are installed in the `Dockerfile` used specifically for those builds): https://docs.python.org/3/library/sqlite3.html There is also a `pysqlite3` package on PyPI that uses the headers, however: 1. It's virtually identical to the stdlib module with Python, and so most apps should just use the stdlib's `sqlite3` module instead. 2. For apps that absolutely must use `pysqlite3`, the maintainer of that package makes a pre-compiled binary version of the package available that can be used without the need for headers: `pysqlite3-binary` 3. For apps that really want to install `pysqlite3` from source (or have some other need for the SQLite headers), they can be installed using the APT buildpack by adding `libsqlite3-dev` and `libsqlite3-0` to the `Aptfile`. Metrics show that only two apps built in the last 60 days use the `pysqlite3` package, confirming that including the headers by default does not make sense. This just leaves the SQLite CLI, which: - Won't be very commonly used, given that most apps don't use SQLite due to the ephemeral storage on dynos - Is outside the scope of what this buildpack should be installing, and is why the APT buildpack exists ...and so is also something the buildpack shouldn't be installing itself. Instead, any apps that need the SQLite CLI should use the APT buildpack and add `sqlite3` to their `Aptfile`. By removing the installation of these packages, we: - Speed up the build (metrics show the install time over last 60 days has a P50 of 4.2s, average of 6.4s, P99.9 of 667s) - Improve build reliability, since it no longer depends on the uptime of the Ubuntu APT mirror - Improve parity with the Python CNB, which doesn't install the SQLite headers or CLI either - (As an added bonus) Reduce the slug size slightly The buildpack already doesn't install these packages when using Python 3.13 (which is the buildpack's default Python version), or when using Poetry or uv. As such, this change only affects apps using Python 3.12 or older, with pip or Pipenv. Closes #1703. GUS-W-18915614. --- CHANGELOG.md | 1 + bin/compile | 11 --- bin/report | 29 +------- bin/steps/sqlite3 | 81 --------------------- bin/warnings | 36 ++++++++- lib/cache.sh | 4 + spec/fixtures/pip_pysqlite3/.python-version | 2 - spec/hatchet/pip_spec.rb | 45 ++++++++++-- spec/hatchet/pipenv_spec.rb | 1 - spec/hatchet/python_version_spec.rb | 3 +- spec/hatchet/stack_spec.rb | 2 +- 11 files changed, 82 insertions(+), 133 deletions(-) delete mode 100755 bin/steps/sqlite3 delete mode 100644 spec/fixtures/pip_pysqlite3/.python-version diff --git a/CHANGELOG.md b/CHANGELOG.md index 0893a5526..56565ae18 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Stopped installing SQLite headers and CLI. ([#1854](https://github.com/heroku/heroku-buildpack-python/pull/1854)) ## [v295] - 2025-08-01 diff --git a/bin/compile b/bin/compile index 71efc7a9a..0a909fe4b 100755 --- a/bin/compile +++ b/bin/compile @@ -186,17 +186,6 @@ case "${package_manager}" in esac meta_time "package_manager_install_duration" "${package_manager_install_start_time}" -# SQLite3 support. -# Installs the sqlite3 dev headers and sqlite3 binary but not the -# libsqlite3-0 library since that exists in the base image. -# We skip this step on Python 3.13 or when using Poetry/uv, as a first step towards removing this feature. -if [[ "${python_major_version}" == +(3.9|3.10|3.11|3.12) && "${package_manager}" != +(poetry|uv) ]]; then - install_sqlite_start_time=$(nowms) - source "${BUILDPACK_DIR}/bin/steps/sqlite3" - buildpack_sqlite3_install - meta_time "sqlite_install_duration" "${install_sqlite_start_time}" -fi - # Install app dependencies. dependencies_install_start_time=$(nowms) case "${package_manager}" in diff --git a/bin/report b/bin/report index 5ae6839dd..884e5c70b 100755 --- a/bin/report +++ b/bin/report @@ -16,6 +16,10 @@ # facing build logs) to avoid breaking builds unnecessarily / causing confusion. To debug # issues check the internal build system logs for `buildpack.report.failed` events, or # when developing run `make run` in this repo locally, which runs `bin/report` too. +# +# Note: The build system doesn't source the `export` script before running this script, +# so Python/the package manager won't be on PATH. If we ever need to run them during +# `bin/report` in the future, we'll need to manually source the export script again. set -euo pipefail shopt -s inherit_errexit @@ -25,14 +29,6 @@ CACHE_DIR="${2}" # The absolute path to the root of the buildpack. BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) -# The build system doesn't source the `export` script before running this script, so we have to do -# so manually (if it exists) so that buildpack Python/pip can be found (if the build succeeded). -EXPORT_FILE="${BUILDPACK_DIR}/export" -if [[ -f "${EXPORT_FILE}" ]]; then - # shellcheck source=/dev/null - source "${EXPORT_FILE}" -fi - source "${BUILDPACK_DIR}/lib/metadata.sh" meta_init "${CACHE_DIR}" "python" @@ -96,7 +92,6 @@ ALL_OTHER_FIELDS=( python_version_outdated python_version_pinned setup_py_only - sqlite_install_duration total_duration ) @@ -109,19 +104,3 @@ for field in "${ALL_OTHER_FIELDS[@]}"; do # shellcheck disable=SC2312 # TODO: Invoke this command separately to avoid masking its return value. kv_pair "${field}" "$(meta_get "${field}")" done - -# If the build failed, pip might not have been installed yet. -if command -v pip >/dev/null; then - # Determine pysqlite3 usage since it's the only package that requires the sqlite3 headers. - if pip show pysqlite3 &>/dev/null; then - kv_pair pysqlite3_installed true - else - kv_pair pysqlite3_installed false - fi - - if pip show pysqlite3-binary &>/dev/null; then - kv_pair pysqlite3_binary_installed true - else - kv_pair pysqlite3_binary_installed false - fi -fi diff --git a/bin/steps/sqlite3 b/bin/steps/sqlite3 deleted file mode 100755 index 1a94d5965..000000000 --- a/bin/steps/sqlite3 +++ /dev/null @@ -1,81 +0,0 @@ -#!/usr/bin/env bash -# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. - -# TODO: Remove this entirely since the Python stdlib now includes modern sqlite support, -# and the APT buildpack should be used if an app needs the sqlite CLI/headers. - -sqlite3_install() { - HEROKU_PYTHON_DIR="$1" - HEADERS_ONLY="${3:-}" - - mkdir -p "$HEROKU_PYTHON_DIR" - - APT_CACHE_DIR="$HEROKU_PYTHON_DIR/apt/cache" - APT_STATE_DIR="$HEROKU_PYTHON_DIR/apt/state" - - mkdir -p "$APT_CACHE_DIR/archives/partial" - mkdir -p "$APT_STATE_DIR/lists/partial" - - APT_OPTIONS=( - "--option=debug::nolocking=true" - "--option=dir::cache=${APT_CACHE_DIR}" - "--option=dir::state=${APT_STATE_DIR}" - "--option=dir::etc::sourcelist=/etc/apt/sources.list" - ) - - apt-get "${APT_OPTIONS[@]}" update >/dev/null 2>&1 - if [[ -z "$HEADERS_ONLY" ]]; then - apt-get "${APT_OPTIONS[@]}" -y -d --reinstall install libsqlite3-dev sqlite3 >/dev/null 2>&1 - else - apt-get "${APT_OPTIONS[@]}" -y -d --reinstall install libsqlite3-dev - fi - - find "$APT_CACHE_DIR/archives/" -name "*.deb" -exec dpkg -x {} "$HEROKU_PYTHON_DIR/sqlite3/" \; - - mkdir -p "$HEROKU_PYTHON_DIR/include" - mkdir -p "$HEROKU_PYTHON_DIR/lib" - - # remove old sqlite3 libraries/binaries - find "$HEROKU_PYTHON_DIR/include/" -name "sqlite3*.h" -exec rm -f {} \; - find "$HEROKU_PYTHON_DIR/lib/" -name "libsqlite3.*" -exec rm -f {} \; - rm -f "$HEROKU_PYTHON_DIR/lib/pkgconfig/sqlite3.pc" - rm -f "$HEROKU_PYTHON_DIR/bin/sqlite3" - - # eg: `x86_64` or `aarch64` - GNU_ARCH=$(arch) - - # copy over sqlite3 headers & bins and setup linking against the stack image library - mv "$HEROKU_PYTHON_DIR/sqlite3/usr/include/"* "$HEROKU_PYTHON_DIR/include/" - mv "$HEROKU_PYTHON_DIR/sqlite3/usr/lib/${GNU_ARCH}-linux-gnu"/libsqlite3.*a "$HEROKU_PYTHON_DIR/lib/" - mkdir -p "$HEROKU_PYTHON_DIR/lib/pkgconfig" - # set the right prefix/lib directories - sed -e 's/prefix=\/usr/prefix=\/app\/.heroku\/python/' -e "s/\/${GNU_ARCH}-linux-gnu//" "$HEROKU_PYTHON_DIR/sqlite3/usr/lib/${GNU_ARCH}-linux-gnu/pkgconfig/sqlite3.pc" >"$HEROKU_PYTHON_DIR/lib/pkgconfig/sqlite3.pc" - # need to point the libsqlite3.so to the stack image library for /usr/bin/ld -lsqlite3 - SQLITE3_LIBFILE="/usr/lib/${GNU_ARCH}-linux-gnu/$(readlink -n "$HEROKU_PYTHON_DIR/sqlite3/usr/lib/${GNU_ARCH}-linux-gnu/libsqlite3.so")" - ln -s "$SQLITE3_LIBFILE" "$HEROKU_PYTHON_DIR/lib/libsqlite3.so" - if [[ -z "$HEADERS_ONLY" ]]; then - mv "$HEROKU_PYTHON_DIR/sqlite3/usr/bin"/* "$HEROKU_PYTHON_DIR/bin/" - fi - - # cleanup - rm -rf "$HEROKU_PYTHON_DIR/sqlite3/" - rm -rf "$HEROKU_PYTHON_DIR/apt/" -} - -buildpack_sqlite3_install() { - output::step "Installing SQLite3" - - # TODO: This never actual prints failure or even aborts the build, since - # the conditional disables `set -e` inside the called function: - # https://stackoverflow.com/q/19789102 - # ...plus whoever wrote this forgot the `exit 1` in the `else` anyway. - # shellcheck disable=SC2310 # TODO: This function is invoked in an 'if' condition so set -e will be disabled. - # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. - if sqlite3_install "$BUILD_DIR/.heroku/python"; then - # mcount "success.python.sqlite3" - : - else - echo "Sqlite3 failed to install." - # mcount "failure.python.sqlite3" - fi -} diff --git a/bin/warnings b/bin/warnings index 0c1d861fd..6992cc48e 100755 --- a/bin/warnings +++ b/bin/warnings @@ -4,15 +4,42 @@ show-warnings() { # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. if grep -qi 'Could not find gdal-config' "${WARNINGS_LOG}"; then output::error <<-'EOF' - Error: Package installation failed since the GDAL library was not found. + Error: Package installation failed since the GDAL library wasn't found. For GDAL, GEOS and PROJ support, use the Geo buildpack alongside the Python buildpack: https://github.com/heroku/heroku-geo-buildpack EOF - fi + meta_set "failure_detail" "Could not find gdal-config" + elif grep -qi 'sqlite3.h: No such file or directory' "${WARNINGS_LOG}"; then + output::error <<-'EOF' + Error: Package installation failed since SQLite headers weren't found. - # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. - if grep -qi 'Please use pip<24.1 if you need to use this version' "${WARNINGS_LOG}"; then + The Python buildpack no longer installs the SQLite headers + package since most apps don't require it. + + If you're trying to install the `pysqlite3` package, we + recommend using the virtually identical `sqlite3` module in + Python's standard library instead: + https://docs.python.org/3/library/sqlite3.html + + To do this: + 1. Remove the `pysqlite3` package from your dependencies. + 2. Replace any `pysqlite3` imports in your app with `sqlite3`. + + Alternatively, if you can't use the `sqlite3` stdlib module, + switch from the `pysqlite3` package to `pysqlite3-binary`, + which is pre-compiled and doesn't need the SQLite headers to + be installed. + + If instead you need the SQLite headers for another reason + (or wish to continue to compile the `pysqlite3` package from + source), then install the `libsqlite3-dev` and `libsqlite3-0` + packages using the APT buildpack (make sure the APT buildpack + runs before the Python buildpack, not after): + https://github.com/heroku/heroku-buildpack-apt + EOF + meta_set "failure_detail" "sqlite3.h: No such file or directory" + elif grep -qi 'Please use pip<24.1 if you need to use this version' "${WARNINGS_LOG}"; then output::error <<-'EOF' Error: One of your dependencies contains broken metadata. @@ -24,5 +51,6 @@ show-warnings() { For more help, see: https://devcenter.heroku.com/changelog-items/3073 EOF + meta_set "failure_detail" "Please use pip<24.1 if you need to use this version" fi } diff --git a/lib/cache.sh b/lib/cache.sh index 421470790..c3038d087 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -135,6 +135,10 @@ function cache::restore() { esac fi + if [[ -f "${cache_dir}/.heroku/python/include/sqlite3.h" ]]; then + cache_invalidation_reasons+=("The legacy SQLite3 headers and CLI binary need to be uninstalled") + fi + if [[ -n "${cache_invalidation_reasons[*]}" ]]; then output::step "Discarding cache since:" local reason diff --git a/spec/fixtures/pip_pysqlite3/.python-version b/spec/fixtures/pip_pysqlite3/.python-version deleted file mode 100644 index 56ada5c99..000000000 --- a/spec/fixtures/pip_pysqlite3/.python-version +++ /dev/null @@ -1,2 +0,0 @@ -# TODO: Switch this to using Python 3.13 as part of the sqlite removal. -3.12 diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index b4982ec16..ccea61d64 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -252,7 +252,6 @@ remote: remote: -----> Installing Python 3.9.0 remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.14.1 (from -r requirements.txt (line 2)) remote: Downloading typing_extensions-4.14.1-py3-none-any.whl.metadata (3.0 kB) @@ -314,7 +313,7 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: note: This error originates from a subprocess, and is likely not a problem with pip. remote: - remote: ! Error: Package installation failed since the GDAL library was not found. + remote: ! Error: Package installation failed since the GDAL library wasn't found. remote: ! remote: ! For GDAL, GEOS and PROJ support, use the Geo buildpack alongside the Python buildpack: remote: ! https://github.com/heroku/heroku-geo-buildpack @@ -359,13 +358,47 @@ end end - # TODO: Switch this to using Python 3.13 as part of the sqlite removal. context 'when requirements.txt contains pysqlite3' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pip_pysqlite3') } + let(:app) { Hatchet::Runner.new('spec/fixtures/pip_pysqlite3', allow_failure: true) } - it 'installs successfully using pip' do + it 'outputs instructions for how to resolve the build failure' do app.deploy do |app| - expect(app.output).to include('Building wheel for pysqlite3') + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: ERROR: Failed to build installable wheels for some pyproject.toml based projects (pysqlite3) + remote: + remote: ! Error: Package installation failed since SQLite headers weren't found. + remote: ! + remote: ! The Python buildpack no longer installs the SQLite headers + remote: ! package since most apps don't require it. + remote: ! + remote: ! If you're trying to install the `pysqlite3` package, we + remote: ! recommend using the virtually identical `sqlite3` module in + remote: ! Python's standard library instead: + remote: ! https://docs.python.org/3/library/sqlite3.html + remote: ! + remote: ! To do this: + remote: ! 1. Remove the `pysqlite3` package from your dependencies. + remote: ! 2. Replace any `pysqlite3` imports in your app with `sqlite3`. + remote: ! + remote: ! Alternatively, if you can't use the `sqlite3` stdlib module, + remote: ! switch from the `pysqlite3` package to `pysqlite3-binary`, + remote: ! which is pre-compiled and doesn't need the SQLite headers to + remote: ! be installed. + remote: ! + remote: ! If instead you need the SQLite headers for another reason + remote: ! (or wish to continue to compile the `pysqlite3` package from + remote: ! source), then install the `libsqlite3-dev` and `libsqlite3-0` + remote: ! packages using the APT buildpack (make sure the APT buildpack + remote: ! runs before the Python buildpack, not after): + remote: ! https://github.com/heroku/heroku-buildpack-apt + remote: + remote: + remote: ! Error: Unable to install dependencies using pip. + remote: ! + remote: ! See the log output above for more information. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT end end end diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index deb0ab449..de611337d 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -135,7 +135,6 @@ remote: remote: -----> Installing Python 3.9.0 remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Saving cache diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 0793353a4..ac0d4a686 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -20,7 +20,6 @@ remote: -----> Using Python #{requested_version} specified in .python-version remote: -----> Installing Python #{resolved_version} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT @@ -150,6 +149,7 @@ remote: -----> Discarding cache since: remote: - The Python version has changed from 3.12.7 to #{LATEST_PYTHON_3_12} remote: - The pip version has changed from 24.0 to #{PIP_VERSION} + remote: - The legacy SQLite3 headers and CLI binary need to be uninstalled remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} OUTPUT @@ -275,7 +275,6 @@ remote: remote: -----> Installing Python #{LATEST_PYTHON_3_9} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index cc9975386..b444b367d 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -67,9 +67,9 @@ remote: - The stack has changed from heroku-22 to heroku-24 remote: - The Python version has changed from 3.12.3 to #{LATEST_PYTHON_3_12} remote: - The pip version has changed + remote: - The legacy SQLite3 headers and CLI binary need to be uninstalled remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing SQLite3 remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT From 921b938b9d0e0422b5f9e391b85eed38f4e3fab3 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 6 Aug 2025 09:20:05 +0100 Subject: [PATCH 182/406] Bump rubocop from 1.79.1 to 1.79.2 in the ruby-dependencies group (#1855) Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.79.1 to 1.79.2 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.79.1...v1.79.2) --- updated-dependencies: - dependency-name: rubocop dependency-version: 1.79.2 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 3f5119fbd..dcba3c5e0 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -41,7 +41,7 @@ GEM racc (1.8.1) rainbow (3.1.1) rate_throttle_client (0.1.2) - regexp_parser (2.10.0) + regexp_parser (2.11.0) rrrretry (1.0.0) rspec-core (3.13.5) rspec-support (~> 3.13.0) @@ -51,7 +51,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.4) - rubocop (1.79.1) + rubocop (1.79.2) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) From 22fc2cdc0389881fc748ed119a07d86f67b2c898 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 6 Aug 2025 08:37:09 +0000 Subject: [PATCH 183/406] Prepare release v296 (#1856) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 56565ae18..f9a31b39d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v296] - 2025-08-06 + - Stopped installing SQLite headers and CLI. ([#1854](https://github.com/heroku/heroku-buildpack-python/pull/1854)) ## [v295] - 2025-08-01 @@ -1270,7 +1273,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v295...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v296...main +[v296]: https://github.com/heroku/heroku-buildpack-python/compare/v295...v296 [v295]: https://github.com/heroku/heroku-buildpack-python/compare/v294...v295 [v294]: https://github.com/heroku/heroku-buildpack-python/compare/v293...v294 [v293]: https://github.com/heroku/heroku-buildpack-python/compare/v292...v293 From d65647266f3155369c17f1d65dcc4376b849abf7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 6 Aug 2025 10:55:08 +0000 Subject: [PATCH 184/406] Bump pip from 25.1.1 to 25.2 (#1848) * Bump pip from 25.1.1 to 25.2 Bumps [pip](https://github.com/pypa/pip) from 25.1.1 to 25.2. - [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst) - [Commits](https://github.com/pypa/pip/compare/25.1.1...25.2) --- updated-dependencies: - dependency-name: pip dependency-version: '25.2' dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Add changelog entry * Update build log assertion for new pip output --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/pip.txt | 2 +- spec/hatchet/pip_spec.rb | 3 ++- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f9a31b39d..6dfb21e43 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated pip from 25.1.1 to 25.2. ([#1848](https://github.com/heroku/heroku-buildpack-python/pull/1848)) ## [v296] - 2025-08-06 diff --git a/requirements/pip.txt b/requirements/pip.txt index e85a84b2e..7d4044f41 100644 --- a/requirements/pip.txt +++ b/requirements/pip.txt @@ -1 +1 @@ -pip==25.1.1 +pip==25.2 diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index ccea61d64..7ce9eb067 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -364,7 +364,8 @@ it 'outputs instructions for how to resolve the build failure' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: ERROR: Failed to build installable wheels for some pyproject.toml based projects (pysqlite3) + remote: × Failed to build installable wheels for some pyproject.toml based projects + remote: ╰─> pysqlite3 remote: remote: ! Error: Package installation failed since SQLite headers weren't found. remote: ! From 817b1e85502b8f411d30b9bc7c961a574cac6fa7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 6 Aug 2025 11:23:38 +0000 Subject: [PATCH 185/406] Bump poetry from 2.1.3 to 2.1.4 (#1858) * Bump poetry from 2.1.3 to 2.1.4 Bumps [poetry](https://github.com/python-poetry/poetry) from 2.1.3 to 2.1.4. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](https://github.com/python-poetry/poetry/compare/2.1.3...2.1.4) --- updated-dependencies: - dependency-name: poetry dependency-version: 2.1.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/poetry.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6dfb21e43..fc0889fec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Updated pip from 25.1.1 to 25.2. ([#1848](https://github.com/heroku/heroku-buildpack-python/pull/1848)) +- Updated Poetry from 2.1.3 to 2.1.4. ([#1858](https://github.com/heroku/heroku-buildpack-python/pull/1858)) ## [v296] - 2025-08-06 diff --git a/requirements/poetry.txt b/requirements/poetry.txt index 3a370aa53..67f219c60 100644 --- a/requirements/poetry.txt +++ b/requirements/poetry.txt @@ -1 +1 @@ -poetry==2.1.3 +poetry==2.1.4 From 95cc4da415fbe9aeb59fd36ed46344fd163e32c9 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 6 Aug 2025 11:56:51 +0000 Subject: [PATCH 186/406] Bump uv from 0.8.4 to 0.8.5 (#1857) * Bump uv from 0.8.4 to 0.8.5 Bumps [uv](https://github.com/astral-sh/uv) from 0.8.4 to 0.8.5. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.8.4...0.8.5) --- updated-dependencies: - dependency-name: uv dependency-version: 0.8.5 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fc0889fec..1faed0e0b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ - Updated pip from 25.1.1 to 25.2. ([#1848](https://github.com/heroku/heroku-buildpack-python/pull/1848)) - Updated Poetry from 2.1.3 to 2.1.4. ([#1858](https://github.com/heroku/heroku-buildpack-python/pull/1858)) +- Updated uv from 0.8.4 to 0.8.5. ([#1857](https://github.com/heroku/heroku-buildpack-python/pull/1857)) ## [v296] - 2025-08-06 diff --git a/requirements/uv.txt b/requirements/uv.txt index 04d297a45..1d5bfff75 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.8.4 +uv==0.8.5 From ec807ccffa8e5577050d0dfe1a286477358d21ef Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 6 Aug 2025 12:27:38 +0000 Subject: [PATCH 187/406] Prepare release v297 (#1859) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1faed0e0b..45ffad469 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v297] - 2025-08-06 + - Updated pip from 25.1.1 to 25.2. ([#1848](https://github.com/heroku/heroku-buildpack-python/pull/1848)) - Updated Poetry from 2.1.3 to 2.1.4. ([#1858](https://github.com/heroku/heroku-buildpack-python/pull/1858)) - Updated uv from 0.8.4 to 0.8.5. ([#1857](https://github.com/heroku/heroku-buildpack-python/pull/1857)) @@ -1276,7 +1279,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v296...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v297...main +[v297]: https://github.com/heroku/heroku-buildpack-python/compare/v296...v297 [v296]: https://github.com/heroku/heroku-buildpack-python/compare/v295...v296 [v295]: https://github.com/heroku/heroku-buildpack-python/compare/v294...v295 [v294]: https://github.com/heroku/heroku-buildpack-python/compare/v293...v294 From 533db1025b3012761056c741b94e88e8f714e7e8 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 6 Aug 2025 19:41:11 +0100 Subject: [PATCH 188/406] Fix the stack check error message for Heroku-20 (#1860) The `heroku-20` entry was removed completely in #1778 rather than moving it to the EOL branch of the conditional, causing the "stack not recognised" message variant to be shown, instead of the "stack no longer supported" version. This error messages can never occur on Heroku (since our build system can only scheduled builds for supported stacks), but these cases can be hit for non-Heroku uses of this buildpack (such as Dokku). GUS-W-19274871. --- CHANGELOG.md | 1 + lib/checks.sh | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 45ffad469..88210c87a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Fixed the stack version check to correctly show the "stack not supported" error message for the EOL Heroku-20, instead of "stack not recognised". ([#1860](https://github.com/heroku/heroku-buildpack-python/pull/1860)) ## [v297] - 2025-08-06 diff --git a/lib/checks.sh b/lib/checks.sh index e5f0bc478..ca24e027a 100644 --- a/lib/checks.sh +++ b/lib/checks.sh @@ -7,7 +7,7 @@ function checks::ensure_supported_stack() { heroku-22 | heroku-24) return 0 ;; - cedar* | heroku-16 | heroku-18) + cedar* | heroku-16 | heroku-18 | heroku-20) # This error will only ever be seen on non-Heroku environments, since the # Heroku build system rejects builds using EOL stacks. output::error <<-EOF From 1a7e472542ea932139c7c36c3ca559fce5751e58 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 6 Aug 2025 23:58:47 +0100 Subject: [PATCH 189/406] Add support for Python 3.13.6 (#1861) Release announcement: https://blog.python.org/2025/08/python-3136-is-now-available.html Changelog: https://docs.python.org/release/3.13.6/whatsnew/changelog.html#python-3-13-6-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/16789298980 GUS-W-17604349. --- CHANGELOG.md | 1 + lib/python_version.sh | 2 +- spec/hatchet/pipenv_spec.rb | 1 + spec/spec_helper.rb | 2 +- 4 files changed, 4 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 88210c87a..1e3bf3430 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- The Python 3.13 version alias now resolves to Python 3.13.6. ([#1861](https://github.com/heroku/heroku-buildpack-python/pull/1861)) - Fixed the stack version check to correctly show the "stack not supported" error message for the EOL Heroku-20, instead of "stack not recognised". ([#1860](https://github.com/heroku/heroku-buildpack-python/pull/1860)) ## [v297] - 2025-08-06 diff --git a/lib/python_version.sh b/lib/python_version.sh index 3fcb7afcc..26444bded 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -8,7 +8,7 @@ LATEST_PYTHON_3_9="3.9.23" LATEST_PYTHON_3_10="3.10.18" LATEST_PYTHON_3_11="3.11.13" LATEST_PYTHON_3_12="3.12.11" -LATEST_PYTHON_3_13="3.13.5" +LATEST_PYTHON_3_13="3.13.6" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=9 NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=13 diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index de611337d..797d9a0a2 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -398,6 +398,7 @@ remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock remote: -----> Discarding cache since: + remote: - The Python version has changed from 3.13.5 to #{DEFAULT_PYTHON_FULL_VERSION} remote: - The Pipenv version has changed from 2024.0.1 to #{PIPENV_VERSION} remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index ff1a39195..bcaec2882 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -13,7 +13,7 @@ LATEST_PYTHON_3_10 = '3.10.18' LATEST_PYTHON_3_11 = '3.11.13' LATEST_PYTHON_3_12 = '3.12.11' -LATEST_PYTHON_3_13 = '3.13.5' +LATEST_PYTHON_3_13 = '3.13.6' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_13 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From a6075a474badec170b3f1efac041cd8bb8986d6e Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 6 Aug 2025 23:04:17 +0000 Subject: [PATCH 190/406] Prepare release v298 (#1862) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1e3bf3430..bdfc307d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v298] - 2025-08-06 + - The Python 3.13 version alias now resolves to Python 3.13.6. ([#1861](https://github.com/heroku/heroku-buildpack-python/pull/1861)) - Fixed the stack version check to correctly show the "stack not supported" error message for the EOL Heroku-20, instead of "stack not recognised". ([#1860](https://github.com/heroku/heroku-buildpack-python/pull/1860)) @@ -1281,7 +1284,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v297...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v298...main +[v298]: https://github.com/heroku/heroku-buildpack-python/compare/v297...v298 [v297]: https://github.com/heroku/heroku-buildpack-python/compare/v296...v297 [v296]: https://github.com/heroku/heroku-buildpack-python/compare/v295...v296 [v295]: https://github.com/heroku/heroku-buildpack-python/compare/v294...v295 From edf0e01edc2fc04dc84e906a4cea335b80d6ecf4 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 12 Aug 2025 08:06:15 +0000 Subject: [PATCH 191/406] Bump uv from 0.8.5 to 0.8.9 (#1866) * Bump uv from 0.8.5 to 0.8.9 Bumps [uv](https://github.com/astral-sh/uv) from 0.8.5 to 0.8.9. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.8.5...0.8.9) --- updated-dependencies: - dependency-name: uv dependency-version: 0.8.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bdfc307d8..f9838b614 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.8.5 to 0.8.9. ([#1866](https://github.com/heroku/heroku-buildpack-python/pull/1866)) ## [v298] - 2025-08-06 diff --git a/requirements/uv.txt b/requirements/uv.txt index 1d5bfff75..8a5fd6f41 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.8.5 +uv==0.8.9 From 3cbdc9328f95332d532293c33541cea1853831e7 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 13 Aug 2025 08:42:42 +0000 Subject: [PATCH 192/406] Prepare release v299 (#1867) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f9838b614..1fbd185c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v299] - 2025-08-13 + - Updated uv from 0.8.5 to 0.8.9. ([#1866](https://github.com/heroku/heroku-buildpack-python/pull/1866)) ## [v298] - 2025-08-06 @@ -1285,7 +1288,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v298...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v299...main +[v299]: https://github.com/heroku/heroku-buildpack-python/compare/v298...v299 [v298]: https://github.com/heroku/heroku-buildpack-python/compare/v297...v298 [v297]: https://github.com/heroku/heroku-buildpack-python/compare/v296...v297 [v296]: https://github.com/heroku/heroku-buildpack-python/compare/v295...v296 From 3e0b8642ec05c4ebe9e2544005d0a7f6e65310fc Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 15 Aug 2025 13:49:39 +0100 Subject: [PATCH 193/406] Add support for Python 3.13.7 (#1868) Release announcement: https://blog.python.org/2025/08/python-3140rc2-and-3137-are-go.html Changelog: https://docs.python.org/release/3.13.7/whatsnew/changelog.html#python-3-13-7-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/16984434876 GUS-W-19343640. --- CHANGELOG.md | 1 + lib/python_version.sh | 2 +- spec/spec_helper.rb | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1fbd185c2..c72074df7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- The Python 3.13 version alias now resolves to Python 3.13.7. ([#1868](https://github.com/heroku/heroku-buildpack-python/pull/1868)) ## [v299] - 2025-08-13 diff --git a/lib/python_version.sh b/lib/python_version.sh index 26444bded..5ec9eb05e 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -8,7 +8,7 @@ LATEST_PYTHON_3_9="3.9.23" LATEST_PYTHON_3_10="3.10.18" LATEST_PYTHON_3_11="3.11.13" LATEST_PYTHON_3_12="3.12.11" -LATEST_PYTHON_3_13="3.13.6" +LATEST_PYTHON_3_13="3.13.7" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=9 NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=13 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index bcaec2882..88fb52018 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -13,7 +13,7 @@ LATEST_PYTHON_3_10 = '3.10.18' LATEST_PYTHON_3_11 = '3.11.13' LATEST_PYTHON_3_12 = '3.12.11' -LATEST_PYTHON_3_13 = '3.13.6' +LATEST_PYTHON_3_13 = '3.13.7' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_13 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From 51712df06645b891fdfd27147ef3b31ecb826978 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Fri, 15 Aug 2025 13:00:59 +0000 Subject: [PATCH 194/406] Prepare release v300 (#1869) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c72074df7..3da11583a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v300] - 2025-08-15 + - The Python 3.13 version alias now resolves to Python 3.13.7. ([#1868](https://github.com/heroku/heroku-buildpack-python/pull/1868)) ## [v299] - 2025-08-13 @@ -1289,7 +1292,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v299...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v300...main +[v300]: https://github.com/heroku/heroku-buildpack-python/compare/v299...v300 [v299]: https://github.com/heroku/heroku-buildpack-python/compare/v298...v299 [v298]: https://github.com/heroku/heroku-buildpack-python/compare/v297...v298 [v297]: https://github.com/heroku/heroku-buildpack-python/compare/v296...v297 From b0974da25ce23c3e329eadeaa910aea20ad90da5 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 18 Aug 2025 14:42:29 +0100 Subject: [PATCH 195/406] Record cache restore duration for the empty cache case too (#1871) So that all builds have this field populated for consistency. --- lib/cache.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/cache.sh b/lib/cache.sh index c3038d087..0b0dc8370 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -33,13 +33,15 @@ function cache::restore() { local python_full_version="${5}" local package_manager="${6}" + local cache_restore_start_time + cache_restore_start_time=$(nowms) + if [[ ! -d "${cache_dir}/.heroku/python" ]]; then meta_set "cache_status" "empty" + meta_time "cache_restore_duration" "${cache_restore_start_time}" return 0 fi - local cache_restore_start_time - cache_restore_start_time=$(nowms) local cache_invalidation_reasons=() local cached_stack From e375ad389ae4cb491f4a96c9a39f52cacff5bf22 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 18 Aug 2025 15:06:33 +0100 Subject: [PATCH 196/406] Simplify handling of caches written by older buildpack versions (#1870) It's been a year since the current metadata store implementation was added, so it's fine to now simplify the handling/messaging around reading older cache versions. This cleanup will also help with the upcoming switch to the new JSON metadata store implementation. GUS-W-19361163. --- CHANGELOG.md | 1 + lib/cache.sh | 37 ++++++++----------------------------- spec/hatchet/stack_spec.rb | 2 +- 3 files changed, 10 insertions(+), 30 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3da11583a..6b2c000aa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Simplified the handling of caches written by older buildpack versions. ([#1870](https://github.com/heroku/heroku-buildpack-python/pull/1870)) ## [v300] - 2025-08-15 diff --git a/lib/cache.sh b/lib/cache.sh index 0b0dc8370..5d92af962 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -54,35 +54,21 @@ function cache::restore() { cache_invalidation_reasons+=("The Python version has changed from ${cached_python_full_version:-"unknown"} to ${python_full_version}") fi - # The metadata store only exists in caches created in v252+ of the buildpack (released 2024-06-17), - # so here and below we have to handle the case where `meta_prev_get` returns the empty string. local cached_package_manager cached_package_manager="$(meta_prev_get "package_manager")" if [[ -z "${cached_package_manager}" ]]; then - # Using `compgen` since `[[ -d ... ]]` doesn't work with globs. - if compgen -G "${cache_dir}/.heroku/python/lib/python*/site-packages/pipenv" >/dev/null; then - cached_package_manager="pipenv" - elif compgen -G "${cache_dir}/.heroku/python/lib/python*/site-packages/pip" >/dev/null; then - cached_package_manager="pip" - fi - fi - - if [[ "${cached_package_manager}" != "${package_manager}" ]]; then + # The metadata store only exists in caches created by v252+ of the buildpack (released 2024-06-17). + cache_invalidation_reasons+=("The buildpack cache format has changed") + elif [[ "${cached_package_manager}" != "${package_manager}" ]]; then cache_invalidation_reasons+=("The package manager has changed from ${cached_package_manager:-"unknown"} to ${package_manager}") else case "${package_manager}" in pip) local cached_pip_version cached_pip_version="$(meta_prev_get "pip_version")" - # Handle caches written by buildpack versions older than v252 (see above). - if [[ -z "${cached_pip_version}" ]]; then - # Whilst we don't know the old version, we know the pip version has definitely - # changed since buildpack v251. - cache_invalidation_reasons+=("The pip version has changed") - elif [[ "${cached_pip_version}" != "${PIP_VERSION:?}" ]]; then - cache_invalidation_reasons+=("The pip version has changed from ${cached_pip_version} to ${PIP_VERSION}") + if [[ "${cached_pip_version}" != "${PIP_VERSION:?}" ]]; then + cache_invalidation_reasons+=("The pip version has changed from ${cached_pip_version:-"unknown"} to ${PIP_VERSION}") fi - # We invalidate the cache if requirements.txt changes since pip is a package installer # rather than a project/environment manager, and so does not deterministically manage # installed Python packages. For example, if a package entry in a requirements file is @@ -97,19 +83,14 @@ function cache::restore() { pipenv) local cached_pipenv_version cached_pipenv_version="$(meta_prev_get "pipenv_version")" - # Handle caches written by buildpack versions older than v252 (see above). - if [[ -z "${cached_pipenv_version}" ]]; then - # Whilst we don't know the old version, we know the Pipenv version has definitely - # changed since buildpack v251. - cache_invalidation_reasons+=("The Pipenv version has changed") - elif [[ "${cached_pipenv_version}" != "${PIPENV_VERSION:?}" ]]; then - cache_invalidation_reasons+=("The Pipenv version has changed from ${cached_pipenv_version} to ${PIPENV_VERSION}") + if [[ "${cached_pipenv_version}" != "${PIPENV_VERSION:?}" ]]; then + cache_invalidation_reasons+=("The Pipenv version has changed from ${cached_pipenv_version:-"unknown"} to ${PIPENV_VERSION}") fi # `pipenv {install,sync}` by design don't actually uninstall packages on their own (!!): # and we can't use `pipenv clean` since it isn't compatible with `--system`. # https://github.com/pypa/pipenv/issues/3365 # We have to explicitly check for the presence of the Pipfile.lock.sha256 file, - # since we only started writing it to the build cache as of buildpack v292+. + # since we only started writing it to the build cache as of buildpack v292 (released 2025-07-23). local pipfile_lock_checksum_file="${cache_dir}/.heroku/python/Pipfile.lock.sha256" if [[ -f "${pipfile_lock_checksum_file}" ]] && ! sha256sum --check --strict --status "${pipfile_lock_checksum_file}"; then cache_invalidation_reasons+=("The contents of Pipfile.lock changed") @@ -118,7 +99,6 @@ function cache::restore() { poetry) local cached_poetry_version cached_poetry_version="$(meta_prev_get "poetry_version")" - # Poetry support was added after the metadata store, so we'll always have the version here. if [[ "${cached_poetry_version}" != "${POETRY_VERSION:?}" ]]; then cache_invalidation_reasons+=("The Poetry version has changed from ${cached_poetry_version:-"unknown"} to ${POETRY_VERSION}") fi @@ -126,7 +106,6 @@ function cache::restore() { uv) local cached_uv_version cached_uv_version="$(meta_prev_get "uv_version")" - # uv support was added after the metadata store, so we'll always have the version here. if [[ "${cached_uv_version}" != "${UV_VERSION:?}" ]]; then cache_invalidation_reasons+=("The uv version has changed from ${cached_uv_version:-"unknown"} to ${UV_VERSION}") fi diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index b444b367d..ccca341cb 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -66,7 +66,7 @@ remote: -----> Discarding cache since: remote: - The stack has changed from heroku-22 to heroku-24 remote: - The Python version has changed from 3.12.3 to #{LATEST_PYTHON_3_12} - remote: - The pip version has changed + remote: - The buildpack cache format has changed remote: - The legacy SQLite3 headers and CLI binary need to be uninstalled remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} From c1f9208cde06cdea629d735ede413e0af437e261 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Mon, 18 Aug 2025 16:09:30 +0000 Subject: [PATCH 197/406] Prepare release v301 (#1872) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b2c000aa..217db11f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v301] - 2025-08-18 + - Simplified the handling of caches written by older buildpack versions. ([#1870](https://github.com/heroku/heroku-buildpack-python/pull/1870)) ## [v300] - 2025-08-15 @@ -1293,7 +1296,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v300...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v301...main +[v301]: https://github.com/heroku/heroku-buildpack-python/compare/v300...v301 [v300]: https://github.com/heroku/heroku-buildpack-python/compare/v299...v300 [v299]: https://github.com/heroku/heroku-buildpack-python/compare/v298...v299 [v298]: https://github.com/heroku/heroku-buildpack-python/compare/v297...v298 From 82379bd42cceaa5af4fd733e213b182bf9bd33f0 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 19 Aug 2025 16:05:22 +0100 Subject: [PATCH 198/406] Remove support for `BUILDPACK_S3_BASE_URL` (#1875) Since: - Metrics show it to be unused on Heroku. - It's not supported by the CNB, so removing it improves parity. - As a feature it's pretty fragile, since unless someone managed to keep up with all changes we make to our S3 bucket, builds would break every time we bump versions. (And as such, anyone using it should be carefully curating the buildpack versions, and so can patch the buildpack as part of that repackaging/automation process.) GUS-W-19372843. --- CHANGELOG.md | 1 + bin/report | 1 - lib/python.sh | 13 ++----------- 3 files changed, 3 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 217db11f4..d3b1eae1b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Removed support for `BUILDPACK_S3_BASE_URL`. ([#1875](https://github.com/heroku/heroku-buildpack-python/pull/1875)) ## [v301] - 2025-08-18 diff --git a/bin/report b/bin/report index 884e5c70b..e93fb0c34 100755 --- a/bin/report +++ b/bin/report @@ -79,7 +79,6 @@ STRING_FIELDS=( ALL_OTHER_FIELDS=( cache_restore_duration cache_save_duration - custom_s3_base_url dependencies_install_duration django_collectstatic_duration nltk_downloader_duration diff --git a/lib/python.sh b/lib/python.sh index bbf38878d..64497f555 100644 --- a/lib/python.sh +++ b/lib/python.sh @@ -4,7 +4,7 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail -DEFAULT_S3_BASE_URL="https://heroku-buildpack-python.s3.us-east-1.amazonaws.com" +S3_BASE_URL="https://heroku-buildpack-python.s3.us-east-1.amazonaws.com" function python::install() { local build_dir="${1}" @@ -24,21 +24,12 @@ function python::install() { mkdir -p "${install_dir}" - # Note: This can't be used via app config vars, since it doesn't reference the value from ENV_DIR. - # TODO: Remove this for parity with the Python CNB, if metrics show it to be unused on Heroku. - if [[ -v BUILDPACK_S3_BASE_URL ]]; then - local s3_base_url="${BUILDPACK_S3_BASE_URL}" - meta_set "custom_s3_base_url" "true" - else - local s3_base_url="${DEFAULT_S3_BASE_URL}" - fi - # Calculating the Ubuntu version from the stack name saves having to shell out to `lsb_release`. local ubuntu_version="${stack/heroku-/}.04" local arch arch=$(dpkg --print-architecture) # e.g.: https://heroku-buildpack-python.s3.us-east-1.amazonaws.com/python-3.13.0-ubuntu-24.04-amd64.tar.zst - local python_url="${s3_base_url}/python-${python_full_version}-ubuntu-${ubuntu_version}-${arch}.tar.zst" + local python_url="${S3_BASE_URL}/python-${python_full_version}-ubuntu-${ubuntu_version}-${arch}.tar.zst" local error_log error_log=$(mktemp) From fc152f66c4bc382e7a64047768137e31373e02db Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 20 Aug 2025 13:19:44 +0100 Subject: [PATCH 199/406] Stop setting the `PYTHONHASHSEED` env var (#1876) Since hash randomisation has been enabled by default since Python 3.3, so setting `PYTHONHASHSEED=random` is the same as the default, and so redundant. See: - https://docs.python.org/3/using/cmdline.html#envvar-PYTHONHASHSEED - https://docs.python.org/3/reference/datamodel.html#object.__hash__ - https://docs.python.org/3/whatsnew/3.3.html#builtin-functions-and-types GUS-W-19372908. --- CHANGELOG.md | 1 + bin/compile | 2 -- spec/hatchet/ci_spec.rb | 8 -------- spec/hatchet/pip_spec.rb | 1 - spec/hatchet/pipenv_spec.rb | 1 - spec/hatchet/poetry_spec.rb | 1 - spec/hatchet/profile_d_scripts_spec.rb | 3 --- spec/hatchet/uv_spec.rb | 1 - 8 files changed, 1 insertion(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d3b1eae1b..959a2fd7d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Stopped setting the `PYTHONHASHSEED` env var. ([#1876](https://github.com/heroku/heroku-buildpack-python/pull/1876)) - Removed support for `BUILDPACK_S3_BASE_URL`. ([#1875](https://github.com/heroku/heroku-buildpack-python/pull/1875)) ## [v301] - 2025-08-18 diff --git a/bin/compile b/bin/compile index 0a909fe4b..32934a968 100755 --- a/bin/compile +++ b/bin/compile @@ -231,8 +231,6 @@ set_env LIBRARY_PATH "\${HOME}/.heroku/python/lib\${LIBRARY_PATH:+:\${LIBRARY_PA set_env LD_LIBRARY_PATH "\${HOME}/.heroku/python/lib\${LD_LIBRARY_PATH:+:\${LD_LIBRARY_PATH}}" # Locale. set_default_env LANG en_US.UTF-8 -# The Python hash seed is set to random. -set_default_env PYTHONHASHSEED random # Tell Python to look for Python modules in the /app dir. Don't change this. set_default_env PYTHONPATH "\${HOME}" diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 576fee12e..bd1d333b5 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -38,7 +38,6 @@ LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ - PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true @@ -52,7 +51,6 @@ LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ - PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true @@ -117,7 +115,6 @@ PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PIPENV_SYSTEM=1 PIPENV_VERBOSITY=-1 - PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true @@ -134,7 +131,6 @@ PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ PIPENV_SYSTEM=1 PIPENV_VERBOSITY=-1 - PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true @@ -204,7 +200,6 @@ PATH=/app/.heroku/python/bin:/tmp/cache.+/.heroku/python-poetry/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ POETRY_VIRTUALENVS_CREATE=false POETRY_VIRTUALENVS_USE_POETRY_PYTHON=true - PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true @@ -218,7 +213,6 @@ LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ - PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true @@ -287,7 +281,6 @@ LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/tmp/cache.+/.heroku/python-uv:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ - PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true @@ -304,7 +297,6 @@ LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ - PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 7ce9eb067..2df8cf7da 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -26,7 +26,6 @@ remote: LD_LIBRARY_PATH=/app/.heroku/python/lib remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - remote: PYTHONHASHSEED=random remote: PYTHONHOME=/app/.heroku/python remote: PYTHONPATH=/app remote: PYTHONUNBUFFERED=true diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 797d9a0a2..20a1b76ac 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -24,7 +24,6 @@ remote: PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PIPENV_SYSTEM=1 remote: PIPENV_VERBOSITY=-1 - remote: PYTHONHASHSEED=random remote: PYTHONHOME=/app/.heroku/python remote: PYTHONPATH=/app remote: PYTHONUNBUFFERED=true diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 5497e8cab..c10ac9eba 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -28,7 +28,6 @@ remote: PATH=/app/.heroku/python/bin:/tmp/codon/tmp/cache/.heroku/python-poetry/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: POETRY_VIRTUALENVS_CREATE=false remote: POETRY_VIRTUALENVS_USE_POETRY_PYTHON=true - remote: PYTHONHASHSEED=random remote: PYTHONHOME=/app/.heroku/python remote: PYTHONPATH=/app remote: PYTHONUNBUFFERED=true diff --git a/spec/hatchet/profile_d_scripts_spec.rb b/spec/hatchet/profile_d_scripts_spec.rb index 0e36a1efb..263a247a6 100644 --- a/spec/hatchet/profile_d_scripts_spec.rb +++ b/spec/hatchet/profile_d_scripts_spec.rb @@ -23,7 +23,6 @@ LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin PWD=/app - PYTHONHASHSEED=random PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true @@ -42,7 +41,6 @@ 'LD_LIBRARY_PATH=/this-should-be-preserved', 'LIBRARY_PATH=/this-should-be-preserved', 'PATH=/this-should-be-preserved:/usr/local/bin:/usr/bin:/bin', - 'PYTHONHASHSEED=this-should-be-preserved', 'PYTHONHOME=/this-should-be-overridden', 'PYTHONPATH=/this-should-be-preserved', 'PYTHONUNBUFFERED=this-should-be-overridden', @@ -59,7 +57,6 @@ LIBRARY_PATH=/app/.heroku/python/lib:/this-should-be-preserved PATH=/app/.heroku/python/bin:/this-should-be-preserved:/usr/local/bin:/usr/bin:/bin PWD=/app - PYTHONHASHSEED=this-should-be-preserved PYTHONHOME=/app/.heroku/python PYTHONPATH=/this-should-be-preserved PYTHONUNBUFFERED=true diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 821ae0d20..863ad67d4 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -26,7 +26,6 @@ remote: LD_LIBRARY_PATH=/app/.heroku/python/lib remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin:/tmp/codon/tmp/cache/.heroku/python-uv:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - remote: PYTHONHASHSEED=random remote: PYTHONHOME=/app/.heroku/python remote: PYTHONPATH=/app remote: PYTHONUNBUFFERED=true From 3c27de9acc7a7bbcaf27d281e1381cd81c793999 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 20 Aug 2025 15:24:51 +0100 Subject: [PATCH 200/406] Add tests for build metadata contents (#1877) It's not possible to test `bin/report` functionality directly using Hatchet tests, however, we can dump the raw build metadata store values to at least test that we're storing the expected keys/values. This complements the non-Hatchet container CI test that checks that `bin/report` exits zero. This has been split out of the upcoming `bin/report` refactor PR to keep that PR smaller, and to more easily demonstrate the change in behaviour after that PR. GUS-W-19384622. --- spec/fixtures/pip_basic/bin/compile | 4 +++ spec/fixtures/pipenv_basic/bin/compile | 4 +++ spec/fixtures/poetry_basic/bin/compile | 4 +++ spec/fixtures/uv_basic/bin/compile | 4 +++ spec/hatchet/pip_spec.rb | 38 ++++++++++++++++++++------ spec/hatchet/pipenv_spec.rb | 22 +++++++++++++++ spec/hatchet/poetry_spec.rb | 34 +++++++++++++++++++---- spec/hatchet/uv_spec.rb | 22 +++++++++++++++ 8 files changed, 118 insertions(+), 14 deletions(-) diff --git a/spec/fixtures/pip_basic/bin/compile b/spec/fixtures/pip_basic/bin/compile index cb66e67da..942fe8171 100755 --- a/spec/fixtures/pip_basic/bin/compile +++ b/spec/fixtures/pip_basic/bin/compile @@ -6,6 +6,7 @@ set -euo pipefail BUILD_DIR="${1}" +CACHE_DIR="${2}" cd "${BUILD_DIR}" @@ -21,3 +22,6 @@ pip list --disable-pip-version-check echo python -c 'import typing_extensions; print(typing_extensions)' +echo + +sort "${CACHE_DIR}/build-data/python" diff --git a/spec/fixtures/pipenv_basic/bin/compile b/spec/fixtures/pipenv_basic/bin/compile index aa11dd877..de32b7386 100755 --- a/spec/fixtures/pipenv_basic/bin/compile +++ b/spec/fixtures/pipenv_basic/bin/compile @@ -6,6 +6,7 @@ set -euo pipefail BUILD_DIR="${1}" +CACHE_DIR="${2}" cd "${BUILD_DIR}" @@ -22,3 +23,6 @@ pip list --disable-pip-version-check --exclude pip echo python -c 'import typing_extensions; print(typing_extensions)' +echo + +sort "${CACHE_DIR}/build-data/python" diff --git a/spec/fixtures/poetry_basic/bin/compile b/spec/fixtures/poetry_basic/bin/compile index 9491a868e..77b8afa7a 100755 --- a/spec/fixtures/poetry_basic/bin/compile +++ b/spec/fixtures/poetry_basic/bin/compile @@ -6,6 +6,7 @@ set -euo pipefail BUILD_DIR="${1}" +CACHE_DIR="${2}" cd "${BUILD_DIR}" @@ -22,3 +23,6 @@ poetry show | grep -v ' (!) ' echo python -c 'import typing_extensions; print(typing_extensions)' +echo + +sort "${CACHE_DIR}/build-data/python" diff --git a/spec/fixtures/uv_basic/bin/compile b/spec/fixtures/uv_basic/bin/compile index 0e3ad460c..7668d373b 100755 --- a/spec/fixtures/uv_basic/bin/compile +++ b/spec/fixtures/uv_basic/bin/compile @@ -6,6 +6,7 @@ set -euo pipefail BUILD_DIR="${1}" +CACHE_DIR="${2}" cd "${BUILD_DIR}" @@ -20,3 +21,6 @@ uv pip list echo python -c 'import typing_extensions; print(typing_extensions)' +echo + +sort "${CACHE_DIR}/build-data/python" diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 2df8cf7da..39196d2c9 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -9,15 +9,15 @@ it 're-uses packages from the cache' do app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 5)) - remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) - remote: Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) + remote: Collecting typing-extensions==4.12.2 \\(from -r requirements.txt \\(line 5\\)\\) + remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata \\(3.0 kB\\) + remote: Downloading typing_extensions-4.12.2-py3-none-any.whl \\(37 kB\\) remote: Installing collected packages: typing-extensions remote: Successfully installed typing-extensions-4.12.2 remote: -----> Saving cache @@ -30,21 +30,43 @@ remote: PYTHONPATH=/app remote: PYTHONUNBUFFERED=true remote: - remote: ['', + remote: \\['', remote: '/app', remote: '/app/.heroku/python/lib/python313.zip', remote: '/app/.heroku/python/lib/python3.13', remote: '/app/.heroku/python/lib/python3.13/lib-dynload', - remote: '/app/.heroku/python/lib/python3.13/site-packages'] + remote: '/app/.heroku/python/lib/python3.13/site-packages'\\] remote: - remote: pip #{PIP_VERSION} from /app/.heroku/python/lib/python3.13/site-packages/pip (python 3.13) + remote: pip #{PIP_VERSION} from /app/.heroku/python/lib/python3.13/site-packages/pip \\(python 3.13\\) remote: Package Version remote: ----------------- ------- remote: pip #{PIP_VERSION} remote: typing_extensions 4.12.2 remote: remote: - OUTPUT + remote: + remote: + remote: cache_restore_duration=[0-9.]+ + remote: cache_save_duration=[0-9.]+ + remote: cache_status=empty + remote: dependencies_install_duration=[0-9.]+ + remote: django_collectstatic_duration=[0-9.]+ + remote: nltk_downloader_duration=[0-9.]+ + remote: package_manager_install_duration=[0-9.]+ + remote: package_manager=pip + remote: pip_version=#{PIP_VERSION} + remote: post_compile_hook=false + remote: pre_compile_hook=false + remote: python_install_duration=[0-9.]+ + remote: python_version=#{DEFAULT_PYTHON_FULL_VERSION} + remote: python_version_major=3.13 + remote: python_version_origin=.python-version + remote: python_version_outdated=false + remote: python_version_pinned=false + remote: python_version_requested=3.13 + remote: setup_py_only=false + remote: total_duration=[0-9.]+ + REGEX app.commit! app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 20a1b76ac..170b141ef 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -43,6 +43,28 @@ remote: typing_extensions 4.12.2 remote: remote: + remote: + remote: + remote: cache_restore_duration=[0-9.]+ + remote: cache_save_duration=[0-9.]+ + remote: cache_status=empty + remote: dependencies_install_duration=[0-9.]+ + remote: django_collectstatic_duration=[0-9.]+ + remote: nltk_downloader_duration=[0-9.]+ + remote: package_manager_install_duration=[0-9.]+ + remote: package_manager=pipenv + remote: pipenv_version=#{PIPENV_VERSION} + remote: post_compile_hook=false + remote: pre_compile_hook=false + remote: python_install_duration=[0-9.]+ + remote: python_version=#{DEFAULT_PYTHON_FULL_VERSION} + remote: python_version_major=3.13 + remote: python_version_origin=Pipfile.lock + remote: python_version_outdated=false + remote: python_version_pinned=false + remote: python_version_requested=3.13 + remote: setup_py_only=false + remote: total_duration=[0-9.]+ REGEX app.commit! app.push! diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index c10ac9eba..ee9fd238f 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -9,7 +9,7 @@ it 'installs successfully using Poetry and on rebuilds uses the cache' do app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} @@ -19,7 +19,7 @@ remote: remote: Package operations: 1 install, 0 updates, 0 removals remote: - remote: - Installing typing-extensions (4.12.2) + remote: - Installing typing-extensions \\(4.12.2\\) remote: -----> Saving cache remote: -----> Inline app detected remote: LANG=en_US.UTF-8 @@ -32,19 +32,41 @@ remote: PYTHONPATH=/app remote: PYTHONUNBUFFERED=true remote: - remote: ['', + remote: \\['', remote: '/app', remote: '/app/.heroku/python/lib/python313.zip', remote: '/app/.heroku/python/lib/python3.13', remote: '/app/.heroku/python/lib/python3.13/lib-dynload', - remote: '/app/.heroku/python/lib/python3.13/site-packages'] + remote: '/app/.heroku/python/lib/python3.13/site-packages'\\] remote: - remote: Poetry (version #{POETRY_VERSION}) + remote: Poetry \\(version #{POETRY_VERSION}\\) remote: Skipping virtualenv creation, as specified in config file. remote: typing-extensions 4.12.2 Backported and Experimental Type Hints for Python ... remote: remote: - OUTPUT + remote: + remote: + remote: cache_restore_duration=[0-9.]+ + remote: cache_save_duration=[0-9.]+ + remote: cache_status=empty + remote: dependencies_install_duration=[0-9.]+ + remote: django_collectstatic_duration=[0-9.]+ + remote: nltk_downloader_duration=[0-9.]+ + remote: package_manager_install_duration=[0-9.]+ + remote: package_manager=poetry + remote: poetry_version=#{POETRY_VERSION} + remote: post_compile_hook=false + remote: pre_compile_hook=false + remote: python_install_duration=[0-9.]+ + remote: python_version=#{DEFAULT_PYTHON_FULL_VERSION} + remote: python_version_major=3.13 + remote: python_version_origin=.python-version + remote: python_version_outdated=false + remote: python_version_pinned=false + remote: python_version_requested=3.13 + remote: setup_py_only=false + remote: total_duration=[0-9.]+ + REGEX app.commit! app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 863ad67d4..3dd7a1fd2 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -48,6 +48,28 @@ remote: typing-extensions 4.13.2 remote: remote: + remote: + remote: + remote: cache_restore_duration=[0-9.]+ + remote: cache_save_duration=[0-9.]+ + remote: cache_status=empty + remote: dependencies_install_duration=[0-9.]+ + remote: django_collectstatic_duration=[0-9.]+ + remote: nltk_downloader_duration=[0-9.]+ + remote: package_manager_install_duration=[0-9.]+ + remote: package_manager=uv + remote: post_compile_hook=false + remote: pre_compile_hook=false + remote: python_install_duration=[0-9.]+ + remote: python_version=#{DEFAULT_PYTHON_FULL_VERSION} + remote: python_version_major=3.13 + remote: python_version_origin=.python-version + remote: python_version_outdated=false + remote: python_version_pinned=false + remote: python_version_requested=3.13 + remote: setup_py_only=false + remote: total_duration=[0-9.]+ + remote: uv_version=#{UV_VERSION} REGEX app.commit! app.push! From db9788b989134e3a5a060f9208b86b2ddcf48ddb Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 21 Aug 2025 16:05:42 +0100 Subject: [PATCH 201/406] Refactor buildpack data store and `bin/report` (#1878) The Node.js buildpack had a set of Bash helper utils for managing a key-value store of metadata, that could be added to during the build, and then output when bin/report was run. When the Python buildpack added support for bin/report, its implementation was based on the Node.js one, with some further small improvements. However, there remains a number of issues with the design: 1. It handles different data types (string vs int vs float vs bool) at `bin/report` time, so requires a hardcoded mapping of values which can (and does) get out of sync. (Plus doesn't work well for some cases, like jvm-common being wrapped by the other JVM related buildpacks.) 2. It tries to escape newlines and carriage returns, but there are a number of edge cases still not handled properly which result in some fields not being sent to Honeycomb properly. In order to work around the above, Manuel created a jq based approach for use by the JVM buildpacks: https://github.com/heroku/heroku-buildpack-jvm-common/blob/4fd6e4f8f258a5766a8b2e9ffec2690675b00879/lib/metrics.sh However, that doesn't have the full API of the existing solution (it's missing the ability to fetch previous values; something which Python needs), and I spotted a few more places where things can be optimised. In addition, now that we're using jq and JSON internally in the buildpack, it makes sense for us to update cytokine to support the build report being emitted as JSON too (in addition to YAML), particularly since cytokine then submits the report to Vacuole as JSON afterwards. And in fact, since JSON is roughly a subset of YAML (at least for the simple data types and schema we're using for the build report), we can start emitting the build report as JSON now, rather than needing to wait for the cytokine change to land first. (We'll still want to update cytokine to use a JSON parser though, just to avoid any edge cases in the future). The new implementation: - Sets the data type at the call site where the metadata is set, rather than relying on a hardcoded mapping at time of retrieval. - Uses jq to do the string escaping, so much less fragile than the hand-rolled implementation. - Uses a simplified initialisation/setup API, which avoids the need to call both `init` and `setup`. - Has been rewritten to avoid unnecessary processes calls, redirections, and patterns that required disabling a number of shellcheck warnings. I've confirmed that metrics still appear in Honeycomb with the changes here. After this merges, I'll open PRs against some of the other language buildpacks that don't yet implement `bin/report`, using the improved implementation from here (with simplifications since other buildpacks won't need the legacy data store handling that Python needs to support caches written by older buildpack versions). GUS-W-19344754. Signed-off-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + bin/compile | 45 ++++--- bin/report | 91 ++------------ bin/steps/collectstatic | 23 ++-- bin/steps/nltk | 11 +- bin/warnings | 6 +- lib/build_data.sh | 157 +++++++++++++++++++++++++ lib/cache.sh | 34 +++--- lib/checks.sh | 12 +- lib/hooks.sh | 10 +- lib/kvstore.sh | 101 ---------------- lib/metadata.sh | 78 ------------ lib/package_manager.sh | 10 +- lib/pip.sh | 10 +- lib/pipenv.sh | 8 +- lib/poetry.sh | 8 +- lib/python.sh | 12 +- lib/python_version.sh | 40 +++---- lib/utils.sh | 6 +- lib/uv.sh | 6 +- spec/fixtures/pip_basic/bin/compile | 2 +- spec/fixtures/pipenv_basic/bin/compile | 2 +- spec/fixtures/poetry_basic/bin/compile | 2 +- spec/fixtures/uv_basic/bin/compile | 2 +- spec/hatchet/pip_spec.rb | 43 +++---- spec/hatchet/pipenv_spec.rb | 43 +++---- spec/hatchet/poetry_spec.rb | 43 +++---- spec/hatchet/uv_spec.rb | 43 +++---- vendor/buildpack-stdlib_v8.sh | 5 - 29 files changed, 375 insertions(+), 479 deletions(-) create mode 100644 lib/build_data.sh delete mode 100644 lib/kvstore.sh delete mode 100644 lib/metadata.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 959a2fd7d..aa406181e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ - Stopped setting the `PYTHONHASHSEED` env var. ([#1876](https://github.com/heroku/heroku-buildpack-python/pull/1876)) - Removed support for `BUILDPACK_S3_BASE_URL`. ([#1875](https://github.com/heroku/heroku-buildpack-python/pull/1875)) +- Refactored buildpack data store and `bin/report` to simplify the implementation and fix some string escaping bugs. ([#1878](https://github.com/heroku/heroku-buildpack-python/pull/1878)) ## [v301] - 2025-08-18 diff --git a/bin/compile b/bin/compile index 32934a968..7b7abe13b 100755 --- a/bin/compile +++ b/bin/compile @@ -20,10 +20,10 @@ BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) source "${BUILDPACK_DIR}/bin/utils" source "${BUILDPACK_DIR}/lib/utils.sh" +source "${BUILDPACK_DIR}/lib/build_data.sh" source "${BUILDPACK_DIR}/lib/cache.sh" source "${BUILDPACK_DIR}/lib/checks.sh" source "${BUILDPACK_DIR}/lib/hooks.sh" -source "${BUILDPACK_DIR}/lib/metadata.sh" source "${BUILDPACK_DIR}/lib/output.sh" source "${BUILDPACK_DIR}/lib/package_manager.sh" source "${BUILDPACK_DIR}/lib/pip.sh" @@ -33,13 +33,12 @@ source "${BUILDPACK_DIR}/lib/python_version.sh" source "${BUILDPACK_DIR}/lib/python.sh" source "${BUILDPACK_DIR}/lib/uv.sh" -compile_start_time=$(nowms) +compile_start_time=$(build_data::current_unix_time_ms) -# Initialise the buildpack metadata store. -# This is used to track state across builds (for cache invalidation and messaging when build -# configuration changes) and also so that `bin/report` can generate the build report. -meta_init "${CACHE_DIR}" "python" -meta_setup +# Initialise the build data store used to track state across builds (for cache invalidation +# and messaging when build configuration changes) and also so that `bin/report` can generate +# a build report that will be consumed by the build system for observability. +build_data::setup checks::ensure_supported_stack "${STACK:?Required env var STACK is not set}" checks::duplicate_python_buildpack "${BUILD_DIR}" @@ -105,15 +104,15 @@ hooks::run_hook "pre_compile" checks::existing_python_dir_present "${BUILD_DIR}" package_manager="$(package_manager::determine_package_manager "${BUILD_DIR}")" -meta_set "package_manager" "${package_manager}" +build_data::set_string "package_manager" "${package_manager}" cached_python_full_version="$(cache::cached_python_full_version "${CACHE_DIR}")" # We use the Bash 4.3+ `nameref` feature to pass back multiple values from this function # without having to hardcode globals. See: https://stackoverflow.com/a/38997681 python_version::read_requested_python_version "${BUILD_DIR}" "${package_manager}" "${cached_python_full_version}" requested_python_version python_version_origin -meta_set "python_version_requested" "${requested_python_version}" -meta_set "python_version_origin" "${python_version_origin}" +build_data::set_string "python_version_requested" "${requested_python_version}" +build_data::set_string "python_version_origin" "${python_version_origin}" case "${python_version_origin}" in default) @@ -129,13 +128,13 @@ esac python_full_version="$(python_version::resolve_python_version "${requested_python_version}" "${python_version_origin}")" python_major_version="${python_full_version%.*}" -meta_set "python_version" "${python_full_version}" -meta_set "python_version_major" "${python_major_version}" +build_data::set_string "python_version" "${python_full_version}" +build_data::set_string "python_version_major" "${python_major_version}" if [[ "${requested_python_version}" == "${python_full_version}" ]]; then - meta_set "python_version_pinned" "true" + build_data::set_raw "python_version_pinned" "true" else - meta_set "python_version_pinned" "false" + build_data::set_raw "python_version_pinned" "false" fi # We wait until after Python version resolution to show these warnings, to avoid causing confusion @@ -166,7 +165,7 @@ fi python::install "${BUILD_DIR}" "${STACK}" "${python_full_version}" "${python_major_version}" "${python_version_origin}" # Install the package manager and related tools. -package_manager_install_start_time=$(nowms) +package_manager_install_start_time=$(build_data::current_unix_time_ms) case "${package_manager}" in pip) pip::install_pip_setuptools_wheel "${python_home}" "${python_major_version}" @@ -184,10 +183,10 @@ case "${package_manager}" in utils::abort_internal_error "Unhandled package manager: ${package_manager}" ;; esac -meta_time "package_manager_install_duration" "${package_manager_install_start_time}" +build_data::set_duration "package_manager_install_duration" "${package_manager_install_start_time}" # Install app dependencies. -dependencies_install_start_time=$(nowms) +dependencies_install_start_time=$(build_data::current_unix_time_ms) case "${package_manager}" in pip) pip::install_dependencies @@ -205,18 +204,18 @@ case "${package_manager}" in utils::abort_internal_error "Unhandled package manager: ${package_manager}" ;; esac -meta_time "dependencies_install_duration" "${dependencies_install_start_time}" +build_data::set_duration "dependencies_install_duration" "${dependencies_install_start_time}" # Support for NLTK corpora. -nltk_downloader_start_time=$(nowms) +nltk_downloader_start_time=$(build_data::current_unix_time_ms) sub_env "${BUILDPACK_DIR}/bin/steps/nltk" -meta_time "nltk_downloader_duration" "${nltk_downloader_start_time}" +build_data::set_duration "nltk_downloader_duration" "${nltk_downloader_start_time}" # Django collectstatic support. # The buildpack automatically runs collectstatic for Django applications. -collectstatic_start_time=$(nowms) +collectstatic_start_time=$(build_data::current_unix_time_ms) sub_env "${BUILDPACK_DIR}/bin/steps/collectstatic" -meta_time "django_collectstatic_duration" "${collectstatic_start_time}" +build_data::set_duration "django_collectstatic_duration" "${collectstatic_start_time}" # Programmatically create .profile.d script for application runtime environment variables. @@ -291,4 +290,4 @@ hooks::run_hook "post_compile" cache::save "${BUILD_DIR}" "${CACHE_DIR}" "${STACK}" "${python_full_version}" "${package_manager}" -meta_time "total_duration" "${compile_start_time}" +build_data::set_duration "total_duration" "${compile_start_time}" diff --git a/bin/report b/bin/report index e93fb0c34..f73f96c68 100755 --- a/bin/report +++ b/bin/report @@ -1,16 +1,11 @@ #!/usr/bin/env bash # Usage: bin/report -# Produces a build report containing metadata about the build, that's consumed by the build system. -# This script is run for both successful and failing builds, so it should not assume the build ran -# to completion (e.g. Python or other tools may not even have been installed). +# Produces a build report emitted to stdout, containing metadata about the build, that's +# consumed by the build system. See the pip_spec.rb for an example build report payload. # -# Metadata must be emitted to stdout as valid YAML key-value pairs. Any fields that should always -# be typed as a string must be explicitly quoted. -# -# Example valid stdout: -# python_version: 'X.Y.Z' -# python_install_duration: 1.234 +# This script is run for both successful and failing builds, so it should not assume the +# build ran to completion (e.g. Python or other tools may not even have been installed). # # Failures in this script don't cause the overall build to fail (and won't appear in user # facing build logs) to avoid breaking builds unnecessarily / causing confusion. To debug @@ -18,8 +13,7 @@ # when developing run `make run` in this repo locally, which runs `bin/report` too. # # Note: The build system doesn't source the `export` script before running this script, -# so Python/the package manager won't be on PATH. If we ever need to run them during -# `bin/report` in the future, we'll need to manually source the export script again. +# so Python/the package manager won't be on PATH by default. set -euo pipefail shopt -s inherit_errexit @@ -29,77 +23,6 @@ CACHE_DIR="${2}" # The absolute path to the root of the buildpack. BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) -source "${BUILDPACK_DIR}/lib/metadata.sh" -meta_init "${CACHE_DIR}" "python" - -# Emit the key / value pair unquoted to stdout. Skips if the value is empty. -# Based on: https://github.com/heroku/heroku-buildpack-nodejs/blob/main/bin/report -kv_pair() { - local key="${1}" - local value="${2}" - if [[ -n "${value}" ]]; then - echo "${key}: ${value}" - fi -} - -# Emit the key / value pair to stdout, safely quoting the string. Skips if the value is empty. -# Based on: https://github.com/heroku/heroku-buildpack-nodejs/blob/main/bin/report -# (Though instead uses single quotes instead of double to avoid escaping issues.) -kv_pair_string() { - local key="${1}" - local value="${2}" - if [[ -n "${value}" ]]; then - # Escape any existing single quotes, which for YAML means replacing `'` with `''`. - value="${value//\'/\'\'}" - echo "${key}: '${value}'" - fi -} - -STRING_FIELDS=( - cache_status - django_collectstatic - failure_detail - failure_reason - nltk_downloader - package_manager - package_manager_multiple_found - pip_version - pipenv_version - poetry_version - python_version_major - python_version_origin - python_version_requested - python_version - setuptools_version - uv_version - wheel_version -) - -# We don't want to quote numeric or boolean fields. -ALL_OTHER_FIELDS=( - cache_restore_duration - cache_save_duration - dependencies_install_duration - django_collectstatic_duration - nltk_downloader_duration - package_manager_install_duration - post_compile_hook - post_compile_hook_duration - pre_compile_hook - pre_compile_hook_duration - python_install_duration - python_version_outdated - python_version_pinned - setup_py_only - total_duration -) - -for field in "${STRING_FIELDS[@]}"; do - # shellcheck disable=SC2312 # TODO: Invoke this command separately to avoid masking its return value. - kv_pair_string "${field}" "$(meta_get "${field}")" -done +source "${BUILDPACK_DIR}/lib/build_data.sh" -for field in "${ALL_OTHER_FIELDS[@]}"; do - # shellcheck disable=SC2312 # TODO: Invoke this command separately to avoid masking its return value. - kv_pair "${field}" "$(meta_get "${field}")" -done +build_data::print_bin_report_json diff --git a/bin/steps/collectstatic b/bin/steps/collectstatic index bcd0ef4aa..efea7ac98 100755 --- a/bin/steps/collectstatic +++ b/bin/steps/collectstatic @@ -16,12 +16,9 @@ set -euo pipefail BUILDPACK_DIR=$(cd "$(dirname "$(dirname "$(dirname "${BASH_SOURCE[0]}")")")" && pwd) source "${BUILDPACK_DIR}/bin/utils" +source "${BUILDPACK_DIR}/lib/build_data.sh" source "${BUILDPACK_DIR}/lib/output.sh" -# Required for `meta_set`. -source "${BUILDPACK_DIR}/lib/metadata.sh" -meta_init "${CACHE_DIR:?}" "python" - if [[ -f .heroku/collectstatic_disabled ]]; then output::step "Skipping Django collectstatic since the file '.heroku/collectstatic_disabled' exists." output::warning <<-'EOF' @@ -29,13 +26,13 @@ if [[ -f .heroku/collectstatic_disabled ]]; then Please remove the file and set the env var DISABLE_COLLECTSTATIC=1 instead. EOF - meta_set "django_collectstatic" "disabled-file" + build_data::set_string "django_collectstatic" "disabled-file" exit 0 fi if [[ "${DISABLE_COLLECTSTATIC:-0}" != "0" ]]; then output::step "Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set." - meta_set "django_collectstatic" "disabled-env-var" + build_data::set_string "django_collectstatic" "disabled-env-var" exit 0 fi @@ -51,11 +48,11 @@ MANAGE_FILE=${MANAGE_FILE:-fakepath} if [[ ! -f "${MANAGE_FILE}" ]]; then output::step "Skipping Django collectstatic since no manage.py file found." - meta_set "django_collectstatic" "skipped-no-manage-py" + build_data::set_string "django_collectstatic" "skipped-no-manage-py" exit 0 fi -meta_set "django_collectstatic" "enabled" +build_data::set_string "django_collectstatic" "enabled" output::step "$ python $MANAGE_FILE collectstatic --noinput" @@ -76,15 +73,15 @@ fi # Display a warning if collectstatic failed. if grep -q 'SyntaxError' "$COLLECTSTATIC_LOG"; then - meta_set "failure_reason" "collectstatic-syntax-error" + build_data::set_string "failure_reason" "collectstatic-syntax-error" elif grep -q 'ImproperlyConfigured' "$COLLECTSTATIC_LOG"; then - meta_set "failure_reason" "collectstatic-improper-configuration" + build_data::set_string "failure_reason" "collectstatic-improper-configuration" elif grep -q 'The CSS file' "$COLLECTSTATIC_LOG"; then - meta_set "failure_reason" "collectstatic-fancy-references" + build_data::set_string "failure_reason" "collectstatic-fancy-references" elif grep -q 'OSError' "$COLLECTSTATIC_LOG"; then - meta_set "failure_reason" "collectstatic-missing-file" + build_data::set_string "failure_reason" "collectstatic-missing-file" else - meta_set "failure_reason" "collectstatic-other" + build_data::set_string "failure_reason" "collectstatic-other" fi output::error <<-EOF diff --git a/bin/steps/nltk b/bin/steps/nltk index 202d71433..24ee8a1ff 100755 --- a/bin/steps/nltk +++ b/bin/steps/nltk @@ -5,12 +5,9 @@ set -euo pipefail BUILDPACK_DIR=$(cd "$(dirname "$(dirname "$(dirname "${BASH_SOURCE[0]}")")")" && pwd) source "${BUILDPACK_DIR}/bin/utils" +source "${BUILDPACK_DIR}/lib/build_data.sh" source "${BUILDPACK_DIR}/lib/output.sh" -# Required for `meta_set`. -source "${BUILDPACK_DIR}/lib/metadata.sh" -meta_init "${CACHE_DIR:?}" "python" - # These are required by `set_env`. PROFILE_PATH="${BUILD_DIR:?}/.profile.d/python.sh" EXPORT_PATH="${BUILDPACK_DIR}/export" @@ -23,7 +20,7 @@ if is_module_available 'nltk'; then nltk_packages_definition="${BUILD_DIR}/nltk.txt" if [[ -f "${nltk_packages_definition}" ]]; then - meta_set "nltk_downloader" "enabled" + build_data::set_string "nltk_downloader" "enabled" readarray -t nltk_packages <"${nltk_packages_definition}" output::step "Downloading NLTK packages: ${nltk_packages[*]}" @@ -39,13 +36,13 @@ if is_module_available 'nltk'; then See the log output above for more information. EOF - meta_set "failure_reason" "nltk-downloader" + build_data::set_string "failure_reason" "nltk-downloader" exit 1 fi set_env NLTK_DATA "${nltk_data_dir}" else - meta_set "nltk_downloader" "skipped-no-nltk-file" + build_data::set_string "nltk_downloader" "skipped-no-nltk-file" echo " 'nltk.txt' not found, not downloading any corpora" fi fi diff --git a/bin/warnings b/bin/warnings index 6992cc48e..6b65fe7cb 100755 --- a/bin/warnings +++ b/bin/warnings @@ -9,7 +9,7 @@ show-warnings() { For GDAL, GEOS and PROJ support, use the Geo buildpack alongside the Python buildpack: https://github.com/heroku/heroku-geo-buildpack EOF - meta_set "failure_detail" "Could not find gdal-config" + build_data::set_string "failure_detail" "Could not find gdal-config" elif grep -qi 'sqlite3.h: No such file or directory' "${WARNINGS_LOG}"; then output::error <<-'EOF' Error: Package installation failed since SQLite headers weren't found. @@ -38,7 +38,7 @@ show-warnings() { runs before the Python buildpack, not after): https://github.com/heroku/heroku-buildpack-apt EOF - meta_set "failure_detail" "sqlite3.h: No such file or directory" + build_data::set_string "failure_detail" "sqlite3.h: No such file or directory" elif grep -qi 'Please use pip<24.1 if you need to use this version' "${WARNINGS_LOG}"; then output::error <<-'EOF' Error: One of your dependencies contains broken metadata. @@ -51,6 +51,6 @@ show-warnings() { For more help, see: https://devcenter.heroku.com/changelog-items/3073 EOF - meta_set "failure_detail" "Please use pip<24.1 if you need to use this version" + build_data::set_string "failure_detail" "Please use pip<24.1 if you need to use this version" fi } diff --git a/lib/build_data.sh b/lib/build_data.sh new file mode 100644 index 000000000..ae50fddc1 --- /dev/null +++ b/lib/build_data.sh @@ -0,0 +1,157 @@ +#!/usr/bin/env bash + +# This is technically redundant, since all consumers of this lib will have enabled these, +# however, it helps Shellcheck realise the options under which these functions will run. +set -euo pipefail + +BUILD_DATA_FILE="${CACHE_DIR:?}/build-data/python.json" +PREVIOUS_BUILD_DATA_FILE="${CACHE_DIR:?}/build-data/python-prev.json" + +# Legacy `key=value` format file used by older Python buildpack versions. +LEGACY_BUILD_DATA_FILE="${CACHE_DIR:?}/build-data/python" + +# Initializes the build data store, preserving the file from the previous build if it exists. +# Call this at the start of `bin/compile` before using any other functions from this file. +# +# Usage: +# ``` +# build_data::setup +# ``` +function build_data::setup() { + if [[ -f "${BUILD_DATA_FILE}" ]]; then + # Rename the existing build data file rather than overwriting it, so we can lookup values + # from the previous build (such as when determining whether to invalidate the cache). + mv "${BUILD_DATA_FILE}" "${PREVIOUS_BUILD_DATA_FILE}" + else + mkdir -p "$(dirname "${BUILD_DATA_FILE}")" + fi + + echo "{}" >"${BUILD_DATA_FILE}" +} + +# Sets a string build data value. The value will be wrapped in double quotes and escaped for JSON. +# +# Usage: +# ``` +# build_data::set_string "python_version" "1.2.3" +# build_data::set_string "failure_reason" "install-dependencies::pip" +# ``` +function build_data::set_string() { + local key="${1}" + local value="${2}" + build_data::_set "${key}" "${value}" "true" +} + +# Sets a build data value for the elapsed time in seconds between the provided start time and the +# current time, represented as a float with milliseconds precision. +# +# Usage: +# ``` +# local dependencies_install_start_time=$(build_data::current_unix_time_ms) +# # ... some operation ... +# build_data::set_duration "dependencies_install_duration" "${dependencies_install_start_time}" +# ``` +function build_data::set_duration() { + local key="${1}" + local start_time="${2}" + local end_time duration + end_time="$(build_data::current_unix_time_ms)" + duration="$(awk -v start="${start_time}" -v end="${end_time}" 'BEGIN { printf "%.3f", (end - start)/1000 }')" + build_data::set_raw "${key}" "${duration}" +} + +# Sets a build data value as raw JSON data. The value parameter must be valid JSON value, that's also +# a supported Honeycomb data type (string, integer, float, or boolean only; no arrays or objects). +# For strings, use `build_data::set_string` instead since it will handle the escaping/quoting for you. +# And for durations, use `build_data::set_duration`. +# +# Usage: +# ``` +# build_data::set_raw "python_version_outdated" "true" +# build_data::set_raw "foo_size_mb" "42.5" +# ``` +function build_data::set_raw() { + local key="${1}" + local value="${2}" + build_data::_set "${key}" "${value}" "false" +} + +# Internal helper to write a key/value pair to the build data store. The buildpack shouldn't call this directly. +# Takes a key, value, and a boolean flag indicating whether the value needs to be quoted. +# +# Usage: +# ``` +# build_data::_set "foo_string" "quote me" "true" +# build_data::_set "bar_number" "99" "false" +# ``` +function build_data::_set() { + local key="${1}" + # Truncate the value to an arbitrary 200 characters since it will sometimes contain user-provided + # inputs which may be unbounded in size. Ideally individual call sites will perform more aggressive + # truncation themselves based on the expected value size, however this is here as a fallback. + # (Honeycomb supports string fields up to 64KB in size, however, it's not worth filling up the + # build data store or bloating the payload passed back to Vacuole/submitted to Honeycomb given the + # extra content in those cases is not normally useful.) + local value="${2:0:200}" + local needs_quoting="${3}" + + if [[ "${needs_quoting}" == "true" ]]; then + # Values passed using `--arg` are treated as strings, and so have double quotes added and any JSON + # special characters (such as newlines, carriage returns, double quotes, backslashes) are escaped. + local jq_args=(--arg value "${value}") + else + # Values passed using `--argjson` are treated as raw JSON values, and so aren't escaped or quoted. + local jq_args=(--argjson value "${value}") + fi + + local new_data_file_contents + new_data_file_contents="$(jq --arg key "${key}" "${jq_args[@]}" '. + { ($key): ($value) }' "${BUILD_DATA_FILE}")" + echo "${new_data_file_contents}" >"${BUILD_DATA_FILE}" +} + +# Retrieve the value of an entry in the build data store from the previous successful build. +# Returns the empty string if the key wasn't found in the store. +# +# Usage: +# ``` +# build_data::get_previous "python_version" +# ``` +function build_data::get_previous() { + local key="${1}" + + # Older versions of this buildpack used a `key=value` format file instead of JSON, + # so we need to support this file format/location too, so older caches can be read. + # We check for this file first, so that we correctly handle the case where an app + # downgraded and then re-upgraded buildpack version, so has both files in the cache. + if [[ -f "${LEGACY_BUILD_DATA_FILE}" ]]; then + # The legacy file contains one entry per line, of form `key=value`. Entries were written in an + # append-only manner so there could be duplicate entries for each key, so we return only the + # last matching entry in the file. The empty string is returned if the key wasn't found. + tac "${LEGACY_BUILD_DATA_FILE}" | { grep --perl-regexp --only-matching --max-count=1 "^${key}=\K.*$" || true; } + elif [[ -f "${PREVIOUS_BUILD_DATA_FILE}" ]]; then + # The `// empty` ensures we return the empty string rather than `null` if the key doesn't exist. + jq --raw-output ".${key} // empty" "${PREVIOUS_BUILD_DATA_FILE}" + fi +} + +# Returns the current time in milliseconds since the UNIX Epoch. +# +# Usage: +# ``` +# local dependencies_install_start_time=$(build_data::current_unix_time_ms) +# # ... some operation ... +# build_data::set_duration "dependencies_install_duration" "${dependencies_install_start_time}" +# ``` +function build_data::current_unix_time_ms() { + date +%s%3N +} + +# Prints the contents of the build data store in sorted JSON format. +# +# Usage: +# ``` +# build_data::print_bin_report_json +# ``` +function build_data::print_bin_report_json() { + jq --sort-keys '.' "${BUILD_DATA_FILE}" +} diff --git a/lib/cache.sh b/lib/cache.sh index 5d92af962..c657f4771 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -34,11 +34,11 @@ function cache::restore() { local package_manager="${6}" local cache_restore_start_time - cache_restore_start_time=$(nowms) + cache_restore_start_time=$(build_data::current_unix_time_ms) if [[ ! -d "${cache_dir}/.heroku/python" ]]; then - meta_set "cache_status" "empty" - meta_time "cache_restore_duration" "${cache_restore_start_time}" + build_data::set_string "cache_status" "empty" + build_data::set_duration "cache_restore_duration" "${cache_restore_start_time}" return 0 fi @@ -55,9 +55,9 @@ function cache::restore() { fi local cached_package_manager - cached_package_manager="$(meta_prev_get "package_manager")" + cached_package_manager="$(build_data::get_previous "package_manager")" if [[ -z "${cached_package_manager}" ]]; then - # The metadata store only exists in caches created by v252+ of the buildpack (released 2024-06-17). + # The build data store only exists in caches created by v252+ of the buildpack (released 2024-06-17). cache_invalidation_reasons+=("The buildpack cache format has changed") elif [[ "${cached_package_manager}" != "${package_manager}" ]]; then cache_invalidation_reasons+=("The package manager has changed from ${cached_package_manager:-"unknown"} to ${package_manager}") @@ -65,7 +65,7 @@ function cache::restore() { case "${package_manager}" in pip) local cached_pip_version - cached_pip_version="$(meta_prev_get "pip_version")" + cached_pip_version="$(build_data::get_previous "pip_version")" if [[ "${cached_pip_version}" != "${PIP_VERSION:?}" ]]; then cache_invalidation_reasons+=("The pip version has changed from ${cached_pip_version:-"unknown"} to ${PIP_VERSION}") fi @@ -82,7 +82,7 @@ function cache::restore() { ;; pipenv) local cached_pipenv_version - cached_pipenv_version="$(meta_prev_get "pipenv_version")" + cached_pipenv_version="$(build_data::get_previous "pipenv_version")" if [[ "${cached_pipenv_version}" != "${PIPENV_VERSION:?}" ]]; then cache_invalidation_reasons+=("The Pipenv version has changed from ${cached_pipenv_version:-"unknown"} to ${PIPENV_VERSION}") fi @@ -98,14 +98,14 @@ function cache::restore() { ;; poetry) local cached_poetry_version - cached_poetry_version="$(meta_prev_get "poetry_version")" + cached_poetry_version="$(build_data::get_previous "poetry_version")" if [[ "${cached_poetry_version}" != "${POETRY_VERSION:?}" ]]; then cache_invalidation_reasons+=("The Poetry version has changed from ${cached_poetry_version:-"unknown"} to ${POETRY_VERSION}") fi ;; uv) local cached_uv_version - cached_uv_version="$(meta_prev_get "uv_version")" + cached_uv_version="$(build_data::get_previous "uv_version")" if [[ "${cached_uv_version}" != "${UV_VERSION:?}" ]]; then cache_invalidation_reasons+=("The uv version has changed from ${cached_uv_version:-"unknown"} to ${UV_VERSION}") fi @@ -135,7 +135,7 @@ function cache::restore() { "${cache_dir}/.heroku/python-version" \ "${cache_dir}/.heroku/requirements.txt" - meta_set "cache_status" "discarded" + build_data::set_string "cache_status" "discarded" else output::step "Restoring cache" mkdir -p "${build_dir}/.heroku" @@ -143,16 +143,18 @@ function cache::restore() { # build directory are on the same filesystem mount. The Python directory is guaranteed # to not already exist thanks to the earlier `checks::existing_python_dir_present()`. mv "${cache_dir}/.heroku/python" "${build_dir}/.heroku/" - meta_set "cache_status" "reused" + build_data::set_string "cache_status" "reused" fi # Remove any legacy cache contents written by older buildpack versions. rm -rf \ + "${cache_dir}/build-data/python" \ + "${cache_dir}/build-data/python-prev" \ "${cache_dir}/.heroku/python-sqlite3-version" \ "${cache_dir}/.heroku/src" \ "${cache_dir}/.heroku/vendor" - meta_time "cache_restore_duration" "${cache_restore_start_time}" + build_data::set_duration "cache_restore_duration" "${cache_restore_start_time}" } # Copies Python and dependencies from the build directory to the cache, for use by subsequent builds. @@ -164,7 +166,7 @@ function cache::save() { local package_manager="${5}" local cache_save_start_time - cache_save_start_time=$(nowms) + cache_save_start_time=$(build_data::current_unix_time_ms) output::step "Saving cache" @@ -179,8 +181,8 @@ function cache::save() { cp --recursive "${build_dir}/.heroku/python" "${cache_dir}/.heroku/" # Metadata used by subsequent builds to determine whether the cache can be reused. - # These are written/consumed via separate files and not the metadata store for compatibility - # with buildpack versions prior to the metadata store existing (which was only added in v252). + # These are written/consumed via separate files and not the build data store for compatibility + # with buildpack versions prior to the build data store existing (which was only added in v252). echo "${stack}" >"${cache_dir}/.heroku/python-stack" # For historical reasons the Python version was always stored with a `python-` prefix. # We continue to use that format so that the file can be read by older buildpack versions. @@ -197,5 +199,5 @@ function cache::save() { sha256sum Pipfile.lock >"${cache_dir}/.heroku/python/Pipfile.lock.sha256" fi - meta_time "cache_save_duration" "${cache_save_start_time}" + build_data::set_duration "cache_save_duration" "${cache_save_start_time}" } diff --git a/lib/checks.sh b/lib/checks.sh index ca24e027a..d2757fdbb 100644 --- a/lib/checks.sh +++ b/lib/checks.sh @@ -19,8 +19,8 @@ function checks::ensure_supported_stack() { Upgrade to a newer stack to continue using this buildpack. EOF - meta_set "failure_reason" "stack::eol" - meta_set "failure_detail" "${stack}" + build_data::set_string "failure_reason" "stack::eol" + build_data::set_string "failure_detail" "${stack}" exit 1 ;; *) @@ -34,8 +34,8 @@ function checks::ensure_supported_stack() { https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references EOF - meta_set "failure_reason" "stack::unknown" - meta_set "failure_detail" "${stack}" + build_data::set_string "failure_reason" "stack::unknown" + build_data::set_string "failure_detail" "${stack}" exit 1 ;; esac @@ -67,7 +67,7 @@ function checks::duplicate_python_buildpack() { Note: This error replaces the deprecation warning which was displayed in build logs starting 13th December 2024. EOF - meta_set "failure_reason" "checks::duplicate-python-buildpack" + build_data::set_string "failure_reason" "checks::duplicate-python-buildpack" exit 1 fi } @@ -98,7 +98,7 @@ function checks::existing_python_dir_present() { Note: This error replaces the deprecation warning which was displayed in build logs starting 13th December 2024. EOF - meta_set "failure_reason" "checks::existing-python-dir" + build_data::set_string "failure_reason" "checks::existing-python-dir" exit 1 fi } diff --git a/lib/hooks.sh b/lib/hooks.sh index 19eb89839..1018081f4 100644 --- a/lib/hooks.sh +++ b/lib/hooks.sh @@ -12,9 +12,9 @@ function hooks::run_hook() { if [[ -f "${script_path}" ]]; then local hook_start_time - hook_start_time=$(nowms) + hook_start_time=$(build_data::current_unix_time_ms) output::step "Running ${script_path} hook" - meta_set "${hook_name}_hook" "true" + build_data::set_raw "${hook_name}_hook" "true" chmod +x "${script_path}" # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. @@ -30,12 +30,12 @@ function hooks::run_hook() { Fix any errors output by your script above, or remove/rename the script to prevent it from being run during the build. EOF - meta_set "failure_reason" "hooks::${hook_name}" + build_data::set_string "failure_reason" "hooks::${hook_name}" exit 1 fi - meta_time "${hook_name}_hook_duration" "${hook_start_time}" + build_data::set_duration "${hook_name}_hook_duration" "${hook_start_time}" else - meta_set "${hook_name}_hook" "false" + build_data::set_raw "${hook_name}_hook" "false" fi } diff --git a/lib/kvstore.sh b/lib/kvstore.sh deleted file mode 100644 index a6377f77f..000000000 --- a/lib/kvstore.sh +++ /dev/null @@ -1,101 +0,0 @@ -#!/usr/bin/env bash - -# This is technically redundant, since all consumers of this lib will have enabled these, -# however, it helps Shellcheck realise the options under which these functions will run. -set -euo pipefail - -# TODO: Switch this file to using namespaced functions like `kvstore::`. - -# Taken from: https://github.com/heroku/heroku-buildpack-nodejs/blob/main/lib/kvstore.sh - -kv_create() { - local f="${1}" - mkdir -p "$(dirname "${f}")" - touch "${f}" -} - -kv_clear() { - local f="${1}" - echo "" >"${f}" -} - -kv_set() { - # TODO: Stop ignoring an incorrect number of passed arguments. - if [[ $# -eq 3 ]]; then - local f="${1}" - local key="${2}" - - # Truncate the value to an arbitrary 100 characters since it will sometimes contain user-provided - # inputs which may be unbounded in size. Ideally individual call sites will perform more aggressive - # truncation themselves based on the expected value size, however this is here as a fallback. - # (Honeycomb supports string fields up to 64KB in size, however, it's not worth filling up the - # metadata store or bloating the payload passed back to Vacuole/submitted to Honeycomb given the - # extra content in those cases is not normally useful.) - local value="${3:0:100}" - # Replace newlines and carriage returns since the data store file format requires that keys don't - # span multiple lines. - value="${value//$'\n'/\\n}" - value="${value//$'\r'/\\r}" - - if [[ -f "${f}" ]]; then - echo "${key}=${value}" >>"${f}" - fi - fi -} - -# Returns a key from the key-value store file, or else the empty string -# if the file doesn't exist or the key wasn't found. -kv_get() { - # TODO: Stop ignoring an incorrect number of passed arguments. - if [[ $# -eq 2 ]]; then - local f="${1}" - if [[ -f "${f}" ]]; then - # shellcheck disable=SC2310 # This function is invoked in an || condition so set -e will be disabled. - grep "^${2}=" "${f}" | sed -e "s/^${2}=//" | tail -n 1 || true - fi - fi -} - -# get the value, but wrap it in quotes if it contains a space -kv_get_escaped() { - local value - value=$(kv_get "${1}" "${2}") - if [[ "${value}" =~ [[:space:]]+ ]]; then - echo "\"${value}\"" - else - echo "${value}" - fi -} - -kv_keys() { - local f="${1}" - local keys=() - - if [[ -f "${f}" ]]; then - # Iterate over each line, splitting on the '=' character - # - # The || [[ -n "${key}" ]] statement addresses an issue with reading the last line - # of a file when there is no newline at the end. This will not happen if the file - # is created with this module, but can happen if it is written by hand. - # See: https://stackoverflow.com/questions/12916352/shell-script-read-missing-last-line - while IFS="=" read -r key value || [[ -n "${key}" ]]; do - # if there are any empty lines in the store, skip them - if [[ -n "${key}" ]]; then - keys+=("${key}") - fi - done <"${f}" - - echo "${keys[@]}" | tr ' ' '\n' | sort -u - fi -} - -kv_list() { - local f="${1}" - - kv_keys "${f}" | tr ' ' '\n' | while read -r key; do - if [[ -n "${key}" ]]; then - # shellcheck disable=SC2312 # TODO: Invoke this command separately to avoid masking its return value. - echo "${key}=$(kv_get_escaped "${f}" "${key}")" - fi - done -} diff --git a/lib/metadata.sh b/lib/metadata.sh deleted file mode 100644 index 99f1919a3..000000000 --- a/lib/metadata.sh +++ /dev/null @@ -1,78 +0,0 @@ -#!/usr/bin/env bash - -# This is technically redundant, since all consumers of this lib will have enabled these, -# however, it helps Shellcheck realise the options under which these functions will run. -set -euo pipefail - -# TODO: Switch this file to using namespaced functions like `metadata::`. - -# Based on: https://github.com/heroku/heroku-buildpack-nodejs/blob/main/lib/metadata.sh - -source "${BUILDPACK_DIR:?}/lib/kvstore.sh" - -# Variables shared by this whole module -BUILD_DATA_FILE="" -PREVIOUS_BUILD_DATA_FILE="" - -# Must be called before you can use any other methods -meta_init() { - local cache_dir="${1}" - local buildpack_name="${2}" - BUILD_DATA_FILE="${cache_dir}/build-data/${buildpack_name}" - PREVIOUS_BUILD_DATA_FILE="${cache_dir}/build-data/${buildpack_name}-prev" -} - -# Moves the data from the last build into the correct place, and clears the store -# This should be called after meta_init in bin/compile -meta_setup() { - # if the file already exists because it's from the last build, save it - if [[ -f "${BUILD_DATA_FILE}" ]]; then - cp "${BUILD_DATA_FILE}" "${PREVIOUS_BUILD_DATA_FILE}" - fi - - kv_create "${BUILD_DATA_FILE}" - kv_clear "${BUILD_DATA_FILE}" -} - -# Force removal of exiting data file state. This is mostly useful during testing and not -# expected to be used during buildpack execution. -meta_force_clear() { - [[ -f "${BUILD_DATA_FILE}" ]] && rm "${BUILD_DATA_FILE}" - [[ -f "${PREVIOUS_BUILD_DATA_FILE}" ]] && rm "${PREVIOUS_BUILD_DATA_FILE}" -} - -meta_get() { - kv_get "${BUILD_DATA_FILE}" "${1}" -} - -meta_set() { - kv_set "${BUILD_DATA_FILE}" "${1}" "${2}" -} - -# Similar to mtime from buildpack-stdlib -meta_time() { - local key="${1}" - local start="${2}" - local end="${3:-$(nowms)}" - local time - time="$(echo "${start}" "${end}" | awk '{ printf "%.3f", ($2 - $1)/1000 }')" - kv_set "${BUILD_DATA_FILE}" "${key}" "${time}" -} - -# Retrieve a value from a previous build if it exists -# This is useful to give the user context about what changed if the -# build has failed. Ex: -# - changed stacks -# - deployed with a new major version of Node -# - etc -meta_prev_get() { - kv_get "${PREVIOUS_BUILD_DATA_FILE}" "${1}" -} - -log_meta_data() { - # print all values on one line in logfmt format - # https://brandur.org/logfmt - # the echo call ensures that all values are printed on a single line - # shellcheck disable=SC2005,SC2046,SC2312 - echo $(kv_list "${BUILD_DATA_FILE}") -} diff --git a/lib/package_manager.sh b/lib/package_manager.sh index 9c2fd9bfb..46cef2d7e 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -33,7 +33,7 @@ function package_manager::determine_package_manager() { Note: This error replaces the warning which was displayed in build logs starting 12th November 2024. EOF - meta_set "failure_reason" "package-manager::pipenv-missing-lockfile" + build_data::set_string "failure_reason" "package-manager::pipenv-missing-lockfile" exit 1 fi @@ -62,9 +62,9 @@ function package_manager::determine_package_manager() { if ((${#package_managers_found[@]} == 0)) && [[ -f "${build_dir}/setup.py" ]]; then package_managers_found+=(pip) package_managers_found_display_text+=("setup.py (pip)") - meta_set "setup_py_only" "true" + build_data::set_raw "setup_py_only" "true" else - meta_set "setup_py_only" "false" + build_data::set_raw "setup_py_only" "false" fi local num_package_managers_found=${#package_managers_found[@]} @@ -106,7 +106,7 @@ function package_manager::determine_package_manager() { https://devcenter.heroku.com/articles/getting-started-with-python https://devcenter.heroku.com/articles/python-support EOF - meta_set "failure_reason" "package-manager::none-found" + build_data::set_string "failure_reason" "package-manager::none-found" exit 1 ;; *) @@ -156,7 +156,7 @@ function package_manager::determine_package_manager() { EOF fi - meta_set "package_manager_multiple_found" "$( + build_data::set_string "package_manager_multiple_found" "$( IFS=, echo "${package_managers_found[*]}" )" diff --git a/lib/pip.sh b/lib/pip.sh index 4a7b3cde8..f5330de9e 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -17,7 +17,7 @@ function pip::install_pip_setuptools_wheel() { local bundled_pip_module_path bundled_pip_module_path="$(utils::bundled_pip_module_path "${python_home}" "${python_major_version}")" - meta_set "pip_version" "${PIP_VERSION}" + build_data::set_string "pip_version" "${PIP_VERSION}" local packages_to_install=( "pip==${PIP_VERSION}" @@ -32,8 +32,8 @@ function pip::install_pip_setuptools_wheel() { # - Most of the Python ecosystem has stopped installing them for Python 3.12+ already. # See the Python CNB's removal for more details: https://github.com/heroku/buildpacks-python/pull/243 if [[ "${python_major_version}" == +(3.9|3.10|3.11|3.12) ]]; then - meta_set "setuptools_version" "${SETUPTOOLS_VERSION}" - meta_set "wheel_version" "${WHEEL_VERSION}" + build_data::set_string "setuptools_version" "${SETUPTOOLS_VERSION}" + build_data::set_string "wheel_version" "${WHEEL_VERSION}" packages_to_install+=( "setuptools==${SETUPTOOLS_VERSION}" "wheel==${WHEEL_VERSION}" @@ -68,7 +68,7 @@ function pip::install_pip_setuptools_wheel() { If that doesn't help, check the status of PyPI here: https://status.python.org EOF - meta_set "failure_reason" "install-package-manager::pip" + build_data::set_string "failure_reason" "install-package-manager::pip" exit 1 fi } @@ -134,7 +134,7 @@ function pip::install_dependencies() { See the log output above for more information. EOF - meta_set "failure_reason" "install-dependencies::pip" + build_data::set_string "failure_reason" "install-dependencies::pip" exit 1 fi } diff --git a/lib/pipenv.sh b/lib/pipenv.sh index 88d272e47..afdf281e9 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -25,7 +25,7 @@ function pipenv::install_pipenv() { local pipenv_bin_dir="${pipenv_root}/bin" local pipenv_venv_dir="${pipenv_root}/venv" - meta_set "pipenv_version" "${PIPENV_VERSION}" + build_data::set_string "pipenv_version" "${PIPENV_VERSION}" # The earlier buildpack cache invalidation step will have already handled the case where the # Pipenv version has changed, so here we only need to check that a Pipenv install exists. @@ -50,7 +50,7 @@ function pipenv::install_pipenv() { See the log output above for more information. EOF - meta_set "failure_reason" "create-venv::pipenv" + build_data::set_string "failure_reason" "create-venv::pipenv" exit 1 fi @@ -82,7 +82,7 @@ function pipenv::install_pipenv() { If that doesn't help, check the status of PyPI here: https://status.python.org EOF - meta_set "failure_reason" "install-package-manager::pipenv" + build_data::set_string "failure_reason" "install-package-manager::pipenv" exit 1 fi @@ -170,7 +170,7 @@ function pipenv::install_dependencies() { See the log output above for more information. EOF - meta_set "failure_reason" "install-dependencies::pipenv" + build_data::set_string "failure_reason" "install-dependencies::pipenv" exit 1 fi } diff --git a/lib/poetry.sh b/lib/poetry.sh index a50294422..92ad18687 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -21,7 +21,7 @@ function poetry::install_poetry() { local poetry_bin_dir="${poetry_root}/bin" local poetry_venv_dir="${poetry_root}/venv" - meta_set "poetry_version" "${POETRY_VERSION}" + build_data::set_string "poetry_version" "${POETRY_VERSION}" # The earlier buildpack cache invalidation step will have already handled the case where the # Poetry version has changed, so here we only need to check that a valid Poetry install exists. @@ -56,7 +56,7 @@ function poetry::install_poetry() { See the log output above for more information. EOF - meta_set "failure_reason" "create-venv::poetry" + build_data::set_string "failure_reason" "create-venv::poetry" exit 1 fi @@ -88,7 +88,7 @@ function poetry::install_poetry() { If that doesn't help, check the status of PyPI here: https://status.python.org EOF - meta_set "failure_reason" "install-package-manager::poetry" + build_data::set_string "failure_reason" "install-package-manager::poetry" exit 1 fi @@ -160,7 +160,7 @@ function poetry::install_dependencies() { See the log output above for more information. EOF - meta_set "failure_reason" "install-dependencies::poetry" + build_data::set_string "failure_reason" "install-dependencies::poetry" exit 1 fi } diff --git a/lib/python.sh b/lib/python.sh index 64497f555..f194734c6 100644 --- a/lib/python.sh +++ b/lib/python.sh @@ -14,7 +14,7 @@ function python::install() { local python_version_origin="${5}" local install_python_start_time - install_python_start_time=$(nowms) + install_python_start_time=$(build_data::current_unix_time_ms) local install_dir="${build_dir}/.heroku/python" if [[ -f "${install_dir}/bin/python" ]]; then @@ -97,8 +97,8 @@ function python::install() { This will allow your app to receive the latest available Python patch version automatically, and prevent this type of error. EOF - meta_set "failure_reason" "python-version::unknown-patch" - meta_set "failure_detail" "${python_full_version}" + build_data::set_string "failure_reason" "python-version::unknown-patch" + build_data::set_string "failure_detail" "${python_full_version}" else output::error <<-EOF Error: Unable to download/install Python. @@ -117,14 +117,14 @@ function python::install() { Then try building again to see if the error resolves itself. EOF - meta_set "failure_reason" "install-python" + build_data::set_string "failure_reason" "install-python" # e.g.: 'curl: (6) Could not resolve host: heroku-buildpack-python.s3.us-east-1.amazonaws.com' - meta_set "failure_detail" "$(head --lines=1 "${error_log}" || true)" + build_data::set_string "failure_detail" "$(head --lines=1 "${error_log}" || true)" fi exit 1 fi fi - meta_time "python_install_duration" "${install_python_start_time}" + build_data::set_duration "python_install_duration" "${install_python_start_time}" } diff --git a/lib/python_version.sh b/lib/python_version.sh index 5ec9eb05e..ad95d2003 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -117,8 +117,8 @@ function python_version::parse_runtime_txt() { ${instructions} EOF - meta_set "failure_reason" "runtime-txt::invalid-version" - meta_set "failure_detail" "${contents:0:50}" + build_data::set_string "failure_reason" "runtime-txt::invalid-version" + build_data::set_string "failure_detail" "${contents:0:50}" exit 1 fi } @@ -168,8 +168,8 @@ function python_version::parse_python_version_file() { version and so stop your app from receiving security updates each time it builds. EOF - meta_set "failure_reason" "python-version-file::invalid-version" - meta_set "failure_detail" "${line:0:50}" + build_data::set_string "failure_reason" "python-version-file::invalid-version" + build_data::set_string "failure_detail" "${line:0:50}" exit 1 fi ;; @@ -189,8 +189,8 @@ function python_version::parse_python_version_file() { If the file already contains a version, check the line doesn't begin with a '#', otherwise it will be treated as a comment. EOF - meta_set "failure_reason" "python-version-file::no-version" - meta_set "failure_detail" "${contents:0:50}" + build_data::set_string "failure_reason" "python-version-file::no-version" + build_data::set_string "failure_detail" "${contents:0:50}" exit 1 ;; *) @@ -214,8 +214,8 @@ function python_version::parse_python_version_file() { If you have added comments to the file, make sure that those lines begin with a '#', so that they are ignored. EOF - meta_set "failure_reason" "python-version-file::multiple-versions" - meta_set "failure_detail" "$( + build_data::set_string "failure_reason" "python-version-file::multiple-versions" + build_data::set_string "failure_detail" "$( IFS=, echo "${first_five_version_lines[*]}" )" @@ -245,8 +245,8 @@ function python_version::read_pipenv_python_version() { Run 'pipenv lock' to regenerate/fix the lockfile. EOF - meta_set "failure_reason" "pipfile-lock::invalid-json" - meta_set "failure_detail" "${jq_error_message:0:100}" + build_data::set_string "failure_reason" "pipfile-lock::invalid-json" + build_data::set_string "failure_detail" "${jq_error_message:0:100}" exit 1 fi @@ -287,8 +287,8 @@ function python_version::read_pipenv_python_version() { For more information, see: https://pipenv.pypa.io/en/stable/specifiers.html#specifying-versions-of-python EOF - meta_set "failure_reason" "pipfile-lock::invalid-version" - meta_set "failure_detail" "${version:0:50}" + build_data::set_string "failure_reason" "pipfile-lock::invalid-version" + build_data::set_string "failure_detail" "${version:0:50}" exit 1 fi } @@ -347,8 +347,8 @@ function python_version::resolve_python_version() { since it contains many performance and usability improvements. EOF fi - meta_set "failure_reason" "python-version::eol" - meta_set "failure_detail" "${major}.${minor}" + build_data::set_string "failure_reason" "python-version::eol" + build_data::set_string "failure_detail" "${major}.${minor}" exit 1 fi @@ -395,8 +395,8 @@ function python_version::resolve_python_version() { by changing the version in your ${python_version_origin} file. EOF fi - meta_set "failure_reason" "python-version::unknown-major" - meta_set "failure_detail" "${major}.${minor}" + build_data::set_string "failure_reason" "python-version::unknown-major" + build_data::set_string "failure_detail" "${major}.${minor}" exit 1 fi @@ -447,7 +447,7 @@ function python_version::warn_or_error_if_python_version_file_missing() { pin your app to an exact Python version and so stop your app from receiving security updates each time it builds. EOF - meta_set "failure_reason" "python-version-file::not-found" + build_data::set_string "failure_reason" "python-version-file::not-found" exit 1 fi @@ -501,7 +501,7 @@ function python_version::warn_or_error_if_python_version_file_missing() { pin your app to an exact Python version and so stop your app from receiving security updates each time it builds. EOF - meta_set "failure_reason" "runtime-txt::not-supported" + build_data::set_string "failure_reason" "runtime-txt::not-supported" exit 1 fi @@ -615,8 +615,8 @@ function python_version::warn_if_patch_update_available() { This will allow your app to receive the latest available Python patch version automatically and prevent this warning. EOF - meta_set "python_version_outdated" "true" + build_data::set_raw "python_version_outdated" "true" else - meta_set "python_version_outdated" "false" + build_data::set_raw "python_version_outdated" "false" fi } diff --git a/lib/utils.sh b/lib/utils.sh index 2345961e4..c81af858f 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -51,7 +51,7 @@ function utils::bundled_pip_module_path() { $(find "${bundled_wheels_dir}/" 2>&1 || find "${python_home}/" -type d 2>&1 || true) EOF - meta_set "failure_reason" "bundled-pip-not-found" + build_data::set_string "failure_reason" "bundled-pip-not-found" exit 1 fi } @@ -62,7 +62,7 @@ function utils::abort_internal_error() { output::error <<-EOF Internal error: ${message}. EOF - meta_set "failure_reason" "internal-error" - meta_set "failure_detail" "${message}" + build_data::set_string "failure_reason" "internal-error" + build_data::set_string "failure_detail" "${message}" exit 1 } diff --git a/lib/uv.sh b/lib/uv.sh index c8e48a72a..8dcd71cfd 100644 --- a/lib/uv.sh +++ b/lib/uv.sh @@ -14,7 +14,7 @@ function uv::install_uv() { # We store uv in the build cache, since we only need it during the build. local uv_dir="${cache_dir}/.heroku/python-uv" - meta_set "uv_version" "${UV_VERSION}" + build_data::set_string "uv_version" "${UV_VERSION}" # The earlier buildpack cache invalidation step will have already handled the case where # the uv version has changed, so here we only need to check whether the uv binary exists. @@ -68,7 +68,7 @@ function uv::install_uv() { If that doesn't help, check the status of GitHub here: https://www.githubstatus.com EOF - meta_set "failure_reason" "install-package-manager::uv" + build_data::set_string "failure_reason" "install-package-manager::uv" exit 1 fi fi @@ -159,7 +159,7 @@ function uv::install_dependencies() { See the log output above for more information. EOF - meta_set "failure_reason" "install-dependencies::uv" + build_data::set_string "failure_reason" "install-dependencies::uv" exit 1 fi } diff --git a/spec/fixtures/pip_basic/bin/compile b/spec/fixtures/pip_basic/bin/compile index 942fe8171..0eecc561b 100755 --- a/spec/fixtures/pip_basic/bin/compile +++ b/spec/fixtures/pip_basic/bin/compile @@ -24,4 +24,4 @@ echo python -c 'import typing_extensions; print(typing_extensions)' echo -sort "${CACHE_DIR}/build-data/python" +jq --sort-keys '.' "${CACHE_DIR}/build-data/python.json" diff --git a/spec/fixtures/pipenv_basic/bin/compile b/spec/fixtures/pipenv_basic/bin/compile index de32b7386..a9c2a9af7 100755 --- a/spec/fixtures/pipenv_basic/bin/compile +++ b/spec/fixtures/pipenv_basic/bin/compile @@ -25,4 +25,4 @@ echo python -c 'import typing_extensions; print(typing_extensions)' echo -sort "${CACHE_DIR}/build-data/python" +jq --sort-keys '.' "${CACHE_DIR}/build-data/python.json" diff --git a/spec/fixtures/poetry_basic/bin/compile b/spec/fixtures/poetry_basic/bin/compile index 77b8afa7a..af757a8b4 100755 --- a/spec/fixtures/poetry_basic/bin/compile +++ b/spec/fixtures/poetry_basic/bin/compile @@ -25,4 +25,4 @@ echo python -c 'import typing_extensions; print(typing_extensions)' echo -sort "${CACHE_DIR}/build-data/python" +jq --sort-keys '.' "${CACHE_DIR}/build-data/python.json" diff --git a/spec/fixtures/uv_basic/bin/compile b/spec/fixtures/uv_basic/bin/compile index 7668d373b..823440ee2 100755 --- a/spec/fixtures/uv_basic/bin/compile +++ b/spec/fixtures/uv_basic/bin/compile @@ -23,4 +23,4 @@ echo python -c 'import typing_extensions; print(typing_extensions)' echo -sort "${CACHE_DIR}/build-data/python" +jq --sort-keys '.' "${CACHE_DIR}/build-data/python.json" diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 39196d2c9..d741830c2 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -45,27 +45,28 @@ remote: remote: remote: - remote: - remote: cache_restore_duration=[0-9.]+ - remote: cache_save_duration=[0-9.]+ - remote: cache_status=empty - remote: dependencies_install_duration=[0-9.]+ - remote: django_collectstatic_duration=[0-9.]+ - remote: nltk_downloader_duration=[0-9.]+ - remote: package_manager_install_duration=[0-9.]+ - remote: package_manager=pip - remote: pip_version=#{PIP_VERSION} - remote: post_compile_hook=false - remote: pre_compile_hook=false - remote: python_install_duration=[0-9.]+ - remote: python_version=#{DEFAULT_PYTHON_FULL_VERSION} - remote: python_version_major=3.13 - remote: python_version_origin=.python-version - remote: python_version_outdated=false - remote: python_version_pinned=false - remote: python_version_requested=3.13 - remote: setup_py_only=false - remote: total_duration=[0-9.]+ + remote: { + remote: "cache_restore_duration": [0-9.]+, + remote: "cache_save_duration": [0-9.]+, + remote: "cache_status": "empty", + remote: "dependencies_install_duration": [0-9.]+, + remote: "django_collectstatic_duration": [0-9.]+, + remote: "nltk_downloader_duration": [0-9.]+, + remote: "package_manager": "pip", + remote: "package_manager_install_duration": [0-9.]+, + remote: "pip_version": "#{PIP_VERSION}", + remote: "post_compile_hook": false, + remote: "pre_compile_hook": false, + remote: "python_install_duration": [0-9.]+, + remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", + remote: "python_version_major": "3.13", + remote: "python_version_origin": ".python-version", + remote: "python_version_outdated": false, + remote: "python_version_pinned": false, + remote: "python_version_requested": "3.13", + remote: "setup_py_only": false, + remote: "total_duration": [0-9.]+ + remote: } REGEX app.commit! app.push! diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 170b141ef..e21703f31 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -44,27 +44,28 @@ remote: remote: remote: - remote: - remote: cache_restore_duration=[0-9.]+ - remote: cache_save_duration=[0-9.]+ - remote: cache_status=empty - remote: dependencies_install_duration=[0-9.]+ - remote: django_collectstatic_duration=[0-9.]+ - remote: nltk_downloader_duration=[0-9.]+ - remote: package_manager_install_duration=[0-9.]+ - remote: package_manager=pipenv - remote: pipenv_version=#{PIPENV_VERSION} - remote: post_compile_hook=false - remote: pre_compile_hook=false - remote: python_install_duration=[0-9.]+ - remote: python_version=#{DEFAULT_PYTHON_FULL_VERSION} - remote: python_version_major=3.13 - remote: python_version_origin=Pipfile.lock - remote: python_version_outdated=false - remote: python_version_pinned=false - remote: python_version_requested=3.13 - remote: setup_py_only=false - remote: total_duration=[0-9.]+ + remote: { + remote: "cache_restore_duration": [0-9.]+, + remote: "cache_save_duration": [0-9.]+, + remote: "cache_status": "empty", + remote: "dependencies_install_duration": [0-9.]+, + remote: "django_collectstatic_duration": [0-9.]+, + remote: "nltk_downloader_duration": [0-9.]+, + remote: "package_manager": "pipenv", + remote: "package_manager_install_duration": [0-9.]+, + remote: "pipenv_version": "#{PIPENV_VERSION}", + remote: "post_compile_hook": false, + remote: "pre_compile_hook": false, + remote: "python_install_duration": [0-9.]+, + remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", + remote: "python_version_major": "3.13", + remote: "python_version_origin": "Pipfile.lock", + remote: "python_version_outdated": false, + remote: "python_version_pinned": false, + remote: "python_version_requested": "3.13", + remote: "setup_py_only": false, + remote: "total_duration": [0-9.]+ + remote: } REGEX app.commit! app.push! diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index ee9fd238f..d7e5e7580 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -45,27 +45,28 @@ remote: remote: remote: - remote: - remote: cache_restore_duration=[0-9.]+ - remote: cache_save_duration=[0-9.]+ - remote: cache_status=empty - remote: dependencies_install_duration=[0-9.]+ - remote: django_collectstatic_duration=[0-9.]+ - remote: nltk_downloader_duration=[0-9.]+ - remote: package_manager_install_duration=[0-9.]+ - remote: package_manager=poetry - remote: poetry_version=#{POETRY_VERSION} - remote: post_compile_hook=false - remote: pre_compile_hook=false - remote: python_install_duration=[0-9.]+ - remote: python_version=#{DEFAULT_PYTHON_FULL_VERSION} - remote: python_version_major=3.13 - remote: python_version_origin=.python-version - remote: python_version_outdated=false - remote: python_version_pinned=false - remote: python_version_requested=3.13 - remote: setup_py_only=false - remote: total_duration=[0-9.]+ + remote: { + remote: "cache_restore_duration": [0-9.]+, + remote: "cache_save_duration": [0-9.]+, + remote: "cache_status": "empty", + remote: "dependencies_install_duration": [0-9.]+, + remote: "django_collectstatic_duration": [0-9.]+, + remote: "nltk_downloader_duration": [0-9.]+, + remote: "package_manager": "poetry", + remote: "package_manager_install_duration": [0-9.]+, + remote: "poetry_version": "#{POETRY_VERSION}", + remote: "post_compile_hook": false, + remote: "pre_compile_hook": false, + remote: "python_install_duration": [0-9.]+, + remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", + remote: "python_version_major": "3.13", + remote: "python_version_origin": ".python-version", + remote: "python_version_outdated": false, + remote: "python_version_pinned": false, + remote: "python_version_requested": "3.13", + remote: "setup_py_only": false, + remote: "total_duration": [0-9.]+ + remote: } REGEX app.commit! app.push! diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 3dd7a1fd2..1c33b8390 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -49,27 +49,28 @@ remote: remote: remote: - remote: - remote: cache_restore_duration=[0-9.]+ - remote: cache_save_duration=[0-9.]+ - remote: cache_status=empty - remote: dependencies_install_duration=[0-9.]+ - remote: django_collectstatic_duration=[0-9.]+ - remote: nltk_downloader_duration=[0-9.]+ - remote: package_manager_install_duration=[0-9.]+ - remote: package_manager=uv - remote: post_compile_hook=false - remote: pre_compile_hook=false - remote: python_install_duration=[0-9.]+ - remote: python_version=#{DEFAULT_PYTHON_FULL_VERSION} - remote: python_version_major=3.13 - remote: python_version_origin=.python-version - remote: python_version_outdated=false - remote: python_version_pinned=false - remote: python_version_requested=3.13 - remote: setup_py_only=false - remote: total_duration=[0-9.]+ - remote: uv_version=#{UV_VERSION} + remote: { + remote: "cache_restore_duration": [0-9.]+, + remote: "cache_save_duration": [0-9.]+, + remote: "cache_status": "empty", + remote: "dependencies_install_duration": [0-9.]+, + remote: "django_collectstatic_duration": [0-9.]+, + remote: "nltk_downloader_duration": [0-9.]+, + remote: "package_manager": "uv", + remote: "package_manager_install_duration": [0-9.]+, + remote: "post_compile_hook": false, + remote: "pre_compile_hook": false, + remote: "python_install_duration": [0-9.]+, + remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", + remote: "python_version_major": "3.13", + remote: "python_version_origin": ".python-version", + remote: "python_version_outdated": false, + remote: "python_version_pinned": false, + remote: "python_version_requested": "3.13", + remote: "setup_py_only": false, + remote: "total_duration": [0-9.]+, + remote: "uv_version": "#{UV_VERSION}" + remote: } REGEX app.commit! app.push! diff --git a/vendor/buildpack-stdlib_v8.sh b/vendor/buildpack-stdlib_v8.sh index 343f9b28d..2d13c6f46 100755 --- a/vendor/buildpack-stdlib_v8.sh +++ b/vendor/buildpack-stdlib_v8.sh @@ -73,8 +73,3 @@ sub_env() { "$@" ) } - -# Returns the current time, in milliseconds. -nowms() { - date +%s%3N -} From 5ffcadf6c834e31c8d2f85c1b4633a6bf13028f8 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 21 Aug 2025 15:21:19 +0000 Subject: [PATCH 202/406] Prepare release v302 (#1879) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index aa406181e..780e41b4d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v302] - 2025-08-21 + - Stopped setting the `PYTHONHASHSEED` env var. ([#1876](https://github.com/heroku/heroku-buildpack-python/pull/1876)) - Removed support for `BUILDPACK_S3_BASE_URL`. ([#1875](https://github.com/heroku/heroku-buildpack-python/pull/1875)) - Refactored buildpack data store and `bin/report` to simplify the implementation and fix some string escaping bugs. ([#1878](https://github.com/heroku/heroku-buildpack-python/pull/1878)) @@ -1299,7 +1302,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v301...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v302...main +[v302]: https://github.com/heroku/heroku-buildpack-python/compare/v301...v302 [v301]: https://github.com/heroku/heroku-buildpack-python/compare/v300...v301 [v300]: https://github.com/heroku/heroku-buildpack-python/compare/v299...v300 [v299]: https://github.com/heroku/heroku-buildpack-python/compare/v298...v299 From 5671ed7c535b7901acfaacf44ea2931c10520e16 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 22 Aug 2025 08:49:06 +0000 Subject: [PATCH 203/406] Bump uv from 0.8.9 to 0.8.13 (#1880) * Bump uv from 0.8.9 to 0.8.13 Bumps [uv](https://github.com/astral-sh/uv) from 0.8.9 to 0.8.13. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.8.9...0.8.13) --- updated-dependencies: - dependency-name: uv dependency-version: 0.8.13 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 780e41b4d..d58e086fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.8.9 to 0.8.13. ([#1880](https://github.com/heroku/heroku-buildpack-python/pull/1880)) ## [v302] - 2025-08-21 diff --git a/requirements/uv.txt b/requirements/uv.txt index 8a5fd6f41..a98c809f6 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.8.9 +uv==0.8.13 From 30086f2e8da2bf50d1b669b2e9ca82d479bf0892 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20Z=C3=BClke?= Date: Fri, 22 Aug 2025 18:14:06 +0200 Subject: [PATCH 204/406] Use EPOCHREALTIME in build_data timing logic (#1881) Built-in Bash (5.0+) variable, a float with microseconds precision. Faster, simpler, and more portable (no longer requires GNU date). Logging it with six digits worth of precision instead of the previous three, but a float is a float ;) GUS-W-19404492. Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + bin/compile | 10 +++++----- lib/build_data.sh | 20 ++++++++++++-------- lib/cache.sh | 4 ++-- lib/hooks.sh | 2 +- lib/python.sh | 2 +- 6 files changed, 22 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d58e086fd..f36dae1c1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Updated uv from 0.8.9 to 0.8.13. ([#1880](https://github.com/heroku/heroku-buildpack-python/pull/1880)) +- Switched to Bash 5.0's `EPOCHREALTIME` for buildpack data store timing logic. ([#1881](https://github.com/heroku/heroku-buildpack-python/pull/1881)) ## [v302] - 2025-08-21 diff --git a/bin/compile b/bin/compile index 7b7abe13b..f5d45f673 100755 --- a/bin/compile +++ b/bin/compile @@ -33,7 +33,7 @@ source "${BUILDPACK_DIR}/lib/python_version.sh" source "${BUILDPACK_DIR}/lib/python.sh" source "${BUILDPACK_DIR}/lib/uv.sh" -compile_start_time=$(build_data::current_unix_time_ms) +compile_start_time=$(build_data::current_unix_realtime) # Initialise the build data store used to track state across builds (for cache invalidation # and messaging when build configuration changes) and also so that `bin/report` can generate @@ -165,7 +165,7 @@ fi python::install "${BUILD_DIR}" "${STACK}" "${python_full_version}" "${python_major_version}" "${python_version_origin}" # Install the package manager and related tools. -package_manager_install_start_time=$(build_data::current_unix_time_ms) +package_manager_install_start_time=$(build_data::current_unix_realtime) case "${package_manager}" in pip) pip::install_pip_setuptools_wheel "${python_home}" "${python_major_version}" @@ -186,7 +186,7 @@ esac build_data::set_duration "package_manager_install_duration" "${package_manager_install_start_time}" # Install app dependencies. -dependencies_install_start_time=$(build_data::current_unix_time_ms) +dependencies_install_start_time=$(build_data::current_unix_realtime) case "${package_manager}" in pip) pip::install_dependencies @@ -207,13 +207,13 @@ esac build_data::set_duration "dependencies_install_duration" "${dependencies_install_start_time}" # Support for NLTK corpora. -nltk_downloader_start_time=$(build_data::current_unix_time_ms) +nltk_downloader_start_time=$(build_data::current_unix_realtime) sub_env "${BUILDPACK_DIR}/bin/steps/nltk" build_data::set_duration "nltk_downloader_duration" "${nltk_downloader_start_time}" # Django collectstatic support. # The buildpack automatically runs collectstatic for Django applications. -collectstatic_start_time=$(build_data::current_unix_time_ms) +collectstatic_start_time=$(build_data::current_unix_realtime) sub_env "${BUILDPACK_DIR}/bin/steps/collectstatic" build_data::set_duration "django_collectstatic_duration" "${collectstatic_start_time}" diff --git a/lib/build_data.sh b/lib/build_data.sh index ae50fddc1..960c4f5c3 100644 --- a/lib/build_data.sh +++ b/lib/build_data.sh @@ -43,11 +43,11 @@ function build_data::set_string() { } # Sets a build data value for the elapsed time in seconds between the provided start time and the -# current time, represented as a float with milliseconds precision. +# current time, represented as a float with microseconds precision. # # Usage: # ``` -# local dependencies_install_start_time=$(build_data::current_unix_time_ms) +# local dependencies_install_start_time=$(build_data::current_unix_realtime) # # ... some operation ... # build_data::set_duration "dependencies_install_duration" "${dependencies_install_start_time}" # ``` @@ -55,8 +55,8 @@ function build_data::set_duration() { local key="${1}" local start_time="${2}" local end_time duration - end_time="$(build_data::current_unix_time_ms)" - duration="$(awk -v start="${start_time}" -v end="${end_time}" 'BEGIN { printf "%.3f", (end - start)/1000 }')" + end_time="$(build_data::current_unix_realtime)" + duration="$(awk -v start="${start_time}" -v end="${end_time}" 'BEGIN { printf "%f", (end - start) }')" build_data::set_raw "${key}" "${duration}" } @@ -134,16 +134,20 @@ function build_data::get_previous() { fi } -# Returns the current time in milliseconds since the UNIX Epoch. +# Returns the current time since the UNIX Epoch, as a float with microseconds precision # # Usage: # ``` -# local dependencies_install_start_time=$(build_data::current_unix_time_ms) +# local dependencies_install_start_time=$(build_data::current_unix_realtime) # # ... some operation ... # build_data::set_duration "dependencies_install_duration" "${dependencies_install_start_time}" # ``` -function build_data::current_unix_time_ms() { - date +%s%3N +function build_data::current_unix_realtime() { + # We use a subshell with `LC_ALL=C` to ensure the output format isn't affected by system locale. + ( + LC_ALL=C + echo "${EPOCHREALTIME}" + ) } # Prints the contents of the build data store in sorted JSON format. diff --git a/lib/cache.sh b/lib/cache.sh index c657f4771..15dd29c8a 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -34,7 +34,7 @@ function cache::restore() { local package_manager="${6}" local cache_restore_start_time - cache_restore_start_time=$(build_data::current_unix_time_ms) + cache_restore_start_time=$(build_data::current_unix_realtime) if [[ ! -d "${cache_dir}/.heroku/python" ]]; then build_data::set_string "cache_status" "empty" @@ -166,7 +166,7 @@ function cache::save() { local package_manager="${5}" local cache_save_start_time - cache_save_start_time=$(build_data::current_unix_time_ms) + cache_save_start_time=$(build_data::current_unix_realtime) output::step "Saving cache" diff --git a/lib/hooks.sh b/lib/hooks.sh index 1018081f4..68c002774 100644 --- a/lib/hooks.sh +++ b/lib/hooks.sh @@ -12,7 +12,7 @@ function hooks::run_hook() { if [[ -f "${script_path}" ]]; then local hook_start_time - hook_start_time=$(build_data::current_unix_time_ms) + hook_start_time=$(build_data::current_unix_realtime) output::step "Running ${script_path} hook" build_data::set_raw "${hook_name}_hook" "true" chmod +x "${script_path}" diff --git a/lib/python.sh b/lib/python.sh index f194734c6..084eb12cd 100644 --- a/lib/python.sh +++ b/lib/python.sh @@ -14,7 +14,7 @@ function python::install() { local python_version_origin="${5}" local install_python_start_time - install_python_start_time=$(build_data::current_unix_time_ms) + install_python_start_time=$(build_data::current_unix_realtime) local install_dir="${build_dir}/.heroku/python" if [[ -f "${install_dir}/bin/python" ]]; then From 49e5bbc8f90a20023eea34acc62c4a00f59cb865 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 25 Aug 2025 19:08:50 +0100 Subject: [PATCH 205/406] Use contractions in user-facing messages (#1882) Per the CX team's style guide. Only the `package_manager.sh` change is user-facing, but I've updated the other places to reduce false positives when grepping. --- bin/compile | 2 +- builds/build_python_runtime.sh | 2 +- builds/test_python_runtime.sh | 4 ++-- etc/publish.sh | 2 +- lib/package_manager.sh | 2 +- spec/hatchet/pipenv_spec.rb | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/bin/compile b/bin/compile index f5d45f673..c00a1405e 100755 --- a/bin/compile +++ b/bin/compile @@ -40,7 +40,7 @@ compile_start_time=$(build_data::current_unix_realtime) # a build report that will be consumed by the build system for observability. build_data::setup -checks::ensure_supported_stack "${STACK:?Required env var STACK is not set}" +checks::ensure_supported_stack "${STACK:?"Required env var STACK isn't set"}" checks::duplicate_python_buildpack "${BUILD_DIR}" # Prepend proper path for old-school virtualenv hackery. diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index 1be5a1fad..c7ab303a3 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -36,7 +36,7 @@ case "${STACK:?}" in esac if [[ " ${SUPPORTED_PYTHON_VERSIONS[*]} " != *" ${PYTHON_MAJOR_VERSION} "* ]]; then - abort "Python ${PYTHON_MAJOR_VERSION} is not supported on ${STACK}!" + abort "Python ${PYTHON_MAJOR_VERSION} isn't supported on ${STACK}!" fi # The release keys can be found on https://www.python.org/downloads/ -> "OpenPGP Public Keys". diff --git a/builds/test_python_runtime.sh b/builds/test_python_runtime.sh index 190c7b62d..020b4dc32 100755 --- a/builds/test_python_runtime.sh +++ b/builds/test_python_runtime.sh @@ -36,7 +36,7 @@ set +x UNEXPECTED_BIN_FILES="$(find "${INSTALL_DIR}/bin" -type 'f,l' -not -name 'python*')" if [[ -n "${UNEXPECTED_BIN_FILES}" ]]; then echo "${UNEXPECTED_BIN_FILES}" - abort "The above files were found in the bin/ directory but were not expected!" + abort "The above files were found in the bin/ directory but weren't expected!" else echo "No unexpected files found in the bin/ directory." fi @@ -44,7 +44,7 @@ fi # Check that all dynamically linked libraries exist in the run image (since it has fewer packages than the build image). LDD_OUTPUT=$(find "${INSTALL_DIR}" -type f,l \( -name 'python3' -o -name '*.so*' \) -exec ldd '{}' +) if grep 'not found' <<<"${LDD_OUTPUT}" | sort --unique; then - abort "The above dynamically linked libraries were not found!" + abort "The above dynamically linked libraries weren't found!" else echo "All dynamically linked libraries were found." fi diff --git a/etc/publish.sh b/etc/publish.sh index d40b12b85..019bf60d3 100755 --- a/etc/publish.sh +++ b/etc/publish.sh @@ -32,7 +32,7 @@ current_github_release_version=$(gh release view --json tagName --jq '.tagName' current_registry_version="$(heroku buildpacks:versions "${buildpack_registry_name}" | awk 'FNR == 3 { print $1 }')" if [[ "${current_github_release_version}" != "${current_registry_version}" ]]; then - abort "The current GitHub release version (v${current_github_release_version}) does not match the registry version (v${current_registry_version}), likely due to a registry rollback. Fix this first!" + abort "The current GitHub release version (v${current_github_release_version}) doesn't match the registry version (v${current_registry_version}), likely due to a registry rollback. Fix this first!" fi new_version="$((current_github_release_version + 1))" diff --git a/lib/package_manager.sh b/lib/package_manager.sh index 46cef2d7e..a4fcac638 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -17,7 +17,7 @@ function package_manager::determine_package_manager() { Error: No 'Pipfile.lock' found! A 'Pipfile' file was found, however, the associated 'Pipfile.lock' - Pipenv lockfile was not. This means your app dependency versions + Pipenv lockfile wasn't. This means your app dependency versions aren't pinned, which means the package versions used on Heroku might not match those installed in other environments. diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index e21703f31..fe7c261a6 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -253,7 +253,7 @@ remote: ! Error: No 'Pipfile.lock' found! remote: ! remote: ! A 'Pipfile' file was found, however, the associated 'Pipfile.lock' - remote: ! Pipenv lockfile was not. This means your app dependency versions + remote: ! Pipenv lockfile wasn't. This means your app dependency versions remote: ! aren't pinned, which means the package versions used on Heroku remote: ! might not match those installed in other environments. remote: ! From 577a6199df69f38b6fcf7e2b0268685f8626ae8e Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 26 Aug 2025 07:59:38 +0100 Subject: [PATCH 206/406] Fix remaining shellcheck SC2250 instances (#1883) https://www.shellcheck.net/wiki/SC2250 GUS-W-19413416. --- bin/compile | 21 ++++++++++---------- bin/steps/collectstatic | 15 +++++++------- bin/utils | 1 - lib/build_data.sh | 2 +- vendor/buildpack-stdlib_v8.sh | 37 +++++++++++++++++------------------ 5 files changed, 36 insertions(+), 40 deletions(-) diff --git a/bin/compile b/bin/compile index c00a1405e..775fab6ca 100755 --- a/bin/compile +++ b/bin/compile @@ -1,7 +1,6 @@ #!/usr/bin/env bash # Usage: bin/compile # See: https://devcenter.heroku.com/articles/buildpack-api -# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. set -euo pipefail shopt -s inherit_errexit @@ -45,7 +44,7 @@ checks::duplicate_python_buildpack "${BUILD_DIR}" # Prepend proper path for old-school virtualenv hackery. # This may not be necessary. -export PATH=:/usr/local/bin:$PATH +export PATH=":/usr/local/bin:${PATH}" # Exported for use in subshells, such as the steps run via sub_env. export BUILD_DIR CACHE_DIR ENV_DIR @@ -72,10 +71,10 @@ source "${BUILDPACK_DIR}/bin/warnings" # More on this topic later. mkdir -p /app/.heroku -PROFILE_PATH="$BUILD_DIR/.profile.d/python.sh" +PROFILE_PATH="${BUILD_DIR}/.profile.d/python.sh" EXPORT_PATH="${BUILDPACK_DIR}/export" -GUNICORN_PROFILE_PATH="$BUILD_DIR/.profile.d/python.gunicorn.sh" -WEB_CONCURRENCY_PROFILE_PATH="$BUILD_DIR/.profile.d/WEB_CONCURRENCY.sh" +GUNICORN_PROFILE_PATH="${BUILD_DIR}/.profile.d/python.gunicorn.sh" +WEB_CONCURRENCY_PROFILE_PATH="${BUILD_DIR}/.profile.d/WEB_CONCURRENCY.sh" python_home='/app/.heroku/python' # NB: Python must be added to PATH using the symlinked `/app` location and not its actual location @@ -94,7 +93,7 @@ export LIBRARY_PATH="/app/.heroku/python/lib${LIBRARY_PATH:+:${LIBRARY_PATH}}" export LD_LIBRARY_PATH="/app/.heroku/python/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" export PKG_CONFIG_PATH="/app/.heroku/python/lib/pkg-config${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}" -cd "$BUILD_DIR" +cd "${BUILD_DIR}" # Runs a `bin/pre_compile` script if found in the app source, allowing build customisation. hooks::run_hook "pre_compile" @@ -148,7 +147,7 @@ python_version::warn_if_patch_update_available "${python_full_version}" "${pytho cache::restore "${BUILD_DIR}" "${CACHE_DIR}" "${STACK}" "${cached_python_full_version}" "${python_full_version}" "${package_manager}" # The directory for the .profile.d scripts. -mkdir -p "$(dirname "$PROFILE_PATH")" +mkdir -p "$(dirname "${PROFILE_PATH}")" # On Heroku CI, builds happen in `/app`. Otherwise, on the Heroku platform, # they occur in a temp directory. Because Python is not portable, we must create @@ -159,7 +158,7 @@ mkdir -p "$(dirname "$PROFILE_PATH")" if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then # python expects to reside in /app, so set up symlinks # we will not remove these later so subsequent buildpacks can still invoke it - ln -nsf "$BUILD_DIR/.heroku/python" /app/.heroku/python + ln -nsf "${BUILD_DIR}/.heroku/python" /app/.heroku/python fi python::install "${BUILD_DIR}" "${STACK}" "${python_full_version}" "${python_major_version}" "${python_version_origin}" @@ -235,7 +234,7 @@ set_default_env PYTHONPATH "\${HOME}" # Python expects to be in /app, if at runtime, it is not, set # up symlinks… this can occur when the subdir buildpack is used. -cat <>"$PROFILE_PATH" +cat <>"${PROFILE_PATH}" if [[ \$HOME != "/app" ]]; then mkdir -p /app/.heroku ln -nsf "\$HOME/.heroku/python" /app/.heroku/python @@ -282,8 +281,8 @@ if [[ "${BUILD_DIR}" != "/app" ]]; then fi # Install sane-default script for $WEB_CONCURRENCY and $FORWARDED_ALLOW_IPS. -cp "${BUILDPACK_DIR}/vendor/WEB_CONCURRENCY.sh" "$WEB_CONCURRENCY_PROFILE_PATH" -cp "${BUILDPACK_DIR}/vendor/python.gunicorn.sh" "$GUNICORN_PROFILE_PATH" +cp "${BUILDPACK_DIR}/vendor/WEB_CONCURRENCY.sh" "${WEB_CONCURRENCY_PROFILE_PATH}" +cp "${BUILDPACK_DIR}/vendor/python.gunicorn.sh" "${GUNICORN_PROFILE_PATH}" # Runs a `bin/post_compile` script if found in the app source, allowing build customisation. hooks::run_hook "post_compile" diff --git a/bin/steps/collectstatic b/bin/steps/collectstatic index efea7ac98..6b6c6e828 100755 --- a/bin/steps/collectstatic +++ b/bin/steps/collectstatic @@ -1,5 +1,4 @@ #!/usr/bin/env bash -# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. # Django Collectstatic runner. If you have Django installed, collectstatic will # automatically be executed as part of the build process. If collectstatic @@ -54,14 +53,14 @@ fi build_data::set_string "django_collectstatic" "enabled" -output::step "$ python $MANAGE_FILE collectstatic --noinput" +output::step "$ python ${MANAGE_FILE} collectstatic --noinput" -PYTHONPATH=${PYTHONPATH:-.} +PYTHONPATH="${PYTHONPATH:-.}" export PYTHONPATH COLLECTSTATIC_LOG=$(mktemp) set +e -python "$MANAGE_FILE" collectstatic --noinput --traceback 2>&1 | tee "$COLLECTSTATIC_LOG" | sed --unbuffered '/^Post-processed/d;/^Copying/d;/^$/d' | output::indent +python "${MANAGE_FILE}" collectstatic --noinput --traceback 2>&1 | tee "${COLLECTSTATIC_LOG}" | sed --unbuffered '/^Post-processed/d;/^Copying/d;/^$/d' | output::indent COLLECTSTATIC_STATUS="${PIPESTATUS[0]}" set -e @@ -72,13 +71,13 @@ if [[ "${COLLECTSTATIC_STATUS}" == 0 ]]; then fi # Display a warning if collectstatic failed. -if grep -q 'SyntaxError' "$COLLECTSTATIC_LOG"; then +if grep -q 'SyntaxError' "${COLLECTSTATIC_LOG}"; then build_data::set_string "failure_reason" "collectstatic-syntax-error" -elif grep -q 'ImproperlyConfigured' "$COLLECTSTATIC_LOG"; then +elif grep -q 'ImproperlyConfigured' "${COLLECTSTATIC_LOG}"; then build_data::set_string "failure_reason" "collectstatic-improper-configuration" -elif grep -q 'The CSS file' "$COLLECTSTATIC_LOG"; then +elif grep -q 'The CSS file' "${COLLECTSTATIC_LOG}"; then build_data::set_string "failure_reason" "collectstatic-fancy-references" -elif grep -q 'OSError' "$COLLECTSTATIC_LOG"; then +elif grep -q 'OSError' "${COLLECTSTATIC_LOG}"; then build_data::set_string "failure_reason" "collectstatic-missing-file" else build_data::set_string "failure_reason" "collectstatic-other" diff --git a/bin/utils b/bin/utils index 195172a7a..3eedc1f81 100755 --- a/bin/utils +++ b/bin/utils @@ -1,5 +1,4 @@ #!/usr/bin/env bash -# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. # Be careful about moving these to bin/compile, since this utils script is sourced # directly by other scripts run via subshells, and not only from bin/compile. diff --git a/lib/build_data.sh b/lib/build_data.sh index 960c4f5c3..6d36fe839 100644 --- a/lib/build_data.sh +++ b/lib/build_data.sh @@ -134,7 +134,7 @@ function build_data::get_previous() { fi } -# Returns the current time since the UNIX Epoch, as a float with microseconds precision +# Returns the current time since the UNIX Epoch, as a float with microseconds precision. # # Usage: # ``` diff --git a/vendor/buildpack-stdlib_v8.sh b/vendor/buildpack-stdlib_v8.sh index 2d13c6f46..e2fdec6a3 100755 --- a/vendor/buildpack-stdlib_v8.sh +++ b/vendor/buildpack-stdlib_v8.sh @@ -1,6 +1,5 @@ #!/usr/bin/env bash # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. -# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. set -euo pipefail @@ -14,46 +13,46 @@ set -euo pipefail # NOTICE: Expects PROFILE_PATH & EXPORT_PATH to be set! set_env() { # TODO: automatically create profile path directory if it doesn't exist. - echo "export $1=$2" >>"$PROFILE_PATH" - echo "export $1=$2" >>"$EXPORT_PATH" + echo "export $1=$2" >>"${PROFILE_PATH}" + echo "export $1=$2" >>"${EXPORT_PATH}" } # Usage: $ set-default-env key value # NOTICE: Expects PROFILE_PATH & EXPORT_PATH to be set! set_default_env() { - echo "export $1=\${$1:-$2}" >>"$PROFILE_PATH" - echo "export $1=\${$1:-$2}" >>"$EXPORT_PATH" + echo "export $1=\${$1:-$2}" >>"${PROFILE_PATH}" + echo "export $1=\${$1:-$2}" >>"${EXPORT_PATH}" } # Usage: $ un-set-env key # NOTICE: Expects PROFILE_PATH to be set! un_set_env() { - echo "unset $1" >>"$PROFILE_PATH" + echo "unset $1" >>"${PROFILE_PATH}" } # Usage: $ _env-blacklist pattern # Outputs a regex of default blacklist env vars. _env_blacklist() { - local regex=${1:-''} - if [[ -n "$regex" ]]; then - regex="|$regex" + local regex="${1:-}" + if [[ -n "${regex}" ]]; then + regex="|${regex}" fi - echo "^(PATH|CPATH|CPPATH|LD_PRELOAD|LIBRARY_PATH|LD_LIBRARY_PATH|PYTHONHOME$regex)$" + echo "^(PATH|CPATH|CPPATH|LD_PRELOAD|LIBRARY_PATH|LD_LIBRARY_PATH|PYTHONHOME${regex})$" } # Usage: $ export-env ENV_DIR WHITELIST BLACKLIST # Exports the environment variables defined in the given directory. export_env() { - local env_dir=${1:-$ENV_DIR} - local whitelist=${2:-''} + local env_dir="${1:-${ENV_DIR}}" + local whitelist="${2:-}" local blacklist - blacklist="$(_env_blacklist "$3")" - if [[ -d "$env_dir" ]]; then + blacklist="$(_env_blacklist "${3}")" + if [[ -d "${env_dir}" ]]; then # Environment variable names won't contain characters affected by: # shellcheck disable=SC2045 - for e in $(ls "$env_dir"); do - echo "$e" | grep -E "$whitelist" | grep -qvE "$blacklist" \ - && export "$e=$(cat "$env_dir/$e")" + for e in $(ls "${env_dir}"); do + echo "${e}" | grep -E "${whitelist}" | grep -qvE "${blacklist}" \ + && export "${e}=$(cat "${env_dir}/${e}")" : done fi @@ -68,8 +67,8 @@ export_env() { sub_env() { ( # TODO: Fix https://github.com/heroku/buildpack-stdlib/issues/37 - export_env "$ENV_DIR" "${WHITELIST:-}" "${BLACKLIST:-}" + export_env "${ENV_DIR}" "${WHITELIST:-}" "${BLACKLIST:-}" - "$@" + "${@}" ) } From f5893f47662945d557edad609e4bb66b4125ac10 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 26 Aug 2025 08:00:41 +0100 Subject: [PATCH 207/406] Adjust curl timeouts, retries and logging (#1884) In #1439, the timeouts used with curl were set to higher values than we needed for Heroku's build system, in order to prevent timeouts in non-Heroku environments such as Dokku running on underpowered devices (or devices with poor internet connections). However, as seen in recent Honeycomb metrics, these higher timeouts cause longer delays on Heroku before a curl retry if there is a transient S3 connection issue - since the request will wait the full 10 seconds `connect_timeout` before retrying. As such, the defaults have been lowered again to be optimised for running on Heroku, and support for overriding the defaults via env var added to still allow running in other environments. Specifically: - The default connection timeout has been lowered from 10 seconds to 3, but can now be overridden via the env var `CURL_CONNECT_TIMEOUT` - The default total request timeout has been lowered from 120 seconds to 60, but can be overridden via `CURL_TIMEOUT` These env vars cannot be set via Heroku config vars, and must instead be set in the build system environment (or by an earlier buildpack). The env var names match those used by the classic Ruby buildpack: https://github.com/heroku/heroku-buildpack-ruby/blob/a6d124492aafe7f7ee0dd42d7a79284b6552d675/lib/language_pack/fetcher.rb#L47-L53 In addition, the usages of `--silent --show-error` have been replaced with `--no-progress-meter`, which is otherwise identical apart from it allowing warnings to be shown, such as the status messages output during retry attempts (of form: `Will retry in 1 seconds. 3 retries left` etc). Before: ``` -----> Installing Python 3.13.7 curl: (6) Could not resolve host: heroku-buildpack-python.s3.us-east-1.amazonaws.com curl: (6) Could not resolve host: heroku-buildpack-python.s3.us-east-1.amazonaws.com ... ``` After: ``` -----> Installing Python 3.13.7 curl: (7) Failed to connect to heroku-buildpack-python.s3.us-east-1.amazonaws.com port 443 after 3 ms: Couldn't connect to server Warning: Problem : connection refused. Will retry in 1 seconds. 5 retries left. curl: (7) Failed to connect to heroku-buildpack-python.s3.us-east-1.amazonaws.com port 443 after 4 ms: Couldn't connect to server Warning: Problem : connection refused. Will retry in 2 seconds. 4 retries left. ... ``` Lastly, the maximum number of retries has been increased slightly. Note: The curl options used in `build_python_runtime.sh` have also been adjusted, however, since these are only used by buildpack maintainers (typically by running them on GitHub Actions), there is no need to make those timeouts configurable. GUS-W-19414036. --- CHANGELOG.md | 3 +++ builds/build_python_runtime.sh | 8 ++++++-- lib/python.sh | 23 +++++++++++------------ lib/uv.sh | 24 ++++++++++++------------ 4 files changed, 32 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f36dae1c1..51b5dafe7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,9 @@ ## [Unreleased] - Updated uv from 0.8.9 to 0.8.13. ([#1880](https://github.com/heroku/heroku-buildpack-python/pull/1880)) +- Reduced default curl timeouts for faster retries of any transient connection issues on Heroku. ([#1884](https://github.com/heroku/heroku-buildpack-python/pull/1884)) +- Added support for overriding the default curl timeouts using `CURL_CONNECT_TIMEOUT` and `CURL_TIMEOUT`. These are intended for use in non-Heroku environments with slow connections, and so must be set via the build system rather than app config vars. ([#1884](https://github.com/heroku/heroku-buildpack-python/pull/1884)) +- Improved log output during curl retry attempts. ([#1884](https://github.com/heroku/heroku-buildpack-python/pull/1884)) - Switched to Bash 5.0's `EPOCHREALTIME` for buildpack data store timing logic. ([#1881](https://github.com/heroku/heroku-buildpack-python/pull/1881)) ## [v302] - 2025-08-21 diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index c7ab303a3..7def990af 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -1,5 +1,9 @@ #!/usr/bin/env bash +# This script is used by buildpack maintainers to compile new Python versions as they are released, +# ready for upload to S3. (It isn't run as part of normal buildpack execution during app builds.) +# See `builds/README.md` for how to invoke this via GitHub Actions. + set -euo pipefail shopt -s inherit_errexit @@ -71,8 +75,8 @@ set -o xtrace mkdir -p "${SRC_DIR}" "${INSTALL_DIR}" "${UPLOAD_DIR}" -curl --fail --retry 3 --retry-connrefused --connect-timeout 10 --max-time 60 -o python.tgz "${SOURCE_URL}" -curl --fail --retry 3 --retry-connrefused --connect-timeout 10 --max-time 60 -o python.tgz.asc "${SIGNATURE_URL}" +curl --fail --retry 5 --retry-connrefused --connect-timeout 3 --max-time 30 -o python.tgz "${SOURCE_URL}" +curl --fail --retry 5 --retry-connrefused --connect-timeout 3 --max-time 30 -o python.tgz.asc "${SIGNATURE_URL}" gpg --batch --verbose --recv-keys "${GPG_KEY_FINGERPRINT}" gpg --batch --verify python.tgz.asc python.tgz diff --git a/lib/python.sh b/lib/python.sh index 084eb12cd..ea87c2c63 100644 --- a/lib/python.sh +++ b/lib/python.sh @@ -37,21 +37,20 @@ function python::install() { # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { { - # We set max-time for improved UX/metrics for hanging downloads compared to relying - # on the build system timeout. The Python archives are only ~10 MB so take < 1s to - # download on Heroku's build system, however, we use much higher timeouts so that - # the buildpack works in non-Heroku environments that may be far from `us-east-1` - # or have a slower connection. We don't use `--speed-limit` since it gives worse - # error messages when used with retries and piping to tar. + # We set max-time for improved UX/metrics for hanging downloads compared to relying on the build + # system timeout. We don't use `--speed-limit` since it gives worse error messages when used with + # retries and piping to tar. The Python archives are ~10 MB so only take ~1s to download on Heroku, + # so we set low timeouts to reduce delays before retries. However, we allow customising the timeouts + # to support non-Heroku environments that may be far from `us-east-1` or have a slower connection. + # We use `--no-progress-meter` rather than `--silent` so that retry status messages are printed. curl \ - --connect-timeout 10 \ + --connect-timeout "${CURL_CONNECT_TIMEOUT:-3}" \ --fail \ - --max-time 120 \ - --retry-max-time 120 \ - --retry 3 \ + --max-time "${CURL_TIMEOUT:-60}" \ + --no-progress-meter \ + --retry-max-time "${CURL_TIMEOUT:-60}" \ + --retry 5 \ --retry-connrefused \ - --show-error \ - --silent \ "${python_url}" \ | tar \ --directory "${install_dir}" \ diff --git a/lib/uv.sh b/lib/uv.sh index 8dcd71cfd..c8d86b619 100644 --- a/lib/uv.sh +++ b/lib/uv.sh @@ -30,22 +30,22 @@ function uv::install_uv() { local uv_url="https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-${gnu_arch}-unknown-linux-gnu.tar.gz" if ! { - # We set max-time for improved UX for hanging downloads compared to relying on the build system - # timeout. The uv archive is only ~15 MB so takes < 1s to download on Heroku's build system, - # however, we use much higher timeouts so that the buildpack works in non-Heroku or local - # environments that may have a slower connection. We don't use `--speed-limit` since it gives - # worse error messages when used with retries and piping to tar. - # We have to use `--strip-components` since the archive contents are nested under a subdirectory. + # We set max-time for improved UX/metrics for hanging downloads compared to relying on the build + # system timeout. We don't use `--speed-limit` since it gives worse error messages when used with + # retries and piping to tar. The Python archives are ~10 MB so only take ~1s to download on Heroku, + # so we set low timeouts to reduce delays before retries. However, we allow customising the timeouts + # to support non-Heroku environments that may be far from `us-east-1` or have a slower connection. + # We use `--no-progress-meter` rather than `--silent` so that retry status messages are printed. + # We have to use `--strip-components` since the uv binary is nested under a subdirectory. curl \ - --connect-timeout 10 \ + --connect-timeout "${CURL_CONNECT_TIMEOUT:-3}" \ --fail \ --location \ - --max-time 120 \ - --retry-max-time 120 \ - --retry 3 \ + --max-time "${CURL_TIMEOUT:-60}" \ + --no-progress-meter \ + --retry-max-time "${CURL_TIMEOUT:-60}" \ + --retry 5 \ --retry-connrefused \ - --show-error \ - --silent \ "${uv_url}" \ | tar \ --directory "${uv_dir}" \ From 11101740c081cdeed82a5c97d2d96a3ed47d3449 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 26 Aug 2025 07:21:16 +0000 Subject: [PATCH 208/406] Prepare release v303 (#1885) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 51b5dafe7..b9de51635 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v303] - 2025-08-26 + - Updated uv from 0.8.9 to 0.8.13. ([#1880](https://github.com/heroku/heroku-buildpack-python/pull/1880)) - Reduced default curl timeouts for faster retries of any transient connection issues on Heroku. ([#1884](https://github.com/heroku/heroku-buildpack-python/pull/1884)) - Added support for overriding the default curl timeouts using `CURL_CONNECT_TIMEOUT` and `CURL_TIMEOUT`. These are intended for use in non-Heroku environments with slow connections, and so must be set via the build system rather than app config vars. ([#1884](https://github.com/heroku/heroku-buildpack-python/pull/1884)) @@ -1307,7 +1310,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v302...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v303...main +[v303]: https://github.com/heroku/heroku-buildpack-python/compare/v302...v303 [v302]: https://github.com/heroku/heroku-buildpack-python/compare/v301...v302 [v301]: https://github.com/heroku/heroku-buildpack-python/compare/v300...v301 [v300]: https://github.com/heroku/heroku-buildpack-python/compare/v299...v300 From c2ba267963c35de0e812fb39ae29f37d16941138 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 29 Aug 2025 16:13:03 +0100 Subject: [PATCH 209/406] Improve Django collectstatic tests (#1887) This backports the more comprehensive Django collectstatic tests from the Python CNB, along with a few further tests and adjustments given the implementation here differs (eg supports nested `manage.py` and collectstatic disablement). These additional tests are a precursor to: 1. (Short term) Fixing the issue seen in #1886 2. (Longer term) Backporting the improved collectstatic implementation from the Python CNB. With the new tests added, the Python getting started guide repo is no longer required, since everything it tested is now tested via feature-specific tests committed to the repo. The guide is still tested via its own Heroku CI runs, plus the separate periodic Direwolf tests. GUS-W-19452886. --- .editorconfig | 3 - .gitignore | 1 - bin/steps/collectstatic | 1 + hatchet.json | 9 +- hatchet.lock | 4 +- .../manage.py | 22 ++ .../django_invalid_settings_module/manage.py | 22 ++ .../requirements.txt | 0 .../requirements.txt | 0 .../manage.py | 22 ++ .../requirements.txt | 1 + .../testproject/__init__.py | 0 .../testproject/settings.py | 15 + .../backend/manage.py | 22 ++ .../backend/requirements.txt | 1 + .../backend/testapp/__init__.py | 0 .../backend/testapp/static/robots.txt | 0 .../backend/testproject/__init__.py | 0 .../backend/testproject/settings.py | 25 ++ .../requirements.txt | 1 + .../.python-version | 1 + .../manage.py | 2 +- .../requirements.txt | 3 + .../testapp/__init__.py | 0 .../testapp/static/robots.txt | 0 .../testproject/__init__.py | 0 .../testproject/settings.py | 28 ++ .../manage.py | 22 ++ .../requirements.txt | 1 + .../testproject/__init__.py | 0 .../testproject/settings.py | 6 + spec/fixtures/pip_basic/manage.py | 2 + spec/hatchet/django_spec.rb | 306 +++++++++++++----- 33 files changed, 419 insertions(+), 101 deletions(-) create mode 100644 spec/fixtures/django_collectstatic_disabled_file/manage.py create mode 100644 spec/fixtures/django_invalid_settings_module/manage.py rename spec/fixtures/{django_broken_project => django_invalid_settings_module}/requirements.txt (100%) rename spec/fixtures/{django_latest => django_no_manage_py}/requirements.txt (100%) create mode 100644 spec/fixtures/django_staticfiles_app_not_enabled/manage.py create mode 100644 spec/fixtures/django_staticfiles_app_not_enabled/requirements.txt create mode 100644 spec/fixtures/django_staticfiles_app_not_enabled/testproject/__init__.py create mode 100644 spec/fixtures/django_staticfiles_app_not_enabled/testproject/settings.py create mode 100644 spec/fixtures/django_staticfiles_latest_django/backend/manage.py create mode 100644 spec/fixtures/django_staticfiles_latest_django/backend/requirements.txt create mode 100644 spec/fixtures/django_staticfiles_latest_django/backend/testapp/__init__.py create mode 100644 spec/fixtures/django_staticfiles_latest_django/backend/testapp/static/robots.txt create mode 100644 spec/fixtures/django_staticfiles_latest_django/backend/testproject/__init__.py create mode 100644 spec/fixtures/django_staticfiles_latest_django/backend/testproject/settings.py create mode 120000 spec/fixtures/django_staticfiles_latest_django/requirements.txt create mode 100644 spec/fixtures/django_staticfiles_legacy_django/.python-version rename spec/fixtures/{django_broken_project => django_staticfiles_legacy_django}/manage.py (69%) create mode 100644 spec/fixtures/django_staticfiles_legacy_django/requirements.txt create mode 100644 spec/fixtures/django_staticfiles_legacy_django/testapp/__init__.py create mode 100644 spec/fixtures/django_staticfiles_legacy_django/testapp/static/robots.txt create mode 100644 spec/fixtures/django_staticfiles_legacy_django/testproject/__init__.py create mode 100644 spec/fixtures/django_staticfiles_legacy_django/testproject/settings.py create mode 100644 spec/fixtures/django_staticfiles_misconfigured/manage.py create mode 100644 spec/fixtures/django_staticfiles_misconfigured/requirements.txt create mode 100644 spec/fixtures/django_staticfiles_misconfigured/testproject/__init__.py create mode 100644 spec/fixtures/django_staticfiles_misconfigured/testproject/settings.py create mode 100644 spec/fixtures/pip_basic/manage.py diff --git a/.editorconfig b/.editorconfig index 0fb218967..0fa1b37cf 100644 --- a/.editorconfig +++ b/.editorconfig @@ -23,9 +23,6 @@ indent_style = tab shell_variant = bash switch_case_indent = true -[.hatchet/repos/**] -ignore = true - # The setup-ruby GitHub Action creates this directory when caching is enabled, and if # its not ignored will cause false positives when running shfmt in the CI lint job. [vendor/bundle/**] diff --git a/.gitignore b/.gitignore index e415b62c2..4cecc5fe2 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,4 @@ __pycache__/ -.hatchet/repos/ .venv/ # The setup-ruby GitHub Action creates this directory when caching is enabled, so we ignore # it here so it does not show up in the output of `git ls-files` for `make lint-scripts`. diff --git a/bin/steps/collectstatic b/bin/steps/collectstatic index 6b6c6e828..b21affb8c 100755 --- a/bin/steps/collectstatic +++ b/bin/steps/collectstatic @@ -104,6 +104,7 @@ if [[ "${DEBUG_COLLECTSTATIC:-0}" == "1" ]]; then echo echo "****** Collectstatic environment variables:" echo + # TODO: Sort the displayed env vars to make the order deterministic. env | output::indent fi diff --git a/hatchet.json b/hatchet.json index a9f518db2..0967ef424 100644 --- a/hatchet.json +++ b/hatchet.json @@ -1,8 +1 @@ -{ - "hatchet": { - "directory": ".hatchet/" - }, - "heroku": [ - "https://github.com/heroku/python-getting-started" - ] -} +{} diff --git a/hatchet.lock b/hatchet.lock index 51aa7c698..dcd024e99 100644 --- a/hatchet.lock +++ b/hatchet.lock @@ -1,3 +1 @@ ---- -- - ".hatchet/repos/heroku/python-getting-started" - - main +--- [] diff --git a/spec/fixtures/django_collectstatic_disabled_file/manage.py b/spec/fixtures/django_collectstatic_disabled_file/manage.py new file mode 100644 index 000000000..5ed9a654d --- /dev/null +++ b/spec/fixtures/django_collectstatic_disabled_file/manage.py @@ -0,0 +1,22 @@ +#!/usr/bin/env python +"""Django's command-line utility for administrative tasks.""" +import os +import sys + + +def main(): + """Run administrative tasks.""" + os.environ.setdefault("DJANGO_SETTINGS_MODULE", "nonexistent-module.settings") + try: + from django.core.management import execute_from_command_line + except ImportError as exc: + raise ImportError( + "Couldn't import Django. Are you sure it's installed and " + "available on your PYTHONPATH environment variable? Did you " + "forget to activate a virtual environment?" + ) from exc + execute_from_command_line(sys.argv) + + +if __name__ == "__main__": + main() diff --git a/spec/fixtures/django_invalid_settings_module/manage.py b/spec/fixtures/django_invalid_settings_module/manage.py new file mode 100644 index 000000000..5ed9a654d --- /dev/null +++ b/spec/fixtures/django_invalid_settings_module/manage.py @@ -0,0 +1,22 @@ +#!/usr/bin/env python +"""Django's command-line utility for administrative tasks.""" +import os +import sys + + +def main(): + """Run administrative tasks.""" + os.environ.setdefault("DJANGO_SETTINGS_MODULE", "nonexistent-module.settings") + try: + from django.core.management import execute_from_command_line + except ImportError as exc: + raise ImportError( + "Couldn't import Django. Are you sure it's installed and " + "available on your PYTHONPATH environment variable? Did you " + "forget to activate a virtual environment?" + ) from exc + execute_from_command_line(sys.argv) + + +if __name__ == "__main__": + main() diff --git a/spec/fixtures/django_broken_project/requirements.txt b/spec/fixtures/django_invalid_settings_module/requirements.txt similarity index 100% rename from spec/fixtures/django_broken_project/requirements.txt rename to spec/fixtures/django_invalid_settings_module/requirements.txt diff --git a/spec/fixtures/django_latest/requirements.txt b/spec/fixtures/django_no_manage_py/requirements.txt similarity index 100% rename from spec/fixtures/django_latest/requirements.txt rename to spec/fixtures/django_no_manage_py/requirements.txt diff --git a/spec/fixtures/django_staticfiles_app_not_enabled/manage.py b/spec/fixtures/django_staticfiles_app_not_enabled/manage.py new file mode 100644 index 000000000..8bd034f0d --- /dev/null +++ b/spec/fixtures/django_staticfiles_app_not_enabled/manage.py @@ -0,0 +1,22 @@ +#!/usr/bin/env python +"""Django's command-line utility for administrative tasks.""" +import os +import sys + + +def main(): + """Run administrative tasks.""" + os.environ.setdefault("DJANGO_SETTINGS_MODULE", "testproject.settings") + try: + from django.core.management import execute_from_command_line + except ImportError as exc: + raise ImportError( + "Couldn't import Django. Are you sure it's installed and " + "available on your PYTHONPATH environment variable? Did you " + "forget to activate a virtual environment?" + ) from exc + execute_from_command_line(sys.argv) + + +if __name__ == "__main__": + main() diff --git a/spec/fixtures/django_staticfiles_app_not_enabled/requirements.txt b/spec/fixtures/django_staticfiles_app_not_enabled/requirements.txt new file mode 100644 index 000000000..94a0e8344 --- /dev/null +++ b/spec/fixtures/django_staticfiles_app_not_enabled/requirements.txt @@ -0,0 +1 @@ +Django diff --git a/spec/fixtures/django_staticfiles_app_not_enabled/testproject/__init__.py b/spec/fixtures/django_staticfiles_app_not_enabled/testproject/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/django_staticfiles_app_not_enabled/testproject/settings.py b/spec/fixtures/django_staticfiles_app_not_enabled/testproject/settings.py new file mode 100644 index 000000000..5922fd00c --- /dev/null +++ b/spec/fixtures/django_staticfiles_app_not_enabled/testproject/settings.py @@ -0,0 +1,15 @@ +from pathlib import Path + +BASE_DIR = Path(__file__).resolve().parent.parent + +INSTALLED_APPS = [ + "django.contrib.auth", + "django.contrib.contenttypes", + "django.contrib.sessions", + "django.contrib.messages", + # The staticfiles app (which is what provides the collectstatic command) is not enabled. + # "django.contrib.staticfiles", +] + +STATIC_ROOT = BASE_DIR / "staticfiles" +STATIC_URL = "static/" diff --git a/spec/fixtures/django_staticfiles_latest_django/backend/manage.py b/spec/fixtures/django_staticfiles_latest_django/backend/manage.py new file mode 100644 index 000000000..8bd034f0d --- /dev/null +++ b/spec/fixtures/django_staticfiles_latest_django/backend/manage.py @@ -0,0 +1,22 @@ +#!/usr/bin/env python +"""Django's command-line utility for administrative tasks.""" +import os +import sys + + +def main(): + """Run administrative tasks.""" + os.environ.setdefault("DJANGO_SETTINGS_MODULE", "testproject.settings") + try: + from django.core.management import execute_from_command_line + except ImportError as exc: + raise ImportError( + "Couldn't import Django. Are you sure it's installed and " + "available on your PYTHONPATH environment variable? Did you " + "forget to activate a virtual environment?" + ) from exc + execute_from_command_line(sys.argv) + + +if __name__ == "__main__": + main() diff --git a/spec/fixtures/django_staticfiles_latest_django/backend/requirements.txt b/spec/fixtures/django_staticfiles_latest_django/backend/requirements.txt new file mode 100644 index 000000000..94a0e8344 --- /dev/null +++ b/spec/fixtures/django_staticfiles_latest_django/backend/requirements.txt @@ -0,0 +1 @@ +Django diff --git a/spec/fixtures/django_staticfiles_latest_django/backend/testapp/__init__.py b/spec/fixtures/django_staticfiles_latest_django/backend/testapp/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/django_staticfiles_latest_django/backend/testapp/static/robots.txt b/spec/fixtures/django_staticfiles_latest_django/backend/testapp/static/robots.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/django_staticfiles_latest_django/backend/testproject/__init__.py b/spec/fixtures/django_staticfiles_latest_django/backend/testproject/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/django_staticfiles_latest_django/backend/testproject/settings.py b/spec/fixtures/django_staticfiles_latest_django/backend/testproject/settings.py new file mode 100644 index 000000000..75021f41d --- /dev/null +++ b/spec/fixtures/django_staticfiles_latest_django/backend/testproject/settings.py @@ -0,0 +1,25 @@ +import os +import sys +from pathlib import Path +from pprint import pprint + +BASE_DIR = Path(__file__).resolve().parent.parent + +INSTALLED_APPS = [ + "django.contrib.staticfiles", + "testapp", +] + +STATIC_ROOT = BASE_DIR / "staticfiles" +STATIC_URL = "static/" + +pprint({ + k: v + for k, v in os.environ.items() + if not k in {"_", "DYNO", "OLDPWD", "REQUEST_ID", "SHLVL"} +}) +print() +pprint(sys.path) + +# Tests that app env vars are passed to the 'manage.py' script invocations. +assert "EXPECTED_ENV_VAR" in os.environ diff --git a/spec/fixtures/django_staticfiles_latest_django/requirements.txt b/spec/fixtures/django_staticfiles_latest_django/requirements.txt new file mode 120000 index 000000000..ed17bf41b --- /dev/null +++ b/spec/fixtures/django_staticfiles_latest_django/requirements.txt @@ -0,0 +1 @@ +backend/requirements.txt \ No newline at end of file diff --git a/spec/fixtures/django_staticfiles_legacy_django/.python-version b/spec/fixtures/django_staticfiles_legacy_django/.python-version new file mode 100644 index 000000000..bd28b9c5c --- /dev/null +++ b/spec/fixtures/django_staticfiles_legacy_django/.python-version @@ -0,0 +1 @@ +3.9 diff --git a/spec/fixtures/django_broken_project/manage.py b/spec/fixtures/django_staticfiles_legacy_django/manage.py similarity index 69% rename from spec/fixtures/django_broken_project/manage.py rename to spec/fixtures/django_staticfiles_legacy_django/manage.py index eb3ab8af5..97ed576b6 100644 --- a/spec/fixtures/django_broken_project/manage.py +++ b/spec/fixtures/django_staticfiles_legacy_django/manage.py @@ -3,7 +3,7 @@ import sys if __name__ == "__main__": - os.environ.setdefault("DJANGO_SETTINGS_MODULE", "gettingstarted.settings") + os.environ.setdefault("DJANGO_SETTINGS_MODULE", "testproject.settings") from django.core.management import execute_from_command_line diff --git a/spec/fixtures/django_staticfiles_legacy_django/requirements.txt b/spec/fixtures/django_staticfiles_legacy_django/requirements.txt new file mode 100644 index 000000000..e258a24fc --- /dev/null +++ b/spec/fixtures/django_staticfiles_legacy_django/requirements.txt @@ -0,0 +1,3 @@ +# This is the oldest Django version that works on Python 3.9 (which is the +# oldest Python that is available on all of our supported builders). +Django==1.8.19 diff --git a/spec/fixtures/django_staticfiles_legacy_django/testapp/__init__.py b/spec/fixtures/django_staticfiles_legacy_django/testapp/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/django_staticfiles_legacy_django/testapp/static/robots.txt b/spec/fixtures/django_staticfiles_legacy_django/testapp/static/robots.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/django_staticfiles_legacy_django/testproject/__init__.py b/spec/fixtures/django_staticfiles_legacy_django/testproject/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/django_staticfiles_legacy_django/testproject/settings.py b/spec/fixtures/django_staticfiles_legacy_django/testproject/settings.py new file mode 100644 index 000000000..9547fe35a --- /dev/null +++ b/spec/fixtures/django_staticfiles_legacy_django/testproject/settings.py @@ -0,0 +1,28 @@ +import os +import sys +from pathlib import Path +from pprint import pprint + +BASE_DIR = Path(__file__).resolve().parent.parent + +INSTALLED_APPS = [ + "django.contrib.staticfiles", + "testapp", +] + +STATIC_ROOT = BASE_DIR / "staticfiles" +STATIC_URL = "static/" + +# Older versions of Django require that `SECRET_KEY` is set when running collectstatic. +SECRET_KEY = "example" + +pprint({ + k: v + for k, v in os.environ.items() + if not k in {"_", "DYNO", "OLDPWD", "REQUEST_ID", "SHLVL"} +}) +print() +pprint(sys.path) + +# Tests that app env vars are passed to the 'manage.py' script invocations. +assert "EXPECTED_ENV_VAR" in os.environ diff --git a/spec/fixtures/django_staticfiles_misconfigured/manage.py b/spec/fixtures/django_staticfiles_misconfigured/manage.py new file mode 100644 index 000000000..8bd034f0d --- /dev/null +++ b/spec/fixtures/django_staticfiles_misconfigured/manage.py @@ -0,0 +1,22 @@ +#!/usr/bin/env python +"""Django's command-line utility for administrative tasks.""" +import os +import sys + + +def main(): + """Run administrative tasks.""" + os.environ.setdefault("DJANGO_SETTINGS_MODULE", "testproject.settings") + try: + from django.core.management import execute_from_command_line + except ImportError as exc: + raise ImportError( + "Couldn't import Django. Are you sure it's installed and " + "available on your PYTHONPATH environment variable? Did you " + "forget to activate a virtual environment?" + ) from exc + execute_from_command_line(sys.argv) + + +if __name__ == "__main__": + main() diff --git a/spec/fixtures/django_staticfiles_misconfigured/requirements.txt b/spec/fixtures/django_staticfiles_misconfigured/requirements.txt new file mode 100644 index 000000000..94a0e8344 --- /dev/null +++ b/spec/fixtures/django_staticfiles_misconfigured/requirements.txt @@ -0,0 +1 @@ +Django diff --git a/spec/fixtures/django_staticfiles_misconfigured/testproject/__init__.py b/spec/fixtures/django_staticfiles_misconfigured/testproject/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/spec/fixtures/django_staticfiles_misconfigured/testproject/settings.py b/spec/fixtures/django_staticfiles_misconfigured/testproject/settings.py new file mode 100644 index 000000000..1a521e130 --- /dev/null +++ b/spec/fixtures/django_staticfiles_misconfigured/testproject/settings.py @@ -0,0 +1,6 @@ +INSTALLED_APPS = [ + "django.contrib.staticfiles", +] + +# This is an invalid STATIC_ROOT value if collectstatic is being used. +STATIC_ROOT = None diff --git a/spec/fixtures/pip_basic/manage.py b/spec/fixtures/pip_basic/manage.py new file mode 100644 index 000000000..db7e7d588 --- /dev/null +++ b/spec/fixtures/pip_basic/manage.py @@ -0,0 +1,2 @@ +# Tests that manage.py alone doesn't trigger Django collectstatic. +raise RuntimeError("This is not a Django app, so manage.py should not be run!") diff --git a/spec/hatchet/django_spec.rb b/spec/hatchet/django_spec.rb index 6372ea097..1821ed421 100644 --- a/spec/hatchet/django_spec.rb +++ b/spec/hatchet/django_spec.rb @@ -4,124 +4,260 @@ # Tests that broken user-provided env vars don't take precedence over those set by this buildpack # and break running Python. This is particularly important when using shared builds of Python, -# since they rely upon `LD_LIBRARY_PATH` being correct. This list of env vars is based on those -# that used to be set to different values by `bin/release` in very old versions of the buildpack: +# since they rely upon `LD_LIBRARY_PATH` being correct. Some of these are based on the env vars +# that used to be set by `bin/release` by very old versions of the buildpack: # https://github.com/heroku/heroku-buildpack-python/blob/27abdfe7d7ad104dabceb45641415251e965671c/bin/release#L11-L18 BROKEN_CONFIG_VARS = { + BUILD_DIR: '/invalid-path', + C_INCLUDE_PATH: '/invalid-path', + # TODO: This currently fails xref: https://github.com/heroku/heroku-buildpack-python/issues/1886 + # CACHE_DIR: '/invalid-path', + CPLUS_INCLUDE_PATH: '/invalid-path', + ENV_DIR: '/invalid-path', LD_LIBRARY_PATH: '/invalid-path', LIBRARY_PATH: '/invalid-path', PATH: '/invalid-path', + PKG_CONFIG_PATH: '/invalid-path', PYTHONHOME: '/invalid-path', PYTHONPATH: '/invalid-path', }.freeze RSpec.describe 'Django support' do - describe 'collectstatic' do - context 'when building a Django project' do - let(:app) { Hatchet::Runner.new('python-getting-started', config: BROKEN_CONFIG_VARS) } - - it 'runs collectstatic' do - app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) - remote: -----> \\$ python manage.py collectstatic --noinput - remote: WARNING:root:No DATABASE_URL environment variable set, and so no databases setup - remote: \\d+ static files? copied to '/tmp/build_.*/staticfiles', \\d+ post-processed. - REGEX - end + context 'when building latest Django with the app nested inside a subfolder' do + # Also tests that app config vars are passed to the 'manage.py' script invocations. + let(:config) { { EXPECTED_ENV_VAR: '1' } } + let(:app) { Hatchet::Runner.new('spec/fixtures/django_staticfiles_latest_django', config:) } + + it 'runs collectstatic' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: Successfully installed Django-.+ + remote: -----> \\$ python backend/manage.py collectstatic --noinput + remote: \\{'BUILDPACK_LOG_FILE': '/dev/null', + remote: 'BUILD_DIR': '/tmp/build_\\w+', + remote: 'CACHE_DIR': '/tmp/codon/tmp/cache', + remote: 'CPLUS_INCLUDE_PATH': '/app/.heroku/python/include', + remote: 'C_INCLUDE_PATH': '/app/.heroku/python/include', + remote: 'DJANGO_SETTINGS_MODULE': 'testproject.settings', + remote: 'ENV_DIR': '/tmp/.+', + remote: 'EXPECTED_ENV_VAR': '1', + remote: 'HOME': '/app', + remote: 'LANG': 'en_US.UTF-8', + remote: 'LD_LIBRARY_PATH': '/app/.heroku/python/lib', + remote: 'LIBRARY_PATH': '/app/.heroku/python/lib', + remote: 'PATH': '/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', + remote: 'PKG_CONFIG_PATH': '/app/.heroku/python/lib/pkg-config', + remote: 'PWD': '/tmp/build_\\w+', + remote: 'PYTHONPATH': '\\.', + remote: 'PYTHONUNBUFFERED': '1', + remote: 'SOURCE_VERSION': '.+', + remote: 'STACK': '#{app.stack}'\\} + remote: \\['/tmp/build_\\w+/backend', + remote: '/tmp/build_\\w+', + remote: '/app/.heroku/python/lib/python313.zip', + remote: '/app/.heroku/python/lib/python3.13', + remote: '/app/.heroku/python/lib/python3.13/lib-dynload', + remote: '/app/.heroku/python/lib/python3.13/site-packages'\\] + remote: 1 static file copied to '/tmp/build_\\w+/backend/staticfiles'. + remote: + remote: -----> Saving cache + REGEX end end + end - context 'when Django is installed but manage.py does not exist' do - let(:app) { Hatchet::Runner.new('spec/fixtures/django_latest') } + context 'when building legacy Django with broken env vars set' do + # Also tests that app config vars are passed to the 'manage.py' script invocations. + let(:config) { BROKEN_CONFIG_VARS.merge(EXPECTED_ENV_VAR: '1') } + let(:app) { Hatchet::Runner.new('spec/fixtures/django_staticfiles_legacy_django', config:) } - it 'skips collectstatic' do - app.deploy do |app| - expect(app.output).to include('Skipping Django collectstatic since no manage.py file found.') - expect(app.output).not_to include('manage.py collectstatic') - end + it 'runs collectstatic' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: Successfully installed Django-.+ + remote: -----> \\$ python manage.py collectstatic --noinput + remote: \\{'BUILDPACK_LOG_FILE': '/dev/null', + remote: 'BUILD_DIR': '/invalid-path', + remote: 'CACHE_DIR': '/tmp/codon/tmp/cache', + remote: 'CPLUS_INCLUDE_PATH': '/invalid-path', + remote: 'C_INCLUDE_PATH': '/invalid-path', + remote: 'DJANGO_SETTINGS_MODULE': 'testproject.settings', + remote: 'ENV_DIR': '/invalid-path', + remote: 'EXPECTED_ENV_VAR': '1', + remote: 'HOME': '/app', + remote: 'LANG': 'en_US.UTF-8', + remote: 'LD_LIBRARY_PATH': '/app/.heroku/python/lib', + remote: 'LIBRARY_PATH': '/app/.heroku/python/lib', + remote: 'PATH': '/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', + remote: 'PKG_CONFIG_PATH': '/invalid-path', + remote: 'PWD': '/tmp/build_\\w+', + remote: 'PYTHONPATH': '/invalid-path', + remote: 'PYTHONUNBUFFERED': '1', + remote: 'SOURCE_VERSION': '.+', + remote: 'STACK': '#{app.stack}'\\} + remote: \\['/tmp/build_\\w+', + remote: '/invalid-path', + remote: '/app/.heroku/python/lib/python39.zip', + remote: '/app/.heroku/python/lib/python3.9', + remote: '/app/.heroku/python/lib/python3.9/lib-dynload', + remote: '/app/.heroku/python/lib/python3.9/site-packages'\\] + remote: 1 static file copied to '/tmp/build_\\w+/staticfiles'. + remote: + remote: -----> Saving cache + REGEX end end + end + + context 'when Django is installed but manage.py does not exist' do + let(:app) { Hatchet::Runner.new('spec/fixtures/django_no_manage_py') } - context 'when DISABLE_COLLECTSTATIC=1' do - let(:app) do - Hatchet::Runner.new('spec/fixtures/django_latest', config: { 'DISABLE_COLLECTSTATIC' => '1' }) + it 'skips collectstatic' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: Successfully installed Django-.+ + remote: -----> Skipping Django collectstatic since no manage.py file found. + remote: -----> Saving cache + REGEX end + end + end + + context 'when DISABLE_COLLECTSTATIC=1' do + let(:config) { { DISABLE_COLLECTSTATIC: '1' } } + let(:app) { Hatchet::Runner.new('spec/fixtures/django_invalid_settings_module', config:) } - it 'skips collectstatic' do - app.deploy do |app| - expect(app.output).to include('Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set.') - expect(app.output).not_to include('manage.py collectstatic') - end + it 'skips collectstatic' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: Successfully installed Django-.+ + remote: -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. + remote: -----> Saving cache + REGEX end end + end - context 'when DISABLE_COLLECTSTATIC=0' do - let(:app) { Hatchet::Runner.new('python-getting-started', config: { 'DISABLE_COLLECTSTATIC' => '0' }) } + context 'when .heroku/collectstatic_disabled exists' do + let(:app) { Hatchet::Runner.new('spec/fixtures/django_collectstatic_disabled_file') } - it 'still runs collectstatic' do - app.deploy do |app| - expect(app.output).to include('manage.py collectstatic') - end + it 'skips collectstatic with a deprecation warning' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: Successfully installed Django-.+ + remote: -----> Skipping Django collectstatic since the file '.heroku/collectstatic_disabled' exists. + remote: + remote: ! Warning: The .heroku/collectstatic_disabled file is deprecated. + remote: ! + remote: ! Please remove the file and set the env var DISABLE_COLLECTSTATIC=1 instead. + remote: + remote: -----> Saving cache + REGEX end end + end - context 'when DISABLE_COLLECTSTATIC is null' do - let(:app) { Hatchet::Runner.new('python-getting-started', config: { 'DISABLE_COLLECTSTATIC' => '' }) } + # TODO: Backport the Python CNB implementation that allows skipping collectstatic automatically for this case. + context 'when Django and manage.py exist but the Django staticfiles app is not enabled' do + let(:app) { Hatchet::Runner.new('spec/fixtures/django_staticfiles_app_not_enabled', allow_failure: true) } - it 'still runs collectstatic' do - app.deploy do |app| - expect(app.output).to include('manage.py collectstatic') - end + it 'fails collectstatic' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: Successfully installed Django-.+ + remote: -----> \\$ python manage.py collectstatic --noinput + remote: Unknown command: 'collectstatic' + remote: Type 'manage.py help' for usage. + remote: + remote: + remote: ! Error: Unable to generate Django static files. + remote: ! + remote: ! The 'python manage.py collectstatic --noinput' Django + remote: ! management command to generate static files failed. + remote: ! + remote: ! See the traceback above for details. + remote: ! + remote: ! You may need to update application code to resolve this error. + remote: ! Or, you can disable collectstatic for this application: + remote: ! + remote: ! \\$ heroku config:set DISABLE_COLLECTSTATIC=1 + remote: ! + remote: ! https://devcenter.heroku.com/articles/django-assets + remote: + remote: ! Push rejected, failed to compile Python app. + REGEX end end + end - context 'when .heroku/collectstatic_disabled exists' do - let(:app) { Hatchet::Runner.new('spec/fixtures/django_collectstatic_disabled_file') } - - it 'skips collectstatic with a deprecation warning' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Skipping Django collectstatic since the file '.heroku/collectstatic_disabled' exists. - remote: - remote: ! Warning: The .heroku/collectstatic_disabled file is deprecated. - remote: ! - remote: ! Please remove the file and set the env var DISABLE_COLLECTSTATIC=1 instead. - remote: - OUTPUT - expect(app.output).not_to include('manage.py collectstatic') - end + # For now this case produces the same error message as the one below, but once we backport the + # Python CNB implementation that will change, so we want a dedicated test for this case. + context 'when manage.py is configured with an invalid settings module' do + let(:app) { Hatchet::Runner.new('spec/fixtures/django_invalid_settings_module', allow_failure: true) } + + it 'fails collectstatic with an informative error message' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + remote: -----> \\$ python manage.py collectstatic --noinput + remote: Traceback \\(most recent call last\\): + remote: .+ + remote: ModuleNotFoundError: No module named 'nonexistent-module' + remote: + remote: + remote: ! Error: Unable to generate Django static files. + remote: ! + remote: ! The 'python manage.py collectstatic --noinput' Django + remote: ! management command to generate static files failed. + remote: ! + remote: ! See the traceback above for details. + remote: ! + remote: ! You may need to update application code to resolve this error. + remote: ! Or, you can disable collectstatic for this application: + remote: ! + remote: ! \\$ heroku config:set DISABLE_COLLECTSTATIC=1 + remote: ! + remote: ! https://devcenter.heroku.com/articles/django-assets + remote: + remote: ! Push rejected, failed to compile Python app. + REGEX end end + end + + context 'when the staticfiles app is misconfigured and DEBUG_COLLECTSTATIC=1' do + let(:config) { { DEBUG_COLLECTSTATIC: '1' } } + let(:app) { Hatchet::Runner.new('spec/fixtures/django_staticfiles_misconfigured', config:, allow_failure: true) } - context 'when building a broken Django project' do - let(:app) { Hatchet::Runner.new('spec/fixtures/django_broken_project', allow_failure: true) } - - it 'fails collectstatic with an informative error message' do - app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) - remote: -----> \\$ python manage.py collectstatic --noinput - remote: Traceback \\(most recent call last\\): - remote: .+ - remote: ModuleNotFoundError: No module named 'gettingstarted' - remote: - remote: - remote: ! Error: Unable to generate Django static files. - remote: ! - remote: ! The 'python manage.py collectstatic --noinput' Django - remote: ! management command to generate static files failed. - remote: ! - remote: ! See the traceback above for details. - remote: ! - remote: ! You may need to update application code to resolve this error. - remote: ! Or, you can disable collectstatic for this application: - remote: ! - remote: ! \\$ heroku config:set DISABLE_COLLECTSTATIC=1 - remote: ! - remote: ! https://devcenter.heroku.com/articles/django-assets - remote: - remote: ! Push rejected, failed to compile Python app. - REGEX - end + # TODO: Sort the displayed env vars to make the order deterministic and then we can test more of the output. + it 'fails collectstatic with an informative error message and prints env vars' do + app.deploy do |app| + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + remote: -----> \\$ python manage.py collectstatic --noinput + remote: Traceback \\(most recent call last\\): + remote: .+ + remote: django.core.exceptions.ImproperlyConfigured: You're using the staticfiles app without having set the required STATIC_URL setting. + remote: + remote: + remote: ! Error: Unable to generate Django static files. + remote: ! + remote: ! The 'python manage.py collectstatic --noinput' Django + remote: ! management command to generate static files failed. + remote: ! + remote: ! See the traceback above for details. + remote: ! + remote: ! You may need to update application code to resolve this error. + remote: ! Or, you can disable collectstatic for this application: + remote: ! + remote: ! \\$ heroku config:set DISABLE_COLLECTSTATIC=1 + remote: ! + remote: ! https://devcenter.heroku.com/articles/django-assets + remote: + remote: + remote: \\*\\*\\*\\*\\*\\* Collectstatic environment variables: + remote: + remote: .+ + remote: ! Push rejected, failed to compile Python app. + REGEX end end end From d0d55d06c4e0b312ecf0d13e7168de6e22bad1d4 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 1 Sep 2025 12:41:27 +0100 Subject: [PATCH 210/406] Fix collectstatic and NLTK downloader for apps with `CACHE_DIR` set (#1888) By default, user-provided env vars (app config vars) are not loaded into the buildpack environment by the build system, and instead buildpacks must load them explicitly when required: https://devcenter.heroku.com/articles/buildpack-api#bin-compile In general, this buildpack (like many of the others) tries to limit the places where user-provided env vars are loaded, in order to prevent them from breaking the build process. However, there are key parts of the build where they must be loaded, so that the app runs as expected. For example, when running the Django collectstatic command (to generate the app's static assets), the Django app's config will be loaded, and parts of it may reference/depend upon the app config vars. For some time, the entire collectstatic step was run in a subshell with the user config vars loaded, which meant there was a chance for env var shadowing. In particular, if an app had the env var `CACHE_DIR` set, it would cause the build data store writes to fail. Until the recent `bin/report` refactor these failing writes were silently ignored, however, those failures now (intentionally) are surfaced, which caused the build to fail as seen in #1886. The Django collectstatic and NLTK downloader steps are the last two parts of the buildpack that haven't been refactored and moved to `lib/` and so are still using the old/fragile implementations written many years ago. I will be refactoring them more fully in the future, but for now the `CACHE_DIR` issue has been resolved by more narrowly scoping where user provided env vars are loaded. Fixes #1886. GUS-W-19443067. --- CHANGELOG.md | 1 + bin/compile | 6 ++++-- bin/steps/collectstatic | 11 +++++++++-- bin/steps/nltk | 5 +++-- spec/hatchet/django_spec.rb | 5 ++--- vendor/buildpack-stdlib_v8.sh | 2 +- 6 files changed, 20 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b9de51635..15faa2f08 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Fixed Django collectstatic and NLTK downloader support for apps that use config vars that shadow internal buildpack variable names (such as `CACHE_DIR`). ([#1888](https://github.com/heroku/heroku-buildpack-python/pull/1888)) ## [v303] - 2025-08-26 diff --git a/bin/compile b/bin/compile index 775fab6ca..7f9a84b80 100755 --- a/bin/compile +++ b/bin/compile @@ -207,13 +207,15 @@ build_data::set_duration "dependencies_install_duration" "${dependencies_install # Support for NLTK corpora. nltk_downloader_start_time=$(build_data::current_unix_realtime) -sub_env "${BUILDPACK_DIR}/bin/steps/nltk" +# TODO: Migrate this script to functions under `lib/` and stop running it in a subshell. +"${BUILDPACK_DIR}/bin/steps/nltk" build_data::set_duration "nltk_downloader_duration" "${nltk_downloader_start_time}" # Django collectstatic support. # The buildpack automatically runs collectstatic for Django applications. collectstatic_start_time=$(build_data::current_unix_realtime) -sub_env "${BUILDPACK_DIR}/bin/steps/collectstatic" +# TODO: Migrate this script to functions under `lib/` and stop running it in a subshell. +"${BUILDPACK_DIR}/bin/steps/collectstatic" build_data::set_duration "django_collectstatic_duration" "${collectstatic_start_time}" # Programmatically create .profile.d script for application runtime environment variables. diff --git a/bin/steps/collectstatic b/bin/steps/collectstatic index b21affb8c..cd8f7b710 100755 --- a/bin/steps/collectstatic +++ b/bin/steps/collectstatic @@ -10,14 +10,21 @@ # - $DISABLE_COLLECTSTATIC: disables this functionality. # - $DEBUG_COLLECTSTATIC: upon failure, print out environment variables. -# This script is run in a subshell via sub_env so doesn't inherit the options/vars/utils from `bin/compile`. -# TODO: Stop running this script in a subshell. +# This script is run in a subshell so doesn't inherit the options/vars/utils from `bin/compile`. +# TODO: Migrate this script to functions under `lib/` and stop running the entire script in a subshell, +# and instead only the parts that need the user-provided config vars loaded. set -euo pipefail BUILDPACK_DIR=$(cd "$(dirname "$(dirname "$(dirname "${BASH_SOURCE[0]}")")")" && pwd) source "${BUILDPACK_DIR}/bin/utils" source "${BUILDPACK_DIR}/lib/build_data.sh" source "${BUILDPACK_DIR}/lib/output.sh" +# This must be after the imports above, to prevent user-provided config vars from overwriting +# env vars used by the buildpack such as `CACHE_DIR`: +# https://github.com/heroku/heroku-buildpack-python/issues/972 +# TODO: Reduce the scope of this even further, as part of refactoring this file. +export_env "${ENV_DIR}" + if [[ -f .heroku/collectstatic_disabled ]]; then output::step "Skipping Django collectstatic since the file '.heroku/collectstatic_disabled' exists." output::warning <<-'EOF' diff --git a/bin/steps/nltk b/bin/steps/nltk index 24ee8a1ff..a1317ebf7 100755 --- a/bin/steps/nltk +++ b/bin/steps/nltk @@ -1,7 +1,7 @@ #!/usr/bin/env bash # This script is run in a subshell via sub_env so doesn't inherit the options/vars/utils from `bin/compile`. -# TODO: Stop running this script in a subshell. +# TODO: Migrate this script to functions under `lib/` and stop running the entire script in a subshell. set -euo pipefail BUILDPACK_DIR=$(cd "$(dirname "$(dirname "$(dirname "${BASH_SOURCE[0]}")")")" && pwd) source "${BUILDPACK_DIR}/bin/utils" @@ -27,7 +27,8 @@ if is_module_available 'nltk'; then nltk_data_dir="/app/.heroku/python/nltk_data" - if ! python -m nltk.downloader -d "${nltk_data_dir}" "${nltk_packages[@]}" |& output::indent; then + # TODO: Does this even need user-provided env vars, or can we remove the sub_env usage here? + if ! sub_env python -m nltk.downloader -d "${nltk_data_dir}" "${nltk_packages[@]}" |& output::indent; then output::error <<-EOF Error: Unable to download NLTK data. diff --git a/spec/hatchet/django_spec.rb b/spec/hatchet/django_spec.rb index 1821ed421..c41eeba2e 100644 --- a/spec/hatchet/django_spec.rb +++ b/spec/hatchet/django_spec.rb @@ -10,8 +10,7 @@ BROKEN_CONFIG_VARS = { BUILD_DIR: '/invalid-path', C_INCLUDE_PATH: '/invalid-path', - # TODO: This currently fails xref: https://github.com/heroku/heroku-buildpack-python/issues/1886 - # CACHE_DIR: '/invalid-path', + CACHE_DIR: '/invalid-path', CPLUS_INCLUDE_PATH: '/invalid-path', ENV_DIR: '/invalid-path', LD_LIBRARY_PATH: '/invalid-path', @@ -78,7 +77,7 @@ remote: -----> \\$ python manage.py collectstatic --noinput remote: \\{'BUILDPACK_LOG_FILE': '/dev/null', remote: 'BUILD_DIR': '/invalid-path', - remote: 'CACHE_DIR': '/tmp/codon/tmp/cache', + remote: 'CACHE_DIR': '/invalid-path', remote: 'CPLUS_INCLUDE_PATH': '/invalid-path', remote: 'C_INCLUDE_PATH': '/invalid-path', remote: 'DJANGO_SETTINGS_MODULE': 'testproject.settings', diff --git a/vendor/buildpack-stdlib_v8.sh b/vendor/buildpack-stdlib_v8.sh index e2fdec6a3..ad8558f06 100755 --- a/vendor/buildpack-stdlib_v8.sh +++ b/vendor/buildpack-stdlib_v8.sh @@ -46,7 +46,7 @@ export_env() { local env_dir="${1:-${ENV_DIR}}" local whitelist="${2:-}" local blacklist - blacklist="$(_env_blacklist "${3}")" + blacklist="$(_env_blacklist "${3:-}")" if [[ -d "${env_dir}" ]]; then # Environment variable names won't contain characters affected by: # shellcheck disable=SC2045 From 8cbd8779f4fa2babe24b9829aa9ee43ff7103552 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Mon, 1 Sep 2025 13:00:42 +0000 Subject: [PATCH 211/406] Prepare release v304 (#1889) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 15faa2f08..c00d2e8b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v304] - 2025-09-01 + - Fixed Django collectstatic and NLTK downloader support for apps that use config vars that shadow internal buildpack variable names (such as `CACHE_DIR`). ([#1888](https://github.com/heroku/heroku-buildpack-python/pull/1888)) ## [v303] - 2025-08-26 @@ -1311,7 +1314,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v303...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v304...main +[v304]: https://github.com/heroku/heroku-buildpack-python/compare/v303...v304 [v303]: https://github.com/heroku/heroku-buildpack-python/compare/v302...v303 [v302]: https://github.com/heroku/heroku-buildpack-python/compare/v301...v302 [v301]: https://github.com/heroku/heroku-buildpack-python/compare/v300...v301 From f557e8eb89a22ef567e92ddee726d8491e52a559 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 2 Sep 2025 10:56:45 +0100 Subject: [PATCH 212/406] Output a warning if an existing virtual environment is found (#1890) Since: - Python virtual environments are specific to a single machine (they contain OS and architecture specific binaries, and hardcoded local paths), so should never be copied from machine to machine. - Including them at best bloats the slug for no reason, and at worst can cause hard to debug errors if the package manager auto-detects them and uses them instead of the Python install managed by this buildpack (as seen in https://heroku.support/1604888) A venv can end up in the app source either if someone forgets to `.gitignore` it, but also if they use an alternative deployment means that bypasses Git (such as Travis dpl), and haven't used another means to exclude unwanted files. This will be made an error fairly soon (once metrics confirm how widespread the pattern is, or whether there are third-party buildpacks that create a venv during the build). GUS-W-19127248. --- CHANGELOG.md | 1 + bin/compile | 5 ++- lib/checks.sh | 33 ++++++++++++++ spec/fixtures/venv_in_app_source/.gitignore | 2 + .../venv_in_app_source/.python-version | 1 + .../venv_in_app_source/.venv/bin/python | 7 +++ .../venv_in_app_source/.venv/pyvenv.cfg | 1 + .../venv_in_app_source/requirements.txt | 0 spec/hatchet/checks_spec.rb | 43 ++++++++++++++++++- 9 files changed, 89 insertions(+), 4 deletions(-) create mode 100644 spec/fixtures/venv_in_app_source/.gitignore create mode 100644 spec/fixtures/venv_in_app_source/.python-version create mode 100644 spec/fixtures/venv_in_app_source/.venv/bin/python create mode 100644 spec/fixtures/venv_in_app_source/.venv/pyvenv.cfg create mode 100644 spec/fixtures/venv_in_app_source/requirements.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index c00d2e8b0..3118e17ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Added a warning if an existing Python virtual environment is found in the build directory at `.venv/` or `venv/`. In the future this warning will be made an error. ([#1890](https://github.com/heroku/heroku-buildpack-python/pull/1890)) ## [v304] - 2025-09-01 diff --git a/bin/compile b/bin/compile index 7f9a84b80..a81032b02 100755 --- a/bin/compile +++ b/bin/compile @@ -98,9 +98,10 @@ cd "${BUILD_DIR}" # Runs a `bin/pre_compile` script if found in the app source, allowing build customisation. hooks::run_hook "pre_compile" -# This check must be after the pre_compile hook, so that we can check not only the original -# app source, but also that the hook hasn't written to '.heroku/python/' either. +# These checks are after the pre_compile hook, so that we can check not only the original +# app source, but also that the hook hasn't written to these locations either. checks::existing_python_dir_present "${BUILD_DIR}" +checks::existing_venv_dir_present "${BUILD_DIR}" package_manager="$(package_manager::determine_package_manager "${BUILD_DIR}")" build_data::set_string "package_manager" "${package_manager}" diff --git a/lib/checks.sh b/lib/checks.sh index d2757fdbb..dd0a75221 100644 --- a/lib/checks.sh +++ b/lib/checks.sh @@ -102,3 +102,36 @@ function checks::existing_python_dir_present() { exit 1 fi } + +function checks::existing_venv_dir_present() { + local build_dir="${1}" + + for venv_name in ".venv" "venv"; do + if [[ -f "${build_dir}/${venv_name}/pyvenv.cfg" ]]; then + output::warning <<-EOF + Warning: Existing '${venv_name}/' directory found. + + Your app's source code contains an existing directory named + '${venv_name}/', which looks like a Python virtual environment: + + $(find "${venv_name}/" -maxdepth 2 | sort || true) + + Including a virtual environment directory in your app source + isn't supported since the files within it are specific to a + single machine and so won't work when run somewhere else. + + If you've committed a '${venv_name}/' directory to your Git repo, you + must delete it and add the directory to your .gitignore file: + https://docs.github.com/en/get-started/git-basics/ignoring-files + + If the directory was created by a 'bin/pre_compile' hook or an + earlier buildpack, you must update them to create the virtual + environment in a different location. + + In future versions of the buildpack, this warning will be turned + into an error. + EOF + build_data::set_string "existing_venv_dir_present" "${venv_name}" + fi + done +} diff --git a/spec/fixtures/venv_in_app_source/.gitignore b/spec/fixtures/venv_in_app_source/.gitignore new file mode 100644 index 000000000..04e429ac0 --- /dev/null +++ b/spec/fixtures/venv_in_app_source/.gitignore @@ -0,0 +1,2 @@ +# Overrides the repo root's .gitignore, which includes `.venv/` +!* diff --git a/spec/fixtures/venv_in_app_source/.python-version b/spec/fixtures/venv_in_app_source/.python-version new file mode 100644 index 000000000..24ee5b1be --- /dev/null +++ b/spec/fixtures/venv_in_app_source/.python-version @@ -0,0 +1 @@ +3.13 diff --git a/spec/fixtures/venv_in_app_source/.venv/bin/python b/spec/fixtures/venv_in_app_source/.venv/bin/python new file mode 100644 index 000000000..03879ebe8 --- /dev/null +++ b/spec/fixtures/venv_in_app_source/.venv/bin/python @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +# This file emulates a Python venv having been committed to the app's Git repo. + +set -euo pipefail + +exit 0 diff --git a/spec/fixtures/venv_in_app_source/.venv/pyvenv.cfg b/spec/fixtures/venv_in_app_source/.venv/pyvenv.cfg new file mode 100644 index 000000000..2cefa6c36 --- /dev/null +++ b/spec/fixtures/venv_in_app_source/.venv/pyvenv.cfg @@ -0,0 +1 @@ +# This file emulates a Python venv having been committed to the app's Git repo. diff --git a/spec/fixtures/venv_in_app_source/requirements.txt b/spec/fixtures/venv_in_app_source/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/hatchet/checks_spec.rb b/spec/hatchet/checks_spec.rb index c83f3c14f..2972c8947 100644 --- a/spec/hatchet/checks_spec.rb +++ b/spec/hatchet/checks_spec.rb @@ -7,7 +7,7 @@ let(:buildpacks) { %i[default default] } let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_unspecified', buildpacks:, allow_failure: true) } - it 'fails detection' do + it 'fails the build with an informative error message' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected @@ -38,7 +38,7 @@ context 'when the app source contains a broken Python install' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_in_app_source', allow_failure: true) } - it 'fails detection' do + it 'fails the build with an informative error message' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected @@ -70,4 +70,43 @@ end end end + + context 'when the app source contains a venv directory' do + let(:app) { Hatchet::Runner.new('spec/fixtures/venv_in_app_source') } + + it 'outputs a warning before continuing to build' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: + remote: ! Warning: Existing '.venv/' directory found. + remote: ! + remote: ! Your app's source code contains an existing directory named + remote: ! '.venv/', which looks like a Python virtual environment: + remote: ! + remote: ! .venv/ + remote: ! .venv/bin + remote: ! .venv/bin/python + remote: ! .venv/pyvenv.cfg + remote: ! + remote: ! Including a virtual environment directory in your app source + remote: ! isn't supported since the files within it are specific to a + remote: ! single machine and so won't work when run somewhere else. + remote: ! + remote: ! If you've committed a '.venv/' directory to your Git repo, you + remote: ! must delete it and add the directory to your .gitignore file: + remote: ! https://docs.github.com/en/get-started/git-basics/ignoring-files + remote: ! + remote: ! If the directory was created by a 'bin/pre_compile' hook or an + remote: ! earlier buildpack, you must update them to create the virtual + remote: ! environment in a different location. + remote: ! + remote: ! In future versions of the buildpack, this warning will be turned + remote: ! into an error. + remote: + remote: -----> Using Python 3.13 specified in .python-version + OUTPUT + end + end + end end From b525a25d058753ee504503ec0cfd204d292b0717 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 2 Sep 2025 10:18:58 +0000 Subject: [PATCH 213/406] Prepare release v305 (#1892) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3118e17ac..2126876a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v305] - 2025-09-02 + - Added a warning if an existing Python virtual environment is found in the build directory at `.venv/` or `venv/`. In the future this warning will be made an error. ([#1890](https://github.com/heroku/heroku-buildpack-python/pull/1890)) ## [v304] - 2025-09-01 @@ -1315,7 +1318,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v304...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v305...main +[v305]: https://github.com/heroku/heroku-buildpack-python/compare/v304...v305 [v304]: https://github.com/heroku/heroku-buildpack-python/compare/v303...v304 [v303]: https://github.com/heroku/heroku-buildpack-python/compare/v302...v303 [v302]: https://github.com/heroku/heroku-buildpack-python/compare/v301...v302 From 1172f9e4216a6f2e97101b2fef4955db1acc68e0 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 8 Sep 2025 13:48:56 +0100 Subject: [PATCH 214/406] Delegate Ruby whitespace handling to RuboCop (#1893) We use heredocs to assert against Hatchet output, and sometimes that output contains trailing newlines. These trailing newlines are permitted in the RuboCop config, however, until now were being removed if people used tooling locally that used the `.editorconfig` instead (such as the VSCode `.editorconfig` extension). See also: https://github.com/heroku/heroku-buildpack-dotnet/pull/118 GUS-W-19580687. --- .editorconfig | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.editorconfig b/.editorconfig index 0fa1b37cf..0d78a23d9 100644 --- a/.editorconfig +++ b/.editorconfig @@ -8,6 +8,11 @@ indent_style = space insert_final_newline = true trim_trailing_whitespace = true +[*.rb] +# Required since we use heredocs to assert against Hatchet output, and sometimes that output +# contains trailing newlines. Our RuboCop config ensures these are only allowed in heredocs. +trim_trailing_whitespace = false + [*.sh] binary_next_line = true # We sadly have to use tabs in shell scripts otherwise we can't indent here documents: From d60370be3c95611a2d659d3b5748ee90ba7f23bc Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 9 Sep 2025 10:11:03 +0100 Subject: [PATCH 215/406] Improve env var and package manager tests (#1895) The buildpack configures env vars for: - pre/post-compile hooks - Django collectstatic - subsequent buildpacks (via the `export` file) - app run-time (via `profile.d/` scripts) The tests for all of these have been improved, by backporting some of the new test matrix strategies from the Python CNB. In particular, we now test the run-time and hooks env var values with all four supported package managers (and not just when using pip). In addition, we now test that Poetry and uv aren't installed at run-time. GUS-W-19591873. --- spec/fixtures/ci_pip/bin/post_compile | 1 + spec/fixtures/ci_pip/bin/print-env-vars.sh | 2 +- spec/fixtures/ci_pipenv/bin/post_compile | 1 + spec/fixtures/ci_pipenv/bin/print-env-vars.sh | 2 +- spec/fixtures/ci_poetry/bin/post_compile | 1 + spec/fixtures/ci_poetry/bin/print-env-vars.sh | 2 +- spec/fixtures/ci_uv/bin/post_compile | 1 + spec/fixtures/ci_uv/bin/print-env-vars.sh | 2 +- .../backend/testproject/settings.py | 17 ++-- .../testproject/settings.py | 17 ++-- spec/fixtures/hooks/bin/post_compile | 7 -- spec/fixtures/hooks/bin/pre_compile | 7 -- spec/fixtures/hooks/bin/print-env-vars.sh | 10 --- .../requirements.txt | 0 .../bin/pre_compile | 0 .../requirements.txt | 0 .../bin/post_compile | 0 .../bin/pre_compile | 7 ++ .../requirements.txt | 0 spec/fixtures/pip_basic/bin/compile | 2 +- spec/fixtures/pip_basic/bin/post_compile | 6 ++ spec/fixtures/pip_basic/bin/print-env-vars.sh | 5 ++ spec/fixtures/pipenv_basic/bin/compile | 2 +- spec/fixtures/pipenv_basic/bin/post_compile | 6 ++ .../pipenv_basic/bin/print-env-vars.sh | 5 ++ spec/fixtures/poetry_basic/bin/compile | 2 +- spec/fixtures/poetry_basic/bin/post_compile | 6 ++ .../poetry_basic/bin/print-env-vars.sh | 5 ++ spec/fixtures/uv_basic/bin/compile | 2 +- spec/fixtures/uv_basic/bin/post_compile | 6 ++ spec/fixtures/uv_basic/bin/print-env-vars.sh | 5 ++ spec/hatchet/ci_spec.rb | 20 ++++- spec/hatchet/django_spec.rb | 16 +--- spec/hatchet/hooks_spec.rb | 72 +++++------------ spec/hatchet/package_manager_spec.rb | 1 + spec/hatchet/pip_spec.rb | 49 ++++++++--- spec/hatchet/pipenv_spec.rb | 81 ++++++++++++++----- spec/hatchet/poetry_spec.rb | 69 ++++++++++++---- spec/hatchet/profile_d_scripts_spec.rb | 34 ++------ spec/hatchet/python_version_spec.rb | 9 ++- spec/hatchet/uv_spec.rb | 45 ++++++++++- spec/spec_helper.rb | 1 + 42 files changed, 335 insertions(+), 191 deletions(-) delete mode 100644 spec/fixtures/hooks/bin/post_compile delete mode 100644 spec/fixtures/hooks/bin/pre_compile delete mode 100755 spec/fixtures/hooks/bin/print-env-vars.sh delete mode 100644 spec/fixtures/hooks_failing_pre_compile/requirements.txt rename spec/fixtures/{hooks_failing_pre_compile => hooks_pre_compile_fail}/bin/pre_compile (100%) rename spec/fixtures/{hooks => hooks_pre_compile_fail}/requirements.txt (100%) rename spec/fixtures/{hooks_failing_post_compile => hooks_pre_compile_pass_post_compile_fail}/bin/post_compile (100%) create mode 100644 spec/fixtures/hooks_pre_compile_pass_post_compile_fail/bin/pre_compile rename spec/fixtures/{hooks_failing_post_compile => hooks_pre_compile_pass_post_compile_fail}/requirements.txt (100%) create mode 100644 spec/fixtures/pip_basic/bin/post_compile create mode 100755 spec/fixtures/pip_basic/bin/print-env-vars.sh create mode 100644 spec/fixtures/pipenv_basic/bin/post_compile create mode 100755 spec/fixtures/pipenv_basic/bin/print-env-vars.sh create mode 100644 spec/fixtures/poetry_basic/bin/post_compile create mode 100755 spec/fixtures/poetry_basic/bin/print-env-vars.sh create mode 100644 spec/fixtures/uv_basic/bin/post_compile create mode 100755 spec/fixtures/uv_basic/bin/print-env-vars.sh diff --git a/spec/fixtures/ci_pip/bin/post_compile b/spec/fixtures/ci_pip/bin/post_compile index 15362f6b1..15dc8c2f3 100644 --- a/spec/fixtures/ci_pip/bin/post_compile +++ b/spec/fixtures/ci_pip/bin/post_compile @@ -2,4 +2,5 @@ set -euo pipefail +# This intentionally uses a relative path to test that the CWD is correct. exec bin/print-env-vars.sh diff --git a/spec/fixtures/ci_pip/bin/print-env-vars.sh b/spec/fixtures/ci_pip/bin/print-env-vars.sh index 9e0bebe6b..6cc32b420 100755 --- a/spec/fixtures/ci_pip/bin/print-env-vars.sh +++ b/spec/fixtures/ci_pip/bin/print-env-vars.sh @@ -2,4 +2,4 @@ set -euo pipefail -printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|BUILD_DIR|CACHE_DIR|CI_NODE_.+|DYNO|ENV_DIR|HEROKU_TEST_RUN_.+|HOME|OLDPWD|PORT|PWD|SHLVL|STACK|TERM)=' +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|CI_NODE_.+|DYNO|HEROKU_TEST_RUN_.+|HOME|OLDPWD|PORT|PWD|SHLVL|STACK|TERM)=' diff --git a/spec/fixtures/ci_pipenv/bin/post_compile b/spec/fixtures/ci_pipenv/bin/post_compile index 15362f6b1..15dc8c2f3 100644 --- a/spec/fixtures/ci_pipenv/bin/post_compile +++ b/spec/fixtures/ci_pipenv/bin/post_compile @@ -2,4 +2,5 @@ set -euo pipefail +# This intentionally uses a relative path to test that the CWD is correct. exec bin/print-env-vars.sh diff --git a/spec/fixtures/ci_pipenv/bin/print-env-vars.sh b/spec/fixtures/ci_pipenv/bin/print-env-vars.sh index 9e0bebe6b..6cc32b420 100755 --- a/spec/fixtures/ci_pipenv/bin/print-env-vars.sh +++ b/spec/fixtures/ci_pipenv/bin/print-env-vars.sh @@ -2,4 +2,4 @@ set -euo pipefail -printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|BUILD_DIR|CACHE_DIR|CI_NODE_.+|DYNO|ENV_DIR|HEROKU_TEST_RUN_.+|HOME|OLDPWD|PORT|PWD|SHLVL|STACK|TERM)=' +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|CI_NODE_.+|DYNO|HEROKU_TEST_RUN_.+|HOME|OLDPWD|PORT|PWD|SHLVL|STACK|TERM)=' diff --git a/spec/fixtures/ci_poetry/bin/post_compile b/spec/fixtures/ci_poetry/bin/post_compile index 15362f6b1..15dc8c2f3 100644 --- a/spec/fixtures/ci_poetry/bin/post_compile +++ b/spec/fixtures/ci_poetry/bin/post_compile @@ -2,4 +2,5 @@ set -euo pipefail +# This intentionally uses a relative path to test that the CWD is correct. exec bin/print-env-vars.sh diff --git a/spec/fixtures/ci_poetry/bin/print-env-vars.sh b/spec/fixtures/ci_poetry/bin/print-env-vars.sh index 9e0bebe6b..6cc32b420 100755 --- a/spec/fixtures/ci_poetry/bin/print-env-vars.sh +++ b/spec/fixtures/ci_poetry/bin/print-env-vars.sh @@ -2,4 +2,4 @@ set -euo pipefail -printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|BUILD_DIR|CACHE_DIR|CI_NODE_.+|DYNO|ENV_DIR|HEROKU_TEST_RUN_.+|HOME|OLDPWD|PORT|PWD|SHLVL|STACK|TERM)=' +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|CI_NODE_.+|DYNO|HEROKU_TEST_RUN_.+|HOME|OLDPWD|PORT|PWD|SHLVL|STACK|TERM)=' diff --git a/spec/fixtures/ci_uv/bin/post_compile b/spec/fixtures/ci_uv/bin/post_compile index 15362f6b1..15dc8c2f3 100644 --- a/spec/fixtures/ci_uv/bin/post_compile +++ b/spec/fixtures/ci_uv/bin/post_compile @@ -2,4 +2,5 @@ set -euo pipefail +# This intentionally uses a relative path to test that the CWD is correct. exec bin/print-env-vars.sh diff --git a/spec/fixtures/ci_uv/bin/print-env-vars.sh b/spec/fixtures/ci_uv/bin/print-env-vars.sh index 9e0bebe6b..6cc32b420 100755 --- a/spec/fixtures/ci_uv/bin/print-env-vars.sh +++ b/spec/fixtures/ci_uv/bin/print-env-vars.sh @@ -2,4 +2,4 @@ set -euo pipefail -printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|BUILD_DIR|CACHE_DIR|CI_NODE_.+|DYNO|ENV_DIR|HEROKU_TEST_RUN_.+|HOME|OLDPWD|PORT|PWD|SHLVL|STACK|TERM)=' +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|CI_NODE_.+|DYNO|HEROKU_TEST_RUN_.+|HOME|OLDPWD|PORT|PWD|SHLVL|STACK|TERM)=' diff --git a/spec/fixtures/django_staticfiles_latest_django/backend/testproject/settings.py b/spec/fixtures/django_staticfiles_latest_django/backend/testproject/settings.py index 75021f41d..91b07ab13 100644 --- a/spec/fixtures/django_staticfiles_latest_django/backend/testproject/settings.py +++ b/spec/fixtures/django_staticfiles_latest_django/backend/testproject/settings.py @@ -13,11 +13,18 @@ STATIC_ROOT = BASE_DIR / "staticfiles" STATIC_URL = "static/" -pprint({ - k: v - for k, v in os.environ.items() - if not k in {"_", "DYNO", "OLDPWD", "REQUEST_ID", "SHLVL"} -}) +ENV_VARS_TO_OMIT = { + "_", + "BUILDPACK_LOG_FILE", + "DYNO", + "HOME", + "OLDPWD", + "REQUEST_ID", + "SHLVL", + "SOURCE_VERSION", + "STACK", +} +pprint({k: v for k, v in os.environ.items() if k not in ENV_VARS_TO_OMIT}) print() pprint(sys.path) diff --git a/spec/fixtures/django_staticfiles_legacy_django/testproject/settings.py b/spec/fixtures/django_staticfiles_legacy_django/testproject/settings.py index 9547fe35a..ad916f8e8 100644 --- a/spec/fixtures/django_staticfiles_legacy_django/testproject/settings.py +++ b/spec/fixtures/django_staticfiles_legacy_django/testproject/settings.py @@ -16,11 +16,18 @@ # Older versions of Django require that `SECRET_KEY` is set when running collectstatic. SECRET_KEY = "example" -pprint({ - k: v - for k, v in os.environ.items() - if not k in {"_", "DYNO", "OLDPWD", "REQUEST_ID", "SHLVL"} -}) +ENV_VARS_TO_OMIT = { + "_", + "BUILDPACK_LOG_FILE", + "DYNO", + "HOME", + "OLDPWD", + "REQUEST_ID", + "SHLVL", + "SOURCE_VERSION", + "STACK", +} +pprint({k: v for k, v in os.environ.items() if k not in ENV_VARS_TO_OMIT}) print() pprint(sys.path) diff --git a/spec/fixtures/hooks/bin/post_compile b/spec/fixtures/hooks/bin/post_compile deleted file mode 100644 index 4bd8ed0e8..000000000 --- a/spec/fixtures/hooks/bin/post_compile +++ /dev/null @@ -1,7 +0,0 @@ -#!/usr/bin/env bash - -set -euo pipefail - -echo '~ post_compile ran with env vars:' -bin/print-env-vars.sh -echo '~ post_compile complete' diff --git a/spec/fixtures/hooks/bin/pre_compile b/spec/fixtures/hooks/bin/pre_compile deleted file mode 100644 index 9f3737bef..000000000 --- a/spec/fixtures/hooks/bin/pre_compile +++ /dev/null @@ -1,7 +0,0 @@ -#!/usr/bin/env bash - -set -euo pipefail - -echo '~ pre_compile ran with env vars:' -bin/print-env-vars.sh -echo '~ pre_compile complete' diff --git a/spec/fixtures/hooks/bin/print-env-vars.sh b/spec/fixtures/hooks/bin/print-env-vars.sh deleted file mode 100755 index af6afded8..000000000 --- a/spec/fixtures/hooks/bin/print-env-vars.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/usr/bin/env bash - -set -euo pipefail - -printenv | sort \ - | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|OLDPWD|REQUEST_ID|SHLVL)=' \ - | sed --regexp-extended \ - --expression 's#(=/tmp/build_)[^:/]+#\1#' \ - --expression 's#^(ENV_DIR=/tmp/).*#\1...#' \ - --expression 's#^(SOURCE_VERSION=).*#\1...#' diff --git a/spec/fixtures/hooks_failing_pre_compile/requirements.txt b/spec/fixtures/hooks_failing_pre_compile/requirements.txt deleted file mode 100644 index e69de29bb..000000000 diff --git a/spec/fixtures/hooks_failing_pre_compile/bin/pre_compile b/spec/fixtures/hooks_pre_compile_fail/bin/pre_compile similarity index 100% rename from spec/fixtures/hooks_failing_pre_compile/bin/pre_compile rename to spec/fixtures/hooks_pre_compile_fail/bin/pre_compile diff --git a/spec/fixtures/hooks/requirements.txt b/spec/fixtures/hooks_pre_compile_fail/requirements.txt similarity index 100% rename from spec/fixtures/hooks/requirements.txt rename to spec/fixtures/hooks_pre_compile_fail/requirements.txt diff --git a/spec/fixtures/hooks_failing_post_compile/bin/post_compile b/spec/fixtures/hooks_pre_compile_pass_post_compile_fail/bin/post_compile similarity index 100% rename from spec/fixtures/hooks_failing_post_compile/bin/post_compile rename to spec/fixtures/hooks_pre_compile_pass_post_compile_fail/bin/post_compile diff --git a/spec/fixtures/hooks_pre_compile_pass_post_compile_fail/bin/pre_compile b/spec/fixtures/hooks_pre_compile_pass_post_compile_fail/bin/pre_compile new file mode 100644 index 000000000..d1401761e --- /dev/null +++ b/spec/fixtures/hooks_pre_compile_pass_post_compile_fail/bin/pre_compile @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +set -euo pipefail + +echo '~ pre_compile ran with env vars:' +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|OLDPWD|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK)=' +echo '~ pre_compile complete' diff --git a/spec/fixtures/hooks_failing_post_compile/requirements.txt b/spec/fixtures/hooks_pre_compile_pass_post_compile_fail/requirements.txt similarity index 100% rename from spec/fixtures/hooks_failing_post_compile/requirements.txt rename to spec/fixtures/hooks_pre_compile_pass_post_compile_fail/requirements.txt diff --git a/spec/fixtures/pip_basic/bin/compile b/spec/fixtures/pip_basic/bin/compile index 0eecc561b..9dc156690 100755 --- a/spec/fixtures/pip_basic/bin/compile +++ b/spec/fixtures/pip_basic/bin/compile @@ -10,7 +10,7 @@ CACHE_DIR="${2}" cd "${BUILD_DIR}" -printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|OLDPWD|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK)=' +bin/print-env-vars.sh echo python -c 'import pprint, sys; pprint.pp(sys.path)' diff --git a/spec/fixtures/pip_basic/bin/post_compile b/spec/fixtures/pip_basic/bin/post_compile new file mode 100644 index 000000000..15dc8c2f3 --- /dev/null +++ b/spec/fixtures/pip_basic/bin/post_compile @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +set -euo pipefail + +# This intentionally uses a relative path to test that the CWD is correct. +exec bin/print-env-vars.sh diff --git a/spec/fixtures/pip_basic/bin/print-env-vars.sh b/spec/fixtures/pip_basic/bin/print-env-vars.sh new file mode 100755 index 000000000..ebc9ad735 --- /dev/null +++ b/spec/fixtures/pip_basic/bin/print-env-vars.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -euo pipefail + +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|OLDPWD|PORT|PS1|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK|TERM)=' diff --git a/spec/fixtures/pipenv_basic/bin/compile b/spec/fixtures/pipenv_basic/bin/compile index a9c2a9af7..306c295fc 100755 --- a/spec/fixtures/pipenv_basic/bin/compile +++ b/spec/fixtures/pipenv_basic/bin/compile @@ -10,7 +10,7 @@ CACHE_DIR="${2}" cd "${BUILD_DIR}" -printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|OLDPWD|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK)=' +bin/print-env-vars.sh echo python -c 'import pprint, sys; pprint.pp(sys.path)' diff --git a/spec/fixtures/pipenv_basic/bin/post_compile b/spec/fixtures/pipenv_basic/bin/post_compile new file mode 100644 index 000000000..15dc8c2f3 --- /dev/null +++ b/spec/fixtures/pipenv_basic/bin/post_compile @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +set -euo pipefail + +# This intentionally uses a relative path to test that the CWD is correct. +exec bin/print-env-vars.sh diff --git a/spec/fixtures/pipenv_basic/bin/print-env-vars.sh b/spec/fixtures/pipenv_basic/bin/print-env-vars.sh new file mode 100755 index 000000000..ebc9ad735 --- /dev/null +++ b/spec/fixtures/pipenv_basic/bin/print-env-vars.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -euo pipefail + +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|OLDPWD|PORT|PS1|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK|TERM)=' diff --git a/spec/fixtures/poetry_basic/bin/compile b/spec/fixtures/poetry_basic/bin/compile index af757a8b4..8b23ed9ce 100755 --- a/spec/fixtures/poetry_basic/bin/compile +++ b/spec/fixtures/poetry_basic/bin/compile @@ -10,7 +10,7 @@ CACHE_DIR="${2}" cd "${BUILD_DIR}" -printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|OLDPWD|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK)=' +bin/print-env-vars.sh echo python -c 'import pprint, sys; pprint.pp(sys.path)' diff --git a/spec/fixtures/poetry_basic/bin/post_compile b/spec/fixtures/poetry_basic/bin/post_compile new file mode 100644 index 000000000..15dc8c2f3 --- /dev/null +++ b/spec/fixtures/poetry_basic/bin/post_compile @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +set -euo pipefail + +# This intentionally uses a relative path to test that the CWD is correct. +exec bin/print-env-vars.sh diff --git a/spec/fixtures/poetry_basic/bin/print-env-vars.sh b/spec/fixtures/poetry_basic/bin/print-env-vars.sh new file mode 100755 index 000000000..ebc9ad735 --- /dev/null +++ b/spec/fixtures/poetry_basic/bin/print-env-vars.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -euo pipefail + +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|OLDPWD|PORT|PS1|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK|TERM)=' diff --git a/spec/fixtures/uv_basic/bin/compile b/spec/fixtures/uv_basic/bin/compile index 823440ee2..5a6fb09f4 100755 --- a/spec/fixtures/uv_basic/bin/compile +++ b/spec/fixtures/uv_basic/bin/compile @@ -10,7 +10,7 @@ CACHE_DIR="${2}" cd "${BUILD_DIR}" -printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|OLDPWD|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK)=' +bin/print-env-vars.sh echo python -c 'import pprint, sys; pprint.pp(sys.path)' diff --git a/spec/fixtures/uv_basic/bin/post_compile b/spec/fixtures/uv_basic/bin/post_compile new file mode 100644 index 000000000..15dc8c2f3 --- /dev/null +++ b/spec/fixtures/uv_basic/bin/post_compile @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +set -euo pipefail + +# This intentionally uses a relative path to test that the CWD is correct. +exec bin/print-env-vars.sh diff --git a/spec/fixtures/uv_basic/bin/print-env-vars.sh b/spec/fixtures/uv_basic/bin/print-env-vars.sh new file mode 100755 index 000000000..ebc9ad735 --- /dev/null +++ b/spec/fixtures/uv_basic/bin/print-env-vars.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -euo pipefail + +printenv | sort | grep -vE '^(_|BUILDPACK_LOG_FILE|DYNO|HOME|OLDPWD|PORT|PS1|PWD|REQUEST_ID|SHLVL|SOURCE_VERSION|STACK|TERM)=' diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index bd1d333b5..02c466ffa 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -20,10 +20,13 @@ Successfully installed .+ pytest-.+ typing-extensions-.+ -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running bin/post_compile hook + BUILD_DIR=/app + CACHE_DIR=/tmp/cache.+ CI=true CPLUS_INCLUDE_PATH=/app/.heroku/python/include C_INCLUDE_PATH=/app/.heroku/python/include DISABLE_COLLECTSTATIC=1 + ENV_DIR=/tmp/env.+ INSTALL_TEST=1 LANG=en_US.UTF-8 LC_ALL=C.UTF-8 @@ -92,10 +95,13 @@ Installing dependencies from Pipfile.lock \\(.+\\)... -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running bin/post_compile hook + BUILD_DIR=/app + CACHE_DIR=/tmp/cache.+ CI=true CPLUS_INCLUDE_PATH=/app/.heroku/python/include C_INCLUDE_PATH=/app/.heroku/python/include DISABLE_COLLECTSTATIC=1 + ENV_DIR=/tmp/env.+ INSTALL_TEST=1 LANG=en_US.UTF-8 LC_ALL=C.UTF-8 @@ -178,16 +184,19 @@ - Installing (pytest|typing-extensions) .+ -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running bin/post_compile hook + BUILD_DIR=/app + CACHE_DIR=/tmp/cache.+ CI=true CPLUS_INCLUDE_PATH=/app/.heroku/python/include C_INCLUDE_PATH=/app/.heroku/python/include DISABLE_COLLECTSTATIC=1 + ENV_DIR=/tmp/env.+ INSTALL_TEST=1 LANG=en_US.UTF-8 LC_ALL=C.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib - PATH=/tmp/cache.+/.heroku/python-poetry/bin:/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PATH=/tmp/cache\\w+/.heroku/python-poetry/bin:/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config POETRY_VIRTUALENVS_CREATE=false POETRY_VIRTUALENVS_USE_POETRY_PYTHON=true @@ -197,7 +206,7 @@ LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib - PATH=/app/.heroku/python/bin:/tmp/cache.+/.heroku/python-poetry/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PATH=/app/.heroku/python/bin:/tmp/cache\\w+/.heroku/python-poetry/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ POETRY_VIRTUALENVS_CREATE=false POETRY_VIRTUALENVS_USE_POETRY_PYTHON=true PYTHONHOME=/app/.heroku/python @@ -260,16 +269,19 @@ \\+ (pytest|typing-extensions)==.+ -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running bin/post_compile hook + BUILD_DIR=/app + CACHE_DIR=/tmp/cache.+ CI=true CPLUS_INCLUDE_PATH=/app/.heroku/python/include C_INCLUDE_PATH=/app/.heroku/python/include DISABLE_COLLECTSTATIC=1 + ENV_DIR=/tmp/env.+ INSTALL_TEST=1 LANG=en_US.UTF-8 LC_ALL=C.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib - PATH=/tmp/cache.+/.heroku/python-uv:/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PATH=/tmp/cache\\w+/.heroku/python-uv:/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 UV_NO_MANAGED_PYTHON=1 @@ -280,7 +292,7 @@ LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib - PATH=/app/.heroku/python/bin:/tmp/cache.+/.heroku/python-uv:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PATH=/app/.heroku/python/bin:/tmp/cache\\w+/.heroku/python-uv:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true diff --git a/spec/hatchet/django_spec.rb b/spec/hatchet/django_spec.rb index c41eeba2e..4f80d0995 100644 --- a/spec/hatchet/django_spec.rb +++ b/spec/hatchet/django_spec.rb @@ -32,15 +32,13 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: Successfully installed Django-.+ remote: -----> \\$ python backend/manage.py collectstatic --noinput - remote: \\{'BUILDPACK_LOG_FILE': '/dev/null', - remote: 'BUILD_DIR': '/tmp/build_\\w+', + remote: \\{'BUILD_DIR': '/tmp/build_\\w+', remote: 'CACHE_DIR': '/tmp/codon/tmp/cache', remote: 'CPLUS_INCLUDE_PATH': '/app/.heroku/python/include', remote: 'C_INCLUDE_PATH': '/app/.heroku/python/include', remote: 'DJANGO_SETTINGS_MODULE': 'testproject.settings', remote: 'ENV_DIR': '/tmp/.+', remote: 'EXPECTED_ENV_VAR': '1', - remote: 'HOME': '/app', remote: 'LANG': 'en_US.UTF-8', remote: 'LD_LIBRARY_PATH': '/app/.heroku/python/lib', remote: 'LIBRARY_PATH': '/app/.heroku/python/lib', @@ -48,9 +46,7 @@ remote: 'PKG_CONFIG_PATH': '/app/.heroku/python/lib/pkg-config', remote: 'PWD': '/tmp/build_\\w+', remote: 'PYTHONPATH': '\\.', - remote: 'PYTHONUNBUFFERED': '1', - remote: 'SOURCE_VERSION': '.+', - remote: 'STACK': '#{app.stack}'\\} + remote: 'PYTHONUNBUFFERED': '1'\\} remote: \\['/tmp/build_\\w+/backend', remote: '/tmp/build_\\w+', remote: '/app/.heroku/python/lib/python313.zip', @@ -75,15 +71,13 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: Successfully installed Django-.+ remote: -----> \\$ python manage.py collectstatic --noinput - remote: \\{'BUILDPACK_LOG_FILE': '/dev/null', - remote: 'BUILD_DIR': '/invalid-path', + remote: \\{'BUILD_DIR': '/invalid-path', remote: 'CACHE_DIR': '/invalid-path', remote: 'CPLUS_INCLUDE_PATH': '/invalid-path', remote: 'C_INCLUDE_PATH': '/invalid-path', remote: 'DJANGO_SETTINGS_MODULE': 'testproject.settings', remote: 'ENV_DIR': '/invalid-path', remote: 'EXPECTED_ENV_VAR': '1', - remote: 'HOME': '/app', remote: 'LANG': 'en_US.UTF-8', remote: 'LD_LIBRARY_PATH': '/app/.heroku/python/lib', remote: 'LIBRARY_PATH': '/app/.heroku/python/lib', @@ -91,9 +85,7 @@ remote: 'PKG_CONFIG_PATH': '/invalid-path', remote: 'PWD': '/tmp/build_\\w+', remote: 'PYTHONPATH': '/invalid-path', - remote: 'PYTHONUNBUFFERED': '1', - remote: 'SOURCE_VERSION': '.+', - remote: 'STACK': '#{app.stack}'\\} + remote: 'PYTHONUNBUFFERED': '1'\\} remote: \\['/tmp/build_\\w+', remote: '/invalid-path', remote: '/app/.heroku/python/lib/python39.zip', diff --git a/spec/hatchet/hooks_spec.rb b/spec/hatchet/hooks_spec.rb index eb20a7e83..e58a7969c 100644 --- a/spec/hatchet/hooks_spec.rb +++ b/spec/hatchet/hooks_spec.rb @@ -3,74 +3,44 @@ require_relative '../spec_helper' RSpec.describe 'Compile hooks' do - context 'when an app has bin/pre_compile and bin/post_compile scripts' do - let(:app) { Hatchet::Runner.new('spec/fixtures/hooks', config: { 'SOME_APP_CONFIG_VAR' => '1' }) } + # This spec skips testing the passing post_compile hook case, since that's already tested + # via the package manager and CI tests. - it 'runs the hooks with the correct environment' do + # Tests two of the four hooks result permutations in the same test to reduce end to end time. + context 'when an app has a passing bin/pre_compile and a failing bin/post_compile script' do + let(:config) { { SOME_APP_CONFIG_VAR: '1' } } + let(:app) { Hatchet::Runner.new('spec/fixtures/hooks_pre_compile_pass_post_compile_fail', config:, allow_failure: true) } + + it 'runs the pre_compile hook but aborts the build during the post_compile hook with a suitable error message' do app.deploy do |app| output = clean_output(app.output) - expect(output).to include(<<~OUTPUT) + expect(output).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected remote: -----> Running bin/pre_compile hook remote: ~ pre_compile ran with env vars: - remote: BUILD_DIR=/tmp/build_ + remote: BUILD_DIR=/tmp/build_.+ remote: CACHE_DIR=/tmp/codon/tmp/cache remote: C_INCLUDE_PATH=/app/.heroku/python/include remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include - remote: ENV_DIR=/tmp/... - remote: HOME=/app + remote: ENV_DIR=/tmp/.+ remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/python/lib remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config - remote: PWD=/tmp/build_ remote: PYTHONUNBUFFERED=1 remote: SOME_APP_CONFIG_VAR=1 - remote: SOURCE_VERSION=... - remote: STACK=#{app.stack} remote: ~ pre_compile complete - OUTPUT + REGEX expect(output).to include(<<~OUTPUT) - remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: -----> Running bin/post_compile hook - remote: ~ post_compile ran with env vars: - remote: BUILD_DIR=/tmp/build_ - remote: CACHE_DIR=/tmp/codon/tmp/cache - remote: C_INCLUDE_PATH=/app/.heroku/python/include - remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include - remote: ENV_DIR=/tmp/... - remote: HOME=/app - remote: LANG=en_US.UTF-8 - remote: LD_LIBRARY_PATH=/app/.heroku/python/lib - remote: LIBRARY_PATH=/app/.heroku/python/lib - remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config - remote: PWD=/tmp/build_ - remote: PYTHONUNBUFFERED=1 - remote: SOME_APP_CONFIG_VAR=1 - remote: SOURCE_VERSION=... - remote: STACK=#{app.stack} - remote: ~ post_compile complete - OUTPUT - end - end - end - - context 'when an app has a failing bin/pre_compile script' do - let(:app) { Hatchet::Runner.new('spec/fixtures/hooks_failing_pre_compile', allow_failure: true) } - - it 'aborts the build with a suitable error message' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Running bin/pre_compile hook - remote: Some pre_compile error! + remote: Some post_compile error! remote: - remote: ! Error: Failed to run the bin/pre_compile script. + remote: ! Error: Failed to run the bin/post_compile script. remote: ! - remote: ! We found a 'bin/pre_compile' script in your app source, so ran + remote: ! We found a 'bin/post_compile' script in your app source, so ran remote: ! it to allow for customisation of the build process. remote: ! remote: ! However, this script exited with a non-zero exit status. @@ -84,18 +54,18 @@ end end - context 'when an app has a failing bin/post_compile script' do - let(:app) { Hatchet::Runner.new('spec/fixtures/hooks_failing_post_compile', allow_failure: true) } + context 'when an app has a failing bin/pre_compile script' do + let(:app) { Hatchet::Runner.new('spec/fixtures/hooks_pre_compile_fail', allow_failure: true) } it 'aborts the build with a suitable error message' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Running bin/post_compile hook - remote: Some post_compile error! + remote: -----> Running bin/pre_compile hook + remote: Some pre_compile error! remote: - remote: ! Error: Failed to run the bin/post_compile script. + remote: ! Error: Failed to run the bin/pre_compile script. remote: ! - remote: ! We found a 'bin/post_compile' script in your app source, so ran + remote: ! We found a 'bin/pre_compile' script in your app source, so ran remote: ! it to allow for customisation of the build process. remote: ! remote: ! However, this script exited with a non-zero exit status. diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index b689946b8..6d6c5a92a 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -130,6 +130,7 @@ it 'clears the cache before installing with the new package manager' do app.deploy do |app| + FileUtils.rm('bin/post_compile') FileUtils.rm('requirements.txt') FileUtils.cp(FIXTURE_DIR.join('uv_basic/pyproject.toml'), '.') FileUtils.cp(FIXTURE_DIR.join('uv_basic/uv.lock'), '.') diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index d741830c2..b9f9b73d0 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -7,7 +7,7 @@ let(:buildpacks) { [:default, 'heroku-community/inline'] } let(:app) { Hatchet::Runner.new('spec/fixtures/pip_basic', buildpacks:) } - it 're-uses packages from the cache' do + it 'installs successfully using pip and on rebuilds uses the cache' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected @@ -20,6 +20,18 @@ remote: Downloading typing_extensions-4.12.2-py3-none-any.whl \\(37 kB\\) remote: Installing collected packages: typing-extensions remote: Successfully installed typing-extensions-4.12.2 + remote: -----> Running bin/post_compile hook + remote: BUILD_DIR=/tmp/build_.+ + remote: CACHE_DIR=/tmp/codon/tmp/cache + remote: C_INCLUDE_PATH=/app/.heroku/python/include + remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include + remote: ENV_DIR=/tmp/.+ + remote: LANG=en_US.UTF-8 + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib + remote: LIBRARY_PATH=/app/.heroku/python/lib + remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config + remote: PYTHONUNBUFFERED=1 remote: -----> Saving cache remote: -----> Inline app detected remote: LANG=en_US.UTF-8 @@ -45,7 +57,7 @@ remote: remote: remote: - remote: { + remote: \\{ remote: "cache_restore_duration": [0-9.]+, remote: "cache_save_duration": [0-9.]+, remote: "cache_status": "empty", @@ -55,7 +67,8 @@ remote: "package_manager": "pip", remote: "package_manager_install_duration": [0-9.]+, remote: "pip_version": "#{PIP_VERSION}", - remote: "post_compile_hook": false, + remote: "post_compile_hook": true, + remote: "post_compile_hook_duration": [0-9.]+, remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", @@ -66,24 +79,41 @@ remote: "python_version_requested": "3.13", remote: "setup_py_only": false, remote: "total_duration": [0-9.]+ - remote: } + remote: \\} REGEX + app.commit! app.push! - expect(clean_output(app.output)).to include(<<~OUTPUT) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Restoring cache remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Requirement already satisfied: typing-extensions==4.12.2 (from -r requirements.txt (line 5)) (4.12.2) + remote: Requirement already satisfied: typing-extensions==4.12.2 \\(from -r requirements.txt \\(line 5\\)\\) \\(4.12.2\\) + remote: -----> Running bin/post_compile hook + remote: .+ remote: -----> Saving cache remote: -----> Inline app detected + REGEX + + # For historical reasons pip is made available at run-time too, unlike some of the other package managers. + expect(app.run('bin/print-env-vars.sh && pip --version')).to eq(<<~OUTPUT) + DYNO_RAM=512 + FORWARDED_ALLOW_IPS=* + GUNICORN_CMD_ARGS=--access-logfile - + LANG=en_US.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib + PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin + PYTHONHOME=/app/.heroku/python + PYTHONPATH=/app + PYTHONUNBUFFERED=true + WEB_CONCURRENCY=2 + pip #{PIP_VERSION} from /app/.heroku/python/lib/python3.13/site-packages/pip (python 3.13) OUTPUT - # Test that pip is available at run-time too (since for historical reasons it's been - # made available after the build, and users now rely on this). - expect(app.run('pip --version')).to include("pip #{PIP_VERSION}") + expect($CHILD_STATUS.exitstatus).to eq(0) end end end @@ -96,6 +126,7 @@ # The test fixture's requirements.txt is a symlink to a requirements file in a subdirectory in # order to test that symlinked requirements files work in general and with cache invalidation. File.write('requirements/prod.txt', 'six==1.17.0', mode: 'a') + FileUtils.rm('bin/post_compile') app.commit! app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index fe7c261a6..789278999 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -7,7 +7,7 @@ let(:buildpacks) { [:default, 'heroku-community/inline'] } let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_basic', buildpacks:) } - it 'builds with the specified python_version and re-uses packages from the cache' do + it 'installs successfully using Pipenv and on rebuilds uses the cache' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected @@ -16,6 +16,21 @@ remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Running bin/post_compile hook + remote: BUILD_DIR=/tmp/build_.+ + remote: CACHE_DIR=/tmp/codon/tmp/cache + remote: C_INCLUDE_PATH=/app/.heroku/python/include + remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include + remote: ENV_DIR=/tmp/.+ + remote: LANG=en_US.UTF-8 + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib + remote: LIBRARY_PATH=/app/.heroku/python/lib + remote: PATH=/app/.heroku/python/pipenv/bin:/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PIPENV_SYSTEM=1 + remote: PIPENV_VERBOSITY=-1 + remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config + remote: PYTHONUNBUFFERED=1 + remote: VIRTUAL_ENV=/app/.heroku/python remote: -----> Saving cache remote: -----> Inline app detected remote: LANG=en_US.UTF-8 @@ -44,7 +59,7 @@ remote: remote: remote: - remote: { + remote: \\{ remote: "cache_restore_duration": [0-9.]+, remote: "cache_save_duration": [0-9.]+, remote: "cache_status": "empty", @@ -54,7 +69,8 @@ remote: "package_manager": "pipenv", remote: "package_manager_install_duration": [0-9.]+, remote: "pipenv_version": "#{PIPENV_VERSION}", - remote: "post_compile_hook": false, + remote: "post_compile_hook": true, + remote: "post_compile_hook_duration": [0-9.]+, remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", @@ -65,11 +81,12 @@ remote: "python_version_requested": "3.13", remote: "setup_py_only": false, remote: "total_duration": [0-9.]+ - remote: } + remote: \\} REGEX + app.commit! app.push! - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock remote: -----> Restoring cache @@ -77,12 +94,31 @@ remote: -----> Using cached Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... + remote: -----> Running bin/post_compile hook + remote: .+ remote: -----> Saving cache remote: -----> Inline app detected REGEX - # Test that Pipenv is available at run-time too (since for historical reasons it's been - # made available after the build, and users now rely on this). - expect(app.run('pipenv --version')).to include("version #{PIPENV_VERSION}") + + # For historical reasons Pipenv is made available at run-time too, unlike some of the other package managers. + expect(clean_output(app.run('bin/print-env-vars.sh && pipenv --version'))).to eq(<<~OUTPUT) + DYNO_RAM=512 + FORWARDED_ALLOW_IPS=* + GUNICORN_CMD_ARGS=--access-logfile - + LANG=en_US.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib + PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin + PIPENV_SYSTEM=1 + PIPENV_VERBOSITY=-1 + PYTHONHOME=/app/.heroku/python + PYTHONPATH=/app + PYTHONUNBUFFERED=true + VIRTUAL_ENV=/app/.heroku/python + WEB_CONCURRENCY=2 + pipenv, version #{PIPENV_VERSION} + OUTPUT + expect($CHILD_STATUS.exitstatus).to eq(0) end end end @@ -93,6 +129,7 @@ it 'clears the cache before installing the packages again' do app.deploy do |app| File.write('Pipfile.lock', "\n", mode: 'a') + FileUtils.rm('bin/post_compile') app.commit! app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) @@ -105,6 +142,7 @@ remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Saving cache + remote: -----> Discovering process types REGEX end end @@ -414,6 +452,7 @@ it 'clears the cache before installing' do app.deploy do |app| update_buildpacks(app, [:default]) + FileUtils.rm('bin/post_compile') app.commit! app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) @@ -510,21 +549,21 @@ end end + # This is disabled since it's currently broken upstream: https://github.com/pypa/pipenv/issues/6403 # This tests that Pipenv doesn't fall back to system Python if the Python version in # pyproject.toml doesn't match that in Pipfile / Pipfile.lock. - context 'when requires-python in pyproject.toml is incompatible with .python-version', - skip: 'this is currently broken upstream: https://github.com/pypa/pipenv/issues/6403' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_mismatched_python_version', allow_failure: true) } - - it 'fails the build' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: - OUTPUT - end - end - end + # context 'when requires-python in pyproject.toml is incompatible with .python-version' do + # let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_mismatched_python_version', allow_failure: true) } + # + # it 'fails the build' do + # app.deploy do |app| + # expect(clean_output(app.output)).to include(<<~OUTPUT) + # remote: -----> Installing dependencies using 'pipenv install --deploy' + # remote: + # OUTPUT + # end + # end + # end # This tests not only our handling of failing dependency installation, but also that we're running # Pipenv in such a way that it errors if the lockfile is out of sync, rather than simply updating it. diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index d7e5e7580..644f6c92b 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -20,6 +20,20 @@ remote: Package operations: 1 install, 0 updates, 0 removals remote: remote: - Installing typing-extensions \\(4.12.2\\) + remote: -----> Running bin/post_compile hook + remote: BUILD_DIR=/tmp/build_.+ + remote: CACHE_DIR=/tmp/codon/tmp/cache + remote: C_INCLUDE_PATH=/app/.heroku/python/include + remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include + remote: ENV_DIR=/tmp/.+ + remote: LANG=en_US.UTF-8 + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib + remote: LIBRARY_PATH=/app/.heroku/python/lib + remote: PATH=/tmp/codon/tmp/cache/.heroku/python-poetry/bin:/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config + remote: POETRY_VIRTUALENVS_CREATE=false + remote: POETRY_VIRTUALENVS_USE_POETRY_PYTHON=true + remote: PYTHONUNBUFFERED=1 remote: -----> Saving cache remote: -----> Inline app detected remote: LANG=en_US.UTF-8 @@ -45,7 +59,7 @@ remote: remote: remote: - remote: { + remote: \\{ remote: "cache_restore_duration": [0-9.]+, remote: "cache_save_duration": [0-9.]+, remote: "cache_status": "empty", @@ -55,7 +69,8 @@ remote: "package_manager": "poetry", remote: "package_manager_install_duration": [0-9.]+, remote: "poetry_version": "#{POETRY_VERSION}", - remote: "post_compile_hook": false, + remote: "post_compile_hook": true, + remote: "post_compile_hook_duration": [0-9.]+, remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", @@ -66,11 +81,12 @@ remote: "python_version_requested": "3.13", remote: "setup_py_only": false, remote: "total_duration": [0-9.]+ - remote: } + remote: \\} REGEX + app.commit! app.push! - expect(clean_output(app.output)).to include(<<~OUTPUT) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Restoring cache @@ -80,9 +96,27 @@ remote: Installing dependencies from lock file remote: remote: No dependencies to install or update + remote: -----> Running bin/post_compile hook + remote: .+ remote: -----> Saving cache remote: -----> Inline app detected + REGEX + + command = 'bin/print-env-vars.sh && if command -v poetry; then echo "Poetry unexpectedly found!" && exit 1; fi' + expect(app.run(command)).to eq(<<~OUTPUT) + DYNO_RAM=512 + FORWARDED_ALLOW_IPS=* + GUNICORN_CMD_ARGS=--access-logfile - + LANG=en_US.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib + PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin + PYTHONHOME=/app/.heroku/python + PYTHONPATH=/app + PYTHONUNBUFFERED=true + WEB_CONCURRENCY=2 OUTPUT + expect($CHILD_STATUS.exitstatus).to eq(0) end end end @@ -94,6 +128,7 @@ it 'clears the cache before installing' do app.deploy do |app| update_buildpacks(app, [:default]) + FileUtils.rm('bin/post_compile') app.commit! app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) @@ -270,21 +305,21 @@ end end + # This is disabled since it's currently broken upstream: https://github.com/python-poetry/poetry/issues/10226 # This tests that Poetry doesn't download its own Python or fall back to system Python # if the Python version in pyproject.toml doesn't match that in .python-version. - context 'when requires-python in pyproject.toml is incompatible with .python-version', - skip: 'this is currently broken upstream: https://github.com/python-poetry/poetry/issues/10226' do - let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_mismatched_python_version', allow_failure: true) } - - it 'fails the build' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Installing dependencies using 'poetry sync --only main' - remote: - OUTPUT - end - end - end + # context 'when requires-python in pyproject.toml is incompatible with .python-version' do + # let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_mismatched_python_version', allow_failure: true) } + # + # it 'fails the build' do + # app.deploy do |app| + # expect(clean_output(app.output)).to include(<<~OUTPUT) + # remote: -----> Installing dependencies using 'poetry sync --only main' + # remote: + # OUTPUT + # end + # end + # end # This tests not only our handling of failing dependency installation, but also that we're running # Poetry in such a way that it errors if the lockfile is out of sync, rather than simply updating it. diff --git a/spec/hatchet/profile_d_scripts_spec.rb b/spec/hatchet/profile_d_scripts_spec.rb index 263a247a6..6d9cd7633 100644 --- a/spec/hatchet/profile_d_scripts_spec.rb +++ b/spec/hatchet/profile_d_scripts_spec.rb @@ -5,38 +5,18 @@ RSpec.describe '.profile.d/ scripts' do it 'sets the required run-time env vars' do Hatchet::Runner.new('spec/fixtures/procfile', run_multi: true).deploy do |app| - # These are written as a single test to reduce end to end test time. This repo uses parallel_split_test, - # so we can't perform app setup in a `before(:all)` and have multiple tests run against the single app. - - list_envs_cmd = 'env | sort | grep -vE "^(_|DYNO|PORT|PS1|SHLVL|TERM)="' - - # Check all env vars are set correctly when there are no user-provided env vars. - # Also checks that the WEB_CONCURRENCY related log output is not shown for one-off dynos. - app.run_multi(list_envs_cmd) do |output, _| - expect(output).to eq(<<~OUTPUT) - DYNO_RAM=512 - FORWARDED_ALLOW_IPS=* - GUNICORN_CMD_ARGS=--access-logfile - - HOME=/app - LANG=en_US.UTF-8 - LD_LIBRARY_PATH=/app/.heroku/python/lib - LIBRARY_PATH=/app/.heroku/python/lib - PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin - PWD=/app - PYTHONHOME=/app/.heroku/python - PYTHONPATH=/app - PYTHONUNBUFFERED=true - WEB_CONCURRENCY=2 - OUTPUT - end + # These are written as a single test to reduce end to end test time. (This repo uses parallel_split_test, + # so we can't perform app setup in a `before(:all)` and have multiple tests run against the single app.) + # These tests supplement the run-time env var tests performed for each package manager, so intentionally + # don't duplicate those. # Check user-provided env var values are preserved/overridden as appropriate. # Also checks that the WEB_CONCURRENCY related log output is not shown for worker dynos. + list_envs_cmd = 'printenv | sort | grep -vE "^(_|DYNO|HOME|PORT|PS1|PWD|SHLVL|TERM)="' user_env_vars = [ 'DYNO_RAM=this-should-be-overridden', 'FORWARDED_ALLOW_IPS=this-should-be-overridden', 'GUNICORN_CMD_ARGS=this-should-be-preserved', - 'HOME=this-should-be-overridden', 'LANG=this-should-be-overridden', 'LD_LIBRARY_PATH=/this-should-be-preserved', 'LIBRARY_PATH=/this-should-be-preserved', @@ -51,12 +31,10 @@ DYNO_RAM=512 FORWARDED_ALLOW_IPS=* GUNICORN_CMD_ARGS=this-should-be-preserved - HOME=/app LANG=C.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib:/this-should-be-preserved LIBRARY_PATH=/app/.heroku/python/lib:/this-should-be-preserved PATH=/app/.heroku/python/bin:/this-should-be-preserved:/usr/local/bin:/usr/bin:/bin - PWD=/app PYTHONHOME=/app/.heroku/python PYTHONPATH=/this-should-be-preserved PYTHONUNBUFFERED=true @@ -64,7 +42,7 @@ OUTPUT end - list_concurrency_envs_cmd = 'env | sort | grep -E "^(DYNO_RAM|WEB_CONCURRENCY)="' + list_concurrency_envs_cmd = 'printenv | sort | grep -E "^(DYNO_RAM|WEB_CONCURRENCY)="' # Check WEB_CONCURRENCY support when using a Standard-1X dyno. # We set the process type to `web` so that we can test the web-dyno-only log output. diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index ac0d4a686..cfb987a9c 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -24,7 +24,8 @@ remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT end - expect(app.run('python -V')).to include("Python #{resolved_version}") + expect(app.run('python -V')).to eq("Python #{resolved_version}\n") + expect($CHILD_STATUS.exitstatus).to eq(0) end end end @@ -153,7 +154,8 @@ remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} OUTPUT - expect(app.run('python -V')).to include("Python #{LATEST_PYTHON_3_12}") + expect(app.run('python -V')).to eq("Python #{LATEST_PYTHON_3_12}\n") + expect($CHILD_STATUS.exitstatus).to eq(0) end end end @@ -278,7 +280,8 @@ remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT - expect(app.run('python -V')).to include("Python #{LATEST_PYTHON_3_9}") + expect(app.run('python -V')).to eq("Python #{LATEST_PYTHON_3_9}\n") + expect($CHILD_STATUS.exitstatus).to eq(0) end end end diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 1c33b8390..60e561f1e 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -20,6 +20,22 @@ remote: Installed 1 package in .+s remote: Bytecode compiled 1 file in .+s remote: \\+ typing-extensions==4.13.2 + remote: -----> Running bin/post_compile hook + remote: BUILD_DIR=/tmp/build_.+ + remote: CACHE_DIR=/tmp/codon/tmp/cache + remote: C_INCLUDE_PATH=/app/.heroku/python/include + remote: CPLUS_INCLUDE_PATH=/app/.heroku/python/include + remote: ENV_DIR=/tmp/.+ + remote: LANG=en_US.UTF-8 + remote: LD_LIBRARY_PATH=/app/.heroku/python/lib + remote: LIBRARY_PATH=/app/.heroku/python/lib + remote: PATH=/tmp/codon/tmp/cache/.heroku/python-uv:/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config + remote: PYTHONUNBUFFERED=1 + remote: UV_CACHE_DIR=/tmp/uv-cache + remote: UV_NO_MANAGED_PYTHON=1 + remote: UV_PROJECT_ENVIRONMENT=/app/.heroku/python + remote: UV_PYTHON_DOWNLOADS=never remote: -----> Saving cache remote: -----> Inline app detected remote: LANG=en_US.UTF-8 @@ -49,7 +65,7 @@ remote: remote: remote: - remote: { + remote: \\{ remote: "cache_restore_duration": [0-9.]+, remote: "cache_save_duration": [0-9.]+, remote: "cache_status": "empty", @@ -58,7 +74,8 @@ remote: "nltk_downloader_duration": [0-9.]+, remote: "package_manager": "uv", remote: "package_manager_install_duration": [0-9.]+, - remote: "post_compile_hook": false, + remote: "post_compile_hook": true, + remote: "post_compile_hook_duration": [0-9.]+, remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", @@ -70,11 +87,12 @@ remote: "setup_py_only": false, remote: "total_duration": [0-9.]+, remote: "uv_version": "#{UV_VERSION}" - remote: } + remote: \\} REGEX + app.commit! app.push! - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Restoring cache @@ -83,9 +101,27 @@ remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' remote: Resolved 7 packages in .+s remote: Bytecode compiled 1 file in .+s + remote: -----> Running bin/post_compile hook + remote: .+ remote: -----> Saving cache remote: -----> Inline app detected REGEX + + command = 'bin/print-env-vars.sh && if command -v uv; then echo "uv unexpectedly found!" && exit 1; fi' + expect(app.run(command)).to eq(<<~OUTPUT) + DYNO_RAM=512 + FORWARDED_ALLOW_IPS=* + GUNICORN_CMD_ARGS=--access-logfile - + LANG=en_US.UTF-8 + LD_LIBRARY_PATH=/app/.heroku/python/lib + LIBRARY_PATH=/app/.heroku/python/lib + PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin + PYTHONHOME=/app/.heroku/python + PYTHONPATH=/app + PYTHONUNBUFFERED=true + WEB_CONCURRENCY=2 + OUTPUT + expect($CHILD_STATUS.exitstatus).to eq(0) end end end @@ -97,6 +133,7 @@ it 'clears the cache before installing' do app.deploy do |app| update_buildpacks(app, [:default]) + FileUtils.rm('bin/post_compile') app.commit! app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 88fb52018..09beb2fa1 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -3,6 +3,7 @@ ENV['HATCHET_BUILDPACK_BASE'] ||= 'https://github.com/heroku/heroku-buildpack-python.git' ENV['HATCHET_DEFAULT_STACK'] ||= 'heroku-24' +require 'English' # for $CHILD_STATUS require 'rspec/core' require 'rspec/retry' require 'hatchet' From 3f9caa0ecdf251d7998cc911a53a65fe28876e82 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 9 Sep 2025 09:25:32 +0000 Subject: [PATCH 216/406] Bump uv from 0.8.13 to 0.8.15 (#1894) * Bump uv from 0.8.13 to 0.8.15 Bumps [uv](https://github.com/astral-sh/uv) from 0.8.13 to 0.8.15. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.8.13...0.8.15) --- updated-dependencies: - dependency-name: uv dependency-version: 0.8.15 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2126876a4..cc153991b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.8.13 to 0.8.15. ([#1894](https://github.com/heroku/heroku-buildpack-python/pull/1894)) ## [v305] - 2025-09-02 diff --git a/requirements/uv.txt b/requirements/uv.txt index a98c809f6..b8c7e955d 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.8.13 +uv==0.8.15 From 5dcc5c4691904270267cc3d33416d6392caec6ca Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 9 Sep 2025 11:25:39 +0000 Subject: [PATCH 217/406] Prepare release v306 (#1896) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cc153991b..baca9244b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v306] - 2025-09-09 + - Updated uv from 0.8.13 to 0.8.15. ([#1894](https://github.com/heroku/heroku-buildpack-python/pull/1894)) ## [v305] - 2025-09-02 @@ -1319,7 +1322,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v305...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v306...main +[v306]: https://github.com/heroku/heroku-buildpack-python/compare/v305...v306 [v305]: https://github.com/heroku/heroku-buildpack-python/compare/v304...v305 [v304]: https://github.com/heroku/heroku-buildpack-python/compare/v303...v304 [v303]: https://github.com/heroku/heroku-buildpack-python/compare/v302...v303 From 487f511c5bfc86b042d982de308feb903b3c45ff Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 10 Sep 2025 14:46:30 +0100 Subject: [PATCH 218/406] Deprecate implicit setup.py support (#1897) Currently if an app has only a `setup.py` file and no Python package manager files (such as `requirements.txt`, `Pipfile.lock`, `Poetry.lock` or `uv.lock`), the buildpack will install the project using `pip install --editable .` However, there is nothing pip-specific about `setup.py` - the file is not a package manager dependency file but instead a setuptools package file, and could easily be installed by any package manager. In addition, some projects may want to install this file in standard rather than editable mode. As such, it's best that the buildpack doesn't guess which package manager and mode to use, and instead we require explicit configuration from the user. For example in the form of a `requirements.txt` file that references the package definition, via this single line in the requirements file: ``` --editable . ``` That said, in general we recommend people don't use `setup.py` to declare their dependencies, since: (a) the file is deprecated in favour of `pyproject.toml` (b) it's intended more for libraries rather than applications, and it's much less common (and practical) to list all transitive dependencies in a `setup.py`, meaning apps using the file typically have unpinned dependencies, which is a production reliability risk. (Apps should either be using a package manager that supports lockfiles, or else using one of the pip requirements files substitutes like pip-tools or `pip freeze` etc). The existing metrics for this fallback show usage to be very low, so we'll likely not wait long before converting this warning to an error (particularly since fixing the error is a case of adding a single line requirements file, so fairly simple). GUS-W-19275438. --- CHANGELOG.md | 1 + lib/package_manager.sh | 26 ++++++++++++++++++++++---- lib/pip.sh | 2 +- spec/hatchet/package_manager_spec.rb | 24 +++++++++++++++++++++++- spec/hatchet/pip_spec.rb | 1 - spec/hatchet/pipenv_spec.rb | 1 - spec/hatchet/poetry_spec.rb | 1 - spec/hatchet/uv_spec.rb | 1 - 8 files changed, 47 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index baca9244b..7ace5e147 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Deprecated support for falling back to installing dependencies from a `setup.py` file if no Python package manager files were found. ([#1897](https://github.com/heroku/heroku-buildpack-python/pull/1897)) ## [v306] - 2025-09-09 diff --git a/lib/package_manager.sh b/lib/package_manager.sh index a4fcac638..be5e79730 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -57,14 +57,32 @@ function package_manager::determine_package_manager() { package_managers_found_display_text+=("uv.lock (uv)") fi - # TODO: Deprecate/sunset this fallback, since using setup.py declared dependencies is - # not a best practice, and we can only guess as to which package manager to use. if ((${#package_managers_found[@]} == 0)) && [[ -f "${build_dir}/setup.py" ]]; then package_managers_found+=(pip) package_managers_found_display_text+=("setup.py (pip)") + output::warning <<-EOF + Warning: Implicit setup.py file support is deprecated. + + Your app currently only has a setup.py file and no Python + package manager files. This means that the buildpack has + to guess which package manager you want to use and also + whether to install your project in editable mode or not. + + For now, we will use pip to install your dependencies in + editable mode, however, this fallback is deprecated and + will be removed in the future. + + Please add an explicit package manager file to your app. + + To continue using pip in editable mode, create a new file + in the root directory of your app named 'requirements.txt' + containing the requirement '--editable .' (without quotes). + + Alternatively, if you wish to switch to another package + manager, we highly recommend uv: + https://docs.astral.sh/uv/ + EOF build_data::set_raw "setup_py_only" "true" - else - build_data::set_raw "setup_py_only" "false" fi local num_package_managers_found=${#package_managers_found[@]} diff --git a/lib/pip.sh b/lib/pip.sh index f5330de9e..ffb983c80 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -90,7 +90,7 @@ function pip::install_dependencies() { install ) - # TODO: Deprecate/sunset this missing requirements file fallback. + # Support for the setup.py fallback is deprecated and will be removed in the future. if [[ -f setup.py && ! -f requirements.txt ]]; then pip_install_command+=(--editable .) else diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index 6d6c5a92a..63b1dafae 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -49,7 +49,7 @@ end end - # TODO: Deprecate/sunset the setup.py file fallback. + # This case will be turned into an error in the future. context 'when there is only a setup.py' do let(:app) { Hatchet::Runner.new('spec/fixtures/setup_py_only') } @@ -57,6 +57,28 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected + remote: + remote: ! Warning: Implicit setup.py file support is deprecated. + remote: ! + remote: ! Your app currently only has a setup.py file and no Python + remote: ! package manager files. This means that the buildpack has + remote: ! to guess which package manager you want to use and also + remote: ! whether to install your project in editable mode or not. + remote: ! + remote: ! For now, we will use pip to install your dependencies in + remote: ! editable mode, however, this fallback is deprecated and + remote: ! will be removed in the future. + remote: ! + remote: ! Please add an explicit package manager file to your app. + remote: ! + remote: ! To continue using pip in editable mode, create a new file + remote: ! in the root directory of your app named 'requirements.txt' + remote: ! containing the requirement '--editable .' \\(without quotes\\). + remote: ! + remote: ! Alternatively, if you wish to switch to another package + remote: ! manager, we highly recommend uv: + remote: ! https://docs.astral.sh/uv/ + remote: remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION} diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index b9f9b73d0..8dc68798a 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -77,7 +77,6 @@ remote: "python_version_outdated": false, remote: "python_version_pinned": false, remote: "python_version_requested": "3.13", - remote: "setup_py_only": false, remote: "total_duration": [0-9.]+ remote: \\} REGEX diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 789278999..220d92155 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -79,7 +79,6 @@ remote: "python_version_outdated": false, remote: "python_version_pinned": false, remote: "python_version_requested": "3.13", - remote: "setup_py_only": false, remote: "total_duration": [0-9.]+ remote: \\} REGEX diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 644f6c92b..7bbe29f5f 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -79,7 +79,6 @@ remote: "python_version_outdated": false, remote: "python_version_pinned": false, remote: "python_version_requested": "3.13", - remote: "setup_py_only": false, remote: "total_duration": [0-9.]+ remote: \\} REGEX diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 60e561f1e..5b32fccf5 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -84,7 +84,6 @@ remote: "python_version_outdated": false, remote: "python_version_pinned": false, remote: "python_version_requested": "3.13", - remote: "setup_py_only": false, remote: "total_duration": [0-9.]+, remote: "uv_version": "#{UV_VERSION}" remote: \\} From 719eb305382434b1ca5d69a4a5d364d8c1a41ab0 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 10 Sep 2025 14:02:52 +0000 Subject: [PATCH 219/406] Prepare release v307 (#1898) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ace5e147..0de2973f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v307] - 2025-09-10 + - Deprecated support for falling back to installing dependencies from a `setup.py` file if no Python package manager files were found. ([#1897](https://github.com/heroku/heroku-buildpack-python/pull/1897)) ## [v306] - 2025-09-09 @@ -1323,7 +1326,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v306...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v307...main +[v307]: https://github.com/heroku/heroku-buildpack-python/compare/v306...v307 [v306]: https://github.com/heroku/heroku-buildpack-python/compare/v305...v306 [v305]: https://github.com/heroku/heroku-buildpack-python/compare/v304...v305 [v304]: https://github.com/heroku/heroku-buildpack-python/compare/v303...v304 From b866992395d52f3968974e06891fa6ff559a75b7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 17 Sep 2025 10:36:37 +0000 Subject: [PATCH 220/406] Bump uv from 0.8.15 to 0.8.17 (#1899) * Bump uv from 0.8.15 to 0.8.17 Bumps [uv](https://github.com/astral-sh/uv) from 0.8.15 to 0.8.17. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.8.15...0.8.17) --- updated-dependencies: - dependency-name: uv dependency-version: 0.8.17 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0de2973f1..30fdc0fab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.8.15 to 0.8.17. ([#1899](https://github.com/heroku/heroku-buildpack-python/pull/1899)) ## [v307] - 2025-09-10 diff --git a/requirements/uv.txt b/requirements/uv.txt index b8c7e955d..822727fe0 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.8.15 +uv==0.8.17 From 189463cb9ceed4b4009bb4a2704c959446673d36 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 17 Sep 2025 10:55:35 +0000 Subject: [PATCH 221/406] Bump poetry from 2.1.4 to 2.2.0 (#1900) * Bump poetry from 2.1.4 to 2.2.0 Bumps [poetry](https://github.com/python-poetry/poetry) from 2.1.4 to 2.2.0. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](https://github.com/python-poetry/poetry/compare/2.1.4...2.2.0) --- updated-dependencies: - dependency-name: poetry dependency-version: 2.2.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Add changelog entry * Convert some Poetry test fixtures to PEP 735 syntax Poetry 2.2.0 added support for PEP 735. In order to test this, some of the Poetry test fixtures now use dependency groups syntax to specify optional dependencies rather than Poetry's previous proprietary syntax. See: https://github.com/python-poetry/poetry/releases/tag/2.2.0 https://peps.python.org/pep-0735/ --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/poetry.txt | 2 +- spec/fixtures/ci_poetry/poetry.lock | 15 +++++++++++---- spec/fixtures/ci_poetry/pyproject.toml | 21 ++++++++++++++------- spec/fixtures/poetry_basic/poetry.lock | 4 ++-- spec/fixtures/poetry_basic/pyproject.toml | 12 +++++++----- 6 files changed, 36 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 30fdc0fab..8c2a4d775 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated Poetry from 2.1.4 to 2.2.0. ([#1900](https://github.com/heroku/heroku-buildpack-python/pull/1900)) - Updated uv from 0.8.15 to 0.8.17. ([#1899](https://github.com/heroku/heroku-buildpack-python/pull/1899)) ## [v307] - 2025-09-10 diff --git a/requirements/poetry.txt b/requirements/poetry.txt index 67f219c60..b2589c103 100644 --- a/requirements/poetry.txt +++ b/requirements/poetry.txt @@ -1 +1 @@ -poetry==2.1.4 +poetry==2.2.0 diff --git a/spec/fixtures/ci_poetry/poetry.lock b/spec/fixtures/ci_poetry/poetry.lock index 7fe5d0794..dc989760e 100644 --- a/spec/fixtures/ci_poetry/poetry.lock +++ b/spec/fixtures/ci_poetry/poetry.lock @@ -1,4 +1,4 @@ -# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand. +# This file is automatically @generated by Poetry 2.2.0 and should not be changed by hand. [[package]] name = "colorama" @@ -6,6 +6,8 @@ version = "0.4.6" description = "Cross-platform colored terminal text." optional = false python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,>=2.7" +groups = ["dev"] +markers = "sys_platform == \"win32\"" files = [ {file = "colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"}, {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"}, @@ -17,6 +19,7 @@ version = "2.0.0" description = "brain-dead simple config-ini parsing" optional = false python-versions = ">=3.7" +groups = ["dev"] files = [ {file = "iniconfig-2.0.0-py3-none-any.whl", hash = "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374"}, {file = "iniconfig-2.0.0.tar.gz", hash = "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3"}, @@ -28,6 +31,7 @@ version = "24.2" description = "Core utilities for Python packages" optional = false python-versions = ">=3.8" +groups = ["dev"] files = [ {file = "packaging-24.2-py3-none-any.whl", hash = "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759"}, {file = "packaging-24.2.tar.gz", hash = "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f"}, @@ -39,6 +43,7 @@ version = "1.5.0" description = "plugin and hook calling mechanisms for python" optional = false python-versions = ">=3.8" +groups = ["dev"] files = [ {file = "pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669"}, {file = "pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1"}, @@ -54,6 +59,7 @@ version = "8.3.4" description = "pytest: simple powerful testing with Python" optional = false python-versions = ">=3.8" +groups = ["dev"] files = [ {file = "pytest-8.3.4-py3-none-any.whl", hash = "sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6"}, {file = "pytest-8.3.4.tar.gz", hash = "sha256:965370d062bce11e73868e0335abac31b4d3de0e82f4007408d242b4f8610761"}, @@ -74,12 +80,13 @@ version = "4.12.2" description = "Backported and Experimental Type Hints for Python 3.8+" optional = false python-versions = ">=3.8" +groups = ["main"] files = [ {file = "typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d"}, {file = "typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"}, ] [metadata] -lock-version = "2.0" -python-versions = "^3.13" -content-hash = "56ec6342f4a39b402d2b65bd08cfb3af4136a02bb58674df49eac8d317376a7f" +lock-version = "2.1" +python-versions = ">=3.13" +content-hash = "029e0ee94dae38c88285bb0a557d4b77d92e76d30aa8ca91d17ac51fcf62b7f1" diff --git a/spec/fixtures/ci_poetry/pyproject.toml b/spec/fixtures/ci_poetry/pyproject.toml index ad7398c6e..99fed51a1 100644 --- a/spec/fixtures/ci_poetry/pyproject.toml +++ b/spec/fixtures/ci_poetry/pyproject.toml @@ -1,9 +1,16 @@ -[tool.poetry] -package-mode = false +[project] +name = "ci-poetry" +version = "0.0.0" +requires-python = ">=3.13" +dependencies = [ + "typing-extensions", +] + +[dependency-groups] +dev = [ + "pytest", +] -[tool.poetry.dependencies] -python = "^3.13" -typing-extensions = "*" -[tool.poetry.group.test.dependencies] -pytest = "*" +[tool.poetry] +package-mode = false diff --git a/spec/fixtures/poetry_basic/poetry.lock b/spec/fixtures/poetry_basic/poetry.lock index 166f8fea8..dc989760e 100644 --- a/spec/fixtures/poetry_basic/poetry.lock +++ b/spec/fixtures/poetry_basic/poetry.lock @@ -1,4 +1,4 @@ -# This file is automatically @generated by Poetry 2.1.1 and should not be changed by hand. +# This file is automatically @generated by Poetry 2.2.0 and should not be changed by hand. [[package]] name = "colorama" @@ -89,4 +89,4 @@ files = [ [metadata] lock-version = "2.1" python-versions = ">=3.13" -content-hash = "8e82be2391208aa1b8599893f6abec30c0aa5702a846842fd442f50de45481e2" +content-hash = "029e0ee94dae38c88285bb0a557d4b77d92e76d30aa8ca91d17ac51fcf62b7f1" diff --git a/spec/fixtures/poetry_basic/pyproject.toml b/spec/fixtures/poetry_basic/pyproject.toml index 85bd48b9d..3da2a688e 100644 --- a/spec/fixtures/poetry_basic/pyproject.toml +++ b/spec/fixtures/poetry_basic/pyproject.toml @@ -3,12 +3,14 @@ name = "poetry-basic" version = "0.0.0" requires-python = ">=3.13" dependencies = [ - "typing-extensions" + "typing-extensions", +] + +[dependency-groups] +# This group shouldn't be installed due to us passing `--only main`. +dev = [ + "pytest", ] [tool.poetry] package-mode = false - -# This group shouldn't be installed due to us passing `--only main`. -[tool.poetry.group.dev.dependencies] -pytest = "*" From 04d8c0d1fb1dba24adfe738ee6674e7a306832a7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 18 Sep 2025 11:40:41 +0100 Subject: [PATCH 222/406] Bump uv from 0.8.17 to 0.8.18 (#1901) * Bump uv from 0.8.17 to 0.8.18 Bumps [uv](https://github.com/astral-sh/uv) from 0.8.17 to 0.8.18. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.8.17...0.8.18) --- updated-dependencies: - dependency-name: uv dependency-version: 0.8.18 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Update changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 2 +- requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c2a4d775..64eb122fe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,7 @@ ## [Unreleased] - Updated Poetry from 2.1.4 to 2.2.0. ([#1900](https://github.com/heroku/heroku-buildpack-python/pull/1900)) -- Updated uv from 0.8.15 to 0.8.17. ([#1899](https://github.com/heroku/heroku-buildpack-python/pull/1899)) +- Updated uv from 0.8.15 to 0.8.18. ([#1899](https://github.com/heroku/heroku-buildpack-python/pull/1899) and [#1901](https://github.com/heroku/heroku-buildpack-python/pull/1901)) ## [v307] - 2025-09-10 diff --git a/requirements/uv.txt b/requirements/uv.txt index 822727fe0..cb5116d05 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.8.17 +uv==0.8.18 From 42534908b4bc67e10e9ae19e9e346a4ad6307294 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 18 Sep 2025 20:49:59 +0100 Subject: [PATCH 223/406] Speed up cache saving (#1902) At the end of the build, the Python installation including the site-packages directory (which contains the app's dependencies) has to be copied from the build directory to the cache directory, so that it's cached for the next build. For apps with very large dependencies this can take a considerable amount of time (Honeycomb reports `python.cache_save_duration` as having a P99.9 of 28 seconds). I'd previously benchmarked using `cp --reflink=auto` in the hopes of improving this in a way that would be compatible with all configurations (including Heroku CI and `build-in-app-dir`), but found it made performance worse for standard builds with the current production build system's filesystems and mount configurations. This leaves `cp --link` (which uses hardlinks) as the last way we can improve performance - but using this means: 1. We need to check whether the cache and build directory are on the same filesystem mounts (since otherwise the copy will fail with a `Invalid cross-device link` error). 2. Any modifications made to the files in one location affects the other. (eg by later buildpacks) However, given the Honeycomb data shows the time spent saving the cache really is significant for some apps, it's clear that using `--link` is worth the complexity/mutability trade-offs. In addition, Python package managers typically fully uninstall a package before reinstalling it at a different version - so if any apps perform `pip install` type operations in later buildpacks (which isn't something we officially support anyway), the cache should still be left in a valid state. Note: We use `df` to determine whether both locations are on the same filesystem mount since `stat`'s device number reports the same value for separate filesystems that happen to be mounted on the same backing device. See: - https://manpages.ubuntu.com/manpages/noble/en/man1/cp.1.html - https://manpages.ubuntu.com/manpages/noble/en/man1/df.1.html GUS-W-19603153. --- CHANGELOG.md | 1 + lib/cache.sh | 25 ++++++++++++++++++------- 2 files changed, 19 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 64eb122fe..324eee7bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ - Updated Poetry from 2.1.4 to 2.2.0. ([#1900](https://github.com/heroku/heroku-buildpack-python/pull/1900)) - Updated uv from 0.8.15 to 0.8.18. ([#1899](https://github.com/heroku/heroku-buildpack-python/pull/1899) and [#1901](https://github.com/heroku/heroku-buildpack-python/pull/1901)) +- Improved performance of Python build cache saving. ([#1902](https://github.com/heroku/heroku-buildpack-python/pull/1902)) ## [v307] - 2025-09-10 diff --git a/lib/cache.sh b/lib/cache.sh index 15dd29c8a..9dc5d35cd 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -171,14 +171,25 @@ function cache::save() { output::step "Saving cache" mkdir -p "${cache_dir}/.heroku" - rm -rf "${cache_dir}/.heroku/python" - # In theory we should be able to use `--reflink=auto` here for improved performance, however, - # initial benchmarking showed it to be slower with the file system type / mounts used by the - # Heroku build system for some reason. (Copying was faster using `--link`, however, that fails - # when copying cross-mount such as for Heroku CI and build-in-app-dir, plus hardlinks could - # result in unintended cache mutation if later buildpacks add/remove packages etc.) - cp --recursive "${build_dir}/.heroku/python" "${cache_dir}/.heroku/" + + local build_dir_filesystem cache_dir_filesystem + build_dir_filesystem="$(df --output=target "${build_dir}")" + cache_dir_filesystem="$(df --output=target "${cache_dir}")" + + # For improved performance, we copy using hard-links if possible. This requires that the build + # and cache directory are on the same filesystem mount - which is the case for standard builds + # but not Heroku CI or build-in-app-dir. Ideally we would be able to use `--reflink=auto` here + # (which would avoid the need for a conditional and also mean accidental edits by users in later + # buildpacks to one location doesn't affect the other), however, with the current filesystems + # used in production benchmarking showed `--reflinks=auto` was much slower than hardlinks. + if [[ "${build_dir_filesystem}" == "${cache_dir_filesystem}" ]]; then + local additional_copy_args=(--link) + else + local additional_copy_args=() + fi + + cp --recursive "${additional_copy_args[@]}" "${build_dir}/.heroku/python" "${cache_dir}/.heroku/" # Metadata used by subsequent builds to determine whether the cache can be reused. # These are written/consumed via separate files and not the build data store for compatibility From 2e852f480215383e68856537310461f4e58966ff Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Fri, 19 Sep 2025 07:38:26 +0000 Subject: [PATCH 224/406] Prepare release v308 (#1903) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 324eee7bd..aaaa5d81a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v308] - 2025-09-19 + - Updated Poetry from 2.1.4 to 2.2.0. ([#1900](https://github.com/heroku/heroku-buildpack-python/pull/1900)) - Updated uv from 0.8.15 to 0.8.18. ([#1899](https://github.com/heroku/heroku-buildpack-python/pull/1899) and [#1901](https://github.com/heroku/heroku-buildpack-python/pull/1901)) - Improved performance of Python build cache saving. ([#1902](https://github.com/heroku/heroku-buildpack-python/pull/1902)) @@ -1329,7 +1332,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v307...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v308...main +[v308]: https://github.com/heroku/heroku-buildpack-python/compare/v307...v308 [v307]: https://github.com/heroku/heroku-buildpack-python/compare/v306...v307 [v306]: https://github.com/heroku/heroku-buildpack-python/compare/v305...v306 [v305]: https://github.com/heroku/heroku-buildpack-python/compare/v304...v305 From d99a3376e6865e7022296406d07a195379863b28 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 19 Sep 2025 17:29:45 +0100 Subject: [PATCH 225/406] Add metrics for misspelled `.python-version` files (#1904) Record the names of files similar to .python-version in the root of the app, to determine how often that file is misspelled, as a temporary first step before deciding whether to add a new warning/error (and if so, which for which misspelled filenames it should check). GUS-W-19271516. --- CHANGELOG.md | 1 + lib/python_version.sh | 11 +++++++++++ spec/hatchet/pip_spec.rb | 1 + spec/hatchet/poetry_spec.rb | 1 + spec/hatchet/uv_spec.rb | 1 + 5 files changed, 15 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index aaaa5d81a..e660c79e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Added metrics for misspelled `.python-version` files. ([#1904](https://github.com/heroku/heroku-buildpack-python/pull/1904)) ## [v308] - 2025-09-19 diff --git a/lib/python_version.sh b/lib/python_version.sh index ad95d2003..1884167de 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -50,6 +50,17 @@ function python_version::read_requested_python_version() { declare -n origin="${5}" local contents + # Record the names of files similar to .python-version in the root of the app, to determine + # how often that file is misspelled, as a temporary first step before deciding whether to add + # a new warning/error (and if so, which for which misspelled filenames it should check). + local python_version_files + python_version_files="$( + find . -maxdepth 1 -type f -iregex '\./\.?python.?version.*' -printf '%P\n' | sort | tr '\n' ',' || true + )" + if [[ -n "${python_version_files}" ]]; then + build_data::set_string "python_version_files" "${python_version_files}" + fi + local runtime_txt_path="${build_dir}/runtime.txt" if [[ -f "${runtime_txt_path}" ]]; then contents="$(utils::read_file_with_special_chars_substituted "${runtime_txt_path}")" diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 8dc68798a..d30e14e98 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -72,6 +72,7 @@ remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", + remote: "python_version_files": ".python-version,", remote: "python_version_major": "3.13", remote: "python_version_origin": ".python-version", remote: "python_version_outdated": false, diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 7bbe29f5f..8bfdfdb40 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -74,6 +74,7 @@ remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", + remote: "python_version_files": ".python-version,", remote: "python_version_major": "3.13", remote: "python_version_origin": ".python-version", remote: "python_version_outdated": false, diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 5b32fccf5..16fa03817 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -79,6 +79,7 @@ remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", + remote: "python_version_files": ".python-version,", remote: "python_version_major": "3.13", remote: "python_version_origin": ".python-version", remote: "python_version_outdated": false, From 4d20ad3e50f23e635c46120c1b013df4c1c44f8d Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Fri, 19 Sep 2025 20:18:10 +0100 Subject: [PATCH 226/406] Prepare release v309 (#1905) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e660c79e3..4261b0245 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v309] - 2025-09-19 + - Added metrics for misspelled `.python-version` files. ([#1904](https://github.com/heroku/heroku-buildpack-python/pull/1904)) ## [v308] - 2025-09-19 @@ -1333,7 +1336,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v308...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v309...main +[v309]: https://github.com/heroku/heroku-buildpack-python/compare/v308...v309 [v308]: https://github.com/heroku/heroku-buildpack-python/compare/v307...v308 [v307]: https://github.com/heroku/heroku-buildpack-python/compare/v306...v307 [v306]: https://github.com/heroku/heroku-buildpack-python/compare/v305...v306 From 2debd95a495264aed35c5fc54a3080aa046d703e Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 22 Sep 2025 12:34:50 +0100 Subject: [PATCH 227/406] Improve metrics for failed uv archive downloads (#1908) For parity with what already exists for Python archive downloads: https://github.com/heroku/heroku-buildpack-python/blob/4d20ad3e50f23e635c46120c1b013df4c1c44f8d/lib/python.sh#L120-L121 Example failure this would now track: ``` -----> Installing uv 0.8.18 curl: (56) Recv failure: Connection reset by peer gzip: stdin: unexpected end of file tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now ``` GUS-W-19702437. --- CHANGELOG.md | 1 + lib/uv.sh | 56 +++++++++++++++++++++++++++++++--------------------- 2 files changed, 34 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4261b0245..1296b05f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Improved metrics for failed uv archive downloads. ([#1908](https://github.com/heroku/heroku-buildpack-python/pull/1908)) ## [v309] - 2025-09-19 diff --git a/lib/uv.sh b/lib/uv.sh index c8d86b619..2456291c1 100644 --- a/lib/uv.sh +++ b/lib/uv.sh @@ -29,30 +29,38 @@ function uv::install_uv() { gnu_arch=$(arch) local uv_url="https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-${gnu_arch}-unknown-linux-gnu.tar.gz" + local error_log + error_log=$(mktemp) + + # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { - # We set max-time for improved UX/metrics for hanging downloads compared to relying on the build - # system timeout. We don't use `--speed-limit` since it gives worse error messages when used with - # retries and piping to tar. The Python archives are ~10 MB so only take ~1s to download on Heroku, - # so we set low timeouts to reduce delays before retries. However, we allow customising the timeouts - # to support non-Heroku environments that may be far from `us-east-1` or have a slower connection. - # We use `--no-progress-meter` rather than `--silent` so that retry status messages are printed. - # We have to use `--strip-components` since the uv binary is nested under a subdirectory. - curl \ - --connect-timeout "${CURL_CONNECT_TIMEOUT:-3}" \ - --fail \ - --location \ - --max-time "${CURL_TIMEOUT:-60}" \ - --no-progress-meter \ - --retry-max-time "${CURL_TIMEOUT:-60}" \ - --retry 5 \ - --retry-connrefused \ - "${uv_url}" \ - | tar \ - --directory "${uv_dir}" \ - --extract \ - --gzip \ - --no-anchored \ - --strip-components 1 + { + # We set max-time for improved UX/metrics for hanging downloads compared to relying on the build + # system timeout. We don't use `--speed-limit` since it gives worse error messages when used with + # retries and piping to tar. The uv archives are ~20 MB so only take ~1s to download on Heroku, + # so we set low timeouts to reduce delays before retries. However, we allow customising the timeouts + # to support non-Heroku environments that may be far from `us-east-1` or have a slower connection. + # We use `--no-progress-meter` rather than `--silent` so that retry status messages are printed. + # We have to use `--strip-components` since the uv binary is nested under a subdirectory. + curl \ + --connect-timeout "${CURL_CONNECT_TIMEOUT:-3}" \ + --fail \ + --location \ + --max-time "${CURL_TIMEOUT:-60}" \ + --no-progress-meter \ + --retry-max-time "${CURL_TIMEOUT:-60}" \ + --retry 5 \ + --retry-connrefused \ + "${uv_url}" \ + | tar \ + --directory "${uv_dir}" \ + --extract \ + --gzip \ + --no-anchored \ + --strip-components 1 + } \ + |& tee "${error_log}" \ + |& output::indent }; then output::error <<-EOF Error: Unable to install uv. @@ -69,6 +77,8 @@ function uv::install_uv() { https://www.githubstatus.com EOF build_data::set_string "failure_reason" "install-package-manager::uv" + # e.g.: 'curl: (56) Recv failure: Connection reset by peer' + build_data::set_string "failure_detail" "$(head --lines=1 "${error_log}" || true)" exit 1 fi fi From 62f49737caa1f0a856a1e26d7d1dba27634c3806 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 23 Sep 2025 11:21:21 +0100 Subject: [PATCH 228/406] Use `cp --no-dereference` when saving the cache (#1909) When copying files, `cp` can either follow (dereference) symlinks or preserve them. If no explicit `--dereference` or `--no-dereference` option is passed, the default behaviour is currently unspecified by the POSIX specification, and in practice arbitrarily varies based on other options that are passed. Specifically, when using `--link`, cp now defaults to following symlinks, which: (a) means the files aren't copied/linked verbatim (b) means the copy will fail if any of the symlinks are broken For the files owned by the buildpack there are no broken symlinks, however, Python packages in the wild can contain broken symlinks, which causes the build to fail with errors like: ``` -----> Saving cache cp: cannot stat '/tmp/build_123/.heroku/python/src/django-admin-autocomplete-list-filter/.pylintrc': No such file or directory ``` Now, `--no-dereference` is passed, to explicitly disable dereferencing, so we're not dependant on the unspecified defaults. GUS-W-19714036. --- CHANGELOG.md | 1 + lib/cache.sh | 5 ++++- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1296b05f1..86b210a2d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Fixed errors saving the build cache when installed packages contain broken symlinks. ([#1909](https://github.com/heroku/heroku-buildpack-python/pull/1909)) - Improved metrics for failed uv archive downloads. ([#1908](https://github.com/heroku/heroku-buildpack-python/pull/1908)) ## [v309] - 2025-09-19 diff --git a/lib/cache.sh b/lib/cache.sh index 9dc5d35cd..61e0e7e9e 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -189,7 +189,10 @@ function cache::save() { local additional_copy_args=() fi - cp --recursive "${additional_copy_args[@]}" "${build_dir}/.heroku/python" "${cache_dir}/.heroku/" + # We must explicitly use `--no-dereference` since the default cp behaviour varies based on other + # options used (such as `--link`), and we don't want symlinks to be resolved since otherwise the + # copy will fail when copying packages that contain broken symlinks. + cp --recursive --no-dereference "${additional_copy_args[@]}" "${build_dir}/.heroku/python" "${cache_dir}/.heroku/" # Metadata used by subsequent builds to determine whether the cache can be reused. # These are written/consumed via separate files and not the build data store for compatibility From 337ffbcb1f6ba5c58df516352df649a20e84cea6 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 23 Sep 2025 10:25:55 +0000 Subject: [PATCH 229/406] Bump uv from 0.8.18 to 0.8.20 (#1910) * Bump uv from 0.8.18 to 0.8.20 Bumps [uv](https://github.com/astral-sh/uv) from 0.8.18 to 0.8.20. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.8.18...0.8.20) --- updated-dependencies: - dependency-name: uv dependency-version: 0.8.20 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 86b210a2d..c924ae9b9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.8.18 to 0.8.20. ([#1910](https://github.com/heroku/heroku-buildpack-python/pull/1910)) - Fixed errors saving the build cache when installed packages contain broken symlinks. ([#1909](https://github.com/heroku/heroku-buildpack-python/pull/1909)) - Improved metrics for failed uv archive downloads. ([#1908](https://github.com/heroku/heroku-buildpack-python/pull/1908)) diff --git a/requirements/uv.txt b/requirements/uv.txt index cb5116d05..71dd98fb9 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.8.18 +uv==0.8.20 From 42910fbe31ab0022dcd7de40d53f38a9e90f193c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 23 Sep 2025 10:35:32 +0000 Subject: [PATCH 230/406] Bump poetry from 2.2.0 to 2.2.1 (#1907) * Bump poetry from 2.2.0 to 2.2.1 Bumps [poetry](https://github.com/python-poetry/poetry) from 2.2.0 to 2.2.1. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](https://github.com/python-poetry/poetry/compare/2.2.0...2.2.1) --- updated-dependencies: - dependency-name: poetry dependency-version: 2.2.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/poetry.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c924ae9b9..46a0e54f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated Poetry from 2.2.0 to 2.2.1. ([#1907](https://github.com/heroku/heroku-buildpack-python/pull/1907)) - Updated uv from 0.8.18 to 0.8.20. ([#1910](https://github.com/heroku/heroku-buildpack-python/pull/1910)) - Fixed errors saving the build cache when installed packages contain broken symlinks. ([#1909](https://github.com/heroku/heroku-buildpack-python/pull/1909)) - Improved metrics for failed uv archive downloads. ([#1908](https://github.com/heroku/heroku-buildpack-python/pull/1908)) diff --git a/requirements/poetry.txt b/requirements/poetry.txt index b2589c103..6bd2fa4e5 100644 --- a/requirements/poetry.txt +++ b/requirements/poetry.txt @@ -1 +1 @@ -poetry==2.2.0 +poetry==2.2.1 From 732efcbe2cd734197d89fa0b37f95ee8facdda06 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 23 Sep 2025 10:40:26 +0000 Subject: [PATCH 231/406] Prepare release v310 (#1911) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 46a0e54f5..8246639e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v310] - 2025-09-23 + - Updated Poetry from 2.2.0 to 2.2.1. ([#1907](https://github.com/heroku/heroku-buildpack-python/pull/1907)) - Updated uv from 0.8.18 to 0.8.20. ([#1910](https://github.com/heroku/heroku-buildpack-python/pull/1910)) - Fixed errors saving the build cache when installed packages contain broken symlinks. ([#1909](https://github.com/heroku/heroku-buildpack-python/pull/1909)) @@ -1340,7 +1343,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v309...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v310...main +[v310]: https://github.com/heroku/heroku-buildpack-python/compare/v309...v310 [v309]: https://github.com/heroku/heroku-buildpack-python/compare/v308...v309 [v308]: https://github.com/heroku/heroku-buildpack-python/compare/v307...v308 [v307]: https://github.com/heroku/heroku-buildpack-python/compare/v306...v307 From b2c9f8c4d596a4b91f6ee0d5a8cc997498e48573 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 29 Sep 2025 12:32:42 +0100 Subject: [PATCH 232/406] Remove unnecessary tar `--no-anchored` (#1912) Since it was a leftover from when this tar command also passed the uv binary name to only extract the single file from the archive. --- lib/uv.sh | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/uv.sh b/lib/uv.sh index 2456291c1..48953c0a9 100644 --- a/lib/uv.sh +++ b/lib/uv.sh @@ -56,7 +56,6 @@ function uv::install_uv() { --directory "${uv_dir}" \ --extract \ --gzip \ - --no-anchored \ --strip-components 1 } \ |& tee "${error_log}" \ From a3caefbc1b4e0dc9a20742de081b6bec6debdb87 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 29 Sep 2025 12:32:58 +0100 Subject: [PATCH 233/406] Use a consistent prefix for checks `failure_reason`s (#1913) Since all of the other checks use `checks::*`. --- lib/checks.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/checks.sh b/lib/checks.sh index dd0a75221..36f7f2206 100644 --- a/lib/checks.sh +++ b/lib/checks.sh @@ -19,7 +19,7 @@ function checks::ensure_supported_stack() { Upgrade to a newer stack to continue using this buildpack. EOF - build_data::set_string "failure_reason" "stack::eol" + build_data::set_string "failure_reason" "checks::stack::eol" build_data::set_string "failure_detail" "${stack}" exit 1 ;; @@ -34,7 +34,7 @@ function checks::ensure_supported_stack() { https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references EOF - build_data::set_string "failure_reason" "stack::unknown" + build_data::set_string "failure_reason" "checks::stack::unknown" build_data::set_string "failure_detail" "${stack}" exit 1 ;; From 136c3a138f2e0a8c2b851e3af217aeaa9ad5491e Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 29 Sep 2025 12:35:53 +0100 Subject: [PATCH 234/406] Further expand buildpack detection known file list (#1914) Similar to #1729, this adds more files to the buildpack's detection list - based on files commonly seen for apps that fail detection due to eg misspelled files. GUS-W-19769721. --- CHANGELOG.md | 1 + bin/detect | 9 ++++++--- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8246639e5..45eb7468e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Added more Python project related file and directory names to the list recognised by buildpack detection. ([#1914](https://github.com/heroku/heroku-buildpack-python/pull/1914)) ## [v310] - 2025-09-23 diff --git a/bin/detect b/bin/detect index a45389ffc..9e3f1e753 100755 --- a/bin/detect +++ b/bin/detect @@ -37,14 +37,17 @@ KNOWN_PYTHON_PROJECT_FILES=( uv.lock # Commonly seen misspellings of requirements.txt. (Which occur since pip doesn't # create/manage requirements files itself, so the filenames are manually typed.) + requeriments.txt requirement.txt - Requirements.txt + requirements requirements.text + Requirements.txt requirements.txt.txt requirments.txt - # Whilst virtual environments shouldn't be committed to Git (and so shouldn't - # normally be present during the build), they are often present for beginner + # Whilst the pyc cache and virtual environments shouldn't be committed to Git (and so + # shouldn't normally be present during the build), they are often present for beginner # Python apps that are missing all of the other Python related files above. + __pycache__/ .venv/ venv/ ) From 05767465af8b69e8357d64c83dfafd5edc109143 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 29 Sep 2025 16:56:37 +0100 Subject: [PATCH 235/406] Use `pip install --isolated` when bootstrapping pip/Pipenv/Poetry (#1915) So that any invalid global pip config set by earlier buildpacks (either via `PIP_*` env vars or global pip config files in the HOME directory) don't affect/break the bootstrapping of the package managers. (Such config is still loaded during the later `pip install` used to install the app's own dependencies; this change only affects the internal buildpack bootstrapping stage.) This will help resolve issues seen via Honeycomb such as the bootstrapping step attempting to use custom package index URLs configured via third-party buildpacks like: https://github.com/meetalbert/albert-public-codeartifact-buildpack (which succeeds eventually, but only after delays from timeouts/retries) See: - https://pip.pypa.io/en/stable/cli/pip/#cmdoption-isolated - https://pip.pypa.io/en/stable/topics/configuration/ GUS-W-19770470. --- CHANGELOG.md | 1 + lib/pip.sh | 3 +++ lib/pipenv.sh | 3 +++ lib/poetry.sh | 3 +++ 4 files changed, 10 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 45eb7468e..1efdfa1ea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Changed the `pip install` command used to install the pip, Pipenv and Poetry package managers to now use `--isolated` mode. ([#1915](https://github.com/heroku/heroku-buildpack-python/pull/1915)) - Added more Python project related file and directory names to the list recognised by buildpack detection. ([#1914](https://github.com/heroku/heroku-buildpack-python/pull/1914)) ## [v310] - 2025-09-23 diff --git a/lib/pip.sh b/lib/pip.sh index ffb983c80..147c5f1ba 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -46,11 +46,14 @@ function pip::install_pip_setuptools_wheel() { # app's requirements.txt in the last build). The install will be a no-op if the versions match. output::step "Installing ${packages_display_text}" + # `--isolated`: Prevents any custom pip configuration added by third party buildpacks (via env + # vars or global config files) from breaking package manager bootstrapping. # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { python "${bundled_pip_module_path}" \ install \ --disable-pip-version-check \ + --isolated \ --no-cache-dir \ --no-input \ --quiet \ diff --git a/lib/pipenv.sh b/lib/pipenv.sh index afdf281e9..f81b7100e 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -60,11 +60,14 @@ function pipenv::install_pipenv() { # We must call the venv Python directly here, rather than relying on pip's `--python` # option, since `--python` was only added in pip v22.3, so isn't supported by the older # pip versions bundled with Python 3.9/3.10. + # `--isolated`: Prevents any custom pip configuration added by third party buildpacks (via env + # vars or global config files) from breaking package manager bootstrapping. # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { "${pipenv_venv_dir}/bin/python" "${bundled_pip_module_path}" \ install \ --disable-pip-version-check \ + --isolated \ --no-cache-dir \ --no-input \ --quiet \ diff --git a/lib/poetry.sh b/lib/poetry.sh index 92ad18687..9998f1d04 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -66,11 +66,14 @@ function poetry::install_poetry() { # We must call the venv Python directly here, rather than relying on pip's `--python` # option, since `--python` was only added in pip v22.3, so isn't supported by the older # pip versions bundled with Python 3.9/3.10. + # `--isolated`: Prevents any custom pip configuration added by third party buildpacks (via env + # vars or global config files) from breaking package manager bootstrapping. # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { "${poetry_venv_dir}/bin/python" "${bundled_pip_module_path}" \ install \ --disable-pip-version-check \ + --isolated \ --no-cache-dir \ --no-input \ --quiet \ From 40c627213f65cf3a9ef62d63207727a7668fc661 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 30 Sep 2025 12:51:00 +0100 Subject: [PATCH 236/406] Always use `|&` with `output::indent` (#1917) To ensure any stderr is intended too. (All other call sites currently use `|&`) See: https://www.gnu.org/software/bash/manual/html_node/Pipelines.html --- bin/steps/collectstatic | 2 +- lib/output.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/bin/steps/collectstatic b/bin/steps/collectstatic index cd8f7b710..269c1bf51 100755 --- a/bin/steps/collectstatic +++ b/bin/steps/collectstatic @@ -67,7 +67,7 @@ export PYTHONPATH COLLECTSTATIC_LOG=$(mktemp) set +e -python "${MANAGE_FILE}" collectstatic --noinput --traceback 2>&1 | tee "${COLLECTSTATIC_LOG}" | sed --unbuffered '/^Post-processed/d;/^Copying/d;/^$/d' | output::indent +python "${MANAGE_FILE}" collectstatic --noinput --traceback |& tee "${COLLECTSTATIC_LOG}" |& sed --unbuffered '/^Post-processed/d;/^Copying/d;/^$/d' |& output::indent COLLECTSTATIC_STATUS="${PIPESTATUS[0]}" set -e diff --git a/lib/output.sh b/lib/output.sh index 84aff374b..a4eb9431e 100644 --- a/lib/output.sh +++ b/lib/output.sh @@ -23,7 +23,7 @@ function output::step() { # # Usage: # ``` -# pip install ... | output::indent +# pip install ... |& output::indent # ``` function output::indent() { sed --unbuffered "s/^/ /" From 6dbce78c01e8549178ec6338025d07fa8836774a Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 30 Sep 2025 12:54:23 +0100 Subject: [PATCH 237/406] Stop rewriting Django collectstatic command log output (#1918) Since: - The previously removed output may be useful for debugging. - Ideally output shouldn't differ from what users see locally. - Newer versions of Django are less verbose anyway. GUS-W-19780921. --- CHANGELOG.md | 1 + bin/steps/collectstatic | 2 +- spec/hatchet/django_spec.rb | 5 +++++ 3 files changed, 7 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1efdfa1ea..63eed823c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Stopped rewriting Django collectstatic command log output. ([#1918](https://github.com/heroku/heroku-buildpack-python/pull/1918)) - Changed the `pip install` command used to install the pip, Pipenv and Poetry package managers to now use `--isolated` mode. ([#1915](https://github.com/heroku/heroku-buildpack-python/pull/1915)) - Added more Python project related file and directory names to the list recognised by buildpack detection. ([#1914](https://github.com/heroku/heroku-buildpack-python/pull/1914)) diff --git a/bin/steps/collectstatic b/bin/steps/collectstatic index 269c1bf51..0abed5a56 100755 --- a/bin/steps/collectstatic +++ b/bin/steps/collectstatic @@ -67,7 +67,7 @@ export PYTHONPATH COLLECTSTATIC_LOG=$(mktemp) set +e -python "${MANAGE_FILE}" collectstatic --noinput --traceback |& tee "${COLLECTSTATIC_LOG}" |& sed --unbuffered '/^Post-processed/d;/^Copying/d;/^$/d' |& output::indent +python "${MANAGE_FILE}" collectstatic --noinput --traceback |& tee "${COLLECTSTATIC_LOG}" |& output::indent COLLECTSTATIC_STATUS="${PIPESTATUS[0]}" set -e diff --git a/spec/hatchet/django_spec.rb b/spec/hatchet/django_spec.rb index 4f80d0995..29588d44a 100644 --- a/spec/hatchet/django_spec.rb +++ b/spec/hatchet/django_spec.rb @@ -47,12 +47,14 @@ remote: 'PWD': '/tmp/build_\\w+', remote: 'PYTHONPATH': '\\.', remote: 'PYTHONUNBUFFERED': '1'\\} + remote: remote: \\['/tmp/build_\\w+/backend', remote: '/tmp/build_\\w+', remote: '/app/.heroku/python/lib/python313.zip', remote: '/app/.heroku/python/lib/python3.13', remote: '/app/.heroku/python/lib/python3.13/lib-dynload', remote: '/app/.heroku/python/lib/python3.13/site-packages'\\] + remote: remote: 1 static file copied to '/tmp/build_\\w+/backend/staticfiles'. remote: remote: -----> Saving cache @@ -86,12 +88,15 @@ remote: 'PWD': '/tmp/build_\\w+', remote: 'PYTHONPATH': '/invalid-path', remote: 'PYTHONUNBUFFERED': '1'\\} + remote: remote: \\['/tmp/build_\\w+', remote: '/invalid-path', remote: '/app/.heroku/python/lib/python39.zip', remote: '/app/.heroku/python/lib/python3.9', remote: '/app/.heroku/python/lib/python3.9/lib-dynload', remote: '/app/.heroku/python/lib/python3.9/site-packages'\\] + remote: Copying '/tmp/build_\\w+/testapp/static/robots.txt' + remote: remote: 1 static file copied to '/tmp/build_\\w+/staticfiles'. remote: remote: -----> Saving cache From 4370ee711dd1052fb6e02119079ee19769069e05 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 30 Sep 2025 20:12:24 +0100 Subject: [PATCH 238/406] Prepare release v311 (#1919) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 63eed823c..550bf622c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v311] - 2025-09-30 + - Stopped rewriting Django collectstatic command log output. ([#1918](https://github.com/heroku/heroku-buildpack-python/pull/1918)) - Changed the `pip install` command used to install the pip, Pipenv and Poetry package managers to now use `--isolated` mode. ([#1915](https://github.com/heroku/heroku-buildpack-python/pull/1915)) - Added more Python project related file and directory names to the list recognised by buildpack detection. ([#1914](https://github.com/heroku/heroku-buildpack-python/pull/1914)) @@ -1346,7 +1349,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v310...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v311...main +[v311]: https://github.com/heroku/heroku-buildpack-python/compare/v310...v311 [v310]: https://github.com/heroku/heroku-buildpack-python/compare/v309...v310 [v309]: https://github.com/heroku/heroku-buildpack-python/compare/v308...v309 [v308]: https://github.com/heroku/heroku-buildpack-python/compare/v307...v308 From 8888cd40a7507531a0b976c80b7f8eaee5c46206 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 1 Oct 2025 15:10:56 +0100 Subject: [PATCH 239/406] Bump actions/checkout from 4 to 5 (#1921) Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4...v5) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/build_python_runtime.yml | 4 ++-- .github/workflows/check_changelog.yml | 2 +- .github/workflows/ci.yml | 6 +++--- .github/workflows/hatchet_app_cleaner.yml | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/.github/workflows/build_python_runtime.yml b/.github/workflows/build_python_runtime.yml index ad6b05a98..11ff2f571 100644 --- a/.github/workflows/build_python_runtime.yml +++ b/.github/workflows/build_python_runtime.yml @@ -42,7 +42,7 @@ jobs: STACK_VERSION: "22" steps: - name: Checkout - uses: actions/checkout@v4 + uses: actions/checkout@v5 - name: Build Docker image run: docker build --platform="linux/amd64" --pull --tag buildenv --build-arg=STACK_VERSION builds/ - name: Compile and package Python runtime @@ -67,7 +67,7 @@ jobs: STACK_VERSION: "24" steps: - name: Checkout - uses: actions/checkout@v4 + uses: actions/checkout@v5 - name: Build Docker image run: docker build --platform="linux/${{ matrix.arch }}" --pull --tag buildenv --build-arg=STACK_VERSION builds/ - name: Compile and package Python runtime diff --git a/.github/workflows/check_changelog.yml b/.github/workflows/check_changelog.yml index e528aae03..f4b8c100e 100644 --- a/.github/workflows/check_changelog.yml +++ b/.github/workflows/check_changelog.yml @@ -13,7 +13,7 @@ jobs: if: (!contains(github.event.pull_request.labels.*.name, 'skip changelog')) steps: - name: Checkout - uses: actions/checkout@v4 + uses: actions/checkout@v5 - name: Check that CHANGELOG is touched run: | git fetch origin ${{ github.base_ref }} --depth 1 && \ diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 90310deb8..867e0ab61 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -19,7 +19,7 @@ jobs: runs-on: ubuntu-24.04 steps: - name: Checkout - uses: actions/checkout@v4 + uses: actions/checkout@v5 - name: Install Ruby and dependencies uses: ruby/setup-ruby@v1 with: @@ -51,7 +51,7 @@ jobs: RSPEC_RETRY_RETRY_COUNT: 1 steps: - name: Checkout - uses: actions/checkout@v4 + uses: actions/checkout@v5 - name: Install Ruby and dependencies uses: ruby/setup-ruby@v1 with: @@ -67,7 +67,7 @@ jobs: runs-on: ubuntu-24.04 steps: - name: Checkout - uses: actions/checkout@v4 + uses: actions/checkout@v5 # These test both the local development `make run` workflow and that `bin/report` completes successfully # for both passing and failing builds (since `bin/report` can't easily be tested via Hatchet tests). - name: Run buildpack using default app fixture diff --git a/.github/workflows/hatchet_app_cleaner.yml b/.github/workflows/hatchet_app_cleaner.yml index aa7a88512..8cd69aa75 100644 --- a/.github/workflows/hatchet_app_cleaner.yml +++ b/.github/workflows/hatchet_app_cleaner.yml @@ -19,7 +19,7 @@ jobs: HEROKU_DISABLE_AUTOUPDATE: 1 steps: - name: Checkout - uses: actions/checkout@v4 + uses: actions/checkout@v5 - name: Install Ruby and dependencies uses: ruby/setup-ruby@v1 with: From 62ac1f97cf26324f8f9019901d053fc18537ca98 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 1 Oct 2025 15:11:06 +0100 Subject: [PATCH 240/406] Bump the ruby-dependencies group with 2 updates (#1920) Bumps the ruby-dependencies group with 2 updates: [rubocop](https://github.com/rubocop/rubocop) and [rubocop-rspec](https://github.com/rubocop/rubocop-rspec). Updates `rubocop` from 1.79.2 to 1.81.1 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.79.2...v1.81.1) Updates `rubocop-rspec` from 3.6.0 to 3.7.0 - [Release notes](https://github.com/rubocop/rubocop-rspec/releases) - [Changelog](https://github.com/rubocop/rubocop-rspec/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop-rspec/compare/v3.6.0...v3.7.0) --- updated-dependencies: - dependency-name: rubocop dependency-version: 1.81.1 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies - dependency-name: rubocop-rspec dependency-version: 3.7.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index dcba3c5e0..7058320cf 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -20,7 +20,7 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.13.2) + json (2.15.0) language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) @@ -37,11 +37,11 @@ GEM heroics (~> 0.1.1) moneta (~> 1.0.0) rate_throttle_client (~> 0.1.0) - prism (1.4.0) + prism (1.5.1) racc (1.8.1) rainbow (3.1.1) rate_throttle_client (0.1.2) - regexp_parser (2.11.0) + regexp_parser (2.11.3) rrrretry (1.0.0) rspec-core (3.13.5) rspec-support (~> 3.13.0) @@ -51,7 +51,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.4) - rubocop (1.79.2) + rubocop (1.81.1) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) @@ -59,21 +59,21 @@ GEM parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) regexp_parser (>= 2.9.3, < 3.0) - rubocop-ast (>= 1.46.0, < 2.0) + rubocop-ast (>= 1.47.1, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 4.0) - rubocop-ast (1.46.0) + rubocop-ast (1.47.1) parser (>= 3.3.7.2) prism (~> 1.4) - rubocop-rspec (3.6.0) + rubocop-rspec (3.7.0) lint_roller (~> 1.1) rubocop (~> 1.72, >= 1.72.1) ruby-progressbar (1.13.0) thor (1.4.0) threaded (0.0.4) - unicode-display_width (3.1.4) - unicode-emoji (~> 4.0, >= 4.0.4) - unicode-emoji (4.0.4) + unicode-display_width (3.2.0) + unicode-emoji (~> 4.1) + unicode-emoji (4.1.0) webrick (1.9.1) PLATFORMS From 540b3c93dbbfc765cec27a540cab8791ca49e4f3 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 3 Oct 2025 08:10:09 +0000 Subject: [PATCH 241/406] Bump uv from 0.8.20 to 0.8.22 (#1916) * Bump uv from 0.8.20 to 0.8.22 Bumps [uv](https://github.com/astral-sh/uv) from 0.8.20 to 0.8.22. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.8.20...0.8.22) --- updated-dependencies: - dependency-name: uv dependency-version: 0.8.22 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 550bf622c..9881170d4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.8.20 to 0.8.22. ([#1916](https://github.com/heroku/heroku-buildpack-python/pull/1916)) ## [v311] - 2025-09-30 diff --git a/requirements/uv.txt b/requirements/uv.txt index 71dd98fb9..fc6c3fbc1 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.8.20 +uv==0.8.22 From eee15727d29b4af255992d8355e59f61a1480d63 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 5 Oct 2025 09:25:06 +0000 Subject: [PATCH 242/406] Bump uv from 0.8.22 to 0.8.23 (#1922) * Bump uv from 0.8.22 to 0.8.23 Bumps [uv](https://github.com/astral-sh/uv) from 0.8.22 to 0.8.23. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.8.22...0.8.23) --- updated-dependencies: - dependency-name: uv dependency-version: 0.8.23 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Update changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 2 +- requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9881170d4..34ce68d72 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## [Unreleased] -- Updated uv from 0.8.20 to 0.8.22. ([#1916](https://github.com/heroku/heroku-buildpack-python/pull/1916)) +- Updated uv from 0.8.20 to 0.8.23. ([#1916](https://github.com/heroku/heroku-buildpack-python/pull/1916) and [#1922](https://github.com/heroku/heroku-buildpack-python/pull/1922)) ## [v311] - 2025-09-30 diff --git a/requirements/uv.txt b/requirements/uv.txt index fc6c3fbc1..61e7fc365 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.8.22 +uv==0.8.23 From a361dfacd967c87fae198cfac24134651a09a844 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Sun, 5 Oct 2025 23:32:12 +0100 Subject: [PATCH 243/406] Prepare release v312 (#1923) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 34ce68d72..0a6068f8a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v312] - 2025-10-05 + - Updated uv from 0.8.20 to 0.8.23. ([#1916](https://github.com/heroku/heroku-buildpack-python/pull/1916) and [#1922](https://github.com/heroku/heroku-buildpack-python/pull/1922)) ## [v311] - 2025-09-30 @@ -1350,7 +1353,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v311...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v312...main +[v312]: https://github.com/heroku/heroku-buildpack-python/compare/v311...v312 [v311]: https://github.com/heroku/heroku-buildpack-python/compare/v310...v311 [v310]: https://github.com/heroku/heroku-buildpack-python/compare/v309...v310 [v309]: https://github.com/heroku/heroku-buildpack-python/compare/v308...v309 From 5f8ddb69cae167c5dc4ca8af8d9549ec3f7e80b6 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 6 Oct 2025 12:15:05 +0100 Subject: [PATCH 244/406] Switch from GPG to Sigstore for Python source verification (#1924) The build scripts that download and compile the Python source archives (for upload to S3, where they are then consumed by the buildpack during customer builds) currently use GPG to verify the Python source archive downloads. However, use of PGP signatures for Python artifact verification was deprecated previously in PEP 761, in favour of Sigstore: https://peps.python.org/pep-0761/ https://www.python.org/downloads/metadata/sigstore/ Until now the PGP signatures have still been available for all stable releases, however, as of Python 3.14 (due to be released this week), Sigstore will be the only supported verification mechanism: https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-no-more-pgp As such, we must now switch over to Sigstore. We use the `cosign` CLI for verification since it's a standalone binary available via a Docker image, rather than the Python `sigstore` CLI which requires a Python environment (and so would need pip, venv etc, and more setup to ensure it stays isolated from the Python we're trying to build). See: - https://www.python.org/downloads/metadata/sigstore/ - https://docs.sigstore.dev/cosign/system_config/installation/#container-images - https://docs.sigstore.dev/cosign/verifying/verify/ - https://github.com/sigstore/cosign/blob/main/doc/cosign_verify-blob.md GUS-W-18244071. --- .github/dependabot.yml | 8 ++++++++ builds/Dockerfile | 2 ++ builds/build_python_runtime.sh | 31 +++++++++++++++---------------- 3 files changed, 25 insertions(+), 16 deletions(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index ee3475f73..fa1d4813e 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -13,6 +13,14 @@ updates: update-types: - "minor" - "patch" + - package-ecosystem: "docker" + directory: "/" + schedule: + interval: "monthly" + labels: + - "dependencies" + - "docker" + - "skip changelog" - package-ecosystem: "github-actions" directory: "/" schedule: diff --git a/builds/Dockerfile b/builds/Dockerfile index cc16e9b25..8e83a738a 100644 --- a/builds/Dockerfile +++ b/builds/Dockerfile @@ -1,4 +1,5 @@ ARG STACK_VERSION="24" +FROM ghcr.io/sigstore/cosign/cosign:v2.6.1@sha256:68839b7f13dac5a6744a5d8818e984dd39183374e37855c19e14d623d9bc9037 AS cosign FROM heroku/heroku:${STACK_VERSION}-build ARG STACK_VERSION @@ -13,6 +14,7 @@ RUN apt-get update --error-on=any \ libreadline-dev \ libsqlite3-dev \ && rm -rf /var/lib/apt/lists/* +COPY --from=cosign /ko-app/cosign /usr/local/bin/cosign WORKDIR /tmp COPY build_python_runtime.sh python-3.13-ubuntu-22.04-libexpat-workaround.patch . diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index 7def990af..18a8dc5ea 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -43,23 +43,19 @@ if [[ " ${SUPPORTED_PYTHON_VERSIONS[*]} " != *" ${PYTHON_MAJOR_VERSION} "* ]]; t abort "Python ${PYTHON_MAJOR_VERSION} isn't supported on ${STACK}!" fi -# The release keys can be found on https://www.python.org/downloads/ -> "OpenPGP Public Keys". +# Sigstore identities taken from: https://www.python.org/downloads/metadata/sigstore/ case "${PYTHON_MAJOR_VERSION}" in - 3.13) - # https://github.com/Yhg1s.gpg - GPG_KEY_FINGERPRINT='7169605F62C751356D054A26A821E680E5FA6305' - ;; - 3.12) - # https://github.com/Yhg1s.gpg - GPG_KEY_FINGERPRINT='7169605F62C751356D054A26A821E680E5FA6305' + 3.12 | 3.13) + SIGSTORE_IDENTITY='thomas@python.org' + SIGSTORE_ISSUER='https://accounts.google.com' ;; 3.10 | 3.11) - # https://keybase.io/pablogsal/ - GPG_KEY_FINGERPRINT='A035C8C19219BA821ECEA86B64E628F8D684696D' + SIGSTORE_IDENTITY='pablogsal@python.org' + SIGSTORE_ISSUER='https://accounts.google.com' ;; 3.9) - # https://keybase.io/ambv/ - GPG_KEY_FINGERPRINT='E3FF2839C048B25C084DEBE9B26995E310250568' + SIGSTORE_IDENTITY='lukasz@langa.pl' + SIGSTORE_ISSUER='https://github.com/login/oauth' ;; *) abort "Unsupported Python version '${PYTHON_MAJOR_VERSION}'!" @@ -69,17 +65,20 @@ esac echo "Building Python ${PYTHON_VERSION} for ${STACK} (${ARCH})..." SOURCE_URL="https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz" -SIGNATURE_URL="${SOURCE_URL}.asc" +SIGSTORE_BUNDLE_URL="${SOURCE_URL}.sigstore" set -o xtrace mkdir -p "${SRC_DIR}" "${INSTALL_DIR}" "${UPLOAD_DIR}" curl --fail --retry 5 --retry-connrefused --connect-timeout 3 --max-time 30 -o python.tgz "${SOURCE_URL}" -curl --fail --retry 5 --retry-connrefused --connect-timeout 3 --max-time 30 -o python.tgz.asc "${SIGNATURE_URL}" +curl --fail --retry 5 --retry-connrefused --connect-timeout 3 --max-time 30 -o python.tgz.sigstore "${SIGSTORE_BUNDLE_URL}" -gpg --batch --verbose --recv-keys "${GPG_KEY_FINGERPRINT}" -gpg --batch --verify python.tgz.asc python.tgz +cosign verify-blob \ + --bundle python.tgz.sigstore \ + --certificate-identity "${SIGSTORE_IDENTITY}" \ + --certificate-oidc-issuer "${SIGSTORE_ISSUER}" \ + python.tgz tar --extract --file python.tgz --strip-components=1 --directory "${SRC_DIR}" cd "${SRC_DIR}" From 227fcbf2f8156cfc9f676edc373d9b114c865e82 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 6 Oct 2025 14:16:03 +0100 Subject: [PATCH 245/406] Stop compiling Python using `--with-system-expat` (#1925) The upstream Python test suite (which gets run when compiling with PGO enabled) fails with the `libexpat` in Ubuntu 22.04. In #1661, I previously added what I hoped would be a temporarily workaround until the failures were fixed upstream, however, the Python maintainers say they don't guarantee compatibility with distro `expat`, and that it's up to us to test for compatibility and patch if we want to use the distro version. However, this isn't viable given that we're neither a Linux distro maintainer, a CPython maintainer or an expat maintainer. Instead, like the upstream Docker Hub Python images (who were also affected by this issue), we will switch the `expat` bundled within the CPython sources, which is actually what the upstream CPython project tests in its CI. This means users won't get security updates for free via the base image, and will instead need to update their Python patch versions instead as newer versions are vendored in CPython. However, this is the least worst alternative for now. I'm doing this now, since otherwise I'll need to generate another patch series for the soon to be released Python 3.14. Note: This change only affects Python versions compiled/released after this merges. Existing Python versions on S3 are unaffected for now (unless they ever get recompiled in the future). For more details, see: https://github.com/python/cpython/issues/125067#issuecomment-2517445480 GUS-W-17414073. --- CHANGELOG.md | 1 + builds/Dockerfile | 2 +- builds/build_python_runtime.sh | 11 --------- ....13-ubuntu-22.04-libexpat-workaround.patch | 24 ------------------- 4 files changed, 2 insertions(+), 36 deletions(-) delete mode 100644 builds/python-3.13-ubuntu-22.04-libexpat-workaround.patch diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a6068f8a..ac8b53a59 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Stopped using `--with-system-expat` when compiling new Python versions. ([#1925](https://github.com/heroku/heroku-buildpack-python/pull/1925)) ## [v312] - 2025-10-05 diff --git a/builds/Dockerfile b/builds/Dockerfile index 8e83a738a..e9f0703fc 100644 --- a/builds/Dockerfile +++ b/builds/Dockerfile @@ -17,4 +17,4 @@ RUN apt-get update --error-on=any \ COPY --from=cosign /ko-app/cosign /usr/local/bin/cosign WORKDIR /tmp -COPY build_python_runtime.sh python-3.13-ubuntu-22.04-libexpat-workaround.patch . +COPY build_python_runtime.sh . diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index 18a8dc5ea..f78e72b35 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -83,14 +83,6 @@ cosign verify-blob \ tar --extract --file python.tgz --strip-components=1 --directory "${SRC_DIR}" cd "${SRC_DIR}" -# Work around PGO profile test failures with Python 3.13 on Ubuntu 22.04, due to the tests -# checking the raw libexpat version which doesn't account for Ubuntu backports: -# https://github.com/heroku/heroku-buildpack-python/pull/1661#issuecomment-2405259352 -# https://github.com/python/cpython/issues/125067 -if [[ "${PYTHON_MAJOR_VERSION}" == "3.13" && "${STACK}" == "heroku-22" ]]; then - patch -p1 Date: Thu, 9 Oct 2025 14:19:38 +0100 Subject: [PATCH 246/406] Add support for Python 3.14 (#1927) Add support for Python 3.14, and release 3.14.0. The default Python version remains unchanged (at 3.13) for now. Release announcement: https://blog.python.org/2025/10/python-3140-final-is-here.html https://www.python.org/downloads/release/python-3140/ Details on what's new in Python 3.14: https://docs.python.org/3.14/whatsnew/3.14.html The stdlib now has native zstd support, so I've updated the binary testing script to test that (optional) module was compiled. I've also expanded the command name tests to test the version specific binary exists too (eg `python3.14` and not just `python3` / `python`). Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/18317304753 Python 3.14 readiness status of the top 360 packages on PyPI: https://pyreadiness.org/3.14/ Note: We don't yet support the new free-threaded build variants (e.g.`3.14t`) since they will require further changes to version handling etc. Closes #1926. GUS-W-17595543. GUS-W-17595545. --- CHANGELOG.md | 1 + README.md | 1 + builds/build_python_runtime.sh | 5 +++++ builds/test_python_runtime.sh | 14 ++++++++++++-- lib/python_version.sh | 4 +++- spec/fixtures/python_3.14/.python-version | 2 ++ spec/fixtures/python_3.14/requirements.txt | 2 ++ spec/hatchet/python_version_spec.rb | 16 +++++++++++----- spec/spec_helper.rb | 1 + 9 files changed, 38 insertions(+), 8 deletions(-) create mode 100644 spec/fixtures/python_3.14/.python-version create mode 100644 spec/fixtures/python_3.14/requirements.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index ac8b53a59..630e05201 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Added support for Python 3.14. ([#1927](https://github.com/heroku/heroku-buildpack-python/pull/1927)) - Stopped using `--with-system-expat` when compiling new Python versions. ([#1925](https://github.com/heroku/heroku-buildpack-python/pull/1925)) ## [v312] - 2025-10-05 diff --git a/README.md b/README.md index 506c28646..d8a273ef7 100644 --- a/README.md +++ b/README.md @@ -47,6 +47,7 @@ The current default Python version is: 3.13 The supported Python versions are: +- Python 3.14 - Python 3.13 - Python 3.12 - Python 3.11 diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index f78e72b35..6dbb8d045 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -32,6 +32,7 @@ case "${STACK:?}" in "3.11" "3.12" "3.13" + "3.14" ) ;; *) @@ -45,6 +46,10 @@ fi # Sigstore identities taken from: https://www.python.org/downloads/metadata/sigstore/ case "${PYTHON_MAJOR_VERSION}" in + 3.14) + SIGSTORE_IDENTITY='hugo@python.org' + SIGSTORE_ISSUER='https://github.com/login/oauth' + ;; 3.12 | 3.13) SIGSTORE_IDENTITY='thomas@python.org' SIGSTORE_ISSUER='https://accounts.google.com' diff --git a/builds/test_python_runtime.sh b/builds/test_python_runtime.sh index 020b4dc32..adc701e4f 100755 --- a/builds/test_python_runtime.sh +++ b/builds/test_python_runtime.sh @@ -21,10 +21,12 @@ export LD_LIBRARY_PATH="${INSTALL_DIR}/lib/" tar --zstd --extract --verbose --file "${ARCHIVE_FILEPATH}" --directory "${INSTALL_DIR}" -# Check Python is able to start and is usable via both the default `python3` command and the -# `python` symlink we create during the build. We use the full filepath rather than adding the +# Check Python is able to start and is usable via both the default `python3` + `python3.XX` commands +# and the `python` symlink we create during the build. We use the full filepath rather than adding the # directory to PATH to ensure the test doesn't pass because of falling through to system Python. "${INSTALL_DIR}/bin/python3" --version +major_python_version="$("${INSTALL_DIR}/bin/python3" -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')")" +"${INSTALL_DIR}/bin/python${major_python_version}" --version "${INSTALL_DIR}/bin/python" --version # Check the Python config script still exists/works after the deletion of scripts with broken shebang lines. @@ -65,6 +67,14 @@ optional_stdlib_modules=( xml.parsers.expat zlib ) + +# https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-zstandard +if [[ "${major_python_version}" == "3.14" ]]; then + optional_stdlib_modules+=( + compression.zstd + ) +fi + if "${INSTALL_DIR}/bin/python3" -c "import $( IFS=, echo "${optional_stdlib_modules[*]}" diff --git a/lib/python_version.sh b/lib/python_version.sh index 1884167de..2739f1369 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -9,9 +9,10 @@ LATEST_PYTHON_3_10="3.10.18" LATEST_PYTHON_3_11="3.11.13" LATEST_PYTHON_3_12="3.12.11" LATEST_PYTHON_3_13="3.13.7" +LATEST_PYTHON_3_14="3.14.0" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=9 -NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=13 +NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=14 DEFAULT_PYTHON_FULL_VERSION="${LATEST_PYTHON_3_13}" DEFAULT_PYTHON_MAJOR_VERSION="${DEFAULT_PYTHON_FULL_VERSION%.*}" @@ -420,6 +421,7 @@ function python_version::resolve_python_version() { 3.11) echo "${LATEST_PYTHON_3_11}" ;; 3.12) echo "${LATEST_PYTHON_3_12}" ;; 3.13) echo "${LATEST_PYTHON_3_13}" ;; + 3.14) echo "${LATEST_PYTHON_3_14}" ;; *) utils::abort_internal_error "Unhandled Python major version: ${requested_python_version}" ;; esac } diff --git a/spec/fixtures/python_3.14/.python-version b/spec/fixtures/python_3.14/.python-version new file mode 100644 index 000000000..70dc33adb --- /dev/null +++ b/spec/fixtures/python_3.14/.python-version @@ -0,0 +1,2 @@ +# Comments are supported +3.14 diff --git a/spec/fixtures/python_3.14/requirements.txt b/spec/fixtures/python_3.14/requirements.txt new file mode 100644 index 000000000..eec3a2223 --- /dev/null +++ b/spec/fixtures/python_3.14/requirements.txt @@ -0,0 +1,2 @@ +# This package has been picked since it has no dependencies and is small/fast to install. +typing-extensions==4.12.2 diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index cfb987a9c..dddac2886 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -5,12 +5,12 @@ RSpec.shared_examples 'builds with the requested Python version' do |requested_version, resolved_version| it "builds with Python #{requested_version}" do app.deploy do |app| - if requested_version == '3.13' + if ['3.9', '3.10', '3.11', '3.12'].include?(requested_version) expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python #{requested_version} specified in .python-version remote: -----> Installing Python #{resolved_version} - remote: -----> Installing pip #{PIP_VERSION} + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT @@ -19,7 +19,7 @@ remote: -----> Python app detected remote: -----> Using Python #{requested_version} specified in .python-version remote: -----> Installing Python #{resolved_version} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) OUTPUT @@ -242,7 +242,7 @@ remote: ! Alternatively, request an older Python version by creating remote: ! a .python-version file in the root directory of your app, remote: ! that contains a Python version like: - remote: ! 3.13 + remote: ! 3.14 remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -310,6 +310,12 @@ it_behaves_like 'builds with the requested Python version', '3.13', LATEST_PYTHON_3_13 end + context 'when .python-version contains Python 3.14' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.14') } + + it_behaves_like 'builds with the requested Python version', '3.14', LATEST_PYTHON_3_14 + end + context 'when .python-version contains an invalid Python version string' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_file_invalid_version', allow_failure: true) } @@ -460,7 +466,7 @@ remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#view-your-buildpacks remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#classic-buildpacks-references remote: ! - remote: ! Otherwise, switch to a supported version (such as Python 3.13) + remote: ! Otherwise, switch to a supported version (such as Python 3.14) remote: ! by changing the version in your .python-version file. remote: remote: ! Push rejected, failed to compile Python app. diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 09beb2fa1..07cb7500b 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -15,6 +15,7 @@ LATEST_PYTHON_3_11 = '3.11.13' LATEST_PYTHON_3_12 = '3.12.11' LATEST_PYTHON_3_13 = '3.13.7' +LATEST_PYTHON_3_14 = '3.14.0' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_13 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From aed5485ac01d6278e4db7d59c6a465fbb23c5c49 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 9 Oct 2025 17:20:50 +0100 Subject: [PATCH 247/406] Add support for Python 3.13.8 (#1928) Release announcement: https://blog.python.org/2025/10/python-3138-is-now-available.html Changelog: https://docs.python.org/release/3.13.8/whatsnew/changelog.html#python-3-13-8 Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/18379072906 GUS-W-17604362. --- CHANGELOG.md | 1 + lib/python_version.sh | 2 +- spec/spec_helper.rb | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 630e05201..6c97dcc9d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Added support for Python 3.14. ([#1927](https://github.com/heroku/heroku-buildpack-python/pull/1927)) +- The Python 3.13 version alias now resolves to Python 3.13.8. ([#1928](https://github.com/heroku/heroku-buildpack-python/pull/1928)) - Stopped using `--with-system-expat` when compiling new Python versions. ([#1925](https://github.com/heroku/heroku-buildpack-python/pull/1925)) ## [v312] - 2025-10-05 diff --git a/lib/python_version.sh b/lib/python_version.sh index 2739f1369..ae193b615 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -8,7 +8,7 @@ LATEST_PYTHON_3_9="3.9.23" LATEST_PYTHON_3_10="3.10.18" LATEST_PYTHON_3_11="3.11.13" LATEST_PYTHON_3_12="3.12.11" -LATEST_PYTHON_3_13="3.13.7" +LATEST_PYTHON_3_13="3.13.8" LATEST_PYTHON_3_14="3.14.0" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=9 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 07cb7500b..d722363f6 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -14,7 +14,7 @@ LATEST_PYTHON_3_10 = '3.10.18' LATEST_PYTHON_3_11 = '3.11.13' LATEST_PYTHON_3_12 = '3.12.11' -LATEST_PYTHON_3_13 = '3.13.7' +LATEST_PYTHON_3_13 = '3.13.8' LATEST_PYTHON_3_14 = '3.14.0' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_13 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From 5583affb64208528edb8764d3d1ebc5e61ad68b1 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 9 Oct 2025 20:55:51 +0100 Subject: [PATCH 248/406] Add support for Python 3.12.12, 3.11.14, 3.10.19, 3.9.24 (#1929) Changelogs: https://docs.python.org/release/3.12.12/whatsnew/changelog.html#python-3-12-12-final https://docs.python.org/release/3.11.14/whatsnew/changelog.html#python-3-11-14-final https://docs.python.org/release/3.10.19/whatsnew/changelog.html#python-3-10-19-final https://docs.python.org/release/3.9.24/whatsnew/changelog.html#python-3-9-24-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/18379078616 https://github.com/heroku/heroku-buildpack-python/actions/runs/18383971694 https://github.com/heroku/heroku-buildpack-python/actions/runs/18383611414 https://github.com/heroku/heroku-buildpack-python/actions/runs/18383614298 GUS-W-19859395. --- CHANGELOG.md | 4 ++++ lib/python_version.sh | 8 ++++---- spec/spec_helper.rb | 8 ++++---- 3 files changed, 12 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c97dcc9d..426f8d1a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,10 @@ - Added support for Python 3.14. ([#1927](https://github.com/heroku/heroku-buildpack-python/pull/1927)) - The Python 3.13 version alias now resolves to Python 3.13.8. ([#1928](https://github.com/heroku/heroku-buildpack-python/pull/1928)) +- The Python 3.12 version alias now resolves to Python 3.12.12. ([#1929](https://github.com/heroku/heroku-buildpack-python/pull/1929)) +- The Python 3.11 version alias now resolves to Python 3.11.14. ([#1929](https://github.com/heroku/heroku-buildpack-python/pull/1929)) +- The Python 3.10 version alias now resolves to Python 3.10.19. ([#1929](https://github.com/heroku/heroku-buildpack-python/pull/1929)) +- The Python 3.9 version alias now resolves to Python 3.9.24. ([#1929](https://github.com/heroku/heroku-buildpack-python/pull/1929)) - Stopped using `--with-system-expat` when compiling new Python versions. ([#1925](https://github.com/heroku/heroku-buildpack-python/pull/1925)) ## [v312] - 2025-10-05 diff --git a/lib/python_version.sh b/lib/python_version.sh index ae193b615..69e0841e8 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -4,10 +4,10 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail -LATEST_PYTHON_3_9="3.9.23" -LATEST_PYTHON_3_10="3.10.18" -LATEST_PYTHON_3_11="3.11.13" -LATEST_PYTHON_3_12="3.12.11" +LATEST_PYTHON_3_9="3.9.24" +LATEST_PYTHON_3_10="3.10.19" +LATEST_PYTHON_3_11="3.11.14" +LATEST_PYTHON_3_12="3.12.12" LATEST_PYTHON_3_13="3.13.8" LATEST_PYTHON_3_14="3.14.0" diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index d722363f6..3233ca69e 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -10,10 +10,10 @@ FIXTURE_DIR = Pathname.new(__FILE__).parent.join('fixtures') -LATEST_PYTHON_3_9 = '3.9.23' -LATEST_PYTHON_3_10 = '3.10.18' -LATEST_PYTHON_3_11 = '3.11.13' -LATEST_PYTHON_3_12 = '3.12.11' +LATEST_PYTHON_3_9 = '3.9.24' +LATEST_PYTHON_3_10 = '3.10.19' +LATEST_PYTHON_3_11 = '3.11.14' +LATEST_PYTHON_3_12 = '3.12.12' LATEST_PYTHON_3_13 = '3.13.8' LATEST_PYTHON_3_14 = '3.14.0' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_13 From 72690d3b3c07ad873e1f3fb8662ea6281f9e9a9b Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 9 Oct 2025 21:11:56 +0100 Subject: [PATCH 249/406] Prepare release v313 (#1930) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 426f8d1a8..4026ec280 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v313] - 2025-10-09 + - Added support for Python 3.14. ([#1927](https://github.com/heroku/heroku-buildpack-python/pull/1927)) - The Python 3.13 version alias now resolves to Python 3.13.8. ([#1928](https://github.com/heroku/heroku-buildpack-python/pull/1928)) - The Python 3.12 version alias now resolves to Python 3.12.12. ([#1929](https://github.com/heroku/heroku-buildpack-python/pull/1929)) @@ -1360,7 +1363,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v312...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v313...main +[v313]: https://github.com/heroku/heroku-buildpack-python/compare/v312...v313 [v312]: https://github.com/heroku/heroku-buildpack-python/compare/v311...v312 [v311]: https://github.com/heroku/heroku-buildpack-python/compare/v310...v311 [v310]: https://github.com/heroku/heroku-buildpack-python/compare/v309...v310 From fc63f31ff9aa39963cd33cbda70f645008f5575a Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 14 Oct 2025 14:49:37 +0100 Subject: [PATCH 250/406] Use a consistent `failure_reason` prefix for all internal errors (#1932) Always use the prefix `internal-error::` for the buildpack build data event `failure_reason` field for internal errors, in preparation for adding an ERR trap with a new `internal-error::` variant. GUS-W-19792272. --- lib/pipenv.sh | 2 +- lib/poetry.sh | 2 +- lib/utils.sh | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/pipenv.sh b/lib/pipenv.sh index f81b7100e..0cb42ab58 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -50,7 +50,7 @@ function pipenv::install_pipenv() { See the log output above for more information. EOF - build_data::set_string "failure_reason" "create-venv::pipenv" + build_data::set_string "failure_reason" "internal-error::create-venv::pipenv" exit 1 fi diff --git a/lib/poetry.sh b/lib/poetry.sh index 9998f1d04..ce39b602e 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -56,7 +56,7 @@ function poetry::install_poetry() { See the log output above for more information. EOF - build_data::set_string "failure_reason" "create-venv::poetry" + build_data::set_string "failure_reason" "internal-error::create-venv::poetry" exit 1 fi diff --git a/lib/utils.sh b/lib/utils.sh index c81af858f..804e9b935 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -51,7 +51,7 @@ function utils::bundled_pip_module_path() { $(find "${bundled_wheels_dir}/" 2>&1 || find "${python_home}/" -type d 2>&1 || true) EOF - build_data::set_string "failure_reason" "bundled-pip-not-found" + build_data::set_string "failure_reason" "internal-error::bundled-pip-not-found" exit 1 fi } @@ -62,7 +62,7 @@ function utils::abort_internal_error() { output::error <<-EOF Internal error: ${message}. EOF - build_data::set_string "failure_reason" "internal-error" + build_data::set_string "failure_reason" "internal-error::abort" build_data::set_string "failure_detail" "${message}" exit 1 } From 258dfa07055c5643a2d81e402248497d1594b20a Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 14 Oct 2025 22:46:04 +0100 Subject: [PATCH 251/406] Improve error messages and metrics for unhandled errors (#1933) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a Bash ERR trap to catch any unhandled buildpack errors, which displays a more helpful error message (that includes the command being run and the stack trace), as well as setting an appropriate failure reason and detail in the build event payload. For example: ``` -----> Using Python 3.13 specified in .python-version -----> Installing Python 3.13.8 mkdir: cannot create directory ‘/tmp/build_1/.heroku’: Not a directory ! Internal Error: An unhandled buildpack error occurred. ! ! An unhandled error occurred while executing the command: ! mkdir -p "${install_dir}" ! ! If this issue persists, please open a support ticket or file ! an issue on the buildpack's GitHub repository: ! https://help.heroku.com/ ! https://github.com/heroku/heroku-buildpack-python/issues ! ! Stack trace: ! python::install @ /tmp/buildpack/lib/python.sh:25 ! main @ ./bin/compile:170 ~ Report: { ... "failure_detail": "mkdir -p \"${install_dir}\"", "failure_reason": "internal-error::unhandled", ... } ``` See: - https://www.gnu.org/software/bash/manual/html_node/Bourne-Shell-Builtins.html#index-trap - https://www.gnu.org/software/bash/manual/html_node/Bash-Builtins.html#index-caller - https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html - https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html GUS-W-19792272. GUS-W-19792285. --- CHANGELOG.md | 1 + bin/compile | 6 +++++- lib/build_data.sh | 13 +++++++++++++ lib/utils.sh | 36 ++++++++++++++++++++++++++++++++++++ 4 files changed, 55 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4026ec280..845a06a04 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Improved error messages and metrics for unhandled buildpack errors. ([#1933](https://github.com/heroku/heroku-buildpack-python/pull/1933)) ## [v313] - 2025-10-09 diff --git a/bin/compile b/bin/compile index a81032b02..1bd3cd36b 100755 --- a/bin/compile +++ b/bin/compile @@ -2,7 +2,9 @@ # Usage: bin/compile # See: https://devcenter.heroku.com/articles/buildpack-api -set -euo pipefail +# We use `errtrace` and `inherit_errexit` to ensure the ERR trap and exit on error behaviour +# propagates to buildpack functions run in subshells, such when using command substitutions. +set -eu -o pipefail -o errtrace shopt -s inherit_errexit # Note: This can't be enabled via app config vars, since at this point they haven't been loaded from ENV_DIR. @@ -39,6 +41,8 @@ compile_start_time=$(build_data::current_unix_realtime) # a build report that will be consumed by the build system for observability. build_data::setup +trap 'utils::err_trap' ERR + checks::ensure_supported_stack "${STACK:?"Required env var STACK isn't set"}" checks::duplicate_python_buildpack "${BUILD_DIR}" diff --git a/lib/build_data.sh b/lib/build_data.sh index 6d36fe839..9841e0f45 100644 --- a/lib/build_data.sh +++ b/lib/build_data.sh @@ -109,6 +109,19 @@ function build_data::_set() { echo "${new_data_file_contents}" >"${BUILD_DATA_FILE}" } +# Check whether an entry exists in the build data store for the current build. +# Returns zero if the key was found and non-zero otherwise. +# +# Usage: +# ``` +# build_data::has "failure_reason" +# ``` +function build_data::has() { + local key="${1}" + + jq --exit-status "has(\"${key}\")" "${BUILD_DATA_FILE}" >/dev/null +} + # Retrieve the value of an entry in the build data store from the previous successful build. # Returns the empty string if the key wasn't found in the store. # diff --git a/lib/utils.sh b/lib/utils.sh index 804e9b935..2d2af50af 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -66,3 +66,39 @@ function utils::abort_internal_error() { build_data::set_string "failure_detail" "${message}" exit 1 } + +function utils::err_trap() { + # We use `errtrace` which means the ERR trap can fire multiple times, such as when an error + # occurs in a buildpack function called from within a command substitution. + # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. + if build_data::has "failure_reason"; then + return + fi + + local failing_command="${BASH_COMMAND}" + local stack_trace + stack_trace=$( + local frame=0 + while read -r line_number function_name source_file < <(caller "${frame}" || true); do + echo "${function_name} @ ${source_file}:${line_number}" + ((++frame)) + done + ) + + output::error <<-EOF + Internal Error: An unhandled buildpack error occurred. + + An unhandled error occurred while executing the command: + ${failing_command} + + If this issue persists, please open a support ticket or file + an issue on the buildpack's GitHub repository: + https://help.heroku.com/ + https://github.com/heroku/heroku-buildpack-python/issues + + Stack trace: + ${stack_trace} + EOF + build_data::set_string "failure_reason" "internal-error::unhandled" + build_data::set_string "failure_detail" "${failing_command}" +} From 5ac2b109362f6932107e931ea685b07acc08f492 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 15 Oct 2025 11:18:38 +0100 Subject: [PATCH 252/406] Add support for Python 3.13.9 (#1934) Release announcement: https://blog.python.org/2025/10/python-3139-is-now-available.html Changelog: https://docs.python.org/release/3.13.9/whatsnew/changelog.html#python-3-13-9-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/18524770867 GUS-W-19897888. --- CHANGELOG.md | 1 + lib/python_version.sh | 2 +- spec/spec_helper.rb | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 845a06a04..58214ccb7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- The Python 3.13 version alias now resolves to Python 3.13.9. ([#1934](https://github.com/heroku/heroku-buildpack-python/pull/1934)) - Improved error messages and metrics for unhandled buildpack errors. ([#1933](https://github.com/heroku/heroku-buildpack-python/pull/1933)) ## [v313] - 2025-10-09 diff --git a/lib/python_version.sh b/lib/python_version.sh index 69e0841e8..136db89fc 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -8,7 +8,7 @@ LATEST_PYTHON_3_9="3.9.24" LATEST_PYTHON_3_10="3.10.19" LATEST_PYTHON_3_11="3.11.14" LATEST_PYTHON_3_12="3.12.12" -LATEST_PYTHON_3_13="3.13.8" +LATEST_PYTHON_3_13="3.13.9" LATEST_PYTHON_3_14="3.14.0" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=9 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 3233ca69e..520777262 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -14,7 +14,7 @@ LATEST_PYTHON_3_10 = '3.10.19' LATEST_PYTHON_3_11 = '3.11.14' LATEST_PYTHON_3_12 = '3.12.12' -LATEST_PYTHON_3_13 = '3.13.8' +LATEST_PYTHON_3_13 = '3.13.9' LATEST_PYTHON_3_14 = '3.14.0' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_13 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From 874e4681609ae97b8e96432a671cc3f9ddbc6e82 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 15 Oct 2025 10:48:51 +0000 Subject: [PATCH 253/406] Prepare release v314 (#1935) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 58214ccb7..572b779c4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v314] - 2025-10-15 + - The Python 3.13 version alias now resolves to Python 3.13.9. ([#1934](https://github.com/heroku/heroku-buildpack-python/pull/1934)) - Improved error messages and metrics for unhandled buildpack errors. ([#1933](https://github.com/heroku/heroku-buildpack-python/pull/1933)) @@ -1365,7 +1368,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v313...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v314...main +[v314]: https://github.com/heroku/heroku-buildpack-python/compare/v313...v314 [v313]: https://github.com/heroku/heroku-buildpack-python/compare/v312...v313 [v312]: https://github.com/heroku/heroku-buildpack-python/compare/v311...v312 [v311]: https://github.com/heroku/heroku-buildpack-python/compare/v310...v311 From 2fe232c4b1d8a123df4d02ec22067dfeb1e04381 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 15 Oct 2025 14:22:24 +0100 Subject: [PATCH 254/406] Update Dependabot config for Docker (#1936) Since it appears the docker updater doesn't traverse the repo and needs the specific directory containing the `Dockerfile`. Fixes: https://github.com/heroku/heroku-buildpack-python/network/updates/1118250366 GUS-W-19898523. --- .github/dependabot.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index fa1d4813e..04310cda2 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -14,7 +14,7 @@ updates: - "minor" - "patch" - package-ecosystem: "docker" - directory: "/" + directory: "/builds" schedule: interval: "monthly" labels: From 99b8ba60b68737bd51862f7b4f72592c3066e542 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 15 Oct 2025 14:25:07 +0100 Subject: [PATCH 255/406] Remove redundant internal error handling for venv creation (#1937) Since: - The underlying cause of the errors (a bug in older `ensurepip`) was fixed in #1761 and friends. - As of #1933 the generic error trap now handles all internal errors, meaning we don't custom handling for them any more. GUS-W-19898848. --- CHANGELOG.md | 1 + lib/pipenv.sh | 14 +------------- lib/poetry.sh | 14 +------------- 3 files changed, 3 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 572b779c4..4de5e64a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Removed redundant internal error handling for venv creation. ([#1937](https://github.com/heroku/heroku-buildpack-python/pull/1937)) ## [v314] - 2025-10-15 diff --git a/lib/pipenv.sh b/lib/pipenv.sh index 0cb42ab58..ff7e164bd 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -40,19 +40,7 @@ function pipenv::install_pipenv() { # We use the pip wheel bundled within Python's standard library to install Pipenv, # since Pipenv vendors its own pip, so doesn't need an install in the venv. - # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. - if ! python -m venv --without-pip "${pipenv_venv_dir}" |& output::indent; then - output::error <<-EOF - Internal Error: Unable to create virtual environment for Pipenv. - - The 'python -m venv' command to create a virtual environment did - not exit successfully. - - See the log output above for more information. - EOF - build_data::set_string "failure_reason" "internal-error::create-venv::pipenv" - exit 1 - fi + python -m venv --without-pip "${pipenv_venv_dir}" local bundled_pip_module_path bundled_pip_module_path="$(utils::bundled_pip_module_path "${python_home}" "${python_major_version}")" diff --git a/lib/poetry.sh b/lib/poetry.sh index ce39b602e..b88a37f82 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -46,19 +46,7 @@ function poetry::install_poetry() { # it bundles its own copy for use as a fallback. As such we don't need to install pip # into the Poetry venv (and in fact, Poetry wouldn't use this install anyway, since # it only finds an external pip if it exists in the target venv). - # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. - if ! python -m venv --without-pip "${poetry_venv_dir}" |& output::indent; then - output::error <<-EOF - Internal Error: Unable to create virtual environment for Poetry. - - The 'python -m venv' command to create a virtual environment did - not exit successfully. - - See the log output above for more information. - EOF - build_data::set_string "failure_reason" "internal-error::create-venv::poetry" - exit 1 - fi + python -m venv --without-pip "${poetry_venv_dir}" local bundled_pip_module_path bundled_pip_module_path="$(utils::bundled_pip_module_path "${python_home}" "${python_major_version}")" From 0ef4ac273267817836106dc5b4a56c38bc91ed47 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 15 Oct 2025 15:34:25 +0100 Subject: [PATCH 256/406] Bump sigstore/cosign/cosign from v2.6.1 to v3.0.2 in /builds (#1938) Bumps sigstore/cosign/cosign from v2.6.1 to v3.0.2. --- updated-dependencies: - dependency-name: sigstore/cosign/cosign dependency-version: v3.0.2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- builds/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/builds/Dockerfile b/builds/Dockerfile index e9f0703fc..531dfbd26 100644 --- a/builds/Dockerfile +++ b/builds/Dockerfile @@ -1,5 +1,5 @@ ARG STACK_VERSION="24" -FROM ghcr.io/sigstore/cosign/cosign:v2.6.1@sha256:68839b7f13dac5a6744a5d8818e984dd39183374e37855c19e14d623d9bc9037 AS cosign +FROM ghcr.io/sigstore/cosign/cosign:v3.0.2@sha256:b29487e48205d875c324c79583e2806d9d269c0fa299e0861bbec023d8430c8b AS cosign FROM heroku/heroku:${STACK_VERSION}-build ARG STACK_VERSION From 1f0c3c91bf2d8769ed816e3147c758c719c4f358 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 16 Oct 2025 14:41:32 +0100 Subject: [PATCH 257/406] Stop using a global warnings log file (#1939) In favour of per-install step temporary log files. This is a first step towards refactoring warnings handling. GUS-W-19969173. --- bin/compile | 6 ------ bin/warnings | 8 +++++--- lib/pip.sh | 7 +++++-- lib/pipenv.sh | 7 +++++-- lib/poetry.sh | 7 +++++-- lib/uv.sh | 7 +++++-- 6 files changed, 25 insertions(+), 17 deletions(-) diff --git a/bin/compile b/bin/compile index 1bd3cd36b..244ec72a2 100755 --- a/bin/compile +++ b/bin/compile @@ -53,12 +53,6 @@ export PATH=":/usr/local/bin:${PATH}" # Exported for use in subshells, such as the steps run via sub_env. export BUILD_DIR CACHE_DIR ENV_DIR -# Common Problem Warnings: -# This section creates a temporary file in which to stick the output of `pip install`. -# The `warnings` subscript then greps through this for common problems and guides -# the user towards resolution of known issues. -WARNINGS_LOG=$(mktemp) - # Sanitize externally-provided environment variables: # The following environment variables are either problematic or simply unnecessary # for the buildpack to have knowledge of, so we unset them, to keep the environment diff --git a/bin/warnings b/bin/warnings index 6b65fe7cb..07bd54daa 100755 --- a/bin/warnings +++ b/bin/warnings @@ -1,8 +1,10 @@ #!/usr/bin/env bash show-warnings() { + local install_log="${1}" + # shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. - if grep -qi 'Could not find gdal-config' "${WARNINGS_LOG}"; then + if grep -qi 'Could not find gdal-config' "${install_log}"; then output::error <<-'EOF' Error: Package installation failed since the GDAL library wasn't found. @@ -10,7 +12,7 @@ show-warnings() { https://github.com/heroku/heroku-geo-buildpack EOF build_data::set_string "failure_detail" "Could not find gdal-config" - elif grep -qi 'sqlite3.h: No such file or directory' "${WARNINGS_LOG}"; then + elif grep -qi 'sqlite3.h: No such file or directory' "${install_log}"; then output::error <<-'EOF' Error: Package installation failed since SQLite headers weren't found. @@ -39,7 +41,7 @@ show-warnings() { https://github.com/heroku/heroku-buildpack-apt EOF build_data::set_string "failure_detail" "sqlite3.h: No such file or directory" - elif grep -qi 'Please use pip<24.1 if you need to use this version' "${WARNINGS_LOG}"; then + elif grep -qi 'Please use pip<24.1 if you need to use this version' "${install_log}"; then output::error <<-'EOF' Error: One of your dependencies contains broken metadata. diff --git a/lib/pip.sh b/lib/pip.sh index 147c5f1ba..335024be6 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -109,6 +109,9 @@ function pip::install_dependencies() { # We only display the most relevant command args here, to improve the signal to noise ratio. output::step "Installing dependencies using '${pip_install_command[*]}'" + local install_log + install_log=$(mktemp) + # TODO: Remove --disable-pip-version-check in favour of exporting PIP_DISABLE_PIP_VERSION_CHECK. # The sed usage is to reduce the verbosity of output lines like: # ...when using Python 3.10 and older: @@ -124,13 +127,13 @@ function pip::install_dependencies() { --no-input \ --progress-bar off \ --src='/app/.heroku/python/src' \ - |& tee "${WARNINGS_LOG:?}" \ + |& tee "${install_log}" \ |& sed --unbuffered --regexp-extended \ --expression 's# in (/app|\.)/\.heroku/python/lib/python[0-9.]+/site-packages##' \ |& output::indent }; then # TODO: Overhaul warnings and combine them with error handling. - show-warnings + show-warnings "${install_log}" output::error <<-EOF Error: Unable to install dependencies using pip. diff --git a/lib/pipenv.sh b/lib/pipenv.sh index ff7e164bd..0b99ac077 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -142,6 +142,9 @@ function pipenv::install_dependencies() { # We only display the most relevant command args here, to improve the signal to noise ratio. output::step "Installing dependencies using '${pipenv_install_command[*]}'" + local install_log + install_log=$(mktemp) + # TODO: Expose app config vars to the install command as part of doing so for all package managers. # `PIPENV_NOSPIN`: Disable progress spinners. # `PIP_SRC`: Override the editable VCS repo location from its default of inside the build directory @@ -150,11 +153,11 @@ function pipenv::install_dependencies() { if ! { PIPENV_NOSPIN="1" PIP_SRC="/app/.heroku/python/src" \ "${pipenv_install_command[@]}" \ - |& tee "${WARNINGS_LOG:?}" \ + |& tee "${install_log}" \ |& output::indent }; then # TODO: Overhaul warnings and combine them with error handling. - show-warnings + show-warnings "${install_log}" output::error <<-EOF Error: Unable to install dependencies using Pipenv. diff --git a/lib/poetry.sh b/lib/poetry.sh index b88a37f82..38df296a0 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -130,6 +130,9 @@ function poetry::install_dependencies() { # We only display the most relevant command args here, to improve the signal to noise ratio. output::step "Installing dependencies using '${poetry_install_command[*]}'" + local install_log + install_log=$(mktemp) + # `--compile`: Compiles Python bytecode, to improve app boot times (pip does this by default). # `--no-ansi`: Whilst we'd prefer to enable colour if possible, Poetry also emits ANSI escape # codes for redrawing lines, which renders badly in persisted build logs. @@ -139,12 +142,12 @@ function poetry::install_dependencies() { --compile \ --no-ansi \ --no-interaction \ - |& tee "${WARNINGS_LOG:?}" \ + |& tee "${install_log}" \ |& sed --unbuffered --expression '/Skipping virtualenv creation/d' \ |& output::indent }; then # TODO: Overhaul warnings and combine them with error handling. - show-warnings + show-warnings "${install_log}" output::error <<-EOF Error: Unable to install dependencies using Poetry. diff --git a/lib/uv.sh b/lib/uv.sh index 48953c0a9..927c84821 100644 --- a/lib/uv.sh +++ b/lib/uv.sh @@ -149,6 +149,9 @@ function uv::install_dependencies() { # We only display the most relevant command args here, to improve the signal to noise ratio. output::step "Installing dependencies using '${uv_install_command[*]}'" + local install_log + install_log=$(mktemp) + # TODO: Expose app config vars to the install command as part of doing so for all package managers. # `--compile-bytecode`: Improves app boot times (pip does this by default). # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. @@ -157,11 +160,11 @@ function uv::install_dependencies() { --color always \ --compile-bytecode \ --no-progress \ - |& tee "${WARNINGS_LOG:?}" \ + |& tee "${install_log}" \ |& output::indent }; then # TODO: Overhaul warnings and combine them with error handling (for all package managers). - show-warnings + show-warnings "${install_log}" output::error <<-EOF Error: Unable to install dependencies using uv. From 38f7d1f01ee34dd9d4a261de58bb9d00a54d6d51 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 16 Oct 2025 14:43:26 +0100 Subject: [PATCH 258/406] Clean up the install pip step (#1940) Some small refactorings split out of another PR: - Move the bundled pip lookup step later, for consistency with the ordering used for the other package managers. - Rename the function to not mention setuptools/wheel, now that it no longer always installs those packages. --- bin/compile | 2 +- lib/pip.sh | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/bin/compile b/bin/compile index 244ec72a2..adbb6de2b 100755 --- a/bin/compile +++ b/bin/compile @@ -166,7 +166,7 @@ python::install "${BUILD_DIR}" "${STACK}" "${python_full_version}" "${python_maj package_manager_install_start_time=$(build_data::current_unix_realtime) case "${package_manager}" in pip) - pip::install_pip_setuptools_wheel "${python_home}" "${python_major_version}" + pip::install_pip "${python_home}" "${python_major_version}" ;; pipenv) pipenv::install_pipenv "${python_home}" "${python_major_version}" "${EXPORT_PATH}" "${PROFILE_PATH}" diff --git a/lib/pip.sh b/lib/pip.sh index 335024be6..6db9eb3b6 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -8,15 +8,10 @@ PIP_VERSION=$(utils::get_requirement_version 'pip') SETUPTOOLS_VERSION=$(utils::get_requirement_version 'setuptools') WHEEL_VERSION=$(utils::get_requirement_version 'wheel') -function pip::install_pip_setuptools_wheel() { +function pip::install_pip() { local python_home="${1}" local python_major_version="${2}" - # We use the pip wheel bundled within Python's standard library to install our chosen - # pip version, since it's faster than `ensurepip` followed by an upgrade in place. - local bundled_pip_module_path - bundled_pip_module_path="$(utils::bundled_pip_module_path "${python_home}" "${python_major_version}")" - build_data::set_string "pip_version" "${PIP_VERSION}" local packages_to_install=( @@ -46,6 +41,11 @@ function pip::install_pip_setuptools_wheel() { # app's requirements.txt in the last build). The install will be a no-op if the versions match. output::step "Installing ${packages_display_text}" + # We use the pip wheel bundled within Python's standard library to install our chosen + # pip version, since it's faster than `ensurepip` followed by an upgrade in place. + local bundled_pip_module_path + bundled_pip_module_path="$(utils::bundled_pip_module_path "${python_home}" "${python_major_version}")" + # `--isolated`: Prevents any custom pip configuration added by third party buildpacks (via env # vars or global config files) from breaking package manager bootstrapping. # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. From 58f10eb0ad1ae72cf0b55a8eb67589d22d1bc331 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 21 Oct 2025 12:01:43 +0100 Subject: [PATCH 259/406] Pin Poetry's `dulwich` version (#1943) After the recent upstream `dulwich` 0.24.6 release, CI in this repo started failing in the test that runs Poetry with our oldest supported Python patch version (3.9.0). eg: ``` remote: -----> Installing Poetry 2.2.1 remote: -----> Installing dependencies using 'poetry sync --only main' remote: remote: unhashable type: 'list' remote: remote: ! Error: Unable to install dependencies using Poetry. remote: ! remote: ! See the log output above for more information. remote: remote: ! Push rejected, failed to compile Python app. ``` Investigation showed this only affects Python 3.9.0 and 3.9.1, which are multiple years behind the latest 3.9.x patch version (3.9.24). For more details see the upstream issue I opened: https://github.com/jelmer/dulwich/issues/1948 Our options are either to drop support for 3.9.0 and 3.9.1 (and require use of Python 3.9.2+), or to pin the dulwich version. Given we'll be dropping support for Python 3.9 in January 2026, for now we'll temporarily pin the dulwich version. (It's also possible the dulwich project might adjust it's `requires-python` and yank the 0.24.6 release, in which case we'll be able to remove the pin before then.) GUS-W-20000280. --- CHANGELOG.md | 1 + lib/poetry.sh | 3 +++ 2 files changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4de5e64a6..39294fdd8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Pinned `dulwich` version when using Poetry to work around an incompatibility with Python <3.9.2. ([#1943](https://github.com/heroku/heroku-buildpack-python/pull/1943)) - Removed redundant internal error handling for venv creation. ([#1937](https://github.com/heroku/heroku-buildpack-python/pull/1937)) ## [v314] - 2025-10-15 diff --git a/lib/poetry.sh b/lib/poetry.sh index 38df296a0..5573c9be8 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -56,6 +56,8 @@ function poetry::install_poetry() { # pip versions bundled with Python 3.9/3.10. # `--isolated`: Prevents any custom pip configuration added by third party buildpacks (via env # vars or global config files) from breaking package manager bootstrapping. + # We pin to an older dulwich version to work around https://github.com/jelmer/dulwich/issues/1948 + # on Python 3.9.0/3.9.1. TODO: Remove this pin when we drop support for Python 3.9 in Jan 2026. # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { "${poetry_venv_dir}/bin/python" "${bundled_pip_module_path}" \ @@ -66,6 +68,7 @@ function poetry::install_poetry() { --no-input \ --quiet \ "poetry==${POETRY_VERSION}" \ + dulwich==0.24.5 \ |& output::indent }; then output::error <<-EOF From 088bce0d919e018ae6bf44767d13aecb0f70336a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 21 Oct 2025 11:06:53 +0000 Subject: [PATCH 260/406] Bump the ruby-dependencies group across 1 directory with 2 updates (#1944) Bumps the ruby-dependencies group with 2 updates in the / directory: [rspec-core](https://github.com/rspec/rspec) and [rubocop](https://github.com/rubocop/rubocop). Updates `rspec-core` from 3.13.5 to 3.13.6 - [Changelog](https://github.com/rspec/rspec/blob/rspec-core-v3.13.6/rspec-core/Changelog.md) - [Commits](https://github.com/rspec/rspec/compare/rspec-core-v3.13.5...rspec-core-v3.13.6) Updates `rubocop` from 1.81.1 to 1.81.6 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.81.1...v1.81.6) --- updated-dependencies: - dependency-name: rspec-core dependency-version: 3.13.6 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies - dependency-name: rubocop dependency-version: 1.81.6 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 7058320cf..a77ab3abb 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -20,7 +20,7 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.15.0) + json (2.15.1) language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) @@ -37,21 +37,21 @@ GEM heroics (~> 0.1.1) moneta (~> 1.0.0) rate_throttle_client (~> 0.1.0) - prism (1.5.1) + prism (1.6.0) racc (1.8.1) rainbow (3.1.1) rate_throttle_client (0.1.2) regexp_parser (2.11.3) rrrretry (1.0.0) - rspec-core (3.13.5) + rspec-core (3.13.6) rspec-support (~> 3.13.0) rspec-expectations (3.13.5) diff-lcs (>= 1.2.0, < 2.0) rspec-support (~> 3.13.0) rspec-retry (0.6.2) rspec-core (> 3.3) - rspec-support (3.13.4) - rubocop (1.81.1) + rspec-support (3.13.6) + rubocop (1.81.6) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) From 9311dd8a37d4abc3ad686bdff2f8f37948282eb8 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 21 Oct 2025 11:54:47 +0000 Subject: [PATCH 261/406] Bump uv from 0.8.23 to 0.9.4 (#1942) * Bump uv from 0.8.23 to 0.9.4 Bumps [uv](https://github.com/astral-sh/uv) from 0.8.23 to 0.9.4. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.8.23...0.9.4) --- updated-dependencies: - dependency-name: uv dependency-version: 0.9.4 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 39294fdd8..ac1bec5e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.8.23 to 0.9.4. ([#1942](https://github.com/heroku/heroku-buildpack-python/pull/1942)) - Pinned `dulwich` version when using Poetry to work around an incompatibility with Python <3.9.2. ([#1943](https://github.com/heroku/heroku-buildpack-python/pull/1943)) - Removed redundant internal error handling for venv creation. ([#1937](https://github.com/heroku/heroku-buildpack-python/pull/1937)) diff --git a/requirements/uv.txt b/requirements/uv.txt index 61e7fc365..7c1615c00 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.8.23 +uv==0.9.4 From b77cfee2f4a8b367042e0d01164cdc4cf940ff93 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 22 Oct 2025 00:17:55 +0100 Subject: [PATCH 262/406] Bump uv from 0.9.4 to 0.9.5 in /requirements (#1945) * Bump uv from 0.9.4 to 0.9.5 in /requirements Bumps [uv](https://github.com/astral-sh/uv) from 0.9.4 to 0.9.5. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.9.4...0.9.5) --- updated-dependencies: - dependency-name: uv dependency-version: 0.9.5 dependency-type: direct:production ... Signed-off-by: dependabot[bot] * Update changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 2 +- requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ac1bec5e0..3191503e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## [Unreleased] -- Updated uv from 0.8.23 to 0.9.4. ([#1942](https://github.com/heroku/heroku-buildpack-python/pull/1942)) +- Updated uv from 0.8.23 to 0.9.5. ([#1942](https://github.com/heroku/heroku-buildpack-python/pull/1942) and [#1945](https://github.com/heroku/heroku-buildpack-python/pull/1945)) - Pinned `dulwich` version when using Poetry to work around an incompatibility with Python <3.9.2. ([#1943](https://github.com/heroku/heroku-buildpack-python/pull/1943)) - Removed redundant internal error handling for venv creation. ([#1937](https://github.com/heroku/heroku-buildpack-python/pull/1937)) diff --git a/requirements/uv.txt b/requirements/uv.txt index 7c1615c00..c74371459 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.9.4 +uv==0.9.5 From a8b7f3deab20f66d9ad2ab68c562844e3df752fc Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 22 Oct 2025 10:10:36 +0100 Subject: [PATCH 263/406] Prepare release v315 (#1946) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3191503e3..63123169b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v315] - 2025-10-22 + - Updated uv from 0.8.23 to 0.9.5. ([#1942](https://github.com/heroku/heroku-buildpack-python/pull/1942) and [#1945](https://github.com/heroku/heroku-buildpack-python/pull/1945)) - Pinned `dulwich` version when using Poetry to work around an incompatibility with Python <3.9.2. ([#1943](https://github.com/heroku/heroku-buildpack-python/pull/1943)) - Removed redundant internal error handling for venv creation. ([#1937](https://github.com/heroku/heroku-buildpack-python/pull/1937)) @@ -1371,7 +1374,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v314...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v315...main +[v315]: https://github.com/heroku/heroku-buildpack-python/compare/v314...v315 [v314]: https://github.com/heroku/heroku-buildpack-python/compare/v313...v314 [v313]: https://github.com/heroku/heroku-buildpack-python/compare/v312...v313 [v312]: https://github.com/heroku/heroku-buildpack-python/compare/v311...v312 From 3b25a5b34d65c3ed395802cd772995fbb536dda0 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 24 Oct 2025 14:22:43 +0100 Subject: [PATCH 264/406] Improve Python version error message for invisible Unicode whitespace (#1947) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Looking through errors in Honeycomb I noticed some of the instances of builds failing due to an invalid Python version (in either their `.python-version` or `runtime.txt` file) were due to leading/trailing invisible Unicode characters such as zero width space / zero width non-breaking space. Previously the error message for those was hard to understand, since the invisible nature of those characters mean they don't show up in the message, eg: ``` ! Error: Invalid Python version in runtime.txt. ! ! The Python version specified in your runtime.txt file isn't ! in the correct format. ! ! The following file contents were found, which aren't valid: ! python-3.11.9 ... ``` The reason for this is that `sed` uses the current locale, and so the `[:space:]` character class used by the sed call in `utils::read_file_with_special_chars_substituted()` was previously matching Unicode whitespace. (In contrast, the Bash extended regex used when parsing the read contents does not.) After this change, the unicode replacement character will now be shown (like it already is for other Unicode non-whitespace characters and ASCII control characters), eg: ``` ! The following file contents were found, which aren't valid: ! ���python-3.11.9 ``` See: https://www.gnu.org/software/sed/manual/html_node/Locale-Considerations.html GUS-W-20034162. --- CHANGELOG.md | 1 + lib/utils.sh | 4 +++- .../.python-version | Bin 379 -> 394 bytes .../runtime_txt_invalid_version/runtime.txt | 2 +- spec/hatchet/python_version_spec.rb | 4 ++-- 5 files changed, 7 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 63123169b..d36e0adfd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Improved the error message when a `.python-version` or `runtime.txt` file contains invisible Unicode whitespace characters. ([#1947](https://github.com/heroku/heroku-buildpack-python/pull/1947)) ## [v315] - 2025-10-22 diff --git a/lib/utils.sh b/lib/utils.sh index 2d2af50af..a640bd2ca 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -20,9 +20,11 @@ function utils::get_requirement_version() { # replacement character, so they are visible in error messages. Also removes any carriage # return characters, to prevent them interfering with the rendering of error messages that # include the raw file contents. +# `LC_ALL=C` ensures that `[:space:]` only matches ASCII whitespace, since we want Unicode +# whitespace (such as the invisible Unicode zero width space character) to be replaced too. function utils::read_file_with_special_chars_substituted() { local file="${1}" - sed --regexp-extended --expression 's/[^[:print:][:space:]]/�/g' --expression 's/\r$//' "${file}" + LC_ALL=C sed --regexp-extended --expression 's/[^[:print:][:space:]]/�/g' --expression 's/\r$//' "${file}" } # Python bundles pip within its standard library, which we can use to install our chosen diff --git a/spec/fixtures/python_version_file_invalid_version/.python-version b/spec/fixtures/python_version_file_invalid_version/.python-version index 3a618f7a5803e3afdb1121474ed4a3150573950a..36fc6073580ae57001fd33ac77b068b0be906358 100644 GIT binary patch delta 90 zcmey()WtmEgJxA~QNBWXW=cthLUBQ2a;iddMq*K7a!G0tFPDOXvVyLHU#O2lVqVI` s_hO92lT{e46dpBn8|yI`8tK&=NGm9Sl<{)$aw#hq>6z;pTJUlK0AisTXaE2J delta 93 zcmeBT{>?PugNci)k3w>OUP)1YjzV%qVo_ppNoo-B$C+R Date: Mon, 27 Oct 2025 11:55:46 +0000 Subject: [PATCH 265/406] Record the file encoding of `.python-version` and `runtime.txt` (#1948) To help work out what file encodings are present in the wild, in order to improve the UX for file parsing related errors that are believe to be due to encoding issues. The `file --brief` command will return values like: - `ASCII text` - `Unicode text, UTF-8 (with BOM) text` - `Unicode text, UTF-16, little-endian text, with CRLF line terminators` These metrics will be removed later. See: https://manpages.ubuntu.com/manpages/noble/en/man1/file.1.html GUS-W-20049083. --- CHANGELOG.md | 1 + lib/python_version.sh | 3 +++ spec/hatchet/pip_spec.rb | 1 + spec/hatchet/poetry_spec.rb | 1 + spec/hatchet/uv_spec.rb | 1 + 5 files changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d36e0adfd..ca0c9b9e6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Improved the error message when a `.python-version` or `runtime.txt` file contains invisible Unicode whitespace characters. ([#1947](https://github.com/heroku/heroku-buildpack-python/pull/1947)) +- Added metrics for the file encoding of `.python-version` and `runtime.txt` files. ([#1948](https://github.com/heroku/heroku-buildpack-python/pull/1948)) ## [v315] - 2025-10-22 diff --git a/lib/python_version.sh b/lib/python_version.sh index 136db89fc..7169ce0d4 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -64,6 +64,8 @@ function python_version::read_requested_python_version() { local runtime_txt_path="${build_dir}/runtime.txt" if [[ -f "${runtime_txt_path}" ]]; then + # For example: "ASCII text" or "Unicode text, UTF-16, little-endian text, with CRLF line terminators" + build_data::set_string "runtime_txt_encoding" "$(file --brief "${runtime_txt_path}" || true)" contents="$(utils::read_file_with_special_chars_substituted "${runtime_txt_path}")" version="$(python_version::parse_runtime_txt "${contents}")" origin="runtime.txt" @@ -72,6 +74,7 @@ function python_version::read_requested_python_version() { local python_version_file_path="${build_dir}/.python-version" if [[ -f "${python_version_file_path}" ]]; then + build_data::set_string "python_version_file_encoding" "$(file --brief "${python_version_file_path}" || true)" contents="$(utils::read_file_with_special_chars_substituted "${python_version_file_path}")" version="$(python_version::parse_python_version_file "${contents}")" origin=".python-version" diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index d30e14e98..1885ef953 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -72,6 +72,7 @@ remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", + remote: "python_version_file_encoding": "ASCII text", remote: "python_version_files": ".python-version,", remote: "python_version_major": "3.13", remote: "python_version_origin": ".python-version", diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 8bfdfdb40..bc2bb1d5c 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -74,6 +74,7 @@ remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", + remote: "python_version_file_encoding": "ASCII text", remote: "python_version_files": ".python-version,", remote: "python_version_major": "3.13", remote: "python_version_origin": ".python-version", diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 16fa03817..813409a59 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -79,6 +79,7 @@ remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", + remote: "python_version_file_encoding": "ASCII text", remote: "python_version_files": ".python-version,", remote: "python_version_major": "3.13", remote: "python_version_origin": ".python-version", From 850c60d9e08ae2c1722cf351c587a5d51067c99e Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Mon, 27 Oct 2025 12:16:03 +0000 Subject: [PATCH 266/406] Prepare release v316 (#1949) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ca0c9b9e6..a590682d3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v316] - 2025-10-27 + - Improved the error message when a `.python-version` or `runtime.txt` file contains invisible Unicode whitespace characters. ([#1947](https://github.com/heroku/heroku-buildpack-python/pull/1947)) - Added metrics for the file encoding of `.python-version` and `runtime.txt` files. ([#1948](https://github.com/heroku/heroku-buildpack-python/pull/1948)) @@ -1376,7 +1379,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v315...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v316...main +[v316]: https://github.com/heroku/heroku-buildpack-python/compare/v315...v316 [v315]: https://github.com/heroku/heroku-buildpack-python/compare/v314...v315 [v314]: https://github.com/heroku/heroku-buildpack-python/compare/v313...v314 [v313]: https://github.com/heroku/heroku-buildpack-python/compare/v312...v313 From dfe88afff641ab4bc2c00a3d3163ffb048ed3bc3 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 31 Oct 2025 10:02:48 +0000 Subject: [PATCH 267/406] Bump uv from 0.9.5 to 0.9.7 (#1952) * Bump uv from 0.9.5 to 0.9.7 Bumps [uv](https://github.com/astral-sh/uv) from 0.9.5 to 0.9.7. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.9.5...0.9.7) --- updated-dependencies: - dependency-name: uv dependency-version: 0.9.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a590682d3..ddc5be69d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.9.5 to 0.9.7. ([#1952](https://github.com/heroku/heroku-buildpack-python/pull/1952)) ## [v316] - 2025-10-27 diff --git a/requirements/uv.txt b/requirements/uv.txt index c74371459..e09f8b1ba 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.9.5 +uv==0.9.7 From 070b7da3bad3789ba62ce2f37a3bac3f96010d12 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 3 Nov 2025 17:11:48 +0000 Subject: [PATCH 268/406] Pin GDAL version in tests (#1954) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit To work around: ``` remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting GDAL (from -r requirements.txt (line 1)) remote: Downloading gdal-3.12.0.tar.gz (229 kB) remote: Installing build dependencies: started remote: Installing build dependencies: finished with status 'done' remote: Getting requirements to build wheel: started remote: Getting requirements to build wheel: finished with status 'error' remote: error: subprocess-exited-with-error remote: remote: × Getting requirements to build wheel did not run successfully. remote: │ exit code: 1 remote: ╰─> [12 lines of output] remote: Using numpy 2.3.4 remote: /tmp/pip-build-env-5ffoncnf/overlay/lib/python3.13/site-packages/setuptools/config/_apply_pyprojecttoml.py:82: SetuptoolsWarning: `license` overwritten by `pyproject.toml` remote: corresp(dist, value, root_dir) remote: /tmp/pip-build-env-5ffoncnf/overlay/lib/python3.13/site-packages/setuptools/config/_apply_pyprojecttoml.py:82: SetuptoolsWarning: `extras_require` overwritten in `pyproject.toml` (optional-dependencies) remote: corresp(dist, value, root_dir) remote: running egg_info remote: writing gdal-utils/GDAL.egg-info/PKG-INFO remote: writing dependency_links to gdal-utils/GDAL.egg-info/dependency_links.txt remote: writing entry points to gdal-utils/GDAL.egg-info/entry_points.txt remote: writing requirements to gdal-utils/GDAL.egg-info/requires.txt remote: writing top-level names to gdal-utils/GDAL.egg-info/top_level.txt remote: error: package directory 'osgeo' does not exist remote: [end of output] ``` (https://github.com/heroku/heroku-buildpack-python/actions/runs/18997765580/job/54260036112#step:5:409) Which started now because GDAL 3.12.0rc0 seems to have accidentally been released as a stable version instead of a release candidate: https://github.com/OSGeo/gdal/issues/13329 GUS-W-20111893. --- spec/fixtures/pip_gdal/requirements.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/spec/fixtures/pip_gdal/requirements.txt b/spec/fixtures/pip_gdal/requirements.txt index 2642fd732..0059e4a71 100644 --- a/spec/fixtures/pip_gdal/requirements.txt +++ b/spec/fixtures/pip_gdal/requirements.txt @@ -1 +1,2 @@ -GDAL +# Pinned due to https://github.com/OSGeo/gdal/issues/13329 +GDAL==3.11.4 From ef05ca9c1b5cf94e7b5cdf75e40b1d813d6ec2b0 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 3 Nov 2025 17:15:19 +0000 Subject: [PATCH 269/406] Bump rubocop from 1.81.6 to 1.81.7 in the ruby-dependencies group (#1953) * Bump rubocop from 1.81.6 to 1.81.7 in the ruby-dependencies group Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.81.6 to 1.81.7 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.81.6...v1.81.7) --- updated-dependencies: - dependency-name: rubocop dependency-version: 1.81.7 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] * Refresh lockfile --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- Gemfile.lock | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index a77ab3abb..389747d96 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -5,7 +5,7 @@ GEM base64 (0.3.0) diff-lcs (1.6.2) erubis (2.7.0) - excon (1.2.8) + excon (1.3.0) logger heroics (0.1.3) base64 @@ -20,7 +20,7 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.15.1) + json (2.15.2) language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) @@ -30,7 +30,7 @@ GEM parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) - parser (3.3.9.0) + parser (3.3.10.0) ast (~> 2.4.1) racc platform-api (3.8.0) @@ -51,7 +51,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.6) - rubocop (1.81.6) + rubocop (1.81.7) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) @@ -89,7 +89,7 @@ DEPENDENCIES rubocop-rspec RUBY VERSION - ruby 3.4.5p51 + ruby 3.4.7p58 BUNDLED WITH - 2.7.1 + 2.7.2 From b2f3bdf213dec77a644c5eaa3aafe41cd8a30647 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 3 Nov 2025 17:18:29 +0000 Subject: [PATCH 270/406] Add support for Python 3.9.25 (#1955) Changelog: https://docs.python.org/release/3.9.25/whatsnew/changelog.html#python-3-9-25-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/19042194319 GUS-W-20112543. --- CHANGELOG.md | 1 + lib/python_version.sh | 2 +- spec/spec_helper.rb | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ddc5be69d..a79e1d4fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- The Python 3.9 version alias now resolves to Python 3.9.25. ([#1955](https://github.com/heroku/heroku-buildpack-python/pull/1955)) - Updated uv from 0.9.5 to 0.9.7. ([#1952](https://github.com/heroku/heroku-buildpack-python/pull/1952)) ## [v316] - 2025-10-27 diff --git a/lib/python_version.sh b/lib/python_version.sh index 7169ce0d4..9e48ad6ca 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -4,7 +4,7 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail -LATEST_PYTHON_3_9="3.9.24" +LATEST_PYTHON_3_9="3.9.25" LATEST_PYTHON_3_10="3.10.19" LATEST_PYTHON_3_11="3.11.14" LATEST_PYTHON_3_12="3.12.12" diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 520777262..dd19c7fb3 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -10,7 +10,7 @@ FIXTURE_DIR = Pathname.new(__FILE__).parent.join('fixtures') -LATEST_PYTHON_3_9 = '3.9.24' +LATEST_PYTHON_3_9 = '3.9.25' LATEST_PYTHON_3_10 = '3.10.19' LATEST_PYTHON_3_11 = '3.11.14' LATEST_PYTHON_3_12 = '3.12.12' From 5cdf8f2ca6328b5817fbfe931f6bb0cae51fb2d4 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Mon, 3 Nov 2025 17:22:52 +0000 Subject: [PATCH 271/406] Prepare release v317 (#1956) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a79e1d4fa..a6dae745d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v317] - 2025-11-03 + - The Python 3.9 version alias now resolves to Python 3.9.25. ([#1955](https://github.com/heroku/heroku-buildpack-python/pull/1955)) - Updated uv from 0.9.5 to 0.9.7. ([#1952](https://github.com/heroku/heroku-buildpack-python/pull/1952)) @@ -1381,7 +1384,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v316...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v317...main +[v317]: https://github.com/heroku/heroku-buildpack-python/compare/v316...v317 [v316]: https://github.com/heroku/heroku-buildpack-python/compare/v315...v316 [v315]: https://github.com/heroku/heroku-buildpack-python/compare/v314...v315 [v314]: https://github.com/heroku/heroku-buildpack-python/compare/v313...v314 From cde4b675bb0fb83f390d88984ddc85ec1f57cbb5 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 12 Nov 2025 17:26:15 +0000 Subject: [PATCH 272/406] Improve `.python-version` and `runtime.txt` error handling (#1958) - Improves the error message shown when a `.python-version` file uses an unsupported file encoding. - Improves the error message shown when a `.python-version` or `runtime.txt` file contain an invalid Python version. - Adds support for commented lines in `runtime.txt`, for parity with `.python-version`. - Adjust the metrics so now the raw (unescaped) invalid strings are recorded, for easier debugging. GUS-W-19246790. GUS-W-19280955. --- .github/workflows/ci.yml | 2 +- CHANGELOG.md | 3 + lib/python_version.sh | 147 +++++++++++++----- lib/utils.sh | 11 -- .../.python-version | Bin 394 -> 518 bytes .../.python-version | 5 +- .../.python-version | 2 + .../requirements.txt | 0 .../runtime.txt | 5 + .../runtime_txt_invalid_version/runtime.txt | 8 +- spec/hatchet/pip_spec.rb | 1 - spec/hatchet/poetry_spec.rb | 1 - spec/hatchet/python_version_spec.rb | 46 ++++-- spec/hatchet/uv_spec.rb | 1 - 14 files changed, 160 insertions(+), 72 deletions(-) create mode 100644 spec/fixtures/python_version_file_unsupported_encoding/.python-version create mode 100644 spec/fixtures/python_version_file_unsupported_encoding/requirements.txt diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 867e0ab61..51e167ef9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -47,7 +47,7 @@ jobs: HEROKU_API_KEY: ${{ secrets.HEROKU_API_KEY }} HEROKU_API_USER: ${{ secrets.HEROKU_API_USER }} HEROKU_DISABLE_AUTOUPDATE: 1 - PARALLEL_SPLIT_TEST_PROCESSES: 85 + PARALLEL_SPLIT_TEST_PROCESSES: 90 RSPEC_RETRY_RETRY_COUNT: 1 steps: - name: Checkout diff --git a/CHANGELOG.md b/CHANGELOG.md index a6dae745d..a6510657e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] +- Improved the error message shown when a `.python-version` file uses an unsupported file encoding. ([#1958](https://github.com/heroku/heroku-buildpack-python/pull/1958)) +- Improved the error message shown when a `.python-version` or `runtime.txt` file contain an invalid Python version. ([#1958](https://github.com/heroku/heroku-buildpack-python/pull/1958)) +- Added support for commented lines in `runtime.txt`, for parity with `.python-version`. ([#1958](https://github.com/heroku/heroku-buildpack-python/pull/1958)) ## [v317] - 2025-11-03 diff --git a/lib/python_version.sh b/lib/python_version.sh index 9e48ad6ca..85bdafd57 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -49,7 +49,6 @@ function python_version::read_requested_python_version() { # without having to hardcode globals. See: https://stackoverflow.com/a/38997681 declare -n version="${4}" declare -n origin="${5}" - local contents # Record the names of files similar to .python-version in the root of the app, to determine # how often that file is misspelled, as a temporary first step before deciding whether to add @@ -64,19 +63,14 @@ function python_version::read_requested_python_version() { local runtime_txt_path="${build_dir}/runtime.txt" if [[ -f "${runtime_txt_path}" ]]; then - # For example: "ASCII text" or "Unicode text, UTF-16, little-endian text, with CRLF line terminators" - build_data::set_string "runtime_txt_encoding" "$(file --brief "${runtime_txt_path}" || true)" - contents="$(utils::read_file_with_special_chars_substituted "${runtime_txt_path}")" - version="$(python_version::parse_runtime_txt "${contents}")" + version="$(python_version::parse_runtime_txt "${runtime_txt_path}")" origin="runtime.txt" return 0 fi local python_version_file_path="${build_dir}/.python-version" if [[ -f "${python_version_file_path}" ]]; then - build_data::set_string "python_version_file_encoding" "$(file --brief "${python_version_file_path}" || true)" - contents="$(utils::read_file_with_special_chars_substituted "${python_version_file_path}")" - version="$(python_version::parse_python_version_file "${contents}")" + version="$(python_version::parse_python_version_file "${python_version_file_path}")" origin=".python-version" return 0 fi @@ -104,25 +98,26 @@ function python_version::read_requested_python_version() { # Parse the contents of a runtime.txt file and return the Python version substring (e.g. `3.12` or `3.12.0`). function python_version::parse_runtime_txt() { - local contents="${1}" + local runtime_txt_path="${1}" + local trimmed_contents + # The file contents with commented lines and leading/trailing whitespace removed. + trimmed_contents="$(python_version::read_trimmed_version_lines "${runtime_txt_path}")" - # The file must contain a string of form `python-N.N` or `python-N.N.N`. - # Leading/trailing whitespace is permitted. - if [[ "${contents}" =~ ^[[:space:]]*python-(${PYTHON_VERSION_REGEX})[[:space:]]*$ ]]; then + # The version must be of form `python-N.N` or `python-N.N.N`. + if [[ "${trimmed_contents}" =~ ^python-(${PYTHON_VERSION_REGEX})$ ]]; then local version="${BASH_REMATCH[1]}" echo "${version}" else local instructions instructions="$(python_version::python_version_file_instructions "runtime.txt" "${DEFAULT_PYTHON_MAJOR_VERSION}")" + # We intentionally don't display the current contents of the file here, to prevent users + # from copying that into the .python-version file instead of our example valid version. output::error <<-EOF Error: Invalid Python version in runtime.txt. The Python version specified in your runtime.txt file isn't in the correct format. - The following file contents were found, which aren't valid: - ${contents:0:100} - However, the runtime.txt file is deprecated since it has been replaced by the more widely supported .python-version file: https://devcenter.heroku.com/changelog-items/3141 @@ -133,31 +128,60 @@ function python_version::parse_runtime_txt() { ${instructions} EOF build_data::set_string "failure_reason" "runtime-txt::invalid-version" - build_data::set_string "failure_detail" "${contents:0:50}" + build_data::set_string "failure_detail" "${trimmed_contents:0:100}" exit 1 fi } # Parse the contents of a .python-version file and return the Python version substring (e.g. `3.12` or `3.12.0`). function python_version::parse_python_version_file() { - local contents="${1}" - local version_lines=() + local python_version_file_path="${1}" + local versions=() + local version - while IFS= read -r line; do - # Ignore lines that only contain whitespace and/or comments. - if [[ ! "${line}" =~ ^[[:space:]]*(#.*)?$ ]]; then - version_lines+=("${line}") + # shellcheck disable=SC2312 # Shellcheck doesn't take the `wait $!` into account. + mapfile -t versions < <(python_version::read_trimmed_version_lines "${python_version_file_path}") + # Ensure that if `python_version::read_trimmed_version_lines` fails inside the process + # substitution that the non-zero exit status is propagated. + wait $! + + # We validate all of the version lines up front, since in the case where there are multiple invalid + # versions, we want to show an "invalid version" error rather than the "multiple versions" error. + for version in "${versions[@]}"; do + if [[ "${version}" =~ ^${PYTHON_VERSION_REGEX}$ ]]; then + continue fi - done <<<"${contents}" - case "${#version_lines[@]}" in - 1) - local line="${version_lines[0]}" - if [[ "${line}" =~ ^[[:space:]]*(${PYTHON_VERSION_REGEX})[[:space:]]*$ ]]; then - local version="${BASH_REMATCH[1]}" - echo "${version}" - return 0 - else + # If we didn't find a valid Python version string, we check the file encoding so that we + # can display a more helpful error message if it turns out that the version was valid but + # that the file was just saved in the wrong encoding. + # + # Example valid values: + # `ASCII text` + # `ASCII text, with CRLF line terminators` + # `ASCII text, with no line terminators` + # `Unicode text, UTF-8 text` + # + # Example invalid values: + # `Unicode text, UTF-8 (with BOM) text` + # `Unicode text, UTF-16, little-endian text, with CRLF line terminators` + # `data` (for example when NUL or CTRL characters found) + # + # Note: File can also return `very short file (no magic)` (eg a file that contains just a newline) + # and `empty`, but we won't see those here since we're iterating over trimmed lines. + local file_encoding + file_encoding="$(file --brief --dereference "${python_version_file_path}")" + + case "${file_encoding}" in + *"ASCII text"* | *"UTF-8 text"*) + # Replace everything but printable ASCII, spaces and tabs with the Unicode replacement + # character, so any invisible unwanted characters (such as ASCII control codes or the + # Unicode zero width space character) are visible in the error message. + local escaped_version + escaped_version="$( + LC_ALL=C + echo "${version//[^[:print:][:blank:]]/�}" + )" output::error <<-EOF Error: Invalid Python version in .python-version. @@ -165,7 +189,7 @@ function python_version::parse_python_version_file() { isn't in the correct format. The following version was found: - ${line} + ${escaped_version} However, the Python version must be specified as either: 1. The major version only, for example: ${DEFAULT_PYTHON_MAJOR_VERSION} (recommended) @@ -184,13 +208,43 @@ function python_version::parse_python_version_file() { each time it builds. EOF build_data::set_string "failure_reason" "python-version-file::invalid-version" - build_data::set_string "failure_detail" "${line:0:50}" + build_data::set_string "failure_detail" "${version:0:100}" exit 1 - fi + ;; + *) + output::error <<-EOF + Error: Unable to read .python-version. + + Your .python-version file couldn't be read because it's using + an unsupported file encoding: + ${file_encoding} + + Configure your editor to save files as UTF-8, without a BOM, + then delete and recreate the file using the correct encoding. + + If that doesn't work, make sure you don't have a .gitattributes + file that's overriding the file encoding. + + Note: On Windows, if you pipe or redirect output to a file + it can result in the file being encoded in UTF-16 LE when + using certain terminals and Windows settings. We recommend + you create the file using a text editor instead. + EOF + build_data::set_string "failure_reason" "python-version-file::invalid-encoding" + build_data::set_string "failure_detail" "${file_encoding}" + exit 1 + ;; + esac + done + + case "${#versions[@]}" in + 1) + echo "${versions[0]}" + return 0 ;; 0) output::error <<-EOF - Error: Invalid Python version in .python-version. + Error: No Python version found in .python-version. No Python version was found in your .python-version file. @@ -205,19 +259,17 @@ function python_version::parse_python_version_file() { begin with a '#', otherwise it will be treated as a comment. EOF build_data::set_string "failure_reason" "python-version-file::no-version" - build_data::set_string "failure_detail" "${contents:0:50}" exit 1 ;; *) - local first_five_version_lines=("${version_lines[@]:0:5}") output::error <<-EOF - Error: Invalid Python version in .python-version. + Error: Multiple Python versions found in .python-version. Multiple versions were found in your .python-version file: $( IFS=$'\n' - echo "${first_five_version_lines[*]}" + echo "${versions[*]}" ) Update the file so it contains only one Python version. @@ -225,20 +277,29 @@ function python_version::parse_python_version_file() { For example, to request the latest version of Python ${DEFAULT_PYTHON_MAJOR_VERSION}, update your .python-version file so it contains exactly: ${DEFAULT_PYTHON_MAJOR_VERSION} - - If you have added comments to the file, make sure that those - lines begin with a '#', so that they are ignored. EOF build_data::set_string "failure_reason" "python-version-file::multiple-versions" build_data::set_string "failure_detail" "$( IFS=, - echo "${first_five_version_lines[*]}" + echo "${versions[*]}" )" exit 1 ;; esac } +# Outputs all populated (non-empty and not commented with '#') lines from the passed file, +# with leading/trailing whitespace (including Unicode whitespace) trimmed from each line. +function python_version::read_trimmed_version_lines() { + local file="${1}" + LC_ALL=C.UTF-8 sed \ + --regexp-extended \ + --expression 's/^[[:space:]]+//' \ + --expression 's/[[:space:]]+$//' \ + --expression '/^(#|$)/d' \ + "${file}" +} + # Read the Python version from a Pipfile.lock, which can exist in one of two optional fields, # `python_full_version` (as N.N.N) and `python_version` (as N.N). If both fields are # defined, we will use the value set in `python_full_version`. See: diff --git a/lib/utils.sh b/lib/utils.sh index a640bd2ca..904532768 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -16,17 +16,6 @@ function utils::get_requirement_version() { echo "${requirement_version}" } -# Replaces any invisible unwanted characters (such as ASCII control codes) with the Unicode -# replacement character, so they are visible in error messages. Also removes any carriage -# return characters, to prevent them interfering with the rendering of error messages that -# include the raw file contents. -# `LC_ALL=C` ensures that `[:space:]` only matches ASCII whitespace, since we want Unicode -# whitespace (such as the invisible Unicode zero width space character) to be replaced too. -function utils::read_file_with_special_chars_substituted() { - local file="${1}" - LC_ALL=C sed --regexp-extended --expression 's/[^[:print:][:space:]]/�/g' --expression 's/\r$//' "${file}" -} - # Python bundles pip within its standard library, which we can use to install our chosen # pip version from PyPI, saving us from having to download the usual pip bootstrap script. function utils::bundled_pip_module_path() { diff --git a/spec/fixtures/python_version_file_invalid_version/.python-version b/spec/fixtures/python_version_file_invalid_version/.python-version index 36fc6073580ae57001fd33ac77b068b0be906358..8641e5c56e55c8e1646d52b30f8541e233da9161 100644 GIT binary patch delta 213 zcmX9&zYf7b9FDMP(kHNdi2(^skzlp4h=k2_?fvM5yQ|(+Vlx?7h!^lIY#zgtxW@SX z|1RZo<@TP}u}p-)fifesf=P6~C{*A30g0+L6m|a0ZSs^4Nb+Ry{hn!!ik3wQzih_%)k3w>OUP)1YjzV%qVo_ppNorB? r Python app detected @@ -330,7 +330,7 @@ remote: ! isn't in the correct format. remote: ! remote: ! The following version was found: - remote: ! ���3.�12.�0� + remote: ! ���python - ��3.12.0�[0m remote: ! remote: ! However, the Python version must be specified as either: remote: ! 1. The major version only, for example: #{DEFAULT_PYTHON_MAJOR_VERSION} (recommended) @@ -354,6 +354,37 @@ end end + context 'when .python-version is saved using an unsupported file encoding' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_file_unsupported_encoding', allow_failure: true) } + + it 'aborts the build with an invalid file encoding message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: + remote: ! Error: Unable to read .python-version. + remote: ! + remote: ! Your .python-version file couldn't be read because it's using + remote: ! an unsupported file encoding: + remote: ! Unicode text, UTF-8 (with BOM) text, with CRLF line terminators + remote: ! + remote: ! Configure your editor to save files as UTF-8, without a BOM, + remote: ! then delete and recreate the file using the correct encoding. + remote: ! + remote: ! If that doesn't work, make sure you don't have a .gitattributes + remote: ! file that's overriding the file encoding. + remote: ! + remote: ! Note: On Windows, if you pipe or redirect output to a file + remote: ! it can result in the file being encoded in UTF-16 LE when + remote: ! using certain terminals and Windows settings. We recommend + remote: ! you create the file using a text editor instead. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + context 'when .python-version does not contain a Python version' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_file_no_version', allow_failure: true) } @@ -362,7 +393,7 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: - remote: ! Error: Invalid Python version in .python-version. + remote: ! Error: No Python version found in .python-version. remote: ! remote: ! No Python version was found in your .python-version file. remote: ! @@ -390,11 +421,10 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: - remote: ! Error: Invalid Python version in .python-version. + remote: ! Error: Multiple Python versions found in .python-version. remote: ! remote: ! Multiple versions were found in your .python-version file: remote: ! - remote: ! // invalid comment remote: ! 3.12 remote: ! 2.7 remote: ! @@ -403,9 +433,6 @@ remote: ! For example, to request the latest version of Python #{DEFAULT_PYTHON_MAJOR_VERSION}, remote: ! update your .python-version file so it contains exactly: remote: ! #{DEFAULT_PYTHON_MAJOR_VERSION} - remote: ! - remote: ! If you have added comments to the file, make sure that those - remote: ! lines begin with a '#', so that they are ignored. remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -561,9 +588,6 @@ remote: ! The Python version specified in your runtime.txt file isn't remote: ! in the correct format. remote: ! - remote: ! The following file contents were found, which aren't valid: - remote: ! ���python-3.12.0� - remote: ! remote: ! However, the runtime.txt file is deprecated since it has been remote: ! replaced by the more widely supported .python-version file: remote: ! https://devcenter.heroku.com/changelog-items/3141 diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 813409a59..16fa03817 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -79,7 +79,6 @@ remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", - remote: "python_version_file_encoding": "ASCII text", remote: "python_version_files": ".python-version,", remote: "python_version_major": "3.13", remote: "python_version_origin": ".python-version", From 947eb153e2889eb8f57101f796e95be735e6a7b4 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 12 Nov 2025 17:35:02 +0000 Subject: [PATCH 273/406] Prepare release v318 (#1959) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a6510657e..b96b07173 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v318] - 2025-11-12 + - Improved the error message shown when a `.python-version` file uses an unsupported file encoding. ([#1958](https://github.com/heroku/heroku-buildpack-python/pull/1958)) - Improved the error message shown when a `.python-version` or `runtime.txt` file contain an invalid Python version. ([#1958](https://github.com/heroku/heroku-buildpack-python/pull/1958)) - Added support for commented lines in `runtime.txt`, for parity with `.python-version`. ([#1958](https://github.com/heroku/heroku-buildpack-python/pull/1958)) @@ -1387,7 +1390,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v317...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v318...main +[v318]: https://github.com/heroku/heroku-buildpack-python/compare/v317...v318 [v317]: https://github.com/heroku/heroku-buildpack-python/compare/v316...v317 [v316]: https://github.com/heroku/heroku-buildpack-python/compare/v315...v316 [v315]: https://github.com/heroku/heroku-buildpack-python/compare/v314...v315 From 2f603c8010baa08bb77d4784c59d6e33fd932f76 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 13 Nov 2025 12:19:23 +0000 Subject: [PATCH 274/406] Bump uv from 0.9.7 to 0.9.9 (#1961) * Bump uv from 0.9.7 to 0.9.9 Bumps [uv](https://github.com/astral-sh/uv) from 0.9.7 to 0.9.9. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.9.7...0.9.9) --- updated-dependencies: - dependency-name: uv dependency-version: 0.9.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b96b07173..1e5f4b7af 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.9.7 to 0.9.9. ([#1961](https://github.com/heroku/heroku-buildpack-python/pull/1961)) ## [v318] - 2025-11-12 diff --git a/requirements/uv.txt b/requirements/uv.txt index e09f8b1ba..99caef278 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.9.7 +uv==0.9.9 From 9ca9559958ca5d281f2d9121c74ba737fc86c5d4 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 13 Nov 2025 13:47:19 +0000 Subject: [PATCH 275/406] Always use `local` for vars inside functions (#1960) Adds a couple of stray missing `local` declarations for completeness. (Previously they were implicitly created as global via the `for` loop definition.) --- lib/checks.sh | 1 + vendor/buildpack-stdlib_v8.sh | 1 + 2 files changed, 2 insertions(+) diff --git a/lib/checks.sh b/lib/checks.sh index 36f7f2206..65c6727c9 100644 --- a/lib/checks.sh +++ b/lib/checks.sh @@ -105,6 +105,7 @@ function checks::existing_python_dir_present() { function checks::existing_venv_dir_present() { local build_dir="${1}" + local venv_name for venv_name in ".venv" "venv"; do if [[ -f "${build_dir}/${venv_name}/pyvenv.cfg" ]]; then diff --git a/vendor/buildpack-stdlib_v8.sh b/vendor/buildpack-stdlib_v8.sh index ad8558f06..8f82fc152 100755 --- a/vendor/buildpack-stdlib_v8.sh +++ b/vendor/buildpack-stdlib_v8.sh @@ -48,6 +48,7 @@ export_env() { local blacklist blacklist="$(_env_blacklist "${3:-}")" if [[ -d "${env_dir}" ]]; then + local e # Environment variable names won't contain characters affected by: # shellcheck disable=SC2045 for e in $(ls "${env_dir}"); do From 94209ab059e65992282a28ab8d23d4a6cd09a6d5 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 14 Nov 2025 13:25:51 +0000 Subject: [PATCH 276/406] Further improve `.python-version` and `runtime.txt` error handling (#1962) Some further improvements on-top of those made in #1958, based on build error message scenarios seen in Honeycomb. Now, if `.python-version` contains a single ESC control code (which gets categorised as "very short file"), or contains any of the ASCI control codes that result in the file being categorised as "data" (such as NUL), then the "invalid python version" error message variant is shown instead of the "invalid text encoding" variant. In addition, any NULs in the file are substituted with a placeholder to avoid this Bash warning: ``` /tmp/buildpack/lib/python_version.sh: line 104: warning: command substitution: ignored null byte in input ``` See: https://manpages.ubuntu.com/manpages/noble/en/man1/file.1.html GUS-W-20220514. --- CHANGELOG.md | 2 ++ lib/python_version.sh | 27 ++++++++++-------- .../runtime_txt_invalid_version/runtime.txt | Bin 144 -> 146 bytes spec/hatchet/python_version_spec.rb | 4 +-- 4 files changed, 19 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1e5f4b7af..b3417b0e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,8 @@ ## [Unreleased] - Updated uv from 0.9.7 to 0.9.9. ([#1961](https://github.com/heroku/heroku-buildpack-python/pull/1961)) +- Improved the error message shown for `.python-version` files that contain unexpected ASCII control code characters. ([#1962](https://github.com/heroku/heroku-buildpack-python/pull/1962)) +- Fixed Bash command substitution warnings from being shown if `runtime.txt` contains null byte characters. ([#1962](https://github.com/heroku/heroku-buildpack-python/pull/1962)) ## [v318] - 2025-11-12 diff --git a/lib/python_version.sh b/lib/python_version.sh index 85bdafd57..a55c883f6 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -152,28 +152,28 @@ function python_version::parse_python_version_file() { continue fi - # If we didn't find a valid Python version string, we check the file encoding so that we + # If we didn't find a valid Python version string, we check the text encoding so that we # can display a more helpful error message if it turns out that the version was valid but - # that the file was just saved in the wrong encoding. + # that the file was just saved in the wrong encoding (such as UTF-8 with BOM or UTF-16). # - # Example valid values: + # Example values `file` can return: # `ASCII text` # `ASCII text, with CRLF line terminators` # `ASCII text, with no line terminators` # `Unicode text, UTF-8 text` - # - # Example invalid values: # `Unicode text, UTF-8 (with BOM) text` # `Unicode text, UTF-16, little-endian text, with CRLF line terminators` - # `data` (for example when NUL or CTRL characters found) + # `data` (such as when the file contains a NUL or other control code characters) + # `very short file (no magic)` (such as when the file contains a single ESC character) # - # Note: File can also return `very short file (no magic)` (eg a file that contains just a newline) - # and `empty`, but we won't see those here since we're iterating over trimmed lines. + # Note: File can also return `empty` but in that case we wouldn't be iterating over found lines. local file_encoding - file_encoding="$(file --brief --dereference "${python_version_file_path}")" + # We exclude some file type tests to avoid false positives, since we only need the encoding. + file_encoding="$(file --brief --dereference --exclude json --exclude soft "${python_version_file_path}")" case "${file_encoding}" in - *"ASCII text"* | *"UTF-8 text"*) + # Cases where the text encoding isn't the issue, and so the version itself must be invalid. + *"ASCII text"* | *"UTF-8 text"* | *"very short file"* | "data") # Replace everything but printable ASCII, spaces and tabs with the Unicode replacement # character, so any invisible unwanted characters (such as ASCII control codes or the # Unicode zero width space character) are visible in the error message. @@ -211,19 +211,20 @@ function python_version::parse_python_version_file() { build_data::set_string "failure_detail" "${version:0:100}" exit 1 ;; + # Unsupported text encodings such as UTF-8 with BOM or UTF-16. *) output::error <<-EOF Error: Unable to read .python-version. Your .python-version file couldn't be read because it's using - an unsupported file encoding: + an unsupported text encoding: ${file_encoding} Configure your editor to save files as UTF-8, without a BOM, then delete and recreate the file using the correct encoding. If that doesn't work, make sure you don't have a .gitattributes - file that's overriding the file encoding. + file that's overriding the text encoding. Note: On Windows, if you pipe or redirect output to a file it can result in the file being encoded in UTF-16 LE when @@ -290,12 +291,14 @@ function python_version::parse_python_version_file() { # Outputs all populated (non-empty and not commented with '#') lines from the passed file, # with leading/trailing whitespace (including Unicode whitespace) trimmed from each line. +# We replace any NUL characters with a placeholder since Bash variables can't store them. function python_version::read_trimmed_version_lines() { local file="${1}" LC_ALL=C.UTF-8 sed \ --regexp-extended \ --expression 's/^[[:space:]]+//' \ --expression 's/[[:space:]]+$//' \ + --expression 's/\x0/␀/' \ --expression '/^(#|$)/d' \ "${file}" } diff --git a/spec/fixtures/runtime_txt_invalid_version/runtime.txt b/spec/fixtures/runtime_txt_invalid_version/runtime.txt index f4ae593e64d2b5f2b89d0a1a65a4e1fbef91fe17..14025c7f8d862c84b9f1ad10faf9056b4a186093 100644 GIT binary patch delta 9 QcmbQhIEit>1SSS901ib0Q2+n{ delta 6 NcmbQlIDv7(1ONwW0!07- diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 663e54d90..13cd9c51c 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -365,14 +365,14 @@ remote: ! Error: Unable to read .python-version. remote: ! remote: ! Your .python-version file couldn't be read because it's using - remote: ! an unsupported file encoding: + remote: ! an unsupported text encoding: remote: ! Unicode text, UTF-8 (with BOM) text, with CRLF line terminators remote: ! remote: ! Configure your editor to save files as UTF-8, without a BOM, remote: ! then delete and recreate the file using the correct encoding. remote: ! remote: ! If that doesn't work, make sure you don't have a .gitattributes - remote: ! file that's overriding the file encoding. + remote: ! file that's overriding the text encoding. remote: ! remote: ! Note: On Windows, if you pipe or redirect output to a file remote: ! it can result in the file being encoded in UTF-16 LE when From b732e6ffe1496ddf1f73dfbd80799de6fbd8cfb8 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 14 Nov 2025 14:48:32 +0000 Subject: [PATCH 277/406] Improve the error message when the build data file has been deleted (#1963) On Honeycomb I've seen cases of builds failing due to a pre/post-compile hook having deleted the buildpack's internal build data file - typically as part of the user trying to clean up the build cache. This file is required for the buildpack to function correctly, so deleting it isn't supported. However, we can show a clearer error message. Before: ``` -----> Running bin/post_compile hook + rm -rf /tmp/cache jq: error: Could not open file /tmp/cache/build-data/python.json: No such file or directory jq: error: Could not open file /tmp/cache/build-data/python.json: No such file or directory ! Internal Error: An unhandled buildpack error occurred. ! ! An unhandled error occurred while executing the command: ! jq --exit-status --arg key "${key}" "${jq_args[@]}" '. + { ($key): ($value) }' "${BUILD_DATA_FILE}" ! ! If this issue persists, please open a support ticket or file ! an issue on the buildpack's GitHub repository: ! https://help.heroku.com/ ! https://github.com/heroku/heroku-buildpack-python/issues ! ! Stack trace: ! build_data::_set @ /tmp/buildpack/lib/build_data.sh:109 ! build_data::set_raw @ /tmp/buildpack/lib/build_data.sh:76 ! build_data::set_duration @ /tmp/buildpack/lib/build_data.sh:60 ! hooks::run_hook @ /tmp/buildpack/lib/hooks.sh:37 ! main @ ./bin/compile:289 jq: error: Could not open file /tmp/cache/build-data/python.json: No such file or directory jq: error: Could not open file /tmp/cache/build-data/python.json: No such file or directory ! Internal Error: An unhandled buildpack error occurred. ! ! An unhandled error occurred while executing the command: ! new_data_file_contents="$(jq --exit-status --arg key "${key}" "${jq_args[@]}" '. + { ($key): ($value) }' "${BUILD_DATA_FILE}")" ! ! If this issue persists, please open a support ticket or file ! an issue on the buildpack's GitHub repository: ! https://help.heroku.com/ ! https://github.com/heroku/heroku-buildpack-python/issues ! ! Stack trace: ! build_data::_set @ /tmp/buildpack/lib/build_data.sh:109 ! build_data::set_raw @ /tmp/buildpack/lib/build_data.sh:76 ! build_data::set_duration @ /tmp/buildpack/lib/build_data.sh:60 ! hooks::run_hook @ /tmp/buildpack/lib/hooks.sh:37 ! main @ ./bin/compile:289 jq: error: Could not open file /tmp/cache/build-data/python.json: No such file or directory ``` After: ``` -----> Running bin/post_compile hook + rm -rf /tmp/cache ! Error: The buildpack's build data file has been deleted. ! ! The Python buildpack's build data file can't be found: ! /tmp/cache/build-data/python.json ! ! This file is required for the buildpack to work correctly, ! and so you must not delete it when removing files from the ! build cache or /tmp directories. ``` Lastly, we also now use `--exit-status` in more places to prevent invalid empty build data files from being silently ignored. GUS-W-20220731. --- CHANGELOG.md | 1 + lib/build_data.sh | 25 ++++++++++++++++--- .../hooks_delete_cache_dir/bin/post_compile | 6 +++++ .../hooks_delete_cache_dir/requirements.txt | 0 spec/hatchet/hooks_spec.rb | 24 ++++++++++++++++++ 5 files changed, 52 insertions(+), 4 deletions(-) create mode 100644 spec/fixtures/hooks_delete_cache_dir/bin/post_compile create mode 100644 spec/fixtures/hooks_delete_cache_dir/requirements.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index b3417b0e8..6dd2ef56f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ - Updated uv from 0.9.7 to 0.9.9. ([#1961](https://github.com/heroku/heroku-buildpack-python/pull/1961)) - Improved the error message shown for `.python-version` files that contain unexpected ASCII control code characters. ([#1962](https://github.com/heroku/heroku-buildpack-python/pull/1962)) - Fixed Bash command substitution warnings from being shown if `runtime.txt` contains null byte characters. ([#1962](https://github.com/heroku/heroku-buildpack-python/pull/1962)) +- Improved the error message shown if the buildpack's build data file is deleted by a pre/post-compile hook. ([#1963](https://github.com/heroku/heroku-buildpack-python/pull/1963)) ## [v318] - 2025-11-12 diff --git a/lib/build_data.sh b/lib/build_data.sh index 9841e0f45..93434ac38 100644 --- a/lib/build_data.sh +++ b/lib/build_data.sh @@ -104,9 +104,25 @@ function build_data::_set() { local jq_args=(--argjson value "${value}") fi - local new_data_file_contents - new_data_file_contents="$(jq --arg key "${key}" "${jq_args[@]}" '. + { ($key): ($value) }' "${BUILD_DATA_FILE}")" - echo "${new_data_file_contents}" >"${BUILD_DATA_FILE}" + if [[ -f "${BUILD_DATA_FILE}" ]]; then + local new_data_file_contents + new_data_file_contents="$(jq --exit-status --arg key "${key}" "${jq_args[@]}" '. + { ($key): ($value) }' "${BUILD_DATA_FILE}")" + echo "${new_data_file_contents}" >"${BUILD_DATA_FILE}" + else + output::error <<-EOF + Error: Can't find the buildpack's build data file. + + The Python buildpack's internal build data file is missing: + ${BUILD_DATA_FILE} + + This file is required for the buildpack to work correctly, + and so you must not delete it when removing files from the + build cache or /tmp directories. + EOF + build_data::setup + build_data::set_string "failure_reason" "build-data::data-file-deleted" + exit 1 + fi } # Check whether an entry exists in the build data store for the current build. @@ -143,6 +159,7 @@ function build_data::get_previous() { tac "${LEGACY_BUILD_DATA_FILE}" | { grep --perl-regexp --only-matching --max-count=1 "^${key}=\K.*$" || true; } elif [[ -f "${PREVIOUS_BUILD_DATA_FILE}" ]]; then # The `// empty` ensures we return the empty string rather than `null` if the key doesn't exist. + # We don't use `--exit-status` since `false` is a valid value for us to retrieve. jq --raw-output ".${key} // empty" "${PREVIOUS_BUILD_DATA_FILE}" fi } @@ -170,5 +187,5 @@ function build_data::current_unix_realtime() { # build_data::print_bin_report_json # ``` function build_data::print_bin_report_json() { - jq --sort-keys '.' "${BUILD_DATA_FILE}" + jq --exit-status --sort-keys '.' "${BUILD_DATA_FILE}" } diff --git a/spec/fixtures/hooks_delete_cache_dir/bin/post_compile b/spec/fixtures/hooks_delete_cache_dir/bin/post_compile new file mode 100644 index 000000000..383d0656b --- /dev/null +++ b/spec/fixtures/hooks_delete_cache_dir/bin/post_compile @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +set -euo pipefail + +set -x +rm -rf "${CACHE_DIR:?}" diff --git a/spec/fixtures/hooks_delete_cache_dir/requirements.txt b/spec/fixtures/hooks_delete_cache_dir/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/hatchet/hooks_spec.rb b/spec/hatchet/hooks_spec.rb index e58a7969c..18fa2eed6 100644 --- a/spec/hatchet/hooks_spec.rb +++ b/spec/hatchet/hooks_spec.rb @@ -78,4 +78,28 @@ end end end + + context 'when an app tries to delete the whole cache directory including the build data file' do + let(:app) { Hatchet::Runner.new('spec/fixtures/hooks_delete_cache_dir', allow_failure: true) } + + it 'aborts the build with a suitable error message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Running bin/post_compile hook + remote: + rm -rf /tmp/codon/tmp/cache + remote: + remote: ! Error: Can't find the buildpack's build data file. + remote: ! + remote: ! The Python buildpack's internal build data file is missing: + remote: ! /tmp/codon/tmp/cache/build-data/python.json + remote: ! + remote: ! This file is required for the buildpack to work correctly, + remote: ! and so you must not delete it when removing files from the + remote: ! build cache or /tmp directories. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end end From 9962336f0f5969388e7562f66579db50fbf579db Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 14 Nov 2025 19:37:39 +0000 Subject: [PATCH 278/406] Work around tmpfs default permissions regression in runc 1.3.3 (#1964) The Python classic repo's CI just started failing in the container-test job with: `mkdir: cannot create directory '/app/.heroku': Permission denied` eg: https://github.com/heroku/heroku-buildpack-python/actions/runs/19368179568/job/55418539741 After updating Docker locally, I was able to reproduce the error, and have found it's due to the recent runc 1.3.3 release: https://github.com/opencontainers/runc/releases/tag/v1.3.3 This runc release includes a number of security fixes - however, one of which has a regression: https://github.com/opencontainers/runc/issues/4971 There is a fix for this upstream: https://github.com/opencontainers/runc/pull/4973 ...but it's not released yet. However, we can work around the issue by explicitly setting the previous tmpfs permissions using `:mode=1777`: https://docs.docker.com/engine/storage/tmpfs/#options-for---tmpfs GUS-W-20221627. --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 6ba94fef7..ea5971da4 100644 --- a/Makefile +++ b/Makefile @@ -25,7 +25,7 @@ format: run: @echo "Running buildpack using: STACK=$(STACK) FIXTURE=$(FIXTURE)" - @docker run --rm -v $(PWD):/src:ro --tmpfs /app -e "HOME=/app" -e "STACK=$(STACK)" "$(STACK_IMAGE_TAG)" \ + @docker run --rm -v $(PWD):/src:ro --tmpfs /app:mode=1777 -e "HOME=/app" -e "STACK=$(STACK)" "$(STACK_IMAGE_TAG)" \ bash -euo pipefail -O dotglob -c '\ mkdir /tmp/buildpack /tmp/cache /tmp/env; \ cp -r /src/{bin,lib,requirements,vendor} /tmp/buildpack; \ From 1e300966d3b707511556cfe90cc8eb8db87a9eea Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Fri, 14 Nov 2025 20:42:44 +0000 Subject: [PATCH 279/406] Prepare release v319 (#1965) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6dd2ef56f..6a5b27ab9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v319] - 2025-11-14 + - Updated uv from 0.9.7 to 0.9.9. ([#1961](https://github.com/heroku/heroku-buildpack-python/pull/1961)) - Improved the error message shown for `.python-version` files that contain unexpected ASCII control code characters. ([#1962](https://github.com/heroku/heroku-buildpack-python/pull/1962)) - Fixed Bash command substitution warnings from being shown if `runtime.txt` contains null byte characters. ([#1962](https://github.com/heroku/heroku-buildpack-python/pull/1962)) @@ -1394,7 +1397,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v318...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v319...main +[v319]: https://github.com/heroku/heroku-buildpack-python/compare/v318...v319 [v318]: https://github.com/heroku/heroku-buildpack-python/compare/v317...v318 [v317]: https://github.com/heroku/heroku-buildpack-python/compare/v316...v317 [v316]: https://github.com/heroku/heroku-buildpack-python/compare/v315...v316 From d9c73e2c53c8bf7eac7edb10a451b73103242b42 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 17 Nov 2025 14:05:16 +0000 Subject: [PATCH 280/406] Fixed regex for null bytes character replacement (#1966) So it replaces all occurrences in each line, rather than just the first. GUS-W-20241349. --- CHANGELOG.md | 1 + lib/python_version.sh | 2 +- .../runtime_txt_invalid_version/runtime.txt | Bin 146 -> 147 bytes 3 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a5b27ab9..6a87c4fad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Fixed regex for null bytes character replacement to replace all occurrences, not just the first. ([#1966](https://github.com/heroku/heroku-buildpack-python/pull/1966)) ## [v319] - 2025-11-14 diff --git a/lib/python_version.sh b/lib/python_version.sh index a55c883f6..2971686b2 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -298,7 +298,7 @@ function python_version::read_trimmed_version_lines() { --regexp-extended \ --expression 's/^[[:space:]]+//' \ --expression 's/[[:space:]]+$//' \ - --expression 's/\x0/␀/' \ + --expression 's/\x0/␀/g' \ --expression '/^(#|$)/d' \ "${file}" } diff --git a/spec/fixtures/runtime_txt_invalid_version/runtime.txt b/spec/fixtures/runtime_txt_invalid_version/runtime.txt index 14025c7f8d862c84b9f1ad10faf9056b4a186093..75986f4ede99ad35623d222c8c27fdf42eabaaf6 100644 GIT binary patch delta 9 QcmbQlIGJ(6L?#9<01jdTRR910 delta 8 PcmbQtIEit>L`E(E3>N}a From b4260ada2ea84dd54b318117ba94183ba5c41e03 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 17 Nov 2025 14:19:27 +0000 Subject: [PATCH 281/406] Unpin Poetry's `dulwich` version (#1967) Since the incompatibility with older Python has now been fixed upstream: https://github.com/jelmer/dulwich/issues/1948 https://github.com/jelmer/dulwich/pull/1949 GUS-W-20241443. --- CHANGELOG.md | 1 + lib/poetry.sh | 1 - 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a87c4fad..3ff23c398 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Fixed regex for null bytes character replacement to replace all occurrences, not just the first. ([#1966](https://github.com/heroku/heroku-buildpack-python/pull/1966)) +- Unpinned Poetry's `dulwich` version, since the upstream incompatibility with older Python has been fixed. ([#1967](https://github.com/heroku/heroku-buildpack-python/pull/1967)) ## [v319] - 2025-11-14 diff --git a/lib/poetry.sh b/lib/poetry.sh index 5573c9be8..eec1d3942 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -68,7 +68,6 @@ function poetry::install_poetry() { --no-input \ --quiet \ "poetry==${POETRY_VERSION}" \ - dulwich==0.24.5 \ |& output::indent }; then output::error <<-EOF From 2bf1d7d8651cfe1cb4e223a1cf89545233be9d4f Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 20 Nov 2025 13:57:30 +0000 Subject: [PATCH 282/406] Add a check for misspelled `.python-version` files (#1970) We recommend that apps use a `.python-version` file to specify their Python version. However, as seen on Honeycomb after #1904, a small number of apps in the wild have spelt the file incorrectly - and sometimes with names that look very similar to the correct name, so the mistake would be easy to miss. Some of the mistakes seen include: - Omitting the leading `.` in the filename. - Adding a file extension such as `.txt`. - Using an underscore or a period instead of the hyphen. - Having trailing whitespace in the filename. The metrics show that some apps have both a correctly spelled file and an incorrectly spelled one, which I suspect would lead to a confusing UX if the wrong file was ever updated ("why isn't Heroku using the Python version I've specified?") - particularly in the case where the user doesn't have dotfiles visible in eg Finder and the spelling mistake was omitting the leading `.`. As such, we still error if a misspelled file is found, even if a correctly named one also exists. GUS-W-19687420. --- CHANGELOG.md | 1 + lib/python_version.sh | 64 +++++++++++++++---- .../.python-version | 1 + .../.python-version | 1 + .../requirements.txt | 0 spec/hatchet/pip_spec.rb | 1 - spec/hatchet/poetry_spec.rb | 1 - spec/hatchet/python_version_spec.rb | 24 +++++++ spec/hatchet/uv_spec.rb | 1 - 9 files changed, 80 insertions(+), 14 deletions(-) create mode 100644 spec/fixtures/python_version_file_misspelled/.python-version create mode 100644 spec/fixtures/python_version_file_misspelled/.python-version create mode 100644 spec/fixtures/python_version_file_misspelled/requirements.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index 3ff23c398..948cf799c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Added a check for misspelled `.python-version` files. ([#1970](https://github.com/heroku/heroku-buildpack-python/pull/1970)) - Fixed regex for null bytes character replacement to replace all occurrences, not just the first. ([#1966](https://github.com/heroku/heroku-buildpack-python/pull/1966)) - Unpinned Poetry's `dulwich` version, since the upstream incompatibility with older Python has been fixed. ([#1967](https://github.com/heroku/heroku-buildpack-python/pull/1967)) diff --git a/lib/python_version.sh b/lib/python_version.sh index 2971686b2..33bf394a8 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -24,6 +24,27 @@ PYTHON_VERSION_REGEX="${INT_REGEX}\.${INT_REGEX}(\.${INT_REGEX})?" # Versions of form N.N.N only. PYTHON_FULL_VERSION_REGEX="${INT_REGEX}\.${INT_REGEX}\.${INT_REGEX}" +# Misspellings of the `.python-version` file seen in the wild. +MISSPELLED_PYTHON_VERSION_FILE_NAMES=( + ".python_version.txt" + ".python_version" + ".python-version " + ".python-version." + ".python-version.py" + ".python-version.rtf" + ".python-version.txt" + ".python-version.TXT" + ".Python-version" + ".python.version" + ".pythonversion" + "'.python-version'" + "python_version.txt" + "python_version" + "python-version." + "python-version.txt" + "python-version" +) + # Determine what Python version has been requested for the project. # # Returns a version request of form N.N or N.N.N, with basic validation checks that the version @@ -50,17 +71,6 @@ function python_version::read_requested_python_version() { declare -n version="${4}" declare -n origin="${5}" - # Record the names of files similar to .python-version in the root of the app, to determine - # how often that file is misspelled, as a temporary first step before deciding whether to add - # a new warning/error (and if so, which for which misspelled filenames it should check). - local python_version_files - python_version_files="$( - find . -maxdepth 1 -type f -iregex '\./\.?python.?version.*' -printf '%P\n' | sort | tr '\n' ',' || true - )" - if [[ -n "${python_version_files}" ]]; then - build_data::set_string "python_version_files" "${python_version_files}" - fi - local runtime_txt_path="${build_dir}/runtime.txt" if [[ -f "${runtime_txt_path}" ]]; then version="$(python_version::parse_runtime_txt "${runtime_txt_path}")" @@ -68,6 +78,28 @@ function python_version::read_requested_python_version() { return 0 fi + local misspelled_python_version_file_name + for misspelled_python_version_file_name in "${MISSPELLED_PYTHON_VERSION_FILE_NAMES[@]}"; do + if [[ -f "${build_dir}/${misspelled_python_version_file_name}" ]]; then + # We use quotes around the current filename since one of the misspellings seen in the + # wild is where the filename contains trailing whitespace. + output::error <<-EOF + Error: Your .python-version file is spelled incorrectly. + + Your app's .python-version file currently has the filename: + '${misspelled_python_version_file_name}' + + However, the correct spelling is (without quotes): + '.python-version' + + You must rename your file to the correct name. + EOF + build_data::set_string "failure_reason" "python-version-file::misspelled" + build_data::set_string "failure_detail" "${misspelled_python_version_file_name}" + exit 1 + fi + done + local python_version_file_path="${build_dir}/.python-version" if [[ -f "${python_version_file_path}" ]]; then version="$(python_version::parse_python_version_file "${python_version_file_path}")" @@ -75,6 +107,16 @@ function python_version::read_requested_python_version() { return 0 fi + # Record the names of files similar to `.python-version` in the root of the app, so we can track + # additional misspellings that might need to be added to `MISSPELLED_PYTHON_VERSION_FILE_NAMES`. + local python_version_files + python_version_files="$( + find . -maxdepth 1 -type f -iregex '\./.*python.?version.*' -printf '%P\n' | sort | tr '\n' ',' || true + )" + if [[ -n "${python_version_files}" ]]; then + build_data::set_string "python_version_files" "${python_version_files}" + fi + if [[ "${package_manager}" == "pipenv" ]]; then version="$(python_version::read_pipenv_python_version "${build_dir}")" # The Python version fields in a Pipfile.lock are optional. diff --git a/spec/fixtures/python_version_file_misspelled/.python-version b/spec/fixtures/python_version_file_misspelled/.python-version new file mode 100644 index 000000000..6324d401a --- /dev/null +++ b/spec/fixtures/python_version_file_misspelled/.python-version @@ -0,0 +1 @@ +3.14 diff --git a/spec/fixtures/python_version_file_misspelled/.python-version b/spec/fixtures/python_version_file_misspelled/.python-version new file mode 100644 index 000000000..f7549fdca --- /dev/null +++ b/spec/fixtures/python_version_file_misspelled/.python-version @@ -0,0 +1 @@ +# This file has a trailing space in its filename diff --git a/spec/fixtures/python_version_file_misspelled/requirements.txt b/spec/fixtures/python_version_file_misspelled/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index d30e14e98..8dc68798a 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -72,7 +72,6 @@ remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", - remote: "python_version_files": ".python-version,", remote: "python_version_major": "3.13", remote: "python_version_origin": ".python-version", remote: "python_version_outdated": false, diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 8bfdfdb40..7bbe29f5f 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -74,7 +74,6 @@ remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", - remote: "python_version_files": ".python-version,", remote: "python_version_major": "3.13", remote: "python_version_origin": ".python-version", remote: "python_version_outdated": false, diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 13cd9c51c..6652e3741 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -316,6 +316,30 @@ it_behaves_like 'builds with the requested Python version', '3.14', LATEST_PYTHON_3_14 end + context 'when .python-version is misspelled' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_file_misspelled', allow_failure: true) } + + it 'aborts the build with an invalid Python version message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: + remote: ! Error: Your .python-version file is spelled incorrectly. + remote: ! + remote: ! Your app's .python-version file currently has the filename: + remote: ! '.python-version ' + remote: ! + remote: ! However, the correct spelling is (without quotes): + remote: ! '.python-version' + remote: ! + remote: ! You must rename your file to the correct name. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + context 'when .python-version contains an invalid Python version string' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_file_invalid_version', allow_failure: true) } diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 16fa03817..5b32fccf5 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -79,7 +79,6 @@ remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", - remote: "python_version_files": ".python-version,", remote: "python_version_major": "3.13", remote: "python_version_origin": ".python-version", remote: "python_version_outdated": false, From 7131c23bdc26e17e1c68f9d0c0cbc12659c4b371 Mon Sep 17 00:00:00 2001 From: Richard Schneeman Date: Thu, 20 Nov 2025 08:20:26 -0600 Subject: [PATCH 283/406] Refactor Bash output helpers (#1968) 1. Prefer ANSI-C style escape quoting. `\e` is the ESC character (ASCII 27), same as `\033`: https://www.gnu.org/software/bash/manual/html_node/ANSI_002dC-Quoting.html 2. Prefer sed over while loop for output formatting: https://github.com/heroku/heroku-buildpack-ruby/pull/1663/files/84841382115e238cec5e0aaac70a7361ae24bde3#r2538926478 --------- Signed-off-by: Ed Morley <501702+edmorley@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- lib/output.sh | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/lib/output.sh b/lib/output.sh index a4eb9431e..1f6dda760 100644 --- a/lib/output.sh +++ b/lib/output.sh @@ -4,10 +4,10 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail -ANSI_BLUE='\033[1;34m' -ANSI_RED='\033[1;31m' -ANSI_YELLOW='\033[1;33m' -ANSI_RESET='\033[0m' +ANSI_BLUE=$'\e[1;34m' +ANSI_RED=$'\e[1;31m' +ANSI_YELLOW=$'\e[1;33m' +ANSI_RESET=$'\e[0m' # Output a single line step message to stdout. # @@ -40,11 +40,8 @@ function output::indent() { # EOF # ``` function output::notice() { - local line echo >&2 - while IFS= read -r line; do - echo -e "${ANSI_BLUE} ! ${line}${ANSI_RESET}" >&2 - done + sed --expression "s/^/${ANSI_BLUE} ! /" --expression "s/$/${ANSI_RESET}/" >&2 echo >&2 } @@ -59,11 +56,8 @@ function output::notice() { # EOF # ``` function output::warning() { - local line echo >&2 - while IFS= read -r line; do - echo -e "${ANSI_YELLOW} ! ${line}${ANSI_RESET}" >&2 - done + sed --expression "s/^/${ANSI_YELLOW} ! /" --expression "s/$/${ANSI_RESET}/" >&2 echo >&2 } @@ -78,10 +72,7 @@ function output::warning() { # EOF # ``` function output::error() { - local line echo >&2 - while IFS= read -r line; do - echo -e "${ANSI_RED} ! ${line}${ANSI_RESET}" >&2 - done + sed --expression "s/^/${ANSI_RED} ! /" --expression "s/$/${ANSI_RESET}/" >&2 echo >&2 } From 89efadca2f392ff050bf6a3c075ac15bcbf71b9b Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 20 Nov 2025 14:26:18 +0000 Subject: [PATCH 284/406] Prepare release v320 (#1971) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 948cf799c..b5858554b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v320] - 2025-11-20 + - Added a check for misspelled `.python-version` files. ([#1970](https://github.com/heroku/heroku-buildpack-python/pull/1970)) - Fixed regex for null bytes character replacement to replace all occurrences, not just the first. ([#1966](https://github.com/heroku/heroku-buildpack-python/pull/1966)) - Unpinned Poetry's `dulwich` version, since the upstream incompatibility with older Python has been fixed. ([#1967](https://github.com/heroku/heroku-buildpack-python/pull/1967)) @@ -1400,7 +1403,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v319...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v320...main +[v320]: https://github.com/heroku/heroku-buildpack-python/compare/v319...v320 [v319]: https://github.com/heroku/heroku-buildpack-python/compare/v318...v319 [v318]: https://github.com/heroku/heroku-buildpack-python/compare/v317...v318 [v317]: https://github.com/heroku/heroku-buildpack-python/compare/v316...v317 From 38c81f8c8ea5517c2f554b45ae5ba5427705302f Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 21 Nov 2025 09:32:20 +0000 Subject: [PATCH 285/406] Deprecate support for Python 3.10 (#1972) Since Python 3.10 reaches upstream EOL in October 2026: https://devguide.python.org/versions/#supported-versions The Python version support policy is that supported versions follows the upstream EOL lifecycle: https://devcenter.heroku.com/articles/python-support#python-version-support-policy To raise awareness of this, a deprecation warning is now shown, similar to that for Python 3.9. In addition the Python 3.9 warning has been updated to use the past tense to refer to the upstream EOL date, since it's now in the past. GUS-W-17595557. --- CHANGELOG.md | 1 + lib/python_version.sh | 21 +++++++++++++++-- spec/hatchet/pip_spec.rb | 4 ++-- spec/hatchet/pipenv_spec.rb | 4 ++-- spec/hatchet/poetry_spec.rb | 4 ++-- spec/hatchet/python_version_spec.rb | 35 ++++++++++++++++++++++++++--- spec/hatchet/uv_spec.rb | 4 ++-- 7 files changed, 60 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b5858554b..b9526b0f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Deprecated support for Python 3.10. ([#1972](https://github.com/heroku/heroku-buildpack-python/pull/1972)) ## [v320] - 2025-11-20 diff --git a/lib/python_version.sh b/lib/python_version.sh index 33bf394a8..d5e007b1c 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -692,8 +692,8 @@ function python_version::warn_if_deprecated_major_version() { output::warning <<-EOF Warning: Support for Python 3.9 is ending soon! - Python 3.9 will reach its upstream end-of-life in October 2025, - at which point it will no longer receive security updates: + Python 3.9 reached its upstream end-of-life on 31st October 2025, + and so no longer receives security updates: https://devguide.python.org/versions/#supported-versions As such, support for Python 3.9 will be removed from this @@ -702,6 +702,23 @@ function python_version::warn_if_deprecated_major_version() { Upgrade to a newer Python version as soon as possible, by changing the version in your ${version_origin} file. + For more information, see: + https://devcenter.heroku.com/articles/python-support#supported-python-versions + EOF + elif [[ "${requested_major_version}" == "3.10" ]]; then + output::warning <<-EOF + Warning: Support for Python 3.10 is deprecated! + + Python 3.10 will reach its upstream end-of-life in October 2026, + at which point it will no longer receive security updates: + https://devguide.python.org/versions/#supported-versions + + As such, support for Python 3.10 will be removed from this + buildpack on 6th January 2027. + + Upgrade to a newer Python version as soon as possible, by + changing the version in your ${version_origin} file. + For more information, see: https://devcenter.heroku.com/articles/python-support#supported-python-versions EOF diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 8dc68798a..e2c14616a 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -272,8 +272,8 @@ remote: remote: ! Warning: Support for Python 3.9 is ending soon! remote: ! - remote: ! Python 3.9 will reach its upstream end-of-life in October 2025, - remote: ! at which point it will no longer receive security updates: + remote: ! Python 3.9 reached its upstream end-of-life on 31st October 2025, + remote: ! and so no longer receives security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! remote: ! As such, support for Python 3.9 will be removed from this diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 220d92155..21fc26fc5 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -162,8 +162,8 @@ remote: remote: ! Warning: Support for Python 3.9 is ending soon! remote: ! - remote: ! Python 3.9 will reach its upstream end-of-life in October 2025, - remote: ! at which point it will no longer receive security updates: + remote: ! Python 3.9 reached its upstream end-of-life on 31st October 2025, + remote: ! and so no longer receives security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! remote: ! As such, support for Python 3.9 will be removed from this diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 7bbe29f5f..3db7c1d0f 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -260,8 +260,8 @@ remote: remote: ! Warning: Support for Python 3.9 is ending soon! remote: ! - remote: ! Python 3.9 will reach its upstream end-of-life in October 2025, - remote: ! at which point it will no longer receive security updates: + remote: ! Python 3.9 reached its upstream end-of-life on 31st October 2025, + remote: ! and so no longer receives security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! remote: ! As such, support for Python 3.9 will be removed from this diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 6652e3741..6b279b3a7 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -262,8 +262,8 @@ remote: remote: ! Warning: Support for Python 3.9 is ending soon! remote: ! - remote: ! Python 3.9 will reach its upstream end-of-life in October 2025, - remote: ! at which point it will no longer receive security updates: + remote: ! Python 3.9 reached its upstream end-of-life on 31st October 2025, + remote: ! and so no longer receives security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! remote: ! As such, support for Python 3.9 will be removed from this @@ -289,7 +289,36 @@ context 'when .python-version contains Python 3.10' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.10') } - it_behaves_like 'builds with the requested Python version', '3.10', LATEST_PYTHON_3_10 + it 'builds with Python 3.10 but shows a deprecation warning' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.10 specified in .python-version + remote: + remote: ! Warning: Support for Python 3.10 is deprecated! + remote: ! + remote: ! Python 3.10 will reach its upstream end-of-life in October 2026, + remote: ! at which point it will no longer receive security updates: + remote: ! https://devguide.python.org/versions/#supported-versions + remote: ! + remote: ! As such, support for Python 3.10 will be removed from this + remote: ! buildpack on 6th January 2027. + remote: ! + remote: ! Upgrade to a newer Python version as soon as possible, by + remote: ! changing the version in your .python-version file. + remote: ! + remote: ! For more information, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions + remote: + remote: -----> Installing Python #{LATEST_PYTHON_3_10} + remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing dependencies using 'pip install -r requirements.txt' + remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) + OUTPUT + expect(app.run('python -V')).to eq("Python #{LATEST_PYTHON_3_10}\n") + expect($CHILD_STATUS.exitstatus).to eq(0) + end + end end context 'when .python-version contains Python 3.11' do diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 5b32fccf5..1cd4121ee 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -262,8 +262,8 @@ remote: remote: ! Warning: Support for Python 3.9 is ending soon! remote: ! - remote: ! Python 3.9 will reach its upstream end-of-life in October 2025, - remote: ! at which point it will no longer receive security updates: + remote: ! Python 3.9 reached its upstream end-of-life on 31st October 2025, + remote: ! and so no longer receives security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! remote: ! As such, support for Python 3.9 will be removed from this From 03cc02eba635732734374334cf521f9ef51591f7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 21 Nov 2025 09:44:00 +0000 Subject: [PATCH 286/406] Bump uv from 0.9.9 to 0.9.11 (#1973) * Bump uv from 0.9.9 to 0.9.11 Bumps [uv](https://github.com/astral-sh/uv) from 0.9.9 to 0.9.11. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.9.9...0.9.11) --- updated-dependencies: - dependency-name: uv dependency-version: 0.9.11 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b9526b0f5..abcfff382 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Deprecated support for Python 3.10. ([#1972](https://github.com/heroku/heroku-buildpack-python/pull/1972)) +- Updated uv from 0.9.9 to 0.9.11. ([#1973](https://github.com/heroku/heroku-buildpack-python/pull/1973)) ## [v320] - 2025-11-20 diff --git a/requirements/uv.txt b/requirements/uv.txt index 99caef278..bb72abdf9 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.9.9 +uv==0.9.11 From d668dbe38ffbe1fb1a223e085e1209c469694c98 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 21 Nov 2025 11:17:16 +0000 Subject: [PATCH 287/406] Change the way pip version checks are disabled (#1974) Switches from the pip CLI arg `--disable-pip-version-check` to the env var `PIP_DISABLE_PIP_VERSION_CHECK=1`, since: - It ensures the unnecessary version check is disabled if the user calls pip outside the main buildpack install step. - The env var is what the CNB is already using. See: https://pip.pypa.io/en/stable/cli/pip/#cmdoption-disable-pip-version-check GUS-W-20289539. --- CHANGELOG.md | 1 + bin/compile | 2 +- lib/pip.sh | 18 ++++++++++++++++-- spec/fixtures/pip_basic/bin/compile | 5 ++--- spec/hatchet/ci_spec.rb | 3 +++ spec/hatchet/django_spec.rb | 2 ++ spec/hatchet/pip_spec.rb | 3 +++ spec/hatchet/profile_d_scripts_spec.rb | 1 + 8 files changed, 29 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index abcfff382..b2eb9cbab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ - Deprecated support for Python 3.10. ([#1972](https://github.com/heroku/heroku-buildpack-python/pull/1972)) - Updated uv from 0.9.9 to 0.9.11. ([#1973](https://github.com/heroku/heroku-buildpack-python/pull/1973)) +- Switched from using `--disable-pip-version-check` to the env var `PIP_DISABLE_PIP_VERSION_CHECK=1` when using pip. ([#1974](https://github.com/heroku/heroku-buildpack-python/pull/1974)) ## [v320] - 2025-11-20 diff --git a/bin/compile b/bin/compile index adbb6de2b..a758d6f36 100755 --- a/bin/compile +++ b/bin/compile @@ -166,7 +166,7 @@ python::install "${BUILD_DIR}" "${STACK}" "${python_full_version}" "${python_maj package_manager_install_start_time=$(build_data::current_unix_realtime) case "${package_manager}" in pip) - pip::install_pip "${python_home}" "${python_major_version}" + pip::install_pip "${python_home}" "${python_major_version}" "${EXPORT_PATH}" "${PROFILE_PATH}" ;; pipenv) pipenv::install_pipenv "${python_home}" "${python_major_version}" "${EXPORT_PATH}" "${PROFILE_PATH}" diff --git a/lib/pip.sh b/lib/pip.sh index 6db9eb3b6..f427d9198 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -11,6 +11,8 @@ WHEEL_VERSION=$(utils::get_requirement_version 'wheel') function pip::install_pip() { local python_home="${1}" local python_major_version="${2}" + local export_file="${3}" + local profile_d_file="${4}" build_data::set_string "pip_version" "${PIP_VERSION}" @@ -74,6 +76,20 @@ function pip::install_pip() { build_data::set_string "failure_reason" "install-package-manager::pip" exit 1 fi + + # Saves us having to pass `--disable-pip-version-check` each time pip is used. Note: We can't use + # this for the pip install usage above, since `--isolated` disables reading of `PIP_` env vars. + export PIP_DISABLE_PIP_VERSION_CHECK="1" + + # Set the same env vars in the environment used by later buildpacks. + cat >>"${export_file}" <<-EOF + export PIP_DISABLE_PIP_VERSION_CHECK="1" + EOF + + # And the environment used at app run-time. + cat >>"${profile_d_file}" <<-EOF + export PIP_DISABLE_PIP_VERSION_CHECK="1" + EOF } function pip::install_dependencies() { @@ -112,7 +128,6 @@ function pip::install_dependencies() { local install_log install_log=$(mktemp) - # TODO: Remove --disable-pip-version-check in favour of exporting PIP_DISABLE_PIP_VERSION_CHECK. # The sed usage is to reduce the verbosity of output lines like: # ...when using Python 3.10 and older: # "Requirement already satisfied: typing-extensions==4.12.2 in ./.heroku/python/lib/python3.10/site-packages (from -r requirements.txt (line 2)) (4.12.2)" @@ -121,7 +136,6 @@ function pip::install_dependencies() { # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { "${pip_install_command[@]}" \ - --disable-pip-version-check \ --exists-action=w \ --no-cache-dir \ --no-input \ diff --git a/spec/fixtures/pip_basic/bin/compile b/spec/fixtures/pip_basic/bin/compile index 9dc156690..7d1d552df 100755 --- a/spec/fixtures/pip_basic/bin/compile +++ b/spec/fixtures/pip_basic/bin/compile @@ -16,9 +16,8 @@ echo python -c 'import pprint, sys; pprint.pp(sys.path)' echo -# TODO: Remove --disable-pip-version-check in favour of exporting PIP_DISABLE_PIP_VERSION_CHECK -pip --version --disable-pip-version-check -pip list --disable-pip-version-check +pip --version +pip list echo python -c 'import typing_extensions; print(typing_extensions)' diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 02c466ffa..ce762e560 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -33,6 +33,7 @@ LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PIP_DISABLE_PIP_VERSION_CHECK=1 PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 -----> Saving cache @@ -41,6 +42,7 @@ LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PIP_DISABLE_PIP_VERSION_CHECK=1 PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true @@ -54,6 +56,7 @@ LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ + PIP_DISABLE_PIP_VERSION_CHECK=1 PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true diff --git a/spec/hatchet/django_spec.rb b/spec/hatchet/django_spec.rb index 29588d44a..ad3e67207 100644 --- a/spec/hatchet/django_spec.rb +++ b/spec/hatchet/django_spec.rb @@ -43,6 +43,7 @@ remote: 'LD_LIBRARY_PATH': '/app/.heroku/python/lib', remote: 'LIBRARY_PATH': '/app/.heroku/python/lib', remote: 'PATH': '/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', + remote: 'PIP_DISABLE_PIP_VERSION_CHECK': '1', remote: 'PKG_CONFIG_PATH': '/app/.heroku/python/lib/pkg-config', remote: 'PWD': '/tmp/build_\\w+', remote: 'PYTHONPATH': '\\.', @@ -84,6 +85,7 @@ remote: 'LD_LIBRARY_PATH': '/app/.heroku/python/lib', remote: 'LIBRARY_PATH': '/app/.heroku/python/lib', remote: 'PATH': '/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', + remote: 'PIP_DISABLE_PIP_VERSION_CHECK': '1', remote: 'PKG_CONFIG_PATH': '/invalid-path', remote: 'PWD': '/tmp/build_\\w+', remote: 'PYTHONPATH': '/invalid-path', diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index e2c14616a..6ce168caa 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -30,6 +30,7 @@ remote: LD_LIBRARY_PATH=/app/.heroku/python/lib remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PIP_DISABLE_PIP_VERSION_CHECK=1 remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config remote: PYTHONUNBUFFERED=1 remote: -----> Saving cache @@ -38,6 +39,7 @@ remote: LD_LIBRARY_PATH=/app/.heroku/python/lib remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PIP_DISABLE_PIP_VERSION_CHECK=1 remote: PYTHONHOME=/app/.heroku/python remote: PYTHONPATH=/app remote: PYTHONUNBUFFERED=true @@ -106,6 +108,7 @@ LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin + PIP_DISABLE_PIP_VERSION_CHECK=1 PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true diff --git a/spec/hatchet/profile_d_scripts_spec.rb b/spec/hatchet/profile_d_scripts_spec.rb index 6d9cd7633..da3f325c0 100644 --- a/spec/hatchet/profile_d_scripts_spec.rb +++ b/spec/hatchet/profile_d_scripts_spec.rb @@ -35,6 +35,7 @@ LD_LIBRARY_PATH=/app/.heroku/python/lib:/this-should-be-preserved LIBRARY_PATH=/app/.heroku/python/lib:/this-should-be-preserved PATH=/app/.heroku/python/bin:/this-should-be-preserved:/usr/local/bin:/usr/bin:/bin + PIP_DISABLE_PIP_VERSION_CHECK=1 PYTHONHOME=/app/.heroku/python PYTHONPATH=/this-should-be-preserved PYTHONUNBUFFERED=true From d8a984deb5c72a6a5d10c33c1b213e994462656e Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 21 Nov 2025 16:53:23 +0000 Subject: [PATCH 288/406] Add more messaging to the build log recommending uv (#1975) The package manager uv offers many benefits for users over the alternatives, such as it: - Supporting lockfiles (unlike pip) - Being much faster - Giving more helpful error messages - Being actively maintained by a full time team As such we want to encourage users who may not have heard of it (or be aware that the buildpack now supports it) to try it out - and so now mention uv at the env of the build log for successful builds. GUS-W-19124925. --- CHANGELOG.md | 1 + bin/compile | 14 ++++++++++++++ bin/detect | 5 +++++ lib/package_manager.sh | 6 ++++-- spec/hatchet/ci_spec.rb | 33 +++++++++++++++++++++++++++++++++ spec/hatchet/detect_spec.rb | 5 +++++ spec/hatchet/pip_spec.rb | 18 +++++++++++++++--- spec/hatchet/pipenv_spec.rb | 26 +++++++++++++++++++------- spec/hatchet/poetry_spec.rb | 19 +++++++++++++++---- 9 files changed, 111 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b2eb9cbab..1a967158e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ - Deprecated support for Python 3.10. ([#1972](https://github.com/heroku/heroku-buildpack-python/pull/1972)) - Updated uv from 0.9.9 to 0.9.11. ([#1973](https://github.com/heroku/heroku-buildpack-python/pull/1973)) +- Added more messaging to the build log recommending uv. ([#1975](https://github.com/heroku/heroku-buildpack-python/pull/1975)) - Switched from using `--disable-pip-version-check` to the env var `PIP_DISABLE_PIP_VERSION_CHECK=1` when using pip. ([#1974](https://github.com/heroku/heroku-buildpack-python/pull/1974)) ## [v320] - 2025-11-20 diff --git a/bin/compile b/bin/compile index a758d6f36..345548aa7 100755 --- a/bin/compile +++ b/bin/compile @@ -290,4 +290,18 @@ hooks::run_hook "post_compile" cache::save "${BUILD_DIR}" "${CACHE_DIR}" "${STACK}" "${python_full_version}" "${package_manager}" +if [[ "${package_manager}" != "uv" ]]; then + output::notice <<-EOF + Note: We recently added support for the package manager uv: + https://devcenter.heroku.com/changelog-items/3238 + + It's now our recommended Python package manager, since it + supports lockfiles, is faster, gives more helpful error + messages, and is actively maintained by a full-time team. + + If you haven't tried it yet, we suggest you take a look! + https://docs.astral.sh/uv/ + EOF +fi + build_data::set_duration "total_duration" "${compile_start_time}" diff --git a/bin/detect b/bin/detect index 9e3f1e753..86da2ddc0 100755 --- a/bin/detect +++ b/bin/detect @@ -86,6 +86,11 @@ If your app already has a package manager file, check that it: Otherwise, add a package manager file to your app. If your app has no dependencies, then create an empty 'requirements.txt' file. +If you aren't sure which package manager to use, we recommend +trying uv, since it supports lockfiles, is extremely fast, and +is actively maintained by a full-time team: +https://docs.astral.sh/uv/ + For help with using Python on Heroku, see: https://devcenter.heroku.com/articles/getting-started-with-python https://devcenter.heroku.com/articles/python-support diff --git a/lib/package_manager.sh b/lib/package_manager.sh index be5e79730..b9e8d1dbf 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -30,8 +30,10 @@ function package_manager::determine_package_manager() { delete your 'Pipfile' and then add either a 'requirements.txt', 'poetry.lock' or 'uv.lock' file. - Note: This error replaces the warning which was displayed in - build logs starting 12th November 2024. + If you aren't sure which package manager to use, we recommend + trying uv, since it supports lockfiles, is extremely fast, and + is actively maintained by a full-time team: + https://docs.astral.sh/uv/ EOF build_data::set_string "failure_reason" "package-manager::pipenv-missing-lockfile" exit 1 diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index ce762e560..1e6be74bc 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -37,6 +37,17 @@ PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 -----> Saving cache + + ! Note: We recently added support for the package manager uv: + ! https://devcenter.heroku.com/changelog-items/3238 + ! + ! It's now our recommended Python package manager, since it + ! supports lockfiles, is faster, gives more helpful error + ! messages, and is actively maintained by a full-time team. + ! + ! If you haven't tried it yet, we suggest you take a look! + ! https://docs.astral.sh/uv/ + -----> Inline app detected LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib @@ -117,6 +128,17 @@ PYTHONUNBUFFERED=1 VIRTUAL_ENV=/app/.heroku/python -----> Saving cache + + ! Note: We recently added support for the package manager uv: + ! https://devcenter.heroku.com/changelog-items/3238 + ! + ! It's now our recommended Python package manager, since it + ! supports lockfiles, is faster, gives more helpful error + ! messages, and is actively maintained by a full-time team. + ! + ! If you haven't tried it yet, we suggest you take a look! + ! https://docs.astral.sh/uv/ + -----> Inline app detected LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib @@ -205,6 +227,17 @@ POETRY_VIRTUALENVS_USE_POETRY_PYTHON=true PYTHONUNBUFFERED=1 -----> Saving cache + + ! Note: We recently added support for the package manager uv: + ! https://devcenter.heroku.com/changelog-items/3238 + ! + ! It's now our recommended Python package manager, since it + ! supports lockfiles, is faster, gives more helpful error + ! messages, and is actively maintained by a full-time team. + ! + ! If you haven't tried it yet, we suggest you take a look! + ! https://docs.astral.sh/uv/ + -----> Inline app detected LANG=en_US.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib diff --git a/spec/hatchet/detect_spec.rb b/spec/hatchet/detect_spec.rb index 04b83ba49..2a44089c2 100644 --- a/spec/hatchet/detect_spec.rb +++ b/spec/hatchet/detect_spec.rb @@ -38,6 +38,11 @@ remote: ! Otherwise, add a package manager file to your app. If your app has remote: ! no dependencies, then create an empty 'requirements.txt' file. remote: ! + remote: ! If you aren't sure which package manager to use, we recommend + remote: ! trying uv, since it supports lockfiles, is extremely fast, and + remote: ! is actively maintained by a full-time team: + remote: ! https://docs.astral.sh/uv/ + remote: ! remote: ! For help with using Python on Heroku, see: remote: ! https://devcenter.heroku.com/articles/getting-started-with-python remote: ! https://devcenter.heroku.com/articles/python-support diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 6ce168caa..a319f9da2 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -34,6 +34,17 @@ remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config remote: PYTHONUNBUFFERED=1 remote: -----> Saving cache + remote: + remote: ! Note: We recently added support for the package manager uv: + remote: ! https://devcenter.heroku.com/changelog-items/3238 + remote: ! + remote: ! It's now our recommended Python package manager, since it + remote: ! supports lockfiles, is faster, gives more helpful error + remote: ! messages, and is actively maintained by a full-time team. + remote: ! + remote: ! If you haven't tried it yet, we suggest you take a look! + remote: ! https://docs.astral.sh/uv/ + remote: remote: -----> Inline app detected remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/python/lib @@ -96,7 +107,6 @@ remote: -----> Running bin/post_compile hook remote: .+ remote: -----> Saving cache - remote: -----> Inline app detected REGEX # For historical reasons pip is made available at run-time too, unlike some of the other package managers. @@ -162,7 +172,7 @@ it 'rewrites .pth, .egg-link and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Running bin/post_compile hook remote: easy-install.pth:/app/.heroku/python/src/gunicorn remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py @@ -174,6 +184,7 @@ remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) remote: -----> Saving cache + .+ remote: -----> Inline app detected remote: easy-install.pth:/app/.heroku/python/src/gunicorn remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py @@ -202,7 +213,7 @@ # Test that the cached .pth files work correctly. app.commit! app.push! - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Running bin/post_compile hook remote: easy-install.pth:/app/.heroku/python/src/gunicorn remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py @@ -214,6 +225,7 @@ remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) remote: -----> Saving cache + .+ remote: -----> Inline app detected remote: easy-install.pth:/app/.heroku/python/src/gunicorn remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 21fc26fc5..d1f6621b3 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -32,6 +32,17 @@ remote: PYTHONUNBUFFERED=1 remote: VIRTUAL_ENV=/app/.heroku/python remote: -----> Saving cache + remote: + remote: ! Note: We recently added support for the package manager uv: + remote: ! https://devcenter.heroku.com/changelog-items/3238 + remote: ! + remote: ! It's now our recommended Python package manager, since it + remote: ! supports lockfiles, is faster, gives more helpful error + remote: ! messages, and is actively maintained by a full-time team. + remote: ! + remote: ! If you haven't tried it yet, we suggest you take a look! + remote: ! https://docs.astral.sh/uv/ + remote: remote: -----> Inline app detected remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/python/lib @@ -96,7 +107,6 @@ remote: -----> Running bin/post_compile hook remote: .+ remote: -----> Saving cache - remote: -----> Inline app detected REGEX # For historical reasons Pipenv is made available at run-time too, unlike some of the other package managers. @@ -141,7 +151,6 @@ remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Saving cache - remote: -----> Discovering process types REGEX end end @@ -303,8 +312,10 @@ remote: ! delete your 'Pipfile' and then add either a 'requirements.txt', remote: ! 'poetry.lock' or 'uv.lock' file. remote: ! - remote: ! Note: This error replaces the warning which was displayed in - remote: ! build logs starting 12th November 2024. + remote: ! If you aren't sure which package manager to use, we recommend + remote: ! trying uv, since it supports lockfiles, is extremely fast, and + remote: ! is actively maintained by a full-time team: + remote: ! https://docs.astral.sh/uv/ remote: remote: ! Push rejected, failed to compile Python app. REGEX @@ -465,7 +476,6 @@ remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Saving cache - remote: -----> Discovering process types REGEX end end @@ -477,7 +487,7 @@ it 'rewrites .pth and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Running bin/post_compile hook @@ -491,6 +501,7 @@ remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Saving cache + .+ remote: -----> Inline app detected remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} @@ -519,7 +530,7 @@ # Test that the cached .pth files work correctly. app.commit! app.push! - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Running bin/post_compile hook @@ -533,6 +544,7 @@ remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Saving cache + .+ remote: -----> Inline app detected remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 3db7c1d0f..54b2e25bd 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -35,6 +35,17 @@ remote: POETRY_VIRTUALENVS_USE_POETRY_PYTHON=true remote: PYTHONUNBUFFERED=1 remote: -----> Saving cache + remote: + remote: ! Note: We recently added support for the package manager uv: + remote: ! https://devcenter.heroku.com/changelog-items/3238 + remote: ! + remote: ! It's now our recommended Python package manager, since it + remote: ! supports lockfiles, is faster, gives more helpful error + remote: ! messages, and is actively maintained by a full-time team. + remote: ! + remote: ! If you haven't tried it yet, we suggest you take a look! + remote: ! https://docs.astral.sh/uv/ + remote: remote: -----> Inline app detected remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/python/lib @@ -98,7 +109,6 @@ remote: -----> Running bin/post_compile hook remote: .+ remote: -----> Saving cache - remote: -----> Inline app detected REGEX command = 'bin/print-env-vars.sh && if command -v poetry; then echo "Poetry unexpectedly found!" && exit 1; fi' @@ -145,7 +155,6 @@ remote: remote: - Installing typing-extensions (4.12.2) remote: -----> Saving cache - remote: -----> Discovering process types OUTPUT end end @@ -157,7 +166,7 @@ it 'rewrites .pth and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Installing dependencies using 'poetry sync --only main' remote: Installing dependencies from lock file remote: @@ -180,6 +189,7 @@ remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Saving cache + .+ remote: -----> Inline app detected remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} @@ -208,7 +218,7 @@ # Test that the cached .pth files work correctly. app.commit! app.push! - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Installing dependencies using 'poetry sync --only main' remote: Installing dependencies from lock file remote: @@ -230,6 +240,7 @@ remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Saving cache + .+ remote: -----> Inline app detected remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} From 5fec34c81b68841c7d8eaccc48c0b96bb10d5176 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Fri, 21 Nov 2025 19:23:15 +0000 Subject: [PATCH 289/406] Prepare release v321 (#1976) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1a967158e..1f983d072 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v321] - 2025-11-21 + - Deprecated support for Python 3.10. ([#1972](https://github.com/heroku/heroku-buildpack-python/pull/1972)) - Updated uv from 0.9.9 to 0.9.11. ([#1973](https://github.com/heroku/heroku-buildpack-python/pull/1973)) - Added more messaging to the build log recommending uv. ([#1975](https://github.com/heroku/heroku-buildpack-python/pull/1975)) @@ -1407,7 +1410,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v320...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v321...main +[v321]: https://github.com/heroku/heroku-buildpack-python/compare/v320...v321 [v320]: https://github.com/heroku/heroku-buildpack-python/compare/v319...v320 [v319]: https://github.com/heroku/heroku-buildpack-python/compare/v318...v319 [v318]: https://github.com/heroku/heroku-buildpack-python/compare/v317...v318 From 1baf44d5f3214950f87156978917b3d7c708a526 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 2 Dec 2025 10:55:57 +0000 Subject: [PATCH 290/406] Bump actions/checkout from 5 to 6 (#1978) Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/build_python_runtime.yml | 4 ++-- .github/workflows/check_changelog.yml | 2 +- .github/workflows/ci.yml | 6 +++--- .github/workflows/hatchet_app_cleaner.yml | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/.github/workflows/build_python_runtime.yml b/.github/workflows/build_python_runtime.yml index 11ff2f571..fa87c1fbf 100644 --- a/.github/workflows/build_python_runtime.yml +++ b/.github/workflows/build_python_runtime.yml @@ -42,7 +42,7 @@ jobs: STACK_VERSION: "22" steps: - name: Checkout - uses: actions/checkout@v5 + uses: actions/checkout@v6 - name: Build Docker image run: docker build --platform="linux/amd64" --pull --tag buildenv --build-arg=STACK_VERSION builds/ - name: Compile and package Python runtime @@ -67,7 +67,7 @@ jobs: STACK_VERSION: "24" steps: - name: Checkout - uses: actions/checkout@v5 + uses: actions/checkout@v6 - name: Build Docker image run: docker build --platform="linux/${{ matrix.arch }}" --pull --tag buildenv --build-arg=STACK_VERSION builds/ - name: Compile and package Python runtime diff --git a/.github/workflows/check_changelog.yml b/.github/workflows/check_changelog.yml index f4b8c100e..419c9fa6a 100644 --- a/.github/workflows/check_changelog.yml +++ b/.github/workflows/check_changelog.yml @@ -13,7 +13,7 @@ jobs: if: (!contains(github.event.pull_request.labels.*.name, 'skip changelog')) steps: - name: Checkout - uses: actions/checkout@v5 + uses: actions/checkout@v6 - name: Check that CHANGELOG is touched run: | git fetch origin ${{ github.base_ref }} --depth 1 && \ diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 51e167ef9..f32322a11 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -19,7 +19,7 @@ jobs: runs-on: ubuntu-24.04 steps: - name: Checkout - uses: actions/checkout@v5 + uses: actions/checkout@v6 - name: Install Ruby and dependencies uses: ruby/setup-ruby@v1 with: @@ -51,7 +51,7 @@ jobs: RSPEC_RETRY_RETRY_COUNT: 1 steps: - name: Checkout - uses: actions/checkout@v5 + uses: actions/checkout@v6 - name: Install Ruby and dependencies uses: ruby/setup-ruby@v1 with: @@ -67,7 +67,7 @@ jobs: runs-on: ubuntu-24.04 steps: - name: Checkout - uses: actions/checkout@v5 + uses: actions/checkout@v6 # These test both the local development `make run` workflow and that `bin/report` completes successfully # for both passing and failing builds (since `bin/report` can't easily be tested via Hatchet tests). - name: Run buildpack using default app fixture diff --git a/.github/workflows/hatchet_app_cleaner.yml b/.github/workflows/hatchet_app_cleaner.yml index 8cd69aa75..caf9388d9 100644 --- a/.github/workflows/hatchet_app_cleaner.yml +++ b/.github/workflows/hatchet_app_cleaner.yml @@ -19,7 +19,7 @@ jobs: HEROKU_DISABLE_AUTOUPDATE: 1 steps: - name: Checkout - uses: actions/checkout@v5 + uses: actions/checkout@v6 - name: Install Ruby and dependencies uses: ruby/setup-ruby@v1 with: From 2a761abc66e8fde11d8cc04ec5aee4e61ab53111 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 2 Dec 2025 11:16:46 +0000 Subject: [PATCH 291/406] Bump rubocop-rspec from 3.7.0 to 3.8.0 in the ruby-dependencies group (#1980) * Bump rubocop-rspec from 3.7.0 to 3.8.0 in the ruby-dependencies group Bumps the ruby-dependencies group with 1 update: [rubocop-rspec](https://github.com/rubocop/rubocop-rspec). Updates `rubocop-rspec` from 3.7.0 to 3.8.0 - [Release notes](https://github.com/rubocop/rubocop-rspec/releases) - [Changelog](https://github.com/rubocop/rubocop-rspec/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop-rspec/compare/v3.7.0...v3.8.0) --- updated-dependencies: - dependency-name: rubocop-rspec dependency-version: 3.8.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] * `bundle update` --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- Gemfile.lock | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 389747d96..5a97e9454 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -5,7 +5,7 @@ GEM base64 (0.3.0) diff-lcs (1.6.2) erubis (2.7.0) - excon (1.3.0) + excon (1.3.1) logger heroics (0.1.3) base64 @@ -20,12 +20,12 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.15.2) + json (2.16.0) language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) moneta (1.0.0) - multi_json (1.17.0) + multi_json (1.18.0) parallel (1.27.0) parallel_split_test (0.10.0) parallel (>= 0.5.13) @@ -62,19 +62,19 @@ GEM rubocop-ast (>= 1.47.1, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 4.0) - rubocop-ast (1.47.1) + rubocop-ast (1.48.0) parser (>= 3.3.7.2) prism (~> 1.4) - rubocop-rspec (3.7.0) + rubocop-rspec (3.8.0) lint_roller (~> 1.1) - rubocop (~> 1.72, >= 1.72.1) + rubocop (~> 1.81) ruby-progressbar (1.13.0) thor (1.4.0) threaded (0.0.4) unicode-display_width (3.2.0) unicode-emoji (~> 4.1) unicode-emoji (4.1.0) - webrick (1.9.1) + webrick (1.9.2) PLATFORMS ruby From 777d498bbbba7232087361110a4ebde8ba4b6182 Mon Sep 17 00:00:00 2001 From: Rune Soerensen Date: Tue, 2 Dec 2025 09:36:40 -0500 Subject: [PATCH 292/406] Use buildpack banner from `heroku/buildpacks` (#1981) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d8a273ef7..14518f23f 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -![python](https://cloud.githubusercontent.com/assets/51578/13712821/b68a42ce-e793-11e5-96b0-d8eb978137ba.png) +![python](https://raw.githubusercontent.com/heroku/buildpacks/refs/heads/main/assets/images/buildpack-banner-python.png) # Heroku Buildpack: Python From cd9e50579c784598aa5cb30444bcb244c6b6270c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 2 Dec 2025 17:37:56 +0000 Subject: [PATCH 293/406] Bump uv from 0.9.11 to 0.9.14 (#1979) * Bump uv from 0.9.11 to 0.9.14 Bumps [uv](https://github.com/astral-sh/uv) from 0.9.11 to 0.9.14. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.9.11...0.9.14) --- updated-dependencies: - dependency-name: uv dependency-version: 0.9.14 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f983d072..ffadeb6d9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.9.11 to 0.9.14. ([#1979](https://github.com/heroku/heroku-buildpack-python/pull/1979)) ## [v321] - 2025-11-21 diff --git a/requirements/uv.txt b/requirements/uv.txt index bb72abdf9..7ff1f7436 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.9.11 +uv==0.9.14 From e6b2065666310e5581f6bbab74e29fd934c83bdf Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 2 Dec 2025 18:08:33 +0000 Subject: [PATCH 294/406] Add support for Python 3.14.1 and 3.13.10 (#1982) Release announcement: https://blog.python.org/2025/12/python-3141-is-now-available.html Changelogs: https://docs.python.org/release/3.14.1/whatsnew/changelog.html#python-3-14-1-final https://docs.python.org/release/3.13.10/whatsnew/changelog.html#python-3-13-10-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/19867766099 https://github.com/heroku/heroku-buildpack-python/actions/runs/19867768866 GUS-W-17604371. GUS-W-20201648. --- CHANGELOG.md | 2 ++ lib/python_version.sh | 4 ++-- spec/spec_helper.rb | 4 ++-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ffadeb6d9..1fff387c0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## [Unreleased] +- The Python 3.14 version alias now resolves to Python 3.14.1. ([#1982](https://github.com/heroku/heroku-buildpack-python/pull/1982)) +- The Python 3.13 version alias now resolves to Python 3.13.10. ([#1982](https://github.com/heroku/heroku-buildpack-python/pull/1982)) - Updated uv from 0.9.11 to 0.9.14. ([#1979](https://github.com/heroku/heroku-buildpack-python/pull/1979)) ## [v321] - 2025-11-21 diff --git a/lib/python_version.sh b/lib/python_version.sh index d5e007b1c..a4544b1c2 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -8,8 +8,8 @@ LATEST_PYTHON_3_9="3.9.25" LATEST_PYTHON_3_10="3.10.19" LATEST_PYTHON_3_11="3.11.14" LATEST_PYTHON_3_12="3.12.12" -LATEST_PYTHON_3_13="3.13.9" -LATEST_PYTHON_3_14="3.14.0" +LATEST_PYTHON_3_13="3.13.10" +LATEST_PYTHON_3_14="3.14.1" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=9 NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=14 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index dd19c7fb3..b162130ae 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -14,8 +14,8 @@ LATEST_PYTHON_3_10 = '3.10.19' LATEST_PYTHON_3_11 = '3.11.14' LATEST_PYTHON_3_12 = '3.12.12' -LATEST_PYTHON_3_13 = '3.13.9' -LATEST_PYTHON_3_14 = '3.14.0' +LATEST_PYTHON_3_13 = '3.13.10' +LATEST_PYTHON_3_14 = '3.14.1' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_13 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From 209c77b28f66579592847469054e56c1452585b2 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 2 Dec 2025 18:18:34 +0000 Subject: [PATCH 295/406] Prepare release v322 (#1983) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1fff387c0..fbb01456f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v322] - 2025-12-02 + - The Python 3.14 version alias now resolves to Python 3.14.1. ([#1982](https://github.com/heroku/heroku-buildpack-python/pull/1982)) - The Python 3.13 version alias now resolves to Python 3.13.10. ([#1982](https://github.com/heroku/heroku-buildpack-python/pull/1982)) - Updated uv from 0.9.11 to 0.9.14. ([#1979](https://github.com/heroku/heroku-buildpack-python/pull/1979)) @@ -1413,7 +1416,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v321...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v322...main +[v322]: https://github.com/heroku/heroku-buildpack-python/compare/v321...v322 [v321]: https://github.com/heroku/heroku-buildpack-python/compare/v320...v321 [v320]: https://github.com/heroku/heroku-buildpack-python/compare/v319...v320 [v319]: https://github.com/heroku/heroku-buildpack-python/compare/v318...v319 From 2b1f8a0ef26a1ac4529d29bb891da6745eb60fda Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 4 Dec 2025 22:56:10 +0000 Subject: [PATCH 296/406] Use Python 3.14 as the default Python version for new apps (#1984) Previously if a new app did not specify a Python version (e.g. via a `.python-version` file), then the buildpack would default to the latest Python 3.13 release. Now the buildpack will use the latest Python 3.14 release. This does not affect: * apps that specify an explicit Python version * existing apps that have already had a build using the previous default version (since that version is remembered from one build to the next, via the "version pinning" / "sticky versions" feature). Use-cases involving new apps that aren't compatible with Python 3.14 will now need to specify Python 3.13 explicitly: https://devcenter.heroku.com/articles/python-support#specify-a-python-version However most apps should be compatible, since Python 3.14 was released a couple of months ago (2025-10-07), does not contain significant breaking changes over Python 3.13, and has been happily used by the python-getting-started project since 2025-10-10. See: https://docs.python.org/3.14/whatsnew/3.14.html GUS-W-17595747. --- CHANGELOG.md | 1 + README.md | 6 +- lib/pip.sh | 4 +- lib/python_version.sh | 2 +- spec/fixtures/ci_pip/.python-version | 2 +- spec/fixtures/ci_pip/requirements.txt | 2 +- spec/fixtures/ci_pipenv/.python-version | 2 +- spec/fixtures/ci_pipenv/Pipfile.lock | 44 +++++++----- spec/fixtures/ci_poetry/.python-version | 2 +- spec/fixtures/ci_poetry/poetry.lock | 72 +++++++++++-------- spec/fixtures/ci_poetry/pyproject.toml | 2 +- spec/fixtures/ci_uv/.python-version | 2 +- spec/fixtures/ci_uv/pyproject.toml | 2 +- spec/fixtures/ci_uv/uv.lock | 38 ++++++---- .../multiple_package_managers/.python-version | 2 +- spec/fixtures/pip_basic/.python-version | 2 +- spec/fixtures/pip_basic/requirements/prod.txt | 2 +- .../pip_oldest_python/requirements.txt | 2 +- spec/fixtures/pipenv_basic/Pipfile | 2 +- spec/fixtures/pipenv_basic/Pipfile.lock | 18 ++--- spec/fixtures/pipenv_editable/Pipfile.lock | 8 +-- spec/fixtures/pipenv_editable/pyproject.toml | 2 +- .../.python-version | 2 +- .../pipenv_lockfile_out_of_sync/Pipfile | 4 +- .../pipenv_lockfile_out_of_sync/Pipfile.lock | 14 +--- .../Pipfile.lock | 8 +-- spec/fixtures/pipenv_no_lockfile/Pipfile | 2 +- .../pipenv_python_full_version/Pipfile.lock | 8 +-- .../Pipfile.lock | 8 +-- .../Pipfile.lock | 8 +-- .../pipenv_python_version_eol/Pipfile.lock | 6 +- .../Pipfile.lock | 8 +-- .../Pipfile.lock | 8 +-- spec/fixtures/poetry_basic/.python-version | 2 +- spec/fixtures/poetry_basic/poetry.lock | 72 +++++++++++-------- spec/fixtures/poetry_basic/pyproject.toml | 2 +- spec/fixtures/poetry_editable/poetry.lock | 16 ++--- spec/fixtures/poetry_editable/pyproject.toml | 2 +- .../poetry_lockfile_out_of_sync/poetry.lock | 8 +-- .../pyproject.toml | 2 +- .../poetry.lock | 12 ++-- .../fixtures/poetry_oldest_python/poetry.lock | 10 +-- spec/fixtures/python_3.10/requirements.txt | 2 +- spec/fixtures/python_3.11/requirements.txt | 2 +- spec/fixtures/python_3.12/requirements.txt | 2 +- spec/fixtures/python_3.13/requirements.txt | 2 +- spec/fixtures/python_3.14/requirements.txt | 2 +- spec/fixtures/python_3.9/requirements.txt | 2 +- .../python_in_app_source/.python-version | 2 +- .../requirements.txt | 2 +- spec/fixtures/setup_py_only/.python-version | 2 +- spec/fixtures/uv_basic/.python-version | 2 +- spec/fixtures/uv_basic/pyproject.toml | 2 +- spec/fixtures/uv_basic/uv.lock | 38 ++++++---- spec/fixtures/uv_editable/.python-version | 2 +- spec/fixtures/uv_editable/pyproject.toml | 2 +- spec/fixtures/uv_editable/uv.lock | 6 +- .../uv_lockfile_out_of_sync/.python-version | 2 +- .../uv_lockfile_out_of_sync/pyproject.toml | 2 +- spec/fixtures/uv_lockfile_out_of_sync/uv.lock | 4 +- .../uv_mismatched_python_version/uv.lock | 2 +- .../uv_no_python_version_file/uv.lock | 2 +- spec/fixtures/uv_oldest_python/uv.lock | 6 +- spec/fixtures/uv_runtime_txt/uv.lock | 4 +- .../venv_in_app_source/.python-version | 2 +- spec/hatchet/checks_spec.rb | 2 +- spec/hatchet/ci_spec.rb | 10 +-- spec/hatchet/django_spec.rb | 8 +-- spec/hatchet/package_manager_spec.rb | 4 +- spec/hatchet/pip_spec.rb | 48 ++++++------- spec/hatchet/pipenv_spec.rb | 30 ++++---- spec/hatchet/poetry_spec.rb | 40 +++++------ spec/hatchet/python_version_spec.rb | 10 +-- spec/hatchet/stack_spec.rb | 10 +-- spec/hatchet/uv_spec.rb | 44 ++++++------ spec/spec_helper.rb | 2 +- 76 files changed, 386 insertions(+), 337 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fbb01456f..cd661b357 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Changed the default Python version for new apps from 3.13 to 3.14. ([#1984](https://github.com/heroku/heroku-buildpack-python/pull/1984)) ## [v322] - 2025-12-02 diff --git a/README.md b/README.md index 14518f23f..6463c686e 100644 --- a/README.md +++ b/README.md @@ -25,9 +25,9 @@ When using the package manager [uv](https://docs.astral.sh/uv/) a `.python-versi We recommend that you specify a Python version for your app rather than relying on the buildpack's default Python version. -For example, to request the latest patch release of Python 3.13, create a `.python-version` file in +For example, to request the latest patch release of Python 3.14, create a `.python-version` file in the root directory of your app containing: -`3.13` +`3.14` We strongly recommend that you use the major version form instead of pinning to an exact version, since it will allow your app to receive Python security updates. @@ -43,7 +43,7 @@ If none of those are found, the buildpack will use a default Python version for build of an app, and then subsequent builds of that app will be pinned to that version unless the build cache is cleared or you request a different version. -The current default Python version is: 3.13 +The current default Python version is: 3.14 The supported Python versions are: diff --git a/lib/pip.sh b/lib/pip.sh index f427d9198..829e34db0 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -130,9 +130,9 @@ function pip::install_dependencies() { # The sed usage is to reduce the verbosity of output lines like: # ...when using Python 3.10 and older: - # "Requirement already satisfied: typing-extensions==4.12.2 in ./.heroku/python/lib/python3.10/site-packages (from -r requirements.txt (line 2)) (4.12.2)" + # "Requirement already satisfied: typing-extensions==4.15.0 in ./.heroku/python/lib/python3.10/site-packages (from -r requirements.txt (line 2)) (4.15.0)" # ...when using Python 3.11+: - # "Requirement already satisfied: typing-extensions==4.12.2 in /app/.heroku/python/lib/python3.13/site-packages (from -r requirements.txt (line 5)) (4.12.2)" + # "Requirement already satisfied: typing-extensions==4.15.0 in /app/.heroku/python/lib/python3.14/site-packages (from -r requirements.txt (line 5)) (4.15.0)" # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { "${pip_install_command[@]}" \ diff --git a/lib/python_version.sh b/lib/python_version.sh index a4544b1c2..3eb24691c 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -14,7 +14,7 @@ LATEST_PYTHON_3_14="3.14.1" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=9 NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=14 -DEFAULT_PYTHON_FULL_VERSION="${LATEST_PYTHON_3_13}" +DEFAULT_PYTHON_FULL_VERSION="${LATEST_PYTHON_3_14}" DEFAULT_PYTHON_MAJOR_VERSION="${DEFAULT_PYTHON_FULL_VERSION%.*}" # Integer with no redundant leading zeros. diff --git a/spec/fixtures/ci_pip/.python-version b/spec/fixtures/ci_pip/.python-version index 24ee5b1be..6324d401a 100644 --- a/spec/fixtures/ci_pip/.python-version +++ b/spec/fixtures/ci_pip/.python-version @@ -1 +1 @@ -3.13 +3.14 diff --git a/spec/fixtures/ci_pip/requirements.txt b/spec/fixtures/ci_pip/requirements.txt index eec3a2223..f6e499a84 100644 --- a/spec/fixtures/ci_pip/requirements.txt +++ b/spec/fixtures/ci_pip/requirements.txt @@ -1,2 +1,2 @@ # This package has been picked since it has no dependencies and is small/fast to install. -typing-extensions==4.12.2 +typing-extensions==4.15.0 diff --git a/spec/fixtures/ci_pipenv/.python-version b/spec/fixtures/ci_pipenv/.python-version index 24ee5b1be..6324d401a 100644 --- a/spec/fixtures/ci_pipenv/.python-version +++ b/spec/fixtures/ci_pipenv/.python-version @@ -1 +1 @@ -3.13 +3.14 diff --git a/spec/fixtures/ci_pipenv/Pipfile.lock b/spec/fixtures/ci_pipenv/Pipfile.lock index a7d99ba86..f86d87a9d 100644 --- a/spec/fixtures/ci_pipenv/Pipfile.lock +++ b/spec/fixtures/ci_pipenv/Pipfile.lock @@ -16,47 +16,55 @@ "default": { "typing-extensions": { "hashes": [ - "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", - "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" + "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", + "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548" ], "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==4.12.2" + "markers": "python_version >= '3.9'", + "version": "==4.15.0" } }, "develop": { "iniconfig": { "hashes": [ - "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3", - "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374" + "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", + "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12" ], - "markers": "python_version >= '3.7'", - "version": "==2.0.0" + "markers": "python_version >= '3.10'", + "version": "==2.3.0" }, "packaging": { "hashes": [ - "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759", - "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f" + "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484", + "sha256:d443872c98d677bf60f6a1f2f8c1cb748e8fe762d2bf9d3148b5599295b0fc4f" ], "markers": "python_version >= '3.8'", - "version": "==24.2" + "version": "==25.0" }, "pluggy": { "hashes": [ - "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1", - "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669" + "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", + "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746" + ], + "markers": "python_version >= '3.9'", + "version": "==1.6.0" + }, + "pygments": { + "hashes": [ + "sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887", + "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b" ], "markers": "python_version >= '3.8'", - "version": "==1.5.0" + "version": "==2.19.2" }, "pytest": { "hashes": [ - "sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6", - "sha256:965370d062bce11e73868e0335abac31b4d3de0e82f4007408d242b4f8610761" + "sha256:3e9c069ea73583e255c3b21cf46b8d3c56f6e3a1a8f6da94ccb0fcf57b9d73c8", + "sha256:67be0030d194df2dfa7b556f2e56fb3c3315bd5c8822c6951162b92b32ce7dad" ], "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==8.3.4" + "markers": "python_version >= '3.10'", + "version": "==9.0.1" } } } diff --git a/spec/fixtures/ci_poetry/.python-version b/spec/fixtures/ci_poetry/.python-version index 24ee5b1be..6324d401a 100644 --- a/spec/fixtures/ci_poetry/.python-version +++ b/spec/fixtures/ci_poetry/.python-version @@ -1 +1 @@ -3.13 +3.14 diff --git a/spec/fixtures/ci_poetry/poetry.lock b/spec/fixtures/ci_poetry/poetry.lock index dc989760e..51cc64181 100644 --- a/spec/fixtures/ci_poetry/poetry.lock +++ b/spec/fixtures/ci_poetry/poetry.lock @@ -1,4 +1,4 @@ -# This file is automatically @generated by Poetry 2.2.0 and should not be changed by hand. +# This file is automatically @generated by Poetry 2.2.1 and should not be changed by hand. [[package]] name = "colorama" @@ -15,78 +15,94 @@ files = [ [[package]] name = "iniconfig" -version = "2.0.0" +version = "2.3.0" description = "brain-dead simple config-ini parsing" optional = false -python-versions = ">=3.7" +python-versions = ">=3.10" groups = ["dev"] files = [ - {file = "iniconfig-2.0.0-py3-none-any.whl", hash = "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374"}, - {file = "iniconfig-2.0.0.tar.gz", hash = "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3"}, + {file = "iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12"}, + {file = "iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730"}, ] [[package]] name = "packaging" -version = "24.2" +version = "25.0" description = "Core utilities for Python packages" optional = false python-versions = ">=3.8" groups = ["dev"] files = [ - {file = "packaging-24.2-py3-none-any.whl", hash = "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759"}, - {file = "packaging-24.2.tar.gz", hash = "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f"}, + {file = "packaging-25.0-py3-none-any.whl", hash = "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484"}, + {file = "packaging-25.0.tar.gz", hash = "sha256:d443872c98d677bf60f6a1f2f8c1cb748e8fe762d2bf9d3148b5599295b0fc4f"}, ] [[package]] name = "pluggy" -version = "1.5.0" +version = "1.6.0" description = "plugin and hook calling mechanisms for python" optional = false -python-versions = ">=3.8" +python-versions = ">=3.9" groups = ["dev"] files = [ - {file = "pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669"}, - {file = "pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1"}, + {file = "pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746"}, + {file = "pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3"}, ] [package.extras] dev = ["pre-commit", "tox"] -testing = ["pytest", "pytest-benchmark"] +testing = ["coverage", "pytest", "pytest-benchmark"] + +[[package]] +name = "pygments" +version = "2.19.2" +description = "Pygments is a syntax highlighting package written in Python." +optional = false +python-versions = ">=3.8" +groups = ["dev"] +files = [ + {file = "pygments-2.19.2-py3-none-any.whl", hash = "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b"}, + {file = "pygments-2.19.2.tar.gz", hash = "sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887"}, +] + +[package.extras] +windows-terminal = ["colorama (>=0.4.6)"] [[package]] name = "pytest" -version = "8.3.4" +version = "9.0.1" description = "pytest: simple powerful testing with Python" optional = false -python-versions = ">=3.8" +python-versions = ">=3.10" groups = ["dev"] files = [ - {file = "pytest-8.3.4-py3-none-any.whl", hash = "sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6"}, - {file = "pytest-8.3.4.tar.gz", hash = "sha256:965370d062bce11e73868e0335abac31b4d3de0e82f4007408d242b4f8610761"}, + {file = "pytest-9.0.1-py3-none-any.whl", hash = "sha256:67be0030d194df2dfa7b556f2e56fb3c3315bd5c8822c6951162b92b32ce7dad"}, + {file = "pytest-9.0.1.tar.gz", hash = "sha256:3e9c069ea73583e255c3b21cf46b8d3c56f6e3a1a8f6da94ccb0fcf57b9d73c8"}, ] [package.dependencies] -colorama = {version = "*", markers = "sys_platform == \"win32\""} -iniconfig = "*" -packaging = "*" +colorama = {version = ">=0.4", markers = "sys_platform == \"win32\""} +iniconfig = ">=1.0.1" +packaging = ">=22" pluggy = ">=1.5,<2" +pygments = ">=2.7.2" [package.extras] -dev = ["argcomplete", "attrs (>=19.2)", "hypothesis (>=3.56)", "mock", "pygments (>=2.7.2)", "requests", "setuptools", "xmlschema"] +dev = ["argcomplete", "attrs (>=19.2)", "hypothesis (>=3.56)", "mock", "requests", "setuptools", "xmlschema"] [[package]] name = "typing-extensions" -version = "4.12.2" -description = "Backported and Experimental Type Hints for Python 3.8+" +version = "4.15.0" +description = "Backported and Experimental Type Hints for Python 3.9+" optional = false -python-versions = ">=3.8" +python-versions = ">=3.9" groups = ["main"] files = [ - {file = "typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d"}, - {file = "typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"}, + {file = "typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548"}, + {file = "typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466"}, ] [metadata] lock-version = "2.1" -python-versions = ">=3.13" -content-hash = "029e0ee94dae38c88285bb0a557d4b77d92e76d30aa8ca91d17ac51fcf62b7f1" +python-versions = ">=3.14" +content-hash = "a0f8d6facd04ce28abc860e56bb4af6e1acac71a1f2a93e4e6ed4e13a299e055" diff --git a/spec/fixtures/ci_poetry/pyproject.toml b/spec/fixtures/ci_poetry/pyproject.toml index 99fed51a1..161499343 100644 --- a/spec/fixtures/ci_poetry/pyproject.toml +++ b/spec/fixtures/ci_poetry/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "ci-poetry" version = "0.0.0" -requires-python = ">=3.13" +requires-python = ">=3.14" dependencies = [ "typing-extensions", ] diff --git a/spec/fixtures/ci_uv/.python-version b/spec/fixtures/ci_uv/.python-version index 24ee5b1be..6324d401a 100644 --- a/spec/fixtures/ci_uv/.python-version +++ b/spec/fixtures/ci_uv/.python-version @@ -1 +1 @@ -3.13 +3.14 diff --git a/spec/fixtures/ci_uv/pyproject.toml b/spec/fixtures/ci_uv/pyproject.toml index 9be1eb818..443befffc 100644 --- a/spec/fixtures/ci_uv/pyproject.toml +++ b/spec/fixtures/ci_uv/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "ci-uv" version = "0.0.0" -requires-python = ">=3.13" +requires-python = ">=3.14" dependencies = [ "typing-extensions", ] diff --git a/spec/fixtures/ci_uv/uv.lock b/spec/fixtures/ci_uv/uv.lock index ffd69a888..da352f67b 100644 --- a/spec/fixtures/ci_uv/uv.lock +++ b/spec/fixtures/ci_uv/uv.lock @@ -1,6 +1,6 @@ version = 1 -revision = 2 -requires-python = ">=3.13" +revision = 3 +requires-python = ">=3.14" [[package]] name = "ci-uv" @@ -32,11 +32,11 @@ wheels = [ [[package]] name = "iniconfig" -version = "2.1.0" +version = "2.3.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/f2/97/ebf4da567aa6827c909642694d71c9fcf53e5b504f2d96afea02718862f3/iniconfig-2.1.0.tar.gz", hash = "sha256:3abbd2e30b36733fee78f9c7f7308f2d0050e88f0087fd25c2645f63c773e1c7", size = 4793, upload-time = "2025-03-19T20:09:59.721Z" } +sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/2c/e1/e6716421ea10d38022b952c159d5161ca1193197fb744506875fbb87ea7b/iniconfig-2.1.0-py3-none-any.whl", hash = "sha256:9deba5723312380e77435581c6bf4935c94cbfab9b1ed33ef8d238ea168eb760", size = 6050, upload-time = "2025-03-19T20:10:01.071Z" }, + { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, ] [[package]] @@ -50,33 +50,43 @@ wheels = [ [[package]] name = "pluggy" -version = "1.5.0" +version = "1.6.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/96/2d/02d4312c973c6050a18b314a5ad0b3210edb65a906f868e31c111dede4a6/pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1", size = 67955, upload-time = "2024-04-20T21:34:42.531Z" } +sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/88/5f/e351af9a41f866ac3f1fac4ca0613908d9a41741cfcf2228f4ad853b697d/pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669", size = 20556, upload-time = "2024-04-20T21:34:40.434Z" }, + { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, +] + +[[package]] +name = "pygments" +version = "2.19.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b0/77/a5b8c569bf593b0140bde72ea885a803b82086995367bf2037de0159d924/pygments-2.19.2.tar.gz", hash = "sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887", size = 4968631, upload-time = "2025-06-21T13:39:12.283Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c7/21/705964c7812476f378728bdf590ca4b771ec72385c533964653c68e86bdc/pygments-2.19.2-py3-none-any.whl", hash = "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b", size = 1225217, upload-time = "2025-06-21T13:39:07.939Z" }, ] [[package]] name = "pytest" -version = "8.3.5" +version = "9.0.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "colorama", marker = "sys_platform == 'win32'" }, { name = "iniconfig" }, { name = "packaging" }, { name = "pluggy" }, + { name = "pygments" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/ae/3c/c9d525a414d506893f0cd8a8d0de7706446213181570cdbd766691164e40/pytest-8.3.5.tar.gz", hash = "sha256:f4efe70cc14e511565ac476b57c279e12a855b11f48f212af1080ef2263d3845", size = 1450891, upload-time = "2025-03-02T12:54:54.503Z" } +sdist = { url = "https://files.pythonhosted.org/packages/07/56/f013048ac4bc4c1d9be45afd4ab209ea62822fb1598f40687e6bf45dcea4/pytest-9.0.1.tar.gz", hash = "sha256:3e9c069ea73583e255c3b21cf46b8d3c56f6e3a1a8f6da94ccb0fcf57b9d73c8", size = 1564125, upload-time = "2025-11-12T13:05:09.333Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/30/3d/64ad57c803f1fa1e963a7946b6e0fea4a70df53c1a7fed304586539c2bac/pytest-8.3.5-py3-none-any.whl", hash = "sha256:c69214aa47deac29fad6c2a4f590b9c4a9fdb16a403176fe154b79c0b4d4d820", size = 343634, upload-time = "2025-03-02T12:54:52.069Z" }, + { url = "https://files.pythonhosted.org/packages/0b/8b/6300fb80f858cda1c51ffa17075df5d846757081d11ab4aa35cef9e6258b/pytest-9.0.1-py3-none-any.whl", hash = "sha256:67be0030d194df2dfa7b556f2e56fb3c3315bd5c8822c6951162b92b32ce7dad", size = 373668, upload-time = "2025-11-12T13:05:07.379Z" }, ] [[package]] name = "typing-extensions" -version = "4.13.2" +version = "4.15.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/f6/37/23083fcd6e35492953e8d2aaaa68b860eb422b34627b13f2ce3eb6106061/typing_extensions-4.13.2.tar.gz", hash = "sha256:e6c81219bd689f51865d9e372991c540bda33a0379d5573cddb9a3a23f7caaef", size = 106967, upload-time = "2025-04-10T14:19:05.416Z" } +sdist = { url = "https://files.pythonhosted.org/packages/72/94/1a15dd82efb362ac84269196e94cf00f187f7ed21c242792a923cdb1c61f/typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", size = 109391, upload-time = "2025-08-25T13:49:26.313Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/8b/54/b1ae86c0973cc6f0210b53d508ca3641fb6d0c56823f288d108bc7ab3cc8/typing_extensions-4.13.2-py3-none-any.whl", hash = "sha256:a439e7c04b49fec3e5d3e2beaa21755cadbbdc391694e28ccdd36ca4a1408f8c", size = 45806, upload-time = "2025-04-10T14:19:03.967Z" }, + { url = "https://files.pythonhosted.org/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548", size = 44614, upload-time = "2025-08-25T13:49:24.86Z" }, ] diff --git a/spec/fixtures/multiple_package_managers/.python-version b/spec/fixtures/multiple_package_managers/.python-version index 24ee5b1be..6324d401a 100644 --- a/spec/fixtures/multiple_package_managers/.python-version +++ b/spec/fixtures/multiple_package_managers/.python-version @@ -1 +1 @@ -3.13 +3.14 diff --git a/spec/fixtures/pip_basic/.python-version b/spec/fixtures/pip_basic/.python-version index 24ee5b1be..6324d401a 100644 --- a/spec/fixtures/pip_basic/.python-version +++ b/spec/fixtures/pip_basic/.python-version @@ -1 +1 @@ -3.13 +3.14 diff --git a/spec/fixtures/pip_basic/requirements/prod.txt b/spec/fixtures/pip_basic/requirements/prod.txt index 8a7dafe83..728919fa5 100644 --- a/spec/fixtures/pip_basic/requirements/prod.txt +++ b/spec/fixtures/pip_basic/requirements/prod.txt @@ -2,4 +2,4 @@ # in order to test that symlinked requirements files work. # This package has been picked since it has no dependencies and is small/fast to install. -typing-extensions==4.12.2 +typing-extensions==4.15.0 diff --git a/spec/fixtures/pip_oldest_python/requirements.txt b/spec/fixtures/pip_oldest_python/requirements.txt index 1d83d0de9..f6e499a84 100644 --- a/spec/fixtures/pip_oldest_python/requirements.txt +++ b/spec/fixtures/pip_oldest_python/requirements.txt @@ -1,2 +1,2 @@ # This package has been picked since it has no dependencies and is small/fast to install. -typing-extensions==4.14.1 +typing-extensions==4.15.0 diff --git a/spec/fixtures/pipenv_basic/Pipfile b/spec/fixtures/pipenv_basic/Pipfile index 0aee61f1c..8c06ae360 100644 --- a/spec/fixtures/pipenv_basic/Pipfile +++ b/spec/fixtures/pipenv_basic/Pipfile @@ -10,4 +10,4 @@ typing-extensions = "*" [dev-packages] [requires] -python_version = "3.13" +python_version = "3.14" diff --git a/spec/fixtures/pipenv_basic/Pipfile.lock b/spec/fixtures/pipenv_basic/Pipfile.lock index 1236083b0..223afa115 100644 --- a/spec/fixtures/pipenv_basic/Pipfile.lock +++ b/spec/fixtures/pipenv_basic/Pipfile.lock @@ -1,11 +1,11 @@ { "_meta": { "hash": { - "sha256": "64d6057c60457235901a48fab172d14e37f8cde498b38d7b4b643882e65c215a" + "sha256": "8f362a119150f17832636f824f9f334a4d5580c6713519ed0865b89c4cc4e845" }, "pipfile-spec": 6, "requires": { - "python_version": "3.13" + "python_version": "3.14" }, "sources": [ { @@ -18,21 +18,21 @@ "default": { "certifi": { "hashes": [ - "sha256:6b31f564a415d79ee77df69d757bb49a5bb53bd9f756cbbe24394ffd6fc1f4b2", - "sha256:8ea99dbdfaaf2ba2f9bac77b9249ef62ec5218e7c2b2e903378ed5fccf765995" + "sha256:97de8790030bbd5c2d96b7ec782fc2f7820ef8dba6db909ccf95449f2d062d4b", + "sha256:d8ab5478f2ecd78af242878415affce761ca6bc54a22a27e026d7c25357c3316" ], "index": "pypi", "markers": "python_version >= '3.7'", - "version": "==2025.7.14" + "version": "==2025.11.12" }, "typing-extensions": { "hashes": [ - "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", - "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" + "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", + "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548" ], "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==4.12.2" + "markers": "python_version >= '3.9'", + "version": "==4.15.0" } }, "develop": {} diff --git a/spec/fixtures/pipenv_editable/Pipfile.lock b/spec/fixtures/pipenv_editable/Pipfile.lock index 8c4b604a3..1ebf46629 100644 --- a/spec/fixtures/pipenv_editable/Pipfile.lock +++ b/spec/fixtures/pipenv_editable/Pipfile.lock @@ -18,7 +18,7 @@ "editable": true, "git": "git+https://github.com/benoitc/gunicorn", "markers": "python_version >= '3.7'", - "ref": "bacbf8aa5152b94e44aa5d2a94aeaf0318a85248" + "ref": "56b5ad87f8d72a674145c273ed8f547513c2b409" }, "local-package-pyproject-toml": { "editable": true, @@ -30,11 +30,11 @@ }, "packaging": { "hashes": [ - "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759", - "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f" + "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484", + "sha256:d443872c98d677bf60f6a1f2f8c1cb748e8fe762d2bf9d3148b5599295b0fc4f" ], "markers": "python_version >= '3.8'", - "version": "==24.2" + "version": "==25.0" }, "pipenv-editable": { "editable": true, diff --git a/spec/fixtures/pipenv_editable/pyproject.toml b/spec/fixtures/pipenv_editable/pyproject.toml index f6dd79f06..934632a78 100644 --- a/spec/fixtures/pipenv_editable/pyproject.toml +++ b/spec/fixtures/pipenv_editable/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "pipenv-editable" version = "0.0.0" -requires-python = ">=3.13" +requires-python = ">=3.14" [project.scripts] pipenv-editable = "pipenv_editable:main" diff --git a/spec/fixtures/pipenv_lockfile_out_of_sync/.python-version b/spec/fixtures/pipenv_lockfile_out_of_sync/.python-version index 24ee5b1be..6324d401a 100644 --- a/spec/fixtures/pipenv_lockfile_out_of_sync/.python-version +++ b/spec/fixtures/pipenv_lockfile_out_of_sync/.python-version @@ -1 +1 @@ -3.13 +3.14 diff --git a/spec/fixtures/pipenv_lockfile_out_of_sync/Pipfile b/spec/fixtures/pipenv_lockfile_out_of_sync/Pipfile index 73f50fc86..c8c201a39 100644 --- a/spec/fixtures/pipenv_lockfile_out_of_sync/Pipfile +++ b/spec/fixtures/pipenv_lockfile_out_of_sync/Pipfile @@ -4,9 +4,7 @@ verify_ssl = true name = "pypi" [packages] +# This dependency isn't in the lockfile. urllib3 = "*" [dev-packages] - -[requires] -python_version = "3.13" diff --git a/spec/fixtures/pipenv_lockfile_out_of_sync/Pipfile.lock b/spec/fixtures/pipenv_lockfile_out_of_sync/Pipfile.lock index 4c426c2ae..d2c2cba4c 100644 --- a/spec/fixtures/pipenv_lockfile_out_of_sync/Pipfile.lock +++ b/spec/fixtures/pipenv_lockfile_out_of_sync/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "a95c318e4395270fbd1e0f52dd8f12185db4c3258d4e03dd80de54b7c7aad8b1" + "sha256": "ebffa69a1fa192d1cef7cb42ad79231ca976565c5ce371a70160b3048d3cbc06" }, "pipfile-spec": 6, "requires": {}, @@ -13,16 +13,6 @@ } ] }, - "default": { - "urllib3": { - "hashes": [ - "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", - "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" - ], - "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==2.2.1" - } - }, + "default": {}, "develop": {} } diff --git a/spec/fixtures/pipenv_mismatched_python_version/Pipfile.lock b/spec/fixtures/pipenv_mismatched_python_version/Pipfile.lock index 6fdc90f5c..517924633 100644 --- a/spec/fixtures/pipenv_mismatched_python_version/Pipfile.lock +++ b/spec/fixtures/pipenv_mismatched_python_version/Pipfile.lock @@ -18,12 +18,12 @@ "default": { "typing-extensions": { "hashes": [ - "sha256:a439e7c04b49fec3e5d3e2beaa21755cadbbdc391694e28ccdd36ca4a1408f8c", - "sha256:e6c81219bd689f51865d9e372991c540bda33a0379d5573cddb9a3a23f7caaef" + "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", + "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548" ], "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==4.13.2" + "markers": "python_version >= '3.9'", + "version": "==4.15.0" } }, "develop": {} diff --git a/spec/fixtures/pipenv_no_lockfile/Pipfile b/spec/fixtures/pipenv_no_lockfile/Pipfile index 7a763f509..a62d277f4 100644 --- a/spec/fixtures/pipenv_no_lockfile/Pipfile +++ b/spec/fixtures/pipenv_no_lockfile/Pipfile @@ -9,4 +9,4 @@ typing-extensions = "*" [dev-packages] [requires] -python_version = "3.13" +python_version = "3.14" diff --git a/spec/fixtures/pipenv_python_full_version/Pipfile.lock b/spec/fixtures/pipenv_python_full_version/Pipfile.lock index 3269fa9ce..f71891fe1 100644 --- a/spec/fixtures/pipenv_python_full_version/Pipfile.lock +++ b/spec/fixtures/pipenv_python_full_version/Pipfile.lock @@ -19,12 +19,12 @@ "default": { "typing-extensions": { "hashes": [ - "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", - "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" + "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", + "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548" ], "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==4.12.2" + "markers": "python_version >= '3.9'", + "version": "==4.15.0" } }, "develop": {} diff --git a/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock b/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock index a9d6269b5..983c376a9 100644 --- a/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock +++ b/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock @@ -18,12 +18,12 @@ "default": { "typing-extensions": { "hashes": [ - "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", - "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" + "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", + "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548" ], "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==4.12.2" + "markers": "python_version >= '3.9'", + "version": "==4.15.0" } }, "develop": {} diff --git a/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile.lock b/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile.lock index c505f1d3c..257b94ada 100644 --- a/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile.lock +++ b/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile.lock @@ -18,12 +18,12 @@ "default": { "typing-extensions": { "hashes": [ - "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", - "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" + "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", + "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548" ], "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==4.12.2" + "markers": "python_version >= '3.9'", + "version": "==4.15.0" } }, "develop": {} diff --git a/spec/fixtures/pipenv_python_version_eol/Pipfile.lock b/spec/fixtures/pipenv_python_version_eol/Pipfile.lock index ef45647a8..026a1b0b1 100644 --- a/spec/fixtures/pipenv_python_version_eol/Pipfile.lock +++ b/spec/fixtures/pipenv_python_version_eol/Pipfile.lock @@ -18,12 +18,12 @@ "default": { "typing-extensions": { "hashes": [ - "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", - "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" + "sha256:a439e7c04b49fec3e5d3e2beaa21755cadbbdc391694e28ccdd36ca4a1408f8c", + "sha256:e6c81219bd689f51865d9e372991c540bda33a0379d5573cddb9a3a23f7caaef" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==4.12.2" + "version": "==4.13.2" } }, "develop": {} diff --git a/spec/fixtures/pipenv_python_version_invalid/Pipfile.lock b/spec/fixtures/pipenv_python_version_invalid/Pipfile.lock index 054e8d6cc..44f3e9fc6 100644 --- a/spec/fixtures/pipenv_python_version_invalid/Pipfile.lock +++ b/spec/fixtures/pipenv_python_version_invalid/Pipfile.lock @@ -18,12 +18,12 @@ "default": { "typing-extensions": { "hashes": [ - "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", - "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" + "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", + "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548" ], "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==4.12.2" + "markers": "python_version >= '3.9'", + "version": "==4.15.0" } }, "develop": {} diff --git a/spec/fixtures/pipenv_python_version_unspecified/Pipfile.lock b/spec/fixtures/pipenv_python_version_unspecified/Pipfile.lock index e744fd450..2b7b232d6 100644 --- a/spec/fixtures/pipenv_python_version_unspecified/Pipfile.lock +++ b/spec/fixtures/pipenv_python_version_unspecified/Pipfile.lock @@ -16,12 +16,12 @@ "default": { "typing-extensions": { "hashes": [ - "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", - "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8" + "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", + "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548" ], "index": "pypi", - "markers": "python_version >= '3.8'", - "version": "==4.12.2" + "markers": "python_version >= '3.9'", + "version": "==4.15.0" } }, "develop": {} diff --git a/spec/fixtures/poetry_basic/.python-version b/spec/fixtures/poetry_basic/.python-version index 24ee5b1be..6324d401a 100644 --- a/spec/fixtures/poetry_basic/.python-version +++ b/spec/fixtures/poetry_basic/.python-version @@ -1 +1 @@ -3.13 +3.14 diff --git a/spec/fixtures/poetry_basic/poetry.lock b/spec/fixtures/poetry_basic/poetry.lock index dc989760e..51cc64181 100644 --- a/spec/fixtures/poetry_basic/poetry.lock +++ b/spec/fixtures/poetry_basic/poetry.lock @@ -1,4 +1,4 @@ -# This file is automatically @generated by Poetry 2.2.0 and should not be changed by hand. +# This file is automatically @generated by Poetry 2.2.1 and should not be changed by hand. [[package]] name = "colorama" @@ -15,78 +15,94 @@ files = [ [[package]] name = "iniconfig" -version = "2.0.0" +version = "2.3.0" description = "brain-dead simple config-ini parsing" optional = false -python-versions = ">=3.7" +python-versions = ">=3.10" groups = ["dev"] files = [ - {file = "iniconfig-2.0.0-py3-none-any.whl", hash = "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374"}, - {file = "iniconfig-2.0.0.tar.gz", hash = "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3"}, + {file = "iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12"}, + {file = "iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730"}, ] [[package]] name = "packaging" -version = "24.2" +version = "25.0" description = "Core utilities for Python packages" optional = false python-versions = ">=3.8" groups = ["dev"] files = [ - {file = "packaging-24.2-py3-none-any.whl", hash = "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759"}, - {file = "packaging-24.2.tar.gz", hash = "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f"}, + {file = "packaging-25.0-py3-none-any.whl", hash = "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484"}, + {file = "packaging-25.0.tar.gz", hash = "sha256:d443872c98d677bf60f6a1f2f8c1cb748e8fe762d2bf9d3148b5599295b0fc4f"}, ] [[package]] name = "pluggy" -version = "1.5.0" +version = "1.6.0" description = "plugin and hook calling mechanisms for python" optional = false -python-versions = ">=3.8" +python-versions = ">=3.9" groups = ["dev"] files = [ - {file = "pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669"}, - {file = "pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1"}, + {file = "pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746"}, + {file = "pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3"}, ] [package.extras] dev = ["pre-commit", "tox"] -testing = ["pytest", "pytest-benchmark"] +testing = ["coverage", "pytest", "pytest-benchmark"] + +[[package]] +name = "pygments" +version = "2.19.2" +description = "Pygments is a syntax highlighting package written in Python." +optional = false +python-versions = ">=3.8" +groups = ["dev"] +files = [ + {file = "pygments-2.19.2-py3-none-any.whl", hash = "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b"}, + {file = "pygments-2.19.2.tar.gz", hash = "sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887"}, +] + +[package.extras] +windows-terminal = ["colorama (>=0.4.6)"] [[package]] name = "pytest" -version = "8.3.4" +version = "9.0.1" description = "pytest: simple powerful testing with Python" optional = false -python-versions = ">=3.8" +python-versions = ">=3.10" groups = ["dev"] files = [ - {file = "pytest-8.3.4-py3-none-any.whl", hash = "sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6"}, - {file = "pytest-8.3.4.tar.gz", hash = "sha256:965370d062bce11e73868e0335abac31b4d3de0e82f4007408d242b4f8610761"}, + {file = "pytest-9.0.1-py3-none-any.whl", hash = "sha256:67be0030d194df2dfa7b556f2e56fb3c3315bd5c8822c6951162b92b32ce7dad"}, + {file = "pytest-9.0.1.tar.gz", hash = "sha256:3e9c069ea73583e255c3b21cf46b8d3c56f6e3a1a8f6da94ccb0fcf57b9d73c8"}, ] [package.dependencies] -colorama = {version = "*", markers = "sys_platform == \"win32\""} -iniconfig = "*" -packaging = "*" +colorama = {version = ">=0.4", markers = "sys_platform == \"win32\""} +iniconfig = ">=1.0.1" +packaging = ">=22" pluggy = ">=1.5,<2" +pygments = ">=2.7.2" [package.extras] -dev = ["argcomplete", "attrs (>=19.2)", "hypothesis (>=3.56)", "mock", "pygments (>=2.7.2)", "requests", "setuptools", "xmlschema"] +dev = ["argcomplete", "attrs (>=19.2)", "hypothesis (>=3.56)", "mock", "requests", "setuptools", "xmlschema"] [[package]] name = "typing-extensions" -version = "4.12.2" -description = "Backported and Experimental Type Hints for Python 3.8+" +version = "4.15.0" +description = "Backported and Experimental Type Hints for Python 3.9+" optional = false -python-versions = ">=3.8" +python-versions = ">=3.9" groups = ["main"] files = [ - {file = "typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d"}, - {file = "typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"}, + {file = "typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548"}, + {file = "typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466"}, ] [metadata] lock-version = "2.1" -python-versions = ">=3.13" -content-hash = "029e0ee94dae38c88285bb0a557d4b77d92e76d30aa8ca91d17ac51fcf62b7f1" +python-versions = ">=3.14" +content-hash = "a0f8d6facd04ce28abc860e56bb4af6e1acac71a1f2a93e4e6ed4e13a299e055" diff --git a/spec/fixtures/poetry_basic/pyproject.toml b/spec/fixtures/poetry_basic/pyproject.toml index 3da2a688e..3a23e1a00 100644 --- a/spec/fixtures/poetry_basic/pyproject.toml +++ b/spec/fixtures/poetry_basic/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "poetry-basic" version = "0.0.0" -requires-python = ">=3.13" +requires-python = ">=3.14" dependencies = [ "typing-extensions", ] diff --git a/spec/fixtures/poetry_editable/poetry.lock b/spec/fixtures/poetry_editable/poetry.lock index 30df026b9..9e21fd4e9 100644 --- a/spec/fixtures/poetry_editable/poetry.lock +++ b/spec/fixtures/poetry_editable/poetry.lock @@ -1,11 +1,11 @@ -# This file is automatically @generated by Poetry 2.1.1 and should not be changed by hand. +# This file is automatically @generated by Poetry 2.2.1 and should not be changed by hand. [[package]] name = "gunicorn" version = "23.0.0" description = "WSGI HTTP Server for UNIX" optional = false -python-versions = ">=3.7" +python-versions = ">=3.10" groups = ["main"] files = [] develop = true @@ -24,7 +24,7 @@ tornado = ["tornado (>=0.2)"] type = "git" url = "https://github.com/benoitc/gunicorn.git" reference = "HEAD" -resolved_reference = "bacbf8aa5152b94e44aa5d2a94aeaf0318a85248" +resolved_reference = "56b5ad87f8d72a674145c273ed8f547513c2b409" [[package]] name = "local-package-pyproject-toml" @@ -56,17 +56,17 @@ url = "packages/local_package_setup_py" [[package]] name = "packaging" -version = "24.2" +version = "25.0" description = "Core utilities for Python packages" optional = false python-versions = ">=3.8" groups = ["main"] files = [ - {file = "packaging-24.2-py3-none-any.whl", hash = "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759"}, - {file = "packaging-24.2.tar.gz", hash = "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f"}, + {file = "packaging-25.0-py3-none-any.whl", hash = "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484"}, + {file = "packaging-25.0.tar.gz", hash = "sha256:d443872c98d677bf60f6a1f2f8c1cb748e8fe762d2bf9d3148b5599295b0fc4f"}, ] [metadata] lock-version = "2.1" -python-versions = "^3.13" -content-hash = "f3fb8b00f781abfc5c963f9a266d16757b9b92cc415866eb28f156b4083fbf48" +python-versions = "^3.14" +content-hash = "f0f202f553940509c2ae9bfc77fdd0df2448f9df01eee351f823249128699160" diff --git a/spec/fixtures/poetry_editable/pyproject.toml b/spec/fixtures/poetry_editable/pyproject.toml index 8df4cea76..cc56aaad4 100644 --- a/spec/fixtures/poetry_editable/pyproject.toml +++ b/spec/fixtures/poetry_editable/pyproject.toml @@ -5,7 +5,7 @@ description = "" authors = [] [tool.poetry.dependencies] -python = "^3.13" +python = "^3.14" gunicorn = { git = "https://github.com/benoitc/gunicorn.git", develop = true } local-package-pyproject-toml = { path = "packages/local_package_pyproject_toml", develop = true } local-package-setup-py = { path = "packages/local_package_setup_py", develop = true } diff --git a/spec/fixtures/poetry_lockfile_out_of_sync/poetry.lock b/spec/fixtures/poetry_lockfile_out_of_sync/poetry.lock index e9d42af7d..9b4be8159 100644 --- a/spec/fixtures/poetry_lockfile_out_of_sync/poetry.lock +++ b/spec/fixtures/poetry_lockfile_out_of_sync/poetry.lock @@ -1,7 +1,7 @@ -# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand. +# This file is automatically @generated by Poetry 2.2.1 and should not be changed by hand. package = [] [metadata] -lock-version = "2.0" -python-versions = "^3.13" -content-hash = "f01b553f3895e558c34b4f10542e05acdef39bf0527c8090bd136d914dc73f94" +lock-version = "2.1" +python-versions = "^3.14" +content-hash = "d3f696e8f01aec1733802da3db722ca7c5e48e3d967cd7e2ce28ee65396abe6c" diff --git a/spec/fixtures/poetry_lockfile_out_of_sync/pyproject.toml b/spec/fixtures/poetry_lockfile_out_of_sync/pyproject.toml index 032945f60..1c9cd7013 100644 --- a/spec/fixtures/poetry_lockfile_out_of_sync/pyproject.toml +++ b/spec/fixtures/poetry_lockfile_out_of_sync/pyproject.toml @@ -2,7 +2,7 @@ package-mode = false [tool.poetry.dependencies] -python = "^3.13" +python = "^3.14" # This dependency isn't in the lockfile. typing-extensions = "*" diff --git a/spec/fixtures/poetry_mismatched_python_version/poetry.lock b/spec/fixtures/poetry_mismatched_python_version/poetry.lock index 5da81ae0f..d7e662543 100644 --- a/spec/fixtures/poetry_mismatched_python_version/poetry.lock +++ b/spec/fixtures/poetry_mismatched_python_version/poetry.lock @@ -1,15 +1,15 @@ -# This file is automatically @generated by Poetry 2.1.3 and should not be changed by hand. +# This file is automatically @generated by Poetry 2.2.1 and should not be changed by hand. [[package]] name = "typing-extensions" -version = "4.13.2" -description = "Backported and Experimental Type Hints for Python 3.8+" +version = "4.15.0" +description = "Backported and Experimental Type Hints for Python 3.9+" optional = false -python-versions = ">=3.8" +python-versions = ">=3.9" groups = ["main"] files = [ - {file = "typing_extensions-4.13.2-py3-none-any.whl", hash = "sha256:a439e7c04b49fec3e5d3e2beaa21755cadbbdc391694e28ccdd36ca4a1408f8c"}, - {file = "typing_extensions-4.13.2.tar.gz", hash = "sha256:e6c81219bd689f51865d9e372991c540bda33a0379d5573cddb9a3a23f7caaef"}, + {file = "typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548"}, + {file = "typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466"}, ] [metadata] diff --git a/spec/fixtures/poetry_oldest_python/poetry.lock b/spec/fixtures/poetry_oldest_python/poetry.lock index 5f96ba1c8..c9a78e867 100644 --- a/spec/fixtures/poetry_oldest_python/poetry.lock +++ b/spec/fixtures/poetry_oldest_python/poetry.lock @@ -2,13 +2,13 @@ [[package]] name = "typing-extensions" -version = "4.12.2" -description = "Backported and Experimental Type Hints for Python 3.8+" +version = "4.15.0" +description = "Backported and Experimental Type Hints for Python 3.9+" optional = false -python-versions = ">=3.8" +python-versions = ">=3.9" files = [ - {file = "typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d"}, - {file = "typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"}, + {file = "typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548"}, + {file = "typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466"}, ] [metadata] diff --git a/spec/fixtures/python_3.10/requirements.txt b/spec/fixtures/python_3.10/requirements.txt index eec3a2223..f6e499a84 100644 --- a/spec/fixtures/python_3.10/requirements.txt +++ b/spec/fixtures/python_3.10/requirements.txt @@ -1,2 +1,2 @@ # This package has been picked since it has no dependencies and is small/fast to install. -typing-extensions==4.12.2 +typing-extensions==4.15.0 diff --git a/spec/fixtures/python_3.11/requirements.txt b/spec/fixtures/python_3.11/requirements.txt index eec3a2223..f6e499a84 100644 --- a/spec/fixtures/python_3.11/requirements.txt +++ b/spec/fixtures/python_3.11/requirements.txt @@ -1,2 +1,2 @@ # This package has been picked since it has no dependencies and is small/fast to install. -typing-extensions==4.12.2 +typing-extensions==4.15.0 diff --git a/spec/fixtures/python_3.12/requirements.txt b/spec/fixtures/python_3.12/requirements.txt index eec3a2223..f6e499a84 100644 --- a/spec/fixtures/python_3.12/requirements.txt +++ b/spec/fixtures/python_3.12/requirements.txt @@ -1,2 +1,2 @@ # This package has been picked since it has no dependencies and is small/fast to install. -typing-extensions==4.12.2 +typing-extensions==4.15.0 diff --git a/spec/fixtures/python_3.13/requirements.txt b/spec/fixtures/python_3.13/requirements.txt index eec3a2223..f6e499a84 100644 --- a/spec/fixtures/python_3.13/requirements.txt +++ b/spec/fixtures/python_3.13/requirements.txt @@ -1,2 +1,2 @@ # This package has been picked since it has no dependencies and is small/fast to install. -typing-extensions==4.12.2 +typing-extensions==4.15.0 diff --git a/spec/fixtures/python_3.14/requirements.txt b/spec/fixtures/python_3.14/requirements.txt index eec3a2223..f6e499a84 100644 --- a/spec/fixtures/python_3.14/requirements.txt +++ b/spec/fixtures/python_3.14/requirements.txt @@ -1,2 +1,2 @@ # This package has been picked since it has no dependencies and is small/fast to install. -typing-extensions==4.12.2 +typing-extensions==4.15.0 diff --git a/spec/fixtures/python_3.9/requirements.txt b/spec/fixtures/python_3.9/requirements.txt index eec3a2223..f6e499a84 100644 --- a/spec/fixtures/python_3.9/requirements.txt +++ b/spec/fixtures/python_3.9/requirements.txt @@ -1,2 +1,2 @@ # This package has been picked since it has no dependencies and is small/fast to install. -typing-extensions==4.12.2 +typing-extensions==4.15.0 diff --git a/spec/fixtures/python_in_app_source/.python-version b/spec/fixtures/python_in_app_source/.python-version index 24ee5b1be..6324d401a 100644 --- a/spec/fixtures/python_in_app_source/.python-version +++ b/spec/fixtures/python_in_app_source/.python-version @@ -1 +1 @@ -3.13 +3.14 diff --git a/spec/fixtures/python_version_unspecified/requirements.txt b/spec/fixtures/python_version_unspecified/requirements.txt index eec3a2223..f6e499a84 100644 --- a/spec/fixtures/python_version_unspecified/requirements.txt +++ b/spec/fixtures/python_version_unspecified/requirements.txt @@ -1,2 +1,2 @@ # This package has been picked since it has no dependencies and is small/fast to install. -typing-extensions==4.12.2 +typing-extensions==4.15.0 diff --git a/spec/fixtures/setup_py_only/.python-version b/spec/fixtures/setup_py_only/.python-version index 24ee5b1be..6324d401a 100644 --- a/spec/fixtures/setup_py_only/.python-version +++ b/spec/fixtures/setup_py_only/.python-version @@ -1 +1 @@ -3.13 +3.14 diff --git a/spec/fixtures/uv_basic/.python-version b/spec/fixtures/uv_basic/.python-version index 24ee5b1be..6324d401a 100644 --- a/spec/fixtures/uv_basic/.python-version +++ b/spec/fixtures/uv_basic/.python-version @@ -1 +1 @@ -3.13 +3.14 diff --git a/spec/fixtures/uv_basic/pyproject.toml b/spec/fixtures/uv_basic/pyproject.toml index ed1942882..b0b1ffa60 100644 --- a/spec/fixtures/uv_basic/pyproject.toml +++ b/spec/fixtures/uv_basic/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "uv-basic" version = "0.0.0" -requires-python = ">=3.13" +requires-python = ">=3.14" dependencies = [ "typing-extensions", ] diff --git a/spec/fixtures/uv_basic/uv.lock b/spec/fixtures/uv_basic/uv.lock index 87eb42270..163be6bca 100644 --- a/spec/fixtures/uv_basic/uv.lock +++ b/spec/fixtures/uv_basic/uv.lock @@ -1,6 +1,6 @@ version = 1 -revision = 2 -requires-python = ">=3.13" +revision = 3 +requires-python = ">=3.14" [[package]] name = "colorama" @@ -13,11 +13,11 @@ wheels = [ [[package]] name = "iniconfig" -version = "2.1.0" +version = "2.3.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/f2/97/ebf4da567aa6827c909642694d71c9fcf53e5b504f2d96afea02718862f3/iniconfig-2.1.0.tar.gz", hash = "sha256:3abbd2e30b36733fee78f9c7f7308f2d0050e88f0087fd25c2645f63c773e1c7", size = 4793, upload-time = "2025-03-19T20:09:59.721Z" } +sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/2c/e1/e6716421ea10d38022b952c159d5161ca1193197fb744506875fbb87ea7b/iniconfig-2.1.0-py3-none-any.whl", hash = "sha256:9deba5723312380e77435581c6bf4935c94cbfab9b1ed33ef8d238ea168eb760", size = 6050, upload-time = "2025-03-19T20:10:01.071Z" }, + { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, ] [[package]] @@ -31,35 +31,45 @@ wheels = [ [[package]] name = "pluggy" -version = "1.5.0" +version = "1.6.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/96/2d/02d4312c973c6050a18b314a5ad0b3210edb65a906f868e31c111dede4a6/pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1", size = 67955, upload-time = "2024-04-20T21:34:42.531Z" } +sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/88/5f/e351af9a41f866ac3f1fac4ca0613908d9a41741cfcf2228f4ad853b697d/pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669", size = 20556, upload-time = "2024-04-20T21:34:40.434Z" }, + { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, +] + +[[package]] +name = "pygments" +version = "2.19.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b0/77/a5b8c569bf593b0140bde72ea885a803b82086995367bf2037de0159d924/pygments-2.19.2.tar.gz", hash = "sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887", size = 4968631, upload-time = "2025-06-21T13:39:12.283Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c7/21/705964c7812476f378728bdf590ca4b771ec72385c533964653c68e86bdc/pygments-2.19.2-py3-none-any.whl", hash = "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b", size = 1225217, upload-time = "2025-06-21T13:39:07.939Z" }, ] [[package]] name = "pytest" -version = "8.3.5" +version = "9.0.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "colorama", marker = "sys_platform == 'win32'" }, { name = "iniconfig" }, { name = "packaging" }, { name = "pluggy" }, + { name = "pygments" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/ae/3c/c9d525a414d506893f0cd8a8d0de7706446213181570cdbd766691164e40/pytest-8.3.5.tar.gz", hash = "sha256:f4efe70cc14e511565ac476b57c279e12a855b11f48f212af1080ef2263d3845", size = 1450891, upload-time = "2025-03-02T12:54:54.503Z" } +sdist = { url = "https://files.pythonhosted.org/packages/07/56/f013048ac4bc4c1d9be45afd4ab209ea62822fb1598f40687e6bf45dcea4/pytest-9.0.1.tar.gz", hash = "sha256:3e9c069ea73583e255c3b21cf46b8d3c56f6e3a1a8f6da94ccb0fcf57b9d73c8", size = 1564125, upload-time = "2025-11-12T13:05:09.333Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/30/3d/64ad57c803f1fa1e963a7946b6e0fea4a70df53c1a7fed304586539c2bac/pytest-8.3.5-py3-none-any.whl", hash = "sha256:c69214aa47deac29fad6c2a4f590b9c4a9fdb16a403176fe154b79c0b4d4d820", size = 343634, upload-time = "2025-03-02T12:54:52.069Z" }, + { url = "https://files.pythonhosted.org/packages/0b/8b/6300fb80f858cda1c51ffa17075df5d846757081d11ab4aa35cef9e6258b/pytest-9.0.1-py3-none-any.whl", hash = "sha256:67be0030d194df2dfa7b556f2e56fb3c3315bd5c8822c6951162b92b32ce7dad", size = 373668, upload-time = "2025-11-12T13:05:07.379Z" }, ] [[package]] name = "typing-extensions" -version = "4.13.2" +version = "4.15.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/f6/37/23083fcd6e35492953e8d2aaaa68b860eb422b34627b13f2ce3eb6106061/typing_extensions-4.13.2.tar.gz", hash = "sha256:e6c81219bd689f51865d9e372991c540bda33a0379d5573cddb9a3a23f7caaef", size = 106967, upload-time = "2025-04-10T14:19:05.416Z" } +sdist = { url = "https://files.pythonhosted.org/packages/72/94/1a15dd82efb362ac84269196e94cf00f187f7ed21c242792a923cdb1c61f/typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", size = 109391, upload-time = "2025-08-25T13:49:26.313Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/8b/54/b1ae86c0973cc6f0210b53d508ca3641fb6d0c56823f288d108bc7ab3cc8/typing_extensions-4.13.2-py3-none-any.whl", hash = "sha256:a439e7c04b49fec3e5d3e2beaa21755cadbbdc391694e28ccdd36ca4a1408f8c", size = 45806, upload-time = "2025-04-10T14:19:03.967Z" }, + { url = "https://files.pythonhosted.org/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548", size = 44614, upload-time = "2025-08-25T13:49:24.86Z" }, ] [[package]] diff --git a/spec/fixtures/uv_editable/.python-version b/spec/fixtures/uv_editable/.python-version index 24ee5b1be..6324d401a 100644 --- a/spec/fixtures/uv_editable/.python-version +++ b/spec/fixtures/uv_editable/.python-version @@ -1 +1 @@ -3.13 +3.14 diff --git a/spec/fixtures/uv_editable/pyproject.toml b/spec/fixtures/uv_editable/pyproject.toml index 74aec087e..15a7b2365 100644 --- a/spec/fixtures/uv_editable/pyproject.toml +++ b/spec/fixtures/uv_editable/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "uv-editable" version = "0.0.0" -requires-python = ">=3.13" +requires-python = ">=3.14" dependencies = [ "gunicorn", "local-package-pyproject-toml", diff --git a/spec/fixtures/uv_editable/uv.lock b/spec/fixtures/uv_editable/uv.lock index f44782cdb..506143ee7 100644 --- a/spec/fixtures/uv_editable/uv.lock +++ b/spec/fixtures/uv_editable/uv.lock @@ -1,11 +1,11 @@ version = 1 -revision = 2 -requires-python = ">=3.13" +revision = 3 +requires-python = ">=3.14" [[package]] name = "gunicorn" version = "23.0.0" -source = { git = "https://github.com/benoitc/gunicorn#a86ea1e4e6c271d1cd1823c7e14490123f9238fe" } +source = { git = "https://github.com/benoitc/gunicorn#56b5ad87f8d72a674145c273ed8f547513c2b409" } dependencies = [ { name = "packaging" }, ] diff --git a/spec/fixtures/uv_lockfile_out_of_sync/.python-version b/spec/fixtures/uv_lockfile_out_of_sync/.python-version index 24ee5b1be..6324d401a 100644 --- a/spec/fixtures/uv_lockfile_out_of_sync/.python-version +++ b/spec/fixtures/uv_lockfile_out_of_sync/.python-version @@ -1 +1 @@ -3.13 +3.14 diff --git a/spec/fixtures/uv_lockfile_out_of_sync/pyproject.toml b/spec/fixtures/uv_lockfile_out_of_sync/pyproject.toml index aca766082..8bd3574b4 100644 --- a/spec/fixtures/uv_lockfile_out_of_sync/pyproject.toml +++ b/spec/fixtures/uv_lockfile_out_of_sync/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "uv-lockfile-out-of-sync" version = "0.0.0" -requires-python = ">=3.13" +requires-python = ">=3.14" # This dependency isn't in the lockfile. dependencies = [ diff --git a/spec/fixtures/uv_lockfile_out_of_sync/uv.lock b/spec/fixtures/uv_lockfile_out_of_sync/uv.lock index 475542093..50a8225c6 100644 --- a/spec/fixtures/uv_lockfile_out_of_sync/uv.lock +++ b/spec/fixtures/uv_lockfile_out_of_sync/uv.lock @@ -1,6 +1,6 @@ version = 1 -revision = 2 -requires-python = ">=3.13" +revision = 3 +requires-python = ">=3.14" [[package]] name = "uv-lockfile-out-of-sync" diff --git a/spec/fixtures/uv_mismatched_python_version/uv.lock b/spec/fixtures/uv_mismatched_python_version/uv.lock index 604350e24..5982e7c5e 100644 --- a/spec/fixtures/uv_mismatched_python_version/uv.lock +++ b/spec/fixtures/uv_mismatched_python_version/uv.lock @@ -1,5 +1,5 @@ version = 1 -revision = 2 +revision = 3 requires-python = "==3.12.*" [[package]] diff --git a/spec/fixtures/uv_no_python_version_file/uv.lock b/spec/fixtures/uv_no_python_version_file/uv.lock index 2c9c3cf8d..b1a4d90e0 100644 --- a/spec/fixtures/uv_no_python_version_file/uv.lock +++ b/spec/fixtures/uv_no_python_version_file/uv.lock @@ -1,5 +1,5 @@ version = 1 -revision = 2 +revision = 3 requires-python = "==3.12.*" [[package]] diff --git a/spec/fixtures/uv_oldest_python/uv.lock b/spec/fixtures/uv_oldest_python/uv.lock index 27cbd3836..1200f8421 100644 --- a/spec/fixtures/uv_oldest_python/uv.lock +++ b/spec/fixtures/uv_oldest_python/uv.lock @@ -4,11 +4,11 @@ requires-python = "==3.9.*" [[package]] name = "typing-extensions" -version = "4.13.2" +version = "4.15.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/f6/37/23083fcd6e35492953e8d2aaaa68b860eb422b34627b13f2ce3eb6106061/typing_extensions-4.13.2.tar.gz", hash = "sha256:e6c81219bd689f51865d9e372991c540bda33a0379d5573cddb9a3a23f7caaef", size = 106967, upload-time = "2025-04-10T14:19:05.416Z" } +sdist = { url = "https://files.pythonhosted.org/packages/72/94/1a15dd82efb362ac84269196e94cf00f187f7ed21c242792a923cdb1c61f/typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", size = 109391, upload-time = "2025-08-25T13:49:26.313Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/8b/54/b1ae86c0973cc6f0210b53d508ca3641fb6d0c56823f288d108bc7ab3cc8/typing_extensions-4.13.2-py3-none-any.whl", hash = "sha256:a439e7c04b49fec3e5d3e2beaa21755cadbbdc391694e28ccdd36ca4a1408f8c", size = 45806, upload-time = "2025-04-10T14:19:03.967Z" }, + { url = "https://files.pythonhosted.org/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548", size = 44614, upload-time = "2025-08-25T13:49:24.86Z" }, ] [[package]] diff --git a/spec/fixtures/uv_runtime_txt/uv.lock b/spec/fixtures/uv_runtime_txt/uv.lock index f5d9d9025..8d1d51bcf 100644 --- a/spec/fixtures/uv_runtime_txt/uv.lock +++ b/spec/fixtures/uv_runtime_txt/uv.lock @@ -1,6 +1,6 @@ version = 1 -revision = 2 -requires-python = ">=3.13" +revision = 3 +requires-python = "==3.12.*" [[package]] name = "uv-runtime-txt" diff --git a/spec/fixtures/venv_in_app_source/.python-version b/spec/fixtures/venv_in_app_source/.python-version index 24ee5b1be..6324d401a 100644 --- a/spec/fixtures/venv_in_app_source/.python-version +++ b/spec/fixtures/venv_in_app_source/.python-version @@ -1 +1 @@ -3.13 +3.14 diff --git a/spec/hatchet/checks_spec.rb b/spec/hatchet/checks_spec.rb index 2972c8947..a7acf46da 100644 --- a/spec/hatchet/checks_spec.rb +++ b/spec/hatchet/checks_spec.rb @@ -104,7 +104,7 @@ remote: ! In future versions of the buildpack, this warning will be turned remote: ! into an error. remote: - remote: -----> Using Python 3.13 specified in .python-version + remote: -----> Using Python 3.14 specified in .python-version OUTPUT end end diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 1e6be74bc..4f01a43d2 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -202,7 +202,7 @@ -----> Installing dependencies using 'poetry sync' Installing dependencies from lock file - Package operations: 5 installs, 0 updates, 0 removals + Package operations: 6 installs, 0 updates, 0 removals .+ - Installing (pytest|typing-extensions) .+ @@ -296,9 +296,9 @@ -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing uv #{UV_VERSION} -----> Installing dependencies using 'uv sync --locked' - Resolved 7 packages in .+s - Prepared 5 packages in .+s - Installed 5 packages in .+s + Resolved 8 packages in .+s + Prepared 6 packages in .+s + Installed 6 packages in .+s Bytecode compiled .+ files in .+s .+ \\+ (pytest|typing-extensions)==.+ @@ -361,7 +361,7 @@ -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Using cached uv #{UV_VERSION} -----> Installing dependencies using 'uv sync --locked' - Resolved 7 packages in .+s + Resolved 8 packages in .+s Bytecode compiled .+ files in .+s -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running bin/post_compile hook diff --git a/spec/hatchet/django_spec.rb b/spec/hatchet/django_spec.rb index ad3e67207..d68ca4fda 100644 --- a/spec/hatchet/django_spec.rb +++ b/spec/hatchet/django_spec.rb @@ -51,10 +51,10 @@ remote: remote: \\['/tmp/build_\\w+/backend', remote: '/tmp/build_\\w+', - remote: '/app/.heroku/python/lib/python313.zip', - remote: '/app/.heroku/python/lib/python3.13', - remote: '/app/.heroku/python/lib/python3.13/lib-dynload', - remote: '/app/.heroku/python/lib/python3.13/site-packages'\\] + remote: '/app/.heroku/python/lib/python314.zip', + remote: '/app/.heroku/python/lib/python3.14', + remote: '/app/.heroku/python/lib/python3.14/lib-dynload', + remote: '/app/.heroku/python/lib/python3.14/site-packages'\\] remote: remote: 1 static file copied to '/tmp/build_\\w+/backend/staticfiles'. remote: diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index 63b1dafae..da99ac2e6 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -166,11 +166,11 @@ remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing uv #{UV_VERSION} remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' - remote: Resolved 7 packages in .+s + remote: Resolved .+ packages in .+s remote: Prepared 1 package in .+s remote: Installed 1 package in .+s remote: Bytecode compiled 1 file in .+s - remote: \\+ typing-extensions==4.13.2 + remote: \\+ typing-extensions==4.15.0 remote: -----> Saving cache remote: -----> Discovering process types REGEX diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index a319f9da2..2d8ec44bb 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -15,11 +15,11 @@ remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting typing-extensions==4.12.2 \\(from -r requirements.txt \\(line 5\\)\\) - remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata \\(3.0 kB\\) - remote: Downloading typing_extensions-4.12.2-py3-none-any.whl \\(37 kB\\) + remote: Collecting typing-extensions==4.15.0 \\(from -r requirements.txt \\(line 5\\)\\) + remote: Downloading typing_extensions-4.15.0-py3-none-any.whl.metadata \\(3.3 kB\\) + remote: Downloading typing_extensions-4.15.0-py3-none-any.whl \\(44 kB\\) remote: Installing collected packages: typing-extensions - remote: Successfully installed typing-extensions-4.12.2 + remote: Successfully installed typing-extensions-4.15.0 remote: -----> Running bin/post_compile hook remote: BUILD_DIR=/tmp/build_.+ remote: CACHE_DIR=/tmp/codon/tmp/cache @@ -57,18 +57,18 @@ remote: remote: \\['', remote: '/app', - remote: '/app/.heroku/python/lib/python313.zip', - remote: '/app/.heroku/python/lib/python3.13', - remote: '/app/.heroku/python/lib/python3.13/lib-dynload', - remote: '/app/.heroku/python/lib/python3.13/site-packages'\\] + remote: '/app/.heroku/python/lib/python314.zip', + remote: '/app/.heroku/python/lib/python3.14', + remote: '/app/.heroku/python/lib/python3.14/lib-dynload', + remote: '/app/.heroku/python/lib/python3.14/site-packages'\\] remote: - remote: pip #{PIP_VERSION} from /app/.heroku/python/lib/python3.13/site-packages/pip \\(python 3.13\\) + remote: pip #{PIP_VERSION} from /app/.heroku/python/lib/python3.14/site-packages/pip \\(python 3.14\\) remote: Package Version remote: ----------------- ------- remote: pip #{PIP_VERSION} - remote: typing_extensions 4.12.2 + remote: typing_extensions 4.15.0 remote: - remote: + remote: remote: remote: \\{ remote: "cache_restore_duration": [0-9.]+, @@ -85,11 +85,11 @@ remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", - remote: "python_version_major": "3.13", + remote: "python_version_major": "3.14", remote: "python_version_origin": ".python-version", remote: "python_version_outdated": false, remote: "python_version_pinned": false, - remote: "python_version_requested": "3.13", + remote: "python_version_requested": "3.14", remote: "total_duration": [0-9.]+ remote: \\} REGEX @@ -103,7 +103,7 @@ remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Requirement already satisfied: typing-extensions==4.12.2 \\(from -r requirements.txt \\(line 5\\)\\) \\(4.12.2\\) + remote: Requirement already satisfied: typing-extensions==4.15.0 \\(from -r requirements.txt \\(line 5\\)\\) \\(4.15.0\\) remote: -----> Running bin/post_compile hook remote: .+ remote: -----> Saving cache @@ -123,7 +123,7 @@ PYTHONPATH=/app PYTHONUNBUFFERED=true WEB_CONCURRENCY=2 - pip #{PIP_VERSION} from /app/.heroku/python/lib/python3.13/site-packages/pip (python 3.13) + pip #{PIP_VERSION} from /app/.heroku/python/lib/python3.14/site-packages/pip (python 3.14) OUTPUT expect($CHILD_STATUS.exitstatus).to eq(0) end @@ -149,14 +149,14 @@ remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 5)) - remote: Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) + remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 5)) + remote: Downloading typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB) remote: Collecting six==1.17.0 (from -r requirements.txt (line 6)) remote: Downloading six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB) - remote: Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) + remote: Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) remote: Downloading six-1.17.0-py2.py3-none-any.whl (11 kB) remote: Installing collected packages: typing-extensions, six - remote: Successfully installed six-1.17.0 typing-extensions-4.12.2 + remote: Successfully installed six-1.17.0 typing-extensions-4.15.0 remote: -----> Saving cache OUTPUT end @@ -320,11 +320,11 @@ remote: -----> Installing Python 3.9.0 remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting typing-extensions==4.14.1 (from -r requirements.txt (line 2)) - remote: Downloading typing_extensions-4.14.1-py3-none-any.whl.metadata (3.0 kB) - remote: Downloading typing_extensions-4.14.1-py3-none-any.whl (43 kB) + remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) + remote: Downloading typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB) + remote: Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) remote: Installing collected packages: typing-extensions - remote: Successfully installed typing-extensions-4.14.1 + remote: Successfully installed typing-extensions-4.15.0 remote: -----> Saving cache OUTPUT app.commit! @@ -334,7 +334,7 @@ # regex variant can be removed once support for Python 3.10 and older is dropped. expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Requirement already satisfied: typing-extensions==4.14.1 (from -r requirements.txt (line 2)) (4.14.1) + remote: Requirement already satisfied: typing-extensions==4.15.0 (from -r requirements.txt (line 2)) (4.15.0) remote: -----> Saving cache OUTPUT end diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index d1f6621b3..49bc4dfd5 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -57,18 +57,18 @@ remote: remote: \\['', remote: '/app', - remote: '/app/.heroku/python/lib/python313.zip', - remote: '/app/.heroku/python/lib/python3.13', - remote: '/app/.heroku/python/lib/python3.13/lib-dynload', - remote: '/app/.heroku/python/lib/python3.13/site-packages'\\] + remote: '/app/.heroku/python/lib/python314.zip', + remote: '/app/.heroku/python/lib/python3.14', + remote: '/app/.heroku/python/lib/python3.14/lib-dynload', + remote: '/app/.heroku/python/lib/python3.14/site-packages'\\] remote: remote: pipenv, version #{PIPENV_VERSION} remote: Package Version remote: ----------------- -+ - remote: certifi 2025.7.14 - remote: typing_extensions 4.12.2 + remote: certifi 2025.11.12 + remote: typing_extensions 4.15.0 remote: - remote: + remote: remote: remote: \\{ remote: "cache_restore_duration": [0-9.]+, @@ -85,11 +85,11 @@ remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", - remote: "python_version_major": "3.13", + remote: "python_version_major": "3.14", remote: "python_version_origin": "Pipfile.lock", remote: "python_version_outdated": false, remote: "python_version_pinned": false, - remote: "python_version_requested": "3.13", + remote: "python_version_requested": "3.14", remote: "total_duration": [0-9.]+ remote: \\} REGEX @@ -455,8 +455,9 @@ end end - context 'when the Pipenv version has changed since the last build' do - let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v291'] } + # TODO: Rename this test description back this when the Pipenv version next changes. + context 'when the Python version has changed since the last build' do + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v313'] } let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_basic', buildpacks:) } it 'clears the cache before installing' do @@ -467,11 +468,10 @@ app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in Pipfile.lock + remote: -----> Using Python 3.14 specified in Pipfile.lock remote: -----> Discarding cache since: - remote: - The Python version has changed from 3.13.5 to #{DEFAULT_PYTHON_FULL_VERSION} - remote: - The Pipenv version has changed from 2024.0.1 to #{PIPENV_VERSION} - remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} + remote: - The Python version has changed from 3.14.0 to #{LATEST_PYTHON_3_14} + remote: -----> Installing Python #{LATEST_PYTHON_3_14} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 54b2e25bd..932c7bf46 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -19,7 +19,7 @@ remote: remote: Package operations: 1 install, 0 updates, 0 removals remote: - remote: - Installing typing-extensions \\(4.12.2\\) + remote: - Installing typing-extensions \\(4.15.0\\) remote: -----> Running bin/post_compile hook remote: BUILD_DIR=/tmp/build_.+ remote: CACHE_DIR=/tmp/codon/tmp/cache @@ -59,16 +59,16 @@ remote: remote: \\['', remote: '/app', - remote: '/app/.heroku/python/lib/python313.zip', - remote: '/app/.heroku/python/lib/python3.13', - remote: '/app/.heroku/python/lib/python3.13/lib-dynload', - remote: '/app/.heroku/python/lib/python3.13/site-packages'\\] + remote: '/app/.heroku/python/lib/python314.zip', + remote: '/app/.heroku/python/lib/python3.14', + remote: '/app/.heroku/python/lib/python3.14/lib-dynload', + remote: '/app/.heroku/python/lib/python3.14/site-packages'\\] remote: remote: Poetry \\(version #{POETRY_VERSION}\\) remote: Skipping virtualenv creation, as specified in config file. - remote: typing-extensions 4.12.2 Backported and Experimental Type Hints for Python ... + remote: typing-extensions 4.15.0 Backported and Experimental Type Hints for Python ... remote: - remote: + remote: remote: remote: \\{ remote: "cache_restore_duration": [0-9.]+, @@ -85,11 +85,11 @@ remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", - remote: "python_version_major": "3.13", + remote: "python_version_major": "3.14", remote: "python_version_origin": ".python-version", remote: "python_version_outdated": false, remote: "python_version_pinned": false, - remote: "python_version_requested": "3.13", + remote: "python_version_requested": "3.14", remote: "total_duration": [0-9.]+ remote: \\} REGEX @@ -130,8 +130,9 @@ end end - context 'when the Poetry and Python versions have changed since the last build' do - let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v275'] } + # TODO: Rename this test description back this when the Poetry version next changes. + context 'when the Python version has changed since the last build' do + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v313'] } let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_basic', buildpacks:) } it 'clears the cache before installing' do @@ -142,18 +143,17 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python 3.13 specified in .python-version + remote: -----> Using Python 3.14 specified in .python-version remote: -----> Discarding cache since: - remote: - The Python version has changed from 3.13.1 to #{LATEST_PYTHON_3_13} - remote: - The Poetry version has changed from 2.0.1 to #{POETRY_VERSION} - remote: -----> Installing Python #{LATEST_PYTHON_3_13} + remote: - The Python version has changed from 3.14.0 to #{LATEST_PYTHON_3_14} + remote: -----> Installing Python #{LATEST_PYTHON_3_14} remote: -----> Installing Poetry #{POETRY_VERSION} remote: -----> Installing dependencies using 'poetry sync --only main' remote: Installing dependencies from lock file remote: remote: Package operations: 1 install, 0 updates, 0 removals remote: - remote: - Installing typing-extensions (4.12.2) + remote: - Installing typing-extensions (4.15.0) remote: -----> Saving cache OUTPUT end @@ -172,8 +172,8 @@ remote: remote: Package operations: 4 installs, 0 updates, 0 removals remote: - remote: - Installing packaging \\(24.2\\) - remote: - Installing gunicorn \\(23.0.0 bacbf8a\\) + remote: - Installing packaging \\(25.0\\) + remote: - Installing gunicorn \\(23.0.0 56b5ad8\\) remote: - Installing local-package-pyproject-toml \\(0.0.1 /tmp/build_.+/packages/local_package_pyproject_toml\\) remote: - Installing local-package-setup-py \\(0.0.1 /tmp/build_.+/packages/local_package_setup_py\\) remote: @@ -224,7 +224,7 @@ remote: remote: Package operations: 0 installs, 3 updates, 0 removals remote: - remote: - Updating gunicorn \\(23.0.0 /app/.heroku/python/src/gunicorn -> 23.0.0 bacbf8a\\) + remote: - Updating gunicorn \\(23.0.0 /app/.heroku/python/src/gunicorn -> 23.0.0 56b5ad8\\) remote: - Updating local-package-pyproject-toml \\(0.0.1 /tmp/build_.+/packages/local_package_pyproject_toml -> 0.0.1 /tmp/build_.+/packages/local_package_pyproject_toml\\) remote: - Updating local-package-setup-py \\(0.0.1 /tmp/build_.+/packages/local_package_setup_py -> 0.0.1 /tmp/build_.+/packages/local_package_setup_py\\) remote: @@ -308,7 +308,7 @@ remote: remote: Package operations: 1 install, 0 updates, 0 removals remote: - remote: - Installing typing-extensions (4.12.2) + remote: - Installing typing-extensions (4.15.0) remote: -----> Saving cache OUTPUT end diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 6b279b3a7..a0987e59b 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -12,7 +12,7 @@ remote: -----> Installing Python #{resolved_version} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) + remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) OUTPUT else expect(clean_output(app.output)).to include(<<~OUTPUT) @@ -21,7 +21,7 @@ remote: -----> Installing Python #{resolved_version} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) + remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) OUTPUT end expect(app.run('python -V')).to eq("Python #{resolved_version}\n") @@ -278,7 +278,7 @@ remote: -----> Installing Python #{LATEST_PYTHON_3_9} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) + remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) OUTPUT expect(app.run('python -V')).to eq("Python #{LATEST_PYTHON_3_9}\n") expect($CHILD_STATUS.exitstatus).to eq(0) @@ -313,7 +313,7 @@ remote: -----> Installing Python #{LATEST_PYTHON_3_10} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) + remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) OUTPUT expect(app.run('python -V')).to eq("Python #{LATEST_PYTHON_3_10}\n") expect($CHILD_STATUS.exitstatus).to eq(0) @@ -768,7 +768,7 @@ remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) + remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) OUTPUT end end diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index ccca341cb..76471daf8 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -71,14 +71,14 @@ remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) + remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) OUTPUT end end end context 'when the stack is downgraded from Heroku-24 to Heroku-22', stacks: %w[heroku-24] do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.13') } + let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.14') } it 'clears the cache before installing again' do app.deploy do |app| @@ -88,13 +88,13 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python 3.13 specified in .python-version + remote: -----> Using Python 3.14 specified in .python-version remote: -----> Discarding cache since: remote: - The stack has changed from heroku-24 to heroku-22 - remote: -----> Installing Python #{LATEST_PYTHON_3_13} + remote: -----> Installing Python #{LATEST_PYTHON_3_14} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting typing-extensions==4.12.2 (from -r requirements.txt (line 2)) + remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) OUTPUT end end diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 1cd4121ee..5ff148819 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -15,11 +15,11 @@ remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing uv #{UV_VERSION} remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' - remote: Resolved 7 packages in .+s + remote: Resolved .+ packages in .+s remote: Prepared 1 package in .+s remote: Installed 1 package in .+s remote: Bytecode compiled 1 file in .+s - remote: \\+ typing-extensions==4.13.2 + remote: \\+ typing-extensions==4.15.0 remote: -----> Running bin/post_compile hook remote: BUILD_DIR=/tmp/build_.+ remote: CACHE_DIR=/tmp/codon/tmp/cache @@ -52,18 +52,18 @@ remote: remote: \\['', remote: '/app', - remote: '/app/.heroku/python/lib/python313.zip', - remote: '/app/.heroku/python/lib/python3.13', - remote: '/app/.heroku/python/lib/python3.13/lib-dynload', - remote: '/app/.heroku/python/lib/python3.13/site-packages'\\] + remote: '/app/.heroku/python/lib/python314.zip', + remote: '/app/.heroku/python/lib/python3.14', + remote: '/app/.heroku/python/lib/python3.14/lib-dynload', + remote: '/app/.heroku/python/lib/python3.14/site-packages'\\] remote: remote: uv #{UV_VERSION} remote: Using Python #{DEFAULT_PYTHON_FULL_VERSION} environment at: /app/.heroku/python remote: Package Version remote: ----------------- ------- - remote: typing-extensions 4.13.2 + remote: typing-extensions 4.15.0 remote: - remote: + remote: remote: remote: \\{ remote: "cache_restore_duration": [0-9.]+, @@ -79,11 +79,11 @@ remote: "pre_compile_hook": false, remote: "python_install_duration": [0-9.]+, remote: "python_version": "#{DEFAULT_PYTHON_FULL_VERSION}", - remote: "python_version_major": "3.13", + remote: "python_version_major": "3.14", remote: "python_version_origin": ".python-version", remote: "python_version_outdated": false, remote: "python_version_pinned": false, - remote: "python_version_requested": "3.13", + remote: "python_version_requested": "3.14", remote: "total_duration": [0-9.]+, remote: "uv_version": "#{UV_VERSION}" remote: \\} @@ -98,7 +98,7 @@ remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Using cached uv #{UV_VERSION} remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' - remote: Resolved 7 packages in .+s + remote: Resolved .+ packages in .+s remote: Bytecode compiled 1 file in .+s remote: -----> Running bin/post_compile hook remote: .+ @@ -126,7 +126,7 @@ end context 'when the uv and Python versions have changed since the last build' do - let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v286'] } + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v313'] } let(:app) { Hatchet::Runner.new('spec/fixtures/uv_basic', buildpacks:) } it 'clears the cache before installing' do @@ -137,18 +137,18 @@ app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python 3.13 specified in .python-version + remote: -----> Using Python 3.14 specified in .python-version remote: -----> Discarding cache since: - remote: - The Python version has changed from 3.13.3 to #{LATEST_PYTHON_3_13} - remote: - The uv version has changed from 0.7.3 to #{UV_VERSION} - remote: -----> Installing Python #{LATEST_PYTHON_3_13} + remote: - The Python version has changed from 3.14.0 to #{LATEST_PYTHON_3_14} + remote: - The uv version has changed from 0.8.23 to #{UV_VERSION} + remote: -----> Installing Python #{LATEST_PYTHON_3_14} remote: -----> Installing uv #{UV_VERSION} remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' - remote: Resolved 7 packages in .+s + remote: Resolved .+ packages in .+s remote: Prepared 1 package in .+s remote: Installed 1 package in .+s remote: Bytecode compiled 1 file in .+s - remote: \\+ typing-extensions==4.13.2 + remote: \\+ typing-extensions==4.15.0 remote: -----> Saving cache remote: -----> Discovering process types REGEX @@ -172,7 +172,7 @@ remote: Prepared 5 packages in .+s remote: Installed 5 packages in .+s remote: Bytecode compiled .+ files in .+s - remote: \\+ gunicorn==23.0.0 \\(from git\\+https://github.com/benoitc/gunicorn@a86ea1e4e6c271d1cd1823c7e14490123f9238fe\\) + remote: \\+ gunicorn==23.0.0 \\(from git\\+https://github.com/benoitc/gunicorn@56b5ad87f8d72a674145c273ed8f547513c2b409\\) remote: \\+ local-package-pyproject-toml==0.0.1 \\(from file:///tmp/build_.+/packages/local_package_pyproject_toml\\) remote: \\+ local-package-setup-py==0.0.1 \\(from file:///tmp/build_.+/packages/local_package_setup_py\\) remote: \\+ packaging==25.0 @@ -299,7 +299,7 @@ remote: Prepared 1 package in .+s remote: Installed 1 package in .+s remote: Bytecode compiled 1 file in .+s - remote: \\+ typing-extensions==4.13.2 + remote: \\+ typing-extensions==4.15.0 remote: -----> Saving cache REGEX end @@ -433,8 +433,8 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' - remote: Using CPython #{DEFAULT_PYTHON_FULL_VERSION} interpreter at: /app/.heroku/python/bin/python#{DEFAULT_PYTHON_MAJOR_VERSION} - remote: error: The Python request from `.python-version` resolved to Python #{DEFAULT_PYTHON_FULL_VERSION}, which is incompatible with the project's Python requirement: `==3.12.*` (from `project.requires-python`) + remote: Using CPython #{LATEST_PYTHON_3_13} interpreter at: /app/.heroku/python/bin/python3.13 + remote: error: The Python request from `.python-version` resolved to Python #{LATEST_PYTHON_3_13}, which is incompatible with the project's Python requirement: `==3.12.*` (from `project.requires-python`) remote: Use `uv python pin` to update the `.python-version` file to a compatible version remote: remote: ! Error: Unable to install dependencies using uv. diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index b162130ae..a4e2eed2d 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -16,7 +16,7 @@ LATEST_PYTHON_3_12 = '3.12.12' LATEST_PYTHON_3_13 = '3.13.10' LATEST_PYTHON_3_14 = '3.14.1' -DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_13 +DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_14 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') # The requirement versions are effectively buildpack constants, however, we want From 65a57e9d85db97cb216aabe3bc9f976042dd36ec Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 5 Dec 2025 21:25:26 +0000 Subject: [PATCH 297/406] Add support for Python 3.14.2 and 3.13.11 (#1985) Changelogs: https://docs.python.org/release/3.14.2/whatsnew/changelog.html#python-3-14-2-final https://docs.python.org/release/3.13.11/whatsnew/changelog.html#python-3-13-11-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/19975436334 https://github.com/heroku/heroku-buildpack-python/actions/runs/19975433926 GUS-W-20450666. GUS-W-20450669. --- CHANGELOG.md | 2 ++ lib/python_version.sh | 4 ++-- spec/spec_helper.rb | 4 ++-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cd661b357..a6adb0edb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,8 @@ ## [Unreleased] - Changed the default Python version for new apps from 3.13 to 3.14. ([#1984](https://github.com/heroku/heroku-buildpack-python/pull/1984)) +- The Python 3.14 version alias now resolves to Python 3.14.2. ([#1985](https://github.com/heroku/heroku-buildpack-python/pull/1985)) +- The Python 3.13 version alias now resolves to Python 3.13.11. ([#1985](https://github.com/heroku/heroku-buildpack-python/pull/1985)) ## [v322] - 2025-12-02 diff --git a/lib/python_version.sh b/lib/python_version.sh index 3eb24691c..0e00e4ca3 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -8,8 +8,8 @@ LATEST_PYTHON_3_9="3.9.25" LATEST_PYTHON_3_10="3.10.19" LATEST_PYTHON_3_11="3.11.14" LATEST_PYTHON_3_12="3.12.12" -LATEST_PYTHON_3_13="3.13.10" -LATEST_PYTHON_3_14="3.14.1" +LATEST_PYTHON_3_13="3.13.11" +LATEST_PYTHON_3_14="3.14.2" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=9 NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=14 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index a4e2eed2d..cf389801d 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -14,8 +14,8 @@ LATEST_PYTHON_3_10 = '3.10.19' LATEST_PYTHON_3_11 = '3.11.14' LATEST_PYTHON_3_12 = '3.12.12' -LATEST_PYTHON_3_13 = '3.13.10' -LATEST_PYTHON_3_14 = '3.14.1' +LATEST_PYTHON_3_13 = '3.13.11' +LATEST_PYTHON_3_14 = '3.14.2' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_14 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From 40e6dfa5f0817d60e50f25feb6ba3ce3af2639e7 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Fri, 5 Dec 2025 22:06:21 +0000 Subject: [PATCH 298/406] Prepare release v323 (#1986) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a6adb0edb..8c7dc7986 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v323] - 2025-12-05 + - Changed the default Python version for new apps from 3.13 to 3.14. ([#1984](https://github.com/heroku/heroku-buildpack-python/pull/1984)) - The Python 3.14 version alias now resolves to Python 3.14.2. ([#1985](https://github.com/heroku/heroku-buildpack-python/pull/1985)) - The Python 3.13 version alias now resolves to Python 3.13.11. ([#1985](https://github.com/heroku/heroku-buildpack-python/pull/1985)) @@ -1419,7 +1422,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v322...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v323...main +[v323]: https://github.com/heroku/heroku-buildpack-python/compare/v322...v323 [v322]: https://github.com/heroku/heroku-buildpack-python/compare/v321...v322 [v321]: https://github.com/heroku/heroku-buildpack-python/compare/v320...v321 [v320]: https://github.com/heroku/heroku-buildpack-python/compare/v319...v320 From f3b29ee5af1bf2f51c4c51632978429f5662e920 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 8 Dec 2025 11:30:35 +0000 Subject: [PATCH 299/406] Stop adding the current working directory to `PATH` (#1987) Currently for historic reasons the buildpack adds the current working directory to `PATH`, due to adding an empty colon-delimited segment before the default Ubuntu `PATH`. This can cause errors if the app has a script/binary in its root directory that has the same name as a command the buildpack tries to run. For example, on Honeycomb I've seen internal errors when the buildpack attempts to locate the bundled pip, due to the `tail` command being run not being the command we expected: ``` -----> Installing pip 25.2, setuptools 70.3.0 and wheel 0.45.1 ./tail: line 1: heroku: command not found ! Internal Error: Unable to locate the Python stdlib's bundled pip. ``` Removing this customisation of `PATH` is a breaking change, but safer in the long run (and also matches the Python CNB behaviour). Any apps that call a script or binary that's in the root directory of their app from `bin/pre_compile` or `bin/post_compile` hook (or any other subprocess called by the buildpack) will now need to do so via an explicit relative path. For example, use `./my_command` not `my_command` when calling the script/binary. This change doesn't affect: - app behaviour at run-time (only during the build) - apps that call scripts in a subdirectory (eg `foo/my_command`) - apps that already use a relative path for scripts in the app's root directory (eg `./my_command`) GUS-W-20472356. --- CHANGELOG.md | 1 + bin/compile | 4 ---- spec/fixtures/pip_basic/cat | 4 ++++ spec/hatchet/ci_spec.rb | 8 ++++---- spec/hatchet/django_spec.rb | 4 ++-- spec/hatchet/hooks_spec.rb | 2 +- spec/hatchet/pip_spec.rb | 2 +- spec/hatchet/pipenv_spec.rb | 2 +- spec/hatchet/poetry_spec.rb | 2 +- spec/hatchet/uv_spec.rb | 2 +- 10 files changed, 16 insertions(+), 15 deletions(-) create mode 100755 spec/fixtures/pip_basic/cat diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c7dc7986..70c04f6f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Stopped adding the current working directory to `PATH`. ([#1987](https://github.com/heroku/heroku-buildpack-python/pull/1987)) ## [v323] - 2025-12-05 diff --git a/bin/compile b/bin/compile index 345548aa7..443a6b97b 100755 --- a/bin/compile +++ b/bin/compile @@ -46,10 +46,6 @@ trap 'utils::err_trap' ERR checks::ensure_supported_stack "${STACK:?"Required env var STACK isn't set"}" checks::duplicate_python_buildpack "${BUILD_DIR}" -# Prepend proper path for old-school virtualenv hackery. -# This may not be necessary. -export PATH=":/usr/local/bin:${PATH}" - # Exported for use in subshells, such as the steps run via sub_env. export BUILD_DIR CACHE_DIR ENV_DIR diff --git a/spec/fixtures/pip_basic/cat b/spec/fixtures/pip_basic/cat new file mode 100755 index 000000000..d213c595e --- /dev/null +++ b/spec/fixtures/pip_basic/cat @@ -0,0 +1,4 @@ +#!/usr/bin/env bash + +echo "This custom script shouldn't take precedence over /usr/bin/cat!" >&2 +exit 1 diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 4f01a43d2..2cc0b9acc 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -32,7 +32,7 @@ LC_ALL=C.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib - PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PIP_DISABLE_PIP_VERSION_CHECK=1 PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 @@ -121,7 +121,7 @@ LC_ALL=C.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib - PATH=/app/.heroku/python/pipenv/bin:/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PATH=/app/.heroku/python/pipenv/bin:/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PIPENV_SYSTEM=1 PIPENV_VERBOSITY=-1 PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config @@ -221,7 +221,7 @@ LC_ALL=C.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib - PATH=/tmp/cache\\w+/.heroku/python-poetry/bin:/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PATH=/tmp/cache\\w+/.heroku/python-poetry/bin:/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config POETRY_VIRTUALENVS_CREATE=false POETRY_VIRTUALENVS_USE_POETRY_PYTHON=true @@ -317,7 +317,7 @@ LC_ALL=C.UTF-8 LD_LIBRARY_PATH=/app/.heroku/python/lib LIBRARY_PATH=/app/.heroku/python/lib - PATH=/tmp/cache\\w+/.heroku/python-uv:/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ + PATH=/tmp/cache\\w+/.heroku/python-uv:/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 UV_NO_MANAGED_PYTHON=1 diff --git a/spec/hatchet/django_spec.rb b/spec/hatchet/django_spec.rb index d68ca4fda..8c613896c 100644 --- a/spec/hatchet/django_spec.rb +++ b/spec/hatchet/django_spec.rb @@ -42,7 +42,7 @@ remote: 'LANG': 'en_US.UTF-8', remote: 'LD_LIBRARY_PATH': '/app/.heroku/python/lib', remote: 'LIBRARY_PATH': '/app/.heroku/python/lib', - remote: 'PATH': '/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', + remote: 'PATH': '/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', remote: 'PIP_DISABLE_PIP_VERSION_CHECK': '1', remote: 'PKG_CONFIG_PATH': '/app/.heroku/python/lib/pkg-config', remote: 'PWD': '/tmp/build_\\w+', @@ -84,7 +84,7 @@ remote: 'LANG': 'en_US.UTF-8', remote: 'LD_LIBRARY_PATH': '/app/.heroku/python/lib', remote: 'LIBRARY_PATH': '/app/.heroku/python/lib', - remote: 'PATH': '/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', + remote: 'PATH': '/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', remote: 'PIP_DISABLE_PIP_VERSION_CHECK': '1', remote: 'PKG_CONFIG_PATH': '/invalid-path', remote: 'PWD': '/tmp/build_\\w+', diff --git a/spec/hatchet/hooks_spec.rb b/spec/hatchet/hooks_spec.rb index 18fa2eed6..8cf91df8d 100644 --- a/spec/hatchet/hooks_spec.rb +++ b/spec/hatchet/hooks_spec.rb @@ -27,7 +27,7 @@ remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/python/lib remote: LIBRARY_PATH=/app/.heroku/python/lib - remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PATH=/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config remote: PYTHONUNBUFFERED=1 remote: SOME_APP_CONFIG_VAR=1 diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 2d8ec44bb..67a0f1674 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -29,7 +29,7 @@ remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/python/lib remote: LIBRARY_PATH=/app/.heroku/python/lib - remote: PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PATH=/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PIP_DISABLE_PIP_VERSION_CHECK=1 remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config remote: PYTHONUNBUFFERED=1 diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 49bc4dfd5..99005766a 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -25,7 +25,7 @@ remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/python/lib remote: LIBRARY_PATH=/app/.heroku/python/lib - remote: PATH=/app/.heroku/python/pipenv/bin:/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PATH=/app/.heroku/python/pipenv/bin:/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PIPENV_SYSTEM=1 remote: PIPENV_VERBOSITY=-1 remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 932c7bf46..b963b444a 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -29,7 +29,7 @@ remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/python/lib remote: LIBRARY_PATH=/app/.heroku/python/lib - remote: PATH=/tmp/codon/tmp/cache/.heroku/python-poetry/bin:/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PATH=/tmp/codon/tmp/cache/.heroku/python-poetry/bin:/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config remote: POETRY_VIRTUALENVS_CREATE=false remote: POETRY_VIRTUALENVS_USE_POETRY_PYTHON=true diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 5ff148819..8f73d00b2 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -29,7 +29,7 @@ remote: LANG=en_US.UTF-8 remote: LD_LIBRARY_PATH=/app/.heroku/python/lib remote: LIBRARY_PATH=/app/.heroku/python/lib - remote: PATH=/tmp/codon/tmp/cache/.heroku/python-uv:/app/.heroku/python/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + remote: PATH=/tmp/codon/tmp/cache/.heroku/python-uv:/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config remote: PYTHONUNBUFFERED=1 remote: UV_CACHE_DIR=/tmp/uv-cache From 8997ea7429d278f1b0ee0d634952f0c399a8130c Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 8 Dec 2025 18:00:26 +0000 Subject: [PATCH 300/406] Convert the existing virtual environment warning to an error (#1990) In #1890 a warning was added if an existing Python virtual environment was found at `.venv/` or `venv/` in the root of the app dir, for the reasons described in that PR. Now that 3 months have passed, this warning is being converted to an error. Users with an inadvertently committed venv will need to follow the instructions in the error message to remove the venv directory from the Git index and add it to their `.gitignore` file. GUS-W-19127259. --- CHANGELOG.md | 1 + bin/compile | 11 ++++++++--- lib/checks.sh | 32 +++++++++++++++++++++++--------- spec/hatchet/checks_spec.rb | 32 ++++++++++++++++++++++---------- 4 files changed, 54 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 70c04f6f2..3d18e3988 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- The build now errors if an existing Python virtual environment is found in the build directory at `.venv/` or `venv/`. This replaces the warning displayed since September 2025. ([#1990](https://github.com/heroku/heroku-buildpack-python/pull/1990)) - Stopped adding the current working directory to `PATH`. ([#1987](https://github.com/heroku/heroku-buildpack-python/pull/1987)) ## [v323] - 2025-12-05 diff --git a/bin/compile b/bin/compile index 443a6b97b..966c032a6 100755 --- a/bin/compile +++ b/bin/compile @@ -92,14 +92,19 @@ cd "${BUILD_DIR}" # Runs a `bin/pre_compile` script if found in the app source, allowing build customisation. hooks::run_hook "pre_compile" -# These checks are after the pre_compile hook, so that we can check not only the original -# app source, but also that the hook hasn't written to these locations either. +# These later checks are after the pre_compile hook so that we can check not only the +# original app source, but also that the hook hasn't written to these locations either. checks::existing_python_dir_present "${BUILD_DIR}" -checks::existing_venv_dir_present "${BUILD_DIR}" package_manager="$(package_manager::determine_package_manager "${BUILD_DIR}")" build_data::set_string "package_manager" "${package_manager}" +# This check is after the package manager step to improve the UX when the package manager file +# is missing and the venv directory was the only reason the buildpack passed detection. +# (Since once they remove the venv detection will fail, which would prevent us from being able +# to display the more helpful "missing package manager file" error message on the next build). +checks::existing_venv_dir_present "${BUILD_DIR}" + cached_python_full_version="$(cache::cached_python_full_version "${CACHE_DIR}")" # We use the Bash 4.3+ `nameref` feature to pass back multiple values from this function diff --git a/lib/checks.sh b/lib/checks.sh index 65c6727c9..e6f222954 100644 --- a/lib/checks.sh +++ b/lib/checks.sh @@ -109,8 +109,8 @@ function checks::existing_venv_dir_present() { for venv_name in ".venv" "venv"; do if [[ -f "${build_dir}/${venv_name}/pyvenv.cfg" ]]; then - output::warning <<-EOF - Warning: Existing '${venv_name}/' directory found. + output::error <<-EOF + Error: Existing '${venv_name}/' directory found. Your app's source code contains an existing directory named '${venv_name}/', which looks like a Python virtual environment: @@ -122,17 +122,31 @@ function checks::existing_venv_dir_present() { single machine and so won't work when run somewhere else. If you've committed a '${venv_name}/' directory to your Git repo, you - must delete it and add the directory to your .gitignore file: + must delete it and add the directory to your .gitignore file. + + To do this: + 1. Run 'git rm --cached -r ${venv_name}/' to remove the directory + from the Git index. + 2. Create a '.gitignore' file in the root of your repository + if it doesn't already exist. + 3. Add the '${venv_name}/' directory to the .gitignore file as a + new entry on its own line (don't include the quotes). + 4. Stage the change using 'git add .gitignore' and then + 'git commit' all changes. + + For more information, see: https://docs.github.com/en/get-started/git-basics/ignoring-files - If the directory was created by a 'bin/pre_compile' hook or an - earlier buildpack, you must update them to create the virtual - environment in a different location. + If the directory was created by a 'bin/pre_compile' hook or + an earlier buildpack, you must instead update them to create + the virtual environment in a different location. - In future versions of the buildpack, this warning will be turned - into an error. + Note: This error replaces the previous warning which had been + displayed in build logs since 2nd September 2025. EOF - build_data::set_string "existing_venv_dir_present" "${venv_name}" + build_data::set_string "failure_reason" "checks::existing-venv-dir" + build_data::set_string "failure_detail" "${venv_name}" + exit 1 fi done } diff --git a/spec/hatchet/checks_spec.rb b/spec/hatchet/checks_spec.rb index a7acf46da..bf8bbdc17 100644 --- a/spec/hatchet/checks_spec.rb +++ b/spec/hatchet/checks_spec.rb @@ -72,14 +72,14 @@ end context 'when the app source contains a venv directory' do - let(:app) { Hatchet::Runner.new('spec/fixtures/venv_in_app_source') } + let(:app) { Hatchet::Runner.new('spec/fixtures/venv_in_app_source', allow_failure: true) } - it 'outputs a warning before continuing to build' do + it 'fails the build with an informative error message' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: - remote: ! Warning: Existing '.venv/' directory found. + remote: ! Error: Existing '.venv/' directory found. remote: ! remote: ! Your app's source code contains an existing directory named remote: ! '.venv/', which looks like a Python virtual environment: @@ -94,17 +94,29 @@ remote: ! single machine and so won't work when run somewhere else. remote: ! remote: ! If you've committed a '.venv/' directory to your Git repo, you - remote: ! must delete it and add the directory to your .gitignore file: + remote: ! must delete it and add the directory to your .gitignore file. + remote: ! + remote: ! To do this: + remote: ! 1. Run 'git rm --cached -r .venv/' to remove the directory + remote: ! from the Git index. + remote: ! 2. Create a '.gitignore' file in the root of your repository + remote: ! if it doesn't already exist. + remote: ! 3. Add the '.venv/' directory to the .gitignore file as a + remote: ! new entry on its own line (don't include the quotes). + remote: ! 4. Stage the change using 'git add .gitignore' and then + remote: ! 'git commit' all changes. + remote: ! + remote: ! For more information, see: remote: ! https://docs.github.com/en/get-started/git-basics/ignoring-files remote: ! - remote: ! If the directory was created by a 'bin/pre_compile' hook or an - remote: ! earlier buildpack, you must update them to create the virtual - remote: ! environment in a different location. + remote: ! If the directory was created by a 'bin/pre_compile' hook or + remote: ! an earlier buildpack, you must instead update them to create + remote: ! the virtual environment in a different location. remote: ! - remote: ! In future versions of the buildpack, this warning will be turned - remote: ! into an error. + remote: ! Note: This error replaces the previous warning which had been + remote: ! displayed in build logs since 2nd September 2025. remote: - remote: -----> Using Python 3.14 specified in .python-version + remote: ! Push rejected, failed to compile Python app. OUTPUT end end From b9c2dc02ca0ba459a4546108d57aba5e9abde5e3 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 9 Dec 2025 11:00:27 +0000 Subject: [PATCH 301/406] Prepare release v324 (#1991) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d18e3988..be7d1a7e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v324] - 2025-12-09 + - The build now errors if an existing Python virtual environment is found in the build directory at `.venv/` or `venv/`. This replaces the warning displayed since September 2025. ([#1990](https://github.com/heroku/heroku-buildpack-python/pull/1990)) - Stopped adding the current working directory to `PATH`. ([#1987](https://github.com/heroku/heroku-buildpack-python/pull/1987)) @@ -1424,7 +1427,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v323...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v324...main +[v324]: https://github.com/heroku/heroku-buildpack-python/compare/v323...v324 [v323]: https://github.com/heroku/heroku-buildpack-python/compare/v322...v323 [v322]: https://github.com/heroku/heroku-buildpack-python/compare/v321...v322 [v321]: https://github.com/heroku/heroku-buildpack-python/compare/v320...v321 From 6d06c3b58909859ccaed55e2d388d821caf448e9 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 9 Dec 2025 15:13:16 +0000 Subject: [PATCH 302/406] Sunset implicit setup.py support (#1992) Previously if an app had only a `setup.py` file and no Python package manager files (such as `requirements.txt`, `Pipfile.lock`, `Poetry.lock` or `uv.lock`), the buildpack would install the project using `pip install --editable .` However, this implicit fallback doesn't make sense now that the buildpack supports multiple package managers, and so has to guess which package manager to use, and whether to install the project in editable mode or not. As such, in #1897 this fallback was deprecated, and is now being sunset. This also brings the classic Python buildpack's behaviour in line with the Python CNB. Apps that only have a `setup.py` file will now need to add an explicit `requirements.txt` file containing: ``` --editable . ``` GUS-W-19275444. --- CHANGELOG.md | 1 + lib/cache.sh | 7 +-- lib/package_manager.sh | 58 ++++++++++--------- lib/pip.sh | 8 +-- .../multiple_package_managers/setup.py | 1 + spec/fixtures/pip_basic/setup.py | 3 +- spec/fixtures/pipenv_basic/setup.py | 2 - spec/fixtures/poetry_basic/setup.py | 2 - spec/fixtures/setup_py_only/setup.py | 6 -- spec/hatchet/package_manager_spec.rb | 46 +++++++-------- spec/hatchet/pipenv_spec.rb | 2 +- 11 files changed, 57 insertions(+), 79 deletions(-) delete mode 100644 spec/fixtures/pipenv_basic/setup.py delete mode 100644 spec/fixtures/poetry_basic/setup.py diff --git a/CHANGELOG.md b/CHANGELOG.md index be7d1a7e9..1afc4010f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Sunset the previously deprecated support for falling back to installing dependencies from a `setup.py` file if no Python package manager files were found. ([#1992](https://github.com/heroku/heroku-buildpack-python/pull/1992)) ## [v324] - 2025-12-09 diff --git a/lib/cache.sh b/lib/cache.sh index 61e0e7e9e..dae064b53 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -74,9 +74,8 @@ function cache::restore() { # installed Python packages. For example, if a package entry in a requirements file is # later removed, pip will not uninstall the package. This check can be removed if we # ever switch to only caching pip's HTTP/wheel cache rather than site-packages. - # TODO: Remove the `-f` check once the setup.py fallback feature is removed. # TODO: Switch this to using sha256sum like the Pipenv implementation. - if [[ -f "${build_dir}/requirements.txt" ]] && ! cmp --silent "${cache_dir}/.heroku/requirements.txt" "${build_dir}/requirements.txt"; then + if ! cmp --silent "${cache_dir}/.heroku/requirements.txt" "${build_dir}/requirements.txt"; then cache_invalidation_reasons+=("The contents of requirements.txt changed") fi ;; @@ -202,9 +201,7 @@ function cache::save() { # We continue to use that format so that the file can be read by older buildpack versions. echo "python-${python_full_version}" >"${cache_dir}/.heroku/python-version" - # TODO: Simplify this once multiple package manager files being found is turned into an - # error and the setup.py fallback feature is removed. - if [[ "${package_manager}" == "pip" && -f "${build_dir}/requirements.txt" ]]; then + if [[ "${package_manager}" == "pip" ]]; then # TODO: Switch this to using sha256sum like the Pipenv implementation. cp "${build_dir}/requirements.txt" "${cache_dir}/.heroku/" elif [[ "${package_manager}" == "pipenv" ]]; then diff --git a/lib/package_manager.sh b/lib/package_manager.sh index b9e8d1dbf..039b95262 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -59,34 +59,6 @@ function package_manager::determine_package_manager() { package_managers_found_display_text+=("uv.lock (uv)") fi - if ((${#package_managers_found[@]} == 0)) && [[ -f "${build_dir}/setup.py" ]]; then - package_managers_found+=(pip) - package_managers_found_display_text+=("setup.py (pip)") - output::warning <<-EOF - Warning: Implicit setup.py file support is deprecated. - - Your app currently only has a setup.py file and no Python - package manager files. This means that the buildpack has - to guess which package manager you want to use and also - whether to install your project in editable mode or not. - - For now, we will use pip to install your dependencies in - editable mode, however, this fallback is deprecated and - will be removed in the future. - - Please add an explicit package manager file to your app. - - To continue using pip in editable mode, create a new file - in the root directory of your app named 'requirements.txt' - containing the requirement '--editable .' (without quotes). - - Alternatively, if you wish to switch to another package - manager, we highly recommend uv: - https://docs.astral.sh/uv/ - EOF - build_data::set_raw "setup_py_only" "true" - fi - local num_package_managers_found=${#package_managers_found[@]} case "${num_package_managers_found}" in @@ -95,6 +67,36 @@ function package_manager::determine_package_manager() { return 0 ;; 0) + if [[ -f "${build_dir}/setup.py" ]]; then + output::error <<-EOF + Error: Implicit setup.py file support has been sunset. + + Your app currently only has a setup.py file and no Python + package manager files. This means that the buildpack can't + tell which package manager you want to use, and whether to + install your project in editable mode or not. + + Previously the buildpack guessed and used pip to install your + dependencies in editable mode. However, this fallback was + deprecated in September 2025 and has now been sunset. + + You must now add an explicit package manager file to your app, + such as a requirements.txt, poetry.lock or uv.lock file. + + To continue using your setup.py file with pip in editable + mode, create a new file in the root directory of your app + named 'requirements.txt' containing the requirement + '--editable .' (without quotes). + + Alternatively, if you wish to switch to another package + manager, we recommend uv, since it supports lockfiles, is + faster, and is actively maintained by a full-time team: + https://docs.astral.sh/uv/ + EOF + build_data::set_string "failure_reason" "package-manager::setup-py-only" + exit 1 + fi + output::error <<-EOF Error: Couldn't find any supported Python package manager files. diff --git a/lib/pip.sh b/lib/pip.sh index 829e34db0..edc049968 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -107,15 +107,9 @@ function pip::install_dependencies() { local pip_install_command=( pip install + -r requirements.txt ) - # Support for the setup.py fallback is deprecated and will be removed in the future. - if [[ -f setup.py && ! -f requirements.txt ]]; then - pip_install_command+=(--editable .) - else - pip_install_command+=(-r requirements.txt) - fi - # Install test dependencies too when the buildpack is invoked via `bin/test-compile` on Heroku CI. # We install both requirements files at the same time to allow pip to resolve version conflicts. if [[ -v INSTALL_TEST && -f requirements-test.txt ]]; then diff --git a/spec/fixtures/multiple_package_managers/setup.py b/spec/fixtures/multiple_package_managers/setup.py index e69de29bb..ab45d8b72 100644 --- a/spec/fixtures/multiple_package_managers/setup.py +++ b/spec/fixtures/multiple_package_managers/setup.py @@ -0,0 +1 @@ +# This file tests that the setup.py sunset error is not shown when other package manager files exist. diff --git a/spec/fixtures/pip_basic/setup.py b/spec/fixtures/pip_basic/setup.py index b5dc486bd..ab45d8b72 100644 --- a/spec/fixtures/pip_basic/setup.py +++ b/spec/fixtures/pip_basic/setup.py @@ -1,2 +1 @@ -# This tests that the setup.py fallback is not used when other package manager files exist. -raise RuntimeError("setup.py should not be run!") +# This file tests that the setup.py sunset error is not shown when other package manager files exist. diff --git a/spec/fixtures/pipenv_basic/setup.py b/spec/fixtures/pipenv_basic/setup.py deleted file mode 100644 index b5dc486bd..000000000 --- a/spec/fixtures/pipenv_basic/setup.py +++ /dev/null @@ -1,2 +0,0 @@ -# This tests that the setup.py fallback is not used when other package manager files exist. -raise RuntimeError("setup.py should not be run!") diff --git a/spec/fixtures/poetry_basic/setup.py b/spec/fixtures/poetry_basic/setup.py deleted file mode 100644 index b5dc486bd..000000000 --- a/spec/fixtures/poetry_basic/setup.py +++ /dev/null @@ -1,2 +0,0 @@ -# This tests that the setup.py fallback is not used when other package manager files exist. -raise RuntimeError("setup.py should not be run!") diff --git a/spec/fixtures/setup_py_only/setup.py b/spec/fixtures/setup_py_only/setup.py index 8d636d69a..e69de29bb 100644 --- a/spec/fixtures/setup_py_only/setup.py +++ b/spec/fixtures/setup_py_only/setup.py @@ -1,6 +0,0 @@ -from setuptools import setup - -setup( - name='test', - install_requires=['six'], -) diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index da99ac2e6..63cf23fcb 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -49,46 +49,40 @@ end end - # This case will be turned into an error in the future. context 'when there is only a setup.py' do - let(:app) { Hatchet::Runner.new('spec/fixtures/setup_py_only') } + let(:app) { Hatchet::Runner.new('spec/fixtures/setup_py_only', allow_failure: true) } - it 'installs packages from setup.py using pip' do + it 'fails the build with an informative error message' do app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) + expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: - remote: ! Warning: Implicit setup.py file support is deprecated. + remote: ! Error: Implicit setup.py file support has been sunset. remote: ! remote: ! Your app currently only has a setup.py file and no Python - remote: ! package manager files. This means that the buildpack has - remote: ! to guess which package manager you want to use and also - remote: ! whether to install your project in editable mode or not. + remote: ! package manager files. This means that the buildpack can't + remote: ! tell which package manager you want to use, and whether to + remote: ! install your project in editable mode or not. remote: ! - remote: ! For now, we will use pip to install your dependencies in - remote: ! editable mode, however, this fallback is deprecated and - remote: ! will be removed in the future. + remote: ! Previously the buildpack guessed and used pip to install your + remote: ! dependencies in editable mode. However, this fallback was + remote: ! deprecated in September 2025 and has now been sunset. remote: ! - remote: ! Please add an explicit package manager file to your app. + remote: ! You must now add an explicit package manager file to your app, + remote: ! such as a requirements.txt, poetry.lock or uv.lock file. remote: ! - remote: ! To continue using pip in editable mode, create a new file - remote: ! in the root directory of your app named 'requirements.txt' - remote: ! containing the requirement '--editable .' \\(without quotes\\). + remote: ! To continue using your setup.py file with pip in editable + remote: ! mode, create a new file in the root directory of your app + remote: ! named 'requirements.txt' containing the requirement + remote: ! '--editable .' (without quotes). remote: ! remote: ! Alternatively, if you wish to switch to another package - remote: ! manager, we highly recommend uv: + remote: ! manager, we recommend uv, since it supports lockfiles, is + remote: ! faster, and is actively maintained by a full-time team: remote: ! https://docs.astral.sh/uv/ remote: - remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version - remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing pip #{PIP_VERSION} - remote: -----> Installing dependencies using 'pip install --editable .' - remote: Obtaining file:///tmp/build_.* - remote: .+ - remote: Installing collected packages: six, test - remote: Successfully installed six-.+ test-0.0.0 - remote: -----> Saving cache - REGEX + remote: ! Push rejected, failed to compile Python app. + OUTPUT end end end diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 99005766a..d18344d34 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -291,7 +291,7 @@ context 'without a Pipfile.lock' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_no_lockfile', allow_failure: true) } - it 'builds with the default Python version using just the Pipfile' do + it 'fails the build with an informative error message' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected From b66b202f8bb3f446a87f471ea564476fc945fd48 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 10 Dec 2025 13:28:10 +0000 Subject: [PATCH 303/406] Error when multiple package manager files are found (#1993) Currently the buildpack will use whichever package manager it finds first, if the files of multiple package managers are found. This occasionally results in support tickets where the user believes the build to not be installing dependencies correctly, when in fact they are adding dependencies to the wrong package manager file. It also means historic users of third-party Poetry and uv buildpacks might not realise that the buildpack now natively supports those package managers (in a much more efficient way), if they missed our release announcements. As such, in November 2024 in #1692 a warning was added, which we're now converting to an error. Users with multiple package manager files committed to their Git repo will need to pick one package manager and delete the files relating to the others. And users who are still using the third-party Poetry or uv buildpacks will need to remove those buildpacks from their app: https://devcenter.heroku.com/articles/managing-buildpacks#remove-classic-buildpacks Closes #1691. GUS-W-18915632. --- CHANGELOG.md | 1 + lib/package_manager.sh | 56 ++++++------------- .../multiple_package_managers/.python-version | 1 - .../multiple_package_managers/Pipfile | 6 -- .../multiple_package_managers/Pipfile.lock | 18 ------ spec/hatchet/package_manager_spec.rb | 54 +++++++----------- 6 files changed, 38 insertions(+), 98 deletions(-) delete mode 100644 spec/fixtures/multiple_package_managers/.python-version diff --git a/CHANGELOG.md b/CHANGELOG.md index 1afc4010f..54cc217a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- The build now errors if the files for multiple package managers are found. This replaces the warning displayed since November 2024. ([#1993](https://github.com/heroku/heroku-buildpack-python/pull/1993)) - Sunset the previously deprecated support for falling back to installing dependencies from a `setup.py` file if no Python package manager files were found. ([#1992](https://github.com/heroku/heroku-buildpack-python/pull/1992)) ## [v324] - 2025-12-09 diff --git a/lib/package_manager.sh b/lib/package_manager.sh index 039b95262..95c4868b6 100644 --- a/lib/package_manager.sh +++ b/lib/package_manager.sh @@ -44,11 +44,6 @@ function package_manager::determine_package_manager() { package_managers_found_display_text+=("requirements.txt (pip)") fi - # This must be after the requirements.txt check, so that the requirements.txt exported by - # `python-poetry-buildpack` takes precedence over poetry.lock, for consistency with the - # behaviour prior to this buildpack supporting Poetry natively. In the future the presence - # of multiple package manager files will be turned into an error, at which point the - # ordering here won't matter. if [[ -f "${build_dir}/poetry.lock" ]]; then package_managers_found+=(poetry) package_managers_found_display_text+=("poetry.lock (Poetry)") @@ -132,57 +127,38 @@ function package_manager::determine_package_manager() { exit 1 ;; *) - # If multiple package managers are found, use the first found above. - # TODO: Turn this case into an error since it results in support tickets from users - # who don't realise they have multiple package manager files and think their changes - # aren't taking effect. (We'll need to wait until after Poetry support has landed, - # and people have had a chance to migrate from the Poetry buildpack mentioned above.) - echo "${package_managers_found[0]}" - - output::warning <<-EOF - Warning: Multiple Python package manager files were found. + output::error <<-EOF + Error: Multiple Python package manager files were found. Exactly one package manager file should be present in your app's source code, however, several were found: $(printf -- "%s\n" "${package_managers_found_display_text[@]}") - For now, we will build your app using the first package manager - listed above, however, in the future this warning will become - an error. + Previously, the buildpack guessed which package manager to use + and installed your dependencies with the first package manager + listed above. However, this implicit behaviour was deprecated + in November 2024 and is now no longer supported. - Decide which package manager you want to use with your app, and - then delete the file(s) and any config from the others. + You must decide which package manager you want to use with your + app, and then delete the file(s) and any config from the others. If you aren't sure which package manager to use, we recommend trying uv, since it supports lockfiles, is extremely fast, and is actively maintained by a full-time team: https://docs.astral.sh/uv/ - EOF - - if [[ "${package_managers_found[*]}" == *"poetry"* ]]; then - output::notice <<-EOF - Note: We recently added support for the package manager Poetry. - If you are using a third-party Poetry buildpack you must remove - it, otherwise the requirements.txt file it generates will cause - the warning above. - EOF - fi - if [[ "${package_managers_found[*]}" == *"uv"* ]]; then - output::notice <<-EOF - Note: We recently added support for the package manager uv. - If you are using a third-party uv buildpack you must remove - it, otherwise the requirements.txt file it generates will cause - the warning above. - EOF - fi - - build_data::set_string "package_manager_multiple_found" "$( + Note: If you use a third-party uv or Poetry buildpack, you must + remove it from your app, since it's no longer required and the + requirements.txt file it generates will trigger this error. See: + https://devcenter.heroku.com/articles/managing-buildpacks#remove-classic-buildpacks + EOF + build_data::set_string "failure_reason" "package-manager::multiple-found" + build_data::set_string "failure_detail" "$( IFS=, echo "${package_managers_found[*]}" )" - return 0 + exit 1 ;; esac } diff --git a/spec/fixtures/multiple_package_managers/.python-version b/spec/fixtures/multiple_package_managers/.python-version deleted file mode 100644 index 6324d401a..000000000 --- a/spec/fixtures/multiple_package_managers/.python-version +++ /dev/null @@ -1 +0,0 @@ -3.14 diff --git a/spec/fixtures/multiple_package_managers/Pipfile b/spec/fixtures/multiple_package_managers/Pipfile index 6a061ae6a..e69de29bb 100644 --- a/spec/fixtures/multiple_package_managers/Pipfile +++ b/spec/fixtures/multiple_package_managers/Pipfile @@ -1,6 +0,0 @@ -[[source]] -url = "https://pypi.org/simple" -verify_ssl = true -name = "pypi" - -[packages] diff --git a/spec/fixtures/multiple_package_managers/Pipfile.lock b/spec/fixtures/multiple_package_managers/Pipfile.lock index d2c2cba4c..e69de29bb 100644 --- a/spec/fixtures/multiple_package_managers/Pipfile.lock +++ b/spec/fixtures/multiple_package_managers/Pipfile.lock @@ -1,18 +0,0 @@ -{ - "_meta": { - "hash": { - "sha256": "ebffa69a1fa192d1cef7cb42ad79231ca976565c5ce371a70160b3048d3cbc06" - }, - "pipfile-spec": 6, - "requires": {}, - "sources": [ - { - "name": "pypi", - "url": "https://pypi.org/simple", - "verify_ssl": true - } - ] - }, - "default": {}, - "develop": {} -} diff --git a/spec/hatchet/package_manager_spec.rb b/spec/hatchet/package_manager_spec.rb index 63cf23fcb..02f68b990 100644 --- a/spec/hatchet/package_manager_spec.rb +++ b/spec/hatchet/package_manager_spec.rb @@ -87,56 +87,44 @@ end end - # This case will be turned into an error in the future. context 'when there are multiple package manager files' do - let(:app) { Hatchet::Runner.new('spec/fixtures/multiple_package_managers') } + let(:app) { Hatchet::Runner.new('spec/fixtures/multiple_package_managers', allow_failure: true) } - it 'outputs a warning and builds with the first listed' do + it 'fails the build with an informative error message' do app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: - remote: ! Warning: Multiple Python package manager files were found. + remote: ! Error: Multiple Python package manager files were found. remote: ! remote: ! Exactly one package manager file should be present in your app's remote: ! source code, however, several were found: remote: ! - remote: ! Pipfile.lock \\(Pipenv\\) - remote: ! requirements.txt \\(pip\\) - remote: ! poetry.lock \\(Poetry\\) - remote: ! uv.lock \\(uv\\) + remote: ! Pipfile.lock (Pipenv) + remote: ! requirements.txt (pip) + remote: ! poetry.lock (Poetry) + remote: ! uv.lock (uv) remote: ! - remote: ! For now, we will build your app using the first package manager - remote: ! listed above, however, in the future this warning will become - remote: ! an error. + remote: ! Previously, the buildpack guessed which package manager to use + remote: ! and installed your dependencies with the first package manager + remote: ! listed above. However, this implicit behaviour was deprecated + remote: ! in November 2024 and is now no longer supported. remote: ! - remote: ! Decide which package manager you want to use with your app, and - remote: ! then delete the file\\(s\\) and any config from the others. + remote: ! You must decide which package manager you want to use with your + remote: ! app, and then delete the file(s) and any config from the others. remote: ! remote: ! If you aren't sure which package manager to use, we recommend remote: ! trying uv, since it supports lockfiles, is extremely fast, and remote: ! is actively maintained by a full-time team: remote: ! https://docs.astral.sh/uv/ + remote: ! + remote: ! Note: If you use a third-party uv or Poetry buildpack, you must + remote: ! remove it from your app, since it's no longer required and the + remote: ! requirements.txt file it generates will trigger this error. See: + remote: ! https://devcenter.heroku.com/articles/managing-buildpacks#remove-classic-buildpacks remote: - remote: - remote: ! Note: We recently added support for the package manager Poetry. - remote: ! If you are using a third-party Poetry buildpack you must remove - remote: ! it, otherwise the requirements.txt file it generates will cause - remote: ! the warning above. - remote: - remote: - remote: ! Note: We recently added support for the package manager uv. - remote: ! If you are using a third-party uv buildpack you must remove - remote: ! it, otherwise the requirements.txt file it generates will cause - remote: ! the warning above. - remote: - remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version - remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} - remote: -----> Installing Pipenv #{PIPENV_VERSION} - remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: Installing dependencies from Pipfile.lock \\(.+\\)... - remote: -----> Saving cache - REGEX + remote: ! Push rejected, failed to compile Python app. + OUTPUT end end end From e6ebda92199579f6c868d4e96e50127ec28dc40e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 10 Dec 2025 13:36:52 +0000 Subject: [PATCH 304/406] Bump uv from 0.9.14 to 0.9.17 (#1995) * Bump uv from 0.9.14 to 0.9.17 Bumps [uv](https://github.com/astral-sh/uv) from 0.9.14 to 0.9.17. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.9.14...0.9.17) --- updated-dependencies: - dependency-name: uv dependency-version: 0.9.17 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 54cc217a6..6ae9d2a09 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ - The build now errors if the files for multiple package managers are found. This replaces the warning displayed since November 2024. ([#1993](https://github.com/heroku/heroku-buildpack-python/pull/1993)) - Sunset the previously deprecated support for falling back to installing dependencies from a `setup.py` file if no Python package manager files were found. ([#1992](https://github.com/heroku/heroku-buildpack-python/pull/1992)) +- Updated uv from 0.9.14 to 0.9.17. ([#1995](https://github.com/heroku/heroku-buildpack-python/pull/1995)) ## [v324] - 2025-12-09 diff --git a/requirements/uv.txt b/requirements/uv.txt index 7ff1f7436..2573d52a5 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.9.14 +uv==0.9.17 From 2f3fb4f7c1f64bf02c3cb322ac40892b7d69b12f Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 10 Dec 2025 13:45:43 +0000 Subject: [PATCH 305/406] Prepare release v325 (#1996) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6ae9d2a09..ee400edf3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v325] - 2025-12-10 + - The build now errors if the files for multiple package managers are found. This replaces the warning displayed since November 2024. ([#1993](https://github.com/heroku/heroku-buildpack-python/pull/1993)) - Sunset the previously deprecated support for falling back to installing dependencies from a `setup.py` file if no Python package manager files were found. ([#1992](https://github.com/heroku/heroku-buildpack-python/pull/1992)) - Updated uv from 0.9.14 to 0.9.17. ([#1995](https://github.com/heroku/heroku-buildpack-python/pull/1995)) @@ -1430,7 +1433,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v324...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v325...main +[v325]: https://github.com/heroku/heroku-buildpack-python/compare/v324...v325 [v324]: https://github.com/heroku/heroku-buildpack-python/compare/v323...v324 [v323]: https://github.com/heroku/heroku-buildpack-python/compare/v322...v323 [v322]: https://github.com/heroku/heroku-buildpack-python/compare/v321...v322 From a3bfa424ee866b543cc2e677b76131b5ec15a8cb Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 16 Dec 2025 08:47:36 +0000 Subject: [PATCH 306/406] Unpin GDAL version in tests (#1998) Since the issue that required it be pinned (in #1954) has since been fixed upstream in: https://github.com/OSGeo/gdal/commit/4dcd9099b4fc39ddce1ba9f214f72d661b6a8fb7 GUS-W-20582833. --- spec/fixtures/pip_gdal/requirements.txt | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/spec/fixtures/pip_gdal/requirements.txt b/spec/fixtures/pip_gdal/requirements.txt index 0059e4a71..2642fd732 100644 --- a/spec/fixtures/pip_gdal/requirements.txt +++ b/spec/fixtures/pip_gdal/requirements.txt @@ -1,2 +1 @@ -# Pinned due to https://github.com/OSGeo/gdal/issues/13329 -GDAL==3.11.4 +GDAL From 689ddf5f5f5d63ae9acb6412c3b1aaab6404cb94 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 4 Jan 2026 23:32:10 +0000 Subject: [PATCH 307/406] Bump sigstore/cosign/cosign from v3.0.2 to v3.0.3 in /builds (#2002) Bumps sigstore/cosign/cosign from v3.0.2 to v3.0.3. --- updated-dependencies: - dependency-name: sigstore/cosign/cosign dependency-version: v3.0.3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- builds/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/builds/Dockerfile b/builds/Dockerfile index 531dfbd26..0e7195939 100644 --- a/builds/Dockerfile +++ b/builds/Dockerfile @@ -1,5 +1,5 @@ ARG STACK_VERSION="24" -FROM ghcr.io/sigstore/cosign/cosign:v3.0.2@sha256:b29487e48205d875c324c79583e2806d9d269c0fa299e0861bbec023d8430c8b AS cosign +FROM ghcr.io/sigstore/cosign/cosign:v3.0.3@sha256:774391ac9f0c137ee419ce56522df5fd3b1f52be90c5b77e97f7c053bdd67a67 AS cosign FROM heroku/heroku:${STACK_VERSION}-build ARG STACK_VERSION From ed233c7ff44bc270ab45d74f26c3ac39f9bf4944 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 4 Jan 2026 23:32:52 +0000 Subject: [PATCH 308/406] Bump rubocop from 1.81.7 to 1.82.1 in the ruby-dependencies group (#2001) Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.81.7 to 1.82.1 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.81.7...v1.82.1) --- updated-dependencies: - dependency-name: rubocop dependency-version: 1.82.1 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 5a97e9454..98c8704da 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -20,7 +20,7 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.16.0) + json (2.18.0) language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) @@ -37,7 +37,7 @@ GEM heroics (~> 0.1.1) moneta (~> 1.0.0) rate_throttle_client (~> 0.1.0) - prism (1.6.0) + prism (1.7.0) racc (1.8.1) rainbow (3.1.1) rate_throttle_client (0.1.2) @@ -51,7 +51,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.6) - rubocop (1.81.7) + rubocop (1.82.1) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) @@ -59,12 +59,12 @@ GEM parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) regexp_parser (>= 2.9.3, < 3.0) - rubocop-ast (>= 1.47.1, < 2.0) + rubocop-ast (>= 1.48.0, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 4.0) - rubocop-ast (1.48.0) + rubocop-ast (1.49.0) parser (>= 3.3.7.2) - prism (~> 1.4) + prism (~> 1.7) rubocop-rspec (3.8.0) lint_roller (~> 1.1) rubocop (~> 1.81) @@ -73,7 +73,7 @@ GEM threaded (0.0.4) unicode-display_width (3.2.0) unicode-emoji (~> 4.1) - unicode-emoji (4.1.0) + unicode-emoji (4.2.0) webrick (1.9.2) PLATFORMS From 4ee96db635a17c700a1f797101acbea597e80f86 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 5 Jan 2026 11:03:20 +0000 Subject: [PATCH 309/406] Bump uv from 0.9.17 to 0.9.21 (#2003) * Bump uv from 0.9.17 to 0.9.21 Bumps [uv](https://github.com/astral-sh/uv) from 0.9.17 to 0.9.21. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.9.17...0.9.21) --- updated-dependencies: - dependency-name: uv dependency-version: 0.9.21 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ee400edf3..5648068a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.9.17 to 0.9.21. ([#2003](https://github.com/heroku/heroku-buildpack-python/pull/2003)) ## [v325] - 2025-12-10 diff --git a/requirements/uv.txt b/requirements/uv.txt index 2573d52a5..f25857315 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.9.17 +uv==0.9.21 From 2833d0357901aad91c977a820678e4aa243a7a4a Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Mon, 5 Jan 2026 11:06:51 +0000 Subject: [PATCH 310/406] Prepare release v326 (#2004) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5648068a5..908e2d479 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v326] - 2026-01-05 + - Updated uv from 0.9.17 to 0.9.21. ([#2003](https://github.com/heroku/heroku-buildpack-python/pull/2003)) ## [v325] - 2025-12-10 @@ -1434,7 +1437,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v325...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v326...main +[v326]: https://github.com/heroku/heroku-buildpack-python/compare/v325...v326 [v325]: https://github.com/heroku/heroku-buildpack-python/compare/v324...v325 [v324]: https://github.com/heroku/heroku-buildpack-python/compare/v323...v324 [v323]: https://github.com/heroku/heroku-buildpack-python/compare/v322...v323 From 2161b9bca31e3e75a13d85790cb122a99fea1e3b Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 7 Jan 2026 12:55:43 +0000 Subject: [PATCH 311/406] Adjust the error message shown for missing SQLite headers (#2006) Since `pysqlite3` 0.6.0 was just released, that now ships with precompiled wheels, which means: 1. Our test for the error handling now started failing, since the package now works out of the box rather than erroring. 2. Our advice for which package to use has now changed, since users now only need to update to a newer version of `pysqlite3` rather than switch to the `pysqlite3-binary` package. GUS-W-20804528. --- CHANGELOG.md | 1 + bin/warnings | 13 +++---------- spec/fixtures/pip_pysqlite3/requirements.txt | 4 +++- spec/hatchet/pip_spec.rb | 13 +++---------- 4 files changed, 10 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 908e2d479..eb3d7839c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Adjusted the error message shown if SQLite headers aren't found during package installation. ([#2006](https://github.com/heroku/heroku-buildpack-python/pull/2006)) ## [v326] - 2026-01-05 diff --git a/bin/warnings b/bin/warnings index 07bd54daa..04b453a2d 100755 --- a/bin/warnings +++ b/bin/warnings @@ -29,16 +29,9 @@ show-warnings() { 2. Replace any `pysqlite3` imports in your app with `sqlite3`. Alternatively, if you can't use the `sqlite3` stdlib module, - switch from the `pysqlite3` package to `pysqlite3-binary`, - which is pre-compiled and doesn't need the SQLite headers to - be installed. - - If instead you need the SQLite headers for another reason - (or wish to continue to compile the `pysqlite3` package from - source), then install the `libsqlite3-dev` and `libsqlite3-0` - packages using the APT buildpack (make sure the APT buildpack - runs before the Python buildpack, not after): - https://github.com/heroku/heroku-buildpack-apt + update your `pysqlite3` package to 0.6.0 or newer, since the + newer versions are published with pre-compiled wheels and so + don't need the SQLite headers to be installed. EOF build_data::set_string "failure_detail" "sqlite3.h: No such file or directory" elif grep -qi 'Please use pip<24.1 if you need to use this version' "${install_log}"; then diff --git a/spec/fixtures/pip_pysqlite3/requirements.txt b/spec/fixtures/pip_pysqlite3/requirements.txt index 864b10382..043c5fdbe 100644 --- a/spec/fixtures/pip_pysqlite3/requirements.txt +++ b/spec/fixtures/pip_pysqlite3/requirements.txt @@ -1 +1,3 @@ -pysqlite3 +# We have to use an older version to test the custom SQLite headers error message, +# since pysqlite3 0.6.0+ ships with precompiled wheels and so doesn't error. +pysqlite3==0.5.4 diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 67a0f1674..c46f762f2 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -449,16 +449,9 @@ remote: ! 2. Replace any `pysqlite3` imports in your app with `sqlite3`. remote: ! remote: ! Alternatively, if you can't use the `sqlite3` stdlib module, - remote: ! switch from the `pysqlite3` package to `pysqlite3-binary`, - remote: ! which is pre-compiled and doesn't need the SQLite headers to - remote: ! be installed. - remote: ! - remote: ! If instead you need the SQLite headers for another reason - remote: ! (or wish to continue to compile the `pysqlite3` package from - remote: ! source), then install the `libsqlite3-dev` and `libsqlite3-0` - remote: ! packages using the APT buildpack (make sure the APT buildpack - remote: ! runs before the Python buildpack, not after): - remote: ! https://github.com/heroku/heroku-buildpack-apt + remote: ! update your `pysqlite3` package to 0.6.0 or newer, since the + remote: ! newer versions are published with pre-compiled wheels and so + remote: ! don't need the SQLite headers to be installed. remote: remote: remote: ! Error: Unable to install dependencies using pip. From 24eee50489621a97497ad22c9e75128cb5ef2c69 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 7 Jan 2026 13:09:42 +0000 Subject: [PATCH 312/406] Remove Python 3.9 support (#2005) Python 3.9 reached upstream EOL on 31st October 2025: https://devguide.python.org/versions/#supported-versions The Python version support policy is that supported versions follows the upstream EOL lifecycle: https://devcenter.heroku.com/articles/python-support#python-version-support-policy And Python 3.9 support has been deprecated since 8th January 2025: https://devcenter.heroku.com/changelog-items/3095 As such, builds of apps using Python 3.9 will now fail with an error message explaining they must be upgraded to a newer/supported Python version. Apps using Python 3.9 that aren't able to upgrade immediately will need to pin to an older buildpack version temporarily. GUS-W-17595553. --- CHANGELOG.md | 1 + README.md | 3 +- builds/build_python_runtime.sh | 76 ++++++------------- lib/pip.sh | 2 +- lib/pipenv.sh | 3 - lib/poetry.sh | 5 -- lib/python_version.sh | 23 +----- .../.python-version | 2 +- .../requirements.txt | 5 +- spec/fixtures/pip_oldest_python/runtime.txt | 2 +- .../pipenv_python_full_version/Pipfile | 2 +- .../pipenv_python_full_version/Pipfile.lock | 4 +- .../poetry_oldest_python/.python-version | 2 +- .../fixtures/poetry_oldest_python/poetry.lock | 6 +- .../poetry_oldest_python/pyproject.toml | 2 +- spec/fixtures/python_3.9/.python-version | 2 - spec/fixtures/python_3.9/requirements.txt | 2 - .../python_version_eol/.python-version | 2 +- .../python_version_eol_cached/bin/pre_compile | 6 +- .../fixtures/uv_oldest_python/.python-version | 2 +- spec/fixtures/uv_oldest_python/pyproject.toml | 2 +- spec/fixtures/uv_oldest_python/uv.lock | 2 +- spec/hatchet/django_spec.rb | 9 +-- spec/hatchet/pip_spec.rb | 26 +++---- spec/hatchet/pipenv_spec.rb | 26 +++---- spec/hatchet/poetry_spec.rb | 22 +++--- spec/hatchet/python_version_spec.rb | 65 ++++------------ spec/hatchet/uv_spec.rb | 22 +++--- spec/spec_helper.rb | 1 - 29 files changed, 112 insertions(+), 215 deletions(-) delete mode 100644 spec/fixtures/python_3.9/.python-version delete mode 100644 spec/fixtures/python_3.9/requirements.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index eb3d7839c..c07f8ed7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Removed support for Python 3.9. ([#2005](https://github.com/heroku/heroku-buildpack-python/pull/2005)) - Adjusted the error message shown if SQLite headers aren't found during package installation. ([#2006](https://github.com/heroku/heroku-buildpack-python/pull/2006)) ## [v326] - 2026-01-05 diff --git a/README.md b/README.md index 6463c686e..4f8703ea4 100644 --- a/README.md +++ b/README.md @@ -51,11 +51,10 @@ The supported Python versions are: - Python 3.13 - Python 3.12 - Python 3.11 -- Python 3.10 These Python versions are deprecated on Heroku: -- Python 3.9 +- Python 3.10 Python versions older than those listed above are no longer supported, since they have reached end-of-life [upstream](https://devguide.python.org/versions/#supported-versions). diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index 6dbb8d045..461d8d24b 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -27,7 +27,6 @@ function abort() { case "${STACK:?}" in heroku-22 | heroku-24) SUPPORTED_PYTHON_VERSIONS=( - "3.9" "3.10" "3.11" "3.12" @@ -58,10 +57,6 @@ case "${PYTHON_MAJOR_VERSION}" in SIGSTORE_IDENTITY='pablogsal@python.org' SIGSTORE_ISSUER='https://accounts.google.com' ;; - 3.9) - SIGSTORE_IDENTITY='lukasz@langa.pl' - SIGSTORE_ISSUER='https://github.com/login/oauth' - ;; *) abort "Unsupported Python version '${PYTHON_MAJOR_VERSION}'!" ;; @@ -102,42 +97,34 @@ CONFIGURE_OPTS=( "--enable-optimizations" # Make autoconf's configure option validation more strict. "--enable-option-checking=fatal" + # Shared builds are beneficial for a number of reasons: + # - Reduces the size of the build, since it avoids the duplication between + # the Python binary and the static library. + # - Permits use-cases that only work with the shared Python library, + # and not the static library (such as `pycall.rb` or `PyO3`). + # - More consistent with the official Python Docker images and other distributions. + # + # Shared builds are slower unless `no-semantic-interposition`and LTO is used, + # however, as of Python 3.10 `no-semantic-interposition` is enabled by default: + # https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup + # https://github.com/python/cpython/issues/83161 + "--enable-shared" # Install Python into `/tmp/python` rather than the default of `/usr/local`. "--prefix=${INSTALL_DIR}" # Skip running `ensurepip` as part of install, since the buildpack installs a curated # version of pip itself (which ensures it's consistent across Python patch releases). "--with-ensurepip=no" + "--with-lto" + # Counter-intuitively, the static library is still generated by default even when + # the shared library is enabled, so we disable it to reduce the build size. + # This option only exists for Python 3.10+. + "--without-static-libpython" ) -if [[ "${PYTHON_MAJOR_VERSION}" != +(3.9) ]]; then - CONFIGURE_OPTS+=( - # Shared builds are beneficial for a number of reasons: - # - Reduces the size of the build, since it avoids the duplication between - # the Python binary and the static library. - # - Permits use-cases that only work with the shared Python library, - # and not the static library (such as `pycall.rb` or `PyO3`). - # - More consistent with the official Python Docker images and other distributions. - # - # However, shared builds are slower unless `no-semantic-interposition`and LTO is used: - # https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup - # https://github.com/python/cpython/issues/83161 - # - # It's only as of Python 3.10 that `no-semantic-interposition` is enabled by default, - # so we only use shared builds on Python 3.10+ to avoid needing to override the default - # compiler flags. - "--enable-shared" - "--with-lto" - # Counter-intuitively, the static library is still generated by default even when - # the shared library is enabled, so we disable it to reduce the build size. - # This option only exists for Python 3.10+. - "--without-static-libpython" - ) -fi - -if [[ "${PYTHON_MAJOR_VERSION}" != +(3.9|3.10) ]]; then +if [[ "${PYTHON_MAJOR_VERSION}" != +(3.10) ]]; then CONFIGURE_OPTS+=( # Skip building the test modules, since we remove them after the build anyway. - # This feature was added in Python 3.10+, however it wasn't until Python 3.11 + # This feature was added in Python 3.10, however it wasn't until Python 3.11 # that compatibility issues between it and PGO were fixed: # https://github.com/python/cpython/pull/29315 "--disable-test-modules" @@ -155,31 +142,14 @@ fi # - https://wiki.ubuntu.com/ToolChain/CompilerFlags # - https://wiki.debian.org/Hardening # - https://github.com/docker-library/python/issues/810 -# We only use `dpkg-buildflags` for Python versions where we build in shared mode (Python 3.9+), -# since some of the options it enables interferes with the stripping of static libraries. -if [[ "${PYTHON_MAJOR_VERSION}" == +(3.9) ]]; then - EXTRA_CFLAGS='' - LDFLAGS='-Wl,--strip-all' -else - EXTRA_CFLAGS="$(dpkg-buildflags --get CFLAGS)" - LDFLAGS="$(dpkg-buildflags --get LDFLAGS) -Wl,--strip-all" -fi +EXTRA_CFLAGS="$(dpkg-buildflags --get CFLAGS)" +LDFLAGS="$(dpkg-buildflags --get LDFLAGS) -Wl,--strip-all" CPU_COUNT="$(nproc)" make -j "${CPU_COUNT}" "EXTRA_CFLAGS=${EXTRA_CFLAGS}" "LDFLAGS=${LDFLAGS}" make install -if [[ "${PYTHON_MAJOR_VERSION}" == +(3.9) ]]; then - # On older versions of Python we're still building the static library, which has to be - # manually stripped since the linker stripping enabled in LDFLAGS doesn't cover them. - # We're using `--strip-unneeded` since `--strip-all` would remove the `.symtab` section - # that is required for static libraries to be able to be linked. - # `find` is used since there are multiple copies of the static library in version-specific - # locations, eg: - # - `lib/libpython3.9.a` - # - `lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.a` - find "${INSTALL_DIR}" -type f -name '*.a' -print -exec strip --strip-unneeded '{}' + -elif ! find "${INSTALL_DIR}" -type f -name '*.a' -print -exec false '{}' +; then +if ! find "${INSTALL_DIR}" -type f -name '*.a' -print -exec false '{}' +; then abort "Unexpected static libraries found!" fi @@ -220,7 +190,7 @@ LD_LIBRARY_PATH="${SRC_DIR}" "${SRC_DIR}/python" -m compileall -f --invalidation # (e.g. `python -m pydoc`) if needed. rm "${INSTALL_DIR}"/bin/{idle,pydoc}* # The 2to3 module and entrypoint was removed from the stdlib in Python 3.13. -if [[ "${PYTHON_MAJOR_VERSION}" == +(3.9|3.10|3.11|3.12) ]]; then +if [[ "${PYTHON_MAJOR_VERSION}" == +(3.10|3.11|3.12) ]]; then rm "${INSTALL_DIR}"/bin/2to3* fi diff --git a/lib/pip.sh b/lib/pip.sh index edc049968..21e5a9e15 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -28,7 +28,7 @@ function pip::install_pip() { # them installed. # - Most of the Python ecosystem has stopped installing them for Python 3.12+ already. # See the Python CNB's removal for more details: https://github.com/heroku/buildpacks-python/pull/243 - if [[ "${python_major_version}" == +(3.9|3.10|3.11|3.12) ]]; then + if [[ "${python_major_version}" == +(3.10|3.11|3.12) ]]; then build_data::set_string "setuptools_version" "${SETUPTOOLS_VERSION}" build_data::set_string "wheel_version" "${WHEEL_VERSION}" packages_to_install+=( diff --git a/lib/pipenv.sh b/lib/pipenv.sh index 0b99ac077..84b00ad4b 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -45,9 +45,6 @@ function pipenv::install_pipenv() { local bundled_pip_module_path bundled_pip_module_path="$(utils::bundled_pip_module_path "${python_home}" "${python_major_version}")" - # We must call the venv Python directly here, rather than relying on pip's `--python` - # option, since `--python` was only added in pip v22.3, so isn't supported by the older - # pip versions bundled with Python 3.9/3.10. # `--isolated`: Prevents any custom pip configuration added by third party buildpacks (via env # vars or global config files) from breaking package manager bootstrapping. # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. diff --git a/lib/poetry.sh b/lib/poetry.sh index eec1d3942..507185b03 100644 --- a/lib/poetry.sh +++ b/lib/poetry.sh @@ -51,13 +51,8 @@ function poetry::install_poetry() { local bundled_pip_module_path bundled_pip_module_path="$(utils::bundled_pip_module_path "${python_home}" "${python_major_version}")" - # We must call the venv Python directly here, rather than relying on pip's `--python` - # option, since `--python` was only added in pip v22.3, so isn't supported by the older - # pip versions bundled with Python 3.9/3.10. # `--isolated`: Prevents any custom pip configuration added by third party buildpacks (via env # vars or global config files) from breaking package manager bootstrapping. - # We pin to an older dulwich version to work around https://github.com/jelmer/dulwich/issues/1948 - # on Python 3.9.0/3.9.1. TODO: Remove this pin when we drop support for Python 3.9 in Jan 2026. # shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled. if ! { "${poetry_venv_dir}/bin/python" "${bundled_pip_module_path}" \ diff --git a/lib/python_version.sh b/lib/python_version.sh index 0e00e4ca3..283a04c1f 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -4,14 +4,13 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail -LATEST_PYTHON_3_9="3.9.25" LATEST_PYTHON_3_10="3.10.19" LATEST_PYTHON_3_11="3.11.14" LATEST_PYTHON_3_12="3.12.12" LATEST_PYTHON_3_13="3.13.11" LATEST_PYTHON_3_14="3.14.2" -OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=9 +OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=10 NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=14 DEFAULT_PYTHON_FULL_VERSION="${LATEST_PYTHON_3_14}" @@ -525,7 +524,6 @@ function python_version::resolve_python_version() { # Otherwise map major version specifiers to the latest patch release. case "${requested_python_version}" in *.*.*) echo "${requested_python_version}" ;; - 3.9) echo "${LATEST_PYTHON_3_9}" ;; 3.10) echo "${LATEST_PYTHON_3_10}" ;; 3.11) echo "${LATEST_PYTHON_3_11}" ;; 3.12) echo "${LATEST_PYTHON_3_12}" ;; @@ -688,24 +686,7 @@ function python_version::warn_if_deprecated_major_version() { local requested_major_version="${1}" local version_origin="${2}" - if [[ "${requested_major_version}" == "3.9" ]]; then - output::warning <<-EOF - Warning: Support for Python 3.9 is ending soon! - - Python 3.9 reached its upstream end-of-life on 31st October 2025, - and so no longer receives security updates: - https://devguide.python.org/versions/#supported-versions - - As such, support for Python 3.9 will be removed from this - buildpack on 7th January 2026. - - Upgrade to a newer Python version as soon as possible, by - changing the version in your ${version_origin} file. - - For more information, see: - https://devcenter.heroku.com/articles/python-support#supported-python-versions - EOF - elif [[ "${requested_major_version}" == "3.10" ]]; then + if [[ "${requested_major_version}" == "3.10" ]]; then output::warning <<-EOF Warning: Support for Python 3.10 is deprecated! diff --git a/spec/fixtures/django_staticfiles_legacy_django/.python-version b/spec/fixtures/django_staticfiles_legacy_django/.python-version index bd28b9c5c..c8cfe3959 100644 --- a/spec/fixtures/django_staticfiles_legacy_django/.python-version +++ b/spec/fixtures/django_staticfiles_legacy_django/.python-version @@ -1 +1 @@ -3.9 +3.10 diff --git a/spec/fixtures/django_staticfiles_legacy_django/requirements.txt b/spec/fixtures/django_staticfiles_legacy_django/requirements.txt index e258a24fc..7f5462e9c 100644 --- a/spec/fixtures/django_staticfiles_legacy_django/requirements.txt +++ b/spec/fixtures/django_staticfiles_legacy_django/requirements.txt @@ -1,3 +1,2 @@ -# This is the oldest Django version that works on Python 3.9 (which is the -# oldest Python that is available on all of our supported builders). -Django==1.8.19 +# This is the oldest Django version that works on Python 3.10 (our oldest supported Python version). +Django==2.1.15 diff --git a/spec/fixtures/pip_oldest_python/runtime.txt b/spec/fixtures/pip_oldest_python/runtime.txt index f72c5111f..fadb07024 100644 --- a/spec/fixtures/pip_oldest_python/runtime.txt +++ b/spec/fixtures/pip_oldest_python/runtime.txt @@ -1 +1 @@ -python-3.9.0 +python-3.10.0 diff --git a/spec/fixtures/pipenv_python_full_version/Pipfile b/spec/fixtures/pipenv_python_full_version/Pipfile index f3e9b3e9e..1b3a10cc1 100644 --- a/spec/fixtures/pipenv_python_full_version/Pipfile +++ b/spec/fixtures/pipenv_python_full_version/Pipfile @@ -10,7 +10,7 @@ typing-extensions = "*" [requires] # Uses the oldest Python version supported by all stacks, to validate Pipenv works with it. -python_full_version = "3.9.0" +python_full_version = "3.10.0" # `python_full_version` should take precedence over this. python_version = "3.12" diff --git a/spec/fixtures/pipenv_python_full_version/Pipfile.lock b/spec/fixtures/pipenv_python_full_version/Pipfile.lock index f71891fe1..8b1a4f32a 100644 --- a/spec/fixtures/pipenv_python_full_version/Pipfile.lock +++ b/spec/fixtures/pipenv_python_full_version/Pipfile.lock @@ -1,11 +1,11 @@ { "_meta": { "hash": { - "sha256": "20aa9cfaef42c18684f934bc2c4e062167465b9a062fa284441acf1fe712dc60" + "sha256": "e9ceff57b2002f3a53ac8e84fc449e7aacbb62139f45e8299bde332e3ef9f6b2" }, "pipfile-spec": 6, "requires": { - "python_full_version": "3.9.0", + "python_full_version": "3.10.0", "python_version": "3.12" }, "sources": [ diff --git a/spec/fixtures/poetry_oldest_python/.python-version b/spec/fixtures/poetry_oldest_python/.python-version index a5c4c7633..30291cba2 100644 --- a/spec/fixtures/poetry_oldest_python/.python-version +++ b/spec/fixtures/poetry_oldest_python/.python-version @@ -1 +1 @@ -3.9.0 +3.10.0 diff --git a/spec/fixtures/poetry_oldest_python/poetry.lock b/spec/fixtures/poetry_oldest_python/poetry.lock index c9a78e867..e8576cea7 100644 --- a/spec/fixtures/poetry_oldest_python/poetry.lock +++ b/spec/fixtures/poetry_oldest_python/poetry.lock @@ -1,4 +1,4 @@ -# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand. +# This file is automatically @generated by Poetry 1.8.5 and should not be changed by hand. [[package]] name = "typing-extensions" @@ -13,5 +13,5 @@ files = [ [metadata] lock-version = "2.0" -python-versions = "^3.9" -content-hash = "3b8ea94b618b2c3590bcbb224f2e2c723e6c14c8da90ba6917ca4a4fdcb716cf" +python-versions = "^3.10" +content-hash = "cfec7fa4755d6e9851dbc7848a76babbc6c288c7f13cc2bed2e60f83783d4fd2" diff --git a/spec/fixtures/poetry_oldest_python/pyproject.toml b/spec/fixtures/poetry_oldest_python/pyproject.toml index c69fd9272..e193f693f 100644 --- a/spec/fixtures/poetry_oldest_python/pyproject.toml +++ b/spec/fixtures/poetry_oldest_python/pyproject.toml @@ -2,5 +2,5 @@ package-mode = false [tool.poetry.dependencies] -python = "^3.9" +python = "^3.10" typing-extensions = "*" diff --git a/spec/fixtures/python_3.9/.python-version b/spec/fixtures/python_3.9/.python-version deleted file mode 100644 index 54f2f0246..000000000 --- a/spec/fixtures/python_3.9/.python-version +++ /dev/null @@ -1,2 +0,0 @@ -# Comments are supported -3.9 diff --git a/spec/fixtures/python_3.9/requirements.txt b/spec/fixtures/python_3.9/requirements.txt deleted file mode 100644 index f6e499a84..000000000 --- a/spec/fixtures/python_3.9/requirements.txt +++ /dev/null @@ -1,2 +0,0 @@ -# This package has been picked since it has no dependencies and is small/fast to install. -typing-extensions==4.15.0 diff --git a/spec/fixtures/python_version_eol/.python-version b/spec/fixtures/python_version_eol/.python-version index 2237ff9e6..033763cdf 100644 --- a/spec/fixtures/python_version_eol/.python-version +++ b/spec/fixtures/python_version_eol/.python-version @@ -4,5 +4,5 @@ # So are empty lines, and leading/trailing whitespace. - 3.8 + 3.9 diff --git a/spec/fixtures/python_version_eol_cached/bin/pre_compile b/spec/fixtures/python_version_eol_cached/bin/pre_compile index 5bce49644..63422ee0b 100755 --- a/spec/fixtures/python_version_eol_cached/bin/pre_compile +++ b/spec/fixtures/python_version_eol_cached/bin/pre_compile @@ -2,8 +2,6 @@ set -euo pipefail -# Emulate the cache from a previous build that used a now-EOL Python version, since we can't -# actually perform a build using Python 3.8, since it was only supported until Heroku-20. -# TODO: Switch to doing a real build once Python 3.9 EOLs. +# Emulate the cache from a previous build that used a now-EOL Python version. mkdir -p "${CACHE_DIR:?}/.heroku" -echo "python-3.8.20" >"${CACHE_DIR}/.heroku/python-version" +echo "python-3.9.25" >"${CACHE_DIR}/.heroku/python-version" diff --git a/spec/fixtures/uv_oldest_python/.python-version b/spec/fixtures/uv_oldest_python/.python-version index a5c4c7633..30291cba2 100644 --- a/spec/fixtures/uv_oldest_python/.python-version +++ b/spec/fixtures/uv_oldest_python/.python-version @@ -1 +1 @@ -3.9.0 +3.10.0 diff --git a/spec/fixtures/uv_oldest_python/pyproject.toml b/spec/fixtures/uv_oldest_python/pyproject.toml index 08b9a31bc..bf4f99549 100644 --- a/spec/fixtures/uv_oldest_python/pyproject.toml +++ b/spec/fixtures/uv_oldest_python/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "uv-oldest-python" version = "0.0.0" -requires-python = "==3.9.*" +requires-python = "==3.10.*" dependencies = [ "typing-extensions", ] diff --git a/spec/fixtures/uv_oldest_python/uv.lock b/spec/fixtures/uv_oldest_python/uv.lock index 1200f8421..47873756d 100644 --- a/spec/fixtures/uv_oldest_python/uv.lock +++ b/spec/fixtures/uv_oldest_python/uv.lock @@ -1,6 +1,6 @@ version = 1 revision = 2 -requires-python = "==3.9.*" +requires-python = "==3.10.*" [[package]] name = "typing-extensions" diff --git a/spec/hatchet/django_spec.rb b/spec/hatchet/django_spec.rb index 8c613896c..664410dd7 100644 --- a/spec/hatchet/django_spec.rb +++ b/spec/hatchet/django_spec.rb @@ -93,11 +93,10 @@ remote: remote: \\['/tmp/build_\\w+', remote: '/invalid-path', - remote: '/app/.heroku/python/lib/python39.zip', - remote: '/app/.heroku/python/lib/python3.9', - remote: '/app/.heroku/python/lib/python3.9/lib-dynload', - remote: '/app/.heroku/python/lib/python3.9/site-packages'\\] - remote: Copying '/tmp/build_\\w+/testapp/static/robots.txt' + remote: '/app/.heroku/python/lib/python310.zip', + remote: '/app/.heroku/python/lib/python3.10', + remote: '/app/.heroku/python/lib/python3.10/lib-dynload', + remote: '/app/.heroku/python/lib/python3.10/site-packages'\\] remote: remote: 1 static file copied to '/tmp/build_\\w+/staticfiles'. remote: diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index c46f762f2..fbab2c174 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -252,7 +252,7 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python 3.9.0 specified in runtime.txt + remote: -----> Using Python 3.10.0 specified in runtime.txt remote: remote: ! Warning: The runtime.txt file is deprecated. remote: ! @@ -272,9 +272,9 @@ remote: ! In the new file, specify your app's major Python version number remote: ! only. Don't include quotes or a 'python-' prefix. remote: ! - remote: ! For example, to request the latest version of Python 3.9, + remote: ! For example, to request the latest version of Python 3.10, remote: ! update your .python-version file so it contains exactly: - remote: ! 3.9 + remote: ! 3.10 remote: ! remote: ! We strongly recommend that you don't specify the Python patch remote: ! version number, since it will pin your app to an exact Python @@ -285,14 +285,14 @@ remote: ! this warning will be made an error. remote: remote: - remote: ! Warning: Support for Python 3.9 is ending soon! + remote: ! Warning: Support for Python 3.10 is deprecated! remote: ! - remote: ! Python 3.9 reached its upstream end-of-life on 31st October 2025, - remote: ! and so no longer receives security updates: + remote: ! Python 3.10 will reach its upstream end-of-life in October 2026, + remote: ! at which point it will no longer receive security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! - remote: ! As such, support for Python 3.9 will be removed from this - remote: ! buildpack on 7th January 2026. + remote: ! As such, support for Python 3.10 will be removed from this + remote: ! buildpack on 6th January 2027. remote: ! remote: ! Upgrade to a newer Python version as soon as possible, by remote: ! changing the version in your runtime.txt file. @@ -303,8 +303,8 @@ remote: remote: ! Warning: A Python patch update is available! remote: ! - remote: ! Your app is using Python 3.9.0, however, there is a newer - remote: ! patch release of Python 3.9 available: #{LATEST_PYTHON_3_9} + remote: ! Your app is using Python 3.10.0, however, there is a newer + remote: ! patch release of Python 3.10 available: #{LATEST_PYTHON_3_10} remote: ! remote: ! It is important to always use the latest patch version of remote: ! Python to keep your app secure. @@ -312,12 +312,12 @@ remote: ! Update your runtime.txt file to use the new version. remote: ! remote: ! We strongly recommend that you don't pin your app to an - remote: ! exact Python version such as 3.9.0, and instead only specify - remote: ! the major Python version of 3.9 in your runtime.txt file. + remote: ! exact Python version such as 3.10.0, and instead only specify + remote: ! the major Python version of 3.10 in your runtime.txt file. remote: ! This will allow your app to receive the latest available Python remote: ! patch version automatically and prevent this warning. remote: - remote: -----> Installing Python 3.9.0 + remote: -----> Installing Python 3.10.0 remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index d18344d34..f0bc4c5f3 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -160,23 +160,23 @@ # 1. That `python_full_version` takes precedence over the `python_version` field. # 2. That Pipenv works on the oldest Python version supported by all stacks. # 3. That the security update available message works for Pipenv too. - context 'with a Pipfile.lock containing python_full_version 3.9.0' do + context 'with a Pipfile.lock containing python_full_version 3.10.0' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_full_version') } it 'builds with the outdated Python version specified and displays a deprecation warning' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python 3.9.0 specified in Pipfile.lock + remote: -----> Using Python 3.10.0 specified in Pipfile.lock remote: - remote: ! Warning: Support for Python 3.9 is ending soon! + remote: ! Warning: Support for Python 3.10 is deprecated! remote: ! - remote: ! Python 3.9 reached its upstream end-of-life on 31st October 2025, - remote: ! and so no longer receives security updates: + remote: ! Python 3.10 will reach its upstream end-of-life in October 2026, + remote: ! at which point it will no longer receive security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! - remote: ! As such, support for Python 3.9 will be removed from this - remote: ! buildpack on 7th January 2026. + remote: ! As such, support for Python 3.10 will be removed from this + remote: ! buildpack on 6th January 2027. remote: ! remote: ! Upgrade to a newer Python version as soon as possible, by remote: ! changing the version in your Pipfile.lock file. @@ -187,8 +187,8 @@ remote: remote: ! Warning: A Python patch update is available! remote: ! - remote: ! Your app is using Python 3.9.0, however, there is a newer - remote: ! patch release of Python 3.9 available: #{LATEST_PYTHON_3_9} + remote: ! Your app is using Python 3.10.0, however, there is a newer + remote: ! patch release of Python 3.10 available: #{LATEST_PYTHON_3_10} remote: ! remote: ! It is important to always use the latest patch version of remote: ! Python to keep your app secure. @@ -196,12 +196,12 @@ remote: ! Update your Pipfile.lock file to use the new version. remote: ! remote: ! We strongly recommend that you don't pin your app to an - remote: ! exact Python version such as 3.9.0, and instead only specify - remote: ! the major Python version of 3.9 in your Pipfile.lock file. + remote: ! exact Python version such as 3.10.0, and instead only specify + remote: ! the major Python version of 3.10 in your Pipfile.lock file. remote: ! This will allow your app to receive the latest available Python remote: ! patch version automatically and prevent this warning. remote: - remote: -----> Installing Python 3.9.0 + remote: -----> Installing Python 3.10.0 remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Installing dependencies from Pipfile.lock \\(.+\\)... @@ -443,7 +443,7 @@ remote: ! As such, it's no longer supported by this buildpack: remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: ! - remote: ! Please upgrade to at least Python 3.9 by changing the + remote: ! Please upgrade to at least Python 3.10 by changing the remote: ! version in your Pipfile.lock file. remote: ! remote: ! If possible, we recommend upgrading all the way to Python #{DEFAULT_PYTHON_MAJOR_VERSION}, diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index b963b444a..5bd26d664 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -267,16 +267,16 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python 3.9.0 specified in .python-version + remote: -----> Using Python 3.10.0 specified in .python-version remote: - remote: ! Warning: Support for Python 3.9 is ending soon! + remote: ! Warning: Support for Python 3.10 is deprecated! remote: ! - remote: ! Python 3.9 reached its upstream end-of-life on 31st October 2025, - remote: ! and so no longer receives security updates: + remote: ! Python 3.10 will reach its upstream end-of-life in October 2026, + remote: ! at which point it will no longer receive security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! - remote: ! As such, support for Python 3.9 will be removed from this - remote: ! buildpack on 7th January 2026. + remote: ! As such, support for Python 3.10 will be removed from this + remote: ! buildpack on 6th January 2027. remote: ! remote: ! Upgrade to a newer Python version as soon as possible, by remote: ! changing the version in your .python-version file. @@ -287,8 +287,8 @@ remote: remote: ! Warning: A Python patch update is available! remote: ! - remote: ! Your app is using Python 3.9.0, however, there is a newer - remote: ! patch release of Python 3.9 available: #{LATEST_PYTHON_3_9} + remote: ! Your app is using Python 3.10.0, however, there is a newer + remote: ! patch release of Python 3.10 available: #{LATEST_PYTHON_3_10} remote: ! remote: ! It is important to always use the latest patch version of remote: ! Python to keep your app secure. @@ -296,12 +296,12 @@ remote: ! Update your .python-version file to use the new version. remote: ! remote: ! We strongly recommend that you don't pin your app to an - remote: ! exact Python version such as 3.9.0, and instead only specify - remote: ! the major Python version of 3.9 in your .python-version file. + remote: ! exact Python version such as 3.10.0, and instead only specify + remote: ! the major Python version of 3.10 in your .python-version file. remote: ! This will allow your app to receive the latest available Python remote: ! patch version automatically and prevent this warning. remote: - remote: -----> Installing Python 3.9.0 + remote: -----> Installing Python 3.10.0 remote: -----> Installing Poetry #{POETRY_VERSION} remote: -----> Installing dependencies using 'poetry sync --only main' remote: Installing dependencies from lock file diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index a0987e59b..73ddec145 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -5,7 +5,7 @@ RSpec.shared_examples 'builds with the requested Python version' do |requested_version, resolved_version| it "builds with Python #{requested_version}" do app.deploy do |app| - if ['3.9', '3.10', '3.11', '3.12'].include?(requested_version) + if ['3.10', '3.11', '3.12'].include?(requested_version) expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python #{requested_version} specified in .python-version @@ -161,9 +161,7 @@ end context 'with an app last built using an EOL Python version' do - # This fixture emulates the cache from a previous build that used a now-EOL Python version, - # since we can't actually perform a build using Python 3.8, since it was only supported - # until Heroku-20. TODO: Switch to doing a real EOL cached rebuild once Python 3.9 EOLs. + # This fixture emulates the cache from a previous build that used a now-EOL Python version. let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_eol_cached', allow_failure: true) } it 'aborts the build with an EOL message' do @@ -171,21 +169,21 @@ expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Running bin/pre_compile hook - remote: -----> No Python version was specified. Using the same major version as the last build: Python 3.8 + remote: -----> No Python version was specified. Using the same major version as the last build: Python 3.9 remote: remote: ! Error: The cached Python version has reached end-of-life. remote: ! remote: ! Your app doesn't specify a Python version, and so normally - remote: ! would use the version cached from the last build (3.8). + remote: ! would use the version cached from the last build (3.9). remote: ! - remote: ! However, Python 3.8 has reached its upstream end-of-life, + remote: ! However, Python 3.9 has reached its upstream end-of-life, remote: ! and is therefore no longer receiving security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! remote: ! As such, it's no longer supported by this buildpack: remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: ! - remote: ! Please upgrade to at least Python 3.9 by configuring an + remote: ! Please upgrade to at least Python 3.10 by configuring an remote: ! explicit Python version for your app. remote: ! remote: ! Create a new file in the root directory of your app named: @@ -197,9 +195,9 @@ remote: ! In the new file, specify your app's major Python version number remote: ! only. Don't include quotes or a 'python-' prefix. remote: ! - remote: ! For example, to request the latest version of Python 3.9, + remote: ! For example, to request the latest version of Python 3.10, remote: ! update your .python-version file so it contains exactly: - remote: ! 3.9 + remote: ! 3.10 remote: ! remote: ! We strongly recommend that you don't specify the Python patch remote: ! version number, since it will pin your app to an exact Python @@ -251,41 +249,6 @@ end end - context 'when .python-version contains Python 3.9' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.9') } - - it 'builds with Python 3.9 but shows a deprecation warning' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python 3.9 specified in .python-version - remote: - remote: ! Warning: Support for Python 3.9 is ending soon! - remote: ! - remote: ! Python 3.9 reached its upstream end-of-life on 31st October 2025, - remote: ! and so no longer receives security updates: - remote: ! https://devguide.python.org/versions/#supported-versions - remote: ! - remote: ! As such, support for Python 3.9 will be removed from this - remote: ! buildpack on 7th January 2026. - remote: ! - remote: ! Upgrade to a newer Python version as soon as possible, by - remote: ! changing the version in your .python-version file. - remote: ! - remote: ! For more information, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions - remote: - remote: -----> Installing Python #{LATEST_PYTHON_3_9} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} - remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) - OUTPUT - expect(app.run('python -V')).to eq("Python #{LATEST_PYTHON_3_9}\n") - expect($CHILD_STATUS.exitstatus).to eq(0) - end - end - end - context 'when .python-version contains Python 3.10' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.10') } @@ -500,18 +463,18 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python 3.8 specified in .python-version + remote: -----> Using Python 3.9 specified in .python-version remote: remote: ! Error: The requested Python version has reached end-of-life. remote: ! - remote: ! Python 3.8 has reached its upstream end-of-life, and is + remote: ! Python 3.9 has reached its upstream end-of-life, and is remote: ! therefore no longer receiving security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! remote: ! As such, it's no longer supported by this buildpack: remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: ! - remote: ! Please upgrade to at least Python 3.9 by changing the + remote: ! Please upgrade to at least Python 3.10 by changing the remote: ! version in your .python-version file. remote: ! remote: ! If possible, we recommend upgrading all the way to Python #{DEFAULT_PYTHON_MAJOR_VERSION}, @@ -691,7 +654,7 @@ remote: ! As such, it's no longer supported by this buildpack: remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: ! - remote: ! Please upgrade to at least Python 3.9 by changing the + remote: ! Please upgrade to at least Python 3.10 by changing the remote: ! version in your runtime.txt file. remote: ! remote: ! If possible, we recommend upgrading all the way to Python #{DEFAULT_PYTHON_MAJOR_VERSION}, @@ -753,7 +716,7 @@ end context 'when the requested Python version has changed since the last build' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.9') } + let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.10') } it 'builds with the new Python version after removing the old install' do app.deploy do |app| @@ -764,7 +727,7 @@ remote: -----> Python app detected remote: -----> Using Python 3.13 specified in .python-version remote: -----> Discarding cache since: - remote: - The Python version has changed from #{LATEST_PYTHON_3_9} to #{LATEST_PYTHON_3_13} + remote: - The Python version has changed from #{LATEST_PYTHON_3_10} to #{LATEST_PYTHON_3_13} remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 8f73d00b2..df8c40782 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -258,16 +258,16 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python 3.9.0 specified in .python-version + remote: -----> Using Python 3.10.0 specified in .python-version remote: - remote: ! Warning: Support for Python 3.9 is ending soon! + remote: ! Warning: Support for Python 3.10 is deprecated! remote: ! - remote: ! Python 3.9 reached its upstream end-of-life on 31st October 2025, - remote: ! and so no longer receives security updates: + remote: ! Python 3.10 will reach its upstream end-of-life in October 2026, + remote: ! at which point it will no longer receive security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! - remote: ! As such, support for Python 3.9 will be removed from this - remote: ! buildpack on 7th January 2026. + remote: ! As such, support for Python 3.10 will be removed from this + remote: ! buildpack on 6th January 2027. remote: ! remote: ! Upgrade to a newer Python version as soon as possible, by remote: ! changing the version in your .python-version file. @@ -278,8 +278,8 @@ remote: remote: ! Warning: A Python patch update is available! remote: ! - remote: ! Your app is using Python 3.9.0, however, there is a newer - remote: ! patch release of Python 3.9 available: #{LATEST_PYTHON_3_9} + remote: ! Your app is using Python 3.10.0, however, there is a newer + remote: ! patch release of Python 3.10 available: #{LATEST_PYTHON_3_10} remote: ! remote: ! It is important to always use the latest patch version of remote: ! Python to keep your app secure. @@ -287,12 +287,12 @@ remote: ! Update your .python-version file to use the new version. remote: ! remote: ! We strongly recommend that you don't pin your app to an - remote: ! exact Python version such as 3.9.0, and instead only specify - remote: ! the major Python version of 3.9 in your .python-version file. + remote: ! exact Python version such as 3.10.0, and instead only specify + remote: ! the major Python version of 3.10 in your .python-version file. remote: ! This will allow your app to receive the latest available Python remote: ! patch version automatically and prevent this warning. remote: - remote: -----> Installing Python 3.9.0 + remote: -----> Installing Python 3.10.0 remote: -----> Installing uv #{UV_VERSION} remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' remote: Resolved 2 packages in .+s diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index cf389801d..9f7bc22aa 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -10,7 +10,6 @@ FIXTURE_DIR = Pathname.new(__FILE__).parent.join('fixtures') -LATEST_PYTHON_3_9 = '3.9.25' LATEST_PYTHON_3_10 = '3.10.19' LATEST_PYTHON_3_11 = '3.11.14' LATEST_PYTHON_3_12 = '3.12.12' From 76f07d92abadf8d85685f6850dae30ce075426f8 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 7 Jan 2026 13:25:39 +0000 Subject: [PATCH 313/406] Prepare release v327 (#2007) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c07f8ed7a..768240539 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v327] - 2026-01-07 + - Removed support for Python 3.9. ([#2005](https://github.com/heroku/heroku-buildpack-python/pull/2005)) - Adjusted the error message shown if SQLite headers aren't found during package installation. ([#2006](https://github.com/heroku/heroku-buildpack-python/pull/2006)) @@ -1439,7 +1442,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v326...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v327...main +[v327]: https://github.com/heroku/heroku-buildpack-python/compare/v326...v327 [v326]: https://github.com/heroku/heroku-buildpack-python/compare/v325...v326 [v325]: https://github.com/heroku/heroku-buildpack-python/compare/v324...v325 [v324]: https://github.com/heroku/heroku-buildpack-python/compare/v323...v324 From 66de50f88d44088dbad31abaefd943a8403bab4b Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 7 Jan 2026 23:41:28 +0000 Subject: [PATCH 314/406] Update pip from 25.2 to 25.3 (#2008) Changelog: https://pip.pypa.io/en/stable/news/#v25-3 Additionally, we now no longer install the `wheel` package when using pip with Python 3.12 and older, since: - With modern setuptools, it's no longer necessary to manually install wheel, since setuptools will trigger its installation if needed. - As of pip 25.3, pip prepares/builds dependencies inside an isolated ephemeral build environment (~venv) that means even if wheel is installed globally, it won't be used anyway. - The Python CNB (and much of the rest of the Python ecosystem) doesn't install wheel globally any more either. This doesn't affect apps using either a different package manager, or that are using pip with Python 3.13 or newer, since we already didn't install the wheel package in those environments. GUS-W-17605676. GUS-W-20804745. --- CHANGELOG.md | 2 + lib/pip.sh | 13 ++--- requirements/pip.txt | 2 +- requirements/wheel.txt | 1 - spec/fixtures/pip_editable/.python-version | 4 +- .../pip_editable/bin/test-entrypoints.sh | 4 +- spec/fixtures/pip_editable/requirements.txt | 2 +- spec/hatchet/pip_spec.rb | 54 ++++++++----------- spec/hatchet/python_version_spec.rb | 6 +-- spec/hatchet/stack_spec.rb | 2 +- spec/spec_helper.rb | 1 - 11 files changed, 37 insertions(+), 54 deletions(-) delete mode 100644 requirements/wheel.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index 768240539..6a065ac4f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## [Unreleased] +- Updated pip from 25.2 to 25.3. ([#2008](https://github.com/heroku/heroku-buildpack-python/pull/2008)) +- Stopped installing wheel when using pip with Python 3.12 and older. ([#2008](https://github.com/heroku/heroku-buildpack-python/pull/2008)) ## [v327] - 2026-01-07 diff --git a/lib/pip.sh b/lib/pip.sh index 21e5a9e15..dc26c39e5 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -6,7 +6,6 @@ set -euo pipefail PIP_VERSION=$(utils::get_requirement_version 'pip') SETUPTOOLS_VERSION=$(utils::get_requirement_version 'setuptools') -WHEEL_VERSION=$(utils::get_requirement_version 'wheel') function pip::install_pip() { local python_home="${1}" @@ -21,21 +20,17 @@ function pip::install_pip() { ) local packages_display_text="pip ${PIP_VERSION}" - # We only install setuptools and wheel on Python 3.12 and older, since: - # - If either is not installed, pip will automatically install them into an isolated build - # environment if needed when installing packages from an sdist. This means that for - # all packages that correctly declare their metadata, it's no longer necessary to have - # them installed. + # We only install setuptools on Python 3.12 and older, since: + # - pip now uses isolated build environments into which it installed setuptools and wheel + # if needed when installing packages from an sdist. # - Most of the Python ecosystem has stopped installing them for Python 3.12+ already. # See the Python CNB's removal for more details: https://github.com/heroku/buildpacks-python/pull/243 if [[ "${python_major_version}" == +(3.10|3.11|3.12) ]]; then build_data::set_string "setuptools_version" "${SETUPTOOLS_VERSION}" - build_data::set_string "wheel_version" "${WHEEL_VERSION}" packages_to_install+=( "setuptools==${SETUPTOOLS_VERSION}" - "wheel==${WHEEL_VERSION}" ) - packages_display_text+=", setuptools ${SETUPTOOLS_VERSION} and wheel ${WHEEL_VERSION}" + packages_display_text+=" and setuptools ${SETUPTOOLS_VERSION}" fi # Note: We still perform this install step even if the cache was reused, since we have no guarantee diff --git a/requirements/pip.txt b/requirements/pip.txt index 7d4044f41..0b3d671e6 100644 --- a/requirements/pip.txt +++ b/requirements/pip.txt @@ -1 +1 @@ -pip==25.2 +pip==25.3 diff --git a/requirements/wheel.txt b/requirements/wheel.txt deleted file mode 100644 index 0a0864cb4..000000000 --- a/requirements/wheel.txt +++ /dev/null @@ -1 +0,0 @@ -wheel==0.45.1 diff --git a/spec/fixtures/pip_editable/.python-version b/spec/fixtures/pip_editable/.python-version index 9b9fc4643..6324d401a 100644 --- a/spec/fixtures/pip_editable/.python-version +++ b/spec/fixtures/pip_editable/.python-version @@ -1,3 +1 @@ -# Note: This test intentionally uses Python 3.12, so that we test *.egg-link -# path rewriting using older globally installed setuptools. -3.12 +3.14 diff --git a/spec/fixtures/pip_editable/bin/test-entrypoints.sh b/spec/fixtures/pip_editable/bin/test-entrypoints.sh index 43afcb553..0e66f44bb 100755 --- a/spec/fixtures/pip_editable/bin/test-entrypoints.sh +++ b/spec/fixtures/pip_editable/bin/test-entrypoints.sh @@ -4,8 +4,8 @@ set -euo pipefail cd .heroku/python/lib/python*/site-packages/ -# List any path like strings in the .egg-link, .pth, and finder files in site-packages. -grep --extended-regexp --only-matching -- '/\S+' *.egg-link *.pth __editable___*_finder.py | sort +# List any path like strings in the .pth and finder files in site-packages. +grep --extended-regexp --only-matching -- '/\S+' *.pth __editable___*_finder.py | sort echo echo -n "Running entrypoint for the pyproject.toml-based local package: " diff --git a/spec/fixtures/pip_editable/requirements.txt b/spec/fixtures/pip_editable/requirements.txt index 54de167d1..a514dabc1 100644 --- a/spec/fixtures/pip_editable/requirements.txt +++ b/spec/fixtures/pip_editable/requirements.txt @@ -2,4 +2,4 @@ # https://github.com/pypa/setuptools/issues/3535 -e ./packages/local_package_pyproject_toml -e ./packages/local_package_setup_py --e git+https://github.com/benoitc/gunicorn@20.1.0#egg=gunicorn +-e git+https://github.com/benoitc/gunicorn@56b5ad87f8d72a674145c273ed8f547513c2b409#egg=gunicorn diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index fbab2c174..428c26a63 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -163,51 +163,42 @@ end end - # This test intentionally uses Python 3.12, so that we test rewriting using older globally installed - # setuptools (which causes .egg-link files to be created too). The Pipenv and Poetry equivalents of - # this test covers the PEP-517/518 setuptools case. context 'when requirements.txt contains editable requirements (both VCS and local package)' do let(:buildpacks) { [:default, 'heroku-community/inline'] } let(:app) { Hatchet::Runner.new('spec/fixtures/pip_editable', buildpacks:) } - it 'rewrites .pth, .egg-link and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do + it 'rewrites .pth and finder paths correctly for hooks, later buildpacks, runtime and cached builds' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Running bin/post_compile hook - remote: easy-install.pth:/app/.heroku/python/src/gunicorn - remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/python/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Saving cache .+ remote: -----> Inline app detected - remote: easy-install.pth:/app/.heroku/python/src/gunicorn - remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/python/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) REGEX # Test rewritten paths work at runtime. expect(app.run('bin/test-entrypoints.sh')).to include(<<~OUTPUT) - easy-install.pth:/app/.heroku/python/src/gunicorn - easy-install.pth:/app/packages/local_package_setup_py + __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} __editable___local_package_pyproject_toml_0_0_1_finder.py:/app/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - gunicorn.egg-link:/app/.heroku/python/src/gunicorn - local-package-setup-py.egg-link:/app/packages/local_package_setup_py + __editable___local_package_setup_py_0_0_1_finder.py:/app/packages/local_package_setup_py/local_package_setup_py'} Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! Running entrypoint for the setup.py-based local package: Hello from setup.py! - Running entrypoint for the VCS package: gunicorn (version 20.1.0) + Running entrypoint for the VCS package: gunicorn (version 23.0.0) OUTPUT # Test that the cached .pth files work correctly. @@ -215,30 +206,29 @@ app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Running bin/post_compile hook - remote: easy-install.pth:/app/.heroku/python/src/gunicorn - remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/python/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) remote: -----> Saving cache .+ remote: -----> Inline app detected - remote: easy-install.pth:/app/.heroku/python/src/gunicorn - remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} - remote: gunicorn.egg-link:/app/.heroku/python/src/gunicorn - remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py + remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} remote: remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! remote: Running entrypoint for the setup.py-based local package: Hello from setup.py! - remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\) + remote: Running entrypoint for the VCS package: gunicorn \\(version 23.0.0\\) REGEX # Test that the VCS repo checkout was cached correctly. - expect(app.output).to include('Updating /app/.heroku/python/src/gunicorn clone (to revision 20.1.0)') + expect(app.output).to include(<<~OUTPUT) + remote: Obtaining gunicorn from git+https://github.com/benoitc/gunicorn@56b5ad87f8d72a674145c273ed8f547513c2b409#egg=gunicorn (from -r requirements.txt (line 5)) + remote: Skipping because already up-to-date. + OUTPUT end end end @@ -318,7 +308,7 @@ remote: ! patch version automatically and prevent this warning. remote: remote: -----> Installing Python 3.10.0 - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} and setuptools #{SETUPTOOLS_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) remote: Downloading typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB) @@ -378,7 +368,7 @@ it 'outputs instructions for how to resolve the build failure' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: note: This error originates from a subprocess, and is likely not a problem with pip. + remote: ERROR: Failed to build 'GDAL' when getting requirements to build wheel remote: remote: ! Error: Package installation failed since the GDAL library wasn't found. remote: ! diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 73ddec145..2b70356ab 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -10,7 +10,7 @@ remote: -----> Python app detected remote: -----> Using Python #{requested_version} specified in .python-version remote: -----> Installing Python #{resolved_version} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} and setuptools #{SETUPTOOLS_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) OUTPUT @@ -152,7 +152,7 @@ remote: - The pip version has changed from 24.0 to #{PIP_VERSION} remote: - The legacy SQLite3 headers and CLI binary need to be uninstalled remote: -----> Installing Python #{LATEST_PYTHON_3_12} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} and setuptools #{SETUPTOOLS_VERSION} OUTPUT expect(app.run('python -V')).to eq("Python #{LATEST_PYTHON_3_12}\n") expect($CHILD_STATUS.exitstatus).to eq(0) @@ -274,7 +274,7 @@ remote: ! https://devcenter.heroku.com/articles/python-support#supported-python-versions remote: remote: -----> Installing Python #{LATEST_PYTHON_3_10} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} and setuptools #{SETUPTOOLS_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) OUTPUT diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index 76471daf8..8fe115496 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -69,7 +69,7 @@ remote: - The buildpack cache format has changed remote: - The legacy SQLite3 headers and CLI binary need to be uninstalled remote: -----> Installing Python #{LATEST_PYTHON_3_12} - remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} + remote: -----> Installing pip #{PIP_VERSION} and setuptools #{SETUPTOOLS_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) OUTPUT diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 9f7bc22aa..fba5874f7 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -29,7 +29,6 @@ def get_requirement_version(package_name) PIP_VERSION = get_requirement_version('pip') SETUPTOOLS_VERSION = get_requirement_version('setuptools') -WHEEL_VERSION = get_requirement_version('wheel') PIPENV_VERSION = get_requirement_version('pipenv') POETRY_VERSION = get_requirement_version('poetry') UV_VERSION = get_requirement_version('uv') From 6a5c28da53126037e7e04cdad6b88f69d2ac7f39 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 7 Jan 2026 23:44:57 +0000 Subject: [PATCH 315/406] Prepare release v328 (#2009) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a065ac4f..57c9891cc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v328] - 2026-01-07 + - Updated pip from 25.2 to 25.3. ([#2008](https://github.com/heroku/heroku-buildpack-python/pull/2008)) - Stopped installing wheel when using pip with Python 3.12 and older. ([#2008](https://github.com/heroku/heroku-buildpack-python/pull/2008)) @@ -1444,7 +1447,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v327...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v328...main +[v328]: https://github.com/heroku/heroku-buildpack-python/compare/v327...v328 [v327]: https://github.com/heroku/heroku-buildpack-python/compare/v326...v327 [v326]: https://github.com/heroku/heroku-buildpack-python/compare/v325...v326 [v325]: https://github.com/heroku/heroku-buildpack-python/compare/v324...v325 From 38713f330684cc1bbf719a23a7f22e6008aa1e81 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 8 Jan 2026 10:00:50 +0000 Subject: [PATCH 316/406] Bump pipenv from 2025.0.4 to 2026.0.3 (#2000) * Bump pipenv from 2025.0.4 to 2026.0.3 Bumps [pipenv](https://github.com/pypa/pipenv) from 2025.0.4 to 2026.0.3. - [Release notes](https://github.com/pypa/pipenv/releases) - [Changelog](https://github.com/pypa/pipenv/blob/main/CHANGELOG.md) - [Commits](https://github.com/pypa/pipenv/compare/v2025.0.4...v2026.0.3) --- updated-dependencies: - dependency-name: pipenv dependency-version: 2026.0.3 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] * Add changelog entry * Update tests * Remove workarounds for bugs that are now fixed --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + lib/pipenv.sh | 12 ----------- requirements/pipenv.txt | 2 +- spec/hatchet/ci_spec.rb | 6 ------ spec/hatchet/pipenv_spec.rb | 43 +++++++++++++++++++------------------ 5 files changed, 24 insertions(+), 40 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 57c9891cc..2960a8c41 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated Pipenv from 2025.0.4 to 2026.0.3. ([#2000](https://github.com/heroku/heroku-buildpack-python/pull/2000)) ## [v328] - 2026-01-07 diff --git a/lib/pipenv.sh b/lib/pipenv.sh index 84b00ad4b..1a87d4f49 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -81,29 +81,17 @@ function pipenv::install_pipenv() { export PATH="${pipenv_bin_dir}:${PATH}" # Force Pipenv to manage the system Python site-packages instead of using venvs. export PIPENV_SYSTEM="1" - # Hide Pipenv's notice about finding/using an existing virtual environment. - export PIPENV_VERBOSITY="-1" - # Work around a Pipenv bug when using `--system`, whereby it doesn't correctly install - # dependencies that happen to also be a dependency of Pipenv (such as `certifi`). - # In general Pipenv's support for its `--system` mode seems very buggy. Longer term we - # should explore moving to venvs, however, that will need to be coordinated across all - # package managers and also change paths for Python which could break other use cases. - export VIRTUAL_ENV="${python_home}" # Set the same env vars in the environment used by later buildpacks. cat >>"${export_file}" <<-EOF export PATH="${pipenv_bin_dir}:\${PATH}" export PIPENV_SYSTEM="1" - export PIPENV_VERBOSITY="-1" - export VIRTUAL_ENV="${python_home}" EOF # And the environment used at app run-time. cat >>"${profile_d_file}" <<-EOF export PATH="${pipenv_bin_dir}:\${PATH}" export PIPENV_SYSTEM="1" - export PIPENV_VERBOSITY="-1" - export VIRTUAL_ENV="${python_home}" EOF } diff --git a/requirements/pipenv.txt b/requirements/pipenv.txt index b239da87f..faed680ba 100644 --- a/requirements/pipenv.txt +++ b/requirements/pipenv.txt @@ -1 +1 @@ -pipenv==2025.0.4 +pipenv==2026.0.3 diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 2cc0b9acc..e44effbf0 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -123,10 +123,8 @@ LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/pipenv/bin:/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PIPENV_SYSTEM=1 - PIPENV_VERBOSITY=-1 PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 - VIRTUAL_ENV=/app/.heroku/python -----> Saving cache ! Note: We recently added support for the package manager uv: @@ -145,11 +143,9 @@ LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PIPENV_SYSTEM=1 - PIPENV_VERBOSITY=-1 PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true - VIRTUAL_ENV=/app/.heroku/python -----> No test-setup command provided. Skipping. -----> Running test command `./bin/print-env-vars.sh && pytest --version`... CI=true @@ -161,11 +157,9 @@ LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ PIPENV_SYSTEM=1 - PIPENV_VERBOSITY=-1 PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true - VIRTUAL_ENV=/app/.heroku/python WEB_CONCURRENCY=5 pytest .+ -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index f0bc4c5f3..d922f136b 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -27,10 +27,8 @@ remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/pipenv/bin:/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PIPENV_SYSTEM=1 - remote: PIPENV_VERBOSITY=-1 remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config remote: PYTHONUNBUFFERED=1 - remote: VIRTUAL_ENV=/app/.heroku/python remote: -----> Saving cache remote: remote: ! Note: We recently added support for the package manager uv: @@ -49,11 +47,9 @@ remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PIPENV_SYSTEM=1 - remote: PIPENV_VERBOSITY=-1 remote: PYTHONHOME=/app/.heroku/python remote: PYTHONPATH=/app remote: PYTHONUNBUFFERED=true - remote: VIRTUAL_ENV=/app/.heroku/python remote: remote: \\['', remote: '/app', @@ -119,11 +115,9 @@ LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin PIPENV_SYSTEM=1 - PIPENV_VERBOSITY=-1 PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true - VIRTUAL_ENV=/app/.heroku/python WEB_CONCURRENCY=2 pipenv, version #{PIPENV_VERSION} OUTPUT @@ -455,8 +449,7 @@ end end - # TODO: Rename this test description back this when the Pipenv version next changes. - context 'when the Python version has changed since the last build' do + context 'when the Pipenv and Python versions have changed since the last build' do let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v313'] } let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_basic', buildpacks:) } @@ -471,6 +464,7 @@ remote: -----> Using Python 3.14 specified in Pipfile.lock remote: -----> Discarding cache since: remote: - The Python version has changed from 3.14.0 to #{LATEST_PYTHON_3_14} + remote: - The Pipenv version has changed from 2025.0.4 to #{PIPENV_VERSION} remote: -----> Installing Python #{LATEST_PYTHON_3_14} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' @@ -560,21 +554,28 @@ end end - # This is disabled since it's currently broken upstream: https://github.com/pypa/pipenv/issues/6403 # This tests that Pipenv doesn't fall back to system Python if the Python version in # pyproject.toml doesn't match that in Pipfile / Pipfile.lock. - # context 'when requires-python in pyproject.toml is incompatible with .python-version' do - # let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_mismatched_python_version', allow_failure: true) } - # - # it 'fails the build' do - # app.deploy do |app| - # expect(clean_output(app.output)).to include(<<~OUTPUT) - # remote: -----> Installing dependencies using 'pipenv install --deploy' - # remote: - # OUTPUT - # end - # end - # end + context 'when requires-python in pyproject.toml is incompatible with .python-version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_mismatched_python_version', allow_failure: true) } + + it 'fails the build' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Installing dependencies using 'pipenv install --deploy' + remote: Warning: Your Pipfile requires "python_version" 3.12, but you are using #{LATEST_PYTHON_3_13} + remote: from //app/./python/bin/python3. + remote: Usage: pipenv install [OPTIONS] [PACKAGES]... + remote: + remote: ERROR:: Aborting deploy + remote: + remote: ! Error: Unable to install dependencies using Pipenv. + remote: ! + remote: ! See the log output above for more information. + OUTPUT + end + end + end # This tests not only our handling of failing dependency installation, but also that we're running # Pipenv in such a way that it errors if the lockfile is out of sync, rather than simply updating it. From f2c677374c57b8790f38c2a03ebf41d6af92b5d7 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 8 Jan 2026 10:05:51 +0000 Subject: [PATCH 317/406] Prepare release v329 (#2010) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2960a8c41..e412f0077 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v329] - 2026-01-08 + - Updated Pipenv from 2025.0.4 to 2026.0.3. ([#2000](https://github.com/heroku/heroku-buildpack-python/pull/2000)) ## [v328] - 2026-01-07 @@ -1448,7 +1451,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v328...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v329...main +[v329]: https://github.com/heroku/heroku-buildpack-python/compare/v328...v329 [v328]: https://github.com/heroku/heroku-buildpack-python/compare/v327...v328 [v327]: https://github.com/heroku/heroku-buildpack-python/compare/v326...v327 [v326]: https://github.com/heroku/heroku-buildpack-python/compare/v325...v326 From 1b26c62ba9549939e9db9212e52cea66c83dc254 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 8 Jan 2026 16:40:46 +0000 Subject: [PATCH 318/406] Re-apply Pipenv `--system` bug workaround (#2011) In #2000, Pipenv was updated to v2026.0.3, which contained a number of fixes for the historically very buggy `--system` mode. Since that Pipenv release was meant to fix many of the `--system` bugs, one of the workarounds for those bugs was removed, since even without the workaround the regression test for that bug still passed (implying the upstream fixes had worked as expected). However, it appears that the bug still exists in some form that was not being picked up by the original testcase, as reported in: https://github.com/heroku/heroku-buildpack-python/pull/2000#discussion_r2672712400 As such, I've had to re-add the workaround. This also affects the logs, since in the new Pipenv version, if `PIPENV_VERBOSITY="-1"` is set (which is required to stop an annoying multi-line warning about a venv being present), then the `Installing dependencies from ...` line is no longer printed. It seems the bug now only reproduces with different versions of the `certifi` package, so I've updated the test fixture accordingly. (My first thought was that perhaps the difference is whether our testcase's `certifi` matches the Pipenv vendored version exactly or not, however, varying to another older version also didn't reproduce?). I've also added `packaging` as another testcase dependency, in the hope that in the future at least one of these packages will be able to reproduce the issue, should we try and remove the workaround again. Longer term, these continual Pipenv bugs are making me inclined to deprecate/strongly discourage Pipenv usage, given there are now much more reliable/better maintained/faster alternatives (such as uv) that don't suffer from these chronic bugs. GUS-W-20819376. --- CHANGELOG.md | 3 ++- lib/pipenv.sh | 14 ++++++++++++++ spec/fixtures/pipenv_basic/Pipfile | 5 ++++- spec/fixtures/pipenv_basic/Pipfile.lock | 17 +++++++++++++---- spec/fixtures/pipenv_basic/bin/compile | 4 ++++ spec/hatchet/ci_spec.rb | 10 ++++++---- spec/hatchet/pipenv_spec.rb | 20 ++++++++++---------- 7 files changed, 53 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e412f0077..ab7918cb8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Re-apply a workaround for a Pipenv bug when using `--system`, that causes packages to not be installed correctly if they are also a dependency of Pipenv (such as `certifi` or `packaging`). ([#2011](https://github.com/heroku/heroku-buildpack-python/pull/2011)) ## [v329] - 2026-01-08 @@ -197,7 +198,7 @@ ## [v293] - 2025-07-23 -- Work around a Pipenv bug when using `--system`, that causes packages to not be installed correctly if they are also a dependency of Pipenv (such as `certifi` ). ([#1842](https://github.com/heroku/heroku-buildpack-python/pull/1842)) +- Work around a Pipenv bug when using `--system`, that causes packages to not be installed correctly if they are also a dependency of Pipenv (such as `certifi`). ([#1842](https://github.com/heroku/heroku-buildpack-python/pull/1842)) ## [v292] - 2025-07-23 diff --git a/lib/pipenv.sh b/lib/pipenv.sh index 1a87d4f49..073f00e87 100644 --- a/lib/pipenv.sh +++ b/lib/pipenv.sh @@ -81,17 +81,31 @@ function pipenv::install_pipenv() { export PATH="${pipenv_bin_dir}:${PATH}" # Force Pipenv to manage the system Python site-packages instead of using venvs. export PIPENV_SYSTEM="1" + # Hide Pipenv's notice about finding/using an existing virtual environment. + export PIPENV_VERBOSITY="-1" + # Work around a Pipenv bug when using `--system`, whereby it doesn't correctly install dependencies + # that happen to also be a dependency of Pipenv (such as `certifi` and `packaging`). In general + # Pipenv's support for its `--system` mode is very buggy. Longer term we should explore moving + # to venvs, however, that will need to be coordinated across all package managers and will also + # change paths for Python which could break other use cases. Be careful removing this even if the + # `pip_basic` test that installs certifi/packaging still passes, since the repro seems to depend + # on specific package version combinations / other factors and so the testcase is very fragile. + export VIRTUAL_ENV="${python_home}" # Set the same env vars in the environment used by later buildpacks. cat >>"${export_file}" <<-EOF export PATH="${pipenv_bin_dir}:\${PATH}" export PIPENV_SYSTEM="1" + export PIPENV_VERBOSITY="-1" + export VIRTUAL_ENV="${python_home}" EOF # And the environment used at app run-time. cat >>"${profile_d_file}" <<-EOF export PATH="${pipenv_bin_dir}:\${PATH}" export PIPENV_SYSTEM="1" + export PIPENV_VERBOSITY="-1" + export VIRTUAL_ENV="${python_home}" EOF } diff --git a/spec/fixtures/pipenv_basic/Pipfile b/spec/fixtures/pipenv_basic/Pipfile index 8c06ae360..ec52c677c 100644 --- a/spec/fixtures/pipenv_basic/Pipfile +++ b/spec/fixtures/pipenv_basic/Pipfile @@ -4,8 +4,11 @@ verify_ssl = true name = "pypi" [packages] -certifi = "*" typing-extensions = "*" +# Test that dependencies that happen to also be vendored dependencies of Pipenv are correctly installed, +# since Pipenv's --system mode is buggy and requires a workaround to ensure they aren't skipped. +certifi = "*" +packaging = "*" [dev-packages] diff --git a/spec/fixtures/pipenv_basic/Pipfile.lock b/spec/fixtures/pipenv_basic/Pipfile.lock index 223afa115..059ed0e08 100644 --- a/spec/fixtures/pipenv_basic/Pipfile.lock +++ b/spec/fixtures/pipenv_basic/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "8f362a119150f17832636f824f9f334a4d5580c6713519ed0865b89c4cc4e845" + "sha256": "0ef98f1284f466d096b821c028367648e5141cc11150e42daa0b60172cbe5f43" }, "pipfile-spec": 6, "requires": { @@ -18,12 +18,21 @@ "default": { "certifi": { "hashes": [ - "sha256:97de8790030bbd5c2d96b7ec782fc2f7820ef8dba6db909ccf95449f2d062d4b", - "sha256:d8ab5478f2ecd78af242878415affce761ca6bc54a22a27e026d7c25357c3316" + "sha256:9943707519e4add1115f44c2bc244f782c0249876bf51b6599fee1ffbedd685c", + "sha256:ac726dd470482006e014ad384921ed6438c457018f4b3d204aea4281258b2120" ], "index": "pypi", "markers": "python_version >= '3.7'", - "version": "==2025.11.12" + "version": "==2026.1.4" + }, + "packaging": { + "hashes": [ + "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484", + "sha256:d443872c98d677bf60f6a1f2f8c1cb748e8fe762d2bf9d3148b5599295b0fc4f" + ], + "index": "pypi", + "markers": "python_version >= '3.8'", + "version": "==25.0" }, "typing-extensions": { "hashes": [ diff --git a/spec/fixtures/pipenv_basic/bin/compile b/spec/fixtures/pipenv_basic/bin/compile index 306c295fc..31c5a74fb 100755 --- a/spec/fixtures/pipenv_basic/bin/compile +++ b/spec/fixtures/pipenv_basic/bin/compile @@ -23,6 +23,10 @@ pip list --disable-pip-version-check --exclude pip echo python -c 'import typing_extensions; print(typing_extensions)' +# Test that dependencies that happen to also be vendored dependencies of Pipenv are correctly installed, +# since Pipenv's --system mode is buggy and requires a workaround to ensure they aren't skipped. +python -c 'import certifi; print(certifi)' +python -c 'import packaging; print(packaging)' echo jq --sort-keys '.' "${CACHE_DIR}/build-data/python.json" diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index e44effbf0..7d74aa6e3 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -105,8 +105,6 @@ -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing Pipenv #{PIPENV_VERSION} -----> Installing dependencies using 'pipenv install --deploy --dev' - Installing dependencies from Pipfile.lock \\(.+\\)... - Installing dependencies from Pipfile.lock \\(.+\\)... -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running bin/post_compile hook BUILD_DIR=/app @@ -123,8 +121,10 @@ LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/pipenv/bin:/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PIPENV_SYSTEM=1 + PIPENV_VERBOSITY=-1 PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 + VIRTUAL_ENV=/app/.heroku/python -----> Saving cache ! Note: We recently added support for the package manager uv: @@ -143,9 +143,11 @@ LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PIPENV_SYSTEM=1 + PIPENV_VERBOSITY=-1 PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true + VIRTUAL_ENV=/app/.heroku/python -----> No test-setup command provided. Skipping. -----> Running test command `./bin/print-env-vars.sh && pytest --version`... CI=true @@ -157,9 +159,11 @@ LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/ PIPENV_SYSTEM=1 + PIPENV_VERBOSITY=-1 PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true + VIRTUAL_ENV=/app/.heroku/python WEB_CONCURRENCY=5 pytest .+ -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully @@ -173,8 +177,6 @@ -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Using cached Pipenv #{PIPENV_VERSION} -----> Installing dependencies using 'pipenv install --deploy --dev' - Installing dependencies from Pipfile.lock \\(.+\\)... - Installing dependencies from Pipfile.lock \\(.+\\)... -----> Skipping Django collectstatic since the env var DISABLE_COLLECTSTATIC is set. -----> Running bin/post_compile hook REGEX diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index d922f136b..c1ad3b0cd 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -15,7 +15,6 @@ remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Running bin/post_compile hook remote: BUILD_DIR=/tmp/build_.+ remote: CACHE_DIR=/tmp/codon/tmp/cache @@ -27,8 +26,10 @@ remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/pipenv/bin:/app/.heroku/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PIPENV_SYSTEM=1 + remote: PIPENV_VERBOSITY=-1 remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config remote: PYTHONUNBUFFERED=1 + remote: VIRTUAL_ENV=/app/.heroku/python remote: -----> Saving cache remote: remote: ! Note: We recently added support for the package manager uv: @@ -47,9 +48,11 @@ remote: LIBRARY_PATH=/app/.heroku/python/lib remote: PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin remote: PIPENV_SYSTEM=1 + remote: PIPENV_VERBOSITY=-1 remote: PYTHONHOME=/app/.heroku/python remote: PYTHONPATH=/app remote: PYTHONUNBUFFERED=true + remote: VIRTUAL_ENV=/app/.heroku/python remote: remote: \\['', remote: '/app', @@ -61,10 +64,13 @@ remote: pipenv, version #{PIPENV_VERSION} remote: Package Version remote: ----------------- -+ - remote: certifi 2025.11.12 + remote: certifi 2026.1.4 + remote: packaging 25.0 remote: typing_extensions 4.15.0 remote: remote: + remote: + remote: remote: remote: \\{ remote: "cache_restore_duration": [0-9.]+, @@ -99,7 +105,6 @@ remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Using cached Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Running bin/post_compile hook remote: .+ remote: -----> Saving cache @@ -115,9 +120,11 @@ LIBRARY_PATH=/app/.heroku/python/lib PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin PIPENV_SYSTEM=1 + PIPENV_VERBOSITY=-1 PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true + VIRTUAL_ENV=/app/.heroku/python WEB_CONCURRENCY=2 pipenv, version #{PIPENV_VERSION} OUTPUT @@ -143,7 +150,6 @@ remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Saving cache REGEX end @@ -198,7 +204,6 @@ remote: -----> Installing Python 3.10.0 remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Saving cache REGEX end @@ -216,7 +221,6 @@ remote: -----> Installing Python #{LATEST_PYTHON_3_13} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Saving cache REGEX end @@ -275,7 +279,6 @@ remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Saving cache REGEX end @@ -468,7 +471,6 @@ remote: -----> Installing Python #{LATEST_PYTHON_3_14} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Saving cache REGEX end @@ -483,7 +485,6 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Running bin/post_compile hook remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} @@ -526,7 +527,6 @@ app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Installing dependencies using 'pipenv install --deploy' - remote: Installing dependencies from Pipfile.lock \\(.+\\)... remote: -----> Running bin/post_compile hook remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} From f02fbdb9559af0de1ed7707988c4aca79fbb1d84 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 8 Jan 2026 16:46:00 +0000 Subject: [PATCH 319/406] Prepare release v330 (#2012) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ab7918cb8..f72f75f8d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v330] - 2026-01-08 + - Re-apply a workaround for a Pipenv bug when using `--system`, that causes packages to not be installed correctly if they are also a dependency of Pipenv (such as `certifi` or `packaging`). ([#2011](https://github.com/heroku/heroku-buildpack-python/pull/2011)) ## [v329] - 2026-01-08 @@ -1452,7 +1455,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v329...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v330...main +[v330]: https://github.com/heroku/heroku-buildpack-python/compare/v329...v330 [v329]: https://github.com/heroku/heroku-buildpack-python/compare/v328...v329 [v328]: https://github.com/heroku/heroku-buildpack-python/compare/v327...v328 [v327]: https://github.com/heroku/heroku-buildpack-python/compare/v326...v327 From 47610636a69dfe2df9336db6f0bfd5923cf0bdfb Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 11 Jan 2026 11:40:25 +0000 Subject: [PATCH 320/406] Bump uv from 0.9.21 to 0.9.24 (#2013) * Bump uv from 0.9.21 to 0.9.24 Bumps [uv](https://github.com/astral-sh/uv) from 0.9.21 to 0.9.24. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.9.21...0.9.24) --- updated-dependencies: - dependency-name: uv dependency-version: 0.9.24 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f72f75f8d..981118102 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.9.21 to 0.9.24. ([#2013](https://github.com/heroku/heroku-buildpack-python/pull/2013)) ## [v330] - 2026-01-08 diff --git a/requirements/uv.txt b/requirements/uv.txt index f25857315..55bf7c7ea 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.9.21 +uv==0.9.24 From 4878b111ba81630cf1b76d11dc64ca339c86321c Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Sun, 11 Jan 2026 11:44:13 +0000 Subject: [PATCH 321/406] Prepare release v331 (#2014) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 981118102..710273e9e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v331] - 2026-01-11 + - Updated uv from 0.9.21 to 0.9.24. ([#2013](https://github.com/heroku/heroku-buildpack-python/pull/2013)) ## [v330] - 2026-01-08 @@ -1456,7 +1459,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v330...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v331...main +[v331]: https://github.com/heroku/heroku-buildpack-python/compare/v330...v331 [v330]: https://github.com/heroku/heroku-buildpack-python/compare/v329...v330 [v329]: https://github.com/heroku/heroku-buildpack-python/compare/v328...v329 [v328]: https://github.com/heroku/heroku-buildpack-python/compare/v327...v328 From 4e00e124ae2a86bb8e4f75e497717c1a9340344b Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 13 Jan 2026 13:46:35 +0000 Subject: [PATCH 322/406] Set `WEB_CONCURRENCY_SET_BY` if appropriate (#2015) The Python buildpack sets the `WEB_CONCURRENCY` env var at app boot based on the current dyno size, so long as the user hasn't specified a custom value via the app's config vars. Now, in addition to setting the `WEB_CONCURRENCY` env var, the buildpack will also set `WEB_CONCURRENCY_SET_BY=heroku/python` to allow the app, other boot time scripts, or humans more easily differentiate between user and buildpack provided `WEB_CONCURRENCY` values (plus determine which buildpack actually set it). In addition to assisting with debugging, this allows for UX improvements in other buildpacks such as the PHP buildpack, which can now check for `WEB_CONCURRENCY_SET_BY` in its boot time Apache/Nginx `heroku-php-...` scripts - helping detect the case where users have ordered the buildpacks on their app in the wrong order. (The "primary" language is supposed to be listed last, otherwise the wrong concurrency value will be used.) See also: - https://github.com/heroku/heroku-buildpack-php/pull/883 - https://github.com/heroku/heroku-buildpack-nodejs/pull/932 - https://github.com/heroku/heroku-buildpack-ruby/pull/1700 GUS-W-20866598. --- CHANGELOG.md | 1 + spec/hatchet/ci_spec.rb | 4 ++++ spec/hatchet/pip_spec.rb | 1 + spec/hatchet/pipenv_spec.rb | 1 + spec/hatchet/poetry_spec.rb | 1 + spec/hatchet/profile_d_scripts_spec.rb | 9 ++++++++- spec/hatchet/uv_spec.rb | 1 + vendor/WEB_CONCURRENCY.sh | 3 +++ 8 files changed, 20 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 710273e9e..5ed572bac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- The web server concurrency calculation profile script now sets the env var `WEB_CONCURRENCY_SET_BY="heroku/python"` if `WEB_CONCURRENCY` was set automatically by the buildpack. ([#2015](https://github.com/heroku/heroku-buildpack-python/pull/2015)) ## [v331] - 2026-01-11 diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 7d74aa6e3..60319499b 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -72,6 +72,7 @@ PYTHONPATH=/app PYTHONUNBUFFERED=true WEB_CONCURRENCY=5 + WEB_CONCURRENCY_SET_BY=heroku/python pytest .+ -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully REGEX @@ -165,6 +166,7 @@ PYTHONUNBUFFERED=true VIRTUAL_ENV=/app/.heroku/python WEB_CONCURRENCY=5 + WEB_CONCURRENCY_SET_BY=heroku/python pytest .+ -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully REGEX @@ -258,6 +260,7 @@ PYTHONPATH=/app PYTHONUNBUFFERED=true WEB_CONCURRENCY=5 + WEB_CONCURRENCY_SET_BY=heroku/python pytest .+ -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully REGEX @@ -345,6 +348,7 @@ PYTHONPATH=/app PYTHONUNBUFFERED=true WEB_CONCURRENCY=5 + WEB_CONCURRENCY_SET_BY=heroku/python pytest .+ -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully REGEX diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 428c26a63..5252aa9ee 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -123,6 +123,7 @@ PYTHONPATH=/app PYTHONUNBUFFERED=true WEB_CONCURRENCY=2 + WEB_CONCURRENCY_SET_BY=heroku/python pip #{PIP_VERSION} from /app/.heroku/python/lib/python3.14/site-packages/pip (python 3.14) OUTPUT expect($CHILD_STATUS.exitstatus).to eq(0) diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index c1ad3b0cd..a5cda0cf8 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -126,6 +126,7 @@ PYTHONUNBUFFERED=true VIRTUAL_ENV=/app/.heroku/python WEB_CONCURRENCY=2 + WEB_CONCURRENCY_SET_BY=heroku/python pipenv, version #{PIPENV_VERSION} OUTPUT expect($CHILD_STATUS.exitstatus).to eq(0) diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 5bd26d664..b93fb0c57 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -124,6 +124,7 @@ PYTHONPATH=/app PYTHONUNBUFFERED=true WEB_CONCURRENCY=2 + WEB_CONCURRENCY_SET_BY=heroku/python OUTPUT expect($CHILD_STATUS.exitstatus).to eq(0) end diff --git a/spec/hatchet/profile_d_scripts_spec.rb b/spec/hatchet/profile_d_scripts_spec.rb index da3f325c0..bb805a69f 100644 --- a/spec/hatchet/profile_d_scripts_spec.rb +++ b/spec/hatchet/profile_d_scripts_spec.rb @@ -43,7 +43,7 @@ OUTPUT end - list_concurrency_envs_cmd = 'printenv | sort | grep -E "^(DYNO_RAM|WEB_CONCURRENCY)="' + list_concurrency_envs_cmd = 'printenv | sort | grep -E "^(DYNO_RAM|WEB_CONCURRENCY.*)="' # Check WEB_CONCURRENCY support when using a Standard-1X dyno. # We set the process type to `web` so that we can test the web-dyno-only log output. @@ -53,6 +53,7 @@ Python buildpack: Defaulting WEB_CONCURRENCY to 2 based on the available memory. DYNO_RAM=512 WEB_CONCURRENCY=2 + WEB_CONCURRENCY_SET_BY=heroku/python OUTPUT end @@ -63,6 +64,7 @@ Python buildpack: Defaulting WEB_CONCURRENCY to 4 based on the available memory. DYNO_RAM=1024 WEB_CONCURRENCY=4 + WEB_CONCURRENCY_SET_BY=heroku/python OUTPUT end @@ -73,6 +75,7 @@ Python buildpack: Defaulting WEB_CONCURRENCY to 5 based on the number of CPU cores. DYNO_RAM=2560 WEB_CONCURRENCY=5 + WEB_CONCURRENCY_SET_BY=heroku/python OUTPUT end @@ -83,6 +86,7 @@ Python buildpack: Defaulting WEB_CONCURRENCY to 17 based on the number of CPU cores. DYNO_RAM=14336 WEB_CONCURRENCY=17 + WEB_CONCURRENCY_SET_BY=heroku/python OUTPUT end @@ -93,6 +97,7 @@ Python buildpack: Defaulting WEB_CONCURRENCY to 9 based on the number of CPU cores. DYNO_RAM=30720 WEB_CONCURRENCY=9 + WEB_CONCURRENCY_SET_BY=heroku/python OUTPUT end @@ -103,6 +108,7 @@ Python buildpack: Defaulting WEB_CONCURRENCY to 17 based on the number of CPU cores. DYNO_RAM=63488 WEB_CONCURRENCY=17 + WEB_CONCURRENCY_SET_BY=heroku/python OUTPUT end @@ -113,6 +119,7 @@ Python buildpack: Defaulting WEB_CONCURRENCY to 33 based on the number of CPU cores. DYNO_RAM=129024 WEB_CONCURRENCY=33 + WEB_CONCURRENCY_SET_BY=heroku/python OUTPUT end diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index df8c40782..766df4dca 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -119,6 +119,7 @@ PYTHONPATH=/app PYTHONUNBUFFERED=true WEB_CONCURRENCY=2 + WEB_CONCURRENCY_SET_BY=heroku/python OUTPUT expect($CHILD_STATUS.exitstatus).to eq(0) end diff --git a/vendor/WEB_CONCURRENCY.sh b/vendor/WEB_CONCURRENCY.sh index c4ea72096..803b7217f 100755 --- a/vendor/WEB_CONCURRENCY.sh +++ b/vendor/WEB_CONCURRENCY.sh @@ -75,6 +75,9 @@ if [[ -v WEB_CONCURRENCY ]]; then return 0 fi +# Make it possible to differentiate between user and buildpack set WEB_CONCURRENCY values. +export WEB_CONCURRENCY_SET_BY="heroku/python" + minimum_memory_per_process_in_mb=256 # Prevents WEB_CONCURRENCY being set to zero if the environment is extremely memory constrained. From 3bddce4cab7fff4f5b5163e47ee1b1ffd0587b1c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 26 Jan 2026 09:51:44 +0000 Subject: [PATCH 323/406] Bump rubocop-rspec from 3.8.0 to 3.9.0 in the ruby-dependencies group (#2018) Bumps the ruby-dependencies group with 1 update: [rubocop-rspec](https://github.com/rubocop/rubocop-rspec). Updates `rubocop-rspec` from 3.8.0 to 3.9.0 - [Release notes](https://github.com/rubocop/rubocop-rspec/releases) - [Changelog](https://github.com/rubocop/rubocop-rspec/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop-rspec/compare/v3.8.0...v3.9.0) --- updated-dependencies: - dependency-name: rubocop-rspec dependency-version: 3.9.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 98c8704da..0bc43ef6c 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -30,14 +30,14 @@ GEM parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) - parser (3.3.10.0) + parser (3.3.10.1) ast (~> 2.4.1) racc platform-api (3.8.0) heroics (~> 0.1.1) moneta (~> 1.0.0) rate_throttle_client (~> 0.1.0) - prism (1.7.0) + prism (1.8.0) racc (1.8.1) rainbow (3.1.1) rate_throttle_client (0.1.2) @@ -65,7 +65,7 @@ GEM rubocop-ast (1.49.0) parser (>= 3.3.7.2) prism (~> 1.7) - rubocop-rspec (3.8.0) + rubocop-rspec (3.9.0) lint_roller (~> 1.1) rubocop (~> 1.81) ruby-progressbar (1.13.0) From e830581e02e751269c41a07cc66865bc39810b3e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 26 Jan 2026 09:57:00 +0000 Subject: [PATCH 324/406] Bump sigstore/cosign/cosign from v3.0.3 to v3.0.4 in /builds (#2020) Bumps sigstore/cosign/cosign from v3.0.3 to v3.0.4. --- updated-dependencies: - dependency-name: sigstore/cosign/cosign dependency-version: v3.0.4 dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- builds/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/builds/Dockerfile b/builds/Dockerfile index 0e7195939..be7742e9e 100644 --- a/builds/Dockerfile +++ b/builds/Dockerfile @@ -1,5 +1,5 @@ ARG STACK_VERSION="24" -FROM ghcr.io/sigstore/cosign/cosign:v3.0.3@sha256:774391ac9f0c137ee419ce56522df5fd3b1f52be90c5b77e97f7c053bdd67a67 AS cosign +FROM ghcr.io/sigstore/cosign/cosign:v3.0.4@sha256:0b015a3557a64a751712da8a6395534160018eaaa2d969882a85a336de9adb70 AS cosign FROM heroku/heroku:${STACK_VERSION}-build ARG STACK_VERSION From 4011224bb2282f16e7850a119c31beefd3cedab4 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 26 Jan 2026 11:00:25 +0000 Subject: [PATCH 325/406] Bump poetry from 2.2.1 to 2.3.1 (#2019) * Bump poetry from 2.2.1 to 2.3.1 Bumps [poetry](https://github.com/python-poetry/poetry) from 2.2.1 to 2.3.1. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](https://github.com/python-poetry/poetry/compare/2.2.1...2.3.1) --- updated-dependencies: - dependency-name: poetry dependency-version: 2.3.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/poetry.txt | 2 +- spec/hatchet/poetry_spec.rb | 4 ++-- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ed572bac..7e0488020 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated Poetry from 2.2.1 to 2.3.1. ([#2019](https://github.com/heroku/heroku-buildpack-python/pull/2019)) - The web server concurrency calculation profile script now sets the env var `WEB_CONCURRENCY_SET_BY="heroku/python"` if `WEB_CONCURRENCY` was set automatically by the buildpack. ([#2015](https://github.com/heroku/heroku-buildpack-python/pull/2015)) ## [v331] - 2026-01-11 diff --git a/requirements/poetry.txt b/requirements/poetry.txt index 6bd2fa4e5..d29004da8 100644 --- a/requirements/poetry.txt +++ b/requirements/poetry.txt @@ -1 +1 @@ -poetry==2.2.1 +poetry==2.3.1 diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index b93fb0c57..74832d7f1 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -131,8 +131,7 @@ end end - # TODO: Rename this test description back this when the Poetry version next changes. - context 'when the Python version has changed since the last build' do + context 'when the Poetry and Python versions have changed since the last build' do let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v313'] } let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_basic', buildpacks:) } @@ -147,6 +146,7 @@ remote: -----> Using Python 3.14 specified in .python-version remote: -----> Discarding cache since: remote: - The Python version has changed from 3.14.0 to #{LATEST_PYTHON_3_14} + remote: - The Poetry version has changed from 2.2.1 to #{POETRY_VERSION} remote: -----> Installing Python #{LATEST_PYTHON_3_14} remote: -----> Installing Poetry #{POETRY_VERSION} remote: -----> Installing dependencies using 'poetry sync --only main' From 4f9718c61c260f2184252b5a1abea111f4482e06 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 26 Jan 2026 11:05:45 +0000 Subject: [PATCH 326/406] Bump uv from 0.9.24 to 0.9.26 (#2016) * Bump uv from 0.9.24 to 0.9.26 Bumps [uv](https://github.com/astral-sh/uv) from 0.9.24 to 0.9.26. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.9.24...0.9.26) --- updated-dependencies: - dependency-name: uv dependency-version: 0.9.26 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7e0488020..dd98b0a45 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Updated Poetry from 2.2.1 to 2.3.1. ([#2019](https://github.com/heroku/heroku-buildpack-python/pull/2019)) +- Updated uv from 0.9.24 to 0.9.26. ([#2016](https://github.com/heroku/heroku-buildpack-python/pull/2016)) - The web server concurrency calculation profile script now sets the env var `WEB_CONCURRENCY_SET_BY="heroku/python"` if `WEB_CONCURRENCY` was set automatically by the buildpack. ([#2015](https://github.com/heroku/heroku-buildpack-python/pull/2015)) ## [v331] - 2026-01-11 diff --git a/requirements/uv.txt b/requirements/uv.txt index 55bf7c7ea..8664a395f 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.9.24 +uv==0.9.26 From f9ce6a681e5f6ff71e1b5dc30e2c090377f1269a Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Mon, 26 Jan 2026 11:23:21 +0000 Subject: [PATCH 327/406] Prepare release v332 (#2021) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dd98b0a45..c0a609c4a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v332] - 2026-01-26 + - Updated Poetry from 2.2.1 to 2.3.1. ([#2019](https://github.com/heroku/heroku-buildpack-python/pull/2019)) - Updated uv from 0.9.24 to 0.9.26. ([#2016](https://github.com/heroku/heroku-buildpack-python/pull/2016)) - The web server concurrency calculation profile script now sets the env var `WEB_CONCURRENCY_SET_BY="heroku/python"` if `WEB_CONCURRENCY` was set automatically by the buildpack. ([#2015](https://github.com/heroku/heroku-buildpack-python/pull/2015)) @@ -1462,7 +1465,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v331...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v332...main +[v332]: https://github.com/heroku/heroku-buildpack-python/compare/v331...v332 [v331]: https://github.com/heroku/heroku-buildpack-python/compare/v330...v331 [v330]: https://github.com/heroku/heroku-buildpack-python/compare/v329...v330 [v329]: https://github.com/heroku/heroku-buildpack-python/compare/v328...v329 From ab4d77dee5ed2b5d7611932c64b2d0df07621387 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 2 Feb 2026 10:57:56 +0000 Subject: [PATCH 328/406] Bump rubocop from 1.82.1 to 1.84.0 in the ruby-dependencies group (#2022) Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.82.1 to 1.84.0 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.82.1...v1.84.0) --- updated-dependencies: - dependency-name: rubocop dependency-version: 1.84.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 0bc43ef6c..1d69d2744 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -37,7 +37,7 @@ GEM heroics (~> 0.1.1) moneta (~> 1.0.0) rate_throttle_client (~> 0.1.0) - prism (1.8.0) + prism (1.9.0) racc (1.8.1) rainbow (3.1.1) rate_throttle_client (0.1.2) @@ -51,7 +51,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.6) - rubocop (1.82.1) + rubocop (1.84.0) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) @@ -59,7 +59,7 @@ GEM parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) regexp_parser (>= 2.9.3, < 3.0) - rubocop-ast (>= 1.48.0, < 2.0) + rubocop-ast (>= 1.49.0, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 4.0) rubocop-ast (1.49.0) From 827b129bf38e587a6b23f0012045585fca055f8e Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 4 Feb 2026 10:21:04 +0000 Subject: [PATCH 329/406] Add support for Python 3.14.3 and 3.13.12 (#2027) * Add support for Python 3.14.3 and 3.13.12 Release announcement: https://blog.python.org/2026/02/python-3143-and-31312-are-now-available.html Changelogs: https://docs.python.org/release/3.14.3/whatsnew/changelog.html#python-3-14-3-final https://docs.python.org/release/3.13.12/whatsnew/changelog.html#python-3-13-12-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/21649829905 https://github.com/heroku/heroku-buildpack-python/actions/runs/21649839617 GUS-W-20590488. GUS-W-20590539. * Disable Pipenv test until upstream Pipenv bug fixed Since the new Python versions happened to expose a remaining Pipenv bug that makes this test fail: https://github.com/pypa/pipenv/issues/6514 --- CHANGELOG.md | 2 ++ lib/python_version.sh | 4 ++-- .../Pipfile | 2 +- .../Pipfile.lock | 4 ++-- spec/hatchet/pipenv_spec.rb | 13 ++++++++++--- spec/spec_helper.rb | 4 ++-- 6 files changed, 19 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c0a609c4a..ff985b21a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## [Unreleased] +- The Python 3.14 version alias now resolves to Python 3.14.3. ([#2027](https://github.com/heroku/heroku-buildpack-python/pull/2027)) +- The Python 3.13 version alias now resolves to Python 3.13.12. ([#2027](https://github.com/heroku/heroku-buildpack-python/pull/2027)) ## [v332] - 2026-01-26 diff --git a/lib/python_version.sh b/lib/python_version.sh index 283a04c1f..d3e97631b 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -7,8 +7,8 @@ set -euo pipefail LATEST_PYTHON_3_10="3.10.19" LATEST_PYTHON_3_11="3.11.14" LATEST_PYTHON_3_12="3.12.12" -LATEST_PYTHON_3_13="3.13.11" -LATEST_PYTHON_3_14="3.14.2" +LATEST_PYTHON_3_13="3.13.12" +LATEST_PYTHON_3_14="3.14.3" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=10 NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=14 diff --git a/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile b/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile index 5afe25b60..d9d1ea61b 100644 --- a/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile +++ b/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile @@ -10,4 +10,4 @@ typing-extensions = "*" [requires] # The version in .python-version should take precedence over this. -python_version = "3.11" +python_version = "3.13" diff --git a/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile.lock b/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile.lock index 257b94ada..2a3e9cd35 100644 --- a/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile.lock +++ b/spec/fixtures/pipenv_python_version_and_python_version_file/Pipfile.lock @@ -1,11 +1,11 @@ { "_meta": { "hash": { - "sha256": "5a5846099fd5ceb0291e704a94f49cf0b8a226109fdae915a61253b56eaf3ed6" + "sha256": "83d7242edaa31ec24731c102c5debc217f3e5f5fa5b4e992de07d4d25805715e" }, "pipfile-spec": 6, "requires": { - "python_version": "3.11" + "python_version": "3.13" }, "sources": [ { diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index a5cda0cf8..e0b46a839 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -211,6 +211,8 @@ end end + # TODO: Delete this test once pipenv_mismatched_python_version is re-enabled, + # since they mostly duplicate each other. context 'when there is a both a Pipfile.lock python_version and a .python-version file' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_version_and_python_version_file') } @@ -556,13 +558,18 @@ end # This tests that Pipenv doesn't fall back to system Python if the Python version in - # pyproject.toml doesn't match that in Pipfile / Pipfile.lock. - context 'when requires-python in pyproject.toml is incompatible with .python-version' do + # pyproject.toml doesn't match that in Pipfile / Pipfile.lock. It also tests which of + # .python-version and Pipfile.lock take precedence for the installed Python version. + context 'when python_version in Pipfile.lock is incompatible with .python-version' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_mismatched_python_version', allow_failure: true) } - it 'fails the build' do + it 'fails the build', skip: 'https://github.com/pypa/pipenv/issues/6514' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python #{LATEST_PYTHON_3_13} specified in .python-version + remote: -----> Installing Python #{LATEST_PYTHON_3_13} + remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing dependencies using 'pipenv install --deploy' remote: Warning: Your Pipfile requires "python_version" 3.12, but you are using #{LATEST_PYTHON_3_13} remote: from //app/./python/bin/python3. diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index fba5874f7..5652c9ff3 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -13,8 +13,8 @@ LATEST_PYTHON_3_10 = '3.10.19' LATEST_PYTHON_3_11 = '3.11.14' LATEST_PYTHON_3_12 = '3.12.12' -LATEST_PYTHON_3_13 = '3.13.11' -LATEST_PYTHON_3_14 = '3.14.2' +LATEST_PYTHON_3_13 = '3.13.12' +LATEST_PYTHON_3_14 = '3.14.3' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_14 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From c43df603416e0ff33bf267427d1d1335046d3760 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 4 Feb 2026 10:53:55 +0000 Subject: [PATCH 330/406] Prepare release v333 (#2028) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ff985b21a..b14769721 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v333] - 2026-02-04 + - The Python 3.14 version alias now resolves to Python 3.14.3. ([#2027](https://github.com/heroku/heroku-buildpack-python/pull/2027)) - The Python 3.13 version alias now resolves to Python 3.13.12. ([#2027](https://github.com/heroku/heroku-buildpack-python/pull/2027)) @@ -1467,7 +1470,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v332...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v333...main +[v333]: https://github.com/heroku/heroku-buildpack-python/compare/v332...v333 [v332]: https://github.com/heroku/heroku-buildpack-python/compare/v331...v332 [v331]: https://github.com/heroku/heroku-buildpack-python/compare/v330...v331 [v330]: https://github.com/heroku/heroku-buildpack-python/compare/v329...v330 From 0c7b05cf770fe292f30e4d0a76f3562ee19fef51 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 4 Feb 2026 13:02:27 +0000 Subject: [PATCH 331/406] Bump poetry from 2.3.1 to 2.3.2 (#2024) * Bump poetry from 2.3.1 to 2.3.2 Bumps [poetry](https://github.com/python-poetry/poetry) from 2.3.1 to 2.3.2. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](https://github.com/python-poetry/poetry/compare/2.3.1...2.3.2) --- updated-dependencies: - dependency-name: poetry dependency-version: 2.3.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/poetry.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b14769721..001751840 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated Poetry from 2.3.1 to 2.3.2. ([#2024](https://github.com/heroku/heroku-buildpack-python/pull/2024)) ## [v333] - 2026-02-04 diff --git a/requirements/poetry.txt b/requirements/poetry.txt index d29004da8..f69508193 100644 --- a/requirements/poetry.txt +++ b/requirements/poetry.txt @@ -1 +1 @@ -poetry==2.3.1 +poetry==2.3.2 From 9d0be84cd391d2a0b89ab0e8cc93aee9bc419c71 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 4 Feb 2026 13:16:24 +0000 Subject: [PATCH 332/406] Bump uv from 0.9.26 to 0.9.29 (#2029) Bumps [uv](https://github.com/astral-sh/uv) from 0.9.26 to 0.9.29. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.9.26...0.9.29) --- updated-dependencies: - dependency-name: uv dependency-version: 0.9.29 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 001751840..5f3ec1712 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Updated Poetry from 2.3.1 to 2.3.2. ([#2024](https://github.com/heroku/heroku-buildpack-python/pull/2024)) +- Updated uv from 0.9.26 to 0.9.29. ([#2029](https://github.com/heroku/heroku-buildpack-python/pull/2029)) ## [v333] - 2026-02-04 diff --git a/requirements/uv.txt b/requirements/uv.txt index 8664a395f..ae44d13f1 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.9.26 +uv==0.9.29 From 53b747d4274227f7422d7c36344f722fc9c23f68 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 4 Feb 2026 13:21:01 +0000 Subject: [PATCH 333/406] Prepare release v334 (#2030) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5f3ec1712..c1848321b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v334] - 2026-02-04 + - Updated Poetry from 2.3.1 to 2.3.2. ([#2024](https://github.com/heroku/heroku-buildpack-python/pull/2024)) - Updated uv from 0.9.26 to 0.9.29. ([#2029](https://github.com/heroku/heroku-buildpack-python/pull/2029)) @@ -1472,7 +1475,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v333...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v334...main +[v334]: https://github.com/heroku/heroku-buildpack-python/compare/v333...v334 [v333]: https://github.com/heroku/heroku-buildpack-python/compare/v332...v333 [v332]: https://github.com/heroku/heroku-buildpack-python/compare/v331...v332 [v331]: https://github.com/heroku/heroku-buildpack-python/compare/v330...v331 From 2d975f8a96a3666005c0d3ea8a852e8200c8a2c6 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 10 Feb 2026 12:19:38 +0000 Subject: [PATCH 334/406] Bump uv from 0.9.29 to 0.10.1 (#2031) * Bump uv from 0.9.29 to 0.10.1 Bumps [uv](https://github.com/astral-sh/uv) from 0.9.29 to 0.10.1. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.9.29...0.10.1) --- updated-dependencies: - dependency-name: uv dependency-version: 0.10.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c1848321b..35a3b6aca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.9.29 to 0.10.1. ([#2031](https://github.com/heroku/heroku-buildpack-python/pull/2031)) ## [v334] - 2026-02-04 diff --git a/requirements/uv.txt b/requirements/uv.txt index ae44d13f1..cc5692614 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.9.29 +uv==0.10.1 From 8eaa83cf0f9e196abd13e895bc667274183909a6 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 10 Feb 2026 12:26:51 +0000 Subject: [PATCH 335/406] Prepare release v335 (#2033) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 35a3b6aca..c013734d3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v335] - 2026-02-10 + - Updated uv from 0.9.29 to 0.10.1. ([#2031](https://github.com/heroku/heroku-buildpack-python/pull/2031)) ## [v334] - 2026-02-04 @@ -1476,7 +1479,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v334...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v335...main +[v335]: https://github.com/heroku/heroku-buildpack-python/compare/v334...v335 [v334]: https://github.com/heroku/heroku-buildpack-python/compare/v333...v334 [v333]: https://github.com/heroku/heroku-buildpack-python/compare/v332...v333 [v332]: https://github.com/heroku/heroku-buildpack-python/compare/v331...v332 From 96e5bea5603b78b40d6c2b59e738cc2b934d8d3c Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 23 Feb 2026 16:06:46 +0000 Subject: [PATCH 336/406] Switch to AWS' dual-stack S3 URLs (#2035) The default AWS S3 URLs only support IPv4, whereas the dual-stack URLs support both IPv4 and IPv6: https://docs.aws.amazon.com/AmazonS3/latest/API/ipv6-access.html https://docs.aws.amazon.com/AmazonS3/latest/API/dual-stack-endpoints.html By switching to the dual-stack URLs, it means systems that support IPv6 will now use it instead of IPv4, which is helpful in IPv6 environments where IPv4 traffic has to fall back to being routed via NAT gateways (which has performance and ingress cost implications). See also: https://salesforce-internal.slack.com/archives/C01R6FJ738U/p1770827176094169 GUS-W-21335961. --- CHANGELOG.md | 1 + lib/python.sh | 6 +++--- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c013734d3..c08d57b1c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Changed the S3 URL used to download Python to use AWS' dual-stack (IPv6 compatible) endpoint. ([#2035](https://github.com/heroku/heroku-buildpack-python/pull/2035)) ## [v335] - 2026-02-10 diff --git a/lib/python.sh b/lib/python.sh index ea87c2c63..0fa91862d 100644 --- a/lib/python.sh +++ b/lib/python.sh @@ -4,7 +4,7 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail -S3_BASE_URL="https://heroku-buildpack-python.s3.us-east-1.amazonaws.com" +S3_BASE_URL="https://heroku-buildpack-python.s3.dualstack.us-east-1.amazonaws.com" function python::install() { local build_dir="${1}" @@ -28,7 +28,7 @@ function python::install() { local ubuntu_version="${stack/heroku-/}.04" local arch arch=$(dpkg --print-architecture) - # e.g.: https://heroku-buildpack-python.s3.us-east-1.amazonaws.com/python-3.13.0-ubuntu-24.04-amd64.tar.zst + # e.g.: https://heroku-buildpack-python.s3.dualstack.us-east-1.amazonaws.com/python-3.14.0-ubuntu-24.04-amd64.tar.zst local python_url="${S3_BASE_URL}/python-${python_full_version}-ubuntu-${ubuntu_version}-${arch}.tar.zst" local error_log @@ -117,7 +117,7 @@ function python::install() { Then try building again to see if the error resolves itself. EOF build_data::set_string "failure_reason" "install-python" - # e.g.: 'curl: (6) Could not resolve host: heroku-buildpack-python.s3.us-east-1.amazonaws.com' + # e.g.: 'curl: (6) Could not resolve host: heroku-buildpack-python.s3.dualstack.us-east-1.amazonaws.com' build_data::set_string "failure_detail" "$(head --lines=1 "${error_log}" || true)" fi From 361215e907719287c821d69639203fe615d7560d Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 2 Mar 2026 17:14:39 +0000 Subject: [PATCH 337/406] Work around NLTK v3.9.3 regression (#2042) In NLTK v3.9.3, a new security check was added for the Zip Slip issue: https://github.com/nltk/nltk/pull/3468 This change unfortunately contained a bug that causes false positive security errors when packages are downloaded to a symlinked path, since the new check doesn't use `abspath` vs `realpath` consistently: https://github.com/nltk/nltk/issues/3509 This causes errors like: ``` -----> Downloading NLTK packages: punkt punkt_tab :128: RuntimeWarning: 'nltk.downloader' found in sys.modules after import of package 'nltk', but prior to execution of 'nltk.downloader'; this may result in unpredictable behaviour [nltk_data] Downloading package punkt to [nltk_data] /app/.heroku/python/nltk_data... [nltk_data] Unzipping tokenizers/punkt.zip. [nltk_data] Zip Slip blocked: punkt/ Error installing package. Retry? [n/y/e] Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/app/.heroku/python/lib/python3.12/site-packages/nltk/downloader.py", line 2631, in rv = downloader.download( ^^^^^^^^^^^^^^^^^^^^ File "/app/.heroku/python/lib/python3.12/site-packages/nltk/downloader.py", line 773, in download choice = input().strip() ^^^^^^^ EOFError: EOF when reading a line ``` See also: https://github.com/heroku/heroku-buildpack-python/actions/runs/22545824206/job/65308000022#step:5:539 Until upstream fix this regression, we can work around it by always passing the raw path for Python home instead of the symlinked path. (We generally prefer using the symlinked path where possible, to ensure the paths used/displayed match between build time and run time, given the build directory is different from the run directory.) Longer term I will be deprecating and that removing support for `nltk.txt` since it's a Heroku-proprietary invention that's unnecessary given most users instead either commit the NLTK corpora or use the post_compile hook feature instead. Fixes #2037. GUS-W-21410908. --- CHANGELOG.md | 1 + bin/steps/nltk | 11 +++++++---- spec/hatchet/nltk_spec.rb | 4 ++-- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c08d57b1c..fc7b10b64 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Added a workaround for `nltk.txt` package downloader errors caused by an upstream regression in NLTK v3.9.3. ([#2041](https://github.com/heroku/heroku-buildpack-python/pull/2041)) - Changed the S3 URL used to download Python to use AWS' dual-stack (IPv6 compatible) endpoint. ([#2035](https://github.com/heroku/heroku-buildpack-python/pull/2035)) ## [v335] - 2026-02-10 diff --git a/bin/steps/nltk b/bin/steps/nltk index a1317ebf7..0e72cdd23 100755 --- a/bin/steps/nltk +++ b/bin/steps/nltk @@ -25,10 +25,11 @@ if is_module_available 'nltk'; then readarray -t nltk_packages <"${nltk_packages_definition}" output::step "Downloading NLTK packages: ${nltk_packages[*]}" - nltk_data_dir="/app/.heroku/python/nltk_data" - + # Note: We have to use the raw build directory path here and not the symlinked `/app` path, + # otherwise it will cause a false positive in NLTK v3.9.3's new Zip Slip security check, + # which doesn't handle symlinked paths correctly: https://github.com/nltk/nltk/issues/3509 # TODO: Does this even need user-provided env vars, or can we remove the sub_env usage here? - if ! sub_env python -m nltk.downloader -d "${nltk_data_dir}" "${nltk_packages[@]}" |& output::indent; then + if ! sub_env python -m nltk.downloader -d "${BUILD_DIR}/.heroku/python/nltk_data" "${nltk_packages[@]}" |& output::indent; then output::error <<-EOF Error: Unable to download NLTK data. @@ -41,7 +42,9 @@ if is_module_available 'nltk'; then exit 1 fi - set_env NLTK_DATA "${nltk_data_dir}" + # Since this will be used at runtime, we must use the symlinked `/app` path and not + # the raw build directory path. + set_env NLTK_DATA "/app/.heroku/python/nltk_data" else build_data::set_string "nltk_downloader" "skipped-no-nltk-file" echo " 'nltk.txt' not found, not downloading any corpora" diff --git a/spec/hatchet/nltk_spec.rb b/spec/hatchet/nltk_spec.rb index 56ed95ce4..1c5a5a59d 100644 --- a/spec/hatchet/nltk_spec.rb +++ b/spec/hatchet/nltk_spec.rb @@ -13,10 +13,10 @@ remote: -----> Downloading NLTK packages: city_database stopwords remote: .*: RuntimeWarning: 'nltk.downloader' found in sys.modules after import of package 'nltk', but prior to execution of 'nltk.downloader'; this may result in unpredictable behaviour remote: \\[nltk_data\\] Downloading package city_database to - remote: \\[nltk_data\\] /app/.heroku/python/nltk_data... + remote: \\[nltk_data\\] /tmp/build_.+/.heroku/python/nltk_data... remote: \\[nltk_data\\] Unzipping corpora/city_database.zip. remote: \\[nltk_data\\] Downloading package stopwords to - remote: \\[nltk_data\\] /app/.heroku/python/nltk_data... + remote: \\[nltk_data\\] /tmp/build_.+/.heroku/python/nltk_data... remote: \\[nltk_data\\] Unzipping corpora/stopwords.zip. REGEX From 3af8b100bad4a9a5a0a6a4213ab378766e5590c5 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 2 Mar 2026 17:19:37 +0000 Subject: [PATCH 338/406] Bump rubocop from 1.84.0 to 1.85.0 in the ruby-dependencies group (#2039) Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.84.0 to 1.85.0 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.84.0...v1.85.0) --- updated-dependencies: - dependency-name: rubocop dependency-version: 1.85.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 1d69d2744..9d76f3600 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,8 +1,11 @@ GEM remote: https://rubygems.org/ specs: + addressable (2.8.9) + public_suffix (>= 2.0.2, < 8.0) ast (2.4.3) base64 (0.3.0) + bigdecimal (4.0.1) diff-lcs (1.6.2) erubis (2.7.0) excon (1.3.1) @@ -20,17 +23,22 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.18.0) + json (2.18.1) + json-schema (6.1.0) + addressable (~> 2.8) + bigdecimal (>= 3.1, < 5) language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) + mcp (0.7.1) + json-schema (>= 4.1) moneta (1.0.0) multi_json (1.18.0) parallel (1.27.0) parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) - parser (3.3.10.1) + parser (3.3.10.2) ast (~> 2.4.1) racc platform-api (3.8.0) @@ -38,6 +46,7 @@ GEM moneta (~> 1.0.0) rate_throttle_client (~> 0.1.0) prism (1.9.0) + public_suffix (7.0.2) racc (1.8.1) rainbow (3.1.1) rate_throttle_client (0.1.2) @@ -51,10 +60,11 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.6) - rubocop (1.84.0) + rubocop (1.85.0) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) + mcp (~> 0.6) parallel (~> 1.10) parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) From d06b98c6b84c868169993656d34a1c238cc2eb1c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 2 Mar 2026 17:20:36 +0000 Subject: [PATCH 339/406] Bump sigstore/cosign/cosign from v3.0.4 to v3.0.5 in /builds (#2040) Bumps sigstore/cosign/cosign from v3.0.4 to v3.0.5. --- updated-dependencies: - dependency-name: sigstore/cosign/cosign dependency-version: v3.0.5 dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- builds/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/builds/Dockerfile b/builds/Dockerfile index be7742e9e..53610332e 100644 --- a/builds/Dockerfile +++ b/builds/Dockerfile @@ -1,5 +1,5 @@ ARG STACK_VERSION="24" -FROM ghcr.io/sigstore/cosign/cosign:v3.0.4@sha256:0b015a3557a64a751712da8a6395534160018eaaa2d969882a85a336de9adb70 AS cosign +FROM ghcr.io/sigstore/cosign/cosign:v3.0.5@sha256:be924970ba7438c22e18067dec5637946d6566eac711f5bedd1584e7137008fb AS cosign FROM heroku/heroku:${STACK_VERSION}-build ARG STACK_VERSION From a6e6ac0d0a8241bbfb953f620b060eddb05c891d Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Mon, 2 Mar 2026 17:24:05 +0000 Subject: [PATCH 340/406] Prepare release v336 (#2038) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fc7b10b64..8d9953849 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v336] - 2026-03-02 + - Added a workaround for `nltk.txt` package downloader errors caused by an upstream regression in NLTK v3.9.3. ([#2041](https://github.com/heroku/heroku-buildpack-python/pull/2041)) - Changed the S3 URL used to download Python to use AWS' dual-stack (IPv6 compatible) endpoint. ([#2035](https://github.com/heroku/heroku-buildpack-python/pull/2035)) @@ -1481,7 +1484,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v335...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v336...main +[v336]: https://github.com/heroku/heroku-buildpack-python/compare/v335...v336 [v335]: https://github.com/heroku/heroku-buildpack-python/compare/v334...v335 [v334]: https://github.com/heroku/heroku-buildpack-python/compare/v333...v334 [v333]: https://github.com/heroku/heroku-buildpack-python/compare/v332...v333 From 522aeb09f57783855d881d193957c31fff79e6fe Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 2 Mar 2026 23:14:59 +0000 Subject: [PATCH 341/406] Bump pip from 25.3 to 26.0.1 (#2032) * Bump pip from 25.3 to 26.0.1 Bumps [pip](https://github.com/pypa/pip) from 25.3 to 26.0.1. - [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst) - [Commits](https://github.com/pypa/pip/compare/25.3...26.0.1) --- updated-dependencies: - dependency-name: pip dependency-version: 26.0.1 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] * Add changelog entry * Update test assertion --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/pip.txt | 2 +- spec/hatchet/pip_spec.rb | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d9953849..85fe04187 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated pip from 25.3 to 26.0.1. ([#2032](https://github.com/heroku/heroku-buildpack-python/pull/2032)) ## [v336] - 2026-03-02 diff --git a/requirements/pip.txt b/requirements/pip.txt index 0b3d671e6..c1dd99e28 100644 --- a/requirements/pip.txt +++ b/requirements/pip.txt @@ -1 +1 @@ -pip==25.3 +pip==26.0.1 diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 5252aa9ee..0f37dba4c 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -349,7 +349,7 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Installing dependencies using 'pip install -r requirements.txt' - remote: ERROR: Invalid requirement: 'an-invalid-requirement!': Expected end or semicolon (after name and no valid version specifier) + remote: ERROR: Invalid requirement: 'an-invalid-requirement!': Expected semicolon (after name with no version specifier) or end remote: an-invalid-requirement! remote: ^ (from line 1 of requirements.txt) remote: From 3b5d531100faea66fdd4d7169ad7304d7313bf58 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 3 Mar 2026 15:08:57 +0000 Subject: [PATCH 342/406] Update to Ruby 4.0 (#2043) For the repo's development/testing tooling. Plus refresh the lockfile. GUS-W-21429033. --- .github/workflows/ci.yml | 4 +- .github/workflows/hatchet_app_cleaner.yml | 2 +- Gemfile | 2 +- Gemfile.lock | 63 +++++++++++++++++++---- 4 files changed, 58 insertions(+), 13 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f32322a11..f92ac8a18 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -24,7 +24,7 @@ jobs: uses: ruby/setup-ruby@v1 with: bundler-cache: true - ruby-version: "3.4" + ruby-version: "4.0" - name: Run ShellCheck run: make lint-scripts - name: Run shfmt @@ -56,7 +56,7 @@ jobs: uses: ruby/setup-ruby@v1 with: bundler-cache: true - ruby-version: "3.4" + ruby-version: "4.0" - name: Hatchet setup run: bundle exec hatchet ci:setup - name: Run Hatchet integration tests diff --git a/.github/workflows/hatchet_app_cleaner.yml b/.github/workflows/hatchet_app_cleaner.yml index caf9388d9..5844b15ca 100644 --- a/.github/workflows/hatchet_app_cleaner.yml +++ b/.github/workflows/hatchet_app_cleaner.yml @@ -24,7 +24,7 @@ jobs: uses: ruby/setup-ruby@v1 with: bundler-cache: true - ruby-version: "3.4" + ruby-version: "4.0" - name: Run Hatchet destroy # Only apps older than 10 minutes are destroyed, to ensure that any # in progress CI runs are not interrupted. diff --git a/Gemfile b/Gemfile index 2abf2b7cb..7ed84f7d1 100644 --- a/Gemfile +++ b/Gemfile @@ -2,7 +2,7 @@ source 'https://rubygems.org' -ruby '>= 3.2', '< 3.5' +ruby '>= 3.2', '< 4.1' group :test, :development do gem 'heroku_hatchet' diff --git a/Gemfile.lock b/Gemfile.lock index 9d76f3600..4c7138d8f 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -8,7 +8,7 @@ GEM bigdecimal (4.0.1) diff-lcs (1.6.2) erubis (2.7.0) - excon (1.3.1) + excon (1.4.0) logger heroics (0.1.3) base64 @@ -30,10 +30,10 @@ GEM language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) - mcp (0.7.1) + mcp (0.8.0) json-schema (>= 4.1) moneta (1.0.0) - multi_json (1.18.0) + multi_json (1.19.1) parallel (1.27.0) parallel_split_test (0.10.0) parallel (>= 0.5.13) @@ -41,12 +41,12 @@ GEM parser (3.3.10.2) ast (~> 2.4.1) racc - platform-api (3.8.0) + platform-api (3.9.0) heroics (~> 0.1.1) moneta (~> 1.0.0) rate_throttle_client (~> 0.1.0) prism (1.9.0) - public_suffix (7.0.2) + public_suffix (7.0.5) racc (1.8.1) rainbow (3.1.1) rate_throttle_client (0.1.2) @@ -59,7 +59,7 @@ GEM rspec-support (~> 3.13.0) rspec-retry (0.6.2) rspec-core (> 3.3) - rspec-support (3.13.6) + rspec-support (3.13.7) rubocop (1.85.0) json (~> 2.3) language_server-protocol (~> 3.17.0.2) @@ -79,7 +79,7 @@ GEM lint_roller (~> 1.1) rubocop (~> 1.81) ruby-progressbar (1.13.0) - thor (1.4.0) + thor (1.5.0) threaded (0.0.4) unicode-display_width (3.2.0) unicode-emoji (~> 4.1) @@ -87,7 +87,9 @@ GEM webrick (1.9.2) PLATFORMS + arm64-darwin-24 ruby + x86_64-linux DEPENDENCIES heroku_hatchet @@ -98,8 +100,51 @@ DEPENDENCIES rubocop rubocop-rspec +CHECKSUMS + addressable (2.8.9) sha256=cc154fcbe689711808a43601dee7b980238ce54368d23e127421753e46895485 + ast (2.4.3) sha256=954615157c1d6a382bc27d690d973195e79db7f55e9765ac7c481c60bdb4d383 + base64 (0.3.0) sha256=27337aeabad6ffae05c265c450490628ef3ebd4b67be58257393227588f5a97b + bigdecimal (4.0.1) sha256=8b07d3d065a9f921c80ceaea7c9d4ae596697295b584c296fe599dd0ad01c4a7 + diff-lcs (1.6.2) sha256=9ae0d2cba7d4df3075fe8cd8602a8604993efc0dfa934cff568969efb1909962 + erubis (2.7.0) sha256=63653f5174a7997f6f1d6f465fbe1494dcc4bdab1fb8e635f6216989fb1148ba + excon (1.4.0) sha256=5d2bc9d2c79511a562e7fcac77cc7a40acd9cebcc55b80e537975ad8187f2924 + heroics (0.1.3) sha256=31d792e8706ecc6f78299f52d0b0f8ab55489a13b5f4f76c3050b07912563562 + heroku_hatchet (8.0.6) sha256=886a7d7d686859db9fac19df1194a35c5349285956ca4d134054c5c376accea3 + json (2.18.1) sha256=fe112755501b8d0466b5ada6cf50c8c3f41e897fa128ac5d263ec09eedc9f986 + json-schema (6.1.0) sha256=6bf70a2cfb6dfd5a06da28093fa8190f324c88eabd36a7f47097f227321dc702 + language_server-protocol (3.17.0.5) sha256=fd1e39a51a28bf3eec959379985a72e296e9f9acfce46f6a79d31ca8760803cc + lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87 + logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203 + mcp (0.8.0) sha256=ae8bd146bb8e168852866fd26f805f52744f6326afb3211e073f78a95e0c34fb + moneta (1.0.0) sha256=2224e5a68156e8eceb525fb0582c8c4e0f29f67cae86507cdcfb406abbb1fc5d + multi_json (1.19.1) sha256=7aefeff8f2c854bf739931a238e4aea64592845e0c0395c8a7d2eea7fdd631b7 + parallel (1.27.0) sha256=4ac151e1806b755fb4e2dc2332cbf0e54f2e24ba821ff2d3dcf86bf6dc4ae130 + parallel_split_test (0.10.0) sha256=4abd6bedc6a1b169ecb93a1bf982fa7fbd437b66c360594d56d107c25f53b05f + parser (3.3.10.2) sha256=6f60c84aa4bdcedb6d1a2434b738fe8a8136807b6adc8f7f53b97da9bc4e9357 + platform-api (3.9.0) sha256=be3b919955c52649fd931a9f62d571bd6f06613de630970dc7255fc95eb7d962 + prism (1.9.0) sha256=7b530c6a9f92c24300014919c9dcbc055bf4cdf51ec30aed099b06cd6674ef85 + public_suffix (7.0.5) sha256=1a8bb08f1bbea19228d3bed6e5ed908d1cb4f7c2726d18bd9cadf60bc676f623 + racc (1.8.1) sha256=4a7f6929691dbec8b5209a0b373bc2614882b55fc5d2e447a21aaa691303d62f + rainbow (3.1.1) sha256=039491aa3a89f42efa1d6dec2fc4e62ede96eb6acd95e52f1ad581182b79bc6a + rate_throttle_client (0.1.2) sha256=f9de968b892fea9272154f6182b4f5cfb74292585e66763fb8a8510181ec83ee + regexp_parser (2.11.3) sha256=ca13f381a173b7a93450e53459075c9b76a10433caadcb2f1180f2c741fc55a4 + rrrretry (1.0.0) sha256=3c60784501701a49d8ad499af7e76dbddf9a8be916beffe885bd0f443ad1c749 + rspec-core (3.13.6) sha256=a8823c6411667b60a8bca135364351dda34cd55e44ff94c4be4633b37d828b2d + rspec-expectations (3.13.5) sha256=33a4d3a1d95060aea4c94e9f237030a8f9eae5615e9bd85718fe3a09e4b58836 + rspec-retry (0.6.2) sha256=6101ba23a38809811ae3484acde4ab481c54d846ac66d5037ccb40131a60d858 + rspec-support (3.13.7) sha256=0640e5570872aafefd79867901deeeeb40b0c9875a36b983d85f54fb7381c47c + rubocop (1.85.0) sha256=317407feb681a07d54f64d2f9e1d6b6af1ce7678e51cd658e3ad8bd66da48c01 + rubocop-ast (1.49.0) sha256=49c3676d3123a0923d333e20c6c2dbaaae2d2287b475273fddee0c61da9f71fd + rubocop-rspec (3.9.0) sha256=8fa70a3619408237d789aeecfb9beef40576acc855173e60939d63332fdb55e2 + ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33 + thor (1.5.0) sha256=e3a9e55fe857e44859ce104a84675ab6e8cd59c650a49106a05f55f136425e73 + threaded (0.0.4) sha256=8f32c11d29f7c7237aabeb36d286c15ac495137ce06ace3357de743aafd4f0e3 + unicode-display_width (3.2.0) sha256=0cdd96b5681a5949cdbc2c55e7b420facae74c4aaf9a9815eee1087cb1853c42 + unicode-emoji (4.2.0) sha256=519e69150f75652e40bf736106cfbc8f0f73aa3fb6a65afe62fefa7f80b0f80f + webrick (1.9.2) sha256=beb4a15fc474defed24a3bda4ffd88a490d517c9e4e6118c3edce59e45864131 + RUBY VERSION - ruby 3.4.7p58 + ruby 4.0.1 BUNDLED WITH - 2.7.2 + 4.0.7 From a64b2d902e84e5f1e95f46f39ab0098046b0d129 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 3 Mar 2026 16:21:44 +0000 Subject: [PATCH 343/406] Explicitly configure uv to use hard links (#2044) Since in newer versions of uv the default link mode has changed from `hardlink` to `clone` (aka reflink), and reflinks are slower in some environments (such as Heroku's build system). See: https://github.com/astral-sh/uv/issues/18259 Even if the uv default ends up being changed back to hard links again, it seems worth us explicitly requesting that link mode, given that we know it's the best mode for our use-case. GUS-W-21429394. --- CHANGELOG.md | 1 + lib/uv.sh | 4 ++++ spec/hatchet/ci_spec.rb | 2 ++ spec/hatchet/uv_spec.rb | 2 ++ 4 files changed, 9 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 85fe04187..c9c1d2011 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Updated pip from 25.3 to 26.0.1. ([#2032](https://github.com/heroku/heroku-buildpack-python/pull/2032)) +- Explicitly configured uv to use hard links to maintain the behaviour of previous versions. ([#2044](https://github.com/heroku/heroku-buildpack-python/pull/2044)) ## [v336] - 2026-03-02 diff --git a/lib/uv.sh b/lib/uv.sh index 927c84821..53d66e990 100644 --- a/lib/uv.sh +++ b/lib/uv.sh @@ -88,6 +88,9 @@ function uv::install_uv() { # Prevent uv from downloading/using its own Python installation. export UV_NO_MANAGED_PYTHON="1" export UV_PYTHON_DOWNLOADS="never" + # Force uv to use hardlinks rather than the new default of reflinks, since the latter are + # significantly slower on Kodon: https://github.com/astral-sh/uv/issues/18259 + export UV_LINK_MODE="hardlink" # Set the same env vars in the environment used by later buildpacks. cat >>"${export_file}" <<-EOF @@ -95,6 +98,7 @@ function uv::install_uv() { export UV_PROJECT_ENVIRONMENT="${python_home}" export UV_NO_MANAGED_PYTHON="1" export UV_PYTHON_DOWNLOADS="never" + export UV_LINK_MODE="hardlink" EOF # As a performance optimisation, uv attempts to use hardlinks instead of copying files from its diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 60319499b..9b4ba8af9 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -319,6 +319,7 @@ PATH=/tmp/cache\\w+/.heroku/python-uv:/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/ PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config PYTHONUNBUFFERED=1 + UV_LINK_MODE=hardlink UV_NO_MANAGED_PYTHON=1 UV_PROJECT_ENVIRONMENT=/app/.heroku/python UV_PYTHON_DOWNLOADS=never @@ -331,6 +332,7 @@ PYTHONHOME=/app/.heroku/python PYTHONPATH=/app PYTHONUNBUFFERED=true + UV_LINK_MODE=hardlink UV_NO_MANAGED_PYTHON=1 UV_PROJECT_ENVIRONMENT=/app/.heroku/python UV_PYTHON_DOWNLOADS=never diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 766df4dca..3b0d74c21 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -33,6 +33,7 @@ remote: PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config remote: PYTHONUNBUFFERED=1 remote: UV_CACHE_DIR=/tmp/uv-cache + remote: UV_LINK_MODE=hardlink remote: UV_NO_MANAGED_PYTHON=1 remote: UV_PROJECT_ENVIRONMENT=/app/.heroku/python remote: UV_PYTHON_DOWNLOADS=never @@ -46,6 +47,7 @@ remote: PYTHONPATH=/app remote: PYTHONUNBUFFERED=true remote: UV_CACHE_DIR=/tmp/uv-cache + remote: UV_LINK_MODE=hardlink remote: UV_NO_MANAGED_PYTHON=1 remote: UV_PROJECT_ENVIRONMENT=/app/.heroku/python remote: UV_PYTHON_DOWNLOADS=never From 26565e47e78c6e7b204223f728da41e9de1b3f28 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 3 Mar 2026 17:24:53 +0000 Subject: [PATCH 344/406] Bump uv from 0.10.1 to 0.10.7 (#2041) * Bump uv from 0.10.1 to 0.10.7 Bumps [uv](https://github.com/astral-sh/uv) from 0.10.1 to 0.10.7. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.10.1...0.10.7) --- updated-dependencies: - dependency-name: uv dependency-version: 0.10.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c9c1d2011..ed0b8a998 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Updated pip from 25.3 to 26.0.1. ([#2032](https://github.com/heroku/heroku-buildpack-python/pull/2032)) +- Updated uv from 0.10.1 to 0.10.7. ([#2041](https://github.com/heroku/heroku-buildpack-python/pull/2041)) - Explicitly configured uv to use hard links to maintain the behaviour of previous versions. ([#2044](https://github.com/heroku/heroku-buildpack-python/pull/2044)) ## [v336] - 2026-03-02 diff --git a/requirements/uv.txt b/requirements/uv.txt index cc5692614..971e116c0 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.10.1 +uv==0.10.7 From ab4ed894d417d6624e6205f5ac1323f69319d4a4 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 3 Mar 2026 17:37:30 +0000 Subject: [PATCH 345/406] Prepare release v337 (#2045) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ed0b8a998..fbf0a7937 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v337] - 2026-03-03 + - Updated pip from 25.3 to 26.0.1. ([#2032](https://github.com/heroku/heroku-buildpack-python/pull/2032)) - Updated uv from 0.10.1 to 0.10.7. ([#2041](https://github.com/heroku/heroku-buildpack-python/pull/2041)) - Explicitly configured uv to use hard links to maintain the behaviour of previous versions. ([#2044](https://github.com/heroku/heroku-buildpack-python/pull/2044)) @@ -1487,7 +1490,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v336...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v337...main +[v337]: https://github.com/heroku/heroku-buildpack-python/compare/v336...v337 [v336]: https://github.com/heroku/heroku-buildpack-python/compare/v335...v336 [v335]: https://github.com/heroku/heroku-buildpack-python/compare/v334...v335 [v334]: https://github.com/heroku/heroku-buildpack-python/compare/v333...v334 From 07bf8f0a81dbbae24065a8aa5b282bd894b9a372 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 3 Mar 2026 20:11:24 +0000 Subject: [PATCH 346/406] Add support for Python 3.12.13, 3.11.15 and 3.10.20 (#2046) Release announcement: https://blog.python.org/2026/03/python-31213-31115-31020/ Changelogs: https://docs.python.org/release/3.12.13/whatsnew/changelog.html#python-3-12-13-final https://docs.python.org/release/3.11.15/whatsnew/changelog.html#python-3-11-15-final https://docs.python.org/release/3.10.20/whatsnew/changelog.html#python-3-10-20-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/22639900939 https://github.com/heroku/heroku-buildpack-python/actions/runs/22639906742 https://github.com/heroku/heroku-buildpack-python/actions/runs/22639923074 GUS-W-21433843. --- CHANGELOG.md | 3 +++ lib/python_version.sh | 6 +++--- spec/spec_helper.rb | 6 +++--- 3 files changed, 9 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fbf0a7937..391fa2a46 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] +- The Python 3.12 version alias now resolves to Python 3.12.13. ([#2046](https://github.com/heroku/heroku-buildpack-python/pull/2046)) +- The Python 3.11 version alias now resolves to Python 3.11.15. ([#2046](https://github.com/heroku/heroku-buildpack-python/pull/2046)) +- The Python 3.10 version alias now resolves to Python 3.10.20. ([#2046](https://github.com/heroku/heroku-buildpack-python/pull/2046)) ## [v337] - 2026-03-03 diff --git a/lib/python_version.sh b/lib/python_version.sh index d3e97631b..a12943e8f 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -4,9 +4,9 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail -LATEST_PYTHON_3_10="3.10.19" -LATEST_PYTHON_3_11="3.11.14" -LATEST_PYTHON_3_12="3.12.12" +LATEST_PYTHON_3_10="3.10.20" +LATEST_PYTHON_3_11="3.11.15" +LATEST_PYTHON_3_12="3.12.13" LATEST_PYTHON_3_13="3.13.12" LATEST_PYTHON_3_14="3.14.3" diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 5652c9ff3..1fed97721 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -10,9 +10,9 @@ FIXTURE_DIR = Pathname.new(__FILE__).parent.join('fixtures') -LATEST_PYTHON_3_10 = '3.10.19' -LATEST_PYTHON_3_11 = '3.11.14' -LATEST_PYTHON_3_12 = '3.12.12' +LATEST_PYTHON_3_10 = '3.10.20' +LATEST_PYTHON_3_11 = '3.11.15' +LATEST_PYTHON_3_12 = '3.12.13' LATEST_PYTHON_3_13 = '3.13.12' LATEST_PYTHON_3_14 = '3.14.3' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_14 From 855d7e66d99357f814f74a1e0920f6c0f377f64f Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 3 Mar 2026 20:18:59 +0000 Subject: [PATCH 347/406] Prepare release v338 (#2047) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 391fa2a46..5678fc956 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v338] - 2026-03-03 + - The Python 3.12 version alias now resolves to Python 3.12.13. ([#2046](https://github.com/heroku/heroku-buildpack-python/pull/2046)) - The Python 3.11 version alias now resolves to Python 3.11.15. ([#2046](https://github.com/heroku/heroku-buildpack-python/pull/2046)) - The Python 3.10 version alias now resolves to Python 3.10.20. ([#2046](https://github.com/heroku/heroku-buildpack-python/pull/2046)) @@ -1493,7 +1496,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v337...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v338...main +[v338]: https://github.com/heroku/heroku-buildpack-python/compare/v337...v338 [v337]: https://github.com/heroku/heroku-buildpack-python/compare/v336...v337 [v336]: https://github.com/heroku/heroku-buildpack-python/compare/v335...v336 [v335]: https://github.com/heroku/heroku-buildpack-python/compare/v334...v335 From 4f3aa881acd4df1c1ef58b886c79bb6d6d077f2d Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 4 Mar 2026 09:41:18 +0000 Subject: [PATCH 348/406] Update test fixture after change in S3 bucket permissions (#2048) S3 directory listing permissions were removed from the buildpack's S3 bucket, which means non-existent files now return 403 rather than 404. The implementation for error handling already works with both: https://github.com/heroku/heroku-buildpack-python/blob/855d7e66d99357f814f74a1e0920f6c0f377f64f/lib/python.sh#L74-L76 However, the test assertion needs to be loosened to resolve this CI failure: https://github.com/heroku/heroku-buildpack-python/actions/runs/22640940619/job/65680836050#step:5:497 GUS-W-21443369. --- spec/hatchet/python_version_spec.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 2b70356ab..0f4a13350 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -527,7 +527,7 @@ remote: -----> Python app detected remote: -----> Using Python 3.12.999 specified in .python-version remote: -----> Installing Python 3.12.999 - remote: curl: \\(22\\) The requested URL returned error: 404.* + remote: curl: \\(22\\) The requested URL returned error: .+ remote: zstd: /\\*stdin\\*\\\\: unexpected end of file remote: tar: Child returned status 1 remote: tar: Error is not recoverable: exiting now From 75dd5af244b2ef12091dd8b5941c8454566596c4 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 12 Mar 2026 09:02:28 +0000 Subject: [PATCH 349/406] Bump uv from 0.10.7 to 0.10.9 (#2049) * Bump uv from 0.10.7 to 0.10.9 Bumps [uv](https://github.com/astral-sh/uv) from 0.10.7 to 0.10.9. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.10.7...0.10.9) --- updated-dependencies: - dependency-name: uv dependency-version: 0.10.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry * Update detection test output Since the wording of the build system part of the error message has since been changed in as part of fixing links to Dev Center. --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- spec/hatchet/detect_spec.rb | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5678fc956..c33d56de7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.10.7 to 0.10.9. ([#2049](https://github.com/heroku/heroku-buildpack-python/pull/2049)) ## [v338] - 2026-03-03 diff --git a/requirements/uv.txt b/requirements/uv.txt index 971e116c0..6f2d9f3e5 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.10.7 +uv==0.10.9 diff --git a/spec/hatchet/detect_spec.rb b/spec/hatchet/detect_spec.rb index 2a44089c2..4a93262ee 100644 --- a/spec/hatchet/detect_spec.rb +++ b/spec/hatchet/detect_spec.rb @@ -48,7 +48,7 @@ remote: ! https://devcenter.heroku.com/articles/python-support remote: remote: - remote: More info: https://devcenter.heroku.com/articles/buildpacks#detection-failure + remote: More info: https://devcenter.heroku.com/articles/buildpacks#buildpack-auto-detection OUTPUT end end From 108a4166f71bc341d25f6814730ef5a8c59b69c8 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 12 Mar 2026 10:18:55 +0000 Subject: [PATCH 350/406] Download uv from `releases.astral.sh` instead of GitHub releases (#2050) Since Astral now mirror uv releases to `releases.astral.sh`, which is backed by Cloudflare R2 / CDN, and so is more reliable than GitHub releases. See: https://github.com/astral-sh/uv/pull/18207#issuecomment-3997795493 GUS-W-21561065. --- CHANGELOG.md | 1 + lib/uv.sh | 10 +++++----- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c33d56de7..a52014743 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Updated uv from 0.10.7 to 0.10.9. ([#2049](https://github.com/heroku/heroku-buildpack-python/pull/2049)) +- Switched to downloading uv from `releases.astral.sh` instead of GitHub releases. ([#2050](https://github.com/heroku/heroku-buildpack-python/pull/2050)) ## [v338] - 2026-03-03 diff --git a/lib/uv.sh b/lib/uv.sh index 53d66e990..a3bc2d2bb 100644 --- a/lib/uv.sh +++ b/lib/uv.sh @@ -27,7 +27,7 @@ function uv::install_uv() { local gnu_arch # eg: `x86_64` or `aarch64`. gnu_arch=$(arch) - local uv_url="https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-${gnu_arch}-unknown-linux-gnu.tar.gz" + local uv_url="https://releases.astral.sh/github/uv/releases/download/${UV_VERSION}/uv-${gnu_arch}-unknown-linux-gnu.tar.gz" local error_log error_log=$(mktemp) @@ -64,16 +64,16 @@ function uv::install_uv() { output::error <<-EOF Error: Unable to install uv. - Failed to download/install uv from GitHub: + Failed to download/install uv from: ${uv_url} In some cases, this happens due to a temporary issue with - the network connection or GitHub's API/CDN. + the network connection or Astral's CDN, Cloudflare. Try building again to see if the error resolves itself. - If that doesn't help, check the status of GitHub here: - https://www.githubstatus.com + If that doesn't help, check the status of Cloudflare here: + https://www.cloudflarestatus.com EOF build_data::set_string "failure_reason" "install-package-manager::uv" # e.g.: 'curl: (56) Recv failure: Connection reset by peer' From 9a9cb664c1ebc61b767de8e6b97d3f64a4064448 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 13 Mar 2026 12:40:41 +0000 Subject: [PATCH 351/406] Add more misspellings to the `.python-version` filename check (#2051) Based on misspellings seen in Honeycomb, that affected more than one app. Having this fixed list of misspellings isn't ideal (given there will be a long tail of misspellings not on it), however, short term it allows us to offer a very specific error message so the majority of apps are fixed by their owners, whilst still maintaining a low false positive rate (vs a looser regex). Longer term we'll likely be making the `.python-version` file mandatory, at which point the error message for that can list out the files in the app source root (the same way we do for detection errors), which will help with the long tail cases. (We can also do things like replace unicode or stray whitespace in the file listing, to make those confusing for the user cases easier to spot too.) GUS-W-21320154. --- CHANGELOG.md | 1 + lib/python_version.sh | 3 +++ 2 files changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a52014743..0b18cadb0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ - Updated uv from 0.10.7 to 0.10.9. ([#2049](https://github.com/heroku/heroku-buildpack-python/pull/2049)) - Switched to downloading uv from `releases.astral.sh` instead of GitHub releases. ([#2050](https://github.com/heroku/heroku-buildpack-python/pull/2050)) +- Updated the check for misspelled `.python-version` files with additional misspellings seen in the wild. ([#2051](https://github.com/heroku/heroku-buildpack-python/pull/2051)) ## [v338] - 2026-03-03 diff --git a/lib/python_version.sh b/lib/python_version.sh index a12943e8f..a0a778320 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -25,9 +25,12 @@ PYTHON_FULL_VERSION_REGEX="${INT_REGEX}\.${INT_REGEX}\.${INT_REGEX}" # Misspellings of the `.python-version` file seen in the wild. MISSPELLED_PYTHON_VERSION_FILE_NAMES=( + " .python-version" + ". python-version" ".python_version.txt" ".python_version" ".python-version " + ",python-version" ".python-version." ".python-version.py" ".python-version.rtf" From 76fa6db5acc2d5f8a5aa2726794b8cb11083fd1b Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Fri, 13 Mar 2026 12:44:48 +0000 Subject: [PATCH 352/406] Prepare release v339 (#2052) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b18cadb0..d5c82b3d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v339] - 2026-03-13 + - Updated uv from 0.10.7 to 0.10.9. ([#2049](https://github.com/heroku/heroku-buildpack-python/pull/2049)) - Switched to downloading uv from `releases.astral.sh` instead of GitHub releases. ([#2050](https://github.com/heroku/heroku-buildpack-python/pull/2050)) - Updated the check for misspelled `.python-version` files with additional misspellings seen in the wild. ([#2051](https://github.com/heroku/heroku-buildpack-python/pull/2051)) @@ -1499,7 +1502,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v338...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v339...main +[v339]: https://github.com/heroku/heroku-buildpack-python/compare/v338...v339 [v338]: https://github.com/heroku/heroku-buildpack-python/compare/v337...v338 [v337]: https://github.com/heroku/heroku-buildpack-python/compare/v336...v337 [v336]: https://github.com/heroku/heroku-buildpack-python/compare/v335...v336 From 3303da8d68d49b7cfe2a87fb5572d95b3bf7772c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 19 Mar 2026 13:06:13 +0000 Subject: [PATCH 353/406] Bump json from 2.18.1 to 2.19.2 (#2055) Bumps [json](https://github.com/ruby/json) from 2.18.1 to 2.19.2. - [Release notes](https://github.com/ruby/json/releases) - [Changelog](https://github.com/ruby/json/blob/master/CHANGES.md) - [Commits](https://github.com/ruby/json/compare/v2.18.1...v2.19.2) --- updated-dependencies: - dependency-name: json dependency-version: 2.19.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 4c7138d8f..b18ebec83 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -23,7 +23,7 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.18.1) + json (2.19.2) json-schema (6.1.0) addressable (~> 2.8) bigdecimal (>= 3.1, < 5) @@ -110,7 +110,7 @@ CHECKSUMS excon (1.4.0) sha256=5d2bc9d2c79511a562e7fcac77cc7a40acd9cebcc55b80e537975ad8187f2924 heroics (0.1.3) sha256=31d792e8706ecc6f78299f52d0b0f8ab55489a13b5f4f76c3050b07912563562 heroku_hatchet (8.0.6) sha256=886a7d7d686859db9fac19df1194a35c5349285956ca4d134054c5c376accea3 - json (2.18.1) sha256=fe112755501b8d0466b5ada6cf50c8c3f41e897fa128ac5d263ec09eedc9f986 + json (2.19.2) sha256=e7e1bd318b2c37c4ceee2444841c86539bc462e81f40d134cf97826cb14e83cf json-schema (6.1.0) sha256=6bf70a2cfb6dfd5a06da28093fa8190f324c88eabd36a7f47097f227321dc702 language_server-protocol (3.17.0.5) sha256=fd1e39a51a28bf3eec959379985a72e296e9f9acfce46f6a79d31ca8760803cc lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87 From 8b3b0e857fc3e29d91c1ee465dae4d671d59bc16 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 20 Mar 2026 15:08:56 +0000 Subject: [PATCH 354/406] Add binary build support for Heroku-26 (#2056) Adds support for building the binaries for the upcoming Heroku-26 stack based on Ubuntu 26.04. A later PR will add support for the stack to the buildpack itself, once the Heroku build system supports Heroku-26 (since until that point we can't run Hatchet in CI anyway). The `libcrypt-dev` library had to be added to the `Dockerfile` used for binary building, since the headers package is no longer in `heroku/heroku:26-build` as it was previously only a transitive dependency in the build image, and with the upstream Ubuntu 26.04 package changes is now no longer pulled in. (From searching it doesn't appear to be widely used, so I don't think it's worth adding back to the Heroku-26 build image.) Doing so fixes this warning on Python 3.12 and older (the `crypt` module only exists in older Python versions): ``` *** WARNING: renaming "_crypt" since importing it failed: /tmp/src/build/lib.linux-x86_64-3.10/_crypt.cpython-310-x86_64-linux-gnu.so: undefined symbol: crypt ... Following modules built successfully but were removed because they could not be imported: _crypt ``` Dry-run test builds using this branch succeeded. e.g.: https://github.com/heroku/heroku-buildpack-python/actions/runs/23343385460 https://github.com/heroku/heroku-buildpack-python/actions/runs/23343379302 GUS-W-20775546. --- .github/workflows/build_python_runtime.yml | 26 ++++++++++++++++++++++ builds/Dockerfile | 4 ++++ builds/build_python_runtime.sh | 2 +- builds/test_python_runtime.sh | 9 ++++++++ 4 files changed, 40 insertions(+), 1 deletion(-) diff --git a/.github/workflows/build_python_runtime.yml b/.github/workflows/build_python_runtime.yml index fa87c1fbf..378229093 100644 --- a/.github/workflows/build_python_runtime.yml +++ b/.github/workflows/build_python_runtime.yml @@ -15,6 +15,7 @@ on: - auto - heroku-22 - heroku-24 + - heroku-26 default: auto required: false dry_run: @@ -80,3 +81,28 @@ jobs: - name: Upload Python runtime archive to S3 if: (!inputs.dry_run) run: aws s3 sync ./upload "s3://${S3_BUCKET}" + + heroku-26: + if: inputs.stack == 'heroku-26' || inputs.stack == 'auto' + strategy: + fail-fast: false + matrix: + arch: ["amd64", "arm64"] + runs-on: ${{ matrix.arch == 'arm64' && 'pub-hk-ubuntu-24.04-arm-xlarge' || 'pub-hk-ubuntu-24.04-xlarge' }} + env: + STACK_VERSION: "26" + steps: + - name: Checkout + uses: actions/checkout@v6 + - name: Build Docker image + run: docker build --platform="linux/${{ matrix.arch }}" --pull --tag buildenv --build-arg=STACK_VERSION builds/ + - name: Compile and package Python runtime + run: docker run --rm --volume="${PWD}/upload:/tmp/upload" buildenv ./build_python_runtime.sh "${{ inputs.python_version }}" + - name: Test Python runtime + run: | + RUN_IMAGE='heroku/heroku:${{ env.STACK_VERSION }}' + ARCHIVE_FILENAME='python-${{ inputs.python_version }}-ubuntu-${{ env.STACK_VERSION }}.04-${{ matrix.arch }}.tar.zst' + docker run --rm --volume="${PWD}/upload:/upload:ro" --volume="${PWD}/builds:/builds:ro" "${RUN_IMAGE}" /builds/test_python_runtime.sh "/upload/${ARCHIVE_FILENAME}" + - name: Upload Python runtime archive to S3 + if: (!inputs.dry_run) + run: aws s3 sync ./upload "s3://${S3_BUCKET}" diff --git a/builds/Dockerfile b/builds/Dockerfile index 53610332e..44bfb88c8 100644 --- a/builds/Dockerfile +++ b/builds/Dockerfile @@ -8,8 +8,12 @@ ENV STACK="heroku-${STACK_VERSION}" # For Heroku-24 and newer, the build image sets a non-root default `USER`. USER root +# `libcrypt-dev` can be removed once we drop support for Python 3.12, +# since the `crypt` module was removed in Python 3.13: +# https://docs.python.org/3.13/whatsnew/3.13.html#whatsnew313-pep594 RUN apt-get update --error-on=any \ && apt-get install -y --no-install-recommends \ + libcrypt-dev \ libdb-dev \ libreadline-dev \ libsqlite3-dev \ diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index 461d8d24b..6e0882b58 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -25,7 +25,7 @@ function abort() { } case "${STACK:?}" in - heroku-22 | heroku-24) + heroku-22 | heroku-24 | heroku-26) SUPPORTED_PYTHON_VERSIONS=( "3.10" "3.11" diff --git a/builds/test_python_runtime.sh b/builds/test_python_runtime.sh index adc701e4f..a1ddc92a8 100755 --- a/builds/test_python_runtime.sh +++ b/builds/test_python_runtime.sh @@ -68,6 +68,7 @@ optional_stdlib_modules=( zlib ) +# zstd support was added in Python 3.14: # https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-zstandard if [[ "${major_python_version}" == "3.14" ]]; then optional_stdlib_modules+=( @@ -75,6 +76,14 @@ if [[ "${major_python_version}" == "3.14" ]]; then ) fi +# crypt was removed in Python 3.13: +# https://docs.python.org/3.13/whatsnew/3.13.html#whatsnew313-pep594 +if [[ "${major_python_version}" == +(3.10|3.11|3.12) ]]; then + optional_stdlib_modules+=( + crypt + ) +fi + if "${INSTALL_DIR}/bin/python3" -c "import $( IFS=, echo "${optional_stdlib_modules[*]}" From f6f11c00f991c45911f2bc356b019ce909d64e1f Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 23 Mar 2026 20:54:04 +0000 Subject: [PATCH 355/406] Fix GUSINFO metadata in CODEOWNERS (#2059) Since the Languages team was renamed to `Heroku - Languages` in GUS some time ago. GUS-W-21708465. --- .github/CODEOWNERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index fa79aa89b..9971b391e 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,6 +1,6 @@ # Default to requesting pull request reviews from the Heroku Languages team. #ECCN:Open Source -#GUSINFO:Languages,Heroku Python Platform +#GUSINFO:Heroku - Languages,Heroku Python Platform * @heroku/languages # However, request review from the language owner instead for files that are updated From e8b7c62ebff6db57fa98f1bef117497cba81f40d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 27 Mar 2026 20:47:16 +0000 Subject: [PATCH 356/406] Bump rubocop from 1.85.0 to 1.86.0 in the ruby-dependencies group (#2061) Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.85.0 to 1.86.0 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.85.0...v1.86.0) --- updated-dependencies: - dependency-name: rubocop dependency-version: 1.86.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 31 ++++++++----------------------- 1 file changed, 8 insertions(+), 23 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index b18ebec83..13b3f38a5 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,11 +1,8 @@ GEM remote: https://rubygems.org/ specs: - addressable (2.8.9) - public_suffix (>= 2.0.2, < 8.0) ast (2.4.3) base64 (0.3.0) - bigdecimal (4.0.1) diff-lcs (1.6.2) erubis (2.7.0) excon (1.4.0) @@ -23,22 +20,17 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.19.2) - json-schema (6.1.0) - addressable (~> 2.8) - bigdecimal (>= 3.1, < 5) + json (2.19.3) language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) - mcp (0.8.0) - json-schema (>= 4.1) moneta (1.0.0) multi_json (1.19.1) parallel (1.27.0) parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) - parser (3.3.10.2) + parser (3.3.11.1) ast (~> 2.4.1) racc platform-api (3.9.0) @@ -46,7 +38,6 @@ GEM moneta (~> 1.0.0) rate_throttle_client (~> 0.1.0) prism (1.9.0) - public_suffix (7.0.5) racc (1.8.1) rainbow (3.1.1) rate_throttle_client (0.1.2) @@ -60,11 +51,10 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.7) - rubocop (1.85.0) + rubocop (1.86.0) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) - mcp (~> 0.6) parallel (~> 1.10) parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) @@ -72,7 +62,7 @@ GEM rubocop-ast (>= 1.49.0, < 2.0) ruby-progressbar (~> 1.7) unicode-display_width (>= 2.4.0, < 4.0) - rubocop-ast (1.49.0) + rubocop-ast (1.49.1) parser (>= 3.3.7.2) prism (~> 1.7) rubocop-rspec (3.9.0) @@ -101,29 +91,24 @@ DEPENDENCIES rubocop-rspec CHECKSUMS - addressable (2.8.9) sha256=cc154fcbe689711808a43601dee7b980238ce54368d23e127421753e46895485 ast (2.4.3) sha256=954615157c1d6a382bc27d690d973195e79db7f55e9765ac7c481c60bdb4d383 base64 (0.3.0) sha256=27337aeabad6ffae05c265c450490628ef3ebd4b67be58257393227588f5a97b - bigdecimal (4.0.1) sha256=8b07d3d065a9f921c80ceaea7c9d4ae596697295b584c296fe599dd0ad01c4a7 diff-lcs (1.6.2) sha256=9ae0d2cba7d4df3075fe8cd8602a8604993efc0dfa934cff568969efb1909962 erubis (2.7.0) sha256=63653f5174a7997f6f1d6f465fbe1494dcc4bdab1fb8e635f6216989fb1148ba excon (1.4.0) sha256=5d2bc9d2c79511a562e7fcac77cc7a40acd9cebcc55b80e537975ad8187f2924 heroics (0.1.3) sha256=31d792e8706ecc6f78299f52d0b0f8ab55489a13b5f4f76c3050b07912563562 heroku_hatchet (8.0.6) sha256=886a7d7d686859db9fac19df1194a35c5349285956ca4d134054c5c376accea3 - json (2.19.2) sha256=e7e1bd318b2c37c4ceee2444841c86539bc462e81f40d134cf97826cb14e83cf - json-schema (6.1.0) sha256=6bf70a2cfb6dfd5a06da28093fa8190f324c88eabd36a7f47097f227321dc702 + json (2.19.3) sha256=289b0bb53052a1fa8c34ab33cc750b659ba14a5c45f3fcf4b18762dc67c78646 language_server-protocol (3.17.0.5) sha256=fd1e39a51a28bf3eec959379985a72e296e9f9acfce46f6a79d31ca8760803cc lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87 logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203 - mcp (0.8.0) sha256=ae8bd146bb8e168852866fd26f805f52744f6326afb3211e073f78a95e0c34fb moneta (1.0.0) sha256=2224e5a68156e8eceb525fb0582c8c4e0f29f67cae86507cdcfb406abbb1fc5d multi_json (1.19.1) sha256=7aefeff8f2c854bf739931a238e4aea64592845e0c0395c8a7d2eea7fdd631b7 parallel (1.27.0) sha256=4ac151e1806b755fb4e2dc2332cbf0e54f2e24ba821ff2d3dcf86bf6dc4ae130 parallel_split_test (0.10.0) sha256=4abd6bedc6a1b169ecb93a1bf982fa7fbd437b66c360594d56d107c25f53b05f - parser (3.3.10.2) sha256=6f60c84aa4bdcedb6d1a2434b738fe8a8136807b6adc8f7f53b97da9bc4e9357 + parser (3.3.11.1) sha256=d17ace7aabe3e72c3cc94043714be27cc6f852f104d81aa284c2281aecc65d54 platform-api (3.9.0) sha256=be3b919955c52649fd931a9f62d571bd6f06613de630970dc7255fc95eb7d962 prism (1.9.0) sha256=7b530c6a9f92c24300014919c9dcbc055bf4cdf51ec30aed099b06cd6674ef85 - public_suffix (7.0.5) sha256=1a8bb08f1bbea19228d3bed6e5ed908d1cb4f7c2726d18bd9cadf60bc676f623 racc (1.8.1) sha256=4a7f6929691dbec8b5209a0b373bc2614882b55fc5d2e447a21aaa691303d62f rainbow (3.1.1) sha256=039491aa3a89f42efa1d6dec2fc4e62ede96eb6acd95e52f1ad581182b79bc6a rate_throttle_client (0.1.2) sha256=f9de968b892fea9272154f6182b4f5cfb74292585e66763fb8a8510181ec83ee @@ -133,8 +118,8 @@ CHECKSUMS rspec-expectations (3.13.5) sha256=33a4d3a1d95060aea4c94e9f237030a8f9eae5615e9bd85718fe3a09e4b58836 rspec-retry (0.6.2) sha256=6101ba23a38809811ae3484acde4ab481c54d846ac66d5037ccb40131a60d858 rspec-support (3.13.7) sha256=0640e5570872aafefd79867901deeeeb40b0c9875a36b983d85f54fb7381c47c - rubocop (1.85.0) sha256=317407feb681a07d54f64d2f9e1d6b6af1ce7678e51cd658e3ad8bd66da48c01 - rubocop-ast (1.49.0) sha256=49c3676d3123a0923d333e20c6c2dbaaae2d2287b475273fddee0c61da9f71fd + rubocop (1.86.0) sha256=4ff1186fe16ebe9baff5e7aad66bb0ad4cabf5cdcd419f773146dbba2565d186 + rubocop-ast (1.49.1) sha256=4412f3ee70f6fe4546cc489548e0f6fcf76cafcfa80fa03af67098ffed755035 rubocop-rspec (3.9.0) sha256=8fa70a3619408237d789aeecfb9beef40576acc855173e60939d63332fdb55e2 ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33 thor (1.5.0) sha256=e3a9e55fe857e44859ce104a84675ab6e8cd59c650a49106a05f55f136425e73 From 3310064f18d660ba0b578781588ead6442f7f53e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 2 Apr 2026 12:10:37 +0000 Subject: [PATCH 357/406] Bump uv from 0.10.9 to 0.11.3 (#2067) * Bump uv from 0.10.9 to 0.11.3 Bumps [uv](https://github.com/astral-sh/uv) from 0.10.9 to 0.11.3. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.10.9...0.11.3) --- updated-dependencies: - dependency-name: uv dependency-version: 0.11.3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Add changelog entry * Update expected test output --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- spec/hatchet/uv_spec.rb | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d5c82b3d1..5fe186a15 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.10.9 to 0.11.3. ([#2067](https://github.com/heroku/heroku-buildpack-python/pull/2067)) ## [v339] - 2026-03-13 diff --git a/requirements/uv.txt b/requirements/uv.txt index 6f2d9f3e5..0aaa7b861 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.10.9 +uv==0.11.3 diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 3b0d74c21..9eedb5af6 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -59,7 +59,7 @@ remote: '/app/.heroku/python/lib/python3.14/lib-dynload', remote: '/app/.heroku/python/lib/python3.14/site-packages'\\] remote: - remote: uv #{UV_VERSION} + remote: uv #{UV_VERSION} \\(x86_64-unknown-linux-gnu\\) remote: Using Python #{DEFAULT_PYTHON_FULL_VERSION} environment at: /app/.heroku/python remote: Package Version remote: ----------------- ------- From c5545e9d35d6eda13324ae9519f7766fc3d11bf7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 2 Apr 2026 13:17:37 +0100 Subject: [PATCH 358/406] Bump poetry from 2.3.2 to 2.3.3 (#2064) * Bump poetry from 2.3.2 to 2.3.3 Bumps [poetry](https://github.com/python-poetry/poetry) from 2.3.2 to 2.3.3. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](https://github.com/python-poetry/poetry/compare/2.3.2...2.3.3) --- updated-dependencies: - dependency-name: poetry dependency-version: 2.3.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/poetry.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5fe186a15..2db80e784 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated Poetry from 2.3.2 to 2.3.3. ([#2064](https://github.com/heroku/heroku-buildpack-python/pull/2064)) - Updated uv from 0.10.9 to 0.11.3. ([#2067](https://github.com/heroku/heroku-buildpack-python/pull/2067)) ## [v339] - 2026-03-13 diff --git a/requirements/poetry.txt b/requirements/poetry.txt index f69508193..32a21352c 100644 --- a/requirements/poetry.txt +++ b/requirements/poetry.txt @@ -1 +1 @@ -poetry==2.3.2 +poetry==2.3.3 From d02e0cba645b41e68d4a888905ca520d6d3ff289 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 2 Apr 2026 13:25:20 +0100 Subject: [PATCH 359/406] Bump pipenv from 2026.0.3 to 2026.5.1 (#2065) * Bump pipenv from 2026.0.3 to 2026.5.1 Bumps [pipenv](https://github.com/pypa/pipenv) from 2026.0.3 to 2026.5.1. - [Release notes](https://github.com/pypa/pipenv/releases) - [Changelog](https://github.com/pypa/pipenv/blob/main/CHANGELOG.md) - [Commits](https://github.com/pypa/pipenv/compare/v2026.0.3...v2026.5.1) --- updated-dependencies: - dependency-name: pipenv dependency-version: 2026.5.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/pipenv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2db80e784..8c05194e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated Pipenv from 2026.0.3 to 2026.5.1. ([#2065](https://github.com/heroku/heroku-buildpack-python/pull/2065)) - Updated Poetry from 2.3.2 to 2.3.3. ([#2064](https://github.com/heroku/heroku-buildpack-python/pull/2064)) - Updated uv from 0.10.9 to 0.11.3. ([#2067](https://github.com/heroku/heroku-buildpack-python/pull/2067)) diff --git a/requirements/pipenv.txt b/requirements/pipenv.txt index faed680ba..69a9727aa 100644 --- a/requirements/pipenv.txt +++ b/requirements/pipenv.txt @@ -1 +1 @@ -pipenv==2026.0.3 +pipenv==2026.5.1 From 20fd6330ce28e1f12ad354516c34f0a8275c2f78 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 2 Apr 2026 13:37:21 +0100 Subject: [PATCH 360/406] Prepare release v340 (#2068) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c05194e8..6fe85c2d0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v340] - 2026-04-02 + - Updated Pipenv from 2026.0.3 to 2026.5.1. ([#2065](https://github.com/heroku/heroku-buildpack-python/pull/2065)) - Updated Poetry from 2.3.2 to 2.3.3. ([#2064](https://github.com/heroku/heroku-buildpack-python/pull/2064)) - Updated uv from 0.10.9 to 0.11.3. ([#2067](https://github.com/heroku/heroku-buildpack-python/pull/2067)) @@ -1505,7 +1508,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v339...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v340...main +[v340]: https://github.com/heroku/heroku-buildpack-python/compare/v339...v340 [v339]: https://github.com/heroku/heroku-buildpack-python/compare/v338...v339 [v338]: https://github.com/heroku/heroku-buildpack-python/compare/v337...v338 [v337]: https://github.com/heroku/heroku-buildpack-python/compare/v336...v337 From 843f76fd438bb7881f555bdf842488240982ab95 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 2 Apr 2026 14:01:19 +0100 Subject: [PATCH 361/406] Reduce Dependabot frequency for Python dependencies (#2069) From weekly to monthly, to reduce toil and match the frequency used for other dependencies - now that uv is more mature and we no longer need to keep as rapidly up to date. We can still trigger manual Dependabot runs as needed (and GitHub will continue to open out-of-schedule PRs for any security updates if needed). GUS-W-21870812. Signed-off-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- .github/dependabot.yml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 04310cda2..d3903c048 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -32,9 +32,7 @@ updates: - package-ecosystem: "pip" directory: "/" schedule: - # We set this to a more frequent interval than the above since uv is still under rapid - # iteration, and we'll generally want to be on a recent (if not the latest) version. - interval: "weekly" + interval: "monthly" ignore: # We're not updating to setuptools v71+ due to its new approach to vendored dependencies: # https://github.com/heroku/heroku-buildpack-python/pull/1630#issuecomment-2324236653 From 845b74afc14e0572a58c91ada0fb4bfeec457010 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 7 Apr 2026 23:34:44 +0100 Subject: [PATCH 362/406] Fix typos and other inconsistencies in comments, tests and CHANGELOG (#2070) As found by a Claude Code review pass of docs/comments/tests. --- .editorconfig | 2 +- CHANGELOG.md | 12 ++++++------ bin/compile | 7 +++---- builds/build_python_runtime.sh | 2 +- builds/test_python_runtime.sh | 2 +- lib/cache.sh | 2 +- lib/hooks.sh | 2 +- lib/python_version.sh | 2 +- lib/utils.sh | 2 +- lib/uv.sh | 2 +- spec/fixtures/uv_editable/bin/test-entrypoints.sh | 2 +- spec/hatchet/pipenv_spec.rb | 4 ++-- spec/hatchet/python_version_spec.rb | 8 ++++---- spec/spec_helper.rb | 2 +- 14 files changed, 25 insertions(+), 26 deletions(-) diff --git a/.editorconfig b/.editorconfig index 0d78a23d9..67289a444 100644 --- a/.editorconfig +++ b/.editorconfig @@ -29,7 +29,7 @@ shell_variant = bash switch_case_indent = true # The setup-ruby GitHub Action creates this directory when caching is enabled, and if -# its not ignored will cause false positives when running shfmt in the CI lint job. +# it's not ignored will cause false positives when running shfmt in the CI lint job. [vendor/bundle/**] ignore = true diff --git a/CHANGELOG.md b/CHANGELOG.md index 6fe85c2d0..52958b1b8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -124,7 +124,7 @@ ## [v318] - 2025-11-12 - Improved the error message shown when a `.python-version` file uses an unsupported file encoding. ([#1958](https://github.com/heroku/heroku-buildpack-python/pull/1958)) -- Improved the error message shown when a `.python-version` or `runtime.txt` file contain an invalid Python version. ([#1958](https://github.com/heroku/heroku-buildpack-python/pull/1958)) +- Improved the error message shown when a `.python-version` or `runtime.txt` file contains an invalid Python version. ([#1958](https://github.com/heroku/heroku-buildpack-python/pull/1958)) - Added support for commented lines in `runtime.txt`, for parity with `.python-version`. ([#1958](https://github.com/heroku/heroku-buildpack-python/pull/1958)) ## [v317] - 2025-11-03 @@ -259,7 +259,7 @@ ## [v292] - 2025-07-23 - Updated Pipenv from 2024.0.1 to 2025.0.4. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840)) -- Fixed the way Pipenv is installed, so that it and its dependencies are installed into a separate virtual environment rather than same environment as the app. If your app inadvertently depended on Pipenv's internal dependencies, you will need to add those dependencies explicitly to your `Pipfile`. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840)) +- Fixed the way Pipenv is installed, so that it and its dependencies are installed into a separate virtual environment rather than the same environment as the app. If your app inadvertently depended on Pipenv's internal dependencies, you will need to add those dependencies explicitly to your `Pipfile`. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840)) - Stopped installing pip when Pipenv is the chosen package manager. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840)) - The build cache is now cleared when using Pipenv if the contents of `Pipfile.lock` has changed since the last build. This is required to work around Pipenv not uninstalling packages when they are removed from the lockfile. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840)) - The build now errors when using Pipenv without its lockfile (`Pipfile.lock`). This replaces the warning displayed since November 2024. ([#1833](https://github.com/heroku/heroku-buildpack-python/pull/1833)) @@ -555,7 +555,7 @@ ## [v243] - 2024-02-07 -- Added support for Python 3.11.8 and 3.12.2. ([#1538](https://github.com/heroku/heroku-buildpack-python/pull/1538)). +- Added support for Python 3.11.8 and 3.12.2. ([#1538](https://github.com/heroku/heroku-buildpack-python/pull/1538)) - Changed the default Python version for new apps from 3.12.1 to 3.12.2. ([#1538](https://github.com/heroku/heroku-buildpack-python/pull/1538)) ## [v242] - 2024-01-11 @@ -566,8 +566,8 @@ ## [v241] - 2023-12-08 -- Changed the default Python version for new apps from Python 3.11 to Python 3.12. ([#1516](https://github.com/heroku/heroku-buildpack-python/pull/1516)). -- Added support for Python 3.11.7 and 3.12.1. ([#1517](https://github.com/heroku/heroku-buildpack-python/pull/1517) and [#1518](https://github.com/heroku/heroku-buildpack-python/pull/1518)). +- Changed the default Python version for new apps from Python 3.11 to Python 3.12. ([#1516](https://github.com/heroku/heroku-buildpack-python/pull/1516)) +- Added support for Python 3.11.7 and 3.12.1. ([#1517](https://github.com/heroku/heroku-buildpack-python/pull/1517) and [#1518](https://github.com/heroku/heroku-buildpack-python/pull/1518)) - Added a deprecation warning for Python 3.8. ([#1515](https://github.com/heroku/heroku-buildpack-python/pull/1515)) ## [v240] - 2023-11-30 @@ -1443,7 +1443,7 @@ Updated pip and Setuptools. Added warnings for lack of Procfile. -## v72 - 2015-12-07 +## v73 - 2015-12-07 Updated default Python to 2.7.11. diff --git a/bin/compile b/bin/compile index 966c032a6..1009f6ccf 100755 --- a/bin/compile +++ b/bin/compile @@ -3,7 +3,7 @@ # See: https://devcenter.heroku.com/articles/buildpack-api # We use `errtrace` and `inherit_errexit` to ensure the ERR trap and exit on error behaviour -# propagates to buildpack functions run in subshells, such when using command substitutions. +# propagates to buildpack functions run in subshells, such as when using command substitutions. set -eu -o pipefail -o errtrace shopt -s inherit_errexit @@ -56,7 +56,6 @@ export BUILD_DIR CACHE_DIR ENV_DIR unset PYTHONHOME PYTHONPATH # Import the warnings script, which contains the `pip install` user warning mechanisms -# (mentioned and explained above) source "${BUILDPACK_DIR}/bin/warnings" # Make the directory in which we will create symlinks from the temporary build directory @@ -76,7 +75,7 @@ python_home='/app/.heroku/python' # using `/app` paths which will still work at run-time after relocation. Amongst other things, this # ensures that the shebang lines in the entrypoint scripts of installed packages are correct. export PATH="/app/.heroku/python/bin:${PATH}" -# Tell Python to not buffer it's stdin/stdout. +# Tell Python to not buffer its stdin/stdout. export PYTHONUNBUFFERED=1 # Ensure Python uses a Unicode locale, to prevent the issues described in: # https://github.com/docker-library/python/pull/570 @@ -266,7 +265,7 @@ EOT # # In addition to ensuring dependencies work for subsequent buildpacks and at runtime, they must also # work for cached rebuilds. Most package managers will reinstall editable dependencies regardless on -# next install, which means we can avoid having to rewrite paths on cache restore from the old build +# the next install, which means we can avoid having to rewrite paths on cache restore from the old build # directory to the new location (`/tmp/build_`). However, Pipenv has a bug when using # PEP660 style editable VCS dependencies where it won't reinstall if it's missing (or in our case, the # path has changed), which means we must make sure that VCS src paths stored in the cache do use the diff --git a/builds/build_python_runtime.sh b/builds/build_python_runtime.sh index 6e0882b58..c19f155bb 100755 --- a/builds/build_python_runtime.sh +++ b/builds/build_python_runtime.sh @@ -104,7 +104,7 @@ CONFIGURE_OPTS=( # and not the static library (such as `pycall.rb` or `PyO3`). # - More consistent with the official Python Docker images and other distributions. # - # Shared builds are slower unless `no-semantic-interposition`and LTO is used, + # Shared builds are slower unless `no-semantic-interposition` and LTO are used, # however, as of Python 3.10 `no-semantic-interposition` is enabled by default: # https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup # https://github.com/python/cpython/issues/83161 diff --git a/builds/test_python_runtime.sh b/builds/test_python_runtime.sh index a1ddc92a8..36c3d114f 100755 --- a/builds/test_python_runtime.sh +++ b/builds/test_python_runtime.sh @@ -88,7 +88,7 @@ if "${INSTALL_DIR}/bin/python3" -c "import $( IFS=, echo "${optional_stdlib_modules[*]}" )"; then - echo "Successful imported: ${optional_stdlib_modules[*]}" + echo "Successfully imported: ${optional_stdlib_modules[*]}" else abort "The above optional stdlib module failed to import! Check the compile logs to see if it was skipped due to missing libraries/headers." fi diff --git a/lib/cache.sh b/lib/cache.sh index dae064b53..f2103a069 100644 --- a/lib/cache.sh +++ b/lib/cache.sh @@ -181,7 +181,7 @@ function cache::save() { # but not Heroku CI or build-in-app-dir. Ideally we would be able to use `--reflink=auto` here # (which would avoid the need for a conditional and also mean accidental edits by users in later # buildpacks to one location doesn't affect the other), however, with the current filesystems - # used in production benchmarking showed `--reflinks=auto` was much slower than hardlinks. + # used in production benchmarking showed `--reflink=auto` was much slower than hardlinks. if [[ "${build_dir_filesystem}" == "${cache_dir_filesystem}" ]]; then local additional_copy_args=(--link) else diff --git a/lib/hooks.sh b/lib/hooks.sh index 68c002774..b99dc4950 100644 --- a/lib/hooks.sh +++ b/lib/hooks.sh @@ -4,7 +4,7 @@ # however, it helps Shellcheck realise the options under which these functions will run. set -euo pipefail -# Used to run the `bin/pre_compile` and `bin/post_compile`s scripts if found in the app source, +# Used to run the `bin/pre_compile` and `bin/post_compile` scripts if found in the app source, # allowing for build customisation. function hooks::run_hook() { local hook_name="${1}" diff --git a/lib/python_version.sh b/lib/python_version.sh index a0a778320..6c463f0e9 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -128,7 +128,7 @@ function python_version::read_requested_python_version() { fi fi - # Protect against unsupported (eg PyPy) or invalid versions being found in the cache metadata. + # Protect against unsupported (e.g. PyPy) or invalid versions being found in the cache metadata. if [[ "${cached_python_full_version}" =~ ^${PYTHON_FULL_VERSION_REGEX}$ ]]; then local cached_python_major_version="${cached_python_full_version%.*}" version="${cached_python_major_version}" diff --git a/lib/utils.sh b/lib/utils.sh index 904532768..4a05e2897 100644 --- a/lib/utils.sh +++ b/lib/utils.sh @@ -27,7 +27,7 @@ function utils::bundled_pip_module_path() { # We have to use a glob since the bundled wheel filename contains the pip version, which differs # between Python versions. We use compgen to avoid having to set nullglob, since there may be no # matches in the case of a broken Python install. We also have to handle the case where there are - # multiple matching pip wheels, since in some versions of Python (eg 3.9.0) multiple versions of + # multiple matching pip wheels, since in some versions of Python (e.g. 3.9.0) multiple versions of # pip were accidentally bundled upstream (we use tail since we want the newest pip version). if bundled_pip_wheel="$(compgen -G "${bundled_wheels_dir}/pip-*.whl" | tail --lines=1)"; then # The pip module exists inside the pip wheel (which is a zip file), however, Python can load diff --git a/lib/uv.sh b/lib/uv.sh index a3bc2d2bb..cdbef4acb 100644 --- a/lib/uv.sh +++ b/lib/uv.sh @@ -25,7 +25,7 @@ function uv::install_uv() { mkdir -p "${uv_dir}" local gnu_arch - # eg: `x86_64` or `aarch64`. + # e.g.: `x86_64` or `aarch64`. gnu_arch=$(arch) local uv_url="https://releases.astral.sh/github/uv/releases/download/${UV_VERSION}/uv-${gnu_arch}-unknown-linux-gnu.tar.gz" diff --git a/spec/fixtures/uv_editable/bin/test-entrypoints.sh b/spec/fixtures/uv_editable/bin/test-entrypoints.sh index 20085b398..2d7fab200 100755 --- a/spec/fixtures/uv_editable/bin/test-entrypoints.sh +++ b/spec/fixtures/uv_editable/bin/test-entrypoints.sh @@ -4,7 +4,7 @@ set -euo pipefail cd .heroku/python/lib/python*/site-packages/ -# List any path like strings in the .pth, and finder files in site-packages. +# List any path like strings in the .pth and finder files in site-packages. grep --extended-regexp --only-matching -- '/\S+' *.pth __editable___*_finder.py | sort echo diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index e0b46a839..c451b1d35 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -213,7 +213,7 @@ # TODO: Delete this test once pipenv_mismatched_python_version is re-enabled, # since they mostly duplicate each other. - context 'when there is a both a Pipfile.lock python_version and a .python-version file' do + context 'when there is both a Pipfile.lock python_version and a .python-version file' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_version_and_python_version_file') } it 'builds with the Python version from the .python-version file' do @@ -558,7 +558,7 @@ end # This tests that Pipenv doesn't fall back to system Python if the Python version in - # pyproject.toml doesn't match that in Pipfile / Pipfile.lock. It also tests which of + # .python-version doesn't match that in Pipfile / Pipfile.lock. It also tests which of # .python-version and Pipfile.lock take precedence for the installed Python version. context 'when python_version in Pipfile.lock is incompatible with .python-version' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_mismatched_python_version', allow_failure: true) } diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 0f4a13350..e779fffbc 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -215,7 +215,7 @@ # Python version than the version supported by this buildpack version. let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_non_existent_major_cached', allow_failure: true) } - it 'aborts the build with an EOL message' do + it 'aborts the build with an unrecognised Python version message' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected @@ -311,7 +311,7 @@ context 'when .python-version is misspelled' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_file_misspelled', allow_failure: true) } - it 'aborts the build with an invalid Python version message' do + it 'aborts the build with a misspelled .python-version file message' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected @@ -486,10 +486,10 @@ end end - context 'when .python-version contains an non-existent Python major version' do + context 'when .python-version contains a non-existent Python major version' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_non_existent_major', allow_failure: true) } - it 'aborts the build with an invalid .python-version message' do + it 'aborts the build with an unrecognised Python version message' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 1fed97721..bb429f7e9 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -49,7 +49,7 @@ def get_requirement_version(package_name) config.filter_run_when_matching :focus # Allows declaring on which stacks a test/group should run by tagging it with `stacks`. config.filter_run_excluding stacks: ->(stacks) { !stacks.include?(ENV.fetch('HATCHET_DEFAULT_STACK')) } - # Make rspec-retry output a retry message when its had to retry a test. + # Make rspec-retry output a retry message when it's had to retry a test. config.verbose_retry = true end From 255d0a8cb677c3b37368918521cdf26c824f024e Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 7 Apr 2026 23:55:48 +0100 Subject: [PATCH 363/406] Add support for Python 3.14.4 and 3.13.13 (#2071) Release announcement: https://blog.python.org/2026/04/python-3150a8-3144-31313/ Changelogs: https://docs.python.org/release/3.14.4/whatsnew/changelog.html#python-3-14-4-final https://docs.python.org/release/3.13.13/whatsnew/changelog.html#python-3-13-13-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/24106929682 https://github.com/heroku/heroku-buildpack-python/actions/runs/24106935699 GUS-W-20590498. GUS-W-20590547. --- CHANGELOG.md | 2 ++ lib/python_version.sh | 4 ++-- spec/spec_helper.rb | 4 ++-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 52958b1b8..15eef3a3a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## [Unreleased] +- The Python 3.14 version alias now resolves to Python 3.14.4. ([#2071](https://github.com/heroku/heroku-buildpack-python/pull/2071)) +- The Python 3.13 version alias now resolves to Python 3.13.13. ([#2071](https://github.com/heroku/heroku-buildpack-python/pull/2071)) ## [v340] - 2026-04-02 diff --git a/lib/python_version.sh b/lib/python_version.sh index 6c463f0e9..695a76350 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -7,8 +7,8 @@ set -euo pipefail LATEST_PYTHON_3_10="3.10.20" LATEST_PYTHON_3_11="3.11.15" LATEST_PYTHON_3_12="3.12.13" -LATEST_PYTHON_3_13="3.13.12" -LATEST_PYTHON_3_14="3.14.3" +LATEST_PYTHON_3_13="3.13.13" +LATEST_PYTHON_3_14="3.14.4" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=10 NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=14 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index bb429f7e9..25ea11c6d 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -13,8 +13,8 @@ LATEST_PYTHON_3_10 = '3.10.20' LATEST_PYTHON_3_11 = '3.11.15' LATEST_PYTHON_3_12 = '3.12.13' -LATEST_PYTHON_3_13 = '3.13.12' -LATEST_PYTHON_3_14 = '3.14.3' +LATEST_PYTHON_3_13 = '3.13.13' +LATEST_PYTHON_3_14 = '3.14.4' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_14 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From 240f6b82e5f17f7a6ec95910e963a4b5712dc1f2 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Tue, 7 Apr 2026 23:00:25 +0000 Subject: [PATCH 364/406] Prepare release v341 (#2072) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 15eef3a3a..fc63d204e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v341] - 2026-04-07 + - The Python 3.14 version alias now resolves to Python 3.14.4. ([#2071](https://github.com/heroku/heroku-buildpack-python/pull/2071)) - The Python 3.13 version alias now resolves to Python 3.13.13. ([#2071](https://github.com/heroku/heroku-buildpack-python/pull/2071)) @@ -1510,7 +1513,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v340...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v341...main +[v341]: https://github.com/heroku/heroku-buildpack-python/compare/v340...v341 [v340]: https://github.com/heroku/heroku-buildpack-python/compare/v339...v340 [v339]: https://github.com/heroku/heroku-buildpack-python/compare/v338...v339 [v338]: https://github.com/heroku/heroku-buildpack-python/compare/v337...v338 From ffd66f371d50e94352e7abd387237b385d98a13b Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 9 Apr 2026 23:19:18 +0100 Subject: [PATCH 365/406] Add buildpack support for Heroku-26 (#2073) Following on from #2056 (which added support for building and releasing Python binaries for Heroku-26), this adds support for the new stack to the buildpack itself. Note: Since Ubuntu 26.04 isn't GA yet, Heroku-26 is currently only available on Heroku for use by internal users. GUS-W-20775616. --- .github/workflows/ci.yml | 2 +- CHANGELOG.md | 1 + Makefile | 2 +- lib/checks.sh | 2 +- .../python_version_outdated/.python-version | 2 +- spec/hatchet/pipenv_spec.rb | 5 ++++- spec/hatchet/poetry_spec.rb | 5 ++++- spec/hatchet/python_version_spec.rb | 22 +++++++++++-------- spec/hatchet/stack_spec.rb | 16 +++++++------- spec/hatchet/uv_spec.rb | 5 ++++- spec/spec_helper.rb | 2 +- 11 files changed, 39 insertions(+), 25 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f92ac8a18..8826ce8e3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -39,7 +39,7 @@ jobs: strategy: fail-fast: false matrix: - stack: ["heroku-22", "heroku-24"] + stack: ["heroku-22", "heroku-24", "heroku-26"] env: HATCHET_APP_LIMIT: 300 HATCHET_DEFAULT_STACK: ${{ matrix.stack }} diff --git a/CHANGELOG.md b/CHANGELOG.md index fc63d204e..c68979af4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Added support for Heroku-26. ([#2073](https://github.com/heroku/heroku-buildpack-python/pull/2073)) ## [v341] - 2026-04-07 diff --git a/Makefile b/Makefile index ea5971da4..f1afaf93d 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ # These targets are not files .PHONY: lint lint-scripts lint-ruby check-format format run publish -STACK ?= heroku-24 +STACK ?= heroku-26 FIXTURE ?= spec/fixtures/python_version_unspecified # Allow overriding the exit code in CI, so we can test bin/report works for failing builds. COMPILE_FAILURE_EXIT_CODE ?= 1 diff --git a/lib/checks.sh b/lib/checks.sh index e6f222954..5e9e4b65e 100644 --- a/lib/checks.sh +++ b/lib/checks.sh @@ -4,7 +4,7 @@ function checks::ensure_supported_stack() { local stack="${1}" case "${stack}" in - heroku-22 | heroku-24) + heroku-22 | heroku-24 | heroku-26) return 0 ;; cedar* | heroku-16 | heroku-18 | heroku-20) diff --git a/spec/fixtures/python_version_outdated/.python-version b/spec/fixtures/python_version_outdated/.python-version index 3e388a4ac..f982feb41 100644 --- a/spec/fixtures/python_version_outdated/.python-version +++ b/spec/fixtures/python_version_outdated/.python-version @@ -1 +1 @@ -3.13.2 +3.14.0 diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index c451b1d35..2ee6cb83b 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -455,7 +455,10 @@ end end - context 'when the Pipenv and Python versions have changed since the last build' do + # TODO: Enable on Heroku-26 after the Pipenv and default Python versions next change, + # since for now there isn't a historic buildpack version we can use in this test whose + # stack check permits Heroku-26 and that also uses older Pipenv/Python versions. + context 'when the Pipenv and Python versions have changed since the last build', stacks: %w[heroku-22 heroku-24] do let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v313'] } let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_basic', buildpacks:) } diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 74832d7f1..25aad1ada 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -131,7 +131,10 @@ end end - context 'when the Poetry and Python versions have changed since the last build' do + # TODO: Enable on Heroku-26 after the Poetry and default Python versions next change, + # since for now there isn't a historic buildpack version we can use in this test whose + # stack check permits Heroku-26 and that also uses older Poetry/Python versions. + context 'when the Poetry and Python versions have changed since the last build', stacks: %w[heroku-22 heroku-24] do let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v313'] } let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_basic', buildpacks:) } diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index e779fffbc..93db7cda9 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -88,17 +88,21 @@ end end - context 'with an app last built using an older default Python version' do + # TODO: Enable on Heroku-26 after the default Python version next changes (for the 3.14.5 + # release), since for now there isn't a historic buildpack version we can use in this test + # whose stack check permits Heroku-26 and also has a different default Python version so + # that we can test the sticky versions feature. + context 'with an app last built using an older default Python version', stacks: %w[heroku-22 heroku-24] do # This test performs an initial build using an older buildpack version, followed # by a build using the current version. This ensures that: # - The current buildpack can successfully read the version metadata # written to the build cache by older buildpack versions. # - If no Python version is specified, the same major version as the - # last build is used (sticky versioning). + # last build is used (sticky major versioning). # - Changes in the pip version are handled correctly. let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v267'] } - it 'builds with the same Python version as the last build' do + it 'builds with the same major Python version as the last build' do app.deploy do |app| update_buildpacks(app, [:default]) app.commit! @@ -566,12 +570,12 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python 3.13.2 specified in .python-version + remote: -----> Using Python 3.14.0 specified in .python-version remote: remote: ! Warning: A Python patch update is available! remote: ! - remote: ! Your app is using Python 3.13.2, however, there is a newer - remote: ! patch release of Python 3.13 available: #{LATEST_PYTHON_3_13} + remote: ! Your app is using Python 3.14.0, however, there is a newer + remote: ! patch release of Python 3.14 available: #{LATEST_PYTHON_3_14} remote: ! remote: ! It is important to always use the latest patch version of remote: ! Python to keep your app secure. @@ -579,12 +583,12 @@ remote: ! Update your .python-version file to use the new version. remote: ! remote: ! We strongly recommend that you don't pin your app to an - remote: ! exact Python version such as 3.13.2, and instead only specify - remote: ! the major Python version of 3.13 in your .python-version file. + remote: ! exact Python version such as 3.14.0, and instead only specify + remote: ! the major Python version of 3.14 in your .python-version file. remote: ! This will allow your app to receive the latest available Python remote: ! patch version automatically and prevent this warning. remote: - remote: -----> Installing Python 3.13.2 + remote: -----> Installing Python 3.14.0 remote: -----> Installing pip #{PIP_VERSION} OUTPUT end diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index 8fe115496..0aebe7179 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -3,7 +3,7 @@ require_relative '../spec_helper' RSpec.describe 'Stack changes' do - context 'when the stack is upgraded from Heroku-22 to Heroku-24', stacks: %w[heroku-22] do + context 'when the stack is upgraded from Heroku-24 to Heroku-26', stacks: %w[heroku-24] do # This test performs an initial build using an older buildpack version, followed by a build # using the current version. This ensures that the current buildpack can successfully read # the stack metadata written to the build cache in the past. The buildpack version chosen is @@ -14,8 +14,8 @@ it 'clears the cache before installing again whilst preserving the sticky Python version' do app.deploy do |app| - expect(app.output).to include('Building on the Heroku-22 stack') - app.update_stack('heroku-24') + expect(app.output).to include('Building on the Heroku-24 stack') + app.update_stack('heroku-26') update_buildpacks(app, [:default]) app.commit! app.push! @@ -64,7 +64,7 @@ remote: ! file and this warning will be made an error. remote: remote: -----> Discarding cache since: - remote: - The stack has changed from heroku-22 to heroku-24 + remote: - The stack has changed from heroku-24 to heroku-26 remote: - The Python version has changed from 3.12.3 to #{LATEST_PYTHON_3_12} remote: - The buildpack cache format has changed remote: - The legacy SQLite3 headers and CLI binary need to be uninstalled @@ -77,20 +77,20 @@ end end - context 'when the stack is downgraded from Heroku-24 to Heroku-22', stacks: %w[heroku-24] do + context 'when the stack is downgraded from Heroku-26 to Heroku-24', stacks: %w[heroku-26] do let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.14') } it 'clears the cache before installing again' do app.deploy do |app| - expect(app.output).to include('Building on the Heroku-24 stack') - app.update_stack('heroku-22') + expect(app.output).to include('Building on the Heroku-26 stack') + app.update_stack('heroku-24') app.commit! app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected remote: -----> Using Python 3.14 specified in .python-version remote: -----> Discarding cache since: - remote: - The stack has changed from heroku-24 to heroku-22 + remote: - The stack has changed from heroku-26 to heroku-24 remote: -----> Installing Python #{LATEST_PYTHON_3_14} remote: -----> Installing pip #{PIP_VERSION} remote: -----> Installing dependencies using 'pip install -r requirements.txt' diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 9eedb5af6..716afee90 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -128,7 +128,10 @@ end end - context 'when the uv and Python versions have changed since the last build' do + # TODO: Enable on Heroku-26 after the uv and default Python versions next change, + # since for now there isn't a historic buildpack version we can use in this test + # whose stack check permits Heroku-26 and that also uses older uv/Python versions. + context 'when the uv and Python versions have changed since the last build', stacks: %w[heroku-22 heroku-24] do let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v313'] } let(:app) { Hatchet::Runner.new('spec/fixtures/uv_basic', buildpacks:) } diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 25ea11c6d..617c73d04 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -1,7 +1,7 @@ # frozen_string_literal: true ENV['HATCHET_BUILDPACK_BASE'] ||= 'https://github.com/heroku/heroku-buildpack-python.git' -ENV['HATCHET_DEFAULT_STACK'] ||= 'heroku-24' +ENV['HATCHET_DEFAULT_STACK'] ||= 'heroku-26' require 'English' # for $CHILD_STATUS require 'rspec/core' From b609ad5042ed099b7acd42d7f84ac2122436cc2c Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 9 Apr 2026 22:24:22 +0000 Subject: [PATCH 366/406] Prepare release v342 (#2074) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c68979af4..3df91dd9a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v342] - 2026-04-09 + - Added support for Heroku-26. ([#2073](https://github.com/heroku/heroku-buildpack-python/pull/2073)) ## [v341] - 2026-04-07 @@ -1514,7 +1517,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v341...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v342...main +[v342]: https://github.com/heroku/heroku-buildpack-python/compare/v341...v342 [v341]: https://github.com/heroku/heroku-buildpack-python/compare/v340...v341 [v340]: https://github.com/heroku/heroku-buildpack-python/compare/v339...v340 [v339]: https://github.com/heroku/heroku-buildpack-python/compare/v338...v339 From 62fb8545c340a3e0a5007ea61513235101bf0c4c Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 10 Apr 2026 13:28:27 +0100 Subject: [PATCH 367/406] Fix outdated NLTK `sub_env` comment (#2075) Since the whole script is no longer run via `sub_env` since: https://github.com/heroku/heroku-buildpack-python/pull/1888 --- bin/steps/nltk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/steps/nltk b/bin/steps/nltk index 0e72cdd23..67040d103 100755 --- a/bin/steps/nltk +++ b/bin/steps/nltk @@ -1,6 +1,6 @@ #!/usr/bin/env bash -# This script is run in a subshell via sub_env so doesn't inherit the options/vars/utils from `bin/compile`. +# This script is run in a subshell so doesn't inherit the options/vars/utils from `bin/compile`. # TODO: Migrate this script to functions under `lib/` and stop running the entire script in a subshell. set -euo pipefail BUILDPACK_DIR=$(cd "$(dirname "$(dirname "$(dirname "${BASH_SOURCE[0]}")")")" && pwd) From ab1d03f9f7aab4c5651c5ab4616cdf8b37048a90 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 10 Apr 2026 13:30:10 +0100 Subject: [PATCH 368/406] Stop using global variable in `warn_or_error_if_python_version_file_missing()` (#2076) Longer term I'll refactor `bin/compile` to reduce the number of globals set, to prevent accidentally relying on globals in the library functions. --- bin/compile | 2 +- lib/python_version.sh | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/bin/compile b/bin/compile index 1009f6ccf..90cba823d 100755 --- a/bin/compile +++ b/bin/compile @@ -139,7 +139,7 @@ fi # as to what was a warning vs an error. In addition, several of the error messages contain similar # content to the warnings (such as encouraging use of .python-version and major version syntax), # which would mean duplicate content if we showed both. -python_version::warn_or_error_if_python_version_file_missing "${python_version_origin}" "${python_major_version}" +python_version::warn_or_error_if_python_version_file_missing "${python_version_origin}" "${python_major_version}" "${package_manager}" python_version::warn_if_deprecated_major_version "${python_major_version}" "${python_version_origin}" python_version::warn_if_patch_update_available "${python_full_version}" "${python_major_version}" "${python_version_origin}" diff --git a/lib/python_version.sh b/lib/python_version.sh index 695a76350..715ce1fa1 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -539,6 +539,7 @@ function python_version::resolve_python_version() { function python_version::warn_or_error_if_python_version_file_missing() { local python_version_origin="${1}" local python_major_version="${2}" + local package_manager="${3}" if [[ "${python_version_origin}" == ".python-version" ]]; then return 0 From 81237145bcd5c866c8c33c9194a5255c55d064ad Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 10 Apr 2026 13:40:35 +0100 Subject: [PATCH 369/406] Fix `build_data::get_previous()` discarding stored `false` values (#2077) The jq `//` (alternative) operator treats both `null` and `false` as falsy, so a stored boolean `false` would return empty string instead of `"false"`. Fixed by replacing with an explicit `has()` check. Though in practice, no callers of `get_previously()` retrieve boolean fields, so currently nothing was affected by this. Also use `--arg` for the jq key parameter in both `build_data::has()` and `build_data::get_previous()`, instead of string interpolation, for consistency with `build_data::_set()`, and to avoid issues if the key name contained any special characters (though in practice they shouldn't given the names aren't user controlled). GUS-W-21977968. GUS-W-21977978. --- lib/build_data.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/build_data.sh b/lib/build_data.sh index 93434ac38..856bb3df3 100644 --- a/lib/build_data.sh +++ b/lib/build_data.sh @@ -135,7 +135,7 @@ function build_data::_set() { function build_data::has() { local key="${1}" - jq --exit-status "has(\"${key}\")" "${BUILD_DATA_FILE}" >/dev/null + jq --exit-status --arg key "${key}" 'has($key)' "${BUILD_DATA_FILE}" >/dev/null } # Retrieve the value of an entry in the build data store from the previous successful build. @@ -158,9 +158,9 @@ function build_data::get_previous() { # last matching entry in the file. The empty string is returned if the key wasn't found. tac "${LEGACY_BUILD_DATA_FILE}" | { grep --perl-regexp --only-matching --max-count=1 "^${key}=\K.*$" || true; } elif [[ -f "${PREVIOUS_BUILD_DATA_FILE}" ]]; then - # The `// empty` ensures we return the empty string rather than `null` if the key doesn't exist. + # By default jq will return `null` if the key isn't found, so we must handle this explicitly. # We don't use `--exit-status` since `false` is a valid value for us to retrieve. - jq --raw-output ".${key} // empty" "${PREVIOUS_BUILD_DATA_FILE}" + jq --raw-output --arg key "${key}" 'if has($key) then .[$key] else empty end' "${PREVIOUS_BUILD_DATA_FILE}" fi } From 065e162e76754a51154a1b39d946a92b46986bcd Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 13 Apr 2026 10:54:38 +0000 Subject: [PATCH 370/406] Bump uv from 0.11.3 to 0.11.6 in /requirements (#2078) * Bump uv from 0.11.3 to 0.11.6 in /requirements Bumps [uv](https://github.com/astral-sh/uv) from 0.11.3 to 0.11.6. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.11.3...0.11.6) --- updated-dependencies: - dependency-name: uv dependency-version: 0.11.6 dependency-type: direct:production ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3df91dd9a..2fa1c951c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.11.3 to 0.11.6. ([#2078](https://github.com/heroku/heroku-buildpack-python/pull/2078)) ## [v342] - 2026-04-09 diff --git a/requirements/uv.txt b/requirements/uv.txt index 0aaa7b861..3dd4ad9bd 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.11.3 +uv==0.11.6 From 7990ed75f6891744aeea228ed69f144a01b1c308 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 13 Apr 2026 11:07:00 +0000 Subject: [PATCH 371/406] Bump poetry from 2.3.3 to 2.3.4 (#2079) * Bump poetry from 2.3.3 to 2.3.4 Bumps [poetry](https://github.com/python-poetry/poetry) from 2.3.3 to 2.3.4. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](https://github.com/python-poetry/poetry/compare/2.3.3...2.3.4) --- updated-dependencies: - dependency-name: poetry dependency-version: 2.3.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/poetry.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2fa1c951c..5c9119d9e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated Poetry from 2.3.3 to 2.3.4. ([#2079](https://github.com/heroku/heroku-buildpack-python/pull/2079)) - Updated uv from 0.11.3 to 0.11.6. ([#2078](https://github.com/heroku/heroku-buildpack-python/pull/2078)) ## [v342] - 2026-04-09 diff --git a/requirements/poetry.txt b/requirements/poetry.txt index 32a21352c..eb883feb4 100644 --- a/requirements/poetry.txt +++ b/requirements/poetry.txt @@ -1 +1 @@ -poetry==2.3.3 +poetry==2.3.4 From 36b45f7b6c6e3f185bcbd97eecb6d58c364d74be Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 13 Apr 2026 13:03:15 +0100 Subject: [PATCH 372/406] Bump pipenv from 2026.5.1 to 2026.5.2 (#2080) * Bump pipenv from 2026.5.1 to 2026.5.2 Bumps [pipenv](https://github.com/pypa/pipenv) from 2026.5.1 to 2026.5.2. - [Release notes](https://github.com/pypa/pipenv/releases) - [Changelog](https://github.com/pypa/pipenv/blob/main/CHANGELOG.md) - [Commits](https://github.com/pypa/pipenv/compare/v2026.5.1...v2026.5.2) --- updated-dependencies: - dependency-name: pipenv dependency-version: 2026.5.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/pipenv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5c9119d9e..386539290 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated Pipenv from 2026.5.1 to 2026.5.2. ([#2080](https://github.com/heroku/heroku-buildpack-python/pull/2080)) - Updated Poetry from 2.3.3 to 2.3.4. ([#2079](https://github.com/heroku/heroku-buildpack-python/pull/2079)) - Updated uv from 0.11.3 to 0.11.6. ([#2078](https://github.com/heroku/heroku-buildpack-python/pull/2078)) diff --git a/requirements/pipenv.txt b/requirements/pipenv.txt index 69a9727aa..269dce251 100644 --- a/requirements/pipenv.txt +++ b/requirements/pipenv.txt @@ -1 +1 @@ -pipenv==2026.5.1 +pipenv==2026.5.2 From b8216105773cb602c576eef22d6afe8200b507e1 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Mon, 13 Apr 2026 12:07:12 +0000 Subject: [PATCH 373/406] Prepare release v343 (#2081) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 386539290..752d33687 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v343] - 2026-04-13 + - Updated Pipenv from 2026.5.1 to 2026.5.2. ([#2080](https://github.com/heroku/heroku-buildpack-python/pull/2080)) - Updated Poetry from 2.3.3 to 2.3.4. ([#2079](https://github.com/heroku/heroku-buildpack-python/pull/2079)) - Updated uv from 0.11.3 to 0.11.6. ([#2078](https://github.com/heroku/heroku-buildpack-python/pull/2078)) @@ -1520,7 +1523,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v342...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v343...main +[v343]: https://github.com/heroku/heroku-buildpack-python/compare/v342...v343 [v342]: https://github.com/heroku/heroku-buildpack-python/compare/v341...v342 [v341]: https://github.com/heroku/heroku-buildpack-python/compare/v340...v341 [v340]: https://github.com/heroku/heroku-buildpack-python/compare/v339...v340 From 88cd62e49856d977041bb0622d55c65a8e14df09 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 7 May 2026 13:53:09 +0100 Subject: [PATCH 374/406] Update Pipenv editable test assertion (#2087) Since the filename of the Pipenv `.pth` file has changed after the release of `editables` 0.6 on 2026-04-14: https://github.com/pfmoore/editables/blob/main/CHANGELOG.md#release-06 https://github.com/pfmoore/editables/commit/08ac62831524a2a8161e5635248a14e7ccf8a36d ...which was causing CI on `main` to fail, eg: https://github.com/heroku/heroku-buildpack-python/actions/runs/25225985308/job/74807485584?pr=2086#step:5:629 GUS-W-22398957. --- spec/hatchet/pipenv_spec.rb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 2ee6cb83b..7bdaacb69 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -493,9 +493,9 @@ remote: -----> Installing dependencies using 'pipenv install --deploy' remote: -----> Running bin/post_compile hook remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} + remote: _editable_impl_pipenv_editable.pth:/tmp/build_.+ remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} - remote: _pipenv_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the current package: Hello from pipenv-editable! remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! @@ -505,9 +505,9 @@ .+ remote: -----> Inline app detected remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} + remote: _editable_impl_pipenv_editable.pth:/tmp/build_.+ remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} - remote: _pipenv_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the current package: Hello from pipenv-editable! remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! @@ -518,9 +518,9 @@ # Test rewritten paths work at runtime. expect(app.run('bin/test-entrypoints.sh')).to include(<<~OUTPUT) __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} + _editable_impl_pipenv_editable.pth:/app __editable___local_package_pyproject_toml_0_0_1_finder.py:/app/packages/local_package_pyproject_toml/local_package_pyproject_toml'} __editable___local_package_setup_py_0_0_1_finder.py:/app/packages/local_package_setup_py/local_package_setup_py'} - _pipenv_editable.pth:/app Running entrypoint for the current package: Hello from pipenv-editable! Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! @@ -535,9 +535,9 @@ remote: -----> Installing dependencies using 'pipenv install --deploy' remote: -----> Running bin/post_compile hook remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} + remote: _editable_impl_pipenv_editable.pth:/tmp/build_.+ remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} - remote: _pipenv_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the current package: Hello from pipenv-editable! remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! @@ -547,9 +547,9 @@ .+ remote: -----> Inline app detected remote: __editable___gunicorn_23_0_0_finder.py:/app/.heroku/python/src/gunicorn/gunicorn'} + remote: _editable_impl_pipenv_editable.pth:/tmp/build_.+ remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'} remote: __editable___local_package_setup_py_0_0_1_finder.py:/tmp/build_.+/packages/local_package_setup_py/local_package_setup_py'} - remote: _pipenv_editable.pth:/tmp/build_.+ remote: remote: Running entrypoint for the current package: Hello from pipenv-editable! remote: Running entrypoint for the pyproject.toml-based local package: Hello from pyproject.toml! From 14acb968a2b8c2427b336a8a200b79b0891f2a9f Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 7 May 2026 14:03:45 +0100 Subject: [PATCH 375/406] Bump sigstore/cosign/cosign from v3.0.5 to v3.0.6 in /builds (#2086) Bumps sigstore/cosign/cosign from v3.0.5 to v3.0.6. --- updated-dependencies: - dependency-name: sigstore/cosign/cosign dependency-version: v3.0.6 dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- builds/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/builds/Dockerfile b/builds/Dockerfile index 44bfb88c8..5a17355ce 100644 --- a/builds/Dockerfile +++ b/builds/Dockerfile @@ -1,5 +1,5 @@ ARG STACK_VERSION="24" -FROM ghcr.io/sigstore/cosign/cosign:v3.0.5@sha256:be924970ba7438c22e18067dec5637946d6566eac711f5bedd1584e7137008fb AS cosign +FROM ghcr.io/sigstore/cosign/cosign:v3.0.6@sha256:de9c65609e6bde17e6b48de485ee788407c9502fa08b8f4459f595b21f56cd00 AS cosign FROM heroku/heroku:${STACK_VERSION}-build ARG STACK_VERSION From ed0226a19533dc410075ac7e8b051153612a5617 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 7 May 2026 13:07:35 +0000 Subject: [PATCH 376/406] Bump rubocop in the ruby-dependencies group across 1 directory (#2083) Bumps the ruby-dependencies group with 1 update in the / directory: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.86.0 to 1.86.1 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.86.0...v1.86.1) --- updated-dependencies: - dependency-name: rubocop dependency-version: 1.86.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 13b3f38a5..b355c0362 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -20,13 +20,13 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.19.3) + json (2.19.5) language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) moneta (1.0.0) multi_json (1.19.1) - parallel (1.27.0) + parallel (2.1.0) parallel_split_test (0.10.0) parallel (>= 0.5.13) rspec-core (>= 3.9.0) @@ -41,7 +41,7 @@ GEM racc (1.8.1) rainbow (3.1.1) rate_throttle_client (0.1.2) - regexp_parser (2.11.3) + regexp_parser (2.12.0) rrrretry (1.0.0) rspec-core (3.13.6) rspec-support (~> 3.13.0) @@ -51,11 +51,11 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.7) - rubocop (1.86.0) + rubocop (1.86.1) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) - parallel (~> 1.10) + parallel (>= 1.10) parser (>= 3.3.0.2) rainbow (>= 2.2.2, < 4.0) regexp_parser (>= 2.9.3, < 3.0) @@ -98,13 +98,13 @@ CHECKSUMS excon (1.4.0) sha256=5d2bc9d2c79511a562e7fcac77cc7a40acd9cebcc55b80e537975ad8187f2924 heroics (0.1.3) sha256=31d792e8706ecc6f78299f52d0b0f8ab55489a13b5f4f76c3050b07912563562 heroku_hatchet (8.0.6) sha256=886a7d7d686859db9fac19df1194a35c5349285956ca4d134054c5c376accea3 - json (2.19.3) sha256=289b0bb53052a1fa8c34ab33cc750b659ba14a5c45f3fcf4b18762dc67c78646 + json (2.19.5) sha256=218a18553e4801d579ca7e0f5bc72bafd776d7397238a1fb4e74db5b0a812c59 language_server-protocol (3.17.0.5) sha256=fd1e39a51a28bf3eec959379985a72e296e9f9acfce46f6a79d31ca8760803cc lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87 logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203 moneta (1.0.0) sha256=2224e5a68156e8eceb525fb0582c8c4e0f29f67cae86507cdcfb406abbb1fc5d multi_json (1.19.1) sha256=7aefeff8f2c854bf739931a238e4aea64592845e0c0395c8a7d2eea7fdd631b7 - parallel (1.27.0) sha256=4ac151e1806b755fb4e2dc2332cbf0e54f2e24ba821ff2d3dcf86bf6dc4ae130 + parallel (2.1.0) sha256=b35258865c2e31134c5ecb708beaaf6772adf9d5efae28e93e99260877b09356 parallel_split_test (0.10.0) sha256=4abd6bedc6a1b169ecb93a1bf982fa7fbd437b66c360594d56d107c25f53b05f parser (3.3.11.1) sha256=d17ace7aabe3e72c3cc94043714be27cc6f852f104d81aa284c2281aecc65d54 platform-api (3.9.0) sha256=be3b919955c52649fd931a9f62d571bd6f06613de630970dc7255fc95eb7d962 @@ -112,13 +112,13 @@ CHECKSUMS racc (1.8.1) sha256=4a7f6929691dbec8b5209a0b373bc2614882b55fc5d2e447a21aaa691303d62f rainbow (3.1.1) sha256=039491aa3a89f42efa1d6dec2fc4e62ede96eb6acd95e52f1ad581182b79bc6a rate_throttle_client (0.1.2) sha256=f9de968b892fea9272154f6182b4f5cfb74292585e66763fb8a8510181ec83ee - regexp_parser (2.11.3) sha256=ca13f381a173b7a93450e53459075c9b76a10433caadcb2f1180f2c741fc55a4 + regexp_parser (2.12.0) sha256=35a916a1d63190ab5c9009457136ae5f3c0c7512d60291d0d1378ba18ce08ebb rrrretry (1.0.0) sha256=3c60784501701a49d8ad499af7e76dbddf9a8be916beffe885bd0f443ad1c749 rspec-core (3.13.6) sha256=a8823c6411667b60a8bca135364351dda34cd55e44ff94c4be4633b37d828b2d rspec-expectations (3.13.5) sha256=33a4d3a1d95060aea4c94e9f237030a8f9eae5615e9bd85718fe3a09e4b58836 rspec-retry (0.6.2) sha256=6101ba23a38809811ae3484acde4ab481c54d846ac66d5037ccb40131a60d858 rspec-support (3.13.7) sha256=0640e5570872aafefd79867901deeeeb40b0c9875a36b983d85f54fb7381c47c - rubocop (1.86.0) sha256=4ff1186fe16ebe9baff5e7aad66bb0ad4cabf5cdcd419f773146dbba2565d186 + rubocop (1.86.1) sha256=44415f3f01d01a21e01132248d2fd0867572475b566ca188a0a42133a08d4531 rubocop-ast (1.49.1) sha256=4412f3ee70f6fe4546cc489548e0f6fcf76cafcfa80fa03af67098ffed755035 rubocop-rspec (3.9.0) sha256=8fa70a3619408237d789aeecfb9beef40576acc855173e60939d63332fdb55e2 ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33 From 0152f96f7a7c091faa0e9a82e588ed0cb7feca5b Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 7 May 2026 13:24:02 +0000 Subject: [PATCH 377/406] Bump uv from 0.11.6 to 0.11.11 (#2090) * Bump uv from 0.11.6 to 0.11.11 Bumps [uv](https://github.com/astral-sh/uv) from 0.11.6 to 0.11.11. - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.11.6...0.11.11) --- updated-dependencies: - dependency-name: uv dependency-version: 0.11.11 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/uv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 752d33687..4fcff9d2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated uv from 0.11.6 to 0.11.11. ([#2090](https://github.com/heroku/heroku-buildpack-python/pull/2090)) ## [v343] - 2026-04-13 diff --git a/requirements/uv.txt b/requirements/uv.txt index 3dd4ad9bd..f5139606e 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.11.6 +uv==0.11.11 From 233b55c7a53d238ec05dd48aa0d8fe328422610e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 7 May 2026 14:45:24 +0100 Subject: [PATCH 378/406] Bump pip from 26.0.1 to 26.1.1 (#2089) * Bump pip from 26.0.1 to 26.1.1 Bumps [pip](https://github.com/pypa/pip) from 26.0.1 to 26.1.1. - [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst) - [Commits](https://github.com/pypa/pip/compare/26.0.1...26.1.1) --- updated-dependencies: - dependency-name: pip dependency-version: 26.1.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/pip.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4fcff9d2a..67b6e570d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Updated pip from 26.0.1 to 26.1.1. ([#2089](https://github.com/heroku/heroku-buildpack-python/pull/2089)) - Updated uv from 0.11.6 to 0.11.11. ([#2090](https://github.com/heroku/heroku-buildpack-python/pull/2090)) ## [v343] - 2026-04-13 diff --git a/requirements/pip.txt b/requirements/pip.txt index c1dd99e28..074276481 100644 --- a/requirements/pip.txt +++ b/requirements/pip.txt @@ -1 +1 @@ -pip==26.0.1 +pip==26.1.1 From 7eb4d6bbc8a48d7760a43aaa69c0f6bc18da0e7a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 7 May 2026 13:53:39 +0000 Subject: [PATCH 379/406] Bump poetry from 2.3.4 to 2.4.0 (#2088) Bumps [poetry](https://github.com/python-poetry/poetry) from 2.3.4 to 2.4.0. - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](https://github.com/python-poetry/poetry/compare/2.3.4...2.4.0) --- updated-dependencies: - dependency-name: poetry dependency-version: 2.4.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/poetry.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 67b6e570d..fceaf175d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Updated pip from 26.0.1 to 26.1.1. ([#2089](https://github.com/heroku/heroku-buildpack-python/pull/2089)) +- Updated Poetry from 2.3.4 to 2.4.0. ([#2088](https://github.com/heroku/heroku-buildpack-python/pull/2088)) - Updated uv from 0.11.6 to 0.11.11. ([#2090](https://github.com/heroku/heroku-buildpack-python/pull/2090)) ## [v343] - 2026-04-13 diff --git a/requirements/poetry.txt b/requirements/poetry.txt index eb883feb4..1d3276616 100644 --- a/requirements/poetry.txt +++ b/requirements/poetry.txt @@ -1 +1 @@ -poetry==2.3.4 +poetry==2.4.0 From 4fa8d49ea4bc926d6ce1a6013918675a081f8326 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 7 May 2026 14:17:43 +0000 Subject: [PATCH 380/406] Bump pipenv from 2026.5.2 to 2026.6.1 (#2082) * Bump pipenv from 2026.5.2 to 2026.6.1 Bumps [pipenv](https://github.com/pypa/pipenv) from 2026.5.2 to 2026.6.1. - [Release notes](https://github.com/pypa/pipenv/releases) - [Changelog](https://github.com/pypa/pipenv/blob/main/CHANGELOG.md) - [Commits](https://github.com/pypa/pipenv/compare/v2026.5.2...v2026.6.1) --- updated-dependencies: - dependency-name: pipenv dependency-version: 2026.6.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Add changelog entry --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 1 + requirements/pipenv.txt | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fceaf175d..8f02962e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] - Updated pip from 26.0.1 to 26.1.1. ([#2089](https://github.com/heroku/heroku-buildpack-python/pull/2089)) +- Updated Pipenv from 2026.5.2 to 2026.6.1. ([#2082](https://github.com/heroku/heroku-buildpack-python/pull/2082)) - Updated Poetry from 2.3.4 to 2.4.0. ([#2088](https://github.com/heroku/heroku-buildpack-python/pull/2088)) - Updated uv from 0.11.6 to 0.11.11. ([#2090](https://github.com/heroku/heroku-buildpack-python/pull/2090)) diff --git a/requirements/pipenv.txt b/requirements/pipenv.txt index 269dce251..b61f78890 100644 --- a/requirements/pipenv.txt +++ b/requirements/pipenv.txt @@ -1 +1 @@ -pipenv==2026.5.2 +pipenv==2026.6.1 From 27b1abbc704fcba556ec45ec1807f42ba900daa7 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 7 May 2026 15:59:07 +0100 Subject: [PATCH 381/406] Group Python Dependabot PRs too (#2091) To reduce the amount of PR rebasing that has to occur each month when we update the Python package manager versions, given that they need CHANGELOG.md entries each, which otherwise cause conflicts. See: https://docs.github.com/en/code-security/reference/supply-chain-security/dependabot-options-reference#groups-- GUS-W-22400243. --- .github/dependabot.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index d3903c048..d2dcc8f44 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -40,3 +40,7 @@ updates: labels: - "dependencies" - "python" + groups: + # We don't limit grouping for Python to just minor/patch versions, since pip doesn't follow + # semver, plus having to update CHANGELOG.md means it's easier to just have a single PR. + python-dependencies: From bb2c6739a24df90e4544e9677b2ca3a9c4c21318 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 7 May 2026 16:20:15 +0100 Subject: [PATCH 382/406] Fix Dependabot config file syntax (#2092) Since the schema requires at least one property to be specified under the `groups` key, even if we're not wanting to add filters. See: https://www.schemastore.org/dependabot-2.0.json GUS-W-22400243. --- .github/dependabot.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index d2dcc8f44..f59c458c9 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -44,3 +44,5 @@ updates: # We don't limit grouping for Python to just minor/patch versions, since pip doesn't follow # semver, plus having to update CHANGELOG.md means it's easier to just have a single PR. python-dependencies: + patterns: + - "*" From b123a446378f6c8295de82fa1058f5f537df6574 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 7 May 2026 16:36:59 +0000 Subject: [PATCH 383/406] Prepare release v344 (#2093) * Prepare release v344 * Fix misc typo --------- Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- lib/pip.sh | 2 +- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8f02962e9..160616a93 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v344] - 2026-05-07 + - Updated pip from 26.0.1 to 26.1.1. ([#2089](https://github.com/heroku/heroku-buildpack-python/pull/2089)) - Updated Pipenv from 2026.5.2 to 2026.6.1. ([#2082](https://github.com/heroku/heroku-buildpack-python/pull/2082)) - Updated Poetry from 2.3.4 to 2.4.0. ([#2088](https://github.com/heroku/heroku-buildpack-python/pull/2088)) @@ -1527,7 +1530,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v343...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v344...main +[v344]: https://github.com/heroku/heroku-buildpack-python/compare/v343...v344 [v343]: https://github.com/heroku/heroku-buildpack-python/compare/v342...v343 [v342]: https://github.com/heroku/heroku-buildpack-python/compare/v341...v342 [v341]: https://github.com/heroku/heroku-buildpack-python/compare/v340...v341 diff --git a/lib/pip.sh b/lib/pip.sh index dc26c39e5..62585f8b4 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -21,7 +21,7 @@ function pip::install_pip() { local packages_display_text="pip ${PIP_VERSION}" # We only install setuptools on Python 3.12 and older, since: - # - pip now uses isolated build environments into which it installed setuptools and wheel + # - pip now uses isolated build environments into which it installs setuptools and wheel # if needed when installing packages from an sdist. # - Most of the Python ecosystem has stopped installing them for Python 3.12+ already. # See the Python CNB's removal for more details: https://github.com/heroku/buildpacks-python/pull/243 From 058ba27a88395e8070c30a5049d5ddedaa52b04a Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 11 May 2026 23:11:28 +0100 Subject: [PATCH 384/406] Add support for Python 3.14.5 (#2094) Release announcement: https://blog.python.org/2026/05/python-3145-is-out/ Changelog: https://docs.python.org/release/3.14.5/whatsnew/changelog.html#python-3-14-5-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/25693905266 GUS-W-22439823. --- CHANGELOG.md | 1 + lib/python_version.sh | 2 +- spec/spec_helper.rb | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 160616a93..3305a744e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- The Python 3.14 version alias now resolves to Python 3.14.5. ([#2094](https://github.com/heroku/heroku-buildpack-python/pull/2094)) ## [v344] - 2026-05-07 diff --git a/lib/python_version.sh b/lib/python_version.sh index 715ce1fa1..2e96549b4 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -8,7 +8,7 @@ LATEST_PYTHON_3_10="3.10.20" LATEST_PYTHON_3_11="3.11.15" LATEST_PYTHON_3_12="3.12.13" LATEST_PYTHON_3_13="3.13.13" -LATEST_PYTHON_3_14="3.14.4" +LATEST_PYTHON_3_14="3.14.5" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=10 NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=14 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 617c73d04..1fcb598ff 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -14,7 +14,7 @@ LATEST_PYTHON_3_11 = '3.11.15' LATEST_PYTHON_3_12 = '3.12.13' LATEST_PYTHON_3_13 = '3.13.13' -LATEST_PYTHON_3_14 = '3.14.4' +LATEST_PYTHON_3_14 = '3.14.5' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_14 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From 1bdfe9b402cf439a605271aa031e9df27ce0e85e Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Mon, 11 May 2026 23:17:06 +0100 Subject: [PATCH 385/406] Prepare release v345 (#2095) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3305a744e..537edc013 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v345] - 2026-05-11 + - The Python 3.14 version alias now resolves to Python 3.14.5. ([#2094](https://github.com/heroku/heroku-buildpack-python/pull/2094)) ## [v344] - 2026-05-07 @@ -1531,7 +1534,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v344...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v345...main +[v345]: https://github.com/heroku/heroku-buildpack-python/compare/v344...v345 [v344]: https://github.com/heroku/heroku-buildpack-python/compare/v343...v344 [v343]: https://github.com/heroku/heroku-buildpack-python/compare/v342...v343 [v342]: https://github.com/heroku/heroku-buildpack-python/compare/v341...v342 From d18722b0ad13e60213fb4a913eace645dd9643b2 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 4 Jun 2026 12:51:56 +0000 Subject: [PATCH 386/406] Bump rubocop from 1.86.1 to 1.87.0 in the ruby-dependencies group (#2097) * Bump rubocop from 1.86.1 to 1.87.0 in the ruby-dependencies group Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.86.1 to 1.87.0 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.86.1...v1.87.0) --- updated-dependencies: - dependency-name: rubocop dependency-version: 1.87.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] * Fix and refresh lockfile --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- Gemfile.lock | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index b355c0362..8fe3f0aeb 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -5,7 +5,7 @@ GEM base64 (0.3.0) diff-lcs (1.6.2) erubis (2.7.0) - excon (1.4.0) + excon (1.5.0) logger heroics (0.1.3) base64 @@ -20,12 +20,12 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.19.5) + json (2.19.8) language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) moneta (1.0.0) - multi_json (1.19.1) + multi_json (1.21.1) parallel (2.1.0) parallel_split_test (0.10.0) parallel (>= 0.5.13) @@ -51,7 +51,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.7) - rubocop (1.86.1) + rubocop (1.87.0) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) @@ -93,17 +93,18 @@ DEPENDENCIES CHECKSUMS ast (2.4.3) sha256=954615157c1d6a382bc27d690d973195e79db7f55e9765ac7c481c60bdb4d383 base64 (0.3.0) sha256=27337aeabad6ffae05c265c450490628ef3ebd4b67be58257393227588f5a97b + bundler (4.0.13) sha256=19f08be7f27022cf0b89f27da0b044ae075e8270a9ef44ad248a932614e1ca3b diff-lcs (1.6.2) sha256=9ae0d2cba7d4df3075fe8cd8602a8604993efc0dfa934cff568969efb1909962 erubis (2.7.0) sha256=63653f5174a7997f6f1d6f465fbe1494dcc4bdab1fb8e635f6216989fb1148ba - excon (1.4.0) sha256=5d2bc9d2c79511a562e7fcac77cc7a40acd9cebcc55b80e537975ad8187f2924 + excon (1.5.0) sha256=c503ad1d0123bc8ab2a062ff3789dc891ec368cb9e13765ab88a9c58c8bb6d50 heroics (0.1.3) sha256=31d792e8706ecc6f78299f52d0b0f8ab55489a13b5f4f76c3050b07912563562 heroku_hatchet (8.0.6) sha256=886a7d7d686859db9fac19df1194a35c5349285956ca4d134054c5c376accea3 - json (2.19.5) sha256=218a18553e4801d579ca7e0f5bc72bafd776d7397238a1fb4e74db5b0a812c59 + json (2.19.8) sha256=6354310fd76ef69b87d5bd1f38b40d730613baf90b6803d2d0a48f618d32dfaa language_server-protocol (3.17.0.5) sha256=fd1e39a51a28bf3eec959379985a72e296e9f9acfce46f6a79d31ca8760803cc lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87 logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203 moneta (1.0.0) sha256=2224e5a68156e8eceb525fb0582c8c4e0f29f67cae86507cdcfb406abbb1fc5d - multi_json (1.19.1) sha256=7aefeff8f2c854bf739931a238e4aea64592845e0c0395c8a7d2eea7fdd631b7 + multi_json (1.21.1) sha256=e6126a31808e3b4d19f483c775ceac34df190dffa62adfb63a165ee14ba68080 parallel (2.1.0) sha256=b35258865c2e31134c5ecb708beaaf6772adf9d5efae28e93e99260877b09356 parallel_split_test (0.10.0) sha256=4abd6bedc6a1b169ecb93a1bf982fa7fbd437b66c360594d56d107c25f53b05f parser (3.3.11.1) sha256=d17ace7aabe3e72c3cc94043714be27cc6f852f104d81aa284c2281aecc65d54 @@ -118,7 +119,7 @@ CHECKSUMS rspec-expectations (3.13.5) sha256=33a4d3a1d95060aea4c94e9f237030a8f9eae5615e9bd85718fe3a09e4b58836 rspec-retry (0.6.2) sha256=6101ba23a38809811ae3484acde4ab481c54d846ac66d5037ccb40131a60d858 rspec-support (3.13.7) sha256=0640e5570872aafefd79867901deeeeb40b0c9875a36b983d85f54fb7381c47c - rubocop (1.86.1) sha256=44415f3f01d01a21e01132248d2fd0867572475b566ca188a0a42133a08d4531 + rubocop (1.87.0) sha256=b9d9ddf55116a513f8ef2c7ae660662d8b49301f118d3f0df61865b33a5c188d rubocop-ast (1.49.1) sha256=4412f3ee70f6fe4546cc489548e0f6fcf76cafcfa80fa03af67098ffed755035 rubocop-rspec (3.9.0) sha256=8fa70a3619408237d789aeecfb9beef40576acc855173e60939d63332fdb55e2 ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33 @@ -129,7 +130,7 @@ CHECKSUMS webrick (1.9.2) sha256=beb4a15fc474defed24a3bda4ffd88a490d517c9e4e6118c3edce59e45864131 RUBY VERSION - ruby 4.0.1 + ruby 4.0.5 BUNDLED WITH - 4.0.7 + 4.0.13 From 6d2b511c178e689109c05c4f1b40c94294d24d1a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 4 Jun 2026 13:33:42 +0000 Subject: [PATCH 387/406] Bump the python-dependencies group across 1 directory with 3 updates (#2098) * Bump the python-dependencies group across 1 directory with 3 updates Bumps the python-dependencies group with 3 updates in the / directory: [poetry](https://github.com/python-poetry/poetry), [uv](https://github.com/astral-sh/uv) and [pip](https://github.com/pypa/pip). Updates `poetry` from 2.4.0 to 2.4.1 - [Release notes](https://github.com/python-poetry/poetry/releases) - [Changelog](https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md) - [Commits](https://github.com/python-poetry/poetry/compare/2.4.0...2.4.1) Updates `uv` from 0.11.11 to 0.11.19 - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/uv/compare/0.11.11...0.11.19) Updates `pip` from 26.1.1 to 26.1.2 - [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst) - [Commits](https://github.com/pypa/pip/compare/26.1.1...26.1.2) --- updated-dependencies: - dependency-name: pip dependency-version: 26.1.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: poetry dependency-version: 2.4.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: uv dependency-version: 0.11.18 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: python-dependencies ... Signed-off-by: dependabot[bot] * Update changelog --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 3 +++ requirements/pip.txt | 2 +- requirements/poetry.txt | 2 +- requirements/uv.txt | 2 +- 4 files changed, 6 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 537edc013..156907930 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] +- Updated pip from 26.1.1 to 26.1.2. ([#2098](https://github.com/heroku/heroku-buildpack-python/pull/2098)) +- Updated Poetry from 2.4.0 to 2.4.1. ([#2098](https://github.com/heroku/heroku-buildpack-python/pull/2098)) +- Updated uv from 0.11.11 to 0.11.19. ([#2098](https://github.com/heroku/heroku-buildpack-python/pull/2098)) ## [v345] - 2026-05-11 diff --git a/requirements/pip.txt b/requirements/pip.txt index 074276481..657e2159e 100644 --- a/requirements/pip.txt +++ b/requirements/pip.txt @@ -1 +1 @@ -pip==26.1.1 +pip==26.1.2 diff --git a/requirements/poetry.txt b/requirements/poetry.txt index 1d3276616..c3973d4c7 100644 --- a/requirements/poetry.txt +++ b/requirements/poetry.txt @@ -1 +1 @@ -poetry==2.4.0 +poetry==2.4.1 diff --git a/requirements/uv.txt b/requirements/uv.txt index f5139606e..c5db7b48e 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.11.11 +uv==0.11.19 From 871f8a1b11d85c41a3b730b4279d4c4e5369cbb4 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Thu, 4 Jun 2026 15:40:08 +0000 Subject: [PATCH 388/406] Prepare release v345 (#2100) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 156907930..b20283171 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v345] - 2026-06-04 + - Updated pip from 26.1.1 to 26.1.2. ([#2098](https://github.com/heroku/heroku-buildpack-python/pull/2098)) - Updated Poetry from 2.4.0 to 2.4.1. ([#2098](https://github.com/heroku/heroku-buildpack-python/pull/2098)) - Updated uv from 0.11.11 to 0.11.19. ([#2098](https://github.com/heroku/heroku-buildpack-python/pull/2098)) @@ -1539,6 +1542,7 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. [unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v345...main [v345]: https://github.com/heroku/heroku-buildpack-python/compare/v344...v345 +[v345]: https://github.com/heroku/heroku-buildpack-python/compare/v344...v345 [v344]: https://github.com/heroku/heroku-buildpack-python/compare/v343...v344 [v343]: https://github.com/heroku/heroku-buildpack-python/compare/v342...v343 [v342]: https://github.com/heroku/heroku-buildpack-python/compare/v341...v342 From 18826fec0c364d56d65d712e610ef34c1c501ea6 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 5 Jun 2026 15:13:38 +0100 Subject: [PATCH 389/406] Fix CHANGELOG for release that didn't occur (#2101) The original v345 release (prepared in #2095) never ended up being published (I forgot to run the final `make publish` step), and so was superseded by #2100 - which resulted in two v345 entries being present in CHANGELOG. Signed-off-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- CHANGELOG.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b20283171..53b719ebf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,9 +8,6 @@ - Updated pip from 26.1.1 to 26.1.2. ([#2098](https://github.com/heroku/heroku-buildpack-python/pull/2098)) - Updated Poetry from 2.4.0 to 2.4.1. ([#2098](https://github.com/heroku/heroku-buildpack-python/pull/2098)) - Updated uv from 0.11.11 to 0.11.19. ([#2098](https://github.com/heroku/heroku-buildpack-python/pull/2098)) - -## [v345] - 2026-05-11 - - The Python 3.14 version alias now resolves to Python 3.14.5. ([#2094](https://github.com/heroku/heroku-buildpack-python/pull/2094)) ## [v344] - 2026-05-07 @@ -1542,7 +1539,6 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. [unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v345...main [v345]: https://github.com/heroku/heroku-buildpack-python/compare/v344...v345 -[v345]: https://github.com/heroku/heroku-buildpack-python/compare/v344...v345 [v344]: https://github.com/heroku/heroku-buildpack-python/compare/v343...v344 [v343]: https://github.com/heroku/heroku-buildpack-python/compare/v342...v343 [v342]: https://github.com/heroku/heroku-buildpack-python/compare/v341...v342 From 689b4172d7cf1d8c256f482ac7fa13aa4a0a1172 Mon Sep 17 00:00:00 2001 From: Colin Casey Date: Mon, 8 Jun 2026 14:39:11 -0300 Subject: [PATCH 390/406] Add OIDC-based buildpack release workflow (#2099) Calls the shared `_classic-buildpack-publish.yml` workflow from `heroku/languages-github-actions` to publish the `heroku/python` buildpack to the registry via OIDC. Auto-triggers when the Linguist-authored "Prepare release" PR is merged to `main` (guarded on event, branch, repo, and PR author). Also supports manual `workflow_dispatch`. Mirrors the pattern established in heroku/heroku-buildpack-nodejs#1653. GUS-W-22295409 is the reference implementation; this is the python caller workflow for the OIDC-Based Classic Buildpack Publishing epic. GUS-W-22295419. --- .github/workflows/release.yml | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 .github/workflows/release.yml diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 000000000..7c9e834cc --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,31 @@ +name: Release Buildpack + +on: + # Auto-trigger when the "Prepare release" PR (created by the + # _classic-buildpack-prepare-release.yml workflow) is merged. + pull_request: + branches: + - main + types: + - closed + workflow_dispatch: + +permissions: + id-token: write + contents: write + +jobs: + release: + name: Release + # On `pull_request`, only run for the merged auto-generated + # "Prepare release" PR (branch name set by + # _classic-buildpack-prepare-release.yml). Manual dispatches always run. + if: >- + github.event_name == 'workflow_dispatch' || + (github.event.pull_request.merged == true && + github.event.pull_request.head.repo.full_name == github.repository && + github.event.pull_request.head.ref == 'prepare-release' && + github.event.pull_request.user.login == 'heroku-linguist[bot]') + uses: heroku/languages-github-actions/.github/workflows/_classic-buildpack-publish.yml@latest + with: + buildpack_id: "heroku/python" From ccc4d5e4a9c76af07fdab290ea342de3ea111e86 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Mon, 8 Jun 2026 18:58:23 +0100 Subject: [PATCH 391/406] Fix intermittent `profile.d/` scripts test failure from extra trailing newline (#2102) The `profile.d/` scripts test asserts on captured `run_multi` command output using an exact match against a heredoc. The platform log stream that captures the output occasionally appends an extra trailing newline, which intermittently breaks these exact assertions. For example: https://github.com/heroku/heroku-buildpack-python/actions/runs/27144980449/job/80156740097?pr=2099#step:5:628 ``` Failures: 1) .profile.d/ scripts sets the required run-time env vars Failure/Error: expect(output).to eq(<<~OUTPUT) Python buildpack: Detected 512 MB available memory and 8 CPU cores. Python buildpack: Skipping automatic configuration of WEB_CONCURRENCY since it's already set. DYNO_RAM=512 WEB_CONCURRENCY=999 OUTPUT expected: "Python buildpack: Detected 512 MB available memory and 8 CPU cores.\nPython buildpack: Skipping auto...matic configuration of WEB_CONCURRENCY since it's already set.\nDYNO_RAM=512\nWEB_CONCURRENCY=999\n" got: "Python buildpack: Detected 512 MB available memory and 8 CPU cores.\nPython buildpack: Skipping auto...tic configuration of WEB_CONCURRENCY since it's already set.\nDYNO_RAM=512\nWEB_CONCURRENCY=999\n\n" (compared using ==) # ./spec/hatchet/profile_d_scripts_spec.rb:128:in 'block (4 levels) in ' # ./vendor/bundle/ruby/4.0.0/gems/heroku_hatchet-8.0.6/lib/hatchet/app.rb:255:in 'block in Hatchet::App#run_multi' ``` Adds a `normalize_trailing_newlines` spec helper that collapses any trailing newlines down to a single one, and applies it to the exact-match assertions in the test. This absorbs the spurious trailing newline whilst still asserting on the rest of the output exactly. GUS-W-22900269. --- spec/hatchet/profile_d_scripts_spec.rb | 18 +++++++++--------- spec/spec_helper.rb | 6 ++++++ 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/spec/hatchet/profile_d_scripts_spec.rb b/spec/hatchet/profile_d_scripts_spec.rb index bb805a69f..20b90418f 100644 --- a/spec/hatchet/profile_d_scripts_spec.rb +++ b/spec/hatchet/profile_d_scripts_spec.rb @@ -27,7 +27,7 @@ 'WEB_CONCURRENCY=this-should-be-preserved', ] app.run_multi(list_envs_cmd, heroku: { env: user_env_vars.join(';'), type: 'example-worker' }) do |output, _| - expect(output).to eq(<<~OUTPUT) + expect(normalize_trailing_newlines(output)).to eq(<<~OUTPUT) DYNO_RAM=512 FORWARDED_ALLOW_IPS=* GUNICORN_CMD_ARGS=this-should-be-preserved @@ -48,7 +48,7 @@ # Check WEB_CONCURRENCY support when using a Standard-1X dyno. # We set the process type to `web` so that we can test the web-dyno-only log output. app.run_multi(list_concurrency_envs_cmd, heroku: { size: 'standard-1x', type: 'web' }) do |output, _| - expect(output).to eq(<<~OUTPUT) + expect(normalize_trailing_newlines(output)).to eq(<<~OUTPUT) Python buildpack: Detected 512 MB available memory and 8 CPU cores. Python buildpack: Defaulting WEB_CONCURRENCY to 2 based on the available memory. DYNO_RAM=512 @@ -59,7 +59,7 @@ # Standard-2X app.run_multi(list_concurrency_envs_cmd, heroku: { size: 'standard-2x', type: 'web' }) do |output, _| - expect(output).to eq(<<~OUTPUT) + expect(normalize_trailing_newlines(output)).to eq(<<~OUTPUT) Python buildpack: Detected 1024 MB available memory and 8 CPU cores. Python buildpack: Defaulting WEB_CONCURRENCY to 4 based on the available memory. DYNO_RAM=1024 @@ -70,7 +70,7 @@ # Performance-M app.run_multi(list_concurrency_envs_cmd, heroku: { size: 'performance-m', type: 'web' }) do |output, _| - expect(output).to eq(<<~OUTPUT) + expect(normalize_trailing_newlines(output)).to eq(<<~OUTPUT) Python buildpack: Detected 2560 MB available memory and 2 CPU cores. Python buildpack: Defaulting WEB_CONCURRENCY to 5 based on the number of CPU cores. DYNO_RAM=2560 @@ -81,7 +81,7 @@ # Performance-L app.run_multi(list_concurrency_envs_cmd, heroku: { size: 'performance-l', type: 'web' }) do |output, _| - expect(output).to eq(<<~OUTPUT) + expect(normalize_trailing_newlines(output)).to eq(<<~OUTPUT) Python buildpack: Detected 14336 MB available memory and 8 CPU cores. Python buildpack: Defaulting WEB_CONCURRENCY to 17 based on the number of CPU cores. DYNO_RAM=14336 @@ -92,7 +92,7 @@ # Performance-L-RAM app.run_multi(list_concurrency_envs_cmd, heroku: { size: 'performance-l-ram', type: 'web' }) do |output, _| - expect(output).to eq(<<~OUTPUT) + expect(normalize_trailing_newlines(output)).to eq(<<~OUTPUT) Python buildpack: Detected 30720 MB available memory and 4 CPU cores. Python buildpack: Defaulting WEB_CONCURRENCY to 9 based on the number of CPU cores. DYNO_RAM=30720 @@ -103,7 +103,7 @@ # Performance-XL app.run_multi(list_concurrency_envs_cmd, heroku: { size: 'performance-xl', type: 'web' }) do |output, _| - expect(output).to eq(<<~OUTPUT) + expect(normalize_trailing_newlines(output)).to eq(<<~OUTPUT) Python buildpack: Detected 63488 MB available memory and 8 CPU cores. Python buildpack: Defaulting WEB_CONCURRENCY to 17 based on the number of CPU cores. DYNO_RAM=63488 @@ -114,7 +114,7 @@ # Performance-2XL app.run_multi(list_concurrency_envs_cmd, heroku: { size: 'performance-2xl', type: 'web' }) do |output, _| - expect(output).to eq(<<~OUTPUT) + expect(normalize_trailing_newlines(output)).to eq(<<~OUTPUT) Python buildpack: Detected 129024 MB available memory and 16 CPU cores. Python buildpack: Defaulting WEB_CONCURRENCY to 33 based on the number of CPU cores. DYNO_RAM=129024 @@ -125,7 +125,7 @@ # Check that WEB_CONCURRENCY is preserved if set, but that we still set DYNO_RAM. app.run_multi(list_concurrency_envs_cmd, heroku: { env: 'WEB_CONCURRENCY=999', type: 'web' }) do |output, _| - expect(output).to eq(<<~OUTPUT) + expect(normalize_trailing_newlines(output)).to eq(<<~OUTPUT) Python buildpack: Detected 512 MB available memory and 8 CPU cores. Python buildpack: Skipping automatic configuration of WEB_CONCURRENCY since it's already set. DYNO_RAM=512 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 1fcb598ff..416826cf0 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -62,6 +62,12 @@ def clean_output(output) .gsub(/\e\[[0-9;]+m/, '') end +def normalize_trailing_newlines(output) + # The platform log stream that captures `run`/`run_multi` command output occasionally appends + # an extra trailing newline, which otherwise intermittently breaks exact output assertions. + output.sub(/\n+\z/, "\n") +end + def update_buildpacks(app, buildpacks) # Updates the list of buildpacks for an existing app, until Hatchet supports this natively: # https://github.com/heroku/hatchet/issues/166 From fadb85eb70382ff58d4463c8019904384b7e4391 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 9 Jun 2026 15:07:32 +0100 Subject: [PATCH 392/406] Bump rubocop-rspec from 3.9.0 to 3.10.2 in the ruby-dependencies group (#2103) Bumps the ruby-dependencies group with 1 update: [rubocop-rspec](https://github.com/rubocop/rubocop-rspec). Updates `rubocop-rspec` from 3.9.0 to 3.10.2 - [Release notes](https://github.com/rubocop/rubocop-rspec/releases) - [Changelog](https://github.com/rubocop/rubocop-rspec/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop-rspec/compare/v3.9.0...v3.10.2) --- updated-dependencies: - dependency-name: rubocop-rspec dependency-version: 3.10.2 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 8fe3f0aeb..1a2e23f6e 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -65,9 +65,10 @@ GEM rubocop-ast (1.49.1) parser (>= 3.3.7.2) prism (~> 1.7) - rubocop-rspec (3.9.0) + rubocop-rspec (3.10.2) lint_roller (~> 1.1) - rubocop (~> 1.81) + regexp_parser (>= 2.0) + rubocop (~> 1.86, >= 1.86.2) ruby-progressbar (1.13.0) thor (1.5.0) threaded (0.0.4) @@ -121,7 +122,7 @@ CHECKSUMS rspec-support (3.13.7) sha256=0640e5570872aafefd79867901deeeeb40b0c9875a36b983d85f54fb7381c47c rubocop (1.87.0) sha256=b9d9ddf55116a513f8ef2c7ae660662d8b49301f118d3f0df61865b33a5c188d rubocop-ast (1.49.1) sha256=4412f3ee70f6fe4546cc489548e0f6fcf76cafcfa80fa03af67098ffed755035 - rubocop-rspec (3.9.0) sha256=8fa70a3619408237d789aeecfb9beef40576acc855173e60939d63332fdb55e2 + rubocop-rspec (3.10.2) sha256=0b3e2ecc592cd10ecbf0095bb58d1e357905276e069643523cc19eb7495f65e2 ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33 thor (1.5.0) sha256=e3a9e55fe857e44859ce104a84675ab6e8cd59c650a49106a05f55f136425e73 threaded (0.0.4) sha256=8f32c11d29f7c7237aabeb36d286c15ac495137ce06ace3357de743aafd4f0e3 From c332164119699a99d2d8b690c0683a9b3c542f2b Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 10 Jun 2026 18:31:03 +0100 Subject: [PATCH 393/406] Add support for Python 3.14.6 and 3.13.14 (#2105) Release announcement: https://blog.python.org/2026/06/python-3146-31314/ Changelogs: https://docs.python.org/release/3.14.6/whatsnew/changelog.html#python-3-14-6-final https://docs.python.org/release/3.13.14/whatsnew/changelog.html#python-3-13-14-final Binary builds: https://github.com/heroku/heroku-buildpack-python/actions/runs/27281631125 https://github.com/heroku/heroku-buildpack-python/actions/runs/27289906221 GUS-W-20590506. GUS-W-20590551. --- CHANGELOG.md | 2 ++ lib/python_version.sh | 4 ++-- spec/hatchet/python_version_spec.rb | 25 ++++++++++--------------- spec/spec_helper.rb | 4 ++-- 4 files changed, 16 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 53b719ebf..198d909ca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## [Unreleased] +- The Python 3.14 version alias now resolves to Python 3.14.6. ([#2105](https://github.com/heroku/heroku-buildpack-python/pull/2105)) +- The Python 3.13 version alias now resolves to Python 3.13.14. ([#2105](https://github.com/heroku/heroku-buildpack-python/pull/2105)) ## [v345] - 2026-06-04 diff --git a/lib/python_version.sh b/lib/python_version.sh index 2e96549b4..0b814cd61 100644 --- a/lib/python_version.sh +++ b/lib/python_version.sh @@ -7,8 +7,8 @@ set -euo pipefail LATEST_PYTHON_3_10="3.10.20" LATEST_PYTHON_3_11="3.11.15" LATEST_PYTHON_3_12="3.12.13" -LATEST_PYTHON_3_13="3.13.13" -LATEST_PYTHON_3_14="3.14.5" +LATEST_PYTHON_3_13="3.13.14" +LATEST_PYTHON_3_14="3.14.6" OLDEST_SUPPORTED_PYTHON_3_MINOR_VERSION=10 NEWEST_SUPPORTED_PYTHON_3_MINOR_VERSION=14 diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 93db7cda9..bf3ea412c 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -88,11 +88,7 @@ end end - # TODO: Enable on Heroku-26 after the default Python version next changes (for the 3.14.5 - # release), since for now there isn't a historic buildpack version we can use in this test - # whose stack check permits Heroku-26 and also has a different default Python version so - # that we can test the sticky versions feature. - context 'with an app last built using an older default Python version', stacks: %w[heroku-22 heroku-24] do + context 'with an app last built using an older default Python version' do # This test performs an initial build using an older buildpack version, followed # by a build using the current version. This ensures that: # - The current buildpack can successfully read the version metadata @@ -100,7 +96,7 @@ # - If no Python version is specified, the same major version as the # last build is used (sticky major versioning). # - Changes in the pip version are handled correctly. - let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v267'] } + let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v342'] } it 'builds with the same major Python version as the last build' do app.deploy do |app| @@ -109,7 +105,7 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same major version as the last build: Python 3.12 + remote: -----> No Python version was specified. Using the same major version as the last build: Python 3.14 remote: remote: ! Warning: No Python version was specified. remote: ! @@ -131,9 +127,9 @@ remote: ! In the new file, specify your app's major Python version number remote: ! only. Don't include quotes or a 'python-' prefix. remote: ! - remote: ! For example, to request the latest version of Python 3.12, + remote: ! For example, to request the latest version of Python 3.14, remote: ! update your .python-version file so it contains exactly: - remote: ! 3.12 + remote: ! 3.14 remote: ! remote: ! We strongly recommend that you don't specify the Python patch remote: ! version number, since it will pin your app to an exact Python @@ -152,13 +148,12 @@ remote: ! file and this warning will be made an error. remote: remote: -----> Discarding cache since: - remote: - The Python version has changed from 3.12.7 to #{LATEST_PYTHON_3_12} - remote: - The pip version has changed from 24.0 to #{PIP_VERSION} - remote: - The legacy SQLite3 headers and CLI binary need to be uninstalled - remote: -----> Installing Python #{LATEST_PYTHON_3_12} - remote: -----> Installing pip #{PIP_VERSION} and setuptools #{SETUPTOOLS_VERSION} + remote: - The Python version has changed from 3.14.4 to #{LATEST_PYTHON_3_14} + remote: - The pip version has changed from 26.0.1 to #{PIP_VERSION} + remote: -----> Installing Python #{LATEST_PYTHON_3_14} + remote: -----> Installing pip #{PIP_VERSION} OUTPUT - expect(app.run('python -V')).to eq("Python #{LATEST_PYTHON_3_12}\n") + expect(app.run('python -V')).to eq("Python #{LATEST_PYTHON_3_14}\n") expect($CHILD_STATUS.exitstatus).to eq(0) end end diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 416826cf0..7cdcb585a 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -13,8 +13,8 @@ LATEST_PYTHON_3_10 = '3.10.20' LATEST_PYTHON_3_11 = '3.11.15' LATEST_PYTHON_3_12 = '3.12.13' -LATEST_PYTHON_3_13 = '3.13.13' -LATEST_PYTHON_3_14 = '3.14.5' +LATEST_PYTHON_3_13 = '3.13.14' +LATEST_PYTHON_3_14 = '3.14.6' DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_14 DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') From e0a66e55e782fe862a13f0876392db0e36bc7902 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Wed, 10 Jun 2026 18:19:21 +0000 Subject: [PATCH 394/406] Prepare release v346 (#2106) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 198d909ca..055910c08 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v346] - 2026-06-10 + - The Python 3.14 version alias now resolves to Python 3.14.6. ([#2105](https://github.com/heroku/heroku-buildpack-python/pull/2105)) - The Python 3.13 version alias now resolves to Python 3.13.14. ([#2105](https://github.com/heroku/heroku-buildpack-python/pull/2105)) @@ -1539,7 +1542,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v345...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v346...main +[v346]: https://github.com/heroku/heroku-buildpack-python/compare/v345...v346 [v345]: https://github.com/heroku/heroku-buildpack-python/compare/v344...v345 [v344]: https://github.com/heroku/heroku-buildpack-python/compare/v343...v344 [v343]: https://github.com/heroku/heroku-buildpack-python/compare/v342...v343 From 7db11ed4e497fa5dd086042e35b873e59c8c3cb7 Mon Sep 17 00:00:00 2001 From: Colin Casey Date: Tue, 16 Jun 2026 08:00:36 -0300 Subject: [PATCH 395/406] Configure classic publish notifications (#2107) W-22900983 --- .github/workflows/release.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 7c9e834cc..144fc3ef4 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -29,3 +29,7 @@ jobs: uses: heroku/languages-github-actions/.github/workflows/_classic-buildpack-publish.yml@latest with: buildpack_id: "heroku/python" + app_id: ${{ vars.LINGUIST_GH_APP_ID }} + secrets: + app_private_key: ${{ secrets.LINGUIST_GH_PRIVATE_KEY }} + slack_webhook_url: ${{ secrets.BUILDPACK_RELEASE_FAILURE_WEBHOOK_URL }} From 32bf66ec23990f7a7315eb583192b1d8e1bcc452 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Tue, 16 Jun 2026 20:04:14 +0100 Subject: [PATCH 396/406] Fix inaccurate `PYTHONUNBUFFERED` comment (#2108) `PYTHONUNBUFFERED` affects stdout and stderr, not stdin: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUNBUFFERED And updates the comment to actually explain the why not the what. GUS-W-23030029. --- bin/compile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/compile b/bin/compile index 90cba823d..7017cabf3 100755 --- a/bin/compile +++ b/bin/compile @@ -75,7 +75,7 @@ python_home='/app/.heroku/python' # using `/app` paths which will still work at run-time after relocation. Amongst other things, this # ensures that the shebang lines in the entrypoint scripts of installed packages are correct. export PATH="/app/.heroku/python/bin:${PATH}" -# Tell Python to not buffer its stdin/stdout. +# Disable Python's output buffering to ensure logs aren't dropped if an app crashes. export PYTHONUNBUFFERED=1 # Ensure Python uses a Unicode locale, to prevent the issues described in: # https://github.com/docker-library/python/pull/570 From 8095af0e2d3c679f070fdfa44d95b65fd060a1a4 Mon Sep 17 00:00:00 2001 From: Ed Morley Date: Fri, 3 Jul 2026 16:00:50 +0100 Subject: [PATCH 397/406] Route builds/Dockerfile Dependabot bumps to maintainer in CODEOWNERS (#2113) builds/Dockerfile matched only the `* @heroku/languages` catch-all, so Dependabot cosign bumps requested team review. Add an override to route it to the maintainer, like the other Dependabot-touched paths. GUS-W-23276629. --- .github/CODEOWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 9971b391e..3f4e003a1 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -8,4 +8,5 @@ CHANGELOG.md @edmorley Gemfile.lock @edmorley /.github/workflows/ @edmorley +/builds/Dockerfile @edmorley /requirements/ @edmorley From 6d3b48705221f208082aa2c89a1ce77d15b19c34 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 4 Jul 2026 00:02:41 +0100 Subject: [PATCH 398/406] Bump sigstore/cosign/cosign from v3.0.6 to v3.1.1 in /builds (#2112) Bumps sigstore/cosign/cosign from v3.0.6 to v3.1.1. --- updated-dependencies: - dependency-name: sigstore/cosign/cosign dependency-version: v3.1.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- builds/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/builds/Dockerfile b/builds/Dockerfile index 5a17355ce..e2a56b2db 100644 --- a/builds/Dockerfile +++ b/builds/Dockerfile @@ -1,5 +1,5 @@ ARG STACK_VERSION="24" -FROM ghcr.io/sigstore/cosign/cosign:v3.0.6@sha256:de9c65609e6bde17e6b48de485ee788407c9502fa08b8f4459f595b21f56cd00 AS cosign +FROM ghcr.io/sigstore/cosign/cosign:v3.1.1@sha256:6bbe0d281d955c79f85b325f0f7e651c1bcab5a4fa4ad4903d74955178a3b2eb AS cosign FROM heroku/heroku:${STACK_VERSION}-build ARG STACK_VERSION From 814c457825f313ef7118d4d587385670a99c23a7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 3 Jul 2026 23:06:42 +0000 Subject: [PATCH 399/406] Bump actions/checkout from 6 to 7 (#2109) Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v6...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/build_python_runtime.yml | 6 +++--- .github/workflows/check_changelog.yml | 2 +- .github/workflows/ci.yml | 6 +++--- .github/workflows/hatchet_app_cleaner.yml | 2 +- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/workflows/build_python_runtime.yml b/.github/workflows/build_python_runtime.yml index 378229093..6039a5710 100644 --- a/.github/workflows/build_python_runtime.yml +++ b/.github/workflows/build_python_runtime.yml @@ -43,7 +43,7 @@ jobs: STACK_VERSION: "22" steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 - name: Build Docker image run: docker build --platform="linux/amd64" --pull --tag buildenv --build-arg=STACK_VERSION builds/ - name: Compile and package Python runtime @@ -68,7 +68,7 @@ jobs: STACK_VERSION: "24" steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 - name: Build Docker image run: docker build --platform="linux/${{ matrix.arch }}" --pull --tag buildenv --build-arg=STACK_VERSION builds/ - name: Compile and package Python runtime @@ -93,7 +93,7 @@ jobs: STACK_VERSION: "26" steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 - name: Build Docker image run: docker build --platform="linux/${{ matrix.arch }}" --pull --tag buildenv --build-arg=STACK_VERSION builds/ - name: Compile and package Python runtime diff --git a/.github/workflows/check_changelog.yml b/.github/workflows/check_changelog.yml index 419c9fa6a..ee511b3d2 100644 --- a/.github/workflows/check_changelog.yml +++ b/.github/workflows/check_changelog.yml @@ -13,7 +13,7 @@ jobs: if: (!contains(github.event.pull_request.labels.*.name, 'skip changelog')) steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 - name: Check that CHANGELOG is touched run: | git fetch origin ${{ github.base_ref }} --depth 1 && \ diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8826ce8e3..35de00fb4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -19,7 +19,7 @@ jobs: runs-on: ubuntu-24.04 steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 - name: Install Ruby and dependencies uses: ruby/setup-ruby@v1 with: @@ -51,7 +51,7 @@ jobs: RSPEC_RETRY_RETRY_COUNT: 1 steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 - name: Install Ruby and dependencies uses: ruby/setup-ruby@v1 with: @@ -67,7 +67,7 @@ jobs: runs-on: ubuntu-24.04 steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 # These test both the local development `make run` workflow and that `bin/report` completes successfully # for both passing and failing builds (since `bin/report` can't easily be tested via Hatchet tests). - name: Run buildpack using default app fixture diff --git a/.github/workflows/hatchet_app_cleaner.yml b/.github/workflows/hatchet_app_cleaner.yml index 5844b15ca..05343fb8a 100644 --- a/.github/workflows/hatchet_app_cleaner.yml +++ b/.github/workflows/hatchet_app_cleaner.yml @@ -19,7 +19,7 @@ jobs: HEROKU_DISABLE_AUTOUPDATE: 1 steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 - name: Install Ruby and dependencies uses: ruby/setup-ruby@v1 with: From 115dcc84211cee295a1c0cf5a15c74704696f411 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 3 Jul 2026 23:12:41 +0000 Subject: [PATCH 400/406] Bump rubocop from 1.87.0 to 1.88.1 in the ruby-dependencies group (#2110) Bumps the ruby-dependencies group with 1 update: [rubocop](https://github.com/rubocop/rubocop). Updates `rubocop` from 1.87.0 to 1.88.1 - [Release notes](https://github.com/rubocop/rubocop/releases) - [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop/rubocop/compare/v1.87.0...v1.88.1) --- updated-dependencies: - dependency-name: rubocop dependency-version: 1.88.1 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: ruby-dependencies ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Gemfile.lock | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 1a2e23f6e..36e3c353f 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -20,7 +20,7 @@ GEM rrrretry (~> 1) thor (~> 1) threaded (~> 0) - json (2.19.8) + json (2.20.0) language_server-protocol (3.17.0.5) lint_roller (1.1.0) logger (1.7.0) @@ -51,7 +51,7 @@ GEM rspec-retry (0.6.2) rspec-core (> 3.3) rspec-support (3.13.7) - rubocop (1.87.0) + rubocop (1.88.1) json (~> 2.3) language_server-protocol (~> 3.17.0.2) lint_roller (~> 1.1.0) @@ -100,7 +100,7 @@ CHECKSUMS excon (1.5.0) sha256=c503ad1d0123bc8ab2a062ff3789dc891ec368cb9e13765ab88a9c58c8bb6d50 heroics (0.1.3) sha256=31d792e8706ecc6f78299f52d0b0f8ab55489a13b5f4f76c3050b07912563562 heroku_hatchet (8.0.6) sha256=886a7d7d686859db9fac19df1194a35c5349285956ca4d134054c5c376accea3 - json (2.19.8) sha256=6354310fd76ef69b87d5bd1f38b40d730613baf90b6803d2d0a48f618d32dfaa + json (2.20.0) sha256=9362bc6e55a952b056abf9167cf053358181c904cb70cd6eee0808ea830fc32b language_server-protocol (3.17.0.5) sha256=fd1e39a51a28bf3eec959379985a72e296e9f9acfce46f6a79d31ca8760803cc lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87 logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203 @@ -120,7 +120,7 @@ CHECKSUMS rspec-expectations (3.13.5) sha256=33a4d3a1d95060aea4c94e9f237030a8f9eae5615e9bd85718fe3a09e4b58836 rspec-retry (0.6.2) sha256=6101ba23a38809811ae3484acde4ab481c54d846ac66d5037ccb40131a60d858 rspec-support (3.13.7) sha256=0640e5570872aafefd79867901deeeeb40b0c9875a36b983d85f54fb7381c47c - rubocop (1.87.0) sha256=b9d9ddf55116a513f8ef2c7ae660662d8b49301f118d3f0df61865b33a5c188d + rubocop (1.88.1) sha256=726af773d6bc169ed3ff852f3ca020b7c58d39c34e7a8d879a8e0147cc994f26 rubocop-ast (1.49.1) sha256=4412f3ee70f6fe4546cc489548e0f6fcf76cafcfa80fa03af67098ffed755035 rubocop-rspec (3.10.2) sha256=0b3e2ecc592cd10ecbf0095bb58d1e357905276e069643523cc19eb7495f65e2 ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33 From 072ebb9d670385f6ed81eb173db6f6bd77374ffa Mon Sep 17 00:00:00 2001 From: Colin Casey Date: Mon, 6 Jul 2026 13:06:45 -0300 Subject: [PATCH 401/406] Post Honeycomb release marker on release (#2114) Passes `HONEYCOMB_API_KEY` through to the shared `_classic-buildpack-publish.yml@latest` workflow, enabling a Honeycomb release marker on the `builds` dataset for each production publish of this classic buildpack. GUS-W-22846373. --- .github/workflows/release.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 144fc3ef4..16872d5c9 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -33,3 +33,4 @@ jobs: secrets: app_private_key: ${{ secrets.LINGUIST_GH_PRIVATE_KEY }} slack_webhook_url: ${{ secrets.BUILDPACK_RELEASE_FAILURE_WEBHOOK_URL }} + honeycomb_api_key: ${{ secrets.HONEYCOMB_API_KEY }} From 8ef81fcb8d5649112a3e0ae1b316020fa3436afa Mon Sep 17 00:00:00 2001 From: Ed Morley Date: Wed, 8 Jul 2026 19:46:11 +0100 Subject: [PATCH 402/406] Fix intermittent one-off dyno test failures from extra trailing newline (#2115) The platform log stream that captures one-off dyno (`app.run`) output occasionally appends an extra trailing newline, which intermittently breaks exact-match (`.to eq`) output assertions. For example: https://github.com/heroku/heroku-buildpack-python/actions/runs/28526498507/job/85081876325?pr=2110#step:5:589 This is the same flakiness that #2102 fixed for the `profile.d/` scripts test. Reuses the existing `normalize_trailing_newlines` spec helper to absorb the spurious newline while still asserting on the rest of the output exactly, applying it to the seven `app.run` exact-match assertions. GUS-W-23366753. --- spec/hatchet/pip_spec.rb | 2 +- spec/hatchet/pipenv_spec.rb | 2 +- spec/hatchet/poetry_spec.rb | 2 +- spec/hatchet/python_version_spec.rb | 6 +++--- spec/hatchet/uv_spec.rb | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 0f37dba4c..07e53f55c 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -110,7 +110,7 @@ REGEX # For historical reasons pip is made available at run-time too, unlike some of the other package managers. - expect(app.run('bin/print-env-vars.sh && pip --version')).to eq(<<~OUTPUT) + expect(normalize_trailing_newlines(app.run('bin/print-env-vars.sh && pip --version'))).to eq(<<~OUTPUT) DYNO_RAM=512 FORWARDED_ALLOW_IPS=* GUNICORN_CMD_ARGS=--access-logfile - diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index 7bdaacb69..c7080e98e 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -111,7 +111,7 @@ REGEX # For historical reasons Pipenv is made available at run-time too, unlike some of the other package managers. - expect(clean_output(app.run('bin/print-env-vars.sh && pipenv --version'))).to eq(<<~OUTPUT) + expect(normalize_trailing_newlines(app.run('bin/print-env-vars.sh && pipenv --version'))).to eq(<<~OUTPUT) DYNO_RAM=512 FORWARDED_ALLOW_IPS=* GUNICORN_CMD_ARGS=--access-logfile - diff --git a/spec/hatchet/poetry_spec.rb b/spec/hatchet/poetry_spec.rb index 25aad1ada..03b980ddf 100644 --- a/spec/hatchet/poetry_spec.rb +++ b/spec/hatchet/poetry_spec.rb @@ -112,7 +112,7 @@ REGEX command = 'bin/print-env-vars.sh && if command -v poetry; then echo "Poetry unexpectedly found!" && exit 1; fi' - expect(app.run(command)).to eq(<<~OUTPUT) + expect(normalize_trailing_newlines(app.run(command))).to eq(<<~OUTPUT) DYNO_RAM=512 FORWARDED_ALLOW_IPS=* GUNICORN_CMD_ARGS=--access-logfile - diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index bf3ea412c..24a083e04 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -24,7 +24,7 @@ remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) OUTPUT end - expect(app.run('python -V')).to eq("Python #{resolved_version}\n") + expect(normalize_trailing_newlines(app.run('python -V'))).to eq("Python #{resolved_version}\n") expect($CHILD_STATUS.exitstatus).to eq(0) end end @@ -153,7 +153,7 @@ remote: -----> Installing Python #{LATEST_PYTHON_3_14} remote: -----> Installing pip #{PIP_VERSION} OUTPUT - expect(app.run('python -V')).to eq("Python #{LATEST_PYTHON_3_14}\n") + expect(normalize_trailing_newlines(app.run('python -V'))).to eq("Python #{LATEST_PYTHON_3_14}\n") expect($CHILD_STATUS.exitstatus).to eq(0) end end @@ -277,7 +277,7 @@ remote: -----> Installing dependencies using 'pip install -r requirements.txt' remote: Collecting typing-extensions==4.15.0 (from -r requirements.txt (line 2)) OUTPUT - expect(app.run('python -V')).to eq("Python #{LATEST_PYTHON_3_10}\n") + expect(normalize_trailing_newlines(app.run('python -V'))).to eq("Python #{LATEST_PYTHON_3_10}\n") expect($CHILD_STATUS.exitstatus).to eq(0) end end diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 716afee90..874668131 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -109,7 +109,7 @@ REGEX command = 'bin/print-env-vars.sh && if command -v uv; then echo "uv unexpectedly found!" && exit 1; fi' - expect(app.run(command)).to eq(<<~OUTPUT) + expect(normalize_trailing_newlines(app.run(command))).to eq(<<~OUTPUT) DYNO_RAM=512 FORWARDED_ALLOW_IPS=* GUNICORN_CMD_ARGS=--access-logfile - From b38040b00a8f4b87505d50a286fa381ed03c8211 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 10 Jul 2026 11:52:34 +0100 Subject: [PATCH 403/406] Bump the python-dependencies group across 1 directory with 2 updates (#2111) * Bump the python-dependencies group across 1 directory with 2 updates --- updated-dependencies: - dependency-name: pipenv dependency-version: 2026.6.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: uv dependency-version: 0.11.26 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: python-dependencies ... Signed-off-by: dependabot[bot] * Add changelog entries * Fix uv lockfile-out-of-sync spec assertion --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ed Morley --- CHANGELOG.md | 2 ++ requirements/pipenv.txt | 2 +- requirements/uv.txt | 2 +- spec/hatchet/uv_spec.rb | 4 +++- 4 files changed, 7 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 055910c08..c21739031 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## [Unreleased] +- Updated Pipenv from 2026.6.1 to 2026.6.2. ([#2111](https://github.com/heroku/heroku-buildpack-python/pull/2111)) +- Updated uv from 0.11.19 to 0.11.28. ([#2111](https://github.com/heroku/heroku-buildpack-python/pull/2111)) ## [v346] - 2026-06-10 diff --git a/requirements/pipenv.txt b/requirements/pipenv.txt index b61f78890..5392a7319 100644 --- a/requirements/pipenv.txt +++ b/requirements/pipenv.txt @@ -1 +1 @@ -pipenv==2026.6.1 +pipenv==2026.6.2 diff --git a/requirements/uv.txt b/requirements/uv.txt index c5db7b48e..f329fecf5 100644 --- a/requirements/uv.txt +++ b/requirements/uv.txt @@ -1 +1 @@ -uv==0.11.19 +uv==0.11.28 diff --git a/spec/hatchet/uv_spec.rb b/spec/hatchet/uv_spec.rb index 874668131..a3b3e2f6f 100644 --- a/spec/hatchet/uv_spec.rb +++ b/spec/hatchet/uv_spec.rb @@ -463,7 +463,9 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Installing dependencies using 'uv sync --locked --no-default-groups' remote: Resolved 2 packages in .+s - remote: The lockfile at `uv.lock` needs to be updated, but `--locked` was provided. To update the lockfile, run `uv lock`. + remote: error: The lockfile at `uv.lock` needs to be updated, but `--locked` was provided. + remote: + remote: hint: To update the lockfile, run `uv lock`. remote: remote: ! Error: Unable to install dependencies using uv. remote: ! From e83797ae35b375eeafe9c350d3a8e8316501e4f5 Mon Sep 17 00:00:00 2001 From: "heroku-linguist[bot]" <136119646+heroku-linguist[bot]@users.noreply.github.com> Date: Fri, 10 Jul 2026 12:19:05 +0100 Subject: [PATCH 404/406] Prepare release v347 (#2116) Co-authored-by: heroku-linguist[bot] <136119646+heroku-linguist[bot]@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c21739031..3648b20a2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## [Unreleased] + +## [v347] - 2026-07-10 + - Updated Pipenv from 2026.6.1 to 2026.6.2. ([#2111](https://github.com/heroku/heroku-buildpack-python/pull/2111)) - Updated uv from 0.11.19 to 0.11.28. ([#2111](https://github.com/heroku/heroku-buildpack-python/pull/2111)) @@ -1544,7 +1547,8 @@ Default Python is now latest 2.7.10. Updated pip and Distribute. - Setuptools updated to v16.0 - pip updated to v7.0.1 -[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v346...main +[unreleased]: https://github.com/heroku/heroku-buildpack-python/compare/v347...main +[v347]: https://github.com/heroku/heroku-buildpack-python/compare/v346...v347 [v346]: https://github.com/heroku/heroku-buildpack-python/compare/v345...v346 [v345]: https://github.com/heroku/heroku-buildpack-python/compare/v344...v345 [v344]: https://github.com/heroku/heroku-buildpack-python/compare/v343...v344 From f4f329e3a8a8ad0e0ed71ca1b58eaa7058faf461 Mon Sep 17 00:00:00 2001 From: Ed Morley Date: Tue, 14 Jul 2026 09:45:55 +0100 Subject: [PATCH 405/406] Add CODE_OF_CONDUCT.md and SECURITY.md (#2117) To match the templates at: https://github.com/salesforce/oss-template GUS-W-23426985. --- CODE_OF_CONDUCT.md | 105 +++++++++++++++++++++++++++++++++++++++++++++ SECURITY.md | 7 +++ 2 files changed, 112 insertions(+) create mode 100644 CODE_OF_CONDUCT.md create mode 100644 SECURITY.md diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 000000000..9e229777d --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,105 @@ +# Salesforce Open Source Community Code of Conduct + +## About the Code of Conduct + +Equality is a core value at Salesforce. We believe a diverse and inclusive +community fosters innovation and creativity, and are committed to building a +culture where everyone feels included. + +Salesforce open-source projects are committed to providing a friendly, safe, and +welcoming environment for all, regardless of gender identity and expression, +sexual orientation, disability, physical appearance, body size, ethnicity, nationality, +race, age, religion, level of experience, education, socioeconomic status, or +other similar personal characteristics. + +The goal of this code of conduct is to specify a baseline standard of behavior so +that people with different social values and communication styles can work +together effectively, productively, and respectfully in our open source community. +It also establishes a mechanism for reporting issues and resolving conflicts. + +All questions and reports of abusive, harassing, or otherwise unacceptable behavior +in a Salesforce open-source project may be reported by contacting the Salesforce +Open Source Conduct Committee at ossconduct@salesforce.com. + +## Our Pledge + +In the interest of fostering an open and welcoming environment, we as +contributors and maintainers pledge to making participation in our project and +our community a harassment-free experience for everyone, regardless of gender +identity and expression, sexual orientation, disability, physical appearance, +body size, ethnicity, nationality, race, age, religion, level of experience, education, +socioeconomic status, or other similar personal characteristics. + +## Our Standards + +Examples of behavior that contributes to creating a positive environment +include: + +* Using welcoming and inclusive language +* Being respectful of differing viewpoints and experiences +* Gracefully accepting constructive criticism +* Focusing on what is best for the community +* Showing empathy toward other community members + +Examples of unacceptable behavior by participants include: + +* The use of sexualized language or imagery and unwelcome sexual attention or +advances +* Personal attacks, insulting/derogatory comments, or trolling +* Public or private harassment +* Publishing, or threatening to publish, others' private information—such as +a physical or electronic address—without explicit permission +* Other conduct which could reasonably be considered inappropriate in a +professional setting +* Advocating for or encouraging any of the above behaviors + +## Our Responsibilities + +Project maintainers are responsible for clarifying the standards of acceptable +behavior and are expected to take appropriate and fair corrective action in +response to any instances of unacceptable behavior. + +Project maintainers have the right and responsibility to remove, edit, or +reject comments, commits, code, wiki edits, issues, and other contributions +that are not aligned with this Code of Conduct, or to ban temporarily or +permanently any contributor for other behaviors that they deem inappropriate, +threatening, offensive, or harmful. + +## Scope + +This Code of Conduct applies both within project spaces and in public spaces +when an individual is representing the project or its community. Examples of +representing a project or community include using an official project email +address, posting via an official social media account, or acting as an appointed +representative at an online or offline event. Representation of a project may be +further defined and clarified by project maintainers. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported by contacting the Salesforce Open Source Conduct Committee +at ossconduct@salesforce.com. All complaints will be reviewed and investigated +and will result in a response that is deemed necessary and appropriate to the +circumstances. The committee is obligated to maintain confidentiality with +regard to the reporter of an incident. Further details of specific enforcement +policies may be posted separately. + +Project maintainers who do not follow or enforce the Code of Conduct in good +faith may face temporary or permanent repercussions as determined by other +members of the project's leadership and the Salesforce Open Source Conduct +Committee. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][contributor-covenant-home], +version 1.4, available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html. +It includes adaptations and additions from [Go Community Code of Conduct][golang-coc], +[CNCF Code of Conduct][cncf-coc], and [Microsoft Open Source Code of Conduct][microsoft-coc]. + +This Code of Conduct is licensed under the [Creative Commons Attribution 3.0 License][cc-by-3-us]. + +[contributor-covenant-home]: https://www.contributor-covenant.org +[golang-coc]: https://golang.org/conduct +[cncf-coc]: https://github.com/cncf/foundation/blob/master/code-of-conduct.md +[microsoft-coc]: https://opensource.microsoft.com/codeofconduct/ +[cc-by-3-us]: https://creativecommons.org/licenses/by/3.0/us/ diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 000000000..72541d6bd --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,7 @@ +# Security + +Please report any security issue to [https://www.sfdc.co/SubmitVuln](https://www.sfdc.co/SubmitVuln) +as soon as it is discovered. This project limits its runtime dependencies in +order to reduce the total cost of ownership as much as can be, but all consumers +should remain vigilant and have their security stakeholders review all third-party +products (3PP) like this one and their dependencies. From e3b3a4d4ff663dfdcbff15663010221a9c6c51ec Mon Sep 17 00:00:00 2001 From: Ed Morley Date: Fri, 24 Jul 2026 18:19:44 +0100 Subject: [PATCH 406/406] Hardcode the setuptools version instead of using Dependabot (#2119) We're not updating setuptools to v71+ due to its new approach to vendored dependencies, so it doesn't need to be managed by Dependabot: https://github.com/heroku/heroku-buildpack-python/pull/1630#issuecomment-2324236653 Remove the requirements file and Dependabot ignore entry, and hardcode the version directly where it's used. This also stops Dependabot security updates from re-opening PRs for it (which bypass the ignore list). GUS-W-23572996. --- .github/dependabot.yml | 4 ---- lib/pip.sh | 5 ++++- requirements/setuptools.txt | 1 - spec/spec_helper.rb | 4 +++- 4 files changed, 7 insertions(+), 7 deletions(-) delete mode 100644 requirements/setuptools.txt diff --git a/.github/dependabot.yml b/.github/dependabot.yml index f59c458c9..636065174 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -33,10 +33,6 @@ updates: directory: "/" schedule: interval: "monthly" - ignore: - # We're not updating to setuptools v71+ due to its new approach to vendored dependencies: - # https://github.com/heroku/heroku-buildpack-python/pull/1630#issuecomment-2324236653 - - dependency-name: "setuptools" labels: - "dependencies" - "python" diff --git a/lib/pip.sh b/lib/pip.sh index 62585f8b4..479ad5e32 100644 --- a/lib/pip.sh +++ b/lib/pip.sh @@ -5,7 +5,10 @@ set -euo pipefail PIP_VERSION=$(utils::get_requirement_version 'pip') -SETUPTOOLS_VERSION=$(utils::get_requirement_version 'setuptools') +# This is hardcoded rather than managed by Dependabot since we can't update to newer versions due to: +# https://github.com/heroku/heroku-buildpack-python/pull/1630#issuecomment-2324236653 +# Also, once we drop support for older Pythons we'll stop installing setuptools entirely. +SETUPTOOLS_VERSION='70.3.0' function pip::install_pip() { local python_home="${1}" diff --git a/requirements/setuptools.txt b/requirements/setuptools.txt deleted file mode 100644 index d125505ce..000000000 --- a/requirements/setuptools.txt +++ /dev/null @@ -1 +0,0 @@ -setuptools==70.3.0 diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 7cdcb585a..fa6deb277 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -28,10 +28,12 @@ def get_requirement_version(package_name) end PIP_VERSION = get_requirement_version('pip') -SETUPTOOLS_VERSION = get_requirement_version('setuptools') PIPENV_VERSION = get_requirement_version('pipenv') POETRY_VERSION = get_requirement_version('poetry') UV_VERSION = get_requirement_version('uv') +# This is hardcoded rather than managed by Dependabot since we can't update to newer versions due to: +# https://github.com/heroku/heroku-buildpack-python/pull/1630#issuecomment-2324236653 +SETUPTOOLS_VERSION = '70.3.0' # Work around the return value for `default_buildpack` changing after deploy: # https://github.com/heroku/hatchet/issues/180