From 02b22856bfef12e0eaec19da9b9ffd826956bfee Mon Sep 17 00:00:00 2001 From: "David A. Wheeler" Date: Thu, 25 Jun 2026 20:44:35 -0400 Subject: [PATCH] Fix Codecov 0%: fix coverage.json, rm gem codecov Our earlier commit e2de938d created the file coverage/codecov-result.json fixing the "Found 0 coverage files" error, but the site still showed 0%. Codecov's backend rejected the upload as "Unusable report ... incorrect data format": the abandoned codecov 0.6.0 gem's formatter emits the deprecated v2 "network" report format (a filename list, a "<<<<<< network" separator, then one line of coverage JSON) which isn't even valid JSON despite the .json name. The pinned Codecov CLI uploaded it without complaint because `--plugin noop` means it never parses the file, so the upload step looked healthy while the backend silently discarded the data. This commit changes the approach for recording coverage results. Now we write coverage/coverage.json with SimpleCov::Formatter::JSONFormatter (from simplecov_json_formatter), whose output Codecov parses natively, and point the CLI --file at it. As before this runs in test:coverage_gaps rather than test_helper.rb, because CI runs with DEFER_COVERAGE set (swapping in SimpleFormatter) and the merged result is only complete once all parallel + system runs finish. This removes the abandoned codecov gem entirely and drops its dead formatter reference from test_helper.rb (it never ran in CI anyway, since DEFER_COVERAGE overrode it). The codecov gem was the only thing pulling in simplecov, including its <0.22 pin, so we now list simplecov and simplecov_json_formatter explicitly: both are required directly (in test_helper.rb and test:coverage_gaps respectively), so declaring them prevents a future simplecov release that stops bundling the JSON formatter from breaking coverage reporting in a surprising way. Signed-off-by: David A. Wheeler Co-Authored-By: Claude Opus 4.8 Signed-off-by: David A. Wheeler --- .circleci/config.yml | 6 +++--- Gemfile | 8 ++++++-- Gemfile.lock | 7 +++---- lib/tasks/default.rake | 20 +++++++++++--------- test/test_helper.rb | 16 ++++++---------- 5 files changed, 29 insertions(+), 28 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index cac461c9b..c580dc3c2 100755 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -177,8 +177,8 @@ jobs: # use. This matches the dependency-pinning policy stated at the top of this # file and mirrors the Heroku CLI install in the deploy job: we never run # an unpinned, unverified binary fetched over the network. The coverage - # JSON itself is produced by SimpleCov::Formatter::Codecov; see - # test/test_helper.rb. --disable-search + --file restrict the upload to + # JSON itself is produced by SimpleCov::Formatter::JSONFormatter in the + # test:coverage_gaps rake task. --disable-search + --file restrict the upload to # exactly that file, and --plugin noop skips Codecov's coverage-generation # plugins (we already have the report). We do NOT pass --fail-on-error, so # a Codecov outage cannot break an otherwise-green build. The upload token, @@ -208,7 +208,7 @@ jobs: echo "** Hash verified. Uploading coverage." chmod +x codecov ./codecov upload-process --disable-search --plugin noop \ - --file coverage/codecov-result.json + --file coverage/coverage.json rm -f codecov # I haven't found a reliable way to calculate HEROKU_APP just once, diff --git a/Gemfile b/Gemfile index 76d655c59..effa89dee 100644 --- a/Gemfile +++ b/Gemfile @@ -183,7 +183,6 @@ end group :test do gem 'capybara-slow_finder_errors', require: false # ID slow Capybara finders - gem 'codecov', require: false # Report test code coverage # Pin minitest < 6.0 until minitest-reporters supports it. # Minitest 6.0 introduced breaking changes that cause minitest-reporters 1.7.1 # to fail silently (tests don't run). Remove this constraint when @@ -196,7 +195,12 @@ group :test do # Note: Updating 'rails-controller-testing' to '1.0.5' causes failures gem 'rails-controller-testing', '~> 1.0' # for `assigns` and `assert_template` gem 'selenium-webdriver' # Automates browser i/f for Rails system testing - # We don't list "simplecov"; code depends on it & brings it in + # Test coverage. We require both directly: simplecov in test_helper.rb, and + # simplecov_json_formatter in test:coverage_gaps to write coverage/coverage.json + # for the Codecov upload. List both so a future simplecov release that stops + # bundling the JSON formatter can't break us in a surprising way. + gem 'simplecov', require: false + gem 'simplecov_json_formatter', require: false gem 'webmock', '~> 3.0', require: false # Mock HTTP requests for testing end diff --git a/Gemfile.lock b/Gemfile.lock index 02c505cb5..89aaa7f3d 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -138,8 +138,6 @@ GEM concurrent-ruby code_analyzer (0.5.5) sexp_processor - codecov (0.6.0) - simplecov (>= 0.15, < 0.22) coderay (1.1.3) commonmarker (2.6.3) rb_sys (~> 0.9) @@ -532,7 +530,7 @@ GEM concurrent-ruby (~> 1.0, >= 1.0.2) logger sexp_processor (4.17.5) - simplecov (0.21.2) + simplecov (0.22.0) docile (~> 1.1) simplecov-html (~> 0.11) simplecov_json_formatter (~> 0.1) @@ -637,7 +635,6 @@ DEPENDENCIES bundler-audit capybara-slow_finder_errors chartkick (~> 5.2) - codecov commonmarker (~> 2.6.2) dotenv (~> 3.0) eslintrb @@ -690,6 +687,8 @@ DEPENDENCIES selenium-webdriver sentry-rails sentry-ruby + simplecov + simplecov_json_formatter solid_queue (~> 1.1) spring (~> 4.1) sprockets-rails (= 3.5.2) diff --git a/lib/tasks/default.rake b/lib/tasks/default.rake index 368bfc8f1..9ca1bae14 100644 --- a/lib/tasks/default.rake +++ b/lib/tasks/default.rake @@ -963,20 +963,22 @@ task 'test:coverage_gaps' do puts "Combined Coverage: #{result.covered_percent.round(2)}%" - # In CI, write coverage/codecov-result.json from the merged result so the + # In CI, write coverage/coverage.json from the merged result so the # Codecov CLI (see .circleci/config.yml) has a file to upload. This MUST # happen here, not in test_helper.rb: the test workers run with - # DEFER_COVERAGE set, which swaps in the minimal SimpleFormatter, so the - # Codecov formatter never runs during the test processes. The merged + # DEFER_COVERAGE set, which swaps in the minimal SimpleFormatter, so a + # report formatter never runs during the test processes. The merged # result is only complete once all parallel + system test runs finish. # - # We call the gem's inner Codecov::SimpleCov::Formatter directly rather - # than SimpleCov::Formatter::Codecov, because the latter also invokes the - # gem's deprecated built-in network uploader. We upload separately with a - # pinned, hash-verified CLI, so we want ONLY the file-writing half here. + # We use SimpleCov::Formatter::JSONFormatter (from simplecov_json_formatter, + # a dependency of simplecov) because Codecov's backend natively parses its + # output. We previously used the abandoned 'codecov' gem's formatter, but + # it emits the deprecated "network" report format that the current Codecov + # backend rejects as "unusable report ... incorrect data format", silently + # yielding 0% coverage on the site even though the upload itself succeeds. if ENV['CI'] - require 'codecov' - Codecov::SimpleCov::Formatter.new.format(result) + require 'simplecov_json_formatter' + SimpleCov::Formatter::JSONFormatter.new.format(result) end # Iterate through files using SimpleCov's own 'missed_lines' logic diff --git a/test/test_helper.rb b/test/test_helper.rb index d093e1457..45fccceda 100644 --- a/test/test_helper.rb +++ b/test/test_helper.rb @@ -15,16 +15,12 @@ # *MUST* state VERY EARLY that we're in the test environment. ENV['RAILS_ENV'] ||= 'test' -# Configure SimpleCov formatting before we start it -if ENV['CI'] - require 'codecov' - SimpleCov.formatters = [ - SimpleCov::Formatter::HTMLFormatter, - SimpleCov::Formatter::Codecov - ] -else - SimpleCov.formatters = SimpleCov::Formatter::HTMLFormatter -end +# Configure SimpleCov formatting before we start it. +# The HTML report is for humans; the machine-readable JSON report uploaded to +# Codecov is written separately by the test:coverage_gaps rake task, since CI +# runs with DEFER_COVERAGE set (which swaps in SimpleFormatter below) and the +# merged result is only complete once all parallel + system runs finish. +SimpleCov.formatters = SimpleCov::Formatter::HTMLFormatter # Start SimpleCov to track coverage # NOTE: If you change SimpleCov configuration (used locally), you may also