Skip to content

[CI Broken] Multiple test failures on Julia 1.0 and 1.6 #16

Description

@ChrisRackauckas-Claude

CI Health Check: Tests Failing

Summary

The CI is currently broken on the main branch. Automated health check PR #14 revealed multiple failures across different Julia versions.

Failed Jobs

  1. Julia 1.0 on ubuntu-latest: Threading configuration error
  2. Julia 1.6 on ubuntu-latest: Visual regression test failures (3 errored tests)

Detailed Analysis

Julia 1.0 Failure

Root Cause: Invalid threading configuration in CI workflow

Error Message:

ERROR: julia: -t,--threads=<n>[,auto|<m>]; n must be an integer >= 1
Package FiniteVolumeMethod1D errored during testing

Issue: The CI workflow file (.github/workflows/CI.yml) sets JULIA_NUM_THREADS: ${{ matrix.julia-threads }} on line 18, but matrix.julia-threads is never defined in the matrix configuration (lines 22-28). This passes an empty value to Julia's -t flag, causing the error.

Suggested Fix: Either:

  • Remove the JULIA_NUM_THREADS environment variable if threading is not needed
  • Add julia-threads to the matrix configuration with appropriate values

Julia 1.6 Failure

Root Cause: Visual regression test failures due to reference image mismatches

Test Results: 67 passed, 0 failed, 3 errored, 0 broken

Failed Tests:

  1. Porous-Medium - Reference image mismatch at docs/src/figures/porous_surface.png
  2. Robin Diffusion - Reference image mismatch at docs/src/figures/robin_diffusion_surface.png
  3. Reaction-Diffusion - Reference image mismatch at docs/src/figures/dirichlet_source_surface.png

Error Message:

LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")',
or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE`
to "true" and re-run the tests via Pkg.

Possible Causes:

  • Upstream dependency changes affecting plot rendering
  • Julia version differences in plotting libraries
  • Platform-specific rendering differences

Suggested Fix Approach:

  1. Run tests locally on Julia 1.6 to reproduce the issue
  2. Review the actual vs reference images to determine if differences are acceptable
  3. If differences are minor/acceptable: Update reference images by setting JULIA_REFERENCETESTS_UPDATE=true
  4. If differences indicate a real regression: Investigate changes in dependencies or plotting code

Links

Priority

High - CI is broken on main branch, blocking development and future PRs.

cc @ChrisRackauckas

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions