Run doctests in testing suite

[cgarling]

This should run the doctests as part of the test suite. I don't expect this to pass now because of printing / environment stuff that we probably need to be more careful about when writing the doctests.

[giordano]

I don't understand why you need eval: https://github.com/EnzymeAD/Enzyme.jl/blob/c009fa593821cbd9017450a6721259f7dbc6ebf1/test/doctests.jl

[cgarling]

Thanks for the tip. I added the @eval approach to fix failures with Documenter.jl not recognizing the package module (see here). This problem only appeared when I switched to ParallelTestRunner so I may just be configuring something improperly. The errors are things like

│ ┌ Error: Failed to evaluate `CurrentModule = AstrochemicalYields` in `@meta` block.
│ │   exception =
│ │    UndefVarError: `AstrochemicalYields` not defined in `Main`
│ │    Suggestion: check for spelling errors or missing imports.
│ │    Hint: AstrochemicalYields is loaded but not imported in the active module Main.

I fixed this by explicitly importing the package into Main using @eval.

Looking at the code in Enzyme it's not clear to me why this error occurs in my project but not in Enzyme because Enzyme has a similar @meta block in their index.md file. Any tips would be appreciated

[giordano]

JuliaTesting/ParallelTestRunner.jl#68. It's not always necessary, but if the output depends on the exact module where the code is being evaluated, then something like that may be needed.

[icweaver]

Interesting, appreciate this info y'all

[icweaver]

Thanks for opening this, Chris! I think you may be hitting on a more general issue/convention/standard? that I've never really been quite clear on: the distinction between: "CI" tests, and "Documentation" tests.

We currently have these somewhat separated by design with our CI.yml and Documentation.yml in some of our repos, but not all. I assume this is an organizational decision to help separate out concerns, but I could be missing the larger picture that's going on. If we decide to continue with the separate workflows route, then would just adding a doc build badge serve the same purpose that this PR is trying to achieve?

This is something that I have been thinking about exploring more and potentially implementing ecosystem-wide, but just never took the time. Is this something that would be worth chatting about around the fireplace? Apologies if I am misunderstanding the overall intent here

[cgarling]

Without being overly philosophical, my main opinions are

1. Doctests are tests and so should be run with the rest of the test suite. I.e., if a source code change breaks a doctest, that should be considered a test failure and show up in the "CI" action, not the "Documentation" action
2. Since doctests are tests they should be counted in code coverage and should not be duplicated in other test suites (e.g. the main test directory)

For my own packages I usually run doctests in the main CI action (as I'm suggesting here) and then disable them on the "Documentation" action. Basically the "Documentation" action only builds the documentation and all the tests are run in the "CI" action. I find this to be a better separation of concerns (that way running tests and building documentation are truly separated).

If you want to start a fireplace thread you can but I'm also fine with this being a stylistic choice for individual maintainers to make with their repos.

[icweaver]

This is awesome, thanks Chris! These are exactly the kind of convos I've been wanting to have as an org, I'll go ahead and set up a fireplace thread then. While I'm in theory all for folks doing their own thing in their repos, I think there's still a lot of value (at least to new folk) in making these sorts of reasonings more transparent.

And who knows, maybe one day we'll develop our own org style as we become more and more cohesive.

[icweaver]

Currently dealing with module qualification shenanigans between doc runs and test suite runs. I think I am moving back onto the fence for my preference between where doctests should be run from, at least for this particular package.

Will keep playing around with this

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.23%. Comparing base (e3b1ef6) to head (725e88b).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #38      +/-   ##
==========================================
+ Coverage   90.38%   94.23%   +3.84%     
==========================================
  Files           8        8              
  Lines         156      156              
==========================================
+ Hits          141      147       +6     
+ Misses         15        9       -6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[icweaver]

Ok, things are looking green now, but I am not sure if I am loving the duplication of documentation dependencies in the test env, and the conditional juggling of module name qualifications and paths that I needed to introduce.

Am I just overcomplicating things?

[cgarling]

The duplication of dependencies between docs and test doesn't bother me. Generally I only write doctests in docstrings, not in markdown files as is currently done in docs/src/index.md, and I try to keep the doctests small and directed. Would you prefer to make the ```jldoctest block in docs/src/index.md just an @example block instead? That may simplify the setup.

[icweaver]

Ok, that sounds like a reasonable delineation to me. I've switched things over to example blocks and cleaned up a few other sections too