cargo test fails on archive from crates.io

[igor-raits]

error: couldn't read src/css-parsing-tests/component_value_list.json: No such file or directory (os error 2)
   --> src/tests.rs:112:20
    |
112 |     run_json_tests(include_str!("css-parsing-tests/component_value_list.json"), |input| {
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: couldn't read src/css-parsing-tests/one_component_value.json: No such file or directory (os error 2)
   --> src/tests.rs:120:20
    |
120 |     run_json_tests(include_str!("css-parsing-tests/one_component_value.json"), |input| {
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: couldn't read src/css-parsing-tests/declaration_list.json: No such file or directory (os error 2)
   --> src/tests.rs:131:20
    |
131 |     run_json_tests(include_str!("css-parsing-tests/declaration_list.json"), |input| {
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: couldn't read src/css-parsing-tests/one_declaration.json: No such file or directory (os error 2)
   --> src/tests.rs:141:20
    |
141 |     run_json_tests(include_str!("css-parsing-tests/one_declaration.json"), |input| {
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: couldn't read src/css-parsing-tests/rule_list.json: No such file or directory (os error 2)
   --> src/tests.rs:149:20
    |
149 |     run_json_tests(include_str!("css-parsing-tests/rule_list.json"), |input| {
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: couldn't read src/css-parsing-tests/stylesheet.json: No such file or directory (os error 2)
   --> src/tests.rs:159:20
    |
159 |     run_json_tests(include_str!("css-parsing-tests/stylesheet.json"), |input| {
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: couldn't read src/css-parsing-tests/one_rule.json: No such file or directory (os error 2)
   --> src/tests.rs:169:20
    |
169 |     run_json_tests(include_str!("css-parsing-tests/one_rule.json"), |input| {
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: couldn't read src/css-parsing-tests/stylesheet_bytes.json: No such file or directory (os error 2)
   --> src/tests.rs:197:24
    |
197 |     run_raw_json_tests(include_str!("css-parsing-tests/stylesheet_bytes.json"),
    |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: couldn't read src/css-parsing-tests/color3.json: No such file or directory (os error 2)
   --> src/tests.rs:348:21
    |
348 |     run_color_tests(include_str!("css-parsing-tests/color3.json"), |c| c.ok().to_json())
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: couldn't read src/css-parsing-tests/color3_hsl.json: No such file or directory (os error 2)
   --> src/tests.rs:354:21
    |
354 |     run_color_tests(include_str!("css-parsing-tests/color3_hsl.json"), |c| c.ok().to_json())
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: couldn't read src/css-parsing-tests/color3_keywords.json: No such file or directory (os error 2)
   --> src/tests.rs:361:21
    |
361 |     run_color_tests(include_str!("css-parsing-tests/color3_keywords.json"), |c| c.ok().to_json())
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: couldn't read src/css-parsing-tests/An+B.json: No such file or directory (os error 2)
   --> src/tests.rs:367:20
    |
367 |     run_json_tests(include_str!("css-parsing-tests/An+B.json"), |input| {
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: couldn't read src/css-parsing-tests/urange.json: No such file or directory (os error 2)
   --> src/tests.rs:377:20
    |
377 |     run_json_tests(include_str!("css-parsing-tests/urange.json"), |input| {
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: couldn't read src/css-parsing-tests/component_value_list.json: No such file or directory (os error 2)
   --> src/tests.rs:403:20
    |
403 |     run_json_tests(include_str!("css-parsing-tests/component_value_list.json"), |input| {
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to 14 previous errors

[emilio]

I think this was done on purpose on a9497fb to not bloat the archive

[emilio]

(Not that this is necessarily wontfix I guess, just pointing that out)

This was done in #116 to help with https://bugzilla.mozilla.org/show_bug.cgi?id=1336655.

[SimonSapin]

@ignatenkobrain Why is it useful to run tests on the crates.io archive? PRs are merged into master after tests pass on Travis-CI.

[igor-raits]

@SimonSapin, it is useful to verify that crate package within distribution works correctly.

@emilio, I don't really against excluding test files. But if you do that, please make sure that such tests are not run instead of having compilation errors.

[SimonSapin]

@ignatenkobrain There are a few unit tests that don’t rely on these JSON files, so they could run from the crates.io archive. I’m not convinced that doing that (instead of just verifying that the crate builds) is worthwhile, but I’d take a PR that adds a json-tests Cargo feature enabled by default and adds corresponding #[cfg(feature = "json-tests")] attributes on the relevant tests.

That way you could run cargo test --no-default-features from crates.io archives. To try it without publishing a new version, run cargo package then cd into target/package/cssparser-*/.

[emilio]

Yeah I don't see a much better way to do it than what @SimonSapin suggested above... We could check the existence of the json file, but that'd risk tests actually passing on automation if those are deleted, which is something I'd like to avoid.

[herzenschein]

I'm trying to make an experimental package for cssparser-color for Fedora and experienced this issue.

Distros run tests by default to ensure the package is properly built. For example:

• https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites
• https://docs.fedoraproject.org/en-US/packaging-guidelines/Rust/#_running_tests

The test can be manually disabled for now, but in the long run this would be best resolved upstream.

Edit: actually just realized the existing cssparser package also does this workaround: https://src.fedoraproject.org/rpms/rust-cssparser/blob/rawhide/f/rust-cssparser.spec#_103

[decathorpe]

Just stumbled upon this old bug report again when I tried updating the Fedora package to 0.35.0.

Yeah I don't see a much better way to do it than what @SimonSapin suggested above... We could check the existence of the json file, but that'd risk tests actually passing on automation if those are deleted, which is something I'd like to avoid.

One thing that would make things easier for us would be to replace instances of include_str!(path) with something like std::fs::read_to_string(path).unwrap(). Using the include_str! macro means that the tests don't even compile if the referenced paths are missing.

Loading the file contents at runtime instead means we can compile the tests and just selectively --skip the ones that depend on input data that's not available. This would also not create a hazard that makes tests pass accidentally if the files are missing, because they would still fail (just at run-time, not at build-time).

[emilio]

I'd take such patch :)