Decide directory structure for screenshot images

[berland]

[berland]

The current structure is implicitly described in

ert/justfile

Lines 40 to 59 in 969bdaf

	pack_updated_screenshots:
	#!/bin/bash
	staging="updated-screenshots"
	mkdir -p "$staging"
	for test_dir in pytest-mpl_results/*/; do
	# Filenames from pytest-mpl need a lot of massaging in order to
	# reconstruct the directory structure:
	full_dotted_name=$(basename "$test_dir")
	updated_img="${test_dir}/result.png"
	filename="${full_dotted_name##*.}.png"
	path_with_test_func_name="${full_dotted_name%.*}"
	path_dots="${path_with_test_func_name%.*}"
	rel_path="${path_dots//.//}"
	target_dir="$staging/$rel_path/baseline"
	mkdir -p $staging/$rel_path/baseline
	cp "$updated_img" "$target_dir/$filename"
	echo "Mapped $full_dotted_name -> $target_dir/$filename"
	done
	[ -d "/tmp/test_docs_screenshots" ] && cp -r /tmp/test_docs_screenshots/* "$staging"
	find "$staging" -type f -exec ls -l {} +

[eilskra]

Does this approach for naming/reconstructing the directory structure have to remain? Is this a limitation with pytest-mpl?

[berland]

After a minor perturbation to the code triggering some screenshot updates:

$ git clean -xfd
$ <perturb some code>
$ just screenshot-comparison-test
$ find pytest-mpl_results/
pytest-mpl_results/
pytest-mpl_results/tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure7
pytest-mpl_results/tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure7/result.png
pytest-mpl_results/tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure7/result-failed-diff.png
pytest-mpl_results/tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure7/baseline.png
pytest-mpl_results/tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure6
pytest-mpl_results/tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure6/result.png
pytest-mpl_results/tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure6/result-failed-diff.png
pytest-mpl_results/tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure6/baseline.png
pytest-mpl_results/tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure1
pytest-mpl_results/tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure1/result.png
pytest-mpl_results/tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure1/result-failed-diff.png
pytest-mpl_results/tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure1/baseline.png

Here one sees a flat directory structure based on Python package hierarchy in tests, and what we want to store is only the baseline.png file. The current bash code in the justfile maps the package hiearchy into a directory path:

$ just pack-updated-screenshots
Mapped tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure1 -> updated-screenshots/tests/ert/ui_tests/gui/baseline/test_that_plot_images_are_unchanged_plot_figure1.png
Mapped tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure6 -> updated-screenshots/tests/ert/ui_tests/gui/baseline/test_that_plot_images_are_unchanged_plot_figure6.png
Mapped tests.ert.ui_tests.gui.test_plotting_of_snake_oil.test_that_plot_images_are_unchanged_plot_figure7 -> updated-screenshots/tests/ert/ui_tests/gui/baseline/test_that_plot_images_are_unchanged_plot_figure7.png

In the current ert repository this makes some sense, to store the baseline image close the test making it. This becomes less important when moving baselines somewhere else.

[berland]

The directory structure in pytest-mpl_results does not look something we can change, it is well suited for HTML reports of the diff (which we also want).

Where we store the baselines, we do not want this flat directory structure, and neither do we want it, as pytest-mpl is not made to look up existing baselines from such a directory structure. That means we should keep the directory structure more or less as it is today, and we need code like what is done by just pack-updated-screenshots to map the flat directory structure into a deep structure for storage.

If we dislike the bash code too much, or it breaks, then a cleaner solution would be to make a manifest file during test collection writing both filenames --- which would give safer filename remapping.

The workflows the directory structure must support:

• Running pytest-mpl - it assumes baseline png files can be found in a directory called baseline relative to the test filename -- requires a nested structure.
  • This step implies git clone of e.g. ert-testdata and merging a subdirectory of ert-testdata onto the current checked out ert repo - easy when ert-testdata has the same directory structure for baselines.
• Producing HTML image comparison artifacts - could be nested, but it is not when made by pytest-mpl
• Producing Pull request to ert-testdata, makes sense to have the same deep directory structure. This step requires the currently ugly bash code to remap files.
• We have custom screenshot code in test_docs_screenshot.py which is not pytest-mpl.

[berland]

I think we should keep the current setup for baseline file hiearchy when moving baselines to a different repository.