Design MVP for distribution-based regression detection

[gevtushenko]

This issue can be closed with a design for a decision tree that answers the following questions:

• is the data good enough to answer the question below?
• should developer look at the comparison to debug regression or not?

[oleksandr-pavlyk]

I think our pipe dream is to find a criterion that would reduce misclassifications when we know two benchmarks use identical kernels (same SASS).

Under these conditions and with repeated runs of the benchmark, use of this criterion should greatly reduce the fraction of times comparison decides that one run is faster/slower than another compared to the current behavior.

This issue should be closed when such a reduction can be demonstrated.

The criterion should

• allow arriving at FAST/SLOW decision from JSON aggregate values.
• should use full datasets, if available, in cases when FAST/SLOW can not be decided from aggregates
• classify not-fast and not-slow as either same or uncertain

---

The logic used by base-line nvbench_compare.py is described in #316 (comment).

A collection of datasets, with scripts and Jupyter notebooks is checked into internal GitLab repo opavlyk/nvbench-data-compare-study.