Test Strategy

The strategy of test execution targeting NVMe compliance is to facilitate the following requirements to coexists:

1. Once a test case is assigned a reference number it does not have to remain at that reference number in the next release.
2. The test numbering indicates the various levels of test dependencies.
3. Allow hardware vendors to specify which NVMe compliance specification to test against.
4. Create a framework to allow adding new tests to be as simple as reasonably possible.
5. Enforce good coding practices to assure each tests case is well documented.

Test Case Numbering

Before compliance suite tagged release 1.2.0 (approx 3-2-2012), there was a rule which stated that once a test was assigned a reference number forever it must remain at that number. This restriction has been removed. The worry was that allowing a test reference number to change would cause the inability to track issues between compliance suite releases. Thus the test numbers must be constant throughout time. However, this isn't strictly necessary, because a test reference number along with tnvme's revision number will uniquely specify an exact failing test throughout all of time. It may also be prudent to track dnvme's revision number, some day canned test cases will reside within dnvme, as the "hybrid" approach is invoked.

Test Dependencies

Test case numbering, starting with compliance suite tagged release 1.2.0 (approx 3-2-2012), will now indicate test case dependency requirements. Simply by knowing the test case number implies the dependencies which must also be executed to expect the test case to pass.

Test Groups
Test Numbering

Specify Test Compliance

As hardware vendors embark on a development cycle, they may most likely choose the most recently released NVMe compliance specification to conform against. See the NVMe home page at http://www.nvmexpress.org. However, the NVMe Working Group won’t stop progress and it could be that a number of additional specification revisions are generated during the time it takes a hardware vendor to complete its development cycle. It should be evident that creating tests to always comply with the latest released NVMe specification won’t be suitable. So tnvme attempts to allow the execution of tests against a specific release via the command line option --rev. This is known as Revision Control.

Revision control filters out tests, data and documentation which are not directly related to the requested revision. The implementation of this feature is the main reason for adopting C++ to encode tnvme. Using polymorphism, developers can override tests of a prior revision to include or exclude the features that have changed.

Revision Control Naming
Revision Control Implementation

The Framework

One goal of this design is to create a framework for which new test cases may be added with little difficulty.

Enforce Documentation

Documentation is not something well liked within the development community. More often than not it is seen as a burden. However, if many developers will potentially be creating a global compliance suite then documentation will be vital to communication. The task of figuring out how to alleviate this burden was at the heart of consideration when developing a framework.

It was realized that any documentation and the source code behind the test would have the best chance to correlate correctly, if they were placed within the same file. The most reasonable place for this is the the C++ source file, *.cpp. There are 3 components to test case documentation.

1. Short description of the test.
       * This is limited to 66 characters.
2. Long description of the test.
       * No character limitation enforced.
3. The NVMe specification revision and specification section which the test case targets mus be specified.

Therefore as part of writing the core logic behind a test, a developer must update the test description within the class constructor. The framework handles the automated mechanism to extract and display this data to the user for each and every test in the same manner. See command line options --summary and --detail.

Further documentation is available via doxygen execution. The entire framework is documented to a fault. Every method within every class is documented. Each input, return value, special notes of usage and general idea of why the method exists is available for the entire framework. If a method is found not documented then it is deemed that documentation would clutter the interface because the method name makes it blatantly obvious what action it performs. The tnvme/Makefile has a “doc” target to generate the doxygen output. The tnvme/classHierarchy.sh script is available to demonstrate how to view the resulting doxygen output files. The doxygen solution allows dynamic creation of C++ class level inheritance diagrams and so it is a good place to start learning the source code of tnvme.