Data quality checks

[tomoinn]

I'm coding against the YAML files and turning up a lot of errors which could be detected with some integrity checks, mostly in the CC definitions but there are some basic things like numbers sometimes appearing as strings as well (easy to work around coding defensively but ideally wouldn't be there)

I'm working on a fork and happy to send a PR when I'm done including the QA code, but right now there are a lot of definitions which are obviously incorrect, having overlapping duplicate CCs which are easy to detect (and which are making my code throw a fit, correctly). Multiple CC for a single parameter is fine if the ranges don't overlap (and also some ranges are just random text fields i.e. '>0' or '10-120' rather than numeric).

[tomoinn]

(this might sound like a lot of complaining from me, I'm genuinely glad this exists though!)

[tomoinn]

https://github.com/Fenland-Pedals/openmidi/blob/master/openmidi-tooling/src/openmidi_tooling/__init__.py is the little library I've written to help with this. It attempts to merge multiple CCs and reports on conflicts. It also supports the additional cc_groups key I've added to some of the .yaml files to help get better merge results, although the heuristics actually do a pretty good job now.

The following code (in build_strata_metadata.py) loads everything from yaml and json files and complains bitterly about everything that's broken. If you run this code without merging my data changes you'd get even more. I'm aiming to fix all the unknown naming convention errors it produces manually but there are also data quality issues in terms of incorrect CCs (some of which are incorrect because they're NRPN in reality)

from pathlib import Path

from openmidi_tooling import OpenMidi


def main() -> None:
    # Define paths relative to the script directory
    data_dir = Path(__file__).parent.parent.parent / "data"
    om = OpenMidi(data_dir)
    for brand in om.brands:
        print(f"Processing brand: {brand.name}")
        for model in brand.models:
            print(f" - Processing model: {model.name}")
            if model.errors:
                for error in model.errors:
                    print(f"   - Error: {error}")


if __name__ == "__main__":
    main()

Produces a bunch of output - the load process at the moment prints out the merge logic, i.e.

...
Merging ['Cab Left - Fender', 'Cab Left - Marshal JTM50', 'Cab Left - Vox', 'Cab Left - Mesa', 'Cab Left - Kerry Wright', 'Cab Left - Marshal 4x12'] -> name: 'Cab Left' desc: '0-1 : Fender, 2-3 : Marshal JTM50, 4-5 : Vox, 6-7 : Mesa, 8-9 : Kerry Wright, 10-11 : Marshal 4x12'
Merging ['Cab Right - Fender', 'Cab Right - Marshal JTM50', 'Cab Right - Vox', 'Cab Right - Mesa', 'Cab Right - Kerry Wright', 'Cab Right - Marshal 4x12'] -> name: 'Cab Right' desc: '0-1 : Fender, 2-3 : Marshal JTM50, 4-5 : Vox, 6-7 : Mesa, 8-9 : Kerry Wright, 10-11 : Marshal 4x12'
Merging ['Amp Left - Fullerton', 'Amp Left - London', 'Amp Left - Dartford', 'Amp Left - Red', 'Amp Left - Citrus', 'Amp Left- Tread'] -> name: 'Amp' desc: '0 : Left - Fullerton, 1 : Left - London, 2 : Left - Dartford, 3 : Left - Red, 4 : Left - Citrus, 5 : Left- Tread'
Merging ['Amp Right - Fullerton', 'Amp Right - London', 'Amp Right - Dartford', 'Amp Right - Red', 'Amp Right - Citrus', 'Amp Right - Tread'] -> name: 'Amp Right' desc: '0 : Fullerton, 1 : London, 2 : Dartford, 3 : Red, 4 : Citrus, 5 : Tread'
Merging ['Room Select Room', 'Room Select Hall', 'Room Select Spring'] -> name: 'Room Select' desc: '0 : Room, 1 : Hall, 2 : Spring'
Merging ['Tweak Switch: MOD', 'Tweak Switch: TONE', 'Tweak Switch: AGE'] -> name: 'Tweak Switch' desc: '0-42 : MOD, 43-85 : TONE, 86-127 : AGE'
Merging ['A|B|C Switch: A', 'A|B|C Switch: B', 'A|B|C Switch: C'] -> name: 'A|B|C Switch' desc: '0-42 : A, 43-85 : B, 86-127 : C'
...

(mostly so I can see what it's doing and look for mistakes)

The main loop above prints out any validation errors associated with each brand / model, i.e.

Processing brand: DigiTech
 - Processing model: Bass Whammy
 - Processing model: Whammy
 - Processing model: Whammy DT
Processing brand: Dimehead
 - Processing model: NAM
   - Error: CC range collision detected for CC18 : [CC(value=18, name='Mids', min=0, max=127, description=''), CC(value=18, name='Mids', min=0, max=127, description='')]
   - Error: CC range collision detected for CC21 : [CC(value=21, name='FX Loop OFF', min=0, max=0, description=''), CC(value=21, name='FX Loop ON', min=127, max=127, description=''), CC(value=21, name='Disable Boost/FX', min=0, max=0, description='')]
Processing brand: Disaster Area
 - Processing model: DPC Micro
 - Processing model: DPC-8EZ Gen2
 - Processing model: Micro Clock Gen4
 - Processing model: NNB Expanse Bridge

It may not be immediately obvious, but this shows the Dimehead NAM has some duplicate CC metadata (kind of, the names are different but the intent looks obviously the same), there are quite a few like that in there. Hard to pick up accurately automatically though!