Inconsistency between extension attributes for SDTM and CRF specializations

[lexjansen]

In the latest USDM JSON file there is an inconsistency between the reference to the SDTM dataset specialization and the CRF specialization.
Where the SDTM reference lists the endpoint to the latest version of the specialization:
/mdr/specializations/sdtm/datasetspecializations/{specialization}
the CRF reference lists a specifiic package:
/mdr/specializations/crf/packages/{package}/specializations/{specialization}
Example:

"id": "BiomedicalConcept_20",
"extensionAttributes": [
  {
    "id": "ExtensionAttribute_9",
    "url": "http://www.cdisc.org/usdm/extensions/specializations/sdtm",
    "valueString": "/mdr/specializations/sdtm/datasetspecializations/TEMP",
    "instanceType": "ExtensionAttribute"
  },
  {
    "id": "ExtensionAttribute_145",
    "url": "http://www.cdisc.org/usdm/extensions/specializations/crf",
    "valueString": "/mdr/specializations/crf/packages/2026-06-30/specializations/TEMP_DENORMALIZED",
    "instanceType": "ExtensionAttribute"
  }
],

The reason for this inconsistency is because in src/soa_builder/web/app.py different endpoints are being used.
SDTM:
{base_prefix}/mdr/specializations/datasetspecializations?biomedicalconcept={code}
or
{base_prefix}/mdr/specializations/sdtm/datasetspecializations

CRF first finds the latest package:
{base_prefix}/mdr/specializations/crf/packages
and then gets the CRF list:
{base_prefix}/mdr/specializations/crf/packages/{package}/specializations
The list of returned CRF specializations will have the packagedate in the endpoint.

CRF can use the same endpoint as SDTM:
{base_prefix}/mdr/specializations/datasetspecializations?biomedicalconcept={code}
It will return both SDTM and CRF specializations:

{
  "_links": {
    "datasetSpecializations": {
      "sdtm": [
        {
          "href": "/mdr/specializations/sdtm/datasetspecializations/AE",
          "title": "Adverse Event Free Text Format",
          "type": "SDTM Dataset Specialization"
        },
    ...
      ],
      "crf": [
        {
          "href": "/mdr/specializations/crf/specializations/AE_DENORMALIZED",
          "title": "Adverse Event Free Text Format (Denormalized)",
          "type": "CRF Specialization"
        },
    ...
      ]
    },
    "self": {
      "href": "/mdr/specializations/datasetspecializations?biomedicalconcept=C179175",
      "title": "Dataset Specializations that specialize a Biomedical Concept",
      "type": "Dataset Specializations List"
    },
    "parentBiomedicalConcept": {
      "href": "/mdr/bc/biomedicalconcepts/C179175",
      "title": "Solicited Adverse Event",
      "type": "Biomedical Concept"
    }
  },
  "name": "Dataset Specializations (latest version)",
  "label": "Dataset Specializations List"
}

And for the list of latest versions of CRF, use:
{base_prefix}/mdr/specializations/crf/specializations

This would the result in:

"id": "BiomedicalConcept_20",
"extensionAttributes": [
  {
    "id": "ExtensionAttribute_9",
    "url": "http://www.cdisc.org/usdm/extensions/specializations/sdtm",
    "valueString": "/mdr/specializations/sdtm/datasetspecializations/TEMP",
    "instanceType": "ExtensionAttribute"
  },
  {
    "id": "ExtensionAttribute_145",
    "url": "http://www.cdisc.org/usdm/extensions/specializations/crf",
    "valueString": "/mdr/specializations/crf/specializations/TEMP_DENORMALIZED",
    "instanceType": "ExtensionAttribute"
  }
],

[pendingintent]

Thank you for the detailed description. I will fix this in branch pi-fix-crf-spec-url and backfill the incorrect values in the database.

[pendingintent]

Fixed the SDTM/CRF endpoint inconsistency by rewriting fetch_crf_specializations() in app.py to mirror fetch_sdtm_specializations():

• Added a code param that hits the same combined endpoint SDTM uses (/mdr/specializations/datasetspecializations?biomedicalconcept={code}), extracting the crf HAL link array instead of sdtm, with its own per-code cache.

• Replaced the two-step packages→specializations lookup with a single call to /mdr/specializations/crf/specializations (the "latest version" list), matching SDTM's pattern.

• routers/activities.py's ui_crf_options now passes code=concept_code, scoping the CRF dropdown to the activity's biomedical concept like the SDTM/DSS dropdown already does.

• Added _migrate_backfill_crf_href_latest_version in migrate_database.py, an idempotent migration that rewrites existing activity_concept_crf.crf_href values from the old dated-package format to the new format, registered to run at startup.

Added tests: tests/test_fetch_crf_specializations.py (code-branch and latest-list-branch behavior) and tests/test_migrate_backfill_crf_href.py (backfill correctness/idempotency). Full suite: 633 passed; black/flake8 clean.