Replace ccpp-prebuild and ccpp-capgen v0 with ccpp-capgen v1

[climbfuji]

Description

This PR replaces the two code generators ccpp-prebuild and ccpp-capgen (version 0) with the new, unified ccpp-capgen (version 1). It supports all host models (CAM-SIMA, CCPP-SCM, NEPTUNE, UFS) and combines the features from ccpp-capgen v0 with the performance and operational robustness of ccpp-prebuild.

This pull request will be updated continuously with links to updates of the CCPP technical documentation and additional information. See also #759 for background information and further materials.

User interface changes?: Yes (see documentation)

This feature branch feature/capgen-v1 will support the three ccpp-prebuild-based models CCPP-SCM, NEPTUNE, and UFS, which will transition to ccpp-capgen v1 before CAM-SIMA. Once CAM-SIMA transitions, this PR will be merged and feature/capgen-v1 will be locked.

Todo

• The LLM-generated documentation and materials in doc/ need to be reviewed and used to update the existing CCPP technical documentation (https://github.com/ncar/ccpp-doc). Once that process is complete, we can remove or trim down the additional documentation in doc/.
• The LLM-generated code should be reviewed file-by-file. LLMs are known for their verbosity, and we can likely remove or trim down sizeable part sof the code. Open question: Should we do the same for the unit tests?

Issues

• Closes Rethinking horizontal dimensions #760
• Closes Weird Capgen fortran parsing error? #757
• Closes Bug in capgen (test/capgen_test) ? surface_air_pressure outvar in ddt_suite #755
• Closes Document --kind-type in ccpp_capgen.py CLI/help, and consider reversing the syntax for external kind modules #754
• Closes Remove unused metadata attribute state_variable #753
• Closes capgen: Add capability and test for passing mpi communicator as a flat field #751
• Closes capgen: case-sensitivity when comparing metadata to Fortran #746
• Closes Bug in scripts/file_utils.py / move_modified_files routine #744
• Closes Unallocated fields passed into Group Caps #690
• Closes Capgen performance when using many suites #688
• Closes Add support for MPI_Comm (mpi08) object in capgen #686
• Closes Transition UFS weather model to Capgen #682
• Closes Transition CCPP-SCM to Capgen #680
• Closes Transition NEPTUNE to capgen #679
• Closes Enhanced/Expanded Testing  #595
• Closes capgen's Fortran parser does not allow array bounds which are not Fortran identifiers #588
• Closes Add "number of instances" dimension to module-level variables to support multi-instance usage #561
• Closes capgen: check handling of type attribute for ddts #516
• Working towards Simplification of ccpp-framework code #737

To check/may not be implemented yet:

• capgen Fortran parser does not accept internal DDTs as part of a registered DDT #749
• capgen: Invalid character declaration, 'character(len=nf_file_length) ' #748
• capgen: dimension specifier foo(N) is not the same as foo(1:N) in Fortran code #747
• Errors when integer variable is output from one phase and used as an interstitial dimension in another #681
• CCPP Framework to provide API calls for suite inspection by host #380
• Fortran/metadata comparison incorrectly reporting bug when comparing Fortran variable declarations and metadata for derived types #360

Testing

Python docstring tests, unit tests, end-to-end tests

Significantly expanded test coverage with Python docstring tests and 1500-ish unit tests (./unit-tests).

End-to-end tests cover:

• Former ccpp-capgen v0 tests:
  • advection
  • advection_auto_clone (support legacy constituents auto-cloning from metadata)
  • capgen_ng
  • ddthost
  • nested_suite
  • var_compat
• Former ccpp-prebuild tests:
  • chunked_data
  • blocked-data not supported in capgen v0 and capgen v1
  • opt_arg
• Newly added features/tests:
  • multi-instances
  • multi-instances + constituents
  • suite_allocate (exercises suite-owned variables with various types of runtime-known dimensions)
  • constituents_dim (suite- and host-owned variables with dimensions from constituents objects)

CAM-SIMA

Python unit tests pass, system tests pass with Intel and GNU. The system tests are bit-for-bit identical with the reference baseline. Two tests fail that also fail on development.

Pull requests:

• Feature/capgen ng climbfuji/CAM-SIMA#1
• Feature/capgen ng climbfuji/atmospheric_physics#1

Status of pull requests: DRAFT

CCPP-SCM

All suites compile and all manual tests run (a sample of all permutations). CI testing/reproducibility: add info here

Pull requests:

• Transition CCPP framework from ccpp-prebuild to ccpp-capgen v1 ccpp-scm#695
• Replace ccpp-prebuild and ccpp-capgen v0 with ccpp-capgen v1 #762 (this PR)
• Updates for ccpp-capgen v1 ccpp-physics#1221
• Updates for ccpp-capgen v1 TEMPO#31

Status of pull requests: UNDER REVIEW

NEPTUNE

All 500 tests compile and run. Results and performance are identical with the ccpp-prebuild code for regular (low-altitude), mid-altitude, and high-altitude physics.

Pull requests in NRL Enterprise GitHub cannot be referenced here.

Status of pull requests: UNDER REVIEW

UFS

ADD INFO AND LINKS TO PRS HERE