Rethinking horizontal dimensions

[climbfuji]

Description

Currently, ccpp_prebuild and ccpp_capgen require using different horizontal dimensions (standard names) depending on the context: host model metadata always uses horizontal_dimension, scheme metadata uses horizontal_loop_extent or horizontal_loop_begin:horizontal_loop_end for the run phase, and horizontal_dimension for all other phases.

This reflects the inner workings of the CCPP physics calls: During the run phase, the physics may be called with horizontal subsets (chunks) of the entire data owned by an MPI task to support cache reuse and/or parallel processing using OpenMP. However, it is not clear why the user or the physics schemes need to know about these details. For a physics developer or a physics scheme, it doesn't matter what mechanism is used internally - all they care about is that an array is passed with a certain horizontal dimension.

Developers and users have complained about this unnecessary and confusing requirement to use one or the other standard name, depending on the context.

Solution

horizontal_dimension should be the only horizontal dimension variable (standard name) ever being used in the dimensions attribute in the metadata for host and scheme metadata, regardless of the phase. The CCPP framework can handle the necessary logic to pass the correct data to the physics scheme for the given context, as long as the host model provides the necessary information in three variables: horizontal_dimension, horizontal_loop_begin, horizontal_loop_end. The latter two need to be set independently for each OpenMP task if the physics run phase is called by multiple OpenMP threads in parallel. Without OpenMP threading and without chunking of data for the physics run phase, the host model simply sets horizontal_loop_begin=1 and horizontal_loop_end=horizontal_dimension.

Note that there is no need to maintain horizontal_loop_extent internally, since this case is covered by horizontal_loop_begin and horizontal_loop_end.

Alternatives (optional)

Do nothing and keep confusing people.

Related to (optional)

We discussed this issue in a CCPP framework meeting several weeks ago, and we agreed that we can just use horizontal_dimension on the user-facing side.

This will be implemented in the next-generation code generator.

[gold2718]

Could you clarify what this means for horizontal_dimension during the run phase? In particular, is it a different number than horizontal_dimension in the init phase?

How does this interact with suite variables which have a single definition of horizontal_dimension during the entire run?

For schemes which currently have an input variable with a standard name of horizontal_loop_extent, would adopting the new parser be as simple as changing the standard name in that scheme's metadata?

[climbfuji]

Could you clarify what this means for horizontal_dimension during the run phase? In particular, is it a different number than horizontal_dimension in the init phase?

Yes and no - it depends on the host model configuration.

What works great is if the host model defines horizontal_dimension (ncol in many of our tests) as a variable in the type = module table (this is renamed to type = host in #759 for clarity; see explanation further down). This variable is also used as the horizontal dimension for allocating suite-owned variables. That means there is no change regarding suite-owned variables - they continue to be allocated for the entire extent of horizontal columns that an MPI task owns.

The second piece of the puzzle is passing the actual horizontal range in the call ccpp_<phase-descriptor>() argument list. These arguments are defined in a new metadata table type = control (basically what is type = host in capgen now).

Examples:

• https://github.com/climbfuji/ccpp-framework/blob/020415383c357b37071a9c8b41f2f01298b0f550/end-to-end-tests/capgen_ng/test_host.F90#L206 for ccpp_physics_timestep_init
• https://github.com/climbfuji/ccpp-framework/blob/020415383c357b37071a9c8b41f2f01298b0f550/end-to-end-tests/capgen_ng/test_host.F90#L256 for ccpp_physics_run using chunking and multiple OpenMP threads in parallel

The corresponding metadata file is https://github.com/climbfuji/ccpp-framework/blob/feature/capgen-ng/end-to-end-tests/capgen_ng/test_host.meta lines 8 onwards (please ignore lines 1-7, they are no longer needed).

The neat thing about this is that runtime control parameters that are dependent on instance number, OpenMP thread, chunk to be processed, etc. are being passed as arguments to the calls. Hence the renaming from type = host to type = control. This provides a lot of flexibility for calling physics. It's essentially your idea, but with a different name of the table that reflects the purpose of these control parameters.

Alongside, renaming type = module to type = host makes sense (to me ...) because module isn't descriptive about who owns this. A module could be anywhere in a ccpp-physics repo next to a scheme. type = host makes it clear that this is data the host owns. This is really just a rename, it doesn't change the fundamental concept on which the capgen metadata tables were built.

How does this interact with suite variables which have a single definition of horizontal_dimension during the entire run?

See above - the suite variables remain as they are.

For schemes which currently have an input variable with a standard name of horizontal_loop_extent, would adopting the new parser be as simple as changing the standard name in that scheme's metadata?

Correct. I've added a legacy mode switch --legacy-mode that translates horizontal_loop_extent to horizontal_dimension when parsing metadata. It prints a big, fat header upon invocation of the code generator that the legacy mode is enabled and that this is a temporary feature. In a single, confined pull request, we can rename horizontal_loop_extent to horizontal_dimension recursively with sed and turn off the --legacy-mode.

A note on the number of arguments for ccpp_<phase-descriptor>. We have flexibility here regarding what we consider as mandatory and optional. I am usually in favor of being explicit rather than "if it's not provided, we are just going to assume something". That being said, the above examples don't have arguments that control instances. These become mandatory if and only if the host model uses this dimension in any of its metadata. We can take a similar approach for other variables that are currently explicit - I am open to this and looking for input.