Scheduled Halo Exchange

[philip-paul-mueller]

This PR introduces the scheduled exchange feature from GHEX into ICON4Py.

These exchange allows to call the exchange function before all work has been completed, i.e. the exchange will wait until the previous work is done. A similar feature is the "scheduled wait", that allows to initiate the receive without the need to wait on its completion.

In addition to this the function also renamed the functions related to halo exchange:

• exchange() was renamed to start().
• wait() was renamed to finish() (that might now return before the transfer has fully concluded).
• exchange_and_wait() was renamed to exchange().

All of these functions now accepts the an argument called stream, which defaults to DEFAULT_STREAM. It is indicate how synchronization with the stream should be performed.
In case of start() it means that the actual exchange should not start until all work previously submitted to stream has finished. For finish() it means that further work, submitted to stream, should not start until the exchange has ended. For finish() it is also possible to specify BLOCK, which means that finish() waits until the transfer has fully finished.

The orchestrator was not updated, but the change were made in such a way that it continues to work in diffusion, although using the original, blocking behaviour.

Note:
The CI fails for cscs/extra, but it also does this for current main, see See this test PR: #982

[philip-paul-mueller]

cscs-ci run default

[philip-paul-mueller]

cscs-ci run extra

[philip-paul-mueller]

cscs-ci run default

[philip-paul-mueller]

cscs-ci run dace

[philip-paul-mueller]

cscs-ci run extra

[philip-paul-mueller]

cscs-ci run default

[philip-paul-mueller]

cscs-ci run default

[philip-paul-mueller]

cscs-ci run dace

[philip-paul-mueller]

cscs-ci run extra

[philip-paul-mueller]

cscs-ci run default

[philip-paul-mueller]

cscs-ci run dace

[philip-paul-mueller]

cscs-ci run extra

[philip-paul-mueller]

There is a failing in extra, however, this error is also present on main.

See this test PR: #982

[philip-paul-mueller]

cscs-ci run default

[philip-paul-mueller]

cscs-ci run dace

[msimberg]

cscs-ci run default

[msimberg]

cscs-ci run distributed

[msimberg]

cscs-ci run default

[msimberg]

cscs-ci run distributed

[havogt]

Suggested change

	class Block:
	class BlockType:

to avoid accidentally passing Block instead of BLOCK. Or alternatively make call it _Block and use type[BLOCK] as annotation? Not sure which option is best, but currently it's just too tempting to pass Block...

[havogt]

Actually we need to make this a proper Singelton otherwise we might have prblems if someone does Block()

[havogt]

Maybe this? @egparedes

class BlockType:
    _instance = None

    def __new__(cls):
        if cls._instance is None:
            cls._instance = super().__new__(cls)
        return cls._instance

BLOCK = BlockType()

[philip-paul-mueller]

According to SO this should be the correct way, although the SO answer is way more fancy, but do we need that?

[philip-paul-mueller]

I have implemented it, but improvements are appreciated.

[havogt]

lgtm

[havogt]

cscs-ci run default

[havogt]

cscs-ci run distributed

[havogt]

cscs-ci run default

[havogt]

cscs-ci run distributed

[github-actions]

Mandatory Tests

Please make sure you run these tests via comment before you merge!

• cscs-ci run default
• cscs-ci run distributed

Optional Tests

To run benchmarks you can use:

• cscs-ci run benchmark-bencher

To run tests and benchmarks with the DaCe backend you can use:

• cscs-ci run dace

To run test levels ignored by the default test suite (mostly simple datatest for static fields computations) you can use:

• cscs-ci run extra

For more detailed information please look at CI in the EXCLAIM universe.

request was from an early state. we'll address further cleanup in future PRs.