How to send messages between distributed state machines?

[muwaqar]

I was wondering how do you generally recommend sending messages between state machines that are running across different machines?

In addition, in our scenario, the state machines are run as asynchronous operations in response to a user request, and they are released from the CPU contention as they enter a waiting state. Periodically, they may be rehydrated to see if any progress can be made.

In this case, is the recommendation to keep polling the other system/state-machine for task completion? or serialize all events to a state machine to some storage which it can monitor?

[akashlal]

Hey @muwaqar, you are free to choose any communication mechanism for sending messages. We only recommend hiding it behind an interface so that you can supply a mock for testing. See the Raft sample that uses Azure Service Bus for communication but also uses a mock for running tests with multiple "remote" state machines. The mock, in some sense, should capture the messaging guarantee that the real system provides. Use the messaging system that makes most sense for your application/service.

I perhaps don't fully understand your scenario, but let me mention a couple of things. A Coyote StateMachine does not hold any compute resources when it has nothing to do (e.g., its inbox is empty). It only holds on to its in-memory object. If you want to further reclaim its memory then yes, should could store it away and rehydrate when work arrives. Otherwise you can keep it alive. It will wake up as soon as you send a message to it.

Polling can be easy to implement sometimes. Another option is to send out a message on task completion; the other side just waits for this message to arrive. But again, perhaps I don't follow all the details here.

[muwaqar]

This clears up things in my mind; regardless of however we send the message, the underlying nature of the messaging system can be captured in a mock and can be tested. I'll study the Raft sample too.

Thanks for the help.