How to handle a split brain?

Question

I read in the Orleans FAQ when a split brain could happen, but I don’t understand what could happen badly and how to handle it properly.

Frequently asked questions say something vague:

You just need to consider the rare possibility of having two instances of an actor while writing your application.

But how should I consider this and what can happen if I do not?

Orleans Paper ( http://research.microsoft.com/pubs/210931/Orleans-MSR-TR-2014-41.pdf ) says the following:

application

can rely on external persistent storage to provide greater data consistency.

But I do not understand what this means.

Suppose a brain failure occurs. Now I have two copies of one grain. When I send several messages, they can be received by these two (or maybe even more?) Different copies. Assume that each instance had the same state before receiving these messages. Now, after processing these messages, they have different states.

How should they persist in their states? There may be a conflict.

When other instances are destroyed and only one remains, what will happen to the states of the destroyed instances? Will it look like the messages they processed were never processed? Then the client state and server state can be desynchronized by IIUC.

I see this (split brain) as a big problem, and I don't understand why there is so little attention to it.

Answer 1

Orleans uses the integrity of the storage provider. When you call this.WriteStateAsync() from the grain, the storage provider ensures that the grain has seen all previous records. If this is not the case, an exception is thrown. You can catch this exception and call DeactivateOnIdle() and rebuild the exception or call ReadStateAsync() and try again. Therefore, if you have 2 grains during a split-brain script that ever calls WriteStateAsync() , it first prevents another state from being written by first reading the most current state.

Update: starting with Orleans v1.5.0, a kernel that allows an InconsistentStateException to InconsistentStateException returned to the caller will be automatically deactivated when current calls are completed. The mirror can catch and handle the exception to avoid automatic deactivation.