Correlate Message after failover

• Documented failure cases for engine and stream processor. I think almost all possible failure cases I can think of we already handle, except problems on reading, which I think can't be handled.
• Checked what the current issue is with the automated chaos experiments. It seems it is a infra problem. You can check the discussion in #infra. It might be affected due to Infra-1292
• Run a chaos experiment, where we correlate a message after fail over.

Chaos Experiment

• Start our normal setup and deploy a workflow with an intermediate message catch event.
• Publish a message and kill a random broker.
• Create a workflow instance and await the result.

I did this experiment several times and it works without any problems, as far as I can tell. First I was wondering that the message was only correlated to one instance, but this seems to be expected message-correlation.html#message-cardinality So learned something new today about our messages 😁.

I prepared already an automatable chaos experiment for that. Have to fine tune it a bit.

No pictures today.