With Zeebe now supporting the addition and removal of brokers to a running cluster, we wanted to test three things:
TL;DR; Scaling up works even under high load and has low impact on processing performance. After scaling is complete, processing performance improves in both throughput and latency.
We experimented with the first version of dynamic scaling in Zeebe, adding or removing brokers for a running cluster.
Scaling up and down is a high-level operation that consists of many steps that need to be carried co-operatively by all brokers in the cluster. For example, adding new brokers first adds them to the replication group of the assigned partitions and then removes some of the older brokers from the replication group. Additionally, priorities need to be reconfigured to ensure that the cluster approaches balanced leadership eventually.
This orchestration over multiple steps ensures that all partitions are replicated by at least as many brokers as configured with the replicationFactor.
As always, when it comes to orchestrating distributed systems, there are many edge cases and failure modes to consider.
The goal of this experiment was to verify that the operation is resilient to broker restarts. We can accept that operations take longer than usual to complete, but we need to make sure that the operation eventually succeeds with the expected cluster topology as result.
TL;DR; Both scaling up and down is resilient to broker restarts, with the only effect that the operation takes longer than usual to complete.
In this chaos day we experimented with the worker count, since we saw recently that it might affect the performance (throughput) negatively if there are more workers deployed. This is related to #7955 and #8244.
We wanted to prove, that even if we have more workers deployed the throughput of the process instance execution should not have an negative impact.
TL;DR; We were not able to prove our hypothesis. Scaling of workers can have a negative impact on performance. Check out the third chaos experiment.
In this chaos day we wanted to prove the hypothesis that the throughput should not significantly change even if we have bigger state, see zeebe-chaos#64
This came up due observations from the last chaos day. We already had a bigger investigation here zeebe#7955.
TL;DR; We were not able to prove the hypothesis. Bigger state, more than 100k+ process instances in the state, seems to have an big impact on the processing throughput.