Update design proposal to abstract broker and Ref-DB
The implementation of the message broker and the Ref-DB should
be agnostic of the arbitrary choice of Kafka and Zookeper.
This change updates the documentation with a proposal that aims to
abstract over those specific choices and allow a dynamic binding of
different implementations instead.
Feature: Issue 10828
Change-Id: I026d48bd6c36bf512c119445b9a0d638f805e25b
diff --git a/DESIGN.md b/DESIGN.md
index 2c5a539..0995a14 100644
--- a/DESIGN.md
+++ b/DESIGN.md
@@ -257,8 +257,18 @@
The multi-site solution described here depends upon the combined use of different
components:
-- **multi-site plugin**: Enables the replication of Gerrit _indexes_, _caches_,
- and _stream events_ across sites.
+- **multi-site libModule**: exports interfaces as DynamicItems to plug in specific
+implementation of `Brokers` and `Global Ref-DB` plugins.
+
+- **broker plugin**: an implementation of the broker interface, which enables the
+replication of Gerrit _indexes_, _caches_, and _stream events_ across sites.
+When no specific implementation is provided, then the [Broker Noop implementation](#broker-noop-implementation)
+then libModule interfaces are mapped to internal no-ops implementations.
+
+- **Global Ref-DB plugin**: an implementation of the Global Ref-DB interface,
+which enables the detection of out-of-sync refs across gerrit sites.
+When no specific implementation is provided, then the [Global Ref-DB Noop implementation](#global-ref-db-noop-implementation)
+then libModule interfaces are mapped to internal no-ops implementations.
- **replication plugin**: enables the replication of the _Git repositories_ across
sites.
@@ -277,10 +287,78 @@
## Implementation Details
-### Message brokers
-The multi-site plugin adopts an event-sourcing pattern and is based on an
-external message broker. The current implementation uses Apache Kafka.
-It is, however, potentially extensible to others, like RabbitMQ or NATS.
+### Multi-site libModule
+As mentioned earlier there are different components behind the overarching architecture
+of this solution of a distributed multi-site gerrit installation, each one fulfilling
+a specific goal. However, whilst the goal of each component is well-defined, the
+mechanics on how each single component achieves that goal is not: the choice of which
+specific message broker or which Ref-DB to use can depend on different factors,
+such as scalability, maintainability, business standards and costs, to name a few.
+
+For this reason the multi-site component is designed to be explicitly agnostic to
+specific choices of brokers and Global Ref-DB implementations, and it does
+not care how they, specifically, fulfill their task.
+
+Instead, this component takes on only two responsibilities:
+
+* Wrapping the GitRepositoryManager so that every interaction with git can be
+verified by the Global Ref-DB plugin.
+
+* Exposing DynamicItem bindings onto which concrete _Broker_ and a _Global Ref-DB_
+plugins can register their specific implementations.
+When no such plugins are installed, then the initial binding points to no-ops.
+
+* Detect out-of-sync refs across multiple gerrit sites:
+Each change attempting to mutate a ref will be checked against the Ref-DB to
+guarantee that each node has an up-to-date view of the repository state.
+
+### Message brokers plugin
+Each gerrit node in the cluster needs to be informed and inform all other nodes
+about fundamental events, such as indexing of new changes, cache evictions and
+stream events. This component will provide a specific pub/sub broker implementation
+that is able to do so.
+
+When provided, the message broker plugin will override the dynamicItem binding exposed
+by the multi-site module with a specific implementation, such as Kafka, RabbitMQ, NATS, etc.
+
+#### Broker Noop implementation
+The default `Noop` implementation provided by the `Multi-site` libModule does nothing
+upon publishing and producing events. This is useful for setting up a test environment
+and allows multi-site library to be installed independently from any additional
+plugins or the existence of a specific broker installation.
+The Noop implementation can also be useful when there is no need for coordination
+with remote nodes, since it avoids maintaining an external broker altogether:
+for example, using the multi-site plugin purely for the purpose of replicating the Git
+repository to a disaster-recovery site and nothing else.
+
+### Global Ref-DB plugin
+Whilst the replication plugin allows the propagation of the Git repositories across
+sites and the broker plugin provides a mechanism to propagate events, the Global
+Ref-DB ensures correct alignment of refs of the multi-site nodes.
+
+It is the responsibility of this plugin to store atomically key/pairs of refs in
+order to allow the libModule to detect out-of-sync refs across multi sites.
+(aka split brain). This is achieved by storing the most recent `sha` for each
+specific mutable `refs`, by the usage of some sort of atomic _Compare and Set_ operation.
+
+We mentioned earlier the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem),
+which in a nutshell states that a distributed system can only provide two of these
+three properties: _Consistency_, _Availability_ and _Partition tolerance_: the Global
+Ref-DB helps achieving _Consistency_ and _Partition tolerance_ (thus sacrificing
+Availability).
+
+See [Prevent split brain thanks to Global Ref-DB](#prevent-split-brain-thanks-to-global-ref-db)
+For a thorough example on this.
+
+When provided, the Global Ref-DB plugin will override the dynamicItem binding
+exposed by the multi-site module with a specific implementation, such as Zoekeeper,
+etcd, MySQL, Mongo, etc.
+
+#### Global Ref-DB Noop implementation
+The default `Noop` implementation provided by the `Multi-site` libModule accepts
+any refs without checking for consistency. This is useful for setting up a test environment
+and allows multi-site library to be installed independently from any additional
+plugins or the existence of a specific Ref-DB installation.
### Eventual consistency on Git, indexes, caches, and stream events
@@ -408,22 +486,10 @@
**NOTE**: The two options are not exclusive.
-#### Introduce a `DfsRefDatabase`
+#### Prevent split brain thanks to Global Ref-DB
-An implementation of the out-of-sync detection logic could be based on a central
-coordinator holding the _last known status_ of a _mutable ref_ (immutable refs won't
-have to be stored here). This would be, essentially, a DFS base `RefDatabase` or `DfsRefDatabase`.
-
-This component would:
-
-- Contain a subset of the local `RefDatabase` data:
- - Store only _mutable _ `refs`
- - Keep only the most recent `sha` for each specific `ref`
-- Require that atomic _Compare and Set_ operations can be performed on a
-key -> value storage. For example, it could be implemented using `Zookeeper`. (One implementation
-was done by Dave Borowitz some years ago.)
-
-This interaction is illustrated in the diagram below:
+The above scenario can be prevented by using an implementation of the Global Ref-DB
+interface, which will operate as follows:
@@ -469,17 +535,6 @@
able to differentiate the type of traffic and, thus, is forced always to use the
RW site, even though the operation is RO.
-- **Support for different brokers**: Currently, the multi-site plugin supports Kafka.
- More brokers need to be supported in a fashion similar to the
- [ITS-* plugins framework](https://gerrit-review.googlesource.com/admin/repos/q/filter:plugins%252Fits).
- Explicit references to Kafka must be removed from the multi-site plugin. Other plugins may contribute
- implementations to the broker extension point.
-
-- **Split the publishing and subscribing**: Create two separate
- plugins. Combine the generation of the events into the current kafka-
- events plugin. The multi-site plugin will focus on
- consumption of, and sorting of, the replication issues.
-
## Step-2: Move to multi-site Stage #8.
- Auto-reconfigure HAProxy rules based on the projects sharding policy
@@ -487,5 +542,3 @@
- Serve RW/RW traffic based on the project name/ref-name.
- Balance traffic with "locally-aware" policies based on historical data
-
-- Preventing split-brain in case of temporary sites isolation
diff --git a/images/architecture-first-iteration.png b/images/architecture-first-iteration.png
index 1a9fe36..841ee40 100644
--- a/images/architecture-first-iteration.png
+++ b/images/architecture-first-iteration.png
Binary files differ
diff --git a/images/git-replication-split-brain-detected.png b/images/git-replication-split-brain-detected.png
index dba5a81..a49b9be 100644
--- a/images/git-replication-split-brain-detected.png
+++ b/images/git-replication-split-brain-detected.png
Binary files differ
diff --git a/src/.DS_Store b/src/.DS_Store
new file mode 100644
index 0000000..d549e6c
--- /dev/null
+++ b/src/.DS_Store
Binary files differ
diff --git a/src/main/.DS_Store b/src/main/.DS_Store
new file mode 100644
index 0000000..c068634
--- /dev/null
+++ b/src/main/.DS_Store
Binary files differ
diff --git a/src/main/resources/.DS_Store b/src/main/resources/.DS_Store
new file mode 100644
index 0000000..d714c4e
--- /dev/null
+++ b/src/main/resources/.DS_Store
Binary files differ
diff --git a/src/main/resources/Documentation/.DS_Store b/src/main/resources/Documentation/.DS_Store
new file mode 100644
index 0000000..2c69d28
--- /dev/null
+++ b/src/main/resources/Documentation/.DS_Store
Binary files differ
diff --git a/src/main/resources/Documentation/sources/architecture-first-iteration.xml b/src/main/resources/Documentation/sources/architecture-first-iteration.xml
index 0765bad..c3aa4f9 100644
--- a/src/main/resources/Documentation/sources/architecture-first-iteration.xml
+++ b/src/main/resources/Documentation/sources/architecture-first-iteration.xml
@@ -1,2 +1,2 @@
<?xml version="1.0" encoding="UTF-8"?>
-<mxfile modified="2019-05-09T13:38:37.123Z" host="www.draw.io" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" etag="JKQWieSjNJp7nwGUZ6iM" version="10.6.7" type="google"><diagram name="Page-1" id="5f0bae14-7c28-e335-631c-24af17079c00">7V1Zl6M2Fv419Zg6SEIsj93VS3JOOqnpzpyezMscjFU2Hco4GHdX5dePsBEGXYGxjQRe/FKF2Ox7v7tfSXfk4fnlYxos55+SKYvvsDV9uSPv7jDGFvH4n3zkdTuCEHW2I7M0mhZju4Ev0T+sGLSK0XU0ZavahVmSxFm0rA+GyWLBwqw2FqRp8qN+2VMS19+6DGYMDHwJgxiOfo2m2Vz8DMvanfiZRbN58WqPFicmQfjXLE3Wi+J9d5g8bT7b08+BeFZx/WoeTJMflSHy/o48pEmSbf97fnlgcU5cQbbtfR8azpbfO2WLrMsN4ZvP/32aPrIP/4seP/89x9/Y+48/lT/zexCvmfghm6+bvQoSbX4kyx+D7sjbH/MoY1+WQZif/cFRwcfm2XNcnH6K4vghiZOUHy+SBb/o7TRYzcvb4dcufsl3lmbspTJU/IyPLHlmWfrKLynOuh7Z3iIw5xSQ+7FjoOMXP2xe4R0ldgGcAjSz8tk7wvF/CtodQEf//KgocFwQ0XUsQERk2woqCsr2TkT3/IiI60RELiRiAxJ1ERH5gGhsyjVecZik2TyZJYsgfr8bfbsjq1UnIXuJsv/kw/e0OPqzcubdS3HH5uC1OPjGsuy1UPTBOkv40O6tvybJUsGerfLEYcjHV1ma/MUqZ6bOxOFWJT8TpNmbXOnzE2EcrFZRKIY/RHH5pReckNtvjag4/nNzbJXHu+++OXqtHj2yNOLcYGkx2AiUVbJOQ9bGjcKiBemMZS3XOQX2c1a14i5lcZBF3+vGSwWiza2cVsFr5YJlEi2yFcBY+fzjYaeQ1R5ghyqg20GwV9hNKfOmtgp2Hp4QxzmN/aQj/4kW9j+/POYM36krQur6irqSGtp+0eKuHUqacaR+D3br7yGWJYFu+8R+IWj3CcGucNppSJs4Nbha1G0FbH4ga5rDwVkqO1TXdHifpjse1V1B7VMzqHaof+9SCXA0p4AObDuuX3eciF99HLzeq8uC8J+0yoLTrzpuwNg+iI1RJXcFL0J6TDLAh+/bAL3CiesZusiyZSw6BrBI9HikvmfX9S1uV7fHKspDLMHp+twAsrEhb4O/yBy0berWoe0ZgHZByErE+omtVnnGCVtvcw6nAPp1YO+JYBvB0D1CdaQw3yvIVAvzsSJEJbSZ3ydFqDYg2h124iyXimWw4P/P8v9/TcIgf9SHL+Isf1n1AkDZ1TxY5v+Gr3HESZzuJ+9ky4tfJ+VAmdP7fZ3xp7D++ODV5YBaPuCDo0oU6MoTCO9oKG+5ez7heI3odNWIZvQh9esY8CTebn/PyarQFTl5kZSypcSwrDoRbbteU7Dm6nEK+k9TnQw/7bDCwMoi37/3Kx/X7WR0e/P4gIb/+c1jmry8ClU+SRu1uHH7iOvSYsMMLlaZR1ubYkaAJiOVjMMTuI380i4jtlvmAwSjsVQMalC/4Fm2Q++JKz+Mxx+VDzYqbxgK3Oev/PiPNHiKQgAnLhlZHSV1jhW1EkX5JIij2SJ3sTj/Ng5WLmcRd9TeFCeeo+l0A0iV4NZB2ncpBkkFLaSQZGRRKMlYmyR7kC2/XxtbiO/eq/2L4RhDYNH7zXIZR2ylMlBO8JyTaTFZ5X/CebCYsdW1cI9IzLNVZWKLmGQeAsz7zDjzwiBT8a/k117P40I56NQZ6ODBGYgBAx/Xkzhazfng++9sE9xcKbd48FbjVhkjDsYtCtM112fEPF82YTB7Y9aEUXpjC0bIln2+ofniQHF59wAjusvkh6S8HEU3lItNMgPGRSIrb/2yeEoDTvR1mK1Tdi0MkhKQWCEu1Ki4qHr+YDHgt0upAhBJYVHqKThgMt3kQndakPop2aQ9wjKR4/y9TrZEJ09PhH+qQ1t+xCzIib5j1vYhnZ24lK2if4LJ5oJcAopkN7+avr2j73L5WmfJapucQhVxi9lTphC2LM9W5VgKo8Xsj03q6ie7J+uD6xVH5EJpUrFSmzC5WnLq9R48t70Jb9d6eaelU6RjB6hcdg+Y96TMTDqhxyZPbXjYm+V3u1aZ+u/xPAktRX1cc4uQO3bGHwXz49HiD4eWhvY0WD2yEWye65ocxz7oxaOulGjXnA33YNr1GhoMKJJcCwt630Y7DDw4KeaWZVWzTvjhQgCVGXKTWR//bMrzJmaReE5pxrbmgeyxamwxFS8o4MdHimcfPstkNLbC9oByt4lVDh1cSIWPo5Z/r6dt2/Yl9VhYuMYv57Zer6crxveGFrtjW71rQlVgvipRp4i0FzK1SE88atMT/TG7o4wJr2Ik3rsPjeuXLMkTWRa0rl/Z5Avjai5ZXIlBxb4t1S2JIhdp1qQi64Ibezrb1A52cSQzN1FXzYCsc5+8iUS/0pWFTJ5d7/2jolQ0VMiELFixePeAr0Rre7QeBg1ePyol+1rKE6g0iWV9woZm02h9AomJErestsmsNkLecObvNMCIPNgtsW0ysY3w0P4SnNhCPBfmttHRuW1unTFMbndbQaE3Vw0112sv2VVDcnOJ7SsWfTLqqyHoNN/y2w3M83GdeQQyz3A03u/8xzONxstoux5s742026P4RsSMyDRgClS5jZ1779hctuqBPJzUlMzmb5P89IIEzXM8Xa/tBj3pbIT77TG45bM7uGG4q6wJT2IsfjuGwfYtpV1xPikCOe2hrajQQuM3osdLVOdFBASVtdsuBJdVQaIe1LOhOXgtAey23qDHzlCYIR0pDI91uQzAV9Q7dMPXg+tdjQW93hDgRXjwov+wzk5TdnVfk80JThLqLBXGAhLLsuCqAsS5R8dmq3wKlymgtg9DHN0ZKzFhdwRBgMh/CpChPRDrIhiVaHgXau8C4kNWRzIZKNDOdkHPMjYHLykrrVLk0HZd7vut12vS5Yq5zbecYMOU2vrsmeF7XpHYHUCxskC3OPOMJi9RSwomRXq9Wms3OXkJkV5bjgfW3CNeCon48kpIDjm6hRdhcu/sf1xvaU95ORJHDGhW63BKycco/2Lvgiy4LMVg+/KKL6Wm2FNp06ca7F7TTMOqhlOWANYehFAYmlvO8XPBuKpBeP/z+tp0gPjyqwyoBhs6Db9w3L1wweQPfAjCuTqBLRa6uSTdgaS1hpQ916qlhvVpjl6rvANrjhP6vfRrjkbLfITacHdJisan9aY0POlFJnQG7PI9qJ51VirBlrwJoBI8s76Eosq4nqzCNJrkippL49UvQga2iSCKvnjDsblY2OnGtma2EVfaAsEeuj5ctopfgv1liPvursr++o5LgkE9d+zYMA4/cn1jYvutz+nJ8oLXmAnlKeyf3fjrnRKvZ2R45bW+RhDG04vK8DGnIcPn+hOrtTajXxk4qjD+SHecUKSM4vX440TIp9EgnsL8XnPkzr5HYV4QuDiFUfceHLgynUpd6IvdHS114+Mmfh04+ed8HZUOvRMFTDr0TuC+9d2JgMJAytfLaZCpxTy6RK/AlzeAVYQIhqUc7qF2bUyhnswUWIs1zJRet9I9Zjfnm+Zt3E53gKVqjnLlHF/OIrV37MjXU2IiHnR6rRDcvIyevAyxR+kQ01iO602wqQx3pxXu4AZKTEzJUqyJfm32FhFCJcrDVS4MG9x+11usLF3YyzJK56tp9uoPQ7MekJjxXLZ269n8s+k9mpUKzJ9cnVKxsdw4RWGt06xWEfOi5LQWJP1OleRU3LPug6IupnmjCqLYdUzVnapRQWtZhUjsPn/oLAZ8TrMYTsiU7/cTu3qJQhQG9hI9XFcRaM8khtbLdSXD9UC9h+C/j0n7Zwr0zsE/EnOyB0Y6dQ+L/sH1nhGsw+zjmVhIW0oalvs8Dmci+1137aY3ejGQ3cs1VE8odKji8KUN2bHfnkeRryeea0BxkAuaknDKPFPtWw7I89RO6GRAWF4VXF8fQ9u7NJs0OCHhTEyaJ/ViE7HY72AmjepZv+pm0gzVBsRelkPXBogwFx1tGriBFJGgZtUBNxU+E9XB1a1EMNHNZEB3/CtJ3G/Ln1m8/O2P9N/h8+zb4qvITNSWaZWImC+Vuuz+4/OdOBcLFmZFQvSuaNVs7iuoRwhYsSEYVk3H0qhQAVE+sjTNZ9LKOeLPX+/yPXq5ylpwVO1v6d8Dweg5mLE3qyWnXqH8lpUSR5oTdTGLWaXuoUBlK5/3Q7XahU+Mkl2xV3h1XQOZuI/xehYpxlu0gLVfC/ROzxeJlsLPEjva7pF8fSvWw5zCp3WcRT+tok1BRA954QoE/KOywNbmY5AhXrfcvb4djttn21kfuOnn1NPHGmPIx90oTVv8m5NsHh7e5slNFSMwej4ky9lbPdzArzFZPR8GwuM1e90JOlaz50Mvw4DZM0XeoY2YfyZG7HQcD23EoPdm3Ih5qB7MIsVeF6YjNwS16dlbMdLAsDFZsXL9xbMwY90pOlYzVhbMRmnHTqbv0Has3Fds7IbsdCQPbcigwjZuyBAeoSUTfLkkS2Y3cGxUlkwA8CwsWXeKjtaS4UESkaboO7glw1DBjtKSnY5kfZaMH6ZJklXrmPxnzj8lU5Zf8X8=</diagram></mxfile>
\ No newline at end of file
+<mxfile modified="2019-06-07T17:30:04.207Z" host="www.draw.io" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" etag="zx6JNhjWm9CAjpC8XxAK" version="10.7.5" type="google"><diagram id="aLS-0ix4CCXe6ffUliya" name="Page-1">7V1bl6M2Ev41/dg+SOIiHvuSmeSczGY2nezsPGIb2yS08WLc051fv8JGGFTCgI0Ad6tfxshcPFTVV6X6SqUb8vD8+jn2Nqsv0dwPb7Axf70hjzcYY0op+ycdeTuMIGTZh5FlHMyzsePAU/CPnw0a2egumPvb0olJFIVJsCkPzqL12p8lpTEvjqMf5dMWUVh+6sZb+mDgaeaFcPRbME9Wh1FqGcfxn/1gueJPRkb2zdSb/b2Mo906e946WvuHb549fpvs1O3Km0c/CkPkpxvyEEdRcvj0/Prgh+l75W/scN2nim/znxz766TJBc7T9u3lP+Zff5KXnx8/mX/e2d//fYsPd3nxwl32KrIfm7zxd7P/3/npTdANuf+xChL/aePN0m9/MHVgY6vkOcy+XgRh+BCFUcyO9++C3M+97Sq//Dl68ab7O6dHsb8N/ikeR4mXFI6ZnvnFY38eFA8zbSiMwHeSvaYXP07818JQ9o4++9Gzn8Rv7JTs21uuk5kuO1x+P46KgSx+0qqgFW425mXKuMzvfZQK+5AJpoWQiBZSjZAQgUJyTImMLKJKSEgLSRBSLgAuJGxDIdlSIZmKhGQCmfhz5giywyhOVtEyWnvhT8fR+6PUjLKE/Ncg+W86PLGyo++Fbx5fsyv2B2/ZwV9+krxl/s/bJREbOj711yjaSKR/g8liscCzGRvfJnH0t1/4Zm5PbeZs02+8OLlLfSH7YhZ6220w48OfgjD/0Wv2Hg+/Gln8+Pv+2MiPj799f/RWPPrqxwEThh9ng0Pp4TbaxTP/hKxpplnsBSz95JQPzLQiVYWTeh37oZcEL+WYQaaj+0uZLLy3wgmbKFgnW6DC+f0vgB5Dgj12mKTi33hr9nmZfv41YgEPO+3TE/+WPa14AjAOFrJs0o+ztzBgNhDXo9b0YCy/TvOBPED6bZewu3C5jh++bKuEXqbtAvSSgpcyB4OAfC4Ar6ZAdA7IjRYSEGkICRw7OoaE59evKQgcdcxyBCUTlOfwP8ouOupPNbTIH+MgWvbEhjBVEM53rVOnsw+HH9AtiMF4aaS+efTqrVxrDXMiKC6y3Ilb+HOc8k0PFgcUuTPlgVOkn+++xtHrG3d107jSy5WVqMbBXYHfMihwVHkSouipRLDpzlO94zB7KPkrt2nsTHDZpgkVpmEV3gjcyzStCXHEm7EJRuEP94sPFsCH37+x4z9ib8GmSaK2MktLykpYVohsPi6ZonthsFynETOT3z5eTu02YHH3XfbFczCf7/VdhjNlG7iy6X6egxGihlLizLAgDmFlOGRDqf+mpd6p1InjTOTR4oByd4Dc7zabMPC3smjA9p5TKayn2/Sf2cpbL/2tVo5uIEHQDVOWTDdIn7pBISb4TDdmXiJTj1wdaqNIrSDnoIdZ1g+LDq4fLtCPr7tpGGxXbPCnF38/zz5DGT6wkLElOAgeVw4mZAwTpjow6FjqVMwQGDB/2m9YgCFFp6XetcdHWJwFDC52DMT++ABz6VrcZ4jbIiVh2zIaHvcpa5gHvE+FyDy18ct6EXtMprtZsot9Lf9O5F/GeIdKjN3q1dhNoAAyKvRfmgNtFp8LaG4hmFtGVp+5ZQwzeVySi2ifhJzlWVv7f7voIFOyWBD2Vxw6iDv0vVSmR1043KRxWC8KLWPi2NnW/Y31mKLDLom2h0x0cR4Q+otEAhVJmppOVXUWrJd/7PPUtyZUlRGoBjKcsqMn0PZlmqHO9G0grw5YB/75UCLjcApCTjsci2v4weGyusKapnRFwxofgdWYez5dSFkNe0b96WJAJKov4mnK2HdfxHOZMjqdKiPXK1TWK2fsenWWFY1XGelgyign2mxIxBPHynm8trwddlzxdqAUUzFRh2F6VpeyXeyrTbccxpk2nKf1WsuGYZZVMzSDTOB4eiYvQBs6Ac9TCO+xeKR9jTa1cx9/8J2kxuX76zl/QKbdbCS79+hquEfjSE0beD7iGvlQ6wIYeLt0SLhbhSttW9xpOgK4U+v0j7NOnq+mupN0Wrx8jlVXxdB1c7OSzWYmVTTYSxCDznw5YkyplYppzLGw3dSE6agmZgSyAU9JlOaDDRh6fPOnTz4D6Wito41OmGDHFApCiIQx6DneIEMj0wjijQYxw4dYM8bnSbWoRmwlgUl/a8ZIM6JET7RbARzF5YU2Jl9ENtREm0C65PEBa2/WibBJee48OPtNYL2zJj8vYbhoubzBQpLCtV7ZT9ItraA5rlEEHU05ru6DjsuUkXaqjJrjGoUyNuW4FEXAIF2FiO1AkgudTXIhw8SQ5RIAWzHLRSD9oYPvy/21WI5oWrDhTK/RN48DNM01uGrwQDzP7kPV6DfvZA6eER9B3inPK5XTSjUuvy5fNVo6qzefiQnwcYTaE3ouoSW7oWlbihgt9jQByCk6/fOQRU9doIbTMmEeRXNa18xpuU05LR6/jWQiZsJVD5rT6i9LRBAgtQYPLq6mA8towaBxrzDeLEu5U0dI1nZJiQdu2zCMTb1OXqDIAVvXouXnRrrvwDp66qRHndEaBx3ENpSshmljG8MGmVU0RV395Ggtzh3bJNOlBuzy5doTdG5m1iWwbZhluHDWqjg7ayoh2c6a13EegesvqtHeJjZXyJ4cUzPHBAr6mHM/2tS8uB32VvlTEXEJPVptVNPS1Tl5viInBFfz6Pz3MM1Yyot1h1/mYUIKjPfhapaa+Lgrr4kB8g1AmI6E5lImS6vTJTsD+7CP2wIWuWIHWEzOXgLD0GRi19+uK8bAEDsNYj6g1MFZsKXU5yD9XY9MBzSInVoK4AoCy0GthqxXh2Kd0jnDotjc8unclKEYxVNiv2sUIyARZNvnL4lnoIhw7e06gjHMu0/mT+oBwiB59AtT6VcGIOx+D95sJeeReC9MjXHV1JDQ3VS61kkCccr2KLM6pYUGhrjxVvyqh7jKaOcMgHMmYqtOeLfO8I0KD+oB3+AKmFYU+EeGL7jcgfYakcE+7k+76XYWB9PUJzGouKgls84b1asAEZbAIB4lD5Y4sjpddTCsD/MRC9QdmQ9zbYd47zlMx6YJ8wNn7jdDsHvyPh15L/CYflIMMFG6j88bpcY/rvMSW9MOnl6w31WS1LcrkqSOOzWGJPrU45YtSS+cGXwTgmTZBTXBN+E9eHpMLtgwP1qdUfBfgllK/mhgOw1sZSoPw47MMlxTllOwYf5ot5l7iVzGgXZdtRJ2DZHgM3nRwWBCht1CtJAva7zI5ZcXOkESt18Rw8m2FvGFk2ckmjFMqPQrYzUr9o0J7aQtlJ6MV+hcbVEa6qnKGvEtNY9NMhtFvq35/IrnqA1W4UxbI+CFy66xGMkgyIH1CoG8MEqckUBBHnEvfRs1bRQkaeMrS/UK+ZJ855eaojJ1glK0Ph6fVYaNr6kMe7zZmfoy7KYNGB01Hq+to7Jx2Wqc01XYYhcKpwe3xp/ZsSEVrOi4tKH/Zeof0ozqrYOMwjosodU4d/VV5iGeT8wT9gGvtuVPqwtOOzM0mAnTocU+6BMEQzDkYvqNLdQsj9eQOGZI7LtjsxykXEMogbJKNEzt+SRbndgMEl1MpE/rDRI7XaGvVzcORXqKhMT5lCcyxC0AlBGeJx6lNuCGbZh1HJB+S4XyT8I34xgsDlCz0lvHAWOOAzrfj+Y8cCJI7porgVO8gGSlmM0iAUSGDgXgWnCNilmSXBAs7/I+FCxSmCP/7MdxurBRZEF+/8bO+2XNoGfNBFRfhV0jzeDZW/p32w17yRmIbQocYZy++/Uy9AvE4RUImJRnv0iy+SYiEvkqFDAsyyotvhfF+DXcLQPJ+AnTNepNd/ySQ2LigpfQ1Vimsv0fKEwxFRezGJ+Yy2KImDKWWmaZFdFmRJZ1wsNfJDNkYCA0IIS04/pmTO8XG+Wrb7LK6Orgvlwcj2xJD0ze87r43qmp6L1T+Nq/7MIkuN0Ge0o/DKZfovkuNZZLLAL2j2B/stDY2P+l36y8+T4qH8SganS03s64XTUzK2VQiAzoxMZjVi3fcmtrc5o5Iqrq7buwVuY+1fc4fQ+Z5xnIqgaZfdaoaL1VDWdGkmA/jKb7jUd+9xe3j/cFkXYbTHw4Ib+VhTmUzF09v+sy4nRccY2sJPTpd4Ln6gles6UWYuoFN0u9qDNNPcFrKzPsQHPrdYLnwg4TGk4vmMCjEeIppPQ0nkqXvIwOUCHtoAG1RmiDIyrViNqlgM0RMhAaUM9aJjo8A6HxtK3MhmcgEJxVACFcPwNhNqIgHPjilVEQCMF185qDqNXS6+EgEAxUxmNYLd/y1XEQ+SJhTUI00tERkxAINp+oJiG0SFuIdCSUA0Iwbvw4SEls2Nyl32IIxDuL60ikieRpheTHG4nwH3BN9lX1lq8vEsknMToSaaKjI45EuNnoSKRrkY4lEsEfIhmCsCWwaJKkfs8xCNbZkBaWdX3ZEHx92ZDKt3yFMYjOhjQR+BVkQ7DOhigS6VhiEAKBse3CY+k6Xcl6XskCZf651QJlY4LscpdPftCmxef17FNXvyF7llGp3zH6cF53C44v1L3Le3JcoHvlbdCNZrqHSnpXuzj+PSlZQx3LuZCh++FxHM2Q1im3q6k9H9EediVAxB4P/DrnNYhw2tnAiJsb19pArtu1VoDH0dvBpoJSu6dbO4jnm24PXWoQubz5yUV+wNGOQIUjUNQT6owdudt5AnBBT67g8pb4Q7uCllZw1a6gcZsfYo7EDISuPXW+AFxg9tH6HhF3EDOo0Gj8foC9AWCridyhZpmWHJE737yh6kGVOi9e0A/08/TOFUP/R5oF4KazANJ5wuVMMzDLWo2NGugXL+Dbfik2A1i7oHdZVsHhVlJHLljwmKWHi1Rtr5ssS4rF/O3WW6YKwUmlS4iH8csEVfjK8loCWVc+S5FMkAOlAqRwoFcb//dbs5y3wnuhHD2K78VCMuJT3XtpUNDRqqdkieQqYYkEt0ZCguXK0YrX7FtUNqwCflqx/9v85shsXoIrY6QnW0gmb7ddjhuwZO9qOfioYywlTa0PknsM4vTi9SL2mKvfzZJdDOt3xhEWnLDraJ3wuP+qVcd2xJbncMtJZEk0R1kogTpost0T1y1mFahhMN8imS0RIiuHqHS79XMb3oSrPltgj4PiwETYxgL3MWOnsDBmFj2nb/xuPX/ykwa4U7Tl/fJ+uBljY8QJvakf3nuzv5d7dZVV0jS2dP4/uzhCSzdfNYQmHIejC/NHt4QI2JJXz/GbRIvF1lfSPRzRESVrGlfMmA4t5mtumWwQrknZyHfGLeJMNRxdgEB8Bd4QCHSZf5F0JHgnoJD3Ye8AFAi1URmxOwEFLOwaSvsDBHItgKDeeLkwlVMIIjlFxS32Otu6R9wdo5/4AmZY3guUdFbnzqAkX0yaCecWdRRg4NJtTSENoRJMBq2JHBmYOD2BidiWitrCyqOuwER8ECZ1REzFL1OLPpK2vO8Ffaoq9M9CH9LVfEZg2wTtUwk3g9aflkuw0Um4KUxmCLKLkxlj4tZVaQwzleFepJ4n7jxqqpjIsMM4ipKiEsXeZvUlmvvpGf8H</diagram></mxfile>
\ No newline at end of file
diff --git a/src/main/resources/Documentation/sources/git-replication-split-brain-detected.txt b/src/main/resources/Documentation/sources/git-replication-split-brain-detected.txt
index b249b9f..79b0c69 100644
--- a/src/main/resources/Documentation/sources/git-replication-split-brain-detected.txt
+++ b/src/main/resources/Documentation/sources/git-replication-split-brain-detected.txt
@@ -2,16 +2,16 @@
participant Client1
participant Instance1
-participant Ref-DB Coordinator
+participant Global Ref-DB
participant Instance2
participant Client2
state over Client1, Client2, Instance1, Instance2: W0
state over Client1 : W0 -> W1
Client1 -> +Instance1: Push W1
-Instance1 -> +Ref-DB Coordinator: CAS if state == W0 set state W0 -> W1
-state over Ref-DB Coordinator : W0 -> W1
-Ref-DB Coordinator -> -Instance1 : ACK
+Instance1 -> +Global Ref-DB: CAS if state == W0 set state W0 -> W1
+state over Global Ref-DB : W0 -> W1
+Global Ref-DB -> -Instance1 : ACK
state over Instance1 : W0 -> W1
Instance1 -> -Client1: Ack W1
@@ -20,8 +20,8 @@
state over Client2 : W0 -> W2
Client2 -> +Instance2: Push W2
-Instance2 -> +Ref-DB Coordinator: CAS if state == W0 set state W0 -> W2
-Ref-DB Coordinator -> -Instance2 : NACK
+Instance2 -> +Global Ref-DB: CAS if state == W0 set state W0 -> W2
+Global Ref-DB -> -Instance2 : NACK
Instance2 -> -Client2 : Push failed -- RO Mode
@@ -36,9 +36,9 @@
Client2 -> Instance2: Pull W1
state over Client2 : W0 -> W1 -> W2
Client2 -> Instance2: Push W2
-Instance2 -> +Ref-DB Coordinator: CAS if state == W1 set state W1 -> W2
-state over Ref-DB Coordinator: W0 -> W1 -> W2
-Ref-DB Coordinator -> -Instance2 : ACK
+Instance2 -> +Global Ref-DB: CAS if state == W1 set state W1 -> W2
+state over Global Ref-DB: W0 -> W1 -> W2
+Global Ref-DB -> -Instance2 : ACK
state over Instance2: W0 -> W1 -> W2
Instance2 -> -Client2 : ACK
