blob: c8ac4b476ba2f014f0e7010396a730d08ea25b6c [file] [log] [blame] [view]
Luca Milanesio87be34a2019-10-12 22:33:29 +01001# Gerrit high-availability plugin
2
3This plugin allows deploying a cluster of multiple Gerrit masters
Luca Milanesiobc41f9e2021-07-06 11:49:41 +01004on the same data-center sharing the same Git repositories.
Luca Milanesio87be34a2019-10-12 22:33:29 +01005
6Requirements for the Gerrit masters are:
7
8- Gerrit v2.14.20 or later
9- Externally mounted filesystem shared among the cluster
Luca Milanesio87be34a2019-10-12 22:33:29 +010010- Load-balancer (HAProxy or similar)
11
12## License
13
14This plugin is released under the same Apache 2.0 license and copyright holders
15as of the Gerrit Code Review project.
16
17## How to build
18
19Refer to the [build instructions in the plugin documentation](src/main/resources/Documentation/build.md).
20
21## Sample configuration for two Gerrit masters in high-availability
22
23Assuming that the Gerrit masters in the clusters are `gerrit-01.mycompany.com` and
24`gerrit-02.mycompany.com`, listening on the HTTP port 8080, with a shared volume
25mounted under `/shared`, see below the minimal configuration steps.
26
Luca Milanesiobc41f9e2021-07-06 11:49:41 +0100271. Install one Gerrit master on the first node (e.g. `gerrit-01.mycompany.com`)
28 with the repositories location under the shared volume (e.g. `/shared/git`).
29 Init the site in order to create the initial repositories.
Luca Milanesio87be34a2019-10-12 22:33:29 +010030
312. Copy all the files of the first Gerrit master onto the second node (e.g. `gerrit-02.mycompany.com`)
Luca Milanesiobc41f9e2021-07-06 11:49:41 +010032 so that it points to the same repositories location.
Luca Milanesio87be34a2019-10-12 22:33:29 +010033
343. Install the high-availability plugin into the `$GERRIT_SITE/plugins` directory of both
35 the Gerrit servers.
36
374. On `gerrit-01.mycompany.com`, create the `$GERRIT_SITE/etc/high-availability.config` with
38 the following settings:
39
40 ```
41 [main]
42 sharedDirectory = /shared
43
44 [peerInfo]
45 strategy = static
46
47 [peerInfo "static"]
48 url = http://gerrit-02.mycompany.com:8080
49 ```
50
515. On `gerrit-02.mycompany.com`, create the `$GERRIT_SITE/etc/high-availability.config` with
52 the following settings:
53
54 ```
55 [main]
56 sharedDirectory = /shared
57
58 [peerInfo]
59 strategy = static
60
61 [peerInfo "static"]
62 url = http://gerrit-01.mycompany.com:8080
63 ```
64
65For more details on the configuration settings, please refer to the
66[high-availability configuration documentation](src/main/resources/Documentation/config.md).
67
68## Load-balancing configuration
69
70It is possible to distribute the incoming traffic to both Gerrit nodes using any software that can
71perform load-balancing of the incoming connections.
72
73The load-balancing of the HTTP traffic is at L7 (Application) while the
74SSH traffic is balanced at L4 (Transport) level.
75
76### Active-passive configuration
77
78It is the simplest and safest configuration, where only one Gerrit master at a
79time serves the incoming requests.
80In case of failure of the primary master, the traffic is forwarded to the backup.
81
82Assuming a load-balancing implemented using [HAProxy](http://www.haproxy.org/)
83associated with the domain name `gerrit.mycompany.com`, exposing Gerrit cluster nodes
84on ports, 80 (HTTP) and 29418 (SSH), see below the minimal configuration
85steps
86
87Add to the `haproxy.cfg` the frontend configurations associated with the HTTP
88and SSH services:
89
90```
91frontend gerrit_http
92 bind *:80
93 mode http
94 default_backend gerrit_http_nodes
95
96frontend gerrit_ssh
97 bind *:29418
98 mode tcp
99 default_backend gerrit_ssh_nodes
100```
101
102Add to the `haproxy.cfg` the backend configurations pointing to the Gerrit cluster
103nodes:
104
105```
106backend gerrit_http_nodes
107 mode http
108 balance source
109 option forwardfor
110 default-server inter 10s fall 3 rise 2
111 option httpchk GET /config/server/version HTTP/1.0
112 http-check expect status 200
113 server gerrit_http_01 gerrit-01.mycompany.com:8080 check inter 10s
114 server gerrit_http_02 gerrit-01.mycompany.com:8080 check inter 10s backup
115
116ackend ssh
117 mode tcp
118 option httpchk GET /config/server/version HTTP/1.0
119 http-check expect status 200
120 balance source
121 timeout connect 10s
122 timeout server 5m
123 server gerrit_ssh_01 gerrit-01.mycompany.com:29418 check port 8080 inter 10s fall 3 rise 2
124 server gerrit-ssh_02 gerrit-02.mycompany.com:29418 check port 8080 inter 10s fall 3 rise 2 backup
125```
126
127### Active-active configuration
128
129This is an evolution of the previous active-passive configuration, where only one Gerrit master at a
130time serves the HTTP write operations (PUT,POST,DELETE) while the remaining HTTP traffic is sent
131to both.
132In case of failure of one of the nodes, all the traffic is forwarded to the other node.
133
134With regards to the SSH traffic, it cannot be safely sent to both nodes because it is associated
135with a stateful session that can host multiple commands of different nature.
136
137Assuming an active-passive configuration using HAProxy, see below the changes needed to implement
138an active-active scenario.
139
140Add to the `haproxy.cfg` the extra acl settings into the `gerrit_http` frontend configurations
141associated with the HTTP and SSH services:
142
143```
144frontend gerrit_http
145 bind *:80
146 mode http
147 acl http_writes method PUT POST DELETE PATCH
148 use_backend gerrit_http_nodes if http_writes
149 default_backend gerrit_http_nodes_balanced
150```
151
152Add to the `haproxy.cfg` a new backend for serving all read-only HTTP operations from both nodes:
153
154```
155backend gerrit_http_nodes_balanced
156 mode http
157 balance source
158 option forwardfor
159 default-server inter 10s fall 3 rise 2
160 option httpchk GET /config/server/version HTTP/1.0
161 http-check expect status 200
162 server gerrit_http_01 gerrit-01.mycompany.com:8080 check inter 10s
163 server gerrit_http_02 gerrit-01.mycompany.com:8080 check inter 10s
164```
165
166## Gerrit canonical URL and further adjustments
167
168Both Gerrit masters are now part of the same cluster, accessible through the HAProxy load-balancer.
169Set the `gerrit.canoncalWebUrl` on both Gerrit masters to the domain name of HAProxy so that any
170location or URL generated by Gerrit would direct the traffic to the balancer and not to the instance
171that served the incoming call.
172
173Example:
174```
175[gerrit]
176 canonicalWebUrl = http://gerrit.mycompany.com
177```
178
179Secondly, adjust the HTTP listen configuration adding the `proxy-` prefix, to inform Gerrit that
180the traffic is getting filtered through a reverse proxy.
181
182Example:
183```
184[httpd]
185 listenUrl = proxy-http://*:8080/
186```
187
188Last adjustment is associated with the session cookies, because they would need to be bound to
189the domain rather than the individual node.
190
191Example:
192```
193[auth]
194 cookiedomain = .mycompany.com
195```