Use NFS filesystem to host git repos on HA setup
Introduce an NFS server which is hosting the git repos.
This is done since it is closer to a real installation,
and allowed us to find or reproduce issues specific to
NFS. Also adapt the docker-compose file accordingly.
Change-Id: I60c34e458cd3adb1ad2d71bc0db4eb450de67630
diff --git a/src/test/README.md b/src/test/README.md
index 1b8e4a2..fd59f64 100644
--- a/src/test/README.md
+++ b/src/test/README.md
@@ -1,19 +1,31 @@
# Gerrit high-availability docker setup example
The Docker Compose project in the docker directory contains a simple test
-environment of two Gerrit masters in HA configuration.
+environment of two Gerrit masters in HA configuration, with their git repos
+hosted on NFS filesystem.
## How to build
-The project can be built using docker-compose.
+The project can be built using docker-compose (make sure you set the
+`platform` attribute in the docker-compose.yaml file if you're not
+in an amd64 arch).
To build the Docker VMs:
-```
- $ docker-compose build
+```bash
+ # first, remove the buildx if it exists and its not running
+ $ docker buildx inspect docker-ha | grep Status
+ $ docker buildx rm docker-ha
+ # create the docker-ha buildx node, provide your architecture and start it up
+ docker buildx create --name docker-ha --platform "linux/amd64" --driver docker-container --use \
+ && docker buildx inspect --bootstrap \
+ && docker-compose build
```
### Building the Docker VMs using a non-default user id
+First, update the user id in the [NFS Dockerfile](./docker/nfs/Dockerfile).
+This is done simply by modifying the file setting the non-default user id.
+Then, run the following:
```
$ export GERRIT_UID=$(id -u)
$ docker-compose build --build-arg GERRIT_UID
@@ -24,6 +36,9 @@
host is not owned by you. For example, some corporate environments use a
restricted 1000 user (id). In that case, the containerized application
may fail to write towards the host (through volumes).
+**Important:** The user id in gerrit must be the same as the uid in the
+NFS server, otherwise you will encounter file ownership problems on any
+filesystem operation.
That UID will be the one set for the containerized gerrit user. Latter's
group will remain as default (1000). This is because groups known from
@@ -36,9 +51,41 @@
Use the 'up' target to startup the Docker Compose VMs.
```
- $ docker-compose up
+ $ docker-compose up -d
```
+## Background on using an NFS server
+We are using the `erichough/nfs-server` image mainly because it's easy to use
+& we had success with it. The work has been inspired by
+[this blog post](https://nothing2say.co.uk/running-a-linux-based-nfs-server-in-docker-on-windows-b64445d5ada2).
+
+The containers start with the `privileged` flag set, which is a security risk
+but necessary to work around permission issues.
+
+It is worth noting that we are exposing the `/var/gerrit/git` directory as the
+nfs-share. This is because more often than not it's the git directory that's
+shared over the network. You can change this in the nfs server and gerrit
+docker files, and in the `exports.txt` file.
+
+The NFS server is using a static IP. The Docker Compose YAML file defines a
+bridge network with the subnet `192.168.1.0/24` (this is what allows us to
+give the NFS Server a known, static IP).
+
+The `addr=192.168.1.5` option (in the `nfs-client-volume` volume) is the
+reason we need a static IP for the server (and hence a configured subnet
+for the network). Note that using a name (ie. addr=nfs-server) we weren't
+able to get the DNS resolution to work properly.
+
+Also in the Docker Compose file we can see that the `nfs-server` container
+uses a `healthcheck`, this is necessary to control when the `gerrit`
+services will start up (they need to start after the nfs server is fully
+up-and-running).
+
+Finally, we are providing an `exports.txt` file, which again utilises the
+subnet we provided during the bridge network creation. This file is baked
+into the image sacrificing a bit of flexibility, but we feel this is
+a small price to pay to have everything automated.
+
# Gerrit high-availability local setup example
1. Init gerrit instances with high-availability plugin installed:
diff --git a/src/test/docker/docker-compose.yaml b/src/test/docker/docker-compose.yaml
index 42c12c9..59bec58 100644
--- a/src/test/docker/docker-compose.yaml
+++ b/src/test/docker/docker-compose.yaml
@@ -2,16 +2,40 @@
services:
+ nfs-server:
+ build: nfs
+# platform: linux/arm64/v8 # uncomment for Apple Silicon arch
+ privileged: true
+ container_name: nfs-server
+ environment:
+ NFS_LOG_LEVEL: DEBUG
+ hostname: nfs-server
+ healthcheck:
+ test: ["CMD-SHELL", "sleep 10"] # required, otherwise the gerrit service will fail to start with a "connection refused" error
+ interval: 1s
+ timeout: 1m
+ retries: 10
+ ports:
+ - 2049:2049
+ networks:
+ nfs-server-bridge:
+ ipv4_address: 192.168.1.5
+ volumes:
+ - nfs-server-volume:/var/gerrit/git
gerrit-01:
build: gerrit
+ privileged: true
+ depends_on:
+ nfs-server:
+ condition: service_healthy
ports:
- "8081:8080"
- "29411:29418"
networks:
- - gerrit-net
+ nfs-server-bridge: null
volumes:
- /dev/urandom:/dev/random
- - gitvolume:/var/gerrit/git
+ - git-volume:/var/gerrit/git
- shareddir:/var/gerrit/shareddir
- ./etc/gerrit.config:/var/gerrit/etc/gerrit.config.orig
- ./etc/high-availability.gerrit-01.config:/var/gerrit/etc/high-availability.config.orig
@@ -20,16 +44,20 @@
gerrit-02:
build: gerrit
+ privileged: true
ports:
- "8082:8080"
- "29412:29418"
networks:
- - gerrit-net
+ nfs-server-bridge: null
depends_on:
- - gerrit-01
+ gerrit-01:
+ condition: service_started
+ nfs-server:
+ condition: service_healthy
volumes:
- /dev/urandom:/dev/random
- - gitvolume:/var/gerrit/git
+ - git-volume:/var/gerrit/git
- shareddir:/var/gerrit/shareddir
- ./etc/gerrit.config:/var/gerrit/etc/gerrit.config.orig
- ./etc/high-availability.gerrit-02.config:/var/gerrit/etc/high-availability.config.orig
@@ -43,9 +71,9 @@
- "80:80"
- "29418:29418"
networks:
- - gerrit-net
+ nfs-server-bridge: null
volumes:
- - syslog-sidecar
+ - syslog-sidecar:/syslog-sidecar
depends_on:
- syslog-sidecar
- gerrit-01
@@ -54,13 +82,22 @@
syslog-sidecar:
build: docker-syslog-ng-stdout
networks:
- - gerrit-net
+ nfs-server-bridge: null
networks:
- gerrit-net:
- driver: bridge
+ nfs-server-bridge:
+ ipam:
+ driver: default
+ config:
+ - subnet: 192.168.1.0/24
volumes:
syslog-sidecar:
shareddir:
- gitvolume:
+ nfs-server-volume:
+ git-volume:
+ driver: "local"
+ driver_opts:
+ type: nfs
+ o: "addr=192.168.1.5,rw"
+ device: ":/var/gerrit/git"
diff --git a/src/test/docker/gerrit/Dockerfile b/src/test/docker/gerrit/Dockerfile
index 820be4c..30912e9 100644
--- a/src/test/docker/gerrit/Dockerfile
+++ b/src/test/docker/gerrit/Dockerfile
@@ -1,27 +1,52 @@
-FROM gerritcodereview/gerrit:3.5.0-rc2
+FROM almalinux:8.5
-ENV GERRIT_BRANCH=stable-3.5
+# Install dependencies
+RUN yum -y install \
+ git \
+ java-11-openjdk \
+ procps \
+ sudo \
+ passwd \
+ gettext \
+ && yum -y clean all
-ENV GERRIT_CI_URL=https://gerrit-ci.gerritforge.com/job
+ENV GERRIT_VERSION=3.6
+ENV JAVA_HOME /usr/lib/jvm/jre-11-openjdk
-USER root
+# Add gerrit user
+RUN adduser -p -m --uid 1000 gerrit --home-dir /home/gerrit
+RUN echo "gerrit:gerrit" | chpasswd
+RUN echo "gerrit ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/gerrit && \
+ chmod 0440 /etc/sudoers.d/gerrit
-RUN yum install -y iputils nmap curl lsof gettext net-tools sudo
+# Create gerrit installation directory
+RUN mkdir -p /var/gerrit && chown -R gerrit /var/gerrit
+ADD --chown=gerrit \
+ "https://gerrit-ci.gerritforge.com/job/Gerrit-bazel-stable-$GERRIT_VERSION/lastSuccessfulBuild/artifact/gerrit/bazel-bin/release.war" \
+ /tmp/gerrit.war
+
+ADD --chown=gerrit \
+"https://gerrit-ci.gerritforge.com/job/plugin-javamelody-bazel-master-stable-$GERRIT_VERSION/lastSuccessfulBuild/artifact/bazel-bin/plugins/javamelody/javamelody.jar" \
+ /var/gerrit/plugins/javamelody.jar
+ADD --chown=gerrit \
+ "https://gerrit-ci.gerritforge.com/job/plugin-high-availability-bazel-stable-$GERRIT_VERSION/lastSuccessfulBuild/artifact/bazel-bin/plugins/high-availability/high-availability.jar" \
+ /var/gerrit/plugins/high-availability.jar
+
+ADD --chown=gerrit \
+ "https://repo1.maven.org/maven2/com/gerritforge/global-refdb/$GERRIT_VERSION.3.4/global-refdb-$GERRIT_VERSION.3.4.jar" \
+ /tmp
+
+ADD --chown=gerrit:gerrit ./wait-for-it.sh /bin
+
+# Change user
USER gerrit
-ADD --chown=gerrit:gerrit $GERRIT_CI_URL/plugin-javamelody-bazel-master-$GERRIT_BRANCH/lastSuccessfulBuild/artifact/bazel-bin/plugins/javamelody/javamelody.jar /var/gerrit/plugins/javamelody.jar
-ADD --chown=gerrit:gerrit $GERRIT_CI_URL/plugin-high-availability-bazel-master-$GERRIT_BRANCH/lastSuccessfulBuild/artifact/bazel-bin/plugins/high-availability/high-availability.jar /var/gerrit/plugins/high-availability.jar
+# Expose ssh and http ports
+EXPOSE 29418 8080
+
+COPY --chown=gerrit ./entrypoint.sh /bin
USER root
-ADD start.sh /bin/
-ADD wait-for-it.sh /bin/
-
-RUN rm -Rf /var/gerrit/{git,index,cache}/*
-
-ARG GERRIT_UID=1000
-RUN usermod -u ${GERRIT_UID} gerrit &> /dev/null
-
-ENTRYPOINT ["/usr/bin/env"]
-CMD /bin/start.sh
+ENTRYPOINT /bin/entrypoint.sh
\ No newline at end of file
diff --git a/src/test/docker/gerrit/entrypoint.sh b/src/test/docker/gerrit/entrypoint.sh
new file mode 100755
index 0000000..b382a3b
--- /dev/null
+++ b/src/test/docker/gerrit/entrypoint.sh
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+if [[ ! -z "$WAIT_FOR" ]]
+then
+ wait-for-it.sh $WAIT_FOR -t 600 -- echo "$WAIT_FOR is up"
+fi
+
+chown -R gerrit /var/gerrit/etc
+sudo -u gerrit cp /var/gerrit/etc/gerrit.config.orig /var/gerrit/etc/gerrit.config
+sudo -u gerrit cp /var/gerrit/etc/high-availability.config.orig /var/gerrit/etc/high-availability.config
+
+
+echo "Init gerrit..."
+sudo -u gerrit java -jar /tmp/gerrit.war init -d /var/gerrit --batch --install-all-plugins
+chown -R gerrit: /var/gerrit/shareddir
+
+# required until regression is fixed, see https://groups.google.com/g/repo-discuss/c/DH-ftHMiCyE/m/qF88c6KMAAAJ
+echo "Copying global ref db jar into lib"
+sudo -u gerrit cp /tmp/global-refdb-*.jar /var/gerrit/lib
+
+echo "Reindexing Gerrit..."
+cd /var/gerrit && sudo -u gerrit java -jar /var/gerrit/bin/gerrit.war reindex -d /var/gerrit
+sudo -u gerrit git config -f /var/gerrit/etc/gerrit.config gerrit.canonicalWebUrl http://$HOSTNAME/
+sudo -u gerrit touch /var/gerrit/logs/{gc_log,error_log,httpd_log,sshd_log,replication_log} && tail -f /var/gerrit/logs/* | grep --line-buffered -v 'HEAD /' &
+
+echo "Running Gerrit ..."
+sudo -u gerrit /var/gerrit/bin/gerrit.sh run
diff --git a/src/test/docker/gerrit/start.sh b/src/test/docker/gerrit/start.sh
deleted file mode 100755
index d906628..0000000
--- a/src/test/docker/gerrit/start.sh
+++ /dev/null
@@ -1,25 +0,0 @@
-#!/bin/bash -e
-
-if [[ ! -z "$WAIT_FOR" ]]
-then
- wait-for-it.sh $WAIT_FOR -t 600 -- echo "$WAIT_FOR is up"
-fi
-
-sudo -u gerrit cp /var/gerrit/etc/gerrit.config.orig /var/gerrit/etc/gerrit.config
-sudo -u gerrit cp /var/gerrit/etc/high-availability.config.orig /var/gerrit/etc/high-availability.config
-
-if [[ ! -f /var/gerrit/etc/ssh_host_ed25519_key ]]
-then
- echo "Initializing Gerrit site ..."
- sudo -u gerrit java -jar /var/gerrit/bin/gerrit.war init -d /var/gerrit --batch
- chown -R gerrit: /var/gerrit/shareddir
-fi
-
-echo "Reindexing Gerrit ..."
-sudo -u gerrit java -jar /var/gerrit/bin/gerrit.war reindex -d /var/gerrit
-sudo -u gerrit git config -f /var/gerrit/etc/gerrit.config gerrit.canonicalWebUrl http://$HOSTNAME/
-
-sudo -u gerrit touch /var/gerrit/logs/{gc_log,error_log,httpd_log,sshd_log,replication_log} && tail -f /var/gerrit/logs/* | grep --line-buffered -v 'HEAD /' &
-
-echo "Running Gerrit ..."
-sudo -u gerrit /var/gerrit/bin/gerrit.sh run
diff --git a/src/test/docker/nfs/Dockerfile b/src/test/docker/nfs/Dockerfile
new file mode 100644
index 0000000..19744f3
--- /dev/null
+++ b/src/test/docker/nfs/Dockerfile
@@ -0,0 +1,13 @@
+FROM erichough/nfs-server
+
+COPY exports.txt /etc/exports
+
+# To avoid ownership issues, the user must be the same betweeen the
+# server and the client, hence we are creating it explicitly in both.
+RUN adduser --disabled-password --gecos "" --uid 1000 gerrit
+
+# /var/gerrit/git is the shared directory
+RUN mkdir --parents /var/gerrit/git
+
+RUN chown gerrit:gerrit /var/lib/nfs
+RUN chown gerrit:gerrit /var/gerrit/git
\ No newline at end of file
diff --git a/src/test/docker/nfs/exports.txt b/src/test/docker/nfs/exports.txt
new file mode 100644
index 0000000..c31d586
--- /dev/null
+++ b/src/test/docker/nfs/exports.txt
@@ -0,0 +1 @@
+/var/gerrit/git 192.168.1.0/24(rw,no_subtree_check,insecure)
\ No newline at end of file
