Use NFS filesystem to host git repos on HA setup Introduce an NFS server which is hosting the git repos. This is done since it is closer to a real installation, and allowed us to find or reproduce issues specific to NFS. Also adapt the docker-compose file accordingly. Change-Id: I60c34e458cd3adb1ad2d71bc0db4eb450de67630
diff --git a/src/test/README.md b/src/test/README.md index 1b8e4a2..fd59f64 100644 --- a/src/test/README.md +++ b/src/test/README.md
@@ -1,19 +1,31 @@ # Gerrit high-availability docker setup example The Docker Compose project in the docker directory contains a simple test -environment of two Gerrit masters in HA configuration. +environment of two Gerrit masters in HA configuration, with their git repos +hosted on NFS filesystem. ## How to build -The project can be built using docker-compose. +The project can be built using docker-compose (make sure you set the +`platform` attribute in the docker-compose.yaml file if you're not +in an amd64 arch). To build the Docker VMs: -``` - $ docker-compose build +```bash + # first, remove the buildx if it exists and its not running + $ docker buildx inspect docker-ha | grep Status + $ docker buildx rm docker-ha + # create the docker-ha buildx node, provide your architecture and start it up + docker buildx create --name docker-ha --platform "linux/amd64" --driver docker-container --use \ + && docker buildx inspect --bootstrap \ + && docker-compose build ``` ### Building the Docker VMs using a non-default user id +First, update the user id in the [NFS Dockerfile](./docker/nfs/Dockerfile). +This is done simply by modifying the file setting the non-default user id. +Then, run the following: ``` $ export GERRIT_UID=$(id -u) $ docker-compose build --build-arg GERRIT_UID @@ -24,6 +36,9 @@ host is not owned by you. For example, some corporate environments use a restricted 1000 user (id). In that case, the containerized application may fail to write towards the host (through volumes). +**Important:** The user id in gerrit must be the same as the uid in the +NFS server, otherwise you will encounter file ownership problems on any +filesystem operation. That UID will be the one set for the containerized gerrit user. Latter's group will remain as default (1000). This is because groups known from @@ -36,9 +51,41 @@ Use the 'up' target to startup the Docker Compose VMs. ``` - $ docker-compose up + $ docker-compose up -d ``` +## Background on using an NFS server +We are using the `erichough/nfs-server` image mainly because it's easy to use +& we had success with it. The work has been inspired by +[this blog post](https://nothing2say.co.uk/running-a-linux-based-nfs-server-in-docker-on-windows-b64445d5ada2). + +The containers start with the `privileged` flag set, which is a security risk +but necessary to work around permission issues. + +It is worth noting that we are exposing the `/var/gerrit/git` directory as the +nfs-share. This is because more often than not it's the git directory that's +shared over the network. You can change this in the nfs server and gerrit +docker files, and in the `exports.txt` file. + +The NFS server is using a static IP. The Docker Compose YAML file defines a +bridge network with the subnet `192.168.1.0/24` (this is what allows us to +give the NFS Server a known, static IP). + +The `addr=192.168.1.5` option (in the `nfs-client-volume` volume) is the +reason we need a static IP for the server (and hence a configured subnet +for the network). Note that using a name (ie. addr=nfs-server) we weren't +able to get the DNS resolution to work properly. + +Also in the Docker Compose file we can see that the `nfs-server` container +uses a `healthcheck`, this is necessary to control when the `gerrit` +services will start up (they need to start after the nfs server is fully +up-and-running). + +Finally, we are providing an `exports.txt` file, which again utilises the +subnet we provided during the bridge network creation. This file is baked +into the image sacrificing a bit of flexibility, but we feel this is +a small price to pay to have everything automated. + # Gerrit high-availability local setup example 1. Init gerrit instances with high-availability plugin installed:
diff --git a/src/test/docker/docker-compose.yaml b/src/test/docker/docker-compose.yaml index 42c12c9..59bec58 100644 --- a/src/test/docker/docker-compose.yaml +++ b/src/test/docker/docker-compose.yaml
@@ -2,16 +2,40 @@ services: + nfs-server: + build: nfs +# platform: linux/arm64/v8 # uncomment for Apple Silicon arch + privileged: true + container_name: nfs-server + environment: + NFS_LOG_LEVEL: DEBUG + hostname: nfs-server + healthcheck: + test: ["CMD-SHELL", "sleep 10"] # required, otherwise the gerrit service will fail to start with a "connection refused" error + interval: 1s + timeout: 1m + retries: 10 + ports: + - 2049:2049 + networks: + nfs-server-bridge: + ipv4_address: 192.168.1.5 + volumes: + - nfs-server-volume:/var/gerrit/git gerrit-01: build: gerrit + privileged: true + depends_on: + nfs-server: + condition: service_healthy ports: - "8081:8080" - "29411:29418" networks: - - gerrit-net + nfs-server-bridge: null volumes: - /dev/urandom:/dev/random - - gitvolume:/var/gerrit/git + - git-volume:/var/gerrit/git - shareddir:/var/gerrit/shareddir - ./etc/gerrit.config:/var/gerrit/etc/gerrit.config.orig - ./etc/high-availability.gerrit-01.config:/var/gerrit/etc/high-availability.config.orig @@ -20,16 +44,20 @@ gerrit-02: build: gerrit + privileged: true ports: - "8082:8080" - "29412:29418" networks: - - gerrit-net + nfs-server-bridge: null depends_on: - - gerrit-01 + gerrit-01: + condition: service_started + nfs-server: + condition: service_healthy volumes: - /dev/urandom:/dev/random - - gitvolume:/var/gerrit/git + - git-volume:/var/gerrit/git - shareddir:/var/gerrit/shareddir - ./etc/gerrit.config:/var/gerrit/etc/gerrit.config.orig - ./etc/high-availability.gerrit-02.config:/var/gerrit/etc/high-availability.config.orig @@ -43,9 +71,9 @@ - "80:80" - "29418:29418" networks: - - gerrit-net + nfs-server-bridge: null volumes: - - syslog-sidecar + - syslog-sidecar:/syslog-sidecar depends_on: - syslog-sidecar - gerrit-01 @@ -54,13 +82,22 @@ syslog-sidecar: build: docker-syslog-ng-stdout networks: - - gerrit-net + nfs-server-bridge: null networks: - gerrit-net: - driver: bridge + nfs-server-bridge: + ipam: + driver: default + config: + - subnet: 192.168.1.0/24 volumes: syslog-sidecar: shareddir: - gitvolume: + nfs-server-volume: + git-volume: + driver: "local" + driver_opts: + type: nfs + o: "addr=192.168.1.5,rw" + device: ":/var/gerrit/git"
diff --git a/src/test/docker/gerrit/Dockerfile b/src/test/docker/gerrit/Dockerfile index 820be4c..30912e9 100644 --- a/src/test/docker/gerrit/Dockerfile +++ b/src/test/docker/gerrit/Dockerfile
@@ -1,27 +1,52 @@ -FROM gerritcodereview/gerrit:3.5.0-rc2 +FROM almalinux:8.5 -ENV GERRIT_BRANCH=stable-3.5 +# Install dependencies +RUN yum -y install \ + git \ + java-11-openjdk \ + procps \ + sudo \ + passwd \ + gettext \ + && yum -y clean all -ENV GERRIT_CI_URL=https://gerrit-ci.gerritforge.com/job +ENV GERRIT_VERSION=3.6 +ENV JAVA_HOME /usr/lib/jvm/jre-11-openjdk -USER root +# Add gerrit user +RUN adduser -p -m --uid 1000 gerrit --home-dir /home/gerrit +RUN echo "gerrit:gerrit" | chpasswd +RUN echo "gerrit ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/gerrit && \ + chmod 0440 /etc/sudoers.d/gerrit -RUN yum install -y iputils nmap curl lsof gettext net-tools sudo +# Create gerrit installation directory +RUN mkdir -p /var/gerrit && chown -R gerrit /var/gerrit +ADD --chown=gerrit \ + "https://gerrit-ci.gerritforge.com/job/Gerrit-bazel-stable-$GERRIT_VERSION/lastSuccessfulBuild/artifact/gerrit/bazel-bin/release.war" \ + /tmp/gerrit.war + +ADD --chown=gerrit \ +"https://gerrit-ci.gerritforge.com/job/plugin-javamelody-bazel-master-stable-$GERRIT_VERSION/lastSuccessfulBuild/artifact/bazel-bin/plugins/javamelody/javamelody.jar" \ + /var/gerrit/plugins/javamelody.jar +ADD --chown=gerrit \ + "https://gerrit-ci.gerritforge.com/job/plugin-high-availability-bazel-stable-$GERRIT_VERSION/lastSuccessfulBuild/artifact/bazel-bin/plugins/high-availability/high-availability.jar" \ + /var/gerrit/plugins/high-availability.jar + +ADD --chown=gerrit \ + "https://repo1.maven.org/maven2/com/gerritforge/global-refdb/$GERRIT_VERSION.3.4/global-refdb-$GERRIT_VERSION.3.4.jar" \ + /tmp + +ADD --chown=gerrit:gerrit ./wait-for-it.sh /bin + +# Change user USER gerrit -ADD --chown=gerrit:gerrit $GERRIT_CI_URL/plugin-javamelody-bazel-master-$GERRIT_BRANCH/lastSuccessfulBuild/artifact/bazel-bin/plugins/javamelody/javamelody.jar /var/gerrit/plugins/javamelody.jar -ADD --chown=gerrit:gerrit $GERRIT_CI_URL/plugin-high-availability-bazel-master-$GERRIT_BRANCH/lastSuccessfulBuild/artifact/bazel-bin/plugins/high-availability/high-availability.jar /var/gerrit/plugins/high-availability.jar +# Expose ssh and http ports +EXPOSE 29418 8080 + +COPY --chown=gerrit ./entrypoint.sh /bin USER root -ADD start.sh /bin/ -ADD wait-for-it.sh /bin/ - -RUN rm -Rf /var/gerrit/{git,index,cache}/* - -ARG GERRIT_UID=1000 -RUN usermod -u ${GERRIT_UID} gerrit &> /dev/null - -ENTRYPOINT ["/usr/bin/env"] -CMD /bin/start.sh +ENTRYPOINT /bin/entrypoint.sh \ No newline at end of file
diff --git a/src/test/docker/gerrit/entrypoint.sh b/src/test/docker/gerrit/entrypoint.sh new file mode 100755 index 0000000..b382a3b --- /dev/null +++ b/src/test/docker/gerrit/entrypoint.sh
@@ -0,0 +1,27 @@ +#!/bin/sh + +if [[ ! -z "$WAIT_FOR" ]] +then + wait-for-it.sh $WAIT_FOR -t 600 -- echo "$WAIT_FOR is up" +fi + +chown -R gerrit /var/gerrit/etc +sudo -u gerrit cp /var/gerrit/etc/gerrit.config.orig /var/gerrit/etc/gerrit.config +sudo -u gerrit cp /var/gerrit/etc/high-availability.config.orig /var/gerrit/etc/high-availability.config + + +echo "Init gerrit..." +sudo -u gerrit java -jar /tmp/gerrit.war init -d /var/gerrit --batch --install-all-plugins +chown -R gerrit: /var/gerrit/shareddir + +# required until regression is fixed, see https://groups.google.com/g/repo-discuss/c/DH-ftHMiCyE/m/qF88c6KMAAAJ +echo "Copying global ref db jar into lib" +sudo -u gerrit cp /tmp/global-refdb-*.jar /var/gerrit/lib + +echo "Reindexing Gerrit..." +cd /var/gerrit && sudo -u gerrit java -jar /var/gerrit/bin/gerrit.war reindex -d /var/gerrit +sudo -u gerrit git config -f /var/gerrit/etc/gerrit.config gerrit.canonicalWebUrl http://$HOSTNAME/ +sudo -u gerrit touch /var/gerrit/logs/{gc_log,error_log,httpd_log,sshd_log,replication_log} && tail -f /var/gerrit/logs/* | grep --line-buffered -v 'HEAD /' & + +echo "Running Gerrit ..." +sudo -u gerrit /var/gerrit/bin/gerrit.sh run
diff --git a/src/test/docker/gerrit/start.sh b/src/test/docker/gerrit/start.sh deleted file mode 100755 index d906628..0000000 --- a/src/test/docker/gerrit/start.sh +++ /dev/null
@@ -1,25 +0,0 @@ -#!/bin/bash -e - -if [[ ! -z "$WAIT_FOR" ]] -then - wait-for-it.sh $WAIT_FOR -t 600 -- echo "$WAIT_FOR is up" -fi - -sudo -u gerrit cp /var/gerrit/etc/gerrit.config.orig /var/gerrit/etc/gerrit.config -sudo -u gerrit cp /var/gerrit/etc/high-availability.config.orig /var/gerrit/etc/high-availability.config - -if [[ ! -f /var/gerrit/etc/ssh_host_ed25519_key ]] -then - echo "Initializing Gerrit site ..." - sudo -u gerrit java -jar /var/gerrit/bin/gerrit.war init -d /var/gerrit --batch - chown -R gerrit: /var/gerrit/shareddir -fi - -echo "Reindexing Gerrit ..." -sudo -u gerrit java -jar /var/gerrit/bin/gerrit.war reindex -d /var/gerrit -sudo -u gerrit git config -f /var/gerrit/etc/gerrit.config gerrit.canonicalWebUrl http://$HOSTNAME/ - -sudo -u gerrit touch /var/gerrit/logs/{gc_log,error_log,httpd_log,sshd_log,replication_log} && tail -f /var/gerrit/logs/* | grep --line-buffered -v 'HEAD /' & - -echo "Running Gerrit ..." -sudo -u gerrit /var/gerrit/bin/gerrit.sh run
diff --git a/src/test/docker/nfs/Dockerfile b/src/test/docker/nfs/Dockerfile new file mode 100644 index 0000000..19744f3 --- /dev/null +++ b/src/test/docker/nfs/Dockerfile
@@ -0,0 +1,13 @@ +FROM erichough/nfs-server + +COPY exports.txt /etc/exports + +# To avoid ownership issues, the user must be the same betweeen the +# server and the client, hence we are creating it explicitly in both. +RUN adduser --disabled-password --gecos "" --uid 1000 gerrit + +# /var/gerrit/git is the shared directory +RUN mkdir --parents /var/gerrit/git + +RUN chown gerrit:gerrit /var/lib/nfs +RUN chown gerrit:gerrit /var/gerrit/git \ No newline at end of file
diff --git a/src/test/docker/nfs/exports.txt b/src/test/docker/nfs/exports.txt new file mode 100644 index 0000000..c31d586 --- /dev/null +++ b/src/test/docker/nfs/exports.txt
@@ -0,0 +1 @@ +/var/gerrit/git 192.168.1.0/24(rw,no_subtree_check,insecure) \ No newline at end of file