| :linkattrs: | |
| = Gerrit Code Review - Building with Bazel | |
| [[summary]] | |
| == TL;DR | |
| If you have the prerequisites, running | |
| ``` | |
| $ bazel build gerrit | |
| ``` | |
| should generate a .war file under `bazel-bin/gerrit.war`. | |
| [[installation]] | |
| == Prerequisites | |
| To build Gerrit from source, you need: | |
| * A Linux or macOS system (Windows is not supported at this time) | |
| * A JDK for Java 11 or Java 17 | |
| * Python 3 | |
| * link:https://github.com/nodesource/distributions/blob/master/README.md[Node.js (including npm),role=external,window=_blank] | |
| * Bower (`npm install -g bower`) | |
| * link:https://docs.bazel.build/versions/master/install.html[Bazel,role=external,window=_blank] -launched with | |
| link:https://github.com/bazelbuild/bazelisk[Bazelisk,role=external,window=_blank] | |
| * Maven | |
| * zip, unzip | |
| * curl | |
| * gcc | |
| [[bazel]] | |
| === Bazel | |
| link:https://github.com/bazelbuild/bazelisk[Bazelisk,role=external,window=_blank] is a version | |
| manager for link:https://bazel.build/[Bazel,role=external,window=_blank], similar to how `nvm` | |
| manages `npm` versions. It takes care of downloading and installing Bazel itself, so you don't have | |
| to worry about using the correct version of Bazel. Bazelisk can be installed in different | |
| ways: link:https://docs.bazel.build/install-bazelisk.html[Install,role=external,window=_blank] | |
| [[java]] | |
| === Java | |
| Ensure that the link:install.html#Requirements[required Java version] | |
| is installed and that `JAVA_HOME` is set to it. | |
| To check the installed version of Java, open a terminal window and run: | |
| `java -version` | |
| [[java-11]] | |
| ==== Java 11 support | |
| To build Gerrit with Java 11 language level, run: | |
| ``` | |
| $ bazel build --java_toolchain=//tools:error_prone_warnings_toolchain_java11 :release | |
| ``` | |
| [[java-17]] | |
| ==== Java 17 support | |
| Java 17 is supported. To build Gerrit with Java 17, run: | |
| ``` | |
| $ bazel build --config=java17 :release | |
| ``` | |
| To run the tests with Java 17, run: | |
| ``` | |
| $ bazel test --config=java17 //... | |
| ``` | |
| === Node.js and npm packages | |
| See link:https://gerrit.googlesource.com/gerrit/+/master/polygerrit-ui/README.md#installing-node_js-and-npm-packages[Installing Node.js and npm packages,role=external,window=_blank]. | |
| [[build]] | |
| == Building on the Command Line | |
| === Gerrit Development WAR File | |
| To build the Gerrit web application: | |
| ---- | |
| bazel build gerrit | |
| ---- | |
| The output executable WAR will be placed in: | |
| ---- | |
| bazel-bin/gerrit.war | |
| ---- | |
| [[release]] | |
| === Gerrit Release WAR File | |
| To build the Gerrit web application that includes the Gerrit UI, | |
| core plugins and documentation: | |
| ---- | |
| bazel build release | |
| ---- | |
| The output executable WAR will be placed in: | |
| ---- | |
| bazel-bin/release.war | |
| ---- | |
| === Headless Mode | |
| To build Gerrit in headless mode, i.e. without the Gerrit UI: | |
| ---- | |
| bazel build headless | |
| ---- | |
| The output executable WAR will be placed in: | |
| ---- | |
| bazel-bin/headless.war | |
| ---- | |
| === Extension and Plugin API JAR Files | |
| To build the extension, plugin and acceptance-framework JAR files: | |
| ---- | |
| bazel build api | |
| ---- | |
| The output archive that contains Java binaries, Java sources and | |
| Java docs will be placed in: | |
| ---- | |
| bazel-bin/api.zip | |
| ---- | |
| Install {extension,plugin,acceptance-framework}-api to the local | |
| maven repository: | |
| ---- | |
| tools/maven/api.sh install | |
| ---- | |
| Install gerrit.war to the local maven repository: | |
| ---- | |
| tools/maven/api.sh war_install | |
| ---- | |
| === Plugins | |
| ---- | |
| bazel build plugins:core | |
| ---- | |
| The output JAR files for individual plugins will be placed in: | |
| ---- | |
| bazel-bin/plugins/<name>/<name>.jar | |
| ---- | |
| The JAR files will also be packaged in: | |
| ---- | |
| bazel-bin/plugins/core.zip | |
| ---- | |
| To build a specific plugin: | |
| ---- | |
| bazel build plugins/<name> | |
| ---- | |
| The output JAR file will be be placed in: | |
| ---- | |
| bazel-bin/plugins/<name>/<name>.jar | |
| ---- | |
| Note that when building an individual plugin, the `core.zip` package | |
| is not regenerated. | |
| [[IDEs]] | |
| == Using an IDE. | |
| === IntelliJ | |
| The Gerrit build works with Bazel's link:https://ij.bazel.build[IntelliJ plugin,role=external,window=_blank]. | |
| Please follow the instructions on <<dev-intellij#,IntelliJ Setup>>. | |
| === Eclipse | |
| ==== Generating the Eclipse Project | |
| Create the Eclipse project: | |
| ---- | |
| tools/eclipse/project.py | |
| ---- | |
| and then follow the link:dev-eclipse.html#setup[setup instructions]. | |
| ==== Refreshing the Classpath | |
| If an updated classpath is needed, the Eclipse project can be | |
| refreshed and missing dependency JARs can be downloaded by running | |
| `project.py` again. For IntelliJ, you need to click the `Sync Project | |
| with BUILD Files` button of link:https://ij.bazel.build[Bazel plugin,role=external,window=_blank]. | |
| [[documentation]] | |
| == Documentation | |
| To build only the documentation for testing or static hosting: | |
| ---- | |
| bazel build Documentation:searchfree | |
| ---- | |
| The html files will be bundled into `searchfree.zip` in this location: | |
| ---- | |
| bazel-bin/Documentation/searchfree.zip | |
| ---- | |
| To generate HTML files skipping the zip archiving: | |
| ---- | |
| bazel build Documentation | |
| ---- | |
| And open `bazel-bin/Documentation/index.html`. | |
| To build the Gerrit executable WAR with the documentation included: | |
| ---- | |
| bazel build withdocs | |
| ---- | |
| The WAR file will be placed in: | |
| ---- | |
| bazel-bin/withdocs.war | |
| ---- | |
| Alternatively, one can generate the documentation as flat files: | |
| ---- | |
| bazel build Documentation:Documentation | |
| ---- | |
| The html, css, js files are placed in: | |
| ---- | |
| `bazel-bin/Documentation/` | |
| ---- | |
| [[tests]] | |
| == Running Unit Tests | |
| ---- | |
| bazel test --build_tests_only //... | |
| ---- | |
| Debugging tests: | |
| ---- | |
| bazel test --test_output=streamed --test_filter=com.gerrit.TestClass.testMethod testTarget | |
| ---- | |
| Debug test example: | |
| ---- | |
| bazel test --test_output=streamed --test_filter=com.google.gerrit.acceptance.api.change.ChangeIT.getAmbiguous //javatests/com/google/gerrit/acceptance/api/change:api_change | |
| ---- | |
| To run a specific test group, e.g. the rest-account test group: | |
| ---- | |
| bazel test //javatests/com/google/gerrit/acceptance/rest/account:rest_account | |
| ---- | |
| To run only tests that do not use SSH: | |
| ---- | |
| bazel test --test_env=GERRIT_USE_SSH=NO //... | |
| ---- | |
| To exclude tests that have been marked as flaky: | |
| ---- | |
| bazel test --test_tag_filters=-flaky //... | |
| ---- | |
| To exclude tests that require very recent git client version: | |
| ---- | |
| bazel test --test_tag_filters=-git-protocol-v2 //... | |
| ---- | |
| To ignore cached test results: | |
| ---- | |
| bazel test --cache_test_results=NO //... | |
| ---- | |
| To run one or more specific groups of tests: | |
| ---- | |
| bazel test --test_tag_filters=api,git //... | |
| ---- | |
| To run the tests against a specific index backend (LUCENE, FAKE): | |
| ---- | |
| bazel test --test_env=GERRIT_INDEX_TYPE=LUCENE //... | |
| ---- | |
| The following values are currently supported for the group name: | |
| * annotation | |
| * api | |
| * edit | |
| * git | |
| * git-protocol-v2 | |
| * git-upload-archive | |
| * notedb | |
| * pgm | |
| * rest | |
| * server | |
| * ssh | |
| Bazel itself supports a multitude of ways to | |
| link:https://docs.bazel.build/versions/master/guide.html#specifying-targets-to-build[specify targets,role=external,window=_blank] | |
| for fine-grained test selection that can be combined with many of the examples | |
| above. | |
| [[debugging-tests]] | |
| == Debugging Unit Tests | |
| In some cases it may be necessary to debug a test while running it in bazel. For example, when we | |
| observe a different test result in Eclipse and bazel. Using the `--java_debug` option will start the | |
| JVM in debug mode and await for a remote debugger to attach. | |
| Example: | |
| [source,bash] | |
| ---- | |
| bazel test --java_debug --test_tag_filters=delete-project //... | |
| ... | |
| Listening for transport dt_socket at address: 5005 | |
| ... | |
| ---- | |
| Now attach with a debugger to the port `5005`. For example use "Remote Java Application" launch | |
| configuration in Eclipe and specify the port `5005`. | |
| [[logging]] | |
| === Controlling logging level | |
| Per default, logging level is set to `INFO` level for all tests. The `DEBUG` | |
| log level can be enabled for the tests. | |
| In IDE, set `-Dgerrit.logLevel=debug` as a VM argument. With `bazel`, pass | |
| `GERRIT_LOG_LEVEL=debug` environment variable: | |
| ---- | |
| bazel test --test_filter=com.google.gerrit.server.notedb.ChangeNotesTest \ | |
| --test_env=GERRIT_LOG_LEVEL=debug \ | |
| javatests/com/google/gerrit/server:server_tests | |
| ---- | |
| The log results can be found in: | |
| `bazel-testlogs/javatests/com/google/gerrit/server/server_tests/test.log`. | |
| == Dependencies | |
| Dependency JARs are normally downloaded as needed, but you can | |
| download everything upfront. This is useful to enable | |
| subsequent builds to run without network access: | |
| ---- | |
| bazel fetch //... | |
| ---- | |
| When downloading from behind a proxy (which is common in some corporate | |
| environments), it might be necessary to explicitly specify the proxy that | |
| is then used by `curl`: | |
| ---- | |
| export http_proxy=http://<proxy_user_id>:<proxy_password>@<proxy_server>:<proxy_port> | |
| ---- | |
| Redirection to local mirrors of Maven Central and the Gerrit storage | |
| bucket is supported by defining specific properties in | |
| `local.properties`, a file that is not tracked by Git: | |
| ---- | |
| echo download.GERRIT = http://nexus.my-company.com/ >>local.properties | |
| echo download.MAVEN_CENTRAL = http://nexus.my-company.com/ >>local.properties | |
| ---- | |
| The `local.properties` file may be placed in the root of the gerrit repository | |
| being built, or in `~/.gerritcodereview/`. The file in the root of the gerrit | |
| repository has precedence. | |
| == Building against unpublished Maven JARs | |
| To build against unpublished Maven JARs, like PrologCafe, the custom JARs must | |
| be installed in the local Maven repository (`mvn clean install`) and | |
| `maven_jar()` must be updated to point to the `MAVEN_LOCAL` Maven repository for | |
| that artifact: | |
| [source,python] | |
| ---- | |
| maven_jar( | |
| name = 'prolog-runtime', | |
| artifact = 'com.googlecode.prolog-cafe:prolog-runtime:42', | |
| repository = MAVEN_LOCAL, | |
| ) | |
| ---- | |
| == Building against artifacts from custom Maven repositories | |
| To build against custom Maven repositories, two modes of operations are | |
| supported: with rewrite in local.properties and without. | |
| Without rewrite the URL of custom Maven repository can be directly passed | |
| to the maven_jar() function: | |
| [source,python] | |
| ---- | |
| GERRIT_FORGE = 'http://gerritforge.com/snapshot' | |
| maven_jar( | |
| name = 'gitblit', | |
| artifact = 'com.gitblit:gitblit:1.4.0', | |
| sha1 = '1b130dbf5578ace37507430a4a523f6594bf34fa', | |
| repository = GERRIT_FORGE, | |
| ) | |
| ---- | |
| When the custom URL has to be rewritten, then the same logic as with Gerrit | |
| known Maven repository is used: Repo name must be defined that matches an entry | |
| in local.properties file: | |
| ---- | |
| download.GERRIT_FORGE = http://my.company.mirror/gerrit-forge | |
| ---- | |
| And corresponding WORKSPACE excerpt: | |
| [source,python] | |
| ---- | |
| GERRIT_FORGE = 'GERRIT_FORGE:' | |
| maven_jar( | |
| name = 'gitblit', | |
| artifact = 'com.gitblit:gitblit:1.4.0', | |
| sha1 = '1b130dbf5578ace37507430a4a523f6594bf34fa', | |
| repository = GERRIT_FORGE, | |
| ) | |
| ---- | |
| == Building against SNAPSHOT Maven JARs | |
| To build against SNAPSHOT Maven JARs, the complete SNAPSHOT version must be used: | |
| [source,python] | |
| ---- | |
| maven_jar( | |
| name = "pac4j-core", | |
| artifact = "org.pac4j:pac4j-core:3.5.0-SNAPSHOT-20190112.120241-16", | |
| sha1 = "da2b1cb68a8f87bfd40813179abd368de9f3a746", | |
| ) | |
| ---- | |
| [[bazel-local-caches]] | |
| To accelerate builds, several caches are activated per default: | |
| * ~/.gerritcodereview/bazel-cache/downloaded-artifacts | |
| * ~/.gerritcodereview/bazel-cache/repository | |
| * ~/.gerritcodereview/bazel-cache/cas | |
| The `downloaded-artifacts` cache can be relocated by setting the | |
| `GERRIT_CACHE_HOME` environment variable. The other two can be adjusted with | |
| `bazel build` options `--repository_cache` and `--disk_cache` respectively. | |
| Currently none of these caches have a maximum size limit. See | |
| link:https://github.com/bazelbuild/bazel/issues/5139[this bazel issue,role=external,window=_blank] for | |
| details. Users should watch the cache sizes and clean them manually if | |
| necessary. | |
| [[npm-binary]] | |
| == NPM Binaries | |
| Parts of the Gerrit web app build require running NPM-based JavaScript programs | |
| as "binaries". We don't attempt to resolve and download NPM dependencies at | |
| build time, but instead use pre-built bundles of the NPM binary along with all | |
| its dependencies. Some packages on | |
| link:https://docs.npmjs.com/misc/registry[registry.npmjs.org,role=external,window=_blank] come with their | |
| dependencies bundled, but this is the exception rather than the rule. More | |
| commonly, to add a new binary to this list, you will need to bundle the binary | |
| yourself. | |
| [NOTE] | |
| We can only use binaries that meet certain licensing requirements, and that do | |
| not include any native code. | |
| Start by checking that the license and file types of the bundle are acceptable: | |
| [source,bash] | |
| ---- | |
| gerrit_repo=/path/to/gerrit | |
| package=some-npm-package | |
| version=1.2.3 | |
| # Note - yarn must be installed before running the following commands | |
| yarn global add license-checker && \ | |
| rm -rf /tmp/$package-$version && mkdir -p /tmp/$package-$version && \ | |
| cd /tmp/$package-$version && \ | |
| yarn add $package@$version && \ | |
| license-checker | grep licenses: | sort -u | |
| ---- | |
| This will output a list of the different licenses used by the package and all | |
| its transitive dependencies. We can only legally distribute a bundle via our | |
| storage bucket if the licenses allow us to do so. As long as all of the listed | |
| license are allowed by | |
| link:https://opensource.google.com/docs/thirdparty/licenses/[Google's | |
| standards,role=external,window=_blank]. Any `by_exception_only`, commercial, prohibited, or unlisted | |
| licenses are not allowed; otherwise, it is ok to distribute the source. If in | |
| doubt, contact a maintainer who is a Googler. | |
| Next, check the file types: | |
| [source,bash] | |
| ---- | |
| cd /tmp/$package-$version | |
| find . -type f | xargs file | grep -v 'ASCII\|UTF-8\|empty$' | |
| ---- | |
| If you see anything that looks like a native library or binary, then we can't | |
| use the bundle. | |
| If everything looks good, install the package with the following command: | |
| [source, bash] | |
| ---- | |
| # Add to ui_npm. Other packages.json can be updated in the same way | |
| cd $gerrit_repo/polygerrit-ui/app | |
| bazel run @nodejs//:yarn add $package | |
| ---- | |
| Update the `polygerrit-ui/app/node_modules_licenses/licenses.ts` file. You should add licenses | |
| for the package itself and for all transitive depndencies. If you forgot to add a license, the | |
| `Documentation:check_licenses` test will fail. | |
| After the update, commit all changes to the repository (including `yarn.lock`). | |
| [NOTE] | |
| ==== | |
| If a npm package has transitive dependencies (or just several files) with a not allowed | |
| license and you can't avoid use it in release, then you can add this package. | |
| For example some packages contain demo-code with a different license. Another example - optional | |
| dependencies, which are not needed to build the Gerrit web app, but they are installed together | |
| with the package anyway. | |
| In this case you should exclude all files and/or transitive dependencies with a not allowed license. | |
| Adding such package requires additional updates: | |
| - Add dependencies (or files) to the license.ts with an appropriate license marked with | |
| `allowed: false`. | |
| - update package.json postinstall script to remove all non-allowed files (if you don't | |
| update postinstall script, `Documentation:check_licenses` test will fail.) | |
| ==== | |
| === Update NPM Binaries | |
| To update a NPM binary the same actions as for a new one must be done (check licenses, | |
| update `licenses.ts` file, etc...). The only difference is a command to install a package: instead | |
| of `bazel run @nodejs//:yarn add $package` you should run the `bazel run @nodejs//:yarn upgrade ...` | |
| command with correct arguments. You can find the list of arguments in the | |
| link:https://classic.yarnpkg.com/en/docs/cli/upgrade/[yarn upgrade doc,role=external,window=_blank]. | |
| [[RBE]] | |
| == Google Remote Build Support | |
| The Bazel build can be used with Google's Remote Build Execution. | |
| This needs the following setup steps: | |
| ``` | |
| gcloud auth application-default login | |
| gcloud services enable remotebuildexecution.googleapis.com --project=${PROJECT} | |
| ``` | |
| Create a worker pool. The instances should have at least 4 CPUs each | |
| for adequate performance. | |
| ``` | |
| gcloud alpha remote-build-execution worker-pools create default \ | |
| --project=${PROJECT} \ | |
| --instance=default_instance \ | |
| --worker-count=50 \ | |
| --machine-type=e2-standard-4 \ | |
| --disk-size=200 | |
| ``` | |
| Due to outdated Git version in official RBE docker images, a custom RBE docker | |
| image must be used. To build custom docker imager, change to the directory | |
| `tools/platforms` and build and publish custom RBE docker image. | |
| To build the custom RBE docker image, run: | |
| ``` | |
| docker build -t gcr.io/api-project-164060093628/rbe-ubuntu18-04 . | |
| ``` | |
| To publish the custom RBE docker image, run: | |
| ``` | |
| docker push gcr.io/api-project-164060093628/rbe-ubuntu18-04 | |
| [...] | |
| latest: digest: sha256:de5186d4313630a6111f9a2449b72563d0bc59ec9fb60956f063b69a38a76834 size: 1584 | |
| ``` | |
| Re-build rbe_autoconfig project conduct a new release and switch to using it | |
| in `WORKSPACE` file. | |
| Note, to authenticate to the gcr.io registry, the following command must be | |
| used: | |
| ``` | |
| gcloud auth configure-docker | |
| ``` | |
| To see the documentation, developer must be added to this group: | |
| https://groups.google.com/forum/#!forum/rbe-alpha-customers. | |
| Documentation can be found at: | |
| https://cloud.google.com/remote-build-execution/docs. | |
| To use RBE, execute | |
| ``` | |
| bazel test --config=remote \ | |
| --remote_instance_name=projects/${PROJECT}/instances/default_instance \ | |
| javatests/... | |
| ``` | |
| GERRIT | |
| ------ | |
| Part of link:index.html[Gerrit Code Review] | |
| SEARCHBOX | |
| --------- |