| :linkattrs: |
| = Gerrit Code Review - Building with Bazel |
| |
| [[summary]] |
| == TL;DR |
| |
| If you have the prerequisites, running |
| |
| ``` |
| $ bazel build gerrit |
| ``` |
| |
| should generate a .war file under `bazel-bin/gerrit.war`. |
| |
| [[installation]] |
| == Prerequisites |
| |
| To build Gerrit from source, you need: |
| |
| * A Linux or macOS system (Windows is not supported at this time) |
| * A JDK for Java 8|9|10|11|... |
| * Python 2 or 3 |
| * link:https://github.com/nodesource/distributions/blob/master/README.md[Node.js (including npm),role=external,window=_blank] |
| * Bower (`sudo npm install -g bower`) |
| * link:https://docs.bazel.build/versions/master/install.html[Bazel,role=external,window=_blank] -launched with |
| link:https://github.com/bazelbuild/bazelisk[Bazelisk,role=external,window=_blank] |
| * Maven |
| * zip, unzip |
| * curl |
| * gcc |
| |
| [[bazel]] |
| === Bazel |
| |
| link:https://github.com/bazelbuild/bazelisk[Bazelisk,role=external,window=_blank] includes a |
| link:https://bazel.build/[Bazel,role=external,window=_blank] version check and downloads the correct |
| `bazel` version for the git project/repository. Bazelisk is the recommended |
| `bazel` launcher for Gerrit. Once Bazelisk is installed locally, a `bazel` |
| symlink can be created towards it. This is so that every `bazel` command |
| seamlessly uses Bazelisk, which then runs the proper `bazel` binary version. |
| |
| [[java]] |
| === Java |
| |
| ==== MacOS |
| |
| On MacOS, ensure that "Java for MacOS X 10.5 Update 4" (or higher) is installed |
| and that `JAVA_HOME` is set to the |
| link:install.html#Requirements[required Java version]. |
| |
| Java installations can typically be found in |
| "/System/Library/Frameworks/JavaVM.framework/Versions". |
| |
| To check the installed version of Java, open a terminal window and run: |
| |
| `java -version` |
| |
| [[java-13]] |
| ==== Java 13 support |
| |
| Java 13 (and newer) is supported through vanilla java toolchain |
| link:https://docs.bazel.build/versions/master/toolchains.html[Bazel option,role=external,window=_blank]. |
| To build Gerrit with Java 13 and newer, specify vanilla java toolchain and |
| provide the path to JDK home: |
| |
| ``` |
| $ bazel build \ |
| --define=ABSOLUTE_JAVABASE=<path-to-java-13> \ |
| --javabase=@bazel_tools//tools/jdk:absolute_javabase \ |
| --host_javabase=@bazel_tools//tools/jdk:absolute_javabase \ |
| --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla \ |
| --java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla \ |
| :release |
| ``` |
| |
| To run the tests, `--javabase` option must be passed as well, because |
| bazel test runs the test using the target javabase: |
| |
| ``` |
| $ bazel test \ |
| --define=ABSOLUTE_JAVABASE=<path-to-java-13> \ |
| --javabase=@bazel_tools//tools/jdk:absolute_javabase \ |
| --host_javabase=@bazel_tools//tools/jdk:absolute_javabase \ |
| --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla \ |
| --java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla \ |
| //... |
| ``` |
| |
| To avoid passing all those options on every Bazel build invocation, |
| they could be added to ~/.bazelrc resource file: |
| |
| ``` |
| $ cat << EOF > ~/.bazelrc |
| > build --define=ABSOLUTE_JAVABASE=<path-to-java-13> |
| > build --javabase=@bazel_tools//tools/jdk:absolute_javabase |
| > build --host_javabase=@bazel_tools//tools/jdk:absolute_javabase |
| > build --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla |
| > build --java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla |
| > EOF |
| ``` |
| |
| Now, invoking Bazel with just `bazel build :release` would include |
| all those options. |
| |
| [[java-11]] |
| ==== Java 11 support |
| |
| Java 11 is supported through alternative java toolchain |
| link:https://docs.bazel.build/versions/master/toolchains.html[Bazel option]. |
| To build Gerrit with Java 11, specify JDK 11 java toolchain: |
| |
| ``` |
| $ bazel build \ |
| --host_javabase=@bazel_tools//tools/jdk:remote_jdk11 \ |
| --javabase=@bazel_tools//tools/jdk:remote_jdk11 \ |
| --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_java11 \ |
| --java_toolchain=@bazel_tools//tools/jdk:toolchain_java11 \ |
| :release |
| ``` |
| |
| === Node.js and npm packages |
| See link:https://gerrit.googlesource.com/gerrit/+/master/polygerrit-ui/README.md#installing-node_js-and-npm-packages[Installing Node.js and npm packages,role=external,window=_blank]. |
| |
| [[build]] |
| == Building on the Command Line |
| |
| === Gerrit Development WAR File |
| |
| To build the Gerrit web application: |
| |
| ---- |
| bazel build gerrit |
| ---- |
| |
| The output executable WAR will be placed in: |
| |
| ---- |
| bazel-bin/gerrit.war |
| ---- |
| |
| [[release]] |
| === Gerrit Release WAR File |
| |
| To build the Gerrit web application that includes the PolyGerrit UI, |
| core plugins and documentation: |
| |
| ---- |
| bazel build release |
| ---- |
| |
| The output executable WAR will be placed in: |
| |
| ---- |
| bazel-bin/release.war |
| ---- |
| |
| === Headless Mode |
| |
| To build Gerrit in headless mode, i.e. without the PolyGerrit UI: |
| Web UI: |
| |
| ---- |
| bazel build headless |
| ---- |
| |
| The output executable WAR will be placed in: |
| |
| ---- |
| bazel-bin/headless.war |
| ---- |
| |
| === Extension and Plugin API JAR Files |
| |
| To build the extension, plugin and acceptance-framework JAR files: |
| |
| ---- |
| bazel build api |
| ---- |
| |
| The output archive that contains Java binaries, Java sources and |
| Java docs will be placed in: |
| |
| ---- |
| bazel-bin/api.zip |
| ---- |
| |
| Install {extension,plugin,acceptance-framework}-api to the local |
| maven repository: |
| |
| ---- |
| tools/maven/api.sh install |
| ---- |
| |
| Install gerrit.war to the local maven repository: |
| |
| ---- |
| tools/maven/api.sh war_install |
| ---- |
| |
| === Plugins |
| |
| ---- |
| bazel build plugins:core |
| ---- |
| |
| The output JAR files for individual plugins will be placed in: |
| |
| ---- |
| bazel-bin/plugins/<name>/<name>.jar |
| ---- |
| |
| The JAR files will also be packaged in: |
| |
| ---- |
| bazel-bin/plugins/core.zip |
| ---- |
| |
| To build a specific plugin: |
| |
| ---- |
| bazel build plugins/<name> |
| ---- |
| |
| The output JAR file will be be placed in: |
| |
| ---- |
| bazel-bin/plugins/<name>/<name>.jar |
| ---- |
| |
| Note that when building an individual plugin, the `core.zip` package |
| is not regenerated. |
| |
| [[IDEs]] |
| == Using an IDE. |
| |
| === IntelliJ |
| |
| The Gerrit build works with Bazel's link:https://ij.bazel.build[IntelliJ plugin,role=external,window=_blank]. |
| Please follow the instructions on <<dev-intellij#,IntelliJ Setup>>. |
| |
| === Eclipse |
| |
| ==== Generating the Eclipse Project |
| |
| Create the Eclipse project: |
| |
| ---- |
| tools/eclipse/project.py |
| ---- |
| |
| and then follow the link:dev-eclipse.html#setup[setup instructions]. |
| |
| ==== Refreshing the Classpath |
| |
| If an updated classpath is needed, the Eclipse project can be |
| refreshed and missing dependency JARs can be downloaded by running |
| `project.py` again. For IntelliJ, you need to click the `Sync Project |
| with BUILD Files` button of link:https://ij.bazel.build[Bazel plugin,role=external,window=_blank]. |
| |
| [[documentation]] |
| === Documentation |
| |
| To build only the documentation for testing or static hosting: |
| |
| ---- |
| bazel build Documentation:searchfree |
| ---- |
| |
| The html files will be bundled into `searchfree.zip` in this location: |
| |
| ---- |
| bazel-bin/Documentation/searchfree.zip |
| ---- |
| |
| To build the executable WAR with the documentation included: |
| |
| ---- |
| bazel build withdocs |
| ---- |
| |
| The WAR file will be placed in: |
| |
| ---- |
| bazel-bin/withdocs.war |
| ---- |
| |
| [[tests]] |
| == Running Unit Tests |
| |
| ---- |
| bazel test --build_tests_only //... |
| ---- |
| |
| Debugging tests: |
| |
| ---- |
| bazel test --test_output=streamed --test_filter=com.gerrit.TestClass.testMethod testTarget |
| ---- |
| |
| Debug test example: |
| |
| ---- |
| bazel test --test_output=streamed --test_filter=com.google.gerrit.acceptance.api.change.ChangeIT.getAmbiguous //javatests/com/google/gerrit/acceptance/api/change:api_change |
| ---- |
| |
| To run a specific test group, e.g. the rest-account test group: |
| |
| ---- |
| bazel test //javatests/com/google/gerrit/acceptance/rest/account:rest_account |
| ---- |
| |
| To run only tests that do not use SSH: |
| |
| ---- |
| bazel test --test_env=GERRIT_USE_SSH=NO //... |
| ---- |
| |
| To exclude tests that have been marked as flaky: |
| |
| ---- |
| bazel test --test_tag_filters=-flaky //... |
| ---- |
| |
| To exclude tests that require a Docker host: |
| |
| ---- |
| bazel test --test_tag_filters=-docker //... |
| ---- |
| |
| To exclude tests that require very recent git client version: |
| |
| ---- |
| bazel test --test_tag_filters=-git-protocol-v2 //... |
| ---- |
| |
| To ignore cached test results: |
| |
| ---- |
| bazel test --cache_test_results=NO //... |
| ---- |
| |
| To run one or more specific groups of tests: |
| |
| ---- |
| bazel test --test_tag_filters=api,git //... |
| ---- |
| |
| The following values are currently supported for the group name: |
| |
| * annotation |
| * api |
| * docker |
| * edit |
| * elastic |
| * git |
| * git-protocol-v2 |
| * notedb |
| * pgm |
| * rest |
| * server |
| * ssh |
| |
| [[elasticsearch]] |
| === Elasticsearch |
| |
| Successfully running the Elasticsearch tests requires Docker, and |
| may require setting the local virtual memory on |
| link:https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html[linux,role=external,window=_blank] and |
| link:https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_set_vm_max_map_count_to_at_least_262144[macOS,role=external,window=_blank]. |
| |
| On macOS, if using link:https://docs.docker.com/docker-for-mac/[Docker Desktop,role=external,window=_blank], |
| the effective memory value can be set in the Preferences, under the Advanced tab. |
| The default value usually does not suffice and is causing premature container exits. |
| That default is currently 2 GB and should be set to at least 5 (GB). |
| |
| If Docker is not available, the Elasticsearch tests will be skipped. |
| Note that Bazel currently does not show |
| link:https://github.com/bazelbuild/bazel/issues/3476[the skipped tests,role=external,window=_blank]. |
| |
| [[logging]] |
| === Controlling logging level |
| |
| Per default, logging level is set to `INFO` level for all tests. The `DEBUG` |
| log level can be enabled for the tests. |
| |
| In IDE, set `-Dgerrit.logLevel=debug` as a VM argument. With `bazel`, pass |
| `GERRIT_LOG_LEVEL=debug` environment variable: |
| |
| ---- |
| bazel test --test_filter=com.google.gerrit.server.notedb.ChangeNotesTest \ |
| --test_env=GERRIT_LOG_LEVEL=debug \ |
| javatests/com/google/gerrit/server:server_tests |
| ---- |
| |
| The log results can be found in: |
| `bazel-testlogs/javatests/com/google/gerrit/server/server_tests/test.log`. |
| |
| |
| == Dependencies |
| |
| Dependency JARs are normally downloaded as needed, but you can |
| download everything upfront. This is useful to enable |
| subsequent builds to run without network access: |
| |
| ---- |
| bazel fetch //... |
| ---- |
| |
| When downloading from behind a proxy (which is common in some corporate |
| environments), it might be necessary to explicitly specify the proxy that |
| is then used by `curl`: |
| |
| ---- |
| export http_proxy=http://<proxy_user_id>:<proxy_password>@<proxy_server>:<proxy_port> |
| ---- |
| |
| Redirection to local mirrors of Maven Central and the Gerrit storage |
| bucket is supported by defining specific properties in |
| `local.properties`, a file that is not tracked by Git: |
| |
| ---- |
| echo download.GERRIT = http://nexus.my-company.com/ >>local.properties |
| echo download.MAVEN_CENTRAL = http://nexus.my-company.com/ >>local.properties |
| ---- |
| |
| The `local.properties` file may be placed in the root of the gerrit repository |
| being built, or in `~/.gerritcodereview/`. The file in the root of the gerrit |
| repository has precedence. |
| |
| == Building against unpublished Maven JARs |
| |
| To build against unpublished Maven JARs, like PrologCafe, the custom JARs must |
| be installed in the local Maven repository (`mvn clean install`) and |
| `maven_jar()` must be updated to point to the `MAVEN_LOCAL` Maven repository for |
| that artifact: |
| |
| [source,python] |
| ---- |
| maven_jar( |
| name = 'prolog-runtime', |
| artifact = 'com.googlecode.prolog-cafe:prolog-runtime:42', |
| repository = MAVEN_LOCAL, |
| ) |
| ---- |
| |
| == Building against artifacts from custom Maven repositories |
| |
| To build against custom Maven repositories, two modes of operations are |
| supported: with rewrite in local.properties and without. |
| |
| Without rewrite the URL of custom Maven repository can be directly passed |
| to the maven_jar() function: |
| |
| [source,python] |
| ---- |
| GERRIT_FORGE = 'http://gerritforge.com/snapshot' |
| |
| maven_jar( |
| name = 'gitblit', |
| artifact = 'com.gitblit:gitblit:1.4.0', |
| sha1 = '1b130dbf5578ace37507430a4a523f6594bf34fa', |
| repository = GERRIT_FORGE, |
| ) |
| ---- |
| |
| When the custom URL has to be rewritten, then the same logic as with Gerrit |
| known Maven repository is used: Repo name must be defined that matches an entry |
| in local.properties file: |
| |
| ---- |
| download.GERRIT_FORGE = http://my.company.mirror/gerrit-forge |
| ---- |
| |
| And corresponding WORKSPACE excerpt: |
| |
| [source,python] |
| ---- |
| GERRIT_FORGE = 'GERRIT_FORGE:' |
| |
| maven_jar( |
| name = 'gitblit', |
| artifact = 'com.gitblit:gitblit:1.4.0', |
| sha1 = '1b130dbf5578ace37507430a4a523f6594bf34fa', |
| repository = GERRIT_FORGE, |
| ) |
| ---- |
| |
| == Building against SNAPSHOT Maven JARs |
| |
| To build against SNAPSHOT Maven JARs, the complete SNAPSHOT version must be used: |
| |
| [source,python] |
| ---- |
| maven_jar( |
| name = "pac4j-core", |
| artifact = "org.pac4j:pac4j-core:3.5.0-SNAPSHOT-20190112.120241-16", |
| sha1 = "da2b1cb68a8f87bfd40813179abd368de9f3a746", |
| ) |
| ---- |
| |
| [[bazel-local-caches]] |
| |
| To accelerate builds, several caches are activated per default: |
| |
| * ~/.gerritcodereview/bazel-cache/downloaded-artifacts |
| * ~/.gerritcodereview/bazel-cache/repository |
| * ~/.gerritcodereview/bazel-cache/cas |
| |
| Currently none of these caches have a maximum size limit. See |
| link:https://github.com/bazelbuild/bazel/issues/5139[this bazel issue,role=external,window=_blank] for |
| details. Users should watch the cache sizes and clean them manually if |
| necessary. |
| |
| [[npm-binary]] |
| == NPM Binaries |
| |
| Parts of the PolyGerrit build require running NPM-based JavaScript programs as |
| "binaries". We don't attempt to resolve and download NPM dependencies at build |
| time, but instead use pre-built bundles of the NPM binary along with all its |
| dependencies. Some packages on |
| link:https://docs.npmjs.com/misc/registry[registry.npmjs.org,role=external,window=_blank] come with their |
| dependencies bundled, but this is the exception rather than the rule. More |
| commonly, to add a new binary to this list, you will need to bundle the binary |
| yourself. |
| |
| [NOTE] |
| We can only use binaries that meet certain licensing requirements, and that do |
| not include any native code. |
| |
| Start by checking that the license and file types of the bundle are acceptable: |
| [source,bash] |
| ---- |
| gerrit_repo=/path/to/gerrit |
| package=some-npm-package |
| version=1.2.3 |
| |
| # Note - yarn must be installed before running the following commands |
| yarn global add license-checker && \ |
| rm -rf /tmp/$package-$version && mkdir -p /tmp/$package-$version && \ |
| cd /tmp/$package-$version && \ |
| yarn add $package@$version && \ |
| license-checker | grep licenses: | sort -u |
| ---- |
| |
| This will output a list of the different licenses used by the package and all |
| its transitive dependencies. We can only legally distribute a bundle via our |
| storage bucket if the licenses allow us to do so. As long as all of the listed |
| license are allowed by |
| link:https://opensource.google.com/docs/thirdparty/licenses/[Google's |
| standards,role=external,window=_blank]. Any `by_exception_only`, commercial, prohibited, or unlisted |
| licenses are not allowed; otherwise, it is ok to distribute the source. If in |
| doubt, contact a maintainer who is a Googler. |
| |
| Next, check the file types: |
| [source,bash] |
| ---- |
| cd /tmp/$package-$version |
| find . -type f | xargs file | grep -v 'ASCII\|UTF-8\|empty$' |
| ---- |
| |
| If you see anything that looks like a native library or binary, then we can't |
| use the bundle. |
| |
| If everything looks good, install the package with the following command: |
| [source, bash] |
| ---- |
| # Add to ui_npm. Other packages.json can be updated in the same way |
| cd $gerrit_repo/polygerrit-ui/app |
| bazel run @nodejs//:yarn add $package |
| ---- |
| |
| Update the `polygerrit-ui/app/node_modules_licenses/licenses.ts` file. You should add licenses |
| for the package itself and for all transitive depndencies. If you forgot to add a license, the |
| `Documentation:check_licenses` test will fail. |
| |
| After the update, commit all changes to the repository (including `yarn.lock`). |
| |
| [NOTE] |
| ==== |
| If a npm package has transitive dependencies (or just several files) with a not allowed |
| license and you can't avoid use it in release, then you can add this package. |
| For example some packages contain demo-code with a different license. Another example - optional |
| dependencies, which are not needed to build polygerrit, but they are installed together with |
| the package anyway. |
| |
| In this case you should exclude all files and/or transitive dependencies with a not allowed license. |
| Adding such package requires additional updates: |
| |
| - Add dependencies (or files) to the license.ts with an appropriate license marked with |
| `allowed: false`. |
| |
| - update package.json postinstall script to remove all non-allowed files (if you don't |
| update postinstall script, `Documentation:check_licenses` test will fail.) |
| ==== |
| |
| === Update NPM Binaries |
| To update a NPM binary the same actions as for a new one must be done (check licenses, |
| update `licenses.ts` file, etc...). The only difference is a command to install a package: instead |
| of `bazel run @nodejs//:yarn add $package` you should run the `bazel run @nodejs//:yarn upgrade ...` |
| command with correct arguments. You can find the list of arguments in the |
| link:https://classic.yarnpkg.com/en/docs/cli/upgrade/[yarn upgrade doc,role=external,window=_blank]. |
| |
| |
| [[RBE]] |
| == Google Remote Build Support |
| |
| The Bazel build can be used with Google's Remote Build Execution. |
| |
| |
| This needs the following setup steps: |
| |
| ``` |
| gcloud auth application-default login |
| gcloud services enable remotebuildexecution.googleapis.com --project=${PROJECT} |
| ``` |
| |
| Create a worker pool. The instances should have at least 4 CPUs each |
| for adequate performance. |
| |
| ``` |
| gcloud alpha remote-build-execution worker-pools create default \ |
| --project=${PROJECT} \ |
| --instance=default_instance \ |
| --worker-count=50 \ |
| --machine-type=e2-standard-4 \ |
| --disk-size=200 |
| ``` |
| |
| To use RBE, execute |
| |
| ``` |
| bazel test --config=remote \ |
| --remote_instance_name=projects/${PROJECT}/instances/default_instance \ |
| javatests/... |
| ``` |
| |
| |
| GERRIT |
| ------ |
| Part of link:index.html[Gerrit Code Review] |
| |
| SEARCHBOX |
| --------- |