| :linkattrs: |
| = Gerrit Code Review - Building with Bazel |
| |
| [[summary]] |
| == TL;DR |
| |
| If you have the prerequisites, running |
| |
| ``` |
| $ bazel build gerrit |
| ``` |
| |
| should generate a .war file under `bazel-bin/gerrit.war`. |
| |
| [[installation]] |
| == Prerequisites |
| |
| To build Gerrit from source, you need: |
| |
| * A Linux or macOS system (Windows is not supported at this time) |
| * A JDK for Java 11 or Java 17 |
| * Python 3 |
| * link:https://github.com/nodesource/distributions/blob/master/README.md[Node.js (including npm),role=external,window=_blank] |
| * Bower (`npm install -g bower`) |
| * link:https://docs.bazel.build/versions/master/install.html[Bazel,role=external,window=_blank] -launched with |
| link:https://github.com/bazelbuild/bazelisk[Bazelisk,role=external,window=_blank] |
| * Maven |
| * zip, unzip |
| * curl |
| * gcc |
| |
| [[bazel]] |
| === Bazel |
| |
| link:https://github.com/bazelbuild/bazelisk[Bazelisk,role=external,window=_blank] is a version |
| manager for link:https://bazel.build/[Bazel,role=external,window=_blank], similar to how `nvm` |
| manages `npm` versions. It takes care of downloading and installing Bazel itself, so you don't have |
| to worry about using the correct version of Bazel. One particular advantage to |
| using Bazelisk is that you can jump between different versions of Gerrit and not |
| worry about which version of Bazel you need. |
| |
| Bazelisk can be installed in different ways: |
| link:https://docs.bazel.build/install-bazelisk.html[Bazelisk Installation,role=external,window=_blank]. |
| To execute the correct version of Bazel using Bazelisk you simply replace |
| the `bazel` command with `bazelisk`. |
| |
| [[java]] |
| === Java |
| |
| Ensure that the link:install.html#Requirements[required Java version] |
| is installed and that `JAVA_HOME` is set to it. |
| |
| To check the installed version of Java, open a terminal window and run: |
| |
| `java -version` |
| |
| [[java-11]] |
| ==== Java 11 support |
| |
| To build Gerrit with Java 11 language level, run: |
| |
| ``` |
| $ bazelisk build --config=java11 :release |
| ``` |
| |
| [[java-17]] |
| ==== Java 17 support |
| |
| Java 17 is supported. To build Gerrit with Java 17, run: |
| |
| ``` |
| $ bazelisk build :release |
| ``` |
| |
| To run the tests with Java 17, run: |
| |
| ``` |
| $ bazelisk test //... |
| ``` |
| |
| === Node.js and npm packages |
| See link:https://gerrit.googlesource.com/gerrit/+/master/polygerrit-ui/README.md#installing-node_js-and-npm-packages[Installing Node.js and npm packages,role=external,window=_blank]. |
| |
| [[build]] |
| == Building on the Command Line |
| |
| === Gerrit Development WAR File |
| |
| To build the Gerrit web application: |
| |
| ---- |
| bazelisk build gerrit |
| ---- |
| |
| The output executable WAR will be placed in: |
| |
| ---- |
| bazel-bin/gerrit.war |
| ---- |
| |
| [[release]] |
| === Gerrit Release WAR File |
| |
| To build the Gerrit web application that includes the Gerrit UI, |
| core plugins and documentation: |
| |
| ---- |
| bazelisk build release |
| ---- |
| |
| The output executable WAR will be placed in: |
| |
| ---- |
| bazel-bin/release.war |
| ---- |
| |
| === Headless Mode |
| |
| To build Gerrit in headless mode, i.e. without the Gerrit UI: |
| |
| ---- |
| bazelisk build headless |
| ---- |
| |
| The output executable WAR will be placed in: |
| |
| ---- |
| bazel-bin/headless.war |
| ---- |
| |
| === Extension and Plugin API JAR Files |
| |
| To build the extension, plugin and acceptance-framework JAR files: |
| |
| ---- |
| bazelisk build api |
| ---- |
| |
| The output archive that contains Java binaries, Java sources and |
| Java docs will be placed in: |
| |
| ---- |
| bazel-bin/api.zip |
| ---- |
| |
| Install {extension,plugin,acceptance-framework}-api to the local |
| maven repository: |
| |
| ---- |
| tools/maven/api.sh install |
| ---- |
| |
| Install gerrit.war to the local maven repository: |
| |
| ---- |
| tools/maven/api.sh war_install |
| ---- |
| |
| === Plugins |
| |
| ---- |
| bazelisk build plugins:core |
| ---- |
| |
| The output JAR files for individual plugins will be placed in: |
| |
| ---- |
| bazel-bin/plugins/<name>/<name>.jar |
| ---- |
| |
| The JAR files will also be packaged in: |
| |
| ---- |
| bazel-bin/plugins/core.zip |
| ---- |
| |
| To build a specific plugin: |
| |
| ---- |
| bazelisk build plugins/<name> |
| ---- |
| |
| The output JAR file will be be placed in: |
| |
| ---- |
| bazel-bin/plugins/<name>/<name>.jar |
| ---- |
| |
| Note that when building an individual plugin, the `core.zip` package |
| is not regenerated. |
| |
| [[IDEs]] |
| == Using an IDE. |
| |
| === IntelliJ |
| |
| The Gerrit build works with Bazel's link:https://ij.bazel.build[IntelliJ plugin,role=external,window=_blank]. |
| Please follow the instructions on <<dev-intellij#,IntelliJ Setup>>. |
| |
| === Eclipse |
| |
| ==== Generating the Eclipse Project |
| |
| Create the Eclipse project: |
| |
| ---- |
| tools/eclipse/project.py |
| ---- |
| |
| and then follow the link:dev-eclipse.html#setup[setup instructions]. |
| |
| ==== Refreshing the Classpath |
| |
| If an updated classpath is needed, the Eclipse project can be |
| refreshed and missing dependency JARs can be downloaded by running |
| `project.py` again. For IntelliJ, you need to click the `Sync Project |
| with BUILD Files` button of link:https://ij.bazel.build[Bazel plugin,role=external,window=_blank]. |
| |
| [[documentation]] |
| == Documentation |
| |
| To build only the documentation for testing or static hosting: |
| |
| ---- |
| bazelisk build Documentation:searchfree |
| ---- |
| |
| The html files will be bundled into `searchfree.zip` in this location: |
| |
| ---- |
| bazel-bin/Documentation/searchfree.zip |
| ---- |
| |
| To use local fonts with the searchfree target: |
| |
| ---- |
| bazel build Documentation:searchfree_safe |
| ---- |
| |
| The html files will be bundled into `searchfree.zip` or `searchfree_safe.zip` in this location: |
| |
| ---- |
| bazel-bin/Documentation/searchfree.zip |
| bazel-bin/Documentation/searchfree_safe.zip |
| ---- |
| |
| To generate HTML files skipping the zip archiving: |
| |
| ---- |
| bazelisk build Documentation |
| ---- |
| |
| And open `bazel-bin/Documentation/index.html`. |
| |
| To build the Gerrit executable WAR with the documentation included: |
| |
| ---- |
| bazelisk build withdocs |
| ---- |
| |
| The WAR file will be placed in: |
| |
| ---- |
| bazel-bin/withdocs.war |
| ---- |
| |
| Alternatively, one can generate the documentation as flat files: |
| |
| ---- |
| bazelisk build Documentation:Documentation |
| ---- |
| |
| The html, css, js files are placed in: |
| |
| ---- |
| `bazel-bin/Documentation/` |
| ---- |
| |
| [[tests]] |
| == Running Unit Tests |
| |
| Bazel BUILD files define test targets for Gerrit. You can run all declared |
| test targets with: |
| |
| ---- |
| bazelisk test --build_tests_only //... |
| ---- |
| |
| [[testgroups]] |
| === Running Test Groups |
| |
| To run one or more specific labeled groups of tests: |
| |
| ---- |
| bazelisk test --test_tag_filters=api,git //... |
| ---- |
| |
| The following label values are currently supported for the group name: |
| |
| * annotation |
| * api |
| * edit |
| * git |
| * git-protocol-v2 |
| * git-upload-archive |
| * notedb |
| * pgm |
| * rest |
| * server |
| * ssh |
| |
| We can also select tests within a specific BUILD target group. For example |
| `javatests/com/google/gerrit/acceptance/rest/account/BUILD` declares a |
| rest_account test target group: |
| |
| ---- |
| bazelisk test //javatests/com/google/gerrit/acceptance/rest/account:rest_account |
| ---- |
| |
| [[debugtests]] |
| === Debugging Tests |
| |
| To debug specific tests you will need to select the test target containing |
| that test then use `--test_filter` to select the specific test you want. |
| This `--test_filter` is a regex and can be used to select multiple tests |
| out of the target: |
| |
| ---- |
| bazelisk test --test_output=streamed --test_filter=com.gerrit.TestClass.testMethod testTarget |
| ---- |
| |
| For example `javatests/com/google/gerrit/acceptance/api/change/BUILD` |
| defines a test target group for every `*IT.java` file in the directory. |
| We can execute the single `getAmbiguous()` test found in ChangeIT.java using |
| this `--test_filter` and target: |
| |
| ---- |
| bazelisk test --test_output=streamed \ |
| --test_filter=com.google.gerrit.acceptance.api.change.ChangeIT.getAmbiguous \ |
| //javatests/com/google/gerrit/acceptance/api/change:ChangeIT |
| ---- |
| |
| [[additionaltestfiltering]] |
| === Additional Test Filtering |
| |
| To run only tests that do not use SSH: |
| |
| ---- |
| bazelisk test --test_env=GERRIT_USE_SSH=NO //... |
| ---- |
| |
| To exclude tests that have been marked as flaky: |
| |
| ---- |
| bazelisk test --test_tag_filters=-flaky //... |
| ---- |
| |
| To exclude tests that require very recent git client version: |
| |
| ---- |
| bazelisk test --test_tag_filters=-git-protocol-v2 //... |
| ---- |
| |
| To run the tests against a specific index backend (LUCENE, FAKE): |
| ---- |
| bazelisk test --test_env=GERRIT_INDEX_TYPE=LUCENE //... |
| ---- |
| |
| Bazel itself supports a multitude of ways to |
| link:https://bazel.build/run/build#specifying-build-targets[specify targets,role=external,window=_blank] |
| for fine-grained test selection that can be combined with many of the examples |
| above. |
| |
| [[testcaching]] |
| === Test Caching |
| |
| By default Bazel caches test results and will not reexecute tests unless they |
| or their dependencies have been modified. To ignore cached test results and |
| force the tests to rerun: |
| |
| ---- |
| bazelisk test --cache_test_results=NO //... |
| ---- |
| |
| [[plugintests]] |
| === Running Plugin Tests |
| |
| Running tests for Gerrit plugins follows the process above. From within the |
| Gerrit project root with the desired plugins checked out into `plugins/` we |
| execute Bazel with the appropriate target: |
| |
| ---- |
| bazelisk test //plugins/replication/... |
| ---- |
| |
| [[debugging-tests]] |
| == Debugging Unit Tests |
| In some cases it may be necessary to debug a test while running it in bazel. For example, when we |
| observe a different test result in Eclipse and bazel. Using the `--java_debug` option will start the |
| JVM in debug mode and await for a remote debugger to attach. |
| |
| Example: |
| [source,bash] |
| ---- |
| bazelisk test --java_debug --test_tag_filters=delete-project //... |
| ... |
| Listening for transport dt_socket at address: 5005 |
| ... |
| ---- |
| |
| Now attach with a debugger to the port `5005`. For example use "Remote Java Application" launch |
| configuration in Eclipse and specify the port `5005`. |
| |
| [[logging]] |
| === Controlling logging level |
| |
| Per default, logging level is set to `INFO` level for all tests. The `DEBUG` |
| log level can be enabled for the tests. |
| |
| In IDE, set `-Dgerrit.logLevel=debug` as a VM argument. With `bazel`, pass |
| `GERRIT_LOG_LEVEL=debug` environment variable: |
| |
| ---- |
| bazelisk test --test_filter=com.google.gerrit.server.notedb.ChangeNotesTest \ |
| --test_env=GERRIT_LOG_LEVEL=debug \ |
| javatests/com/google/gerrit/server:server_tests |
| ---- |
| |
| The log results can be found in: |
| `bazel-testlogs/javatests/com/google/gerrit/server/server_tests/test.log`. |
| |
| |
| == Dependencies |
| |
| Dependency JARs are normally downloaded as needed, but you can |
| download everything upfront. This is useful to enable |
| subsequent builds to run without network access: |
| |
| ---- |
| bazelisk fetch //... |
| ---- |
| |
| When downloading from behind a proxy (which is common in some corporate |
| environments), it might be necessary to explicitly specify the proxy that |
| is then used by `curl`: |
| |
| ---- |
| export http_proxy=http://<proxy_user_id>:<proxy_password>@<proxy_server>:<proxy_port> |
| ---- |
| |
| Redirection to local mirrors of Maven Central and the Gerrit storage |
| bucket is supported by defining specific properties in |
| `local.properties`, a file that is not tracked by Git: |
| |
| ---- |
| echo download.GERRIT = http://nexus.my-company.com/ >>local.properties |
| echo download.MAVEN_CENTRAL = http://nexus.my-company.com/ >>local.properties |
| ---- |
| |
| The `local.properties` file may be placed in the root of the gerrit repository |
| being built, or in `~/.gerritcodereview/`. The file in the root of the gerrit |
| repository has precedence. |
| |
| == Building against unpublished Maven JARs |
| |
| To build against unpublished Maven JARs, like PrologCafe, the custom JARs must |
| be installed in the local Maven repository (`mvn clean install`) and |
| `maven_jar()` must be updated to point to the `MAVEN_LOCAL` Maven repository for |
| that artifact: |
| |
| [source,python] |
| ---- |
| maven_jar( |
| name = 'prolog-runtime', |
| artifact = 'com.googlecode.prolog-cafe:prolog-runtime:42', |
| repository = MAVEN_LOCAL, |
| ) |
| ---- |
| |
| == Building against artifacts from custom Maven repositories |
| |
| To build against custom Maven repositories, two modes of operations are |
| supported: with rewrite in local.properties and without. |
| |
| Without rewrite the URL of custom Maven repository can be directly passed |
| to the maven_jar() function: |
| |
| [source,python] |
| ---- |
| GERRIT_FORGE = 'http://gerritforge.com/snapshot' |
| |
| maven_jar( |
| name = 'gitblit', |
| artifact = 'com.gitblit:gitblit:1.4.0', |
| sha1 = '1b130dbf5578ace37507430a4a523f6594bf34fa', |
| repository = GERRIT_FORGE, |
| ) |
| ---- |
| |
| When the custom URL has to be rewritten, then the same logic as with Gerrit |
| known Maven repository is used: Repo name must be defined that matches an entry |
| in local.properties file: |
| |
| ---- |
| download.GERRIT_FORGE = http://my.company.mirror/gerrit-forge |
| ---- |
| |
| And corresponding WORKSPACE excerpt: |
| |
| [source,python] |
| ---- |
| GERRIT_FORGE = 'GERRIT_FORGE:' |
| |
| maven_jar( |
| name = 'gitblit', |
| artifact = 'com.gitblit:gitblit:1.4.0', |
| sha1 = '1b130dbf5578ace37507430a4a523f6594bf34fa', |
| repository = GERRIT_FORGE, |
| ) |
| ---- |
| |
| == Building against SNAPSHOT Maven JARs |
| |
| To build against SNAPSHOT Maven JARs, the complete SNAPSHOT version must be used: |
| |
| [source,python] |
| ---- |
| maven_jar( |
| name = "pac4j-core", |
| artifact = "org.pac4j:pac4j-core:3.5.0-SNAPSHOT-20190112.120241-16", |
| sha1 = "da2b1cb68a8f87bfd40813179abd368de9f3a746", |
| ) |
| ---- |
| |
| [[bazel-local-caches]] |
| |
| To accelerate builds, several caches are activated per default: |
| |
| * ~/.gerritcodereview/bazel-cache/downloaded-artifacts |
| * ~/.gerritcodereview/bazel-cache/repository |
| * ~/.gerritcodereview/bazel-cache/cas |
| |
| The `downloaded-artifacts` cache can be relocated by setting the |
| `GERRIT_CACHE_HOME` environment variable. The other two can be adjusted with |
| `bazelisk build` options `--repository_cache` and `--disk_cache` respectively. |
| |
| Currently none of these caches have a maximum size limit. See |
| link:https://github.com/bazelbuild/bazel/issues/5139[this bazel issue,role=external,window=_blank] for |
| details. Users should watch the cache sizes and clean them manually if |
| necessary. |
| |
| [[npm-binary]] |
| == NPM Binaries |
| |
| Parts of the Gerrit web app build require running NPM-based JavaScript programs |
| as "binaries". We don't attempt to resolve and download NPM dependencies at |
| build time, but instead use pre-built bundles of the NPM binary along with all |
| its dependencies. Some packages on |
| link:https://docs.npmjs.com/misc/registry[registry.npmjs.org,role=external,window=_blank] come with their |
| dependencies bundled, but this is the exception rather than the rule. More |
| commonly, to add a new binary to this list, you will need to bundle the binary |
| yourself. |
| |
| [NOTE] |
| We can only use binaries that meet certain licensing requirements, and that do |
| not include any native code. |
| |
| Start by checking that the license and file types of the bundle are acceptable: |
| [source,bash] |
| ---- |
| gerrit_repo=/path/to/gerrit |
| package=some-npm-package |
| version=1.2.3 |
| |
| # Note - yarn must be installed before running the following commands |
| yarn global add license-checker && \ |
| rm -rf /tmp/$package-$version && mkdir -p /tmp/$package-$version && \ |
| cd /tmp/$package-$version && \ |
| yarn add $package@$version && \ |
| license-checker | grep licenses: | sort -u |
| ---- |
| |
| This will output a list of the different licenses used by the package and all |
| its transitive dependencies. We can only legally distribute a bundle via our |
| storage bucket if the licenses allow us to do so. As long as all of the listed |
| license are allowed by |
| link:https://opensource.google.com/docs/thirdparty/licenses/[Google's |
| standards,role=external,window=_blank]. Any `by_exception_only`, commercial, prohibited, or unlisted |
| licenses are not allowed; otherwise, it is ok to distribute the source. If in |
| doubt, contact a maintainer who is a Googler. |
| |
| Next, check the file types: |
| [source,bash] |
| ---- |
| cd /tmp/$package-$version |
| find . -type f | xargs file | grep -v 'ASCII\|UTF-8\|empty$' |
| ---- |
| |
| If you see anything that looks like a native library or binary, then we can't |
| use the bundle. |
| |
| If everything looks good, install the package with the following command: |
| [source, bash] |
| ---- |
| # Add to ui_npm. Other packages.json can be updated in the same way |
| cd $gerrit_repo/polygerrit-ui/app |
| bazelisk run @nodejs//:yarn add $package |
| ---- |
| |
| Update the `polygerrit-ui/app/node_modules_licenses/licenses.ts` file. You should add licenses |
| for the package itself and for all transitive dependencies. If you forgot to add a license, the |
| `Documentation:check_licenses` test will fail. |
| |
| After the update, commit all changes to the repository (including `yarn.lock`). |
| |
| [NOTE] |
| ==== |
| If a npm package has transitive dependencies (or just several files) with a not allowed |
| license and you can't avoid use it in release, then you can add this package. |
| For example some packages contain demo-code with a different license. Another example - optional |
| dependencies, which are not needed to build the Gerrit web app, but they are installed together |
| with the package anyway. |
| |
| In this case you should exclude all files and/or transitive dependencies with a not allowed license. |
| Adding such package requires additional updates: |
| |
| - Add dependencies (or files) to the license.ts with an appropriate license marked with |
| `allowed: false`. |
| |
| - update package.json postinstall script to remove all non-allowed files (if you don't |
| update postinstall script, `Documentation:check_licenses` test will fail.) |
| ==== |
| |
| === Update NPM Binaries |
| To update a NPM binary the same actions as for a new one must be done (check licenses, |
| update `licenses.ts` file, etc...). The only difference is a command to install a package: instead |
| of `bazelisk run @nodejs//:yarn add $package` you should run the `bazelisk run @nodejs//:yarn upgrade ...` |
| command with correct arguments. You can find the list of arguments in the |
| link:https://classic.yarnpkg.com/en/docs/cli/upgrade/[yarn upgrade doc,role=external,window=_blank]. |
| |
| |
| [[RBE]] |
| == Google Remote Build Support |
| |
| The Bazel build can be used with Google's Remote Build Execution. |
| |
| |
| This needs the following setup steps: |
| |
| ``` |
| gcloud auth application-default login |
| gcloud services enable remotebuildexecution.googleapis.com --project=${PROJECT} |
| ``` |
| |
| Create a worker pool. The instances should have at least 4 CPUs each |
| for adequate performance. |
| |
| ``` |
| gcloud alpha remote-build-execution worker-pools create default \ |
| --project=${PROJECT} \ |
| --instance=default_instance \ |
| --worker-count=50 \ |
| --machine-type=e2-standard-4 \ |
| --disk-size=200 |
| ``` |
| |
| Due to outdated Git version in official RBE docker images, a custom RBE docker |
| image must be used. To build custom docker imager, change to the directory |
| `tools/platforms` and build and publish custom RBE docker image. |
| |
| To build the custom RBE docker image, run: |
| |
| ``` |
| docker build -t gcr.io/api-project-164060093628/rbe-ubuntu18-04 . |
| ``` |
| |
| To publish the custom RBE docker image, run: |
| |
| ``` |
| docker push gcr.io/api-project-164060093628/rbe-ubuntu18-04 |
| [...] |
| latest: digest: sha256:de5186d4313630a6111f9a2449b72563d0bc59ec9fb60956f063b69a38a76834 size: 1584 |
| ``` |
| |
| Re-build rbe_autoconfig project conduct a new release and switch to using it |
| in `WORKSPACE` file. |
| |
| Note, to authenticate to the gcr.io registry, the following command must be |
| used: |
| |
| ``` |
| gcloud auth configure-docker |
| ``` |
| |
| To see the documentation, developer must be added to this group: |
| https://groups.google.com/forum/#!forum/rbe-alpha-customers. |
| |
| Documentation can be found at: |
| https://cloud.google.com/remote-build-execution/docs. |
| |
| To use RBE, execute |
| |
| ``` |
| bazelisk test --config=remote \ |
| --remote_instance_name=projects/${PROJECT}/instances/default_instance \ |
| javatests/... |
| ``` |
| |
| |
| GERRIT |
| ------ |
| Part of link:index.html[Gerrit Code Review] |
| |
| SEARCHBOX |
| --------- |