Documentation/dev-bazel.txt

:linkattrs:
= Gerrit Code Review - Building with Bazel

[[summary]]
== TL;DR

If you have the prerequisites, running

```
  $ bazel build gerrit
```

should generate a .war file under `bazel-bin/gerrit.war`.

[[installation]]
== Prerequisites

To build Gerrit from source, you need:

* A Linux or macOS system (Windows is not supported at this time)
* A JDK for Java 8|11|...
* Python 3
* link:https://github.com/nodesource/distributions/blob/master/README.md[Node.js (including npm),role=external,window=_blank]
* Bower (`npm install -g bower`)
* link:https://docs.bazel.build/versions/master/install.html[Bazel,role=external,window=_blank] -launched with
link:https://github.com/bazelbuild/bazelisk[Bazelisk,role=external,window=_blank]
* Maven
* zip, unzip
* curl
* gcc

[[bazel]]
=== Bazel

link:https://github.com/bazelbuild/bazelisk[Bazelisk,role=external,window=_blank] is a version
manager for link:https://bazel.build/[Bazel,role=external,window=_blank], similar to how `nvm`
manages `npm` versions. It takes care of downloading and installing Bazel itself, so you don't have
to worry about using the correct version of Bazel. Bazelisk can be installed in different
ways: link:https://docs.bazel.build/install-bazelisk.html[Install,role=external,window=_blank]

[[java]]
=== Java

Ensure that the link:install.html#Requirements[required Java version]
is installed and that `JAVA_HOME` is set to it.

To check the installed version of Java, open a terminal window and run:

`java -version`

[[java-8]]
==== Java 8 support (deprecated)

Java 8 is a legacy Java release and support for Java 8 will be discontinued
in future gerrit releases. To build Gerrit with Java 8 language level, run:

```
  $ bazel build :release
```

[[java-11]]
==== Java 11 support

To build Gerrit with Java 11 language level, run:

```
  $ bazel build --java_toolchain=//tools:error_prone_warnings_toolchain_java11 :release
```

[[java-13]]
==== Java 13 support

Java 13 (and newer) is supported through vanilla java toolchain
link:https://docs.bazel.build/versions/master/toolchains.html[Bazel option,role=external,window=_blank].
To build Gerrit with Java 13 and newer, specify vanilla java toolchain and
provide the path to JDK home:

```
  $ bazel build \
    --define=ABSOLUTE_JAVABASE=<path-to-java-13> \
    --javabase=@bazel_tools//tools/jdk:absolute_javabase \
    --host_javabase=@bazel_tools//tools/jdk:absolute_javabase \
    --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla \
    --java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla \
    :release
```

To run the tests, `--javabase` option must be passed as well, because
bazel test runs the test using the target javabase:

```
  $ bazel test \
    --define=ABSOLUTE_JAVABASE=<path-to-java-13> \
    --javabase=@bazel_tools//tools/jdk:absolute_javabase \
    --host_javabase=@bazel_tools//tools/jdk:absolute_javabase \
    --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla \
    --java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla \
    //...
```

To avoid passing all those options on every Bazel build invocation,
they could be added to ~/.bazelrc resource file:

```
$ cat << EOF > ~/.bazelrc
build --define=ABSOLUTE_JAVABASE=<path-to-java-13>
build --javabase=@bazel_tools//tools/jdk:absolute_javabase
build --host_javabase=@bazel_tools//tools/jdk:absolute_javabase
build --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla
build --java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla
EOF
```

Now, invoking Bazel with just `bazel build :release` would include
all those options.

=== Node.js and npm packages
See link:https://gerrit.googlesource.com/gerrit/+/master/polygerrit-ui/README.md#installing-node_js-and-npm-packages[Installing Node.js and npm packages,role=external,window=_blank].

[[build]]
== Building on the Command Line

=== Gerrit Development WAR File

To build the Gerrit web application:

----
  bazel build gerrit
----

The output executable WAR will be placed in:

----
  bazel-bin/gerrit.war
----

[[release]]
=== Gerrit Release WAR File

To build the Gerrit web application that includes the Gerrit UI,
core plugins and documentation:

----
  bazel build release
----

The output executable WAR will be placed in:

----
  bazel-bin/release.war
----

=== Headless Mode

To build Gerrit in headless mode, i.e. without the Gerrit UI:

----
  bazel build headless
----

The output executable WAR will be placed in:

----
  bazel-bin/headless.war
----

=== Extension and Plugin API JAR Files

To build the extension, plugin and acceptance-framework JAR files:

----
  bazel build api
----

The output archive that contains Java binaries, Java sources and
Java docs will be placed in:

----
  bazel-bin/api.zip
----

Install {extension,plugin,acceptance-framework}-api to the local
maven repository:

----
  tools/maven/api.sh install
----

Install gerrit.war to the local maven repository:

----
  tools/maven/api.sh war_install
----

=== Plugins

----
  bazel build plugins:core
----

The output JAR files for individual plugins will be placed in:

----
  bazel-bin/plugins/<name>/<name>.jar
----

The JAR files will also be packaged in:

----
  bazel-bin/plugins/core.zip
----

To build a specific plugin:

----
  bazel build plugins/<name>
----

The output JAR file will be be placed in:

----
  bazel-bin/plugins/<name>/<name>.jar
----

Note that when building an individual plugin, the `core.zip` package
is not regenerated.

[[IDEs]]
== Using an IDE.

=== IntelliJ

The Gerrit build works with Bazel's link:https://ij.bazel.build[IntelliJ plugin,role=external,window=_blank].
Please follow the instructions on <<dev-intellij#,IntelliJ Setup>>.

=== Eclipse

==== Generating the Eclipse Project

Create the Eclipse project:

----
  tools/eclipse/project.py
----

and then follow the link:dev-eclipse.html#setup[setup instructions].

==== Refreshing the Classpath

If an updated classpath is needed, the Eclipse project can be
refreshed and missing dependency JARs can be downloaded by running
`project.py` again. For IntelliJ, you need to click the `Sync Project
with BUILD Files` button of link:https://ij.bazel.build[Bazel plugin,role=external,window=_blank].

[[documentation]]
== Documentation

To build only the documentation for testing or static hosting:

----
  bazel build Documentation:searchfree
----

The html files will be bundled into `searchfree.zip` in this location:

----
  bazel-bin/Documentation/searchfree.zip
----

To generate HTML files skipping the zip archiving:

----
  bazel build Documentation
----

And open `bazel-bin/Documentation/index.html`.

To build the Gerrit executable WAR with the documentation included:

----
  bazel build withdocs
----

The WAR file will be placed in:

----
  bazel-bin/withdocs.war
----

Alternatively, one can generate the documentation as flat files:

----
  bazel build Documentation:Documentation
----

The html, css, js files are placed in:

----
 `bazel-bin/Documentation/`
----

[[tests]]
== Running Unit Tests

----
  bazel test --build_tests_only //...
----

Debugging tests:

----
  bazel test --test_output=streamed --test_filter=com.gerrit.TestClass.testMethod testTarget
----

Debug test example:

----
  bazel test --test_output=streamed --test_filter=com.google.gerrit.acceptance.api.change.ChangeIT.getAmbiguous //javatests/com/google/gerrit/acceptance/api/change:api_change
----

To run a specific test group, e.g. the rest-account test group:

----
  bazel test //javatests/com/google/gerrit/acceptance/rest/account:rest_account
----

To run SSH tests using JSch ssh client:

----
  bazel test --test_env=SSH_CLIENT_IMPLEMENTATION=JSCH //...
----

To run SSH tests using Apache MINA ssh client:

----
  bazel test --test_env=SSH_CLIENT_IMPLEMENTATION=APACHE //...
----

To run only tests that do not use SSH:

----
  bazel test --test_env=GERRIT_USE_SSH=NO //...
----

To exclude tests that have been marked as flaky:

----
  bazel test --test_tag_filters=-flaky //...
----

To exclude tests that require a Docker host:

----
  bazel test --test_tag_filters=-docker //...
----

To exclude tests that require very recent git client version:

----
  bazel test --test_tag_filters=-git-protocol-v2 //...
----

To ignore cached test results:

----
  bazel test --cache_test_results=NO //...
----

To run one or more specific groups of tests:

----
  bazel test --test_tag_filters=api,git //...
----

The following values are currently supported for the group name:

* annotation
* api
* docker
* edit
* elastic
* git
* git-protocol-v2
* git-upload-archive
* notedb
* pgm
* rest
* server
* ssh

Bazel itself supports a multitude of ways to
link:https://docs.bazel.build/versions/master/guide.html#specifying-targets-to-build[specify targets,role=external,window=_blank]
for fine-grained test selection that can be combined with many of the examples
above.

[[debugging-tests]]
== Debugging Unit Tests
In some cases it may be necessary to debug a test while running it in bazel. For example, when we
observe a different test result in Eclipse and bazel. Using the `--java_debug` option will start the
JVM in debug mode and await for a remote debugger to attach.

Example:
[source,bash]
----
  bazel test --java_debug --test_tag_filters=delete-project //...
  ...
  Listening for transport dt_socket at address: 5005
  ...
----

Now attach with a debugger to the port `5005`. For example use "Remote Java Application" launch
configuration in Eclipe and specify the port `5005`.

[[elasticsearch]]
=== Elasticsearch

Successfully running the Elasticsearch tests requires Docker, and
may require setting the local virtual memory on
link:https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html[linux,role=external,window=_blank] and
link:https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_set_vm_max_map_count_to_at_least_262144[macOS,role=external,window=_blank].

On macOS, if using link:https://docs.docker.com/docker-for-mac/[Docker Desktop,role=external,window=_blank],
the effective memory value can be set in the Preferences, under the Advanced tab.
The default value usually does not suffice and is causing premature container exits.
That default is currently 2 GB and should be set to at least 5 (GB).

If Docker is not available, the Elasticsearch tests will be skipped.
Note that Bazel currently does not show
link:https://github.com/bazelbuild/bazel/issues/3476[the skipped tests,role=external,window=_blank].

[[logging]]
=== Controlling logging level

Per default, logging level is set to `INFO` level for all tests. The `DEBUG`
log level can be enabled for the tests.

In IDE, set `-Dgerrit.logLevel=debug` as a VM argument. With `bazel`, pass
`GERRIT_LOG_LEVEL=debug` environment variable:

----
  bazel test --test_filter=com.google.gerrit.server.notedb.ChangeNotesTest \
  --test_env=GERRIT_LOG_LEVEL=debug \
  javatests/com/google/gerrit/server:server_tests
----

The log results can be found in:
`bazel-testlogs/javatests/com/google/gerrit/server/server_tests/test.log`.


== Dependencies

Dependency JARs are normally downloaded as needed, but you can
download everything upfront.  This is useful to enable
subsequent builds to run without network access:

----
  bazel fetch //...
----

When downloading from behind a proxy (which is common in some corporate
environments), it might be necessary to explicitly specify the proxy that
is then used by `curl`:

----
  export http_proxy=http://<proxy_user_id>:<proxy_password>@<proxy_server>:<proxy_port>
----

Redirection to local mirrors of Maven Central and the Gerrit storage
bucket is supported by defining specific properties in
`local.properties`, a file that is not tracked by Git:

----
  echo download.GERRIT = http://nexus.my-company.com/ >>local.properties
  echo download.MAVEN_CENTRAL = http://nexus.my-company.com/ >>local.properties
----

The `local.properties` file may be placed in the root of the gerrit repository
being built, or in `~/.gerritcodereview/`.  The file in the root of the gerrit
repository has precedence.

== Building against unpublished Maven JARs

To build against unpublished Maven JARs, like PrologCafe, the custom JARs must
be installed in the local Maven repository (`mvn clean install`) and
`maven_jar()` must be updated to point to the `MAVEN_LOCAL` Maven repository for
that artifact:

[source,python]
----
 maven_jar(
   name = 'prolog-runtime',
   artifact = 'com.googlecode.prolog-cafe:prolog-runtime:42',
   repository = MAVEN_LOCAL,
 )
----

== Building against artifacts from custom Maven repositories

To build against custom Maven repositories, two modes of operations are
supported: with rewrite in local.properties and without.

Without rewrite the URL of custom Maven repository can be directly passed
to the maven_jar() function:

[source,python]
----
  GERRIT_FORGE = 'http://gerritforge.com/snapshot'

  maven_jar(
    name = 'gitblit',
    artifact = 'com.gitblit:gitblit:1.4.0',
    sha1 = '1b130dbf5578ace37507430a4a523f6594bf34fa',
    repository = GERRIT_FORGE,
 )
----

When the custom URL has to be rewritten, then the same logic as with Gerrit
known Maven repository is used: Repo name must be defined that matches an entry
in local.properties file:

----
  download.GERRIT_FORGE = http://my.company.mirror/gerrit-forge
----

And corresponding WORKSPACE excerpt:

[source,python]
----
  GERRIT_FORGE = 'GERRIT_FORGE:'

  maven_jar(
    name = 'gitblit',
    artifact = 'com.gitblit:gitblit:1.4.0',
    sha1 = '1b130dbf5578ace37507430a4a523f6594bf34fa',
    repository = GERRIT_FORGE,
 )
----

== Building against SNAPSHOT Maven JARs

To build against SNAPSHOT Maven JARs, the complete SNAPSHOT version must be used:

[source,python]
----
 maven_jar(
   name = "pac4j-core",
   artifact = "org.pac4j:pac4j-core:3.5.0-SNAPSHOT-20190112.120241-16",
   sha1 = "da2b1cb68a8f87bfd40813179abd368de9f3a746",
 )
----

[[bazel-local-caches]]

To accelerate builds, several caches are activated per default:

* ~/.gerritcodereview/bazel-cache/downloaded-artifacts
* ~/.gerritcodereview/bazel-cache/repository
* ~/.gerritcodereview/bazel-cache/cas

The `downloaded-artifacts` cache can be relocated by setting the
`GERRIT_CACHE_HOME` environment variable. The other two can be adjusted with
`bazel build` options `--repository_cache` and `--disk_cache` respectively.

Currently none of these caches have a maximum size limit. See
link:https://github.com/bazelbuild/bazel/issues/5139[this bazel issue,role=external,window=_blank] for
details. Users should watch the cache sizes and clean them manually if
necessary.

[[npm-binary]]
== NPM Binaries

Parts of the Gerrit web app build require running NPM-based JavaScript programs
as "binaries". We don't attempt to resolve and download NPM dependencies at
build time, but instead use pre-built bundles of the NPM binary along with all
its dependencies. Some packages on
link:https://docs.npmjs.com/misc/registry[registry.npmjs.org,role=external,window=_blank] come with their
dependencies bundled, but this is the exception rather than the rule. More
commonly, to add a new binary to this list, you will need to bundle the binary
yourself.

[NOTE]
We can only use binaries that meet certain licensing requirements, and that do
not include any native code.

Start by checking that the license and file types of the bundle are acceptable:
[source,bash]
----
  gerrit_repo=/path/to/gerrit
  package=some-npm-package
  version=1.2.3

  # Note - yarn must be installed before running the following commands
  yarn global add license-checker && \
  rm -rf /tmp/$package-$version && mkdir -p /tmp/$package-$version && \
  cd /tmp/$package-$version && \
  yarn add $package@$version && \
  license-checker | grep licenses: | sort -u
----

This will output a list of the different licenses used by the package and all
its transitive dependencies. We can only legally distribute a bundle via our
storage bucket if the licenses allow us to do so. As long as all of the listed
license are allowed by
link:https://opensource.google.com/docs/thirdparty/licenses/[Google's
standards,role=external,window=_blank]. Any `by_exception_only`, commercial, prohibited, or unlisted
licenses are not allowed; otherwise, it is ok to distribute the source. If in
doubt, contact a maintainer who is a Googler.

Next, check the file types:
[source,bash]
----
  cd /tmp/$package-$version
  find . -type f | xargs file | grep -v 'ASCII\|UTF-8\|empty$'
----

If you see anything that looks like a native library or binary, then we can't
use the bundle.

If everything looks good, install the package with the following command:
[source, bash]
----
# Add to ui_npm. Other packages.json can be updated in the same way
cd $gerrit_repo/polygerrit-ui/app
bazel run @nodejs//:yarn add $package
----

Update the `polygerrit-ui/app/node_modules_licenses/licenses.ts` file. You should add licenses
for the package itself and for all transitive depndencies. If you forgot to add a license, the
`Documentation:check_licenses` test will fail.

After the update, commit all changes to the repository (including `yarn.lock`).

[NOTE]
====
If a npm package has transitive dependencies (or just several files) with a not allowed
license and you can't avoid use it in release, then you can add this package.
For example some packages contain demo-code with a different license. Another example - optional
dependencies, which are not needed to build the Gerrit web app, but they are installed together
with the package anyway.

In this case you should exclude all files and/or transitive dependencies with a not allowed license.
Adding such package requires additional updates:

- Add dependencies (or files) to the license.ts with an appropriate license marked with
`allowed: false`.

- update package.json postinstall script to remove all non-allowed files (if you don't
update postinstall script, `Documentation:check_licenses` test will fail.)
====

=== Update NPM Binaries
To update a NPM binary the same actions as for a new one must be done (check licenses,
update `licenses.ts` file, etc...). The only difference is a command to install a package: instead
of `bazel run @nodejs//:yarn add $package` you should run the `bazel run @nodejs//:yarn upgrade ...`
command with correct arguments. You can find the list of arguments in the
link:https://classic.yarnpkg.com/en/docs/cli/upgrade/[yarn upgrade doc,role=external,window=_blank].


[[RBE]]
== Google Remote Build Support

The Bazel build can be used with Google's Remote Build Execution.


This needs the following setup steps:

```
gcloud auth application-default login
gcloud services enable remotebuildexecution.googleapis.com  --project=${PROJECT}
```

Create a worker pool. The instances should have at least 4 CPUs each
for adequate performance.

```
gcloud alpha remote-build-execution worker-pools create default \
    --project=${PROJECT} \
    --instance=default_instance \
    --worker-count=50 \
    --machine-type=e2-standard-4 \
    --disk-size=200
```

Due to outdated Git version in official RBE docker images, a custom RBE docker
image must be used. To build custom docker imager, change to the directory
`tools/platforms` and build and publish custom RBE docker image.

To build the custom RBE docker image, run:

```
docker build -t gcr.io/api-project-164060093628/rbe-ubuntu18-04 .
```

To publish the custom RBE docker image, run:

```
docker push gcr.io/api-project-164060093628/rbe-ubuntu18-04
[...]
latest: digest: sha256:de5186d4313630a6111f9a2449b72563d0bc59ec9fb60956f063b69a38a76834 size: 1584
```

Re-build rbe_autoconfig project conduct a new release and switch to using it
in `WORKSPACE` file.

Note, to authenticate to the gcr.io registry, the following command must be
used:

```
gcloud auth configure-docker
```

To see the documentation, developer must be added to this group:
https://groups.google.com/forum/#!forum/rbe-alpha-customers.

Documentation can be found at:
https://cloud.google.com/remote-build-execution/docs.

To use RBE, execute

```
bazel test --config=remote \
    --remote_instance_name=projects/${PROJECT}/instances/default_instance \
    javatests/...
```


GERRIT
------
Part of link:index.html[Gerrit Code Review]

SEARCHBOX
---------