Documentation/dev-buck.txt

= Gerrit Code Review - Building with Buck


== Installation

You need to use Java 7 and Node.js for building gerrit.

There is currently no binary distribution of Buck, so it has to be manually
built and installed.  Apache Ant and gcc are required.  Currently only Linux
and Mac OS are supported.

Clone the git and build it:

----
  git clone https://github.com/facebook/buck
  cd buck
  git checkout $(cat ../gerrit/.buckversion)
  ant
----

If you don't have a `bin/` directory in your home directory, create one:

----
  mkdir ~/bin
----

Add the `~/bin` folder to the path:

----
  PATH=~/bin:$PATH
----

Note that the buck executable needs to be available in all shell sessions,
so also make sure it is appended to the path globally.

Add a symbolic link in `~/bin` to the buck and buckd executables:

----
  ln -s `pwd`/bin/buck ~/bin/
  ln -s `pwd`/bin/buckd ~/bin/
----

Verify that `buck` is accessible:

----
  which buck
----

To enable autocompletion of buck commands, install the autocompletion
script from `./scripts/buck-completion.bash` in the buck project.  Refer
to the script's header comments for installation instructions.

== Prerequisites

Buck requires Python version 2.7 to be installed. The Maven download toolchain
requires `curl` to be installed.

[[eclipse]]
== Eclipse Integration


=== Generating the Eclipse Project

Create the Eclipse project:

----
  tools/eclipse/project.py
----

and then follow the link:dev-eclipse.html#setup[setup instructions].

=== Refreshing the Classpath

If an updated classpath is needed, the Eclipse project can be
refreshed and missing dependency JARs can be downloaded:

----
  tools/eclipse/project.py
----


=== Attaching Sources

Source JARs are downloaded by default. This allows Eclipse to show
documentation or dive into the implementation of a library JAR.

To save time and bandwidth, download of source JARs can be restricted
to only those that are necessary to compile Java source into JavaScript
using the GWT compiler:

----
  tools/eclipse/project.py --no-src
----


[[build]]
== Building on the Command Line


=== Gerrit Development WAR File

To build the Gerrit web application that includes GWT UI and PolyGerrit UI:

----
  buck build gerrit
----

[NOTE]
PolyGerrit UI may require additional tools (such as npm). Please read
the polygerrit-ui/README.md for more info.

The output executable WAR will be placed in:

----
  buck-out/gen/gerrit/gerrit.war
----

To build the Gerrit web application that includes only GWT UI:

----
  buck build gwtgerrit
----

The output executable WAR will be placed in:

----
  buck-out/gen/gwtgerrit/gwtgerrit.war
----


=== Headless Mode

To build Gerrit in headless mode, i.e. without the GWT Web UI:

----
  buck build headless
----

The output executable WAR will be placed in:

----
  buck-out/gen/headless/headless.war
----

=== Extension and Plugin API JAR Files

To build the extension, plugin and GWT API JAR files:

----
  buck build api
----

Java binaries, Java sources and Java docs are generated into corresponding
project directories in `buck-out/gen`, here as example for plugin API:

----
  buck-out/gen/gerrit-plugin-api/plugin-api.jar
  buck-out/gen/gerrit-plugin-api/plugin-api-javadoc/plugin-api-javadoc.jar
  buck-out/gen/gerrit-plugin-api/plugin-api-src.jar
----

Install {extension,plugin,gwt}-api to the local maven repository:

----
  tools/maven/api.sh install
----

Install gerrit.war to the local maven repository:

----
  tools/maven/api.sh war_install
----

=== Plugins

To build all core plugins:

----
  buck build plugins:core
----

The output JAR files for individual plugins will be placed in:

----
  buck-out/gen/plugins/<name>/<name>.jar
----

The JAR files will also be packaged in:

----
  buck-out/gen/plugins/core/core.zip
----

To build a specific plugin:

----
  buck build plugins/<name>:<name>
----

The output JAR file will be be placed in:

----
  buck-out/gen/plugins/<name>/<name>.jar
----

Note that when building an individual plugin, the `core.zip` package
is not regenerated.

Additional plugins with BUCK files can be added to the build
environment by cloning the source repository into the plugins
subdirectory:

----
  git clone https://gerrit.googlesource.com/plugins/<name> plugins/<name>
  echo /plugins/<name> >>.git/info/exclude
----

Additional plugin sources will be automatically added to Eclipse the
next time project.py is run:

----
  tools/eclipse/project.py
----


[[documentation]]
=== Documentation

To build only the documentation for testing or static hosting:

----
  buck build docs
----

The generated html files will NOT come with the search box, and will be
placed in:

----
  buck-out/gen/Documentation/searchfree__tmp/Documentation
----

The html files will also be bundled into `searchfree.zip` in this location:

----
  buck-out/gen/Documentation/searchfree/searchfree.zip
----

To build the executable WAR with the documentation included:

----
  buck build withdocs
----

The WAR file will be placed in:

----
  buck-out/gen/withdocs/withdocs.war
----

[[soyc]]
=== GWT Compile Report

The GWT compiler can output a compile report (or "story of your compile"),
describing the size of the JavaScript and which source classes contributed
to the overall download size.

----
  buck build soyc
----

The report will be written as an HTML page to the extras directory, and
can be opened and viewed in any web browser:

----
  extras/gerrit_ui/soycReport/compile-report/index.html
----

Only the "Split Point Report" is created, "Compiler Metrics" are not output.

[[release]]
=== Gerrit Release WAR File

To build the release of the Gerrit web application, including documentation and
all core plugins:

----
  buck build release
----

The output release WAR will be placed in:

----
  buck-out/gen/release/release.war
----

[[tests]]
== Running Unit Tests

To run all tests including acceptance tests (but not flaky tests):

----
  buck test --exclude flaky
----

To exclude flaky and slow tests:

----
  buck test --exclude flaky slow
----

To run only a specific group of acceptance tests:

----
  buck test --include api
----

The following groups of tests are currently supported:

* acceptance
* api
* edit
* flaky
* git
* pgm
* rest
* server
* ssh
* slow

To run a specific test group, e.g. the rest-account test group:

----
  buck test //gerrit-acceptance-tests/src/test/java/com/google/gerrit/acceptance/rest/account:rest-account
----

To create test coverage report:

----
  buck test --code-coverage --code-coverage-format html --no-results-cache
----

The HTML report is created in `buck-out/gen/jacoco/code-coverage/index.html`.

== Dependencies

Dependency JARs are normally downloaded automatically, but Buck can inspect
its graph and download any missing JAR files.  This is useful to enable
subsequent builds to run without network access:

----
  tools/download_all.py
----

When downloading from behind a proxy (which is common in some corporate
environments), it might be necessary to explicitly specify the proxy that
is then used by `curl`:

----
  export http_proxy=http://<proxy_user_id>:<proxy_password>@<proxy_server>:<proxy_port>
----

Redirection to local mirrors of Maven Central and the Gerrit storage
bucket is supported by defining specific properties in
`local.properties`, a file that is not tracked by Git:

----
  echo download.GERRIT = http://nexus.my-company.com/ >>local.properties
  echo download.MAVEN_CENTRAL = http://nexus.my-company.com/ >>local.properties
----

The `local.properties` file may be placed in the root of the gerrit repository
being built, or in `~/.gerritcodereview/`.  The file in the root of the gerrit
repository has precedence.

== Building against unpublished Maven JARs

To build against unpublished Maven JARs, like gwtorm or PrologCafe, the custom
JARs must be installed in the local Maven repository (`mvn clean install`) and
`maven_jar()` must be updated to point to the `MAVEN_LOCAL` Maven repository for
that artifact:

[source,python]
----
 maven_jar(
   name = 'gwtorm',
   id = 'gwtorm:gwtorm:42',
   license = 'Apache2.0',
   repository = MAVEN_LOCAL,
 )
----

== Building against artifacts from custom Maven repositories

To build against custom Maven repositories, two modes of operations are
supported: with rewrite in local.properties and without.

Without rewrite the URL of custom Maven repository can be directly passed
to the maven_jar() function:

[source,python]
----
  GERRIT_FORGE = 'http://gerritforge.com/snapshot'

  maven_jar(
    name = 'gitblit',
    id = 'com.gitblit:gitblit:1.4.0',
    sha1 = '1b130dbf5578ace37507430a4a523f6594bf34fa',
    license = 'Apache2.0',
    repository = GERRIT_FORGE,
 )
----

When the custom URL has to be rewritten, then the same logic as with Gerrit
known Maven repository is used: Repo name must be defined that matches an entry
in local.properties file:

----
  download.GERRIT_FORGE = http://my.company.mirror/gerrit-forge
----

And corresponding BUCK excerpt:

[source,python]
----
  GERRIT_FORGE = 'GERRIT_FORGE:'

  maven_jar(
    name = 'gitblit',
    id = 'com.gitblit:gitblit:1.4.0',
    sha1 = '1b130dbf5578ace37507430a4a523f6594bf34fa',
    license = 'Apache2.0',
    repository = GERRIT_FORGE,
 )
----

=== Caching Build Results

Build results can be locally cached, saving rebuild time when
switching between Git branches. Buck's documentation covers
caching in link:http://facebook.github.io/buck/concept/buckconfig.html[buckconfig].
The trivial case using a local directory is:

----
  cat >.buckconfig.local <<EOF
  [cache]
    mode = dir
    dir = buck-cache
  EOF
----

[[clean-cache]]
=== Cleaning The Buck Cache

The cache for the Gerrit Code Review project is located in
`~/.gerritcodereview/buck-cache/locally-built-artifacts`.

The Buck cache should never need to be manually deleted. If you find yourself
deleting the Buck cache regularly, then it is likely that there is something
wrong with your environment or your workflow.

If you really do need to clean the cache manually, then:

----
 rm -rf ~/.gerritcodereview/buck-cache/locally-built-artifacts
----

Note that the root `buck-cache` folder should not be deleted as it also contains
the `downloaded-artifacts` directory, which holds the artifacts that got
downloaded (not built locally).

[[buck-daemon]]
=== Using Buck daemon

Buck ships with a daemon command `buckd`, which uses the
link:https://github.com/martylamb/nailgun[Nailgun] protocol for running
Java programs from the command line without incurring the JVM startup
overhead.

Using a Buck daemon can save significant amounts of time as it avoids the
overhead of starting a Java virtual machine, loading the buck class files
and parsing the build files for each command.

It is safe to run several buck daemons started from different project
directories and they will not interfere with each other. Buck's documentation
covers daemon in http://facebook.github.io/buck/command/buckd.html[buckd].

To use `buckd` the additional
link:https://facebook.github.io/watchman[watchman] program must be installed.

To disable `buckd`, the environment variable `NO_BUCKD` must be set. It's not
recommended to put it in the shell config, as it can be forgotten about it and
then assumed Buck was working as it should when it should be using buckd.
Prepend the variable to Buck invocation instead:

----
  NO_BUCKD=1 buck build gerrit
----

[[watchman]]
=== Installing watchman

Watchman is used internally by Buck to monitor directory trees and is needed
for buck daemon to work properly. Because buckd is activated by default in the
latest version of Buck, it searches for the watchman executable in the
path and issues a warning when it is not found and kills buckd.

To prepare watchman installation on Linux:

----
  git clone https://github.com/facebook/watchman.git
  cd watchman
  ./autogen.sh
----

To install it in user home directory (without root privileges):

----
  ./configure --prefix $HOME/watchman
  make install
----

To install it system wide:

----
  ./configure
  make
  sudo make install
----

Put $HOME/watchman/bin/watchman in path or link to $HOME/bin/watchman.

To install watchman on OS X:

----
  brew install --HEAD watchman
----

See the original documentation for more information:
link:https://facebook.github.io/watchman/docs/install.html[Watchman
installation].

=== Override Buck's settings

Additional JVM args for Buck can be set in `.buckjavaargs` in the
project root directory. For example to override Buck's default 1GB
heap size:

----
  cat > .buckjavaargs <<EOF
  -XX:MaxPermSize=512m -Xms8000m -Xmx16000m
  EOF
----

== Rerun unit tests

Test execution results are cached by Buck. If a test that was already run
needs to be repeated, the unit test cache for that test must be removed first:

----
  rm -rf buck-out/bin/gerrit-acceptance-tests/src/test/java/com/google/gerrit/acceptance/rest/account/.rest-account/
----

After clearing the cache, the test can be run again:

----
  buck test //gerrit-acceptance-tests/src/test/java/com/google/gerrit/acceptance/rest/account:rest-account
  [-] TESTING...FINISHED 12,3s (12 PASS/0 FAIL)
  RESULTS FOR //gerrit-acceptance-tests/src/test/java/com/google/gerrit/acceptance/rest/account:rest-account
  PASS     970ms  2 Passed   0 Skipped   0 Failed   com.google.gerrit.acceptance.rest.account.CapabilitiesIT
  PASS     999ms  1 Passed   0 Skipped   0 Failed   com.google.gerrit.acceptance.rest.account.EditPreferencesIT
  PASS      1,2s  1 Passed   0 Skipped   0 Failed   com.google.gerrit.acceptance.rest.account.GetAccountDetailIT
  PASS     951ms  2 Passed   0 Skipped   0 Failed   com.google.gerrit.acceptance.rest.account.GetAccountIT
  PASS      6,4s  2 Passed   0 Skipped   0 Failed   com.google.gerrit.acceptance.rest.account.GetDiffPreferencesIT
  PASS      1,2s  4 Passed   0 Skipped   0 Failed   com.google.gerrit.acceptance.rest.account.PutUsernameIT
  TESTS PASSED
----

An alternative approach is to use Buck's `--filters` (`-f`) option:

----
  buck test -f 'com.google.gerrit.acceptance.rest.account.CapabilitiesIT'
  Using buckd.
  [-] PROCESSING BUCK FILES...FINISHED 1,0s [100%]
  [-] BUILDING...FINISHED 2,8s [100%] (334/701 JOBS, 110 UPDATED, 5,1% CACHE MISS)
  [-] TESTING...FINISHED 9,2s (6 PASS/0 FAIL)
  RESULTS FOR SELECTED TESTS
  PASS      8,0s  2 Passed   0 Skipped   0 Failed   com.google.gerrit.acceptance.rest.account.CapabilitiesIT
  PASS    <100ms  4 Passed   0 Skipped   0 Failed   //tools:util_test
  TESTS PASSED
----

When this option is used, the cache is disabled per design and doesn't need to
be explicitly deleted. Note, that this is a known issue, that python tests are
always executed.

Note that when this option is used, the whole unit test cache is dropped, so
repeating the

----
buck test
----

causes all tests to be executed again.

To run tests without using cached results at all, use the `--no-results-cache`
option:

----
buck test --no-results-cache
----

== Upgrading Buck

The following tests should be executed, when Buck version is upgraded:

* buck build release
* tools/maven/api.sh install
* buck test
* buck build gerrit, change some sources in gerrit-server project,
  repeat buck build gerrit and verify that gerrit.war was updated
* install and verify new gerrit site
* upgrade and verify existing gerrit site
* reindex existing gerrit site
* verify that tools/eclipse/project.py produces sane Eclipse project
* verify that tools/eclipse/project.py --src generates sources as well
* verify that unit test execution from Eclipse works
* verify that daemon started from Eclipse works
* verify that GWT SDM debug session started from Eclipse works

== Known issues and bugs

=== Symbolic links and `watchman`

`Buck` with activated `Watchman` has currently a
[known bug](https://github.com/facebook/buck/issues/341) related to
symbolic links. The symbolic links are used very often with external
plugins, that are linked per symbolic link to the plugins directory.
With this use case Buck is failing to rebuild the plugin artifact
after it was built. All attempts to convince Buck to rebuild will fail.
The only known way to recover is to weep out `buck-out` directory. The
better workaround is to avoid using Watchman in this specific use case.
Watchman can either be de-installed or disabled. See
link:#buck-daemon[Using Buck daemon] section above how to temporarily
disable `buckd`.

== Troubleshooting Buck

In some cases problems with Buck itself need to be investigated. See for example
link:https://gerrit-review.googlesource.com/62411[this attempt to upgrade Buck]
and link:https://github.com/facebook/buck/pull/227[the fix that was needed] to
make the update possible.

To build Gerrit with a custom version of Buck, the following steps are necessary:

1. In the Buck git apply any necessary changes from pull requests
2. Compile Buck with `ant`
3. In the root of the Gerrit project create a `.nobuckcheck` file to prevent Buck
from updating itself
4. Replace the sha1 in Gerrit's `.buckversion` file with the required version from
the custom Buck build
5. Build Gerrit as usual

GERRIT
------
Part of link:index.html[Gerrit Code Review]

SEARCHBOX
---------