Migrate external IDs to NoteDb (part 2)

This is the second part of migrating external IDs from ReviewDb to
NoteDb.

This change:
* migrates the external IDs from ReviewDb to NoteDb (for single
  instance Gerrit servers)
* adds a configuration parameter (user.readExternalIdsFromGit) that
  controls whether external IDs are read from ReviewDb or NoteDb

The new ExternalIds class provides access to external IDs. All code that
needs external IDs is adapted to use this class to retrieve external IDs
(instead of reading directly from the database). ExternalIds gets the
external IDs either directly from the storage backend (via
ExternalIdReader) or from a cache (via ExternalIdCache).

ExternalIdReader reads the external IDs from the storage backend.
Depending on the value of the user.readExternalIdsFromGit parameter the
external IDs are read from ReviewDB or NoteDb.

If reading external IDs from NoteDb is enabled, reading the external IDs
of an account requires parsing all Git notes. This is because external
IDs are keyed by external ID key ('<scheme>:<id>') and the account ID is
only contained in the Git note content. Since parsing all Git notes is
too expensive if it is done frequently, there is a new external ID cache
which makes external IDs accessible by account. This cache is populated
once by reading all external IDs from NoteDb and is then kept up to date
by informing it whenever an external ID is added, updated or deleted.
The external ID cache uses the revision of the refs/meta/external-ids
branch as key, so that all external IDs are reloaded when the
refs/meta/external-ids branch is changed behind Gerrit's back. This
makes it easy to use this cache in a multimaster setup, since an update
of the refs/meta/external-ids branch which is done due to replication
between nodes causes a reload of the external IDs in the receiving node.
The ExternalIdCache is an implementation detail of how external IDs are
read and written, which is why it is package private. Callers should
always use ExternalIds to access external IDs and ExternalIdsUpdate /
ExternalIdsBatchUpdate to update external IDs.

The LocalUsernamesToLowerCase program needs to access all external IDs
only once to update them. After the update they are not accessed again.
Hence the LocalUsernamesToLowerCase program doesn't benefit from caching
external IDs and the external ID cache can be disabled for it.

The external ID cache is defined by the ExternalIdCache interface. It is
implemented by ExternalIdCacheImpl and DisabledExternalIdCache.
DisabledExternalIdCache can be used when an external ID cache is not
needed, e.g. in the LocalUsernamesToLowerCase program or in tests.

Pushing to the refs/meta/external-ids branch, which would only update
the external IDs in NoteDb, is still prevented by a commit validator so
that the external IDs in ReviewDb and NoteDb do not go out of sync.

Change-Id: Ia1dae9306b7ee07388b6c5e1f3dc4a1a5eea4b08
Signed-off-by: Edwin Kempin <ekempin@google.com>
29 files changed
tree: 9f9b2915bee9ad94433667e9bcf4d119f3503588
  1. .settings/
  2. contrib/
  3. Documentation/
  4. gerrit-acceptance-framework/
  5. gerrit-acceptance-tests/
  6. gerrit-antlr/
  7. gerrit-cache-h2/
  8. gerrit-common/
  9. gerrit-elasticsearch/
  10. gerrit-extension-api/
  11. gerrit-gpg/
  12. gerrit-gwtdebug/
  13. gerrit-gwtexpui/
  14. gerrit-gwtui/
  15. gerrit-gwtui-common/
  16. gerrit-httpd/
  17. gerrit-launcher/
  18. gerrit-lucene/
  19. gerrit-main/
  20. gerrit-oauth/
  21. gerrit-openid/
  22. gerrit-patch-commonsnet/
  23. gerrit-patch-jgit/
  24. gerrit-pgm/
  25. gerrit-plugin-api/
  26. gerrit-plugin-gwtui/
  27. gerrit-prettify/
  28. gerrit-reviewdb/
  29. gerrit-server/
  30. gerrit-sshd/
  31. gerrit-test-util/
  32. gerrit-util-cli/
  33. gerrit-util-http/
  34. gerrit-util-ssl/
  35. gerrit-war/
  36. lib/
  37. plugins/
  38. polygerrit-ui/
  39. ReleaseNotes/
  40. tools/
  41. website/
  42. .bazelproject
  43. .editorconfig
  44. .git-blame-ignore-revs
  45. .gitignore
  46. .gitmodules
  47. .mailmap
  48. .pydevproject
  49. BUILD
  50. COPYING
  51. INSTALL
  52. README.md
  53. SUBMITTING_PATCHES
  54. version.bzl
  55. WORKSPACE
README.md

Gerrit Code Review

Gerrit is a code review and project management tool for Git based projects.

Build Status

Objective

Gerrit makes reviews easier by showing changes in a side-by-side display, and allowing inline comments to be added by any reviewer.

Gerrit simplifies Git based project maintainership by permitting any authorized user to submit changes to the master Git repository, rather than requiring all approved changes to be merged in by hand by the project maintainer.

Documentation

For information about how to install and use Gerrit, refer to the documentation.

Source

Our canonical Git repository is located on googlesource.com. There is a mirror of the repository on Github.

Reporting bugs

Please report bugs on the issue tracker.

Contribute

Gerrit is the work of hundreds of contributors. We appreciate your help!

Please read the contribution guidelines.

Note that we do not accept Pull Requests via the Github mirror.

Getting in contact

The IRC channel on freenode is #gerrit. An archive is available at: echelog.com.

The Developer Mailing list is repo-discuss on Google Groups.

License

Gerrit is provided under the Apache License 2.0.

Build

Install Bazel and run the following:

    git clone --recursive https://gerrit.googlesource.com/gerrit
    cd gerrit && bazel build release

Install binary packages (Deb/Rpm)

The instruction how to configure GerritForge/BinTray repositories is here

On Debian/Ubuntu run:

    apt-get update & apt-get install gerrit=<version>-<release>

NOTE: release is a counter that starts with 1 and indicates the number of packages that have been released with the same version of the software.

On CentOS/RedHat run:

    yum clean all && yum install gerrit-<version>[-<release>]

On Fedora run:

    dnf clean all && dnf install gerrit-<version>[-<release>]

Use pre-built Gerrit images on Docker

Docker images of Gerrit are available on DockerHub

To run a CentOS 7 based Gerrit image:

    docker run -p 8080:8080 gerritforge/gerrit-centos7[:version]

To run a Ubuntu 15.04 based Gerrit image:

    docker run -p 8080:8080 gerritforge/gerrit-ubuntu15.04[:version]

NOTE: release is optional. Last released package of the version is installed if the release number is omitted.