Documentation/note-db.txt

:linkattrs:
= Gerrit Code Review - NoteDb Backend

NoteDb is the storage backend for code review metadata. It is based on
Git, so code reviews are stored together with the code under review.
NoteDb replaced the traditional SQL backend for change, account and group
metadata that was used in the 2.x series.

.Advantages
- *Simplicity*: All data is stored in one location in the site directory, rather
  than being split between the site directory and a possibly external database
  server.
- *Consistency*: Replication and backups can use a snapshot of the Git
  repository refs, which will include both the branch and patch set refs, and
  the change metadata that points to them.
- *Auditability*: Rather than storing mutable rows in a database, modifications
  to changes are stored as a sequence of Git commits, automatically preserving
  history of the metadata. +
  There are no strict guarantees, and meta refs may be rewritten, but the
  default assumption is that all operations are logged.
- *Extensibility*: Plugin developers can add new fields to metadata without the
  core database schema having to know about them.
- *New features*: Enables simple federation between Gerrit servers, as well as
  offline code review and interoperation with other tools.

For an example NoteDb change, poke around at this one:
----
  git fetch https://gerrit.googlesource.com/gerrit refs/changes/70/98070/meta \
      && git log -p FETCH_HEAD
----

== Current Status

NoteDb is the only database format supported by Gerrit 3.0+. The
change data migration tools are only included in Gerrit 2.16; they are
not available in 3.0, so any upgrade from Gerrit 2.x to 3.x must go through
2.16 to effect the NoteDb upgrade.

== Format

Each review ("change") in Gerrit is numbered. The different revisions
("patchsets") of a change 12345 are stored under
----
  refs/changes/45/12345/${PATCHSET_NUMBER}
----

The revisions are stored as commits to the main project, ie. if you
fetch this ref, you can check out the proposed change.

A change 12345 has its review metadata under
----
  refs/changes/45/12345/meta
----
The metadata is a notes branch. The commit messages on the branch hold
modifications to global data of the change (votes, global comments). The inline
comments are in a
link:https://eclipse.gerrithub.io/plugins/gitiles/eclipse-jgit/jgit/\+/refs/heads/master/org.eclipse.jgit/src/org/eclipse/jgit/notes/NoteMap.java[NoteMap],
where the key is the commit SHA-1 of the patchset
that the comment refers to, and the value is JSON data. The format of the
JSON is in the
link:https://gerrit.googlesource.com/gerrit/\+/master/java/com/google/gerrit/server/notedb/RevisionNoteData.java[RevisionNoteData]
which contains 
link:https://gerrit.googlesource.com/gerrit/\+/master/java/com/google/gerrit/entities/Comment.java[Comment] entities.

For example:
----
   {
      "key": {
        "uuid": "c7be1334_47885e36",
        "filename":
"java/com/google/gerrit/server/restapi/project/CommitsCollection.java",
        "patchSetId": 7
      },
      "lineNbr": 158,
      "author": {
        "id": 1026112
      },
      "writtenOn": "2019-11-06T09:00:50Z",
      "side": 1,
      "message": "nit: factor this out in a variable, use
toImmutableList as collector",
      "range": {
        "startLine": 156,
        "startChar": 32,
        "endLine": 158,
        "endChar": 66
      },
      "revId": "071c601d6ee1a2a9f520415fd9efef8e00f9cf60",
      "serverId": "173816e5-2b9a-37c3-8a2e-48639d4f1153",
      "unresolved": true
    },
----

Automated systems may post "robot comments" instead of normal
comments, which are an extension of the previous comment, defined in
the
link:https://gerrit.googlesource.com/gerrit/\+/master/java/com/google/gerrit/entities/RobotComment.java[RobotComment]
class.

[[migration]]
== Migration

Migrating change metadata can take a long time for large sites, so
administrators choose whether to do the migration offline or online, depending
on their available resources and tolerance for downtime.

Only change metadata requires manual steps to migrate it from ReviewDb; account
and group data is migrated automatically by `gerrit.war init`.

[[online-migration]]
=== Online

Note that online migration is only available in 2.x. To do the online migration
from 2.14.x or 2.15.x to 3.0, it is necessary to first upgrade to 2.16.x.

To start the online migration, set the `noteDb.changes.autoMigrate` option in
`gerrit.config` and restart Gerrit:
----
[noteDb "changes"]
  autoMigrate = true
----

Alternatively, pass the `--migrate-to-note-db` flag to
`gerrit.war daemon`:
----
  java -jar gerrit.war daemon -d /path/to/site --migrate-to-note-db
----

Both ways of starting the online migration are equivalent. Once started, it is
safe to restart the server at any time; the migration will pick up where it left
off. Migration progress will be reported to the Gerrit logs.

*Advantages*

* No downtime required.

*Disadvantages*

* Only available in 2.x; not available in Gerrit 3.0.
* Much slower than offline; uses only a single thread, to leave resources
  available for serving traffic.
* Performance may be degraded, particularly of updates; data needs to be written
  to both ReviewDb and NoteDb while the migration is in progress.

[[offline-migration]]
=== Offline

To run the offline migration, run the `migrate-to-note-db` program:
----
  java -jar gerrit.war migrate-to-note-db -d /path/to/site
----

Once started, it is safe to cancel and restart the migration process, or to
switch to the online process.

[NOTE]
Migration requires a heap size comparable to running a Gerrit server. If you
normally run `gerrit.war daemon` with an `-Xmx` flag, pass that to the migration
tool as well.

[NOTE]
Note that by appending `--reindex false` to the above command, you can skip the
lengthy, implicit reindexing step of the migration. This is useful if you plan
to perform further Gerrit upgrades while the server is offline and have to
reindex later anyway (E.g.: a follow-up upgrade to Gerrit 3.2 or newer, which
requires to reindex changes anyway).

*Advantages*

* Much faster than online; can use all available CPUs, since no live traffic
  needs to be served.
* No degraded performance of live servers due to writing data to 2 locations.

*Disadvantages*

* Available in Gerrit 2.15 and 2.16 only.
* May require substantial downtime; takes about twice as long as an
  link:pgm-reindex.html[offline reindex]. (In fact, one of the migration steps is a
  full reindex, so it can't possibly take less time.)

[[trial-migration]]
==== Trial mode

The migration tool also supports "trial mode", where changes are
migrated to NoteDb and read from NoteDb at runtime, but their primary storage
location is still ReviewDb, and data is kept in sync between the two locations.

To run the migration in trial mode, add `--trial` to `migrate-to-note-db` or
`daemon`:
----
  java -jar gerrit.war migrate-to-note-db --trial -d /path/to/site
  # OR
  java -jar gerrit.war daemon -d /path/to/site --migrate-to-note-db --trial
----

Or, set `noteDb.changes.trial=true` in `gerrit.config`.

There are several use cases for trial mode:

* Help test early releases of the migration tool for bugs with lower risk.
* Try out new NoteDb-only features like
  link:rest-api-changes.html#get-hashtags[hashtags] without running the full
  migration.

To continue with the full migration after running the trial migration, use
either the online or offline migration steps as normal. To revert to
ReviewDb-only, remove `noteDb.changes.read` and `noteDb.changes.write` from
`notedb.config` and restart Gerrit.

== Configuration

The migration process works by setting a configuration option in `notedb.config`
for each step in the process, then performing the corresponding data migration.

Config options are read from `notedb.config` first, falling back to
`gerrit.config`. If editing config manually, you may edit either file, but the
migration process itself only touches `notedb.config`. This means if your
`gerrit.config` is managed with Puppet or a similar tool, it can overwrite
`gerrit.config` without affecting the migration process. You should not manage
`notedb.config` with Puppet, but you may copy values back into `gerrit.config`
and delete `notedb.config` at some later point after completing the migration.

In general, users should not set the options described below manually; this
section serves primarily as a reference.

- `noteDb.changes.write=true`: During a ReviewDb write, the state of the change
  in NoteDb is written to the `note_db_state` field in the `Change` entity.
  After the ReviewDb write, this state is written into NoteDb, resulting in
  effectively double the time for write operations. NoteDb write errors are
  dropped on the floor, and no attempt is made to read from ReviewDb or correct
  errors (without additional configuration, below).
- `noteDb.changes.read=true`: Change data is written
  to and read from NoteDb, but ReviewDb is still the source of truth. During
  reads, first read the change from ReviewDb, and compare its `note_db_state`
  with what is in NoteDb. If it doesn't match, immediately "auto-rebuild" the
  change, copying data from ReviewDb to NoteDb and returning the result.
- `noteDb.changes.primaryStorage=NOTE_DB`: New changes are written only to
  NoteDb, but changes whose primary storage is ReviewDb are still supported.
  Continues to read from ReviewDb first as in the previous stage, but if the
  change is not in ReviewDb, falls back to reading from NoteDb. +
  Migration of existing changes is described in the link:#migration[Migration]
  section above. +
  Due to an implementation detail, writes to Changes or related tables still
  result in write calls to the database layer, but they are inside a transaction
  that is always rolled back.
- `noteDb.changes.disableReviewDb=true`: All access to Changes or related tables
  is disabled; reads return no results, and writes are no-ops. Assumes the state
  of all changes in NoteDb is accurate, and so is only safe once all changes are
  NoteDb primary. Otherwise, reading changes only from NoteDb might result in
  inaccurate results, and writing to NoteDb would compound the problem. +

== NoteDB to ReviewDB rollback

In case of rollback from NoteDB to ReviewDB, all the meta refs and the
sequence ref need to be removed.
The link:https://gerrit.googlesource.com/gerrit/+/refs/heads/master/contrib/remove-notedb-refs.sh[remove-notedb-refs.sh,role=external,window=_blank]
script has been written to automate this process.