#summary Notedb Migration Tasks and Challenges

=== What is notedb? ===

Notedb is the successor to !ReviewDb: a replacement database backend for Gerrit. The goal is to read code review metadata from the same set of repositories that store the data. This allows for improved atomicity, consistency of replication, and the creation of new features like federated review and offline review.

This document describes the state of migration (not necessarily completely up to date), the tasks that remain, and notes on some of the challenges we've encountered along the way. This document is *not* a full design document for notedb; if you're that curious, bug dborowitz and he will help you out.

Finally, this document is for core developers. If you are a casual user of Gerrit looking for documentation, you've come to the wrong place.

== Root Tables ==

While !ReviewDb has a lot of tables, there are relatively few "root" tables, that is, tables whose primary key's `get!ParentKey()` method returns null:
 * *Change*: subtables !ChangeMessage, !PatchSet, !PatchSetApproval, !PatchSetAncestor, !PatchLineComment, !TrackingId
 * *Account*: subtables !AccountExternalId, !AccountProjectWatch, !AccountSshKey, !AccountDiffPreference, !StarredChange
 * *!AccountGroup*: subtables !AccountGroupMember

TODO(dborowitz): Document other minor tables, audits, etc.

For each root entity in each of these tables, there is one DAG describing all modifications that have been applied to the change over time. Entities DAGs are stored in a repository corresponding to their function:
 * *Change*: stored in `refs/changes/YZ/XYZ/meta` in the destination repository
 * *Account*: stored in `refs/accounts/YZ/XYZ/meta` (TBD) in `All-Users`
 * *!AccountGroup*: stored in `TBD` in `All-Projects`

Most of this document focuses on Change entities, partly because it's the most complex, but also because that's where most effort to date has been focused.

== Changes: What's Done ==

Current progress, along with some possibly-interesting implementation notes.

 * !ChangeMessages: Stored in commit message body. Currently the subject of the commit message contains a machine-generated-but-readable summary like "Updated patch set 3"; we might decide to eliminate this and just use the !ChangeMessage.
 * !PatchSetApprovals: Stored as a footer "Label: Label-Name=Foo". Instead of storing an implicit 0 review for reviewers, include them explicitly in a separate Reviewer footer. Freeze labels at submit time along with the full submit rule evaluator results using "Submitted-with".
 * !PatchLineComments: The only thing thus far actually stored in a note, on the commit to which it applies. Drafts are stored in a per-user ref in All-Users.
 * !TrackingId: Killed this long ago and use the secondary index. (Just reminded myself I need to rip out the gwtorm entity.)
 * Change: Started storing obvious fields: owner, project, branch, topic.
 * !PatchSet: Storing IDs and created on timestamps right now, not reading from it yet.

== Changes: What Needs to Be Done ==

 * !PatchSetAncestors: Replace with a persistent cache, which should probably be rebuilt in !RebuildNotedb.
 * !PatchSet: Draft field. (Someday I think we should replace Draft with WIP, but I digress.)
 * Change: Kill more fields. Actually implement reading from changes; see challenges section.
 * Some sort of batch parser. If we get 100 changes back from a search, sequentially reading the DAG for each of those might take a while.
 * Benchmark and optimize the heck out of the parsers. Let's say a target is 1ms per change DAG. Related, we may also need to disable (eager) parsing of certain fields, if we can prove with benchmarks that they are problematic. (We already do this for !PatchLineComments, to avoid having to read anything but commits in the common case.)

== JGit Changes ==

 * Teach JGit to pack notedb refs in a separate pack file from everything else. We don't want to hurt locality within the main pack by interleaving metadata commits.
 * Teach JGit to better handle many small, separate commit graphs in the packer. Ordered chronologically, notedb metadata will be spread across a large number of separate DAGs. We will get better kernel buffer cache locality by clustering all commits in each disconnect DAG together. (But this may also hurt batch parsing; see above.)

== Challenges ==

TODO