| = Gerrit Code Review - Repository Maintenance | |
| == Description | |
| Each project in Gerrit is stored in a bare Git repository. Gerrit uses | |
| the JGit library to access (read and write to) these Git repositories. | |
| As modifications are made to a project, Git repository maintenance will | |
| be needed or performance will eventually suffer. When using the Git | |
| command line tool to operate on a Git repository, it will run `git gc` | |
| every now and then on the repository to ensure that Git garbage | |
| collection is performed. However regular maintenance does not happen as | |
| a result of normal Gerrit operations, so this is something that Gerrit | |
| administrators need to plan for. | |
| Gerrit has a built-in feature which allows it to run Git garbage | |
| collection on repositories. This can be | |
| link:config-gerrit.html#gc[configured] to run on a regular basis, and/or | |
| this can be run manually with the link:cmd-gc.html[gerrit gc] ssh | |
| command, or with the link:rest-api-projects.html#run-gc[run-gc] REST API. | |
| Some administrators will opt to run `git gc` or `jgit gc` outside of | |
| Gerrit instead. There are many reasons this might be done, the main one | |
| likely being that when it is run in Gerrit it can be very resource | |
| intensive and scheduling an external job to run Git garbage collection | |
| allows administrators to finely tune the approach and resource usage of | |
| this maintenance. | |
| == Git Garbage Collection Impacts | |
| Unlike a typical server database, access to Git repositories is not | |
| marshalled through a single process or a set of inter communicating | |
| processes. Unfortunately the design of the on-disk layout of a Git | |
| repository does not allow for 100% race free operations when accessed by | |
| multiple actors concurrently. These design shortcomings are more likely | |
| to impact the operations of busy repositories since racy conditions are | |
| more likely to occur when there are more concurrent operations. Since | |
| most Gerrit servers are expected to run without interruptions, Git | |
| garbage collection likely needs to be run during normal operational hours. | |
| When it runs, it adds to the concurrency of the overall accesses. Given | |
| that many of the operations in garbage collection involve deleting files | |
| and directories, it has a higher chance of impacting other ongoing | |
| operations than most other operations. | |
| === Interrupted Operations | |
| When Git garbage collection deletes a file or directory that is | |
| currently in use by an ongoing operation, it can cause that operation to | |
| fail. These sorts of failures are often single shot failures, i.e. the | |
| operation will succeed if tried again. An example of such a failure is | |
| when a pack file is deleted while Gerrit is sending an object in the | |
| file over the network to a user performing a clone or fetch. Usually | |
| pack files are only deleted when the referenced objects in them have | |
| been repacked and thus copied to a new pack file. So performing the same | |
| operation again after the fetch will likely send the same object from | |
| the new pack instead of the deleted one, and the operation will succeed. | |
| === Data Loss | |
| It is possible for data loss to occur when Git garbage collection runs. | |
| This is very rare, but it can happen. This can happen when an object is | |
| believed to be unreferenced when object repacking is running, and then | |
| garbage collection deletes it. This can happen because even though an | |
| object may indeed be unreferenced when object repacking begins and | |
| reachability of all objects is determined, it can become referenced by | |
| another concurrent operation after this unreferenced determination but | |
| before it gets deleted. When this happens, a new reference can be | |
| created which points to a now missing object, and this will result in a | |
| loss. | |
| == Reducing Git Garbage Collection Impacts | |
| JGit has a `preserved` directory feature which is intended to reduce | |
| some of the impacts of Git garbage collection, and Gerrit can take | |
| advantage of the feature too. The `preserved` directory is a | |
| subdirectory of a repository's `objects/pack` directory where JGit will | |
| move pack files that it would normally delete when `jgit gc` is invoked | |
| with the `--preserve-oldpacks` option. It will later delete these files | |
| the next time that `jgit gc` is run if it is invoked with the | |
| `--prune-preserved` option. Using these flags together on every `jgit gc` | |
| invocation means that packfiles will get an extended lifetime by one | |
| full garbage collection cycle. Since an atomic move is used to move these | |
| files, any open references to them will continue to work, even on NFS. On | |
| a busy repository, preserving pack files can make operations much more | |
| reliable, and interrupted operations should almost entirely disappear. | |
| Moving files to the `preserved` directory also has the ability to reduce | |
| data loss. If JGit cannot find an object it needs in its current object | |
| DB, it will look into the `preserved` directory as a last resort. If it | |
| finds the object in a pack file there, it will restore the | |
| slated-to-be-deleted pack file back to the original `objects/pack` | |
| directory effectively "undeleting" it and making all the objects in it | |
| available again. When this happens, data loss is prevented. | |
| One advantage of restoring preserved packfiles in this way when an | |
| object is referenced in them, is that it makes loosening unreferenced | |
| objects during Git garbage collection, which is a potentially expensive, | |
| wasteful, and performance impacting operation, no longer desirable. It | |
| is recommended that if you use Git for garbage collection, that you use | |
| the `-a` option to `git repack` instead of the `-A` option to no longer | |
| perform this loosening. | |
| When Git is used for garbage collection instead of JGit, it is fairly | |
| easy to wrap `git gc` or `git repack` with a small script which has a | |
| `--prune-preserved` option which behaves as mentioned above by deleting | |
| any pack files currently in the preserved directory, and also has a | |
| `--preserve-oldpacks` option which then hardlinks all the currently | |
| existing pack files from the `objects/pack` directory into the | |
| `preserved` directory right before calling the real Git command. This | |
| approach will then behave similarly to `jgit gc` with respect to | |
| preserving pack files. An implementation is available | |
| link:https://gerrit.googlesource.com/gerrit/+/refs/heads/master/contrib/git-gc-preserve[here]. | |
| GERRIT | |
| ------ | |
| Part of link:index.html[Gerrit Code Review] | |
| SEARCHBOX | |
| --------- |