Scaling.wiki - gerrit.wiki - Git at Google

 #summary Scaling Gerrit Installations

 ==== Master ====

 The first step to scaling is to scale your master server.  Some easy, but pricey ways to scale your master are:

   * Adding cores, some of the larger installations use 24 core machines
   * Adding RAM, most of the larger installations have at least 48GM, some have 100+GB
   * Ensure fast disk IO, SSDs help
   * Network, I suspect that most large installs use 10Gb ethernet

 ==== Mirrors/Slaves ====

 Once you have a decent master, it is probably worth adding either some git mirrors (if you do not need ACLs on your repos), or Gerrit slaves to help offload much of your read only requests.  Mirrors/Slaves can also help reduce LAN and WAN traffic if you place them nearer your users/build hosts.  Some of the larger installations have at least 7 of these.

 ==== Multi - Master ====

 Gerrit cannot yet be run in multi master mode, at least not the open source version.  The gerrit-review site does currently run in multi master mode, but it does not use the same code base as the open source version.  This however means that we are mostly aware of what it will take to make Gerrit support MultiMaster.  Any contributions in this direction should help make this a reality more quickly.

 ==== Issues ====

 As you scale, you will always run into some issues.  Depending on your specific setup those issues may be very different from what other people are running into.  Some of the specific axes which people may or may not need to scale on are:

  * Number of Projects: Some Gerrit installs have 10K projects, many have at least 1K
  * Number of Refs: Some Gerrit installs have projects with 250K refs
  * Repository Sizes:  Some Gerrit projects are 40G aggressively packed, these can often cause issues with email templates taking lots of CPU
  * Large Files: Gerrit may have difficulty with some very large files (what size can it handle easily?)
  * Number of Groups: Some Gerrit installs use 6K groups (most issues have been worked out here)
  * Number of Users: Many installs server 1K users, some serve at least 3K.


 ===== Jetty =====

 The default built in container which Gerrit uses is Jetty.  Some installations have had serious "Failed to dispatch" errors which lead to 100%CPU and filled up logs requiring a server reboot.  One way to workaround this issue appears to be using [https://gerrit-review.googlesource.com/#/c/35010/6/Documentation/install-tomcat.txt Tomcat] instead to replace Jetty.  Another solution is to set http.maxqueued = 0 if you are not too concerned with DDOS attacks.


 ===== Repo Syncs =====

 With beefier servers, many people have [http://groups.google.com/group/repo-discuss/browse_thread/thread/c8f003f2247d7157/ad6915f5558df8f5?lnk=gst&q=repo+sync+error#ad6915f5558df8f5 seen] channel master issues with ssh.  You may set GIT_SSH to avoid these:

   export GIT_SSH=$(which ssh)


 ===== Java GC =====

 If you overload your server too much, eventually java gc may block.  Large blocking times of over 1 min can be seen under extreme loads.  Limiting the ssh max threads can help prevent this.  Another solution may be to tweak the jvm gc parameters (I have not had luck with that).


 ===== Replication =====

 There are many scalibility issues which can plague replication, most are related to high ref counts, those are not specifically mentioned here, so you will likely need to first be familiar with the "High Ref Counts" section to make replication run smoothly.

 == JSch ==

 Jsch has threading issues which seem to serialize replication even across worker groups.  This has lead some teams to perform replication without using ssh (Jsch is the ssh implementation used inside jgit).  To do this, you may setup a "write only" git deamon on your slaves with a port only open to your Gerrit master and replicate via git daemon without authentication or encryption.  This is particularly useful if you have sites which replicate to at very different speeds.

 == Failed To Lock ==

 If you have a fairly smooth replication setup, your replication can start running into contention and failing with "Failed to Lock" errors in your logs.  This can happen when 2 separate threads attempt to replicate the same project/branch combination.  The Gerrit replication code does not yet retry these failures and it can result in your mirrors being out of date.

 ===== High Ref Counts =====

 High ref counts can have impacts in many places in the git/jgit/Gerrit stacks.  There are many ongoing fixes and tweaks to alleviated many of these problems, but some of them still remain.  Some can be "unofficially" worked around.

 == DB Slaves ==

 I am not aware of anyone using DB slaves yet, but it should help offload much of the WAN traffic to remote slaves.  This could be particularly helpful to help reduce some WAN traffic related to high ref counts.  A potential [https://gerrit-review.googlesource.com/#/c/35220 workaround].

 == git daemon mirrors ==

 Current versions (prior to git 1.7.11) will use an [http://marc.info/?l=git&m=133310001303068&w=2 excessive amount of CPU] when receiving pushes on sites with high ref counts.  Upgrading git there can help drastically reduce your replication time in these cases.

 == git ==

 Suggest to your users that they use the latest git possible, many of the older versions (which are still the defaults on many distros) have severe problems with high ref counts.  Particularly [http://marc.info/?l=git&m=131552810309538&w=2 bad] versions are between 1.7.4.1 and 1.7.7.  Git 1.8.1 seems to have some speedups in fetches of high ref counts compared to even 1.7.8.

 == jgit ==

 jGit still has a [http://groups.google.com/group/repo-discuss/browse_thread/thread/d0914922dc565516 performance problem] with high refs.  An experimental [https://git.eclipse.org/r/5491 patch] has been proposed and will drastically reduce upload and replication times in Gerrit if applied for repos with many (> 60K?) patch sets.

 There are some very high performance patches which make jgit extremely fast.

 == Tags ==

 If you have android repositories, you likely use around 400 of them.  Cross project tagging can be [http://marc.info/?l=git&m=133772533609422&w=2 problematic].  There are no good solutions yet to this problem.

 == Changes ==

 == ACLS ==


 ===== Disk Space =====

 Installations which do not have enough spare disk space for their repos can run into problems easily.  Be aware that git repos contain highly compressed data and that at times this data may need to be uncompressed.  It is easy to underestimate the temporary needs of repositories because git is so good at compressing this data.  However, minor changes can cause repositories to "explode" so it is good to plan for this and leave a lot of free space for this to never be an issue.  This is particularly important for those using SSDs where they might be more likely to skimp on space.

 == Git GC Repo Explosions ==

 Under certain conditions git gc can cause a repo explosion, primarily when unreferenced objects are removed from pack files and are placed as loose refs in the filesystem.  Eventually git gc should prune these, but until that happens serious problems can occur.

 Some of the situations which can cause many unreferenced objects:

   *  A user uploads a change to the wrong repository and it gets rejected by Gerrit
   *  Tags are [http://marc.info/?l=git&m=131829057610072&w=2 deleted] from the linux repo


 == Git GC ==

 Running GC regularly is important, jgit does not have built in gc so unlike a regular git repo it will not auto run every now and then.  Setting up a crontab is probably a good idea.  If you do run gc too often, however excessive pack file churn can also be a problem.  A potential [https://gerrit-review.googlesource.com/#/c/35215/ solution] for packfile churn:

   nice -n 19 ionice -c 3 gitexproll.sh -r 5 ...

 Pack file churn can lead to several issues, RAM utilization, Disk utilization and excessive BW utilization from filesystem mirroring scripts (such as rysnc).

 == Keep and Noz files ==

 Currently, Gerrit may leave behind some temporary files in your git repos when it shuts down (particularly if ungraceful).  There are some temporary files which begin with "noz", this can consume disk space if left uncleaned.  There are also some .keep files in the objects/pack directories which can be left behind, these don't in themselves don't take space, but they will prevent git gc from repacking the packfile they are associated with which can lead to poor disk space utilization and performance issues.


 == ~/.gerritcodereview ==

 The temporary unjared war files in here can build up.  (This has been move to review_site/tmp in Gerrit 2.5+)

 ===== hooks =====

 Servers with lots of RAM are succeptible to slow forks which can delay each hook invocation quite a bit.  When java uses over 10G of memory, it may add at least a second to each hook invocation.  Using java 7 seems to avoid this problem and makes hooks blazingly fast again.
	#summary Scaling Gerrit Installations

	==== Master ====

	The first step to scaling is to scale your master server. Some easy, but pricey ways to scale your master are:

	* Adding cores, some of the larger installations use 24 core machines
	* Adding RAM, most of the larger installations have at least 48GM, some have 100+GB
	* Ensure fast disk IO, SSDs help
	* Network, I suspect that most large installs use 10Gb ethernet

	==== Mirrors/Slaves ====

	Once you have a decent master, it is probably worth adding either some git mirrors (if you do not need ACLs on your repos), or Gerrit slaves to help offload much of your read only requests. Mirrors/Slaves can also help reduce LAN and WAN traffic if you place them nearer your users/build hosts. Some of the larger installations have at least 7 of these.

	==== Multi - Master ====

	Gerrit cannot yet be run in multi master mode, at least not the open source version. The gerrit-review site does currently run in multi master mode, but it does not use the same code base as the open source version. This however means that we are mostly aware of what it will take to make Gerrit support MultiMaster. Any contributions in this direction should help make this a reality more quickly.

	==== Issues ====

	As you scale, you will always run into some issues. Depending on your specific setup those issues may be very different from what other people are running into. Some of the specific axes which people may or may not need to scale on are:

	* Number of Projects: Some Gerrit installs have 10K projects, many have at least 1K
	* Number of Refs: Some Gerrit installs have projects with 250K refs
	* Repository Sizes: Some Gerrit projects are 40G aggressively packed, these can often cause issues with email templates taking lots of CPU
	* Large Files: Gerrit may have difficulty with some very large files (what size can it handle easily?)
	* Number of Groups: Some Gerrit installs use 6K groups (most issues have been worked out here)
	* Number of Users: Many installs server 1K users, some serve at least 3K.





	===== Jetty =====

	The default built in container which Gerrit uses is Jetty. Some installations have had serious "Failed to dispatch" errors which lead to 100%CPU and filled up logs requiring a server reboot. One way to workaround this issue appears to be using [https://gerrit-review.googlesource.com/#/c/35010/6/Documentation/install-tomcat.txt Tomcat] instead to replace Jetty. Another solution is to set http.maxqueued = 0 if you are not too concerned with DDOS attacks.


	===== Repo Syncs =====

	With beefier servers, many people have [http://groups.google.com/group/repo-discuss/browse_thread/thread/c8f003f2247d7157/ad6915f5558df8f5?lnk=gst&q=repo+sync+error#ad6915f5558df8f5 seen] channel master issues with ssh. You may set GIT_SSH to avoid these:

	export GIT_SSH=$(which ssh)


	===== Java GC =====

	If you overload your server too much, eventually java gc may block. Large blocking times of over 1 min can be seen under extreme loads. Limiting the ssh max threads can help prevent this. Another solution may be to tweak the jvm gc parameters (I have not had luck with that).


	===== Replication =====

	There are many scalibility issues which can plague replication, most are related to high ref counts, those are not specifically mentioned here, so you will likely need to first be familiar with the "High Ref Counts" section to make replication run smoothly.

	== JSch ==

	Jsch has threading issues which seem to serialize replication even across worker groups. This has lead some teams to perform replication without using ssh (Jsch is the ssh implementation used inside jgit). To do this, you may setup a "write only" git deamon on your slaves with a port only open to your Gerrit master and replicate via git daemon without authentication or encryption. This is particularly useful if you have sites which replicate to at very different speeds.

	== Failed To Lock ==

	If you have a fairly smooth replication setup, your replication can start running into contention and failing with "Failed to Lock" errors in your logs. This can happen when 2 separate threads attempt to replicate the same project/branch combination. The Gerrit replication code does not yet retry these failures and it can result in your mirrors being out of date.

	===== High Ref Counts =====

	High ref counts can have impacts in many places in the git/jgit/Gerrit stacks. There are many ongoing fixes and tweaks to alleviated many of these problems, but some of them still remain. Some can be "unofficially" worked around.

	== DB Slaves ==

	I am not aware of anyone using DB slaves yet, but it should help offload much of the WAN traffic to remote slaves. This could be particularly helpful to help reduce some WAN traffic related to high ref counts. A potential [https://gerrit-review.googlesource.com/#/c/35220 workaround].

	== git daemon mirrors ==

	Current versions (prior to git 1.7.11) will use an [http://marc.info/?l=git&m=133310001303068&w=2 excessive amount of CPU] when receiving pushes on sites with high ref counts. Upgrading git there can help drastically reduce your replication time in these cases.

	== git ==

	Suggest to your users that they use the latest git possible, many of the older versions (which are still the defaults on many distros) have severe problems with high ref counts. Particularly [http://marc.info/?l=git&m=131552810309538&w=2 bad] versions are between 1.7.4.1 and 1.7.7. Git 1.8.1 seems to have some speedups in fetches of high ref counts compared to even 1.7.8.

	== jgit ==

	jGit still has a [http://groups.google.com/group/repo-discuss/browse_thread/thread/d0914922dc565516 performance problem] with high refs. An experimental [https://git.eclipse.org/r/5491 patch] has been proposed and will drastically reduce upload and replication times in Gerrit if applied for repos with many (> 60K?) patch sets.

	There are some very high performance patches which make jgit extremely fast.

	== Tags ==

	If you have android repositories, you likely use around 400 of them. Cross project tagging can be [http://marc.info/?l=git&m=133772533609422&w=2 problematic]. There are no good solutions yet to this problem.

	== Changes ==

	== ACLS ==


	===== Disk Space =====

	Installations which do not have enough spare disk space for their repos can run into problems easily. Be aware that git repos contain highly compressed data and that at times this data may need to be uncompressed. It is easy to underestimate the temporary needs of repositories because git is so good at compressing this data. However, minor changes can cause repositories to "explode" so it is good to plan for this and leave a lot of free space for this to never be an issue. This is particularly important for those using SSDs where they might be more likely to skimp on space.

	== Git GC Repo Explosions ==

	Under certain conditions git gc can cause a repo explosion, primarily when unreferenced objects are removed from pack files and are placed as loose refs in the filesystem. Eventually git gc should prune these, but until that happens serious problems can occur.

	Some of the situations which can cause many unreferenced objects:

	* A user uploads a change to the wrong repository and it gets rejected by Gerrit
	* Tags are [http://marc.info/?l=git&m=131829057610072&w=2 deleted] from the linux repo


	== Git GC ==

	Running GC regularly is important, jgit does not have built in gc so unlike a regular git repo it will not auto run every now and then. Setting up a crontab is probably a good idea. If you do run gc too often, however excessive pack file churn can also be a problem. A potential [https://gerrit-review.googlesource.com/#/c/35215/ solution] for packfile churn:

	nice -n 19 ionice -c 3 gitexproll.sh -r 5 ...

	Pack file churn can lead to several issues, RAM utilization, Disk utilization and excessive BW utilization from filesystem mirroring scripts (such as rysnc).

	== Keep and Noz files ==

	Currently, Gerrit may leave behind some temporary files in your git repos when it shuts down (particularly if ungraceful). There are some temporary files which begin with "noz", this can consume disk space if left uncleaned. There are also some .keep files in the objects/pack directories which can be left behind, these don't in themselves don't take space, but they will prevent git gc from repacking the packfile they are associated with which can lead to poor disk space utilization and performance issues.


	== ~/.gerritcodereview ==

	The temporary unjared war files in here can build up. (This has been move to review_site/tmp in Gerrit 2.5+)

	===== hooks =====

	Servers with lots of RAM are succeptible to slow forks which can delay each hook invocation quite a bit. When java uses over 10G of memory, it may add at least a second to each hook invocation. Using java 7 seems to avoid this problem and makes hooks blazingly fast again.