| commit | d0e872ef2835ce439f020322fad9678cb96f7e1b | [log] [tgz] |
|---|---|---|
| author | Prudhvi Akhil Alahari <prudhvi.alahari@linaro.org> | Fri Feb 10 10:27:03 2023 +0530 |
| committer | Prudhvi Akhil Alahari <prudhvi.alahari@linaro.org> | Mon Feb 13 12:18:51 2023 +0530 |
| tree | ed5b48f09441b8708235a178debd9bb3c856878c | |
| parent | 8a25e22d7e82bdd7e27ec05a6f6698445bd13ff7 [diff] |
Evict ref from the ref cache even when ref update fails If for some reason there is a miss-match between a persisted ref value and the cached value of that ref, it is likely to result in a ref-update failure. There are many places in Gerrit where code is retried on ref-update failures with the hopes that the failure can be recovered from. To make this recovery possible, it is essential to ensure that the outdated cached ref value no longer be used, and evicting the cached value even on failures makes this possible. Some reasons where cached ref values could be outdated are: 3rd party actors modifying the repos outside of Gerrit, multi-primary setups. For example, consider a multi-primary Gerrit setup with 2 primaries pointing to the repositories on NFS. When one primary receives a review command and at the same time another primary updates the same change with a new patchset and succeeds, then the review command would fail with Lock failure. In this case, review command would retry the operation, but it fails again because the cached ref value is still outdated. In order for a retry operation to succeed, we need to invalidate the ref from the cache when ref update fails. Change-Id: I724d5ba3d756e6a22a0b8c64be7ebd4f82d69d14
When Serialize AccountCache series was introduced it simplified the cache eviction by always reaching out to JGit for data. Unfortunately it comes with price which is especially high when All-Users repository is accessed through the NFS and core.trustFolderStat = false is configured in ${GERRIT_SITE}/etc/jgit.config (quite common setup for HA/Multi-Site ens).
This plugin was developed to introduce the in-memory cache (managed by Gerrit so that evictions could be coordinated to multiple nodes) that reduces the price for reaching to refs in JGit. It is a Gerrit native alternative (that can be applied to Gerrit 3.2) to work that is currently under progress for caching Refs in JGit.
Here is the short comparison of heavy-refs-related operations performance. The test scenario was to get random change details (over the same REST API that is used in Gerrit's details page) in 8 parallel threads over 5mins period of time. The core.trustFolderStat = false was set in ${GERRIT_SITE}/etc/jgit.config. It was called against:
stable-3.1 in the results)stable-3.2 in the results)stable-3.2-libCache in the results).Note that TRS is Reqs/Sec for each Thread.
| version | TRS Avg | TRS Std Dev | TRS Max | Total Reqs/sec | Transfer/sec(MB)| | --------------------------------- | ------- | ----------- | ------- | -------------- | --------------- | | stable-3.1 | 57,33 | 8,26 | 80 | 456,95 | 4,34 | | stable-3.2 | 13,87 | 4,92 | 20 | 110,18 | 1,07 | | stable-3.2-libCache | 105,27 | 14,55 | 150 | 834,88 | 8,41 | | stable-3.1 vs stable-3.2 | 313,34% | 67,89% | 300,00% | 314,73% | 305,61% | | stable-3.2-libCache vs stable-3.2 | 658,98% | 195,73% | 650,00% | 657,74% | 685,98% | | stable-3.2-libCache vs stable-3.1 | 83,62% | 76,15% | 87,50% | 82,71% | 93,78% |
One can clearly see that in this setup using this library module outperforms both Gerrit 3.2 and 3.1 by factor of 6 and 2 correspondingly. The test script, detailed description and more results are available here.
Clone or link this plugin to the plugins directory of Gerrit‘s source tree, and then run bazel build on the plugin’s directory.
Example:
git clone --recursive https://gerrit.googlesource.com/gerrit cd plugins git clone "https://gerrit.googlesource.com/modules/cached-refdb" cd .. && bazel build plugins/cached-refdb
The output plugin jar is created in:
bazel-bin/plugins/cached-refdb/cached-refdb.jar
Copy the cached-refdb.jar into the ${GERRIT_SITE}/lib/ so that it is being loaded when the Gerrit instance is started. Note that the following configuration options need to be added
git config --file ${GERRIT_SITE}/etc/gerrit.config --add gerrit.installDbModule\
com.googlesource.gerrit.plugins.cachedrefdb.LibDbModule
git config --file ${GERRIT_SITE}/etc/gerrit.config --add gerrit.installModule\
com.googlesource.gerrit.plugins.cachedrefdb.LibSysModule
NOTE: There are situations where the binding of the module to the Gerrit's GitRepositoryManager is not desired; e.g., when using this module together with others that are trying to override it at the same time.
It is possible to just load the module using the following two options:
git config --file ${GERRIT_SITE}/etc/gerrit.config --add gerrit.installDbModule\ com.googlesource.gerrit.plugins.cachedrefdb.LibModule git config --file ${GERRIT_SITE}/etc/gerrit.config --add gerrit.installModule\ com.googlesource.gerrit.plugins.cachedrefdb.LibSysModule
By default cache can hold up to 1024 refs which will not be sufficient for any production site therefore one can configure it through the standard Gerrit cache configuration means e.g.
git config --file ${GERRIT_SITE}/etc/gerrit.config cache.ref_by_name.memoryLimit 10240
Note that library module requires the Gerrit instance restart in order to pick up the configuration changes.