commit e7a09e316dbbfff5d23cbb9687f7b0a5d44e7da0
author Martin Fick <quic_mfick@quicinc.com> Mon Aug 14 21:48:55 2023 -0600
committer Matthias Sohn <matthias.sohn@sap.com> Sat Aug 26 16:16:43 2023 +0200
tree 77cf601c487d2c79e5615f83b758ecaf2f46bd7f
parent f8f5979aa24668e0668b5dc0983ce871c430e846

Introduce core.packedIndexGitUseStrongRefs config key

Introduce a core.packedIndexGitUseStrongRefs configuration key, which
defaults to true so that the current behavior does not change. However,
setting it to false allows soft references to be used for Pack indices
instead of strong references so that they can be garbage collected when
there is memory pressure.

Pack objects can be large when associated with pack files with large
object counts, and this memory is not really accounted for or tracked by
the WindowCache and it can be very substantial at times, especially with
many large object count projects. A particularly problematic use case is
Gerrit's ls-projects command which loads very little data in the
WindowCache via ByteWindows, but ends up loading and holding many entire
indices in memory, sometimes even after the ByteWindows for their Pack
objects have already been garbage collected since they won't get cleared
until after a new ByteWindow is loaded. By using SoftReferences, single
use indices can get cleared when there is memory pressure and OOMs can
be easily avoided, drastically reducing the amount of memory required to
perform an ls-projects on large sites with many projects and large
object counts.

On one of our test sites, an ls-projects command with strong index
references requires more than 66GB of heap to complete successfully,
with soft index references it requires less than 23GB.

Change-Id: I3cb3df52f4ce1b8c554d378807218f199077d80b
Signed-off-by: Martin Fick <quic_mfick@quicinc.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>