tree 02bbb3f8f23ba91cb7fa1b6e11996b9459ee8be7
parent f371bfb3eb9b212c735aa7985b98eff89e34da80
author Matthias Sohn <matthias.sohn@sap.com> 1680176597 +0200
committer Matthias Sohn <matthias.sohn@sap.com> 1681914584 +0200

Prevent infinite loop rescanning the pack list on PackMismatchException

We found, when analysing an incident where Gerrit's gc runner thread got
stuck, that we can end up in an infinite loop in
ObjectDirectory#openPackedObject which tries to rescan the pack
list and starts over trying to open a packed object in an unconfined
loop if it catches a PackMismatchException.

Here the relevant part of a thread dump we created while the gc runner
was stuck:

"WorkQueue-2[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@350812a3[Not
completed,
task = java.util.concurrent.Executors$RunnableAdapter@5425d7ee]]" #72
tid=0x00007f73cee1c800 nid=0x584
runnable  [0x00007f7392d57000]
   java.lang.Thread.State: RUNNABLE
	at org.eclipse.jgit.internal.storage.file.WindowCache.removeAll(WindowCache.java:716)
	at org.eclipse.jgit.internal.storage.file.WindowCache.purge(WindowCache.java:399)
	at org.eclipse.jgit.internal.storage.file.PackFile.close(PackFile.java:296)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.reuseMap(ObjectDirectory.java:973)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.scanPacksImpl(ObjectDirectory.java:904)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.scanPacks(ObjectDirectory.java:895)
	- locked <0x000000050a498f60> (a
java.util.concurrent.atomic.AtomicReference)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.searchPacksAgain(ObjectDirectory.java:794)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:465)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:417)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:408)
	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
	at org.eclipse.jgit.lib.ObjectReader$1.open(ObjectReader.java:279)
	at org.eclipse.jgit.revwalk.RevWalk$2.next(RevWalk.java:1031)
	at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPack(PackWriter.java:1911)
	at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:960)
	at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:876)
	at org.eclipse.jgit.internal.storage.file.GC.writePack(GC.java:1168)
	at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:852)
	at org.eclipse.jgit.internal.storage.file.GC.doGc(GC.java:269)
	at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:220)
	at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:179)
	at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:112)
	at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:75)
	at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:71)
	at com.google.gerrit.server.git.GarbageCollectionRunner.run(GarbageCollectionRunner.java:76)
	at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:103)
	at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.18/Executors.java:515)
	at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.18/FutureTask.java:305)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.18/ScheduledThreadPoolExecutor.java:305)
	at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

The code in ObjectDirectory#openPackedObject [1] apparently assumes that
this is caused by a transient problem which it can resume from by
retrying. We use `core.trustFolderStat = false` on this server since it
uses NFS. The incident we had showed that we can enter into an infinite
loop here if there is a permanent mismatch between a pack file and its
corresponding pack index. I am not yet sure how this can happen.

Break the infinite loop by limiting the number of attempts rescanning
the pack list to 5 retries.  When we exceed this threshold set the type
of the PackMismatchException to permanent and rethrow it which breaks
the infinite loop.

Also apply the same limit in #getPackedObjectSize
and #selectObjectRepresentation where we use similar retry loops.

[1] https://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+/011c26ff36b9e76c84fc2459e337f159c0f55a9a/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/ObjectDirectory.java#465

Change-Id: I20fb63bcc1fdc3a03d39b963f06a90e6f0ba73dc