commit 081a9914a3b75e33d94f8d176882c044879895b0
author Luca Milanesio <luca.milanesio@gmail.com> Thu Nov 30 23:23:35 2023 +0000
committer Luca Milanesio <luca.milanesio@gmail.com> Sat Dec 02 02:27:44 2023 +0000
tree 107a211651ef2ff625b50926de4005adb93d2376
parent 53291cf2e490df210733c33a4ee95b81d69cba41

Make the indexing operation fail upon StorageException(s)

Change 270450 caused the blanking of the Lucene document
upon reindexing if any field caused a StorageException.

Whilst the overall intention was good, the implementation
caused the Lucene index replace operation to continue with
a Document without any fields instead of making the whole
operation fail.

StorageExceptions are thrown when the underlying storage,
being a filesystem or anything else, returns some errors.
Whether the failure is permanent or temporary (e.g. concurrent
GCs, repacking or pruning may cause sporadic StorageException(s))
returning a blank Lucene document was incorrect because, instead
of failing the operation, it was causing the change entry to be
completely removed from the index.

Let the StorageException fail the indexing operation, so that
existing entries may continue to exist, allowing the caller
to retry the operation.

The previous implementation, returning an empty Document, did
not allow any retry, because once the change entry was removed
from the index, it could not be discovered and reindexed anymore
for example by a `gerrit index changes <changenum>`.

Tested manually, applying a randomic StorageException thrown
during the fetching of the ChangeField extensions:

public static Set<String> getExtensions(ChangeData cd) {
  if (new Random().nextBoolean()) {
    throw new StorageException("Simulated storage exception");
  }
  return extensions(cd).collect(toSet());
}

Before this change, every time one change indexing throws a
StorageException, it disappears from the index. Eventually,
all changes will be eliminated and only an off-line reindex
or the reindex of all changes in the project would allow to
recover them.

After this change, some of the indexing operations are
successful and other fails. However, retrying the reindexing
operation would allow to reindex all of them.

Even if the above test case looks strange at first sight,
it simulates a real-life scenario where a low percentage
of indexing operation (0.1%) may fail because of sporadic
StorageExceptions. Before this change, some index entries
were missing on a daily basis (5 to 10 changes per day),
whilst after this change all indexing operation can be
retried and do not result in any indexing entry loss.

Bug: Issue 314113030
Release-Notes: Fail the change reindex operation upon StorageException(s)
Change-Id: Ia121f47f7a68c290849a22dea657804743a26b0d