commit ba22868bcf9c5802381d77f4f8a8e8f57049156f
author Dave Borowitz <dborowitz@google.com> Wed Aug 03 12:54:38 2016 -0400
committer Dave Borowitz <dborowitz@google.com> Wed Aug 03 12:54:38 2016 -0400
tree 32656213b8bda3d6c662cc8addc7df045c6f359c
parent c66120af5aa776bf1cb4168dc0675e7351bbd90b

Fix racy rebuilding when the winning writer fails

Consider the following scenario:

1. Thread 1 rebuilds in-memory to state A, and successfully updates
   the state in ReviewDb to A.
2. Thread 1 attempts the ref update but fails.
3. Thread 2 rebuilds in-memory and gets A again, which throws
   AbortUpdateException because the state in ReviewDb is already A.

Now we are in a state where step (3) repeats indefinitely until there
is a write to the change that triggers a NoteDbChangeState update in
ReviewDb. At that point it will skip the update of the NoteDb ref,
since it's out of date, but the *next* read will go back to step (1)
and have another shot at rebuilding it.

Fix this by checking the actual state of the refs after an
AbortUpdateException, and retrying the ref update if it doesn't match.
This has the slight downside that if N threads are trying to rebuild
the change at once, likely N-1 or N of them will still fail. However,
there's not much we can do in this case: there is no way to tell it to
rebuild only if we think there isn't going to be contention on the
ref. Because we return the staged results and ignore errors coming from
ChangeRebuilder#execute, this shouldn't affect correctness, just
performance for the requests doing the rebuild.

Change-Id: I6243f94d3e99535d1c1abbea0a1b3f075557d93a