Locks
=====

Each entry of the locks table consists of a:

| Name | Description | Type |
| ------------- |:-------------|:-------------|
`project` | Project name | `String`
`ref` | Ref path | `String`
`heartbeat` | Timestamp updated every few seconds to identify staleness | `Timestamp`
`token` | Timestamp used to identify the lock version | `Timestamp`
`owner` | Gerrit instance identifier | `String`

The primary key is a composite of (project, ref).


Design
------

When the global-refdb attempts to acquire a lock on a given entry, we first try
to insert the lock into the locks table.

If the insertion succeeds:
* Set the Lock object's token value to match the token in the database
    * The token is a unique identifier for the Lock which establishes versioning.
* Start a new process to update the heartbeat value every few seconds
    * The heartbeat ensures freshness of the lock and is used when deciding
      whether or not a Lock is stale.

If the insertion fails due to a lock on that project/ref already existing:
* Check the timestamp of the existing lock.
	* If it is sufficiently fresh, pass
          on attempting to reclaim the lock and end here.
	* If the timestamp is stale, attempt to reclaim the lock as described
          below.

Reclaiming an existing stale lock:
* Check the existing token (now known as oldToken) of the lock
* In one read/write transaction:
  * Check that the token of the lock is still oldToken.
  * Attempt to insert a meta reclaim-lock into the table to mark that the old
    lock is being reclaimed.
  * Refresh the old lock, issuing an update with our new heartbeat, token, and
    owner.
  * If any of those steps fail, our information is outdated and the attempt
    will fail. As the read/write transaction is atomic, either every buffered
    mutation succeeds or all fail.
* If all succeed, the lock has been reclaimed and we set up the heartbeat and
  clean up (deleting the meta reclaim-lock).

Updating a heartbeat:
* Check if the tokens of the in-database lock and the Lock object match.
  * If so, update the heartbeat to the current timestamp.
  * If not, shut down the heartbeat process; the lock has been reclaimed.

When the lock is closed:
* Shut down the heartbeat process
* Check if the tokens of the in-database lock and the Lock object match.
  * If so, delete the in-database lock.
  * If not, stop - the lock has been reclaimed.

Example
-------
Reclaim process:
| Project | Ref | Heartbeat | Token | Description
| --------|-----|:----------|:------|:-----------
`project1` | `ref1` | `Stale timestamp` | `T1` | When a new process attempts to lock `project1-ref1`, it finds a stale heartbeat and can then insert a reclaim lock.
`RECLAIM_project1_T1` | `ref1` | `Tx` | `Tx` | While this reclaim lock exists, no other process can succeed in the reclaim process.

Result:
| Project | Ref | Heartbeat | Token | Description
| --------|-----|:----------|:------|:-----------
`project1` | `ref1` | `Fresh timestamp` | `T2` | Once the RECLAIM_ lock has successfully been inserted, the old lock is then updated and now belongs to the new process. If another process now attempts to reclaim the original lock, it will find a fresh timestamp and new token, indicating the lock is not stale and cannot be reclaimed.

Note:
If a lock belonging to a frozen process is successfully reclaimed but the
original owner resumes (i.e. after a long gc process), then its calling
process may be unaware that it no longer holds the lock.

Sources:
* [Distributed locking](https://hazelcast.com/blog/long-live-distributed-locks/)
* [Spindle, a distributed locking system built for Cloud Spanner in Go](https://github.com/flowerinthenight/spindle)