Locks

Each entry of the locks table consists of a:

Name	Description	Type
project	Project name	String
ref	Ref path	String
heartbeat	Timestamp updated every few seconds to identify staleness	Timestamp
token	Timestamp used to identify the lock version	Timestamp
owner	Gerrit instance identifier	String

The primary key is a composite of (project, ref).

Design

When the global-refdb attempts to acquire a lock on a given entry, we first try to insert the lock into the locks table.

If the insertion succeeds:

• Set the Lock object's token value to match the token in the database
  • The token is a unique identifier for the Lock which establishes versioning.
• Start a new process to update the heartbeat value every few seconds
  • The heartbeat ensures freshness of the lock and is used when deciding whether or not a Lock is stale.

If the insertion fails due to a lock on that project/ref already existing:

• Check the timestamp of the existing lock.
  • If it is sufficiently fresh, pass on attempting to reclaim the lock and end here.
  • If the timestamp is stale, attempt to reclaim the lock as described below.

Reclaiming an existing stale lock:

• Check the existing token (now known as oldToken) of the lock
• In one read/write transaction:
  • Check that the token of the lock is still oldToken.
  • Attempt to insert a meta reclaim-lock into the table to mark that the old lock is being reclaimed.
  • Refresh the old lock, issuing an update with our new heartbeat, token, and owner.
  • If any of those steps fail, our information is outdated and the attempt will fail. As the read/write transaction is atomic, either every buffered mutation succeeds or all fail.
• If all succeed, the lock has been reclaimed and we set up the heartbeat and clean up (deleting the meta reclaim-lock).

Updating a heartbeat:

• Check if the tokens of the in-database lock and the Lock object match.
  • If so, update the heartbeat to the current timestamp.
  • If not, shut down the heartbeat process; the lock has been reclaimed.

When the lock is closed:

• Shut down the heartbeat process
• Check if the tokens of the in-database lock and the Lock object match.
  • If so, delete the in-database lock.
  • If not, stop - the lock has been reclaimed.

Example

Reclaim process: | Project | Ref | Heartbeat | Token | Description | --------|-----|:----------|:------|:----------- project1 | ref1 | Stale timestamp | T1 | When a new process attempts to lock project1-ref1, it finds a stale heartbeat and can then insert a reclaim lock. RECLAIM_project1_T1 | ref1 | Tx | Tx | While this reclaim lock exists, no other process can succeed in the reclaim process.

Result: | Project | Ref | Heartbeat | Token | Description | --------|-----|:----------|:------|:----------- project1 | ref1 | Fresh timestamp | T2 | Once the RECLAIM_ lock has successfully been inserted, the old lock is then updated and now belongs to the new process. If another process now attempts to reclaim the original lock, it will find a fresh timestamp and new token, indicating the lock is not stale and cannot be reclaimed.

Note: If a lock belonging to a frozen process is successfully reclaimed but the original owner resumes (i.e. after a long gc process), then its calling process may be unaware that it no longer holds the lock.

Sources:

• Distributed locking
• Spindle, a distributed locking system built for Cloud Spanner in Go