Lock reftable auto-refresh to ensure consistency

Ensure that reftable auto-refresh operations, clearing the database
cache and reloading the reftable stack are executed in an exclusive
critical section under lock. Previously, these steps were performed
without an exclusive critical section, creating a window where
concurrent threads could interfere with each other.

In a race condition, one thread might clear the cache and before it had
a chance of reloading the stack, another thread could repopulate the
cache with stale data, keeping a reference to the open BlockSource
channel to the underlying tables that are subsequently removed when the
first thread reloads the stack.

The above race condition resulted in attempts to access closed resources
and lead to ClosedChannelException errors.

As an example, consider the following scenario:

* T0 - Thread-1 is executing auto-refresh and it clears the database
       cache

* T1 - The master branch moves forward (for any reason):
         - A new refTable (`R_new`) file is created
         - An existing refTable (`R_old`) file is deleted due to
           auto-compaction.

* T3 - Thread-2 repopulates the database cache before Thread-1 has had a
       chance to reload the refTable stack.

* T4 - Thread-1 finally reloads the refTable stack, causing the closing
       of the BlockSource wrapping the removed `R_old` refTable file.

* T5 - Thread-2 attempts to read from the already-closed `R_old`
       BlockSource and the `j.n.c.ClosedChannelException` is thrown

To reproduce this problem, you can run a script created to craft this
racing condition: I1e78e175cff.

While such errors during concurrent execution might be expected and
tolerable in isolation, the situation becomes more severe when the
`RepositoryCache` is involved, as is the case with Gerrit.

The `FileReftableDatabase` instance is cached within the `Repository`
object. When a `BlockSource` is closed prematurely due to this race
condition, the dangling reference remains in memory until the cached
Repository expires, which is one hour by default.

This means that, once the race occurs, the repository may be unable to
perform any ref lookups for up to an hour, effectively causing a
repository outage.

By introducing a ReentrantLock around both operations, the refresh logic
now guarantees that concurrent readers and writers maintain a consistent
view of the reftable state, eliminating the race condition.

Verified by executing the script provided at I1e78e175cff and no
exceptions are raised anymore.

Bug: jgit-130
Change-Id: I6153528a7b2695115b670bda04d4d4228c1731e1
1 file changed
tree: febca7691a22b51619e46d0945dd97e9043d1efc
  1. .github/
  2. .mvn/
  3. .settings/
  4. Documentation/
  5. lib/
  6. org.eclipse.jgit/
  7. org.eclipse.jgit.ant/
  8. org.eclipse.jgit.ant.test/
  9. org.eclipse.jgit.archive/
  10. org.eclipse.jgit.benchmarks/
  11. org.eclipse.jgit.coverage/
  12. org.eclipse.jgit.gpg.bc/
  13. org.eclipse.jgit.gpg.bc.test/
  14. org.eclipse.jgit.http.apache/
  15. org.eclipse.jgit.http.server/
  16. org.eclipse.jgit.http.test/
  17. org.eclipse.jgit.junit/
  18. org.eclipse.jgit.junit.http/
  19. org.eclipse.jgit.junit.ssh/
  20. org.eclipse.jgit.lfs/
  21. org.eclipse.jgit.lfs.server/
  22. org.eclipse.jgit.lfs.server.test/
  23. org.eclipse.jgit.lfs.test/
  24. org.eclipse.jgit.packaging/
  25. org.eclipse.jgit.pgm/
  26. org.eclipse.jgit.pgm.test/
  27. org.eclipse.jgit.ssh.apache/
  28. org.eclipse.jgit.ssh.apache.agent/
  29. org.eclipse.jgit.ssh.apache.test/
  30. org.eclipse.jgit.ssh.jsch/
  31. org.eclipse.jgit.ssh.jsch.test/
  32. org.eclipse.jgit.test/
  33. org.eclipse.jgit.ui/
  34. tools/
  35. .bazelrc
  36. .bazelversion
  37. .gitattributes
  38. .gitignore
  39. .mailmap
  40. BUILD
  41. CODE_OF_CONDUCT.md
  42. CONTRIBUTING.md
  43. LICENSE
  44. MODULE.bazel
  45. pom.xml
  46. README.md
  47. SECURITY.md
  48. WORKSPACE
README.md

Java Git

An implementation of the Git version control system in pure Java.

This project is licensed under the EDL (Eclipse Distribution License).

JGit can be imported straight into Eclipse and built and tested from there. It can be built from the command line using Maven or Bazel. The CI builds use Maven and run on Jenkins.

  • org.eclipse.jgit

    A pure Java library capable of being run standalone, with no additional support libraries. It provides classes to read and write a Git repository and operate on a working directory.

    All portions of JGit are covered by the EDL. Absolutely no GPL, LGPL or EPL contributions are accepted within this package.

  • org.eclipse.jgit.ant

    Ant tasks based on JGit.

  • org.eclipse.jgit.archive

    Support for exporting to various archive formats (zip etc).

  • org.eclipse.jgit.http.apache

    Apache httpclient support.

  • org.eclipse.jgit.http.server

    Server for the smart and dumb Git HTTP protocol.

  • org.eclipse.jgit.lfs

    Support for LFS (Large File Storage).

  • org.eclipse.jgit.lfs.server

    Basic LFS server support.

  • org.eclipse.jgit.packaging

    Production of Eclipse features and p2 repository for JGit. See the JGit Wiki on why and how to use this module.

  • org.eclipse.jgit.pgm

    Command-line interface Git commands implemented using JGit (“pgm” stands for program).

  • org.eclipse.jgit.ssh.apache

    Client support for the SSH protocol based on Apache Mina sshd.

  • org.eclipse.jgit.ssh.apache.agent

    Optional support for SSH agents for org.eclipse.jgit.ssh.apache.

  • org.eclipse.jgit.ui

    Simple UI for displaying git log.

Tests

  • org.eclipse.jgit.junit, org.eclipse.jgit.junit.http, org.eclipse.jgit.junit.ssh: Helpers for unit testing
  • org.eclipse.jgit.ant.test: Unit tests for org.eclipse.jgit.ant
  • org.eclipse.jgit.http.test: Unit tests for org.eclipse.jgit.http.server
  • org.eclipse.jgit.lfs.server.test: Unit tests for org.eclipse.jgit.lfs.server
  • org.eclipse.jgit.lfs.test: Unit tests for org.eclipse.jgit.lfs
  • org.eclipse.jgit.pgm.test: Unit tests for org.eclipse.jgit.pgm
  • org.eclipse.jgit.ssh.apache.test: Unit tests for org.eclipse.jgit.ssh.apache
  • org.eclipse.jgit.test: Unit tests for org.eclipse.jgit

Warnings/Caveats

  • Native symbolic links are supported, provided the file system supports them. For Windows you must use a non-administrator account and have the SeCreateSymbolicLinkPrivilege.

  • Only the timestamp of the index is used by JGit if the index is dirty.

  • JGit 6.0 and newer requires at least Java 11. Older versions require at least Java 1.8.

  • CRLF conversion is performed depending on the core.autocrlf setting, however Git for Windows by default stores that setting during installation in the “system wide” configuration file. If Git is not installed, use the global or repository configuration for the core.autocrlf setting.

  • The system wide configuration file is located relative to where C Git is installed. Make sure Git can be found via the PATH environment variable. When installing Git for Windows check the “Run Git from the Windows Command Prompt” option. There are other options like Eclipse settings that can be used for pointing out where C Git is installed. Modifying PATH is the recommended option if C Git is installed.

  • We try to use the same notation of $HOME as C Git does. On Windows this is often not the same value as the user.home system property.

Features

  • org.eclipse.jgit

    • Read loose and packed commits, trees, blobs, including deltafied objects.

    • Read objects from shared repositories

    • Write loose commits, trees, blobs.

    • Write blobs from local files or Java InputStreams.

    • Read blobs as Java InputStreams.

    • Copy trees to local directory, or local directory to a tree.

    • Lazily loads objects as necessary.

    • Read and write .git/config files.

    • Create a new repository.

    • Read and write refs, including walking through symrefs.

    • Read, update and write the Git index.

    • Checkout in dirty working directory if trivial.

    • Walk the history from a given set of commits looking for commits introducing changes in files under a specified path.

    • Object transport

      Fetch via ssh, git, http, Amazon S3 and bundles. Push via ssh, git, http, and Amazon S3. JGit does not yet deltify the pushed packs so they may be a lot larger than C Git packs.

    • Garbage collection

    • Merge

    • Rebase

    • And much more

  • org.eclipse.jgit.pgm

    • Assorted set of command line utilities. Mostly for ad-hoc testing of jgit log, glog, fetch etc.
  • org.eclipse.jgit.ant

    • Ant tasks
  • org.eclipse.jgit.archive

    • Support for Zip/Tar and other formats
  • org.eclipse.http

    • HTTP client and server support

Missing Features

There are some missing features:

  • signing push
  • shallow and partial cloning
  • support for remote helpers
  • support for credential helpers
  • support for multiple working trees (git-worktree)
  • using external diff tools
  • support for HTTPS client certificates
  • SHA-256 object IDs
  • git protocol V2 (client side): packfile-uris
  • multi-pack index
  • split index

Support

Post questions, comments or discussions to the jgit-dev@eclipse.org mailing list. You need to be subscribed to post. File bugs and enhancement requests in Bugzilla.

Contributing

See the EGit Contributor Guide.

About Git

More information about Git, its repository format, and the canonical C based implementation can be obtained from the Git website.