Introduce lib that contains Postgres based persistent cache impl
The following change introduces cache-postgres.jar library that contains
Postgres based persistence implementation for Gerrit caches.
It is heavily based on existing Gerrit's H2CacheImpl and could/should
be further optimized in the follow up changes.
One builds it by calling:
bazel build cache-postgres
Installation:
1. create Postgres DB e.g. gerrit_caches
2. copy 'bazel-genfiles/cache-postgres.jar' to GERRIT_SITE/lib dir
3. modify GERRIT_SITE/etc/gerrit.config with the following changes
a) inform Gerrit that main library's module should be loaded
gerrit.installModule
"com.googlesource.gerrit.modules.cache.pg.CacheModule"
b) specify connection to Postgres DB
cache.url
"jdbc:postgresql://localhost:5432/gerrit_caches?user=gerrit&password=gerrit"
4. restart Gerrit
5. remaining configuration parameters are mentioned in README.md
NOTES:
1. current implementation doesn't contain migration step and
existing data will be wiped out.
2. it uses INSERT INTO ... ON CONFLICT DO UPDATE ... upsert syntax [1]
therefore it is Postgres 9.5+ compatible.
[1] https://stackoverflow.com/questions/17267417/how-to-upsert-merge-insert-on-duplicate-update-in-postgresql
Depends-On: I06ab562b9feed66391084aea249b7744217bdb84
Depends-On: I7562b210fad4c5f6dc67887f627cf76815a378cb
Change-Id: I33fa1cd3dba1c1b246aecaa05a052f7348ddd64a
Signed-off-by: Jacek Centkowski <jcentkowski@collab.net>
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..b4fb4c2
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,5 @@
+.classpath
+.project
+.settings
+/eclipse-out
+/bazel-*
diff --git a/BUILD b/BUILD
new file mode 100644
index 0000000..6f34b87
--- /dev/null
+++ b/BUILD
@@ -0,0 +1,36 @@
+load("//tools/bzl:junit.bzl", "junit_tests")
+load(
+ "//tools/bzl:plugin.bzl",
+ "gerrit_plugin",
+ "PLUGIN_DEPS",
+ "PLUGIN_TEST_DEPS",
+)
+
+gerrit_plugin(
+ name = "cache-postgres",
+ srcs = glob(["src/main/java/**/*.java"]),
+ manifest_entries = [
+ "Gerrit-PluginName: cache-postgres",
+ ],
+ resources = glob(["src/main/resources/**/*"]),
+)
+
+junit_tests(
+ name = "cache_postgres_tests",
+ testonly = 1,
+ srcs = glob(["src/test/java/**/*.java"]),
+ tags = ["cache-postgres"],
+ deps = [
+ ":cache-postgres__plugin_test_deps",
+ ],
+)
+
+java_library(
+ name = "cache-postgres__plugin_test_deps",
+ testonly = 1,
+ visibility = ["//visibility:public"],
+ exports = PLUGIN_DEPS + PLUGIN_TEST_DEPS + [
+ ":cache-postgres__plugin",
+ "@mockito//jar",
+ ],
+)
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..0a3788b
--- /dev/null
+++ b/README.md
@@ -0,0 +1,65 @@
+# Gerrit Postgres based persistent cache
+
+Gerrit lib module to swap existing persistent cache implementation
+(that is H2 based) with implementation that stores data in Postgres.
+Note that is uses [INSERT INTO ... ON CONFLICT DO UPDATE ...](https://stackoverflow.com/questions/17267417/how-to-upsert-merge-insert-on-duplicate-update-in-postgresql)
+upsert syntax therefore it is Postgres *9.5+* compatible.
+
+## How to build
+
+Build this module similarly to standalone build for any other,
+bazel based Gerrit plugin:
+
+- Clone the cache-postgres source tree
+- Run ```bazel build cache-postgres```
+- The ```cache-postgres.jar``` module is generated under ```/bazel-genfiles/cache-postgres.jar```
+
+## How install
+
+Copy ```cache-postgres.jar``` library to Gerrit ```/lib``` and add the following
+two extra settings to ```gerrit.config```:
+
+```
+[gerrit]
+ installModule = com.googlesource.gerrit.modules.cache.pg.CacheModule
+
+[cache]
+ url = jdbc:postgresql://localhost:5432/gerrit_caches?user=gerrit&password=gerrit
+```
+
+## Core Gerrit settings: section `cache`
+
+cache.url
+: URI that specifies connection to existing DB (including both
+username and passwword).
+
+cache.poolLimit
+: Maximum number of open database connections. If the server needs
+more than this number, request processing threads will wait up
+to `cache.poolMaxWait` seconds for a connection to be released before
+they abort with an exception.
+Default value is taken from `database.poolLimit`.
+
+cache.poolMinIdle
+: Minimum number of connections to keep idle in the pool.
+Default is `4`.
+
+cache.poolMaxIdle
+: Maximum number of connections to keep idle in the pool. If there
+are more idle connections, connections will be closed instead of
+being returned back to the pool.
+Default is min(`cache.poolLimit`, `16`).
+
+cache.poolMaxWait
+Maximum amount of time a request processing thread will wait to
+acquire a database connection from the pool. If no connection is
+released within this time period, the processing thread will abort
+its current operations and return an error to the client.
+Values should use common unit suffixes to express their setting:
+* ms, milliseconds
+* s, sec, second, seconds
+* m, min, minute, minutes
+* h, hr, hour, hours
+
+If a unit suffix is not specified, `milliseconds` is assumed.
+Default is `30 seconds`.
diff --git a/WORKSPACE b/WORKSPACE
new file mode 100644
index 0000000..5c5e6c0
--- /dev/null
+++ b/WORKSPACE
@@ -0,0 +1,30 @@
+workspace(name = "cache_postgres")
+
+load("//:bazlets.bzl", "load_bazlets")
+
+load_bazlets(
+ commit = "03e26065d0b14cdc1e05dba2403445ff6506a378",
+ # local_path = "/home/<user>/projects/bazlets",
+)
+
+#Snapshot Plugin API
+load(
+ "@com_googlesource_gerrit_bazlets//:gerrit_api_maven_local.bzl",
+ "gerrit_api_maven_local",
+)
+
+# Load snapshot Plugin API
+gerrit_api_maven_local()
+
+# Release Plugin API
+#load(
+# "@com_googlesource_gerrit_bazlets//:gerrit_api.bzl",
+# "gerrit_api",
+#)
+
+# Load release Plugin API
+#gerrit_api()
+
+load("//:external_plugin_deps.bzl", "external_plugin_deps")
+
+external_plugin_deps()
diff --git a/bazlets.bzl b/bazlets.bzl
new file mode 100644
index 0000000..e14e488
--- /dev/null
+++ b/bazlets.bzl
@@ -0,0 +1,17 @@
+NAME = "com_googlesource_gerrit_bazlets"
+
+def load_bazlets(
+ commit,
+ local_path = None
+ ):
+ if not local_path:
+ native.git_repository(
+ name = NAME,
+ remote = "https://gerrit.googlesource.com/bazlets",
+ commit = commit,
+ )
+ else:
+ native.local_repository(
+ name = NAME,
+ path = local_path,
+ )
diff --git a/external_plugin_deps.bzl b/external_plugin_deps.bzl
new file mode 100644
index 0000000..135e5f6
--- /dev/null
+++ b/external_plugin_deps.bzl
@@ -0,0 +1,24 @@
+load("//tools/bzl:maven_jar.bzl", "maven_jar")
+
+def external_plugin_deps():
+ maven_jar(
+ name = "mockito",
+ artifact = "org.mockito:mockito-core:2.5.0",
+ sha1 = "be28d46a52c7f2563580adeca350145e9ce916f8",
+ deps = [
+ "@byte_buddy//jar",
+ "@objenesis//jar",
+ ],
+ )
+
+ maven_jar(
+ name = "byte_buddy",
+ artifact = "net.bytebuddy:byte-buddy:1.5.12",
+ sha1 = "b1ba1d15f102b36ed43b826488114678d6d413da",
+ )
+
+ maven_jar(
+ name = "objenesis",
+ artifact = "org.objenesis:objenesis:2.4",
+ sha1 = "2916b6c96b50c5b3ec4452ed99401db745aabb27",
+ )
diff --git a/src/main/java/com/googlesource/gerrit/modules/cache/pg/CacheModule.java b/src/main/java/com/googlesource/gerrit/modules/cache/pg/CacheModule.java
new file mode 100644
index 0000000..9263e9c
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/modules/cache/pg/CacheModule.java
@@ -0,0 +1,28 @@
+// Copyright (C) 2018 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.googlesource.gerrit.modules.cache.pg;
+
+import com.google.gerrit.lifecycle.LifecycleModule;
+import com.google.gerrit.server.ModuleImpl;
+import com.google.gerrit.server.cache.PersistentCacheFactory;
+
+@ModuleImpl(name = com.google.gerrit.server.cache.CacheModule.PERSISTENT_MODULE)
+public class CacheModule extends LifecycleModule {
+ @Override
+ protected void configure() {
+ bind(PersistentCacheFactory.class).to(PgCacheFactory.class);
+ listener().to(PgCacheFactory.class);
+ }
+}
diff --git a/src/main/java/com/googlesource/gerrit/modules/cache/pg/KeyType.java b/src/main/java/com/googlesource/gerrit/modules/cache/pg/KeyType.java
new file mode 100644
index 0000000..d624f5b
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/modules/cache/pg/KeyType.java
@@ -0,0 +1,94 @@
+// Copyright (C) 2018 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.googlesource.gerrit.modules.cache.pg;
+
+import com.google.common.hash.Funnel;
+import com.google.common.hash.Funnels;
+import com.google.common.hash.PrimitiveSink;
+import com.google.inject.TypeLiteral;
+import java.io.IOException;
+import java.io.ObjectOutputStream;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Types;
+
+class KeyType<K> {
+ String columnType() {
+ return "BYTEA"; // H2 uses OTHER
+ }
+
+ @SuppressWarnings("unchecked")
+ K get(ResultSet rs, int col) throws SQLException {
+ return (K) PgSqlHandle.deserialize(rs.getBytes(col)); // H2 doesn't require deserialization
+ }
+
+ void set(PreparedStatement ps, int col, K value) throws SQLException {
+ ps.setObject(
+ col,
+ PgSqlHandle.serialize(value),
+ Types.BINARY); // H2 uses JAVA_OBJECT and doesn't require serialization
+ }
+
+ Funnel<K> funnel() {
+ return new Funnel<K>() {
+ private static final long serialVersionUID = 1L;
+
+ @Override
+ public void funnel(K from, PrimitiveSink into) {
+ try (ObjectOutputStream ser = new ObjectOutputStream(new SinkOutputStream(into))) {
+ ser.writeObject(from);
+ ser.flush();
+ } catch (IOException err) {
+ throw new RuntimeException("Cannot hash as Serializable", err);
+ }
+ }
+ };
+ }
+
+ @SuppressWarnings("unchecked")
+ static <K> KeyType<K> create(TypeLiteral<K> type) {
+ if (type.getRawType() == String.class) {
+ return (KeyType<K>) STRING;
+ }
+ return (KeyType<K>) OTHER;
+ }
+
+ static final KeyType<?> OTHER = new KeyType<>();
+ static final KeyType<String> STRING =
+ new KeyType<String>() {
+ @Override
+ String columnType() {
+ return "text";
+ }
+
+ @Override
+ String get(ResultSet rs, int col) throws SQLException {
+ return rs.getString(col);
+ }
+
+ @Override
+ void set(PreparedStatement ps, int col, String value) throws SQLException {
+ ps.setString(col, value);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ Funnel<String> funnel() {
+ Funnel<?> s = Funnels.unencodedCharsFunnel();
+ return (Funnel<String>) s;
+ }
+ };
+}
diff --git a/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgCacheBindingProxy.java b/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgCacheBindingProxy.java
new file mode 100644
index 0000000..e648fbc
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgCacheBindingProxy.java
@@ -0,0 +1,111 @@
+// Copyright (C) 2018 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.googlesource.gerrit.modules.cache.pg;
+
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.Weigher;
+import com.google.gerrit.server.cache.CacheBinding;
+import com.google.inject.TypeLiteral;
+import java.util.concurrent.TimeUnit;
+
+class PgCacheBindingProxy<K, V> implements CacheBinding<K, V> {
+ private static final String MSG_NOT_SUPPORTED =
+ "This is a read-only wrapper. Modifications are not supported";
+
+ private final CacheBinding<K, V> source;
+
+ PgCacheBindingProxy(CacheBinding<K, V> source) {
+ this.source = source;
+ }
+
+ @Override
+ public Long expireAfterWrite(TimeUnit unit) {
+ return source.expireAfterWrite(unit);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public Weigher<K, V> weigher() {
+ Weigher<K, V> weigher = source.weigher();
+ if (weigher == null) {
+ return null;
+ }
+
+ // introduce weigher that performs calculations
+ // on value that is being stored not on ValueHolder
+ return (Weigher<K, V>)
+ new Weigher<K, ValueHolder<V>>() {
+ @Override
+ public int weigh(K key, ValueHolder<V> value) {
+ return weigher.weigh(key, value.value);
+ }
+ };
+ }
+
+ @Override
+ public String name() {
+ return source.name();
+ }
+
+ @Override
+ public TypeLiteral<K> keyType() {
+ return source.keyType();
+ }
+
+ @Override
+ public TypeLiteral<V> valueType() {
+ return source.valueType();
+ }
+
+ @Override
+ public long maximumWeight() {
+ return source.maximumWeight();
+ }
+
+ @Override
+ public long diskLimit() {
+ return source.diskLimit();
+ }
+
+ @Override
+ public CacheLoader<K, V> loader() {
+ return source.loader();
+ }
+
+ @Override
+ public CacheBinding<K, V> maximumWeight(long weight) {
+ throw new RuntimeException(MSG_NOT_SUPPORTED);
+ }
+
+ @Override
+ public CacheBinding<K, V> diskLimit(long limit) {
+ throw new RuntimeException(MSG_NOT_SUPPORTED);
+ }
+
+ @Override
+ public CacheBinding<K, V> expireAfterWrite(long duration, TimeUnit durationUnits) {
+ throw new RuntimeException(MSG_NOT_SUPPORTED);
+ }
+
+ @Override
+ public CacheBinding<K, V> loader(Class<? extends CacheLoader<K, V>> clazz) {
+ throw new RuntimeException(MSG_NOT_SUPPORTED);
+ }
+
+ @Override
+ public CacheBinding<K, V> weigher(Class<? extends Weigher<K, V>> clazz) {
+ throw new RuntimeException(MSG_NOT_SUPPORTED);
+ }
+}
diff --git a/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgCacheFactory.java b/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgCacheFactory.java
new file mode 100644
index 0000000..881698c
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgCacheFactory.java
@@ -0,0 +1,180 @@
+// Copyright (C) 2018 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.googlesource.gerrit.modules.cache.pg;
+
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import com.google.gerrit.extensions.events.LifecycleListener;
+import com.google.gerrit.extensions.registration.DynamicMap;
+import com.google.gerrit.server.cache.CacheBinding;
+import com.google.gerrit.server.cache.MemoryCacheFactory;
+import com.google.gerrit.server.cache.PersistentCacheFactory;
+import com.google.gerrit.server.config.GerritServerConfig;
+import com.google.gerrit.server.plugins.Plugin;
+import com.google.inject.Inject;
+import com.google.inject.Provider;
+import com.google.inject.Singleton;
+import com.google.inject.TypeLiteral;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.eclipse.jgit.lib.Config;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Singleton
+public class PgCacheFactory implements PersistentCacheFactory, LifecycleListener {
+ private static final Logger log = LoggerFactory.getLogger(PgCacheFactory.class);
+
+ private final PgSqlSource ds;
+ private final Config cfg;
+ private final MemoryCacheFactory memFactory;
+ private final List<PgCacheImpl<?, ?>> caches;
+ private final DynamicMap<Cache<?, ?>> cacheMap;
+ private final ExecutorService executor;
+ private final ScheduledExecutorService cleanup;
+
+ @Inject
+ PgCacheFactory(
+ PgSqlSource ds,
+ @GerritServerConfig Config cfg,
+ MemoryCacheFactory memFactory,
+ DynamicMap<Cache<?, ?>> cacheMap) {
+ this.ds = ds;
+ this.cfg = cfg;
+ this.memFactory = memFactory;
+ this.caches = new LinkedList<>();
+ this.cacheMap = cacheMap;
+
+ executor =
+ Executors.newFixedThreadPool(
+ 1, new ThreadFactoryBuilder().setNameFormat("PgCache-Store-%d").build());
+ cleanup =
+ Executors.newScheduledThreadPool(
+ 1,
+ new ThreadFactoryBuilder().setNameFormat("PgCache-Prune-%d").setDaemon(true).build());
+ }
+
+ @Override
+ public void start() {
+ if (executor != null) {
+ for (PgCacheImpl<?, ?> cache : caches) {
+ executor.execute(cache::start);
+ @SuppressWarnings("unused")
+ Future<?> possiblyIgnoredError =
+ cleanup.schedule(() -> cache.prune(cleanup), 30, TimeUnit.SECONDS);
+ }
+ }
+ }
+
+ @Override
+ public void stop() {
+ if (executor != null) {
+ try {
+ cleanup.shutdownNow();
+
+ List<Runnable> pending = executor.shutdownNow();
+ if (executor.awaitTermination(15, TimeUnit.MINUTES)) {
+ if (pending != null && !pending.isEmpty()) {
+ log.info("Finishing {} disk cache updates", pending.size());
+ for (Runnable update : pending) {
+ update.run();
+ }
+ }
+ } else {
+ log.info("Timeout waiting for disk cache to close");
+ }
+ } catch (InterruptedException e) {
+ log.warn("Interrupted waiting for disk cache to shutdown");
+ }
+ }
+ synchronized (caches) {
+ for (PgCacheImpl<?, ?> cache : caches) {
+ cache.stop();
+ }
+ }
+ }
+
+ @SuppressWarnings({"unchecked"})
+ @Override
+ public <K, V> Cache<K, V> build(CacheBinding<K, V> in) {
+ long limit = cfg.getLong("cache", in.name(), "diskLimit", in.diskLimit());
+
+ if (limit <= 0) {
+ return memFactory.build(in);
+ }
+
+ PgCacheBindingProxy<K, V> def = new PgCacheBindingProxy<>(in);
+ PgSqlStore<K, V> store =
+ newSqlStore(def.name(), def.keyType(), limit, def.expireAfterWrite(TimeUnit.SECONDS));
+ PgCacheImpl<K, V> cache =
+ new PgCacheImpl<>(
+ def.name(),
+ executor,
+ store,
+ def.keyType(),
+ (Cache<K, ValueHolder<V>>) memFactory.build(def));
+ synchronized (caches) {
+ caches.add(cache);
+ }
+ return cache;
+ }
+
+ @SuppressWarnings({"unchecked"})
+ @Override
+ public <K, V> LoadingCache<K, V> build(CacheBinding<K, V> in, CacheLoader<K, V> loader) {
+ long limit = cfg.getLong("cache", in.name(), "diskLimit", in.diskLimit());
+
+ if (limit <= 0) {
+ return memFactory.build(in, loader);
+ }
+
+ PgCacheBindingProxy<K, V> def = new PgCacheBindingProxy<>(in);
+ PgSqlStore<K, V> store =
+ newSqlStore(def.name(), def.keyType(), limit, def.expireAfterWrite(TimeUnit.SECONDS));
+ Cache<K, ValueHolder<V>> mem =
+ (Cache<K, ValueHolder<V>>)
+ memFactory.build(def, (CacheLoader<K, V>) new PgCacheLoader<>(executor, store, loader));
+ PgCacheImpl<K, V> cache = new PgCacheImpl<>(def.name(), executor, store, def.keyType(), mem);
+ caches.add(cache);
+ return cache;
+ }
+
+ @Override
+ public void onStop(Plugin plugin) {
+ synchronized (caches) {
+ for (Map.Entry<String, Provider<Cache<?, ?>>> entry :
+ cacheMap.byPlugin(plugin.getName()).entrySet()) {
+ Cache<?, ?> cache = entry.getValue().get();
+ if (caches.remove(cache)) {
+ ((PgCacheImpl<?, ?>) cache).stop();
+ }
+ }
+ }
+ }
+
+ private <V, K> PgSqlStore<K, V> newSqlStore(
+ String name, TypeLiteral<K> keyType, long maxSize, Long expireAfterWrite) {
+ return new PgSqlStore<>(
+ ds, name, keyType, maxSize, expireAfterWrite == null ? 0 : expireAfterWrite.longValue());
+ }
+}
diff --git a/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgCacheImpl.java b/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgCacheImpl.java
new file mode 100644
index 0000000..08aa480
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgCacheImpl.java
@@ -0,0 +1,182 @@
+// Copyright (C) 2018 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.googlesource.gerrit.modules.cache.pg;
+
+import com.google.common.cache.AbstractLoadingCache;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheStats;
+import com.google.common.cache.LoadingCache;
+import com.google.gerrit.common.TimeUtil;
+import com.google.gerrit.server.cache.PersistentCache;
+import com.google.inject.TypeLiteral;
+import java.time.LocalDateTime;
+import java.time.OffsetDateTime;
+import java.time.temporal.ChronoUnit;
+import java.time.temporal.TemporalAdjusters;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executor;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class PgCacheImpl<K, V> extends AbstractLoadingCache<K, V> implements PersistentCache {
+ private static final Logger log = LoggerFactory.getLogger(PgCacheImpl.class);
+
+ private final String name;
+ private final Executor executor;
+ private final PgSqlStore<K, V> store;
+ private final TypeLiteral<K> keyType;
+ private final Cache<K, ValueHolder<V>> mem;
+
+ public PgCacheImpl(
+ String name,
+ Executor executor,
+ PgSqlStore<K, V> store,
+ TypeLiteral<K> keyType,
+ Cache<K, ValueHolder<V>> mem) {
+ this.name = name;
+ this.executor = executor;
+ this.store = store;
+ this.keyType = keyType;
+ this.mem = mem;
+ }
+
+ @Override
+ public V get(K key) throws ExecutionException {
+ if (mem instanceof LoadingCache) {
+ return ((LoadingCache<K, ValueHolder<V>>) mem).get(key).value;
+ }
+
+ log.error("Memory cache for persistent backend {} should be of LoadingCache type", name);
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public V getIfPresent(Object objKey) {
+ if (!keyType.getRawType().isInstance(objKey)) {
+ log.warn("Invalid object key type [{}] was provided for cache {}", objKey, name);
+ return null;
+ }
+
+ @SuppressWarnings("unchecked")
+ K key = (K) objKey;
+
+ ValueHolder<V> h = mem.getIfPresent(key);
+ if (h != null) {
+ return h.value;
+ }
+
+ if (store.mightContain(key)) {
+ h = store.getIfPresent(key);
+ if (h != null) {
+ mem.put(key, h);
+ return h.value;
+ }
+ }
+ return null;
+ }
+
+ @Override
+ public V get(K key, Callable<? extends V> valueLoader) throws ExecutionException {
+ return mem.get(
+ key,
+ () -> {
+ if (store.mightContain(key)) {
+ ValueHolder<V> h = store.getIfPresent(key);
+ if (h != null) {
+ return h;
+ }
+ }
+
+ ValueHolder<V> h = new ValueHolder<>(valueLoader.call());
+ h.created = TimeUtil.nowMs();
+ executor.execute(() -> store.put(key, h));
+ return h;
+ })
+ .value;
+ }
+
+ @Override
+ public void put(K key, V val) {
+ final ValueHolder<V> h = new ValueHolder<>(val);
+ h.created = TimeUtil.nowMs();
+ mem.put(key, h);
+ executor.execute(() -> store.put(key, h));
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void invalidate(Object key) {
+ if (keyType.getRawType().isInstance(key) && store.mightContain((K) key)) {
+ executor.execute(() -> store.invalidate((K) key));
+ }
+ mem.invalidate(key);
+ }
+
+ @Override
+ public void invalidateAll() {
+ store.invalidateAll();
+ mem.invalidateAll();
+ }
+
+ @Override
+ public long size() {
+ return mem.size();
+ }
+
+ @Override
+ public CacheStats stats() {
+ return mem.stats();
+ }
+
+ @Override
+ public DiskStats diskStats() {
+ return store.diskStats();
+ }
+
+ void start() {
+ store.open();
+ }
+
+ void stop() {
+ for (Map.Entry<K, ValueHolder<V>> e : mem.asMap().entrySet()) {
+ ValueHolder<V> h = e.getValue();
+ if (!h.clean) {
+ store.put(e.getKey(), h);
+ }
+ }
+ store.close();
+ }
+
+ void prune(ScheduledExecutorService service) {
+ store.prune(mem);
+
+ long delay =
+ LocalDateTime.now()
+ .with(TemporalAdjusters.firstDayOfMonth())
+ .withHour(01)
+ .truncatedTo(ChronoUnit.HOURS)
+ .toInstant(OffsetDateTime.now().getOffset())
+ .minusMillis(TimeUtil.nowMs())
+ .toEpochMilli();
+ @SuppressWarnings("unused")
+ Future<?> possiblyIgnoredError =
+ service.schedule(() -> prune(service), delay, TimeUnit.MILLISECONDS);
+ }
+}
diff --git a/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgCacheLoader.java b/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgCacheLoader.java
new file mode 100644
index 0000000..7e47d6a
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgCacheLoader.java
@@ -0,0 +1,46 @@
+// Copyright (C) 2018 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.googlesource.gerrit.modules.cache.pg;
+
+import com.google.common.cache.CacheLoader;
+import com.google.gerrit.common.TimeUtil;
+import java.util.concurrent.Executor;
+
+public class PgCacheLoader<K, V> extends CacheLoader<K, ValueHolder<V>> {
+ private final Executor executor;
+ private final PgSqlStore<K, V> store;
+ private final CacheLoader<K, V> loader;
+
+ PgCacheLoader(Executor executor, PgSqlStore<K, V> store, CacheLoader<K, V> loader) {
+ this.executor = executor;
+ this.store = store;
+ this.loader = loader;
+ }
+
+ @Override
+ public ValueHolder<V> load(K key) throws Exception {
+ if (store.mightContain(key)) {
+ ValueHolder<V> h = store.getIfPresent(key);
+ if (h != null) {
+ return h;
+ }
+ }
+
+ final ValueHolder<V> h = new ValueHolder<>(loader.load(key));
+ h.created = TimeUtil.nowMs();
+ executor.execute(() -> store.put(key, h));
+ return h;
+ }
+}
diff --git a/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgSqlHandle.java b/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgSqlHandle.java
new file mode 100644
index 0000000..fa78287
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgSqlHandle.java
@@ -0,0 +1,143 @@
+// Copyright (C) 2018 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.googlesource.gerrit.modules.cache.pg;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.ObjectStreamClass;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.sql.Statement;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class PgSqlHandle {
+ private static final Logger log = LoggerFactory.getLogger(PgSqlHandle.class);
+
+ Connection conn;
+ PreparedStatement get;
+ PreparedStatement put;
+ PreparedStatement touch;
+ PreparedStatement invalidate;
+
+ private final String name;
+
+ PgSqlHandle(PgSqlSource store, String name, KeyType<?> type) throws SQLException {
+ this.name = name;
+ this.conn = store.getConnection();
+ try (Statement stmt = conn.createStatement()) {
+ stmt.addBatch(
+ "CREATE TABLE IF NOT EXISTS \"data_"
+ + this.name
+ + "\" (k "
+ + type.columnType()
+ + " NOT NULL PRIMARY KEY" // H2 uses HASH however it is discouraged in Postgres doc
+ + ",v BYTEA NOT NULL"
+ + ",created TIMESTAMP NOT NULL"
+ + ",accessed TIMESTAMP NOT NULL"
+ + ",space BIGINT"
+ + ")");
+ // Postgres has no 'computed columns' concept, however one can achieve that
+ // with trigger
+ stmt.addBatch(
+ "CREATE OR REPLACE FUNCTION compute_space()\n"
+ + "RETURNS trigger\n"
+ + "LANGUAGE plpgsql\n"
+ + "SECURITY DEFINER\n"
+ + "AS $BODY$\n"
+ + "BEGIN\n"
+ + " NEW.space = OCTET_LENGTH(NEW.k) + OCTET_LENGTH(NEW.v);\n"
+ + " RETURN NEW;\n"
+ + "END\n"
+ + "$BODY$;");
+ stmt.addBatch("DROP TRIGGER IF EXISTS computed_space ON \"data_" + this.name + "\"");
+ stmt.addBatch(
+ "CREATE TRIGGER computed_space "
+ + "BEFORE INSERT OR UPDATE "
+ + "ON \"data_"
+ + this.name
+ + "\" "
+ + "FOR EACH ROW "
+ + "EXECUTE PROCEDURE compute_space()");
+ stmt.executeBatch();
+ }
+ }
+
+ void close() {
+ get = closeStatement(get);
+ put = closeStatement(put);
+ touch = closeStatement(touch);
+ invalidate = closeStatement(invalidate);
+
+ if (conn != null) {
+ try {
+ conn.close();
+ } catch (SQLException e) {
+ log.warn("Cannot close connection to " + name, e);
+ } finally {
+ conn = null;
+ }
+ }
+ }
+
+ private PreparedStatement closeStatement(PreparedStatement ps) {
+ if (ps != null) {
+ try {
+ ps.close();
+ } catch (SQLException e) {
+ log.warn("Cannot close statement for " + name, e);
+ }
+ }
+ return null;
+ }
+
+ static Object deserialize(byte[] d) {
+ try {
+ ByteArrayInputStream in = new ByteArrayInputStream(d);
+ ClassLoader loader = Thread.currentThread().getContextClassLoader();
+ ObjectInputStream is =
+ new ObjectInputStream(in) {
+ @Override
+ protected Class<?> resolveClass(ObjectStreamClass desc)
+ throws IOException, ClassNotFoundException {
+ try {
+ return Class.forName(desc.getName(), true, loader);
+ } catch (ClassNotFoundException e) {
+ return super.resolveClass(desc);
+ }
+ }
+ };
+ return is.readObject();
+ } catch (Throwable e) {
+ log.error("Object deserialization failed", e);
+ throw new RuntimeException(e);
+ }
+ }
+
+ static byte[] serialize(Object o) {
+ try (ByteArrayOutputStream out = new ByteArrayOutputStream();
+ ObjectOutputStream os = new ObjectOutputStream(out)) {
+ os.writeObject(o);
+ return out.toByteArray();
+ } catch (Throwable e) {
+ log.error("Object serialization failed {}", o, e);
+ throw new RuntimeException(e);
+ }
+ }
+}
diff --git a/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgSqlSource.java b/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgSqlSource.java
new file mode 100644
index 0000000..fd635e8
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgSqlSource.java
@@ -0,0 +1,63 @@
+// Copyright (C) 2018 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.googlesource.gerrit.modules.cache.pg;
+
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+
+import com.google.gerrit.server.config.ConfigUtil;
+import com.google.gerrit.server.config.GerritServerConfig;
+import com.google.gerrit.server.config.ThreadSettingsConfig;
+import com.google.inject.Inject;
+import com.google.inject.Singleton;
+import java.sql.Connection;
+import java.sql.SQLException;
+import javax.sql.DataSource;
+import org.apache.commons.dbcp.BasicDataSource;
+import org.eclipse.jgit.lib.Config;
+
+@Singleton
+public class PgSqlSource {
+ private static final String CACHE = "cache";
+
+ private final DataSource ds;
+
+ @Inject
+ PgSqlSource(@GerritServerConfig Config cfg, ThreadSettingsConfig threadSettingsConfig) {
+ this.ds = createDs(cfg, threadSettingsConfig);
+ }
+
+ Connection getConnection() throws SQLException {
+ return ds.getConnection();
+ }
+
+ private static final DataSource createDs(Config cfg, ThreadSettingsConfig threadSettingsConfig) {
+ BasicDataSource ds = new BasicDataSource();
+ ds.setUrl(cfg.getString(CACHE, null, "url"));
+ int poolLimit = threadSettingsConfig.getDatabasePoolLimit();
+ ds.setMaxActive(cfg.getInt(CACHE, "poolLimit", poolLimit));
+ ds.setDriverClassName("org.postgresql.Driver");
+ ds.setMinIdle(cfg.getInt(CACHE, "poolminidle", 4));
+ ds.setMaxIdle(cfg.getInt(CACHE, "poolmaxidle", Math.min(poolLimit, 16)));
+ ds.setInitialSize(ds.getMinIdle());
+ ds.setMaxWait(
+ ConfigUtil.getTimeUnit(
+ cfg, CACHE, null, "poolmaxwait", MILLISECONDS.convert(30, SECONDS), MILLISECONDS));
+ long evictIdleTimeMs = 1000L * 60;
+ ds.setMinEvictableIdleTimeMillis(evictIdleTimeMs);
+ ds.setTimeBetweenEvictionRunsMillis(evictIdleTimeMs / 2);
+ return ds;
+ }
+}
diff --git a/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgSqlStore.java b/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgSqlStore.java
new file mode 100644
index 0000000..31ae338
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/modules/cache/pg/PgSqlStore.java
@@ -0,0 +1,375 @@
+// Copyright (C) 2018 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.googlesource.gerrit.modules.cache.pg;
+
+import com.google.common.cache.Cache;
+import com.google.common.hash.BloomFilter;
+import com.google.gerrit.common.TimeUtil;
+import com.google.gerrit.server.cache.PersistentCache.DiskStats;
+import com.google.inject.TypeLiteral;
+import java.io.InvalidClassException;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.sql.Timestamp;
+import java.sql.Types;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.atomic.AtomicLong;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class PgSqlStore<K, V> {
+ static final Logger log = LoggerFactory.getLogger(PgSqlStore.class);
+
+ private final PgSqlSource source;
+ private final String name;
+ private final KeyType<K> keyType;
+ private final long maxSize;
+ private final long expireAfterWrite;
+ private final BlockingQueue<PgSqlHandle> handles;
+ private final AtomicLong hitCount = new AtomicLong();
+ private final AtomicLong missCount = new AtomicLong();
+ private volatile BloomFilter<K> bloomFilter;
+ private int estimatedSize;
+
+ private final String qCount;
+ private final String qKeys;
+ private final String qValue;
+ private final String qTouch;
+ private final String qPut;
+ private final String qInvalidateKey;
+ private final String qInvalidateAll;
+ private final String qSum;
+ private final String qOrderByAccessed;
+ private final String qStats;
+
+ PgSqlStore(
+ PgSqlSource source,
+ String name,
+ TypeLiteral<K> keyType,
+ long maxSize,
+ long expireAfterWrite) {
+ this.source = source;
+ this.name = name;
+ this.keyType = KeyType.create(keyType);
+ this.maxSize = maxSize;
+ this.expireAfterWrite = expireAfterWrite;
+
+ int cores = Runtime.getRuntime().availableProcessors();
+ int keep = Math.min(cores, 16);
+ this.handles = new ArrayBlockingQueue<>(keep);
+
+ // initiate all query strings
+ this.qCount = "SELECT COUNT(*) FROM \"data_" + this.name + "\"";
+ this.qKeys = "SELECT k FROM \"data_" + this.name + "\"";
+ this.qValue = "SELECT v, created FROM \"data_" + this.name + "\" WHERE k=?";
+ this.qTouch = "UPDATE \"data_" + this.name + "\" SET accessed=? WHERE k=?";
+ this.qPut =
+ "INSERT INTO \"data_"
+ + this.name
+ + "\" (k, v, created, accessed) VALUES(?,?,?,?) "
+ + "ON CONFLICT (k) DO UPDATE "
+ + "SET v = EXCLUDED.v, created = EXCLUDED.created, accessed = EXCLUDED.accessed";
+ this.qInvalidateKey = "DELETE FROM \"data_" + this.name + "\" WHERE k=?";
+ this.qInvalidateAll = "DELETE FROM \"data_" + this.name + "\"";
+ this.qSum = "SELECT SUM(space) FROM \"data_" + this.name + "\"";
+ this.qOrderByAccessed =
+ "SELECT k,space,created FROM \"data_" + this.name + "\" ORDER BY accessed";
+ this.qStats = "SELECT COUNT(*),SUM(space) FROM \"data_" + this.name + "\"";
+ }
+
+ synchronized void open() {
+ if (bloomFilter == null) {
+ bloomFilter = buildBloomFilter();
+ }
+ }
+
+ void close() {
+ PgSqlHandle h;
+ while ((h = handles.poll()) != null) {
+ h.close();
+ }
+ }
+
+ boolean mightContain(K key) {
+ BloomFilter<K> b = bloomFilter;
+ if (b == null) {
+ synchronized (this) {
+ b = bloomFilter;
+ if (b == null) {
+ b = buildBloomFilter();
+ bloomFilter = b;
+ }
+ }
+ }
+ return b == null || b.mightContain(key);
+ }
+
+ private BloomFilter<K> buildBloomFilter() {
+ PgSqlHandle c = null;
+ try {
+ c = acquire();
+ try (Statement s = c.conn.createStatement()) {
+ if (estimatedSize <= 0) {
+ try (ResultSet r = s.executeQuery(qCount)) {
+ estimatedSize = r.next() ? r.getInt(1) : 0;
+ }
+ }
+
+ BloomFilter<K> b = newBloomFilter();
+ try (ResultSet r = s.executeQuery(qKeys)) {
+ while (r.next()) {
+ b.put(keyType.get(r, 1));
+ }
+ } catch (SQLException e) {
+ if (e.getCause() instanceof InvalidClassException) {
+ log.warn(
+ "Entries cached for {} "
+ + "have an incompatible class and can't be deserialized. "
+ + "Cache is flushed.",
+ name);
+ invalidateAll();
+ } else {
+ throw e;
+ }
+ }
+ return b;
+ }
+ } catch (SQLException e) {
+ log.warn("Cannot build BloomFilter for {}: {}", name, e.getMessage());
+ c = close(c);
+ return null;
+ } finally {
+ release(c);
+ }
+ }
+
+ ValueHolder<V> getIfPresent(K key) {
+ PgSqlHandle c = null;
+ try {
+ c = acquire();
+ if (c.get == null) {
+ c.get = c.conn.prepareStatement(qValue);
+ }
+ keyType.set(c.get, 1, key);
+ try (ResultSet r = c.get.executeQuery()) {
+ if (!r.next()) {
+ missCount.incrementAndGet();
+ return null;
+ }
+
+ Timestamp created = r.getTimestamp(2);
+ if (expired(created)) {
+ invalidate(key);
+ missCount.incrementAndGet();
+ return null;
+ }
+
+ @SuppressWarnings("unchecked")
+ V val = (V) PgSqlHandle.deserialize(r.getBytes(1));
+ ValueHolder<V> h = new ValueHolder<>(val);
+ h.clean = true;
+ hitCount.incrementAndGet();
+ touch(c, key);
+ return h;
+ } finally {
+ c.get.clearParameters();
+ }
+ } catch (SQLException e) {
+ log.warn("Cannot read cache {} for {}", name, key, e);
+ c = close(c);
+ return null;
+ } finally {
+ release(c);
+ }
+ }
+
+ private boolean expired(Timestamp created) {
+ if (expireAfterWrite == 0) {
+ return false;
+ }
+ long age = TimeUtil.nowMs() - created.getTime();
+ return 1000 * expireAfterWrite < age;
+ }
+
+ private void touch(PgSqlHandle c, K key) throws SQLException {
+ if (c.touch == null) {
+ c.touch = c.conn.prepareStatement(qTouch);
+ }
+ try {
+ c.touch.setTimestamp(1, TimeUtil.nowTs());
+ keyType.set(c.touch, 2, key);
+ c.touch.executeUpdate();
+ } finally {
+ c.touch.clearParameters();
+ }
+ }
+
+ void put(K key, ValueHolder<V> holder) {
+ if (holder.clean) {
+ return;
+ }
+
+ BloomFilter<K> b = bloomFilter;
+ if (b != null) {
+ b.put(key);
+ bloomFilter = b;
+ }
+
+ PgSqlHandle c = null;
+ try {
+ c = acquire();
+ if (c.put == null) {
+ c.put = c.conn.prepareStatement(qPut);
+ }
+ try {
+ keyType.set(c.put, 1, key);
+ c.put.setObject(2, PgSqlHandle.serialize(holder.value), Types.BINARY);
+ c.put.setTimestamp(3, new Timestamp(holder.created));
+ c.put.setTimestamp(4, TimeUtil.nowTs());
+ c.put.executeUpdate();
+ holder.clean = true;
+ } finally {
+ c.put.clearParameters();
+ }
+ } catch (SQLException e) {
+ log.warn("Cannot put into cache {}", name, e);
+ c = close(c);
+ } finally {
+ release(c);
+ }
+ }
+
+ void invalidate(K key) {
+ PgSqlHandle c = null;
+ try {
+ c = acquire();
+ invalidate(c, key);
+ } catch (SQLException e) {
+ log.warn("Cannot invalidate cache {}", name, e);
+ c = close(c);
+ } finally {
+ release(c);
+ }
+ }
+
+ private void invalidate(PgSqlHandle c, K key) throws SQLException {
+ if (c.invalidate == null) {
+ c.invalidate = c.conn.prepareStatement(qInvalidateKey);
+ }
+ try {
+ keyType.set(c.invalidate, 1, key);
+ c.invalidate.executeUpdate();
+ } finally {
+ c.invalidate.clearParameters();
+ }
+ }
+
+ void invalidateAll() {
+ PgSqlHandle c = null;
+ try {
+ c = acquire();
+ try (Statement s = c.conn.createStatement()) {
+ s.executeUpdate(qInvalidateAll);
+ }
+ bloomFilter = newBloomFilter();
+ } catch (SQLException e) {
+ log.warn("Cannot invalidate cache {}", name, e);
+ c = close(c);
+ } finally {
+ release(c);
+ }
+ }
+
+ void prune(Cache<K, ?> mem) {
+ PgSqlHandle c = null;
+ try {
+ c = acquire();
+ try (Statement s = c.conn.createStatement()) {
+ long used = 0;
+ try (ResultSet r = s.executeQuery(qSum)) {
+ used = r.next() ? r.getLong(1) : 0;
+ }
+ if (used <= maxSize) {
+ return;
+ }
+
+ try (ResultSet r = s.executeQuery(qOrderByAccessed)) {
+ while (maxSize < used && r.next()) {
+ K key = keyType.get(r, 1);
+ Timestamp created = r.getTimestamp(3);
+ if (mem.getIfPresent(key) != null && !expired(created)) {
+ touch(c, key);
+ } else {
+ invalidate(c, key);
+ used -= r.getLong(2);
+ }
+ }
+ }
+ }
+ } catch (SQLException e) {
+ log.warn("Cannot prune cache {}", name, e);
+ c = close(c);
+ } finally {
+ release(c);
+ }
+ }
+
+ DiskStats diskStats() {
+ long size = 0;
+ long space = 0;
+ PgSqlHandle c = null;
+ try {
+ c = acquire();
+ try (Statement s = c.conn.createStatement();
+ ResultSet r = s.executeQuery(qStats)) {
+ if (r.next()) {
+ size = r.getLong(1);
+ space = r.getLong(2);
+ }
+ }
+ } catch (SQLException e) {
+ log.warn("Cannot get DiskStats for {}", name, e);
+ c = close(c);
+ } finally {
+ release(c);
+ }
+ return new DiskStats(size, space, hitCount.get(), missCount.get());
+ }
+
+ private PgSqlHandle acquire() throws SQLException {
+ PgSqlHandle h = handles.poll();
+ return h != null ? h : new PgSqlHandle(source, name, keyType);
+ }
+
+ private void release(PgSqlHandle h) {
+ if (h != null && !handles.offer(h)) {
+ h.close();
+ }
+ }
+
+ private PgSqlHandle close(PgSqlHandle h) {
+ if (h != null) {
+ h.close();
+ }
+ return null;
+ }
+
+ private BloomFilter<K> newBloomFilter() {
+ int cnt = Math.max(64 * 1024, 2 * estimatedSize);
+ return BloomFilter.create(keyType.funnel(), cnt);
+ }
+}
diff --git a/src/main/java/com/googlesource/gerrit/modules/cache/pg/SinkOutputStream.java b/src/main/java/com/googlesource/gerrit/modules/cache/pg/SinkOutputStream.java
new file mode 100644
index 0000000..6e1d021
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/modules/cache/pg/SinkOutputStream.java
@@ -0,0 +1,36 @@
+// Copyright (C) 2018 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.googlesource.gerrit.modules.cache.pg;
+
+import com.google.common.hash.PrimitiveSink;
+import java.io.OutputStream;
+
+class SinkOutputStream extends OutputStream {
+ private final PrimitiveSink sink;
+
+ SinkOutputStream(PrimitiveSink sink) {
+ this.sink = sink;
+ }
+
+ @Override
+ public void write(int b) {
+ sink.putByte((byte) b);
+ }
+
+ @Override
+ public void write(byte[] b, int p, int n) {
+ sink.putBytes(b, p, n);
+ }
+}
diff --git a/src/main/java/com/googlesource/gerrit/modules/cache/pg/ValueHolder.java b/src/main/java/com/googlesource/gerrit/modules/cache/pg/ValueHolder.java
new file mode 100644
index 0000000..bf0c71e
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/modules/cache/pg/ValueHolder.java
@@ -0,0 +1,25 @@
+// Copyright (C) 2018 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.googlesource.gerrit.modules.cache.pg;
+
+class ValueHolder<V> {
+ final V value;
+ long created;
+ volatile boolean clean;
+
+ ValueHolder(V value) {
+ this.value = value;
+ }
+}
diff --git a/tools/bazel.rc b/tools/bazel.rc
new file mode 100644
index 0000000..4ed16cf
--- /dev/null
+++ b/tools/bazel.rc
@@ -0,0 +1,2 @@
+build --workspace_status_command=./tools/workspace-status.sh
+test --build_tests_only
diff --git a/tools/bzl/BUILD b/tools/bzl/BUILD
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/bzl/BUILD
diff --git a/tools/bzl/classpath.bzl b/tools/bzl/classpath.bzl
new file mode 100644
index 0000000..d5764f7
--- /dev/null
+++ b/tools/bzl/classpath.bzl
@@ -0,0 +1,4 @@
+load(
+ "@com_googlesource_gerrit_bazlets//tools:classpath.bzl",
+ "classpath_collector",
+)
diff --git a/tools/bzl/junit.bzl b/tools/bzl/junit.bzl
new file mode 100644
index 0000000..3af7e58
--- /dev/null
+++ b/tools/bzl/junit.bzl
@@ -0,0 +1,4 @@
+load(
+ "@com_googlesource_gerrit_bazlets//tools:junit.bzl",
+ "junit_tests",
+)
diff --git a/tools/bzl/maven_jar.bzl b/tools/bzl/maven_jar.bzl
new file mode 100644
index 0000000..2eabedb
--- /dev/null
+++ b/tools/bzl/maven_jar.bzl
@@ -0,0 +1 @@
+load("@com_googlesource_gerrit_bazlets//tools:maven_jar.bzl", "maven_jar")
diff --git a/tools/bzl/plugin.bzl b/tools/bzl/plugin.bzl
new file mode 100644
index 0000000..a2e438f
--- /dev/null
+++ b/tools/bzl/plugin.bzl
@@ -0,0 +1,6 @@
+load(
+ "@com_googlesource_gerrit_bazlets//:gerrit_plugin.bzl",
+ "gerrit_plugin",
+ "PLUGIN_DEPS",
+ "PLUGIN_TEST_DEPS",
+)
diff --git a/tools/eclipse/BUILD b/tools/eclipse/BUILD
new file mode 100644
index 0000000..2d7b28a
--- /dev/null
+++ b/tools/eclipse/BUILD
@@ -0,0 +1,9 @@
+load("//tools/bzl:classpath.bzl", "classpath_collector")
+
+classpath_collector(
+ name = "main_classpath_collect",
+ testonly = 1,
+ deps = [
+ "//:cache-postgres__plugin_test_deps",
+ ],
+)
diff --git a/tools/eclipse/project.sh b/tools/eclipse/project.sh
new file mode 100755
index 0000000..c87efea
--- /dev/null
+++ b/tools/eclipse/project.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+#Copyright (C) 2017 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+`bazel query @com_googlesource_gerrit_bazlets//tools/eclipse:project --output location | sed s/BUILD:.*//`project.py -n external-cache -r .
+
+
diff --git a/tools/workspace-status.sh b/tools/workspace-status.sh
new file mode 100755
index 0000000..1d4e3dd
--- /dev/null
+++ b/tools/workspace-status.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+
+#This script will be run by bazel when the build process starts to
+# generate key-value information that represents the status of the
+# workspace. The output should be like
+#
+# KEY1 VALUE1
+# KEY2 VALUE2
+#
+# If the script exits with non-zero code, it's considered as a failure
+# and the output will be discarded.
+
+function rev() {
+ cd $1; git describe --always --match "v[0-9].*" --dirty
+}
+
+echo STABLE_BUILD_CACHE-POSTGRES_LABEL $(rev .)
