Implement rate limits per account for fetch requests
From time to time we observe excessive load caused by misconfigured
build jobs polling for new changes too frequently or by buggy scripts or
other misbehaving clients. Analyzing and following up such problems
manually is time consuming. Rate limiting can be used to automatically
limit the load an individual user can impose on Gerrit to prevent a
negative impact on other users.
As a first step this change implements rate limits for fetch requests
which we found to be the most frequent cause of excessive load caused by
misbehaving clients.
Fetch request rate limits can be configured in configuration file
quota.config stored in the refs/meta/config branch of All-Projects.
For logged in users rate limits are associated to their accountId. For
anonymous users rate limits are associated to their remote host address.
If multiple anonymous users are accessing Gerrit via the same host (e.g.
a proxy) they share a common rate limit.
Guava's RateLimiter implementation class SmoothRateLimiter.SmoothBursty
allows to collect permits during idle times which can be used to send
bursts of requests exceeding the average rate until the stored permits
are consumed. If the rate per second is 0.2 and you wait 20 seconds you
can acquire 4 permits which in average matches the configured rate limit
of 0.2 requests/second.
Unfortunately the standard implementation doesn't allow any bursts if
the configured permit rate is smaller than 1 per second since the
maximum time which can be used to collect stored permits is hard-coded
to 1 second.
Build jobs fetching updates from Gerrit are typically triggered by
events which can arrive in bursts. Hence the standard RateLimiter seems
not to be the right choice for fetch requests we might want to limit to
a rate of less than 1 request per second per user.
The used constructor can't be accessed through a public method yet hence
use reflection to instantiate it. There is a bug [1] open requesting to
expose this parameter in the public guava API.
For standalone Buck build reference Gerrit API 2.13-SNAPSHOT until
2.13.9 has been released since this change depends on the enhanced
UploadValidationListener interface in
Iaafc0f844c07d823b8a7ef6377f2ede1c5805a08
TODOs:
* add tests for parsing of rate limit configuration entries
[1] https://github.com/google/guava/issues/1974
Depends-On: Iaafc0f844c07d823b8a7ef6377f2ede1c5805a08
Change-Id: Ie1ca2e19e9d8a9a525af534b7ee7d6d4164e27e9
diff --git a/lib/gerrit/BUCK b/lib/gerrit/BUCK
index 4b48569..19326f3 100644
--- a/lib/gerrit/BUCK
+++ b/lib/gerrit/BUCK
@@ -1,7 +1,7 @@
include_defs('//bucklets/maven_jar.bucklet')
-VER = '2.13.3'
-REPO = MAVEN_CENTRAL
+VER = '2.13-SNAPSHOT'
+REPO = MAVEN_LOCAL
maven_jar(
name = 'plugin-api',
diff --git a/src/main/java/com/googlesource/gerrit/plugins/quota/AccountLimitsConfig.java b/src/main/java/com/googlesource/gerrit/plugins/quota/AccountLimitsConfig.java
new file mode 100644
index 0000000..7567855
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/plugins/quota/AccountLimitsConfig.java
@@ -0,0 +1,190 @@
+// Copyright (C) 2017 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.googlesource.gerrit.plugins.quota;
+
+import com.google.common.collect.ArrayTable;
+import com.google.common.collect.Table;
+import java.text.MessageFormat;
+import java.util.Arrays;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import org.eclipse.jgit.lib.Config;
+import org.eclipse.jgit.lib.Config.ConfigEnum;
+import org.eclipse.jgit.lib.Config.SectionParser;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class AccountLimitsConfig {
+ private static final Logger log =
+ LoggerFactory.getLogger(AccountLimitsConfig.class);
+ static final String GROUP_SECTION = "group";
+ static final SectionParser<AccountLimitsConfig> KEY =
+ new SectionParser<AccountLimitsConfig>() {
+ @Override
+ public AccountLimitsConfig parse(final Config cfg) {
+ return new AccountLimitsConfig(cfg);
+ }
+ };
+
+ public static class RateLimit {
+ public Type getType() {
+ return type;
+ }
+
+ public double getRatePerSecond() {
+ return ratePerSecond;
+ }
+
+ public int getMaxBurstSeconds() {
+ return maxBurstSeconds;
+ }
+
+ private Type type;
+ private double ratePerSecond;
+ private int maxBurstSeconds;
+
+ public RateLimit(Type type, double ratePerSecond, int maxBurstSeconds) {
+ this.type = type;
+ this.ratePerSecond = ratePerSecond;
+ this.maxBurstSeconds = maxBurstSeconds;
+ }
+ }
+
+ public static enum Type implements ConfigEnum {
+ UPLOADPACK;
+
+ @Override
+ public String toConfigValue() {
+ return name().toLowerCase(Locale.ROOT);
+ }
+
+ @Override
+ public boolean matchConfigValue(String in) {
+ return name().equalsIgnoreCase(in);
+ }
+ }
+
+ private Table<Type, String, RateLimit> rateLimits;
+
+ private AccountLimitsConfig(final Config c) {
+ Set<String> groups = c.getSubsections(GROUP_SECTION);
+ if (groups.size() == 0) {
+ return;
+ }
+ rateLimits = ArrayTable.create(Arrays.asList(Type.values()), groups);
+ for (String groupName : groups) {
+ Type type = Type.UPLOADPACK;
+ rateLimits.put(type, groupName,
+ parseRateLimit(c, groupName, type, 60, 30));
+ }
+ }
+
+ RateLimit parseRateLimit(Config c, String groupName, Type type,
+ int defaultIntervalSeconds, int defaultBurstCount) {
+ String name = type.toConfigValue();
+ String value = c.getString(GROUP_SECTION, groupName, name).trim();
+ if (value == null) {
+ return defaultRateLimit(type, defaultIntervalSeconds, defaultBurstCount);
+ }
+
+ Matcher m = Pattern.compile("^\\s*(\\d+)\\s*/\\s*(.*)\\s*burst\\s*(\\d+)$")
+ .matcher(value);
+ if (!m.matches()) {
+ log.warn(
+ "Invalid ''{}'' ratelimit configuration ''{}'', use default ratelimit {}/hour",
+ type.toConfigValue(), value, 3600.0D / defaultIntervalSeconds);
+ return defaultRateLimit(type, defaultIntervalSeconds, defaultBurstCount);
+ }
+
+ String digits = m.group(1);
+ String unitName = m.group(2).trim();
+ String storeCountString = m.group(3).trim();
+ long burstCount = defaultBurstCount;
+ try {
+ burstCount = Long.parseLong(storeCountString);
+ } catch (NumberFormatException e) {
+ log.warn(
+ "Invalid ''{}'' ratelimit store configuration ''{}'', use default burst count ''{}''",
+ type.toConfigValue(), storeCountString, burstCount);
+ }
+
+ TimeUnit inputUnit = TimeUnit.HOURS;
+ double ratePerSecond = 1.0D / defaultIntervalSeconds;
+ if (unitName.isEmpty()) {
+ inputUnit = TimeUnit.SECONDS;
+ } else if (match(unitName, "s", "sec", "second")) {
+ inputUnit = TimeUnit.SECONDS;
+ } else if (match(unitName, "m", "min", "minute")) {
+ inputUnit = TimeUnit.MINUTES;
+ } else if (match(unitName, "h", "hr", "hour")) {
+ inputUnit = TimeUnit.HOURS;
+ } else if (match(unitName, "d", "day")) {
+ inputUnit = TimeUnit.DAYS;
+ } else {
+ logNotRateUnit(GROUP_SECTION, groupName, name, value);
+ }
+ try {
+ ratePerSecond = 1.0D * Long.parseLong(digits)
+ / TimeUnit.SECONDS.convert(1, inputUnit);
+ } catch (NumberFormatException nfe) {
+ logNotRateUnit(GROUP_SECTION, groupName, unitName, value);
+ }
+
+ int maxBurstSeconds = (int) (burstCount / ratePerSecond);
+ return new RateLimit(type, ratePerSecond, maxBurstSeconds);
+ }
+
+ private static boolean match(final String a, final String... cases) {
+ for (final String b : cases) {
+ if (b != null && b.equalsIgnoreCase(a)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private void logNotRateUnit(String section, String subsection, String name,
+ String valueString) {
+ if (subsection != null) {
+ log.error(MessageFormat.format("Invalid rate unit value: {0}.{1}.{2}={3}",
+ section, subsection, name, valueString));
+ } else {
+ log.error(MessageFormat.format("Invalid rate unit value: {0}.{1}={2}",
+ section, name, valueString));
+ }
+ }
+
+ private RateLimit defaultRateLimit(Type type, int defaultIntervalSeconds,
+ int defaultStoreCount) {
+ return new RateLimit(type, 1.0D / defaultIntervalSeconds,
+ defaultIntervalSeconds * defaultStoreCount);
+ }
+
+ /**
+ * @param type type of rate limit
+ * @return map of rate limits per group name
+ */
+ Optional<Map<String, RateLimit>> getRatelimits(Type type) {
+ if (rateLimits != null) {
+ return Optional.ofNullable(rateLimits.row(type));
+ }
+ return Optional.empty();
+ }
+}
diff --git a/src/main/java/com/googlesource/gerrit/plugins/quota/AccountLimitsFinder.java b/src/main/java/com/googlesource/gerrit/plugins/quota/AccountLimitsFinder.java
new file mode 100644
index 0000000..751735c
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/plugins/quota/AccountLimitsFinder.java
@@ -0,0 +1,95 @@
+// Copyright (C) 2017 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package com.googlesource.gerrit.plugins.quota;
+
+import static com.googlesource.gerrit.plugins.quota.AccountLimitsConfig.KEY;
+
+import com.google.gerrit.common.data.GroupDescription;
+import com.google.gerrit.server.IdentifiedUser;
+import com.google.gerrit.server.account.GroupMembership;
+import com.google.gerrit.server.group.GroupsCollection;
+import com.google.gerrit.server.project.ProjectCache;
+import com.google.inject.Inject;
+
+import com.googlesource.gerrit.plugins.quota.AccountLimitsConfig.RateLimit;
+import com.googlesource.gerrit.plugins.quota.AccountLimitsConfig.Type;
+
+import org.eclipse.jgit.lib.Config;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+import java.util.Optional;
+
+public class AccountLimitsFinder {
+ private static final Logger log =
+ LoggerFactory.getLogger(AccountLimitsFinder.class);
+
+ private final ProjectCache projectCache;
+ private final GroupsCollection groupsCollection;
+
+ @Inject
+ AccountLimitsFinder(ProjectCache projectCache,
+ GroupsCollection groupsCollection) {
+ this.projectCache = projectCache;
+ this.groupsCollection = groupsCollection;
+ }
+
+ /**
+ * @param type type of rate limit
+ * @param user identified user
+ * @return the rate limit matching the first configured group limit the given
+ * user is a member of
+ */
+ public Optional<RateLimit> firstMatching(AccountLimitsConfig.Type type,
+ IdentifiedUser user) {
+ Optional<Map<String, AccountLimitsConfig.RateLimit>> limits =
+ getRatelimits(type);
+ if (limits.isPresent()) {
+ GroupMembership memberShip = user.getEffectiveGroups();
+ for (String groupName : limits.get().keySet()) {
+ GroupDescription.Basic d = groupsCollection.parseId(groupName);
+ if (d == null) {
+ log.error("Ignoring limits for unknown group ''{}'' in quota.config",
+ groupName);
+ } else if (memberShip.contains(d.getGroupUUID())) {
+ return Optional.ofNullable(limits.get().get(groupName));
+ }
+ }
+ }
+ return Optional.empty();
+ }
+
+ /**
+ * @param type type of rate limit
+ * @param groupName name of group to lookup up rate limit for
+ * @return rate limit
+ */
+ public Optional<RateLimit> getRateLimit(Type type, String groupName) {
+ if (getRatelimits(type).isPresent()) {
+ return Optional.ofNullable(getRatelimits(type).get().get(groupName));
+ }
+ return Optional.empty();
+ }
+
+ /**
+ * @param type type of rate limit
+ * @return map of rate limits per group name
+ */
+ private Optional<Map<String, RateLimit>> getRatelimits(Type type) {
+ Config cfg = projectCache.getAllProjects().getConfig("quota.config").get();
+ AccountLimitsConfig limitsCfg = cfg.get(KEY);
+ return limitsCfg.getRatelimits(type);
+ }
+}
diff --git a/src/main/java/com/googlesource/gerrit/plugins/quota/Module.java b/src/main/java/com/googlesource/gerrit/plugins/quota/Module.java
index 6814967..935498a 100644
--- a/src/main/java/com/googlesource/gerrit/plugins/quota/Module.java
+++ b/src/main/java/com/googlesource/gerrit/plugins/quota/Module.java
@@ -15,24 +15,36 @@
package com.googlesource.gerrit.plugins.quota;
import static com.google.gerrit.server.config.ConfigResource.CONFIG_KIND;
+import static com.google.gerrit.server.group.SystemGroupBackend.ANONYMOUS_USERS;
import static com.google.gerrit.server.project.ProjectResource.PROJECT_KIND;
import static com.googlesource.gerrit.plugins.quota.QuotaResource.QUOTA_KIND;
+import com.google.common.cache.CacheLoader;
+import com.google.common.util.concurrent.RateLimiter;
import com.google.gerrit.extensions.events.GarbageCollectorListener;
import com.google.gerrit.extensions.events.LifecycleListener;
import com.google.gerrit.extensions.events.ProjectDeletedListener;
import com.google.gerrit.extensions.registration.DynamicMap;
import com.google.gerrit.extensions.registration.DynamicSet;
import com.google.gerrit.extensions.restapi.RestApiModule;
+import com.google.gerrit.reviewdb.client.Account;
+import com.google.gerrit.server.IdentifiedUser;
+import com.google.gerrit.server.IdentifiedUser.GenericFactory;
+import com.google.gerrit.server.cache.CacheModule;
import com.google.gerrit.server.git.ReceivePackInitializer;
+import com.google.gerrit.server.git.validators.UploadValidationListener;
+import com.google.gerrit.server.group.SystemGroupBackend;
import com.google.gerrit.server.validators.ProjectCreationValidationListener;
-import com.google.inject.AbstractModule;
+import com.google.inject.Inject;
import com.google.inject.Scopes;
import com.google.inject.internal.UniqueAnnotations;
-
+import com.googlesource.gerrit.plugins.quota.AccountLimitsConfig.RateLimit;
+import java.util.Optional;
import org.eclipse.jgit.transport.PostReceiveHook;
-class Module extends AbstractModule {
+class Module extends CacheModule {
+ static final String CACHE_NAME_ACCOUNTID = "rate_limits_by_account";
+ static final String CACHE_NAME_REMOTEHOST = "rate_limits_by_ip";
@Override
protected void configure() {
@@ -62,5 +74,72 @@
bind(LifecycleListener.class)
.annotatedWith(UniqueAnnotations.create())
.to(PublisherScheduler.class);
+
+ DynamicSet.bind(binder(), UploadValidationListener.class)
+ .to(RateLimitUploadListener.class);
+ cache(CACHE_NAME_ACCOUNTID, Account.Id.class, Holder.class)
+ .loader(LoaderAccountId.class);
+ cache(CACHE_NAME_REMOTEHOST, String.class, Holder.class)
+ .loader(LoaderRemoteHost.class);
+ }
+
+ static class Holder {
+ public static final Holder EMPTY = new Holder(null);
+ private RateLimiter l;
+
+ public Holder(RateLimiter l) {
+ this.l = l;
+ }
+
+ public RateLimiter get() {
+ return l;
+ }
+ }
+
+ private static class LoaderAccountId extends CacheLoader<Account.Id, Holder> {
+ private GenericFactory userFactory;
+ private AccountLimitsFinder finder;
+
+ @Inject
+ LoaderAccountId(IdentifiedUser.GenericFactory userFactory,
+ AccountLimitsFinder finder) {
+ this.userFactory = userFactory;
+ this.finder = finder;
+ }
+
+ @Override
+ public Holder load(Account.Id key) throws Exception {
+ IdentifiedUser user = userFactory.create(key);
+ Optional<RateLimit> limit =
+ finder.firstMatching(AccountLimitsConfig.Type.UPLOADPACK, user);
+ if (limit.isPresent()) {
+ return new Holder(RateLimitUploadListener.createSmoothBurstyRateLimiter(
+ limit.get().getRatePerSecond(), limit.get().getMaxBurstSeconds()));
+ }
+ return Holder.EMPTY;
+ }
+ }
+
+ private static class LoaderRemoteHost extends CacheLoader<String, Holder> {
+ private AccountLimitsFinder finder;
+ private String anonymous;
+
+ @Inject
+ LoaderRemoteHost(SystemGroupBackend systemGroupBackend,
+ AccountLimitsFinder finder) {
+ this.finder = finder;
+ this.anonymous = systemGroupBackend.get(ANONYMOUS_USERS).getName();
+ }
+
+ @Override
+ public Holder load(String key) throws Exception {
+ Optional<RateLimit> limit =
+ finder.getRateLimit(AccountLimitsConfig.Type.UPLOADPACK, anonymous);
+ if (limit.isPresent()) {
+ return new Holder(RateLimitUploadListener.createSmoothBurstyRateLimiter(
+ limit.get().getRatePerSecond(), limit.get().getMaxBurstSeconds()));
+ }
+ return Holder.EMPTY;
+ }
}
}
diff --git a/src/main/java/com/googlesource/gerrit/plugins/quota/RateLimitException.java b/src/main/java/com/googlesource/gerrit/plugins/quota/RateLimitException.java
new file mode 100644
index 0000000..daa6a77
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/plugins/quota/RateLimitException.java
@@ -0,0 +1,25 @@
+// Copyright (C) 2017 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.googlesource.gerrit.plugins.quota;
+
+import com.google.gerrit.server.validators.ValidationException;
+
+public class RateLimitException extends ValidationException {
+ private static final long serialVersionUID = 1L;
+
+ RateLimitException(String msg) {
+ super(msg);
+ }
+}
diff --git a/src/main/java/com/googlesource/gerrit/plugins/quota/RateLimitUploadListener.java b/src/main/java/com/googlesource/gerrit/plugins/quota/RateLimitUploadListener.java
new file mode 100644
index 0000000..64ec2b1
--- /dev/null
+++ b/src/main/java/com/googlesource/gerrit/plugins/quota/RateLimitUploadListener.java
@@ -0,0 +1,168 @@
+// Copyright (C) 2017 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.googlesource.gerrit.plugins.quota;
+
+import static com.googlesource.gerrit.plugins.quota.Module.CACHE_NAME_ACCOUNTID;
+import static com.googlesource.gerrit.plugins.quota.Module.CACHE_NAME_REMOTEHOST;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.cache.LoadingCache;
+import com.google.common.util.concurrent.RateLimiter;
+import com.google.gerrit.reviewdb.client.Account;
+import com.google.gerrit.reviewdb.client.Project;
+import com.google.gerrit.server.CurrentUser;
+import com.google.gerrit.server.git.validators.UploadValidationListener;
+import com.google.gerrit.server.validators.ValidationException;
+import com.google.inject.Inject;
+import com.google.inject.Provider;
+import com.google.inject.name.Named;
+
+import com.googlesource.gerrit.plugins.quota.Module.Holder;
+
+import org.eclipse.jgit.lib.ObjectId;
+import org.eclipse.jgit.lib.Repository;
+import org.eclipse.jgit.transport.UploadPack;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.text.MessageFormat;
+import java.util.Collection;
+import java.util.concurrent.ExecutionException;
+
+public class RateLimitUploadListener implements UploadValidationListener {
+ private static final int SECONDS_PER_HOUR = 3600;
+ private static final Logger log =
+ LoggerFactory.getLogger(RateLimitUploadListener.class);
+ private static final Method createStopwatchMethod;
+ private static final Constructor<?> constructor;
+
+ static {
+ try {
+ Class<?> sleepingStopwatchClass = Class.forName(
+ "com.google.common.util.concurrent.RateLimiter$SleepingStopwatch");
+ createStopwatchMethod =
+ sleepingStopwatchClass.getDeclaredMethod("createFromSystemTimer");
+ createStopwatchMethod.setAccessible(true);
+ Class<?> burstyRateLimiterClass = Class.forName(
+ "com.google.common.util.concurrent.SmoothRateLimiter$SmoothBursty");
+ constructor = burstyRateLimiterClass.getDeclaredConstructors()[0];
+ constructor.setAccessible(true);
+ } catch (ClassNotFoundException | NoSuchMethodException e) {
+ // shouldn't happen
+ throw new RuntimeException(
+ "Failed to prepare loading RateLimiter via reflection", e);
+ }
+ }
+
+ /**
+ * Create a custom instance of RateLimiter by accessing the non-public
+ * constructor of the implementation class SmoothRateLimiter.SmoothBursty
+ * through reflection.
+ *
+ * <p>
+ * RateLimiter's implementation class SmoothRateLimiter.SmoothBursty allows to
+ * collect permits during idle times which can be used to send bursts of
+ * requests exceeding the average rate until the stored permits are consumed.
+ * If the rate per second is 0.2 and you wait 20 seconds you can acquire 4
+ * permits which in average matches the configured rate limit of 0.2
+ * requests/second. If the permitted rate is smaller than 1 per second the
+ * standard implementation doesn't allow any bursts since it hard-codes the
+ * maximum time which can be used to collect stored permits to 1 second.
+ *
+ * <p>
+ * Build jobs fetching updates from Gerrit are typically triggered by events
+ * which can arrive in bursts. Hence the standard RateLimiter seems not to be
+ * the right choice at least for fetch requests where we probably want to
+ * limit the rate to less than 1 request per second per user.
+ *
+ * <p>
+ * The used constructor can't be accessed through a public method yet hence
+ * use reflection to instantiate it.
+ *
+ * @see "https://github.com/google/guava/issues/1974"
+ * @param permitsPerSecond the new stable rate of this {@code RateLimiter}
+ * @param maxBurstSeconds The maximum number of permits that can be saved.
+ * @return a new RateLimiter
+ */
+ @VisibleForTesting
+ static RateLimiter createSmoothBurstyRateLimiter(double permitsPerSecond,
+ double maxBurstSeconds) {
+ RateLimiter rl;
+ try {
+ Object stopwatch = createStopwatchMethod.invoke(null);
+ rl = (RateLimiter) constructor.newInstance(stopwatch, maxBurstSeconds);
+ rl.setRate(permitsPerSecond);
+ } catch (InvocationTargetException | IllegalAccessException
+ | InstantiationException e) {
+ // shouldn't happen
+ throw new RuntimeException(e);
+ }
+ return rl;
+ }
+
+ private final Provider<CurrentUser> user;
+ private final LoadingCache<Account.Id, Holder> limitsPerAccount;
+ private final LoadingCache<String, Holder> limitsPerRemoteHost;
+
+ @Inject
+ RateLimitUploadListener(Provider<CurrentUser> user,
+ @Named(CACHE_NAME_ACCOUNTID) LoadingCache<Account.Id, Holder> limitsPerAccount,
+ @Named(CACHE_NAME_REMOTEHOST) LoadingCache<String, Holder> limitsPerRemoteHost) {
+ this.user = user;
+ this.limitsPerAccount = limitsPerAccount;
+ this.limitsPerRemoteHost = limitsPerRemoteHost;
+ }
+
+ @Override
+ public void onBeginNegotiate(Repository repository, Project project,
+ String remoteHost, UploadPack up, Collection<? extends ObjectId> wants,
+ int cntOffered) throws ValidationException {
+ RateLimiter limiter = null;
+ CurrentUser u = user.get();
+ if (u.isIdentifiedUser()) {
+ Account.Id accountId = u.asIdentifiedUser().getAccountId();
+ try {
+ limiter = limitsPerAccount.get(accountId).get();
+ } catch (ExecutionException e) {
+ String msg = MessageFormat
+ .format("Cannot get rate limits for account ''{}''", accountId);
+ log.warn(msg, e);
+ }
+ } else {
+ try {
+ limiter = limitsPerRemoteHost.get(remoteHost).get();
+ } catch (ExecutionException e) {
+ String msg = MessageFormat.format(
+ "Cannot get rate limits for anonymous access from remote host ''{0}''",
+ remoteHost);
+ log.warn(msg, e);
+ }
+ }
+ if (limiter != null && !limiter.tryAcquire()) {
+ throw new RateLimitException(
+ MessageFormat.format("Exceeded rate limit of {0,number,##.##} fetch requests/hour",
+ limiter.getRate() * SECONDS_PER_HOUR));
+ }
+ }
+
+ @Override
+ public void onPreUpload(Repository repository, Project project,
+ String remoteHost, UploadPack up, Collection<? extends ObjectId> wants,
+ Collection<? extends ObjectId> haves) throws ValidationException {
+ }
+}
diff --git a/src/main/resources/Documentation/about.md b/src/main/resources/Documentation/about.md
index 5091958..a20a452 100644
--- a/src/main/resources/Documentation/about.md
+++ b/src/main/resources/Documentation/about.md
@@ -2,7 +2,8 @@
To protect a Gerrit installation it makes sense to limit the resources
that a project or group can consume. To do this a Gerrit administrator
-can use this plugin to define quotas on project namespaces.
+can use this plugin to define quotas on project namespaces and define
+rate limits per user group.
The @PLUGIN@ plugin supports the following quotas:
@@ -11,3 +12,10 @@
The measured repository sizes can be published periodically to registered
UsageDataPublishedListeners.
+
+The @PLUGIN@ plugin supports the following rate limits:
+
+* `uploadpack` requests which are executed when a client runs a fetch command.
+
+Rate limits define the maximum request rate for users in a given group
+for a given request type.
diff --git a/src/main/resources/Documentation/config.md b/src/main/resources/Documentation/config.md
index 31c32b9..8bf28c1 100644
--- a/src/main/resources/Documentation/config.md
+++ b/src/main/resources/Documentation/config.md
@@ -133,6 +133,90 @@
packedObjectsSize*, where *looseObjectsSize* and *packedObjectsSize* are given
by JGit RepoStatistics. By default, false.
+Rate Limits
+-----------
+
+The defined rate limits are stored in a `quota.config` file in the
+`refs/meta/config` branch of the `All-Projects` root project. Rate
+limits are defined per user group and rate limit type:
+
+Example:
+```
+[group "buildserver"]
+ uploadpack = 10 / min burst 500
+
+[group "Registered Users"]
+ uploadpack = 1 /min burst 180
+
+[group "Anonymous Users"]
+ uploadpack = 6/h burst 12
+```
+
+For logged in users rate limits are associated to their accountId. For
+anonymous users rate limits are associated to their remote host address.
+If multiple anonymous users are accessing Gerrit via the same host (e.g.
+a proxy) they share a common rate limit.
+
+If a user is a member of multiple groups mentioned in `quota.config`
+the limit applies that is defined first in the `quota.config` file.
+This resolves ambiguity in case the user is a member of multiple groups
+used in the configuration.
+
+Use group "Anonymous Users" to define the rate limit for anonymous users.
+Use group "Registered Users" to define the default rate limit for all logged
+in users.
+
+Format of the rate limit entries in `quota.config`:
+```
+[group "<groupName>"]
+ <rateLimitType> = <rateLimit> <rateUnit> burst <storedRequests>
+```
+
+<a id="rateLimitType>">
+`group.<groupName>.<rateLimitType>`
+: identifies which request type is limited by this configuration.
+The following rate limit types are supported:
+* `uploadpack`: rate limit for uploadpack (fetch) requests
+The group can be defined by its name or UUID.
+
+<a id="uploadpack">
+`group.<groupName>.uploadpack`
+: rate limit for uploadpack (fetch) requests for the given group. The
+group can be defined by its name or UUID.
+
+<a id="rateLimit">
+: The rate limit (first parameter) defines the maximum allowed request rate.
+
+<a id="rateUnit">
+: Rate limits can be defined using the following rate units:
+`/s`, `/sec`, `/second`: requests per second
+`/m`, `/min`, `/minute`: requests per minute
+`/h`, `/hr`, `/hour`: requests per hour
+`/d`, `/day`: requests per day
+
+The default unit used if no unit is configured is `/hour`.
+
+<a id="burst">
+The `burst` parameter allows to define how many unused requests can be
+stored for later use during idle times. This allows clients to send
+bursts of requests exceeding their rate limit until all their stored
+requests are consumed.
+
+If a rate limit configuration value is invalid a default rate limit of 1
+request per minute with 30 stored requests is assumed.
+
+Example:
+
+Configure a rate limit of maximum 30 fetch request per hour for
+the group of registered users. Up to 60 unused requests can be stored
+during idle times which may be consumed at a later time to send bursts
+of requests above the maximum request rate.
+
+```
+[group "Registered Users"]
+ uploadpack = 30/hour burst 60
+```
+
Publication Schedule
--------------------
