Implement rate limits per account for fetch requests
From time to time we observe excessive load caused by misconfigured
build jobs polling for new changes too frequently or by buggy scripts or
other misbehaving clients. Analyzing and following up such problems
manually is time consuming. Rate limiting can be used to automatically
limit the load an individual user can impose on Gerrit to prevent a
negative impact on other users.
As a first step this change implements rate limits for fetch requests
which we found to be the most frequent cause of excessive load caused by
misbehaving clients.
Fetch request rate limits can be configured in configuration file
quota.config stored in the refs/meta/config branch of All-Projects.
For logged in users rate limits are associated to their accountId. For
anonymous users rate limits are associated to their remote host address.
If multiple anonymous users are accessing Gerrit via the same host (e.g.
a proxy) they share a common rate limit.
Guava's RateLimiter implementation class SmoothRateLimiter.SmoothBursty
allows to collect permits during idle times which can be used to send
bursts of requests exceeding the average rate until the stored permits
are consumed. If the rate per second is 0.2 and you wait 20 seconds you
can acquire 4 permits which in average matches the configured rate limit
of 0.2 requests/second.
Unfortunately the standard implementation doesn't allow any bursts if
the configured permit rate is smaller than 1 per second since the
maximum time which can be used to collect stored permits is hard-coded
to 1 second.
Build jobs fetching updates from Gerrit are typically triggered by
events which can arrive in bursts. Hence the standard RateLimiter seems
not to be the right choice for fetch requests we might want to limit to
a rate of less than 1 request per second per user.
The used constructor can't be accessed through a public method yet hence
use reflection to instantiate it. There is a bug [1] open requesting to
expose this parameter in the public guava API.
For standalone Buck build reference Gerrit API 2.13-SNAPSHOT until
2.13.9 has been released since this change depends on the enhanced
UploadValidationListener interface in
Iaafc0f844c07d823b8a7ef6377f2ede1c5805a08
TODOs:
* add tests for parsing of rate limit configuration entries
[1] https://github.com/google/guava/issues/1974
Depends-On: Iaafc0f844c07d823b8a7ef6377f2ede1c5805a08
Change-Id: Ie1ca2e19e9d8a9a525af534b7ee7d6d4164e27e9
8 files changed
