commit 5e2325e2deaca8379f251bb6eda64bc4e9465b2a
author Edwin Kempin <ekempin@google.com> Thu Jul 11 12:13:33 2024 +0000
committer Edwin Kempin <ekempin@google.com> Thu Jul 11 14:20:32 2024 +0000
tree b72103a010c48129ef407569a904d4a31a5499cf
parent e0b48f76a7c795f23d9cd3e2fa8c5316f3096b59

Add support for configurable performance metrics

To optimize performance in Gerrit, it's important to know which
operations are expensive, how much latency they cause and how often they
are called. We did have performance metrics for this once, but had to
drop them because it was too expensive to capture this data for all
operations (see https://issues.gerritcodereview.com/issues/40014513).

We still have performance logs, which allows us to find expensive
operations, but knowing their latency distribution (e.g. p90, p99) and
how often they are called across all requests cannot easily be computed
from them.

With this change we re-add the performance metrics, but require that the
operations for which we record them are explicitly configured. This way
a metric explosion can be avoided while we can still get this data for
operations that we know are slow and which we want to optimize.

Having the performance metrics is useful to measure the impact of
performance improvements:

1. Find a slow operation in the performance logs
2. Configure the slow operation to be recorded in the metrics
3. Collect metric data for some days to know the average latency
4. Optimise the performance of the operation
5. Check how the average latency has improved
6. Estimate the total performance gain by multiplying the latency
   improvement with the number of times the operation is called

Alternatively we could add a new specific metric each time we want to
improve something, but this seems more overhead. An example for such a
metric is the existing parent_data_computation metric, which we can drop
once we have confirmed that the configurable performance metrics are
working.

This basically reverts commit d8af41afd1f352f8e25c9f79bff705c05665934c
and 491337d70cad8f3e25ae3477a7723021d395a480 but avoids the metric
explosion by requiring to configure the recorded operations explicitly.

Release-Notes: Added support for configurable performance metrics
Change-Id: I65790d0b17e2ccb1ced9e8ec2a1766c5286dcf91