commit 65f4828c37a7c52363bfe4354411f7957ff514bd
author Ben Rohlfs <brohlfs@google.com> Mon Nov 07 20:02:18 2022 +0100
committer Ben Rohlfs <brohlfs@google.com> Tue Nov 08 09:42:26 2022 +0000
tree 6c23ec804a78c08dcdeb4a4b85eec66c10717e5b
parent 9d49c587cdbacdf0ffd44d05a49e17cc3d325c77

Fix DASHBOARD_DISPLAYED timing

Since Oct 10th we have seen a substantial increase in the 99th
percentile of this latency metric.

I have looked into `DashboardDisplayed` events with elapsed > 10
seconds. Apparently we already had bogus events in September, but only
a few per day. Then on Oct 5 the number of bogus events has
substantially increased. With ~17,000 DashboardDisplayed events per day
the ~200 bogus events per day have started impacting the 99th
percentile massively.

I have then looked into one random sample session with a bogus event.
The difference `event_start - elapsed = 13268` of the first bogus event
clearly pointed to the `NAVIGATION` being where DashboardDisplayed has
started counting, but it was also kind of obvious that most of the time
the dashboard was not actually loading.

So I have then looked into the code of how `DashboardDisplayed` is
measured. There is a bit of a disconnect between where the timer is
started and stopped, and using console logging I have verified that
very often the `reporting.dashboardDisplayed()` call does not actually
report a timing event, because the timer was not started at all. What
is new since Oct 5 (router was refactored) is that it can happen more
often that the timer starts, but does not end, and both together are
the root cause of the issue.

So this issue turns out to be a measuring problem. Gerrit has not
actually become slower. We are fixing the issue such that the router
is only responsible for starting the timer when the web app loads the
dashboard for the first time. All subsequent dashboard loading from
the same component is measured from the component itself. If the
`dashboard-view` is re-used, then there is not much delay from the
router initiating the navigation and the existing component receiving
the state update.

This new way of measuring `DashboardDisplayed` will very likely not
just fix the 99th percentile, but improve the metric overall, because
it will be reported more often.

I think it could be argued that `DashboardDisplayed` is not a very
useful metric to track. More or less it will just measure how long
the change search query to the backend takes.

Release-Notes: skip
Google-Bug-Id: b/254162560
Change-Id: I618559d421f12bd9bcbd181c21e9c80a653a6697