commit b4b73bda1ff749c5f713f1444f563555de76048e
author Kaushik Lingarkar <kaushik.lingarkar@linaro.org> Mon Sep 12 10:05:46 2022 -0700
committer Kaushik Lingarkar <kaushik.lingarkar@linaro.org> Mon Sep 12 10:05:46 2022 -0700
tree a4d7c368b8ce80ae592f974864a156cb51891238
parent db635f9a025c3dda198ed11c3cc42b17e2c0db4b

Paginate no-limit queries

Currently, no-limit queries do a single index search with a rather
large limit (around the size of change index). For Elasticsearch
index backend, this can be problematic as Elasticsearch's REST
client will fail with an error like [1] if too many changes have
to be returned. The REST client has a default limit of 100MB for
content it can process. This effectively implies that no-limit
queries with ES index backend are likely non-functional for most
sites as the 100MB limit is reached by as low as ~50k changes.

Changing this default limit is not recommended as it can overload
ES data nodes and the client as well. Instead, this change updates
no-limit queries to paginate rather than doing a single index search
with a large limit. It is recommended to set an appropriate value
for 'index.maxPageSize' to avoid the error [1] especially when
'index.pageSizeMultiplier' is set to value greater than 1.

Here are some stats with a Lucene based site and ~20k docs in open
changes index and ~4M in closed changes index (with ~1M abandoned
and ~3M merged).

                               status:open     staus:abandoned
                               no-limit        no-limit

without this change            7.6s            436s

with change
paginationType=OFFSET          7.9s            2622s
pageSizeMultiplier=1

with change
paginationType=SEARCH_AFTER    7.8s            480s
pageSizeMultiplier=1

with change
paginationType=OFFSET          7.4s            417s
pageSizeMultiplier=10

with change
paginationType=SEARCH_AFTER    7.7s            418s
pageSizeMultiplier=10

If 'index.pageSizeMultiplier' is set to 1 (default), we update it to
10 for no-limit queries as it helps improve performance and also
prevents no-limit queries from severely degrading when pagination type
is OFFSET.

[1] entity content is too long [..] for the configured buffer limit [104857600]

Release-Notes: no-limit queries are now usable with Elasticsearch index backend
Change-Id: Ifb1f6f5411140c430f2520fb252e688b67d5333c