tree d88ab1063ec19b72171f4bb4d9a5f9caed7850a4
parent dc11ed55eca9e9d9f518a9798f3bab06caca0723
author Keegan Carruthers-Smith <keegan.csmith@gmail.com> 1571436885 +0200
committer Keegan Carruthers-Smith <keegan.csmith@gmail.com> 1572510216 +0200

ctags: skip long lines

Currently we will fail to index a repository if it contains a very large
symbol. This is due to bufio.Scanner returning bufio.ErrTooLong. The
documentation encourages using bufio.Reader to have a different behaviour,
which is what this commit implements.

We introduce a scanner struct which mimics a minimal API of bufio.Scanner. It
directly uses bufio.Reader, but taking care to skip lines that are too long or
empty. It will skip any line larger than 4096 (including newline).

Examples of repositories with very large symbols:
- https://github.com/noshotz/eldorito
- https://github.com/corretto/corretto-8
- https://github.com/ifsnow/facebooklivereaction

Change-Id: Ic60f39c79f63363ef2d4175c4d6de7baae90355c