commit | 5851b63aed4d850f6618b696b89e757f0ae1ec22 | [log] [tgz] |
---|---|---|
author | Keegan Carruthers-Smith <keegan.csmith@gmail.com> | Sat Oct 19 00:14:45 2019 +0200 |
committer | Keegan Carruthers-Smith <keegan.csmith@gmail.com> | Thu Oct 31 10:23:36 2019 +0200 |
tree | d88ab1063ec19b72171f4bb4d9a5f9caed7850a4 | |
parent | dc11ed55eca9e9d9f518a9798f3bab06caca0723 [diff] |
ctags: skip long lines Currently we will fail to index a repository if it contains a very large symbol. This is due to bufio.Scanner returning bufio.ErrTooLong. The documentation encourages using bufio.Reader to have a different behaviour, which is what this commit implements. We introduce a scanner struct which mimics a minimal API of bufio.Scanner. It directly uses bufio.Reader, but taking care to skip lines that are too long or empty. It will skip any line larger than 4096 (including newline). Examples of repositories with very large symbols: - https://github.com/noshotz/eldorito - https://github.com/corretto/corretto-8 - https://github.com/ifsnow/facebooklivereaction Change-Id: Ic60f39c79f63363ef2d4175c4d6de7baae90355c
"Zoekt, en gij zult spinazie eten" - Jan Eertink ("seek, and ye shall eat spinach" - My primary school teacher)
This is a fast text search engine, intended for use with source code. (Pronunciation: roughly as you would pronounce “zooked” in English)
Downloading:
go get github.com/google/zoekt/
Indexing:
go install github.com/google/zoekt/cmd/zoekt-index $GOPATH/bin/zoekt-index .
Searching
go install github.com/google/zoekt/cmd/zoekt $GOPATH/bin/zoekt 'ngram f:READ'
Indexing git repositories:
go install github.com/google/zoekt/cmd/zoekt-git-index $GOPATH/bin/zoekt-git-index -branches master,stable-1.4 -prefix origin/ .
Indexing repo repositories:
go install github.com/google/zoekt/cmd/zoekt-{repo-index,mirror-gitiles} zoekt-mirror-gitiles -dest ~/repos/ https://gfiber.googlesource.com zoekt-repo-index \ -name gfiber \ -base_url https://gfiber.googlesource.com/ \ -manifest_repo ~/repos/gfiber.googlesource.com/manifests.git \ -repo_cache ~/repos \ -manifest_rev_prefix=refs/heads/ --rev_prefix= \ master:default_unrestricted.xml
Starting the web interface
go install github.com/google/zoekt/cmd/zoekt-webserver $GOPATH/bin/zoekt-webserver -listen :6070
A more organized installation on a Linux server should use a systemd unit file, eg.
[Unit] Description=zoekt webserver [Service] ExecStart=/zoekt/bin/zoekt-webserver -index /zoekt/index -listen :443 --ssl_cert /zoekt/etc/cert.pem --ssl_key /zoekt/etc/key.pem Restart=always [Install] WantedBy=default.target
Zoekt comes with a small service management program:
go install github.com/google/zoekt/cmd/zoekt-indexserver cat << EOF > config.json [{"GithubUser": "username"}, {"GithubOrg": "org"}, {"GitilesURL": "https://gerrit.googlesource.com", "Name": "zoekt" } ] EOF $GOPATH/bin/zoekt-server -mirror_config config.json
This will mirror all repos under ‘github.com/username’, ‘github.com/org’, as well as the ‘zoekt’ repository. It will index the repositories.
It takes care of fetching and indexing new data and cleaning up logfiles.
The webserver can be started from a standard service management framework, such as systemd.
It is recommended to install Universal ctags to improve ranking. See here for more information.
Thanks to Alexander Neubeck for coming up with this idea, and helping me flesh it out.
This is not an official Google product