Google Git
Sign in
gerrit / zoekt / 2a6b8144da3091074ec5cbd6ee905ec0a4417278
commit2a6b8144da3091074ec5cbd6ee905ec0a4417278[log] [tgz]
authorHan-Wen Nienhuys <hanwen@google.com>Fri May 25 17:53:49 2018 +0200
committerHan-Wen Nienhuys <hanwen@google.com>Fri May 25 18:04:09 2018 +0200
treea81e1e7d9f8682051209cb550a3d158b960cbeef
parenta23cf64a3697a1a2cc257c8ca00303ced8af96db [diff]
Check for binary content using bytes.Index before building postings

In commit 8029ba ("Include files without text on index"), I bailed
halfway in newSearchableString on finding problems. Unfortunately, the
postingsBuilder was not meant to process a file only halfway. This lead
to OOB access during the rune/byte translation for shards that had
halfway processed files.

Change-Id: I49d6a82bb1e6d4129f03de9a4293fc1cc5dcb4b3
1 file changed
tree: a81e1e7d9f8682051209cb550a3d158b960cbeef
  1. build/
  2. cmd/
  3. ctags/
  4. doc/
  5. gitindex/
  6. query/
  7. shards/
  8. web/
  9. .gitignore
  10. all.bash
  11. api.go
  12. bits.go
  13. bits_test.go
  14. build-deploy.sh
  15. contentprovider.go
  16. CONTRIBUTING
  17. eval.go
  18. hititer.go
  19. index_test.go
  20. indexbuilder.go
  21. indexdata.go
  22. indexfile.go
  23. indexfile_linux.go
  24. LICENSE
  25. matchiter.go
  26. matchtree.go
  27. read.go
  28. read_test.go
  29. README.md
  30. section.go
  31. section_test.go
  32. toc.go
  33. write.go
README.md
"Zoekt, en gij zult spinazie eten" - Jan Eertink

("seek, and ye shall eat spinach" - My primary school teacher)

This is a fast text search engine, intended for use with source code. (Pronunciation: roughly as you would pronounce “zooked” in English)

INSTRUCTIONS

Downloading:

go get github.com/google/zoekt/

Indexing:

go install github.com/google/zoekt/cmd/zoekt-index
$GOPATH/bin/zoekt-index .

Searching

go install github.com/google/zoekt/cmd/zoekt
$GOPATH/bin/zoekt 'ngram f:READ'

Indexing git repositories:

go install github.com/google/zoekt/cmd/zoekt-git-index
$GOPATH/bin/zoekt-git-index -branches master,stable-1.4 -prefix origin/ .

Indexing repo repositories:

go install github.com/google/zoekt/cmd/zoekt-{repo-index,mirror-gitiles}
zoekt-mirror-gitiles -dest ~/repos/ https://gfiber.googlesource.com
zoekt-repo-index \
   -name gfiber \
   -base_url https://gfiber.googlesource.com/ \
   -manifest_repo ~/repos/gfiber.googlesource.com/manifests.git \
   -repo_cache ~/repos \
   -manifest_rev_prefix=refs/heads/ --rev_prefix= \
   master:default_unrestricted.xml

Starting the web interface

go install github.com/google/zoekt/cmd/zoekt-webserver
$GOPATH/bin/zoekt-webserver -listen :6070

A more organized installation on a Linux server should use a systemd unit file, eg.

[Unit]
Description=zoekt webserver

[Service]
ExecStart=/zoekt/bin/zoekt-webserver -index /zoekt/index -listen :443  --ssl_cert /zoekt/etc/cert.pem   --ssl_key /zoekt/etc/key.pem
Restart=always

[Install]
WantedBy=default.target

SEARCH SERVICE

Zoekt comes with a small service management program:

go install github.com/google/zoekt/cmd/zoekt-indexserver

cat << EOF > config.json
[{"GithubUser": "username"},
 {"GitilesURL": "https://gerrit.googlesource.com", Name: "zoekt" }
]
EOF

$GOPATH/bin/zoekt-server -mirror_config config.json

This will mirror all repos under ‘github.com/username’ as well as the ‘zoekt’ repository. It will index the repositories.

It takes care of fetching and indexing new data and cleaning up logfiles.

The webserver can be started from a standard service management framework, such as systemd.

SYMBOL SEARCH

It is recommended to install Universal ctags to improve ranking. See here for more information.

ACKNOWLEDGEMENTS

Thanks to Alexander Neubeck for coming up with this idea, and helping me flesh it out.

DISCLAIMER

This is not an official Google product