Google Git
Sign in
gerrit / zoekt / eed1c916f12d72fe155b945c0360e9050b35df16
commiteed1c916f12d72fe155b945c0360e9050b35df16[log] [tgz]
authorKeegan Carruthers-Smith <keegan.csmith@gmail.com>Mon Feb 10 17:17:26 2020 +0000
committerKeegan Carruthers-Smith <keegan.csmith@gmail.com>Wed Feb 12 15:14:17 2020 +0000
tree2eedc99f6749ceb90edaae9bc7beb1782e51c821
parenta9b6bda45559b9c4ee6ae6bcb5945365fa8d0e56 [diff]
shards: cache ranked slice

On a search request we construct and sort a slice of all shards to
search. This change will cache the slice. We do the caching on the read path
to avoid holding the exclusive lock for a long time when we do writes to the
shard map.

The use of a "rankedVersion" is a bit fiddly. Alternatively we could look
making the replace call a batch version to amortize the cost of creating the
ranked slice.

Change-Id: I5272dbb75a4becebb8aa8727d8459d0666cb9bf9
1 file changed
tree: 2eedc99f6749ceb90edaae9bc7beb1782e51c821
  1. build/
  2. cmd/
  3. ctags/
  4. doc/
  5. gitindex/
  6. query/
  7. shards/
  8. web/
  9. .gitignore
  10. all.bash
  11. api.go
  12. bits.go
  13. bits_test.go
  14. build-deploy.sh
  15. contentprovider.go
  16. CONTRIBUTING
  17. eval.go
  18. go.mod
  19. go.sum
  20. hititer.go
  21. hititer_test.go
  22. index_test.go
  23. indexbuilder.go
  24. indexdata.go
  25. indexfile.go
  26. indexfile_linux.go
  27. LICENSE
  28. matchiter.go
  29. matchtree.go
  30. matchtree_test.go
  31. read.go
  32. read_test.go
  33. README.md
  34. section.go
  35. section_test.go
  36. toc.go
  37. write.go
README.md
"Zoekt, en gij zult spinazie eten" - Jan Eertink

("seek, and ye shall eat spinach" - My primary school teacher)

This is a fast text search engine, intended for use with source code. (Pronunciation: roughly as you would pronounce “zooked” in English)

INSTRUCTIONS

Downloading:

go get github.com/google/zoekt/

Indexing:

go install github.com/google/zoekt/cmd/zoekt-index
$GOPATH/bin/zoekt-index .

Searching

go install github.com/google/zoekt/cmd/zoekt
$GOPATH/bin/zoekt 'ngram f:READ'

Indexing git repositories:

go install github.com/google/zoekt/cmd/zoekt-git-index
$GOPATH/bin/zoekt-git-index -branches master,stable-1.4 -prefix origin/ .

Indexing repo repositories:

go install github.com/google/zoekt/cmd/zoekt-{repo-index,mirror-gitiles}
zoekt-mirror-gitiles -dest ~/repos/ https://gfiber.googlesource.com
zoekt-repo-index \
   -name gfiber \
   -base_url https://gfiber.googlesource.com/ \
   -manifest_repo ~/repos/gfiber.googlesource.com/manifests.git \
   -repo_cache ~/repos \
   -manifest_rev_prefix=refs/heads/ --rev_prefix= \
   master:default_unrestricted.xml

Starting the web interface

go install github.com/google/zoekt/cmd/zoekt-webserver
$GOPATH/bin/zoekt-webserver -listen :6070

A more organized installation on a Linux server should use a systemd unit file, eg.

[Unit]
Description=zoekt webserver

[Service]
ExecStart=/zoekt/bin/zoekt-webserver -index /zoekt/index -listen :443  --ssl_cert /zoekt/etc/cert.pem   --ssl_key /zoekt/etc/key.pem
Restart=always

[Install]
WantedBy=default.target

SEARCH SERVICE

Zoekt comes with a small service management program:

go install github.com/google/zoekt/cmd/zoekt-indexserver

cat << EOF > config.json
[{"GithubUser": "username"},
 {"GithubOrg": "org"},
 {"GitilesURL": "https://gerrit.googlesource.com", "Name": "zoekt" }
]
EOF

$GOPATH/bin/zoekt-server -mirror_config config.json

This will mirror all repos under ‘github.com/username’, ‘github.com/org’, as well as the ‘zoekt’ repository. It will index the repositories.

It takes care of fetching and indexing new data and cleaning up logfiles.

The webserver can be started from a standard service management framework, such as systemd.

SYMBOL SEARCH

It is recommended to install Universal ctags to improve ranking. See here for more information.

ACKNOWLEDGEMENTS

Thanks to Alexander Neubeck for coming up with this idea, and helping me flesh it out.

DISCLAIMER

This is not an official Google product