Slothfs is a FUSE file system that offers a read-only view of a Git tree. It downloads files lazily, so it takes up less diskspace than a full checkout based on repo. Once its caches are seeded, creating fresh workspaces or syncing should take on the order of seconds.
It is developed as open source, and can be used by anyone that uses Gerrit/Gitiles based Git hosting. The design doc explains the background behind its design choices.
It has been tested on Linux, but we expect it works on OSX too.
For the remainder of this document, we will assume that you are trying to compile some flavor of Android.
Get the source code,
go get github.com/google/slothfs/cmd/slothfs-multifs
SlothFS depends git2go, which depends on libgit2. For git2go, we recommend compiling libgit2 in statically, as documented here.
To install all components, run
cd $GOPATH/src/github.com/google/slothfs/ && sh all.bash
The rest of this document assumes this has been done, and $GOPATH/bin/ is in your $PATH.
In addition, install the standard Android clone.json to avoid unnecessary git clones
mkdir -p $HOME/.config/slothfs curl https://raw.githubusercontent.com/google/slothfs/master/android.json \ > $HOME/.config/slothfs/clone.json
The defaults of slothfs are for the public version of Android at android.googlesource.com. Set the -gitiles_url option to change against which version of Android you want to run.
First, create a mountpoint:
sudo mkdir -p /slothfs && sudo chown $USER /slothfs
Then, to mount the file system, run
slothfs-multifs /slothfs
The manifest describes which repositories go into an Android checkout. To make a file system out of this, we must decide at which exact revision each repository should be offered.
This can be done with slothfs-deref-manifest, eg.
slothfs-deref-manifest > /tmp/m.xml
A workspace can be created by a symlinking a deferenced manifest field into the config directory, i.e.
ln -s /tmp/m.xml /slothfs/config/my-workspace
This should create a directory /slothfs/my-workspace holding the tree described in /tmp/m.xml.
On the first time you do this, slothfs will have to fetch the tree data, which is slow, so this might take a while.
To use the read-only tree, create a writable checkout. Let's assume we want to change the art repo:
mkdir -p checkout ; cd checkout git clone https://android.googlesource.com/platform/art
To make checkout into a full-fledged checkout, run
slothfs-populate -ro /slothfs/my-workspace .
This populate the checkout directory with symlinks into /slothfs, yielding a full check out.
If there were symlinks to a previous checkout in the workspace, this will also update timestamps to make incremental builds work.
Advancing your checkout to a different timestamp uses the same commands. To sync to the current state of the Android tree, do the following
slothfs-populate -sync .
This fetches the latest manifest file, finds the project revisions, sets up a workspace for the manifest, and updates the symlinks from your read/write checkout.
Remove a workspace by removing its symlink configuration entry, eg.
rm /slothfs/config/my-workspace
A SlothFS daemon can be unmounted with fusermount
fusermount -u /slothfs
If this reports device busy, check if there any processes holding open files (eg. Jack compilation servers.)
The file system offers the following metadata files:
workspace/.slothfs/manifest.xml - manifest XML workspace/path/to/repo/.slothfs/tree.json - tree listing of this repository workspace/path/to/repo/.slothfs/treeID - hex tree ID of the this repository
In addition, each blob has the user.gitsha1 extended attribute that surfaces the blob's git SHA1 checksum.
SlothFS clones repositories on-demand, as soon as a file in the repository is opened. You can avoid cloning altogether for repositories you know you don't need; the files will then be fetched over HTTP.
Details of which repositories and files trigger cloning are configured through a JSON file.
For example, if you work on Android, and build on a Linux machine, you will never need the Darwin related prebuilts. You can avoid a costly clone for those by doing:
{"Repo": ".*darwin.*", "Clone": false}
Similarly, the build system system will read files (typically called ‘*.mk’) across the entire tree. When any .mk file is opened, this should not trigger a clone. This is achieved with the following entry
{"File": ".*mk$", "Clone": false}
Together, the following config.json file is a good start for working on android:
[{"Repo": ".*darwin.*", "Clone": false},
{"File": ".*mk$", "Clone": false}]
A more elaborate configuration file is included as android.json.
SlothFS loads the configuration data from a directory that can be ste with the -config flag. The following configuration is available
$HOME/.config/clone.json # clone configuration $HOME/.config/manifests/ # configured workspaces
SlothFS caches data in a directory which can be set with -cache flag. The following data are cached:
$HOME/.cache/slothfs/tree # trees $HOME/.cache/slothfs/git # bare git repositories $HOME/.cache/slothfs/blob # blobs
When syncing a checkout with slothfs-populate, an attempt is made to update timestamps of the blobs, so incremental builds will work as expected. However, since blobs are shared between different workspaces, a sync in one workspace may cause spurious rebuilds in other workspaces.
Similarly, interrupting slothfs-populate and then syncing to another workspace may yield unpredictable results.
When the slothfs FUSE daemon is restarted, all timestamp information is lost.