Building Android with SlothFS

Slothfs is a FUSE file system that offers a read-only view of a Git tree. It downloads files lazily, so it takes up less diskspace than a full checkout based on repo. Once its caches are seeded, creating fresh workspaces or syncing should take on the order of seconds.

It is developed as open source, and can be used by anyone that uses Gerrit/Gitiles based Git hosting. The design doc explains the background behind its design choices.

It has been tested on Linux, but we expect it works on OSX too.

For the remainder of this document, we will assume that you are trying to compile some flavor of Android.

Installing

Get the source code,

go get github.com/google/slothfs/cmd/slothfs-multifs

SlothFS depends git2go, which depends on libgit2. For git2go, we recommend compiling libgit2 in statically, as documented here.

To install all components, run

cd $GOPATH/src/github.com/google/slothfs/ && sh all.bash

The rest of this document assumes this has been done, and $GOPATH/bin/ is in your $PATH.

In addition, install the standard Android clone.json to avoid unnecessary git clones

mkdir -p $HOME/.config/slothfs
curl https://raw.githubusercontent.com/google/slothfs/master/android.json \
  >  $HOME/.config/slothfs/clone.json

Selecting the host

The defaults of slothfs are for the public version of Android at android.googlesource.com. Set the -gitiles_url option to change against which version of Android you want to run.

Mounting the filesystem

First, create a mountpoint:

sudo mkdir -p /slothfs && sudo chown $USER /slothfs

Then, to mount the file system, run

slothfs-multifs /slothfs

Dereferencing a manifest

The manifest describes which repositories go into an Android checkout. To make a file system out of this, we must decide at which exact revision each repository should be offered.

This can be done with slothfs-deref-manifest, eg.

slothfs-deref-manifest > /tmp/m.xml

Configuring a workspace

A workspace can be created by a symlinking a deferenced manifest field into the config directory, i.e.

ln -s /tmp/m.xml /slothfs/config/my-workspace

This should create a directory /slothfs/my-workspace holding the tree described in /tmp/m.xml.

On the first time you do this, slothfs will have to fetch the tree data, which is slow, so this might take a while.

Using a workspace

To use the read-only tree, create a writable checkout. Let's assume we want to change the art repo:

mkdir -p checkout ; cd checkout
git clone https://android.googlesource.com/platform/art

To make checkout into a full-fledged checkout, run

slothfs-populate -ro /slothfs/my-workspace .

This populate the checkout directory with symlinks into /slothfs, yielding a full check out.

If there were symlinks to a previous checkout in the workspace, this will also update timestamps to make incremental builds work.

Syncing

Advancing your checkout to a different timestamp uses the same commands. To sync to the current state of the Android tree, do the following

slothfs-populate -sync .

This fetches the latest manifest file, finds the project revisions, sets up a workspace for the manifest, and updates the symlinks from your read/write checkout.

Removing a workspace

Remove a workspace by removing its symlink configuration entry, eg.

rm /slothfs/config/my-workspace

Unmounting slothfs

A SlothFS daemon can be unmounted with fusermount

 fusermount -u /slothfs

If this reports device busy, check if there any processes holding open files (eg. Jack compilation servers.)

Metadata

The file system offers the following metadata files:

 workspace/.slothfs/manifest.xml - manifest XML

 workspace/path/to/repo/.slothfs/tree.json - tree listing of this repository
 workspace/path/to/repo/.slothfs/treeID - hex tree ID of the this repository

In addition, each blob has the user.gitsha1 extended attribute that surfaces the blob's git SHA1 checksum.

Configuring

SlothFS clones repositories on-demand, as soon as a file in the repository is opened. You can avoid cloning altogether for repositories you know you don't need; the files will then be fetched over HTTP.

Details of which repositories and files trigger cloning are configured through a JSON file.

For example, if you work on Android, and build on a Linux machine, you will never need the Darwin related prebuilts. You can avoid a costly clone for those by doing:

{"Repo": ".*darwin.*", "Clone": false}

Similarly, the build system system will read files (typically called ‘*.mk’) across the entire tree. When any .mk file is opened, this should not trigger a clone. This is achieved with the following entry

{"File": ".*mk$", "Clone": false}

Together, the following config.json file is a good start for working on android:

[{"Repo": ".*darwin.*", "Clone": false},
 {"File": ".*mk$", "Clone": false}]

A more elaborate configuration file is included as android.json.

File layout

SlothFS loads the configuration data from a directory that can be ste with the -config flag. The following configuration is available

$HOME/.config/clone.json   # clone configuration
$HOME/.config/manifests/   # configured workspaces

SlothFS caches data in a directory which can be set with -cache flag. The following data are cached:

$HOME/.cache/slothfs/tree  # trees
$HOME/.cache/slothfs/git   # bare git repositories
$HOME/.cache/slothfs/blob  # blobs

Caveats: timestamps

When syncing a checkout with slothfs-populate, an attempt is made to update timestamps of the blobs, so incremental builds will work as expected. However, since blobs are shared between different workspaces, a sync in one workspace may cause spurious rebuilds in other workspaces.

Similarly, interrupting slothfs-populate and then syncing to another workspace may yield unpredictable results.

When the slothfs FUSE daemon is restarted, all timestamp information is lost.