docs/manual.md

Building Android with SlothFS
=============================

[Slothfs](https://github.com/google/slothfs) is a FUSE file system that offers a
read-only view of a Git tree. It downloads files lazily, so it takes up less
diskspace than a full checkout based on `repo`. Once its caches are seeded,
creating fresh workspaces or syncing should take on the order of seconds.

It is developed as open source, and can be used by anyone that uses
Gerrit/Gitiles based Git hosting. The [design doc](design.md) explains the
background behind its design choices.

It has been tested on Linux, but we expect it works on OSX too.

For the remainder of this document, we will assume that you are trying to
compile some flavor of Android.


Installing
==========

Get the source code,

    go get github.com/google/slothfs/cmd/slothfs-repofs

SlothFS depends git2go, which depends on libgit2.  For git2go, we recommend
compiling libgit2 in statically, as documented
[here](https://github.com/libgit2/git2go#from-next).

To install all components, run

    cd $GOPATH/src/github.com/google/slothfs/ && sh all.bash

The rest of this document assumes this has been done, and `$GOPATH/bin/` is in
your `$PATH`.

In addition, install the standard Android `clone.json` to avoid unnecessary git
clones

    mkdir -p $HOME/.config/slothfs
    curl https://raw.githubusercontent.com/google/slothfs/master/android.json \
      >  $HOME/.config/slothfs/clone.json


Selecting the host
==================

The defaults of slothfs are for the public version of Android at
`android.googlesource.com`.  Set the `-gitiles_url` option to change against
which version of Android you want to run.


Mounting the filesystem
=======================

First, create a mountpoint:

    sudo mkdir -p /slothfs && sudo chown $USER /slothfs

Then, to mount the file system, run

    slothfs-repofs /slothfs


Dereferencing a manifest
========================

The manifest describes which repositories go into an Android checkout. To make a
file system out of this, we must decide at which exact revision each repository
should be offered.

This can be done with `slothfs-deref-manifest`, eg.

    slothfs-deref-manifest > /tmp/m.xml


Configuring a workspace
=======================

A workspace can be created by a symlinking a deferenced manifest field into the
`config` directory, i.e.

    ln -s /tmp/m.xml /slothfs/config/my-workspace

This should create a directory `/slothfs/my-workspace` holding the tree
described in `/tmp/m.xml`.

On the first time you do this, slothfs will have to fetch the tree data, which
is slow, so this might take a while.


Using a workspace
=================

To use the read-only tree, create a writable checkout. Let's assume we want to
change the `art` repo:

    mkdir -p checkout ; cd checkout
    git clone https://android.googlesource.com/platform/art

To make `checkout` into a full-fledged checkout, run

    slothfs-populate -ro /slothfs/my-workspace .

This populate the `checkout` directory with symlinks into `/slothfs`, yielding a
full check out.

If there were symlinks to a previous checkout in the workspace, this will also
update timestamps to make incremental builds work.


Syncing
=======

Advancing your checkout to a different timestamp uses the same commands. To sync
to the current state of the Android tree, do the following

    slothfs-populate -sync .

This fetches the latest manifest file, finds the project revisions, sets up a
workspace for the manifest, and updates the symlinks from your read/write
checkout.


Removing a workspace
====================

Remove a workspace by removing its symlink configuration entry, eg.

    rm /slothfs/config/my-workspace

Unmounting slothfs
==================

A SlothFS daemon can be unmounted with `fusermount`

     fusermount -u /slothfs

If this reports `device busy`, check if there any processes holding open files
(eg. Jack compilation servers.)


Metadata
========

The file system offers the following metadata files:

     workspace/.slothfs/manifest.xml - manifest XML

     workspace/path/to/repo/.slothfs/tree.json - tree listing of this repository
     workspace/path/to/repo/.slothfs/treeID - hex tree ID of the this repository

In addition, each blob has the `user.gitsha1` extended attribute that surfaces
the blob's git SHA1 checksum.


Configuring
===========

SlothFS clones repositories on-demand, as soon as a file in the repository is
opened. You can avoid cloning altogether for repositories you know you don't
need; the files will then be fetched over HTTP.

Details of which repositories and files trigger cloning are configured through a
JSON file.

For example, if you work on Android, and build on a Linux machine, you will
never need the Darwin related prebuilts. You can avoid a costly clone for those
by doing:

    {"Repo": ".*darwin.*", "Clone": false}

Similarly, the build system system will read files (typically called '*.mk')
across the entire tree. When any .mk file is opened, this should not trigger a
clone. This is achieved with the following entry

    {"File": ".*mk$", "Clone": false}

Together, the following `config.json` file is a good start for working on
android:

    [{"Repo": ".*darwin.*", "Clone": false},
     {"File": ".*mk$", "Clone": false}]

A more elaborate configuration file is included as `android.json`.


File layout
-----------

SlothFS loads the configuration data from a directory that can be ste with the
`-config` flag. The following configuration is available

    $HOME/.config/clone.json   # clone configuration
    $HOME/.config/manifests/   # configured workspaces

SlothFS caches data in a directory which can be set with `-cache` flag.
The following data are cached:

    $HOME/.cache/slothfs/tree  # trees
    $HOME/.cache/slothfs/git   # bare git repositories
    $HOME/.cache/slothfs/blob  # blobs


Caveats: timestamps
-------------------

When syncing a checkout with `slothfs-populate`, an attempt is made to update
timestamps of the blobs, so incremental builds will work as expected. However,
since blobs are shared between different workspaces, a sync in one workspace may
cause spurious rebuilds in other workspaces.

Similarly, interrupting `slothfs-populate` and then syncing to another workspace
may yield unpredictable results.

When the slothfs FUSE daemon is restarted, all timestamp information is lost.