Add more documentation. Change-Id: I82159ee4741ed3a2997f09aeb520ee1efe239319
diff --git a/README.md b/README.md index 2aefb36..7ad8800 100644 --- a/README.md +++ b/README.md
@@ -1,6 +1,7 @@ -This is a FUSE filesystem that provides light-weight, lazily downloaded, -read-only checkouts of Git repositories. It is intended for use with Android. +SlothFS is a FUSE filesystem that provides light-weight, lazily downloaded, +read-only checkouts of manifest-based Git projects. It is intended for use with +Android. How to use @@ -10,7 +11,7 @@ go install github.com/google/slothfs/cmd/slothfs-multifs mkdir /tmp/mnt - slothfs-multifs -gitiles https://android.googlesource.com/ /tmp/mnt & + slothfs-multifs -gitiles https://android.googlesource.com/ /tmp/mnt & To create a workspace "ws" corresponding to the latest manifest version @@ -19,47 +20,7 @@ > /tmp/m.xml && ln -s /tmp/m.xml /tmp/mnt/config/ws -To populate a checkout - - go install github.com/google/slothfs/cmd/slothfs-populate - mkdir -p checkout/frameworks - cd checkout/frameworks - git clone https://android.googlesource.com/platform/frameworks/base - cd ../ - slothfs-populate -ro /tmp/mnt/ws . - -The filesystem daemon uses an on-disk cache, which by default is stored under -~/.cache/slothfs - - -Configuring -=========== - -The FUSE file system clones repositories on-demand. You can avoid cloning -altogether for repositories you know you don't need. This is configured through -a JSON file. - -For example, if you work on Android, and build on a Linux machine, you will -never need the Darwin related prebuilts. You can avoid a costly clone for those -by doing: - - {"Repo": ".*darwin.*", "Clone": false} - -Similarly, the build system system will read files (typically called '*.mk') -across the entire tree. When any .mk file is opened, this should not trigger a -clone. This is achieved with the following entry - - {"File": ".*mk$", "Clone": false} - -Together, the following `config.json` file is a good start for working on -android: - - [{"Repo": ".*darwin.*", "Clone": false}, - {"File": ".*mk$", "Clone": false}] - -A more elaborate configuration file is included as `android.json`. - -By default, slothfs loads the configuration from =$HOME/.config/slothfs/clone.json=. +More details can be found in the [manual](docs/manual.md). DISCLAIMER
diff --git a/design.md b/docs/design.md similarity index 100% rename from design.md rename to docs/design.md
diff --git a/docs/manual.md b/docs/manual.md new file mode 100644 index 0000000..8ec7e2c --- /dev/null +++ b/docs/manual.md
@@ -0,0 +1,202 @@ + +Building Android with SlothFS +============================= + +[Slothfs](https://github.com/google/slothfs) is a FUSE file system that offers a +read-only view of a Git tree. It downloads files lazily, so it takes up less +diskspace than a full checkout based on `repo`. Once its caches are seeded, +creating fresh workspaces or syncing should take on the order of seconds. + +It is developed as open source, and can be used by anyone that uses +Gerrit/Gitiles based Git hosting. The [design doc](design.md) explains the +background behind its design choices. + +It has been tested on Linux, but we expect it works on OSX too. + +For the remainder of this document, we will assume that you are trying to +compile some flavor of Android. + + +Installing +========== + +Run `all.bash` to compile, test and install all components. The rest of this +document assumes this has been done, and `$GOPATH/bin/` is in your `$PATH`. + +In addition, install the standard Android `clone.json` to avoid unnecessary git +clones + + mkdir -p $HOME/.config/slothfs + curl https://raw.githubusercontent.com/google/slothfs/master/android.json \ + > $HOME/.config/slothfs/clone.json + + +Selecting the host +================== + +The defaults of slothfs are for the public version of Android at +`android.googlesource.com`. Set the `-gitiles_url` option to change against +which version of Android you want to run. + + +Dereferencing a manifest +======================== + +The manifest describes which repositories go into an Android checkout. To make a +file system out of this, we must decide at which exact revision each repository +should be offered. + +This can be done with `slothfs-expand-manifest`, eg. + + slothfs-expand-manifest > /tmp/m.xml + + +Mounting the filesystem +======================= + +First, create a mountpoint: + + sudo mkdir -p /slothfs && sudo chown $USER /slothfs + +Then, to mount the file system, run + + slothfs-multifs /slothfs + + +Configuring a workspace +======================= + +A workspace can be created by a symlinking a deferenced manifest field into the +`config` directory, i.e. + + ln -s /tmp/m.xml /slothfs/config/my-workspace + +This should create a directory `/slothfs/my-workspace` holding the tree +described in `/tmp/m.xml`. + + +Using a workspace +================= + +To use the read-only tree, create a writable checkout. Let's assume we want to +change the `art` repo: + + mkdir -p checkout ; cd checkout + git clone https://android.googlesource.com/platform/art + +To make `checkout` into a full-fledged checkout, run + + slothfs-populate -ro /slothfs/my-workspace . + +This populate the `checkout` directory with symlinks into `/slothfs`, yielding a +full check out. + + +Syncing +======= + +Advancing your checkout to a different timestamp uses the same commands. To sync +to the current state of the Android tree, do the following + + SYNC=$(date -Iminutes) + slothfs-expand-manifest > /tmp/${SYNC}.xml + ln -s /tmp/${SYNC}.xml /slothfs/config/${SYNC} + slothfs-populate -ro /slothfs/${SYNC} . + +this will reroute symlinks in your check to your newly created +workspace. `slothfs-populate` will also update timestamps so incremental builds +keep working. + + +Removing a workspace +==================== + +Remove a workspace by removing its symlink configuration entry, eg. + + rm /slothfs/config/my-workspace + +Unmounting slothfs +================== + +A SlothFS daemon can be unmounted with `fusermount` + + fusermount -u /slothfs + +If this reports `device busy`, check if there any processes holding open files +(eg. Jack compilation servers.) + + +Metadata +======== + +The file system offers the following metadata files: + + workspace/.slothfs/manifest.xml - manifest XML + + workspace/path/to/repo/.slothfs/tree.json - tree listing of this repository + workspace/path/to/repo/.slothfs/treeID - hex tree ID of the this repository + +In addition, each blob has the `user.gitsha1` extended attribute that surfaces +the blob's git SHA1 checksum. + + +Configuring +=========== + +SlothFS clones repositories on-demand, as soon as a file in the repository is +opened. You can avoid cloning altogether for repositories you know you don't +need; the files will then be fetched over HTTP. + +Details of which repositories and files trigger cloning are configured through a +JSON file. + +For example, if you work on Android, and build on a Linux machine, you will +never need the Darwin related prebuilts. You can avoid a costly clone for those +by doing: + + {"Repo": ".*darwin.*", "Clone": false} + +Similarly, the build system system will read files (typically called '*.mk') +across the entire tree. When any .mk file is opened, this should not trigger a +clone. This is achieved with the following entry + + {"File": ".*mk$", "Clone": false} + +Together, the following `config.json` file is a good start for working on +android: + + [{"Repo": ".*darwin.*", "Clone": false}, + {"File": ".*mk$", "Clone": false}] + +A more elaborate configuration file is included as `android.json`. + + +File layout +----------- + +SlothFS loads the configuration data from a directory that can be ste with the +`-config` flag. The following configuration is available + + $HOME/.config/clone.json # clone configuration + $HOME/.config/manifests/ # configured workspaces + +SlothFS caches data in a directory which can be set with `-cache` flag. +The following data are cached: + + $HOME/.cache/slothfs/tree # trees + $HOME/.cache/slothfs/git # bare git repositories + $HOME/.cache/slothfs/blob # blobs + + +Caveats: timestamps +------------------- + +When syncing a checkout with `slothfs-populate`, an attempt is made to update +timestamps of the blobs, so incremental builds will work as expected. However, +since blobs are shared between different workspaces, a sync in one workspace may +cause spurious rebuilds in other workspaces. + +Similarly, interrupting `slothfs-populate` and then syncing to another workspace +may yield unpredictable results. + +When the slothfs FUSE daemon is restarted, all timestamp information is lost.