Expand documentation. Add design doc. Change-Id: Ieafc33436f5de14aa59f440b63068e9844fd03ef
diff --git a/CONTRIBUTING b/CONTRIBUTING index cd0c2db..655bb1f 100644 --- a/CONTRIBUTING +++ b/CONTRIBUTING
@@ -1,4 +1,6 @@ -For contributions, we need a CL, and generally follow the process -documented at +For contributions, we need a change in Gerrit, and generally follow +the process documented at https://gerrit-review.googlesource.com/Documentation/dev-contributing.html + +As a prerequisite, we will need also Google CLA on file.
diff --git a/README.md b/README.md index daeccd4..b36c4f9 100644 --- a/README.md +++ b/README.md
@@ -1,24 +1,35 @@ -This is a FUSE filesystem that provides light-weight, read-only checkouts of -Android. +This is a FUSE filesystem that provides light-weight, read-only checkouts of Git +repositories. It is intended for use with Android. How to use ========== -To start, +To start the file system: - go install github.com/google/gitfs/cmd/gitfs-{multifs,expand-manifest} - gitfs-expand-manifest --gitiles https://android.googlesource.com/ \ - > /tmp/m.xml + go install github.com/google/gitfs/cmd/gitfs-multifs mkdir /tmp/mnt gitfs-multifs -cache /tmp/cache -gitiles https://android.googlesource.com/ /tmp/mnt & -then, in another terminal, execute +To create a workspace "ws" corresponding to the latest manifest version + go install github.com/google/gitfs/cmd/gitfs-expand-manifest + gitfs-expand-manifest --gitiles https://android.googlesource.com/ \ + > /tmp/m.xml && ln -s /tmp/m.xml /tmp/mnt/config/ws -To create a workspace "ws" corresponding to the manifest in m.xml. +To populate a checkout + + go install github.com/google/gitfs/cmd/gitfs-populate + mkdir -p checkout/frameworks + cd checkout/frameworks + git clone https://android.googlesource.com/platform/frameworks/base + cd ../ + gitfs-populate -ro /tmp/mnt/ws . + +The filesystem daemon uses an on-disk cache, which by default is stored under +~/.cache/gitfs Configuring @@ -47,6 +58,7 @@ {"File": ".*mk$", "Clone": true}] + DISCLAIMER ==========
diff --git a/design.md b/design.md new file mode 100644 index 0000000..cf51acb --- /dev/null +++ b/design.md
@@ -0,0 +1,166 @@ + +Goal +==== +Minimize source control overhead for Android developers + + +Background +========== + +Android uses git repositories stitched together with ‘repo’. There are several +hundred repos, with new ones added very frequently. Managing these (syncing, +checking out) is a significant waste of time. There are avoidable +inefficiencies, because the tree contains a significant amount of unused data +for unused host platforms, unused target architectures/devices and unused +history. + +Requirements +============ + +* Released as Open source +* Runs on Linux and Mac +* Both for automated and manual use +* Build and create CLs for the Android tree with minimal overhead +* Easily deployable + +Idea +==== +* Provide a lightweight mechanism to create a read-only snapshot of the Android tree. +* Provide tooling to check out some repositories as git, and create a symlink + forest to the read-only snapshot for the rest. + +Implementation +============== + +Overview: + +1. We provide a FUSE file system with R/O snapshots of the tree. +2. We provide tooling to complete a partial checkout with symlinks to a R/O + snapshot. +3. We provide tooling to sync partial checkout, updating timestamps to + satisfy the build system. + +FUSE filesystem overview +------------------------ + +Provide a FUSE file system for the R/O snapshot. + + * The FS is read-only, except for timestamps. + * The FS uses hardlinks for blobs with the same content. + * The FS populates metadata from the Gitiles JSON REST API. + * When a file from some repo is opened for reading, fetch the content. A heuristic will decide between: + * Full clone (useful for code repositories) + * Shallow clone (prebuilts: history doesn’t matter, but need full tree) + * Individual object downloads from Gitiles (don’t need full tree) + * The FS runs as the user + +The FS can be configured by symlinking manifest.xml or a submodule commit SHA1 + * ln -s ~/android/.repo/manifest.xml /fuse/config/WORKSPACE-NAME + * ln -s ~/submod-android:master /fuse/config/WORKSPACE-NAME + +Optional optimizations: + + * Do periodic git pre-fetches from the FS; minimizes wait time when issuing the sync command. + * Initialize from an existing repo installation + * Create an on-disk Content Addressable Store for source files to minimize time to unpack data on startup. + + +Tooling overview +---------------- +* Provide tooling to calculate diff between 2 trees. Based on this diff, call “touch” on the changed blobs. +* Provide tooling to populate a workspace with symlink forest pointing to the FUSE snapshot, and checking out individual subrepos as normal .git repositories. + + +Motivation +========== + +Why OSX +------- + +This is a request from Android team. It actually makes providing a good solution +more difficult, because OSX and OSXFUSE lack several features: + + * No attribute cache control + * No kernel UnionFS or bind mounts. + * OSXFUSE is buggy + * OSXFUSE lacks performance optimizations (eg. readdirplus, zero roundtrip reads) + +Why FUSE? +--------- + + * FUSE is the only way to avoid downloading data for unused files + * You can also create trees cheaply by hard/soft linking a CAS directly. However, it is easy for users to accidentally edit files in the CAS, leading to build breakage. + * This could be circumvented by asking users to run the CAS under a different UID, but that is bothersome to set up. + +Why readonly? +------------- + +Write access goes through the normal file system. This avoids surfacing a +writable tree in FUSE. + + * Git performance would suffer if routed through FUSE + * A writable git repo would have to be backed by some data on disk. Using a normal git repo is easiest for troubleshooting and for users to understand, but at the same time, the standard posix interface (which uses filenames) is ill-suited to implement a posixly correct file system. We sidestep this problem by not offering a writable tree. + * Preventing writes prevents FS race conditions. + * For a r/o FS, we can set infinite timeouts on attributes and entry data, + minimizing kernel roundtrips. + +Why writable timestamps? +------------------------ + +We must support incremental builds, so syncs must lead to timestamp changes. + + * In OSXFUSE, the FS can’t invalidate attributes, so it is better to change timestamps from the outside. + +Why hardlinks for blobs? +------------------------ + + * multiple checkouts of similar trees can share kernel page cache memory for + the trees. + * reading through FUSE is expensive. Sharing the blobs amortizes reading costs. + * disadvantage: blobs are shared, so when setting timestamps for one + workspace, other workspaces are affected too, leading to spurious rebuilds + in those workspaces. + +Why run FUSE as the user? +------------------------- + + * Simplifies credential management, in case the FS must contact authenticated services. + * The FS doesn’t have to reason about the permissions and owners of the process opening some file. + * Simplifies deployment. User does not need root privileges to use this. + + +Why use gitiles JSON API, and git wire protocol? +------------------------------------------------ + + * Gitiles + git wire protocol are already supported in open-source Gerrit. Zero deployment overhead for external users. + * Git wire protocol is battle tested and well optimized for bulk transfers. + * The gitiles JSON API is sufficient for what we want. + +Open questions +============== + +* How should we integrate this into existing tooling? (repo?) +* Does ‘repo’ get confused when part of the checkout is a symlink forest? +* Do we replace (reimplement) repo, or extend it? + +Implementation steps +==================== + +* Gitiles: + * Add support for ‘size’ field to tree listings. The ‘size’ field is necessary for serving FUSE data. + * Add support for recursive tree traversals. This saves roundtrips. + +* FUSE daemon: + * Add support for serving trees based on Gitiles JSON data + * Add support for lazily cloning .git repositories in gitfs + * Add support for chtimes + * Add support for sharing blobs + * Add support for a CAS + * Add support for surfacing manifest.xml with SHA1s. + * Investigate bazil.org/fuse, which has better community support. + +* Tooling: + * Generate composite workspace of symlink forest and git repos + * Generate chtime() calls based on two manifest.xml (with SHA1s) or submodule SHA1s. + * Write a sync command + * Add ‘checkout’ command