| |
| Goal |
| ==== |
| Minimize source control overhead for Android developers |
| |
| |
| Background |
| ========== |
| |
| Android uses git repositories stitched together with ‘repo’. There are several |
| hundred repos, with new ones added very frequently. Managing these (syncing, |
| checking out) is a significant waste of time. There are avoidable |
| inefficiencies, because the tree contains a significant amount of unused data |
| for unused host platforms, unused target architectures/devices and unused |
| history. |
| |
| Requirements |
| ============ |
| |
| * Released as Open source |
| * Runs on Linux and Mac |
| * Both for automated and manual use |
| * Build and create CLs for the Android tree with minimal overhead |
| * Easily deployable |
| |
| Idea |
| ==== |
| * Provide a lightweight mechanism to create a read-only snapshot of the Android tree. |
| * Provide tooling to check out some repositories as git, and create a symlink |
| forest to the read-only snapshot for the rest. |
| |
| Implementation |
| ============== |
| |
| Overview: |
| |
| 1. We provide a FUSE file system with R/O snapshots of the tree. |
| 2. We provide tooling to complete a partial checkout with symlinks to a R/O |
| snapshot. |
| 3. We provide tooling to sync partial checkout, updating timestamps to |
| satisfy the build system. |
| |
| FUSE filesystem overview |
| ------------------------ |
| |
| Provide a FUSE file system for the R/O snapshot. |
| |
| * The FS is read-only, except for timestamps. |
| * The FS uses hardlinks for blobs with the same content. |
| * The FS populates metadata from the Gitiles JSON REST API. |
| * When a file from some repo is opened for reading, fetch the content. A heuristic will decide between: |
| * Full clone (useful for code repositories) |
| * Shallow clone (prebuilts: history doesn’t matter, but need full tree) |
| * Individual object downloads from Gitiles (don’t need full tree) |
| * The FS runs as the user |
| |
| The FS can be configured by symlinking manifest.xml or a submodule commit SHA1 |
| * ln -s ~/android/.repo/manifest.xml /fuse/config/WORKSPACE-NAME |
| * ln -s ~/submod-android:master /fuse/config/WORKSPACE-NAME |
| |
| Optional optimizations: |
| |
| * Do periodic git pre-fetches from the FS; minimizes wait time when issuing the sync command. |
| * Initialize from an existing repo installation |
| * Create an on-disk Content Addressable Store for source files to minimize time to unpack data on startup. |
| |
| |
| Tooling overview |
| ---------------- |
| * Provide tooling to calculate diff between 2 trees. Based on this diff, call “touch” on the changed blobs. |
| * Provide tooling to populate a workspace with symlink forest pointing to the FUSE snapshot, and checking out individual subrepos as normal .git repositories. |
| |
| |
| Motivation |
| ========== |
| |
| Why OSX |
| ------- |
| |
| This is a request from Android team. It actually makes providing a good solution |
| more difficult, because OSX and OSXFUSE lack several features: |
| |
| * No attribute cache control |
| * No kernel UnionFS or bind mounts. |
| * OSXFUSE is buggy |
| * OSXFUSE lacks performance optimizations (eg. readdirplus, zero roundtrip reads) |
| |
| Why FUSE? |
| --------- |
| |
| * FUSE is the only way to avoid downloading data for unused files |
| * You can also create trees cheaply by hard/soft linking a CAS directly. However, it is easy for users to accidentally edit files in the CAS, leading to build breakage. |
| * This could be circumvented by asking users to run the CAS under a different UID, but that is bothersome to set up. |
| |
| Why readonly? |
| ------------- |
| |
| Write access goes through the normal file system. This avoids surfacing a |
| writable tree in FUSE. |
| |
| * Git performance would suffer if routed through FUSE |
| * A writable git repo would have to be backed by some data on disk. Using a normal git repo is easiest for troubleshooting and for users to understand, but at the same time, the standard posix interface (which uses filenames) is ill-suited to implement a posixly correct file system. We sidestep this problem by not offering a writable tree. |
| * Preventing writes prevents FS race conditions. |
| * For a r/o FS, we can set infinite timeouts on attributes and entry data, |
| minimizing kernel roundtrips. |
| |
| Why writable timestamps? |
| ------------------------ |
| |
| We must support incremental builds, so syncs must lead to timestamp changes. |
| |
| * In OSXFUSE, the FS can’t invalidate attributes, so it is better to change timestamps from the outside. |
| |
| Why hardlinks for blobs? |
| ------------------------ |
| |
| * multiple checkouts of similar trees can share kernel page cache memory for |
| the trees. |
| * reading through FUSE is expensive. Sharing the blobs amortizes reading costs. |
| * disadvantage: blobs are shared, so when setting timestamps for one |
| workspace, other workspaces are affected too, leading to spurious rebuilds |
| in those workspaces. |
| |
| Why run FUSE as the user? |
| ------------------------- |
| |
| * Simplifies credential management, in case the FS must contact authenticated services. |
| * The FS doesn’t have to reason about the permissions and owners of the process opening some file. |
| * Simplifies deployment. User does not need root privileges to use this. |
| |
| |
| Why use gitiles JSON API, and git wire protocol? |
| ------------------------------------------------ |
| |
| * Gitiles + git wire protocol are already supported in open-source Gerrit. Zero deployment overhead for external users. |
| * Git wire protocol is battle tested and well optimized for bulk transfers. |
| * The gitiles JSON API is sufficient for what we want. |
| |
| Open questions |
| ============== |
| |
| * How should we integrate this into existing tooling? (repo?) |
| * Does ‘repo’ get confused when part of the checkout is a symlink forest? |
| * Do we replace (reimplement) repo, or extend it? |
| |
| Implementation steps |
| ==================== |
| |
| * Gitiles: |
| * Add support for ‘size’ field to tree listings. The ‘size’ field is necessary for serving FUSE data. |
| * Add support for recursive tree traversals. This saves roundtrips. |
| |
| * FUSE daemon: |
| * Add support for serving trees based on Gitiles JSON data |
| * Add support for lazily cloning .git repositories |
| * Add support for chtimes |
| * Add support for sharing blobs |
| * Add support for a CAS |
| * Add support for surfacing manifest.xml with SHA1s. |
| * Investigate bazil.org/fuse, which has better community support. |
| |
| * Tooling: |
| * Generate composite workspace of symlink forest and git repos |
| * Generate chtime() calls based on two manifest.xml (with SHA1s) or submodule SHA1s. |
| * Write a sync command |
| * Add ‘checkout’ command |
