Goal

Minimize source control overhead for Android developers

Background

Android uses git repositories stitched together with ‘repo’. There are several hundred repos, with new ones added very frequently. Managing these (syncing, checking out) is a significant waste of time. There are avoidable inefficiencies, because the tree contains a significant amount of unused data for unused host platforms, unused target architectures/devices and unused history.

Requirements

  • Released as Open source
  • Runs on Linux and Mac
  • Both for automated and manual use
  • Build and create CLs for the Android tree with minimal overhead
  • Easily deployable

Idea

  • Provide a lightweight mechanism to create a read-only snapshot of the Android tree.
  • Provide tooling to check out some repositories as git, and create a symlink forest to the read-only snapshot for the rest.

Implementation

Overview:

  1. We provide a FUSE file system with R/O snapshots of the tree.
  2. We provide tooling to complete a partial checkout with symlinks to a R/O snapshot.
  3. We provide tooling to sync partial checkout, updating timestamps to satisfy the build system.

FUSE filesystem overview

Provide a FUSE file system for the R/O snapshot.

  • The FS is read-only, except for timestamps.
  • The FS uses hardlinks for blobs with the same content.
  • The FS populates metadata from the Gitiles JSON REST API.
  • When a file from some repo is opened for reading, fetch the content. A heuristic will decide between:
    • Full clone (useful for code repositories)
    • Shallow clone (prebuilts: history doesn’t matter, but need full tree)
    • Individual object downloads from Gitiles (don’t need full tree)
  • The FS runs as the user

The FS can be configured by symlinking manifest.xml or a submodule commit SHA1

  • ln -s ~/android/.repo/manifest.xml /fuse/config/WORKSPACE-NAME
  • ln -s ~/submod-android:master /fuse/config/WORKSPACE-NAME

Optional optimizations:

  • Do periodic git pre-fetches from the FS; minimizes wait time when issuing the sync command.
  • Initialize from an existing repo installation
  • Create an on-disk Content Addressable Store for source files to minimize time to unpack data on startup.

Tooling overview

  • Provide tooling to calculate diff between 2 trees. Based on this diff, call “touch” on the changed blobs.
  • Provide tooling to populate a workspace with symlink forest pointing to the FUSE snapshot, and checking out individual subrepos as normal .git repositories.

Motivation

Why OSX

This is a request from Android team. It actually makes providing a good solution more difficult, because OSX and OSXFUSE lack several features:

  • No attribute cache control
  • No kernel UnionFS or bind mounts.
  • OSXFUSE is buggy
  • OSXFUSE lacks performance optimizations (eg. readdirplus, zero roundtrip reads)

Why FUSE?

  • FUSE is the only way to avoid downloading data for unused files
  • You can also create trees cheaply by hard/soft linking a CAS directly. However, it is easy for users to accidentally edit files in the CAS, leading to build breakage.
  • This could be circumvented by asking users to run the CAS under a different UID, but that is bothersome to set up.

Why readonly?

Write access goes through the normal file system. This avoids surfacing a writable tree in FUSE.

  • Git performance would suffer if routed through FUSE
  • A writable git repo would have to be backed by some data on disk. Using a normal git repo is easiest for troubleshooting and for users to understand, but at the same time, the standard posix interface (which uses filenames) is ill-suited to implement a posixly correct file system. We sidestep this problem by not offering a writable tree.
  • Preventing writes prevents FS race conditions.
  • For a r/o FS, we can set infinite timeouts on attributes and entry data, minimizing kernel roundtrips.

Why writable timestamps?

We must support incremental builds, so syncs must lead to timestamp changes.

  • In OSXFUSE, the FS can’t invalidate attributes, so it is better to change timestamps from the outside.

Why hardlinks for blobs?

  • multiple checkouts of similar trees can share kernel page cache memory for the trees.
  • reading through FUSE is expensive. Sharing the blobs amortizes reading costs.
  • disadvantage: blobs are shared, so when setting timestamps for one workspace, other workspaces are affected too, leading to spurious rebuilds in those workspaces.

Why run FUSE as the user?

  • Simplifies credential management, in case the FS must contact authenticated services.
  • The FS doesn’t have to reason about the permissions and owners of the process opening some file.
  • Simplifies deployment. User does not need root privileges to use this.

Why use gitiles JSON API, and git wire protocol?

  • Gitiles + git wire protocol are already supported in open-source Gerrit. Zero deployment overhead for external users.
  • Git wire protocol is battle tested and well optimized for bulk transfers.
  • The gitiles JSON API is sufficient for what we want.

Open questions

  • How should we integrate this into existing tooling? (repo?)
  • Does ‘repo’ get confused when part of the checkout is a symlink forest?
  • Do we replace (reimplement) repo, or extend it?

Implementation steps

  • Gitiles:

    • Add support for ‘size’ field to tree listings. The ‘size’ field is necessary for serving FUSE data.
    • Add support for recursive tree traversals. This saves roundtrips.

  • FUSE daemon:

    • Add support for serving trees based on Gitiles JSON data
    • Add support for lazily cloning .git repositories
    • Add support for chtimes
    • Add support for sharing blobs
    • Add support for a CAS
    • Add support for surfacing manifest.xml with SHA1s.
    • Investigate bazil.org/fuse, which has better community support.

  • Tooling:

    • Generate composite workspace of symlink forest and git repos
    • Generate chtime() calls based on two manifest.xml (with SHA1s) or submodule SHA1s.
    • Write a sync command
    • Add ‘checkout’ command