| # AOSP Preupload Hooks |
| |
| This repo holds hooks that get run by [repo] during the upload phase. They |
| perform various checks automatically such as running linters on your code. |
| |
| Note: Currently all hooks are disabled by default. Each repo must explicitly |
| turn on any hook it wishes to enforce. |
| |
| [TOC] |
| |
| ## Usage |
| |
| Normally these execute automatically when you run `repo upload`. If you want to |
| run them by hand, you can execute `pre-upload.py` directly. By default, that |
| will scan the active repo and process all commits that haven't yet been merged. |
| See its help for more info. |
| |
| ### Bypassing |
| |
| Sometimes you might want to bypass the upload checks. While this is **strongly |
| discouraged** (often failures you add will affect others and block them too), |
| sometimes there are valid reasons for this. You can simply use the option |
| `--ignore-hooks` when running `repo upload` to ignore all hook errors. |
| This will ignore **all** hook errors and not just specific ones. |
| |
| # Config Files |
| |
| There are two types of config files: |
| * Repo project-wide settings (e.g. all of AOSP). These set up defaults for all |
| projects that are checked out via a single manifest. |
| * Project-local settings (e.g. a single .git repo). These control settings for |
| the local project you're working on. |
| |
| The merging of these config files control the hooks/checks that get run when |
| running `repo upload`. |
| |
| ## GLOBAL-PREUPLOAD.cfg |
| |
| These are the manifest-wide defaults and can be located in two places: |
| * `.repo/manifests/GLOBAL-PREUPLOAD.cfg`: The manifest git repo. |
| Simply check this in to the manifest git repo and you're done. |
| * `GLOBAL-PREUPLOAD.cfg`: The top level of the repo checkout. |
| For manifests that don't have a project checked out at the top level, |
| you can use repo's `<copyfile>` directive. |
| |
| These config files will be loaded first before stacking `PREUPLOAD.cfg` |
| settings on top. |
| |
| ## PREUPLOAD.cfg |
| |
| This file is checked in the top of a specific git repository. Stacking them |
| in subdirectories (to try and override parent settings) is not supported. |
| |
| ## Example |
| |
| ``` |
| # Per-project `repo upload` hook settings. |
| # https://android.googlesource.com/platform/tools/repohooks |
| |
| [Options] |
| ignore_merged_commits = true |
| |
| [Hook Scripts] |
| name = script --with args ${PREUPLOAD_FILES} |
| |
| [Builtin Hooks] |
| cpplint = true |
| |
| [Builtin Hooks Options] |
| cpplint = --filter=-x ${PREUPLOAD_FILES} |
| |
| [Tool Paths] |
| clang-format = /usr/bin/clang-format |
| ``` |
| |
| ## Environment |
| |
| Hooks are executed in the top directory of the git repository. All paths should |
| generally be relative to that point. |
| |
| A few environment variables are set so scripts don't need to discover things. |
| |
| * `REPO_PROJECT`: The name of the project. |
| e.g. `platform/tools/repohooks` |
| * `REPO_PATH`: The path to the project relative to the root. |
| e.g. `tools/repohooks` |
| * `REPO_REMOTE`: The name of the git remote. |
| e.g. `aosp`. |
| * `REPO_LREV`: The name of the remote revision, translated to a local tracking |
| branch. This is typically latest commit in the remote-tracking branch. |
| e.g. `ec044d3e9b608ce275f02092f86810a3ba13834e` |
| * `REPO_RREV`: The remote revision. |
| e.g. `master` |
| * `PREUPLOAD_COMMIT`: The commit that is currently being checked. |
| e.g. `1f89dce0468448fa36f632d2fc52175cd6940a91` |
| |
| ## Placeholders |
| |
| A few keywords are recognized to pass down settings. These are **not** |
| environment variables, but are expanded inline. Files with whitespace and |
| such will be expanded correctly via argument positions, so do not try to |
| force your own quote handling. |
| |
| * `${PREUPLOAD_FILES}`: List of files to operate on. |
| * `${PREUPLOAD_FILES_PREFIXED}`: A list of files to operate on. |
| Any string preceding/attached to the keyword ${PREUPLOAD_FILES_PREFIXED} |
| will be repeated for each file automatically. If no string is preceding/attached |
| to the keyword, the previous argument will be repeated before each file. |
| * `${PREUPLOAD_COMMIT}`: Commit hash. |
| * `${PREUPLOAD_COMMIT_MESSAGE}`: Commit message. |
| |
| Some variables are available to make it easier to handle OS differences. These |
| are automatically expanded for you: |
| |
| * `${REPO_PATH}`: The path to the project relative to the root. |
| e.g. `tools/repohooks` |
| * `${REPO_PROJECT}`: The name of the project. |
| e.g. `platform/tools/repohooks` |
| * `${REPO_ROOT}`: The absolute path of the root of the repo checkout. If the |
| project is in a submanifest, this points to the root of the submanifest. |
| * `${REPO_OUTER_ROOT}`: The absolute path of the root of the repo checkout. |
| This always points to the root of the overall repo checkout. |
| * `${BUILD_OS}`: The string `darwin-x86` for macOS and the string `linux-x86` |
| for Linux/x86. |
| |
| ### Examples |
| |
| Here are some examples of using the placeholders. |
| Consider this sample config file. |
| ``` |
| [Hook Scripts] |
| lister = ls ${PREUPLOAD_FILES} |
| checker prefix = check --file=${PREUPLOAD_FILES_PREFIXED} |
| checker flag = check --file ${PREUPLOAD_FILES_PREFIXED} |
| ``` |
| |
| With a commit that changes `path1/file1` and `path2/file2`, then this will run |
| programs with the arguments: |
| * `['ls', 'path1/file1', 'path2/file2']` |
| * `['check', '--file=path1/file1', '--file=path2/file2']` |
| * `['check', '--file', 'path1/file1', '--file', 'path2/file2']` |
| |
| ## [Options] |
| |
| This section allows for setting options that affect the overall behavior of the |
| pre-upload checks. The following options are recognized: |
| |
| * `ignore_merged_commits`: If set to `true`, the hooks will not run on commits |
| that are merged. Hooks will still run on the merge commit itself. |
| |
| ## [Hook Scripts] |
| |
| This section allows for completely arbitrary hooks to run on a per-repo basis. |
| |
| The key can be any name (as long as the syntax is valid), as can the program |
| that is executed. The key is used as the name of the hook for reporting purposes, |
| so it should be at least somewhat descriptive. |
| |
| Whitespace in the key name is OK! |
| |
| The keys must be unique as duplicates will silently clobber earlier values. |
| |
| You do not need to send stderr to stdout. The tooling will take care of |
| merging them together for you automatically. |
| |
| ``` |
| [Hook Scripts] |
| my first hook = program --gogog ${PREUPLOAD_FILES} |
| another hook = funtimes --i-need "some space" ${PREUPLOAD_FILES} |
| some fish = linter --ate-a-cat ${PREUPLOAD_FILES} |
| some cat = formatter --cat-commit ${PREUPLOAD_COMMIT} |
| some dog = tool --no-cat-in-commit-message ${PREUPLOAD_COMMIT_MESSAGE} |
| ``` |
| |
| ## [Builtin Hooks] |
| |
| This section allows for turning on common/builtin hooks. There are a bunch of |
| canned hooks already included geared towards AOSP style guidelines. |
| |
| * `aidl_format`: Run AIDL files (.aidl) through `aidl-format`. |
| * `alint`: Run AyeAye (Google Android service) analyzers on the CL to be uploaded. |
| * `aosp_license`: Check if all new-added file have valid AOSP license headers. |
| * `android_test_mapping_format`: Validate TEST_MAPPING files in Android source |
| code. Refer to go/test-mapping for more details. |
| * `black`: Run Python files (.py) through `black`. Settings can be stored in |
| `pyproject.toml` in the root of the project. |
| * `bpfmt`: Run Blueprint files (.bp) through `bpfmt`. |
| * `checkpatch`: Run commits through the Linux kernel's `checkpatch.pl` script. |
| * `clang_format`: Run git-clang-format against the commit. The default style is |
| `file`. |
| * `commit_msg_bug_field`: Require a valid `Bug:` line. |
| * `commit_msg_changeid_field`: Require a valid `Change-Id:` Gerrit line. |
| * `commit_msg_prebuilt_apk_fields`: Require badging and build information for |
| prebuilt APKs. |
| * `commit_msg_relnote_field_format`: Check for possible misspellings of the |
| `Relnote:` field and that multiline release notes are properly formatted with |
| quotes. |
| * `commit_msg_relnote_for_current_txt`: Check that CLs with changes to |
| current.txt or public_plus_experimental_current.txt also contain a |
| `Relnote:` field in the commit message. |
| * `commit_msg_test_field`: Require a `Test:` line. |
| * `cpplint`: Run through the cpplint tool (for C++ code). |
| * `gofmt`: Run Go code through `gofmt`. |
| * `google_java_format`: Run Java code through |
| [`google-java-format`](https://github.com/google/google-java-format). |
| Supports an additional option --include-dirs, which if specified will limit |
| enforcement to only files under the specified directories. |
| * `jsonlint`: Verify JSON code is sane. |
| * `ktfmt`: Run Kotlin code through `ktfmt`. Supports an additional option |
| --include-dirs, which if specified will limit enforcement to only files under |
| the specified directories. |
| * `pylint`: Alias of `pylint3`. |
| * `pylint2`: Ignored for compatibility with old configs. |
| * `pylint3`: Run Python code through `pylint` using Python 3. |
| * `rustfmt`: Run Rust code through `rustfmt`. |
| * `xmllint`: Run XML code through `xmllint`. |
| |
| Note: Builtin hooks tend to match specific filenames (e.g. `.json`). If no |
| files match in a specific commit, then the hook will be skipped for that commit. |
| |
| ``` |
| [Builtin Hooks] |
| # Turn on cpplint checking. |
| cpplint = true |
| # Turn off gofmt checking. |
| gofmt = false |
| ``` |
| |
| ## [Builtin Hooks Options] |
| |
| Used to customize the behavior of specific `[Builtin Hooks]`. Any arguments set |
| here will be passed directly to the linter in question. This will completely |
| override any existing default options, so be sure to include everything you need |
| (especially `${PREUPLOAD_FILES}` -- see below). |
| |
| Quoting is handled naturally. i.e. use `"a b c"` to pass an argument with |
| whitespace. |
| |
| See [Placeholders](#Placeholders) for variables you can expand automatically. |
| |
| ``` |
| [Builtin Hooks Options] |
| # Pass more filter args to cpplint. |
| cpplint = --filter=-x ${PREUPLOAD_FILES} |
| ``` |
| |
| ## [Builtin Hooks Exclude Paths] |
| |
| *** note |
| This section can only be added to the repo project-wide settings |
| [GLOBAL-PREUPLOAD.cfg](#GLOBAL_PREUPLOAD_cfg). |
| *** |
| |
| Used to explicitly exclude some projects when processing a hook. With this |
| section, it is possible to define a hook that should apply to the majority of |
| projects except a few. |
| |
| An entry must completely match the project's `REPO_PATH`. The paths can use the |
| [shell-style wildcards](https://docs.python.org/library/fnmatch.html) and |
| quotes. For advanced cases, it is possible to use a [regular |
| expression](https://docs.python.org/howto/regex.html) by using the `^` prefix. |
| |
| ``` |
| [Builtin Hooks Exclude Paths] |
| # Run cpplint on all projects except ones under external/ and vendor/. |
| # The "external" and "vendor" projects, if they exist, will still run cpplint. |
| cpplint = external/* vendor/* |
| |
| # Run rustfmt on all projects except ones under external/. All projects under |
| # hardware/ will be excluded except for ones starting with hardware/google (due to |
| # the negative regex match). |
| rustfmt = external/ ^hardware/(!?google) |
| ``` |
| |
| ## [Tool Paths] |
| |
| Some builtin hooks need to call external executables to work correctly. By |
| default it will call those tools from the user's `$PATH`, but the paths of those |
| executables can be overridden through `[Tool Paths]`. This is helpful to |
| provide consistent behavior for developers across different OS and Linux |
| distros/versions. The following tools are recognized: |
| |
| * `aidl-format`: used for the `aidl_format` builtin hook. |
| * `alint`: used for the `alint` builtin hook. |
| * `android-test-mapping-format`: used for the `android_test_mapping_format` |
| builtin hook. |
| * `black`: used for the `black` builtin hook. |
| * `bpfmt`: used for the `bpfmt` builtin hook. |
| * `clang-format`: used for the `clang_format` builtin hook. |
| * `cpplint`: used for the `cpplint` builtin hook. |
| * `git-clang-format`: used for the `clang_format` builtin hook. |
| * `gofmt`: used for the `gofmt` builtin hook. |
| * `google-java-format`: used for the `google_java_format` builtin hook. |
| * `google-java-format-diff`: used for the `google_java_format` builtin hook. |
| * `ktfmt`: used for the `ktfmt` builtin hook. |
| * `pylint`: used for the `pylint` builtin hook. |
| * `rustfmt`: used for the `rustfmt` builtin hook. |
| |
| See [Placeholders](#Placeholders) for variables you can expand automatically. |
| |
| ``` |
| [Tool Paths] |
| # Pass absolute paths. |
| clang-format = /usr/bin/clang-format |
| # Or paths relative to the top of the git project. |
| clang-format = prebuilts/bin/clang-format |
| # Or paths relative to the repo root. |
| clang-format = ${REPO_ROOT}/prebuilts/clang/host/${BUILD_OS}/clang-stable/bin/clang-format |
| ``` |
| |
| # Hook Developers |
| |
| These are notes for people updating the `pre-upload.py` hook itself: |
| |
| * Don't worry about namespace collisions. The `pre-upload.py` script is loaded |
| and exec-ed in its own context. The only entry-point that matters is `main`. |
| * New hooks can be added in `rh/hooks.py`. Be sure to keep the list up-to-date |
| with the documentation in this file. |
| * Python versions |
| * Code loaded & run by end users (i.e. during `repo upload`) should stick to |
| older versions of Python. We expect users to run on a variety of platforms |
| where Python is not the latest (e.g. Ubuntu LTS that is years behind). We |
| currently require **Python 3.6**. This aligns with [repo's supported Python |
| versions](https://gerrit.googlesource.com/git-repo/+/HEAD/docs/python-support.md). |
| * Code only run by repohooks developers may use much newer versions of Python |
| to keep things simple, especially as we don't readily test older versions. |
| |
| ## Warnings |
| |
| If the return code of a hook is 77, then it is assumed to be a warning. The |
| output will be printed to the terminal, but uploading will still be allowed |
| without a bypass being required. |
| |
| # TODO/Limitations |
| |
| * Some checkers operate on the files as they exist in the filesystem. This is |
| not easy to fix because the linters require not just the modified file but the |
| entire repo in order to perform full checks. e.g. `pylint` needs to know what |
| other modules exist locally to verify their API. We can support this case by |
| doing a full checkout of the repo in a temp dir, but this can slow things down |
| a lot. Will need to consider a `PREUPLOAD.cfg` knob. |
| * We need to add `pylint` tool to the AOSP manifest and use that local copy |
| instead of relying on the version that is in $PATH. |
| * Should make file extension filters configurable. All hooks currently declare |
| their own list of files like `.cc` and `.py` and `.xml`. |
| * Add more checkers. |
| * `clang-check`: Runs static analyzers against code. |
| * Whitespace checking (trailing/tab mixing/etc...). |
| * Long line checking. |
| * Commit message checks (correct format/BUG/TEST/SOB tags/etc...). |
| * Markdown (gitiles) validator. |
| * Spell checker. |
| |
| [repo]: https://gerrit.googlesource.com/git-repo/ |