Spark ETL to extra analytics data from Gerrit Projects using the Analytics plugin

Clone this repo:



  1. de7d41a Missing license disclaimer by Fabio Ponciroli · 3 weeks ago master
  2. f8245b0 Merge "Using numbers to specify import order in Kibana." by Claudio Pacchiega · 3 weeks ago
  3. 627d74e Using numbers to specify import order in Kibana. by Claudio Pacchiega · 3 weeks ago
  4. b60c57a Fixed parameter text for index name. by Claudio Pacchiega · 3 weeks ago
  5. d0fb7ed Match the CI name by Fabio Ponciroli · 3 weeks ago

Gerrit Analytics ETL

Spark ETL to extra analytics data from Gerrit Projects.

Requires a Gerrit 2.13.x or later with the analytics plugin installed and Apache Spark 2.11 or later.

Job can be launched with the following parameters:

bin/spark-submit \
    --conf \
    $JARS/SparkAnalytics-assembly.jar \
    --since 2000-06-01 \
    --aggregate email_hour \
    --url \
    --events file:///tmp/gerrit-events-export.json
    --writeNotProcessedEventsTo file:///tmp/failed-events
    -e gerrit/analytics

Should ElasticSearch need authentication (i.e.: if X-Pack is enabled), credentials can be passed through the and parameters.


  • since, until, aggregate are the same defined in Gerrit Analytics plugin see:

  • -u --url Gerrit server URL with the analytics plugins installed

  • -p --prefix (optional) Projects prefix. Limit the results to those projects that start with the specified prefix.

  • -e --elasticIndex specify as / to be loaded in Elastic Search if not provided no ES export will be performed

  • -o --out folder location for storing the output as JSON files if not provided data is saved to /analytics- where is the system temporary directory

  • -a --email-aliases (optional) “emails to author alias” input data path.

  • --events location where to load the Gerrit Events If not specified events will be ignored

  • --writeNotProcessedEventsTo location where to write a TSV file containing the events we couldn't process with a description fo the reason why

    CSVs with 3 columns are expected in input.

    Here an example of the required files structure:

    John Smith,,John's Company
    John Smith,,John's Company
    David Smith,,Indipendent
    David Smith,,Indipendent

    You can use the following command to quickly extract the list of authors and emails to create part of an input CSV file:

    echo -e "author,email\n$(git log --pretty="%an,%ae%n%cn,%ce"|sort |uniq )" > /tmp/my_aliases.csv

    Once you have it, you just have to add the organization column.


    • organization will be extracted from the committer email if not specified
    • author will be defaulted to the committer name if not specified

Development environment

A docker compose file is provided to spin up an instance of Elastisearch with Kibana locally. Just run docker-compose up.

Kibana will run on port 5601 and Elastisearch on port 9200


If Elastisearch dies with exit code 137 you might have to give Docker more memory (check this article for more details)

Distribute as Docker Container

To build the gerritforge/spark-gerrit-analytics-etl docker container just run sbt docker. If you want to distribute use sbt dockerBuildAndPush.

The build and distribution override the latest image tag too

Remember to create an annotated tag for a relase. The tag is used to define the docker image tag too