Google Git
Sign in
gerrit / apps / analytics-etl / b9d74bdb9117d764f2bde9912924d6eda1595b64
commitb9d74bdb9117d764f2bde9912924d6eda1595b64[log] [tgz]
authorFabio Ponciroli <ponch78@gmail.com>Thu Nov 09 16:53:17 2017 +0000
committerLuca Milanesio <luca@milanesio.org>Tue Dec 05 06:12:31 2017 -0500
treefa2f9efb765bb3aa15398c39e9590f71b6757df1
parent91c551dcaf0f93aacd1b3da96e64939e076322fa [diff]
Local ES and Kibana for development

It can be handy having a local Kibana and Elastisearch running while developing the Spark ETL

Change-Id: I7ceb54b4bd2ef8223dd4fd0f6565b33ef787a19e
2 files changed
tree: fa2f9efb765bb3aa15398c39e9590f71b6757df1
  1. kibana/
  2. project/
  3. src/
  4. .gitignore
  5. build.sbt
  6. docker-compose.yaml
  7. LICENSE
  8. README.md
README.md

spark-gerrit-analytics-etl

Spark ETL to extra analytics data from Gerrit Projects.

Job can be launched with the following parameters:

bin/spark-submit \
    --conf spark.es.nodes=es.mycompany.com \
    --conf spark.es.net.http.auth.user=elastic \
    --conf spark.es.net.http.auth.pass=changeme \
    $JARS/SparkAnalytics-assembly-1.0.jar \
    --since 2000-06-01 \
    --aggregate email_hour \
    --url http://gerrit.mycompany.com \
    -e gerrit/analytics

Parameters

  • since, until, aggregate are the same defined in Gerrit Analytics plugin see: https://gerrit.googlesource.com/plugins/analytics/+/master/README.md
  • -u --url Gerrit server URL with the analytics plugins installed
  • -e --elasticIndex specify as / to be loaded in Elastic Search if not provided no ES export will be performed
  • -o --out folder location for storing the output as JSON files if not provided data is saved to /analytics- where is the system temporary directory

Development environment

A docker compose file is provided to spin up an instance of Elastisearch with Kibana locally. Just run docker-compose up.

Kibana will run on port 5601 and Elastisearch on port 9200

Default credentials

The Elastisearch default user is elastic and the default password changeme

Caveats

If Elastisearch dies with exit code 137 you might have to give Docker more memory (check this article for more details)