Google Git
Sign in
gerrit / apps / analytics-etl / 099ab16e888090f5b53abea2ac15a57f57cfcd65
commit099ab16e888090f5b53abea2ac15a57f57cfcd65[log] [tgz]
authorLuca Milanesio <luca.milanesio@gmail.com>Sun Aug 27 05:10:35 2017 -0700
committerLuca Milanesio <luca.milanesio@gmail.com>Sun Aug 27 05:10:38 2017 -0700
treee10d09e93b248c9ea97eaf38e7dc1b93c6259b29
parent09e93ac90638bfff1345372854f0149b59b00157 [diff]
Don't run Spark tests in parallel

Spark uses a global context that is shared across threads.
If we run all the tests in parallel we risk to shutdown the
context from one thread whilst is still used by the others.

Change-Id: I7e536ae6058077334b37cab4a3dfdf224bc65a65
1 file changed
tree: e10d09e93b248c9ea97eaf38e7dc1b93c6259b29
  1. project/
  2. src/
  3. .gitignore
  4. build.sbt
  5. LICENSE
  6. README.md
README.md

spark-gerrit-analytics-etl

Spark ETL to extra analytics data from Gerrit Projects.

Job can be launched with the following parameters:

bin/spark-submit \
    --conf spark.es.nodes=es.mycompany.com \
    --conf spark.es.net.http.auth.user=elastic \
    --conf spark.es.net.http.auth.pass=changeme \
    $JARS/SparkAnalytics-assembly-1.0.jar \
    --since 2000-06-01 \
    --aggregate email_hour \
    --url http://gerrit.mycompany.com \
    -e gerrit/analytics

Parameters

  • since, until, aggregate are the same defined in Gerrit Analytics plugin see: https://gerrit.googlesource.com/plugins/analytics/+/master/README.md
  • -u --url Gerrit server URL with the analytics plugins installed
  • -e --elasticIndex specify as / to be loaded in Elastic Search if not provided no ES export will be performed
  • -o --out folder location for storing the output as JSON files if not provided data is saved to /analytics- where is the system temporary directory