Refactor project structure to allow multi build
sbt build can now build multiple project so that this repository can be
used to host multiple ETL jobs.
Moved gitcommits implementation into its own project and created a
skeleton for the auditlog project too.
This change will cause the existing gitcommits docker image to change
its name from:
gerritforge/spark-gerrit-analytics-etl
to a more representative:
gerritforge/gerrit-analytics-etl-gitcommits
Feature: Issue 10074
Change-Id: I285db20ca603825714245d9b5a3286fb09ac7891
diff --git a/README.md b/README.md
index 5a02ca3..db7bd8a 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,26 @@
-# Gerrit Analytics ETL
-Spark ETL to extra analytics data from Gerrit Projects.
+# Intro
+
+This repository provides a set of spark ETL jobs able to extract, transform and persist data from
+gerrit projects with the purpose of performing analytics tasks.
+
+Each job focuses on a specific dataset and it knows how to extract it, filter it, aggregate it,
+transform it and then persist it.
+
+The persistent storage of choice is *elasticsearch*, which plays very well with the *kibana* dashboard for
+visualizing the analytics.
+
+All jobs are configured as separate sbt projects and have in common just a thin layer of core
+dependencies, such as spark, elasticsearch client, test utils, etc.
+
+Each job can be built and published independently, both as a fat jar artifact or a docker image.
+
+# Spark ETL jobs
+
+Here below an exhaustive list of all the spark jobs provided by this repo, along with their documentation.
+
+## Git Commits
+
+Extracts and aggregate git commits data from Gerrit Projects.
Requires a [Gerrit 2.13.x](https://www.gerritcodereview.com/releases/README.md) or later
with the [analytics](https://gerrit.googlesource.com/plugins/analytics/)
@@ -11,14 +32,13 @@
bin/spark-submit \
--class com.gerritforge.analytics.gitcommits.job.Main \
--conf spark.es.nodes=es.mycompany.com \
- $JARS/analytics-etl.jar \
+ $JARS/analytics-etl-gitcommits.jar \
--since 2000-06-01 \
--aggregate email_hour \
--url http://gerrit.mycompany.com \
--events file:///tmp/gerrit-events-export.json \
--writeNotProcessedEventsTo file:///tmp/failed-events \
- -e gerrit
- \
+ -e gerrit \
--username gerrit-api-username \
--password gerrit-api-password
```
@@ -30,11 +50,9 @@
-e ES_HOST="es.mycompany.com" \
-e GERRIT_URL="http://gerrit.mycompany.com" \
-e ANALYTICS_ARGS="--since 2000-06-01 --aggregate email_hour --writeNotProcessedEventsTo file:///tmp/failed-events -e gerrit" \
- gerritforge/spark-gerrit-analytics-etl:latest
+ gerritforge/gerrit-analytics-etl-gitcommits:latest
```
-Should ElasticSearch need authentication (i.e.: if X-Pack is enabled), credentials can be
-passed through the *spark.es.net.http.auth.pass* and *spark.es.net.http.auth.user* parameters.
### Parameters
- since, until, aggregate are the same defined in Gerrit Analytics plugin
see: https://gerrit.googlesource.com/plugins/analytics/+/master/README.md
@@ -73,16 +91,28 @@
* **organization** will be extracted from the committer email if not specified
* **author** will be defaulted to the committer name if not specified
-## Development environment
+### Build
-A docker compose file is provided to spin up an instance of Elastisearch with Kibana locally.
-Just run `docker-compose up`.
+#### JAR
+To build the jar file, simply use
+
+`sbt analyticsETLGitCommits/assembly`
+
+#### Docker
+
+To build the *gerritforge/gerrit-analytics-etl-gitcommits* docker container just run:
+
+`sbt analyticsETLGitCommits/docker`.
+
+If you want to distribute use:
+
+`sbt analyticsETLGitCommits/dockerBuildAndPush`.
+
+The build and distribution override the `latest` image tag too
+Remember to create an annotated tag for a release. The tag is used to define the docker image tag too
### Caveats
-
-* If Elastisearch dies with `exit code 137` you might have to give Docker more memory ([check this article for more details](https://github.com/moby/moby/issues/22211))
-
-* If you want to run the etl job from within docker you need to make elasticsearch and gerrit available to it.
+* If you want to run the git commits ETL job from within docker you need to make elasticsearch and gerrit available to it.
You can do this by:
* spinning the container within the same network used by your elasticsearch container (`analytics-etl_ek` if you used the docker-compose provided by this repo)
@@ -98,9 +128,20 @@
-e ES_HOST="elasticsearch" \
-e GERRIT_URL="http://$HOST_IP:8080" \
-e ANALYTICS_ARGS="--since 2000-06-01 --aggregate email_hour --writeNotProcessedEventsTo file:///tmp/failed-events -e gerrit" \
- gerritforge/spark-gerrit-analytics-etl:latest
+ gerritforge/gerrit-analytics-etl-gitcommits:latest
```
+# Development environment
+
+A docker compose file is provided to spin up an instance of Elastisearch with Kibana locally.
+Just run `docker-compose up`.
+
+## Caveats
+
+* If Elastisearch dies with `exit code 137` you might have to give Docker more memory ([check this article for more details](https://github.com/moby/moby/issues/22211))
+
+* Should ElasticSearch need authentication (i.e.: if X-Pack is enabled), credentials can be passed through the *spark.es.net.http.auth.pass* and *spark.es.net.http.auth.user* parameters.
+
* If the dockerized spark job cannot connect to elasticsearch (also, running on docker) you might need to tell elasticsearch to publish
the host to the cluster using the \_site\_ address.
@@ -117,11 +158,10 @@
See [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#network-interface-values) for more info
-## Distribute as Docker Container
+## Build all
-To build the `gerritforge/spark-gerrit-analytics-etl` docker container just run `sbt docker`. If you want to distribute
-use `sbt dockerBuildAndPush`.
+To perform actions across all jobs simply run the relevant *sbt* task without specifying the job name. For example:
-The build and distribution override the `latest` image tag too
-
-Remember to create an annotated tag for a release. The tag is used to define the docker image tag too
\ No newline at end of file
+* Test all jobs: `sbt test`
+* Build jar for all jobs: `sbt assembly`
+* Build docker for all jobs: `sbt docker`
\ No newline at end of file
diff --git a/auditlog/src/main/scala/com/gerritforge/analytics/auditlog/Main.scala b/auditlog/src/main/scala/com/gerritforge/analytics/auditlog/Main.scala
new file mode 100644
index 0000000..43b48da
--- /dev/null
+++ b/auditlog/src/main/scala/com/gerritforge/analytics/auditlog/Main.scala
@@ -0,0 +1,7 @@
+package com.gerritforge.analytics.auditlog
+
+
+object Main extends App {
+ // TODO: Implement job here
+}
+
diff --git a/build.sbt b/build.sbt
index 3da5061..51c0448 100644
--- a/build.sbt
+++ b/build.sbt
@@ -1,94 +1,45 @@
-import sbt.Keys.version
-import sbtassembly.AssemblyKeys
+import SharedSettings._
+import sbtassembly.AssemblyPlugin.autoImport._
+import sbtdocker.DockerPlugin.autoImport._
-enablePlugins(GitVersioning)
-enablePlugins(DockerPlugin)
+lazy val common = (project in file("common"))
+ .settings(commonSettings: _*)
+ .settings(assembleArtifact in assembly := false)
-git.useGitDescribe := true
+lazy val analyticsETLGitCommits = (project in file("gitcommits"))
+ .enablePlugins(GitVersioning)
+ .enablePlugins(DockerPlugin)
+ .settings(commonSettings: _*)
+ .settings(commonDockerSettings(projectName="gitcommits"))
+ .settings(
+ dockerfile in docker := {
+ val artifact: File = assembly.value
+ val entryPointBase = s"/app"
-organization := "gerritforge"
-
-name := "analytics-etl"
-
-scalaVersion := "2.11.8"
-
-val sparkVersion = "2.3.2"
-
-val gerritApiVersion = "2.13.7"
-
-val pluginName = "analytics-etl"
-
-val mainClassPackage = "com.gerritforge.analytics.gitcommits.job.Main"
-val dockerRepository = "spark-gerrit-analytics-etl"
-
-libraryDependencies ++= Seq(
- "org.apache.spark" %% "spark-core" % sparkVersion % "provided"
- exclude("org.spark-project.spark", "unused"),
- "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
- "org.elasticsearch" %% "elasticsearch-spark-20" % "6.2.0"
- excludeAll ExclusionRule(organization = "org.apache.spark"),
- // json4s still needed by GerritProjects
- "org.json4s" %% "json4s-native" % "3.2.11",
- "com.google.gerrit" % "gerrit-plugin-api" % gerritApiVersion % Provided withSources(),
-
- "com.typesafe.scala-logging" %% "scala-logging" % "3.7.2",
-
- "com.github.scopt" %% "scopt" % "3.6.0",
- "org.scalactic" %% "scalactic" % "3.0.1" % "test",
- "org.scalatest" %% "scalatest" % "3.0.1" % "test"
-)
-
-mainClass in (Compile,run) := Some(mainClassPackage)
-
-assemblyJarName in assembly := s"${name.value}.jar"
-
-parallelExecution in Test := false
-
-// Docker settings
-docker := (docker dependsOn AssemblyKeys.assembly).value
-
-dockerfile in docker := {
- val artifact: File = assembly.value
- val artifactTargetPath = s"/app/${name.value}-assembly.jar"
- val entryPointBase = s"/app"
-
- new Dockerfile {
- from("openjdk:8-alpine")
- label("maintainer" -> "GerritForge <info@gerritforge.com>")
- runRaw("apk --update add curl tar bash && rm -rf /var/lib/apt/lists/* && rm /var/cache/apk/*")
- env("SPARK_VERSION", sparkVersion)
- env("SPARK_HOME", "/usr/local/spark-$SPARK_VERSION-bin-hadoop2.7")
- env("PATH","$PATH:$SPARK_HOME/bin")
- env("SPARK_JAR_PATH", artifactTargetPath)
- env("SPARK_JAR_CLASS",mainClassPackage)
- runRaw("curl -sL \"http://www-eu.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop2.7.tgz\" | tar -xz -C /usr/local")
- copy(baseDirectory(_ / "scripts" / "gerrit-analytics-etl.sh").value, file(s"$entryPointBase/gerrit-analytics-etl.sh"))
- copy(baseDirectory(_ / "scripts" / "wait-for-elasticsearch.sh").value, file(s"$entryPointBase/wait-for-elasticsearch.sh"))
- add(artifact, artifactTargetPath)
- runRaw(s"chmod +x $artifactTargetPath")
- cmd(s"/bin/sh", s"$entryPointBase/gerrit-analytics-etl.sh")
- }
-}
-imageNames in docker := Seq(
- ImageName(
- namespace = Some(organization.value),
- repository = dockerRepository,
- tag = Some("latest")
- ),
-
- ImageName(
- namespace = Some(organization.value),
- repository = dockerRepository,
- tag = Some(version.value)
+ baseDockerfile(projectName="gitcommits", artifact, artifactTargetPath=s"$entryPointBase/${name.value}-assembly.jar")
+ .copy(baseDirectory(_ / "scripts" / "gerrit-analytics-etl-gitcommits.sh").value, file(s"$entryPointBase/gerrit-analytics-etl-gitcommits.sh"))
+ .copy(baseDirectory(_ / "scripts" / "wait-for-elasticsearch.sh").value, file(s"$entryPointBase/wait-for-elasticsearch.sh"))
+ .cmd(s"/bin/sh", s"$entryPointBase/gerrit-analytics-etl-gitcommits.sh")
+ }
)
-)
-buildOptions in docker := BuildOptions(cache = false)
+ .dependsOn(common)
-packageOptions in(Compile, packageBin) += Package.ManifestAttributes(
- ("Gerrit-ApiType", "plugin"),
- ("Gerrit-PluginName", pluginName),
- ("Gerrit-Module", "com.gerritforge.analytics.gitcommits.plugin.Module"),
- ("Gerrit-SshModule", "com.gerritforge.analytics.gitcommits.plugin.SshModule"),
- ("Implementation-Title", "Analytics ETL plugin"),
- ("Implementation-URL", "https://gerrit.googlesource.com/plugins/analytics-etl")
-)
+lazy val analyticsETLAuditLog = (project in file("auditlog"))
+ .enablePlugins(GitVersioning)
+ .enablePlugins(DockerPlugin)
+ .settings(commonSettings: _*)
+ .settings(commonDockerSettings(projectName="auditlog"))
+ .settings(
+ dockerfile in docker := {
+ val artifact: File = assembly.value
+ val entryPointBase = s"/app"
+
+ baseDockerfile(projectName="auditlog", artifact, artifactTargetPath=s"$entryPointBase/${name.value}-assembly.jar")
+ }
+ )
+ .dependsOn(common)
+
+lazy val root = (project in file("."))
+ .disablePlugins(AssemblyPlugin)
+ .settings(test in assembly := {})
+ .aggregate(analyticsETLGitCommits,analyticsETLAuditLog)
diff --git a/src/main/scala/com/gerritforge/analytics/gitcommits/api/gerritApiConnectivity.scala b/common/src/main/scala/com/gerritforge/analytics/common/api/gerritApiConnectivity.scala
similarity index 97%
rename from src/main/scala/com/gerritforge/analytics/gitcommits/api/gerritApiConnectivity.scala
rename to common/src/main/scala/com/gerritforge/analytics/common/api/gerritApiConnectivity.scala
index ece68d1..b3214e2 100644
--- a/src/main/scala/com/gerritforge/analytics/gitcommits/api/gerritApiConnectivity.scala
+++ b/common/src/main/scala/com/gerritforge/analytics/common/api/gerritApiConnectivity.scala
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
-package com.gerritforge.analytics.gitcommits.api
+package com.gerritforge.analytics.common.api
import java.net.URL
diff --git a/scripts/gerrit-analytics-etl.sh b/gitcommits/scripts/gerrit-analytics-etl-gitcommits.sh
similarity index 90%
rename from scripts/gerrit-analytics-etl.sh
rename to gitcommits/scripts/gerrit-analytics-etl-gitcommits.sh
index 5a0de69..e7710a7 100755
--- a/scripts/gerrit-analytics-etl.sh
+++ b/gitcommits/scripts/gerrit-analytics-etl-gitcommits.sh
@@ -9,7 +9,7 @@
# Optional
ES_PORT="${ES_PORT:-9200}"
-SPARK_JAR_PATH="${SPARK_JAR_PATH:-/app/analytics-etl-assembly.jar}"
+SPARK_JAR_PATH="${SPARK_JAR_PATH:-/app/analytics-etl-gitcommits-assembly.jar}"
SPARK_JAR_CLASS="${SPARK_JAR_CLASS:-com.gerritforge.analytics.gitcommits.job.Main}"
echo "* Elastic Search Host: $ES_HOST:$ES_PORT"
diff --git a/scripts/wait-for-elasticsearch.sh b/gitcommits/scripts/wait-for-elasticsearch.sh
similarity index 100%
rename from scripts/wait-for-elasticsearch.sh
rename to gitcommits/scripts/wait-for-elasticsearch.sh
diff --git a/src/main/resources/email-aliasing.input.example b/gitcommits/src/main/resources/email-aliasing.input.example
similarity index 100%
rename from src/main/resources/email-aliasing.input.example
rename to gitcommits/src/main/resources/email-aliasing.input.example
diff --git a/src/main/scala/com/gerritforge/analytics/gitcommits/engine/GerritAnalyticsTransformations.scala b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/engine/GerritAnalyticsTransformations.scala
similarity index 98%
rename from src/main/scala/com/gerritforge/analytics/gitcommits/engine/GerritAnalyticsTransformations.scala
rename to gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/engine/GerritAnalyticsTransformations.scala
index ada6a12..a98e92b 100644
--- a/src/main/scala/com/gerritforge/analytics/gitcommits/engine/GerritAnalyticsTransformations.scala
+++ b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/engine/GerritAnalyticsTransformations.scala
@@ -18,7 +18,7 @@
import java.time.format.DateTimeFormatter
import java.time.{LocalDateTime, ZoneId, ZoneOffset, ZonedDateTime}
-import com.gerritforge.analytics.gitcommits.api.GerritConnectivity
+import com.gerritforge.analytics.common.api.GerritConnectivity
import com.gerritforge.analytics.gitcommits.model._
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.functions.{udf, _}
diff --git a/src/main/scala/com/gerritforge/analytics/gitcommits/engine/events/AggregationStrategy.scala b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/engine/events/AggregationStrategy.scala
similarity index 100%
rename from src/main/scala/com/gerritforge/analytics/gitcommits/engine/events/AggregationStrategy.scala
rename to gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/engine/events/AggregationStrategy.scala
diff --git a/src/main/scala/com/gerritforge/analytics/gitcommits/engine/events/GerritEventsTransformations.scala b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/engine/events/GerritEventsTransformations.scala
similarity index 100%
rename from src/main/scala/com/gerritforge/analytics/gitcommits/engine/events/GerritEventsTransformations.scala
rename to gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/engine/events/GerritEventsTransformations.scala
diff --git a/src/main/scala/com/gerritforge/analytics/gitcommits/engine/events/model.scala b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/engine/events/model.scala
similarity index 100%
rename from src/main/scala/com/gerritforge/analytics/gitcommits/engine/events/model.scala
rename to gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/engine/events/model.scala
diff --git a/src/main/scala/com/gerritforge/analytics/gitcommits/job/Main.scala b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/job/Main.scala
similarity index 100%
rename from src/main/scala/com/gerritforge/analytics/gitcommits/job/Main.scala
rename to gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/job/Main.scala
diff --git a/src/main/scala/com/gerritforge/analytics/gitcommits/model/Email.scala b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/model/Email.scala
similarity index 100%
rename from src/main/scala/com/gerritforge/analytics/gitcommits/model/Email.scala
rename to gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/model/Email.scala
diff --git a/src/main/scala/com/gerritforge/analytics/gitcommits/model/GerritEndpointConfig.scala b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/model/GerritEndpointConfig.scala
similarity index 97%
rename from src/main/scala/com/gerritforge/analytics/gitcommits/model/GerritEndpointConfig.scala
rename to gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/model/GerritEndpointConfig.scala
index 69e8cee..e12ad91 100644
--- a/src/main/scala/com/gerritforge/analytics/gitcommits/model/GerritEndpointConfig.scala
+++ b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/model/GerritEndpointConfig.scala
@@ -17,7 +17,7 @@
import java.time.format.DateTimeFormatter
import java.time.{LocalDate, ZoneOffset}
-import com.gerritforge.analytics.gitcommits.api.GerritConnectivity
+import com.gerritforge.analytics.common.api.GerritConnectivity
import com.gerritforge.analytics.gitcommits.support.ops.AnalyticsTimeOps.AnalyticsDateTimeFormater
case class GerritEndpointConfig(
diff --git a/src/main/scala/com/gerritforge/analytics/gitcommits/model/GerritProject.scala b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/model/GerritProject.scala
similarity index 100%
rename from src/main/scala/com/gerritforge/analytics/gitcommits/model/GerritProject.scala
rename to gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/model/GerritProject.scala
diff --git a/src/main/scala/com/gerritforge/analytics/gitcommits/plugin/GerritConfigSupport.scala b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/plugin/GerritConfigSupport.scala
similarity index 100%
rename from src/main/scala/com/gerritforge/analytics/gitcommits/plugin/GerritConfigSupport.scala
rename to gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/plugin/GerritConfigSupport.scala
diff --git a/src/main/scala/com/gerritforge/analytics/gitcommits/plugin/Module.scala b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/plugin/Module.scala
similarity index 100%
rename from src/main/scala/com/gerritforge/analytics/gitcommits/plugin/Module.scala
rename to gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/plugin/Module.scala
diff --git a/src/main/scala/com/gerritforge/analytics/gitcommits/plugin/ProcessGitCommitsCommand.scala b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/plugin/ProcessGitCommitsCommand.scala
similarity index 100%
rename from src/main/scala/com/gerritforge/analytics/gitcommits/plugin/ProcessGitCommitsCommand.scala
rename to gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/plugin/ProcessGitCommitsCommand.scala
diff --git a/src/main/scala/com/gerritforge/analytics/gitcommits/plugin/SshModule.scala b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/plugin/SshModule.scala
similarity index 100%
rename from src/main/scala/com/gerritforge/analytics/gitcommits/plugin/SshModule.scala
rename to gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/plugin/SshModule.scala
diff --git a/src/main/scala/com/gerritforge/analytics/gitcommits/support/ops/AnalyticsTimeOps.scala b/gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/support/ops/AnalyticsTimeOps.scala
similarity index 100%
rename from src/main/scala/com/gerritforge/analytics/gitcommits/support/ops/AnalyticsTimeOps.scala
rename to gitcommits/src/main/scala/com/gerritforge/analytics/gitcommits/support/ops/AnalyticsTimeOps.scala
diff --git a/src/test/scala/com/gerritforge/analytics/GerritAnalyticsTransformationsSpec.scala b/gitcommits/src/test/scala/com/gerritforge/analytics/GerritAnalyticsTransformationsSpec.scala
similarity index 99%
rename from src/test/scala/com/gerritforge/analytics/GerritAnalyticsTransformationsSpec.scala
rename to gitcommits/src/test/scala/com/gerritforge/analytics/GerritAnalyticsTransformationsSpec.scala
index 65f4571..40c5391 100644
--- a/src/test/scala/com/gerritforge/analytics/GerritAnalyticsTransformationsSpec.scala
+++ b/gitcommits/src/test/scala/com/gerritforge/analytics/GerritAnalyticsTransformationsSpec.scala
@@ -17,7 +17,7 @@
import java.io.{ByteArrayInputStream, File, FileOutputStream, OutputStreamWriter}
import java.nio.charset.StandardCharsets
-import com.gerritforge.analytics.gitcommits.api.GerritConnectivity
+import com.gerritforge.analytics.common.api.GerritConnectivity
import com.gerritforge.analytics.gitcommits.engine.GerritAnalyticsTransformations._
import com.gerritforge.analytics.gitcommits.model.{GerritProject, GerritProjectsSupport, ProjectContributionSource}
import org.apache.spark.sql.Row
diff --git a/src/test/scala/com/gerritforge/analytics/SparkTestSupport.scala b/gitcommits/src/test/scala/com/gerritforge/analytics/SparkTestSupport.scala
similarity index 100%
rename from src/test/scala/com/gerritforge/analytics/SparkTestSupport.scala
rename to gitcommits/src/test/scala/com/gerritforge/analytics/SparkTestSupport.scala
diff --git a/src/test/scala/com/gerritforge/analytics/gitcommits/engine/events/GerritEventsTransformationsSpec.scala b/gitcommits/src/test/scala/com/gerritforge/analytics/gitcommits/engine/events/GerritEventsTransformationsSpec.scala
similarity index 100%
rename from src/test/scala/com/gerritforge/analytics/gitcommits/engine/events/GerritEventsTransformationsSpec.scala
rename to gitcommits/src/test/scala/com/gerritforge/analytics/gitcommits/engine/events/GerritEventsTransformationsSpec.scala
diff --git a/src/test/scala/com/gerritforge/analytics/gitcommits/model/EmailSpec.scala b/gitcommits/src/test/scala/com/gerritforge/analytics/gitcommits/model/EmailSpec.scala
similarity index 100%
rename from src/test/scala/com/gerritforge/analytics/gitcommits/model/EmailSpec.scala
rename to gitcommits/src/test/scala/com/gerritforge/analytics/gitcommits/model/EmailSpec.scala
diff --git a/src/test/scala/com/gerritforge/analytics/gitcommits/model/GerritEndpointConfigTest.scala b/gitcommits/src/test/scala/com/gerritforge/analytics/gitcommits/model/GerritEndpointConfigTest.scala
similarity index 100%
rename from src/test/scala/com/gerritforge/analytics/gitcommits/model/GerritEndpointConfigTest.scala
rename to gitcommits/src/test/scala/com/gerritforge/analytics/gitcommits/model/GerritEndpointConfigTest.scala
diff --git a/src/test/scala/com/gerritforge/analytics/gitcommits/support/ops/AnalyticsTimeOpsSpec.scala b/gitcommits/src/test/scala/com/gerritforge/analytics/gitcommits/support/ops/AnalyticsTimeOpsSpec.scala
similarity index 100%
rename from src/test/scala/com/gerritforge/analytics/gitcommits/support/ops/AnalyticsTimeOpsSpec.scala
rename to gitcommits/src/test/scala/com/gerritforge/analytics/gitcommits/support/ops/AnalyticsTimeOpsSpec.scala
diff --git a/project/SharedSettings.scala b/project/SharedSettings.scala
new file mode 100644
index 0000000..9461d32
--- /dev/null
+++ b/project/SharedSettings.scala
@@ -0,0 +1,88 @@
+import Versions._
+import com.typesafe.sbt.GitPlugin.autoImport.git
+import sbt.Keys._
+import sbt.{Def, ExclusionRule, _}
+import sbtassembly.AssemblyKeys
+import sbtassembly.AssemblyPlugin.autoImport.{assemblyJarName, _}
+import sbtdocker.DockerPlugin.autoImport._
+
+object SharedSettings {
+
+ private val dockerRepositoryPrefix = "gerrit-analytics-etl"
+
+ lazy val commonSettings: Seq[Def.Setting[_]] = Seq(
+ scalaVersion := "2.11.8",
+ organization := "gerritforge",
+ parallelExecution in Test := false,
+ git.useGitDescribe := true,
+ libraryDependencies ++= Seq(
+ "org.apache.spark" %% "spark-core" % sparkVersion % "provided" exclude("org.spark-project.spark", "unused"),
+ "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
+ "org.elasticsearch" %% "elasticsearch-spark-20" % esSpark excludeAll ExclusionRule(organization = "org.apache.spark"),
+ "org.json4s" %% "json4s-native" % json4s,
+ "com.google.gerrit" % "gerrit-plugin-api" % gerritApiVersion % Provided withSources(),
+ "com.typesafe.scala-logging" %% "scala-logging" % scalaLogging,
+ "com.github.scopt" %% "scopt" % scopt,
+ "org.scalactic" %% "scalactic" % scalactic % "test",
+ "org.scalatest" %% "scalatest" % scalaTest % "test"
+ )
+ )
+
+ def commonDockerSettings(projectName: String): Seq[Def.Setting[_]] = {
+ val repositoryName = Seq(dockerRepositoryPrefix, projectName).mkString("-")
+ Seq(
+ name := s"analytics-etl-$projectName",
+ mainClass in (Compile,run) := Some(s"com.gerritforge.analytics.$projectName.job.Main"),
+ packageOptions in(Compile, packageBin) += Package.ManifestAttributes(
+ ("Gerrit-ApiType", "plugin"),
+ ("Gerrit-PluginName", s"analytics-etl-$projectName"),
+ ("Gerrit-Module", s"com.gerritforge.analytics.$projectName.plugin.Module"),
+ ("Gerrit-SshModule", s"com.gerritforge.analytics.$projectName.plugin.SshModule"),
+ ("Implementation-Title", s"Analytics ETL plugin - $projectName"),
+ ("Implementation-URL", "https://gerrit.googlesource.com/plugins/analytics-etl")
+ ),
+ assemblyJarName in assembly := s"${name.value}.jar",
+ docker := (docker dependsOn AssemblyKeys.assembly).value,
+ imageNames in docker := Seq(
+ ImageName(
+ namespace = Some(organization.value),
+ repository = repositoryName,
+ tag = Some("latest")
+ ),
+ ImageName(
+ namespace = Some(organization.value),
+ repository = repositoryName,
+ tag = Some(version.value)
+ )
+ ),
+ buildOptions in docker := BuildOptions(cache = false)
+ )
+ }
+
+ def baseDockerfile(projectName: String, artifact: File, artifactTargetPath: String): Dockerfile = {
+ new Dockerfile {
+ from("openjdk:8-alpine")
+ label("maintainer" -> "GerritForge <info@gerritforge.com>")
+ runRaw("apk --update add curl tar bash && rm -rf /var/lib/apt/lists/* && rm /var/cache/apk/*")
+ env("SPARK_VERSION", sparkVersion)
+ env("SPARK_HOME", "/usr/local/spark-$SPARK_VERSION-bin-hadoop2.7")
+ env("PATH","$PATH:$SPARK_HOME/bin")
+ env("SPARK_JAR_PATH", artifactTargetPath)
+ env("SPARK_JAR_CLASS",s"com.gerritforge.analytics.$projectName.job.Main")
+ runRaw("curl -sL \"http://www-eu.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop2.7.tgz\" | tar -xz -C /usr/local")
+ add(artifact, artifactTargetPath)
+ runRaw(s"chmod +x $artifactTargetPath")
+ }
+ }
+}
+
+object Versions {
+ val sparkVersion = "2.3.2"
+ val gerritApiVersion = "2.13.7"
+ val esSpark = "6.2.0"
+ val scalaLogging = "3.7.2"
+ val scopt = "3.6.0"
+ val scalactic = "3.0.1"
+ val scalaTest = "3.0.1"
+ val json4s = "3.2.11"
+}
