Merge branch 'stable-3.1'
* stable-3.1:
Use nested CF template for Network Stack
Change-Id: I12927e66dfbdae90e86b0eb2dd7cf292b0f7bafa
diff --git a/.gitignore b/.gitignore
index 00e1907..cb80d80 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,8 +1,8 @@
setup.env
-gerrit.setup
-replication.setup
gerrit/plugins/*.jar
gerrit/etc/*key*
gerrit.config
secure.config
replication.config
+.idea
+**/*.pem
diff --git a/Configuration.md b/Configuration.md
new file mode 100644
index 0000000..28efac2
--- /dev/null
+++ b/Configuration.md
@@ -0,0 +1,68 @@
+# Configuration
+
+Each recipe provides a `setup.env.template` file which is a template for configuring the Gerrit stacks.
+Copy that into a `setup.env` and set the correct values for the provided environment variables.
+
+```bash
+cp setup.env.template setup.env
+```
+Here below a list of variables that are common and need to be specified regardless the recipe you want to
+deploy. Please refer to the individual recipes to understand what additional variables need to be set.
+
+## Common parameters
+
+#### Environment
+
+Configuration values affecting deployment environment and cluster properties
+
+* `AWS_REGION`: Optional. Which region to deploy to. `us-east-1` by default.
+* `AWS_PREFIX`: Optional. A string to prefix stacks and resources with. `gerrit` by default.
+* `DOCKER_REGISTRY_URI`: Mandatory. URI of the Docker registry. See the
+ [prerequisites](Prerequisites.md) section for more details.
+* `SSL_CERTIFICATE_ARN`: Mandatory. ARN of the wildcard SSL Certificate, covering both master nodes.
+* `CLUSTER_STACK_NAME`: Optional. Name of the cluster stack. `gerrit-cluster` by default.
+* `DNS_ROUTING_STACK_NAME`: Optional. Name of the DNS routing stack. `gerrit-dns-routing` by default.
+* `HOSTED_ZONE_NAME`: Optional. Name of the hosted zone. `mycompany.com` by default.
+* `GERRIT_KEY_PREFIX` : Optional. Secrets prefix used during the [Import into AWS Secret Manager](#import-into-aws-secret-manager).
+ `gerrit_secret` by default.
+
+#### SPECS
+
+Configuration values to spec up Gerrit containers.
+
+* `CLUSTER_INSTANCE_TYPE`: Optional. The EC2 instance Type used to run the cluster. `m4.10xlarge` by default.
+* `GERRIT_RAM`: RAM allocated (MiB) to the Gerrit container. `70000` by default.
+* `GERRIT_CPU`: vCPU units allocated to the Gerrit container. `10240` by default.
+* `GERRIT_HEAP_LIMIT`: Maximum heap size of the Java process running Gerrit, in bytes.
+ See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#container.heapLimit)
+ `35g` by default.
+* `JGIT_CACHE_SIZE`: Maximum number of bytes to load and cache in memory from pack files.
+ See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#core.packedGitLimit)
+ for more details. `12g` by default.
+
+#### LDAP
+
+Configuration values related to LDAP integration.
+See more details [here](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#ldap)
+
+* `LDAP_SERVER`: Mandatory. URL of the organization’s LDAP server to query for user information and group membership from
+ See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#ldap.server)
+* `LDAP_USERNAME`: Mandatory. Username to bind to the LDAP server with
+ See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#ldap.username)
+* `LDAP_ACCOUNT_BASE`: Mandatory. Root of the tree containing all user accounts
+ See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#ldap.accountBase)
+* `LDAP_GROUP_BASE`: Mandatory. Root of the tree containing all group objects
+ See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#ldap.groupBase)
+
+#### SMTP
+
+Configuration values related to SMTP integration.
+See more details [here](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#sendemail)
+
+* `SMTP_SERVER`: Mandatory. Hostname (or IP address) of a SMTP server that will relay messages generated by Gerrit to end users
+ See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#sendemail.smtpServer)
+* `SMTP_USER`: Mandatory. User name to authenticate with
+ See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#sendemail.smtpUser)
+* `SMTP_DOMAIN`: Mandatory. Domain to be used in the "From" field of any generated email messages
+ See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#sendemail.from)
+
diff --git a/Docker.md b/Docker.md
new file mode 100644
index 0000000..7214357
--- /dev/null
+++ b/Docker.md
@@ -0,0 +1,38 @@
+# Docker Operations
+
+The templates provided by this repo aim to deploy Gerrit (and the relevant infrastructure) as containerized
+applications over Amazon ECS. In order to achieve this the application components, including Gerrit are packaged
+as docker images and stored in ECR, the AWS docker registry.
+
+## Configure Elastic Container Registry (ECR)
+
+Set the `DOCKER_REGISTRY_URI` environment variable in your `setup.env` file. This will be
+
+```bash
+<aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com
+```
+
+The existence of the docker repositories is left as a prerequisite manual step,
+see [documentation](Prerequisites.md)
+
+## Publishing Docker images
+
+The Makefiles provided by these recipes allow to publish docker images to ECR (see below).
+You might want to do this to test building phase without deploying a new cluster, however you should keep in mind that
+publishing a new docker image will _not_ make it available to ECS, so it cannot be used for upgrading running instances.
+
+Note that you will need to _cd_ to the recipe directory before running any of the following and that the relevant image
+needs to exist for that specific recipe (for example you can't publish HAProxy from the single-master recipe).
+
+* Gerrit: `make gerrit-publish`
+* SSH Agent: `make git-ssh-publish`
+* Gerrit Daemon: `make git-daemon-publish`
+* Grafana: `make grafana-publish`
+* Prometheus: `make prometheus-publish`
+* HAProxy: `make haproxy-publish`
+* Syslog sidecar: `make syslog-sidecar-publish`
+
+
+
+
+
diff --git a/Makefile.common b/Makefile.common
index 5d5e7c8..550d9ac 100644
--- a/Makefile.common
+++ b/Makefile.common
@@ -7,6 +7,12 @@
aws ec2 create-key-pair --region $(AWS_REGION) --key-name $(CLUSTER_KEYS) --query 'KeyMaterial' --output text > $(CLUSTER_KEYS).pem
upload-common-templates:
- export AWS_PAGER=; aws s3api head-bucket --bucket $(TEMPLATE_BUCKET_NAME) 2>/dev/null || aws s3api create-bucket --bucket $(TEMPLATE_BUCKET_NAME)
+ $(eval CREATE_BUCKET_PARAMS := --bucket $(TEMPLATE_BUCKET_NAME))
+ifneq ("$(AWS_REGION)", "us-east-1")
+ $(eval CREATE_BUCKET_PARAMS := $(CREATE_BUCKET_PARAMS) --create-bucket-configuration LocationConstraint=$(AWS_REGION))
+endif
+
+ export AWS_PAGER=; aws s3api head-bucket --bucket $(TEMPLATE_BUCKET_NAME) 2>/dev/null || \
+ aws s3api create-bucket $(CREATE_BUCKET_PARAMS)
aws s3 cp ../common-templates/cf-gerrit-task-execution-role.yml s3://$(TEMPLATE_BUCKET_NAME)/
aws s3 cp ../common-templates/cf-gerrit-network-stack.yml s3://$(TEMPLATE_BUCKET_NAME)/
diff --git a/Prerequisites.md b/Prerequisites.md
new file mode 100644
index 0000000..6dbccab
--- /dev/null
+++ b/Prerequisites.md
@@ -0,0 +1,29 @@
+# Prerequisites
+
+Before configuring, setting up and deploying a gerrit stack on AWS, there are some one-off operations
+that are required. These includes key creations, certificates, docker registries etc.
+The prerequisites to run this stack are:
+
+* a registered and correctly configured domain in
+[Route53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/getting-started.html)
+
+* Make sure ECR repositories exist
+
+```bash
+aws ecr create-repository --repository-name aws-gerrit/gerrit
+aws ecr create-repository --repository-name aws-gerrit/git-ssh
+aws ecr create-repository --repository-name aws-gerrit/git-daemon
+aws ecr create-repository --repository-name aws-gerrit/haproxy
+aws ecr create-repository --repository-name aws-gerrit/syslog-sidecar
+aws ecr create-repository --repository-name aws-gerrit/prometheus
+aws ecr create-repository --repository-name aws-gerrit/grafana
+```
+
+* to upload required credentials to AWS Secret Manager execute the [secrets configuration documentation steps](Secrets.md).
+
+* an SSL Certificate in AWS Certificate Manager (you can find more information on
+ how to create and handle certificates in AWS [here](https://aws.amazon.com/certificate-manager/getting-started/)
+
+* An S3 bucket is required in order to store cloudformation templates that are used to build the stacks.
+You must edit the `common.env` file and set the `TEMPLATE_BUCKET_NAME`. The bucket will then be created
+the first time you run a deployment. Note that the at the moment the bucket accessibility is public.
diff --git a/README.md b/README.md
index a453575..69bf466 100644
--- a/README.md
+++ b/README.md
@@ -10,6 +10,7 @@
- [Overview](#overview)
- [Pre-requisites](#pre-requisites)
- [Templates](#templates)
+- [External Services](#external-services)
## Overview
@@ -28,8 +29,30 @@
To manage your AWS services via command line you will need to install
[AWS CLI](https://aws.amazon.com/cli/) and set it up to point to your account.
+To build gerrit and related-components' images
+[Docker](https://www.docker.com/)
+
## Templates
* [Standalone Gerrit master sandbox with LDAP authentication](/single-master/README.md)
* [Gerrit master and slave sandbox with LDAP authentication](/master-slave/README.md)
-* [Gerrit dual-master in HA sandbox with LDAP authentication](/multi-master/README.md)
+* [Gerrit dual-master in HA sandbox with LDAP authentication](/dual-master/README.md)
+
+## External services
+
+This is a list of external services that you might need to setup your stack and some suggestions
+on how to easily create them.
+
+#### SMTP Server
+
+If you need to setup a SMTP service Amazon Simple Email Service can be used.
+Details how setup Amazon SES can be found [here](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-set-up.html).
+
+To correctly setup email notifications Gerrit requires ssl protocol on default port 465 to
+be enabled on SMTP Server. It is possible to setup Gerrit to talk to standard SMTP port 25
+but by default all EC2 instances are blocking it. To enable port 25 please follow [this](https://aws.amazon.com/premiumsupport/knowledge-center/ec2-port-25-throttle/) link.
+
+#### LDAP Server
+
+If you need a testing LDAP server you can find details on how to easily
+create one in the [LDAP folder](ldap/README.md).
diff --git a/Secrets.md b/Secrets.md
new file mode 100644
index 0000000..962b84a
--- /dev/null
+++ b/Secrets.md
@@ -0,0 +1,93 @@
+# Store Gerrit Secrets to AWS Secret Manager
+
+[AWS Secret Manager](https://aws.amazon.com/secrets-manager/) is a secure way of
+storing and managing secrets. These scripts make use of it to retrieve sensitive information
+required to run gerrit, such as private keys and passwords.
+
+In order to do that, such secrets must be previously uploaded to the secret manager, so that
+they can be found and used during deployment.
+
+To store the secret you can run the relevant [script](../gerrit/add_secrets_aws_secrets_manager.sh) to
+upload them to AWS Secret Manager:
+`./add_secrets_aws_secrets_manager.sh /path/to/your/keys/directory secret_prefix aws-region-id`
+
+for example:
+
+```bash
+./add_secrets_aws_secrets_manager.sh /tmp/secrets secret_prefix us-east-1`
+```
+
+When `secret_prefix` is omitted, it is set to `gerrit_secret` by default.
+
+The [script](../gerrit/add_secrets_aws_secrets_manager.sh) expects secrets to be available in a
+specified directory (e.g. `/tmp/secrets`).
+
+The expected secrets are the following.
+
+### SSH Host Keys
+
+The SSH keys you will need to add are the one usually created and used by Gerrit:
+* ssh_host_ecdsa_384_key
+* ssh_host_ecdsa_384_key.pub
+* ssh_host_ecdsa_521_key
+* ssh_host_ecdsa_521_key.pub
+* ssh_host_ecdsa_key
+* ssh_host_ecdsa_key.pub
+* ssh_host_ed25519_key
+* ssh_host_ed25519_key.pub
+* ssh_host_rsa_key
+* ssh_host_rsa_key.pub
+
+You will have to create the keys and place them for example in `/tmp/secrets` directory,
+which you can then feed to the `add_secrets_aws_secrets_manager.sh` script.
+These SSH host keys are generated by gerrit during the init script, so you could copy them
+from a previous installation (`etc/*key*` files), if you have one.
+
+ ```bash
+ cp <previous_installation>/etc/*key* /tmp/secrets
+ ```
+
+ If you don't have a previous installation of gerrit you can initialize a new gerrit and copy the
+ generated ones, as follows:
+
+ ```bash
+cd /tmp/
+wget https://gerrit-releases.storage.googleapis.com/gerrit-3.2.2.war
+java -jar gerrit-3.2.2.war init -d /tmp/foobar --dev --batch --no-auto-start
+cp /tmp/foobar/etc/*key* /tmp/secrets
+```
+
+### Email Private Key
+
+You will need to create a secret and put it in a file called `registerEmailPrivateKey`
+in the same directory of the SSH keys (e.g. `/tmp/secrets`).
+
+### LDAP Password
+
+You will need to put the admin LDAP password in a file called `ldapPassword`
+in the same directory of the SSH keys (e.g. `/tmp/secrets`).
+
+### SMTP Password
+
+You will need to put the SMTP password in a file called `smtpPassword`
+in the same directory of the SSH keys (e.g. `/tmp/secrets`).
+
+### Prometheus Bearer Token
+
+Generate a bearer token to be used for monitoring with Prometheus:
+
+```bash
+openssl rand -hex 20 > /tmp/secrets/prometheus_bearer_token
+```
+
+### Private/public SSH key for replication (required only for recipes involving replicas)
+
+You will also need to create private and public SSH keys used by the replication plugin to replicate
+from masters to replicas, for example:
+
+```bash
+ssh-keygen -b 2048 -t rsa -f /tmp/secrets/replication_user_id_rsa -q -N ""
+```
+
+* replication_user_id_rsa
+* replication_user_id_rsa.pub
diff --git a/common.env b/common.env
index 371577f..c5bcc9b 100644
--- a/common.env
+++ b/common.env
@@ -4,10 +4,9 @@
CLUSTER_KEYS=$(AWS_PREFIX)-cluster-keys
# Gerrit version
-GERRIT_VERSION=3.1
-GERRIT_PATCH=6
+GERRIT_VERSION=3.2
+GERRIT_PATCH=2
GERRIT_BRANCH=stable-$(GERRIT_VERSION)
-
# Gerrit CI
GERRIT_CI=https://gerrit-ci.gerritforge.com/view/Plugins-$(GERRIT_BRANCH)/job
LAST_BUILD=lastSuccessfulBuild/artifact/bazel-bin/plugins
@@ -26,3 +25,6 @@
# Nested templates bucket
TEMPLATE_BUCKET_NAME=aws-gerrit-cf-templates
+
+# Elastic Container Service
+CLUSTER_STACK_NAME:=$(AWS_PREFIX)-cluster
diff --git a/dual-master/Makefile b/dual-master/Makefile
index 6200bbb..eda7031 100644
--- a/dual-master/Makefile
+++ b/dual-master/Makefile
@@ -1,3 +1,4 @@
+include ../common.env # Must be included before setup.env because the latter depends on it
include setup.env
include ../Makefile.common
@@ -25,6 +26,10 @@
dns-routing wait-for-dns-routing-creation
cluster: cluster-keys
+ifdef CLUSTER_INSTANCE_TYPE
+ $(eval OPTIONAL_PARAMS := $(OPTIONAL_PARAMS) ParameterKey=InstanceType,ParameterValue=$(CLUSTER_INSTANCE_TYPE))
+endif
+
$(AWS_FC_COMMAND) create-stack \
--stack-name $(CLUSTER_STACK_NAME) \
--capabilities CAPABILITY_IAM \
@@ -33,7 +38,7 @@
--parameters \
ParameterKey=DesiredCapacity,ParameterValue=$(CLUSTER_DESIRED_CAPACITY) \
ParameterKey=ECSKeyName,ParameterValue=$(CLUSTER_KEYS) \
- ParameterKey=InstanceType,ParameterValue=$(CLUSTER_INSTANCE_TYPE)
+ $(OPTIONAL_PARAMS)
service-master-1:
$(AWS_FC_COMMAND) create-stack \
@@ -42,9 +47,18 @@
--template-body file://`pwd`/$(SERVICE_MASTER_TEMPLATE) \
--region $(AWS_REGION) \
--parameters \
+ ParameterKey=LDAPServer,ParameterValue=$(LDAP_SERVER) \
+ ParameterKey=LDAPUsername,ParameterValue=\"$(LDAP_USERNAME)\" \
+ ParameterKey=LDAPAccountBase,ParameterValue=\"$(LDAP_ACCOUNT_BASE)\" \
+ ParameterKey=LDAPGroupBase,ParameterValue=\"$(LDAP_GROUP_BASE)\" \
+ ParameterKey=SMTPServer,ParameterValue=$(SMTP_SERVER) \
+ ParameterKey=SMTPUser,ParameterValue=$(SMTP_USER) \
+ ParameterKey=SMTPDomain,ParameterValue=$(SMTP_DOMAIN) \
ParameterKey=ClusterStackName,ParameterValue=$(CLUSTER_STACK_NAME) \
+ ParameterKey=TemplateBucketName,ParameterValue=$(TEMPLATE_BUCKET_NAME) \
ParameterKey=HostedZoneName,ParameterValue=$(HOSTED_ZONE_NAME) \
ParameterKey=Subdomain,ParameterValue=$(MASTER1_SUBDOMAIN) \
+ ParameterKey=SlaveSubdomain,ParameterValue=$(SLAVE_SUBDOMAIN) \
ParameterKey=DockerRegistryUrl,ParameterValue=$(DOCKER_REGISTRY_URI) \
ParameterKey=CertificateArn,ParameterValue=$(SSL_CERTIFICATE_ARN) \
ParameterKey=HTTPHostPort,ParameterValue=$(HTTP_HOST_PORT_MASTER1) \
@@ -72,9 +86,18 @@
--template-body file://`pwd`/$(SERVICE_MASTER_TEMPLATE) \
--region $(AWS_REGION) \
--parameters \
+ ParameterKey=LDAPServer,ParameterValue=$(LDAP_SERVER) \
+ ParameterKey=LDAPUsername,ParameterValue=\"$(LDAP_USERNAME)\" \
+ ParameterKey=LDAPAccountBase,ParameterValue=\"$(LDAP_ACCOUNT_BASE)\" \
+ ParameterKey=LDAPGroupBase,ParameterValue=\"$(LDAP_GROUP_BASE)\" \
+ ParameterKey=SMTPServer,ParameterValue=$(SMTP_SERVER) \
+ ParameterKey=SMTPUser,ParameterValue=$(SMTP_USER) \
+ ParameterKey=SMTPDomain,ParameterValue=$(SMTP_DOMAIN) \
ParameterKey=ClusterStackName,ParameterValue=$(CLUSTER_STACK_NAME) \
+ ParameterKey=TemplateBucketName,ParameterValue=$(TEMPLATE_BUCKET_NAME) \
ParameterKey=HostedZoneName,ParameterValue=$(HOSTED_ZONE_NAME) \
ParameterKey=Subdomain,ParameterValue=$(MASTER2_SUBDOMAIN) \
+ ParameterKey=SlaveSubdomain,ParameterValue=$(SLAVE_SUBDOMAIN) \
ParameterKey=DockerRegistryUrl,ParameterValue=$(DOCKER_REGISTRY_URI) \
ParameterKey=CertificateArn,ParameterValue=$(SSL_CERTIFICATE_ARN) \
ParameterKey=HTTPHostPort,ParameterValue=$(HTTP_HOST_PORT_MASTER2) \
@@ -102,7 +125,12 @@
--template-body file://`pwd`/$(SERVICE_SLAVE_TEMPLATE) \
--region $(AWS_REGION) \
--parameters \
+ ParameterKey=LDAPServer,ParameterValue=$(LDAP_SERVER) \
+ ParameterKey=LDAPUsername,ParameterValue=\"$(LDAP_USERNAME)\" \
+ ParameterKey=LDAPAccountBase,ParameterValue=\"$(LDAP_ACCOUNT_BASE)\" \
+ ParameterKey=LDAPGroupBase,ParameterValue=\"$(LDAP_GROUP_BASE)\" \
ParameterKey=ClusterStackName,ParameterValue=$(CLUSTER_STACK_NAME) \
+ ParameterKey=TemplateBucketName,ParameterValue=$(TEMPLATE_BUCKET_NAME) \
ParameterKey=HostedZoneName,ParameterValue=$(HOSTED_ZONE_NAME) \
ParameterKey=Subdomain,ParameterValue=$(SLAVE_SUBDOMAIN) \
ParameterKey=DockerRegistryUrl,ParameterValue=$(DOCKER_REGISTRY_URI) \
diff --git a/dual-master/README.md b/dual-master/README.md
index 1f8bd19..f07b040 100644
--- a/dual-master/README.md
+++ b/dual-master/README.md
@@ -5,11 +5,12 @@
## Architecture
-Four templates are provided in this example:
+The following templates are provided in this example:
* `cf-cluster`: define the ECS cluster and the networking stack
-* `cf-service-master-1`: define the service stack running Gerrit master 1
-* `cf-service-master-2`: define the service stack running Gerrit master 2
+* `cf-service-master`: define the service stack running the gerrit master
* `cf-dns-route`: define the DNS routing for the service
+* `cf-service-slave`: define the service stack running the gerrit replica
+* `cf-service-lb`: define the LBs in front of gerrit masters (this includes haproxy as well as NLB)
### Networking
@@ -53,143 +54,71 @@
* All the logs are forwarded to AWS CloudWatch in the LogGroup with the cluster
stack name
+### Monitoring
+
+* Standard CloudWatch monitoring metrics for each component
+* Prometheus and Grafana stack is currently not available for dual-master, but a change is in progress to allow this
+ (see [Issue 12979](https://bugs.chromium.org/p/gerrit/issues/detail?id=12979))
+
## How to run it
-### Setup
+### 0 - Prerequisites
-The `setup.env.template` is an example of setup file for the creation of the stacks.
+Follow the steps described in the [Prerequisites](../Prerequisites.md) section
-Before creating the stacks, create a `setup.env` in the `Makefile` directory and
-set the correct values of the environment variables.
+### 1 - Configuration
-This is the list of available parameters:
+Please refer to the [configuration docs](../Configuration.md) to understand how to set up the
+configuration and what common configuration values are needed.
+On top of that, you might set the additional parameters, specific for this recipe.
-* `DOCKER_REGISTRY_URI`: Mandatory. URI of the Docker registry. See the
- [prerequisites](#prerequisites) section for more details.
-* `SSL_CERTIFICATE_ARN`: Mandatory. ARN of the wildcard SSL Certificate, covering both master nodes.
-* `CLUSTER_STACK_NAME`: Optional. Name of the cluster stack. `gerrit-cluster` by default.
+#### Environment
+
+Configuration values affecting deployment environment and cluster properties
+
* `SERVICE_MASTER1_STACK_NAME`: Optional. Name of the master 1 service stack. `gerrit-service-master-1` by default.
* `SERVICE_MASTER2_STACK_NAME`: Optional. Name of the master 2 service stack. `gerrit-service-master-2` by default.
-* `DNS_ROUTING_STACK_NAME`: Optional. Name of the DNS routing stack. `gerrit-dns-routing` by default.
-* `HOSTED_ZONE_NAME`: Optional. Name of the hosted zone. `mycompany.com` by default.
* `MASTER1_SUBDOMAIN`: Optional. Name of the master 1 sub domain. `gerrit-master-1-demo` by default.
* `MASTER2_SUBDOMAIN`: Optional. Name of the master 2 sub domain. `gerrit-master-2-demo` by default.
-* `CLUSTER_DESIRED_CAPACITY`: Optional. Number of EC2 instances composing the cluster. `1` by default.
-* GERRIT_KEY_PREFIX : Optional. Secrets prefix used during the [Import into AWS Secret Manager](#import-into-aws-secret-manager).
- `gerrit_secret` by default.
-* `GERRIT_RAM`: RAM allocated (MiB) to the Gerrit container. `70000` by default.
-* `GERRIT_CPU`: vCPU units allocated to the Gerrit container. `10240` by default.
-* `GERRIT_HEAP_LIMIT`: Maximum heap size of the Java process running Gerrit, in bytes.
- See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#container.heapLimit)
- `35g` by default.
-* `JGIT_CACHE_SIZE`: Maximum number of bytes to load and cache in memory from pack files.
- See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#core.packedGitLimit)
- for more details. `12g` by default.
+* `CLUSTER_DESIRED_CAPACITY`: Optional. Number of EC2 instances composing the cluster. `1` by default.
+* `HTTP_HOST_PORT_MASTER1`: Optional. Gerrit Host HTTP port for master1 (must be different from master2). `9080` by default.
+* `SSH_HOST_PORT_MASTER1:`: Optional. Gerrit Host SSH port for master1 (must be different from master2). `29418` by default.
+* `HTTP_HOST_PORT_MASTER2`: Optional. Gerrit Host HTTP port for master2 (must be different from master1). `9080` by default.
+* `SSH_HOST_PORT_MASTER2:`: Optional. Gerrit Host SSH port for master2 (must be different from master1). `29418` by default.
+* `SLAVE_SUBDOMAIN`: Mandatory. The subdomain of the Gerrit slave. For example: `<AWS_PREFIX>-slave`
+* `LB_SUBDOMAIN`: Mandatory. The subdomain of the Gerrit load balancer. For example: `<AWS_PREFIX>-dual-master`
-### Prerequisites
-
-The prerequisites to run this stack are:
-* a registered and correctly configured domain in
-[Route53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/getting-started.html)
-* to [publish the Docker image](#publish-custom-gerrit-docker-image) with your
-Gerrit configuration in AWS ECR
-* to [add Gerrit secrets](#add-gerrit-secrets-in-aws-secret-manager) in AWS Secret
-Manager
-* an SSL Certificate in AWS Certificate Manager (you can find more information on
- how to create and handle certificates in AWS [here](https://aws.amazon.com/certificate-manager/getting-started/)
-
-### Add Gerrit Secrets in AWS Secret Manager
-
-[AWS Secret Manager](https://aws.amazon.com/secrets-manager/) is a secure way of
-storing and managing secrets of any type.
-
-The secrets you will have to add are the Gerrit SSH keys and the Register Email
-Private Key set in `secure.config`.
-
-#### SSH Keys
-
-The SSH keys you will need to add are the one usually created and used by Gerrit:
-* ssh_host_ecdsa_384_key
-* ssh_host_ecdsa_384_key.pub
-* ssh_host_ecdsa_521_key
-* ssh_host_ecdsa_521_key.pub
-* ssh_host_ecdsa_key
-* ssh_host_ecdsa_key.pub
-* ssh_host_ed25519_key
-* ssh_host_ed25519_key.pub
-* ssh_host_rsa_key
-* ssh_host_rsa_key.pub
-
-Plus a key used by the replication plugin:
-* replication_user_id_rsa
-* replication_user_id_rsa.pub
-
-You will have to create the keys and place them in a directory.
-
-#### Register Email Private Key
-
-You will need to create a secret and put it in a file called `registerEmailPrivateKey`
-in the same directory of the SSH keys.
-
-#### LDAP Password
-
-You will need to put the admin LDAP password in a file called `ldapPassword`
-in the same directory of the SSH keys.
-
-#### SMTP Password
-
-You will need to put the SMTP password in a file called `smtpPassword`
-in the same directory of the SSH keys.
-
-#### Import into AWS Secret Manager
-
-You can now run the [script](../gerrit/add_secrets_aws_secrets_manager.sh) to
-upload them to AWS Secret Manager:
-`add_secrets_aws_secrets_manager.sh /path/to/your/keys/directory secret_prefix aws-region-id`
-
-When `secret_prefix` is omitted, it is set to `gerrit_secret` by default.
-
-### Publish custom Gerrit Docker image
-
-* Create the repository in the Docker registry:
- `aws ecr create-repository --repository-name aws-gerrit/gerrit`
-* Set the Docker registry URI in `DOCKER_REGISTRY_URI`
-* Create a `gerrit.setup` and set the correct parameters
- * An example of the possible setting are in `gerrit.setup.template`
- * The structure and parameters of `gerrit.setup` are the same as a normal `gerrit.config`
- * Refer to the [Gerrit Configuration Documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html)
- for the meaning of the parameters
-* Add the plugins you want to install in `./gerrit/plugins`
-* Publish the image: `make gerrit-publish`
-
-### Publish custom HAProxy Docker image
-
-* Create the HAProxy and the logging sidecar repository in the Docker registry:
- `aws ecr create-repository --repository-name aws-gerrit/haproxy`
- `aws ecr create-repository --repository-name aws-gerrit/syslog-sidecar`
-* Publish the images:
- `make haproxy-publish`
- `make syslog-sidecar-publish`
-
-### Getting Started
+### 2 - Deploy
* Create the cluster, services and DNS routing stacks:
```
-make create-all
+make [AWS_REGION=a-valid-aws-region] [AWS_PREFIX=some-cluster-prefix] create-all
```
-*NOTE: the creation of the cluster needs an EC2 key pair are useful when you need to connect
+The optional `AWS_REGION` and `AWS_REFIX` allow you to define where it will be deployed and what it will be named.
+
+It might take several minutes to build the stack.
+You can monitor the creations of the stacks in [CloudFormation](https://console.aws.amazon.com/cloudformation/home)
+
+* *NOTE*: the creation of the cluster needs an EC2 key pair are useful when you need to connect
to the EC2 instances for troubleshooting purposes. The key pair is automatically generated
-and store them in a `pem` file on the current directory.
-To use when ssh-ing into your instances as follow: `ssh -i cluster-keys.pem ec2-user@<ec2_instance_ip>`*
+and stored in a `pem` file on the current directory.
+To use when ssh-ing into your instances as follow: `ssh -i cluster-keys.pem ec2-user@<ec2_instance_ip>`
### Cleaning up
```
-make delete-all
+make [AWS_REGION=a-valid-aws-region] [AWS_PREFIX=some-cluster-prefix] delete-all
```
+The optional `AWS_REGION` and `AWS_REFIX` allow you to specify exactly which stack you target for deletion.
+
+Note that this will *not* delete:
+* Secrets stored in Secret Manager
+* SSL certificates
+* ECR repositories
+
### Access your Gerrit instances
Get the URL of your Gerrit master instances this way:
@@ -212,21 +141,11 @@
* HTTP `8080`
* SSH `29418`
-# External services
+### External Services
-This is a list of external services that you might need to setup your stack and some suggestions
-on how to easily create them.
+If you need to setup some external services (maybe for testing purposes, such as SMTP or LDAP),
+you can follow the instructions [here](../README.md#external-services)
-## SMTP Server
+### Docker
-If you need to setup a SMTP service Amazon Simple Email Service can be used.
-Details how setup Amazon SES can be found [here](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-set-up.html).
-
-To correctly setup email notifications Gerrit requires ssl protocol on default port 465 to
-be enabled on SMTP Server. It is possible to setup Gerrit to talk to standard SMTP port 25
-but by default all EC2 instances are blocking it. To enable port 25 please follow [this](https://aws.amazon.com/premiumsupport/knowledge-center/ec2-port-25-throttle/) link.
-
-## LDAP Server
-
-If you need a testing LDAP server you can find details on how to easily
-create one in the [LDAP folder](../ldap/README.md).
+Refer to the [Docker](../Docker.md) section for information on how to setup docker or how to publish images
\ No newline at end of file
diff --git a/dual-master/cf-cluster.yml b/dual-master/cf-cluster.yml
index 3a70b84..f5f07db 100644
--- a/dual-master/cf-cluster.yml
+++ b/dual-master/cf-cluster.yml
@@ -109,6 +109,7 @@
done; (exit $s)
mkdir -p $DIR_TGT/git
mkdir -p $DIR_TGT/high-availability
+ mkdir -p $DIR_TGT/events
chown -R 1000:1000 $DIR_TGT
cp -p /etc/fstab /etc/fstab.back-$(date +%F)
echo -e \"$DIR_SRC:/ \t\t $DIR_TGT \t\t nfs \t\t defaults \t\t 0 \t\t 0\" | tee -a /etc/fstab
diff --git a/dual-master/cf-service-master.yml b/dual-master/cf-service-master.yml
index de62c94..f4ecec5 100644
--- a/dual-master/cf-service-master.yml
+++ b/dual-master/cf-service-master.yml
@@ -11,6 +11,9 @@
Description: Stack name of the ECS cluster to deply the serivces
Type: String
Default: gerrit-cluster
+ TemplateBucketName:
+ Description: S3 bucket containing cloudformation templates
+ Type: String
EnvironmentName:
Description: An environment name used to build the log stream names
Type: String
@@ -54,6 +57,14 @@
Description: Gerrit SSH port
Type: Number
Default: 29418
+ GitPort:
+ Description: Git daemon port
+ Type: Number
+ Default: 9418
+ GitSSHPort:
+ Description: Git ssh port
+ Type: Number
+ Default: 1022
CertificateArn:
Description: SSL Certificates ARN
Type: String
@@ -64,6 +75,9 @@
Description: The subdomain of the Gerrit cluster
Type: String
Default: gerrit-master-demo
+ SlaveSubdomain:
+ Description: The subdomain of the Gerrit slave
+ Type: String
LBSubdomain:
Description: The subdomain of the Gerrit load balancer
Type: String
@@ -77,6 +91,10 @@
Description: Gerrit git volume name
Type: String
Default: gerrit-git-master
+ GerritEventsVolume:
+ Description: Gerrit replication events volume name
+ Type: String
+ Default: gerrit-events-master
GerritWebsessionsVolume:
Description: Gerrit git volume name
Type: String
@@ -121,7 +139,27 @@
Description: JGit cache size
Type: String
Default: 12g
-
+ LDAPServer:
+ Description: LDAP server URL
+ Type: String
+ LDAPUsername:
+ Description: Username to bind to the LDAP server with
+ Type: String
+ LDAPAccountBase:
+ Description: Root of the tree containing all user accounts
+ Type: String
+ LDAPGroupBase:
+ Description: Root of the tree containing all group objects
+ Type: String
+ SMTPServer:
+ Description: SMTP server URL
+ Type: String
+ SMTPUser:
+ Description: User name to authenticate with, if required for relay
+ Type: String
+ SMTPDomain:
+ Description: Domain to be used in the From field
+ Type: String
Resources:
Service:
@@ -177,6 +215,26 @@
Value: !Ref GerritHeapLimit
- Name: JGIT_CACHE_SIZE
Value: !Ref JgitCacheSize
+ - Name: LDAP_SERVER
+ Value: !Ref LDAPServer
+ - Name: LDAP_USERNAME
+ Value: !Ref LDAPUsername
+ - Name: LDAP_ACCOUNT_BASE
+ Value: !Ref LDAPAccountBase
+ - Name: LDAP_GROUP_BASE
+ Value: !Ref LDAPGroupBase
+ - Name: SMTP_SERVER
+ Value: !Ref SMTPServer
+ - Name: SMTP_USER
+ Value: !Ref SMTPUser
+ - Name: SMTP_DOMAIN
+ Value: !Ref SMTPDomain
+ - Name: GIT_PORT
+ Value: !Ref GitPort
+ - Name: GIT_SSH_PORT
+ Value: !Ref GitSSHPort
+ - Name: SLAVE_SUBDOMAIN
+ Value: !Ref SlaveSubdomain
MountPoints:
- SourceVolume: !Ref GerritGitVolume
ContainerPath: /var/gerrit/git
@@ -192,6 +250,8 @@
ContainerPath: /var/gerrit/db
- SourceVolume: !Ref GerritLogsVolume
ContainerPath: /var/gerrit/logs
+ - SourceVolume: !Ref GerritEventsVolume
+ ContainerPath: /var/gerrit/events
Cpu: !Ref GerritCPU
Memory: !Ref GerritRAM
PortMappings:
@@ -211,6 +271,9 @@
- Name: !Ref 'GerritGitVolume'
Host:
SourcePath: "/mnt/efs/gerrit-shared/git"
+ - Name: !Ref 'GerritEventsVolume'
+ Host:
+ SourcePath: "/mnt/efs/gerrit-shared/events"
- Name: !Ref 'GerritWebsessionsVolume'
Host:
SourcePath: "/mnt/efs/gerrit-shared/high-availability"
@@ -308,7 +371,7 @@
ECSTaskExecutionRoleStack:
Type: AWS::CloudFormation::Stack
Properties:
- TemplateURL: https://aws-gerrit-cf-templates.s3.amazonaws.com/cf-gerrit-task-execution-role.yml
+ TemplateURL: !Join [ '', ['https://', !Ref TemplateBucketName, '.s3.amazonaws.com/cf-gerrit-task-execution-role.yml'] ]
TimeoutInMinutes: '5'
Outputs:
diff --git a/dual-master/cf-service-slave.yml b/dual-master/cf-service-slave.yml
index 01da4ac..02ef28c 100644
--- a/dual-master/cf-service-slave.yml
+++ b/dual-master/cf-service-slave.yml
@@ -14,6 +14,9 @@
Description: Stack name of the ECS cluster to deply the serivces
Type: String
Default: gerrit-cluster
+ TemplateBucketName:
+ Description: S3 bucket containing cloudformation templates
+ Type: String
EnvironmentName:
Description: An environment name used to build the log stream names
Type: String
@@ -118,6 +121,18 @@
Description: JGit cache size
Type: String
Default: 12g
+ LDAPServer:
+ Description: LDAP server URL
+ Type: String
+ LDAPUsername:
+ Description: Username to bind to the LDAP server with
+ Type: String
+ LDAPAccountBase:
+ Description: Root of the tree containing all user accounts
+ Type: String
+ LDAPGroupBase:
+ Description: Root of the tree containing all group objects
+ Type: String
Resources:
GerritService:
@@ -173,6 +188,14 @@
Value: !Ref GerritHeapLimit
- Name: JGIT_CACHE_SIZE
Value: !Ref JgitCacheSize
+ - Name: LDAP_SERVER
+ Value: !Ref LDAPServer
+ - Name: LDAP_USERNAME
+ Value: !Ref LDAPUsername
+ - Name: LDAP_ACCOUNT_BASE
+ Value: !Ref LDAPAccountBase
+ - Name: LDAP_GROUP_BASE
+ Value: !Ref LDAPGroupBase
MountPoints:
- SourceVolume: !Ref GerritGitVolume
ContainerPath: /var/gerrit/git
@@ -396,7 +419,7 @@
ECSTaskExecutionRoleStack:
Type: AWS::CloudFormation::Stack
Properties:
- TemplateURL: https://aws-gerrit-cf-templates.s3.amazonaws.com/cf-gerrit-task-execution-role.yml
+ TemplateURL: !Join [ '', ['https://', !Ref TemplateBucketName, '.s3.amazonaws.com/cf-gerrit-task-execution-role.yml'] ]
TimeoutInMinutes: '5'
Outputs:
diff --git a/dual-master/setup.env.template b/dual-master/setup.env.template
index 25f3e87..c340d14 100644
--- a/dual-master/setup.env.template
+++ b/dual-master/setup.env.template
@@ -1,4 +1,3 @@
-CLUSTER_STACK_NAME:=$(AWS_PREFIX)-cluster
CLUSTER_DESIRED_CAPACITY:=3
CLUSTER_INSTANCE_TYPE:=m4.2xlarge
SERVICE_MASTER1_STACK_NAME:=$(AWS_PREFIX)-service-master-1
@@ -20,3 +19,12 @@
GERRIT_CPU=10240
GERRIT_HEAP_LIMIT=35g
JGIT_CACHE_SIZE=12g
+
+LDAP_SERVER:=ldap://yourldap.yourcompany.com
+LDAP_USERNAME:=cn=admin,dc=example,dc=org
+LDAP_ACCOUNT_BASE:=dc=example,dc=org
+LDAP_GROUP_BASE:=dc=example,dc=org
+
+SMTP_SERVER:=yoursmtp.yourcompany.com
+SMTP_USER:=smtpuser
+SMTP_DOMAIN:=mail.yourcompany.com
diff --git a/gerrit/Dockerfile b/gerrit/Dockerfile
index 5586696..26dec1f 100644
--- a/gerrit/Dockerfile
+++ b/gerrit/Dockerfile
@@ -1,16 +1,13 @@
-FROM gerritcodereview/gerrit:$GERRIT_VERSION.$GERRIT_PATCH-centos7
+FROM gerritcodereview/gerrit:$GERRIT_VERSION.$GERRIT_PATCH-centos8
USER root
-RUN yum install -y https://repo.ius.io/ius-release-el7.rpm \
- && yum install -y python36u python36u-libs python36u-devel python36u-pip
+RUN yum install -y python36 python3-libs python36-devel python3-pip
COPY --chown=gerrit:gerrit ssh-config /var/gerrit/.ssh/config
# Installing scripts to get SSH Keys from Secret Manager
COPY --chown=gerrit:gerrit requirements.txt /tmp
-COPY --chown=gerrit:gerrit gerrit.setup /tmp
-COPY --chown=gerrit:gerrit replication.setup /tmp
COPY --chown=gerrit:gerrit setup_gerrit.py /tmp
RUN chmod +x /tmp/setup_gerrit.py \
&& pip3 install -r /tmp/requirements.txt
diff --git a/gerrit/Makefile b/gerrit/Makefile
index 902e93a..a6d6719 100644
--- a/gerrit/Makefile
+++ b/gerrit/Makefile
@@ -15,12 +15,12 @@
|| { echo >&2 "Cannot download metrics-reporter-prometheus plugin: Check internet connection. Aborting"; exit 1; }
@echo "Downloading javamelody plugin $(GERRIT_BRANCH)"
- wget $(GERRIT_CI)/plugin-javamelody-bazel-$(GERRIT_BRANCH)/$(LAST_BUILD)/javamelody/javamelody.jar \
+ wget $(GERRIT_CI)/plugin-javamelody-bazel-master-$(GERRIT_BRANCH)/$(LAST_BUILD)/javamelody/javamelody.jar \
-O ./plugins/javamelody.jar \
|| { echo >&2 "Cannot download javamelody plugin: Check internet connection. Aborting"; exit 1; }
@echo "Downloading HA plugin $(GERRIT_BRANCH)"
- wget $(GERRIT_CI)/plugin-high-availability-bazel-$(GERRIT_BRANCH)/$(LAST_BUILD)/high-availability/high-availability.jar \
+ wget $(GERRIT_CI)/plugin-high-availability-bazel-master-$(GERRIT_BRANCH)/$(LAST_BUILD)/high-availability/high-availability.jar \
-O ./plugins/high-availability.jar \
|| { echo >&2 "Cannot download high-availability plugin: Check internet connection. Aborting"; exit 1; }
diff --git a/gerrit/etc/replication.config.template b/gerrit/etc/replication.config.template
index c03a807..814a4a9 100644
--- a/gerrit/etc/replication.config.template
+++ b/gerrit/etc/replication.config.template
@@ -1,6 +1,8 @@
[replication]
lockErrorMaxRetries = 30
maxRetries = 30
+ eventsDirectory = events
+ distributionInterval = 1
[gerrit]
autoReload = true
diff --git a/gerrit/gerrit.setup.template b/gerrit/gerrit.setup.template
deleted file mode 100644
index b019efe..0000000
--- a/gerrit/gerrit.setup.template
+++ /dev/null
@@ -1,12 +0,0 @@
-[ldap]
-server = ldap://yourldap.yourcompany.com
-username = cn=admin,dc=example,dc=org
-accountBase = dc=example,dc=org
-groupBase = dc=example,dc=org
-
-[smtp]
-server = yoursmtp.yourcompany.com
-user = smtpuser
-password = smtppassword
-domain = yourcompany.com
-
diff --git a/gerrit/replication.setup.template b/gerrit/replication.setup.template
deleted file mode 100644
index e5e018b..0000000
--- a/gerrit/replication.setup.template
+++ /dev/null
@@ -1,3 +0,0 @@
-[remote-slave]
- url = git://subdomain.hostedzonename:9418/${name}.git
- adminUrl = ssh://gerrit@$subdomain.hostedzonename:1022/var/gerrit/git/${name}.git
diff --git a/gerrit/requirements.txt b/gerrit/requirements.txt
index 45831a6..58ee30e 100644
--- a/gerrit/requirements.txt
+++ b/gerrit/requirements.txt
@@ -1,3 +1,2 @@
boto3
jinja2==2.11.1
-configparser==5.0.0
diff --git a/gerrit/setup_gerrit.py b/gerrit/setup_gerrit.py
index 54dabe8..2f15659 100755
--- a/gerrit/setup_gerrit.py
+++ b/gerrit/setup_gerrit.py
@@ -3,7 +3,6 @@
import boto3
import base64
import os
-import configparser
from botocore.exceptions import ClientError
from jinja2 import Environment, FileSystemLoader
@@ -61,7 +60,6 @@
It reads from:
- AWS Secret Manager: Statically defined.
- - gerrit.setup: Statically defined.
- environment variables: Dinamycally defined.
"""
@@ -120,8 +118,6 @@
)
BASE_CONFIG_DIR = "/tmp"
-config = configparser.ConfigParser()
-config.read(BASE_CONFIG_DIR + '/gerrit.setup')
print("Setting Gerrit config in '" + GERRIT_CONFIG_DIRECTORY + "gerrit.config'")
template = env.get_template("gerrit.config.template")
@@ -141,13 +137,13 @@
with open(GERRIT_CONFIG_DIRECTORY + "gerrit.config", 'w',
encoding='utf-8') as f:
config_for_template.update({
- 'LDAP_SERVER': config['ldap']['server'],
- 'LDAP_USERNAME': config['ldap']['username'],
- 'LDAP_ACCOUNT_BASE': config['ldap']['accountBase'],
- 'LDAP_GROUP_BASE': config['ldap']['groupBase'],
- 'SMTP_SERVER': config['smtp']["server"],
- 'SMTP_USER': config['smtp']["user"],
- 'SMTP_DOMAIN': config['smtp']["domain"],
+ 'LDAP_SERVER': os.getenv('LDAP_SERVER'),
+ 'LDAP_USERNAME': os.getenv('LDAP_USERNAME'),
+ 'LDAP_ACCOUNT_BASE': os.getenv('LDAP_ACCOUNT_BASE'),
+ 'LDAP_GROUP_BASE': os.getenv('LDAP_GROUP_BASE'),
+ 'SMTP_SERVER': os.getenv('SMTP_SERVER'),
+ 'SMTP_USER': os.getenv('SMTP_USER'),
+ 'SMTP_DOMAIN': os.getenv('SMTP_DOMAIN'),
'GERRIT_HEAP_LIMIT': os.getenv('GERRIT_HEAP_LIMIT'),
'JGIT_CACHE_SIZE': os.getenv('JGIT_CACHE_SIZE')
})
@@ -157,18 +153,17 @@
if ((not containerSlave) and setupReplication):
print("Setting Replication config in '" +
GERRIT_CONFIG_DIRECTORY + "replication.config'")
- config.read(BASE_CONFIG_DIR + '/replication.setup')
template = env.get_template("replication.config.template")
with open(GERRIT_CONFIG_DIRECTORY + "replication.config", 'w', encoding='utf-8') as f:
+ SLAVE_FQDN = os.getenv('SLAVE_SUBDOMAIN') + "." + os.getenv('HOSTED_ZONE_NAME')
f.write(template.render(
- SLAVE_1_URL=config['remote-slave']['url'],
- SLAVE_1_AMDIN_URL=config['remote-slave']['adminUrl']
+ SLAVE_1_URL="git://" + SLAVE_FQDN + ":" + os.getenv('GIT_PORT') + "/${name}.git",
+ SLAVE_1_AMDIN_URL="ssh://gerrit@" + SLAVE_FQDN + ":" + os.getenv('GIT_SSH_PORT') + "/var/gerrit/git/${name}.git"
))
if (setupHA):
print("Setting HA config in '" +
GERRIT_CONFIG_DIRECTORY + "high-availability.config'")
- config.read(BASE_CONFIG_DIR + '/high-availability.setup')
template = env.get_template("high-availability.config.template")
with open(GERRIT_CONFIG_DIRECTORY + "high-availability.config", 'w', encoding='utf-8') as f:
f.write(template.render(HA_PEER_URL=os.getenv('HA_PEER_URL')))
diff --git a/master-slave/Makefile b/master-slave/Makefile
index 3d3b7fd..2413277 100644
--- a/master-slave/Makefile
+++ b/master-slave/Makefile
@@ -1,5 +1,6 @@
-include ../Makefile.common
+include ../common.env # Must be included before setup.env because the latter depends on it
include setup.env
+include ../Makefile.common
CLUSTER_TEMPLATE:=cf-cluster.yml
SERVICE_MASTER_TEMPLATE:=cf-service-master.yml
@@ -22,6 +23,10 @@
dns-routing wait-for-dns-routing-creation
cluster: cluster-keys
+ifdef CLUSTER_INSTANCE_TYPE
+ $(eval OPTIONAL_PARAMS := $(OPTIONAL_PARAMS) ParameterKey=InstanceType,ParameterValue=$(CLUSTER_INSTANCE_TYPE))
+endif
+
$(AWS_FC_COMMAND) create-stack \
--stack-name $(CLUSTER_STACK_NAME) \
--capabilities CAPABILITY_IAM \
@@ -29,7 +34,8 @@
--region $(AWS_REGION) \
--parameters \
ParameterKey=DesiredCapacity,ParameterValue=$(CLUSTER_DESIRED_CAPACITY) \
- ParameterKey=ECSKeyName,ParameterValue=$(CLUSTER_KEYS)
+ ParameterKey=ECSKeyName,ParameterValue=$(CLUSTER_KEYS) \
+ $(OPTIONAL_PARAMS)
service-master:
$(AWS_FC_COMMAND) create-stack \
@@ -38,9 +44,18 @@
--template-body file://`pwd`/$(SERVICE_MASTER_TEMPLATE) \
--region $(AWS_REGION) \
--parameters \
+ ParameterKey=LDAPServer,ParameterValue=$(LDAP_SERVER) \
+ ParameterKey=LDAPUsername,ParameterValue=\"$(LDAP_USERNAME)\" \
+ ParameterKey=LDAPAccountBase,ParameterValue=\"$(LDAP_ACCOUNT_BASE)\" \
+ ParameterKey=LDAPGroupBase,ParameterValue=\"$(LDAP_GROUP_BASE)\" \
+ ParameterKey=SMTPServer,ParameterValue=$(SMTP_SERVER) \
+ ParameterKey=SMTPUser,ParameterValue=$(SMTP_USER) \
+ ParameterKey=SMTPDomain,ParameterValue=$(SMTP_DOMAIN) \
ParameterKey=ClusterStackName,ParameterValue=$(CLUSTER_STACK_NAME) \
+ ParameterKey=TemplateBucketName,ParameterValue=$(TEMPLATE_BUCKET_NAME) \
ParameterKey=HostedZoneName,ParameterValue=$(HOSTED_ZONE_NAME) \
ParameterKey=Subdomain,ParameterValue=$(MASTER_SUBDOMAIN) \
+ ParameterKey=SlaveSubdomain,ParameterValue=$(SLAVE_SUBDOMAIN) \
ParameterKey=DockerRegistryUrl,ParameterValue=$(DOCKER_REGISTRY_URI) \
ParameterKey=CertificateArn,ParameterValue=$(SSL_CERTIFICATE_ARN) \
ParameterKey=SlaveServiceStackName,ParameterValue=$(SERVICE_SLAVE_STACK_NAME) \
@@ -58,7 +73,12 @@
--template-body file://`pwd`/$(SERVICE_SLAVE_TEMPLATE) \
--region $(AWS_REGION) \
--parameters \
+ ParameterKey=LDAPServer,ParameterValue=$(LDAP_SERVER) \
+ ParameterKey=LDAPUsername,ParameterValue=\"$(LDAP_USERNAME)\" \
+ ParameterKey=LDAPAccountBase,ParameterValue=\"$(LDAP_ACCOUNT_BASE)\" \
+ ParameterKey=LDAPGroupBase,ParameterValue=\"$(LDAP_GROUP_BASE)\" \
ParameterKey=ClusterStackName,ParameterValue=$(CLUSTER_STACK_NAME) \
+ ParameterKey=TemplateBucketName,ParameterValue=$(TEMPLATE_BUCKET_NAME) \
ParameterKey=HostedZoneName,ParameterValue=$(HOSTED_ZONE_NAME) \
ParameterKey=Subdomain,ParameterValue=$(SLAVE_SUBDOMAIN) \
ParameterKey=DockerRegistryUrl,ParameterValue=$(DOCKER_REGISTRY_URI) \
diff --git a/master-slave/README.md b/master-slave/README.md
index 763d5c6..0f44bfa 100644
--- a/master-slave/README.md
+++ b/master-slave/README.md
@@ -54,157 +54,64 @@
## How to run it
-### Setup
+### 0 - Prerequisites
-The `setup.env.template` is an example of setup file for the creation of the stacks.
+Follow the steps described in the [Prerequisites](../Prerequisites.md) section
-Before creating the stacks, create a `setup.env` in the `Makefile` directory and
-correctly set the value of the environment variables.
+### 1 - Configuration
-This is the list of available parameters:
+Please refer to the [configuration docs](../Configuration.md) to understand how to set up the
+configuration and what common configuration values are needed.
+On top of that, you might set the additional parameters, specific for this recipe.
-* `DOCKER_REGISTRY_URI`: Mandatory. URI of the Docker registry. See the
- [prerequisites](#prerequisites) section for more details.
-* `SSL_CERTIFICATE_ARN`: Mandatory. ARN of the SSL Certificate.
-* `CLUSTER_STACK_NAME`: Optional. Name of the cluster stack. `gerrit-cluster` by default.
+#### Environment
+
+Configuration values affecting deployment environment and cluster properties
+
* `SERVICE_MASTER_STACK_NAME`: Optional. Name of the master service stack. `gerrit-service-master` by default.
* `SERVICE_SLAVE_STACK_NAME`: Optional. Name of the slave service stack. `gerrit-service-slave` by default.
-* `DNS_ROUTING_STACK_NAME`: Optional. Name of the DNS routing stack. `gerrit-dns-routing` by default.
-* `HOSTED_ZONE_NAME`: Optional. Name of the hosted zone. `mycompany.com` by default.
* `MASTER_SUBDOMAIN`: Optional. Name of the master sub domain. `gerrit-master-demo` by default.
* `SLAVE_SUBDOMAIN`: Optional. Name of the slave sub domain. `gerrit-slave-demo` by default.
-* GERRIT_KEY_PREFIX : Optional. Secrets prefix used during the [Import into AWS Secret Manager](#import-into-aws-secret-manager).
- `gerrit_secret` by default.
-* `CLUSTER_DESIRED_CAPACITY`: Optional. Number of EC2 instances composing the cluster. `1` by default.
-* `GERRIT_RAM`: RAM allocated (MiB) to the Gerrit container. `70000` by default.
-* `GERRIT_CPU`: vCPU units allocated to the Gerrit container. `10240` by default.
-* `GERRIT_HEAP_LIMIT`: Maximum heap size of the Java process running Gerrit, in bytes.
- See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#container.heapLimit)
- `35g` by default.
-* `JGIT_CACHE_SIZE`: Maximum number of bytes to load and cache in memory from pack files.
- See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#core.packedGitLimit)
- for more details. `12g` by default.
+* `CLUSTER_DESIRED_CAPACITY`: Optional. Number of EC2 instances composing the cluster. `1` by default.
-*NOTE: if you are planning to run the monitoring stack, set the
+*NOTE*: if you are planning to run the monitoring stack, set the
`CLUSTER_DESIRED_CAPACITY` value to at least 2. The resources provided by
a single EC2 instance won't be enough for all the services that will be ran*
-### Prerequisites
+* `PROMETHEUS_SUBDOMAIN`: Optional. Prometheus subdomain. For example: `<AWS_PREFIX>-prometheus`
+* `GRAFANA_SUBDOMAIN`: Optional. Grafana subdomain. For example: `<AWS_PREFIX>-grafana`
-As a prerequisite to run this stack, you will need:
-* a registered and correctly configured domain in
-[Route53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/getting-started.html)
-* to [publish the Docker image](#publish-custom-gerrit-docker-image) with your
-Gerrit configuration in AWS ECR
-* to [publish the SSH Agent Docker image](#publish-ssh-agent) in AWS ECR
-* to [publish the Git Daemon Docker image](#publish-git-daemon) in AWS ECR
-* to [add Gerrit secrets](#add-gerrit-secrets-in-aws-secret-manager) in AWS Secret
-Manager
-* an SSL Certificate in AWS Certificate Manager (you can find more information on
- how to create and handle certificates in AWS [here](https://aws.amazon.com/certificate-manager/getting-started/)
-
-### Add Gerrit Secrets in AWS Secret Manager
-
-[AWS Secret Manager](https://aws.amazon.com/secrets-manager/) is a secure way of
-storing and managing secrets of any type.
-
-The secrets you will have to add are the Gerrit SSH keys and the Register Email
-Private Key set in `secure.config`.
-
-#### SSH Keys
-
-The SSH keys you will need to add are the one usually created and used by Gerrit:
-* ssh_host_ecdsa_384_key
-* ssh_host_ecdsa_384_key.pub
-* ssh_host_ecdsa_521_key
-* ssh_host_ecdsa_521_key.pub
-* ssh_host_ecdsa_key
-* ssh_host_ecdsa_key.pub
-* ssh_host_ed25519_key
-* ssh_host_ed25519_key.pub
-* ssh_host_rsa_key
-* ssh_host_rsa_key.pub
-
-Plus a key used by the replication plugin:
-* replication_user_id_rsa
-* replication_user_id_rsa.pub
-
-Generate a random bearer token to be used for monitoring with Promtetheus:
-* `openssl rand -hex 20 > prometheus_bearer_token`
-
-You will have to create the keys and place them in a directory.
-
-#### Register Email Private Key
-
-You will need to create a secret and put it in a file called `registerEmailPrivateKey`
-in the same directory of the SSH keys.
-
-#### LDAP Password
-
-You will need to put the admin LDAP password in a file called `ldapPassword`
-in the same directory of the SSH keys.
-
-#### SMTP Password
-
-You will need to put the SMTP password in a file called `smtpPassword`
-in the same directory of the SSH keys.
-
-#### Import into AWS Secret Manager
-
-You can now run the [script](../gerrit/add_secrets_aws_secrets_manager.sh) to
-upload them to AWS Secret Manager:
-`add_secrets_aws_secrets_manager.sh /path/to/your/keys/directory secret_prefix aws-region-id`
-
-When `secret_prefix` is omitted, it is set to `gerrit_secret` by default.
-
-### Publish custom Gerrit Docker image
-
-* Create the repository in the Docker registry:
- `aws ecr create-repository --repository-name aws-gerrit/gerrit`
-* Set the Docker registry URI in `DOCKER_REGISTRY_URI`
-* Create a `gerrit.setup` and set the correct parameters
- * An example of the possible setting are in `gerrit.setup.template`
- * The structure and parameters of `gerrit.setup` are the same as a normal `gerrit.config`
- * Refer to the [Gerrit Configuration Documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html)
- for the meaning of the parameters
-* Add the plugins you want to install in `./gerrit/plugins`
-* Publish the image: `make gerrit-publish`
-
-### Publish SSH Agent
-
-* Create the repository in the Docker registry:
- `aws ecr create-repository --repository-name aws-gerrit/git-ssh`
-* Publish the image: `make git-ssh-publish`
-
-### Publish Git Daemon
-
-* Create the repository in the Docker registry:
- `aws ecr create-repository --repository-name aws-gerrit/git-daemon`
-* Publish the image: `make git-daemon-publish`
-
-### Getting Started
+### 2 - Deploy
* Create the cluster, services and DNS routing stacks:
```
-make create-all
+make [AWS_REGION=a-valid-aws-region] [AWS_PREFIX=some-cluster-prefix] create-all
```
-The slave will start with 5 min delay to allow the replication from master of `All-Users`
-and `All-Projects` to happen.
-You can now check in the slave logs to see when the slave is up and running.
+The optional `AWS_REGION` and `AWS_REFIX` allow you to define where it will be deployed and what it will be named.
-*NOTE: the creation of the cluster needs an EC2 key pair are useful when you need to connect
+It might take several minutes to build the stack.
+You can monitor the creations of the stacks in [CloudFormation](https://console.aws.amazon.com/cloudformation/home)
+
+* *NOTE*: the creation of the cluster needs an EC2 key pair are useful when you need to connect
to the EC2 instances for troubleshooting purposes. The key pair is automatically generated
-and store them in a `pem` file on the current directory.
-To use when ssh-ing into your instances as follow: `ssh -i cluster-keys.pem ec2-user@<ec2_instance_ip>`*
+and stored in a `pem` file on the current directory.
+To use when ssh-ing into your instances as follow: `ssh -i cluster-keys.pem ec2-user@<ec2_instance_ip>`
### Cleaning up
```
-make delete-all
+make [AWS_REGION=a-valid-aws-region] [AWS_PREFIX=some-cluster-prefix] delete-all
```
+The optional `AWS_REGION` and `AWS_REFIX` allow you to specify exactly which stack you target for deletion.
+
+Note that this will *not* delete:
+* Secrets stored in Secret Manager
+* SSL certificates
+* ECR repositories
+
### Access your Gerrit instances
Get the URL of your Gerrit master instance this way:
@@ -234,26 +141,16 @@
* HTTP `9080`
* SSH `39418`
-# External services
+### Monitoring
-This is a list of external services that you might need to setup your stack and some suggestions
-on how to easily create them.
-
-## SMTP Server
-
-If you need to setup a SMTP service Amazon Simple Email Service can be used.
-Details how setup Amazon SES can be found [here](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-set-up.html).
-
-To correctly setup email notifications Gerrit requires ssl protocol on default port 465 to
-be enabled on SMTP Server. It is possible to setup Gerrit to talk to standard SMTP port 25
-but by default all EC2 instances are blocking it. To enable port 25 please follow [this](https://aws.amazon.com/premiumsupport/knowledge-center/ec2-port-25-throttle/) link.
-
-## LDAP Server
-
-If you need a testing LDAP server you can find details on how to easily
-create one in the [LDAP folder](../ldap/README.md).
+* Standard CloudWatch monitoring metrics for each component
+* Optionally Prometheus and Grafana stack (see [here](../monitoring/README.md))
## Monitoring
If you want to monitor your system, you can add a Prometheus and Grafana stack.
[Here](../monitoring/README.md) you can find the details on how to add it.
+
+### Docker
+
+Refer to the [Docker](../Docker.md) section for information on how to setup docker or how to publish images
\ No newline at end of file
diff --git a/master-slave/cf-service-master.yml b/master-slave/cf-service-master.yml
index 7a2f4a3..e07b5a5 100644
--- a/master-slave/cf-service-master.yml
+++ b/master-slave/cf-service-master.yml
@@ -7,6 +7,9 @@
SlaveServiceStackName:
Type: String
Default: gerrit-slave
+ TemplateBucketName:
+ Description: S3 bucket containing cloudformation templates
+ Type: String
ClusterStackName:
Description: Stack name of the ECS cluster to deply the serivces
Type: String
@@ -42,6 +45,14 @@
Description: Gerrit SSH port
Type: Number
Default: 29418
+ GitPort:
+ Description: Git daemon port
+ Type: Number
+ Default: 9418
+ GitSSHPort:
+ Description: Git ssh port
+ Type: Number
+ Default: 1022
CertificateArn:
Description: SSL Certificates ARN
Type: String
@@ -52,6 +63,9 @@
Description: The subdomain of the Gerrit cluster
Type: String
Default: gerrit-master-demo
+ SlaveSubdomain:
+ Description: The subdomain of the Gerrit slave
+ Type: String
GerritKeyPrefix:
Description: Gerrit credentials keys prefix
Type: String
@@ -95,6 +109,27 @@
Description: JGit cache size
Type: String
Default: 12g
+ LDAPServer:
+ Description: LDAP server URL
+ Type: String
+ LDAPUsername:
+ Description: Username to bind to the LDAP server with
+ Type: String
+ LDAPAccountBase:
+ Description: Root of the tree containing all user accounts
+ Type: String
+ LDAPGroupBase:
+ Description: Root of the tree containing all group objects
+ Type: String
+ SMTPServer:
+ Description: SMTP server URL
+ Type: String
+ SMTPUser:
+ Description: User name to authenticate with, if required for relay
+ Type: String
+ SMTPDomain:
+ Description: Domain to be used in the From field
+ Type: String
Resources:
Service:
@@ -142,6 +177,28 @@
Value: !Ref GerritHeapLimit
- Name: JGIT_CACHE_SIZE
Value: !Ref JgitCacheSize
+ - Name: LDAP_SERVER
+ Value: !Ref LDAPServer
+ - Name: LDAP_USERNAME
+ Value: !Ref LDAPUsername
+ - Name: LDAP_ACCOUNT_BASE
+ Value: !Ref LDAPAccountBase
+ - Name: LDAP_GROUP_BASE
+ Value: !Ref LDAPGroupBase
+ - Name: SMTP_SERVER
+ Value: !Ref SMTPServer
+ - Name: SMTP_USER
+ Value: !Ref SMTPUser
+ - Name: SMTP_DOMAIN
+ Value: !Ref SMTPDomain
+ - Name: GIT_PORT
+ Value: !Ref GitPort
+ - Name: GIT_SSH_PORT
+ Value: !Ref GitSSHPort
+ - Name: SLAVE_SUBDOMAIN
+ Value: !Ref SlaveSubdomain
+ - Name: HOSTED_ZONE_NAME
+ Value: !Ref HostedZoneName
MountPoints:
- SourceVolume: !Ref GerritGitVolume
ContainerPath: /var/gerrit/git
@@ -274,7 +331,7 @@
ECSTaskExecutionRoleStack:
Type: AWS::CloudFormation::Stack
Properties:
- TemplateURL: https://aws-gerrit-cf-templates.s3.amazonaws.com/cf-gerrit-task-execution-role.yml
+ TemplateURL: !Join [ '', ['https://', !Ref TemplateBucketName, '.s3.amazonaws.com/cf-gerrit-task-execution-role.yml'] ]
TimeoutInMinutes: '5'
Outputs:
diff --git a/master-slave/cf-service-slave.yml b/master-slave/cf-service-slave.yml
index 01da4ac..f02130d 100644
--- a/master-slave/cf-service-slave.yml
+++ b/master-slave/cf-service-slave.yml
@@ -14,6 +14,9 @@
Description: Stack name of the ECS cluster to deply the serivces
Type: String
Default: gerrit-cluster
+ TemplateBucketName:
+ Description: S3 bucket containing cloudformation templates
+ Type: String
EnvironmentName:
Description: An environment name used to build the log stream names
Type: String
@@ -118,6 +121,18 @@
Description: JGit cache size
Type: String
Default: 12g
+ LDAPServer:
+ Description: LDAP server URL
+ Type: String
+ LDAPUsername:
+ Description: Username to bind to the LDAP server with
+ Type: String
+ LDAPAccountBase:
+ Description: Root of the tree containing all user accounts
+ Type: String
+ LDAPGroupBase:
+ Description: Root of the tree containing all group objects
+ Type: String
Resources:
GerritService:
@@ -173,6 +188,14 @@
Value: !Ref GerritHeapLimit
- Name: JGIT_CACHE_SIZE
Value: !Ref JgitCacheSize
+ - Name: LDAP_SERVER
+ Value: !Ref LDAPServer
+ - Name: LDAP_USERNAME
+ Value: !Ref LDAPUsername
+ - Name: LDAP_ACCOUNT_BASE
+ Value: !Ref LDAPAccountBase
+ - Name: LDAP_GROUP_BASE
+ Value: !Ref LDAPGroupBase
MountPoints:
- SourceVolume: !Ref GerritGitVolume
ContainerPath: /var/gerrit/git
@@ -396,7 +419,7 @@
ECSTaskExecutionRoleStack:
Type: AWS::CloudFormation::Stack
Properties:
- TemplateURL: https://aws-gerrit-cf-templates.s3.amazonaws.com/cf-gerrit-task-execution-role.yml
+ TemplateURL: !Join [ '', ['https://', !Ref TemplateBucketName, '.s3.amazonaws.com/cf-gerrit-task-execution-role.yml'] ]
TimeoutInMinutes: '5'
Outputs:
diff --git a/master-slave/git-ssh/setup_ssh.py b/master-slave/git-ssh/setup_ssh.py
index 362df06..c42fb3d 100644
--- a/master-slave/git-ssh/setup_ssh.py
+++ b/master-slave/git-ssh/setup_ssh.py
@@ -53,7 +53,6 @@
It reads from:
- AWS Secret Manager: Statically defined.
- - gerrit.setup: Statically defined.
- environment variables: Dinamycally defined.
"""
diff --git a/master-slave/setup.env.template b/master-slave/setup.env.template
index 4edded0..32fe85e 100644
--- a/master-slave/setup.env.template
+++ b/master-slave/setup.env.template
@@ -1,5 +1,5 @@
-CLUSTER_STACK_NAME:=$(AWS_PREFIX)-cluster
CLUSTER_DESIRED_CAPACITY:=1
+CLUSTER_INSTANCE_TYPE:=m4.10xlarge
SERVICE_MASTER_STACK_NAME:=$(AWS_PREFIX)-service-master
SERVICE_SLAVE_STACK_NAME:=$(AWS_PREFIX)-service-slave
SERVICE_PROMETHEUS_STACK_NAME:=$(AWS_PREFIX)-prometheus
@@ -8,7 +8,7 @@
DNS_ROUTING_MONITORING_STACK_NAME:=$(AWS_PREFIX)-monitoring-dns-routing
HOSTED_ZONE_NAME:=yourcompany.com
MASTER_SUBDOMAIN:=$(AWS_PREFIX)-master.gerrit-demo
-SLAVE_SUBDOMAIN:=$(AWS_PREFIX).gerrit-demo
+SLAVE_SUBDOMAIN:=$(AWS_PREFIX)-slave.gerrit-demo
PROMETHEUS_SUBDOMAIN:=$(AWS_PREFIX)-prometheus.gerrit-demo
GRAFANA_SUBDOMAIN:=$(AWS_PREFIX)-grafana.gerrit-demo
DOCKER_REGISTRY_URI:=<yourAccountId>.dkr.ecr.us-east-1.amazonaws.com
@@ -18,3 +18,12 @@
GERRIT_CPU=10240
GERRIT_HEAP_LIMIT=35g
JGIT_CACHE_SIZE=12g
+
+LDAP_SERVER:=ldap://yourldap.yourcompany.com
+LDAP_USERNAME:=cn=admin,dc=example,dc=org
+LDAP_ACCOUNT_BASE:=dc=example,dc=org
+LDAP_GROUP_BASE:=dc=example,dc=org
+
+SMTP_SERVER:=yoursmtp.yourcompany.com
+SMTP_USER:=smtpuser
+SMTP_DOMAIN:=mail.yourcompany.com
diff --git a/monitoring/Makefile b/monitoring/Makefile
index f703011..5e1cc89 100644
--- a/monitoring/Makefile
+++ b/monitoring/Makefile
@@ -1,3 +1,4 @@
+include ../common.env # Must be included before setup.env because the latter depends on it
include ../$(RECIPE)/setup.env
AWS_FC_COMMAND=export AWS_PAGER=;aws cloudformation
diff --git a/monitoring/README.md b/monitoring/README.md
index e9e098c..f454504 100644
--- a/monitoring/README.md
+++ b/monitoring/README.md
@@ -8,35 +8,12 @@
## Prerequisites
-### Publish custom Prometheus Docker image
-
-* Create the repository in the Docker registry:
- `aws ecr create-repository --repository-name aws-gerrit/prometheus`
-
-* From the main cookbook, publish the image: `make prometheus-publish`
-
-### Publish custom Grafana Docker image
-
-* Create the repository in the Docker registry:
- `aws ecr create-repository --repository-name aws-gerrit/grafana`
-
-* From the main cookbook, publish the image: `make grafana-publish`
-
### Import a Prometheus Bearer Token
* [Generate](https://www.uuidgenerator.net/) a Token
-* Import the Token in AWS secret manager with the provided script:
-```
-> add_prometheus_secrets_to_aws_secret_manager.sh <yourToken>
- Adding Prometheus Bearer Token...
- {
- "ARN": "arn:aws:secretsmanager:us-east-1:<yourAccountId>:secret:gerrit_secret_prometheus_bearer_token-gXpAFL",
- "Name": "gerrit_secret_test_prometheus_bearer_token",
- "VersionId": "e19310a4-8078-4bdb-90b4-74ead48e4339"
- }
-```
+ Import the Token in AWS secret manager with the provided script [here](../Secrets.md#prometheus-bearer-token)
* Add `TOKEN_VERSION` to the main cookbook `setup.env`
- * Its value is the last part of the secret ARN, `gXpAFL` in this case
+* Its value is the last part of the secret ARN, `gXpAFL` in this case
### How to run it
@@ -69,3 +46,7 @@
The default credentials are:
* user `admin`
* password `admin`
+
+### Docker
+
+Refer to the [Docker](../Docker.md) section for information on how to setup docker or how to publish images
\ No newline at end of file
diff --git a/monitoring/add_prometheus_secrets_to_aws_secret_manager.sh b/monitoring/add_prometheus_secrets_to_aws_secret_manager.sh
deleted file mode 100755
index 567e145..0000000
--- a/monitoring/add_prometheus_secrets_to_aws_secret_manager.sh
+++ /dev/null
@@ -1,18 +0,0 @@
-#!/bin/bash -e
-
-PROMETHEUS_BEARER_TOKEN=$1
-if [ -z "$PROMETHEUS_BEARER_TOKEN" ];
-then
- echo "Prometheus Bear Token must be specified"
- exit 1
-fi
-
-# Avoid to open output in less for each AWS command
-export AWS_PAGER=;
-KEY_PREFIX=gerrit_secret
-
-echo "Adding Prometheus Bearer Token..."
-
-aws secretsmanager create-secret --name ${KEY_PREFIX}_prometheus_bearer_token \
- --description "Prometheus Bearer Token" \
- --secret-string ${PROMETHEUS_BEARER_TOKEN}
diff --git a/single-master/Makefile b/single-master/Makefile
index 8d652bbe..c4519a1 100644
--- a/single-master/Makefile
+++ b/single-master/Makefile
@@ -1,3 +1,4 @@
+include ../common.env # Must be included before setup.env because the latter depends on it
include setup.env
include ../Makefile.common
@@ -19,13 +20,18 @@
dns-routing wait-for-dns-routing-creation
cluster: cluster-keys
+ifdef CLUSTER_INSTANCE_TYPE
+ $(eval OPTIONAL_PARAMS := $(OPTIONAL_PARAMS) ParameterKey=InstanceType,ParameterValue=$(CLUSTER_INSTANCE_TYPE))
+endif
+
$(AWS_FC_COMMAND) create-stack \
--stack-name $(CLUSTER_STACK_NAME) \
--capabilities CAPABILITY_IAM \
--template-body file://`pwd`/$(CLUSTER_TEMPLATE) \
--region $(AWS_REGION) \
--parameters \
- ParameterKey=ECSKeyName,ParameterValue=$(CLUSTER_KEYS)
+ ParameterKey=ECSKeyName,ParameterValue=$(CLUSTER_KEYS) \
+ $(OPTIONAL_PARAMS)
service:
$(AWS_FC_COMMAND) create-stack \
@@ -34,7 +40,15 @@
--template-body file://`pwd`/$(SERVICE_TEMPLATE) \
--region $(AWS_REGION) \
--parameters \
+ ParameterKey=LDAPServer,ParameterValue=$(LDAP_SERVER) \
+ ParameterKey=LDAPUsername,ParameterValue=\"$(LDAP_USERNAME)\" \
+ ParameterKey=LDAPAccountBase,ParameterValue=\"$(LDAP_ACCOUNT_BASE)\" \
+ ParameterKey=LDAPGroupBase,ParameterValue=\"$(LDAP_GROUP_BASE)\" \
+ ParameterKey=SMTPServer,ParameterValue=$(SMTP_SERVER) \
+ ParameterKey=SMTPUser,ParameterValue=$(SMTP_USER) \
+ ParameterKey=SMTPDomain,ParameterValue=$(SMTP_DOMAIN) \
ParameterKey=ClusterStackName,ParameterValue=$(CLUSTER_STACK_NAME) \
+ ParameterKey=TemplateBucketName,ParameterValue=$(TEMPLATE_BUCKET_NAME) \
ParameterKey=HostedZoneName,ParameterValue=$(HOSTED_ZONE_NAME) \
ParameterKey=Subdomain,ParameterValue=$(SUBDOMAIN) \
ParameterKey=DockerRegistryUrl,ParameterValue=$(DOCKER_REGISTRY_URI) \
diff --git a/single-master/README.md b/single-master/README.md
index 3d74d80..3cd843e 100644
--- a/single-master/README.md
+++ b/single-master/README.md
@@ -5,7 +5,7 @@
## Architecture
-Two templates are provided in this example:
+Three templates are provided in this example:
* `cf-cluster`: define the ECS cluster and the networking stack
* `cf-service`: defined the service stack running Gerrit
* `cf-dns-route`: defined the DNS routing for the service
@@ -45,6 +45,9 @@
### Monitoring
* Standard CloudWatch monitoring metrics for each component
+* Prometheus and Grafana stack is not available for this recipe yet. However the work has been done for
+the dual-master recipe and it could be easily adapted (you can find the relevant issue
+[here](https://bugs.chromium.org/p/gerrit/issues/detail?id=13092)).
## How to run it
@@ -53,146 +56,64 @@
However, keep reading this guide for a more exhaustive explanation.
-### Setup
+### 0 - Prerequisites
-The `setup.env.template` is an example of setup file for the creation of the stacks.
+Follow the steps described in the [Prerequisites](../Prerequisites.md) section
-Before creating the stacks, create a `setup.env` in the `Makefile` directory and
-correctly set the value of the environment variables.
+### 1 - Configuration
-This is the list of available parameters:
+Please refer to the [configuration docs](../Configuration.md) to understand how to set up the
+configuration and what common configuration values are needed.
+On top of that, you might set the additional parameters, specific for this recipe.
-* `DOCKER_REGISTRY_URI`: Mandatory. URI of the Docker registry. See the
- [prerequisites](#prerequisites) section for more details.
-* `SSL_CERTIFICATE_ARN`: Mandatory. ARN of the SSL Certificate.
-* `CLUSTER_STACK_NAME`: Optional. Name of the cluster stack. `gerrit-cluster` by default.
+#### Environment
+
+Configuration values affecting deployment environment and cluster properties
+
* `SERVICE_STACK_NAME`: Optional. Name of the service stack. `gerrit-service` by default.
-* `DNS_ROUTING_STACK_NAME`: Optional. Name of the DNS routing stack. `gerrit-dns-routing` by default.
-* `HOSTED_ZONE_NAME`: Optional. Name of the hosted zone. `mycompany.com` by default.
-* `SUBDOMAIN`: Optional. Name of the sub domain. `gerrit-master-demo` by default.
-* GERRIT_KEY_PREFIX : Optional. Secrets prefix used during the [Import into AWS Secret Manager](#import-into-aws-secret-manager).
- `gerrit_secret` by default.
-* `GERRIT_RAM`: RAM allocated (MiB) to the Gerrit container. `70000` by default.
-* `GERRIT_CPU`: vCPU units allocated to the Gerrit container. `10240` by default.
-* `GERRIT_HEAP_LIMIT`: Maximum heap size of the Java process running Gerrit, in bytes.
- See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#container.heapLimit)
- `35g` by default.
-* `JGIT_CACHE_SIZE`: Maximum number of bytes to load and cache in memory from pack files.
- See [Gerrit documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#core.packedGitLimit)
- for more details. `12g` by default.
-### Prerequisites
-
-As a prerequisite to run this stack, you will need:
-* a registered and correctly configured domain in
-[Route53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/getting-started.html)
-* to [publish the Docker image](#publish-custom-gerrit-docker-image) with your
-Gerrit configuration
-* to [add Gerrit secrets](#add-gerrit-secrets-in-aws-secret-manager) in AWS Secret
-Manager
-* an SSL Certificate in AWS Certificate Manager (you can find more information on
- how to create and handle certificates in AWS [here](https://aws.amazon.com/certificate-manager/getting-started/)
-
-### Add Gerrit Secrets in AWS Secret Manager
-
-[AWS Secret Manager](https://aws.amazon.com/secrets-manager/) is a secure way of
-storing and managing secrets of any type.
-
-The secrets you will have to add are the Gerrit SSH keys and the Register Email
-Private Key set in `secure.config`.
-
-#### SSH Keys
-
-The SSH keys you will need to add are the one usually created and used by Gerrit:
-* ssh_host_ecdsa_384_key
-* ssh_host_ecdsa_384_key.pub
-* ssh_host_ecdsa_521_key
-* ssh_host_ecdsa_521_key.pub
-* ssh_host_ecdsa_key
-* ssh_host_ecdsa_key.pub
-* ssh_host_ed25519_key
-* ssh_host_ed25519_key.pub
-* ssh_host_rsa_key
-* ssh_host_rsa_key.pub
-
-You will have to create the keys and place them in a directory.
-
-#### Register Email Private Key
-
-You will need to create a secret and put it in a file called `registerEmailPrivateKey`
-in the same directory of the SSH keys.
-
-#### LDAP Password
-
-You will need to put the admin LDAP password in a file called `ldapPassword`
-in the same directory of the SSH keys.
-
-#### SMTP Password
-
-You will need to put the SMTP password in a file called `smtpPassword`
-in the same directory of the SSH keys.
-
-#### Import into AWS Secret Manager
-
-You can now run the [script](../gerrit/add_secrets_aws_secrets_manager.sh) to
-upload them to AWS Secret Manager:
-`add_secrets_aws_secrets_manager.sh /path/to/your/keys/directory secret_prefix aws-region-id`
-
-When `secret_prefix` is omitted, it is set to `gerrit_secret` by default.
-
-### Publish custom Gerrit Docker image
-
-* Create the repository in the Docker registry:
- `aws ecr create-repository --repository-name aws-gerrit/gerrit`
-* Set the Docker registry URI in `DOCKER_REGISTRY_URI`
-* Create a `gerrit.setup` and set the correct parameters
- * An example of the possible setting are in `gerrit.setup.template`
- * The structure and parameters of `gerrit.setup` are the same as a normal `gerrit.config`
- * Refer to the [Gerrit Configuration Documentation](https://gerrit-review.googlesource.com/Documentation/config-gerrit.html)
- for the meaning of the parameters
-* Add the plugins you want to install in `./gerrit/plugins`
-* Publish the image: `make gerrit-publish`
-
-### Getting Started
+### 2 - Deploy
* Create the cluster, service and DNS routing stacks:
```
-make create-all
+make [AWS_REGION=a-valid-aws-region] [AWS_PREFIX=some-cluster-prefix] create-all
```
-*NOTE: the creation of the cluster needs an EC2 key pair are useful when you need to connect
+The optional `AWS_REGION` and `AWS_REFIX` allow you to define where it will be deployed and what it will be named.
+
+It might take several minutes to build the stack.
+You can monitor the creations of the stacks in [CloudFormation](https://console.aws.amazon.com/cloudformation/home)
+
+* *NOTE*: the creation of the cluster needs an EC2 key pair are useful when you need to connect
to the EC2 instances for troubleshooting purposes. The key pair is automatically generated
-and store them in a `pem` file on the current directory.
-To use when ssh-ing into your instances as follow: `ssh -i cluster-keys.pem ec2-user@<ec2_instance_ip>`*
+and stored in a `pem` file on the current directory.
+To use when ssh-ing into your instances as follow: `ssh -i cluster-keys.pem ec2-user@<ec2_instance_ip>`
### Cleaning up
```
-make delete-all
+make [AWS_REGION=a-valid-aws-region] [AWS_PREFIX=some-cluster-prefix] delete-all
```
+The optional `AWS_REGION` and `AWS_REFIX` allow you to specify exactly which stack you target for deletion.
+
+Note that this will *not* delete:
+* Secrets stored in Secret Manager
+* SSL certificates
+* ECR repositories
+
### Access your Gerrit
You Gerrit instance will be available at this URL: `http://<HOSTED_ZONE_NAME>.<SUBDOMAIN>`.
The available ports are `8080` for HTTP and `29418` for SSH.
-# External services
+### External Services
-This is a list of external services that you might need to setup your stack and some suggestions
-on how to easily create them.
+If you need to setup some external services (maybe for testing purposes, such as SMTP or LDAP),
+you can follow the instructions [here](../README.md#external-services)
-## SMTP Server
+### Docker
-If you need to setup a SMTP service Amazon Simple Email Service can be used.
-Details how setup Amazon SES can be found [here](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-set-up.html).
-
-To correctly setup email notifications Gerrit requires ssl protocol on default port 465 to
-be enabled on SMTP Server. It is possible to setup Gerrit to talk to standard SMTP port 25
-but by default all EC2 instances are blocking it. To enable port 25 please follow [this](https://aws.amazon.com/premiumsupport/knowledge-center/ec2-port-25-throttle/) link.
-
-## LDAP Server
-
-If you need a testing LDAP server you can find details on how to easily
-create one in the [LDAP folder](../ldap/README.md).
+Refer to the [Docker](../Docker.md) section for information on how to setup docker or how to publish images
\ No newline at end of file
diff --git a/single-master/cf-service.yml b/single-master/cf-service.yml
index a361e7d..d08ef0e 100644
--- a/single-master/cf-service.yml
+++ b/single-master/cf-service.yml
@@ -8,6 +8,9 @@
Description: Stack name of the ECS cluster to deply the serivces
Type: String
Default: gerrit-cluster
+ TemplateBucketName:
+ Description: S3 bucket containing cloudformation templates
+ Type: String
EnvironmentName:
Description: An environment name that will be prefixed to resource names
Type: String
@@ -88,6 +91,27 @@
Description: JGit cache size
Type: String
Default: 12g
+ LDAPServer:
+ Description: LDAP server URL
+ Type: String
+ LDAPUsername:
+ Description: Username to bind to the LDAP server with
+ Type: String
+ LDAPAccountBase:
+ Description: Root of the tree containing all user accounts
+ Type: String
+ LDAPGroupBase:
+ Description: Root of the tree containing all group objects
+ Type: String
+ SMTPServer:
+ Description: SMTP server URL
+ Type: String
+ SMTPUser:
+ Description: User name to authenticate with, if required for relay
+ Type: String
+ SMTPDomain:
+ Description: Domain to be used in the From field
+ Type: String
Resources:
Service:
@@ -133,6 +157,20 @@
Value: !Ref GerritHeapLimit
- Name: JGIT_CACHE_SIZE
Value: !Ref JgitCacheSize
+ - Name: LDAP_SERVER
+ Value: !Ref LDAPServer
+ - Name: LDAP_USERNAME
+ Value: !Ref LDAPUsername
+ - Name: LDAP_ACCOUNT_BASE
+ Value: !Ref LDAPAccountBase
+ - Name: LDAP_GROUP_BASE
+ Value: !Ref LDAPGroupBase
+ - Name: SMTP_SERVER
+ Value: !Ref SMTPServer
+ - Name: SMTP_USER
+ Value: !Ref SMTPUser
+ - Name: SMTP_DOMAIN
+ Value: !Ref SMTPDomain
MountPoints:
- SourceVolume: !Ref GerritGitVolume
ContainerPath: /var/gerrit/git
diff --git a/single-master/setup.env.template b/single-master/setup.env.template
index 55e5b39..2d15394 100644
--- a/single-master/setup.env.template
+++ b/single-master/setup.env.template
@@ -1,5 +1,5 @@
-CLUSTER_STACK_NAME:=$(AWS_PREFIX)-cluster
SERVICE_STACK_NAME:=$(AWS_PREFIX)-service
+CLUSTER_INSTANCE_TYPE:=m4.10xlarge
DNS_ROUTING_STACK_NAME:=$(AWS_PREFIX)-dns-routing
HOSTED_ZONE_NAME:=mycompany.com
SUBDOMAIN:=$(AWS_PREFIX)-master-demo
@@ -9,3 +9,12 @@
GERRIT_CPU=10240
GERRIT_HEAP_LIMIT=35g
JGIT_CACHE_SIZE=12g
+
+LDAP_SERVER:=ldap://yourldap.yourcompany.com
+LDAP_USERNAME:=cn=admin,dc=example,dc=org
+LDAP_ACCOUNT_BASE:=dc=example,dc=org
+LDAP_GROUP_BASE:=dc=example,dc=org
+
+SMTP_SERVER:=yoursmtp.yourcompany.com
+SMTP_USER:=smtpuser
+SMTP_DOMAIN:=mail.yourcompany.com
\ No newline at end of file