Doc: add architecture documentation Solves: Jira GER-2365 Change-Id: I52ebdc2441d7192682de884d7596448b8b3e1b3f
diff --git a/src/main/resources/Documentation/architecture.md b/src/main/resources/Documentation/architecture.md new file mode 100644 index 0000000..4c38500 --- /dev/null +++ b/src/main/resources/Documentation/architecture.md
@@ -0,0 +1,154 @@ +# Architecture + + + +## Listeners + +Events-eiffel plugin listens to `ref-updated` for `refs/heads/*` to create SCS +events, `ref-updated` for `refs/tags/*` to create tag events and +`patchset-created` to create SCC. + +You can disable event-triggering or configure which refs event creation is +triggered for in the [All-Project configuration](./config-project-level.md). + +## REST API + +The plugin sports a REST API to create missing events manually, also useful if +you want to do a "backfill" of historical commits in a controlled fashion +before enabling the event-triggered creation of Eiffel events: +* [POST SCC](./rest-api-events-eiffel-scc-post.md) +* [POST SCS](./rest-api-events-eiffel-scs-post.md) +* [POST ArtC](./rest-api-events-eiffel-artc-post.md) + +The REST APIs are not protected by the same ref-filter that the listeners are, +so it should be used with caution. For this reason the REST API endpoints +requires the caller to have the ADMINISTRATE_SERVER capability. + +## Parse queue + +Since each parse request potentially can result in walking hundreds of thousand +commits and create twice as many Eiffel events (a new branch in the linux kernel +would result in ~1.6m events). We need to limit the effect it has on the Gerrit +server. The plugin creates it's own work queue for incoming parse-requests. +Queued parse-requests can be monitored through Gerrit's +[show-queue command](../../../Documentation/cmd-show-queue.html). + +Number of executers used to consume this queue can be configured with +[EventParsing.poolSize](../config.md). +A reasonable poolsize for a large Gerrit instance should be between 2-4. + +## Event creation + +### Parsing + +For each Eiffel event creation request first we need to ensure that all the +event's parents are created. To +[find missing parent events](./basic-principle.md#identifying-missing-events) +we need to walk the commit graph until we find a commit that has a +corresponding event of the correct type. + +Parsing is done with custom commit-walkers +`src/main/java/com/googlesource/gerrit/plugins/eventseiffel/parsing/CommitsWalker.java`. + +These custom commit-walkers use an external state (whether the corresponding +Eiffel event is already created or not) to determine when to stop walking. This +state is fetched from the [EiffelEventHub](#event-hub). + +They also make sure to reuse the object database when walking for child events. +E.g. when parsing for SCS event creation we must first parse for corresponding +SCC as they must be created first. Creating and tearing down an object database +once for each event type is computationally costly and uses twice as much memory. + +### Mapping + +Once we have determined which commits we need to create events for we need to +map those commits to an Eiffel event. + +`com/googlesource/gerrit/plugins/eventseiffel/mapping/EiffelEventMapper.java` + +You can configure the mapping get values from the commit instead of using the +index to find change-data to populate the Eiffel event from by setting +[EventMapping.useIndex](../config.md) to `False`. This gives a noticeable +performance increase when doing a backfill of historical events. + +For tag-events (Artc and CD) you can set a namespace by configuring +[EiffelEvent.namespace](./config.md). This `namespace` becomes part of the purl +of the Artc +[data.identity](https://github.com/eiffel-community/eiffel/blob/edition-arica/eiffel-vocabulary/EiffelArtifactCreatedEvent.md#dataidentity) +and the +[data.name](https://github.com/eiffel-community/eiffel/blob/edition-arica/eiffel-vocabulary/EiffelCompositionDefinedEvent.md#dataname) +of the CD event. See example at: +[Tag representation](./basic-principle.md#tag-representation) (in the example +`namespace` is set to `gerrit-host`). + +## Publishing + +### Event hub + +`com/googlesource/gerrit/plugins/eventseiffel/EiffelEventHub.java` + +The event hub has four different functions: +* As an API for querying of existing Eiffel events and their properties. +* Keeping track of the [state](./basic-principle.md#consistency-vs-performance) + of the Eiffel events. +* Maintaining the queue containing the Eiffel events that should be published. +* Updating the local `EiffelEventIdCache` once an event is reported as published + and removed from the queue. +* Handling the lifecycle of the Consumer that feeds events to the Publisher. + +The thread-safe implementation +`com/googlesource/gerrit/plugins/eventseiffel/EiffelEventHubImpl.java` is +loosely modeled around LinkedBlockingQueue.java adopted for the more +complicated data model of Eiffel events. + +### Publish event worker + +`com/googlesource/gerrit/plugins/eventseiffel/PublishEventWorker.java` + +The responsibility of the `PublishEventWorker` is to take events from the queue +and feed them to the `EventPublisher`, which in turn is responsible for +publishing the events to Eiffel, as well as keeping track of if the publishing +was successful + +and report back to the Event Hub. + +### Event Publisher + +There is currently only one implementation of EiffelEventPublisher. This +implementation publishes the event to a RabbitMQ exchange: +`com/googlesource/gerrit/plugins/eventseiffel/mq/RabbitMqPublisher.java` + +### Event ID Cache + +`com/googlesource/gerrit/plugins/eventseiffel/cache/EiffelEventIdCacheImpl.java` + +The cache uses a persisted LoadingCache and is backed by the implementation of +`com/googlesource/gerrit/plugins/eventseiffel/eiffel/api/EventStorage.java` +Currently the only implementation of EventStorage uses: +[Eiffel GraphQL API](https://github.com/eiffel-community/eiffel-graphql-api) +and [Eiffel GoER](https://github.com/eiffel-community/eiffel-goer). + +## Extendability + +### Event Storage + +If you want to use a different EventStorage, e.g. a local DBMS like postgres or similar, you +will need to +* Write a Postgres implementation of the EventStorage interface. +* Write a + [Provider<EventStorage>](https://google.github.io/guice/api-docs/3.0/javadoc/com/google/inject/Provider.html) + for the implementation. +* Add configuration for this event storage and ensure that EventStorage is be + bound to that Provider when configured. + +### Publish to Eiffel + +If you use other means than a RabbitMQ exchange for publishing events to +Eiffel, e.g. +[REMRem](https://github.com/eiffel-community/eiffel-remremhttps://github.com/eiffel-community/eiffel-remrem) +or similar, you will need to: +* Write a REMRem implementation of the + `com/googlesource/gerrit/plugins/eventseiffel/eiffel/api/EiffelEventPublisher.java` + interface. +* Add configuration for this event publisher and ensure that EiffelEventPublisher is + bound to the REMRem implementation when configured for it.
diff --git a/src/main/resources/Documentation/images/architecture.png b/src/main/resources/Documentation/images/architecture.png new file mode 100644 index 0000000..6ee3588 --- /dev/null +++ b/src/main/resources/Documentation/images/architecture.png Binary files differ
