Carry retry metadata in requeued messages Consumers retrying failed messages need to know how many attempts were made, when the last one happened and which Gerrit instance is in charge of the retry. Persist EventMetadata, which contains the Retry state of each message, in the wire format so that consumers don't need to rely on in-memory representation of the retry-state of each message, facilitating handling of retried events. Add EventSerializer and EventDeserializer.deserializeMessage() so broker implementations share one wire format: the metadata is a top-level `brokerMeta` field of the event JSON, omitted when empty. This is a breaking API change: BrokerApi.requeue() and AckAwareConsumer.accept() take an extra EventMetadata parameter, and the default requeue(topic, event) implementation is removed, so broker implementations and ack-aware consumers must be updated. The wire format stays interoperable with older versions: messages without metadata are serialized exactly as before, older readers ignore the extra `brokerMeta` field, and messages written by older versions are read with EventMetadata.EMPTY. Bug: Issue 576781780 Change-Id: I2faaadbaf6ce18c9ae39e27b2a3080b8eb09116d
diff --git a/README.md b/README.md index f9cc10a..6fbc064 100644 --- a/README.md +++ b/README.md
@@ -127,6 +127,28 @@ publishing should fall back to the normal broker behavior, while partition-specific subscription cannot resolve a logical partition and fails if requested. +### Message Metadata + +Every message is delivered to `AckAwareConsumer.accept(event, metadata, ack)` +together with its `EventMetadata`, which is `EventMetadata.EMPTY` for +messages published with `BrokerApi.send()`. + +Consumers that fail to process a message can put it back on the topic with +`BrokerApi.requeue(topic, event, metadata)`, recording the failed attempt with +`metadata.nextRetry(instanceId)`. The `Retry` metadata holds the number of +failed attempts (`retryCount`), the epoch milliseconds of the last one +(`lastAttemptOn`) and the Gerrit instance id in charge of retrying the message +(`retryingInstanceId`). Every requeue publishes a new message carrying exactly +the metadata passed to `requeue()`: the retry state only changes when the +consumer calls `nextRetry()`. + +Broker implementations must persist the metadata passed to +`BrokerApi.requeue()` in the message: they serialize messages with +`EventSerializer` and read them back with +`EventDeserializer.deserializeMessage()`, which store the metadata in the +`brokerMeta` field of the event JSON. The field is omitted when the metadata is +empty, so messages without metadata keep the legacy format. + ### Broker Metrics When `StreamEventPublisher` is used user can optionally bind an implementation of
diff --git a/src/main/java/com/gerritforge/gerrit/eventbroker/AckAwareConsumer.java b/src/main/java/com/gerritforge/gerrit/eventbroker/AckAwareConsumer.java index b2fad0a..a1580e7 100644 --- a/src/main/java/com/gerritforge/gerrit/eventbroker/AckAwareConsumer.java +++ b/src/main/java/com/gerritforge/gerrit/eventbroker/AckAwareConsumer.java
@@ -15,8 +15,8 @@ package com.gerritforge.gerrit.eventbroker; /** - * A consumer that accepts a message and its acknowledgement handle, allowing for explicit - * acknowledgement. + * A consumer that accepts a message, its metadata and its acknowledgement handle, allowing for + * explicit acknowledgement. * * @param <T> the type of the event input to the operation */ @@ -26,8 +26,11 @@ * Performs this operation on the given argument. * * @param t the input argument + * @param metadata the metadata travelling with the message, {@link EventMetadata#EMPTY} when the + * message carries none * @param acknowledgement the message acknowledgement handle * @throws InterruptedException when the thread accepting the message is interrupted */ - void accept(T t, MessageAcknowledgement<T> acknowledgement) throws InterruptedException; + void accept(T t, EventMetadata metadata, MessageAcknowledgement<T> acknowledgement) + throws InterruptedException; }
diff --git a/src/main/java/com/gerritforge/gerrit/eventbroker/BrokerApi.java b/src/main/java/com/gerritforge/gerrit/eventbroker/BrokerApi.java index 4f65e76..b7c4572 100644 --- a/src/main/java/com/gerritforge/gerrit/eventbroker/BrokerApi.java +++ b/src/main/java/com/gerritforge/gerrit/eventbroker/BrokerApi.java
@@ -38,15 +38,16 @@ * Requeue a message into the topic. * * <p>Requeue an existing message to the topic as it was not processed yet or failed to be - * processed. + * processed. Implementations store {@code metadata} in the requeued message and pass it back to + * the consumer through {@link AckAwareConsumer#accept(Object, EventMetadata, + * MessageAcknowledgement)}. * * @param topic topic name * @param message to be re-queued to the topic + * @param metadata to be stored in the re-queued message * @return a future that returns when the message has been sent. */ - default ListenableFuture<Boolean> requeue(String topic, Event message) { - return send(topic, message); - } + ListenableFuture<Boolean> requeue(String topic, Event message, EventMetadata metadata); /** * Receive asynchronously a message from a topic, using an acknowledgement-aware consumer.
diff --git a/src/main/java/com/gerritforge/gerrit/eventbroker/EventDeserializer.java b/src/main/java/com/gerritforge/gerrit/eventbroker/EventDeserializer.java index babc9d7..208fd8a 100644 --- a/src/main/java/com/gerritforge/gerrit/eventbroker/EventDeserializer.java +++ b/src/main/java/com/gerritforge/gerrit/eventbroker/EventDeserializer.java
@@ -19,10 +19,19 @@ import com.google.gerrit.server.events.Event; import com.google.gerrit.server.events.EventGson; import com.google.gson.Gson; +import com.google.gson.JsonObject; import com.google.inject.Inject; public class EventDeserializer { + /** + * An event together with the broker metadata travelling with it. + * + * @param event the deserialized event + * @param metadata the message metadata, {@link EventMetadata#EMPTY} when absent + */ + public record Message(Event event, EventMetadata metadata) {} + private Gson gson; @Inject @@ -37,6 +46,22 @@ return resultEvent; } + /** + * Deserializes a message produced by {@link EventSerializer#serialize(Event, EventMetadata)}. + * + * @param json the serialized message + * @return the event and its metadata + */ + public Message deserializeMessage(String json) { + JsonObject jsonObject = gson.fromJson(json, JsonObject.class); + Event event = gson.fromJson(jsonObject, Event.class); + validate(event); + + EventMetadata metadata = + gson.fromJson(jsonObject.get(EventMetadata.JSON_FIELD), EventMetadata.class); + return new Message(event, metadata == null ? EventMetadata.EMPTY : metadata); + } + private void validate(Event event) { requireNonNull(event.type, "Event type cannot be null"); }
diff --git a/src/main/java/com/gerritforge/gerrit/eventbroker/EventMetadata.java b/src/main/java/com/gerritforge/gerrit/eventbroker/EventMetadata.java new file mode 100644 index 0000000..3340f79 --- /dev/null +++ b/src/main/java/com/gerritforge/gerrit/eventbroker/EventMetadata.java
@@ -0,0 +1,65 @@ +// Copyright (C) 2026 The Android Open Source Project +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package com.gerritforge.gerrit.eventbroker; + +import com.google.gerrit.common.Nullable; +import com.google.gerrit.server.util.time.TimeUtil; + +/** + * Broker metadata travelling with a message, separately from the event it carries. + * + * <p>The metadata is stored in the message by {@link BrokerApi#requeue(String, + * com.google.gerrit.server.events.Event, EventMetadata)} and handed back to consumers through + * {@link AckAwareConsumer#accept(Object, EventMetadata, MessageAcknowledgement)}. Every requeue + * publishes a new message carrying the metadata passed to it. + * + * @param retry retry state of the message, {@code null} when the message was never retried + */ +public record EventMetadata(@Nullable Retry retry) { + /** Name of the JSON field holding the metadata in a serialized message. */ + static final String JSON_FIELD = "brokerMeta"; + + /** Metadata of a message that was never retried. */ + public static final EventMetadata EMPTY = new EventMetadata(null); + + /** + * Retry state of a message whose processing failed at least once. + * + * @param retryCount number of failed processing attempts + * @param lastAttemptOn epoch milliseconds of the last failed processing attempt + * @param retryingInstanceId Gerrit instance id of the node in charge of retrying the message + */ + public record Retry(int retryCount, long lastAttemptOn, String retryingInstanceId) {} + + /** + * Returns whether the message processing failed at least once. + * + * @return {@code true} if the message carries a retry state + */ + public boolean isRetry() { + return retry != null; + } + + /** + * Returns the metadata recording one more failed processing attempt, performed now. + * + * @param retryingInstanceId Gerrit instance id of the node in charge of retrying the message + * @return new metadata with an incremented retry count + */ + public EventMetadata nextRetry(String retryingInstanceId) { + int retryCount = retry == null ? 0 : retry.retryCount(); + return new EventMetadata(new Retry(retryCount + 1, TimeUtil.nowMs(), retryingInstanceId)); + } +}
diff --git a/src/main/java/com/gerritforge/gerrit/eventbroker/EventSerializer.java b/src/main/java/com/gerritforge/gerrit/eventbroker/EventSerializer.java new file mode 100644 index 0000000..f4af78c --- /dev/null +++ b/src/main/java/com/gerritforge/gerrit/eventbroker/EventSerializer.java
@@ -0,0 +1,52 @@ +// Copyright (C) 2026 The Android Open Source Project +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package com.gerritforge.gerrit.eventbroker; + +import com.google.gerrit.server.events.Event; +import com.google.gerrit.server.events.EventGson; +import com.google.gson.Gson; +import com.google.gson.JsonObject; +import com.google.inject.Inject; + +/** + * Serializes events, and their message metadata, into the JSON read by {@link EventDeserializer}. + */ +public class EventSerializer { + + private final Gson gson; + + @Inject + public EventSerializer(@EventGson Gson gson) { + this.gson = gson; + } + + /** + * Serializes an event into a JSON message. + * + * <p>Non-empty metadata is stored in the {@code brokerMeta} field of the event JSON object. + * + * @param event the event to serialize + * @param metadata the metadata travelling with the event + * @return the serialized message + */ + public String serialize(Event event, EventMetadata metadata) { + if (metadata == null || metadata.equals(EventMetadata.EMPTY)) { + return gson.toJson(event); + } + JsonObject json = gson.toJsonTree(event).getAsJsonObject(); + json.add(EventMetadata.JSON_FIELD, gson.toJsonTree(metadata)); + return gson.toJson(json); + } +}
diff --git a/src/test/java/com/gerritforge/gerrit/eventbroker/EventDeserializerTest.java b/src/test/java/com/gerritforge/gerrit/eventbroker/EventDeserializerTest.java index 317b5bf..0058db6 100644 --- a/src/test/java/com/gerritforge/gerrit/eventbroker/EventDeserializerTest.java +++ b/src/test/java/com/gerritforge/gerrit/eventbroker/EventDeserializerTest.java
@@ -32,6 +32,28 @@ } @Test + public void deserializeMessageShouldReturnEmptyMetadataWhenAbsent() { + final String eventJson = "{ \"type\": \"project-created\", \"instanceId\":\"instance-id\" }"; + final EventDeserializer.Message message = deserializer.deserializeMessage(eventJson); + + assertThat(message.event().instanceId).isEqualTo("instance-id"); + assertThat(message.metadata()).isEqualTo(EventMetadata.EMPTY); + } + + @Test + public void nextRetryShouldIncrementRetryCount() { + EventMetadata metadata = EventMetadata.EMPTY.nextRetry("first-instance"); + + assertThat(metadata.retry().retryCount()).isEqualTo(1); + assertThat(metadata.retry().retryingInstanceId()).isEqualTo("first-instance"); + + EventMetadata secondMetadata = metadata.nextRetry("second-instance"); + + assertThat(secondMetadata.retry().retryCount()).isEqualTo(2); + assertThat(secondMetadata.retry().retryingInstanceId()).isEqualTo("second-instance"); + } + + @Test public void eventDeserializerShouldParseEvent() { final String eventJson = "{ \"type\": \"project-created\", \"instanceId\":\"instance-id\" }"; final Event event = deserializer.deserialize(eventJson);
diff --git a/src/test/java/com/gerritforge/gerrit/eventbroker/EventSerializerTest.java b/src/test/java/com/gerritforge/gerrit/eventbroker/EventSerializerTest.java new file mode 100644 index 0000000..5817b30 --- /dev/null +++ b/src/test/java/com/gerritforge/gerrit/eventbroker/EventSerializerTest.java
@@ -0,0 +1,95 @@ +// Copyright (C) 2026 The Android Open Source Project +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package com.gerritforge.gerrit.eventbroker; + +import static com.google.common.truth.Truth.assertThat; + +import com.google.gerrit.server.events.Event; +import com.google.gerrit.server.events.EventGsonProvider; +import com.google.gerrit.server.events.ProjectCreatedEvent; +import com.google.gson.Gson; +import org.junit.Before; +import org.junit.Test; + +public class EventSerializerTest { + private Gson gson; + private EventSerializer serializer; + private EventDeserializer deserializer; + private ProjectCreatedEvent event; + + @Before + public void setUp() { + gson = new EventGsonProvider().get(); + serializer = new EventSerializer(gson); + deserializer = new EventDeserializer(gson); + + event = new ProjectCreatedEvent(); + event.instanceId = "instance-id"; + event.projectName = "project"; + event.headName = "refs/heads/master"; + } + + @Test + public void serializeWithEmptyMetadataShouldRoundtripWithLegacyDeserialize() { + String serialized = serializer.serialize(event, EventMetadata.EMPTY); + + assertThat(serialized).isEqualTo(gson.toJson(event)); + Event deserialized = new EventDeserializer(gson).deserialize(serialized); + assertThat(deserialized).isInstanceOf(ProjectCreatedEvent.class); + ProjectCreatedEvent deserializedEvent = (ProjectCreatedEvent) deserialized; + assertThat(deserializedEvent.instanceId).isEqualTo(event.instanceId); + assertThat(deserializedEvent.projectName).isEqualTo(event.projectName); + assertThat(deserializedEvent.headName).isEqualTo(event.headName); + } + + @Test + public void messageWithMetadataShouldRoundtripWithLegacyDeserialize() { + EventMetadata metadata = + EventMetadata.EMPTY.nextRetry("first-instance").nextRetry("second-instance"); + String serialized = serializer.serialize(event, metadata); + + Event message = deserializer.deserialize(serialized); + ProjectCreatedEvent deserialized = (ProjectCreatedEvent) message; + + assertThat(deserialized.projectName).isEqualTo(event.projectName); + assertThat(deserialized.headName).isEqualTo(event.headName); + assertThat(deserialized.instanceId).isEqualTo(event.instanceId); + } + + @Test + public void messageWithEmptyMetadataShouldRoundtrip() { + String serialized = serializer.serialize(event, EventMetadata.EMPTY); + + EventDeserializer.Message message = deserializer.deserializeMessage(serialized); + + assertThat(message.event()).isInstanceOf(event.getClass()); + assertThat(message.metadata()).isEqualTo(EventMetadata.EMPTY); + } + + @Test + public void messageWithMetadataShouldRoundtrip() { + EventMetadata metadata = + EventMetadata.EMPTY.nextRetry("first-instance").nextRetry("second-instance"); + String serialized = serializer.serialize(event, metadata); + + EventDeserializer.Message message = deserializer.deserializeMessage(serialized); + ProjectCreatedEvent deserialized = (ProjectCreatedEvent) message.event(); + + assertThat(message.metadata()).isEqualTo(metadata); + assertThat(deserialized.projectName).isEqualTo(event.projectName); + assertThat(deserialized.headName).isEqualTo(event.headName); + assertThat(deserialized.instanceId).isEqualTo(event.instanceId); + } +}