The Gerrit User Summit 2019 can definitely be defined as truly innovative in its format and audience.
For the first time in the Gerrit history, the Summit is split into two parts. Volvo Cars have hosted the first in Gothenburg (Sweden) while the second will take place from the 11th to the 17th of November at GerritForge Inc. HQ in Sunnyvale, CA (USA).
The Summit has been repeated on both sides of the Atlantic: the European and US communities come from different background and have different needs. The Gerrit Code Review Community is global and is willing to share experiences and receive feedback from both sides.
We are also innovating on the Hackathon perspective, with three new elements:
The Hackathon is now open to everyone, including the people that have never contributed to Gerrit before. Experienced maintainers have paired with newbies to guide through the very first contributions.
The Hackathon at Volvo Cars has been 100% focused in triaging the massive backlog of open issues and fixing as many bugs as possible for the latest three supported branches: stable-3.0, stable-2.16 and stable-2.15.
The OpenStack and Gerrit communities finally have met and started talking and interacting more closely.
During the Hackathon, David Pursehouse and Luca Milanesio have been working on the release of Gerrit v2.15.16, v2.16.11.1 and v3.0.2, thanks to the precious help from Gert van Dijk for writing and reviewing the release notes. In addition to the bug-fixing activity, James Blair and Monty Taylor (both RedHat, working on the OpenStack project) have presented the project Reno.
The OpenStack project uses it for managing the collection and generation of the release notes automatically, which can be useful for the Gerrit Code Review project. Other interesting discussions have been around the migration process from Gerrit v2.13 or earlier to v3.0.
The talks have been mainly centred on the automotive use-case, characterized by:
All the talks have been recorded and will be soon published on the GerritForgeTV YouTube channel, together with the past two years conferences videos.
Edwin Kempin (Google) is one of the longest-serving maintainers of the Gerrit Code Review project. He is one of the Community Managers working hard to strengthen the collaboration and to improve the way the whole project works.
He has organized an “agile-style” retrospective on what is working well and what can be improved in the whole project, giving complete freedom to anyone to participate either on-site and on-line, using a digital ideas board.
The full report has been published on the Repo-Discuss Mailing List.
Nicholas Mucci (Volvo Cars) is both the main organizer of the Summit in Gothenburg and also a key player in the adoption of Gerrit Code Review for the development of some of the critical components in modern Volvo Cars. He stressed the importance of adopting Open Source for Volvo Cars. Every software component of a car needs to be able to receive updates and security fixes, which means that Volvo needs to have control over the CI/CD pipeline for improving and building code constantly. Cars need to be safe and supported for over 15 years, independently from vendor lock-in, software licenses or black-box closed source components. The CI/CD pipeline of Volvo Cars also includes some key Open Source components, like Zuul, Jenkins and, of course, Gerrit Code Review.
Alice Kober-Sotzek (Google) has presented the current state-of-art of the checks plugin, developed by Google in 2018/2019. The plugin allows other CI/CD tools, such as Jenkins, to publish build progress and results directly into Gerrit Code Review.
There are multiple benefits in having a dedicated API in Gerrit: Gerrit users can get direct feedback on a Change about its build validation status, without having to jump into another tool. Various types of checks (e.g. code-style, unit-test, integration-test, load-tests ...) can be defined and displayed more concisely, without having to configure multiple labels. Build actions can be triggered directly from the Gerrit change screen, simplifying the daily lifecycle of patch-sets validations. There was high interest for this topic, specifically from the OpenStack Community. An existing plugin (verify-status) already covers a similar set of functionality. However, it is mostly incompatible with the recent versions of Gerrit; it doesn't have a PolyGerrit UI and relies on an external relational database. The maintainers of the Zuul project have explicitly shown interest in integrating soon with the checks API and also collaborating on its development.
Luca Milanesio and Marcin Czech (GerritForge) have presented the brand-new multi-site plugin for Gerrit Code Review. Finally, after many years of waiting, the rest of the OpenSource Community can deploy multiple instances of Gerrit Masters across the globe. Previously, only Google was running Gerrit in a multi-master/multi-site fashion. Also, WANdisco announced years ago a commercial multi-site solution, based on a custom fork of Gerrit v2.13 and its proprietary replication solution.
The multi-site plugin is available for any version of Gerrit from v2.16 onwards and requires NoteDb. It has been adopted since May on GerritHub.io. The reliability of the site has jumped to 100% since then, certified by both uptime.gerrithub.io and pingdom.com.
Edwin Kempin (Google) has showcased how to implement Gerrit submit rules without the use of Prolog but using a simple Gerrit plugin. The Prolog language has been a difficult part of the Gerrit submit rules: only one (Luca) in all audience declared to like it as a programming language. However, nobody (even Luca) feels confident in reading and writing it easily. The introduction of the Prolog cookbook has improved the situation in the past. However, people still misuse the tool by simply “copy&paste” random parts of the cookbook and at times generating overload on Gerrit and a massive headache for the Gerrit admins.
Starting from Gerrit v2.16 the submit rules can be implemented via plugins. Edwin showcased a sample plugin (simple-submit-rule) for implementing some useful rules, such as blocking a change if some of the comments are not addressed, and a lot more.
With regards to the future of Gerrit and Prolog, most likely it would be moved from Gerrit core to a plugin for those who still require it in production.
Fabio Ponciroli (GerritForge), aka Ponch, showed the work on implementing a consistent end-to-end test scenario for Gerrit by leveraging the Gatling tool. Testing Gerrit involves the invocation of REST-API by simulating the PolyGerrit UI and also the use of Git/HTTP and Git/SSH protocol. Gatling, however, does not support the Git protocol out-of-the-box. Ponch has introduced the gatling-git project, that extends Gatling to include the Git protocol.
The definition of end-to-end tests is further simplified by using the Gatling “feeders”. Those are sample data in JSON format, which can also be generated from existing Gerrit production logs.
Ponch has then showcased, to Luca's surprise, a real use-case of running load tests against GerritHub.io, and they generated the expected spike of incoming traffic.
Monty Taylor (RedHat) introduced Zuul, an OpenSource project created for running the validation of OpenStack changes on Gerrit Code Review.
The OpenStack project started their CI/CD pipeline adopting Jenkins CI and afterwards implementing some higher-level projects like the Jenkins Job Builder for templating and auto-generating the build jobs.
However, Jenkins wasn't powerful enough to “juggle” the validation of multiple changes together across components, which is a key requirement of the CI validation for OpenStack.
None of the OpenSource alternatives to Jenkins was sufficient to satisfy the OpenStack requirements, and thus they decided back in 2012 to create a brand-new CI system, named Zuul.
Zuul's job definition does not differ much from the Jenkins-Job-Builder YAML but relies on Nodepool and Ansible for job scheduling and execution.
Monty and James have already started to implement Zuul for upstream Gerrit, and we hope to show the progress at the next forthcoming Gerrit User Summit 2019 USA in November in Sunnyvale CA.
Thomas Gerbet (Enelean) is part of the Tuleap Team in Grenoble (France). Gerrit Code Review has been part of their daily development pipeline since the beginning and they shared their experience on regular reviews.
Gerrit has been the central focal point of the collaboration and helped to improve the communication and exchange of ideas. In addition to providing a tool for reviews, it has encouraged people to focus more on communicating effectively, even face-to-face and with mutual respect and constructive behaviours.
The Tuleap Team had adopted Gerrit from the very first versions when the distinctive Android-green theme still characterized it. So far, they have been able to follow all the upgrade cycles, and they are happy to be running on the latest v3.0 with NoteDb.
James E. Blair (RedHat) impressed the audience with its revolutionary TTY-based UI for Gerrit Code Review. It is quite remarkable how James managed to implement the full workflow by simply using the Gerrit REST-API and building a fully functional experience by only using your keyboard.
During the daily work of software development, the keyboard is the central part of where the activity takes place. Changing the context and opening a web-browser with a different experience, can be time-consuming and cause the break of the creative flow. Gertty comes to the rescue and gives to those who feel more productive with a keyboard the ability to do the whole lifecycle in the fast and most efficient way possible.
One extra goodie, which is very important for everyone that is travelling for work, is the ability to fully work off-line and sync back once the connectivity is back. This was also one of the main goals of the Gerrit project founder, Shawn Pearce: using the power of Git for allowing a full peer-to-peer cooperation between the members of the community, including reviews.
Gertty, however, isn't quite there yet, because still relies on a central Gerrit server and the meta-data is stored in a SQLite database on the local computer. However, James was open to the idea to remove the local database and place it by the NoteDb format from Gertty, which would be great as it would implement exactly the way that Shawn initially designed the Gerrit product.
James also made, during the Hackathon, important fixes to Gertty to make it fully compatible with the latest version v3.0.x of Gerrit and with gerrit-review.googlesource.com.
Patrick Hiesel (Google) presented remotely from Munich the work that is currently underway on the master branch of the Gerrit Code Review project.
Google has spent the last year and a half on the stabilization and improvement of the Gerrit user-experience, using a scientific approach:
The targets for *-review.googlesource.com were, for the change display screen, set to 2s and 5s, for the 90th and 95th percentile.
A lot of work has been made on Gerrit for improving the cache effectiveness, switching from Java serialization to protobuf, which preserves compatibility across Gerrit versions and upgrades.
With regards to the access to indexes, dashboards have been moved to make parallel queries so that multiple parts of the page can be loaded concurrently. With regards to the UI rendering, the components are now lazy-loaded. The number of round-trips from the browser to the backend has also been reduced to a minimum.
A lot more work is ongoing for the forthcoming v3.1 of Gerrit, including:
Last but not least, Patrick gave useful recommendations on how to improve performance on your current version of Gerrit, making the right choices for caches, CPUs and heap utilization.
More progress on Gerrit v3.1 and the performance work will be presented at the Gerrit User Summit in Sunnyvale, in November.
The Summit has concluded with yet another exciting talk by Michael Watkins (Softagram).
Just nine months ago, Shane McIntosh (McGill University) presented the BLIMP Tracer research work on the automated structural analysis of code review changes. Now Softagram, a small startup based in Finland, presented an original commercial version of the same concept, production-ready and fully integrated with Gerrit Code Review.
Michael went through the three central values provided by the automated structural change analysis:
Code Review has become a “huge activity”, with over 1 million code reviews created world-wide every day. Moving a lot of repetitive activity from humans to robots is the key to make reviews more accurate and enjoyable, leaving a lot of repetitive checks to robots.
Michael also implemented a fully working plugin for integrating the output produced by Softagram into the Gerrit change screen. That was a fascinating experiment for the following reasons:
The integration will grow soon and, as Gerrit Community, we are fully committed to helping Michael and all the other new contributors to get started with Gerrit development and start contributing for making it better and easier to use.
We are all looking forward to seeing Softagram integration used for the Gerrit Code Review project, as their solution is free for OpenSource projects.
Nicholas Mucci (Volvo Cars) has concluded the event by giving the opportunity to all participants to the Summit to share their feedback and give new ideas on how to make the event better next time.
We should publish the action items from the Hackathon and Summit and act on them.
Edwin Kempin gave immediate progress on this, by publishing all the action items from the retrospective to the Gerrit Code Review Issue Tracker.
We should better advertise the Hackathon and Summit. Outside of the immediate Gerrit community, it could be hard for people to learn about these events.
Luca Milanesio advertising the Summit to the JGit community and Han-Wen (Google) invited members of the Google Development Team working on Git/JGit to come and present their latest innovations.
In expanded advertising, we should include a call for talks.
The Talks are already open to everyone and go through the Gerrit Code Review process. The Summit website is simply a Gitiles view of the summit/2019 repository. However, the process would not have been necessarily clear to everyone, and we will commit to making it more accessible and visible for submissions.
European summits should be in cheaper countries than Scandinavia; it can be hard to get budget for Sweden. The challenge is then finding a host and site.
The problem with Gerrit Code Review adoption in Europe is that it is mainly focussed on the northern countries, with Sweden and Finland in the first places because of the historical use from companies like Nokia and Ericsson.
It was very nice to see from Enalean that Gerrit is also used in France, which could potentially be a cheaper destination for the next Summit in 2020, assuming that a company is willing to host the event.
We should consider piggy-backing on other related conferences to minimize travel costs (e.g. DevOps World, I/O).
DevOps World was in mid-August in San Francisco. However, the associated costs for the location and the time of the year could have been exactly in conflict with the previous proposal of making the travelling less expensive. However, other European events about OpenSource, like FOSDEM 2020 are in a more affordable location and could be an excellent opportunity to get more communities together.
Thank you again to all the attendees of the Gerrit User Summit 2019 in Volvo - Sweden and see you in November at Sunnyvale, for the release of Gerrit Code Review v3.1.
Luca Milanesio (Gerrit Maintainer, Release Manager, ESC member)