How to Go from Copy and Paste Deployments to Full GitOps

InnerSource helped reduce the amount of development work involved when introducing GitOps by sharing company-specific logic, Jemma Hussein Allen said at QCon London. In her talk, she showed how they went from copy and paste deployments to full GitOps. She mentioned that a psychologically safe environment is really important for open and honest discussions that can help resolve pain points and drive innovation.

Their version control tool at the time was Subversion, which was moved to a more popular distributed version control system – Github, the Git based developer platform.

When they started the implementation of GitOps, their servers were running a LAMP stack (Linux, Apache, MySQL and PHP). This was a standard software stack for PHP web applications, Hussein Allen mentioned, which they didn’t want to change as the application was running well and the focus was on deployment automation.

The CI/CD tool of choice was Jenkins because of the flexibility in pipeline building block configuration and the large number of plugin integrations with other tools that were available, Hussein Allen said. Puppet was used for configuration management as it was already implemented for the more recent deployments and worked well, so the decision was made to continue with it, she mentioned.

Hussein Allen mentioned the four GitOps principles:

• Declarative – the desired system state needs to be defined declaratively
• Versioned and Immutable – the desired system state needs to be stored in a way that is immutable and versioned
• Automatic pulls – the desired system state is automatically pulled from source without manual interventions
• Continuous reconciliation – the system state is continuously monitored and reconciled

The principles they found the most important were a declarative system state definition, solid versioning and continuous reconciliation, because of the benefits they brought in terms of faster development and deployments, Hussein Allen said.

The removal of all manual interventions is one that they used more cautiously, Hussein Allen said, as it required a fully comprehensive set of automated tests, high availability, a very mature monitoring solution, and compatibility with team ways of working.

Developers customised their workspaces with Docker, as Hussein Allen explained:

We set up an image registry where developers could download base images and use these as building blocks to develop and test new features locally before integrating them into the multi-environment test and deployment workflow.

Before the changes, developers would change code locally and deploy to the development environment to test, Hussein Allen said. The challenge with this testing method came when multiple developers needed to test different changes in the development environment at the same time, she explained:

When the development environment included multiple changes, it didn’t allow for testing in isolation and didn’t provide reliable results. The image registry meant that developers could test their own changes locally, against the code running in production, in isolation before integrating the change into the main development testing pool.

A common theme that emerged from developer feedback on GitOps was the need for building blocks and a “quick start” guide to help adopt more quickly. Introducing an InnerSource capability encouraged developers to create and contribute to these building blocks and boilerplates, Hussein Allen said. The increased contribution to shared resources positively and directly impacted the speed that developers could adopt new tooling.

As developer requirements evolve, creating an open dialogue within a psychologically safe environment is invaluable for understanding the evolving needs of developers, Hussein Allen said; establishing regular connections and solid offline communication channels, ensuring developers stay up-to-date with the platform roadmap and any alterations that could impact their work. These channels also provide valuable opportunities for developer feedback and suggestions, which can be integrated into the platform strategy, she concluded.

InfoQ interviewed Jemma Hussein Allen about adopting GitOps.

InfoQ: What did you do to help developers with the transition from the old way of working?

Jemma Hussein Allen: We spent time training or pair programming with developers. Some teams took longer to transition to the newer way of working, mainly due to heavy workloads, as it took time for developers to familiarise themselves with the new process and become as efficient using it as they were with the old way of working.

Working with team product owners and stakeholders to show the benefits of the new process helped to give developers the bandwidth to learn and adopt the new tooling into their daily work.

InfoQ: What’s your approach to knowing the needs of the developers and keeping the dialogue going?

Hussein Allen: What we found helpful in understanding developer needs was providing the opportunity for platform engineers and developers to work more closely together. In organisations with a centralised platform team structure, initiatives such as “Walk a day in my shoes” where platform engineers are embedded into product teams for a short time and vice versa can be really valuable to get an understanding of any pain points or improvements that can be made to the platform.