Po Linn Chia presented how they re-used a single development environment to deploy multiple service versions for testing their distributed system in her presentation No QA Environment? No Problem at Dev Summit Boston. A small enablement team, cultural buy-in, and gradual learning helped teams collaborate, reduce cognitive load, and scale testing practices.
Without a dedicated QA environment, teams faced tech and coordination issues when testing a distributed system, Chia explained in the InfoQ news How to Enable Testing a Distributed System on a Single Environment Using Proxy Routing:
A slow, unmaintainable CLI led an organization to shift left with automated testing. They built a tool for versioned deployments using CI and proxy routing, enabling developers to run isolated tests on multiple versions to catch bugs earlier.
Having implemented their internal deployment tool, they realized that their team structure was no longer viable for what they were trying to do. The company has grown, but couldn’t scale due to not having enabling teams that would cross development and operations, Chia explained:
For every one team trying to test one thing, we blocked N minus one teams, and as you scale, that does not scale, and just gets bigger.
Trying to fix problems, they assumed that if they could test the back-end well, that would reduce trickle-down effects to the frontend and mobile testing. They set up a small team, but the issues were too big to fix. They also learned that their test infrastructure would need to change.
Next, they tried testing in production. A problem was that their production systems weren’t ready to accept test data. They did some monitoring, but monitoring can’t replace testing; monitors just tell you things are already on fire, Chia said.
They came up with the idea to set up an enablement team, working like a tiger team. They’re very small, but important and they will clear the way for them to do the work that they need to do, Chia mentioned. The hard part is figuring out the unspoken criteria for your organization, she explained:
- Can’t be too expensive
- Can’t be too complicated
- Must bridge platform and product
- Can’t be too disruptive: must enable different types of existing testing while enabling integration tests
They are not disrupting the current developer experience, but they’re providing them with this shadow realm that will be stable, and the ability to ephemerally spin up things in their PRs and on demand to do whatever testing they want, Chia explained:
We empower mobile and front-end engineers to target versions themselves using the tools they are familiar with.
As with the adoption of any new technology, it was hard to educate people, Chia mentioned. There is a lot of learning, and even the most plugged-in engineer has to deal with endless cognitive load. People are overwhelmed by how much they have to know; until this is embedded in our engineering culture, we have to ask our developers to take on knowledge acquisition.
You can have a small team, but with strategy and political will, you can do great things, Linn Chai said. Try and enable as many types of testing as possible, even if it doesn’t fit the perfect practical testing pyramid with all the nice linear boxes; you’ll get there, but you have to start small, she concluded.
InfoQ interviewed Po Linn Chia about changes in their organization and ways of working.
InfoQ: How do you cycle engineers between development and operations?
Po Linn Chia: In the Classpass organisation within Playlist, we have guilds for different umbrellas of engineering – in our case, backend, frontend, and mobile development – as well as a separate platform operations team. The guilds have permanent staff – for example, I’m the lead for our backend guild – and the guild sits separately from product workstreams. We focus on technical excellence, developer productivity, and collaboration with our platform operations teams.
Engineers can rotate onto the guild temporarily. This allows them to gain experience working on cross-team, cross-platform projects, and often is their first exposure to operations-type work. The backend guild in particular maintains a close working relationship with our platform team, and there’s often a lot of overlap in our work.
InfoQ: What have your experiences been with rotating engineers?
Chia: By and large, these rotations have been well-received! I think engineers enjoy the chance to work on high-leverage technical projects, and it’s a great opportunity to spread working systems/ops knowledge around more broadly, so everyone – including platform operations – wins.
