As a company grows, investing in custom developer tools may become necessary. Initially, standard tools suffice, but as the company scales in engineers, maturity, and complexity, industry tools may no longer meet needs, Carlos Arguelles said at QCon San Francisco. Inflection points, such as a crisis, hyper-growth, or reaching a new market, often trigger these investments, providing opportunities for improving productivity and operational excellence.
Carlos Arguelles gave a talk about Amazon’s inflection points in engineering productivity. When a company first starts, it doesn’t make sense for it to create its own developer tools, as there are plenty of excellent ones available in the industry, he said. But as a company grows (in number of engineers, in maturity, in customer adoption, in domains), investing in its own developer tools starts making sense:
An inflection point is when that investment in engineering productivity that didn’t make sense before now suddenly does. This could be because the industry tools do not scale, or because the internal tools can be optimized to integrate better with the rest of the ecosystem, as Arguelles explained.
A little papercut where each developer is wasting a couple of minutes per day in toil can add to hundreds of millions of dollars of lost productivity in a company like Amazon or Google.
The more obvious inflection point that made investments in engineering productivity become feasible is the number of engineers. Maybe it didn’t make sense for your company to have its own CI/CD proprietary tooling when there were 3000 engineers, but it does when there are 10,000 engineers, because the savings in developer productivity with a toolchain optimized for the ecosystem have a significant return on investment, Arguelles said.
He mentioned that the opposite is true as well. When your company is in hypergrowth it may make sense to have duplicate tools (each organization creating its bespoke tool so that it can independently move fast), but when the company stops growing (which is what happened in 2023 with all the big tech layoffs), it makes sense to consolidate tooling and defragment the world.
Arguelles gave some more examples of inflection points, like reaching a certain level of maturity where you need to raise the bar in terms of engineering or operational excellence can provide an inflection point or entering an entirely different and new market. Sometimes the inflection point is a crisis or even a single operational issue that could have been prevented with the right tooling:
For example, Amazon invested significantly in a number of load, stress, and chaos testing tools after the Prime Day incident of 2018 (where the Amazon Store was unavailable for hours during the busiest shopping day of the year). We had been talking about doing that for years, but that incident helped us sharpen our focus and build a solid case for funding those investments.
Inflections can also happen when an organization experiences hyper-growth:
I saw Amazon double in size every year, from 3000 engineers when I started in 2009, to 60k-70k in 2022. What this meant in practice is that we needed to be thinking about skating to where the puck was going to be, not where it currently was.
Scaling needs and security needs often meant sooner or later we needed to create our own developer tools, Arguelles said. Over time, they developed tools to scale source code repositories and built their own tools for code reviews and CI/CD (including testing and deployment):
Because of that hyperscale, we often found ourselves needing to re-think our architecture much sooner than we had envisioned. But it also provided ample opportunities to innovate and think differently!
Inflection points are inevitable and occur naturally in many situations: a company drastically increasing or shrinking in terms of number of engineers, a crisis, reaching a certain level of maturity where you need to raise the bar in terms of engineering or operational excellence, or entering an entirely different and new market, Arguelles said. He concluded that it is important to have your eyes open, recognize when these inflection points are around the corner, proactively shape your engineering productivity tooling for the future, and seize the opportunities.
