Software architecture is tough because it blends coding, math, and business systems. Due to surprises, architectures tend to become irrelevant over time, Barry O’Reilly said at Goto Copenhagen. He presented residuality theory, where he suggested stressing naive architectures to reveal hidden “attractors” in complex business systems. This allows designs to better survive change and uncertainty.
Software architecture is difficult because it requires a broad skillset, O’Reilly said. We have to master the world of code, math, and logic, and the world of human and business systems, and understand how these two relate to and affect each other.
At university, they only teach us the tech. When we start to design systems, we realise that the complexity of the business environment constantly surprises us, rendering our architectures irrelevant, O’Reilly said. Making something as static and rigid as a software architecture survive in a fluid world that is always changing is not an easy task, he added.
Residuality assumes that a random simulation of stress on a naive architecture will produce a better architecture than the traditional methods around requirements engineering, risk management, or reacting to change as it happens, O’Reilly explained:
This started out as a curious observation, and for the last 10 years, I have had to build theoretical explanations and experiments to show that this actually is the case. Armed with this knowledge, we can think differently about software architecture and build new tools.
As students of Western science, our first recourse is to reduce any complex system to its component parts and study those parts in detail. This is the default for software engineers, O’Reilly said. In complex systems, the number of elements and potential interactions and states makes this kind of detail-oriented analysis impossible. Previous generations of architects have tried to reduce the complexity of the business environment to logic, or to structures, or to the development process.
One of the key aspects of complex systems is that they never visit all the possible states that the combination of their elements allows. Instead, the interactions of the elements constrain the system to a very small number of potential states, which we call “attractors”, O’Reilly said. A complex business system is therefore not modelled as a number of interaction elements and their relationships, but instead as a number of attractors, he explained:
When we build an architecture, it is these attractors that the architecture must survive in. Attractors therefore provide a much simpler and easier and more pragmatic way of interfacing with the complexity of the environment.
The problem is that we don’t know what the attractors are, but by randomly simulating stress, we can discover many of them, O’Reilly said. If you think back to the major architectural failures you’ve seen, you will see that they mostly fail because they missed attractors, he mentioned.
Residuality theory is a very simple process. Sometimes, people are put off because the theoretical work necessary to prove that residuality works is very heavy, but applying it is easy, O’Reilly explained:
We start out with a suggestion, a naive architecture that solves the functional problem. From there we stress the architecture with potential changes in the environment. These stressors allow us to uncover the attractors, often through conversations with domain experts. For each attractor, we identify the residue, what’s left of our architecture in this attractor, and then we change the naive architecture to make it survive better.
We do this many times and, at the end, integrate all of these augmented residues into a coherent architecture. We can then test this to show that it survives unknown forms of stress better than our naive architecture.
In complex business environments with uncertainty, residuality makes it possible to create architectures quickly instead of chasing down stakeholders demanding specific requirements or answers to questions that are unknown by the business itself, O’Reilly said. It pulls technical architects out of details and teaches them to productively engage with a business environment without the lines and boxes of traditional enterprise architecture, he concluded.
InfoQ interviewed Barry O’Reilly about residuality.
InfoQ: How can we prove that the residual architecture that we created is an improvement over the naive architecture?
Barry O’Reilly: A simple test is to use a second set of stressors to check that our residual architecture survives more unknown events than our naive architecture. You can easily see the similarities between this and the training/testing sets of ML. Residuality theory ultimately states that architectures should be trained, not designed.
InfoQ: What benefits have you seen from residual analysis?
O’Reilly: Senior architects report that it gives a theoretical justification for practices that many had already figured out and a shared vocabulary for teams to talk about architecture. Ultimately, it makes architecture more explicit, better defined, and easier to teach. The result is architectures we can believe in and decision making that is traceable.
It has its challenges as well. A small number of developers find the jump from the linear, logical, mathematical world we are trained for to the lateral, imaginative techniques very difficult. Residuality is a subject as big as OOP and requires the same amount of effort to learn.
