Architectural Experimentation in Practice: Frequently Asked Questions

In previous articles, including “Software Architecture and the Art of Experimentation” and “If Architectural Experimentation Is So Great, Why Aren’t You Doing It?”, we have discussed why we think architectural experimentation is an essential tool for evaluating architectural decisions. The most important reason is to reduce the cost and probability that bad architectural decisions will result in catastrophic increases in the cost of a system.

For example, while an MVP is an experiment about business value, whether customers find a possible solution valuable, an MVA is an experiment about whether that MVP can be sustainably delivered and supported over time.

We are always experimenting, in a sense, but most of it is unintentional. With intentional experimentation, we actually know that we are running an experiment, and we’ve put some goals and boundaries around it to improve our ability to learn from experience. Accidental experimentation is somewhat random and usually only recognized after things have gone “off the rails”.

In the interest of being more specific but without being prescriptive, this article explores the most frequently asked questions about architectural experimentation.

What decisions need experiments?

In short, those decisions that could be costly to reverse if they turn out to be wrong, especially if the likelihood of being wrong is high. Ranking decisions by some measure of overall impact, such as the probability of being wrong times the cost of being wrong, can be useful in deciding what to tackle first.

The hard part about deciding when an experiment is needed is overcoming cognitive bias and recognizing which decisions create the greatest risk. When challenging a decision, look at its underlying assumptions and ask, “How do we know that’s true?” If it turns out that the decision is mostly supported by opinion, it might be worthwhile to run an experiment just to make sure.

Decisions that people don’t feel comfortable with are usually good candidates for experiments. This especially applies to “expert opinions”. Sometimes, a team will be unfamiliar with a particular challenge, with the exception of one individual, the “expert” on that topic. The “expert” might be right, and it’s great if they are, but the rest of the team needs more than “trust me” to be certain.

Everyone has blind spots, even “experts”, and the experiment will help the team expand its technical knowledge and experience. Transparency is important – a few dominant team members may push their viewpoints and force decisions that other team members feel uncomfortable with. The other team members need a way to voice their concerns. If any team members feel uncomfortable or uncertain about an architectural decision, an experiment will help resolve their uncertainty or discomfort and can be a way for the quieter voices to be heard.

Some decisions are costly to change, but they aren’t necessarily complex, and by complex we mean “intellectually challenging” or “likely to really mess things up if you make the wrong decision”. In other words, rewriting code isn’t complex–rethinking the concepts behind the code is where real complexity arises. Architectural decisions always involve complex trade-offs.

What kinds of decisions usually deserve scrutiny?

The kinds of decisions that are generally worth deeper consideration include:

• Decisions about the fundamental concepts the system uses and its key abstractions as represented in the data structures (e.g. classes, types, …) it uses to share information across the entire system, and even between systems.
• Decisions about the way these data structures are used, i.e., the fundamental algorithms that access and manipulate the data structures.

In addition to algorithms and data structures that represent key concepts, other choices play critical roles in shaping the architecture, including decisions relating to:

• Changes to the messaging paradigm – e.g. synchronous to asynchronous
• Changes to response time commitments – e.g. non-real-time to real-time
• Changes to concurrency/consistency strategies, e.g. optimistic versus pessimistic resource locking
• Changes to transaction control algorithms – e.g. fail/retry strategies
• Changes to data distribution that affect latency
• Changes to cache coherency strategies, especially for federated data
• Changes to security models, especially the granularity of security access when it extends to individual objects or elements.

These types of decisions involve knowledge sharing between components, which Vlad Khononov, author of Balancing Coupling in Software Design: Universal Design Principles for Architecting Modular Software Systems, refers to as “integration strength”. The more knowledge shared between two components, the greater the integration strength and the greater the coupling. Therefore, experiments can help teams optimize component coupling in their systems.

Even when decisions involve the use of technologies that are mature and well-understood, experimentation may still be needed if the technologies will be used in novel ways. For example, organizational architectural standards may require the use of a traditional SQL database. If the team needs to use that product in a way that is not typical, for example, by handling unstructured data, they should run experiments to prove that using the technology in novel ways will meet their architectural goals. If it does not, they will be able to look for other alternatives and, perhaps most importantly, they will have evidence to argue their case for deviating from mandated standards.

Finally, decisions to introduce new technologies should be subject to experiments. For instance, if a team wants to use an event store database as part of an MVA, and that product is not on the list of approved software, an experiment would show whether it’s worth the effort to use the new technology.

When should you not experiment?

When the cost of reversing a decision is low or trivial, experimentation does not reduce cost very much and may actually increase cost. Prior experience with certain kinds of decisions usually guides the choice; if team members have worked on similar systems or technical challenges, they will have an understanding of how easily a decision can be reversed.

In addition, some decisions are expensive to change but not very complex, such as:

• Redesigning the user interface for an application. Even when using a UI framework, changing visual metaphors can be time-consuming and expensive to modify, but it is rarely complex so long as the changes don’t affect the fundamental concepts the system deals with.
• Exchanging one major component or subsystem with another of equivalent functionality. An example of this is switching from one vendor’s SQL database to another vendor’s SQL database. These changes can take work, but conversion tools help, as does staying away from proprietary features. So long as the new component/subsystem supports the same fundamental concepts as the old one, the change doesn’t alter the architecture of the system.
• Changing programming languages may not even be architecturally significant so long as the languages support the same abstractions and programming language concepts. In other words, syntax changes aren’t architecturally significant, but changes to fundamental concepts or metaphors are. For example, moving from an object-oriented language to a functional language warrants an experiment.
• With the appropriate conversion tools, reversing these kinds of decisions might not be very costly. It used to be that rewriting a user interface was expensive but modern UI design tools and frameworks have made this sort of work relatively inexpensive. Deciding what UI framework, SQL database, or programming language to use is an implementation detail, not an architectural decision. Those are all significant decisions but they do not rise to the level of architectural decisions.

What are common misconceptions about experiments?

Experiments are more than just playing around with technology. There is a place for playing with new ideas and technologies in an unstructured, exploratory way, and people often say that they are “experimenting” when they are doing this. When we talk about experimentation, we mean a process that involves forming a hypothesis and then building something that tests this hypothesis, either accepting or rejecting it. We prefer to call the other approach “unstructured exploratory learning”, a category that includes hackathons, “10% Time”, and other professional development opportunities.

For example, playing with Generative AI may be necessary to learn how to use it, but it’s not an experiment until you are testing an LLM to see if it meets some specific need and enables you to support the delivery of an MVP.

What information do you need to get started?

As stated in the previous section, experimentation is a disciplined process that involves forming a hypothesis and then building something that tests this hypothesis, either accepting or rejecting it. To get started, you’ll need the following information:

• A decision you want to test – maybe because someone “does not feel good” about it.
• A precise time frame for the experiment, including when you plan to start and the duration.
• An explicit hypothesis that limits the scope of the experiment.
• Expected results (the hypothesis) you will compare with actual results to know if your experiment succeeded or failed. There is a danger of extrapolating the results from experiments beyond their context.
• Team consensus on the goals and boundaries for the experiment, so that the team knows when the experiment is successful, and when it has wandered off-course.

When is it time to “pull the plug” on an experiment?

If you have an experiment that never seems to end, such as in cases where the team feels that they need “just a little more time” to get results, it’s a sign that the experiment has lost focus. Experiments should have a clear duration and purpose. When you find an experiment that’s not yielding results in the desired timeframe, it’s time to stop it and design something else to test your hypothesis that will yield more conclusive results. The “failed” experiment can still yield useful information, as it may indicate that the hypothesis is difficult to prove or may influence subsequent, more clearly defined experiments.

What do you do with the results of an experiment?

There are three possible outcomes from an experiment:

• The experiment failed, meaning that your decision needs to be re-examined. You will need to go back to reconsider the decision and possible alternatives. In the worst case you may have to scrap the MVA (and possibly the MVP) and start over.
• The experiment succeeded, in which case your decision is confirmed and you can move on to look at other decisions.
• The experiment is inconclusive, in which case you may need to redesign it to determine if your decision is correct. Redesign is important, and you should consider the result as a new experiment and not a continuation of the original experiment.

Conclusion

Experiments help teams to test architectural decisions. They can reduce the cost of a system by reducing the waste associated with having to reverse a decision that turns out to be incorrect. As a result, not all architectural decisions need experiments, just the ones that will be very costly to reverse; other changes may not be worth the cost of experimentation.

Experiments are more than just playing around with technology. Team members often use the word “experiment” to characterize the important work of learning a technology, using exploration. Experimentation is different and is focused specifically on testing decisions.

Experimental results should tell you whether a decision is acceptable or needs to be reconsidered. If they don’t, the experiment is not specific enough and needs to be redesigned.