Elisabeth Hendrickson on Systems Thinking for Quality Engineering

Transcript

Shane Hastie: This is Shane Hastie for the InfoQ Engineering Culture Podcast. Today, I had the pleasure of sitting down with somebody who I have known for, wow, it’s a couple of decades. Can we admit that? Elisabeth Hendrickson. Elisabeth, welcome. Thanks for taking the time to talk to us today.

Introductions [00:55]

Elisabeth Hendrickson: Oh, thank you so much. I am so delighted to be here. And yes, I think it was a couple of decades ago at one of the conferences in Australia, I think, was where we first met.

Shane Hastie: One of the early Software Testing Australia and New Zealand conferences, STANZ. So obviously, you and I know each other pretty well, but there’s a fair number of our audience who might not have come across you. So what’s the 30-second overview? Who’s Elisabeth?

Elisabeth Hendrickson: Yes, there’s no reason they should know who I am, but if they do, it might be as test obsessed. I am test obsessed in a variety of places. My company is Curious Duck Digital Laboratory, and these days, I am a consultant and folks call me when their organization is having a hard time delivering software.

Shane Hastie: Having a hard time delivering software.

Elisabeth Hendrickson: Yes.

Shane Hastie: Why do organizations have hard times delivering software?

Why Organizations Struggle with Software Delivery [01:48]

Elisabeth Hendrickson: Oh my goodness, all the reasons and I’m sure you and everybody watching this has their own whole list of reasons. There can be late breaking surprises. So, we thought we were done, but then we got to final test and discovered, “No, no, we’re not so done”. There can be huge interruptions. So, we got all the way through to production and then stuff started breaking. And so now the development team that is supposed to be working on the next thing is struggling under the weight of all of the surprises that were found in production and angry customers and incidents and pager duty going off, etcetera.

There can also be problems all the way at the frontend where the teams are suffering from analysis-paralysis and because there can be so many reasons why the organization is having trouble delivering software and they often masquerade as a different reason. So, I’ll hear, “Oh, we’ve got huge problems with requirements analysis and our product managers can’t make up their minds”. And what I’ll discover is that the product managers are actually really clear on what they wanted to begin with, but there are technical limitations and there needs to be a better partnership. Many reasons why an organization might have trouble delivering software.

Shane Hastie: Now, I know that you are a systems thinker and-

Elisabeth Hendrickson: True fact.

Shane Hastie: … system thinking is something that you use to explore this stuff, but what is systems thinking?

What is Systems Thinking? [03:08]

Elisabeth Hendrickson: So, systems thinking is, at the most basic level, the recognition that the behavior of the system as a whole isn’t about the individual parts, but the relationship between things. For example, when you have all of those late-breaking surprises happening at the end of the cycle, there isn’t a simple linear reason why a developer made simple mistakes. That’s extremely unlikely. It’s much more likely that there was a cumulative set of effects that happened because of decisions that got made at various points that all seemed like a really good idea at the time that led to late-breaking surprises. And so being a systems thinker means that I have a whole bunch of tools in my back pocket for looking at the holistic system as opposed to getting stuck on just one piece of it and having that myopic view.

Shane Hastie: So what are some of those tools? If I’m a team lead, a manager in an organization struggling with some of these issues that you’ve mentioned or others, where do I start?

Tools for Systems Thinking [04:15]

Elisabeth Hendrickson: My personal take is usually I’m going to start in one of two places. Either I’ve got data already or I think I do and I’m going to start with a causal map to understand the relationships between things. If I don’t feel like I actually have data, I will probably start with getting the data because my gut might be telling me something and the problem is that guts are sometimes remarkably accurate and sometimes they lie. And so for example, if I am the team lead and I’m very frustrated because QA keeps finding bugs and I’m feeling like, “We must be producing crap. How do we have so many bugs downstream?” I’m first going to check my assumptions, “Maybe I’m only looking at bug counts and I’m not actually understanding what the content of those bugs are”.

Once I have some data though, I can start using a causal diagram to say, “Okay, I had a lot of bugs over here. What is my current working hypothesis about what contributed to those bugs?” and just start mapping out the series of forces within the organization that would lead to where we are. Incidentally, one of the things that I had done not too long before you and I met was I wrote a paper called Better Testing, Worse Quality. That was exactly an example of using a causal diagram to figure out, “Hey, we have a lot of bugs, but we’re testing better than we ever tested before. So it makes sense that there would be a lot of bugs because we’re testing better, but wait, no, we’re getting worse results out the other end of this process”.

And it turned out that in that organization, and in many organizations, when you strengthen the independent testing muscle without strengthening the rest of the organization, you create the conditions where you’ve got a more target-rich environment. So while I was busy patting ourselves collectively on the back for having done an amazing job of testing, it turned out that we were contributing to making the outcomes worse. And that was a really bad day at work. When I realized the better I got at my job, the worse it was for the company that employed me. That was sad.

Shane Hastie: How do I trace it backwards?

Elisabeth Hendrickson: Oh, how do you trace it backwards?

Shane Hastie: Don’t we do five whys, root cause or any of those?

Better Testing, Worse Quality [06:33]

Elisabeth Hendrickson: Those are all great examples of tools that relate to systems thinking. And so five whys is a great tool, but the thing that’s different there is then starting to use five whys to start to map out the other contributing causes. So at the simplest level, we’ll take my example from Better Testing, Worse Quality, at the simplest level, we put more emphasis on independent testing, “Well, why? Why did that happen?” “Well, because there was a perception … I got hired because there was a perception that there was a quality problem”. “Okay, what was that perception from?” “Okay, customer complaints”.

And so you can start to see how things start to map out and then now, I can go forward, “Okay, we put more emphasis. What happened?” “Well, we found more bugs”. “Oh, okay, great. So we found more bugs. So we’re shipping fewer bugs, right?” “No, the data doesn’t support that”. “Oh, well now I know that there’s something missing from the model because my intuition says, if we find more bugs, we ship fewer bugs, right? Okay, there must be more bugs to be found. Well, how did that happen?”

And so you can start to see how at each stage you can use tools like five whys could use root cause analysis tools like Ishikawa diagrams, but they’re going to start to inform, “Oh, there’s a force there that we weren’t taking into account”. And so what looked like a really good, simple, straightforward decision at the time had unintended consequences.

Shane Hastie: So if we track back your example of the, “We found more bugs and things got worse”, when you eventually dug through, what were the linkages?

Elisabeth Hendrickson: So the hidden linkage that I did not see until after I had gotten hauled into my boss’ office, so my boss, VP of engineering, hauled me into his office and said, “Hey, we hired you. We spent a bunch of money. We hired a bunch of testers that you said we needed. We outfitted an entire test lab and yet we have a worse outcome”, that was the moment that I realized that I was missing a linkage, and when I dug through, the linkage that was missing is, before I showed up, the programmers were terrified because there was no really good testing happening. And after I showed up and we made all of this investment in the testing, I realized that I could actually remember overhearing hallway conversations that went like, “Why are you still testing that thing? You’re behind on the schedule. Get that into QA. That’s what they’re here for”.

And so the testing at the level where, I’m not going to say it was unit testing because this was the 1990s, unit testing was a thing, but not at this company, but the developers had been doing a lot of desk checks locally. They would run the stuff locally and make sure that before they checked in that the code actually did what they intended it to do and they stopped doing that because they had so much schedule pressure, which I did not include that piece in the paper that I wrote back in 2001, but I will say that it’s ironic that the more pressure you put on an organization to go faster, you would think that that’s like a gas pedal and it’ll make the organization go faster.

The unintended consequence is actually it can make the organization go way slower because you end up with a whole lot more rework and a whole lot of chaos and a whole lot of arguments and bugs flooding into later stages. And then you’re having the bug triage meetings just arguing, “Was that important enough to fix before the release all because of pressure?”

Shane Hastie: So, what’s the state of quality engineering today?

The Current State of Quality Engineering [10:12]

Elisabeth Hendrickson: I fundamentally believe that quality engineering is a key and critical strategic advantage for any organization, but I don’t define it as the work that some group with quality in their name does. The head of quality engineering in any organization is really the head that is responsible for the product delivery. So that could be a VP of engineering, it might be a CTO, it might be a head of product delivery, whatever that has product managers and engineers. But wherever the delivery responsibility lives, that is the head of quality engineering, even if they have a head of quality, VP quality, whatever title. So, if you take that model, that quality engineering is the set of activities that we do in order to deliver good stuff.

And I’ll frame that as the result of quality engineering is delivering value sustainably. Everything that we do in service of deliver value sustainably, that is an aspect of quality engineering. So that includes all of the things that we do to make sure that we have actually figured out what to build, all of the work that we do within development to do test-driven development, disciplined engineering practices, continuous integration, continuous delivery. And all of the things that we’re doing in operations to ensure that we’ve got the observability so that we can tell, when logins go down by half, something has gone wrong. All of these are aspects of quality engineering. So, if I define it that way, the state of quality engineering is more awesome today than it has ever been before in the history of our field.

Shane Hastie: And what’s at the leading edge of quality engineering today? What are the challenges and what are the opportunities?

Challenges in Quality Engineering [12:05]

Elisabeth Hendrickson: I think one of the biggest challenges is in … We, for the last 20 years, a little more than 20, have been doing agile transformations. We’ve been doing DevOps transformations, massive process transformations. We’ve been, as an industry, focusing on shift left, etcetera. All of these sound like great things, but the challenges in the organizations that didn’t actually make the transition. They seem like they did because they have all of the right words on all of the right things, but when you actually dig in under the covers, they still are getting quality via inspection. And you can tell that this is happening because there is some set of people or things responsible for actually checking and its quality gate and stuff backs up in front of the quality gate.

Just to make this a simpler example, let’s say you do still have an independent QA function and that independent QA function is always running behind, always. There’s so much pressure on them because that’s where the bottleneck in the system is. That is an organization that didn’t actually fully transition no matter what the PowerPoint slides say. And they’re in a world of hurt right now because things just keep running faster and faster.

Shane Hastie: So how do we address that? We do another transformation?

Addressing Problems Without Big Transformations [13:28]

Elisabeth Hendrickson: Oh, I’m so tired of transformations and I’m pretty sure everyone else is too. And there are those who will say, “Rub some AI on it and it’ll be fine”. So rather than suggesting transformation and big initiative, I’m going to suggest what we do about it is take it one symptom at a time. It is entirely possible that the thing that your intuition tells you is the wrong thing is actually the thing that’s going to fix the problem. So for example, let’s say work keeps piling up in QA. Well, your intuition might say, “Hey, we obviously have a bottleneck in QA and the answer is we need to hire more QA people. So we’re going to expand the flow of work through the system. We’re going to expand the capacity there, because obviously, that’s our bottleneck, so we need to fix that bottleneck”.

“Well, you could do that, and next month or six months from now, they’re going to be the bottleneck again”, because it turns out that we, as an industry, at this point are capable of producing massive amounts of stuff. And then if the only time we can tell that the stuff was what we actually wanted is that this very late stage. And if that stage is so separated from the rest of all the upstream activities that they now are saying, “Well, this is a bug”. “Well, no, that’s what we expected”. “Well, the requirements didn’t state that”. “Oh, we changed stuff. You’re just not paying attention, right? All of this leads to that capacity, that you bought more capacity and now you’re sending more stuff down and now … Okay, I guess we have to ..”.

And the next thing you know, you’ve got this massive QA function that’s not going to help, but getting rid of that QA function would be a disaster because you’re going to end up shipping something that’s wholly untested. I’m going to suggest you probably don’t want to do that either. So instead, if you start looking at, “Okay, let’s take a really hard look at why and get into the not big transformation, but actually get into the details. What are we testing? How do we know that we’re supposed to be testing that? What of those responsibilities would we be able to invest in earlier testing that runs through our CI/CD pipelines automatically? Oh, and those tests go red. Okay, how can we make a small investment in the culture that it doesn’t go to these people until it’s actually green and commenting out tests that are read is cheating? That’s not allowed”.

So each one of the steps that you would take to solve the problem is a natural progression from an observation as opposed to, “We’re going to roll out the new ..”. I don’t care what flavor of new hot methodology, whatever or tool. It requires actually engaging and paying attention and addressing it one symptom at a time.

Shane Hastie: But this is the shift left we’ve been trying to do for 20+ years.

Elisabeth Hendrickson: So I guess the natural follow-on question is, why hasn’t that worked?

Shane Hastie: Yes.

Why Shift-Left Hasn’t Worked: Policy vs Culture [16:41]

Elisabeth Hendrickson: That’s a great question and this is why I still have a business because the answer is different everywhere. But I will say that if you’re a leader and you’re wondering, “Wait, we’ve been trying to do shift left for 20 years. How have we not achieved the ultimate amazing result?” You have two lovers. Two. Only two. You have policy and you have culture. If policy and culture are in conflict, culture wins. No doubt about it. You can affect both. And if you are in a leadership position and thinking, “Oh, well I can’t affect culture”, mm-mm, I’m not going to let you off the hook like that.

The way you affect culture though is by feeding what you want to see grow, very gently taping down occasionally on the stuff you don’t want, but you can’t issue edicts. The culture isn’t what you tell people it should be. It isn’t the PowerPoint slide that’s got our four values on it. The culture is the implicit behavioral norms that everyone operates by and it is the cumulative effect of the incentive structures in the organization, what gets people promoted, what gets them a raise, what gets them recognition, what gets them the kudos from their peers, right?

So it’s the incentive structures and it’s the tone and the rituals and those are all of the reinforcing things in the culture. And if you want to shift the culture, don’t try to do it with an all-hands-in-a-PowerPoint slide. Start feeding what you want to see grow. I was doing a transformation with an organization that I was responsible for and I noticed that there was a culture of heroism and there was actually antipathy towards live collaboration, like, “That guy over there talks to too many people”, that kind of culture that was actually punished in the culture talking to people, well, that’s actually what I want to grow.

And so one tiny thing that I did that telegraphed very strongly when I took responsibility for this group, I happened to have a little bit of budget and I did a peer nomination award and the nominations had to be, “Tell me about a time that this person went above and beyond collaborating. Not what did they achieve all on their own, tell me about something they did that was collaborative”.

And so as a leader, you can influence culture by feeding what you want to see grow and feeding it consistently. It’s not a one-and-done. You got to keep going. And then on the policy side, that’s the hammer. You do actually have the ability to set policy, but you have to be willing to live with enforcing that policy. If you say, “Our policy is we have a unit tests checked in with every code change and then there’s nothing that will reject a pull request that doesn’t have tests”, it’s not actually a policy. It’s a wish that is written on a confluence page somewhere. That’s not a policy. How do we actually achieve this? You have two levers, policy and culture, and you have to be operating both levers. So it’s like flying a helicopter, no problem.

Shane Hastie: Yes, two simple levers that are really complex.

Elisabeth Hendrickson: But that’s why systems thinking is important.

Shane Hastie: What are you coding with and in at the moment?

Current Coding Projects and AI Tools [20:00]

Elisabeth Hendrickson: Let me start with what I’m coding. My day job, the things that people bring me in to do, they don’t … I sometimes get to pair in with people and that’s a lot of fun, but nobody hires me to write their code for them. I ended up developing a project that was really a solution in search of a problem, I’ll be honest about that, but it was so much fun and I couldn’t get it out of my head. I built a simulation of work flowing through an organization and it has the notion of teams and work and interruptions and individuals and skills and it’s got a bunch of different concepts built into this simulation.

It’s the backend engine. It’s like a game engine and that’s written in pure Ruby. And the frontend is a Ruby on Rails application that uses stimulus and web sockets to get the information and then display it from the backend simulation as it’s running. And I started just using RubyMine as my IDE, but less than a year ago, I decided, “All right, it’s time to actually dig in and learn Cursor”. And so I’ve been using Cursor in my development and it’s been amazeballs. I’m a fan.

Shane Hastie: What’s it taught you?

Elisabeth Hendrickson: One of the things that it can teach me is having AI as a thinking partner. It’s gotten encyclopedic knowledge of idiomatic programming concepts, at least for very popular languages. If you’ve got something that’s pretty niche, it’s not going to be as good, but Ruby, there’s a lot of Ruby code out there. And so I can have a conversation with it about the structure of my game engine, for example, and it will come back with intelligent suggestions for how I might think about the class hierarchy there. It’s been useful to either reinforce knowledge that I had about architectural patterns, or in some cases, in terms of what has it taught me about programming, it’s a super useful thinking partner.

In terms of what it taught me about working with AI, the answer is … I wrote an essay, AI to me is a mech suit. It’s like that thing that in a sci-fi movie you strap yourself into, so that you can lift the two-ton thing, but it’s still very much the human driving. That’s my mental model of AI because it can help me do heavy lifting, but don’t trust it. You’ll do all kinds of things on your behalf that you didn’t want it to do.

Shane Hastie: The age of verification and validation, this was something we were chatting about earlier on. So what is happening with this incorporation of AI? You made the point, don’t trust it.

Elisabeth Hendrickson: No.

Shane Hastie: How do we figure it out?

Why AI Makes Quality Practices More Critical Than Ever [22:39]

Elisabeth Hendrickson: So you’re talking about an essay that I’ve got coming out in the IT Revolution fall journal with a group of co-authors. This was not just me, but I’m super stoked about this. We were working on it just this weekend and we finally delivered it to final edit and the title of this essay is Revenge of QA: AI is the Age of Verification and Validation. And the core thesis of this is we can’t exactly trust AI and that’s why all of these quality activities that we were talking about before are even more important now, because if AI speeds up producing more code and if QA was already the bottleneck, now you’ve got an even bigger problem because we can produce that much more code.

And so taking that holistic view of quality engineering, that it’s not just what happens downstream, that it’s all of the activities and one of the most powerful things you can do is not actually write production code for stuff, right? Eliminating code is hugely valuable if what you’re trying to get is productivity improvements and so running tiny experiments upfront before you commit to producing features that then will fail because they won’t integrate with everything else in the system, for example. That’s why I view this age of AI is the age in which all of these quality practices, quality engineering shift left, everything that we’ve been talking about for the last 20 some odd years is even more critical now than ever before.

Shane Hastie: But we want it quicker.

Elisabeth Hendrickson: It’s good to want things. If you actually want it quicker, the way to get value like outcomes, not just output. The way to get value faster isn’t to remove all of the constraints because you’ll get stuff faster. You want represent value.

Shane Hastie: So when we do this shift left in the age of AI, how do we incorporate the AI tools in that space?

Incorporating AI in Shift-Left Practices [24:47]

Elisabeth Hendrickson: So one of my favorite ways to use AI is as a thinking partner just in general. So whether I’m coding and I’m using Cursor as the thinking partner or whether I am doing research, I have a company and so I do sometimes go off and do industry market research using AI as a thinking partner to understand where are the opportunities. That is one of my favorite uses of AI. And so I do see tremendous uses of AI early on, not just with respect to industry analysis or coming up with ideas, but also telling you about risks in the ideas potentially, helping you form testable hypotheses like, “If we decide to do this initiative and invest in it, give me a list”.

One of my favorite tricks with any of these, Claude, ChatGPT, pick your favorite, “Give me a list of 20 testable hypotheses that might be true, including ones that I don’t want to hear”, so the negative things like, “The one testable hypothesis is that if we invest in this, we actually lose customers”, for example. So AI is useful for the technical stuff, it’s useful for the analysis stuff, and at the other end, if you’ve got a ton of data coming out the other end, it is absolutely useful as the blender to send that data through to see if you can extract meaning out of it. So like I said, while I fully recognize the many problems with AI that we have to start addressing, I am a huge fan and view it as a tremendous point of leverage. It’s a great tool with the humans in charge.

Shane Hastie: With the humans in charge.

Elisabeth Hendrickson: Leave the humans in charge, please.

Shane Hastie: What’s the important question I haven’t asked you today?

Elisabeth Hendrickson: Ooh, that’s a great question. I don’t know what else might be interesting that you would want to talk about. What do you make of all of this? Well, the industry is changing so fast.

Shane Hastie: Where are we headed?

Future Outlook and Optimistic Vision [26:48]

Elisabeth Hendrickson: Where are we headed? I’m going to give the optimistic answer because there’s a lot of negative potential answers that could get really depressing really fast and we don’t have time for that. Optimistically, I think that we are potentially headed into an era of phenomenal innovation where we have tools available to us that can support us by doing the more laborious toilsome aspects of the work, so that the humans can really focus on the interesting and groundbreaking activities that frankly bring tremendous job satisfaction. Everybody’s got a study on the impact of AI on development out these days. There’s many of them.

And one of the findings that I’ve seen repeated more than once was that developers reported greater job satisfaction. They didn’t necessarily get productivity improvements, but that idea of greater job satisfaction for the humans, because we let the robots deal with the toil, I hope we’re heading into that world.

Shane Hastie: Elisabeth, it’s an absolute pleasure to catch up. Thank you so much for taking the time to talk to us today. If people want to continue the conversation, where do they find you?

Elisabeth Hendrickson: Well, let’s see. I’m on Mastodon and I think my handle is testobsessed and I’m on Ruby.social and we can get the link into the show notes or at my website curiousduck.io or on LinkedIn.

Shane Hastie: Wonderful. Again, thank you so much.

Elisabeth Hendrickson: Thank you.

Mentioned: