AI Amplifies Team Strengths and Weaknesses in Software Development

Transcript

Shane Hastie: Good day, folks. This is Shane Hastie for the InfoQ Engineering Culture Podcast. Today I have the great privilege of sitting down with Anita Zbieg and Jon Kern. Welcome. Thanks for taking the time to talk to us today.

Jon Kern: Thank you, Shane.

Anita Zbieg: Huge pleasure.

Shane Hastie: I like to start these conversations with getting to know who am I talking to. Now, Jon, you and I met over the decades and you have a huge reputation, but I’m sure that there’s at least three people in our audience who haven’t come across you. And Anita, you probably don’t have quite the same profile, but let’s start with you. Anita, who’s Anita.

Anita’s Work in Team Collaboration Research [01:22]

Anita Zbieg: So I am the CEO at Network Perspective, and I spend most of my time researching how teams work and how the collaboration patterns form, where friction hides, how we can improve deliver flow and outcomes without adding more processes. And basically, I use two tools to do that, DevEx surveys and system log collaboration data. Those are tools we’ve built so that teams can answer the questions like where the delivery flow gets blocked or how much time we have for deep work, or how much time do we spend on context switching? And what’s most important, how do we collaborate within teams and across teams? And that’s basically what I am doing for more than 10 years already.

Shane Hastie: Thank you. And Jon?

Jon’s Systems Thinking Approach [02:22]

Jon Kern: Well, I’ll just echo that. The work that Anita does, we’ve probably been working together, I don’t know, two or three years. There’s a major overlap of the value that she was just talking about, the collaboration especially. And my background is a bit of an odd one in that I’m an aeronautical engineer, and I’m thankful all day long that I learned about treating systems as holistic, and it’s a super important element of my practice. I still write code today. I work with teams. You might call me a consultant per se, but it’s more about mentoring and actually being in work with them and writing code for firefighters. I have an actor production app that’s at least 15 years, maybe a little bit more. Started life as J2EE. So I try to stay honest by coding and helping teams code. And the work that Anita and I do together is really powerful, I think.

Shane Hastie: You’ve been doing some research. What are the key things that are coming out, the key trends, particularly when we look at developer experience, and Anita, you were talking about collaboration, friction, smoothing the flow, and how AI is impacting that?

AI as Amplifier Rather Than Magic Solution [03:43]

Anita Zbieg: So we ask about 400 tech leaders what is speeding up and what is slowing down their teams, the speed ease and quality of delivery across their teams. And we also ask about that in the context of AI that is now entering the tech world. And one may say big thing, another may say obvious thing we found out. We found out that AI is not a cure a magic, but AI is an amplifier. And when you think about it a little bit deeper, it comes out that AI amplifies both delivery efficiency, but also deliver weaknesses. And that’s the point when we started thinking about fundamental key things that you might want to focus on when you think about AI as a turbocharger or amplifier. And it is really close to what Jon is doing for years already. So it’s about focusing on people and keeping adapting.

The Complexity-Experience Quadrant [05:10]

Jon Kern: Yes. I would tend to agree. I was looking for something to put into a visual today as I was talking to my colleagues about my experience with replit.com, which is an AI-powered, amazing tool that I’m at the moment infatuated with but my blinders are not on. One of the thoughts that I had while using it is if you maybe think of the complexity of what you’re working on on the Y-axis and maybe the experience that you have or your team has on the X-axis … I drew a bit of a quadrant, like a classic type of a quadrant because I think when things are simple and you’ve got a powerful AI tool, you can probably have some success because it’s simple and it’ll probably lead you down the right path and it’ll accelerate flow undoubtedly. But if you’re not experienced and you’re dealing with something highly complex, it’s probably a danger. You might get initial like, “Wow. That was really cool”., and then crash and burn later because you’re not aware enough or experienced enough to know the pitfalls that might have occurred.

So I think it’s in the hands of a team that’s really good at collaborating and has lots of experience. I think you’ll see the acceleration being positive and that would be how I would interpret what Anita was saying about the findings. That’s how I think of it is you can do a lot more damage way more quickly if you don’t have enough battle scars and you’re doing something hard, or you can do a lot of amazing things, which I’ve been experiencing lately, and I don’t think it’s going to supplant or replace me or others, but it’ll help me accelerate being able to explore and proof of concepts and really powerful things and get deeper and richer collaboration more quickly with stakeholders and the internal folks. It’s really a great topic to talk about.

Shane Hastie: One of the things that I’m hearing and these conversations on social media and all over is that this is destroying opportunities for junior folks. Companies are not taking on juniors because the tools are doing that instead. How do we make sure that people learn what’s needed to use it as an amplifier of the good? How do we build those base skills that you were talking about?

Impact on Junior Developer Opportunities [07:56]

Anita Zbieg: I would say that you need to think about the delivery as the whole complex system. There’s everything inside from the specification to code, quality testing and having the user feedback. And the developer’s role is also changing into an orchestrator rather than just creator of a smaller part of what we are working on. And in this context, I think that we should provide people with as much context and understanding of what is going on in the whole system as possible to make them really powerful in what they are doing as a small part of the whole delivery system.

The Developer as Orchestrator [08:56]

Jon Kern: Yes. I think there you’re touching on something that I really try to reinforce. We’re working with about 40 developers recently, and I would say my system is simple, but it’s not easy. It also has a lot of recursive things, one of which is really understanding the direction, the vision, where we’re trying to go, whether that’s the whole product we’re building or down to an individual feature that somebody might be assigned to. That sense of allowing them to see the big picture not just be compartmentalized. So I think for the companies who are shortsighted and think that they can save a few bucks by not hiring a junior person, I will say the things that I’ve had Replit do while doing some of this proof of concept and experimenting and building a very complex system with an AI engine underneath the actual system, not just the one that Replit helps, not a chance myself or the best developers I’ve worked with could hold a candle to the speed with which I could ideate and produce and reflect and iterate.

And I look, “Oh, wow, I spent $50 today”. This is not a way to supplant the junior. So I think the more companies that think that way, they’re just on the way out, in my opinion. Versus I’ve also discovered I’m not that savvy. I’m a C, C++ Java, not a React and some of the newer stuff. And one of the products that I’m experimenting with Replit, oh, it’s like TypeScript. And I’m even finding … I use my knowledge to say, “Hey, I’d like to do X, Y, and Z”. I just don’t know the lingo and it does it for you. So I’m even able to learn. So I think bringing junior people on, having a more holistic way of onboarding people in the beginning, and you’re not trying to pigeonhole people where they can’t be as productive.

So I think it’s a mistake to assume that you don’t need junior people because somehow you got to backfill. And yes. I have stories about onboarding junior people and within six months they’re building a nice, small but decent feature all on their own, so to speak, because of the ways that the system is architected, the way I approach things, and the same thing with bringing them up to speed quickly such that they’re able to own something and do it and make … Pride’s not a good thing, but make me proud, most people would say. And those are junior people less than a year out of college. There’s no way that you can replace that kind of attitude and aptitude just with an AI tool because a lot of that was dealing with customers and dealing with actual problems on the fly that AI is not going to help with those things. So anyway, long-winded way to say it’s do it at your own peril, tossing the juniors aside.

Fundamental Team Struggles Beyond AI [12:03]

Anita Zbieg: And I would even say it’s not only about junior people. I was talking with senior people already using AI. In our research, we found that many dev teams struggle with clear specification and shared ownership, fast feedback, and effective collaboration, and they already have AI, they already have senior people. And those are the basics, I would say, that are even more important right now when code creation can really speed up.

Shane Hastie: What does it mean for me as programmer today that I now have to become an orchestrator? What am I doing differently?

Orchestration and Rapid Feedback [12:55]

Anita Zbieg: The first thing is that probably you need to collaborate more because you not only collaborate with your AI agents, but also with much more people to orchestrate everything. That is one thing. Another thing is about you need much more and faster feedback to know if you are orchestrating and if you are going into the right direction. As Jon would say it, to know or sense if you are going into the right direction or not, if you are getting warmer or colder like children are playing.

Jon Kern: Right. The speed of the feedback is important. I guess orchestration strikes maybe a slightly better note than a methodical process-wise, unit-wise approach, which assumes that … This is another one of those times when you want to have … One of the dimensions I’m talking about is the level of complexity. Sure, things that are super simple, they should take a simple approach and maybe even automate it, it’s so simple. Generally, what I’m talking about is when the work and the system become more complex, the more you have to rely on the sense and respond and the more of the, what I would call the sense of orchestrating because it’s a little bit more of that holistic engineering feel that I often have where there are multiple interrelated things you can’t know at all and you have to have a sense of the inner relationships, and that only comes through closer collaboration, more rapid feedback, taking smaller steps, the typical things that … Funny enough, that’s one thing you have to guard against AI.

It loves to help, and sometimes I have to fight it a little bit to not go quite so far, which is an interesting thing that how do I have a sense of that? I don’t know, but I do and not everybody will. So it’ll happily go more chaotic than you might want. So I think for me, the orchestration bit is that deeper sense of instead of predicting planning harder, like, “Oh, we just need to do more. That’s why we missed”. No, no, no. You actually need to do less and relax a little bit more, take in the input and sense and respond more rapidly. And I would call that a sense of orchestration style.

Shane Hastie: One of the previous guests on the podcast were talking about some of the quality surveys and quality investigation, and the size of the pull requests get larger, 300% more code, 400% more bugs. So how do we manage the tools so that we’re not getting that overburden of too much stuff?

Managing Code Volume and Pull Request Size [16:00]

Jon Kern: Even before the advent of AI and code generators and whatnot, those were telltale signs, as you mentioned, a huge pull request that nobody wants to review. My obvious wisdom is to avoid it as opposed to figure out what to do about it. Maybe reject it, tell them to cut it apart and give it smaller. But nonetheless, certainly the propensity of AI code generation, I’ve already had to coerce it and beat it back a little bit to not go so far so fast and to take smaller steps. That’s one of my superpowers is I often say I can out small anybody because that’s so key of making features smaller and smaller and smaller. Because as I always say, you can always add more if you realize, because we’re coupling this with more sense and respond, more orchestration, more rapid feedback. So you can work with feature flippers.

I often work with maybe a customer who wants this particular feature so you build a rapport and you can get very early, “Well, it’s almost there”. And you probably already knew where you wanted to take it and they can either help reinforce that, “Yep, okay, I do need to do that next step. Thank you for the feedback. I’ll do that next”. And that gives them a sense that we listen to them, we build something that’s quick and … Or they never mention it. You go, “Oh, wow, guess we’re done. I didn’t need to build that after all. ” So you can never get back time you spent building too much and you can never recover that lack of learning that you didn’t even know it was too much. I think for me, it’s all about trying to avoid that in the first place. And if it happens, it certainly is a probable sign that you bit off too much.

Shane Hastie: So what you’re telling me is go back to the principles that you and others came up with in 2001.

Jon Kern: Could be.

Shane Hastie: Simple ideas of the agile manifesto.

Returning to Agile Manifesto Principles [18:04]

Jon Kern: Yes. I’m pretty sure that fundamental’s a good word. Tell me what’s changed and what doesn’t work anymore. If anything, folks need to spend more time trying to understand it, because I admit it’s ambiguous. There’s no start here. Step number four says this. It requires a certain mindset to even understand it, but it’s worth trying to struggle through it if it’s still not quite clear because there is a lot of gold. And what Anita and company have focused on around collaboration, deepening the work, showing if you were in pointless meetings, being able to get input from the voices on the ground. Anita taught me, especially when we use the DevEx survey and there’s freeform comments that she really reinforced and it became very obvious very quickly she was right that there’s a lot of value. When someone takes the time to tell you something for real in those comments, yes, you get little Likert scales, you get little numbers, but boy, the gold is often in the text and what somebody offered anonymously. That’s why it works.

But yes, I think a lot of it is centered around that first principle or the first value, I should say, in the manifesto around individuals and interactions, which is collaboration. That’s another word for that. I love process and I love tools, but I love collaborating to deliver more value. That’s really my passion is I’m a value nut, so to speak. That’s why Anita and I get along so well.

Shane Hastie: So Anita, what are people saying in these surveys about where is the friction? And you said there was the getting better quickly, getting worse quickly. What are the organizations that are getting better quickly doing well?

Continuous Improvement Over New Creation [19:56]

Anita Zbieg: I would say their mindset is more focused on improving things, like focusing the second, third time on the same thing and finding out how to make it really great rather than mindset of producing more and more new things. We really very often have this gratification that we’ve created something new. So we have this tendency to build more and more, create more and more. But what I have observed is that the top tech teams, they are extremely focused on improving things like taking the second look, the third look and the same thing and trying to find out how to do it 5% better, 2% better, but doing iteration after iteration after iteration.

Shane Hastie: And what are the gotchas to avoid? What are the things that we can advise our listeners that, hey, here are some patterns, here are some mistakes that other people are making with the adoption of AI tools into that developer process in particular?

Common AI Adoption Pitfalls [21:17]

Anita Zbieg: I would say there are two things we found out. The first one is that many tech teams adopt AI locally, like local optimizations. We can produce code faster, but that’s only one stage of the delivery process. And the success of adopting AI depends on how the whole delivery process works. Because if you produce locally more at the stage of producing code, then your bottleneck moves into the code review or the bugging. It is what Jon is always telling us, take a look holistically on what’s going on. And that is one trap, I would say. And another one is measure and imagine how the great or good enough looks like and try to find any numbers, signals, opinion … Maybe not opinions about that. Because another trave is that we may perceive that we work faster or better with AI tools. But in reality, when you have senior engineers, they may work slower and still perceive they are going faster. So have any signals of what you are doing while adopting AI.

The Importance of Measuring Flow [22:53]

Jon Kern: That is a great point. As you might imagine, Shane, that folks often come and ask help for their team or teams, and it’s not like people are doing good and can you help us get to great? It’s often can we try to maybe get from not so good to just possibly maybe up to good. In those situations, it’s more likely that they don’t even have any data to wonder, how do you measure yourself? Some of the things we often do early on with folks we engage with is try to get some metrics in there and even try to get a sense. They don’t always have it measured, but things like reverse flows that are often going on where stuff goes down the path and whoop, go back to go like the Monopoly game, but so little is measured.

And then if you inject something like AI onto something that you’re not already very clear about what sort of flow do you currently have, it’s easy to mistake like Anita was saying, sensing that it seems faster. Yet what you were pointing out, Shane, is sometimes you’re actually overloading the system. Now suddenly the pull requests are bigger because it just puked all kinds of code, but hey, it works, which is okay. It’s hard to deny. Working is good. But at what cost? And to Anita’s point, you might be chasing bottlenecks that appear elsewhere in the overall process, but boy, I got my code done and pushed and someone else has to … So it’s that cautionary tale of you can’t improve what you don’t measure and then the typical careful what you measure, you just might get it.

There is no easy way out other than trying to pay attention and trying to be accountable to yourselves and observe. Take some DevEx data points along the way because you can get some good anonymous feedback and then take it again in three months if you’re trying something new and see if anything sticks out as well, this suddenly got worse or this suddenly got better. But yes, I think that’s a common pitfall is people want to improve, but they don’t have any current way of even seeing what’s going on.

And I actually had a colleague of mine when I was a little bit worried about having an impact on a team, not enough of an impact. He said, “Oh, look, you’ve already shown them things that they have never seen”. That’s a start. Even being able to visualize, wow, this is what’s going on. That in and of itself is an improvement because now you can at least start to look at it, watch it, think about it, and do something different. So I think that’s an important lesson.

Shane Hastie: What’s the important question I haven’t asked you in our conversation today? What’s the message to the technical community?

The Widening Gap Between Organizations [25:55]

Anita Zbieg: So when I was summarizing this research around AI, I was thinking can something big, a big change in tech happen with the AI adoption? And I can imagine the gap between strong and weak organization can grow because if AI speeds up good things and bad things, we may see in a couple of years that teams that already have good processes, good collaboration, useful data, and obsession of improving things, AI can make them even faster and better, speeds up this whole process. But when those basics are missing, AI won’t fix it. It may increase confusion, speed up mistakes, create more technical organizational collaboration debt. And that is something crazy I imagined when I was thinking about the report, playing the scenario, future-oriented scenarios.

Jon Kern: Yes. What you’re pointing out is it could increase the gap over time, accelerate the gap between the good and the not so good. It hearkens back to … I’ve been saying this for decades probably. That there are a lot of major companies who have software, especially say 10 or 20 years ago, was a part of the company, but not of the company. So they could get away with having a really crappy software because our biggest businesses just were great at logistics and we’re great at moving goods, and the software’s a small part of that, for example.

Versus another company whose software was maybe the only thing they did, you could go bankrupt if you had too much technical debt, you couldn’t keep up with competitors. So I think it’s a similar thing where depending on your core competencies and where AI fits in, maybe if it’s a small part of your portfolio, might not see too much difference because it’s not the bulk of what you do for a living, but others, it might sink or swim. You might actually be exacerbating the worst and cause you to go bankrupt. So who knows? Time will tell.

Shane Hastie: Anita, Jon, lots and lots to think about this and really good advice and good pointers. If people want to continue the conversation, where do they find you?

Anita Zbieg: On LinkedIn.

Jon Kern: Yes. Me too. Find a step, find us on LinkedIn.

Shane Hastie: I’ll make sure that both of your LinkedIn profiles are in the show notes. Thank you so much for taking the time to talk to us today. We really appreciate it.

Anita Zbieg: Thank you, Shane.

Jon Kern: Thank you, Shane.

Mentioned: