Building a Product-First Engineering Culture in the Age of AI

Transcript

Shane Hastie: Good day folks. This is Shane Hastie for the InfoQ Engineering Culture podcast. Today I get to sit down with Zach Lloyd. Zach, welcome. Thanks for taking the time to talk to us.

Introductions [00:48]

Zach Lloyd: I’m really excited to be here, Shane. Thanks for having me.

Shane Hastie: I’d like to start these conversations with who’s Zach?

Zach Lloyd: Let’s see how I describe myself. So I’m an engineer I’d say first and foremost. I’ve been a software engineer for, oh God, 25-ish years now. I’ve had a career where I’ve gotten the chance to build some really cool stuff. I was a principal engineer at Google. I used to lead engineering for the Google Docs Suite. I helped build really a lot of Google’s spreadsheet product, which was very, very cool experience. I was also an engineering manager and managed a significant team of engineers there.

I have been out of Google. I’m a two-time startup founder and my current company is a company called Warp, which is a AI-powered developer tool. It’s really like a reimagination of the command line terminal, such that as a developer you can, instead of typing commands, you can just tell your terminal what you want it to do and it will agentically do it for you.

I had a brief stint as the CTO at Time Magazine, which was interesting. So I’ve kind of been, it was my only real thing outside of the technology industry.

I’d say what really motivates me is I love building software products. My goal is to just build stuff that is useful for people, whether it’s useful for knowledge workers or useful, most recently for developers, love solving interesting technical problems. But really I’m passionate about building something that other people find useful. That’s a quick summary.

Shane Hastie: Cool.

One of the things that got us in touch was you published the Warp, how we work.

I’d call it a page, a site, a … There are a whole lot of principles and ideas. What made you want to make that so visible?

The “How we Work” document [02:49]

Zach Lloyd: Yes. So what it is it’s basically my accumulated, I guess, knowledge from being an engineer and an engineering manager. I had a period in between the two companies that I founded where I was like, “You know what? I really want to get this all down. I want to get this down from my head, how I think about hiring, how I think about building an engineering culture, how I think about values, all the way into the minutia of how people should structure pull requests and use feature flags, really down into this is my playbook, or if I were to build an engineering team, build an engineering organization or product team, what are the key things that matter to me?”

I put it down initially for myself and then I shared it with friends. I used it a little bit as a basis for doing some consulting for other startups and advising for other startups.

And then when I founded Warp, I was like, this is an awesome thing to publish because from an internal perspective, it’s like our operating system as a team, and then from an external perspective it lets people get a very good sense of the culture we’re building and how we work. And so it lets people who are interested in working with us self-select in. It lets people who are like, “No, I don’t like that culture”, be like, “Okay, that’s not the team for me”. And that’s also, it’s a very useful tool from a hiring standpoint in that regard.

Shane Hastie: So what would you say is the core of culture?

Core engineering culture values [04:33]

Zach Lloyd: I think it starts with the values that we care about. I think different leaders, different cultures are going to value different things, but for me, the type of place that I want to work and the type of people I want to work with, there’s a common set of values that I care about. And for me, those happen to be sort of honesty. Are we able to be very honest and transparent at all times, kind of a no BS type culture? Is it a culture where people are concerned with hierarchy? ‘Cause I really don’t like that. I think that the best ideas are coming from any place in the company. Is it a place where it’s like people’s ideas, the best ideas went out? That really matters to me.

A second value that I, again, this might be different for other people, but for me was being pragmatic, and to me that’s where I see things go wrong. I’ve seen things go wrong in my career is when there’s dogmatism or very rigid thinking in how you do something. I want to work someplace where we realize that solving real world problems is messy, that perfection can actually get in the way and that we’re trying to make a reasonable set of trade-offs and having like-minded people who aren’t so anchored to particular ideas that they won’t adjust them in sort of the face of new information, that really matters to me.

Product-first vs code-first engineers [06:03]

A third value for us is being user-focused or product-focused. The reason I bring this up, so I wrote this essay which I think is maybe somewhat controversial where I distinguish between engineers who are product-first and engineers who are code-first. And as a product-first engineer, what I’m looking for is are you always thinking about the why that you’re building something? What problem is it solving for a user?

And if you can’t name the problem, I think probably going off track. Whereas sometimes what I think of more as a code-first approach is there’s a class of engineering who’s like, they’re really into building with the latest technology. Are my APIs right? Are my abstractions right? And it’s like looking at the code for the sake of the code. And I don’t care about that. I care about good code in the service of a great user experience. I don’t care about … Users don’t use code, let me put it like that. They use the thing that you’ve built. And so I really emphasize that. And that actually is a great filter. Some engineers do not agree with what I’m saying at all here, but to me it’s very, very important value.

So I would start with if you’re someone who’s building a team or you’re hiring or you’re managing, I would always start with what are those core values that you really care about, that you believe in, that you can embody? And then try to build a team of people who subscribe and believe in those same values I think is a good place to start.

Shane Hastie: One of the things that I see in there is just fix small issues because that almost contradict with the product-first versus the code-first engineer?

Zach Lloyd: No, ’cause the idea behind that is that it encourages a culture of ownership. So it’s like the anti-pattern to me would be working someplace where when someone sees an issue, they throw it into Slack or the bug tracker or they’re like, “Hey, I noticed this other engineer on the team broke this thing. They have to fix it”. And so it creates communication overhead and it creates a lack of ownership. So what I’m trying to accomplish with that particular rule or guideline is like we’re all owners of this thing. We also feel responsibility. If you see something, just fix it. And I’m assuming that they’re fixing something that matters to a user. Let me put it that way. If it doesn’t matter to a user, then don’t fix it. But if it’s something that is impacting a user, I would love, I’d love it when engineers just fix stuff.

Shane Hastie: You said that this feeds into your hiring process. How do you hire and how do you hire well?

How do you hire well? [08:59]

Zach Lloyd: Yes, this is really, really hard. I think it’s hard to hire perfectly. So we are looking for generalists, product-focused engineers who have a strong fundamental background in computer science and programming, and also who subscribe to these values, put it that way. So finding those people is like, those are great people. Everyone wants those people. It’s hard to find those people.

When we hire, it basically comes, there’s three sources of people. I don’t know how into the weeds to go here, but you have people who apply. So you have a sort of inbound. You have people who are referred from people who currently work there. And then you have people who aren’t looking for a job, who we have identified is like, “Hey, this person looks like an awesome potential fit. Who we reach out to?” And that’s the three things.

It’s essentially a sales process, whether you’re starting your own company or you’re trying to attract people to your team inside of a big organization. I remember doing this at Google. I was constantly trying to sell great engineers to come work with me. So I think you have to get good at communicating why someone should do that. And then, once you’ve done that, you got to know what you’re looking for. And so for us it’s like product-focused, generalists, really, really strong CS foundations.

The reason I focus on generalists by the way, is it’s partly the domain that I’m in. We’re not doing super specialized stuff. You don’t need PhD-level expertise, although we’re now opening up some roles on the AI side that are a little bit more specialized. But in general, I like people who just have great problem-solving skills, agnostic to the technology ’cause the technologies, we tend to choose the right technology to solve the problem. And so anchoring on a technology ahead of time is like an anti-pattern to me.

I like people who can work fully across the stack. So I never hire back end versus front end. And again, there’s totally valid differing schools of thought, but to me the best way to get a product feature that works well for a user is if the engineer builds the entire thing. There’s some efficiency here to that, but I think it aligns the incentive of the engineer and the user the best. So those are some of the principles.

Shane Hastie: What does great technical leadership look like?

Models of technical leadership [11:29]

Zach Lloyd: Yes, great question. Actually let me ask you this. Are you asking about management? I see leadership. How do you think of it? Or should I just give you my whole lay of the land?

Shane Hastie: Let’s go top to bottom.

Zach Lloyd: Yes. Okay. So there’s this traditional distinction where it’s like you can have people who are tech leads, maybe not managers, but are like your architects. They’re super knowledgeable about the system. And then flip that they have a counterpart who is a engineering manager whose specialty is maybe more like people management type thing. So it’s like how do you help an engineer develop their career? How do you help them get promoted? And at most big tech companies, those roles bifurcate. So you have two separate things.

My personal view on this is that I actually like a combo. So when I was at Google, I was always on what was the individual contributor track, but I managed people and I always found that my authority as a manager came at least in part because I understood the technology very, very deeply and still contributed as an engineer.

That’s my personal preference. However, I’ve seen people be very successful just as pure ICs. If they’re going to do that, I think they really need to excel in terms of teaching other people on their team the technical skills in terms of doing things like great code review, great design documents, really modeling what technical excellence looks like so that the people who they’re trying to get to build things the right way, do that, see what technical excellence is.

On the flip side, if you’re an engineering manager, I think that the thing that makes the most successful engineering managers I know successful is really deeply felt empathy and really aligning with the interests and being very good at understanding what is the motivation of an engineer who’s on their team? Are they trying to up-level their technical skills? Are they trying to get exposed to more leadership opportunities? Are they trying to run bigger projects?

I’m not quite as great at that. I mean I’m not bad at that, but there are some people who I’ve worked with who just excel and really take joy out of helping other engineers on their team succeed and understanding what their goals are. So that’s how I would think about that role.

Shane Hastie:How do we help engineers grow their careers?

Growing engineering careers [14:08]

Zach Lloyd: There’s a bunch of ways to answer this. So depending on where you’re at in your career, there’s probably some next up skillset that you want to improve upon. So when you’re really early in your career, and we hire a bunch of people who are right out of college at work, the first thing I emphasize is just become an excellent software engineer. And what that means is can you write code that is production quality code? Is it well-tested? When there’s a bug in it, are you proactive in fixing it? Do you take code review comments well and adjust? Do you really learn the language?

So at the beginning of the career, my advice is hone your technical skills, just become an excellent IC engineer. Usually when engineers are a couple years into doing that, I think that the focus shifts a bit to taking on more responsibility, shipping things that have more impact, maybe leading smaller teams, building some other skills that you’re going to need if eventually you’re going to be either the technical lead archetype or you’re going to be an engineering manager.

And so I think that’s about finding the right opportunities when people have demonstrated that they have the technical skills to take on projects that have higher scope or projects that require a leadership aspect. As an engineering manager, the trickiness there tends to be like, are there enough opportunities for that type of thing? At Google, I saw some crazy anti-patterns around optimizing for people’s promotions, which was really interesting. At Google, I don’t know how much digressing to this, but at Google, everyone is at a level. So you start as a SWE-2 or SWE-3, become a senior engineer or staff engineer or senior staff engineer or principal engineer. And a lot of what seems to drive career progression is can you get to that next level on the ladder? And each level on the ladder has a very well-defined rubric and that rubric is sliced up into, I think I forget what it is, like four things. It’s like impact, scope, leadership, whatever.

And so there’s a lot of managing to the rubric, which I do think can create these perverse incentives where it’s like you’re not necessarily managing to what the person truly needs in order to develop themselves. It’s like you’re managing so that you can put together a promotion packet that a committee of other managers will look at and be like, “Yes, this person deserves to go up one rung on the ladder”. And there’s huge amounts of money at stake in this, right? It’s a very high stakes type thing. And so that I try to stay away from. We don’t organize Warps career progression around that.

And the other problem with that is that it creates adverse business outcomes because you end up having as a manager to create opportunities for people to demonstrate these checkbox skills on the ladder, which the canonical examples, you make a project that doesn’t need to exist in order to give someone a leadership opportunity to ship a project and you end up, this happens all the time. It’s like a crazy system. I would try to avoid that and try and just focus on genuinely what is it that the person wants to grow at and how can you help them succeed.

Shane Hastie: I know you have some thoughts on, we’re in the world of AI today, on engineers using the generative AI tools.

AI tools for engineers [17:47]

Zach Lloyd: So I just posted something on LinkedIn on this and I got in a lot of trouble, but my thought is it’s not a question of if, it’s a question of when as far as developers really need to learn to use these tools as best they can to ship more software. There is fear around them. There is I think rightful frustration around using them. I don’t know if you’ve used them, but if you used them six months ago, nine months ago and you asked AI to build you a feature, you’d probably get something that didn’t work very well and you might just decide, “Hey, this isn’t worth my time. This isn’t worthwhile. Technology’s not good enough”.

The technology is changing extremely quickly to the point where I think it is useful not for every task. I also think the correct way of thinking about it right now as an engineer is as another tool in your toolbox. So it’s raising the abstraction level from, it’s not that dissimilar, the shift from assembly language to programming language, formal programming language. And then even within formal programming languages, there’s a huge difference between working in C and working in Python or JavaScript.

And this is a step change to where the way that you can work is by simply directing an AI to do some amount of the work for you. You should consider it like a draft where you then review it and iterate and get to a point where it’s at the quality bar where if you had written it by hand that it’s still there. But I basically think as an engineer, if you want to continue in this field, you need to start thinking of yourself not as a coder, but as a producer of software and the way you produce software, you have to use the best possible method for that. And that might not be writing code by hand. That might be guiding AI and hopping in and iterating with it. And so that’s my take on it.

It is a struggle as an engineering manager and leader to change the habits of people on my team to approach programming problems in this way. And this is where I got in trouble on LinkedIn because I was like, “How do you get senior engineers who want to use these tools?” And then a lot of people were like, “The engineers know best about what tools they should be using. Why are you telling them to use these tools? They’ll use them when the tools are good enough”. And there’s truth to that. But I actually think people need a nudge and people need to learn a new skill set ’cause there’s actually skill in how do you prompt the AI, how do you work with it. So that’s my take on it.

Shane Hastie: One of the things we have seen with the take-up of these tools, a couple of things happening. One, pull requests seem to have got bigger. One study done in Australia, 300% more code being produced when using the generative AI tools, and about 400% more bugs.

AI tool challenges [20:54]

Zach Lloyd: Yes. That doesn’t surprise me. More code equals more bugs almost axiomatically. And one of the best engineers I ever worked with a Google, I was like, “What’s your main principle for writing great code?” And he’s like, “Write less code”. The more you can delete, the better. So that’s not a good sign.

I would say that using these tools, you cannot advocate responsibility for the quality of the code any more than if you were using IntelliSense in your IDE and be like, “Oh well, the tab complete, put this function so I just used it”. That’s totally unacceptable. And so you as an engineer using the tool, have to maintain responsibility for the thing that you’re submitting.

I also think one of just the cardinal rules of good software engineering is small PRs, small discrete changes. And so one of the anti-patterns to me in using AI is trying to one shot or zero shot through one prompt a huge thing. And this is where I think there’s actual skill in using this stuff. You should still decompose the problem exactly as though you were going to write the components one by one, but you can just save a ton of time. And that’s actually the interesting part of the engineering a lot of the times, it’s like what’s the right decomposition? And then you should use the AI to help you write the components. But what you shouldn’t do is be like, “Write me this whole app that does this or this whole system that does that”, because the more code there is, the less you’re going to comprehend it, the more bugs that are going to be. And so that’s, I think I diagnosed that as a misuse, not an intrinsic issue with AI. That’s my take. But I don’t know, people probably disagree.

Shane Hastie: For the junior engineer who’s coming into the profession today and these tools are their norm, one of the concerns that I’ve certainly heard and seen is how do we help that person get the underlying skills to do that decomposition well?

The impact on junior engineers? [23:13]

Zach Lloyd: Yes, it’s a great question. I have some friends who run coding bootcamps and I know professors of CS and I think that you, it’s like when I was in college, I learned C, and I’ve never written a professional program in C in my life, but I am glad that I learned C because it lets you learn, understand how memory works, how the function call stack works, basically how computers work. And so what I think will be problematic for junior engineers is if you don’t learn the basics and you are just trying to go straight to the AI that is making your apps for you, I think that’s a recipe for stuff that doesn’t work in a production environment.

I think that’s fine for prototyping. I think that’s fine for low stakes applications. Like you’re making, I don’t know, a landing page or something like that, but I don’t think anyone who’s working at Google or at a bank or at SpaceX or whatever should ever be generating code without an understanding of how code works.

So I would still teach people the fundamentals. And then I think that to your question, how do you teach people to do production software engineering, decomposing things right, I don’t know, the main way that I learned that I still think works, which is I got code reviews and design doc reviews from engineers who knew what they were doing when I didn’t know what I was doing. And I think this is one of the reason why I think it’s important that the more senior engineers learn how to use these AI tools is because they’ll learn how to review and improve the experience of junior developers using these AI tools.

Changes needed in the ways we train engineers [25:11]

I think a bad outcome would be a world where the more experienced generation of developers shuns these tools. The junior developers use these tools. The more advanced or experienced developer just think that the junior developers are misusing these tools and it’s just a mess. So I think there needs to be almost a redesign of the engineering curriculum and engineering like how you teach an engineer in light of the fact that these tools now exist. That’s what I think.

Shane Hastie: Yes. I have a grandson who’s studying computer science at the moment, and they’re making them write their test programs on paper by hand.

Zach Lloyd: Oh my god. What? Is that to prevent cheating with AI or what is that?

Shane Hastie: That is what I believe it’s about.

Zach Lloyd: That’s not good. Wait, that’s not … Yes. I think that the curriculum has to be re-imagined so that you learn the basics, but then you learn the tools you’re going to use as a pro engineer, and AI is definitely going to be one of them, and you learn how to use it correctly. The risk is this stuff is changing so so quickly that I think it’s very hard to know what the heck to do ’cause the technology has advanced tremendously in the last six to nine months, so it’s a hard thing to figure out exactly what to do.

Shane Hastie: Well, Zach, we’ve meandered a lot, a lot of really interesting stuff in there. If people want to continue the conversation, where do they find you?

Zach Lloyd: Yes, I think the easiest thing is to just reach out to me on LinkedIn. I’ll respond to people DMing me there. I’m also on Twitter, X. I don’t use that as much. I think for this group of people, LinkedIn is probably the best place.

Shane Hastie: Well, thank you so much for taking the time to talk to us.

Zach Lloyd: Thank you for having me, Shane. This was awesome. I hope people enjoyed it.

Mentioned: