How Blameless Culture Transforms Engineering Teams

Transcript

Shane Hastie: Good day folks. This is Shane Hastie for the InfoQ Engineering Culture Podcast. Today, I’m sitting down with Tameem Hourani. Tameem, welcome. Thanks for taking the time to talk to us.

Tameem Hourani: Thanks for having me, Shane. Good to be on with you guys.

Shane Hastie: My normal starting point with these conversations is who’s Tameen?

Introductions [00:58]

Tameem Hourani: What version of the story do you want? Let’s stick to tech, I suppose. I’m based in Boston. I grew up in the Middle East on a small island called Bahrain, which most people aren’t too familiar with. I made my way to Boston for college in ’06 and never went back. I was supposed to do four years, go through university and head back home for family business, decided I liked tech too much. I got my first job at EMC running their networks across I would say a few hundred offices, which was really interesting. From there, I made my way into an ops role, which is where a lot of what we do at RapDev began.

Spent a good amount of time running a NOC, SOC, a few other ops roles. And back then, it was very heavy on the technology stacks versus the services as the world has transformed today. We’re responsible for storage, we’re responsible for security, responsible for mail services and et cetera. That over time has evolved into being responsible for different services regardless of the components that build them. That’s basically how I ended up here.

Shane Hastie: And in that journey, what have been some of the key learnings that you would share with the younger you, somebody coming out of university, going into a technical career?

Calm leadership in action – this too shall pass [02:14]

Tameem Hourani: I’ll tell you a story because those tend to do better. I was at EMC at the time, responsible for data centers. We had a major outage, I think one of the main power lines into the data center went out and it was a weekend. There was multiple leaders there. And a few hours into it, I was rushing down one of the aisles in the data center and one of the VPs pulled me over. That was probably like two or three years out of college, and everything was a fire and everything was an emergency. And I got pulled over and he started lecturing me actually. He started telling me that I shouldn’t be stressed and everything will pass and this will be fine.

At the back of my head, “I’m thinking, Ramesh, we got to fix the problem. We got a real issue here”. I look back on it now, and he probably realized that it was just email, it was just a storage array and everything will be fine and the business will continue. To me at the time it was, “Oh my God, the world’s on fire. People can’t send emails. In the grand scheme of things, it really wasn’t that bad”. The irony is Ramesh is now on our board at RapDev. He’s come full circle. He was actually a customer of ours for a while, and I asked him if he’d be interested in helping be an advisor and he graciously accepted. But one of the biggest takeaways for me back then was this too shall pass.

No matter how big or small these issues are they run into, it’s not the end of the world. And today we see outages of all sorts. I think one of the GCP regions went down probably six weeks ago, was that, remember that outage? I think it was six or eight weeks ago, took down half the world with it. But I’m sure there’s a lot of lessons learned. They put out a very nice manifest of what happened. I think very relevant to some of our conversation today is very much a blameless objective approach on how things went down and how we improve them. But even in the bigger, larger outages and scenarios, it’s important to anchor around staying calm and remembering that we’ll figure it out. We’ll work through it.

Shane Hastie: Stay calm, we’ll figure it out, we’ll work through it. Good advice. How do we create well genuinely blameless environment when things go wrong, or when things are going well?

Building a genuinely blameless culture [04:22]

Tameem Hourani: That’s a great question. I think I’ve been fortunate to work in both ends of that spectrum. I worked for a CIO that walked into the NOC once, stared at one of the screens and said, “Well, that’s obvious that users can’t log on to VPN and the CPUs are high or whatever”. I don’t remember the exact reason. “Who’s responsible”, and literally asked for a name in front of a room of a hundred plus people. Of course, somebody had to raise their hand and that person was walked out. And I’ve been fortunate enough to be on the other side of it where I didn’t personally, but somebody took a whole data center down. Customers couldn’t log on to Wayfair, and the simple approach was, it’s fine.

They’ll come back in five minutes and buy their couches. Let’s figure out why the system isn’t resilient enough to withstand what just happened. And in both scenarios, you could argue that the second scenario is actually a broader impact to the business, just given the fact that it was revenue impacting and the first wasn’t. It was just users could log on to VPN during a snow storm in winter Boston, I think of some odd scenario. What was very evident is trying to blame a human for a problem is never a constructive method of improving your system. And it was very interesting to see a whole culture where anytime an outage happened, the first reaction was how do we improve our systems? How do we make them more resilient?

And at first, it’s easy to think that something’s can and something cannot, right? It’s easy to think, “Hey, at the end of the day it’s always a human that’s responsible”. But if you really change that mindset to building more resilient infrastructure, services, front ends, back ends, whatever it may be, you can really land in a place where everything can be attributed to a lack of resiliency in a system. That’s what makes it a very interesting way of approaching outages or approaching downtime or engineering issues regardless of what they’re and what they’re caused by.

Shane Hastie: Why don’t we move towards that culture from the first one?

Why organisations struggle with cultural change [06:25]

Tameem Hourani: Also, great question. A lot of hard questions today. It’s always easier to build a culture than it’s to change your culture. That applies to everything. I think one of the first things you have to get comfortable with as a leader in an organization is not reacting negatively when things go bad. And I think it’s somewhat tied to the first example I was giving around data center outages and really being concerned about getting it back online. And Ramesh at the time being very calm and saying, “Hey, let’s figure it out”.

The first step is not trying to pinpoint who did what, and it’s actually trying to understand why the system failed, not because of an outage, not because of a code deploy, not because of a bad switch, not because of a load balancer. What could have been built more effectively upfront to enable a culture where when outages do happen, you’re working collaboratively to figure out what can I architecturally modify, change, improve to prevent this outage from happening? Look, at the end of the day, most changes are caused by a human, right? And the most downtime and outages are the result of a change that a human has made.

It’s very easy to over index on looking for that one person and blaming them for not being careful, or for not double checking their work or for not testing or for not… And those are not wrong, right? Those are not the reasons to not have failures. But the ability to overcome the human instinct, to blame someone or to look for someone or to point at someone and to look internally and to think what could have been prevented from an engineering standpoint is really, it’s a very hard thing to accomplish, but it’s a very powerful thing to have in a culture.

It enables trust, it enables resiliency, it enables a way of thinking from an engineering perspective that allows you to move much faster, release much more frequently, much more consistently, builds a ton more velocity in your organization and it’s all a result of really understanding how to lean on the technology that’s available to you. And once that’s working, it’s a really, really beautiful thing. But I would say it starts with leadership resisting the urge to try to look for someone to blame for an outage. And we can get very technical in how you do these, right? And you can almost talk about the industry has transformed significantly in the way we manage change.

If you think about ITIL and ITIL, v3 and v4 and all that stuff, maybe 10 years ago even, and look, some organizations still do this, but there’s this mantra that change is bad, right? Change is risky, change brings downtime. We don’t want change. We want to slow down change. That’s not an option anymore. We live in a world where if you can’t embrace and accelerate change, you’re out. And there’s millions of examples, everything from Venmo and PayPal to Bank of America, to Netflix and Blockbuster.

I am sure you’ve got tons and tons of these examples, but the companies that win are the ones that have figured out how to embrace change and really enable engineers to be comfortable, to feel like they can trust each other, to feel like they can build on a platform that is resilient. And that starts at the top. That doesn’t start with engineers, it starts with leaders.

Shane Hastie: It starts with leaders. What if I’m a new leader? What if I’m one of those engineers who’s now moving into a leadership role? Where do I learn? How do I pick this up?

Becoming an engineering leader [09:55]

Tameem Hourani: I’ll tell you what worked for me. That may not work for everyone, of course, and I have no authority to discuss how others should learn. But what worked for me as a shared experience is exposure to other managers. And I say manager is not leaders, because it takes a good amount of time to become a leader that’s impactful across an organization. What’s really cool is the people that generally become managers are the ones that are really good at their jobs, right? So, you’re a great engineer, you’re a great developer, you’re a great network administrator, you’re a great storage, whatever it is, you’re a great front end developer. People naturally gravitate towards you for help.

And over time, an organization would by proxy of the people on your team or your peers coming to you would nominate you to become a manager. Or you may raise your hand and say, “This is something I’m interested in”. I always say, managers have the hardest jobs in organization. Managers are responsible for their team’s outputs with very little control or influence over them. Once you graduate from a manager to managing a manager, whether it’s a senior manager, director, whatever title a company uses, it becomes significantly easier because you’re now working with people that have already been proven to be good managers and are starting to become good leaders.

So, all that is to say becoming a leader is not something that to me didn’t happen overnight and I wouldn’t expect it to happen overnight. I think some of the leaders that are most likely to fail are the ones that appoint themselves as leaders. And that’s not really how you become a leader. You become a leader because you’ve earned enough trust with your team, you’ve earned enough trust with your peers to have enough influence and to be looked to as someone that can help either drive an organization, affect an organization, motivate and advance an organization in ways which people feel is valuable.

But for me, what has worked is just the exposure from peers and exposure to what’s happening in the industry, trends, staying in touch, staying ahead and trying to help everyone that needs it. Don’t force it on anyone else? I’d love to hear your opinion on this because I’m sure you’ve studied it a lot more than I have, but that’s what’s worked for me.

Shane Hastie: Yes. One of the things that I see a lot sadly, and it still happens, is we take that great practitioner who people have started to see, they’ve become the go-to person, and then we put them into that first level leadership role and give them absolutely no support and training.

Tameem Hourani: No guidance.

Shane Hastie: No guidance. And the skill sets are different. What makes me a great engineer does not necessarily make me a great manager of engineering.

Tameem Hourani: Leader. And I completely agree. And I think that’s why one of the things they said is decide to become a manager. A lot of people are pulled to become managers, and that’s not a good recipe. Just because you’re a good engineer doesn’t make a good manager and you may never want to become a manager, and that needs to be very clear, right? But I completely agree I’ve seen that happen. And to counter that, it’s extremely important in an organization to allow engineers to attempt to become managers and allow them to fail and go back to being engineers. A lot of organizations don’t work that way. A lot of organizations will expect you to become a manager. You’ll fail as a manager and you’ll be asked to find another role.

But why? You became a manager because you’re a great engineer. Why not put you back in a role where you’re valuable and successful and extremely high output to the organization you’re in, and very likely happier than the manager role you have put into the first place? And I think organizations need to be open to that. And I’ve only seen this work or I’ve only seen this done at Wayfair. I’ve never seen it anywhere else, but just because you’ve failed in your promotion to manager, it doesn’t mean you should be penalized and sent out the door. Go back to being a great engineer.

Shane Hastie: So, extending this, what makes a great culture in an organization?

What makes a great culture? [13:57]

Tameem Hourani: Again, I’ll talk about my own experiences, not to tell or enforce what I believe should happen, but what’s worked for us. I think the first and foremost most important thing has been transparency and transparency with the good and transparency with the bad. And I really emphasized this because I enjoyed it. When I worked for leaders that were transparent where I knew exactly what was going on, and if I didn’t, I could ask him, I’d get an honest answer. That builds trust and that takes time.

But what starts to happen is it snowballs when you’ve got enough people that truly believe that it’s a transparent culture and can openly talk about anything, you start to build this really cool effect of trust regardless of what your role is in an organization, regardless of where you sit or what you do. And for me personally, I’m very big on being transparent in communications such as Slack. I really don’t like direct messages in Slack, and whenever I get messaged directly at work, I remind people to go back to channels. There’s a ton of value in making this data parseable, structured, repeatable, findable, searchable, and that allows people to really believe that everything is shared.

The only thing I don’t share at RapDev is people’s personal comp and of course anything confidential, but there’s no reason to hide anything. You want to know how much cash you have in the bank, I’m happy to pull it up. You want to know which customers are doing well or not well, our numbers, our margins, our profit, our revenue, everything is extremely, extremely transparent. And if you have that, you start to build a really cool trusting culture. You don’t need to worry about showing up at nine and leaving at five. You don’t need to worry about where you’re working or if you’re online or if it’s a weekend. You trust that people will get their work done and that again comes from transparency.

As long as the work gets done, it doesn’t matter where it’s happening and then that becomes a flywheel. It’s the snowball again. It all really anchors on transparency. Once you’ve got that and you’ve got the trust built, it’s endless what you could do with a group of really sharp engineers that trust each other.

Shane Hastie: I want to say you can build that from the ground up. What if I’m not in that an organization now, how do I nudge it? How do I influence it?

Influencing culture from within [16:21]

Tameem Hourani: I think you got to be a catalyst, really. And I think this is one of those things that doesn’t have to start at the leadership level. We would expect and hope that all of our leaders are transparent, but it’s not an easy thing to change if your whole career has not been that way. But if you have a culture of engineers that are not afraid to fail and not afraid to get penalized for outages or for failures or blamed for things going wrong, you start to inherently build a culture of trust. And if you know you’ve broken something, raise your hand and speak up because you know that that’s the right thing to do and you won’t get blamed for it. You won’t get penalized for it. That’s a thing that doesn’t have to start at the leadership level.

And look, are there leaders that are going to be against it? I’m sure, right? Not everybody’s going to be really appreciative, but I’d like to believe that much more often than not, being transparent and being upfront and honest is actually going to end in a good place with good results. That’s what’s going to drive change. Whether it’s two engineers, whether it’s 10 engineers, whether it’s managers, it’s infectious. And I think it’s a very cool thing to see in an organization that’s trying to transform. And to your point, it’s a lot easier to start like that, right? It’s like with anything, change is hard, but being transparent is something that doesn’t have to start at the top. And by the way, Wayfair was very good at this.

They were really good about sharing any information you wanted anywhere. I think the buzzwords for that today is democratizing data, but we don’t need buzzwords. It’s just transparency. Anybody has access to anything anywhere except for salaries, and we post all of our comp bands on our notion page, on our Wiki. We don’t hide that from anyone. I was at one of our happy hours two months ago in New York and we had two engineers that were asking about applying I think. We were like, we had a fast trending QR code that we were hiring or something. And so, we started talking. I had a T-shirt on and they asked me, “Hey, what do you guys pay?”

So, I pulled up literally our notion on my phone and showed them based on your level, here’s our comp bands. We don’t hide it. There’s nothing to hide. And everybody knows that, “Hey, at RapDev, at this level, here’s what we pay.” That goes a long way. Engineers don’t want to have to play games to switch roles or to join a company. It should all be up. But I really do think transparency can start from anywhere.

Shane Hastie: Another thing that I know you are passionate about is the use of tech for good. How have you applied that?

Tech for good: Project Citizenship [18:45]

Tameem Hourani: The first way we’ve ever applied that there’s actually, we’ve been doing more of it now, which is cool. See, the challenge in the business we’re in is you’re borrowing time. You have a finite bucket of time and you’re always borrowing time. And I guess that’s a problem in any business, not just our business. But in order to build tech that is not revenue generating, but is socially impactful, you’re taking away from time that could be important to the business. And the beauty about not having investors and not being a public company and not having to report to anyone is we can make those decisions. We can choose to lose some money.

We can choose to invest time in something that’s not going to generate a profit, or even a revenue for RapDev. And that’s one of the best things I’ve ever stumbled upon, right? This wasn’t even by design, but we just didn’t take money because we didn’t need it. But I look back now and it’s amazing that we don’t have to respond to anyone financially. But yes, taking time away from other projects is definitely impactful. One of the first projects we did was an organization here in Boston called Project Citizenship. While I was at Wayfair, I had my green card. You got to wait five years to be able to apply for your citizenship to become a US citizen. They had some agreement with HR or something at Wayfair.

An email went out. Turns out if you’re married to a US citizen, that was five years become three, and I didn’t know that and they did. So, they reached out and said, “Hey, look, you’re actually eligible. Here’s why, dah, dah, dah, dah, dah”. And that was a huge change or a huge jump in my life at the time. It meant a lot and it really helped me mobility wise, get around, no Visas, travel, et cetera. And there’s a non-for-profit. They didn’t charge me anything. They’re real lawyers just helping through process. And what I remembered at the time is the process was pretty clunky. It was very manual, it was all forms, it was all PDFs. So, one of our hackathons at RapDev, we invited their CEO into the office.

And the hackathon was you spent 24 hours to build something that could positively help project citizenship. And that started our whole tech for good, if you will, initiative. I just came up with that term, that’s what we call it, but she came to the office. She was a judge on the hackathon. She started by presenting the problem day one and then helped judge at day two. And today, they use ServiceNow to intake, I wouldn’t call them clients, but applicants and automatically generate a 28-page PDF that then gets filed with the government to apply for a citizenship. And that’s saved them a ton of time. It’s allowed them to intake two or three times the amount of volume they would’ve been able to manually.

They used to have these days where they’d invite everyone to a room and they’d have a lawyer paired up and the lawyer would fill a form out for every single applicant. And today it’s just done online through ServiceNow. All the checks and balances are there, and boom, you got the PDF, good to go. Things like that go a long way. It’s a good way to help the community. It’s a good way to give back and they did a lot for me that I wasn’t expecting, and the least we could do is try to help them out. That’s how it started.

Shane Hastie: What’s the impact that doing that has on the people in the organization, in your organization, so the people in the hackathon?

Impact on team culture [21:55]

Tameem Hourani: We do have hackathons. We actually have them coming up on August 14th every year. We have these events twice a year. Back to culture, your topic, we do something called summer week and something called winter week. And they’re basically two weeks a year, where we fly the whole company into Boston and we do all sorts of events, work obviously during the day, and then it’s things that range from community service to karaoke to you name it, dinners, et cetera. During summer week, we always have a hackathon, and that’s the last 24 hours of summer week, and that’s where the judges come into the panel, et cetera. What does it do for the team? I don’t know. I don’t like to think of things as what do I get out of it, right?

I like to think of things as do what’s right and try to do what you would’ve wanted to happen. The results are a trailing indicator versus going into it saying, “Hey, I need to do a hackathon because I need to make people happy. That’s never been the way we approach anything at RapDev, frankly. And even financials, we never went or started RapDev with a specific agenda on financials. We always said, do right by the customer, do good engineering work, and the rest will sort itself out. And it has, we’ve been growing great, and our lodges are great. Our business is healthy. It’s interesting because we don’t have a team of people running culture. We don’t have a chief culture officer.

We don’t have the fun police, like it’s woven into the fabric of the company. And we’re very lucky because I’d like to believe everybody feels some sort of responsibility to maintain it. It’s a very welcoming culture when you come into it. And we hear this time and time again, and there’s nothing better to hear than someone new joining RapDev and dropping a nugget about how different or refreshing or challenging, sometimes intimidating, right? It’s intimidating because of the level of engineering, and that’s exciting for engineers. Engineers want to be challenged. Engineers don’t want to have easy jobs. That’s exactly the type of culture we have. It’s a very constructive culture, but it’s very transparent.

It’s very supportive. Nobody feels like they can’t ask a question or cannot get an answer, but it’s a very high bar. That’s a byproduct of all the hard work that’s gone into what we’ve done so far. By the way, I cannot take credit for this myself. This is a result of years and years of everybody involved in RapDev pulling this together and maintaining a high bar, challenging each other and knowing that they can safely do so without being reprimanded, without being frowned upon, without being looked down upon. It’s a really important thing, and we can’t ever lose that. But it’s not a single-handed achievement or accomplishment that one person has done.

Shane Hastie: What’s the important question I haven’t asked you?

Learning from failures [24:34]

Tameem Hourani: Great question. I think it’s easier to talk about the things that go well and forget about all the things that didn’t on your way to getting it right. And I think it’s something that’s often looked over very quickly and something that we don’t like to go back and really understand how we’ve arrived at where we are, because there’s a lot of failures and there’s a lot of mistakes and there’s a lot of decisions that were wrong, and those are the things that make us a better company today. Those are the things that allow us to reflect, and I think those are the things that help build trust. Talking about those things openly.

When things go bad, when a project doesn’t go well, when you take down a customer’s environment. When you sell a project, it’s not just engineering, it’s the whole company. When you sell a project that we’re in way over our heads or we’ve underscoped or we’re losing money on, those are all important things to be able to discuss and understand and learn from. But again, it’s all about being blameless. You don’t say Who sold this deal? It’s a bad deal. We never do that. We try to understand why it’s a bad deal. Why did we underscope it? And then we use tech to solve it. In a lot of instances in sales, we now use plugins instead of having the ambiguity of a human and that’s what we’ve learned.

Humans are very different. Every human is going to have a different level of estimate for a different set of work. What we’ve learned is you can actually measure most of this, right? So, we built plugins that we give our customers to try to come up with some baseline of what needs to get done here. And that’s made us much better at delivering accurate SOWs and accurate contracts and writing accurate scopes. It’s like a project, right? When you do your points and you do your votes when you’re doing your compound planning, it’s not that different from trying to get an estimate, right? But essentially, we try to use as much technology as we can to be objective in what we’re building.

So, there’s been a lot of things that have not gone right at RapDev and at every company and in every organization and on every team. And I think the way you react to those is a big factor in creating a blameless culture, creating an environment that allows us to learn from each other, that allows us to be constructive. How could we do this better? What could we have changed? Is it technology? Is it checklists? I hate the word process. I don’t like process. Can we add a bullet or remove a bullet? Can we have a better understanding upfront? Do we need more? Is it a capacity problem? But to always be objective and constructive about understanding what went wrong and transparent about things going wrong in the first place.

You don’t want to be in a world where nothing breaks. That’s also wrong, right? That’s ignorant, and that’s egotistic. And you have to be human as a leader and as a manager and as an engineer. You can’t try to be perfect, and that’s what allows you to learn and grow and really build a ton of momentum. But don’t overlook everything that’s gone bad. There’s a lot. Everyone’s got a lot.

Shane Hastie: Tameem, a lot of really deep thoughts here. If people wanted to continue the conversation, where can they find you?

Tameem Hourani: Yes, of course. Always happy to chat. Would love to brainstorm. You can grab my LinkedIn. It’s on the description on the podcast, and feel free to reach out about anything. We’re always hiring good engineers. If you want to change it up, please let me know. But if we’re just brainstorming about anything, tech, culture, business, I’m here. Hit me up.

Shane Hastie: Thanks so much. It’s been a pleasure talking with you.

Tameem Hourani: Likewise, Shane. Thank you for having me.

Mentioned: