Startup Software Architecture – You Never Really Throw It Away: A Conversation with David Gudeman

Transcript

Michael Stiefel: Welcome to the Architect’s Podcast, where we discuss what it means to be an architect and how architects actually do their job. Today’s guest is David Gudeman, who is the co-founder and CTO of VelocityAI, a product that gives every sales rep a real-time AI assistant that can guide complex calls, surface the right insights instantly, and dramatically elevate performance in the moments that matter. He began his career at Actium Health, where he architected a hospital CRM backend from scratch, migrated the company to containerized infrastructure and built CI/CD Pipelines.

As the company grew, he stepped into product management. He then joined Tapity, a children’s science education platform where he built from the ground up their scalable interactive web product with real-time video. Live science lessons became replayable interactive episodes. From there, he joined Triumph Labs as a senior engineer where he led a major technical re-architecture, migrating the backend from JavaScript to TypeScript, rebuilding infrastructure on Google Cloud Run and moving from Firebase real-time DB to Firestore.

He later became engineering manager and helped shape Triumph’s product strategy. In 2023, David joined South Park Commons where he explored the intersection of AI, real-time communication and productivity, an exploration that eventually led to founding VelocityAI. It’s great to have you here on the podcast, and I’d like to start out by asking you, were you trained as an architect? How did you become an architect? It’s not something you decided one morning when you woke up and said, “Today I’m going to be an architect”.

How Did You Become An Architect? [02:20]

David Gudeman: Right. Well, thanks, Michael, for having me. It’s really a pleasure and I’m excited to speak on these topics. But no, I didn’t plan on doing this type of work. In fact, I didn’t even plan on being an engineer. When I graduated college, I had a degree in physical chemistry of all things. I was always interested in entrepreneurship, and I had read about Silicon Valley in the news, and I just decided to actually move without having a lot of experience or a lot of background in software to Palo Alto.

And there, I joined one of these hacker houses, and I met this 18-year-old kid who had what, at the time, I considered a relatively good paying job. And he skipped college and he had done one of these bootcamps. And I asked him, “Well, how long did it take?” He’s like, “Oh, three months”. I had just finished many, many years of schooling and I was like, “Yeah, he had an awesome job”. And so I said, “Well, without really investing, I’m just going to do exactly what he did”. And so I went and did the exact same bootcamp and sure enough, at the end of that intensive program, I ended up joining Actium.

They were just getting started and they were just down the street. They interviewed me, they hired me, and that’s actually what got me to being engineer number one at Actium. And I wanted to give that backstory because when I was thrown into making a lot of these very challenging technical decisions, it was quite overwhelming at the beginning because I really did not have a lot of experience or even a lot of the traditional foundations in order to make these kinds of decisions. So that colors a lot of the way I approach problems in the sense of when you’re tasked with making incredibly important decisions with very little information, you have to get up to speed on the basics and basically make a decision and live with it. Over time, I had done that over and over and people came to depend on me for that.

And so then I got more and more into that role where it started with, okay, are we doing AWS or are we doing Heroku? And I had experience with both in that bootcamp and I was like, “Well, this is supposed to be hospital software, right? So I don’t want to do Heroku. That’s like prescript kiddies or something”. I just knew that was like, I mean, not to use a derogatory term, but I’d known that AWS is for serious big platforms and then Heroku is for making a little LinkedIn clone for your projects. So that gives you some of the background on how I kind of got started in it, but yeah, I basically got tasked with making some of these decisions and I kept getting more and more tasks and then I just ended up becoming the go-to guy.

Decision Making Under Uncertainty [05:12]

Michael Stiefel: Well, it sounds like, and this may be sort of a theme I get by when I read your bio and listen to you talk right now, is that decision-making under uncertainty with not enough, in fact, you’ll never have enough information.

In fact, when I was in graduate school, we had this exercise, I remember this very clearly, where they had us calculate the cost of getting perfect information and it turned out the cost of perfect information was more expensive than just building the product itself. This leads us into talking about architecting for startups. You don’t have enough information. Resources are scarce. Time is scarce. So maybe one way to start out the conversation is based on your experience on a spectrum of one end where you have repeatable and generalizable startup experiences and at the other end, you have startup experiences are always specific to individual situations. Where do you think the truth lies? Does not the constraints of time, money, engineering capacity overwhelm everything else, or is this something that we could sort of extract out for people in general?

David Gudeman: Well, that’s an excellent question. And I think that is actually, at least in my experience, the issue of imperfect information and the reality in startups is that you’ll always be at a very pronounced deficit. And the reality is that actually is what defines the experience for the most part because you’re in that situation across multiple dimensions from the market standpoint, from the product standpoint, from the engineering standpoint, across every important dimension, you’re constantly having to make these decisions with very little information. Sometimes information isn’t knowable, right? If it was knowable, sometimes the big companies would do it. And so it’s kind of the nature of the game. And to your question, is it generalizable? Some parts are and some parts aren’t. I think the idea that you should acknowledge that and then make decisions where you know that you don’t have a lot of information and you want to plan around that fact that it’s very likely you will be wrong.

And so you don’t want to necessarily work yourself into a corner product-wise, engineering-wise, market-wise, especially in the market, you want to find that wedge, but a lot of times you’re revisiting a decision or you’re tweaking it. And so designing around where you aren’t betting the farm that this is the perfect decision, and that’s kind of critical, keeping all your options open. And I saw that repeatedly over time, that when teams are overly sure or they were not leaving room for changing, that was nearly always a mistake. For example, at Actium, the entire premise of what we were doing was a pop health approach. This was when Obamacare was coming out and the idea was that we were going to have these ACOs, accountable care organizations, and there was going to be value-based care, which is different than fee-for-service. Our system is generally fee-for-service. And so there’s going to be this push towards pop health.

The important thing was going to be the next best action. That was the NBA. The entire company was founded on this idea about each patient was going to have a next best action. And that’s a cool idea, and we built a lot around that, but in certain situations, it just really didn’t fit what our customers really needed. And so we were getting into these awkward situations where the problem just didn’t fit into the model. And so I think it was a mistake to really overly fit the company to this as opposed to having a system that was a little bit more flexible, where you base the primitives off what exists in the hospital, and then you layer on top of it, let’s say, a next best action when it makes sense. We were getting into situations where it was discharge care from the emergency department, ED, and Next Best Action just didn’t fit into that because it was like they had a prescribed system that they wanted to codify in our software.

They needed them to be reached out to by the nurse and all this sort of stuff, and that just didn’t fit in. It wasn’t like this AI-driven platform where we were deciding what was best for the patient, which is what pop health is kind of about. You’re supposed to say, “Okay, this person needs a breast cancer screening. This person needs to have their annual visit”. So it was just this mismatch. And the original premise of the idea was this NBA, but I think it was like we were really hobbled by overly fixating on that. And that went down all the way down to the architecture. We were getting to all sorts of weird problems where I got the product requirements that specified there’s going to be NBAs and there’s going to be calls being made from this call center. And then next we’re deploying this into the ED department of a hospital.

And it was like, okay, nothing makes any sense anymore.

Risk Based Software Development [10:44]

Michael Stiefel: So that raises a question in my mind, and I’m perhaps seeing an analogy here, but when you talk about next best action, what I think about is in software development or in this situation where you’re describing here, what you really have to do is look at every point and what thing or what piece of information poses the greatest risk to you at any particular time. And that’s the thing that you have to address. For example, I don’t know if you’re familiar with Barry Boehm’s spiral model of software development. It was the rage for a while. Software architecture goes through these phases from time to time, but the idea behind that was, you look at where you are, the software you have, and where you think you need to go, and you look at what the greatest risk you have at that particular point.

And that might lead you to prototype, that may lead you to do a database analysis, but it’s risk that drives the process. And do you think that’s sort of a better way of looking at software development at a startup?

David Gudeman: Yeah, I think it’s a good approach. And I think the issue is, and I’ve seen a lot of these kinds of frameworks come and go, right? And the reason why they come and go is because this problem is really challenging and there isn’t necessarily a one size fits all sort of solution. A lot of times there is so much variability within each startup. And that’s why people are always trying to develop these frameworks because everybody recognizes that it is quite challenging. And I guess the risk approach, to answer your question, to think about it, is like starting with what you have and then kind of forecasting into the future. And I think that does make sense. It depends a lot on the team. If the people interfacing the biz dev people who oftentimes are collecting the product requirements are very limited technically, it then becomes on you to try to understand the requirements and translate it into a flexible architecture.

Architecture Straddles Product and Engineering [12:54]

That’s why I’ve always kind of straddled both product and engineering because when I’ve operated in these startups, a lot of times you are doing a lot of product work. It’s not just architecture. It bleeds into architecture because a lot of times the product ideas and how you kind of translate the customer’s needs into, “Well, how are we going to productize this? “, heavily determines the architecture. In the case of Actium, it probably would have made sense for me to be more involved earlier on to identify like, “Hey, all that NBA stuff we were discussing, it doesn’t necessarily fit. We should have an entirely different framework”. One of the things that was really problematic in that situation is with NBAs and pop health, the entire premise is that there’s many different things that the patient could do and we want to prioritize them and list them.

Okay? That’s a completely separate problem than the hospital’s legally bound to do X, Y, Z after they’re discharged. We don’t control that, they control that. And so explaining that to non-technical people, it doesn’t really land. The distinction isn’t clear, right? And so that is where the challenge actually is.

Michael Stiefel: Yes. What you’re describing is what is often referred to by Gregory Hohpe as the elevator architect. In other words, the architect has to be the person to explain technology to the business people and the business to the developers and to stick the architect, he likes to say you go from the boiler room to the board of directors, to the executive suite. And what you’re saying is that this applies just as much to a startup and the architect has to be involved with the product people almost from the beginning because otherwise they will make technological assumptions that are just not realistic.

David Gudeman: Or a lot of times in what I’ve seen, because again, I operated in a role as a pure product person for about a year. And one of the lessons I learned from that experience is so when you’re technical and you’re talking amongst your peers, it’s understood people should bring the problem to you, not necessarily the solution. With non-technical people, a lot of times the only language they have is solutioning. So there was a learning curve for me there where I had to not get frustrated because a lot of times they say, “Hey, can you just change this thing?” And I’d be like, “Why do you want it to operate that way?” And it was because they couldn’t step back, take a holistic view, understand the problem from a 30,000-foot view, and then explain that to me. They only know that this thing is causing them problems and they need it to change.

You can’t really expect non-technical people and business people to have that kind of holistic understanding. They’re going to speak to you in solutions and you have to stitch that together, step back, kind of explain it back to them, make sure that that makes sense. And a lot of times they’ll have an aha moment. They’ll be listening to you. They’re like, “Oh yeah, that is kind of what’s going on”. In the case of Actium in the situation where we have this pop health product and it was being kind of forced into this emergency department situation, all they know is that the report doesn’t look right, the interface isn’t quite … It doesn’t fit in with like, “Hey, how do I schedule this call?” And so when I explain it like, “Hey, this entire system is built around completely different assumptions”, then they’ll kind of come along.

And that was actually a really important quality of that role in product. And I guess in some sense I should be clear, I was a technical product manager. I was kind of this architect/understanding the requirements, translating it, building the tickets. And so is that architecture? I don’t really know. It certainly felt like it. It felt like I was doing a lot of the same work I was doing when I was designing these scalable systems.

The Interaction Of Requirements and Financing [17:02]

Michael Stiefel: Well, to me, that’s a perfect architectural role because if you cannot, and this is a whole other discussion perhaps another time, is actually talking about requirements analysis. I like to say that in my software career, which I’ve had for maybe 30, 35 years, I’ve only done three things. Insert levels of indirection, trade off space and time, and try to get my clients to tell me what they really want.

David Gudeman: To me, that’s the hardest. If you’re in a big company, the relationship is mature, there’s a well-defined market, you know where you fit in, those conversations can be facilitated by that environment. When you’re kind of trying to figure out if there is something there, is this even feasible? And what was very problematic and what I’ve seen over and over is there’ll be a lot of context, there’ll be a lot of conversations before I even get involved. And by the time it gets to me, a lot of times it is kind of a mess. And then from that standpoint, there’s a lot of momentum, right? The deal’s going to get done. The deal’s going to get done. Promises have been made. One of the most important clients at Actium was a large health system out of Arizona. It was 400 beds and we were going to be reaching out to patients.

And so part of that was selecting the marketing automation platform that they were going to use, integrating it to our existing CRM and developing their reports, all that sort of stuff. This was one of the most important projects I was on as a PM and I had to do that kind of on the fly because the deal, they just cut a contract with them. And I looked through the SOW and in the SOW, there was picnic management for the employees. It wasn’t discussed, I just happened to read it. Now, that was a very frustrating situation for me because to them, maybe it’s a thing, you just throw it in there, but to me, I don’t have any idea. The implications could be profound on the product. And so ultimately, thank God it was one of these things where they just put it in the contract, it never gets done, nobody talks about it.

And that happens a lot. That happens a lot where in order to get consensus on their side, they have to make a promise. The thing is that Salesforce, we were displacing Salesforce at this hospital and Salesforce is such a flexible platform that when you go to deployments, you will see crazy things like that. There’ll be picnic management for the employees at the hospital. And it’s like, yeah, they can do it. Salesforce has that level of configuration ability. For us, we can’t do that. And it wouldn’t make sense to do it and we don’t want to do it, but there was a lot of tension there because sales was just trying to get the deal done.

Michael Stiefel: They had a checkbox that had to be checked.

David Gudeman: They just did it. They didn’t talk to anybody. They just did it.

Michael Stiefel: Well, I think based on what you’re saying, one of the most important things at a startup is for the architect to be involved as close to the beginning as possible.

Architecture as a Collaborative Effort With Business [20:21]

David Gudeman: That’s how I think it should be done, but let me put it to you this way. Again, to give some grace to the other parts of the organization is the number one thing is to make the deal done, get the deals and get the revenue. That’s number one. Now, if the idea is good, the team’s well organized, there’s good communication, there’s healthy, collaborative atmosphere. Yeah, the architect should be invited, should be listening in, should be providing feedback after the call, advising, “Hey”, what would’ve preferred to happen is taking the salespeople aside, say, “Hey, this caveat could potentially be a major issue. This could drag us into all sorts of design problems. And how important is this? Do we have to include this in the SOW, the statement of work?” So I guess in my experience, the architect was a very strategic role. It wasn’t necessarily just a technical role.

There are a lot of technical qualities to it, and that is interesting and challenging on its own. When I was at Tapity, they had this kind of edutainment system and it wasn’t parameterized. Every time they made an episode, it was a bunch of handcrafted code. When I came in and I recognized, I was like, “Hey, why don’t we just take a step back here? Let’s identify five or six common primitives that you want to do. At this point in the episode, you want to present the child two images for them to select”. That’s one node. What I ended up designing was kind of a graph system with edges and nodes and each node had a type. The type would basically imply a certain type of client side code where you present the user with a single image for them to interact with, two images for them to select from.

It could be a poll. All the children would select images and it would have a real time poll. And so I worked with the founder to determine like, “Hey, let’s just pick five or six of these and I’ll build a system where each episode you can just select the notes and build this graph, upload the images, you can select the images and now you don’t have to write any code. You’re just publishing. You have a publishing platform”. That’s half product, half architecture. I had in my mind, this graph, the admin tool, how are we going to upload the images? How are the images going to be fetched on the client? And I gathered that from observing what the product kind of was as is and how can we really productize this. Is that architecture?

Michael Stiefel: I think what you’re saying is very, very important. This again, it reinforces the idea that the architect is just as much about the product as the technical side of things. And I was also thinking when you were talking about these situations where they throw these things into contracts and they maybe provide difficulty, at least even if they don’t listen to you, they can’t say they weren’t warned. To me, the biggest problems come when you make a decision without knowing what you’re doing.

David Gudeman: Or not knowing the trade-offs.

Michael Stiefel: So I think, again, what you’re saying reinforces this thing and what you said with the last example as well, the architect, especially at a startup, is just as much about the product as is about the technology. And I think the other half is just as important, which we haven’t talked about, but I think is important as well, is the technical people have to understand what the business proposition is and don’t think, “Well, this is stupid. I’m not going to do a good job because, well, maybe picnic management really was the deal breaker”.

David Gudeman: Right, exactly. That’s what I was trying to say is that a lot of times when you’re downstream, you don’t know what took place earlier on. It might have been the case on that particular deal that the decision maker is in charge of this yearly picnic thing. It’s a headache, it’s been dumped on them, and they will not move off of Salesforce unless they can be guaranteed that their life won’t be hell in three, four months. To me, when I looked at the SOW, I was like, “This is completely insane”. That’s my first instinct. But as I’ve gotten more mature in my career, we should have had a conversation, but I can appreciate what might have happened there. And that’s actually part of the reason why I moved into product was I saw, at least at that company, the technical decisions were important, but they paled in comparison to guiding the process and making sure we weren’t designing a product that couldn’t fit into the emergency department.

We were not agreeing to things that had nothing to do with our core business proposition that potentially would be very expensive to develop or have a system that could accommodate picnic management is a drastically more flexible system. You’re reinventing Salesforce, you’re reinventing it. And the point is that we wanted a purpose-built Salesforce for a few vertical cases, the primary cases that were supposed to translate across many hospitals and picnic management was only at this particular customer, nobody else had picnic management. And so that’s where I saw a lot of the challenge at that company.

Business People and Architectural Evolution [26:15]

Michael Stiefel: So one of the other things that come to mind is that business people have no understanding of the difficulty level of a request. I’m in situations where they come and say, “We just want to make this little change”. But this little change, as you mentioned before, contradicts the basic assumptions of the product. So it’s a huge change. And the other times they come and say, “Well, this is probably so difficult and it’s going to take forever to do. ” And because it’s as simple really at the backend, the parameterization of something you’ve already done, you can get it done in no time.

David Gudeman: Right. Again, another very, very common situation, and it can be disheartening sometimes, but I’ve been in situations where I heard they were already communicating to the client, “This is not possible”, or, “This is a big lift”. And then I’m listening, I’m like, “Wait, no, guys, we can do that”.

Michael Stiefel: But don’t tell the client that.

David Gudeman: Right, right. Don’t tell the client that, but I’m thinking, don’t adopt a stance of defeatism here. And whereas other times they’ll be just talking to the client, I’ve been in meetings and they’re just telling, “Oh yeah, we can do this, we can do that”. And I’m having a heart attack because I’m thinking, “You guys, do you know how hard that is? ” Or earlier in my career, I used to really sweat this stuff. As I got older and more experienced, I realized that a lot of times you just need to let those conversations happen. Most of the time they go nowhere. Don’t get upset about it. Don’t create an issue when there isn’t one. Wait till it’s really close to getting done before you raise the issue, because those conversations are very fluid. So assuming we’re at that stage where something might get done and they are coming to you, I think again, there is no silver bullet.

This is, again, part of the challenge is that you can’t be everywhere all the time. You have to have good partners and you have to be able to trust them and you have to be able to communicate to them and you have to have that type of relationship. It’s very relationship oriented. You have to have that type of relationship or create a system in place where they will, in earnest, provide what they understand is going on. They’ll communicate that to you and there’ll be this dialogue going back and forth and it’s safe for them to communicate to you. You cannot just be shutting down people all the time. If they get too much resistance from you, they’ll stop communicating with you and they’ll just make the decision and force it on you. So you have to be receptive, you have to be open and you have to be collaborative and create an environment where people feel safe to come explore ideas with you.

And if they can explore ideas with you and you’re not dropping an anvil on their head every time they say something stupid, then that will open up the door for you to have that type of relationship. And I’ve seen a lot of engineers get very frustrated. They want there to be this clarity of thought that does not exist.

Embrace Ambiguity [29:21]

Michael Stiefel: I think what you’re saying is very interesting for two reasons. One, some of this stems from the basic ambiguity of the English language or for any language. And the second thing is that one of the things you have to be to be a good programmer is being anal-retentive because computers at the end of the day are stupid.

David Gudeman: They’re unforgiving.

Michael Stiefel: They’ll do exactly what you told them to do. So that leads to a mentality which is very good for the engineer, but is not very good for the architect or dealing with product people. It’s as if almost there’s no overlap in the skillset.

David Gudeman: Right. Well, they’re at odds, in fact.

Michael Stiefel: Fact. Yes. Yes. And every engineer that I know that has successfully moved to the role that you have or for many years when I was a consultant, you had to learn how to embrace ambiguity and accept it.

David Gudeman: And that’s critical. So in my capacity as an architect, it was a lot of product. It was a lot of human skills, actually. I ended up in a situation once at Actium where I had done this massive marketing automation integration. It was hugely expensive. And these marketing automation platforms have templating systems that are complicated, they’re mature, you can express conditional logic in them, all sorts of difficult. These are the reasons why they exist. And slowly, I saw us piecemeal rebuilding that requirement by requirement in a different area. And it was done not because they said, “Hey, I need a flexible templating system where these people in a different department can provision them. I can select them from a dropdown and I can use that to determine what type of emails our patients are getting”. That wasn’t the requirement. It started as, “Hey, I need to send this one piece of text.

Can you create a little dropdown for that? ” And then it was like, “Well, can you make a variable for the person’s name?” Then it was like, “Well, in certain situations we needed to show this text and then in other situations we need to show this text. So now we need conditional logic”. And before you know it, you’re rebuilding this insanely expensive platform, but we had already done marketing automation integration. In some sense, is that architecture? To me, I don’t know. It came through my … It was on my lap and I had to introduce that without any frustration. Well, you have that knowledge and nobody else has that knowledge. It’s easy to get frustrated because you’re thinking, why isn’t there any coordination? How is this not happening?

Acquire People Skills [32:13]

There’s no acknowledgement as the architect that you have this kind of authority or it’s your responsibility to identify these situations, but in many cases you are. And so you have to develop those people skills of, “Hey, I know you want this one little feature here, but we’ve done this massive integration over here. And yes, your workflow’s going to look different. The place that you provision the templates is going to be a different place. It might be an external system, but you’ll get all these benefits, all these things that you’ve never thought about, you’re going to have access to that you’re likely going to need”. Because it wasn’t necessarily marketing. The person who’s provisioning the templates was a practicing clinician, but the requirements were more or less the same. So just because it sounds like something they shouldn’t use, marketing automation, but it actually fits with what they needed.

And so the people skills, having the patience, letting people … And again, that’s another judgment call. Sometimes that little thing that they need is just a little thing and you don’t want to overbuild.

Michael Stiefel: This is why we still use humans and not AIs.

David Gudeman: Well, that’s another thing that … I mean, we don’t have to get into that, but sometimes when I see a kind of fear mongering and I listen to people like, “Oh my God, my job”. I think back on all these annoying situations I’ve been in, I’m like, “I can’t wait. Bring it on. AI’s going to do that stuff. I can’t wait”. Because that was very challenging.

Michael Stiefel: Especially when the AIs are very obsequious the way …

David Gudeman: They agree with everything. “Oh, that’s a great idea”. And that’s the funny part with some of these low code kind of vibe coding platforms like, “Okay, yeah, sure”. Half of my job was to really press them on the necessity of those requirements, who was benefiting, how much it was worth. A lot of times with the customer service department, they are very accommodating. They want to say yes to everything. And so a lot of times you find yourself being like Mr. No. And you have to facilitate those conversations where you’re listening to it, you’re giving them time to speak. You’re kind of like, “Well, what would happen if that didn’t exist? What would break down?”

Michael Stiefel: It’s almost like I’m using the Socratic method.

David Gudeman: You’re guiding them for them to agree. You cannot impose these decisions. It has to be consensus driven. And so you have to bring them along to your side.

So you have to say, “Okay, well, between these three things, which one is critical? Which one, if I were to delete tomorrow, would kill this customer?” And sometimes those conversations, they’re unpleasant.

Michael Stiefel: Right. But the best situation is where you can make the solution look like it was their idea.

David Gudeman: Right. And so is that part of architecture? I don’t know.

Will You Have to Throw Your Architecture Away? [35:05]

Michael Stiefel: I definitely think it is because at the end of the day, to me, the architect is sort of responsible for everything that doesn’t fit into somebody’s clear box. All the emerging properties, the ilities, scalability, flexibility, maintainability, which you can’t write a use case for. You can’t write a use case that says, “We shall have a secure system”. Those are the architect’s responsibility. I did a whole talk on this, which actually is on the InfoQ site. But anyway, I’d like to flip to a more global view. One of the things that occurred to me as you’re talking about all this is when you’re architecting for a startup, do you have to plan to throw away this initial architecture?

The Core Will Evolve and the Basic Stuff Will Not [35:56]

David Gudeman: It’s yes and no. And this is the challenge. You’ll never know which part you’re going to throw away. You never know which part you’re going to throw away. Certain things you can assume are going to be conserved as the product evolves, like user management, like onboarding stuff, which is why you kind of want to fast-forward through those things. Some of the very basic stuff, you want to just get right and get right out of the box and just move on and not have to revisit those things. But when you start getting into situations where it looks like this is going to be a very important part of the product, you want to think carefully about it and you want to design it in such a way where if you do have to revisit it, you’re not deleting the whole thing and starting over. That is a bad approach.

So I’ll give you an example where at Velocity, so when we first started, everything was centered around calls. It was a conversational intelligence platform and everything pivoted around Google Meet calls, Zoom calls, Microsoft Teams calls. And so I nested the AI agents under the call. Well, later we integrated with email. Well, now it doesn’t really fit very well. So either you’re overloading the call because I nested it or I’m revisiting that decision and I have to do major surgery. Now, that’s what I ended up doing, but it was very costly. Had I, with perfect information known that conversational intelligence overlaps quite a bit with email intelligence, I would have flattened it and put them side by side, the agents and the calls. But again, the cost of investigating that at the time was still probably not worth it. So I made a decision, I move forward, and you don’t have a lot of time.

So I got it working, we got people using it. That’s the major hurdle you’ve got to get over. The perfect architecture is subordinate to that. So I designed it in such a way where, okay, I didn’t have to delete it and start over. I had to do major surgery, and that’s really the best you can kind of hope for. I think the idea that build it assuming you’ll just throw the whole thing away, it’s almost like click bait because startups are exciting. People are always writing about it. They’re always talking about it. So some of these ideas sometimes get distorted and overly simplified to the point where they’re like, “Yeah, just write whatever you want and you’ll just throw it away and start over”. I’ve never seen it work. And if you’re in a situation where you’ve written some very important piece of software as a WordPress plugin, you’re in trouble, so don’t do that. And don’t think that that’s the right way. Just be wise and thoughtful about it without getting an analysis paralysis. Does that kind of make sense?

Aim for Surgery, Not Excision [39:04]

Michael Stiefel: It makes sense. And obviously you have to give this answer some level of abstraction because it’s going to depend, as you say, on the peculiar situation you’re in. But you said one very important thing is that the goal in the back of your mind should be, if I have to do something, it should be surgery, not excision.

David Gudeman: Right. That’s the rule of thumb, I would say.

Michael Stiefel: And what you’re also doing by having sort of rearrangement is that you’re probably also making a future migration easier because you’re not going to have to completely change everything.

David Gudeman: Right.

Michael Stiefel: If you have to migrate, you can think of, “Okay, first we’ll change the data model and then we’ll change the API”. And then in other words, you’re not doing everything all at once.

You Almost Never Really Throw Working Software Away [40:01]

David Gudeman: Right. So I was part of a one and a half year failed migration at Actium. And that’s what informs my belief is that people tend to underestimate when you write a lot of software, even if it feels throwaway, a lot of times it’s really challenging to actually throw it away, unless it’s like one of these situations where the product gets no utilization and it was just a failed effort, which is not what we’re talking about, I don’t think. We’re talking about when you’re getting utilization, it’s starting to get traction. It’s important, but now you’re in a ball of mud. I think the best way to do it is you can probably at some level avoid throwing away if you use, and this might be helpful advice, is if you use commonly available popular technologies and you compose them to build a solution. And I’ve seen this.

One of the reasons why I’m saying it like this is because sometimes with non-technical founders, they’ll take what they have. They’ll build a solution on top of WordPress or they’ll have a landing page and slowly morph it into a product. And in those situations, you have to rebuild something and then you might find yourself rebuilding major pieces of infrastructure when it’s hugely hard to do, but you have to do it because you have to get off of this other platform. But the challenge, of course, is sometimes it makes sense to piggyback on a mature platform. And then again, you come back to their product requirements where it’s the vision for the company. I’ll give you a specific example. At the very beginning of Actium, there was a decision either to build on top of Salesforce, but then you’re kind of beholden to them or rebuild it from scratch.

Given what I saw later, it probably would’ve made more sense to just actually build it as a Salesforce plugin in some sense because it also, there’s a lot of work that was done by Salesforce modeling complex business processes that we ended up rebuilding from scratch. Earlier on, the conversation might’ve been, “Are we going to have to throw this away and rebuild it from scratch because they’re going to price us out because we have a price floor”. Now, if you’re making an informed strategic choice like that, you’re probably in a good position. Whether or not it’s the right choice or not, you’re probably on track, you shouldn’t worry too much about it. The cases when they have no context and you’re piling on something and you don’t really know what you’re giving up, right?

Client and Production Feedback Into Architecture [42:28]

Michael Stiefel: There’s also a feedback loop from production back into architecture, the mature companies, but certainly in startups where not only you have production problems, but you find out things from users, how do you facilitate that feedback from production back into architecture?

David Gudeman: That’s a very challenging part because the reality is that the traditional solution is you instrument the product, you build all sorts of analytics. That’s what you’re supposed to do. Now, nobody has time for that. If you have a technical founder and they can instrument products, you’re at a huge advantage. Most of the time, most people can’t even get their head around something like that. So a lot of times you just have to kind of do forensics looking at database records and seeing what’s actually being utilized, what type of information’s in there, and just estimating based on what’s in the database. And then going back and talking to these various stakeholders and just saying, “Hey, is this what’s going on? Tell me”. And then hearing, “Why is this not being utilized? Why is this being utilized? Why is this type of information in this field?”

You’ll see things that it’s clearly being overloaded. They’ll be overloading at some task system and it’ll be utilized as the issue escalation or something. Again, you’ve touched on a very challenging part and given infinite resources, yes, you should instrument the product with an analytic system and then take the time to build the dashboards, but most of the time you don’t have time for that. So you’re just doing forensics on the database.

The Importance of Error Handling [44:05]

Michael Stiefel: Yes. And also the problem I’ve always had is to get the engineers to understand the importance of error handling

David Gudeman: Oh yes, and capturing the errors and storing the errors in a place where you can easily find them.

Michael Stiefel: And the disconnect between where the error is detected and what the error actually means.

David Gudeman: Right, right. And so the way I’ve done this, and this is moving to more of a technical solution, is that anything that is stateful has an error field, an error reason, and you stuff it in there, and the code always has to have that. If anything is stateful and it doesn’t have it, you’re setting yourself for a problem. And I’ve coached junior devs on that. I’ll say, “Hey, create a design”, and they’ll only be thinking about the success path, the happy path. First of all, everything should be stateful. That’s another design pattern that I found very advantageous is if any process goes through stages where it’s potentially created, then it’s loading, then it’s finished, you will have a status enum and one of those better be error and you better have an error reason. And you just have to have hygiene around that design pattern and you just have to always implement it and there is no other solution.

And sometimes you’ll see these products like, “We’ll catch all your errors like Datadog”, and it’s like that stuff never works. And when you’re small, that stuff is way too complicated, it’s way too hard. And I wish there was a more satisfying solution, but no, you just have to implement it.

Michael Stiefel: Yeah. I mean, some programmers think that all you have to do is throw an error and it’ll be caught by the great catch handler in the sky.

David Gudeman: No, no. You’ll never know what’s going wrong because you test it locally, it works fine. And then the user’s doing something weird or I don’t know what’s going on and you have to have error status. And then the unfortunate part is a lot of times you have to build UI around that.

One of the design approaches that I built is that there is no separate admin panel. Every feature you build has like a flag and then you can just see what happened. Now, that’s just from trial and error and seeing … Because you think, “Well, we’ll have an admin panel”. And then you’re maintaining two separate code bases and it’s a nightmare and you have two engineers, three engineers, and it’s like, “Okay, this is not going to work”. And so you’re left with just like basically everything has to have an admin view.

Advice For People Thinking About Doing a Startup [46:37]

Michael Stiefel: We just talked about the great catch handler in the sky. If there was somebody who came to you and said, “I’m thinking of doing a startup. I’m an architect. I have my friends who have some product ideas”, what advice would you give that person?

David Gudeman: I would say work very closely with them to model the problem before you write any code because code is expensive, cycles spent on the wrong direction are very expensive and where you want to spend most of your energy is with the business people, making sure that you’re focused on the most important part of the problem that will get traction and set you guys up for success. So spend less time writing code, more time talking.

Michael Stiefel: Yes. Measure twice, cut once.

David Gudeman: Exactly right. And one thing that always stuck in my mind, the CTO I worked with at Actium, the best code is no code.

Michael Stiefel: Yes.

David Gudeman: So as somebody who writes a lot of code, take it to heart that you want to spend time building the one thing that people really want and you want to model it correctly and it should be front loaded with a lot of conversation, a lot of modeling like you’re sitting down with pen and paper, talking through that like, is this what’s going on? How many states does this have? Understand the business process. How is it being solved today? Understand every step from beginning to end. A lot of times you’ll be introduced in the middle without understanding how people review this and how did we get here? Is there some sort of configuration that is being done? And that’s where you get into tricky situations where after you started writing code, they tell you there’s this whole other version of whatever you’re building.

So you want to understand beginning to end before you write any code.

Michael Stiefel:I think that’s very important because there’s a certain tendency, and I’m sure you’ve seen it, and I’ve certainly encountered people like this in the startups that I have consulted for is, we just don’t have the time. Yes, time is constrained, but it’s also a question of using your time wisely.

David Gudeman: Right.

The Architect’s Questionnaire [49:05]

Michael Stiefel: So what I’d like now to do is get to the architect’s questionnaire, which I ask everybody who appears on the podcast. It gives you a sort of more human view of architecture.

What is your favorite part of being an architect?

David Gudeman: It really is, for me, modeling the problem thoroughly and having it successfully solve the problem in an elegant way. If you can really achieve that, that’s a very, very satisfying outcome.

Michael Stiefel: What is your least favorite part of being an architect?

David Gudeman: Thinking about writing tests and things like that, trying to make it right. The creative exploratory part is the fun part. The shoring it up, making it robust, making it safe for updating later. That stuff is always just kind of a chore. It’s like washing the dishes after cooking.

Michael Stiefel: You have to do it.

David Gudeman: You have to do it at some point.

Michael Stiefel: Yes. Is there anything creatively, spiritually, or emotionally about architecture or being an architect?

David Gudeman: I guess each person it’s different, but I would say there is something spiritual about it in the sense that it is an art form and you are expressing yourself and a lot of the decisions are not cut and dry. And so you’re invested in it. And so I guess there’s some sense of spirituality there. I mean, not in maybe the traditional sense, but there is some zen to it when done correctly. Emotional, of course, you don’t want to be too emotional about it because then you’ll make the wrong decision, but you want to be passionate about getting it right and doing a good job. So definitely there’s some spiritual, some zen, some emotion involved. Sure.

Michael Stiefel: What turns you off about architecture or being an architect?

David Gudeman: Well, there is some fatigue sometimes when you’re doing what feels like the same thing over and over. If you’re designing a very similar system over and over, that can get tedious, it can get kind of boring.

Sometimes, at least in startups, you design this amazing cathedral, this big, you make all these efforts, and then it doesn’t get utilized. That can be demoralizing. That’s more a commentary on startups and less about the specific role of being an architect. But I would say it can be tedious sometimes. There are a lot of common design patterns that you find yourself doing over and over and over, and yeah, that can be kind of tedious.

Michael Stiefel: Do you have any favorite technologies?

David Gudeman: Oh, yes. I mean, for the type of work that I do, I’m a huge fan of Firestore. I spoke about this in my presentation at QCon, but it does so much for you. It does permissioning. It does real-time updates. It has an entire dashboard where you can go in and spot check the data. It has excellent guarantees around transactions. It solves so many problems for you and you can just be focused on shipping product. To me, technologies that allow you to do that are incredible and I really have great appreciation for it.

Michael Stiefel: What about architecture do you love?

David Gudeman: Well, it is intellectually challenging. I’ve always enjoyed intellectual stimulation. I’m intellectually curious. I’m always reading about new technologies, seeing where the future’s going. I feel like there’s a lot of interesting developments in architecture. Things have evolved so much over my career from jQuery and imperative UI programming to Declarative React Typescript, single page apps, Firestore, Serverless, Docker.

I’ve seen all these evolutions and it’s fascinating. To me, that’s really what’s so interesting and makes me passionate about it.

Michael Stiefel: What about architecture do you hate?

David Gudeman: If you’re wrong, you’re screwed. That’s what I hate about it. It’s unforgiving. Let’s put it that way.

Michael Stiefel: Especially as a startup.

David Gudeman: Yeah, it’s unforgiving. It’s like a prickly romantic partner. When it’s great, it’s great. When it’s bad, it’s bad.

Michael Stiefel: What profession, other than being an architect, would you like to attempt?

David Gudeman: I’ve done a lot of architecture stuff, done a lot of software stuff, done product stuff. I’m not too interested in design, I’ll be honest.

Michael Stiefel: It can be outside of computers, whatever.

David Gudeman: Yeah. I really do what I love. I’ve interacted with professional investors. Sometimes I wonder, “Could I do that better?” So professional investing, it might be an interesting space.

Michael Stiefel: Do you ever see yourself not being an architect?

David Gudeman: Not necessarily. The type of work that I do, I really enjoy. I guess if I were to take a step back and be more of an advisor, that’d be kind of interesting if I’m retired or something. I don’t know.

Michael Stiefel: When a project is done, what do you like to hear from the clients or your team?

David Gudeman: From the clients, it’s not necessarily praise that you hear. The best is almost radio silence because it’s just working and they’re so invested in it, they just assume it should have always been that way and it’s so aligned with their workflow that it’s almost invisible. Complaining might be an indication. Sometimes there’s no way to avoid it because they’re using it so much and it’s such an important part of their workflow that they can’t help but complain. There’s too many clicks or this or that. You might get a little bit of praise right at the beginning because the delta between what they had before and what they have now is so nice. But typically what you want to hear in the long run is just quiet utilization. And to me, that’s the most satisfying, is that it works. They’re not looking to replace it. Think of it like Google Meet or something.

We assume no one’s talking about it. No one’s marveling at this amazing piece of technology that we can just go to a URL and just have real-time video communication. It’s an incredible feat of engineering. All the codecs and all the hardware … It’s like this amazing piece of technology. Everybody just assumes it’s like an airplane or something. Everyone just complains about “it takes too long to get on the airplane”, but it’s like, this is incredible.

Michael Stiefel: What about from the team?

David Gudeman: Architecture is a bit of a thankless job. People shouldn’t really expect too much praise, in my opinion. Obviously, people that are in the know, they’re going to appreciate the clarity of thought, the in-depth analysis, and the kind of elegance of the solution. I have had that from certain members of certain teams, but in general, the lack of complaints usually means that you’ve done a really, really good job.

All the assumptions, all the work that you went into is facilitating the rest of the productive process.

Michael Stiefel: Well, thank you very much for being on the podcast. I hope this will be of interest to people who are thinking about doing a startup because I know many people have that in the back of their mind, or just have a better appreciation of, this is not something they want to do.

David Gudeman: I will say that startup architecture, it’s challenging and there’s no way around it.

Michael Stiefel: But I hope that people will get an idea of what’s involved, how it’s similar to other pieces of architecture that get done and how it’s very different. Again, thank you very much for being on the podcast.

David Gudeman: Thank you for having me, Michael. I really appreciate it. And I had a good time speaking with you.

Michael Stiefel: It was my pleasure.

Mentioned: