Transcript
Olimpiu Pop: Hello, everybody. I’m Olimpiu Pop, an InfoQ editor, and I have in front of me, Asgaut and Ola, two of the speakers of QCon London this year. The topic they touched upon is whether programming needs to be married to continuous deployment, or vice versa. And obviously, then, we have more questions for them, and this is the opportunity to have it. Gentlemen, please introduce yourself.
Asgaut Mjølne Söderbom: Yes. Thank you very much. My name is Asgaut. I’ve been working as a developer for 19 years, holding various roles, primarily focused on programming. I’ve been interested in clean code and collaboration from the start. Over the last five, six, and seven years, we’ve been moving more towards working with culture across teams.
Ola Hast: Yes. And my name is Ola. I’ve been working as a developer for about 15 years now. I’ve held several different roles, including developer, tech lead, and architect, but I’ve also always been interested in the cultural side. Because if you want to have a good architecture, you have to have a good culture.
Asgaut Mjølne Söderbom: But we both love programming, that’s the essential part.
Olimpiu Pop: Well, one of the questions I asked you initially was, How do you convince your manager? Because, usually, when you’re talking about changes, that’s the question everybody has, and you were both smiling. And at some point, you started laughing and then, “Okay. The management is behind us. They are just supporting us”. And, I was talking to one of our common friends, Rustam, and he said, “That’s why I joined their company”. So I have to ask you, what’s your company like? How is it working?
Ola Hast: Our company is both owned and has 13 banks as customers, so they both own us and they’re our customers. So we can organise ourselves as a tech company. This initiative began in 2012, when our leadership decided that the company’s core philosophy should be based on the lean philosophy. Respect for individuals and a commitment to ongoing improvement. So, they have been on board the entire time. However, one of the reasons we are here today is that, back in 2016, leadership decided they wanted more.
So, they set a pretty much stretch goal that we called 24/30, which was 24 hours from idea to production and 30 minutes from finished code to production. Back in 2016, this was quite ambitious. Back then, we were a dual organisation with a development side and a massive apparatus for taking the material from development and moving it into production. Some of our listeners are probably aware of something called ITIL, which was how our organisation was organised. We had the people responsible for receiving the work we did in development, and then we had to document a lot of things. And I think back then, we had two releases per year. So going from that to 30 minutes from finished code to production was quite a leap.
Asgaut Mjølne Söderbom: I guess one of the most important parts here is that we’re not a bank, but we serve 13 different banks. We’re a tech company, and our leadership is invested in us. They are all tech professionals, all the way up to management, which significantly impacts how we work.
Olimpiu Pop: Okay. So let me try to frame this in one word. You have 13 banks as stakeholders, but also as owners of the company. And from that point on, you’re acting as their technology arm-
Asgaut Mjølne Söderbom: Yes. Those are-
Olimpiu Pop: … so that allows it to work like a technology company. Then-
Asgaut Mjølne Söderbom: Yes. We create all the software, and the software we make for the 13 banks is the same, just with similar changes in products. However, it’s the same software used by all the different banks. It’s called SpareBank 1. It’s like an alliance.
Switching from two releases per day to multiple releases per day in production [04:31]
Olimpiu Pop: Okay. And then, you moved from a more traditional banking ecosystem, where you’re based solely on ITIL. That meant more like a factory line working where you just did some stuff, then you provided it to the next piece in the chain and so on-
Ola Hast: Basic-
Olimpiu Pop: … to a more over-the-wall. You just throw it over the wall, and then you close your ears and you don’t care about what happens. Moving towards a leaner aspect that’s more in line with manufacturing, but you’ve set a very ambitious goal.
Ola Hast: Essentially, we simply increased our delivery cadence. So, we started delivering every month instead of every three months. And then, we started every week, and then we started every day. And as we increased the cadence, we could see where things started to hurt. And obviously, testing and documentation. I don’t think that when you have quarterly releases, anyone reads the entire documentation anyway. But when you have to produce and read the same documentation every week instead, then-
Asgaut Mjølne Söderbom: Or a day.
Ola Hast: … or day, then you realise that this is not sustainable. You have to do something different. And then, we used that as leverage to improve our process.
Olimpiu Pop: As you mentioned, you started in 2016. Right. Nine years down the road?
Olimpiu Pop: You still have five to go. How far along are you on touching that target? Because that’s only a dream come true in 24 hours, from ideation to completion… I know that you’re in Norway, and now the day is very long with a lot of light. However, that doesn’t mean it will allow you to work more and also have, as you said, half an hour to get the code into production once it’s finished. So it’s just… That isn’t just a goal, that’s a dream. How far along are you? Did you touch this?
Asgaut Mjølne Söderbom: That was achieved in 2016-
Ola Hast: Yes, yes. Essentially, by 2017 and 2018, we began transitioning from a virtual machine-based production environment to a Kubernetes-based one. We were there in 2018 and 2019, and we had automatic deployment, at least since 2016.
Asgaut Mjølne Söderbom: Yes. But this is the essential fun part because all the technology was in place, but now it’s the people and the culture. Even though you can move code into production every day, people still use the big branches. They may have worked in a branch for two weeks, so that was the way we worked at that time. So we had the technology, but now we have to start convincing people to work in other ways to make more minor things go more rapidly into production
Ola Hast: And that’s where the lean part comes in as well.
Asgaut Mjølne Söderbom: Yes.
Olimpiu Pop: One of the core principles of lean is continuous improvement, taking out the garbage. Looking at your system or your process and then strongly-
Asgaut Mjølne Söderbom: Yes. Yes. Yes.
Olimpiu Pop: It’s pretty nice, and it’s very relatable for most people.
Identifying areas for improvement by using the “waste” clock [07:39]
Asgaut Mjølne Söderbom: Yes. We have invented a special tool that our team created, which we call the waste clock, because everyone wastes a lot of time. You waste time, and we waste time; management wastes time, and everyone wastes time waiting. We wait for those who don’t attend meetings. We wait for slow software. And as developers, we wait even longer because we wait for things that are the same every time. When I started at SpareBank 1, we had a Maven build that took around two minutes, and I was used to working with Quarkus, which took a couple of seconds.
And this took two minutes, and then I asked, “We can’t work like this. It takes two minutes”, and then they just told me this is just how the world is. And then, the other guys are sitting there, watching how the world is and the different teams around the corner, and they are sitting with the same framework and everything, waiting for two minutes. And then, this just adds up, and the whole company wastes a lot of money. However, the cool thing is that when you transition into pair programming and start working together, perhaps the first time you review this build takes only two minutes, you grab some coffee, and it’s okay. But the next time it’s like, “Yes. We have to fix this. We can’t wait two minutes”. This is a positive consequence of working together: you reduce waste much faster.
Olimpiu Pop: That’s an interesting concept. How does that clock work? I mean-
Ola Hast: Oh, no. It’s not an actual clock. We just talked about it. It’s like a stopwatch. When we wait, we could have used the stopwatch, and every time we start waiting, we just start it, and then it all adds up during the day.
Asgaut Mjølne Söderbom: Yes.
Olimpiu Pop: That’s a concept that you have in terms of a focus point, in terms of introspection and examining how things can be done faster.
Asgaut Mjølne Söderbom: Yes. And we start this watch all the time. As we did yesterday, we had to wait for another team to approve a pull request. We hate pull requests, but that’s a problem when other teams are doing stuff. And then, we sit there for hours. We have to wait, and then the waste clock starts again.
Olimpiu Pop: Okay. That’s usually a challenge when you have interdependencies on others.
Ola Hast: And it’s the same thing. Yes. When we look back at the way we worked before, when we had several releases a year, and then you had to wait for someone to deploy it before you got feedback. And then, you don’t know. Everything is in limbo for a long time, and then you can start working on something else. But then, when the thing that you stopped working on three months ago goes into production, you have to drop everything else to fix bugs that you created three or six months ago.
Asgaut Mjølne Söderbom: And yes, this moves us into the part with feedback and the waiting, and we are very impatient. We are feedback junkies, I guess. This is a focus that we have on everything. We need fast feedback on everything that we do, and that goes all the way from the start, where test-driven development is an excellent way of having quick feedback when we work to “We need a fast build on all the tests. We need a fast build on the CI server”, and so forth. This is a primary focus for us.
Olimpiu Pop: I like this line, feedback junkies. I think we can relate to that. But what… The example that you provided, Asgaut, was feedback in the engineering space. But you know, and I know, everybody knows that the most important one is the customer feedback; it is those who are doing it. How do you incorporate that? I mean, if you go all the way to production, half an hour is speedy-
Asgaut Mjølne Söderbom: No, it’s faster. Now, it’s five minutes. It’s five minutes from push to main to production, it’s five minutes-
Olimpiu Pop: Well-
Asgaut Mjølne Söderbom: … and it has to be five minutes. If it’s more than five minutes, then it’s a problem. Because before launch, we can put code into production maybe five times. We create small things, put them into production, and sometimes behind feature toggles. And then, we move over and over. If it were half an hour, then we’d ruin the process.
Olimpiu Pop: Well, guys, there are still five years until 2030. By 2030, you will likely have a negative balance. You’ll first implement the things, and then you’ll get ideas or something similar.
Asgaut Mjølne Söderbom: Yes. I think if we were at “30 minutes to production” today, we would be caffeine junkies.
Olimpiu Pop: That’s the beauty of technology, because it makes very complex things seem simple, or at least makes them seem so naturally. However, there are many factors involved in all these points. As you mentioned, culture is the most important one because that’s where you have the most friction. Then, you mentioned feature flags, which push things into the platform space, and you’re talking about a lot of tooling that will ensure the proper quality of the code we write.
It’s about testing and feature flags that are likely built directly into the platform space. You then have the operational aspect. This is an iceberg, encompassing everything in its entirety. So how did the company approach it? How did you approach it? Was it like, “This is the destination. You had a target operating model and then you just did it”, or did it just go incrementally step-wise? You just look at one point, then take a step, look at what happened, and then just improve it?
Asgaut Mjølne Söderbom: Yes.
DevOps adoption facilitated by an enablement team [12:47]
Ola Hast: Yes. Back in 2016, when we started with the 24/30, it was our first foray into what I would say is the DevOps space. I wouldn’t know it was the correct way to enter DevOps. However, we created a new team called Optimus, which was responsible for increasing the speed. There were people from development and operations who collaborated to expedite deployment and ensure that everything occurred safely and in the correct sequence.
And there’s another crucial aspect that also touches on culture. Because if you want to change how often you can deploy to production and all that, you need to have IT operations on board as well. Because, especially in IT-led environments, change is often perceived as dangerous. And if you continue to introduce changes to the people responsible for maintaining your systems without explanation or without getting them on board with your thought process and including them, then they will push back.
And then, you’re going to encounter friction and, essentially, conflict. We started with that team and then automated the deployment phase. And then from there, we have gradually improved. We have switched the architecture a few times. Gone from, as I said earlier, from virtual machine-based pet servers to Kubernetes on-prem. And now, we have everything in the cloud on Kubernetes and-
Asgaut Mjølne Söderbom: Yes. We have a fantastic team now working on a platform with AWS. And they are always trying to improve the speed at which things are being deployed to production, ensure everything is secure, and so forth.
Ola Hast: Because even though, as we said earlier, we are a technology company at the end of the line, our systems touch on people’s money, so our systems and our processes have to be secure. We can’t skimp on security or do things half-baked. We have to do it properly. Yes.
Asgaut Mjølne Söderbom: But at the same time, it’s cool to see if you see the whole circle. When we put things into production, we first sit together, and we work together on everything. No task is too small for working together. We work together on absolutely all tasks. Then, we do test-driven development on everything. And as long as we have something that can go into production, it will be sent to the CI server, which builds the main builds. This is the positive consequence that we saw from working together. We are the only ones working on our code bases in our team. And we work together on everything, so why should you build branches on the CI server? It’s just a waste of time, money and energy. We stopped building branches on the CI server and now only make the main branch. And then, if that’s screened, that goes automatically into production.
And then, we have feature flags sometimes. Sometimes, it’d just go directly out, but it’s the small pieces that touch productions all the time that we are… Very, very safe when it goes out. Additionally, we have alarms in place to notify us if something goes wrong. We receive alarms in Slack, and on top of that, we have a team that monitors all bank applications continuously. So we’re very safe, and we’ve been working this way for now three years. This part that we are working on is like the heart of the bank. It’s concerning accounts. In Norway, it’s 1.2 million users and 35 million requests per day. So, if we do something wrong, then we will be in the newspaper. And still, we’re constantly putting things into production with a resting heart rate, and the entire cycle is characterised by fast feedback. It’s a scientific approach to working, and it works brilliantly. Everyone should do it.
Olimpiu Pop: That’s the angle, right? That’s the dot on the horizon for most of us, or at least, that’s what I hope. Let me rewind a bit. Ola mentioned earlier that one of the drivers of change within the company was Optimus, when you said you’re with SpareBank 1. So, it was probably Optimus Prime who just kept the number one in the name. So that’s more or less an enablement team, right?
Ola Hast: Yes.
Olimpiu Pop: Their focus is on driving change, ensuring tasks are completed correctly, and identifying items used across the company. Their main goal was to make things fast and secure. Then, the journey that you mentioned was virtual machines. Like many banks, these days, are moving towards Kubernetes on-premises and then into the cloud space. Then, you mentioned that you have the threshold. Currently, it’s at five minutes. Then, Asgaut mentioned that you’re doing a lot of test-driven development and a lot of pair programming. But what I want to ask you is, how do you do the rest of the testing? You have 13 clients, right? I would expect that it is a product, but each of the 13 banks probably has a different level of customisation.
Ola Hast: No. They have very few differences between them. They have the same branding. They offer similar products. They have a few different things here and there, but that’s usually implemented as their own microservices.
Asgaut Mjølne Söderbom: Yes. And they’re-
Olimpiu Pop: Then, you’re the lucky ones. You’re lucky, guys.
Ola Hast: Yes, I worked in a different company here in Norway where we had a spring beans-based configuration. Basically, each implementation of the application was different based on which bank it was, and we had XML that chose the correct spring bean based on the bank. It was painful.
Olimpiu Pop: Yes. I know, I know. I worked in banking or the finance industry. At that point, investment banking was affected during the MIFID 2 implementation. And that was painful because you had a complete stop at the beginning of January, and you had to implement a lot of stuff. People started customising things only a couple of months in advance, and we had multiple environments, making everything extremely complex. And then, you had a GA candidate, a couple of pilots, and everything was all over the place. That chaotic approach works, but it’s still obviously… Your simplicity is only something to be envious for.
Ola Hast: Yes. We also sometimes have deadlines, based on compliance or new regulations, and so on. And, when you have to work towards a deadline and perhaps can’t put something into production properly before that deadline, it becomes painful when you can’t work the way you want to. But sometimes, we have to because-
Developer joy is the essential metric [19:46]
Asgaut Mjølne Söderbom: The primary benefit of our approach is developer joy. It’s the learning. It’s the fast learning and developer joy. One of our primary goals is to encourage more people to work in this way, to have the same life and workdays that we do. We also undertake a significant amount of work across the company to attract more people in this direction.
Olimpiu Pop: Interestingly, you touch on that. What metric are you using? Because… And your stretch goal is comprehensive, encompassing different key results, objectives, and underlying points. And developer joy, it’s pretty essential.
Asgaut Mjølne Söderbom: Yes.
Olimpiu Pop: It’s hard to measure. Then, it’s also the empirical stuff that you can just say, “How much time did I need?” As you mentioned before, the Maven build, which previously took two minutes, is now something that completes in seconds. How do you orchestrate all this? Because you have two distinct, measurable aspects, you can track and observe their progress, identifying trends. Additionally, there are factors outside the boundaries of your company, such as developer satisfaction and the way things are done. How do you coordinate all this?
Asgaut Mjølne Söderbom: Yes. There’s a lot of stuff there. We like to measure things in our company. We have a service that is always available, asking people how they feel and how they feel about their teams, and how they like to work here. That measures how people enjoy their work. We have scientists and researchers on board for five or six years, I think, who have been researching in efficient teams. And they have been researching across numerous organisations in the Nordic region. They have Ericsson, they have Spotify, and they have many of the big companies that they can compare with as well. But they have been following us, and we have done something exciting together with them.
Because me and Ola, we go around the company and we talk about how we can work together and what make small things work the way we work. But it all comes down to you having to test it out yourself. The idea we came up with, in collaboration with SINTEF, was to have a fixed period of three or four weeks. And then, we have one team or several teams in a tribe or area that are going to test working together. And we set some prerequisites. One requirement was that everyone had to book at least two pair programming sessions per week in the calendar, as that was the standard procedure.
Additionally, we must rotate roles at least every 10 minutes during the pair programming session, with the navigator and driver switching positions. And then, everyone just tests this for three weeks. The researchers conduct surveys before and after. They do observations during the experiment or intervention, as we call it. And then, they do surveys a couple of months later as well. So then, they measure, “How many pair programming sessions did you have before the experiment? How many did you have after? How many did you have three months later?” Additionally, we obtain some data on this topic, as well as all the findings.
Ola Hast: Additionally, during the experiments, we hold weekly retrospectives for the teams involved. And what usually happens is that the problems within the team surface during this time. That sounded very dramatic, but stuff like, “We are too many in the team. We have too much responsibility. We have too many meetings”. All of these issues tend to surface during these experiments because forcing people to work together exposes the difficulties that arise from collaboration.
Asgaut Mjølne Söderbom: Yes. And this has been a success. We have had five different experiments now over the two last years, involving a lot of people around the company. And since the researchers they see a huge benefit here. This is an excellent way of introducing a new technique in a company without forcing it. Because everyone just have to test it for three or four weeks and then you can do whatever they want afterwards. So they’re actually writing a scientific article about this as well and had different presentations of it. They gave us some key findings that we get to say from this and that is, for example, more cross-functional collaboration, reduced waiting time, stronger culture of collaboration, higher quality feedback, improved wellbeing, increased psychological safety. It’s amazing. Then, we have something to measure.
Olimpiu Pop: Is the article available, or is it still in the making?
Asgaut Mjølne Söderbom: I think it’s still in progress.
Olimpiu Pop: Please send it over when you have it.
Asgaut Mjølne Söderbom: Yes, yes.
Olimpiu Pop: Great. So it seems that you have some help. You have people who are focused on observing people who are the referees or coaches from the outside. On one side, you have the enablement team that is focused and drives that change. And on the other side, you have a research company that is focused on observing, coming, and trying to put the priority in place.
Asgaut Mjølne Söderbom: Yes. We are the enablers, me and Ola. We are the ones who do all this stuff and then we just have the researchers that are researching. Your company can try it, everyone can try it. It’s really not harmful. It doesn’t affect the things that you’re planning to do in that period, so we actually made other companies try it out as well.
It’s harder to convince the developers than the management [25:06]
Olimpiu Pop: Well, continuous deployment is one part of the pill that is very hard to imagine for most people, even though it’s something that is the standard in the technology space. However, their programming is something that every manager will likely complain about when they’re simply stating that you have a workforce that is perhaps too crowded in terms of what they need to implement. And then, you just slosh that in half. How did the people react to this kind of change? First of all, the management, and then, secondly, the developers. Because developers are a unique breed, they have their own way of working. Headsets on the heads, doing the coding-
Ola Hast: It’s kind of interesting that you say that. Because in our experience, if you give management a good presentation on the benefits, they are not that difficult to convince. Maybe we’re lucky but they weren’t a problem. Developers are actually harder to convince. Yes.
Asgaut Mjølne Söderbom: Yes. That is why we do these experiments, for example. But to answer your question directly, if two people working together are faster than two people working in parallel, there are several factors at play, but one key aspect is focus. When Ola and I work together on a task for two hours, we focus solely on that task. We don’t check Slack, emails, or anything else. We just work. And we both come into the flow state, which is impressive. Everyone loves to be in a state of flow. So, you get to accomplish a lot in those two hours, and then you produce a measurably high rate of quality when you work together. You frequently discuss design, and this, combined with test-driven development, yields a high-quality output.
The second thing here is the rework. Because when we have done this, we don’t use any time on pull requests. We never use a second on pull requests. We simply create it and then approve it as part of our compliance process, but we never use pull requests. So that time is eliminated because we work together. And then, when the code goes further into production, it has such high quality that it’s very seldom that there are any problems with it. And we know that the problem that arises after something is finished, whether it’s just a bug in the test environment or the production environment, is really… They cost a lot of money. That bug when we’re two people, it’s just an hourly rate for two of us. But the bug in production can involve the whole company and-
Ola Hast: Yes. In the worst-case scenario, it can put all 13 banks in crisis mode, which is really, really, really expensive.
Asgaut Mjølne Söderbom: And I think the measurement of this is challenging to measure the rework after work, when you haven’t worked together, but we can see over time that it’s a measure that’d be much, much lower. We have very few bugs and we have high quality.
Olimpiu Pop: Okay. One of the key measurements of your success is the number of defects, which allows you to see exactly how many bugs you managed to find, whether during the coding session, TDD session, or when transitioning to production, as that’s the most expensive stage.
Asgaut Mjølne Söderbom: Yes. And everything here fits together. Because if we find a bug in production, our code has such high quality that we test during development. And the things we do with [inaudible 00:28:32], drag, and assign, and everything, it’s very easy just to go in, create a test that fails, fix the bug in the implementation. As long as that test goes green, we proceed directly into production. We trust our unit tests so much that we don’t need to perform any end-to-end tests, for example.
Do you need to think about code coverage? [28:50]
Olimpiu Pop: What’s the target? What’s the coverage target that you have?
Asgaut Mjølne Söderbom: Coverage target-
Ola Hast: Coverage targets-
Asgaut Mjølne Söderbom: … never thought about.
Ola Hast: I think coverage targets are somewhat similar to lines of code targets. They don’t have a meaning in themselves. However, since we practice TDD, I believe most of the code within our domains has at least 90% coverage.
Asgaut Mjølne Söderbom: Yes. I would be surprised if it didn’t.
Ola Hast: And the same applies to external dependencies and integrations, which also have high coverage. Let’s say that if we integrate with a database, we create an adapter and then rewrite the integration tests on top of that, using test containers. A Docker-based database to run our tests against. So I guess our coverage target is high but we don’t say 100% or 90%. It’s just-
Olimpiu Pop: So you don’t have something that checks. You already have it boiled down in your culture and you know exactly what it stands for-
Ola Hast: Yes. Say, if I create some code that doesn’t have good enough test coverage, then Asgaut and the other guys in the team will look sternly at me and just-
Olimpiu Pop: Well, that’s a benefit of being co-located and being in the office. But when you’re looking into continuous integration, I would expect you to have some kind of fitness function, some kind of threshold, some kind of tools that are ensuring the quality because-
Ola Hast: No.
Olimpiu Pop: No? Nothing? So you’re just the two of you, coding-
Ola Hast: For-
Olimpiu Pop: … you’re happy?
Ola Hast: Yes, well, we do have stuff that checks for vulnerabilities and stuff like that, so-
Olimpiu Pop: But that’s a whole different thing-
Ola Hast: … varied tests.
Olimpiu Pop: Yes. That’s for sure. You have a unit test. What happens when continuous integration is running? You run the test. You have the unit that you wrote, as well as the entire package of everything. Then, you have probably vulnerability testing… Well, down the line, at some point, you check for dependencies and so on. What else? And is nothing being considered in terms of the sanity of your code, or the architectural perspectives?
Observing production allows you to identify potential issues [30:55]
Asgaut Mjølne Söderbom: I think that’s an important part that is… Since we are constantly putting so much content into production, the difference between what we have locally and what we have in production is very small all the time. It’s very similar, and it encompasses the entire circle, from local machines to CI, to production, and then back to us. That happens all the time, and that’s essentially the end-to-end tests.
Ola Hast: We have lots of metrics in production that tell us if something is wrong. If we have a slow database query or if we have done something that makes our JVMs do lots of garbage collection or we have memory leaks or so on. So we have lots of metrics there, but we don’t have anything in our pipeline that tells us that we are not following some prescribed architecture.
Asgaut Mjølne Söderbom: And this works brilliant-
Ola Hast: For several years. And we have super good control and high quality. We are several teams and the architecture is coherent, but that’s because it’s also baked into our culture.
Asgaut Mjølne Söderbom: The teams are very autonomous in how we work, and one of the main goals is also to avoid any handoffs to other teams. We can work as autonomously as we want, so we handle all the internal architecture, design, and everything. We see that one of the main problems with feedback and speed is the minute that we are connected to another team or we have to wait for another team.
Then, everything slows down immediately. We want to have higher team autonomy, as we believe… If we have to do something together with another team because sometimes we do, the last time we did it, two of the people from the other team were sitting with us in the period. And they were part of some of our meetings and we worked together with them closely in that period. And then, they went back to their team again.
Olimpiu Pop: Okay.
Asgaut Mjølne Söderbom: And we never accept pull requests from other teams, never. If you are in our company, you couldn’t do a pull request into our team. This is part of the team boundary. We said this several times to everyone in the company, “No one can do pull requests into our code in our team. If you want to do a change in the code base that we manage, then you have to come to us, talk to us. And if we agree, then we peer program it together”. People, they just deliver the big pull requests into the code base in our team and we didn’t have time to see it properly. And we just merged it in and we got a mess. So this is a highly much better way of working we think.
Olimpiu Pop: So that feels that you have more or less a micro culture where your teams are highly autonomous, probably even getting close to independence, like you are very responsible for everything that you’re doing. And in case somebody wants to do something, they have to come on board, work with somebody in the team that knows the way of doing this stuff, and that’s your safety net.
Asgaut Mjølne Söderbom: Yes.
Olimpiu Pop: How sensible is that to people who are on holiday, people leaving, or sick? Because it begs the question, like, “Okay. You don’t rely on tooling, you don’t allow people from outside, so everything is very team-dependent”.
Ola Hast: Yes. Since we work together a lot, we have a high level of resilience because everyone knows everything at all times, with four developers.
Asgaut Mjølne Söderbom: Yes, and we are four developers, product leads, and team leads. And we both have children who get sick, and they bring germs home that often make us ill. So we have a lot of experience with people being away. And since we work together on everything, it’s usually not a problem because there are generally more than two people present at all times, so it’s not a significant issue.
Ola Hast: No. We put small things in production all the time. So before a holiday, there’s nothing significant happening. We also have our computers with us, in case something happens, but it never does.
Asgaut Mjølne Söderbom: The technology stack in most of the teams is quite similar. It’s a Java framework called Spring Boot and so on. We strive to adhere to as many industry standards as possible, making onboarding, especially given our work approach, exceptionally easy. So that helps as well. If we need to bring in someone else, then they’re not completely in unfamiliar territory.
Pair programming makes onboarding as smooth as possible [35:29]
Ola Hast: We actually stress test after this, if I can tell you about that. Because the thing we have tested now this spring is something we call job shadowing. So for example, if you wanted to start in our team, what we did, we took a developer from another team and he visited us for a week. And then, there was no preparation, just some communication between his team lead and our team lead. It’s okay if he comes for a week and then you get a cinnamon bun when it comes to the first today. And then, we just started working together. And he is put in directly into a pair or a mob and then we switch every seven minutes who is typing, who is watching.
Before lunch, on the first day, he had a code in production. And then, it was just immediately rolled into the team. And what he said was that in his five-year-long career, he never had so much flow and focus that he had that week. We had another woman who had the same experience a couple of weeks later. And then, it was much better than settling into an external course or something. They can just come to us and work with us for a week.
And then, we saw that this onboarding is highly efficient when we work in the way we do. And this also changes if everyone works similarly to us, then the team mobility inside the company would be completely different because you can move around teams all the time. Earlier, it may have taken two months to learn how the team works and blah blah blah. Now, it immediately onboards, regardless of whether you’re a senior, junior, or whatever.
When not to use pair programming? [37:10]
Olimpiu Pop: Okay. We do test-driven development. We always do pair programming. But what are the exceptions? When are you solo programming?
Asgaut Mjølne Söderbom: Two days ago, when… Because we’re four people and, suddenly, you are alone because someone is in a meeting and one is sick. Ola and I had been sitting working together, and then, suddenly, I was alone. And what happened after about 30 minutes is that you feel like you have to make decisions on your own, which is weird and awkward when you’re used to pair programming. And then, you know, if you make a decision and then continue, you get these follow-up problems that I have to change when he comes back. So, the exceptions are as follows: when he is in a meeting, the other team members are sick, then you may have one to two hours of work alone, which is acceptable. Then, we just have some reviewing when he comes back.
Ola Hast: So usually if one is always away for the whole day, then we just do a mob of three or two if people are-
Asgaut Mjølne Söderbom: Yes. So that’s the thing now. Four people, if we’re four, we’re split into two. If we’re three, we always work in a mob of two, if we work in-
Olimpiu Pop: Okay. That was the next question. We’re looking at the other outlier because obviously, that’s one extreme where you’re just working solo. However, given the size of the team, I expect that when you encounter different challenges, you can simply group with the whole team.
Asgaut Mjølne Söderbom: Yes. And we work remotely as well. We do the same. It doesn’t matter if one is in the office and three are home. We’re working the same way all the time.
Olimpiu Pop: Did the size of the team change with the adoption at some point? Were the teams bigger at any given moment in time?
Ola Hast: We still have a lot of teams that are too big, unfortunately-
Asgaut Mjølne Söderbom: Yes. In the company, we have a model that’s similar to the Spotify model, if you’re familiar with that. You have areas, and within each area, you have teams. And then, the area can share resources such as team leaders and product leaders, and so on, which doesn’t make sense to have one per team. That gives you better flexibility in team size. Instead of having one team of 12 people with one team lead, you may have three or four teams of four people each, with two team leads, each overseeing two teams. This approach allows us to split into smaller teams. Because what we see is that once teams increase in size, their responsibility area usually increases as well, which makes it very difficult to prioritise. And then, you have too much work in progress on the team, and you’re no longer a team; you’re just a group of people working together.
Ola Hast: Yes. The team size, with a maximum of five developers, is fine. That’s what SINTEF found as well-
Asgaut Mjølne Söderbom: What the researchers-
Ola Hast: It can depend somewhat on what you’re creating and whether you need more cross-functional people.
Objective Key Results help the company move in the same direction [40:15]
Olimpiu Pop: Okay. We spoke a lot about things that usually serve as scaffolding, including aspects of culture and everything else. But in the end, as a software development technology company, your goal is to deliver functionality and impact for the organisation. And usually, agile stands… Or for a lot of people, it’s synonymous with Kanban, scrum, and whatever. How do you do that? I mean, how do you approach it? Do you have your way of looking at things? You’re taking more from the Spotify model, where you have a great deal of autonomy and focus on a particular topic. And then, you have people from marketing, as well as individuals from other parts of the organisation, to ensure that you’re just… Have everything to make sure that you can launch it?
Ola Hast: We use something called OKRs, or Objective Key Results, where there are other goals at different levels within the company. The goals always point upwards to the goal above it. So our team, let’s say that management says, “We want to focus on modernising our platform”, and then that cascades down. And for us, it means getting rid of the last VM-based servers we have. So that’s how we manage to have coherent goals throughout the organisation.
Asgaut Mjølne Söderbom: And our team lead and product owner is responsible for ensuring that we follow these OKRs within our team. We speak with her several times a week. However, within our team, we work with a model called Radical Focus. Have you heard of that? You have a Monday commitment where you simply state the tasks you’re going to accomplish this week, and then you have a Friday win where you review what you have done that week, essentially a celebration. And then, you work your ass through the week. And what we see now when we’re working with small tasks and putting things into production all the time, is that the task issue-tracking system becomes-
Ola Hast: Redundant.
Asgaut Mjølne Söderbom: Yes. We spent more time on moving tasks inside Jira from ‘in progress’ to ‘done’ than we did on actually creating new content. So we quit entirely with Jira some years ago. So on Monday, we just have some checklists of what we’re going to do during the week, which are the main tasks, and one essential part is that we never assign people to the functions. It’s the team that’s going to deliver the tasks and not specific people. And then, we just start working and work through the week. And then, we see after a week, we have a small retro and we have this Friday win. And then, we see what we have done and what we’re going to do the next week. And that’s the whole stuff. It’s back to the waste cut. We are reducing waste there as well.
Ola Hast: Because the problem is that many organization, when they implement an agile practice, they go to one of the big frameworks and then they, “Okay. Let’s start by implementing everything”. And then, it kind of just gets stuck there. But instead, we start with as little as possible and then we see how… We need a way to track minor things that we see in the code when we’re working. Okay. We have a whiteboard, so we just put the small tasks on the whiteboard instead. And then, as the week progresses and we finish one task, we just take the next task from the whiteboard instead of creating several issues in a tracking system. Yes.
Asgaut Mjølne Söderbom: Another consequence is that the status meetings are also reduced. Because everyone in the team, we work together and we’re sitting together, we know what we’re doing. So we don’t need knowledge sharing sessions. We don’t need stand-ups. We have two meetings a week, half an hour each with the product owner but not more than that. So we have very few meetings and, of course, developers love when you have a calendar with very few meetings. That’s also developer joy and we get to do more. So everything here just fits together. It just comes to consequences of major parts which is to work together and makes small things all the time.
Ola Hast: Yes. And this goes back to the leadership philosophy as well as respect for people and having trust in people. We’re all adults there and we’re trusted that we do the right thing. We do our job properly.
Asgaut Mjølne Söderbom: One way we managed to come to the level we are now in our company and be as modern and work together across teams is that we have one day every week, which is Thursday, where we just focus on professional development. You can use that to whatever you want as long as it’s improve your professional knowledge and that also means that everyone in the company are available that day. So if you want to work with someone in another team to try a proof of concept or test technology or test something, then they’re always available.
And then, that really builds culture in a good way. It’s like the 20% model that Google have. It’s just fixed to one day a week, which is very important. Then you can step out of your team, do something else, improve stuff, work with others, build culture across the company. And then, you go back to your team and you take this with you and that is one of the reason that we managed to come this far, I think.
Olimpiu Pop: The model was first implemented by Intel, and Andrew Grove brought it. However, you’re right, Google perfected it because Google was probably the largest and most streamlined company that used it, and they used it properly.
Asgaut Mjølne Söderbom: Yes. A small retro element is also essential that we incorporate… Because if you have a retrospective every week, you get to discuss things early, which is also part of the feedback loops. And you also use less time on retros. And most importantly, you remember what happened. That’s very important for the Friday wins: 10 minutes of retrospective.
Is there a place for product owners still? [46:06]
Olimpiu Pop: Okay. And you mentioned product owners. What’s the role in this whole ecosystem? You don’t have Jira, so nothing for them to do
Asgaut Mjølne Söderbom: We must also acknowledge that. Some people use Jira, while others love it, and they can love it. However, we are autonomous, so people can choose how they want to work and what they like to use. Not all teams in SpareBank1 work as we do. Some teams still have testers and prefer to have their own testers, and they don’t work together as much.
Ola Hast: But the role of the product leaders is that they obviously talk to the banks, and they usually are responsible for a product or a set of product or an area of the internet bank. Their job is to communicate to us what goal they want us to achieve. They don’t say how we should achieve that goal. And since we’re co-located and they’re around, we talk to them more than that because they’re here. We work together. We don’t necessarily need a formal meeting to talk. They come with the task but they don’t break it down into several Jira tasks. They say that-
Asgaut Mjølne Söderbom: Does anyone do that still?
Ola Hast: Probably.
Olimpiu Pop: Okay. For me, it sounds like you’re a very focused workshop where you have a lot of craftsmen. Everybody’s very focused on what they’re doing, and they are just looking at the problem. Let’s tackle it, and then whenever you need it, you can simply chat rather than hiding behind a cumbersome process that keeps you stuck between tasks and so on. Would that sum up-
Ola Hast: Areas as… We are somewhat independent in the area we work, as we handle everything related to accounts. We are, in a way, a platform team. However, other teams are working more directly on products, and they are collaborating more across teams to develop products that have a broader impact.
Asgaut Mjølne Söderbom: I guess the product leaders also work together on other-
Ola Hast: Yes. They work together to ensure that things are aligned, and I think they also do a lot of things that we don’t see. They shield us from the outside-
Asgaut Mjølne Söderbom: Boring stuff.
Ola Hast: Yes. They reduce a lot of noise for us so we can-
Asgaut Mjølne Söderbom: Yes. That’s nice-
Ola Hast: … work-
Olimpiu Pop: What’s next? You have five years. You already achieved a lot. What’s next?
Asgaut Mjølne Söderbom: You want to start with the
Ola Hast: We can start with the enjoyable part, which is EU regulations.
Olimpiu Pop: Moving on.
Ola Hast: Actually-
Olimpiu Pop: You’re part of the European economic space, so-
Ola Hast: Yes. I’m not going into politics, but we’re part of the EU. We just don’t have voting rights. But we have to… Now, in finance, we get something called the Digital Operations Resilience Act, which is actually replacing a lot of the compliance stuff we get from the government. So basically, what we’re doing now is looking at how we should work in the best possible way within that framework. And we have to become better at describing our processes, describing the pros of our processes, because being highly automated is actually a really good way of being compliant.
If everything is automated, then you can build compliance into your pipeline. So we have to describe that in a good way. That’s one thing. That’s more short-term. We also want to focus even more on building culture internally to get more people. If that means forcing them to work the way we work, then so be it. They shall be happy.
Asgaut Mjølne Söderbom: Yes. Because, as we see, this is an outcome of all the experiments. People having better days at work, that’s our biggest motivation. And we also think it’s a little bit funny that the most ancient technique in history, which is working together, is now coming up as something significant in IT, and we want to work, as we all are saying, internally to make this even better. Our company motto is better together, so we are back.
Is there any waste to still remove? [50:27]
Olimpiu Pop: Yes. That’s a nice place to wrap. So, you’ve worked up to now on building something that flows nicely within the company. Some teams are more involved than others, but you have a mechanism through which you cross-pollinate your information, and you’re helping others to achieve better flow. But now, it’s also the funny stuff that you need to do. And the amount of automation that you manage to put in place allows you to have snapshots in terms of regulation and compliance that usually require an audit or other stuff like that. But now, having those things automated allows you to do that out of the box.
Asgaut Mjølne Söderbom: Yes. That’s the focus we will continue with… A third thing we want to mention is, of course, AI. We just have to mention AI, of course. As we work together on everything, when we have Copilot inside IntelliJ, it suggests sentences and code as you type. When you’re sitting together, we’ve seen that this interrupts us more than it creates value, so we turn that off. However, we use the chat quite a lot with the Copilot.
Especially when it comes to third-party things that you don’t want to fill your head with anyway, like “How does the Spring life cycle work?”, and stuff like that. Then, we use Copilot, which is really good with things. So that’s where we are right now, I guess.
Ola Hast: Yes. And we haven’t looked that much at agents and stuff like that either. So I guess-
Asgaut Mjølne Söderbom: Oh, that’s coming.
Ola Hast: … we just need to figure out if it’s just hype or not.
Asgaut Mjølne Söderbom: Yes. We’re focusing on making code that is easy to change. That is one of the main goals when you make code. It should be easy to change for others, but it also should be easy to change for us and we see that when we come back to code maybe two years after. It’s important that we can change it easy as well. That’s a focus that we try to talk about in the company as well. Instead of talking about bad code and good code, we talk about, “Is the code easy to change?” Because if the code is easy to change, then you most likely have used the many good design principles and tester in development, everything to make this code in a changeable state. So that is some focus that we have.
Olimpiu Pop: You’re just focusing a lot on the outcome that you want and then you just have layered measures like that. Like, “Okay. Other developers or me tomorrow, it’s one of the stakeholders, so I want to make sure that that’s working”.
Asgaut Mjølne Söderbom: Yes. And that is… We are affected by Dave Fairley there, I guess, with all his talk around this code that is easy to change. So we try to have it with high cohesion and low coupling and abstraction and all of this stuff. But we see that if you’re doing pair programming and tester in development, the design comes as a consequence of working this way as well.
Ola Hast: So far, I don’t see that AI generating lots of code for us, giving us code that’s easy to change too-
Asgaut Mjølne Söderbom: No-
Ola Hast: … come back to that topic.
Asgaut Mjølne Söderbom: But as… We’re paying attention to it, following it and reading about it and testing it.
Olimpiu Pop: Well, you’re in a good place. Norway put a lot of money into researching AI, so you’re on the edge. If you’re happy with the outcome that you have without AI, I’m sure that it’s just one step away to get it on board. Thank you, gentlemen, for your time. It’s very insightful. Good luck in your journey and definitely, we’ll talk more to see where this takes you.
Asgaut Mjølne Söderbom: Thank you.
Mentioned:
.
From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.
