Transcript
Olimpiu Pop: Hello everybody. I’m Olimpiu Pop, an InfoQ editor, and today I have in front of me Brian Fox, CTO and co-founder of Sonatype. Hello, Brian. If you can introduce yourself, I know that you are very involved with open source, so I’ll allow you to choose the accolades that you want to bear, as you have plenty.
Brian Fox: Sure, sure. Thanks. Thanks for having me. So, yes, I’m Brian Fox, as you mentioned, co-founder and CTO at Sonatype. If you haven’t heard of Sonatype, Sonatype runs the Maven Central Repository. We’re also the producers of Nexus and a lot of other software that people tend to know. I got started in open source a long time ago.
Most of my open source street credentials come from the Maven project. I was a contributor early on. I was the PMC chair for several years. I’m still on the PMC, but I haven’t written any code there in a while. I work with the Apache Infrared to run some of that stuff. And more recently, I’ve been on the governing board of the Open Source Security Foundation, the OpenSSF, and I’m also on the board of the FINOS, Financial Open Source Foundation. And through those efforts in recent years, I have worked with various governments, contributing to some of the national cybersecurity drafts. Over the last couple of years, I have also been involved in cyber resiliency and CRA work in Europe. So that’s what I’ve been up to.
Is the EU Cyber Resiliency Act in its final form good for Open Source? [01:42]
Olimpiu Pop: Thank you for the help with the CRA. You were pretty involved with that. You are one of the people who pushed and rang the alarm when you felt that the open source in Europe, at least, is in danger. How about now? Are you happy with the outcome? I know that people felt that they listened, but how do you feel?
Brian Fox: That’s a Pandora’s box, isn’t it? So, for listeners who might not be fully aware, the CRA was attempting to impose obligations on open source, treating it like commercial software, which is generally directionally correct. Right. The problem, and this is something that I’ve been pushing on for years and years, is that organisations use open source software, but they don’t track it. They don’t know what’s going on. So when something like Log4j, they have no idea what’s inside their software, and there’s a collective freak-out. And this is a problem that I’ve been helping organisations with for almost two decades by building services and products for them to do this. And so the CRA maturation is generally good. The problem was in the drafting; they were trying to close loopholes. They literally said they were worried that organisations would get around the penalties by open-sourcing their software, which I’m not sure that’s something we should be worried about, right?
If a big commercial enterprise is willing to open source its software, why would we not think that’s a good thing? But they did. And so, in the result, they tried to define commercial and tried to put obligations on open source producers, which, without specific carve-outs or exceptions, could have meant that individuals who were writing software anywhere in the world could be held liable for significant security vulnerabilities in software that they had no control over. So if you wrote a library and somebody included it in, I don’t know, a car, and it was a self-driving car, and it killed some people because of a bug in your software, you might be held liable even though you didn’t know that your open source was being used in a car. And so it’s one of these cases where you take a good idea, but you try to go too far, and it can create some backlash.
There was also some early language that put Sonatype, the company running Maven Central, which hosts all the world’s open-source Java libraries, suddenly on the hook for the security of everything being transmitted through it. And so what I was standing up and shouting to anybody that would listen a few years ago is like, listen, this is good. But what Europe is about to do is basically box itself off from the rest of the world because you’re not an EU citizen and you’re writing open source. I don’t have to comply with your laws. I can change my license and inform you that you can no longer use it in Europe. Is that what you really want? Do you want to have to find your own forks of Linux and literally everything because open source users worldwide carve you out?
And currently, the only places where you have to restrict your open source, at least as a US citizen, are North Korea, Iran, and Russia. And did Europe really want to add itself to that list by saying you can’t have access to open source? And further, do you want to put yourself in a situation where providers of open source infrastructure like Maven Central would be forced to block it for Europe, lest we become liable for literally every bug and every piece of open source Java? It was insane. Fortunately, it took a long time, but people listened, and they made some exceptions to clarify the matter. However, that wasn’t the primary goal when the legislation was passed. I was content, maybe not happy after the battle. In the meantime, it has become problematic how these standards are created. The legislation called for the definition of standards around a lot of different things.
And there are standards bodies, specifically ones that operate within Europe, and they’re used to working in a very commercial sense. And so what’s actually happening right now is that open source is still struggling to be heard in these standards. So I’ve regressed in my happiness from that contentment to some frustration. And we might backslide because the resulting standards that get defined may only work for commercial entities, as that’s the only group with input to the process right now. And I’m sure people would take issue with that statement, but that’s what I see unfolding, and I’m still in several meetings where this is the conversation that happens nearly every week. So we’ll see.
Olimpiu Pop: The CRA was a battle from the start, and it was an ongoing process. And in the end, the compromise or the point reached was decent and okay with open source supporters. However, as it begins in the execution phase and its implementation, it’ll come into full force two years from now. Commercial entities will be heard rather than having an open table where everyone, including open source representatives, can discuss.
And that’s the most significant danger at this particular point in time, especially as we hear a lot of sovereign in conversations. Also, when it comes to cloud, for instance, it’s a lot of things that are being pushed by. What I was surprised to see now, with the Open Source Summit, and also in the spring with QCon Europe, is that there was a lot of discussion about sovereign cloud, the Linux Foundation EU, and, as you said, factions are being created rather than moving towards a common goal. We should be concerned about what happened and remain vigilant, acting as a group, because as consumers, we need to understand and ensure that it’s in our best interest.
Brian Fox: Yes, it is a very slippery slope. And I haven’t heard of any recently, but there were some open source projects not long after the CRA was signed that basically withdrew all of their open source from the market, because they were basically saying, I’m not a lawyer. I can’t tell that I’m not culpable here. I’m not committing to providing five years of support on all of my stuff. So, effective immediately, I revoke all of my open source. It is a true statement that at least some people have pulled back from open source, not just in Europe but globally, because they can’t understand the implications of this. So it is a real thing. It’s not just sheer mongering. How bad it will be is still to be determined.
Olimpiu Pop: Yes, what you can see in the last period is that software is coming of age, people really understand where it happens. But in comparison with the rest of the infrastructure, I don’t know, let’s think about electrical infrastructure or gas infrastructure, software has the added value and the added ability of being able that a group of people around the globe can act as providers, as infrastructure builders, but nobody will just go and build their own electrical power grids and open source and just allow it. So that’s the advantage, but also the problem with what we have.
Going towards that, I saw that during the Open Source Summit, there was a lot of discussion. The Linux Foundation discussed a lot about what happens in the open source space. Everybody knows and has seen the report that said that 90 per cent of the software is now open source. Frank Nagle is part of the Linux Foundation, and he’ll continue this effort, especially in the AI age. Then, there were reports about the commercial open source and companies like MongoDB, which also take money from their paid products, but provide an open source licensing model. So it’s an enjoyable day, what’s on your plate these days? Because I always have to go always during the autumn times, I’m waiting for the open source report that you’re putting together on the state of open source 10 years from now. This year will be the 11th.
Unveiling the Vulnerabilities in Open-Source Infrastructure [10:02]
Brian Fox: This year marks the 11th anniversary, so we have been publishing the State of Software Supply Chain Security report every year. We have shifted it to Q1, so it’ll be a wintertime thing this year for the 11th. It started that way in the beginning, and we slowly turned into the fall. So we’ll be working on it. We’re going to be looking a lot at AI, agentic AI, how that intersects with open source governance and stuff like that. I’ve written about that on several blog posts lately, if you’re curious about where we’re thinking about that. But that work is coming.
In addition to that, I’ve been speaking and writing about the sustainability of open source infrastructure, which is an interesting topic. Everybody’s used to thinking about open source sustainability, and everybody’s head goes through, right, pay the maintainers, have your developers contribute all the code. Yes, you should do all those things. By and large, most people don’t. Nothing new there.
What has been flying under the radar and not enough people pay attention to is all the infrastructure that open source production depends on these days. These are things like Maven Central, which Sonatype pays for. We run it like it’s a charity. We’re not a not-for-profit, but we treat this part of it historically like it’s a charity, and so do most of the other package registries. A lot of different open-source infrastructure is the same.
It’s donated for open source, or it’s donated to support open source; take your pick. All hyperscalers provide credits to projects and similar initiatives. Everybody’s doing it. But by and large, they’re all operating like charities. There’s no balancing of the consumption and the costs. And so I started digging into this early last year, in 2024 precisely, because I was having conversations with hyperscalers and others about trying to figure out a better model.
Because when I look at our data in Maven Central, 80% of the traffic is coming from one of the big three cloud providers in the US: Amazon, Microsoft, and Google. So their customers collectively produce 80% of the bandwidth that Sonatype pays for. So there are conversations to be had about what to do about that? These are scalers; they have obviously better rates for bandwidth than I can negotiate. However, that conversation led to things like, ‘Well, nobody wants to sign up for an open bill just to pay for unlimited bandwidth.’ What are you doing to curb the abuse then? Nobody ever really asked me that before. And it’s somewhat ironic because everybody thinks that, oh, well, there’s just a bigger fish. They will always just step in and write an unlimited check for these things. And the answer is they won’t. In fact, most of these companies are the sponsors in almost every open source foundation, including the ones I’m a part of and all these things.
So they are giving back, but nobody wants to sign up for running a service that has to be run indefinitely and is critical infrastructure that is just growing unbounded. When I started digging into it, I found that the consumption patterns were massively lopsided, and I forget the exact number, it was somewhere also in the ballpark of 80% of the traffic was coming from less than 1% of the individual IPs worldwide. It was massively skewed towards a handful of weighty users. We had to implement some sensible throttling to bring this under control. And with open source things like this, you have to approach it carefully because people are used to getting a free ride forever. And so you have to be benevolent in your actions and change that. So that led down a journey that went from looking at individual IPs to looking at organisational consumption.
I found that there were substantial non-cloud organisations that were downloading stuff from us across thousands of IP numbers, and in some instances downloading the same set of 10,000 JARs half a million times every two weeks. That’s ridiculous. And so it’s like, why is that happening? And that led me to go further, and I uncovered further. I wouldn’t necessarily call it abuse, but I would say decisions made by other tools in the ecosystem by companies that don’t pay the bill, looking at all of this stuff as just free infrastructure.
And it is the classic case of what we say in the tragedy of the commons, when nobody really sees who’s paying for the thing, then waste ensues. And so I’ve been on a journey down that path for a long time. In talking to other open source infrastructure providers, other package registries, and what have you, everybody is carrying this burden at the moment. We’re having conversations about how we can change that dynamic because none of this is sustainable to run it as an indefinite, unlimited, unbounded charity.
Olimpiu Pop: That’s for sure true. But that’s something that you as a company are supporting, and everybody takes it for granted. And if you look at the big players, you mentioned the big three cloud providers, they have 80% of the downloads going towards that direction, but nobody’s willing to pay, or they don’t have any intent. They expect it to be like clouds, right? They’re in the sky, they’re there. We don’t care about how they get there and all this kind of stuff.
Brian Fox: And to be clear, it’s not so much the clouds that I’m talking about here. It’s the users in general. Many of them are on the cloud, which makes my job more difficult because I don’t easily have a way to find out who they are. I can’t reach out and contact them and ask them to change their behaviour or do anything like that. It’s opaque.
This is not an indictment of the cloud. The cloud, like I said, they pour tons of money into open source. They’re the ones who help start the Alpha Omega fund and know all these other things. But the point is, even in conversations with them, it’s hard to align the business and the cost side of things. And that’s ultimately the problem here, that all of these foundations, and to the extent they’re supporting open source infrastructure, rely on a set of benefactors, I.E., their members who sign up to support a cause.
But the members are not getting into this thinking that, oh, I want to sign up, so a big chunk of my annual dues goes to paying for another Fortune 100 company to download JARs a million times a month. Nobody gets excited about that, but that’s what’s happening because these foundations rely on member dues to pay their operating costs. And on the flip side, all of the consumers can just consume this stuff in an unbounded way.
So it’s a little bit, whether it’s a single company in the case of Sonatype paying for Central or if it’s like members of, take your pick, Eclipse who are paying for the OpenVSX stuff or the members of the Linux Foundation paying for all the distribution, when the consumption is misaligned with the cost, it creates massive amounts of waste at the end of the day. And so that has to change.
And so we’re exploring ways to think about that. Throttling is part of it, but there may be models where, if organisations exceed a certain expected threshold, they have to pay. If they’re distributing proprietary software, which is the case, I find many companies. I call it the unrestricted CDN use case. They’re putting stuff in Central, it’s clearly proprietary, and they’re doing it just because they don’t have to buy or pay for a CDN to distribute their stuff. Nobody would argue that’s fair, but they’re leveraging the distribution and network effect and all of that, that’s great, but if they were paying some, it would help to contribute to it. And so that’s where all this is going, that we need to do a better job of balancing the cost and the consumption, and then some of these things start to work themselves out.
Changes We Can Make To Release the Pressure on Open-Source Infrastructure [18:12]
Olimpiu Pop: We all like the benefits of open source, especially the part with the open, but it’s also about the responsibility. If you use it, you should contribute to that. If you’re not able to contribute money, time or whatever, it would be nice to see how you use it. The patterns, the way you interact with repositories or package managers. It’s a way. What would be a couple of suggestions for developers, people who are just using package managers, on how they can adapt the way they interact with this to make their life easier and their life easier in the end?
Brian Fox: Yes, it’s a great question. And by and large, it’s not the individual developers that are the problem. They’re the ones that we want to be able to have access to this stuff. It’s massive organisations that are not using a repository manager, for example. And so instead of caching the dependencies they use, every single one of their 10,000 CI jobs or in the near future, their agentic development tools are spawning off and re-downloading the same JARs all over the place, because it’s quote unquote free, that’s kind of the point that I’m making.
So if you’re in charge of infrastructure at these organisations, you should be looking closely at how you’re using these open source tools. Just because they’re free for you doesn’t mean they’re free for the world. I equate it to burning carbon. It’s why the cap in taxes is a conceptually valuable thing here, because if it’s effectively free for you to emit unbounded amounts of carbon, yet all of humanity pays the cost.
That’s what we’re talking about. As soon as you start putting a price on it, then people start thinking, well, those efficiency things make sense for me. And so, if there’s no penalty for you to simply download the same components to your cloud all the time, because your ingress costs on your cloud are free, then why would you do anything differently? And that’s the part, it’s somewhat invisible, the costs. And so that’s what we’re talking about. So if you’re in charge of the infrastructure, think about how your organisation is using it, and think about using it more efficiently and contributing to it. When these foundations start making changes to their models, it is unlikely to be an option, but being a champion of it will be helpful. If you’re developing tools that use these services, think carefully about how you’re doing it.
I was shocked to find that many of these giant companies, which were hammering Central, were doing so; they even had repository managers. Some of them, like Nexus, couldn’t effectively use it because of decisions made in the build tools they were using, which made it really, really difficult for them to send their traffic to the repository manager. So they were accidentally doing it. And this was again, because another company created a build tool that didn’t have to pay for Central, so they didn’t think about making it efficient. And we found multiple instances like that. This is why I’m trying to raise awareness. For the most part, nobody is doing this stuff maliciously, but if it’s invisible, then we’re collectively squandering the resources that we have available to us.
Olimpiu Pop: A couple of years ago, a group was writing a carbon almanack of people around the globe, including those in business, marketing, and various other fields. Then they’re trying to provide a basic understanding for everybody of how important carbon management is. And then they’re providing numbers like, okay, a solar panel, it’s needed to use carbon when you build it, but then you get to the point where you get to zero in a couple of months, you already used it, and so on, so forth.
One of the conclusions was about comfort. As you said, it’s not malicious, but it’s comfortable to just not think about these kinds of things. It makes my life easy, and I’m dependent on something that usually matters. And given the size of software infrastructure these days, we are aware of the carbon footprint we create. We are one of the major players, especially with all the AI advancements, and it’s getting worse. We don’t even think about it, and we are doing some pretty stupid things. An example that I was discussing at some point with Holly Cummins from Red Hat, probably IBM now. Some people are simply using 4K on YouTube to listen to music. They are coding, they don’t see the video, but they use 4K, and then all the infrastructure behind it.
Brian Fox: That’s a perfect example. It’s the waste. And why do you do that? Well, because it’s easy.
Olimpiu Pop: Yes, it’s comfortable.
Brian Fox: Yes. I had an example of one person who essentially had their pipeline set up so that every time they committed, it ran through all the tests and all the continuous integration processes. And then it was staging a potential deployment on Maven Central just to run through the validation, because it had failed once before. So they set it up, and so they’re doing this and causing us to have to validate signatures and do all kinds of work when they know they’re going to drop it. And they were doing it on every commit.
And so I reached out and was like, dude, what are you doing? I get it. And at the end, I told them why. I said, “Listen, if you don’t care about wasting Sonatype’s money, at least be a better carbon burner for all of us, because you’re wasting carbon and doing this. This is just ridiculous”. And it’s those little things. And it was, when I pointed it out, they were like, I never thought about that. I thought about it just because it was free. However, when you actually consider who is paying for that CI time, it was probably donated time, possibly from GitHub or GitHub Actions. Someone’s paying for it, and it’s not a free ride.
Olimpiu Pop: Obviously, but let’s not expect everybody to think about it broadly and understand our pain or your pain in this case. But the only thing that’s going through my mind as somebody who was on top of infrastructure in multiple companies is, how about the risk? What if something happens and you accidentally unplug Maven Central? My company is reliant on that, and somebody, I don’t know, the cleaning lady just stomps on the cable that keeps the server, the Maven Central cable, up and running. What happens? I’ll not do anything because I’m reliant on Maven Central. So then that’s one-on-one risk mitigation and the risk management,
Brian Fox: That’s been the message, of course, as our first product was Nexus, the caching proxy. And so it makes those things faster, more reliable, and you’re less reliant on the internet. And the irony is, everybody had those in the early days because Maven Central wasn’t on a highly performant CDN. It was donated time from a university in the US, and so it was not uncommon for it to fail for a while. So everybody had to have a repo manager.
And so in a weird way, by us investing in more and more expensive infrastructure, we’ve worked against people using repo managers. We made it more reliable, and then they no longer felt the need to do it. And so this is a classic win-win because if you start doing these things, your builds will get faster. You may not care that you’re spending money downloading those things, but it still takes time, and it’s always going to be faster to pull it from a 10-gig network connection in a machine in your own rack than it is to pull it over the internet every time. There’s just no way around that.
Supply Chain Attacks Increased 700% in the Last Few Years [25:29]
Olimpiu Pop: That’s for sure. So let me iterate over the potential risks of, I said developers broadly, meaning that people that are consuming, that is organization and everything, but because in the end the developers are the ones that are ringing the alarm bells, they just go to conferences, they listen to things and then just go to their managers, and then they come with the question, how do I convince my manager to do the right stuff? Well, most of the time, except for the AI, because generative AI convinced all the managers to convince all the developers they use generative AI, which is also a different pattern.
However, you mentioned the risk of not having the necessary infrastructure. At some point, the infrastructure is severed, you are out in the cold, and you don’t have it. Then obviously, the global warming that is pressing all of us, and we can see it every day. Another factor to consider is the risk. I particularly looked into the state of the supply chain report that you put together in the last two years, and I reported on it. And what was interesting is that the supply chain attacks were increasing, I don’t know, it was three X, four X, and pretty much everything was what happened in the, I don’t know, in the first years of the report, it was probably five times more in the other part. I don’t remember the numbers, but the scale was ludicrous.
Brian Fox: It has been growing by 700% for multiple consecutive years. Yes.
Olimpiu Pop: Yes. Am I correct when I’m saying that if I were to cache inside of my organisation, I’ll be in a much better place because I’ll be able to make sure that I’m in a safer place, what I’m using, I’ll not be exposed to something open, it’s exposed to everybody?
Brian Fox: Yes, I’ve been talking about that for so long. You can see, I don’t even talk about it anymore. It’s old news, but it’s useful for people to hear it again. What you’re talking about is the rise of intentionally malicious open source, right? And we’ve been talking about that, tracking that, providing solutions for that since at least 2017. I don’t know the latest number. It’s something like 900,000 malicious attacks, and it’s going up by thousands every week. And yes, part of the problem is that so much of this depends directly on cashing or fetching items from the repository, which means that if attackers can get something into the repository, they instantly gain access to millions of items, including NPM and other tools. They prefer the latest. So it doesn’t matter. I don’t have to put it in there and convince people to upgrade.
I just put it in there, and a tool fetches it instantly. Being completely online-dependent with these things in real-time sets you up for a situation where your developers potentially download a malicious component within seconds of it being on the public repository. Tools like Nexus have capabilities that you can put in place for your developers, right? Because we have information. We analyse these things as soon as they hit public repos, not just Maven Central, all of them, because our customers need all of these things. We can tell when something is fishy within minutes of them hitting the repo, and then we can block that from being fetched by the developers. But if you don’t have that infrastructure in place, you don’t have any of these firewalls, if you will, that can stop this stuff from coming in.
I started this conversation focused on just stopping the resources. But you’re right, there are these other, more traditional angles of it, and it will make you more secure. It will make you faster. And so it’s like, why aren’t you doing that? Part of the answer is that some of the tools that they’re using have made the decision not to prioritise that. Tools like Gradle make it hard to use a repository manager, for example. That’s what I’ve uncovered. I didn’t know. I’m a Maven guy, not a Gradle guy. But that’s what I learned in this process when I was working with some of these companies trying to curb their behaviour, and then they came back and said, Yes, we can’t. And so I dug a little deeper and uncovered some of that.
We found, for example, that React Native is compiling JavaScript for native mobile phones and similar applications. We found that what would happen is they’d run an NPM build, and that thing would fork off thousands of Gradle builds. Each one of those Gradle builds was fetching the same components over and over again. And worse, because the Gradle files were generated, they couldn’t insert the line in there that said, go to my repository manager.
I found one instance of a large company with 60 developers, but not their CI systems. Their CI systems were coming from a different CI space, just literally the developer machines in their offices; they were creating more traffic to Maven Central than Many Clouds, not the big three, but clouds like cable modem providers, which is shocking to think about, but it’s not that hard when you actually understand what was happening. One build was recursively forking tens of thousands of other builds, and each one of those was downloading the same things over and over again.
Fortunately, when I reached out to the React team, they made it possible to make changes, to tell the React staff when you generate the Gradle files, and overwrite the URLs to use my repository manager. Now it’s a matter of convincing people to flip those switches, but at least they’ve made the switches. But these are those hidden things, these decisions that nobody thought about that were made and now baked into the tooling that just makes this so much worse. So those are some specifics of what we’ve uncovered, and I’m sure I’ll find more.
Most of the Package Managers Work Together on the Infrastructure Problem [30:48]
Olimpiu Pop: Yes, good luck with that. So I’m thinking about all the ecosystems that I know that were problematic. I know that NPM was one of the worst affected when it came to malicious open source. PyPy was another one. Are you alone in this battle, or is it the common front? I think that NPM has a lot of traction. JavaScript is growing a lot. Python is increasing rapidly these days. Cargo for Rust, again, has gained significant traction with its adoption in the latest kernel and is undoubtedly the shiniest tool in the CNCF ecosystem. Are you working together? Or you’re still alone like Gandalf in the dark, and then try to bring people together? Or should I leave this as a question for a later date?
Brian Fox: Most of the package managers have recently come together, and we’ve discussed the sustainability problem. Everybody is having the same problem. Everybody is unsure how to approach it with their communities. We are therefore working on a joint statement to increase visibility of this issue globally. So we are working together on that. There are working groups within the open SSF where most, if not all, of the package registries do work on things like checklists for how to make the supply chain more secure, who’s implemented MFA and these types of things, for sure.
One of the challenges they face is related to these endogenous decisions. In Maven, there’s the concept of the namespace, the group ID, which Byron from Java, the reverse DNS of your company’s name. This makes it a convenient place for us as we manage Central to make sure we validate that you control the namespace that you’re trying to publish to. You either need to have your own domain or use the project domain name of a foundation or GitHub organisation you’re affiliated with, but we require you to provide proof.
And that takes a whole set of these malicious attacks off the table because it’s much harder for you to pretend to be org Apache Struts than it is if you were just trying to pretend to be Struts. And it makes it less likely that somebody would confuse you as the official provider of these things in these other ecosystems. They don’t have the concept of namespaces, but they are familiar with NPM’s added scopes, which were introduced after the fact and are optional. And most things published into the repo don’t have one. So now you’ve removed one piece of metadata that would also have to be faked. You remove one factor in the identification, if you will. So you’ve got that problem.
And then there’s no validation of the name because there’s not a namespace to tie it to. And so you can easily typo-squat popular projects, and then you combine it with what I mentioned before, which Maven actively discourages you from saying, I want the latest jar. You need to put a version number in. And so just because a new version appears in the repo doesn’t mean your build is fetching it. You would have to update and grab it. So we preferred stability over auto-updating, which, to be fair, comes at its own costs for a long time. I was talking about how long people were still running old, vulnerable versions of Log4J, right? So there are trade-offs to that decision, but the other side of it is ecosystems where it’s going to fetch the latest unless you tell it not to. That’s the ready audience for the attackers.
So those factors, the ease of pretending to be someone you’re not, combined with the audience that’s about to fetch this stuff as soon as you can get it out there. If you were an attacker, why wouldn’t you start there? And so that’s why we see it’s very disproportionate. These intentionally malicious components are appearing in those ecosystems. It’s not because they’re not trying to make it secure. The decisions made in the early design of those tools didn’t provide the levers that they really needed to do things like what we’re able to do with Maven and with Maven Central.
Olimpiu Pop: What I was thinking while listening to you is about the open SSF scorecards. I found that to be very, very good. And also the tools that are just going over your repositories, and it just comes with the score and the suggestions. I know that at some point, I spoke to one of the maintainers of an open source library, and after the presentation, he said, “Okay, well, I learned something new”. Well, I was pleased that I could promote that. And something that could help people, or be on the path towards continuous deployment. There is also this minimal continuesdeployment.org, a manifesto about how we should do things better. Sending out something in that space will help.
Brian Fox: Yes. The push towards continuous everything exacerbates most of the problems I’ve been talking about. And so it’s a case of we take things to the extreme, just like the example I was talking about with the publisher, staging all the way to central, making people aware that those will definitely help. And the other part that I’ve been trying to shine a light on, because I learned this recently about a year ago, and it kind of surprised even me, is that some of these open source projects spend about $60,000 in machine time just to do a single release. And all of that is because all of the integrated testing and all of this kind of stuff, when you’re talking about big distributed databases, distributed file systems, testing that at scale, it’s not cheap. That’s what some of these open source projects are looking at. And it’s like, wow, we’ve moved from a very different place than we were when I started.
When I started, I was donating my time. I already had a computer, an internet connection and electricity, and I just needed time to work on the code. Fast-forward to where we are today, and it’s more or less frowned upon if you do a release from your personal machine because who can trust that your machine hasn’t been compromised? We like to see things in repeatable, auditable, trackable supply chains. And guess what? Those are all running in the cloud, mostly on donated machine time. And that’s the part that crept up on us and has become invisible.
But the expectations the world has of all of these things come at a cost. And so we’re now in a situation where, yes, the developers are still contributing their time, and there are conversations about how they should be compensated for that, of course. But the light I’m shining on is everything that goes underneath that as well, just not free. And so the more people think about that, they will, at least some of them, hopefully, will inevitably start changing their behaviours a little bit to be better overall citizens, to recognise that.
Optimising Builds: The Power of the Repository Manager’s Cache [37:28]
Olimpiu Pop: That’s definitely useful. There are two options to make more money. One of them is to gain more, and the other one is to be more cautious on how you’re spending the money. And if you’re choosing both ways of doing it, it’s definitely for the best. The way you mentioned, it’s definitely the desired way that tremendous players have better governance and a better way of implementing stuff on their side. And we know that it’s not something that you’ll do from today to tomorrow.
And now I’ll go back again to the pawns of this big landscape, the developers. Suggestions that will help them make their lives easier, but also your life. How they start looking at the build is the main subject here, and the way they interact. What should they know? Three things, five things that you think would be useful for them to do, to wrap up with an idea.
Brian Fox: Make sure that you have a repository manager so that it can cache all of these things. That’s true. Even if you’re running in the cloud, the speed of light is still a thing. Additionally, double-click on that now and ensure that your builds are actually using it. This is the thing that I uncovered. People thought they were, and they turned out they weren’t. So look closely at the logs, make sure it is in fact downloading from your Nexus or whatever you have. And that would go a long way. And by the way, it’s going to make all of your stuff faster, and it might even save you money too, because now you’re paying less compute time for all of those build jobs running. You might not pay for the bandwidth, but you’re paying for the CPU time while it’s sitting there, twiddling its thumbs waiting for the downloads.
These are the common-sense things that start to make a difference. And then that also, like I said before, provides you with that leverage point to begin putting protections in place for the malicious open source and things like that. And flip it around if you are an open source developer or you’re developing tools that are leveraging these things, just think carefully about how you’re leveraging them—cache things. Don’t rerun stuff every single time just because you can. Back to the way we used to have to write code when we had limited memory, a limited CPU couldn’t infinitely scale it. We had to be a little bit more judicious in what we were doing. Bringing that back, I think, can help a lot here.
Olimpiu Pop: Let’s adopt an embedded development mentality in the cloud space, promoting greater frugality. The cloud providers have already started slashing their hundreds of services. So it’s about time for us to look at what we have. So before we move, increase the infrastructure, because as you said, it’s very easy these days to just go into a console. You just increase it. All of a sudden, you double the number of run numbers of CPU,s and it’s a lot easier than it used to be. Before you do that-
Brian Fox: We’re on the precipice of an explosion with AI because some of these things were self-limited by how many humans you could add to the problem, right? Imagine you can quadruple your development force or more, just like you can dial up your cloud. If you can dial up or down your AI agents, these small inefficiencies now become even more amplified. So we’re on the precipice of seeing some of that. And I know for sure that some of the package managers are struggling with AI-induced loads for that exact reason.
Olimpiu Pop: Yes. So I remain with my statement, which said that AI is a bicycle. If you’re just going downhill and you don’t wear a helmet, you’ll have big problems if you fall. If you’re doing it the proper way, it might help you. Before we start using AI chaotically, make sure that we are using our own infrastructure and we actually know what we have there. Okay. Anything else, Brian, that we would like to touch upon before we wrap up?
Brian Fox: I think we’ve beaten this one to death. Hopefully, it’s useful for the audience, and they learned a thing or two.
Olimpiu Pop: Great. I’ll touch on the blog post that you mentioned to provide more awareness to everybody. Good luck with your battle, with our struggle. Thank you for being the champion of the open source community, and have a nice day. Thank you.
Brian Fox: Thank you.
Mentioned:
.
From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.
