Why Rust Will Help You Deliver Better Low-latency Systems and Happier Developers

Transcript

Olimpiu Pop: Hello, everybody. I’m Olimpiu Pop, an InfoQ editor, and I have in front of you Andrew Lamb, one of the developers who helped with the transformation of InfluxDB, the whole rewrite that brought us 3.0. Given that they changed their language and their ecosystem, we are curious to see why. Why Rust, mainly, because that’s the language that they chose, together with the whole different stack. So, Andrew, please introduce yourself and then let’s get started.

Andrew Lamb: Yes. Hello, my name is Andrew Lamb. I’m currently a staff engineer at Influx Data, and I’ve worked for five years professionally with Rust. Before that, I worked for another 15 years professionally and mostly with C and C++ on databases and other low-level systems. Recently, I’ve been working extensively on Apache Aero and Apache Data Fusion, and I serve as a PMC chair for both projects. But yes, I’ve been super excited about Rust. I’m thrilled to tell you all about it.

Why use Rust rather than C/C++ for database engine development? [01:19]

Olimpiu Pop: You mentioned that you have a 20-year programming career, with 15 years of experience in C and C++, which is typically the preferred language when discussing database engines, and the last five years in Rust. Moreover, as a maintainer of two open-source libraries from the Apache Foundation, you live and breathe this thing. So everybody says that Rust is a whole different beast. It’s different from everything else that’s out there. What’s your feeling?

Andrew Lamb: I came to Rust because, at the point in my career when I joined Paul at Influx, I needed a job where I could program again. Paul said, “We’re going to do it in Rust.” And so that’s why, from my perspective, we chose Rust because Paul said so, and I just joined it. I need to program at a job. And at that point, I thought it was hipster language. I’d seen a bunch of people discussing it, and I’d attended some early meetups. However, the examples we showed had far too many ampersands, and there was something about a borrowed check, so I didn’t approach it with a deep analysis. I just came at it from like Paul said, I’m excited to build databases again, and that sounds good enough to me. But over the last five years, I’ve come to love it, and now I’m a true believer.

And I can look back, and I can articulate the things that have made it seemingly wonderful to me. Because when you’re first starting with Rust, the pitch from the Rust fanboys or fan fandom is that once the Rust compiler compiles, you never have to chase down race conditions or crashes again. And sometimes they’ll talk about memory safety, sometimes they’ll talk about whatever. But those are the core things that were super valuable at Rust: it does live up to those, and those are broad claims, and you don’t know until you’ve spent a while trying to build a real production system. But over the five years that I’ve been doing this professionally now, I’ve chased, maybe one memory bug, depending on how you count. It’s not a memory bug, and there are zero race conditions.

And compared to my previous experience in C and C++, it’s just like night and day. I spent a significant amount of time chasing down bugs in those classes. The really tricky ones, right? You’re beating on the database for three days, and it crashes after three days. Now you’ve got to go figure out what’s wrong, and it turns out to be some weird use-after-free or whatever. And in Rust, we’ve had none of that. That is the core value proposition for Rust. There are a lot of other things that people talk about, and maybe you like how it looks more functional, you don’t. But at the end of the day, the reason it’s catching on is that you can achieve C, C++-like performance with far better productivity. It has modern tooling, so you don’t have to chase down race conditions, site faults, or other issues.

Performance vs Productivity [04:06]

Olimpiu Pop: You mentioned a lot about performance versus productivity. So, in the end, effectiveness is either about the execution of the programming side or how you, as a developer, run things faster. Let’s frame that. How do you measure productivity, and how do you measure performance?

Andrew Lamb: I’ll start with performance because that’s the easiest thing. So performance is how fast you can get the program to do whatever you want. And these days, that involves multiple cores, you can efficiently split your work up and use them effectively. And the other one is just the raw computational efficiency of your low-level code. In the context of databases, performance often boils down to minimising memory copy and being careful not to recopy data or make too many allocations. And that low-level memory management layer, which is why I think Rust ends up being a perfect choice for databases because unlike a bunch of managed languages like Java or even Golang, you can go so far with performance, but then at some point to get the next 20, 30, 40% of performance, you have to stop copying memory around.

You just need to be more careful about how you’re doing it. And that is when languages like Rust or C++ come into the bank. Additionally, Rust uses the same underlying compiler infrastructure as C++. So, you’ve got Clang, which generates the code, performs all the optimisations, and handles auto-vectorisation and other tasks; that’s LODM. Rust is the same thing. It’s just the front end is different, but then by the time it’s generating code, you’ve got all the same optimisations in years of investment that have gone in. So that’s the performance aspect of it. So from everything I’ve seen, Rust and C++ generate the equivalent code. You’ve got to work at it sometimes, but I’ve never found a place where it’s like, oh, I just can’t make Rust go that much faster. You can always basically drop it into doing whatever you could have done with C++.

Olimpiu Pop: Okay, so to sum up the points that you had. First of all, what you’re saying is that Rust is solid from the point of view of where other languages failed from, I don’t know, segmentation fault, all these problems with memory and all the bugs, all the crazy bugs that are just, you don’t understand why it’s working or it isn’t. And then the other point is in the frame that you are focused mainly on data focused products it’s the fact that you don’t need to move memory from one side, copying memory from one side to another, and that reduces the number of operations that the processor needs to do and that allows it to be a lot more performant.

Andrew Lamb: That’s the performance piece. And then there’s the productivity piece. I’ll just tell you some horror stories of what I went through with C++, and then we can talk about it. I’ll let you know the equivalent in Rust. So, horror stories and C++ are involved, well, let’s start with the build system, right? Have you ever had the joy of messing around with the C++ build system, but I spent a lot of time using makefiles. I realised there’s now CMake, which is slightly better, but I spent far more time than I’d like to admit trying to figure out why the thing wouldn’t link. It couldn’t find the correct symbol, and you had to play games with this WL linker flag. If you don’t know what I’m talking about, that’s good, but it was miserable. The reason is that Make was like a 30-year-old tool at this point.

So, integrating a C++ library can be a real hassle, as you have to ensure all the symbols line up and everything else works out. So it’s not impossible. There’s a large C++ codebase to figure out, but it requires a significant amount of effort. In Rust, they just copied. I don’t mean to diminish it, but they essentially follow the best practices that Python and JS innovated, particularly with package managers. And so if you want to add a dependency on Rust, that’s just built into the standard language. So again, that sounds simple, but it makes a big difference. Now, if you want to get a third-party library, it’s like you add a line to a config file rather than spend an afternoon figuring out a whole different weird build system and hoping that it builds on your machine.

That’s one aspect, right? This is modern tooling. And of course, Rust also comes with similar things. I think C++ is mostly caught up, with integration with all the IDEs, autocomplete, a formatter, a built-in unit test framework, and a documentation generator. Again, you can get this stuff in C++, but you have to pick and choose which parts you want, and some projects do stuff differently. Having that ecosystem and the modern tooling makes a big difference. And then there’s the other thing we touched on briefly: memory safety, which I would lump together.

People could discuss the numerous security benefits associated with that. But I also consider it from the developer productivity perspective. Because I spent a significant amount of hours in my career, like days, weeks, months, chasing down weird race conditions where you start the thing. Everything works fine, or you shut it down, and it falls for some reason, and you spend a bunch of time trying to figure out why it only happens sometimes, only under load, under this, that next thing.

It’s like spending a considerable amount of time debugging what eventually turns out to be a use-after-free or something. And yes, some structures and modifiers make that less likely, but you have to know how to use them. So it’s been night and day. Instead, what you get with Rust is that the compiler will sit there and complain at you. You just can’t. There are specific patterns that prevent you from doing something; provide two different ways to change the same object. In some ways, it’s pretty simple. It just doesn’t let you do patterns that could be dangerous, not necessarily hazardous, but potentially risky. As a result, it rules out a whole class of bugs. It’s been amazing. It’s very frustrating the first three months because I had patterns that I had done in C++ for 15 years, and the Rust compiler just would refuse to compile them. Just say, You just can’t do that and it’s super frustrating. This is fine. I know this is fine. I’ve done this many, many times, and it would say, No, you can’t.

The Rust compiler helps you write robust code. [10:11]

Olimpiu Pop: What would be one of the things that you created with that kind of automation, and then the compiler says no?

Andrew Lamb: Yes, you tried to pass. You have a struct with a couple of fields. You want to modify one function to change a set of fields, and another function to change a different set of fields. Rust won’t let you effectively call one function that changes some subset and then call another one that might change another subgroup. At least it wouldn’t at that point; they might’ve gotten smarter now. Because it says, hey, you might’ve modified it here, and then this other function could modify it as well. But of course, you have to get used to the terminology that Rust uses, which is borrow checker, immutable borrows, which at first is a whole foreign language.

Once you invest in understanding that foreign language and grasp what it’s trying to convey, you might notice aliasing, which is why we want to prevent you from doing this. Instead, you’ll understand the pattern, making it much more straightforward. It was frustrating the first several months, but once you got into the flow and just went with what they suggested or what it was telling you, it’s been great.

Olimpiu Pop: Okay. I recall a presentation where someone wrote on a slide, “In Rust, it’s like doing pair programming with a compiler; it’s your friend.” And one of the examples was the fact that the error messages mean something, and they really help you, rather than all the other programming languages out there.

Andrew Lamb: I think, compared to all the other programming languages I’ve used, which are mostly C++, it’s gotten better since then. But yes, the Rust compiler sets the bar for sure on it doesn’t tell you what’s wrong from the compiler’s perspective. It basically tells you what’s wrong from the program. Basically you as a programmer’s perspective, which is it sounds obvious that that’s how you should do it, but the types of errors you got out of C++ were like, you need this particular syntactic structure. If you as a programmer didn’t know what it was talking about, you would’ve no idea. You have to go Google around, stack over a phone, figure out what the error actually meant. Whereas Rust says, hey, you probably meant to import this thing and you just copy paste the suggestion. It often just works.

How mature is the  Rust ecosystem? [12:24]

Olimpiu Pop: Let’s say that this settles it for the language. You have a decent ecosystem. Probably most of the languages out there now have a decent IDE and you have let’s say a decent way of working with those things. There is a tendency now to have more focus on functional programming languages because we all have more cores and we’re discussing a lot about working in a stateless manner and obviously functional programming is helping with that aspect. You mentioned the memory handling that is different than all the other ones and it makes sense. The other thing that is useful is that 90% of the application that we’re running, because most of us are using libraries. And one of the powers for instance of the Java ecosystem, that is probably one of the best out there because it’s quite broad and it grew a lot in the last almost 30 years, more or less. How is it with Rust?

Andrew Lamb: That’s a good question. If you’re going to build a modern application, you almost certainly need web clients, probably some web server, some sort of application framework, all that sort of stuff. And I agree a hundred percent more or less Java sets the standard for completeness and maturity of that type of ecosystem. I have actually been very pleasantly surprised with the maturity of those systems. They don’t have quite as many features.

Another thing that’s happened is that the Rust ecosystem tends to move fast. So one of the pain points we’ve hit with Rust, so Rust is great, I’m telling you all these things about Rust. I’m not chasing down memory pointers, but what’s some of the downsides of Rust? And part of it’s the ecosystem you’re describing. The ecosystem is still relatively new. It hasn’t had 20 years of battle testing. What that means in practice is that your dependencies upgrade quickly every couple of months.

Some of the foundational pieces like the underlying HTTP handler, we just went through one of these things called Hyper, they made a change, which made some substantial changes to the API. And then there’s a whole stack of dependencies on top of it that also change those APIs. And so what that means is from your application’s perspective, you basically need to continually invest time in keeping up with all the dependencies. And I think a lot of the Rust crates don’t, for example, keep a long-term stable release that they backpour stuff to instead they’re just always kind of pushing forward. So that’s been a challenge. It just takes time. I’m not sure if it’s just how this offered it all in changing these days. You continually have to keep up with dependencies, but that’s also I think basically becoming the state of the art everywhere. So I’m not sure it’s as big a deal as it might’ve been a few years ago. If it is.

Rust and WebAssembly [15:08]

Olimpiu Pop: Over the last three to four years, I’ve made it a habit to visit KubeCon Europe during the spring. When I am roaming around, I find myself hearing the same story over and over again. Well, it’s written in Go. So Go seems to be the lingua franca for the CNCF ecosystem and the cloud native ecosystem. Then, at some point, there were some outliers, mainly people who were doing it in C, and the reason for this is that we didn’t need any assistance. We know what we’re doing, so we are confident in our ability to do it. Then, starting from last year, it began to shift. Everybody moved towards Rust.

Additionally, there are two other points I would like to mention about Rust. One of them is Rust, the first programming language that was accepted in the Linux kernel besides the C family, and that’s a huge deal. The other one is WebAssembly.

Andrew Lamb: Yes.

Olimpiu Pop: WebAssembly and Rust are very good friends.

Andrew Lamb: It’s because there’s excellent tooling in libraries around there and because the memory and management are so crucial in WebAssembly. You’re restricted in what you can do with the memory to satisfy its security model. And so it’s relatively straightforward, the model, the restrictions in Rust. Some people have also invested a lot of time in developing great tooling for building WebAssembly. It works well. The other thing I’ve noticed, by the way, is Python, in addition to WebAssembly.

If you look at the Python ecosystem, there’s a massive amount of Python code, but then if it ever needed performance, it would drop down into C, right? What we’re seeing now is many new projects, Daft. There are a bunch of new linters, new V, and all these Python tools. Instead of dropping into C when they needed performance, they’re falling into Rust. It’s a similar story, with development productivity being significantly better. Consequently, the Python ecosystem is another example like WebAssembly.

Testing in Rust [17:22]

Olimpiu Pop: Yes, it’s pretty nice because, looking at modern history over the last 30 years, since things started to catch up and we had bubble after bubble, we had a dot-com bubble and then waves after waves, it seemed that there was always one language to rule them all. Now we have finally reached a point of maturity where everybody chooses their poison. And then you have some pioneers, like Rust in this case, or Python, which scientific organisations widely adopt; well, currently, these are synonymous with data science. But what people said almost two decades ago – that software is eating the world – is happening.

The programming languages are essential, and now it seems that we can live in harmony. It’s like a global Village where Rust is known for its proper way of handling memory. Then on the website, you have quite a few options because everybody can do that, and you can build one step on top of the other. WebAssembly of the species, so things are happening quite nicely, and it’s quite harmonious, if I can say that.

And you mentioned earlier that one of the things on the productivity checklist was the fact that you have unit testing available. What is a basic experience of a programming day in Rust? At times, when I’m exploring ideas rather than sticking to pen and paper and focusing on the architecture, I start coding, and then I’m doing all this TDD. And things evolved. How easy is it for you to do that, or how would you approach the problem?

Andrew Lamb: It’s super easy because you don’t have to add any new dependencies to your project. You just say, “Hey, these are tests,” and there’s a built-in test runner. So, you run cargo tests, and it’ll run your tests. Which is nice, again, that you don’t have to configure that, and you don’t have a choice really. There are other test runners, but people almost always use the built-in one. And then there are built-in assertion macros that you can put in, which will generate nice errors when they fail. So the tests look very, very straightforward. So it’s basically like precisely what you’d expect. There’s nothing fancy, there’s no hidden things. It’s just very, very simplistic. They allow Rust things in the Linux kernel. Now, why is that? Partly it’s because of this memory safety thing. Still, there’s a broader point that’s important, given how software is developed these days: there are giant piles of open source software that underlie almost every modern application.

And when you have a giant pile of C++ code, especially performance-critical ones where it’s manipulating memory with low-level APIs, you need to ensure that a change to those, like the Linux kernel, is right. To ensure that there’s a change to those code bases doesn’t cause memory problems, that there’s not an inadvertent you have to free or bounds override or something. It takes quite a bit of skill on the part of the reviewer. So, to evaluate the memory safety of a patch in C, forget about all the functional stuff and just consider the memory safety aspect. It is quite a bit higher than if you were using Rust, mainly because the compiler will tell you if there’s something wrong. And so the fact that it compiles is already doing a bunch of the work, versus in C, you have to have an expert who knows the code base, understands C, and then can evaluate the patch in terms of safety.

So, it effectively lowers the effort required to review those patches. Therefore, given the most open source projects, the limiting factor is maintainer capacity, the amount of time people have to review PRs. Rust reduces the effort required for a review, making open source projects more effective and efficient. The data fusion feature in InfluxDB 3 is low-level database stuff. There’s a lot of finagling in there with memory.

And if you told me 15 years ago, when I was working on super cost-based database,s that random people from the internet would show up and provide patches that did low level operations that you’d have to spend evaluating and making sure that wasn’t introducing problems is very, very high versus now I feel like anyone can contribute. We’ve reviewed the PR compiles and are confident it won’t introduce any memory issues. It makes open source more efficient, but what that effectively does is open the contribution doors to many more people. That’s my theory. We’ll see whether that’s borne out in practice, but that’s one of the reasons why Rust is so prevalent in the open source world, too.

The FDAP stack [22:07]

Olimpiu Pop: How should I put it nicely? Rust had friends in high places when the US government introduced legislation for safe memory, and now everybody points to Rust. Okay, that’s the solution. That’s the silver bullet. At least that was the feeling at that point, as they were focusing more on cybersecurity. But more than that, you mentioned that InfluxDB 3.0 is around Rust. Paul made the bet, and the company is reaping the benefits. I suggest rewriting the database from scratch and then getting involved in all these communities. This is because Influx data has numerous open-source components that can be utilised. You also had an acronym for that at home, or?

Andrew Lamb: The Tick Stack or the FDAP Stack?

Olimpiu Pop: FDAP Stack. And then at some point, you were also considering adding another I into that, the iceberg in terms of-

Andrew Lamb: Oh, yes. It’s coming. Let me walk you. Let me walk the people through the FDAP Stack story first, and then we can get to the iceberg piece. So as you said, rewriting a database is a very ambitious undertaking, typically undertaken by startups that don’t know any better. Paul knew better because he’d done it at least once before. And Evan, who was the CEO of Influx, also knew how much of an effort it was. Part of the story of rewriting the database for 3.0 was that they had to find a way to do it more efficiently than they could on their own. All Influx employees are writing it from scratch. They knew that was going to take a long, long time. Instead, the bet was that we wouldn’t have to rewrite the whole thing from scratch. We’re going to build on top of several open-source pieces that already cover most of what we need.

They’re commoditised now. And then we’ll just spend most of our engineering effort building the time series-specific components. So, the common piece across multiple vendors is the FDAP stack. So, that’s like the Flight Arrow, Parquet data fusion, because Influx wasn’t the only one looking for a fast standard execution engine, which is what Data Fusion is, right? A query engine or SQL planner. Again, there are a bunch of other people who are interested in that technology as well. By working together from Influx’s perspective, we can lower the investment required to achieve the same level of technology. There are other considerations, too. But from a pure economic perspective, you just think about how you make a new database if you have to build a whole new database from the ground up, totally integrated, something like DuckDB, it’s possible. Still, it’s a significant investment, and you have to have the right people, too.

Leftyb guys are the database world experts out of CWI. So, yes, they can build a new one, which’s great, but can any arbitrary startup muster that kind of expertise and investment? Probably not. But by working together, you can bring together a bunch of different people from various companies that all have the same needs. You can muster the same level of expertise and investment that you’ve only been able to muster before as part of a big, well-funded company. So that’s the story, FDAP Stack. Why is it so important in the story? You couldn’t have rewritten the database with the same staffing level without the open source pieces.

From early versions of Influx, the company aimed to do so, and it’s very common for other open-source companies to build on similar products. The vast majority of the contribution to the product came from the employees of the company. And then yes, they accepted contributions elsewhere, but most of the development effort is driven by the company’s employees, whereas that’s not the case with Arrow or Data Fusion. That development is driven by a lot of people from other companies, and I certainly contribute, but it’s partly because a library is not a product. So the economics and the dynamics of those projects are pretty different from the original InfluxDB 1 and 2.

Olimpiu Pop: Okay. But still, you started the rewrite, and well, it was a significant endeavour, that’s for sure. You were working with a new programming language, a new ecosystem, and a handful of new libraries, because most of them were just getting started at that point. Not everything went very linearly. What are the things that I don’t know, the war stories, what should we know? The kind of things that you are going to tell your grandchildren, your neighbours are talking about.

Tricks for a fast data engine [26:57]

Andrew Lamb: This kind of stuff. There’s a bunch of Rust-specific stuff which we can talk about. I’ll start with the specific Influx database stuff. So the first lesson, which won’t surprise anyone, is that it takes longer than you ever expect. Something else that was interesting was that when we started, there was a whole class called disaggregated databases. This is basically because of the move to the cloud and services like Amazon S3 as the object store, which became the preferred place for durable data. Partly because you outsource the durability requirements to it, and it’s infinitely scalable, and it’s supposed to be cheap. So there was a move in database architectures, and InfluxDB is the same. Instead of tightly tying your data to the local disk, you disaggregate and put the permanent store on S3 because it’s cheaper and more durable.

You don’t have to worry about crashes. And then you have a local copy there. One thing we underestimated when we started was how slow S3 is, especially for use cases that require low latency. For Influx, this was particularly true, as our bread and butter were low-latency queries to drive dashboards. You run a query and you get the results back in 5 or 10 milliseconds. If you’re just running on S3, the S3 request itself takes like a hundred milliseconds. It varies a lot, too, which is another problem.

The point is that if you care about low latency, the fact that you have to go to S3 once means you’re already blowing your budget. As a result, the caching logic turned out to be a far bigger investment than we initially thought. It turned out that you needed multiple levels of complex caching to achieve the same level of performance. That was one lesson learned. Now it’s much better understood that the trade-offs of things like S3 are significant, but it turns out it’s not as fast as you think, and getting it to perform quickly in low latency is quite a bit of engineering effort.

Olimpiu Pop: Looking now at the project. So, FDAP stands for flight, if I remember correctly, Apache Arrow, Arrow, Flight, Data Fusion and Parquet, right?

Andrew Lamb: Yes, yes.

Olimpiu Pop: And then Arrow is written primarily in C++.

Andrew Lamb: Well, Arrow is like a memory specification. So there are client libraries written in a whole bunch of different languages, including Rust. So there’s a native Rust implementation of it, which is what Influx uses. It all comes down to whether you want to build an analytic database with all this low-level, high-tech technology. A variety of patterns have emerged over the last 15 to 20 years in database systems. I know because I worked at Vertica, an early adopter of this technology. People have now realised that the way to build fast analytic systems is by using columns. Instead of putting data in rows, you put it in columns. The reason is that by putting it in columns, it’s much more amenable to the modern XMD architectures. You build this vectorised database and you use column-oriented file formats.

There are a handful of things like that, rather than re-implementing them over and over again. Since the pattern is the same and well understood, it has been implemented through generations of systems. By the way, all the Hadoop stuff uses similar technologies and techniques. If we can build a shared library that implements those pieces, and then people can build specialised things on top of it, that’s a much more efficient way to go. So, Arrow is a piece of that sort of composable data stack in the sense that because you’re representing data in memory, you have to have a representation of memory data to process it, and you need to operate on it. Compare, look for a string, add it together, whatever you’re doing with it. And so, in these columns, you basically have vectorised engines, which are essentially arrays, right? So you have an array of numbers.

Those are the columns that then you need to do computation on. And so the point of Arrow is that the in-memory layout of the data across different systems is pretty similar. There aren’t that many ways to represent an array of integers. But the details are important. What’s the alignment? How do you represent which ones are null or not? And so the specifics may be less critical for performance, but the fact that everyone agrees is important for interoperability and likewise for strings and for whatever.

So Arrow is basically this common memory format. It’s originally designed for interchange so that Python programs could transfer data back and forth between each other without having to copy it and transform it from whatever format. Pandas especially, was one of the big ones. But what actually happened is once people standardised on those memory layouts, you could then invest a whole lot more time in geeking out about making individual operations on those memory layouts much faster.

So there’s been a whole host of effort that’s put into the Rust implementation of Arrow to make adding to arrays really fast, looking for strings really fast. Maybe implementing regular expression matching really fast. I don’t think it’s explicitly part of the Arrow specification, but the fact that it was a spec and that a bunch of people can use it and therefore can also contribute back to this implementation means that you end up with this library of very high-quality and high-performance kernels. Which, again, any individual company couldn’t afford to build all by itself, but together we can.

How to ensure the high quality of your Rust code? [32:13]

Olimpiu Pop: Let me take a step back now. While listening to you, I’ve noticed that many of your conversations, especially when discussing programming languages, often focus on the language itself, as well as the patterns and best practices that have developed around it. And then on top of the best practices, you start having Linters. You have a full pipeline of things that generally go into continuous integration, which allows you to have a trunk that has a given standard. Is that the case also with Rust? Because you said that the compiler is so strong. Do you need anything else besides the compiler?

Andrew Lamb: Yes, so I mean Rust has a linter or like a formatter, right? I don’t know how many of your listeners are old enough to remember the time. I still remember early jobs where we’d argue about how the freaking braces, where the braces worn down lines. At the end of the day, the conclusion I think was no one really cared. Well, some people care, but it really didn’t matter as long as everyone was consistent, that was what mattered. And so since Rust sort of came of age when linters were a thing or people figured out what those were, it just includes one, it has a Rust format one, which is the lexical piece, and it has something called Clippy, which is the best pattern practice kind of things that are built into the language, and basically everyone uses them because already there, Clippy kind of, it’s definitely opinionated. But at the end of the day, you just follow it, you do what it says, and it’s like we just get used to it.

Olimpiu Pop: Yes, this is the Clippy from old Microsoft Office, just changed jobs and companies.

Andrew Lamb: It’s very unclear why they chose Clippy, right? That parallel has been drawn before and they needed a lovable mascot to hate or something because linters always bring that out in people, I don’t know.

Why use Rust for developing a database [34:02]

Olimpiu Pop: Okay. Is there anything else that we didn’t touch upon and that we should have discussed about why Rust and why Rust for the database space?

Andrew Lamb: Oh, yes. Well, I’ll just get on my soapbox for a few moments to talk about Rust Async, because it gets an unnecessarily bad reputation. It’s phenomenal, let’s put it that way, or at least phenomenal compared to everything else that I’ve ever used. So, if you want to do high-performance network IO in any language, including Rust, the way to set that up is to avoid blocking and waiting for the network to respond. You send a message, wait for the response, but then you don’t want to tie up your thread, so you’re busy waiting for the response to come back on the network. So that’s the high-level primitive that you need. Rust implements this with a cooperative scheduler built into the language. It looks like a network call, and there’s a keyword called Await that just works.

But under the covers, what Rust does is generate control, like a return from the function. Returns to the currently running scheduler and will set up a callback with the appropriate marshalled state to call you back when the function is ready. So, from a programmer’s perspective, you just write straight-line code, but what’s happening on the covers is that your cooperative returns control to the scheduler, and then it calls you back when the data’s ready. The fact that that’s built into the language is a huge deal. JavaScript has always had this feature, and it gets wrapped in the API, which is called the Future, a concept similar to promises in other languages. One of the problems with the Rust implementation is that it immediately delves into the details of polling and related tasks, which is a lot of work.

So, it feels complicated at first, but once you understand the alternative, the Rust model is great. It’s so super great. We used it for scheduling, such as CPU scheduling in a database, which isn’t typically what you would use Async for. You’d use Async for network tasks, and then have a separate system for thread processing. Since I’ve implemented more than one thread scheduler in my past life, I’m so embarrassed to admit they’re hard to get right. They’re hard to tune. You get them working 99.9% of the time, but then they’ll have weird problems on shutdown or race conditions, all sorts of terrible things. The Rust ecosystem includes this phenomenal multi-threaded work-stealing scheduler called tokio. You can use that for both the network stuff, which is excellent, and also because it’s so high quality, you can use a separate thread pool for CPU-bound tasks as well.

Because basically a database, you have the same basic idea where you’re trying to do cooperative scheduling, you do some work, and then you’ve got to return control by schedule and let it figure out what the next thing to do is. I’ve been quite pleased we were able to work that pattern into the data fusion code base.

So now all the CPU work is done on tokio thread pools as well. And you have to be careful not to run the same work on the CPU-bound tasks, and the same thread pool is used for the IO-bound tasks, which takes a little finagling. But at the end of the day, the fact that you get built-in support in your language for doing these continuations is, I think really mind-blowing when you think about what it’s actually doing. Well, it’s mind-blowing compared to the alternative. If you have to write it yourself, that’s when you realise how nice it is to have it done it myself a couple of times. It’s great. Anyway, I love Rust. Rust is your thing. It’s very good.

Olimpiu Pop: Yes, it seems like that. So, what I’m hearing from you is that it’s not just a tool, but more of a one-stop shop. The tool chain that you get with Rust is complete, and it has a lot of the stuff that you’ll generally go somewhere else, and it leaves space for tool-building companies. And that even the scheduling side on the thread side, is that good that you can also use it for internal purposes?

Andrew Lamb: Yes. From my perspective, Rust is a phenomenal choice for building performance-critical infrastructure where performance and resource efficiency matter. That’s one of the reasons why it’s suitable for WebAssembly as well. It’s passable, allowing you to build websites and other digital projects. I’ve seen people; there are web frameworks and stuff, but if I were doing that, I would not build web frameworks with Rust. It’s unnecessary. The web frameworks are never CPU-bound; it’s really about managing complexity and the libraries and stuff. So there are a lot of better choices in my opinion than Rust for the application tier. But if you’re building a database or some database-intensive application or some network thing or something where performance really matters because you’ve got to link a million files, Rust is pretty hard to beat.

Olimpiu Pop: Whenever you’re dealing with resource-intensive tasks or situations where you don’t want to make mistakes, use Rust. For our website, for instance, there are other choices because that was also something they had in mind. Many purists take it to the extreme, making a lot of effort to make something look good, but it’s even more complex than it needs to be.

Andrew Lamb: Rust has the low-level primitives to manage memory and control it carefully, but it comes with the overhead of having to think about that a little bit more. Versus when you’re writing a web handler, you don’t care about the allocation of the intermediate object or whatever. It doesn’t matter in the grand scheme of things. Rust has more power than you need for a lot of the internal web stuff. There’s a subset of people who enjoy low-level worrying about memory ownership and who owns what, and they’re making it significantly faster. There are a bunch of people who think that’s not super interesting, and it’s forcing them to worry about that when it’s not relevant to their job. It basically just puts too high barriers. Far more people that write good websites than we care about the low level details that you have to worry about in Rust. But if you’re down in the database and that stuff really matters, Rust really starts shining.

Olimpiu Pop: Okay, great. And my final question, how about documentation?

Andrew Lamb: Yes, I don’t know how the Rust community did it, but they’re obsessed with documentation. And so the actual Rust Project itself is, I think, very, very well documented. There are Rust’s artefacts, the standard library and the compiler and stuff. So that’s all well documented. And then there’s the overarching sort of community and how the Rust community decides what to do and communicates about the status of their projects. The Rust people have really done a phenomenal job of being open. And if you wanted to know what the current status of anything is, you go read pages and pages of stuff on GitHub. And I think actually that’s basically best practice for open source communities these days.

Like the Apache Software Foundation seems to be very popular for many projects these days. I’ve now been involved in it for many years, has very, very few actual requirements on the projects but one of them that really is basically everything happens publicly. So there’s one or two things that happen non publicly, but everything else is public in some way or otherwise. You’re on public mailing lists or it’s on GitHub issues or whatever. And I think that may not sound that important. I think that alone is really critical to building community trust, but also people trusting enough to that they want to get involved because they can see and they could be part of what the discussions, there’s no magic behind the scenes viewing going on. People have full visibility. The Rust organization itself is very, very good at that as well.

Olimpiu Pop: Okay. And how is the decision being taken, because that was also one of the things that I had in mind, that one of the things that was very useful in the Java space, even though it was slower than expected, was the Java community process. It’s very bureaucratic and it moves slowly, but as you mentioned, the community felt that it’s part of it and people can chip in. It seems that Rust has something similar.

Andrew Lamb: Yes, it’s the same. Very, very similar. There’s a process to make, they call them RFCs in Rust, but it’s the same basic idea where whoever wants to make a change writes up why and explains what, and then there’s a community comment period where lots of people can, well, I don’t know the exact criteria for what’s required to get one of those passed. But in Apache, we basically strive for consensus all the time. And I think most people are pretty reasonable about it. Another important thing is you don’t make everyone sign off on it. You just basically give everyone the opportunity to object, which is a very important difference because it means one person’s sort of apathy can’t slow the whole process down. But they also have an opportunity to participate if they actually want to, but they have to put the work in to participate.

Olimpiu Pop: Okay. Thank you for your time, Andrew. Very insightful conversation.

Andrew Lamb: I hope you have a good evening.

Olimpiu Pop: Thank you.

Mentioned: