Transcript
O’Hara: Advanced Message Queuing Protocol, we’ll talk about what that thing is, but Advanced Message Queuing Politics. We had a keynote here, there was a story about the future. Twenty years ago, AMQP was a story about the future. This is the trip report. It’s probably the best way to think about it. It’s a story of people and passions and visions and competitors and friends. It’s all about people making technology. Once upon a time, there was a bank, it was JPMorgan. Hopefully talking about it is ok here now, all these years later. They had thousands of traders trading equities on five continents. This is the type of picture you’d see in trading rooms around the world. That’s not how they do it today. In fact, banks is not where this happens today.
Back then, we had this problem, we had all these people, and making the technology of the 1990s, early 2000s work was really tough. One of the things we’d done in the bank, was we’d built this amazing messaging middleware system that could transport positions and price information to each individual trader and tailored for them what they were doing. It worked really well. It worked over slow networks, networks of 256 kilobits in Hong Kong. Incredible stuff. It worked so well that the engineers got bored of it and they moved on to do something else. This became a risk for the business.
One enterprising chap in the firm said, I know what, I’ll take this software off you and I’ll build a company around it. With that company, I’ll be able to support you. The bank thought, that sounds like a good idea. That’s what happened. The company went off, managed to commercialize the product quite successfully. The CEO was not making as much money as he wanted to make. He looked around at the bank he had left and thought, hmm. At the next renewal, they came in and they asked for an utterly exorbitant amount of money or they’d turn off the trading system. We wondered why they’d spent the last few years building a license manager into it. That is the business case for AMQP.
Background
I’m John O’Hara. I’m a board member at Adaptive. They make trading technology for the cloud. I’m a board member at the company I founded, Taskize, which is interbank workflow, acquired by Euroclear. I’m an advisor to Fidelity and Man Group. I’m a past executive at all these banks. Today, I do technology advisory and also help mentor startups. That’s what I do. Twenty years ago, I was a distinguished engineer at JPMorgan, and I made software. I was really into software.
JPMorgan (Investment Bank)
JPMorgan, walk along the embankment, and you will come to 60 Victoria Embankment. It’s a beautiful building. The front of it is the old city of London Boy’s School. It’s been preserved immaculately. Behind this, there’s a giant office block, a low-rise office block, incredibly high-tech, and it floats in the mud by the Thames. That’s the building, floats in the mud. Quite amazing. In the 1990s, 2000s, JPMorgan had the saying, “More PhDs than NASA. More software engineers than Microsoft”. I worked there. I didn’t know if that was true, but it certainly felt like it was true. It came home to me after 3 years of working with the bank. I stayed there 16 years. It was only 16 years before I ran out of challenges there. It was an amazing place. They were building an investment bank from scratch. The people were luminous. The challenges were amazing. The technology was great. The funding was ample.
Three years in, writing the SWIFT gateway for a derivatives trading system. We put it live. I watched as the first message came up in the system. The first thing it did was make $2 billion of collateral calls. I just stared at the screen and thought, that’s the most interesting console message I’ll ever see, really brought it home. Over the 16 years, I worked in every part of the bank, front, back, every line of business, chief architect across these different lines of business. All of them used middleware. The middleware was arcane. It often, back in 2002, didn’t work at the scale that we were operating at, and we would frequently write our own.
By 2002, I had written three middlewares in addition to doing my job. We weren’t alone. Other banks were doing this as well. Some were spinning them out and commercializing these things. I thought, this is really annoying. If you were to buy a commercial middleware back then — IBM MQ series came to market in 1993, TIBCO EMS came to market in 1997 — the leading products would charge $25,000 per CPU in 2000. You think, what that would be in today’s money. I thought, this is absolutely nuts. This stuff needs to be everywhere. If you get me going over a pint, you’ll hear me rant on about microservices and how much I think HTTP is a terrible protocol for microservices. You should be using real middleware.
The first thing that banks do when they get an external request from FIX, or from SWIFT, or from anything, is they offload it onto a proper middleware. I wanted someone to sort this out. Someone’s going to solve this problem. It’s a really obvious problem. This stuff should be in the operating system. It wasn’t. Someone’s going to solve this.
Message-Oriented Middleware
First, what is message-oriented middleware? The easiest way to think of it is, it’s managed delivery for your data packaged up into messages and delivered to another computer system. That other computer system might be in a different part of your organization. Might be inside the system you’re running. Might be in a different company entirely. Might be in a different part of the world. That’s managed delivery. We all know how lovely managed delivery is with our Amazon Prime. We can watch things move. Like, look where the van is now. My goodies will be here really soon. Look, they’ve given me a picture of my own front door.
Imagine if we had that for computer systems. Can you imagine watching your data go? Look, it’s in the DMZ now. Look, it’s crossed into the other company. Look, it’s arrived at the system. Look, they’ve confirmed they’re processing this thing now. If it didn’t get there, maybe it would retry for me. It wouldn’t come back to me. It wouldn’t be my problem. The middleware would retry for me. If something terrible happened, like a transformer fire taking out something, it would send it back to me and say, something really bad has happened here. Isn’t it amazing that we have better delivery tracking for packages in the physical world than we do for data packets? Isn’t that amazing?
What it looks like, and this was alluded to earlier, is the lines between the boxes. The lines are the important part. The lines are the difficult part. If you look at a large organization, it’ll have thousands of systems. I think JPMorgan’s current system count is 6,000. Bank of America, 4,000. These systems aren’t uniformly spread around. They’re spread around in different islands of politics. Those political boundaries are really important, because they’re the parts you don’t control.
The system you’re trying to communicate with is owned by someone else. When you go to build, when you go to construct one of these lines, you’re not doing it yourself. You have to negotiate with someone else. You don’t get to choose their technology. You don’t get to choose when they build it. These are the things that we don’t really think about very much as developers. You discover about it when you try and schedule the change. They go, yes, we can do that in 8 months’ time. That’s a real problem. It’s this inter-business. We forget that we’re supporting businesses. Someone’s paying us these salaries, and they’re making more money to do that, and intra-system connections. A big part of our job in IT is to manage these connections. The biggest pain in the head is managing these connections. Wouldn’t it be really good if your operating system came with software that made this really easy? It doesn’t. The nearest thing we have is, bizarrely, Secure File Transfer Protocol.
If you know how that works, you know how really weird and bizarre that is. It’s a bit fragile. It was never really meant for this. People say, “No, John, HTTP APIs”. Lovely idea. The problem is, you have to build too much, and they have to build too much. You go back to what I said earlier. They have to build something. Someone has to pay for those expensive engineers to build it. You have to agree what it is. You’re going to work out a protocol, yes? That’s expensive, slow. You’re not going to get it done. I remember talking to the chief technology officer of my startup, and he was very keen that we’d have an HTTP API. He just went, “Mark, the clients aren’t going to use it. They’re just going to want a file”. Build it. Build it anyway. “We talked to them, they want a file”. Every client wanted a file, because they don’t have the time to build them.
We have solved this now. We are living in the future. Your AMQP is ready. You have seven different implementations of AMQP that you can install with a simple docker pull command. All of them are industrial strength. Some of them are open source. All of the clients of these products can talk to all of the servers without a recompile. That means if I choose to run RabbitMQ, and you choose to run Microsoft Service Bus, we actually don’t need to know. We just need to know we’re doing AMQP. My RabbitMQ will talk to your service bus across the company firewalls. That will work. That’s pretty amazing. How long has it taken to get here? That’s what this is the story of, how we got here. I’ve got a timeline.
Along the bottom, there’s a progress bar. It starts at 2002, which is incredible. It’s older than QCon. It’s older than AWS. It’s older than me. On the right, there’s today. Everyone knows this XKCD cartoon. We all need one standard, one ring to rule them all, and standards plus one. We’re all very aware of this cartoon and how true it is. I really didn’t want to do this. Someone’s got to fix it. I don’t want to do it. I’ve been working on something called FPML, the Financial Product Markup Language. It was a way of describing financial derivative products. It’s really good. We started doing that in 1999. It took 8 years to make FPML version 5, major version 5. That means every version in between was completely incompatible with the version before. Tons of arguing inside our company before we went and argued with other companies. We argued a lot and built protocols that we didn’t like, that were too complicated.
Eventually, we got to FPML version 5, which is really good. It really is. If you want to learn how derivatives work, it’s a fantastic way of learning. I knew how painful this was. I knew how passionately people get involved in this stuff. We like complexity. We get into the problems. We really wrestle with them. Our way is right. Their way is wrong. Solving the problem is easy. Agreeing to do it the same way, that’s really hard. You can write a middleware, it turns out, in a week. It takes a lot longer to get everyone to agree to do it the same way. That’s where the value is. I really didn’t want to do this, but we did.
AMQP Begins
In 2003, the story at the start actually happened. That story really illustrates why you need standards. If you’re exposed, dependent, beholden to a single supplier of a technology, the temptation for that supplier to turn around and extort you becomes ever greater the more successful your application is. Do multi-cloud. I’ve seen this, and I’ve worked in a lot of places, at least 60. Don’t try and guess where these are. I’ve personally been in a room where financial extortion by a vendor has happened four times.
The largest number was $200 million. It’s very expensive not to have standards, as that first story is illustrating it. This is why you have AMD to go with Intel. The U.S. Department of Defense didn’t want a single supplier. They had seen this game before. AMD was able to do x86 as well because of that. Here, we had to solve a business problem. The business negotiated with the vendor really hard. They managed to secure a 3-year extension at great cost. They resolved never to pay the vendor another cent. They were really angry. This had come from the inside of the company. They were properly angry. We had to get this thing out of a dozen systems, and we had 3 years to do it, and a very hard deadline. The business didn’t want to do it. Here we have this opportunity.
First, a lot of the systems could just be moved to market data systems. Standard market data systems were working really well now. This is 2003. TibRV was a thing. The Reuters Feeds were all working. It was great. We could just move a whole bunch of the systems straight onto that. That one system with 1000 traders in 5 continents, the way it worked, it couldn’t just be moved. It needed something special. It needed a specific solution. This represented a really rare opportunity for a business to fund a technology advancement. I made my pitch. I said, we can do this.
The senior manager, rightly wary, they thought, is there any other way? We really don’t want to give you technology guys actual money to write stuff. We’ve seen this game before. It’s high risk. We’ve got 3 years. It’s going to be a literal drop-dead date. They thought, can we negotiate even harder with the vendor? They tried that and they realized that, no, the vendor knew too much about us and they thought that they had us completely. The second thing was, can we turn the system off? Do we need this? We’re about to stop trading equities this way. Program trading, auto trading, high frequency trading, these are all coming. No, we still need this. Can we bypass the license manager? Funny you should say that. My people tell me we can, but no, we’re not going to do that. We’re not going to do that, which says something actually about the firm. We decided to do it. JPMorgan was going to commoditize middleware as a side effect of a business problem.
Just let this sink in. This is Hursley in Winchester. It’s a stately home. It happens to be the home of IBM’s mainframe business and IBM MQ series. MQ series was a $2 billion business, I think. I don’t know for sure, but I think it was $2 billion business a year back then. Like you do, we had an enterprise visit to IBM. The JPMorgan people and the IBM people, we got together and talked stuff. There were 10 of us down one side of a table and 10 of them down the other side of the table in this grand salon. Imagine it. We’re chit-chatting through the day. Halfway through the day, my boss turns to me. He’s a chap called Mark Etherington.
Mark turns to me and says, “John, tell them what we’re going to do”. I look at Mark, telepathy. He goes, “Yes, go on”. I turn to IBM, and just imagine this, I just say, “We’re planning to commoditize middleware”. That was the extent of my presentation. There were youngsters, the young engineers in the room. There was silence first, dead silence. Then smiles broke out among the younger engineers. Then looks of grave concern broke out among the older engineers. That is the only thing I remember from that day.
I’m really going to do this, this thing I didn’t want to do. I had to have a little chat with myself, because I’d seen this rodeo before. First, I’m technical. I knew what I like. I’m going to have ideas. My ideas will probably not survive this process. I am going to be ok with that, I told myself.
The second thing is, the mission is to commoditize middleware, not for us to win. That’s a really interesting difference. If we commoditize middleware, but AMQP is not the answer, that’s ok. That’s a really important thing. The next one is, I knew technology guys were going to get passionate. They were going to give of themselves, give of their souls to this process. They were going to give exorbitant amounts of time, above and beyond the call of duty. They’re going to get really wrapped up in it. I resolved that contributors should be able to leave their mark. Put their fingerprints on it. Go, that’s mine. It’s a little something. Leave a mark.
The last one, I really love technology. My job is to make this happen. To keep it alive. That’s what my job is. Who else cares about standardizing middleware? Who else is that silly? Who else works in an organization that actually knows how to do this stuff really well? Telcos, the actual software vendors, and banks, and only the big ones: JPMorgan, Morgan Stanley, Deutsche, Goldman’s. We know how to do this. Who else does? Who else can mobilize a million dollars a year to make it happen? It’s an unusual set of qualifications. My job is to make it happen.
AMQP was just a side effect of the business project. I was going to call it, A Message Queuing Protocol. We all like our IT jokes. A Message Queuing Protocol. Has to be four letters: HTTP, AMQP, SMTP. Has to be four. It’s the industry standard. FTP, I’ll ignore that. SFTP, that’s why we have SFTP. SSH, I ignore that too. We had a clean room team. Clean room team? We had a messaging team. We had an integration team in Glasgow. We had our test migration team in India. There were 25 people involved in this. At the end of this, we planned to roll this out to the 1000 traders on 5 continents without a hitch. That was the plan.
iMatix and OpenAMQ (Patent Risk)
Patent risk. Does anyone remember being afraid of software patents? You’ve got the Alice Corporation ruling in 2014 to be thankful for, that we’re no longer afraid of software patents. Most software patents, unless they’re bound physically to hardware, are now essentially irrelevant. A business process patent can’t be held. If anyone ever has the opportunity to stop software patents coming back, because people are trying to bring them back, make sure they don’t come back. They’re hell. My point of view may be clear. The patent risk was really real in 2002. We worked out that with patent risk, you’re sued really for making. Making is the bad thing. Making is what’s going to get you in trouble. People who are patent trolls, they go after the guy with the deepest pockets.
Being called JPMorgan doesn’t help at this point. We resolved that we’re going to specify this really well. We’re going to draw on the expertise of the firm and specify this really well, but we would not make it. We’re going to pay another company to actually make it, following that specification. I went and set out to find a company that would be good to make this stuff. I found iMatix. I googled and I found iMatix. I liked Pieter Hintjens’ work. Pieter Hintjens did work on code generation, on state machines, on web servers. He knew middleware. He really got the vibe. I loved his communication style. He loved the vision. We decided to work together. I need to say at this point that Pieter is no longer with us.
Very sadly, he passed away well before his time. He was one heck of a character. Pieter made the first non-JPMorgan contribution. The first fingerprint was Pieter’s. Pieter said, “It must be exciting. We can’t call it A Message Queuing Protocol. It must be the Advanced Message Queuing Protocol”, and so it was. That’s how it got renamed. I contracted iMatix to build the broker in C. Why? You’re all going, why? I wanted to make this an operating system solution. I wanted to make it palatable to those guys in Linux land that apparently have issues with these other programming languages. I thought, we’re going to do the hard one. We’re going to do the difficult one.
Then other people can write the easy ones, Java and stuff. C is so hard. Just build your own bricks, why don’t you? I committed to sign co-copyright of the software to iMatix if we went live successfully. We pay you to build it, and if it goes live well, we’ll give you the software. We’ll pay you for support contracts. We want that business to succeed. We want this to work for you.
ActiveMQ
Right about this time, ActiveMQ appears in the scene. ActiveMQ may have been the first commercial open-source middleware, and built by James Strachan. Ironically, James Strachan had actually worked for JPMorgan at one point, and had actually made a commit to the system we were upgrading. The world’s a small place. James had just come from making the Groovy software language. He’s a really talented guy. I thought, this is fantastic. Maybe I don’t have to do this. Remember, I really don’t want to do this. Maybe I don’t have to do this. I met with James on Fleet Street on a sunny day. We sat down, had a chat. I tried to convince James that standards were the way. James didn’t see the value. James just said, just use ActiveMQ, commercial open source. The commercial imperative to be sticky, to have a USP, to be differentiated, to be able to sell.
This is the fundamental tension between standards and vendor commercial proprietary products. When AMQP started making mindshare, James reacted. He published OpenWire, which was essentially a dump of the symbols from ActiveMQ. Call it protocol.
2005 to 2011 – AMQP’s Journey
We’re still operating in secret at this point. We’re doing everything under NDA. I’m running around talking to other banks. I’m running around talking to all these software companies, IBM, Cisco, all manner of companies, Sonic Software, Solace Systems, 29West, all manner of companies. We’re good at doing that. I have this guy working for me, a guy called John Davies. John’s a serial entrepreneur. People undoubtedly know John. A big character. He was between entrepreneurial gigs working for me. He thought AMQP was the most interesting thing that the bank was doing. He went to New York one year and went to the Web Services on Wall Street Conference in 2005, and he just couldn’t help himself. He blabbed.
In fact, he more than blabbed. He stood up and told everyone. John’s a big guy. He’s got a lot of presence. People sat up and listened. The press were in the room. This is one of the Finextra, “JPMorgan promotes open-source messaging system”. John’s name’s in there. I was annoyed. This was quite a fragile project. We were doing it on the Qt. We were worried about patent risk. I went, this is not good. It was good. Because in that room was David Ingham. David Ingham worked for Microsoft. David thought, that sounds interesting. Also, John Davies, was friends with Alexis Richardson. Alexis was the founder of RabbitMQ. That’s how Alexis learned of AMQP. John, all is forgiven. I think what he did that day was very important to this journey.
2006 was one heck of a year. In 2006, we had AMQP 0.9, a 130-page specification that was really good. I thought that. iMatix were in the middle of coding with C, and they had decided to do it in multithreaded. What’s worse than C? Multithreaded C. Read Rust to understand what’s going on there. They were having some problems. This thing was not passing acceptance testing. Months went past, and it just kept locking up. If you program multithreaded C in the early 2000s, Linux didn’t really do threads properly. Lots of things didn’t work. Today, everything works. Back then, nothing really worked.
My project manager, Steve Connolly, was really getting quite concerned. We sat down and thought, we need a plan B. It’s time for plan B. We have this super hard deadline. Plan B. One of my guys in Glasgow, a chap called Robert Greig, had written the JMS API for AMQP. We were doing the Java stuff because easy. They were doing C stuff because hard. He’d done a really great job on the JMS API. I went to Robert and said, could you turn the JMS API in Java into a complete broker server implementation? He went, I’ll have a go. Two months later, he had done it. He’d done it really well. Robert is a talented guy. OpenAMQ was fixed at that point.
The C one, which was called OpenAMQ, got fixed. It passed its tests in May. We went live in November. The business project is a success. We roll this thing out to 1,000 traders in 5 continents. We have this ability to do a slider: old middleware, new middleware. We had bridging technologies and all sorts. The clean room stuff I talked about earlier was because of the litigious nature of the whole relationship, we had an independent team analyzing what the system did and creating specifications for us to map it onto. It was a big deal. It’s a huge project. We were able to just go slide. The business didn’t skip a heartbeat. It was a tremendous success. The execs were really happy. Because of that, I was really happy because we were able to assign the co-copyright, not exclusive copyright, we had learned that one, to iMatix for OpenAMQ. That was a win.
Lots of stuff happened in 2006. 2006, we came out, we went public. We stopped being secret. The AMQP Working Group, which lasted from 2006 to 2011, started in June 2006. JPMorgan, Cisco, it’s a great quote here, whenever another large middleware company learned of Cisco’s involvement, they called up the guy and said, why are you doing this? Went, you’ve had a good run, but it’s over. iMatix involved IONA, which is now Progress, Red Hat. We had this legal structure. It was a cooperative arrangement. There was no legal entity. No legal entity. It was a whole cooperative contract structure that also had a big amount of licensing patents involved. We had defensive mechanisms built into this.
Red Hat and Cisco had such huge patent portfolios, we felt we were wrapping ourselves in safety, because this is still a problem. 2014, we’re not there yet. Patents are still a thing. The Linux guys were suing each other. Andy Caddel, the chief intellectual property lawyer at JPMorgan helped us do this. It’s a really fantastic set of arrangements. It’s worthwhile looking at the agreement structure. We would meet every Wednesday. Come hell or high water, that call happened. Over time, we grew and grew.
Still 2006, what a hell of a year. Red Hat, Carl Trieloff was IONA, moved to Red Hat. Carl wanted to give Red Hat a middleware story. Enterprise company, middleware. He talked to Pieter at iMatix to acquire AMQP knowledge, because Red Hat didn’t know anything about this. It was moving. We’re kind of part of it, but we’re not part of it. We’re not really in it. We need those skills. How do we get them? He talked to Pieter about some sort of arrangement, and Pieter declined. Pieter declined to partner with Red Hat. He said, we’ll go alone. That was his prerogative, his right. It really upset me. I thought, I want OpenAMQ in Linux.
I like the idea of installing Linux and getting middleware. You can see where this is coming from. It was unacceptable to me that Red Hat did not have skin in the game. We took Blaze, which is what Robert had called our broker. It’s still used. We took Blaze and we licensed it to Red Hat and IONA very discreetly. Red Hat took that and championed it into the Apache Software Foundation. Blaze was a highly trademarked name. Apache said, we can’t have that, rename it, so we called it Qpid.
Apache Qpid came into existence in the incubator in 2006. Became a top-level project in December 2008. It’s very widely used. It’s not got anyone really pushing it, but it’s used all over the place. There are over 40 systems in JPMorgan running Qpid. It’s bulletproof. It’s the Switzerland of AMQP, Qpid. A neutral ground. It turns out Rafael Schloming at Red Hat, one of the engineers there, he really had a thing about working on projects that had Star Trek themes. Qpid is named after Q, this character here, who’s from Star Trek: Next Generation. He’s one of the villains. He’s an omnipotent character. Rafael got his fingerprint on this by naming it Qpid.
Still 2006, one hell of a year, RabbitMQ begins. Alexis Richardson, he’d wanted to build a financial messaging product, and he’d learned about AMQP’s existence from John, from earlier, John Davies. John had gotten a copy of the specification. Alexis was working with a company called LShift. They were a boutique consultancy in Hoxton, here in London.
Matthias Radestock and Tony Garnock-Jones, they were doing the prep work for this whole thing. They were looking at Erlang, and thought, this is really interesting for writing distributed systems. Whenever Matthias saw the AMQP spec, he got the joke, because the AMQP spec is beautifully formatted PDF document, but the source code for it is XML. They realized they were able to code generate a huge amount of implementation directly from the specification. They had a broker put together on Erlang really quickly. In February 2007, RabbitMQ is released to the world. Because it’s leveraging all of Erlang OTP, it’s immediately fast and robust, and it gets this reputation for just working. Alexis and the team were amazing entrepreneurs. I watched them doing developer evangelism. They really ignited passions around this thing.
If it wasn’t for them having chosen AMQP, and them having been so successful with developer evangelism, I don’t think we’d have got as far as we have got. Rabbit was a really big part of the story. Their developer education, everything, the passions people had. To a lot of people, RabbitMQ was AMQP, and AMQP was RabbitMQ, a lot of people. That was interesting.
A machine-readable specification. One of my contributions to the project was to make the spec machine-readable. We had this fanatical thing about making the spec extremely complete and unambiguous. I’d seen ambiguous specifications before. If you look at the IETF and some other places, you’ll see specs. You go, how do you build this thing? There’s just not enough information. I think that’s deliberate. I think some vendors half-specify things to make it impossible for competitors to know how the secret sauce works. That’s not useful. I had actually tried to implement one of the CORBA Event Service one time, and I thought, how does this work? I’ve discovered that no one had ever built it before. We resolved not to do that either. This spec has been built multiple times. Inside it was kind of an IDL.
Leaf through the XML that’s used to generate the PDF, there’s all this data that you can use to write a program to read the spec, to generate code in the language of your choice. You’ve done half the work in a day of making a new client for AMQP. The community around Rabbit really seized on this, because they liked their eclectic technology. A Go client appeared. Everything appeared really quickly, because the community got the joke. Another thing I’d done with it, that I’d insisted in the protocol was to make it asymmetric. It was much easier to write clients than it was to write server, and that really helped.
2007, you see our progress bar. It does get faster towards the end. 2007, acmqueue reached out and asked me to write an article for them, which was great, because it added an air of enhanced legitimacy to what we’re doing. That was the second most cited article in that particular queue. This is an interesting one. I’m sitting at a desk in JPMorgan. It’s the afternoon. My desk phone rings. Desk phones never ring. Why do we have them? In the year 2000, Cisco was the largest telephone handset manufacturer in the world. We all had one. They never rang. I pick it up, go, “Hello?” “Hello, John”. You know an awful lot about me. How’d you get my number? “The Department of Homeland Security here”.
These are not words that strike warmth and coziness into your heart. I thought, what have I done with my visa? Did I leave the little green slip in the visa going to New York or something? They went, “No, we just thought we’d call up and tell you we’ve decided to standardize the Customs and Border Patrol protocols on AMQP”. They processed $2 trillion of revenue a year. I think with Donald’s changes recently, it might be more than that now. They were really big on this. Wolf Tombe, the CIO of the Department of Homeland Security, CBP, they joined the working group and they really were very active participants, very good participants.
Still, 2008, Microsoft joins us. It took them a while, but they join us. David Ingham, the guy in the audience at that conference all those years ago, became the editor of the AMQP specification. It was a neutral role and he performed it fantastically well. Very well liked. We didn’t see much surface area from Microsoft. Our website traffic to amqp.org quintupled the day they joined. They’ve got their own gravitational field. We had the sense that something was happening behind David. They were doing something, but we couldn’t tell what it was. 2008, we sit back in my chair and go, have we just won? Has this worked? Did we actually achieve everything we set out to achieve? Rabbit helped us correct a few errata in AMQP 0.9. We got 0.9.1, which is a very robust and delightful protocol. We have these multiple high-quality AMQP 0.9.1 implementations. It’s being rapidly adopted, but it’s not a standard. It’s just our little project still with lots of people, but not a standard.
If you’ve started a company, and Alexis inspired me to start my startup. Turned down a job offer at Goldman’s, and I started my own company, because life’s too easy. Watching him, that was really inspirational. If you start a company, VCs have this phrase, default dead. You have to come into the office every day and make your company be alive. You’re willing it into existence. The same was true with this. It’s default dead. Every day you have to come in and will it along. People come along throughout because they see you’re dragging it from the front, and they’re clapping. It’s fantastic, keep going. Then one day, it takes on a life of its own. You might not even notice. It’s actually quite hard to notice when something stops being default dead. I realized that AMQP was default alive at 0.9.1. In particular, the AMQP Working Group, their work was not yet done.
The first breaking change. The working group, the experts that were joining, really hardcore experts, they wanted more. They proposed, they wanted a much stronger session transport. It was the first breaking change. They terribly named it 0-10, because it looks like one. Nobody liked this version of the protocol. The people who used 0.9.1 didn’t like it because it was a breaking change. 0.9 was fine. The people who wanted more didn’t like it because it didn’t go far enough. We ended up with this terrible secret that confused the heck out of everybody, we had two protocols. You think, they just evolve. No, from 0.9.1 to 1.0, yes, it is true, it changed completely. This was a massive thing.
It went from being broker-centric, asymmetric, competent, easy-to-write clients, strongly defined queues, impossible to retrofit onto existing products, and we kind of like that. That was a passive-aggressive thing, my point, I think. It was user-friendly. The 1.0 draft was brokered or brokerless. Symmetric, world-class transport. This thing smells like InfiniBand. It’s really hard to read. I’d recommend you look at Clemens Vasters’ documents at Microsoft? He’s got some fantastic descriptions of how the protocol works, but it’s awesome. It’s properly awesome, and not at all easy to understand. Very hard to write clients, loosely defined queues, and therefore much more easy to retrofit to existing products. It’s vendor-friendly. Exchanges are gone. People knew AMQ because, it has exchanges, and no one understood. They’re gone.
In our San Diego offsite in 2009, I stopped and took this picture. I just thought, we have maybe the brain’s thrust of middleware, with the exception of the two largest firms in the space, in this room. We have 29West, we have Solace, we have Goldman’s, we have Credit Suisse, we have JPMorgan, we’ve got Red Hat, we’ve got Cisco. It was like, what? They’re in one room together, and the protocol looked amazing. Great work. The public day we had, a whole bunch. This is San Diego. It’s the middle of nowhere, really. A lot of people flew in to see what was going on, and they approved, they liked what they saw. This is a terrible place to have a conference. Matt Arrott was working with the National Science Foundation, Oceanographic Survey Initiative, and they were using AMQP in submarines and stuff like that. They funded Scripps Institute being able to host the conference that year.
The people involved in this all really liked each other. It was a fantastic camaraderie. We became really friendly. A lot of us became friends, mostly. iMatix goes its own way. Pieter wanted AMQP to stop the 0.9.1, but it was still evolving. The community was taking on, it was out of any one person’s control. I think Pieter thought, I control it, but this had gone out of my control now, remember? I’ll not be upset if it’s not my ideas. iMatix had sales, but RabbitMQ was winning. He got angry and went in the opposite direction to something he could control. 0MQ was born. 0MQ is a lovely product, and its particular niche is in Jupyter Notebooks and all sorts of stuff. The result, 0MQ, the clue is in the name: no queues, no broker. Pieter loudly objected to AMQP’s evolution, and actually used that railing against us to publicize 0MQ. Eventually he abandoned OpenAMQ as well whenever our support contract expired, which is unfortunate. That died. It’s funny, you see products succeed, products dying.
Rabbit, the succeeder, gets even stronger. In 2010, they’re acquired by SpringSource, which is amazing. Then SpringSource is immediately acquired by Pivotal Labs. This is becoming a multibillion-dollar organization. Pivotal Labs is ultimately acquired by VMware. This really burnishes Rabbit’s enterprise credentials, strength to strength.
At this point, IBM notices that all is not well in the middleware world. This was not the plan. We could hear whisperings. They’d never say anything publicly, but we could hear whisperings on the back channel. They released MQTT. MQTT is super simple, super easy to use. It really is genuinely good for IoT, Internet of Things applications. It’s definitely not an MQ series competitor, which means it’s not an AMQP competitor. It was thrown out there to clear the pitch, to distract, to take away attention from us. It’s light. It’s sexy. It’s cool. Internet of Things. I have it on good authority, this was deliberately done to do that. It started tracking the same process as us, as AMQP. It started tracking OASIS. It started tracking Apache. After 16 years of having a whale of a time at JPMorgan, I get headhunted by Bank of America, and they join the working group.
Now we have three of the top five U.S. banks are members of this little AMQP Working Group. Matt Arrott becomes co-chair, because I’ve got this new job, I’ve got to really knuckle down and do things, the actual work. JPMorgan starts to transition Qpid towards Red Hat.
Standardization of AMQP
We now have this period where we’ve got a specification that’s really good, and no products to implement it. We’ve got tons of things to implement 0.9.1, but that’s not what the working group wants. The working group wants 1.0, and no product implements it. We’ve got this 2-year wait while engineers meet in rooms around the world, around tables huddled with laptops and coffee, trying to make their software work together. Lots of connect-a-thons.
In the meantime, while they’re doing that, we get to decide, because we never meant the AMQP Working Group to be permanent. There’s no legal entity. We get to decide where we want to land the standard. Where’s it going to go? We need two independent implementations to qualify. Where’s it going to go? The Internet Engineering Task Force, super cool. A bit anarchic. ISO, super cool, and the other way, quite inaccessible. W3C, webby stuff, not us. No, really, just wasn’t us. OASIS. OASIS is a really nice standards body. We liked them, and we liked their processes. Really easy to work in their construct. Also, they were affiliated with ISO. There’s this chance that maybe we could follow the path up to the big standard. In 2011, we have the launch party. This was at 270 Park Avenue.
If you know New York, this is the building they’re rebuilding. This is the giant edifice that’s being rebuilt in the middle of Midtown. There’s 100 guests there. We had lots of companies, lots of vendors. Software AG, Red Hat, Microsoft, Rabbit, stood up and made commitments in public, attestations of support to the one true protocol. They had running software products. We had live demos in the room that day. Real products, more than two, eligible for Standardization with a capital S. This is who was in the Interop demo: Microsoft, Apache Qpid, RabbitMQ. StormMQ aren’t with us anymore, but they were great people in that. INETCO, who are good friends of mine. Angus Telfer there did a whole bunch of stuff just because he was passionate about technology.
Time passes, 2012, a year passes as the wheels of standardization grind. Nothing changes about the protocol. It doesn’t change at all. We get rubber stamped. We are an OASIS international standard. It just took 10 years. This is 10 years from the starting point. Remember I talked about how difficult it was to write the transport in AMQP 1.0. Apache Qpid ended up being the parking lot for lots of software to make it easy for people to write that transport. Now, we decided to go for ISO. Microsoft helped us. That’s a difficult process because countries vote on whether you should become a standard. A majority of countries have to agree. As an interstitial, it’d be interesting to think about the participation rates.
At the end, 60 people were involved in making that international standard. Participation had grown in every single version of the specification. In the end, 91 people have been credited in some version of the AMQP specifications. There are hundreds more who took part in this. I really want to thank them. We couldn’t have done this without all those people working together. It’s not an engineering problem. It’s a people problem. We become ISO 19464. ISO has a very strong gravitational pull. All the products converge. It takes a while. All the products converge. Eighteen years after that meeting in Hursley, all the products converged. I didn’t notice, it was done very quietly. MQ series 9.3 is AMQP 1.0 compatible. It made me smile. Mission complete. Here are some numbers you need to know.
RabbitMQ is the most widely deployed AMQP implementation with 54,000 customers and 1 billion downloads from Docker Hub. Microsoft have the largest AMQP 1.0 infrastructure. They process 10 trillion messages a day on service bus. They’re all AMQP 1.0. That’s what they were doing quietly in the background. They architected the entire thing around AMQP. It’s amazing.
Lessons Learned
What did we learn? A dedicated leader is necessary. I didn’t expect how much this would be a people thing. Everyone needs a business case. It’s not obvious. You need a business case. Every individual needs to know why they’re there. If they don’t have that, they won’t be able to stay the course. It will take longer than you think. Get the resources. Find a standards body earlier than we did. Specify comprehensively. You will not regret it. A lack of backwards compatibility can be fatal, but it didn’t kill us. A product is not a standard, James. Anybody else? Don’t formalize until things work as expected. Vendors and users have fundamentally different needs. The needs from a vendor and a standard are different, but you have to accommodate them and balance them. That’s what we did. We had AMQP, ISO 19464, world-class messaging for everyone.
See more presentations with transcripts
