A Rust Framework to Simplify Distributed Systems

A Rust Framework to Simplify Distributed Systems Building software that runs across many machines is notoriously difficult. Developers have to grapple with problems such as race conditions, partial failures, and message ordering. Notably, one category of distributed software has largely escaped these burdens. A distributed database can spread a single query across thousands of machines, handling the coordination, failure recovery, and ordering internally. This raises a natural question of why general-purpose distributed programming can’t feel the same way. This is a highly practical problem at AWS https://aws.amazon.com/, because the reliability of cloud infrastructure depends on getting distributed systems right at massive scale. Joe Hellerstein https://x.com/joe_hellersteinspent thirty years as a database and distributed systems researcher at UC Berkeley, where he pioneered much of the foundational thinking on applying database ideas to distributed programming. He is now at AWS, where he works to bring his research into production through Hydro, which is a Rust framework to bring declarative queries to general-purpose distributed programming. In this episode, Joe joins Sean Falconer to discuss how ideas from the database world could make distributed programming dramatically simpler and safer. Sponsorship inquiries: [email protected] mailto:[email protected] The post A Rust Framework to Simplify Distributed Systems https://softwareengineeringdaily.com/podcasts/a-rust-framework-to-simplify-distributed-systems/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

• 0:00Speaker 00

  Building software that runs across many machines is notoriously difficult. Developers have to grapple with problems such as race conditions, partial failures, and message ordering. Notably, one category of distributed software has largely escaped these burdens. A distributed database can spread a single query across thousands of machines, handling the coordination, failure, recovery, and ordering internally. This raises a natural question of why general-purpose distributed programming can't feel the same way. This is a highly practical problem at AWS because the reliability of cloud infrastructure depends on getting distributed systems right at massive scale.

• 0:42Speaker 00

  Joe Hellerstein spent 30 years as a database and distributed systems researcher at UC Berkeley, where he pioneered much of the foundational thinking on applying database ideas to distributed programming. He is now at AWS, where he works to bring his research into production through Hydro, which is a REST framework to bring declarative queries to general-purpose distributed programming. In this episode, Joe joins Sean Falconer to discuss how ideas from the database world could make distributed programming dramatically simpler and safer. This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

• 1:34Speaker 01

  Joe, welcome to the show.

• 1:36Speaker 02

  Good to be here.

• 1:37Speaker 01

  Yeah, thanks for doing this. I'm excited about it. You have a very interesting background. I don't think that we'll even be able to cover everything, certainly, that you've done in your career, but I think we have some interesting things to jump into. You spent 30 years at Berkeley, working on databases, distributed systems, now you're at AWS. I guess, like, how would you describe this thread that has connected your research, maybe going back to the 90s, all the way to what you're working on today, which is Hydro, which we'll start to talk to about later?

• 2:05Speaker 02

  Yeah, well, I think, just to keep it focused, because you work on a lot of things over 30 years, I'll pick the through thread. Like you said, I was trained up as a database person back in the 90s, actually right out of college. I went to IBM Research in the very early 90s and got baptized in the database religion by the IBMers and then by Mike Stonebreaker at Berkeley, who was my master's advisor, and then by the Wisconsin team, which was the database powerhouse at the time, Dave DeWitt and others. So that was my training. And one of the things that was cool about databases was that you could write a query and it would run on any hardware you liked. So today, you can run an SQL query and it'll run on an embedded device like your phone, or it'll run in the cloud across thousands of machines and it's the same query.

• 2:47Speaker 02

  And something that really bothered me as a software person was that programming wasn't like that at all. Programming computers was somehow different, and every time you got a new piece of hardware, you had to start over. And particularly, if you wanted to run something that wasn't on a single node but was on lots of nodes or parallelized, distributed, call it what you will, you had to start from scratch and programming languages weren't there to help you particularly. So I've been wrestling with that gap between declarative queries and imperative programming during the course of my career and trying to narrow, narrow, narrow that gap until we can really talk about programming the cloud, all the computers in the world, if you will, in a way that's portable, just the way we write queries, they're portable.

• 3:27Speaker 01

  I mean, I think that's a really interesting point because if you think about, if I'm running a query against some massively sharded database that's running all over the place, like, as the actual user writing that query, I don't have to think about any of the infrastructure, I don't have to think about the configuration of that, but suddenly, if I really need to run my application, all that complexity is thrown in my face and I have to just deal with it. And I guess there's some bypasses out there that try to hide some of the subtraction to you, but it's really, really hard to get away from, especially as applications tend to scale, more and more of essentially what's under the covers becomes probably, maybe it's 90% of essentially the engineering time is just working on that platform.

• 4:09Speaker 02

  Yeah, I think that's the apologist's response to my question that I asked 30 years ago. It's like, why must it be so much harder to write general-purpose programs than queries? And the apologist says, well, there's a lot of details in there, man, and you have to get them right. And I'm like, well, what details? Because that's what I'm asking. I'm exactly asking that question. Let's not hide this under the rug and say it's too hard. Let's stare at this thing until we can tease out all the little bits that we care about controlling as software engineers, right? That databases are glossing over and making assumptions about. And in studying that, we can bring more and more of this stuff to the point where the engineer maybe only has to specify that stuff when it really, really matters for performance, or maybe not even that. Maybe actually compilers should be better at this than engineers.

• 4:54Speaker 02

  So I really like to leave this stone not unturned.

• 4:57Speaker 01

  What do you think led to us being able to do this with databases but struggling to do it with general programming languages?

• 5:04Speaker 02

  Yeah, so if you roll me back 20 years and ask me this question, I would give you a sharp answer for that. I'd say, well, databases very wisely chose sets as their fundamental model. So a table, a database table is a set of rows. The ordering is not dictated. And the programmer knows that when they write an SQL query, unless they say order by, they'll get their output in some random order and they're cool with that. And I would have said a bunch of years ago that everybody else was getting it wrong. We should all program with sets all the time. I think at some point in my career, I stopped trying to yell that from the mountaintops because it's annoying and nobody wanted to hear it anyway. And I applied my own medicine, which is, no, let's stare at this thing harder and say, well, okay, sequentiality sometimes matters. How are we going to accommodate it in a way that doesn't require all this careful, single machine at a time reasoning?

• 5:49Speaker 02

  And so that has been the story of, let's say, the last five to 10 years and what led up to Hydro was closing that last gap because it's not correct or reasonable to say everybody should write all their programs using these high-level, set-oriented languages.

• 6:04Speaker 01

  Right. I guess, like, what motivated the decision to go from full-time researcher to industry full-time?

• 6:11Speaker 02

  Yeah, well, I think part of that was the fact that we were starting to really answer the final questions here. So the Hydro project, which we'll talk about more, was the end of a long line of efforts that I had in my lab on trying to make programming the cloud easy to get correct, easy to make scalable. And I think we finally got it. And so part of this story was, well, I never felt we were ready for production with this research till the last few years. And now I do feel we are, and I think it can really change the way that we write safe, distributed, correct, high-performance code. There were other considerations, too, about after teaching for 30 years, if you still want to keep teaching and so on, that are more personal. But I think technically, in terms of this arc of research, it's ready to put into production and that's what we get the opportunity to do at AWS.

• 6:55Speaker 02

  I've been saying we should change the way we program the cloud. Okay, let's go to the world's biggest cloud and see if we can change the way they program it, right? Stop pussyfooting and writing research papers.

• 7:04Speaker 01

  Besides the personal stuff, was it kind of like the biggest thing is that by being part of AWS, there's just things that you can do there that you wouldn't be able to do in a university lab?

• 7:11Speaker 02

  I think that's part of it. I think a lot of it is also validation, to be honest. Like, I could build toys on campus. Hey, here's a Twitter clone. Here, I implemented Paxos. You know, like, I can do all these little feats of daring do, but who's really going to believe that, right? And there's always these questions like, well, but if you're doing it at scale and you're doing it for dollars, like, you wouldn't do it that way. So at Amazon, I can remove that objection. We can work with colleagues here at Amazon, retrofit systems that the world relies on with Hydro, show that it gives correctness guarantees that the previous code did not and does not give up performance.

• 7:43Speaker 01

  And we were talking about distributed systems here, and I think a lot of times when people get introduced to distributed systems, they get introduced to, like, the CAP theorem, and that becomes the overall thing of, like, how they think about the world. And there's always this discussion around whatever the investment is or whatever the building, what are the two things that they're guaranteeing out of the three? But I guess, like, you've been a proponent of the COM theorem. Like, how do those things relate? And I guess, like, how does COM potentially change the conversation away from CAP?

• 8:09Speaker 02

  Hmm, so many questions are embedded in there. So I'm going to postpone the specific CAP versus COM thing for a minute because I want to get across something you were hinting at in the first part of the question, which is, when people think about distributed systems, so the thing is, everything's a distributed system now, man. Your phone app is talking to something in the cloud, so there's at least two nodes in almost every app you use. Your desktop app is also probably always talking to the cloud. Nobody writes a single-node system anymore, unless it's pretty much a toy or an experiment. Most things are distributed. And bugs will just creep in, race conditions, failure handling bugs, all this stuff. This is part of writing software now, and it's really leaking into every application's

• 8:54Speaker 02

  problem. Now, it may not be every developer's problem. If you're a front-end person writing JavaScript, you might not run across it, but I bet you somewhere in your product, you are dealing with this one way or another. Having said that, front-end people are really good at asynchronous, so they actually get this stuff very quickly when you teach it to them.

• 9:10Speaker 01

  Yeah, and I would say a front-end, to your point, like, everything being a distributed system, it's also client-server, so there are at least two nodes there.

• 9:16Speaker 02

  Yeah, so actually, I don't mean to dunk on front-end folks at all. They're often quite sophisticated about this stuff. But the bottom line is, everything is distributed, pretty much, and that's exposed to software. It's not just like, everything's distributed in your processor and you don't know it. It's like, no, really, you're programming a distributed system most of the time. Now, to answer your question about CAP and COM, so let's jump into really technical stuff here. So CAP was an effort by my colleague Eric Brewer at Berkeley to try to articulate, and it was early, so he was really feeling his way through it. And I think you have to understand the historical context when he was wrestling with this stuff. So I was there with him at the time. They were building the Ink to Me search engine, which is essentially the predecessor of Google, and trying to build it on clusters, not on a single big box, which is the way previous search engines had been built.

• 10:00Speaker 02

  And their first effort at this was to take a transactional database, I believe it was Informix at the time, which was at that time a very good parallel database, and see if they could build a search engine with it. And all the transactionality guarantees that the database provided were getting in the way of availability. So if a single node in the database went down, the database would say, uh-oh, it's not safe to run transactions, and it would lock you out, right? And they couldn't build a always-on search engine that way. Moreover, a search engine doesn't really need transactional consistency. So Eric was trying to articulate in a semi-formal way, like, what trade-offs do you have to make between some notion of correctness, which transactions are giving you a particular notion of correctness, and the notion of availability?

• 10:44Speaker 02

  And CAP was his effort to codify that. So the words are consistency, availability, and then the P is partition. So think of a network partitioning your set of nodes into two groups. Maybe one node is outside and the rest are inside, or maybe it's split down the middle. So his assertion was you can only get two out of three: consistency, availability, and tolerance of a partition. So that's CAP. And It was not a theorem, it was a way of thinking about the world. Some folks at MIT decided that they would try to prove a theorem version of it formally, but what they reached for was a much narrower view than Eric actually intended. What they proved in the CAP theorem is that linearizability, which is a very specific model of correctness, exhibits these CAP trade-offs.

• 11:28Speaker 02

  The problem then was that the universe was running around thinking about CAP as this bigger umbrella that Eric had defined, while the actual formalism was this very narrow thing. And that led to all sorts of confusion and therefore a lot of discussion, which was pretty illuminating. But over the course of maybe a decade, there were blog posts of people saying, "I bypassed the CAP theorem, I have a system that's all three." And other people saying, "That's not possible, it's a theorem. Look, there's a math paper and it's from MIT." And there's just like this disconnect between the CAP, the spiritual idea that Eric had, and CAP, the Gilbert and Lynch paper that was published by MIT. And Eric subsequently wrote another paper that explained, he didn't really mean the Gilbert and Lynch thing. He meant this bigger picture of like, you gotta think about these trade-offs,

• 12:10Speaker 02

  but they may not be endemic in all definitions of consistency. The consistency that Gilbert and Lynch chose was linearizability. So that's Eric's journey. I come in in the middle of this and say, "Consistency? Well, maybe that's the whole thing. Maybe the definition of consistency, some definitions give you this trade-off where you can't have all three. Some definitions of consistency maybe allow you to have all three." And just saying consistency is too vague. So where COME came out was an effort to try to identify exactly what is the sharp line in this consistency space between programs that succumbed CAP and programs that can avoid CAP. And particularly, I didn't really want to talk about availability anymore. I just wanted to talk about like, will two nodes potentially get into a disagreement? Like a long-term disagreement?

• 12:55Speaker 02

  Whether they're partitioned, whether they're not partitioned? That's really the question, can you keep the system consistent? And so what we observed in the COME theorem, the original conjecture, much like Eric, it started as just an idea, was that I know in database queries how this works. If you have operators that stream, so as they get more input, they produce more output. Those operators you can run and you can eagerly consume their outputs, which means that you can consume it faster in one place than another and not worry about correctness. Okay, so something about these streaming operators, which includes filtering, it includes mapping, it includes joining two tables, which is, you know, that's most of database queries. That stuff is actually not subject to CAP. I know this because I've built these things and they stream and it's okay, it's cool.

• 13:40Speaker 02

  And everybody can see these streams at different rates. But there are things in the database that have to block. A classic one is sorting. If you're gonna sort a table and you haven't read all the items, you don't know what the first item is. So you can't sort something till you've seen everything. Another one is like aggregation. So think about a sum query that's adding up all the numbers in a column. So you see all the numbers, you don't know what the total is, right? So these are things that somehow required you to block and consume the entire input before producing an output. Well, it turns out that, taking this story down the road, this property that as more stuff comes in, more stuff goes out, but all the stuff from before remains true? There's a math name for that, it's called monotonicity. Monotonicity says if you give a bigger input to a function, you get a strictly bigger output than you would have gotten.

• 14:23Speaker 02

  Which is another way to say that is, in a smaller input, you could start emitting its outputs without worrying about retracting them if more inputs come in. So it's this streaming property exactly. That's just said in terms of like, two fixed instances. One input, one bigger input, one output, one bigger output. Yeah, so that's monotonicity. And so COME was a conjecture that you get consistency if your program is monotone and you do not get it if it's not. And that's a sharp line, it's an if and only if. And again, a graduate student, much like with the CAP theorem, proved it about a year later in a very specific way, very much like Gilbert and Lynch. So he proved it for relational languages on these things called relational transducers. So if you're like a data log geek, you might really love this paper by Tom Amalut. And it led to like three or four more papers, much like the CAP theorem.

• 15:08Speaker 02

  So it's a little cottage industry, but still doesn't really apply to like, I'm sitting down in rest and writing a program, how do I know if it's monotone? So just this past six months, I finally decided it was time to clean this up. And I have a, it's like just an archive so far. But we have a notion of what I call complete COME. And it captures in a way that you can analyze, actually in a compiler, we do it in Hydro, for arbitrary programs essentially. It says what kind of monotonicity we're talking about. You don't need a special language, like a logic language or a database language. Any language can be used to implement a spec, but the spec is either monotone or not monotone. And if it's monotone, any language can be used to implement it in a way that is both consistent, so everybody gets the same answer.

• 15:53Speaker 02

  And also, it's available under partition. So you can start outputting answers even if you're in a partitioned state without worrying about the future.

• 16:00Speaker 01

  And I guess like tactically, like if I understand that my program does this monotone specificity, how does that help me as an engineer? Like what do I take away from that?

• 16:11Speaker 02

  Absolutely. So let me work backwards from what we have now in Hydro as a feature and then we can talk about why and how it works. In Hydro, you can point at like an endpoint of the system. So think about a network endpoint in the system that's gonna stream out outputs as it gets inputs, right? So requests are gonna come in, responses are gonna go out. So look at one of these response streams. Hydro will tell you what kind of consistency it offers. So Hydro uses the COME theorem to say, "Hey, guess what? All replicas will agree on the messages that come on this endpoint, but they won't agree on the ordering." That's a sort of set-based consistency. Or Hydro might say, "All replicas will agree on the sequence of outcomes on this endpoint, that's a sequential consistency." Okay, so it's a stronger one. But Hydro can use the CAP theorem to tell you what kind of consistency

• 16:55Speaker 02

  you're getting for each one of your endpoints of your program. And then you can decide what you need to worry about. If all of your replicas are getting the same outputs in the same order, you don't have to think about anything. You're sort of just living in your usual sequential world. If all of your replicas are getting the same set over time, they might get them in different orders. So you have to think about like, is your downstream code that's consuming this endpoint sensitive to order or not sensitive to order? Commutative is the math term. But it's like very actionable. And actually in Hydro, we can check these properties downstream for you too. So we can tell you, "Hey, this was set-wise consistent, but downstream you have a thing that needs a specific order." And we won't compile those together because that's non-deterministic. You're getting a random order being fed into a thing that cares about order.

• 17:40Speaker 02

  That's broken. So the compiler can literally check this stuff for you these days.

• 17:44Speaker 01

  Yeah.

• 17:45Speaker 03

  This episode of Software Engineering Daily is brought to you by WarpBuild. AI is writing more code than ever, which means GitHub Actions is running more than ever. Your GitHub Actions bill is now a function of how much AI code you generate. And every engineer knows the feeling. You push a commit and then you wait. WarpBuild makes GitHub Actions twice as fast at half the cost with a one-line change to your workflow. Linux, macOS, and Windows runners, in WarpBuild's cloud or your own, enterprise-ready, SOC 2, Type 2 attested, and trusted by teams like Sky from Comcast, Bitcoin, and Brain Trust AI. Get started with $50 and free credits at warpbuild.com/sed.

• 18:23Speaker 00

  Think about your mobile app's source code. Once it hits the App Store, it's out in the wild. And without the right protection, decompiling is easy for malicious actors looking to steal your IP or tamper with your software. That's where GuardSquare comes in. GuardSquare provides the highest level of mobile app security for Android and iOS applications and SDKs. Their advanced tools integrate seamlessly into your CI/CD pipeline. We're talking polymorphic, multi-layered code-hardening techniques and automated runtime application self-protection, paired with mobile application security testing and real-time threat monitoring to deliver the highest level of mobile app security without compromise. Don't leave your hard work exposed.

• 19:08Speaker 00

  Secure your mobile applications today. Go to guardsquare.com to learn more.

• 19:15Speaker 03

  If you're running Postgres in production, you've probably felt the moment analytical queries start fighting your transactional workload. Most teams end up adding a second database and all the pipeline complexity that comes with it. Tiger Data, creators of Timescale DB, takes a different approach. We extend Postgres with hybrid row and columnar storage, so one table handles both writes and analytical scans. Native compression cuts storage costs up to 95%. Continuous aggregates keep dashboards live without bash jobs. And it scales to petabytes without you re-architecting. Companies like Cloudflare, Octave Energy, Schneider, Axpo, and Flowco run production workloads on Tiger Data today. No stale data, no second system to operate, just Postgres. Managed for you, ready for the workload you're building toward. Try it free at tigerdata.com.

• 19:58Speaker 01

  So we mentioned Hydro, we've teased it a few times. So it's probably a good time to define like, what is this? What is Hydro?

• 20:05Speaker 02

  Yeah, thanks for asking. Hydro is a programming framework in Rust. So it works in standard Rust, doesn't use any nightly features or anything crazy, standard Rust, for writing distributed code. And its general purpose, which is to say, if you're writing a really fast event handling system, like it's running on network switches, you can use Hydro and it'll be fast enough and low latency enough. If you wanted to run like a big data analytics job, you can also use Hydro. So it can do things like Spark or Flink would do. It doesn't care about the performance shape of your events. They can be many or few, they can be quick or in volume, but it allows you to program a distributed system in a way that like is quite general purpose. And then when you program in Hydro, you get a bunch of kind of metaphors and then a bunch

• 20:49Speaker 02

  of correctness guarantees. So the metaphors are that you're programming locations, not single computers. What's a location? A location could be a process, although we don't say where it's running, but it could be a single process. But a location could also be a cluster of processes. And the reason to have locations is to talk about their behavior. So you can say, for example, if you have two processes and you want to compare a value between them, you can't. You have to send a message, right? And so that's in the type system. So if you try to write down X is less than Y, and X has a location that's different from Y's, that's just a compiler error, right? So the compiler is going to make you put in the networking and then think about the delays and all that stuff. So it's a global model, but communication and

• 21:31Speaker 02

  location are explicit in the model and they're type checked. If you're on a cluster, then sending to a cluster goes to a single member and then you have to figure out like how the members talk to each other. But you have in the program a very clear definition of what this cluster is and properties about it, like can I broadcast to the entire cluster? Or can I only send Unicast? Or is the membership going to change under me? These are all things that are in the type system. They will help you with bugs.

• 21:56Speaker 01

  Is it a case that you're, essentially you do a compile time check to see, could this thing talk to this other thing? Or are you making it so that those two things can talk to each other and I don't have to think about the underlying networking message passing in order to do that?

• 22:10Speaker 02

  It's a little bit of both. I mean, you do have to put in some syntax for networking. So the first thing you'll see is this doesn't compile. You're comparing two things at two different locations. But the syntax in order to get things from one location to the other is like dot send. You just needed to add an extra word essentially to your program and then it'll compile. So we did this in Rust, in part because I think Rust programmers are used to the idea that there are small subtle things you have to fix to get your program right. But if you fix them, you're pretty guaranteed to get correctness. In particular, we like to talk about Hydro as offering distributed safety, much like Rust offers memory safety. And so some of the kinds of bugs we prevent for you are race condition bugs. You can't write race conditions in pure Hydro. It's not possible. It won't compile.

• 22:52Speaker 01

  Right. So in a lot of ways, you're putting in these like guardrails around these known issues that tend to bite people writing and running distributed systems. How does this affect testing? So if I have, let's say that I do have a function that spans multiple machines, like how do I actually write like a unit test?

• 23:10Speaker 02

  Awesome. Great. So let's start with this like location stuff I teed up. Suppose you have a cluster and the code on the cluster, so this is the code that runs on every node in that cluster, says send to Sean, and Sean is a process. Okay. Well, we know because the cluster is a cluster, that the messages will arrive at Sean in a random order from the different cluster members. We can't control the timing of the messages. They'll come when they come. And on different runs, Sean may receive messages in different orders. So the input port at Sean will be marked as a stream that's unordered of messages. Now, if you have downstream code in the Sean process that is order sensitive, like it's accumulating a list, or it's buying stocks,

• 23:51Speaker 02

  we will flag that as a compiler error. So you simply cannot connect this unordered stream of input messages to this ordered operator. Okay. So that's all guaranteed by the type system. There's no testing needed, right? The same way in Rust, you can't walk off the end of an array. You just can't compile that. However, as in Rust, if you really want to do that, you can, and you have to put in a keyword. So in Rust, the memory safety, the keyword is unsafe. In Hydro, it's non-dead, non-deterministic. So to get this unordered stream and shove it into this ordered thing and say, I don't care, man, it's coming in whatever is coming in, I want to run this code, you will put in a non-dead block. And at that point now, the compiler will let you run this thing, compile it and run it. But you have a non-deterministic system. And so

• 24:33Speaker 02

  now you want to test it. So Hydro comes with a batteries included simulation tester, or model checker, if you like that terminology. But it only checks the non-dead stuff, because everything else is correct by assertion with the type system. In particular, if you take an unordered set of messages and you put it in a commutative operation, like some, some doesn't care what order things come in, right? When you're adding stuff up, you can add it up in any order. So if you take unordered and you stick it into some, Hydro won't simulation test that at all. It'll just try any one order and say, yep, any one order is the same as any other. So I'll only test the one order. But if you put in a non-dead, so you have random order going into a thing that's order sensitive, the non-dead that in the simulator will try all

• 25:14Speaker 02

  the orders, run every single one of them, and you can see if your tests pass. So in Hydro, testing is there when the type system doesn't save you. And the type system doesn't save you when you tell it not to. And then we'll do simulation testing for you.

• 25:28Speaker 01

  In the scenario where I have like a single function that's spanning multiple machines, like is Hydro handling the state management between those machines as well? Like a durable execution framework or something like that? And some things go wrong, essentially, it's going to have to back up the checkpoint, right?

• 25:42Speaker 02

  Right. This is where I think a lot of libraries trip, actually, and why we've postponed a general purpose solution for all these years, is that if you start to answer those questions, like you're asking, what about durability? What about replay of dropped messages or things? All these kind of like specific problems typically lead people to specific solutions. So if you look at Spark, or Flink, or Kafka, or many of these frameworks, they only work in their narrow sweet spot. They're distributed, but they only work at the stuff they're good at, and they're really bad at other stuff. You would never implement Paxos with Spark. That would be a terrible idea. And the reason is they answer the questions like the one you just asked. They say, you know what, we're going to need a way to make things durable. Let's bake one in and force people to use it. And it'll make some things

• 26:25Speaker 02

  super easy. It'll make other things completely too slow or inaccessible. So in Hydro, we don't do that. All that stuff that you want to do, like durability, you just write some more code or you plug in a library, because we don't want to be opinionated about architecture. We just want to be opinionated about correctness. So you get to build the architecture that works for your application. If your application does well by like spooling things to disk and then reading them back, cool, do that. But that's going to be too slow for like a packet handler. So for a packet handler, you might instead send a negative acknowledgement to the sender. The sender's job is to send it back. You know, there's all sorts of ways to do failure handling. There's all sorts of ways to do durability. It can be done with replication. It can be done with spinning iron. You know, there's just, these are decisions a language should not make, in my opinion.

• 27:09Speaker 01

  So I guess like, what is the best way to think about Hydro in reference to maybe these existing solutions that people might go to, who have been building these types of systems historically?

• 27:22Speaker 02

  Yeah, great. So I partition this world of distributed frameworks into two. There's the high level frameworks. So think of things like Spark and Flink and Kafka, and durable functions is another great example. And then there's low level frameworks that are really convenience for programming, like actors or RPC. So the high level frameworks tend to be very opinionated. They're really good at one thing and they're quite bad at lots of other things. That's sort of the Spark, Flink, Kafka thing. Those are essentially applications. They are great at what they do well. Okay. But they're not programming frameworks. They're high level programming frameworks. SQL is another example. Like, I really shouldn't be trying to convince the world to write all software in SQL. It's a terrible idea. So not general purpose programming. So that's one bucket. And the reason those things don't generalize

• 28:07Speaker 02

  is because they make these strong architectural assumptions. In the other bucket, like take RPC as the canonical example of a programming tool for distributed systems. It's completely unopinionated. It just says, hey, here's a way to send messages, which is great, or even particularly invoking code on the far side, right? By sending a message and getting a response. But it doesn't offer very much help in assembling a program, and it doesn't offer very much help in checking if the program's correct. And the same is true with actors. Actors are basically an event loop around event handlers. And you know, cool, that's handy, but it doesn't really help you get through like, is my Paxos doing Paxos? You know what I'm saying? It just, it doesn't tell you if you have race conditions. It doesn't tell you how it deals with failure.

• 28:48Speaker 02

  It's just a little event loop. So there's been nothing akin to a real language compiler for distributed systems as general purpose, and solves for you the hard things about your language. So Rust memory safety is such a beautiful example. We've all internalized that. It solved a hard problem we thought we had to enforce at runtime using garbage collection. And it said, no, you know what? We can solve this at compile time. And we want to do that now for all the hard things about distributed systems, failure, parallelism, you know, ordering, retries, all that stuff. And I think with Hydro, we've demonstrated that that's possible.

• 29:21Speaker 01

  And when you talk about, you know, making this simpler, what does that mean for, I guess, end user? Like, is that simpler in terms of less bugs, less lines of code, maybe both? How do you sort of frame the simpler part of that?

• 29:32Speaker 02

  I would argue mostly less bugs. Maybe less lines of code. That's not always a virtue, right? You can write some really dense code that's hard to read. That might not be a good idea. I have used that argument in the past, but I think it's not really a very honest argument. No, fewer bugs. So, and in particular, in the era of agentic programming, you've just generated a distributed program. How do you know it doesn't have race conditions, man? You know, like in your own code, you might have been, well, I assert that it doesn't have race conditions, because I'm really good, you know? And I know people who can do that pretty reliably, but they make mistakes. AI agents are really bad at this right now. They make tons of distributed systems mistakes. They don't even, in my experience, really know what tests to write for distributed systems yet. They'll get better at that. But really, to know that your code is correct in an era where you're not reading the code or writing

• 30:17Speaker 02

  the code, you want your compiler to make sure it's correct, right? So that's kind of where we are with this stuff. You know, this is why it's better to use Rust than Python when you're using an agent, right? Because the Python could have all kinds of bugs and you really can't know.

• 30:32Speaker 01

  I guess like, if I'm going to use this, if I'm using Hydro as a developer, like, what is that process like? How do I kind of plug it in? Like, where do I start?

• 30:39Speaker 02

  Yeah. So there's a like, you know, greenfield version of this, where you go get the Hydro template, you say cargo generate with your Rust tool chain, and it generates you a project, and you start writing Hydro. And that's basically a dialect of Rust that looks a bit functional. So it's a lot of maps and filters and things like that in big long chains. It looks actually quite a bit like Spark or Pandas, except instead of handling data streams, you're handling requests, packets, you know, stuff coming off the network, or off the I/O channel, or what have you. But there's also sort of a brownfield use of Hydra where we don't need to control the network endpoints. You can just have input-output streams essentially of events. And then you embed inside some Rust code and you know, the Hydra bit is Hydro E and

• 31:23Speaker 02

  the rest bits are still plain old naked Rust. And the main thing to think about is Hydra does have this global programming model, which is kind of nice. So where do you want to apply it? And I'll give you an example. A lot of distributed systems have sort of roles defined for them. So if you know your Paxos, there's like proposers and acceptors and learners. There are these different roles and we write a different piece of code to run on those different roles. They each run on their own machines, right? So we write some code for the acceptor. We write some code for the proposer. In Hydra, the code's distributed. So instead you write functional units. You don't write roles. You write like a task and the task might be something like, take a vote, which is a key part of these agreement protocols. Take a vote involves many machines. One machine says, I would

• 32:08Speaker 02

  like to take a vote and broadcasts it to many machines. Then it collects the votes from the machines and counts them up and then it announces the result. That could be just a single Rust function in Hydra. So the modularity then becomes around like reusable distributed components. And that's a nice thing to think about when you're retrofitting your system with some Hydra is like, well, what are the functional units of this distributed program? And how can I tear them out of the little programs I hand wrote for each location and turn them into a reusable service in a library?

• 32:36Speaker 01

  Would this only work if you built this in Rust because of the properties of how Rust works?

• 32:41Speaker 02

  I think it would work quite well and arguably better in a language like Scala. Actually, one of the things that we often are missing in the Rust ecosystem is dependent types, which is, you know, a programming language in ACID that I'm not going to get into the details of, but it does make some of our Hydra internal code kind of hard to write. Like Hydra would have been easier to write in Scala. There are many advanced languages that would be amenable to this kind of programming. The key issue is something called quoting. And the idea is you write some code and we tear it up into code that's going to run on every machine, right? So you wrote a function for vote. It really eventually does have to be binaries that run on every individual machine. So that code that you write, we're going to like, need to kind of take it apart. And that taking apart business

• 33:24Speaker 02

  in our compiler is easier in some languages than others to do it natively.

• 33:29Speaker 01

  But to use this, like, do I, am I only basically building a certain part of my application in Rust using Hydra and I could build other parts of it using whatever language I choose?

• 33:41Speaker 02

  Yeah. I mean, you know, keep in mind, it's the distributed systems world. So you can always take a piece of your system and turn it into a service. And then the endpoints are GRPC or REST or something like that, right? They can be cross language. So certainly that's always an option. You can write a microservice or tear out a piece of your system and turn it into a service and write it in Hydra, even though your service was written in something else. You know, other than that, you're doing the moral equivalent that in memory, say, you know, Java native interface is an example. You can take Java code and embed Rust in it and the rest could be Hydra. But you go through J and I and it's, it's a bit of a drag.

• 34:15Speaker 01

  Yeah, I see. You talked a little bit earlier about this like simulation environment for testing. So how does that kind of work at a high level? Like, where am I running that?

• 34:27Speaker 02

  Yeah, great. It works on your laptop. So when you write unit tests in Hydra, you'll write them as simulator tests. And so you'll write a unit test that says, you know, I want the sum of the numbers to, that I put in to equal the sum of the numbers that I put in. You know what I'm saying? But the way it's going to get simulated is all orders of the messages in your program to do the summing that we talked about will get run by our test harness. And you'll get an answer, not just to one run of that, but all possible runs of the system with all the orderings of the messages. And it'll check every one of them for your invariant. But it's just doing it on your laptop. It's a simulator. So it just looks like a unit test. It's just a super great unit test.

• 35:08Speaker 01

  We were talking about Rust and you were, you were mentioning that some of this stuff might be easier in Scala. Like I think Scala runs on JVM, like Java still sort of dominates the data infrastructure world. But do you see, you know, Rust potentially displacing some of that over time? Like because of these, you know, key properties that Rust has of these compile time guarantees?

• 35:29Speaker 02

  I mean, I'll say for the record that using the JVM for infrastructure was a terrible idea. Terrible idea. It was a movement that was going on in the big data world because, I don't know why, because it was a terrible idea there too. But it kind of was happening around the same time the cloud was emerging. And so what ended up happening was the early clouds got polluted with a ton of Java services and they're slow and they're ugly. And you know, it's just not, it's nobody's favorite language. The garbage collection, despite decades of trying to make garbage collection free is not free. And when it runs at an inconvenient time, it looks like a failure, you know, and distributed systems, the number of quote unquote failures your distributed system deals with because of the JVM is

• 36:13Speaker 02

  going to be much higher than in a non JVM language. And just the overhead of most code in Java, by the time you're done, and it's written by normal developers, not by super duper Java experts. It's slower than Rust code. I was brought up as a C programmer, you know, and the downside of C, of course, is bugs are just all over the place and you don't know where they are. Rust has been a beautiful bringing together of, you know, how do we get memory safety and a high level language with a nice type system that will really help you find your bugs? And I just see no justification for writing anything over the JVM anymore at all. And my experience at, most of the companies I've talked to, AWS among them, is they're in the middle of Rustifying as much Java code as they can

• 36:54Speaker 02

  or C sharp code or what have you, because it is a language with fewer bugs and the agents can do this semi automatically. So let's give the agents a really tight guardrail language to re-implement things in a fast language.

• 37:07Speaker 01

  Yeah. I mean, I think Google's kind of, I guess, like goified by their backend. But I wonder if part of that is like the product of when a lot of these systems were coming up with, like where the state of programming languages at that time, like maybe, like my hypothesis would be like Java gave just an easy way to run these applications across variety of different types of machines and operating systems at that time. And we didn't have as many good ways of doing that.

• 37:31Speaker 02

  I mean, you know, like Oracle used to run on everything. It was written in C. It was expensive to do that, but it's not hard. I mean, it's hard in ergs, yadda la programmers typing a lot of code and testing it. But you know, portability, I think was a sham.

• 37:48Speaker 01

  Yeah.

• 37:49Speaker 02

  I can look at it, that Hadoop is what made Java so successful because it was so wildly hyped. And it was just a mistake. It was, you know, the Google people didn't write MapReduce in Java. Certainly not. They wanted performance. They had to deal with all the world's data. They weren't going to do it in Java. Hadoop was a, intended to be kind of an experiment and it just kind of took off and the hype began. And I think, you know, Java really got huge boost from that. Similarly with Scala and Spark, by the way, but it wasn't a big enough movement to make a language as hard as Scala popular. Like pretty hard to approach. Java was much easier to approach.

• 38:23Speaker 01

  Yeah. I mean, I think if you look at the history of some of the languages that are now like the most dominant languages in the world, Java, JavaScript, probably Python, like their beginnings were not focused at all on the types of workloads that they're handling today. And I think it's, you could make pretty strong arguments that they were not the right, like hindsight being 2020, the right language is to put basically the entire internet behind.

• 38:45Speaker 02

  Yeah. I'm sensitive to this. Like a lot of technologies emerge from lo-fi, right? Because they get adoption and adoption drives usage and usage drives improvement and we iterate to a better world, right? It's just that if you do that without counting the consequences, you can really drive yourself into a corner. And then somebody has to at some point say, can we do this better? Right? So this sort of lo-fi to medium-fi movement is one that I hear a lot. It might be a good thing for certain startup companies. For example, you start at the bottom of the market with an easy to adopt low function solution. Somebody else will probably start at the top of the market, selling a very high function thing for a lot of money. And then very often the market just bifurcates. There's just

• 39:27Speaker 02

  two winners. There's the kind of high end winner and then there's the low end, lo-fi winner. But it's pretty rare in my experience that the lo-fi thing evolves into a really good solution. And so when we talk about system infrastructure, like the cloud, I don't know why you embrace lo-fi technologies. It just seems crazy to me.

• 39:44Speaker 01

  Going back to Rust, like Rust clearly has some great things about it, but like, what are the, what have you found the hard parts of actually building in Rust? Like where does the language kind of fight you when you become from maybe some of these other programming paradigms?

• 39:58Speaker 02

  Yeah. I mean, you know, I suffered with the Rust learning curve pre LLMs, like a bunch of people. I had a master student who tutored me, you know, became a software engineer on my team. I hired him and you know, we paired for a year or something where he's really, really good at Rust and I'm, you know, I'm an old guy. I learn slow, but you know, I was learning to write pretty fancy traits in Rust and we were doing some pretty fancy stuff. And the borrow checker difficulties actually didn't come up so much in the kinds of code we were writing. So I'm still pretty bad at the Rust borrow checker bugs. And then along came LLM coding agents and like, none of those frictions really matter anymore. Like, it's like, oh, you forgot to put a lifetime on this variable. That's okay. I'll do that for you. You know, it's just a lot of those learning curve hurdles

• 40:40Speaker 02

  get smoothed out by the LLMs. And now the challenge is not so much is Rust hard to write, but will you take the time to learn how to read it?

• 40:47Speaker 01

  Yeah. I even think like with a lot of the, you know, we've, as in historically with programming, we've invented a lot of different layers of abstraction, different frameworks and things like that to hide away some of the details. But if LLMs are really, you know, eventually at the details, I would presume a lot of those types of things go away because it's like, you're beginning to probably get better performance if you just write it natively than, you know, if you take, use the front end example, like if I could natively write everything in JavaScript, not having to go through some buffer of like a React framework or something like that is probably going to be more performant. The assumption is that I'm not a good enough programmer to do that in JavaScript without tons of bugs or poor performance. So I can rely on React to kind of do that away where people have already solved those problems for me.

• 41:32Speaker 02

  Yeah. I think that's a reasonable hypothesis as a starting point, but what we're learning now with agentic programming, sort of 12 to 24 months in, is that maintaining that code, even with the help of LLMs is hard. So if you write very verbose, very detailed code that doesn't use componentry, you can get it right, but keeping it right as you grow it gets increasingly difficult for the LLMs. So modularity and elegance and simplicity actually still matter quite a lot, I would say. Language dialect less so, but you know, you can get an LLM to write you some great assembly code, right? You just don't want to maintain that. And probably if you ask the LLM to change it, it'll introduce bugs. So I think, you know, the key components are, is the language going to keep the LLM

• 42:14Speaker 02

  on the rails, right? Guardrails are just so important. And is it evolvable? Which is the second part. And that part I think does still benefit from like, Nice design patterns, libraries, you know, all that good common reuse stuff that we use as humans.

• 42:30Speaker 01

  So with this project, Hydro, where it started at Berkeley, now you're doing this at AWS. Like, I guess, like, how does it change when you take a research project and you turn to making that essentially like a production application? Like, those are pretty different things. I was a researcher. I know a lot of the stuff that I built in research. I would now build it the same way if I was, you know, building it as a product. So, you know, is there a lot that has to change in order to do that?

• 42:54Speaker 02

  Yeah, but I think you have to see it in context. So, you know, Hydro is the result of 25 years of work in my group, something like that. Like, we were not ready to just go to industry 15 years ago. We were ready to go to industry three years ago. So it took a long time to do research. And, you know, part of that was I went off and did our data wrangling startup that had nothing to do with, you know, this whole agenda. And so I lost the better part of a decade to other interesting topics. But it was still many years of work, you know, with sort of three iterations of languages, I'd say. Now that we sort of had ironed out the model and we, we really believed that going to AWS was eye opening in ways I didn't expect. So I expected, you know, we'd get beat over the head with performance. No, you know, we performed just fine. In fact, we're usually way faster simply because we're Rust and we're replacing Java.

• 43:38Speaker 02

  So, you know, that we're using the Rust compiler. I will say the fellow who taught me Rust, he's an incredible Rust programmer and he's generating the code in Hydro. So he's generating code that the Rust compiler is very good at inlining. And so our Rust code is actually very, very fast. So performance, we could think about that on campus and we could measure cycles and all that stuff. What we didn't think about was, oh, there's some weird brownfield code on a router and they want to use Hydro, but they need it to interoperate with Quick, which is a networking protocol, you know, which is used in a minority of settings. But if you really care about stuff, it's used. It's like, oh, well, we had baked in the networking layer into Hydro and we never thought about it running separately from its own networking.

• 44:19Speaker 02

  Like, we have to strip out the core Hydro stuff from its event loop and its network handlers. So there's stuff you learn about, I would say, interoperability in the real world that just wouldn't occur to you, sitting in a office, just designing something beautiful. So I think that's the biggest lesson is really interop. And it's not, in this case, just interop of protocol. It's interop of my code sitting inside your code. You know, it's like compiler interop inside a program. So that's been a lesson. Deployment would be the other big lesson. You know, we started out thinking, eh, we'll write some shell scripts and deploy some stuff. And then we're like, okay, well, if we were in the real world, we'd use like Terraform, Kubernetes, something like that, ECS on Amazon. But, you know, that's just because those are a little easier.

• 45:02Speaker 02

  No, actually, when you start looking at deployment, you really do want to think about it very carefully and it changes what you do. So I'll give you an example like off the bleeding edge. Hydro Simulator now allows you to simulate multiple versions of your program at the same time. So one of the things about deployment that's true in the real world is that you do incremental rollout, which means you have multiple versions of your code running simultaneously. Okay, now all the bugs you worried about in V1 and all the bugs you worried about in V1.1. What about when they're talking to each other? Have you worried about the bugs in V1 and V1.1 working together? And we will simulate that for you. So again, like, we'll just go through all the possible configurations and run a whole bunch of tests for you automatically in our simulator.

• 45:44Speaker 02

  So you can do deployment tests before you launch that incremental rollout. You can sim test to see if it's going to be safe. So that's not a thing compilers or tests usually did. And it's not something we thought of in the office. We only thought of it sitting at AWS doing real stuff.

• 45:59Speaker 01

  What about, you know, advising? Like, you've advised a number of students over the years. You have some very successful students, like three of your students won the Jim Gray Dissertation Award. I'm sure many others are now in prestigious positions. Like, is that something that you've been able to take, translate into your, you know, role and industry as well? Like, whatever your approach is to advising and, you know, hoping these students be successful, is that a skill that's been transferable as well to industry?

• 46:26Speaker 02

  I hope so. You know, I'll say a couple of things. First of all, you know, the success of my students is largely biased by my having had the good luck of getting a job at Berkeley. And the students at Berkeley are tremendous. So many of these successful people are successful because they're amazing, not because I mentored them, right? So I just want to flag that. The other thing I want to flag is that a cool thing about this AWS setup is they think it's beneficial for me to stay at Berkeley advising PhD students. So I actually still have four students in a research group and a lab and an office on campus. And we continue to push the envelope there on things that are not on the time horizons that AWS operates on. And so they're essentially giving a little bit of my time to have a longer horizon in the open source world of research.

• 47:05Speaker 01

  That's fantastic.

• 47:06Speaker 02

  Having said that, you know, there's a lot of mentorship that goes on in companies, as you know. And I do think having mentored students is helpful. I also think I'm learning things about, it's different when you're building together than when you're just kind of educating and PhD students are exploring ideas, because they'll often go off and do speculative stuff. And it doesn't have to be solid. It just has to be interesting. You know, so there's just like different criteria that you're optimizing for on campus than you are in production. And learning the craft of software engineering for production is a bit different than learning the craft of research. So I'm definitely learning about that as I go. My startup years, which were many, helped with that.

• 47:46Speaker 02

  But I didn't run engineering at my startups. And that's kind of where this kind of mentorship that I'm seeing at AWS is, is inside of engineering. So I got a lot to learn, but I hope I can give back some stuff.

• 47:56Speaker 01

  Last question. So I know you're a jazz trumpeter. You've played with some incredible musicians. Does any of that map to how you think about distributed systems? Has that been a transferable skill for you?

• 48:07Speaker 02

  To distributed systems? I'm not so sure about that. But I think to an outlook on like, the balance between structure and creativity, like jazz sits in a particular place because it's so improvisation focused.

• 48:18Speaker 01

  Yeah.

• 48:18Speaker 02

  About, you have a skeleton of a structure, but you, you don't worry about the details till the very last minute, right? You know, you know the chords, but you haven't picked the notes yet. That's like your typical jazz solo, right? You haven't picked the rhythms. You haven't picked the notes. You've just got the chord changes, the harmony. I do think that has something to do with my way of doing computer science, which I'm pretty intuition driven. And I'll skip the details for a while. I know people who are much more disciplined upfront. They remind me of classical musicians. You know, you have to play every note exactly the right way. And that's the value of classical music, you know, in some sense. It's of course not. It's the table stakes for classical music is, you know, at minimum, you play the notes correctly.

• 49:00Speaker 02

  Maximum, obviously, you're making music. In all of these forms of music, the important thing is just to make music, connect, create experiences. But jazz does it, you know, with less attention to setup. And so I do think, you know, there's, there's a role for that in, in research for people to be thinking kind of atmospherically and letting the details fill in as they do. Perhaps my best example of that, my master's advisor was Michael Stonebreaker, who won the Turing Award. He's the postgress guy, among other many feats. That guy could not code his way out of a paper bag. Like, he did not know how to write software. He could barely, he didn't use LaTeX because he didn't use LaTeX. He was too old for LaTeX. He used Enroff. The guy did not program and he was right, like, most of the time. And as a graduate student, it drove me bananas. Like, how could this guy who doesn't know how to program

• 49:46Speaker 02

  be right all the time about these architectural decisions? And he just is really good. So I think there's room for that in the world. And we give Turing Awards to people who behave like that. So it must be one of the things we have to do in computer science to do well. And I do think that there's a connection to the variety of ways to make music and the variety of ways to do computer science. Distributed systems? Nah, I don't know.

• 50:07Speaker 01

  Awesome. Well, thanks so much for being here.

• 50:10Speaker 02

  Yeah, this was a pleasure. Thanks for letting me be here. This was great.

• 50:13Speaker 01

  Cheers.