Did Ethereum Break? with Preston Van Loon & Terence Tsao

Did Ethereum break or what actually happened? We brought on Preston Van Loon & Terence Tsao, two Ethereum Protocol Developers, to help us answer that question. ------ 📣 CONSENSYS | DILIGENCE FUZZING https://bankless.cc/diligence-fuzzing-pod ------ 🚀 Airdrop Alpha is waiting for you on Bankless.com https://bankless.cc/Alpha ------ BANKLESS SPONSOR TOOLS: 🐙KRAKEN | MOST-TRUSTED CRYPTO EXCHANGE https://k.xyz/bankless-pod-q2 🦊METAMASK LEARN | HELPFUL WEB3 RESOURCE https://bankless.cc/MetaMask ⚖️ ARBITRUM | SCALING ETHEREUM https://bankless.cc/Arbitrum 👾STADER LABS | ETHX LIQUID STAKING https://bankless.cc/Stader 🗣️TOKU | CRYPTO EMPLOYMENT SOLUTION https://bankless.cc/Toku 🎮IMMUTABLE | GAMING ECOSYSTEM https://bankless.cc/Immutable 🛞MANTLE | MODULAR LAYER 2 NETWORK https://bankless.cc/Mantle ------ Timestamps: 0:00 Intro 7:53 Did Ethereum Break? 8:22 What Happened? 14:50 Slot Visualization & Connectivity 20:30 Did Users Notice? 23:57 Why Was This Worrisome? 26:19 Preston & Terence's Emotional Response 35:55 Ethereum Resilience 39:30 Was This Good For Ethereum? 41:20 Solana vs. Ethereum Liveness 44:40 Closing & Disclaimers ------ Resources: Is Ethereum Breaking? https://twitter.com/BanklessHQ/status/1657122512516575232 Preston Van Loon https://twitter.com/preston_vanloon Terence Tsao https://twitter.com/terencechain ----- Not financial or tax advice. This channel is strictly educational and is not investment advice or a solicitation to buy or sell any assets or to make any financial decisions. This video is not tax advice. Talk to your accountant. Do your own research. Disclosure. From time-to-time I may add links in this newsletter to products I use. I may receive commission if you make a purchase through one of these links. Additionally, the Bankless writers hold crypto assets. See our investment disclosures here: https://www.bankless.com/disclosures

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:30Ryan Sean Adams?

  In the night with finality on the Ethereum blockchain last week and we want to go all, we want to go over that today with some experts who can help us answer the question as to whether Ethereum broke or not. What does it mean to break? What happened? What does non-finality mean? I think we'll talk about that. And what are the risk conditions that we are exposed to as a result of this? How did Ethereum handle this event is really the central question in my mind. David, who do we have on today?

• 0:59David Hoffman?

  We have Preston Ben Loon and Terence Chow of the co-founders of Prismatic Labs now in Arbitrum, now a part of Off-Chain Labs. And we've brought Preston and other members of Prismatic on to the show before to understand the world of clients because it is a complicated world, but not so complicated that we won't be able to understand it here on the show. So some very important questions with some very smart people who have been with the Ethereum proof of stake effort since it was once upon a time called Ethereum 2. But before we get into the show, we've got to talk to our friends and sponsors at Consensus because we've got to talk about this thing called diligence fuzzing. It's a new auditing tool. Just fuzz it. Yeah. With a sick new name called diligence fuzzing. So what's diligence fuzzing? For smart contracts, solidity devs, auditors, enterprises

• 1:44David Hoffman?

  engaging with Web3 that need to ensure that their smart contract is secure before proceeding to main net, diligence fuzzing, it's an audit-grade security tool that serves as an automated way to find code vulnerabilities for people without hosting their own infrastructure or writing complete complex test cases. So there are many ways to do automated testing, but buzzing is a way, brand new way with quite a lot of sophistication, according to Consensus, that you can go and access right now. So this is an extra layer of protection for your smart contracts in addition to and above and beyond typical code audits. Humans are messy and can miss things, but diligence fuzzing is diligence by robots.

• 2:25David Hoffman?

  Fuzzy robots, apparently.

• 2:26Ryan Sean Adams?

  You did that really well.

• 2:27David Hoffman?

  There's a link in the show notes to get access to this.

• 2:30Ryan Sean Adams?

  David, can you imagine releasing a smart contract and not fuzzing it? I just can't even imagine doing that.

• 2:35David Hoffman?

  I prefer fuzzy smart contracts.

• 2:37Ryan Sean Adams?

  Let's fuzz it. All right, so there was an event that happened last week, and this was an event that happened after the roll up. And I'm wondering if we could kind of tee this up before we bring Preston and Terrence on. So what happened? I think maybe the first sort of mini event was on Thursday and then again Friday. If I'm recalling correctly, all my weeks are blurred, David, but give us the recap before we get in here.

• 3:03David Hoffman?

  Yeah, it happened twice. And so Ethereum failed to finalize, which is an interesting sentence. What does that mean? Blocks were still propagating, transactions were still going through, but finalization was not happening. And just at a high level, finalization is just something that happens when enough clients, enough validators have given the thumbs up about a certain epoch. Epox, which are, I believe Preston's going to come in here and give me the whip. I believe 64 blocks, but it's a collection of blocks. And then if enough validators give the thumbs up, then that epoch finalizes and that is permanently embedded into the blockchain.

• 3:41David Hoffman?

  When a epoch is finalized, it requires slashing conditions to go back and unwind those blocks. And so if a block isn't finalized, it's not as permanently embedded into the blockchain as it otherwise could be. Except with this event, which happened twice, Ethereum failed to finalize because blocks were being missed. Validators were not doing their jobs. I am only able to explain it to some degree at a high level. And so we're going to go talk to Preston and Terence to get a deeper dive into what this world of finalization means.

• 4:14Ryan Sean Adams?

  Should do a disclosure before we get in. Of course, now Prismatic is owned by Offchain Labs, which is a sponsor of the Bankless Podcast, very supportive of what they're doing in Layer 2 and client development work also. David and I are notorious holders of the asset called ETH, so you should know that.

• 4:30David Hoffman?

  Also related.

• 4:31Ryan Sean Adams?

  Podcast, that we are bullish on Ethereum generally and are interested in finding out what happened today. So guys, we're going to get right to the episode with Preston and Terence to talk about whether Ethereum broke or not. But before we do, we want to thank the sponsors that made this episode possible, including our number one recommend exchange, Kraken. Go check it out.

• 4:54David Hoffman?

  Kraken Pro has easily become the best crypto trading platform in the industry. The place I use to check the charts and the crypto prices, even when I'm not looking to place a trade. On Kraken Pro, you'll have access to advanced charting tools, real-time market data, and lightning-fast trade execution, all inside their spiffy new modular interface. Kraken's new customizable modular layout lets you tailor your trading experience to suit your needs. Pick and choose your favorite modules and place them anywhere you want in your screen. With Kraken Pro, you have that power. Whether you are a seasoned pro or just starting out, join thousands of traders who trust Kraken Pro for their crypto trading needs. Visit pro.kraken.com to get started today. Mantle is a brand-new, high-performance Ethereum Layer 2 network, built differently from the other Layer 2s you may be familiar with.

• 5:37David Hoffman?

  Mantle is a modular Layer 2, built on the OP stack, but uses EigenLayers' data availability solution instead of the expensive Ethereum Layer 1. Not only does this reduce Mantle's gas fees by 80% compared to other Layer 2s, but it also reduces gas fee volatility. Mantle has a decentralized sequencer set, eliminating the risk of downtime and censorship on the network. And because Mantle implements multi-party computation nodes, Layer 1 settlement execution is shortened from seven days to as low as just one or two. Mantle is the first Layer 2 built by a DAO and is backed by one of the biggest DAO treasuries in the world: BitDAO. Mantle already has sub-communities from around Web3 onboarded to help the growth of Mantle, like Game7 for Web3 Gaming, or EduDAO for the world of D-Side, and Bybit

• 6:22David Hoffman?

  for TVL, liquidity, and on-ramps. Check out Mantle at mantle.xyz and follow them on Twitter at @zeroxmantle. Immutable is at the forefront of Web3 gaming, on a mission to bring digital ownership to every player, offering the world's best games and game development platform. Immutable lets game builders and players focus on great gaming experiences. So, build your next Web3 game on easy mode with Immutable's leading full-stack Web3 gaming platform. Its built-in UX features, like the Immutable Passport, are designed for games to scale to the next billion players coming to Web3. With Immutable, players can sign up with an email, pay with a credit card, and experience a frictionless purchase flow inside of games. So, discover your next favorite game and explore a network of 150 games building on Immutable, including such

• 7:07David Hoffman?

  titles as "Gotham Chain," "Gilds of Guardians," "Illuvium," "Ember Sword," and "Metalcore." So join Web3's largest ecosystem of games and players. Build, play, and connect at immutable.com.

• 7:20David Hoffman?

  Bankless Nation, Preston Van Loon and Terence Tao are the co-founders of Prismatic Labs, one of the organizations that maintains the Prism Ethereum consensus client. Prismatic Labs was one of the earliest client teams that was formed to start building what was then called Ethereum 2, which we now just call Ethereum, and they've been very key to Ethereum development every step of the way. Preston, Terence, welcome to Bankless.

• 7:41Speaker 02

  Hey, happy to be back. Thanks for having me.

• 7:45David Hoffman?

  Yeah, okay, I'm gonna look forward to being educated about some of the nuances of finality and security, but first, did Ethereum break, Preston?

• 7:55Speaker 02

  No, it did not break, right? What, we did see Ethereum actually work. This, we have a time where Ethereum has difficulty finalizing, and we got to see it actually work in action, right? So this is not like a problem with Ethereum, that specification did not break. It actually, it actually worked, and it worked in a unfavorable condition, and we got to see that happen in mainnet for the, for the first time.

• 8:23David Hoffman?

  So how would you summarize what actually happened? And so I did my best to explain it in the intro.

• 8:30David Hoffman?

  How would you describe to the layman, explain it like I'm five, what actually happened here?

• 8:34Ryan Sean Adams?

  And how many blocks are there in an epoch, Preston? Most importantly.

• 8:38Speaker 02

  There are 32 blocks in an epoch, not 64. At least you got a power of two, so that's nice. What happened was multiple clients, Prism specifically, and Taku had this problem where if it received these sort of old attestations, attestations being what the validators are producing to say, this is what I think the head of the chain is, they received these old attestations which caused Prism in particular to replay a bunch of states, right? It, it had us to go back in time to evaluate whether or not this attestation was valid, or, or how to count this vote.

• 9:19Speaker 02

  It ended up being a pretty expensive operation that somehow bypassed or filled the caches that we had, so we weren't able to. To, you know, keep that work in the client, I had to recompute it every time. So essentially it was an accidental, I guess, like kind of denial of service where the clients were so busy doing these computations. They weren't able to respond. This is the beacon node part of the client. The beacon node was not able to respond to what the validators were asking to do. The validator said, here's the block I want to produce. Here's the attestation I want to produce. These weren't getting out the door because the client was so bogged down with trying to process these

• 10:03Speaker 02

  valid but untimely attestations. That's the gist of it.

• 10:08Ryan Sean Adams?

  This happened to two clients, Preston?

• 10:11Speaker 02

  Yeah, well, two clients have released updates specifically to handle this scenario. Uh, that's Prism and Taku. Lighthouse already does it really well. So, uh, a lot of us have sort of taken inspiration from their model of how they handle these old attestations. In short, they, they basically dropped them to say, this attestation is not going to be very valuable to how we're going to update the, um, fork choice. It's not worth recomputing all this, all this effort. So we sort of copied their model of, uh, of just excluding that attestation from consideration, not doing all the work.

• 10:51Ryan Sean Adams?

  And so why did this happen so suddenly?

• 10:55Speaker 02

  Well, um, we're still trying to track down the exact, you know, trigger to the event. But what we think had happened is that some clients, namely Lighthouse, when they were, when they lose a connection to the execution layer, the layer that the client that handles transactions and the, the state of Ethereum as we think of from, you know, the application layer. That being like Go Ethereum or, or whatever you use for your execution client. When we lose access to that, uh, some clients will behave where they'll just withhold these attestations. And then later produce those, um.

• 11:38Speaker 02

  Just sort of later release them. So what Prism does is they just don't. We just don't attest during that time because we can't really. Come up with, uh, we can't advance our node, so we're not, it would just wouldn't really be a testing there. Other clients will produce this sort of old valid, but you know, not valuable attestations. And that was what actually triggered the event. It's not a common path that happens. Uh, but what we have learned is that it wasn't something that was nefarious. It was just. Some sort of unexpected behavior, valid behavior, but unexpected. That Prism and perhaps Tech who didn't consider or handle very well.

• 12:20Speaker 03

  I think a good like a question here, it's like. Why it didn't happen on test net, right? Like, why it didn't happen before? I think there are several reasons to it. The first reason is that there's a lot of validators now. There is six hundred thousand validators. And validator matters, like the number of validator matters, matters, the more valid you have, the more hashing that you have to do on a state and that's slower. And our test that only has four hundred thousand validators. So even the test then is lower. Then the main net, which is obviously a flaw here. That's something that we're taking back and we're gonna revisit. How do we do this type of test net? The second thing is, is there's a lot more deposit now, right? Because, because now we're able to withdraw.

• 13:04Speaker 03

  And then the deposit key with, and then people are kind of. Uh, getting their top of eve, they can really deposit it. And then with so many deposit, there's a lot more hashing that you have to do. That also increase the latency. So those are the two big factors. And once we reach those two boiling point and we kind of just create this, that's, that's, that's spiral.

• 13:29David Hoffman?

  Okay, so to recap what I'm, what I'm hearing, both, um, there were our new conditions about the state of the beacon chain. A higher number of validators are increasing the resource requirements of nodes. And then also from what, uh, from what Preston was saying, uh, Prism and Lighthouse shared similar properties because Prism just borrowed from Lighthouse. As codebase to do this one part of the client that in Prism and because that part of was shared, this showed up this problem that was triggered by the increased computational requirements and bandwidth or networking requirements of the more and more validators coming in that was triggered. But then because of the shared property of Lighthouse and Prism that showed up in two clients, which is how it actually showed up as a non-finalization event in Ethereum.

• 14:14David Hoffman?

  Is that recap accurate?

• 14:16Speaker 02

  Sort of. So we, we, the fix was to replicate what Lighthouse handles this behavior. It was Prism and Taku that didn't handle this scenario very well. So when the outage happened, right, there were something like 60% or more blocks were missing. So that's an indicator that more than one client is having issues. And we believe that to be Prism and Taku because both clients have released sort of emergency updates or critical updates since the incident on Friday and Thursday.

• 14:47David Hoffman?

  Okay.

• 14:49Ryan Sean Adams?

  I have a question in general about what this looks like for Ethereum to not finalize. So here is a beacon chain visualizer we've shown before on bankless. I think specifically when the, you know, hard fork upgrades happen, you sort of look for these things to go from, from red to green, right? These are proposed blocks that we're looking at. So when the blockchain doesn't finalize, what does, when Ethereum didn't finalize on, I guess, you know, it was Thursday once for a period of time, and I don't know how long that was. And then also Friday, what, first of all, what does this chart look like?

• 15:30Speaker 02

  Well, the, the main indicator is that the, the green bar you see at the bottom of each epoch, justifying or finalize, that's not going to be greater than two thirds. It needed to be greater than two thirds to justify or finalize. In the case of the incident last week, we were missing a ton of blocks. So you can see a couple of red spaces here in the blocks. Those are missing blocks. It's a proposer, whoever they may be, failed to. To produce a block at that slot. And because there were so few blocks, you know, more than half of them were missing. It wasn't possible to pack in all of the attestations and also very likely the same nodes that were suffering from

• 16:13Speaker 02

  not being able to produce the bugs, we're not able to produce the attestations either. So two parts to it, you have to have enough blocks to include attestations and you have to have enough attestations that agree with each other, that being more than two thirds to finalize and justify and finalize an epoch.

• 16:31Ryan Sean Adams?

  And how long were these periods of time where this sort of degraded finalization service was happening?

• 16:38Speaker 02

  And the first time on Thursday, I think it was for four or maybe, maybe five epochs that we had a gap in finality. Usually we're seeing every epoch is finalizing. You know, two epochs later. So in this screenshot, you can see epoch sixty-three is finalized. When the current epoch sixty-five finishes, we're gonna see the sixty-four will finalize and we just have this sort of continuing rotating. Um, you know, the epoch two epochs ago is finalizing. Well, when it wasn't finalizing on Thursday, it took five epochs to get back on track. And then on Friday, it took even longer.

• 17:20Speaker 02

  I don't recall the specific number, maybe Terence knows, but what was significant about Friday is when it goes longer than five epochs, we start to incur a additional penalty that everyone gets penalized. And it's called a, um, a, a, a liveless, liveness leak, I think. Um, Terence, do you want to explain? In detail.

• 17:44Speaker 03

  Sure. So we, we have this inactivity dig. So this inactivity dig is essentially what make Ethereum World War Three resilience, right? So imagine we cut the internet in half. And then there's this, say, Asia and then USA, and they cannot talk to each other, right? In that case, we want, we want either side of our validator to dig their balance faster, to lose their balance faster, so we can finalize faster on each side. So you will begin to lose balance faster if we don't finalize for more than four to five epochs. And then, and then eventually after seventeen days, you will loss probably around sixteen ETH, then you will get ejected from the pool.

• 18:28Speaker 03

  So this is the first time this inactivity, um, uh, inactivity did happen. So I did some math earlier. So we ended up burning about twenty eight ETH. And I don't know if this is something that we should be proud of, because everyone loves spent ETH, but this is slightly different. So on average, every validator, if today, if you are offline at that time, you lost about one dollar. So, yeah.

• 18:55David Hoffman?

  Okay, just to double down on that mechanism, the idea here is Ethereum's philosophy is to withstand World War III. So if we imagine the internet as a gigantic cell and that cell gets cut in half, Ethereum should find a way to, to bridge the gap between the division of the internet. And the, the way here is that if, if there can't be consensus between the nodes across the world, there are, the assumption is that there's going to be at least a few nodes that can span that gap. And for all the nodes that cannot span that gap, they will get inactivity leaked and down until they are kicked out of the validator set. And then the few remaining nodes that can span east and west or whatever division of the internet there is, they will be the ones left standing who are not getting inactivity linked because they are the ones still propagating the blocks.

• 19:39David Hoffman?

  And that's the mechanism that, that entered. Is that correct?

• 19:42Speaker 03

  Yep, that's, that's right.

• 19:44Ryan Sean Adams?

  Well, at least we got to test the inactivity leak, right, guys? There's a silver lining here. I don't think we've ever tested that in the wild. And so this inactivity leak came into play for those who were staking with, was it, is it just Prism and Taku and Lighthouse kind of carried on? Was it just a sort of a subset of, uh, client users?

• 20:04Speaker 03

  I think it's hard to tell. I think every client had some issue at that time because of this high CPU usage. So they're sending their attestations, but their attestations may not be received, may not be processed. So I think like every single client, unless today you run your client on a very, very powerful machine, that that you are likely break even or you'll lose a few cents.

• 20:29Ryan Sean Adams?

  I got it. Okay, so what happened? That that's the view of sort of from the from the client and high level blockchain perspective. We've seen that. What about the average user? Were what happened during this time? Were they able to get transactions through or what happens to users using the using the network? Did they even notice?

• 20:49Speaker 03

  So I guess put it this way, right? If if today it wasn't for Bitcoin Chain's floor, if it wasn't for that side, will we ever find out the issue? Maybe yes, maybe not. But I don't think we'll find out the issue at that time, right? We'll probably find out the issue like a few hours later when someone was using this finality notion before, right? So so with that, basically, we can say that, yeah, it probably doesn't affect most people if today you're you were doing a swap on Uniswap, you're buying something that is, those still goes on. But if today, say today, you're actually using finality, which is a pretty rare set of application, I would say, maybe there too.

• 21:31Speaker 03

  Maybe like, maybe like it's changed as a big amount of withdrawal, then that would be affected. But that's a kind of a small set of the use case. I mean, was it acceptable? No, I don't think it's acceptable for Ethereum to be this global internet of money, which we cannot have this, right? But I'm really happy that I'm really happy that life is still goes on because that's probably like, I would say, close to ninety five percent of the. Of the use case out there.

• 22:02Ryan Sean Adams?

  The transactions were going through completely. It wasn't as if there was a pause in sort of user transactions. It was just this one finality piece that some sort of, you called them settlement applications might use. It might be a layer two, or it's an exchange, maybe doing a big batch process with a lot of money and they, they want to wait until full finality is achieved before they, before they settle it. Those are the kind of apps that were maybe affected, even though that's even hard to see because this is so brief, but users weren't affected at all, like the regular users and liveliness was preserved through this event. Preston, looked like you were going to add something to this.

• 22:41Speaker 02

  Yeah, I wanted to add that the real impact to, for the end users was that there was a diminished amount of block space, right? So when we're packing transactions into blocks, what drives the gas price of Ethereum is the demand for block space and the availability. So I did notice that at least on Thursday, gas prices were slightly higher than the daily average. It was something like 120 Gua, at least on the epoch that I checked, which, you know, in terms of the scale of Ethereum gas pricing is kind of a negligible increase. Maybe it

• 23:20Speaker 02

  was 100 epoch before and now it's 120. We've seen a lot of, you know, daps push the gas price much higher for different reasons. So it didn't seem like anyone really would have noticed, right? It wasn't astronomically high. It's just slightly higher than average during that time due to less available block space for transactions. But everything was still going through pretty, in a timely fashion. You may have noticed your, your transaction took up to a minute, but where it normally takes, you know, 12 seconds, but that was really, really, you didn't really feel that it was down or broken in any way.

• 23:58Ryan Sean Adams?

  So in that case, what's the big deal here? Like, why do we care about finality? What, what would, would have been, what could have been a worse condition? What, what, could we play out a scenario where maybe this happens again and then something actually bad happens? Like, what's the big worry here?

• 24:15Speaker 02

  Well, the concept of finality in proof of stake is is one of the main features, right? It means that we have finality in the blockchain and that we can be assured that that point in time is is final and can never be rewritten, or we cannot go back in time, build on a prior block and sort of exclude this history that's happened. As Terence was saying, that's very important for some transactions. So if you're doing maybe a very large purchase, let's just say, you know, we're at a point where you can purchase property in the real world. You say, I want to close on this property. I'm going to send you my ETH and then I'm or or whatever it might be my tokens.

• 24:59Speaker 02

  And I'm not and we're not going to consider done until there's finality, right? So there there is that aspect where we want it to be final, cannot be rewritten. It's also part of the entire security of Ethereum, right? If if Ethereum aims to be the global settlement layer, you need to have this consistent finality in a timely fashion. And it can also affect, you know, layer twos or other networks that are leveraging Ethereum's security. I was imagining a scenario where what if this played out for a very long time, right? We have optimistic rollups out there with a seven day, I think, a seven day window for the challenge period. So you can imagine that if this incident went longer than seven days,

• 25:44Speaker 02

  what does that mean for the challenge period? Even though it has resolved, it's not entirely final because it's not final until it's finalized. And I thought about that as being sort of like the the medium term worst case. I mean, the the obvious worst case, this goes on longer and longer until we have so much inactivity leaked that a lot of validators are ejected. Eventually Ethereum recovers. L twos would recover and those challenges would eventually be finalized. But you could see that would be a major disruption if it went on longer than seven days.

• 26:19David Hoffman?

  Can you like put us into the headspace of Preston when this was happening? Like, were you freaking out? Were you like, oh, that's a curiosity? Just like, sometimes, yeah, we can talk about like the technical details of finality, but what about the emotional response?

• 26:35Ryan Sean Adams?

  I can give you guys the response of people on Twitter, which was mass panic, freak out, oh my god, Ethereum is broken. That was the response on Twitter, but yeah.

• 26:47Speaker 02

  Yeah, so this is, when it happened on Thursday, it was around like 4pm my time, and I'd been starting my day very early, and I was like, I'm gonna go, I'm gonna go take a nap, like, I'm kinda tired, right? And then I just checked right before I go to make sure nobody needs me for anything, and I see this message I never want to see again from Nishant, he's on our team, Nishant Das, he's sent me these a couple of times. He says, "Maintenance, stop finalizing." And I said, "Oh my god, what? There's no way I'm gonna be able to sleep for days."

• 27:20Ryan Sean Adams?

  Those three words, those three words.

• 27:22Speaker 02

  Maintenance, stop finalizing. And this is how I found out about it the first time. So this isn't the first time mainnet has had a finality delay or finality incident. It happened a few years ago where Prism was unable to produce blocks, and Prism was a supermajority, thanks to client diversity, or not have that risk anymore. But the same message he sent me, he said, "Maintenance, stop finalizing." And that time, it took us, I think, three days to figure it out, to figure out what the issue was, and we were sleeping in rotating shifts, like, maybe a few hours at a time. So I, when I thought, okay, I'm gonna go take a nap, and then he said those three words, I said, "Great, I'm just never gonna sleep again." So emotionally,

• 28:07Speaker 02

  that was, that was challenging.

• 28:10David Hoffman?

  Well, I think this does a very good job illustrating why client teams are so important and why client team maintainers are heroes that should wear capes. So Preston, Terence, thank you for always wearing a cape, even though you don't actually legitimately, literally wear a cape. There's a lot more that I want to ask about, but first, we have to talk about some of these fantastic sponsors that make this show possible, especially MetaMask. If we have used jargon in this episode that is confusing to you, I bet you MetaMask Learn has an explainer for you on their product. There is a link in the show notes. Here we go. Learning about crypto is hard, until now. Introducing MetaMask Learn, an open educational platform about crypto, Web3, self custody, wallet management, and all the other topics needed to onboard people into this crazy

• 28:55David Hoffman?

  world of crypto. MetaMask Learn is an interactive platform with each lesson offering a simulation for the task at hand, giving you actual, practical experience for navigating Web3. The purpose of MetaMask Learn is to teach people the basics of self custody and wallet security in a safe environment. And while MetaMask Learn always takes the time to define Web3-specific vocabulary, it is still a jargon-free experience for the crypto-curious user. Friendly, not scary. MetaMask Learn is available in 10 languages with more to be added soon, and it's meant to cater to a global Web3 audience. So, are you tired of having to explain crypto concepts to your friends? Go to learn.meta mask.io and add MetaMask Learn to your guides to get onboarded into the world of Web3.

• 29:37David Hoffman?

  Hiring people worldwide, paying them in crypto, providing them access to benefits, it all's so complex. But it doesn't have to be. Complying with labor laws, payroll rules, tax obligations, and crypto regulations in every country that you employ someone is difficult, time-consuming, manual, and costly, and is drawing more and more attention from regulators and governments. But, there is good news. Toku is here. Toku is the first employment and compensation platform for the crypto industry that makes this easy. Toku helps you hire employees or contractors and pay them in fiat or crypto legally, compliantly, and with all the taxes handled in over 100 different jurisdictions. So whether you're an early-stage company with just a team of two, or you're an enterprise with 200, Toku has a solution that meets your needs.

• 30:21David Hoffman?

  Toku is already working with the leading companies in the space: Protocol Labs, Hedera, Gitcoin, and many more. So, transform your employment and token payroll operations with Toku. You can reach out to Toku at toku.com/banklist or click the link in the show notes. Introducing ETHX from Stater. ETHX is a liquid staking token designed to maximize rewards, all while securing Ethereum. With Stater, you can run an Ethereum node with just four ETH, an 85% lower capital requirement versus the 32 ETH required for solo staking. With Stater's four ETH nodes, you can get a 35% average higher yield, since you charge fees to those who use your node to stake their ETH. By running a node with Stater, the ETHX staking derivative token can get minted on your validators and can flow into the world of DeFi,

• 31:05David Hoffman?

  which Stater is actively building integrations and partnerships into to increase the utility of ETHX. Stater allows for both permissioned and permissionless nodes to join the network, maximizing its potential scalability for ETHX, while preserving the values of decentralization and openness behind its liquid staking token. Go to staterlabs.com/eth and sign up to get access to the Stater staking protocol. Arbitrum One is pioneering the world of secure Ethereum scalability and is continuing to accelerate the Web3 landscape. Hundreds of projects have already deployed on Arbitrum One, producing flourishing DeFi and NFT ecosystems. With the recent addition of Arbitrum Nova, gaming and social dApps like Reddit are also now calling Arbitrum home.

• 31:46David Hoffman?

  Both Arbitrum One and Nova leverage the security and decentralization of Ethereum and provide a builder experience that's intuitive, familiar, and fully EVM compatible. On Arbitrum, both builders and users will experience faster transaction speeds with significantly lower gas fees. With Arbitrum's recent migration to Arbitrum Nitro, it's also now 10 times faster than before. Visit arbitrum.io, where you can join the community, dive into the developer docs, bridge your assets, and start building your first app. With Arbitrum, experience Web3 development the way it was meant to be: secure, fast, cheap, and friction-free.

• 32:22David Hoffman?

  And we're back. Now, Terence, I'd like to turn to you. What was your story? What is it like, from your perspective, to hear the words "mainnet stopped finalizing" as a client team dev?

• 32:32Speaker 03

  Yeah, it's definitely not great hearing that on Thursday afternoon. Typically, I stop all my meetings after Thursday, so I decided to get coding, have some personal time, read some paper. But, um, so I, yeah, so my president did, I heard from the news, I heard from Telegram. And to lose sixty-five percent of participation, it's, it's a little crazy. Like, when I first read that, I just couldn't believe that. I was like, just how that happened, right? It's like, imagine like sixty-five percent of the nodes disappear, right? It has to be like, that, this sounds very like catastrophic if you really think about it. So, in my mind, I just, I started piecing everything together quickly, right?

• 33:16Speaker 03

  Cause like, sixty-five percent, then I know it's not just one client. Cause like, thanks to client diversity, like, every client is pretty spread even. My president is, sorry, uh, president is thirty-three percent, my house has like thirty-three percent. So I know it's not just one client, so it's not a client bug. Then I thought, okay, maybe it's like a, maybe it's like a target attack, right? And then, maybe it's like, they're really like, someone is attacking the network. Finally, someone is actually attacking the network. And then, but luckily, like, the whole thing, like, went back to normal after like three to four epoch, which is like twenty minutes. So they kind of gave us more time to think about what the issue is, right?

• 33:58Speaker 03

  And then I thought, okay, well, if it's, it's a target attack, like, why did this stop? If, I mean, if they could have done it for longer, like, why stop there? So I, so I didn't really think it was a target attack at that point. Then I thought, okay, maybe there's like a really big exchange or staking pool that runs something on GCP and AWS. And then they shut down or something like that. So that was another reason, but I luckily, I think the whole thing took about like four to five hours. Until we came to the gist of, gist of it, that's basically our software was a little bit, um, under, uh, uh, under, uh, optimized.

• 34:39David Hoffman?

  So how hard of a problem was this to fix? As soon as so it sounds like diagnosing wasn't too difficult, but what was the actual solution process like?

• 34:49Speaker 03

  It wasn't hard to fix. We could have fixed the problem within the first thirty minutes, but we kind of wanted to do the perfect fix, right? Because everything is a trade off. Client software is a trade off. If you optimize one thing, you cannot lose another thing. And this is why client diversity is nice, right? So I'm gonna shout out client diversity again, just Lighthouse. Work fine because of the trade off they they made, right? Today, network is under another scenario. Lighthouse may underperform like versus prison, just everything is a trade off here, right? So we kind of, we kind of picked the fix that we thought had the right trade off that where we like.

• 35:31Speaker 03

  And then again, this is a team effort, like, like probably like five to ten of us, like we got together. We talked about it and then we thought, okay, this is the fix that we want. We also share our fix, our idea with five other awesome client teams as well, right? And then because we're all learning from each other, we are all study each other's design. So, yeah.

• 35:54Ryan Sean Adams?

  Well, one interesting observation I have about this is this is, um, this was sort of a finality event, but it was, but it's also like a non-event. Right? Like, uh, when we asked the question at the beginning of this episode, Preston did Ethereum break and you were just emphatically, no, it didn't break. And so this is maybe a demonstration of the resilience of the Ethereum network. And I, I will.

• 36:21Ryan Sean Adams?

  What came into play? Like, how did the Ethereum network show resilience during this event like that? Terence, you just talked about multiple clients. Was, was that the key thing? Is that the thing that that saved our ass? I guess I would say, or are there other elements that demonstrate the, the resilience of Ethereum? First to you, Terence.

• 36:40Speaker 03

  That's one thing, right? If today it was three years ago when Prism was the majority client, I will be really, really worried, right? But today, we're just like a piece of the pie and I'm also therefore I feel like I can sleep a lot better. The second thing is that we really need to think like this, um, this thing is called LMD Ghost. This is designed by Vitalek and a few brilliant researchers from the Ethereum Foundation. So LMD Ghost is this consensus mechanism, is this for choice rule that we use for E2 or uh, or the Bitcoin chain, basically. This is, so essentially, uh, the Bitcoin chain Ethereum proof of stake consensus is made up two part.

• 37:24Speaker 03

  The first part is LMD Ghost, essentially it favors liveness. It basically ensure today, if we don't have safety, we don't have finality, the blockchain keeps going. And this is very important, right? As it turns out, because before I, I always question, why do we need this? Why can't we just have safety? That's it. Why can't we be like a tandem in chain? Why can't we be like, maybe? Like the other blockchain, they just have safety, right? And this is nearly the reason, this, this is why, because like if things goes wrong, if we don't have finality, we don't wanna affect the use, we don't wanna affect the UX, right? We want people still be able to buy NFTs, they'll be buy copies, they'll be made trade.

• 38:06Speaker 03

  So essentially, to answer your question, this LMD Ghost fortress rule was the one that saved our but.

• 38:13Ryan Sean Adams?

  It, it prioritized liveness. That's what it did specifically. Preston, what would you add to that? What saved us?

• 38:19Speaker 02

  Yeah, I think Terence said it well. It's the specification of Ethereum, LMD Ghost and these things that make Ethereum uh, the way it is, combined with client diversity, right? Like the specification worked here and when there's an incident, it can recover, it can live on. And because there was an incident and we're having client diversity, it was minimized to a subset of the network, right? It could have been worse if it was more of a majority prism. It brings me back to the days when we had these early test nets like Medasha, which was a hundred percent prism and it was just on fire. It was it was terrible when there were incidents and you know,

• 39:02Speaker 02

  thankfully client diversity, we've done a really great job over the last two years, three years, however long to smooth that out. And we could see here today that there was an incident and it recovered on its own without intervention within minutes. And that we were able to uh release mitigations and hot fixes and client updates within days to say, okay, now it's not going to happen again. So I think that went really well. It's a great test in main net.

• 39:32David Hoffman?

  So uh I'm going to pull in uh something from my Bitcoiner world. Um they always love to say this is good for Bitcoin. Uh and the spirit of that, that, that response is that anti fragile crypto economic systems, what doesn't kill them makes them stronger. So Preston, um was this good for Ethereum?

• 39:51Speaker 02

  Yeah, I think, you know, like we're saying, this is the first time we've seen it in main net in production. We knew it would work. We've done it in test nets, but actually see it i- in in action.

• 40:04Speaker 02

  It just goes to show the resiliency that when things are going horribly wrong, it can still be okay and it's still um eventually finalizes and that the mechanisms and the specification and the design of Ethereum proof of stake is resilient. So I thought that was great.

• 40:21David Hoffman?

  And also.

• 40:22Speaker 02

  Yeah.

• 40:24Speaker 03

  So like Ethereum is an evolving protocol as we know, right? It is still constantly heavy under heavy R and D, right? And then we definitely came out here with like a very better understanding on just like very little detail of things that we want to improve of even like future research spec as well. Just I heard your podcast on um M V B Boom, right? This is very good podcast, but because of this, right, we're kind of looking at M V B Boom, we we look at P B S to see how we can do better there as well, because everything is interconnected, like they all, they basically all touch each other. And I definitely feel like we definitely got all of this very strong.

• 41:07David Hoffman?

  And not only that, but uh clients have discovered a weakness that is now removed, correct?

• 41:15David Hoffman?

  Yep.

• 41:17Ryan Sean Adams?

  Cool. Nice. I love anti-fragile systems. My last question is this, and this goes back to Terence, something that you were saying about kind of the the design of Ethereum proof of stake with LMD Go. So it's like way in the weeds for people, but like the priorities that the Ethereum network makes. As we all know, it's not high transactions per second on the base layer on mainnet, right? Instead, Ethereum has made other trade-offs. And one of those trade-offs is liveness, a liveness trade-off. And I want to just focus for a minute on some other outages that have happened in crypto with other Layer 1 blockchains. And we don't have to name names. It's like, I don't want us to,

• 42:02Ryan Sean Adams?

  like, anyone to feel like, you know, the Ethereum's punching down, that kind of thing. Okay. So like, but like, Just just think of the first name that comes to your mind about a blockchain. But but also, just like, just the facts. Okay. So like, I think when people were talking about, and by the way, Bankless included, we're sorry, everyone listening for the the tweet headline. All right, we've made amends on that. That happened this week. But when people equate Ethereum breaking to sort of the downtime and the breaking or temporary halting of other blockchains, it's much different. So I'll just throw the name Solana out there, just just for a minute. We can talk about that as a just the facts kind of use case here. When Solana was

• 42:44Ryan Sean Adams?

  down, because it did not prioritize liveness, when it had some some issues, it was actually offline, right? Like, you couldn't get a transaction through for a period of time. I'm wondering if you guys could just just weigh in on that briefly, because it is a useful contrast point. Okay, Ethereum doesn't have the highest transactions per second on the mainnet, but it does have this feature of liveness. Terrence, could you talk about that for a minute?

• 43:08Speaker 03

  Yep. So I just wanted to add, right? Like, there's nothing like what we are doing today, right? We're working on opens for software, and this is the open network. That means anyone can read any code that we write. We're like chefs, we're cooking in the kitchen, right? And anyone can join the network, right? So it makes you very like vulnerable to attacks, therefore, right? Therefore, I think favoring liveness is very important, just because like you want your life to keep going, right? Say today you have a country that's built on top of Ethereum, I wanna buy coffee. And then I, I, I mean, today Ethereum is dumb. We cannot process transaction, I cannot buy my coffee, then I will be very upset, right?

• 43:52Speaker 03

  So I would say liveness is a very important property, probably the most important property. If you want higher throughput, there is layer two, right? And that's what layer two is for. But Ethereum as a base layer, definitely a liveness.

• 44:06Speaker 02

  Yeah, you know, I think that's what makes a blockchain great is to have liveness, right? It's not useful if you can't use it. So to have liveness is, I think, the most, one of the most important uh, aspects of Ethereum, more so than transactions per second, right? We have solutions for that. We can obstruct to layer two or, or other clever mechanisms. But if you don't have liveness, then none of that works. You have zero transactions per second. So it is the most important uh, aspect of Ethereum.

• 44:38Ryan Sean Adams?

  Preston, Terence, yeah, thanks for summarizing this. Thanks for jumping on the Bankless Podcast, explaining this to the Bankless community and the crypto community. Uh, did Ethereum break? The answer is no. This was maybe- Whoever tweets that is dumb. Yes. This is maybe Ethereum's biggest liveness test yet, though. And so it was good to come out on the other side and see a resilient network. Uh, pretty unique. I think something that's, that's very special in this space and, and worth preserving. And as always, gotta thank you. Whenever we have client devs on, on the Bankless Podcast, we would be remiss, David, if, if you and I did not express our thanks, both to you, Terence, and to you, Preston, and also to any other client dev that is doing this hard and essential work

• 45:24Ryan Sean Adams?

  to build this incredible freedom technology network uh, that we care so much about. Thank you sincerely. Open source superheroes for sure. We appreciate you guys coming on.

• 45:37Speaker 02

  Yeah, it's our pleasure to get to work on it. Really, it is an exciting opportunity. We know that Ethereum is going to change the world and to be working on it at the protocol level. There's nothing better that we could imagine doing. So it's our pleasure to be here.

• 45:52Speaker 03

  Plus one to what Preston said. Thank you for having us.

• 45:55Ryan Sean Adams?

  Recent disclaimers, guys. Of course, got to let you know Ethereum is resilient, but ETH the asset is risky. So is all of crypto. So is DeFi. You could definitely lose what you put in, but we are headed west. This is the frontier. It's not for everyone, but we're glad you're with us on the bankless journey. Thanks a lot.