119 - Dive into Danksharding | Vitalik, Dankrad, Protolambda, Moderated by Tim Beiko

This might be the dankest livestream we’ve done to date. Vitalik Buterin, Dankrad Feist, and Protolambda join Tim Beiko to discuss all things Danksharding and Proto-Danksharding. Sharding what? Don't worry. By the end of this conversation, you’ll know how sharding has evolved over time, why danksharding is possible, how proto-danksharding will get deployed, and so much more. ------ 📣 ALCHEMIX | Get a self-repaying loan today! https://bankless.cc/Alchemix ------ 🚀 SUBSCRIBE TO NEWSLETTER: https://newsletter.banklesshq.com/ 🎙️ SUBSCRIBE TO PODCAST: http://podcast.banklesshq.com/ ------ BANKLESS SPONSOR TOOLS: ⚖️ ARBITRUM | SCALED ETHEREUM https://bankless.cc/Arbitrum ❎ ACROSS | BRIDGE TO LAYER 2 https://bankless.cc/Across 🏦 ALTO IRA | TAX-FREE CRYPTO https://bankless.cc/AltoIRA 👻 AAVE V3 | LEND & BORROW CRYPTO https://bankless.cc/aave ⚡️ LIDO | LIQUID ETH STAKING https://bankless.cc/lido 🔐 LEDGER | NANO S PLUS WALLET https://bankless.cc/Ledger ------ Topics Covered: 0:00 Intro & Danksharding Explained 7:00 Panelist’s Backgrounds 8:34 How Sharding Has Evolved 15:56 Why Danksharding is Possible 23:40 Proto-Danksharding 27:15 Data Availability 35:40 Data Availability Sampling 48:00 Cryptography for Noncryptographers 56:10 How Does All of This Get Deployed 1:00:05 The Big Open Questions in Sharding Land 1:05:30 Closing Thoughts ------ Resources: Vitalik https://twitter.com/VitalikButerin Dankrad https://twitter.com/dankrad Protolambda https://twitter.com/protolambda Tim Beiko https://twitter.com/TimBeiko Proto-Danksharding https://www.eip4844.com/ ----- Not financial or tax advice. This channel is strictly educational and is not investment advice or a solicitation to buy or sell any assets or to make any financial decisions. This video is not tax advice. Talk to your accountant. Do your own research. Disclosure. From time-to-time I may add links in this newsletter to products I use. I may receive commission if you make a purchase through one of these links. Additionally, the Bankless writers hold crypto assets. See our investment disclosures here: https://newsletter.banklesshq.com/p/bankless-disclosures

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:09Ryan Sean Adams?

  Hey, Bankless Nation. Welcome to our special livestream. This is going to be a panel addition. I think, David, this is going to be our dankest panel, maybe the dankest episode we have ever recorded, because we are talking about dank sharding today. And I gotta be honest, going to this episode, I'm not entirely sure what it means. Like, what is dank sharding? But we'll ask some of the participants this. And maybe, David, you could describe the setup of this panel, and who's on it, and how we're going to handle this episode.

• 0:37David Hoffman?

  Yeah, of course. You guys all know Tim Baker. He's the guy on the bottom of the screen. Tim manages All Core Devs Call and is coordinating the lead into the Merge and Beyond. And we've had Tim Baker on before. During, he led a very technical EIP 1559 panel, and asked the questions that Ryan and I are just not smart enough to ask. So this is one of those panels. We're going to get as technical as possible. We have three fantastic panelists who are behind the scenes. We've got Vitalik, Dankrad, and Proto, who are the minds behind dank sharding, and sharding in general, and other Ethereum-related technologies. And Tim is going to be able to ask questions that are the technical questions, the smart questions. But before we get there, Tim, we just wanted to cover some high-level stuff. What is dank sharding?

• 1:22David Hoffman?

  How to get its name? And just like, what does it mean for users?

• 1:26Speaker 02

  Great. Well, first, yeah, thanks for having me on, guys. So dank sharding, and proto dank sharding, which we'll also get into, are iterations over the sharding design for Ethereum. And we'll spend, you know, the bulk of this panel discussing what they are, what the trade-offs are and whatnot. But at a high level, sharding is a way for Ethereum to have more data pass through the network. And because Layer 2 solutions, like ZK rollups and Optimistic rollups, they produce it on a data, if we have a way for them to post that to the Ethereum network more cheaply, it immediately reduces how much they need to charge users for transactions. So all of these kind of flavors of sharding all have

• 2:10Speaker 02

  kind of the same end goal, which is to create a cheap place for Layer 2 solutions to post data on Ethereum. And the impact of that is that the transaction fees that end users end up paying on Layer 2s is lowered by a lot.

• 2:25David Hoffman?

  So this is all about getting transaction fees down, particularly on layer twos, gas fees down on layer twos. And, and Tim, is this related, you mentioned proto dank sharding and dank sharding, you guys will talk about all that in the panel, but is this related to EIP 4844? Because that's another EIP we've heard a lot about.

• 2:45Speaker 02

  Yes, so EIP 4844 is proto dank sharding, basically. And one way to think about it is like, proto dank sharding is like, maybe the first step we get towards sharding, and then big sharding is a simplification that we'd had on the previous roadmap. So like, the more prefixes we have, it's like, the sooner we get them. Yeah.

• 3:06David Hoffman?

  All right. And this has been, like you said, an iterative cycle, an iterative process to get to where we are today. And dank sharding comes from Dankrad, who's on the panel, and proto dank sharding comes from Proto, Proto Lambda, who's also on the panel. So these guys' names have been baked into the name of this EIP itself. Tim, is this the new EIP 1559? Is EIP 4844 going to be the new EIP that we focus about going into the future?

• 3:34Speaker 02

  So there's a lot of stuff we're working on right now. So, you know, it's definitely one of the big ones. And I think for end users, it's one of the most impactful ones, because it directly affects like the gas price you pay. Hopefully, it's much less contentious than 15559 was. But yeah, it's definitely an EIP you're going to be hearing more about over the next few months.

• 3:54David Hoffman?

  And just one last question before we hand it over to the panelists and we bring the panelists online. There's been a bunch of conversations about, you know, when EIP 1559 came and go, people were like, is this going to reduce gas fees? And the answer was no. And then people are like, the Ethereum merge, is that going to reduce gas fees? And the answer is no. This EIP reduces gas fees. This reduces transaction fees, not for the layer one, but for the layer twos, correct?

• 4:18Speaker 02

  Yes, on Air 2s. Yes, that's correct. And basically 4844 is a way for us to get some of the reductions of sharding quicker. And then the full dank sharding rollout gives us even more reductions. But because 4844 aka proto dank sharding is simpler to implement, we can just get that first.

• 4:36David Hoffman?

  Fantastic. All right. I think that is all of my questions and I think that's all of Ryan's questions. So with that, I'm going to ask the panelists to come in from the shadows and turn on their cameras. And me and Ryan are going to actually duck out of here. Excuse me. We're going to dank out of here. And let Tim take over this stream. Guys, welcome to the panel. And Tim, thank you for doing this. And then absolutely just take it away.

• 4:57Speaker 02

  Awesome. Thanks. Um. Okay, sweet. It's just us now. Um, so I guess, yeah, before we get into it, do you each want to just take a minute and kind of talk about like what you work on and who you are? Yeah, Vitalik, we can talk with you.

• 5:16Speaker 03

  Um, yeah. So, hi. I'm Vitalik. I'm the co-founder of Bitcoin Magazine. I write a blog. I contribute to specs once in a while.

• 5:25Speaker 02

  Dankrad.

• 5:29Speaker 04

  Yep. Hi, I'm Dankrad. I'm an Ethereum researcher since 2019. And I'm working on, among others, sharding and

• 5:41Speaker 04

  what else? I work on proof of custody, statelessness, some projects on the roadmap of Ethereum.

• 5:50Speaker 02

  Nice. And Prado?

• 5:53Speaker 05

  Hey, hello. I'm Prado Lempa.

• 5:56Speaker 05

  I used to work at Ethereum Foundation on research there. Now I do the same thing, but I'm not Optimism. I helped with sharding earlier on and now I'm contributing back to Layer 1 while working at Optimism and Layer 2.

• 6:12Speaker 02

  Sweet. Okay, so just to kind of get into sharding generally. Over like the past couple of years, what sharding means that Ethereum has changed a lot. And I think like the biggest one is like this shift from full execution sharding, the only data sharding. Vitalik, do you want to give us just an overview of like how that shift happened in the research roadmap and why we've landed on just doing data sharding?

• 6:42Speaker 03

  Yeah, so I think there's been this ongoing simplification of the sharding roadmap that started really sometime in 2016. So for people who have been in the Ethereum ecosystem for a long time, you might remember some of the scaling docs that we published back in 2015, back in early 2016. Some of the blog posts thinking that came out in 2014. And the stuff that was there in those earlier periods was in a lot of ways really complicated, right? Like there were these ideas around hypercubes and hub and spoke chains and

• 7:21Speaker 03

  in protocol-supported cross-shard transactions that would be routed between like one corner of a hypercube to another corner and where the protocol would help them move from like one shard to another, to a third, to a fourth. There was even thinking about super quadratic sharding, which is basically saying like instead of just having shards, you have shards on top of shards and potentially like in infinite hierarchy of shards instead of shards instead of shards. So actually the sort of stuff that the Telegram Ton project ended up incorporating into their paper, though I guess that it never really ended up coming close to going live, unfortunately.

• 8:02Speaker 03

  But that was the kind of thinking that we had back in 2015 and 2016. And I think after that, the progression has just abandoned this like big, slow process of I guess, increasing pragmatism, increasing appreciation of how complicated it is to develop and actually bring to production just about anything. Like what feels like 10 lines of code actually becomes like hundreds of lines of code once you add all of the A couple of cities that clients have to inevitably have and deal with. You know, how some particular thing interacts with the syncing process, how some particular thing interacts with the fork choice, how

• 8:42Speaker 03

  some particular thing interacts with the database and the need to store it in optimized formats and that sort of stuff. So the process of simplification, I think the big first step was definitely the decision to not bother with anything beyond quadratic sharding and just say we're doing quadratic sharding, right? So not bother with us with doing ever any kind of shards on top of shards and just saying, we have the beacon chain. We have shards, shard headers are connected to the beacon chain and that's it. Like that's the only layer of sharding that ends up actually happening. So that was the first step. Then the second step was the move from this

• 9:26Speaker 03

  concept of like chains that have regular commitment blocks in the beacon chain. I think there was another word for them. I forget what, what, what, what that word is. A cross links was the word, right? You have shard chains that cross link into the beacon chain. So move it from that to a system where you just have every shard block directly getting included in the beacon chain. So that was the second simplification. I forget exactly when that happened. I think that might've been around 2019 or 2020 or so. And the big benefit of that simplification is that it meant that we didn't have to worry about shard chains anymore.

• 10:05Speaker 03

  Then after that, there was the idea that we're going to do data sharding first. This was when I started talking about the rollup centered roadmap, right? Basically, instead of shard blocks actually containing transactions that would be executed at the Ethereum layer, these shard blocks would just contain big blobs of data. And it would be the responsibility of layer two rollup protocols to use that data space in order to create secure and more scalable experiences for their users, right? So Ethereum, the system was. Provide non-scalable computation and scalable data.

• 10:45Speaker 03

  And what a rollup does is it converts scalable data and non-scalable computation into a scalable computation. So we have a somewhat more performant layer one that has extra data space. And then we combine that with this layer two ecosystem and the layer two ecosystem ends up like really bringing all of the scalability to life. So that's the rollup centric roadmap and at the beginning, I think the rollup centric roadmap was phrased in this ambiguous way where it basically said, well, look, the rollup centric roadmap realistically, like data sharding is the obvious prelude to full sharding anyway, right? Like if we're going to implement full sharding with EVMs and all the shards, it's just an obvious first step to have data shards first.

• 11:28Speaker 03

  But it turns out the data shards are actually really good for rollups already. And so we might as well run with that. We might as well realize the rollups are our best hope for short-term scalability. And and just take that direction and try to make the best of it, right? And that still leaves open the door for adding EVM execution shards in the future. But it basically says, well, actually, you know, Ethereum will be fine even if we end up never actually completely doing that, right? So that's rollup centric roadmap. That's another simplification, basically saying we don't have to bother with execution on shards. And then that also allowed some other simplifications, like it made it even

• 12:11Speaker 03

  more possible for shards to not bother with fortress rules, for example. And then the next simplification after that is DING sharding, where it basically said that there is actually this merge proposal mechanism where there's only one proposer that chooses all of the shard blocks and all of the shards that appear within a particular beacon block. And that simplifies things massively in a whole bunch of ways. So it basically means we don't have to deal with like the whole shard proposer bureaucracy, which simplifies that can set this complexity a huge amount. It simplifies some of the economic properties a huge amount. It basically makes the system feel much more similar to like just what a non-scalable chain would look like,

• 12:54Speaker 03

  except it's just more scalable, right? And that's the extra scalability happens in the background. And then proto take sharding finally, like that's not a simplification. That's more a step on the way to full take sharding that gets us maybe half the benefits of sharding. But at a point that's maybe you're kind of like halfway along the timeline to actually getting full sharding out there. So we actually get some of the benefits sooner, right? So that was the general progression, like basically more complexity to less complexity, more more of Ethereum trying to do everything to less Ethereum trying to do everything and more willingness to work with way or two protocols. And those two things together, and that's where we are now.

• 13:36Speaker 02

  Nice. Yeah, thanks. That was that was comprehensive. They cried like talking about bank sharding, like, can you kind of walk us through how? Like this idea that it's okay to assume that the block proposers or the block builders have to track all of the shards because of this separation between proposers and builders that we've seen kind of over the past few years, especially with the rise of MEV. So yeah, just talk us through like, why is it possible to do something like bank sharding and not sacrifice the decentralization properties of the chain?

• 14:13Speaker 04

  Right. Yeah. So I guess, I mean, maybe like if we go a bit into the the history of MEV like, or maybe think about how how it has been recently. So like it started with maybe like, maybe some mining pools doing some stuff like to exploit MEV and like to like, yeah, to to make to get some more than just the transaction fees. And over over the course of time, this has become more and more professionalized. Like nowadays, most of them work with some other entity, for example, flash flashbots that

• 14:53Speaker 04

  sells them complete bundles of, or yeah, maybe selling is the wrong word here, that buys complete slots for bundles to be included in blocks that exploit a certain amount of MEV and then and then the block. Users which are currently the miners, they just get the payment for that, like, like, um, so, uh, yeah, so the, the, the searches are now like lots of independent entities that uh, try to find the best MEV and, um, and mining pools now don't have to bother with us anymore. And, um, and basically, um, it turns out that, uh, if you want to properly decentralize this, which is something we really want to do, like,

• 15:36Speaker 04

  um, so right now there's like maybe, um, a few mining pools, like, um, tens of them or so. And so like the way it works right now is that, uh, basically Flashbots has a a business relationship with each of them, like they, they basically have a trusted relationship and, um, if like, uh, one of, one of the two sides did something naughty, like for example, the, um, the miner could start, um, looking into the strategies and instead of, um, just executing them, they could like exploit the strategies themselves. So for example, if you have a an arbitrage transaction, then like you can often, uh, do much worse things with the individual trades if you, if you want to exploit them.

• 16:19Speaker 04

  Um, or like they could try to figure out what the strategies are by, by what they're being sent and so on. Um. So, um, this, this doesn't scale to a system where instead of like a few tens of mining pools, we have, uh, probably tens of thousands or so of individual validators. Um, because you can't have like a trust relationship with each of them, that's not gonna work. Um, so the only way to translate this, uh, world of MEV into, um, into the, uh, future of proof stake is, is by having some form of proposer builder separation.

• 17:02Speaker 04

  The way proposer builder separation works is that, um, instead of having, uh, traditionally like,

• 17:09Speaker 04

  I guess like five years ago, we all thought of it as the same thing. If you propose a block, you build that block, right? And with proposer builder separation, that's not true anymore. What we do instead is uh someone, a builder builds a block. And the proposer just proposes, oh yeah, I'm gonna propose that block that this guy built. Um. So we separated the two roles. And um in in that way, we can have a professional role of block building, building, building, which is uh this role that will extract MEV or work with all the searches and so on. And we can have the proposer and that is just a normal validator. And the good nice thing is proposing is extremely simple and cheap.

• 17:51Speaker 04

  Because it's basically just selecting the highest bid and saying, yep, uh you get to build the block. And uh whereas building is a complex process where you have to manage uh lots of searches and they have to trust your system that uh their strategies won't get exploited and so on. Um. So that that is more suited for a complex and more uh capitalized entity. And so um um it's it's good that not not everyone has to do it. And uh basically, this is uh this is the, I guess, this is how we're seeing the future of block building on Ethereum. There's not, there's not really currently a viable alternative to this world.

• 18:36Speaker 04

  Um. And um. And what we also know uh is that um sort of building this uh, building scalable system, especially also building this massive uh data availability system, uh becomes a lot easier. Once you assume that there is someone who can handle these massive amounts of data. So once you put that entity into the system and say, well, um, there's someone who can compute this encoding, who can distribute all this data and so on. Then many things become a lot simpler. And so in the past, um, I guess we didn't.

• 19:19Speaker 04

  Really think about these designs because we were like, well, we want Ethereum to be extremely decentralized. And now with the, with this proposal, building, builder separation coming into the design space due to MEV, it's also become available to think about for other things. And basically, this is how we, or like I first thought at the end of last year, well, let's use this, let's use this idea where you have these entities that can handle, for example, large amounts of data. It's not really a problem if you are like running some large machines anyway. It's not like an absolutely insane amount, it's not data center, center kind of amount. It's just like large machines with a good internet connection kind of amount and

• 20:06Speaker 04

  and exploit these entities and let them basically do this building and that allows us to get to a much simpler and more efficient charting design.

• 20:19Speaker 02

  Got it. And so am I right in thinking like, because it's basically very hard to build an optimal block that becomes a specialized industry. But once you do have a block that is, it's very easy to verify its validity, right? Like so,

• 20:33Speaker 04

  Right.

• 20:33Speaker 02

  you know, finding the exact right block to build is hard. And so you need, you know, tons of machines to do that. But once you have found it, then anyone can verify it. It's almost analogous to proof of work in that way. For like, yeah.

• 20:47Speaker 04

  Yeah, yeah, that's absolutely correct. Like basically verifying, I mean, that's, that's the, that's the crux of data availability checks, this idea that there, there is a way to check that this amount of data is available that needs much less work than actually downloading all the data. So there's this asymmetry where like there's someone, somewhere we need to do this work of encoding the data and distributing the data. But then verifying it is much easier.

• 21:17Speaker 02

  Got it. Um, okay, to get us all the way to proto dank sharding now. Uh, so recently, the three of you have have written an EIP, EIP 4844. That's now colloquially known as proto dank sharding. Which helps kind of lay the foundation for this full sharding design. Without require, without requiring this entire kind of shard data network to be live. Um, Pro, do you want to walk us through like? What are the things that 4844 does to get it towards sharding? And also like, how does that help layer twos? Like, how do L2s then use 4844 to provide their users with lower transaction fees?

• 22:03Speaker 05

  Right, so we just ended with how dank sharding basically introduces data availability sampling. And these other new advanced tech features to try and distribute this job across the network better. But this comes with additional complexity, it will take more time to properly test and introduce it to Ethereum. So instead of waiting for the full version of dank sharding. We can reduce the feature sets and go with a amount of data in between these things. We can offer additional data. And we can look at layer twos, like what kind of security properties they need and optimize for that. And they can already make a big win there.

• 22:48Speaker 05

  And then later, with the additional features, we can get to full dank sharding. And so what we start with here is the changes to pay for the data as a layer two. This type of transaction that introduces the data to the network. And we need some changes to distribute the data across the network. But it won't be as much data just yet. So it's manageable by the own network to download. And so we don't need sampling yet. And we can have everyone download the data.

• 23:24Speaker 02

  Got it. And how, yeah, how does a layer two then use that data? Like from say optimism's point of view, how do you actually leverage that? Is it just changing where you post the data that's currently posted in in normal transactions as call data?

• 23:40Speaker 05

  Right.

• 23:41Speaker 02

  Is there more involved?

• 23:43Speaker 05

  You need to optimize what a layer two really needs. You can take apart all the things that the layer two uses. One of those things is publishing the data, making sure that this honest minority that protects the layer two is able to get the data in the first place. And then there's this other property that the layer two uses right now of getting the data long term. But these are very different. Data availability is this property that ensures actors are able to get the data. And this can be for a more limited amount of time. And so you do need to ensure that even with downtime and even with censorship and whatever other

• 24:26Speaker 05

  unforeseen circumstances, these actors on the network on layer two are able to get the data. But then after some amount of time, this should be sufficient to guarantee the security of the layer two. Because you want the actors to be able to reconstruct the states. Because only if a full state and you have the full history, they're able to challenge the operator, challenge the sequencer of the rod.

• 24:53Speaker 02

  Got it. And I think we've mentioned like a few times already, and I want to make sure we we kind of clarify for folks is this idea of data availability. And I think this is something that at least to me was not like clear until I spent way more time looking at the in the sharding is like what, when we say data availability, what exactly do we mean? And how is that different than like the data we store on like the Ethereum blockchain today? And I don't know, Vitalik, do you want to give a quick overview?

• 25:21Speaker 03

  Sure, you know, I think it's definitely a very important and a subtle topic. Like, I think even the, the really big point of comparison a lot of people have is like, what's the difference between what we're doing and IPFS, right? And this, like, IPFS is a platform where if you publish data, then that, like, presumably, you know, if the incentives are right enough, people care about the data, the data gets broadcasted, and then anyone who wants to download the data is able to download the data. The difference between that and what Ethereum is doing and going to be doing is that Ethereum is and will be providing consensus on data availability, basically.

• 26:04Speaker 03

  So you just, you can always have a hard consensus on the question of whether or not a piece of data with a particular hash or a particular commitment actually is available. And what we mean by available basically is, did, what, did the data go through this publication process where it got broadcasted on a public network? And anyone who wanted to actually download the data actually did have a lot of time during which they could have downloaded the data. So, basically, the, the, the difference between that and something like IPFS has to do with the case of a malicious publisher, right? Like, if I'm, I'm a malicious publisher,

• 26:48Speaker 03

  then, like, on IPFS, I could potentially do something where, you know, I yet control some small number of servers, and then I publish to a small number of servers, and then a small number of servers might respond to, and say data is available to some people, but they might not respond to other people. And so, some people get the data, other people don't get the data, but you never actually, like, get this kind of very binary consensus on whether or not the data was actually published. Now, the reason why this kind of concept of consensus on data availability is necessary has to do with A lot of these Layer 2 protocols, where those Layer 2 protocols depend on data being out there

• 27:32Speaker 03

  and not downloaded by everyone by default, but downloadable by anyone in the case that they want to download it for some security properties, right? So one very simple example is Ezekiel Rollup, right? In Ezekiel Rollup, you have a sequencer. That sequencer accepts transactions. That sequencer publishes these blocks that contain state deltas. They get to get a proof. And that sequencer also basically manages this kind of internal state, like what is the state of Ezekiel, like the balances, contracts, whatever, inside of that Ezekiel Rollup. Now, the difference between Ezekiel Rollup and a Validium, right? Is that Ezekiel Rollup has like state deltas or inputs on-chain. In a Validium, you only have the proofs on-chain

• 28:17Speaker 03

  and you have everything else off-chain. From a security point of view of like, can they force invalid stuff to go into the system? Both Rollups and Validiums both protect against that, right? Because the ZK Stark prevents you from actually bringing anything invalid. The place where they're different is what happens if the sequencer disappears? Right? What happens if the sequencer becomes malicious? And they just basically shut off from the network, never talk to anyone again? And the reason why they do this is basically that because they want to just make it not possible for someone else to interact with that system going forward. And so if people have money inside the system, that money gets stuck. Now, in a Validium, this is actually a problem.

• 29:02Speaker 03

  If the Validium operator does this, then they can't steal, but they can make people's money stuck. And so if they're really mean, they could potentially extort and they can basically say like, hey, you know, if the whales don't extend that 20% of their money to a ransom address, then they're just going to make everyone's money stuck forever. In Ezekiel Rollup, on the other hand, there's this guarantee that because either the inputs or the state deltas get published through the chain, if the original sequencer disappears, you can always have a new sequencer. Or come in, read the data from the chain and basically initialize the exact same state that the original secrets or has. And so be in the exact same position and have the exact same capability.

• 29:45Speaker 03

  To then be able to continue providing ZK roll up blocks, processing withdrawals and processing transactions, right? So because that data is on chain and so someone else can come in and reconstructive and like basically swat themselves into the same role. Uh, you don't have this same security problem that voting has have, right? So the difference between the two basically is data like in this like on chain or is it off chain? Now, why does it matter if it's on chain? Because on chain is a very simple, convenient medium where if you see the data is on chain. Even if you don't personally process it, even if you personally don't care about it, you still know that. If something terrible happens and you need it to recover, then you will be able to like actually go on chain and grab that data.

• 30:29Speaker 03

  Right? Um, so what proto ding sharding and like, and then eventually full ding sharding try to do. Is they basically try to like really zero in on providing a platform that provides exactly that capability, right? So the beacon chain would actually only contain hashes of data. And so if you're a client, then you would base, you'd be just downloading the beacon chain and you would get hashes of everything. But when I say hashes here, I mean like hashes of KZG commitments and the, you know, blah, blah, complicated math. But like, think of them as hashes, like actually, yeah, it is a, uh, KZG is a cryptographic valid hash function by, by, yeah, you know. Definitions of collision resistance and pretty much resistance and so forth. Uh, but, uh.

• 31:10Speaker 03

  Basically, yeah, it, the actual full data would instead live in this like sharded system where it would be inside of shards. That would be inside of pure, pure sub networks. Um, and the point of all of this machinery around data availability sampling and that sort of stuff is to basically. Provide a way of, uh, kind of checking and guaranteeing that the data actually has been published through this mechanism where. If in the future you need it, you will be able to get it without actually requiring everyone to like directly download all, uh, all of the data themselves. Right? Now, the chain does not have to store the data, the, or the shards do not have to store that data forever, right?

• 31:48Speaker 03

  So like the plan is for them to delete that data after some period of time, like it's, uh, you know, numbers have been thrown around of like somewhere like could be 30 days, could be a couple of months and then, uh. Basically, but but the point is to give enough time that any mechanism or like anyone that wants to be able to download the data will be able to download the data and like for there to be enough time that for all of the people that would be making backups for related to a particular application to actually have the time to do that, right? So basically, you can create the system that's like really optimized around this idea of like, how can we just provide this exact guarantee of like data availability, like proof that

• 32:29Speaker 03

  the underlying data behind a particular hash has actually been published to this public notice board where if people want it, they they can get it. So that rollups similar to is can take advantage of that for scalability without incurring the complexity costs of like actually trying to shard like full on EVM execution or whatever.

• 32:52Speaker 02

  Right, right. And so it's like the the guarantee of the Ethereum L one protocol is is quite tight. It's like we we guarantee that this data will have been published on the network for this amount of time. Beyond that, obviously, there's still ways to retrieve that data. They're just not guaranteed by the Ethereum protocol, right? And they it could very well be on IPFS, but that's not a guarantee that the protocol can make, you know.

• 33:17Speaker 03

  Right, exactly.

• 33:18Speaker 02

  Yeah. Um, and and and we've we we've touched kind of on this a couple of times already, like this idea of data availability, sampling and only verifying like having each fair each valid or verified parts of the data to make sure that like overall entirety has been published. Dankrad, do you want to give us like a an overview of how does data availability availability sampling work?

• 33:42Speaker 02

  For call it an intermediate audience, you know, not a photographer.

• 33:47Speaker 03

  Yeah. Yeah, yeah. So.

• 33:51Speaker 04

  So the idea between the data availability sampling is that you somehow want a scalable way to to ensure that some amount of data is available, like is basically and available means you could download it if you wanted to, right? And so the obvious, obviously, we know if we download it, which is what we do now, it's available. That's simple, right? Because if you could download it, then you could download it. Okay, but but how do we make it scalable? So scalable means we have the same amount, a constant amount, or maybe increasing a logarithmic amount, but not like a linear amount of resources that we need to do this

• 34:36Speaker 04

  amount of work. So we we need to find some way of doing this in a more efficient way. And the way data availability sampling does this is what what nodes do is that they sample, they they pick these random parts of the data and they say, I want this and this and this, and and I'll try to get it. And only if I can get all of these, or maybe a vast majority of them, then I will consider the data to be available. And naively, like if you just do this on blocks as they are now, then it doesn't work. And why? Because if there's, say, just one piece of the block missing, the probability that

• 35:19Speaker 04

  you catch it is tiny, because you would have to request exactly that piece of the block. And you want to only request a really small part of it, so the probability that you catch it is small. So that doesn't work, because we know, like in blockchains, basically, the thing with blockchains is like, the bad things could happen anywhere, like even one single missing transaction could screw up the whole system. So you cannot allow any part of the data to be missing. Like, just sampling directly doesn't work. So what you have to do instead is, you have to first encode the data, and you encode it in such a way, and this is called Reed-Solomon codes, you encode it in such a way that that

• 36:04Speaker 04

  any any fixed fraction, for example, you can pick 50%, if any 50% of the data are available, then then you can reconstruct the whole from that.

• 36:20Speaker 04

  So you encode it in that way. And now, um, now the scaling becomes different because now you don't have to ensure that all the data is available. You don't have to know that all the samples are available, but you have to know that 50% are available. And that's a task that you can do statistically, because if you download thirty samples or, well, I mean, the correct way is saying like this. If someone is trying to hide, this is the attack we're trying to defend against, right? Someone is trying to hide the data somehow. If they do that, they have to make less than 50% of the samples available. If they make less than 50% of the samples available and you download, for example, thirty,

• 37:01Speaker 04

  then the probability that, uh, that all of those will be available is two to the minus thirty, which is one in a billion, and so it's really small. And by downloading ten more, you decrease it by another factor of ten. So that, that is a scalable way of ensuring data availability, and that's the principle of HardWorks.

• 37:22Speaker 02

  Right, right. Um, and basically, building this entire, this entire system, uh, is why shipping bank charting is, is gonna take a while. Um, Proto, can you walk us through, like, in the meantime, in like the forty-eight, forty-four world? How do we like sidestep that? Why, why can we get away with not, uh, not having all this already live?

• 37:49Speaker 05

  So for background here, first off the merge, uh, we separate Ethereum into a consensus layer, an execution layer. And we are not throwing more data at the execution layer, but rather we continue to scale the consensus layer. And even then, we are only doing so with a limited amount of data. So we're talking about a month or maybe three months, some amount of data that is retained. After the period, we start to prune the data. So we can ensure that this is available for layer twos, for a sufficient amount of time for them to secure their network.

• 38:30Speaker 05

  But then at the same time, it doesn't grow infinitely, like it doesn't grow indefinitely. There, now you have a bounded amount of data to store on consensus nodes. We can distribute this between the different beacon nodes.

• 38:46Speaker 02

  Right, and I guess just to clarify this for the listeners. The amount of data that we make available in proto dank sharding is less than the amount that we make available in full dank sharding, correct?

• 39:03Speaker 05

  Right, so the full dank sharding, we distribute the job of storing and propagating the data between all the nodes on the network, between the different validators. Whereas with EIP 4844, we still require all of the consensus nodes to acquire all of the blob data. But we limit this. We don't make it grow indefinitely, so we can increase the throughput.

• 39:30Speaker 02

  Got it. And can, what if you give me an estimate, like, you know, how much do we lower the cost of storing data for day or twos with 4844? And then how much do we lower it further with a full sharding deployment, roughly?

• 39:46Speaker 05

  Sure. So, current Ethereum blocks are anywhere between like 50, maybe tops like 100 kilobytes. It's very favorable. Like worst case, it could grow a lot larger. But you're paying for call data, data that is going through the EVM and that's available forever. This is very different type of data that a rollup really needs. So instead, you can try and optimize, you can have this different type of data called blob compared to call data. And you can grow it from this order of magnitude from like 50 kilobytes to maybe like a megabyte per block.

• 40:26Speaker 05

  And this is, obviously, this is already a huge increase, that rollups could benefit from a reduced cost. And then the full dank sharding, it can go another order of magnitude larger. Because now we don't have to store all the data on one node, that we can distribute it across 64 nodes. We could have a multiplier here, like how we do the data apart.

• 40:49Speaker 02

  Got it. So it's like we get an order of magnitude increase with just 4H44 in terms of how much data we can have and then we get another order of magnitude with full big sharding. And then one thing that's been interesting to me to learn as I've been spending time on this is the idea that like the demand for storing data in these blobs or in the full sharding system is like independent or at least decordated from the demand to use Ethereum gas, right? Like there might be people who are willing to pay a lot to execute computation and people who are willing to pay a lot to store data, but they don't necessarily overlap. And it's like it creates two different markets. So can, what if you kind of walk us through like how we're like

• 41:35Speaker 02

  designing these two different markets and isolating them from each other to an extent?

• 41:42Speaker 05

  Well, so it starts with the transaction type, right? This additional fee parameter. But with this fee, you create a different market. And so if you really want to, you could separate the transaction pool. And the capacity and the type of resource is very different. And I think Vitalik already wrote a post about a multi-dimensional EIP 1559, where we can try and think of all the different resources in Ethereum as different markets. And I believe Dunkhut already has a post on how EIP 1559 could work for this type of blob data instead of regular gas.

• 42:24Speaker 02

  Right. So it's like there's two auctions happening in parallel. One is people bidding for like transaction computation and the other one is people bidding for storage. And we can use kind of the same mechanism which we have already for gas and call data, which is like weirdly bundled to then to then separate it and have one 1559 that works for gas and one 1559 that works for for shard data.

• 42:50Speaker 02

  Another thing, we haven't touched on this a lot, but Vitalik, I think you mentioned them earlier on, but the Sharn design requires introduction of KSG commitments. And they're kind of like a hash, but not really. I know, Dankrod, you had like a great post about them. Do you want to explain again and sort of to non-cryptographers what these are? And you know how they resemble the cryptography that's currently in Ethereum and differ from it?

• 43:18Speaker 04

  Yeah, yeah, I mean, so I mentioned earlier at the end of explaining sampling, how the data has to be encoded in this way that we call a Reed-Solomon code, which is a way to ensure that any 50% of the data can be used to reconstruct the whole data. So I mean, I guess data is a bit misleading here because there's like the original data, which is the actual payload that we're talking about. But then the code expands the data, so it becomes twice as large in the process as well.

• 43:55Speaker 04

  And so, and so what is a Reed-Solomon code? So Reed-Solomon code is, well, so yeah, we call it, it's a polynomial. So basically, what it means is you've learned about polynomials maybe in your mathematics classes. It's a certain type of function. And basically, this type of function has the property that, that when you know it at a certain number of points, which we call the degree of the polynomial, then you know the whole polynomial.

• 44:33Speaker 04

  So basically, we use that property in order to put this polynomial function through the data. And then if you have like a certain number of samples, which is like half the amount of the full encoding, then you can get all of them.

• 44:54Speaker 04

  And and the reason we need KZG commitments is this, like when you just sample the data, there's one thing that you can't decide from those samples, and that is whether the encoding is correct. What if someone just like Reed Solom calls have a certain structure, right? They have a certain structure that allows us to reconstruct the whole thing. But what if someone encoded it in a different way? What if they just put garbage in it? Then every 50% of the samples would give you different data. And of course, that's not acceptable because the data has to be unique. It has to be that all 50% of the samples give you exactly the same data.

• 45:38Speaker 04

  And and the way we we do that, I mean, there are different approaches. But basically, over the years, we have ended up here where we just found this amazing type of commitment, a KATI or KZG commitment that you can basically see as a hash. It's similar to a hash of data, but with a property that instead of hashing to just data, it hashes a polynomial. So it's a way to hash a polynomial function and and basically reveal any point on it. And so that guarantees that this this correctness of the encoding. And that's why we need these KZG commitments.

• 46:19Speaker 02

  Got it. And as I was reading about KZG commitments, one of the first things you stumble on as a non-cryptographer is they require a trusted setup.

• 46:31Speaker 02

  And, you know, I'd be curious for the viewers to walk through like what what actually, what are the trust assumptions in a trusted setup? You know, like, what are the things that like we are trusting in that setup? And as we as we make one to to kind of enable this on Ethereum, is there things that like end users can do so that they can have kind of a higher assurance that the setup was performed correctly and like minimize the trust assumptions they are making individually?

• 47:03Speaker 04

  Right, yes. Um, yeah, so basically the trusted setup is like, basically what we what we have to do is we have to uh generate uh these elliptic curve points that have a certain relation. That that's like one of the fundamental inputs um of the of the KCG commitment scheme. And uh the trusted setup is basically a way a- and okay, and in addition to that property that they have a certain relation is nobody is allowed to know the actual relation between them. So this has to be a secret, and that's why it has to be this trusted setup. And it's called trusted setup because like one of the ways to do a trusted setup is just to say, hey, we all trust Tim, Tim, you do it.

• 47:47Speaker 04

  And you give us the outputs and then it's done and you throw it away and everything will be fine. Uh but the problem is, of course, that's not really sufficient for uh the Ethereum community. People would be like, well, what what if? Um so instead, we have this way of distributing this trust and saying like um we let many, many people participate in this trusted setup. And uh and we can design trusted setups um in a way so that if even a single one of these uh uh people that participated in it um did it according to the protocol and the protocol means that you you execute this whole thing, you run this program, send your output,

• 48:29Speaker 04

  and then you destroy your data, like you destroy the secret that you've used to do it. You don't you don't keep it. And if even a single person out of the potentially thousands that are going to participate uh did this properly, then the setup is completely safe. So even like, let's say, one thousand people do this, nine hundred ninety nine colluded and they all kept their secret and they come together and try to reconstruct it. But one person did it properly and they don't have it. Even in this case. These nine hundred ninety nine people know absolutely nothing that helps them to break it. So that's the security guarantee which we call like N minus one. So even if N minus one collude, they can't get anything on that last person that participated properly.

• 49:14Speaker 04

  And yes, I mean, obviously, this property has a nice property that if you are really, really worried about this and are like, oh, my God, how can I trust these people? Then you can just participate. So like one obvious way is if you participated and you know you did things properly, then you don't need to trust anyone because you're part of it. And yeah, it's done. So like you can consider it secure. Obviously, there are also all the parts of like making sure that all the software works properly and so on. That we're all very familiar with in our blockchain systems, like all of this we're relying for the function of the team as well. So obviously, it has all to be very well audited.

• 49:56Speaker 04

  And we need several implementations of this as well. But yeah, so that's, I guess, like the other trust assumption that we have to make sure that all the software is safe and works properly.

• 50:07Speaker 02

  Right, so there's like kind of different levels. It's like either, you know, you don't care at all and you just trust that somebody somewhere has been honest. And maybe you just, it's not that you don't care, but you learn about Ethereum ten years from now, which is basically what you have to do because you can't participate. If you're, if you're part of the Ethereum community today, there's going to be an opportunity for people to participate individually. So then as long as you're confident that like your participation was, was correct, then, then the, the whole output should be correct. And then if you're even kind of more concerned, there's a specification from which we can write different implementations. So I assume, you know, if you didn't trust the existing ones, you could write your own and also produce an output.

• 50:50Speaker 02

  Or at the very least, kind of review the code of the different ones and make sure that they, they, they match up. And yeah, you, you would kind of get a high level of certainty that things are correct.

• 51:04Speaker 02

  I'm okay. And I guess that we're coming up on time here. The last thing I did want to talk about is like, how do we actually get this deployed? Which is the part I'm usually get involved in. I know Proto, you started prototyping 4844 along with some other folks. Can you give just kind of a quick summary of like,

• 51:28Speaker 02

  what was done so far in terms of prototyping? And what do you think are like the next steps that the community can expect?

• 51:36Speaker 05

  Right. So earlier this year, it started with an initial write-up of what it could be like. Then during the hackathon in Denver, it transformed into this software where we have an actual implementation of the proposal. And over time, we have been improving that and testing that. And what we need to go and do from here is there are two, these two different branches, right? Where we want to further develop the client software to be able to make a test network. And we want to continue the development on this trusted setup. So that we do have the cryptography, everything there on that side. All set as well for when we do want to deploy this.

• 52:18Speaker 05

  And then once we have both ready, we can make larger and larger test nets. And then eventually include it as an EIP via the Alcor Devs process into Ethereum mainnet.

• 52:34Speaker 02

  Got it. Got it. So we have some initial prototypes. We want to product, like grow them, make them more robust, make sure the trusted setup is working according to plan. And then once we have that, it kind of becomes a normal EIP we need to shepherd through the process. And then one thing that's worth noting about 4844 is what's very neat about it is from the execution layer point of view. So like the kind of smart contract and end user transaction generation point of view. Sharding is basically done then, like there's no more changes that we'll do for people to interact with this blob data. What what will happen is then we need to deploy kind of this entire dank sharding infrastructure on the consensus layer.

• 53:20Speaker 02

  But from applications point of view, that that kind of just happens in the background. But at the consensus layer dank rad, like what are what are the steps to get this deployed? Like, how many how many hard forks do we need to get there? You talked a lot about like proposal builder separation before. So like, what do you see is like the logical set of stepping stones to get us to the full full sharding on on a consensus layer?

• 53:43Speaker 04

  Yeah, I mean, I I hope it's it's two, but I don't know yet. I think like so clearly like, I mean, this is the reason why we chose this stepping stone of proto dank sharding, that it's a it's something that gets us substantially closer to the full implementation. So it will become simpler. It will like the interface, for example, will stay the same. The execution layer changes will be minimal once that's implemented. So that's why it's really nice if we can get that done in the Shanghai hard fork.

• 54:18Speaker 04

  And then, I mean, I think it's very unlike that will be the next hard fork after that. But hopefully relatively soon, we will get, yeah, we'll get this. I, yeah, I don't know. I it's currently still, there there are still like some things we definitely need to work on. Like there's there's a lot of work on the network working to be done to have, yeah, full sharding rolled out.

• 54:47Speaker 04

  I I currently have no exact estimate, but I would hope that that can be done in one hard fork.

• 54:54Speaker 02

  OK, I sympathize with not being able to give direct estimates about complex projects. So I won't push you on there. And and maybe, yeah, to to to kind of close this off, like Vitalik, like if people want to contribute from like the from a researcher, like engineering point of view, like what are like the big open questions in sharding land that they should spend their brain cycles on?

• 55:20Speaker 03

  I think one problem is definitely like figuring out the networking of data availability sampling. Like, there are designs that we have that work in theory, right? Like, there's doing it based on subnets. There is the approach of trying to make a DHT much faster. There's a couple of other techniques in the middle, but like, really taking those ideas and from an engineering perspective, like just trying to optimize it really hard. Like, how do you actually make a, basically, it's like a specialized, scalable DHT that we are publishing and downloading

• 56:01Speaker 03

  can happen extremely quickly. So, that's one problem. And I think in general, like in the Ethereum ecosystem, the networking side is one of the sides that's talked about the least. I mean, possibly just, I think, a bit of an accident, if you ask me, that the Ethereum core research community just happens to, like, never really have people who, like, spent a lot of time thinking about networking stuff. It's generally like they always spend a lot of time thinking about, I mean, cryptography and consensus economics. But it's still a really important problem. And it's a problem where I think it would be amazing if we can have more, like, very active networking expertise in Ethereum. So that's a short term, or that's

• 56:45Speaker 03

  like a very clear problem. Another problem is that with folding sharding, there is this issue of like how to combine it well with proposal builder separation. And there, there are some like economics challenges. There's still the challenge of like, well, how do you actually make a good proposal builder separation protocol? How do you add like censorship resistance lists so you can bypass censoring builders? And once again, there, we have ideas on each of those things, but there's still the question of digging into the details, combining a PBS design with the design of Ethereum's future proof of stake, which is something that at some point will probably start to

• 57:29Speaker 03

  Having to kind of talk and think more about, right, like there's been this increasing effort within the Ethereum research and the protocol community of basically thinking through what would a better proof of stake design look like in the long term? Like, you know, do we want to have single slot finality? How do we achieve single slot finality? What other benefits can we achieve? How can we offer more in protocol of what Lido offers to people, extra protocol, to try to reduce, I mean, stake and pool centralization incentives and just generally increasing simplicity. Right? So I've written a bunch on that. Like, if you just, you know,

• 58:08Speaker 03

  Google for single slot finality, you can probably find it. But the do that and the intersection of that and proposed builder separation, the intersection of that and sharding is going to be another research area going forward. And then the other one is also adjacent to Dake sharding stuff, but and also adjacent to 4444 stuff, which is very critical to proto-dake sharding and Dake sharding actually being viable and just generally Ethereum scaling well is like creating the like as decentralized as possible and as robust as possible systems

• 58:52Speaker 03

  to give people the same guarantees that they come to expect out of Ethereum in terms of history retention, but without requiring participants in the core Ethereum consensus protocol to like all be retaining blocks forever. Right? So there's a team working on Portal. There's things like the graph, like there's this big, long list of projects, right? And trying to figure out, you know, how to make those better or even create better alternatives to those, I think, also an important area.

• 59:22Speaker 03

  Oh, also, one last one I really wanted to throw in, this is important. The switch to Layer 2 is, I think, one where it's very important for the ecosystem to try to maintain and even improve its decentralization going through that switch. And so, you know, we need light clients that can poke into optimism and poke into arbitrage and poke into Starknet. And that's something that there's been a bit of like theoretical thinky work on, but it's the sort of thing that I think there is a lot of room for people to slot themselves in and like really try to improve that ecosystem a lot, right? Like basically try to like really think through if everyone's really going to migrate

• 1:00:03Speaker 03

  over to L2 over the next 12 to 24 months, especially as blockchain sharding comes alive and roll up costs go down even more. Like, how do we really make sure that transition goes well and that it preserves all of the decentralization properties and even improves on those properties that we've come to expect of Ethereum?

• 1:00:24Speaker 02

  Yeah, that's a really good list.

• 1:00:27Speaker 02

  We have like five minutes left, so I wanted to leave a bit of time for the three of you. If there's anything you wanted to share that you think is important about Chardonnay or 4844 or Ethereum generally that like we haven't talked about, the floor is yours.

• 1:00:43Speaker 02

  Whatever you want to rant about or get people to pay attention to.

• 1:00:52Speaker 05

  I mean, the last week has been kind of chaotic, to say the least. I think this is like the time of market where if you feel a little bit down, like maybe read a post, try and get involved with Nissan projects. It's not the bear market, it's the builder market. Read the specs for the EIP. There is the site called eip4844.com, so I hope that you start it. And then there are simple diagrams and all the way down to like very elaborate posts about crypto, cryptography involved. And yeah, just reach out and get building. Stay optimistic, by the way.

• 1:01:38Speaker 02

  I love it. Thank you, Rad, Vitalik. Anything else you want to share?

• 1:01:44Speaker 03

  Stay optimistic, but in the long term, hopefully, I'll stay zero knowledge.

• 1:01:50Speaker 02

  Wow.

• 1:01:54Speaker 02

  Anything I do at Bankrad?

• 1:01:57Speaker 04

  Yeah, I mean, like, I mean, I think, I think the week has shown that

• 1:02:03Speaker 04

  that you need solid designs and that we need to like build things that can actually last and that are, yeah, built to last and won't go away. And that's what we're trying to do here. And yeah, so I'm optimistic on this, but yeah, I am clearly also pessimistic about many other things that are happening in the ecosystem. So we just have to be better and build better.

• 1:02:30Speaker 02

  Love it. Um, yeah, then that's basically a wrap. The Bankless guys did ask me to end with a disclaimer. So here it goes. And I'm reading from the screen now, risk and disclaimers. Crypto is risky. You could lose what you put in, but we're headed west. This is the frontier. It's not for everyone, but we're glad you're with us on the Bankless journey. Thanks a lot. And yeah, thanks to Tan, Proto, Vitalik, Bankrad for coming at a bunch of different hours across your respective time zones. I think this has been really helpful to explain the entire Shari robot to people.

• 1:03:11David Hoffman?

  Hey, we hope you enjoyed the video. If you did, head over to Bankless HQ right now to develop your crypto investing skills and learn how to free yourself from banks and gain your financial independence. We recommend joining our daily newsletter, podcasts, and community as a Bankless premium subscriber to get the most out of your Bankless experience. You'll get access to our market analysis, our alpha leaks, and exclusive content, and even the Bankless token for airdrops, raffles, and unlocks. If you're interested in crypto, the Bankless community is where you want to be. Click the link in the description to become a Bankless premium subscriber today. Also, don't forget to subscribe to the channel for in-depth interviews with industry leaders, Ask Me Anythings, and weekly rollups where we summarize the week in crypto

• 1:03:56David Hoffman?

  and other fantastic content. Thanks everyone for watching and being on the journey as we build out the Bankless nation.