Pod Insights a16z Podcast Preserving Digital History: How to Close the Web's 'Memory Hole' About this episode More than 98% of the information on the web is lost within 20 years, and huge gaps exist in our digital and cultural history. Zoran Basich and Alex Pruden of a16z talk to Brewster Kahle and Sam Williams, who are using different approaches to attack this problem. Brewster cofounded the Internet Archive, which is well known for creating the Wayback Machine that crawls a billion URLs every day. Sam cofounded Arweave, a company that uses decentralized crypto networks to store information forever. For both of them, this issue has implications that go far beyond just data storage. It touches on issues of censorship, government manipulation of information, and how historical context is necessary for well-functioning societies. They discuss how decentralized models offer the promise of building a next-generation web that works better for users. Stay Updated:
Find a16z on YouTube: YouTube
Find a16z on X
Find a16z on LinkedIn
Listen to the a16z Show on Spotify
Listen to the a16z Show on Apple Podcasts
Follow our host: https://twitter.com/eriktorenberg
Please note that the content here is for informational purposes only;
should NOT be taken as legal, business, tax, or investment advice or be
used to evaluate any investment or security; and is not directed at any
investors or potential investors in any a16z fund. a16z and its
affiliates may maintain investments in the companies discussed. For more
details please see a16z.com/disclosures.
Hosted by Simplecast, an AdsWizz company. See https://pcm.adswizz.com
for information about our collection and use of personal data for
advertising.
Transcript 0:00 Speaker 00 Hi, welcome to the A16Z Podcast. I'm Zoran. Conventional wisdom has it that most everything important is accessible online and that it will exist forever. But in fact, more than 98% of the information on the web is lost within 20 years, and huge gaps exist in our cultural history. This includes published works like news reports, books, music, and video. A16Z Investment Partner Alex Pruden and I interviewed Brewster Kahle and Sam Williams, who are using different approaches to attack this problem. Brewster co-founded the Internet Archive. It's well-known for creating the Wayback Machine, which crawls a billion URLs every day. Sam co-founded Arweave, a company that uses decentralized crypto networks to store information forever. For both of them, this issue has implications that go
0:44 Speaker 00 far beyond just data storage. It touches on issues of censorship, government manipulation of information, and how historical context is necessary for well-functioning societies. We talk about the challenges of trying to preserve digital history in an era of deep fakes and fake news, how decentralized models like Arweave can fit in, and what types of economic models will help sustain data preservation. We also discuss base layer protocols versus archive layers, and evolving standards for the cryptographic hash functions crucial for tagging in metadata. But we begin with Brewster and how he started his quest to preserve digital history.
1:19 Speaker 01 The idea was to build the Library of Alexandria for the digital age. Could we go and make all the published works of mankind available to everybody and in computer-readable form so we could build a global brain? Kind of building on the ideas of Vannevar Bush and Ted Nelson, which even by 1980, when I started on this project, it was kind of assumed that it was going to be done or done already. So I thought, okay, let's just go and do that.
1:44 Speaker 00 A lot of people would say, "We already have the internet. All the knowledge that we need is already there and accessible." Why is that true or not true?
1:51 Speaker 01 Oh, it's actually fairly thin. I mean, there's a Wikipedia page on all sorts of things, and there's certainly all sorts of commentary about current events. So the good, bad, and the ugly out there. But a lot of the actual published works aren't there: books, music, video. It's astonishing how little of it, or the 20th century is largely not there. Or if you take webpages, they only last 100 days on average before they're changed. Or deleted. So we're building our culture on shifting sand. So I think it was a cruel joke to call webpages pages, as if they were like a Gutenberg Bible that would last hundreds of years if you didn't pay attention to it. It's just not true. This is all shifting.
2:33 Speaker 01 So we needed to build a library now that people are publishing with this new medium.
2:38 Speaker 00 And so it's kind of notable, here we are in 2020, and we seem to be living in this age of fake news, I guess, where the truth itself can seem slippery. People can't even agree on what sources of info are reliable. What does that mean for the Internet Archive? Is this kind of a unique period in history as you've been doing this for the last four decades? And what are the challenges that that presents for someone like you trying to preserve knowledge?
2:59 Speaker 01 I think it's showing the need for more and more context around what people are looking at, being able to reference real materials. Really, the web grew up in parallel and separate from, say, the book world. The book world just staunchly, you can go buy a book from Amazon, or you can download it on your Kindle, but the web and books never grew up together. So we're trying to make all the footnotes, the references in Wikipedia, live links. So if it's a book and it's got a page number, it opens right to the right page. Or just take Wikipedia's footnotes that go to other webpages. We started, based on talking with Catherine Maher, the exec director of Wikipedia, she was worried that
3:40 Speaker 01 truth might fracture based on basically bad citation availability. So we went and made a robot to fix broken links in Wikipedia. We fixed 11 million broken links. So we've, the idea is to try to make this good idea of the World Wide Web into something you could actually live on and depend on. And in this era of fake news, it's really grown up throughout the web. It shows how fragile information is if you don't have an ecosystem of references, context, reliability, organizations you can believe in. So we're trying to play a role in hooking these things together into this wondrous thing, which is the World Wide Web.
4:24 Speaker 02 One of the reasons I'm personally so excited about what you and Sam are doing is that I firmly believe that human progress is largely dependent on our understanding of our own human history. And I think this notion of how we study history and how we understand events that happened in the past, this is called historiography, I think is absolutely critical and often underappreciated part of that. I think it gets to kind of this, maybe this is the same as when you were referring to it as a global brain, but I guess can you in your mind describe maybe the benefits, maybe tangible benefits that we have by having this better, more granular understanding of events exactly as they happened on the web in the past?
5:03 Speaker 01 Well, let's just take, there's going in understanding the worldwide web's history so that politicians can't just erase their past, or I don't know, when you have pundits that want to go and now be in government, they just want to erase their past. And if you just make it so that they can do that, then people just feel much more free to just make things up. If there's no accountability for what it is, you said, you've done, you've published, you can just erase it, then that's a really scary time. And it gets into the whole George Orwell, try and rewrite history thing, which is we're seeing more and as the governments around the world are swinging right in a very hardcore way, and we're losing newspapers, we're getting more and more control and balkanization of the internet.
5:48 Speaker 01 But let's just take the 20th century. The 20th century documents from the 20th century are largely not online. Copyright in the United States ends in 1924, so if it's published before 1924, it can be public domain. And so our book project, we digitize and make them just publicly downloadable. Google's less so, they kind of still bound up the public domain, but at least a lot of materials pre-1924 are available online. But after that, if you look at the graph of what's digitized and put on the Internet Archive Book Collection, it goes up to 1924 in the craters, and it's basically almost nothing through to the 20th century. And it's not because those are all in print, because if you try to go to Amazon and buy them new, they're not on Amazon either. So the
6:33 Speaker 01 20th century, we're bringing up a generation without access to the information and what happened during the 20th century. It's a very impactful, very important century. And if we don't learn from it, if we can't even reference it, then people can go and say, oh, this didn't happen. Or it really happened this way, or actually, people were really cheerful about going into the Japanese internment camps. It's like, it's just not true at all. But if you don't have referenceable materials, either through the web era or back through the 20th century, you can just make things up, and we're suffering from that right now.
7:10 Speaker 00 Sam, you're working on, or you're driven by some of these same kind of ideas, I think, and you're doing it in a slightly different way than what Brewster has done. Talk a little bit about the decentralized web and, as you call it, the perma web with Arweave.
7:21 Speaker 03 Yeah, I mean, so we definitely started with some of the same ideals. Like, the real question we were trying to answer when we started Arweave was, how is it that we can close the memory hole? As George Orwell puts in 1984, the idea that a politician can instruct someone to take a record of history and then delete it such that nobody could ever prove what happened in the past. Yeah, and so we were thinking about how, well, we were thinking about distributed networks at the time, and then we were thinking about how they could be used in practice. And one of the things they do exceptionally well is replicate data around the world with no single central point of failure. And that's great, but of course, in a typical blockchain system, you could only store a very small amount of data. So with Arweave, we essentially, yeah, built it to address that question.
8:04 Speaker 03 How do we make it so that we can store very large amounts of data trustlessly in exceptionally large numbers of places across the world so it can be deleted by, well, all kinds of people, in fact? So that's what we were going for. And then when we built this, which was mainly driven by this question of how are we going to make verifiable copies of news archives? We realized that the technology underneath was essentially just a permanent hard drive, could really be used for storing all things that should be permanently archived. And then subsequently, yeah, from there, we realized, well, if you make that data storage available inside a web browser... What you have is a version of the web, except all of the data inside it is permanent. And this goes to address some of the problems that Bruce was talking about with the fact that,
8:49 Speaker 03 you know, web links last on average 100 days. And I think what's really striking, 98.4% of the content on the web is lost within 20 years. And 98.4% is essentially the entire thing reset, which is really incredible, given that we're embedding it at the core of our societies as the fundamental way of transmitting information between people. So we see with the permweb that you essentially have a system by which to address this. You make an endowment structure, which pays for the storage of information forever in a network where there is no single centralized point of failure. And you essentially make an open business model that anyone can offer their hard drive to and get paid for offering storage, such that there is no centralized authority
9:33 Speaker 03 that needs to manage that network either. So that's what we're working on with the permweb.
9:37 Speaker 02 I think the question potentially comes up sometimes when people discuss the Internet Archive is the sustainability question. In other words, if the Internet Archive goes away tomorrow, who is going to continue to do this important work? How can we make sure that this important work is continuing to be done? And so, yeah, I would love to hear your thoughts on kind of the approach that you're taking and then this kind of sort of different approach that Sam is taking.
10:00 Speaker 01 Oh, yeah, no, I think there should be lots of approaches. There should be lots of libraries. So the Internet Archive does play a major reference role by building the Wayback Machine. We crawl about a billion URLs every day now, which is kind of amazing to me, a billion a day. It's a total of maybe 800 billion in the Wayback Machine collection at this point. And it's getting large. So we try to collect a copy of every webpage every two months. There's one robot sort of mandate. Another is collecting the top webpages. That's just the sort of homepages plus one. We have active news collecting going on, but we also now have
10:42 Speaker 01 600 organizations, about 1,000 librarians that are directing particular crawls to go and make sure that we've got good collections in their particular subject expertise. And lastly, there's a service called Save Page Now. So if one goes to web.archive.org and puts in a URL, it will archive it and hand you back an archival URL right then. And that's being done now at 80 per second. So we're starting to get woven into the web. The Brave browser, if you hit a 404, it will ask the Wayback Machine, is there a copy of that? And then offer that to you. So it's kind of being woven into making the web a
11:23 Speaker 01 more reliable piece of infrastructure. The Internet Archive stores it in a couple places spinning, but also has partial copies in Canada, in Amsterdam, and in Alexandria, Egypt. So the idea is to have multiple copies in multiple places. And that's not to mention all of the libraries that we're working with that go and get copies of their national collections from the Internet Archive when they contract with us. So for instance, when the Library of Congress contracts with the Internet Archive to collect it, they bring a copy home with them. So I like the concept of lots of copies keeps stuff safe. And what Arweave is trying to do is use more decentralized technologies, which I think is a great idea.
12:07 Speaker 01 The decentralized web has been an initiative of the Internet Archive for several years. There's now been many gatherings. There's a deweb.archive.org, which is a prototype trying to make the Internet Archive but decentralized. And that leverages WebTorrent, IPFS, Gun to go and build a decentralized Internet Archive. Again, still really early days, very exciting project. And Arweave is part of that whole meur of decentralized web technologies.
12:39 Speaker 02 Sam, we'd love to hear kind of your thoughts on, and maybe a specific example of how Arweave solves this problem that we've described. And maybe, I think Arweave is different in architecture certainly, but same goal.
12:51 Speaker 03 Yeah, I mean, I wouldn't really say we're trying to solve the same goal. I see Arweave as like a fundamental base layer protocol, which is offering a service to anyone of any kind that wants to purchase it, which is simply permanent information storage. Whereas- Perhaps Bruce, you could correct me if I'm wrong, but really the Internet Archive is like an archive layer. It's the kind of thing built on top of the technology. So it's deliberately built to be a library. And that's one of the things you can do with Arweave. But Arweave is really just focused on basically building this hard drive that never forgets. But in terms of how you can use this in practice in environments where censorship is rampant, say, yeah, there's a really good instance from just the last month, actually. So we
13:34 Speaker 03 noticed that after the death of Dr. Lee Wenliang, there was a substantial outpouring of early an outcry for free speech inside China on Weibo, which is basically unheard of. This is kind of like breakthrough event that requires a very large number of people to work in unison, which is very difficult to organise under a kind of authoritarian regime, because everybody is scared, right? But once enough people get together and start speaking all at once, the critical mass is such that everyone else can also start to speak. And so for about three to five days, I think it was, there was a pretty large amount of dissent in China over free speech issues. So that was very interesting, we thought. And we figured that we could very quickly just build a small scraper that could go around
14:19 Speaker 03 Weibo and collect up these artefacts, commit them to the essentially collective permanent memory that we're building, distribute them around the world, and also make them available inside China in a censorship-resistant way. And so we got this running very quickly. And then a member of our community thought, oh, wow, that's really interesting. So he built a essentially UI on top of this, that not only showed all of the posts on Weibo that had been stored, but also went and detected when they had been censored inside China, and then highlighted them inside this UI. So now you can go to this webpage, I think they call it Weibo Uncensored, and you can press this button that says Show Only Censored, and this works inside mainland China, and it'll show you all of the things in Weibo
15:02 Speaker 03 that the bots have caught that the Chinese government are attempting to censor. So it actually has this sort of Streisand effect, where it further highlights the information that they were attempting to censor because they were censoring it. And of course, there are hundreds and hundreds of RWEV nodes. So it's extraordinarily difficult for the Chinese government to come along and try and actually block access to that. And those nodes are shifting and changing every single day.
15:25 Speaker 00 I'm curious about this question of centralization. And Brewster, when you think about the Internet Archive and centralization versus decentralization, are there challenges around having single points of failure or single points of contact versus having it dispersed the way it is in an RWEV kind of model?
15:40 Speaker 01 Oh, I think you want both. I think there's reason for having sort of the dot com sector, the dot org sector, dot edu, and working together well. I think they offer different perspectives and points of view. I think the top-level domain names of the Internet are kind of an interesting decomposition of the different types of organizational structures and what they're for. And how, if we can get them all to cooperate well, then that can work very well. So the Internet Archive, by having partners all over the world that are in these established institutions, that is a different form of decentralization. So I think we're in early days in terms of the actually how to go and pass around information
16:24 Speaker 01 peer-to-peer, but terrific progress is being made with, basically with hashes of files. You don't even need to care where you're getting it from because you know if you get a file that matches that hash, that you have the data you're looking for. So there's, with this cryptography that is now commonly available, which wasn't really to Tim Berners-Lee when he was building the web, JavaScript makes it so you can do your browser, it can be a first-class node on the Internet, you don't have to depend on servers. That's a tremendous step forward. And we now have digital cash that allows people to be able to pay for things, which
17:04 Speaker 01 has been a big missing piece on the World Wide Web that has really driven us towards advertising models, which have real problems. So I'm very excited about some of the technologies being brought to bear on some of these cultural areas. So yes, I think there's reasons to have these different systems work together, have the more traditional server-type structures be actively working on decentralization, and having the decentralized organizations, the ones that are sort of born decentralized, start to go and bridge to the more server-based systems.
17:42 Speaker 03 Add on to that, I think particularly of interest, so hashing is fantastic in terms of just being able to identify a piece of data at a later point. But one of the things that really got us excited when we were starting Arweave was how you could use timestamping of data, verifiable timestamping of data using blockchains or blockchain-like ledgers that involve a proof-of-work or proof-of-state mechanism to ensure that a piece of data existed at a certain point in time and hasn't been altered since that time. And so when we started to work with the Internet Archive data, it was natural that we looked at these torrents that you guys produced. And so subsequently, we've just been archiving very large volumes of these torrents over time. And now you can go back and you can say, okay, so this is a piece of data from the Internet
18:26 Speaker 03 Archive collection that is verifiably unchanged since it was added to, since the torrent was added to the archive, you know, like 18 months ago, whenever it was. And so I think that's a really compelling and exciting way that we can use these technologies.
18:39 Speaker 02 Yeah, no, that's great. And one question I have for both of you, and I think so far, I think we've kind of discussed at a high level or a philosophical level, why this is an important problem for people to be thinking about. But, and Brewster, I know you also, you sit on the, I think, I believe you're on the board of the EFF, and so I'm sure you're used to having this conversation, is you know, you're probably trying to convey to folks, hey, why should you care? So I'm curious for both of you, what is, in your mind, the best application or the best use of the technologies that you both are building? Like, how might the average person relate to this in a way that they can understand or apply in their own lives?
19:15 Speaker 01 Oh, gosh. Well, I think of it as a library, and so there are some of us that are old enough to remember what libraries did. So it's, you know, physical libraries. But I think the thing that most people use the Internet Archive for is the Wayback Machine, the free service on archive.org to be able to see past webpages and be able to get to things. So there was a pronouncement by the son-in-law of the president that the National Stockpile of medical equipment wasn't for the states. And within two hours, the homepage of the National Stockpile website
19:52 Speaker 01 Changed its mission and took out states and Indian territories, other territories and Indian reservations. And so somebody found this and they used the Wayback Machine's diff feature to go and surface the new page and how the diff, you know, sort of a red line version of how did it change from a couple hours before that public pronouncement of the change of what the federal stockpile was for. That's, I think, an example of why you'd want an archive of the World Wide Web. And there's similar examples, because we're also archiving television, of going and saying, "No, no, no, he really did say that, actually repeatedly." To be able to think critically, you know, I have kids in school, so to think critically,
20:37 Speaker 01 you need to be able to quote, compare and contrast. You need to be able to go and put your hands on something and be able to quote it, and then you can compare and contrast it with other things. If you can't quote what has been said or what was pronounced, then you can't really think critically, it just flows over. And if it flows over, then there's no accountability and people feel well free to make things up. And what's going on is people are making things up.
21:07 Speaker 00 And I think people have this impression that everything is on YouTube now. It's all out there, don't worry about it, everything's preserved. But that's not really true, there's these massive gaps, isn't there?
21:14 Speaker 01 Oh, yeah. And YouTube also takes things down all the time. Before YouTube, at Google, there was Google Videos, and that had 6 million videos, and they just decided to turn that off. So I had archived those, so those are on the Wayback Machine. There was Yahoo Videos, those are all gone. Apple had a self-hosted site for their people that own their equipment to go and make webpages and the like, and that was called MobileMe, and they just turned that off. So there's whole swaths of the Internet that just go away. And if people just get used to things coming and going and not being reliable, then we'll have people just start to not know what to depend on and what to believe, so this is not a good state.
21:57 Speaker 03 The Wayback Machine is a truly incredible thing, but actually so few people understand about how much else is stored inside the Internet Archive. Like, it's really incredible, the amount of interesting data that is stored there. But most people, I think, just see, you know, the Wayback Machine, essentially. So it's definitely worth exploring. But in terms of R-Weave, I think the the obvious first use cases are two, really. If there's something that you would like to say that you would like to make sure will be around for a very long period of time, like a blog post, a thought you had, something like that, you can just upload it to one of the blogging platforms on the network and it'll just perpetuate it essentially forever, backed by a sustainable endowment economic structure. So that's very valuable. And on the other side, you can also just use it as an accountability
22:41 Speaker 03 ledger for anything. So if you make a contract with someone, for example, or you receive an email and you make a deal over this email, then you can just forward it to a certain address, which someone in the community has set up, and it'll timestamp it in the R-Weave and you'll be able to come back to it later and prove verifiably that you haven't tampered with that data in the intervening period. And this is very valuable when we think about things like people sharing screenshots of stuff, which people do all the time. Those screenshots have no verifiability. And quite often, we see that they've actually been faked. So with R-Weave, you can totally circumvent that problem. That would just be a surface scratch, I guess, but definitely a first use case.
23:17 Speaker 02 One of the, when I excitedly describe R-Weave to folks, as I often do, one of the applications that I often cite is one that a community member has made called FeedWeave. And I think that one I find resonates with folks, just the idea that you can take, you can essentially replicate a social network graph on R-Weave, and the implications of that mean that it's easier to switch between or view that graph in different ways.
23:41 Speaker 03 Yeah, definitely. I think one of the deeper implications of this system is that it's essentially a massive open database that any application can read to and any application can write to, sorry, read from and write to. And that means that you can build applications that are purely composable in nature. So yeah, someone made FeedWeave.co, which is a simple blogging platform, that of course you can then build on top of this and you can say, okay, well, I would like to make a version of FeedWeave that is only for people that are mentioning COVID-19. So somebody has built a COVID-19 quarantine journaling app on top of the R-Weave network, just actually today, I think it was released. And of course, if you lay out the tags correctly, you can get intercrossing views of those data across the different applications.
24:25 Speaker 03 This also highlights another feature, which is if you have these permanent webpages, the applications stored inside those webpages can, well, essentially persist forever. And that has really interesting properties for the user because it essentially means that if I come along to, say, I don't know, Weave Mail, which is essentially an emailing service inside this network, I will always be able to get back to Weave Mail as it exists today in the future. There is simply no way that the author can come along and change it and subsequently perhaps add adverts or make it sell people's data or something like this. So if you've audited and you understand what the application is doing today, you can have very, very strong guarantees that it's going to do that tomorrow or really any time in the future.
25:08 Speaker 03 And when you compare this to typical web two point O services, the contrast is pretty stark. And everyone remembers when they first used Facebook or they first used Gmail. And they think about the way they perceived those services at the time. And now, in hindsight, when we can look back and see the essentially abuses of privacy that have happened on those services, we wonder, well, would I have signed up for it if I'd known that is what they would have eventually done? If that is what the service would have eventually become? And the answer is, in some cases, quite likely no. However, with a perma web app, you just don't get that problem. So weave mail dot app works like it does today. And it will work that way tomorrow and in ten years and in twenty years.
25:48 Speaker 02 I'm really glad that you brought the fact that most web applications are ultimately just wrappers, UI wrappers around a database. And I think that the point that you're making is that when it's a company that owns the database, it's not, you know, it's a proprietary database, it's difficult as a user to take that, your part of the database, the data that's relevant to you and move it to another application. Right? But in this paradigm that you're describing, that's not the case, right? It's the friction, it's essentially frictionless. And therefore much easier for people to, as you say, if they decide that at some point their privacy is not being respected or they just don't like the application, it's easy to switch. They don't have to make the trade off that like, well, I have to leave all my data that's relevant here and start new somewhere else.
26:28 Speaker 03 Right, exactly. Well, there's actually kind of two sides of this, right? So it's the fact that the database itself is open and available for anyone to build on. So it's as if every single application has an API by default and by necessity. So it has to have an API in order to function. And that part is really valuable. And it's kind of interesting in the RVF community, we have two camps about this. I was one of the great things you get with like decentralized communities like this. Some people, and I'm definitely on this side, I would say, think that the fact that the UI itself that accesses this data is permanent and think that that's important. And other people are just more focused on the database side. But certainly, yeah, both are very interesting.
27:06 Speaker 00 Brewster, I wanted to ask you actually about community because Sam mentioned community and how big a part of kind of the decentralized web community is. You know, blockchain protocols tend to build networks and communities of early users who become sort of evangelists. And it strikes me that with Internet Archive, there have been people along the way with you, whether they're volunteers or other advocates, who have helped. Could you talk about community as it relates to the preservation of knowledge?
27:30 Speaker 01 If we don't really have a desire to keep things around, it will just fade away. The example of the Library of Alexandria version one, which ran for 500 years. So it's a pretty good run, but it was a joint project basically between the Greeks and the Egyptians. And they built just an amazing center of knowledge for the ancient world. But by 200, 300, 400 AD, the concept of having universal knowledge or an encyclopedia started to fade as a concept in the sort of rise of some of the newer rebellion movements, really, which was the Christians. It was not supported as much.
28:12 Speaker 01 And there's only about eight pieces of papyrus that we believe exist today that were in the Library of Alexandria of that time. That was told to me by Dr. Sareguldeen, who runs the new Library of Alexandria in Alexandria, Egypt. So we need to keep a culture wanting to have information available. So how do we do that? There's going and having lots of people buy in. I think that's your community question. And I'm really quite happy that last year, 100,000 people donated to the Internet Archive, directly, money. And then there's lots of people that have contributed in other ways.
28:54 Speaker 01 But I'd say, you know, to your point on decentralization, people are supporting it by going and doing. Things on their own that may leverage the Internet Archive, may not leverage the Internet Archive, but we all take this concept of legacy and memory very seriously. I don't think it's just a technological hack that we need to do. There's financial components, there's organizational components. And being able to share, I find difficult for people to adjust to, especially within commercial enterprises. But every which way, people don't like to share in bulk. So how do you go and encourage that and make it, build it in as I think you talked about into the protocol itself?
29:36 Speaker 01 But make it so that there's an ongoing interest in having materials live, I'd say is one of the key parts of the sociological architecture as opposed to just the technological architecture.
29:50 Speaker 02 Yeah, I really, what you just said reminds me of one of my favorite quotes from, I'm a big science fiction fan, and Kim Stanley Robinson, who's a relatively famous sci fi author. He frames technology in terms of it's not just like the physical, you know, it's not just a computer or the way that bits and bytes move around, you know, based on the physics of electricity. Democracy, he defines as a technology, for example, right? It's these social innovations that actually collectively can enable us to do things at far greater scale, enable us to coordinate at far greater scale. And so I think broadening the frame there, I completely agree with. And I do think sometimes it's underappreciated in our current world where technology kind of implies that it's a piece of software or hardware. And of course, it could be a blend of both.
30:35 Speaker 02 Like, take cryptocurrencies and digital tokens, for example. So not only do they rely on cryptography and computer science, technology as we kind of traditionally think of it, but they also have an economic dimension. So like the game theory that underpins the security of that model, and even political and social dimension when it comes to governance, these protocols. And those economic, political, social dimensions are equally important parts.
30:58 Speaker 01 Right, there's new technologies. I love the sort of the token based things. That's completely great to really sort of focus a lot of energy, right? Money is gasoline. So that's tremendous to go and leverage these new technologies for having people coordinate their activities through contributions to these token sales. It's exciting to see this type of energy going into these sort of more traditional topics of cultural preservation.
31:25 Speaker 03 I think there is this like detail here, which is that Technology itself can be money, right? It's not such that the money has to be separated from technology. And in the way that you point out, Alex, about democracy being a technology, really what we're talking about here is just mechanism designs, right? There are incentive structures that we can produce that push rational humans essentially towards some kind of behavior that we prefer. We see this same in democracy and capitalism is essentially that we have adopted competition, helpful competition at the basis of these things in order to produce the best outcomes we can, or at least in theory we can at the global layer.
32:07 Speaker 02 One question that I have for both of you actually is, you know, I think I've read that the amount of data being produced by the human race has increased exponentially over the past couple of decades, particularly as kind of the World Wide Web has come online and more and more people are using it for various things. I'm curious how you view the sustainability of your efforts in terms of how do you keep up in this race of more and more data is created? And therefore there's more and more that's being archived or this permanent hard drive can kind of keep up in terms of storage space. I'm curious how both of you think about that.
32:42 Speaker 03 Because money can be technology. We just built a kind of technological money that creates this endowment structure to pay for storage going forward. So we essentially financialize the lowering cost of storage over time at a very, very conservative rate to ensure that we have this, yeah, it's really an endowment in a traditional sense with a couple of modifications that allows us to pay for storage from the interest from that principle that we put up front when we add a piece of data to the network, rather than actually taking away from that upfront payment. So, yeah, I think these are, I guess, differences in approach. I mean, really, the RWEV approach is a kind of, in some senses, it's more like a traditional business model. It's simply decentralized rather than being a not-for-profit structure like the Internet Archive.
33:28 Speaker 01 And your question about sort of how do you scale? Archives, when the amount of storage that's offered to people producing the things is going exponential. And I think the Internet Archive is about 60 petabytes of data at this point. And it's just what webpages look like. When you talk to the folks at AWS, they don't really think about petabytes, they just think about exabytes, which is kind of awesome. But that's not where we are. So it's just where we could record things that of what the web looked like would be terrific. And it's sort of the promise of the decentralized web. And where are we as part of that whole sphere is can we make the web itself sort of a self-archiving structure?
34:11 Speaker 01 So instead of going and thinking there's a website that's over there and you have to then make a copy of it, snapshots of periods of time, can it be more like how Git works? So that you have a website that evolves over time, but you can go and replay it back to whatever point in time it ran. So you could even fork it. But that's not just the code of the website, but the actual data within it. And have that data, basically, and the code itself for the website, live in many places, not just one. It's too fragile to have things in just one place. Just the idea of a publisher being able to just take a book away and then have it just go away from all the shelves of all of the libraries is kind of dystopian.
34:56 Speaker 01 But it's exactly what the lobbyists for the publishing industry and the authors want to have happen is every reading event is a permissioned event. So it can be turned off at any time. And you can understand why people want that. So think of it as sort of Netflix, right? That Netflix is the only place that you can go and see movies. If that were true, then there's too much power in the hands of Netflix to go and make, oh, some documentary that is not in favor anymore just disappear and nobody can see it again because people hadn't made decentralized copies. So the decentralized copies approach, especially if it's built right into how a next generation
35:39 Speaker 01 web works, that would be a better system than the...
35:44 Speaker 01 Kind of kludge, which is the World Wide Web and then try to back it up separately. So I really applaud the efforts of the decentralized community to try to make it so this is just how the next generation technology works.
35:58 Speaker 03 Okay, so I'd love to get your input on something, Prousta, actually. One of the conversations we have a lot in the decentralization community is about content addressing, right? So should a piece of content be addressed by its location, as it is on the current web, or should it be addressed by the, essentially, the hash of the piece of content itself? Or the way that we see it at Rweave is that it should be really, I mean, no piece of information in the web is just a piece of information. There's much more to it than that that's interesting. There's when did it get there? Who put it there? And also other kinds of metadata. It's very strange to us that on the web at the moment, there's no metadata tags available for stuff. You just get a webpage. And so, yeah, essentially, what we've done with Rweave is just bake in these tags that
36:43 Speaker 03 can be of arbitrary type. And at the beginning, this was really just to add small amounts of metadata. But then we realized the community was picking this up and essentially using the whole network like a massive document store that had these sort of tags associated with stuff that they could query. And then you get these very complex web applications built on top. So I guess, yeah, I wondered if you had some thoughts on that.
37:02 Speaker 01 One of my mentors is Bill Dunn. He started Dow Jones Electronics in the mid 1980s. He was completely inspiring. He said the metadata is more important than the data itself. It's a kind of cool idea. It was really ahead of its time. I mean, you think of how does Google work? And it's all metadata. It's not actually the information in the document. It's often anchor text and usage patterns and all of that. And so Bill Dunn was way ahead of it. He coined, he actually coined the term metadata, which is kind of great to know that somebody actually came up with that and who it was. So there's, I think, context is another way of putting that, is absolutely important. And context shifts over time.
37:44 Speaker 01 So often, you need a mechanism of having materials and then being able to find what are people saying about it. So being able to query based on a hash or something, some sort of persistent identifier, even if the material has moved from place to place, is absolutely critical in allowing new context to be able to be built around it. And of course, all of that context then has to be able to have. Referenceable, where did it come from? And people are able to comment on that. The URL, which is sort of very primitive system that's really location based, is not good enough, that we do need mechanisms to have things be able to live in many different places
38:27 Speaker 01 and over time, and still know what it was, where mechanisms of referring to these materials. And hashes are a little problematic at the moment because they keep changing. You know, okay, well, now we have MD5s. Oh, no, we can't use MD5s anymore. Okay, now it's SHA1s. Great, all right, we standardized on SHA1s. Everybody used SHA1s. And now we need SHA256 because people are breaking the SHA1s. So we have some trouble just based on how do we go and maintain these hashes over time? And I don't have a good answer for that. So that's a real puzzle, but I'm glad you're baking in tags. I hope there's kind of a rendezvous function so that if you have a particular item that you can find other things that refer to that item,
39:13 Speaker 01 that's going to be an important part of a next generation web.
39:16 Speaker 03 Yeah, that makes a lot of sense. I mean, the way that it works in our protocol is we have the sort of base layer that exposes these tags, and then we have indexes on top that are actually very free form. So there's one indexer that uses a kind of a very cryptic, frankly, old type of querying language, but there's a new one that uses GraphQL and another one that looks more like SQL and this kind of thing. And then you can essentially interact with this metadata in a pretty free form fashion. But yeah, absolutely. In terms of metadata being important, I was reminded of this quote, I'm not sure if it, I don't know if we actually know who it came from in the first place, but it was that, quote, "We kill people with metadata or on the basis of metadata."
39:58 Speaker 03 And from in reference to the NSA and, well, the drone strike program that the US runs, which I thought was very powerful. I mean, that's a really... Strong statement about the importance of this kind of data.
40:10 Speaker 02 Yeah, Brewster, I wanted to follow up on one thing that you said, which is the standards keep changing. You know, I think that you mentioned, you know, there was MD5, SHA1, SHA256. In fact, I think SHA3 was recently standardized with NIST. And so I guess it just, you know, this speaks to the fact that the technology is evolving and new innovations are coming out all the time. I'm curious for both of you, what are the technological innovations on the horizon that you think are most relevant to both of your organization's missions? Or put another way, I guess, what are you most excited about in the future?
40:41 Speaker 03 Decentralized autonomous organizations, for sure. This idea that we can program the structure of how an organization should work in code is exceptionally exciting. I mean, we've been pretty keen on this for a few years, even since we started RWE. But then, just in, I think, December time, we decided that we would make a small experiment where we just gave essentially an ownership token in this new DAO that we'd made, so decentralized autonomous organization, to, I think, about 20, 25 of the most core community members in the RWE space. And we gave them a very small amount of money and we just said, hey, you guys just go, basically do what you want with it. And you might have this fear that people would essentially just take the money and run. But quite the opposite happened.
41:23 Speaker 03 They spent almost no money and they've achieved such incredible things in such a short period of time. And they've essentially, they've made like, I think, two or two and a half, depending on how you want to count it, different products. And they've taken a pretty major step in decentralizing the infrastructure of the RWE network. Yeah, on just a tiny amount of money. So I guess what we've found so exceptionally powerful about that was if you truly give people ownership of something, like if you give them responsibility and ownership as a decentralized community that doesn't have to have a formal leader, they're actually able to get things done and thrive and help a community like ours. So that's been really exciting to see. And I'm sure in the future. DAOs will play a bigger and bigger part in our society.
42:07 Speaker 03 But this is probably like 10 to 20 years off, like in the mainstream sense.
42:11 Speaker 01 Culturally, the things that I'm kind of excited about is people learning to work together better, whether it's the Wikipedias, the Brave browsers, hopefully the Chromes and Mozilla's, the Internet archives, that people sort of see that it works better to work together. In terms of what's going on right now, in terms of this whole COVID thing and the economic impact that's going to be, I'm really hoping that we de-financialize more of our systems. So people actually go back to owning things for real. Because when everything's leveraged up to the hilt and it's owned and controlled to go and multiply results and it's all ROI, it's a very fragile piece of infrastructure.
42:52 Speaker 01 So I think that that, in terms of just how to build a more robust society, I think de-financializing, which is sort of trying to undo some of the financial engineering masterpieces that have come out recently and have been showing the fragility and centralization problems that we have. And in the technology area, I agree with Sam in that sort of the exciting things of the decentralization. I want blank but decentralized. Let's take our favorite tools, but let's make it so that it's not going and playing into somebody else's business model to go and hoard data. I use Slack all the time, but it's a little creepy, right? It's all, because it's in a third-party service. So I want Slack but decentralized.
43:35 Speaker 01 Ideally, I'd want Slack to go and make that because they don't need to go and hoard all this information for their business model. Or Google Docs but decentralized, or Google Maps but decentralized. There are these miracles of technologies that we've built centralized versions of. So we know what the future we want is, but can we build blank? But decentralized. I find that another sort of useful shower thought experiment is like, how would I go and do that without having all of this information be hoarded by these organizations that may be profiting for it, buy it, or at least handing it over to people that you don't necessarily think that you want to have that information? How do you have those toll tags that are on our cars that allow us to go zipping through
44:19 Speaker 01 the tolls, go and be paid securely, make sure that they're done, but not leave a trace of that you went through that toll booth at that time? All they should want is just to get paid for somebody to go through that toll booth. How do we make it like cash? So those are the sorts of, I would hope that the technology community really works on. And if it's based on open standards, then there may not be quite the lock-in sort of thing that a lot of investors are looking for. But I think it may be what it is society is looking for. And I think in general, we should be pursuing things that society wants.
44:55 Speaker 00 Bruce, back around 1980, you set a goal. You were pretty specific, I believe. It was October 2020, you had a specific goal about making the internet a library. Can you tell us how close we are to achieving that goal?
45:06 Speaker 01 We're now really depending on this information ecosystem, yet it's not good enough yet. It's not referenceable, doesn't have enough of the published works woven into it. There's a, for older folks, I was trying to think, what did I learn about a library in high school? And there was this thing called The Reader's Guide to Periodical Literature, and it was a subject index. So if you were doing a report on, say, cigarette advertisements from the 1960s, or the McCarthy era of the witch hunts of communists, and you wanted to go and understand, how did people think about it in 1955, right? What party was for it and what party was against it, right? That kind of thing.
45:51 Speaker 01 It's very difficult to answer that kind of question on the World Wide Web now, but we had this thing called The Reader's Guide to Periodical Literature, where you could go and look up McCarthy in 1955, and go and get articles from Time Magazine, New York Times, all these different things, and go and, actually at that time, you went back to microfilm, to go and pull those out, to try to walk back in time and feel like it was 1955 again. What was the media that was out there for you? Wouldn't it be great if we could actually do that again? So people could start to feel empathetic or understand how did things used to work? And that is not available yet on the net.
46:34 Speaker 01 That would be a component, I would say, to making the internet into a library. Another is reference desk, right? How do you go and have that helpful librarian be able to help you find things? Is that really there? Is Google good enough for that? I'd say no, we're really not there. What do we want out of a library? And how do we make it so that the next generation, this generation, our generation? All of us that are turning to our screens to answer questions? So we have as good, a rich, as deep background as to what's going on? What's true and what's not true? Who should you be thinking about? Who should you be reading about, even if they're not currently in print? Those are the aspects that I find a useful thought experiment.
47:19 Speaker 01 And we're using at the Internet Archive and other organizations to try to build a reliable, thoughtful infrastructure to build our culture on as opposed to kind of the slapdash, techie thing that we've ended up now depending on so fully.
Previous Alex Honnold on Human Performance (part 2) – Climbing and Entrepreneurship Next How Transparent Pricing Drives Healthcare Change Page updated October 26, 2026 · transcript and summary are machine-generated