a16z Podcast
Original podcast
Where to listenCopy share link
4,725 words · 2 speakers
Perfect. Thanks. Yeah,
so uh I'm excited to tell you about some work that we've been doing over the last couple years in my lab, which is on a system called Deep Dive, which is what we call a dark data system. Now, uh during this talk, I won't have time to sort of parcel out credit to all the people who deserve it, but there are a bunch of people on this team and a bunch of people who uh I didn't put on the slide. Uh everything I'm gonna talk about is open source. So if there's details that are missing, please go ahead and check out the open source. So uh deep dive is what we call a dark data system. So these are the types of systems that start with things like emails, web pages, unstructured information in tables, increasingly images. And the goal of those unstruc uh the of these dark data systems is to take all that information which is inaccessible to machines and basically create rich structured data out of it. Sort of traditional SQL style databases out of these text tables and figures.
Now, this is a problem that people have been working on for a couple of decades. But very recently, there's been a kind of an exciting development in this area, which is that the quality of these kinds of systems has dramatically improved. And I'll tell you about our work in this area, but there are a number of systems that have shown that it's actually possible to build these ETL pipelines, these pipelines that extract, transform, and load information with higher quality than humans in some in some very simple settings. And we've compared in our work against things like human annotators and science, but also increasingly we've been comparing against people from corporations and inside the enterprise and actually building these ETL pipelines in a short amount of time and in some ways with higher quality than humans. And I'm gonna validate those claims in the middle of the talk. Now, when you look at that claim, you may say, well, this is a little bit strange.
If I think about ETL as a process, there really are sort of three separate problems that people have been working on. And if you go to any conference on this topic, you'll see entire tracks about how to extract data. That's go into the text and find the mentions and entities you're interested in. Another track about how once you've extracted those entities, how do I integrate them with my existing structured database? And once I have those two databases actually merge, how do I clean up the information so that it's usable for all the downstream analytics? And those are sort of three separate problems that people have been studying for a while. Now, in Deep Dive, we took sort of a strange academic approach to this and said, what if we viewed all of them as essentially one statistical inference problem? So we're gonna view all of these three problems holistically together. Now, at first glance, this sounds like an insane solution.
Why would I take three hard problems that really haven't been solved and stick them all together? What could possibly uh be gained by doing this? And really, there are two things that make me think this is not an insane idea. The first one is in every application we actually built, we had all three of these problems. You always have to do extraction, integration, and cleaning. So we really weren't making the problem any more difficult. A second issue is that by doing probabilistic or statistical inference, we could actually focus on the part of the pipeline that was most relevant for end-to-end application quality. So in contrast, if you think about the way that people build these extraction and integration and cleaning systems, they tend to over-extract, over-integrate, and over-clean, because they have no idea if their cleaning or extraction is actually going to prove the end application goal that they're interested in.
In contrast, we can build a pipeline first, which may not be very high quality, and then use probabilistic inference as a way to say where should we spend our effort next. And the reduction in effort can sometimes be orders of magnitude, as we've seen in some of the applications we built in our lab. Okay. Now, another goal of deep dive is to try and make it accessible to people who are not computer scientists as much as possible. And I would not claim that we've succeeded in this goal, but we've made it substantially easier. People in bioengineering, in law enforcement, have actually written programs against deep dive and gotten value out of it. To do that, what we've done is raise dramatically the level of abstraction that you write programs against deep dive compared to other machine learning tools. In particular, you cannot write down an algorithm inside deep dive. You only specify things like the features, the random variables and the correlations, completely decoupling how that inference problem is actually solved.
Now, if you do this, you have a real serious problem. And any time that you remove something with an abstraction, you put more burden on the system. And so what we realized a couple of years ago was that if we were ever going to make this vision succeed, we had to go and build basically huge scalable inference engines. And so in the second part of the talk, I'll describe what we've been doing there to try and take advantage of modern hardware and in particular the parallelism that modern hardware exploits to be able to build these scalable inference engines. So before all that, a problem motivated deep dive. And the problem I'll just motivate for you is a purely scientific question. So as we started to walk around campus, one of the things that we realized were that all the scientists that were around us were able to access information that they had never seen before. So in a way like never before, scientific information was accessible to them.
They know the keywords, they know the topic, they can go out there and get the PDF. The problem they faced was that for even the narrowest of scientific questions, they couldn't possibly read all the information that was relevant to them, or even a significant fraction of it. So this is why I say the scientific knowledge is accessible in a way like never before, but it's not really readable. Now, worse. Today, some of the most pressing problems that we were interested in in our lab and our collaborators were interested in, are what we call macroscopic problems. These are problems where we have to take all this information that's available out there and assemble it in one place, and then and only then can we make an estimate about what we're interested in. So for example, in climate and biodiversity, we would want to build the basically the world's biggest fossil record, where every fossil was found anywhere in the planet, put that in one structured database, and then relatively simple analytics would tell us what's going on with various die-offs over time.
And we could do that analysis if we could get that information which is buried in PDFs and charts into a structured database. Now, climate and biodiversity isn't unique in this way. At Stanford, we've been doing a bunch of work in health, and we have some things I can talk about offline. And also financial markets is another area where these uh there's this integration of a massive amount of information to try and make a best effort. But I'm gonna talk about climate and biodiversity, because it's the first one we actually built. So we asked this very naive question. Since the machine since the literature is not readable, could we build a machine to read for us? Now, I don't mean that it's gonna read like poetry. It's not gonna read and feel like it's getting moved by the words. I literally mean that it's gonna go from the process of the bits that come in that are unstructured to the nice structured database. Now, unfortunately, that process requires inference.
It's not as simple as just copying the bits that appear on the page and putting them in a structured database. And we want to be able to do that process as automatically as possible. Okay. So let me show you something that we'd like to read, and I don't expect you to read it right now. Okay, it's too small. But this is the kind of information we'd like to, a paper we'd like to read, a scientific paper we'd like to read for this biodiversity application. So in this table right here, we have information that we'd like to extract. So here we've actually seen that in this region called Obora, we have this genus, more of Alamacrias was a kind of cockroach. If you love cockroaches, this is really hot stuff. You want to know that this cockroach was found in Obora. Me, I personally don't care, but you'd probably want to know it if you were a cockroach enthusiast. So in any event, if you think about the computer science that you'd have to do to be able to extract this information, you'd have to make a bunch of choices.
So if you think concretely about the program you'd have to write, you'd have to say is Obora a region or not? Is more of Alama Crias a genus, a species? What is it? I'd have to deal with the OCR, the optical character recognition, and make sure I'm grouping the terms correctly. I'd have to make a choice that actually a relation was expressed visually in this uh in this table, and I'd like to be able to learn that pattern so that I can repeat this and save some work. Even if I solve all those low-level issues, I still have another problem. The obora that's in my table here, turns out that there are 40 different regions inside my data set that are informally referred to as OBORA. So even if I solve all what all those low-level or what we call mention level challenges, I still have these higher level entity level challenges to try and resolve the information of which obora I'm talking about. Now, in this example, we get uh deceptively uh lucky.
We're actually much less lucky than it seems here. In the text of the article, the body of the article, we actually get this great sentence that says, the OBORA site is situated, and then it gives a description, and here we get the GPS coordinates. And so if we could go into the text, and indeed we do go into the text and read this information, and again make all of those little choices about what the text means. We could potentially link these two pieces of information and get very high value data, the kind of data that a person would be able to extract from reading this information. So this isn't vaporware, this is actually something we built. It's called Paleo Deep Dive, it's been live for a while now. Here's how it works. So we want to extract this higher coverage fossil record. That's what we're after. And basically, this is what people were doing. Bunch of PDFs come in, and they try to type into a standard SQL database what they read in those PDFs, where they found a fossil and what rock formation in what time period.
What we're gonna do instead is we're gonna take that text, if you like, an OCR and all the rest, and basically do statistical inference to try and predict the most likely database that the humans would have filled out, given all the observations from text and NLP tools and all the rest that we run over the corpus. Okay. Now, our approach is a little bit insane or aggressive. Every one of those choices that you would have made if you had written the ETL script, we're going to basically ask you to write down a random variable. And that random variable is gonna say, I don't know. I don't know which Obora I'm talking about, but it's one of these 45. I don't know if Orbora is the word is a location or if it's a genus, but it's only a set of three or four possibilities. We're gonna let you write down those random variables, all of those choices, and then we're going to basically build a very large factor graph, so a set of random variables and their correlations, and we're gonna process them.
In some of our applications right now, these factor graphs become hundreds of terabytes in size. So if we were to write them all out to disk, which we actually don't. But the point is is that to be able to pull this vision off of being having this very high-level programming language, we're going to have to do very scalable statistical inference. So if you've seen this. Area, you've probably said, well, okay, fine, people have tried approaches like this, but the quality is never very good. It's, you know, you'd still prefer humans. Humans are actually aren't that expensive. Let's just pay them to do it. So a couple years ago, we compared to this database called Paleo DB. And these numbers are a little bit out of date. But this was something where people weren't in pain. They were actually 300 or so volunteers reading papers and putting that information into a database. They spent over a decade on this, and in fact, they spent two continuous person decades since we have the logs, reading the information, typing it into a database.
Okay. So we decided, let's make a machine version of this that we called Paleo Deep Dive. We'll put a single graduate student on it. What could possibly go wrong? So we put this one graduate student who's phenomenal, So Jong, who did this, and he decided he was going to build a system. Now I'm going to skip the two years of him crying tears of blood to get the thing to work, but after that, it turned on. It was great. From my perspective, it was like two minutes. So we turn the machine on and we read about 10 times more documents. Now why do we read 10 times more documents? It's a machine, it doesn't get tired, you pipe in as many bits as possible. The thing runs in now under an hour. We're not constrained by the amount of data that we're reading. You may ask why only a factor of 10, why not a factor of 100? And the usual glib summary I give is that publishers are criminals. I've sort of softened my stance. We can't get access to the data because publishers have said yes to us, but we can't legally get access to about another order of magnitude of data.
Okay. So there's more data out there to read. It's not a technological barrier. Even when in that 10x more data, there's a hundredfold more data that we could extract from that. These are what we call subjective taxonomic opinions. This is I look at a fossil, where does it belong in the tree of life? Relatively boring data, but it's something that is littered throughout these reports and throughout these papers. And what we see here is the first reason why I would argue that you don't want people doing these extraction tasks. We get about an order of magnitude more data. And that's data in the same papers that they were reading, basically, they just weren't extracting it. They were, you know, distracted, they didn't want to do it, it was a painful task. They check the document and say that it's marked off, and it's sort of lost to the ages, some fraction of that information. Now, the second thing to look at is when we extract a piece of information, how accurate is it?
And the first question to really ask is how accurate is a person. So we looked at this one task where someone reads particularly an uh a location, and they try and match the rock formation or location to the real canonical name. So they read it, they read the article and they actually have to put in the correct real world name of a of a location. How often does a volunteer get that right? And we assess this, we found that the Paleo DB volunteers were accurate about 84% of the time. So what are they making mistakes on in that 16%? A rough summary is the following. When they read the paper, they have some background knowledge. These four rock formations change names over time. I read some rock formation, I say, oh my gosh, it's called by a new name. I should call it by the new name, not the old name that's written in the text. When they're right, this is good data. You know, they've improved the quality of your data. It's it's up to date.
When they're wrong, it's kind of catastrophic because you can't possibly rerun the human computation, the two decades long computation to make sure that that thing is fixed up. Even when they're right, it's not great. And the reason it's not great is that it's not done systematically. One annotator is doing it, but another annotator is not. And so your data is kind of a mishmash of different conventions of how to populate this data. Now, our machine, in contrast, is a function of a small number of rules. It's statistical, but it's really only a function of a small number of rules and training data. And so it can be trained by experts to be closer to what they want to see than the volunteers. And so, not surprisingly, our quality is actually higher on this predicate, that is doing this formation extraction than people. Now, in both of these cases, we've picked things where we're better than our competitors, better than the human competitors, and we compared across all the predicates.
In some predicates, it's basically a wash. There's information that humans and machines are equally good at, but we're never any worse. And in some cases, we're substantially better. So our hope when we saw this was that we could actually extend this throughout all of different kinds of sciences. So we not only doing the tree of life, but things in drug repurposing and genomics. We've actually used this inside a couple of companies, which I'm happy to talk about offline, and we even won one of the uh major NLP competitions using this approach last year with Chris Manning's group. Okay. Now, one thing that you may have seen here was that actually there's a database and we're basically adding tuples to it. And one thing that I'll skip over a little bit quickly is that there are situations that we can use this information in which there's no database. And one thing we've been working on a lot is this human trafficking application with DARPA, and basically the idea is to go out and read a bunch of web text that is about people being advertised, sexual services being advertised to try and identify who's trafficked or not.
Okay. Now it turns out to do this that basically what we're doing is building like the world's best prostitution pricing model, which is a little bit weird, but that's what we're doing to try and identify who is potentially trafficked. And one bit about that that's quite interesting is not only can we do this with high quality, but they're actually taking countermeasures against us over time, as it turns out we actually start to make some real arrest in this domain. Okay. So I'll skip over the details of the application, but I will say that it's actually an active use, and we can actually get high precision values, and we're in extending uh pilots to more law enforcement agencies over the next couple months. Okay. Now the last thing that I wanted to talk about was basically the trends that are coming up that make us build the engine the way we have built this engine. And we, as I mentioned, we're getting rid of algorithms, we will choose the algorithm for you.
And our trick is we're gonna choose the dumbest algorithm I can imagine, which is Gibb sampling, and we're just gonna run it as blazingly fast as we can.
Okay.
So to do that, we need a really really fast uh, we really fast engine. And it turns out that modern hardware has one particular feature that's really helpful to us. And that feature is massive amounts of fine-grain parallelism. SIMD parallelism, multi-core parallelism, NUMA, those are all things we need to take advantage of to go in some cases orders of magnitude faster than competitor systems on these tasks. Okay. Now, one thing I want to highlight here is that when we build these systems, it's really a systems problem. It's a computer science systems problem. We're always balancing two factors that are in sort of fighting with each other. And those two factors here are what we call statistical efficiency and hardware efficiency. So statistical efficiency is basically the number of steps the algorithm is gonna take until it converges. This is what optimization and machine learning people have been working on for quite a while.
On the other side of the coin is hardware efficiency. This is basically those primitives that the hardware exposes that in some cases can be orders of magnitude more efficient than the conventional, you know, one step at a time kinds of operations. Okay. So we studied this, and it turns out that various models are in different parts on this trade-off. But I want to illustrate for you why these trade-off spaces entirely different. That is, why it's a new trade-off space in classical trade-off space uh don't sort of suffice. And to do that, I'm gonna show you one result which just illustrates the classical models of uh sequential uh evaluation you can completely throw away. Okay. So the crash course in machine learning for this slide is the following. Uh, we're going to look at these problems where we're trying to find some X, we're gonna minimize X subject to YIs, which are a bunch of data, and we're gonna try and minimize a loss function.
And we're gonna have billions of these terms in this sum. And this captures things like deep learning and classification and recommendation. These are linearly separable problems. Okay. Now, it turns out to solve these problems. There's actually an emerging theory about why this is true, but people basically use one simple, very, very stupid algorithm called stochastic gradient descent. Okay? And the way SGD works is a very simple idea. You have an estimate at some time XK, you could grab one term from the sum, you take a gradient, you compute the direction that's gonna minimize you, the opposite of the direction of the gradient, and you update it. And all that's important for us is you're just gonna update this many, many, many many times and have billions of these tiny little iterations. And so, from a systems perspective, if you're thinking about what the hardware sees, it's seeing billions of these tiny, tiny little iterations.
Okay. Now, why that's important, as I illustrated before, as I talked about before, is that modern hardware hasn't been getting straight line fast for a while. It's just been giving us wider and wider processing units, more parallelism in fine-grained ways. And the problem is this seems like a very natively sequential algorithm. You read an X, you update a few components, and you write it back. It seems like it's got to be natively sequential. It doesn't seem like it's gonna be possible to really exploit parallelism for it. So what we decided to do to do this was a uh kind of a stupid idea, but it actually worked quite well. So if you know how processors work, they communicate with each other by locks. They take a lock and say it's my turn to write the value, it's my turn to read the value. And they go through a locking protocol. What we decided to do for these statistical algorithms was essentially throw away all the locking.
Just comment them out entirely. And this is a technique that we called hogwild, which has an exclamation point and will be important in a slide or two. And basically you just toss away the locks. Now we did slightly more than toss away the locks. We've proved an embarrassing number of theorems that say under what conditions on sparsity and what ways of getting parallelism, SIMD parallelism, you know, we're able to actually do this and get linear speed ups. But we're able to basically prove situations under which throwing away the locking actually doesn't damage the statistical efficiency of the algorithm, and as a result, we can take advantage of all that hardware goodness that's underneath the covers. Okay. Now I have to share one thing with you because this is the height of my academic achievement, and you'll see that it has to do with the exclamation point. So this is a Project Atom uh slide. So if you haven't paid attention to deep learning recently, uh the world goes like this.
Google recognizes cats, Microsoft recognizes dogs, AI overlords take over the universe. So I think that's the that's the clear succession. Anyway, so a while ago, last year, Microsoft released their Project Atom, and they had a great demo where they were showing they were releasing, you know, recognizing shih tzues with Windows phones using deep learning. And the reason I will love Microsoft forever is the following quote that appeared in Wired magazine. They claim that they were using a technology called of all things hog wild. Now, two things about this made me very happy. The first Is they got the exclamation point correct. So I was really, really happy. My troll heart was was filled. And the second thing was they said, the author said, of all things. Like she was genuinely like thought it was insane that a billion dollar corporation would base something as stupid on the name Hog Wild. So I was I was really happy.
I hit all my buttons. Okay. Anyway, but being serious, the reason I show this particular result is that this idea that we can relax consistency in fundamentally new ways to run these statistical algorithms to take advantage of hardware is sort of just in its infancy. We've basically done a couple of parlor tricks, and those parlor tricks have allowed us to get you know 10, 100, 1,000 times faster than competitor systems because we're actually taking advantage of what the hardware is giving us. But there's a lot more to do here, and there's a bunch of great work that's going on with people using these ideas and more in different settings. So to conclude, I basically just told you two things. One is that these dark data systems, these knowledge bases, help with what we call macroscopic questions. We're really interested in this idea of going beyond just look doing fact lookup queries to basically integrating all the information that's available about a particular topic, putting that in one place and then answering questions on top of it.
The other thing that I illustrated for you was that to be able to build this system, we needed to build scalable inference engines. And to do that, we had to be able to relax consistency. And that is one technique, but it's one that we think keeps coming back and may come back in other forms for energy efficiency, for performance, and so on. Thank you so much for your time.
Page updated August 25, 2026 · transcript and summary are machine-generated