EP 52: Jason Warner & Jacob Effron (Redpoint Ventures) - Former GitHub CTO Explains AI for Dummies

In the 52nd episode of the Logan Bartlett Show, Redpoint’s Jason Warner and Jacob Effron join to discuss the quickly expanding AI landscape, what the AI models are, and why we should be paying attention. (0:00) Welcome Jason Warner and Jacob Effron (8:30) Why does the AI boom feel like it’s come out of nowhere? (12:30) How does it actually work? (16:02) Compounding on itself through usage & training reward models (18:08) Two most mainstream AI models (22:20) Why do people care about open source vs closed source? (30:45) Companies at the foundational level (36:05) What’s exciting from a use case standpoint? (44:28) Changes occurring in the workforce (48:15) Who should be in charge of AI policy (54:12) Comparisons to crypto (58:54) Future-proofing yourself as a founder outside the AI landscape Show Notes: Unsupervised Learning Podcast: https://open.spotify.com/episode/6Zn7A4TPHRScBPZ2pA6sZ1?si=avH3lLS8R4iBuocgUAiYxQ Unsupervised Learning Substack: https://unsupervisedlearning.substack.com/ OpenAI Cookbook: https://github.com/openai/openai-cookbook#readme Langchain Overview: https://langchain.readthedocs.io/en/latest/ Attention is All You Need Paper: https://dl.acm.org/doi/epdf/10.5555/3295222.3295349 Henry Kissinger and Eric Schmidt book: https://www.amazon.com/Age-I-Our-Human-Future/dp/0316273805 Mixed and edited: Justin Hrabovsky Produced: Andrew Nadeau and Rashad Assir Executive Producer: Josh Machiz Music: Griff Lawson 🎙 Listen to the show Apple Podcasts: https://podcasts.apple.com/us/podcast/three-cartoon-avatars/id1606770839 Spotify: https://open.spotify.com/show/5WqBqDb4br3LlyVrdqOYYb?si=3076e6c1b5c94d63&nd=1 Google Podcasts: https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5zaW1wbGVjYXN0LmNvbS9zb0hJZkhWbg 🎥 Subscribe on YouTube: https://www.youtube.com/channel/UCugS0jD5IAdoqzjaNYzns7w?sub_confirmation=1 Follow on Socials 📸 Instagram - https://www.instagram.com/theloganbartlettshow 🐦 Twitter - https://twitter.com/loganbartshow 🎬 Clips on TikTok - https://www.tiktok.com/@theloganbartlettshow About the Show Logan Bartlett is a Software Investor at Redpoint Ventures - a Silicon Valley-based VC with $6B AUM and investments in Snowflake, DraftKings, Twilio, Netflix. In each episode, Logan goes behind the scenes with world-class entrepreneurs and investors. If you're interested in the real inside baseball of tech, entrepreneurship, and start-up investing, tune in every Friday for new episodes. Executive Producer: Rashad Assir Producer: Leah Clapper Mixing and editing: Justin Hrabovsky Check out Unsupervised Learning, Redpoint's AI Podcast: https://www.youtube.com/@UCUl-s_Vp-Kkk_XVyDylNwLA 🎥 Subscribe on YouTube: https://www.youtube.com/channel/UCugS0jD5IAdoqzjaNYzns7w?sub_confirmation=1 Follow on Socials 📸 Instagram - https://www.instagram.com/theloganbartlettshow 📱 X - https://twitter.com/loganbartshow 🎬 Clips on TikTok - https://www.tiktok.com/@theloganbartlettshow About the Show Logan Bartlett is a Software Investor at Redpoint Ventures - a Silicon Valley-based VC with $6B AUM and investments in Snowflake, DraftKings, Twilio, and Netflix. In each episode of The Logan Bartlett Show, we sit down with the people behind today’s most important startups and extract the tactics, lessons, and frameworks they’ve learned the hard way. Conversations span hiring to GTM, product, growth, fundraising and everything in between - collectively forming the ultimate playbook to make you a better CEO, investor or board member. Tap follow and enable notifications to stay ahead of the game.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Logan Bartlett?

  Welcome to the 52nd episode of the Logan Bartlett Show. I am your host, Logan Bartlett, a partner at Redpoint Ventures. And on this episode, we're going to do something a little different. So we're going to talk about some of the things we've been working on internally. On episode 46, we had Iman Mostak from Stability AI, one of the biggest contributors to stable diffusion, come on and talk about AI and the advancements that have been made in generative AI and large language models and all the stuff that they're doing. And I think we've talked about what OpenAI is doing, Sam Altman, all of that. And so we've been working on this stuff

• 0:44Logan Bartlett?

  internally. And it seems like this is everywhere you look, there's more and more talk about OpenAI. Now, ChatGPT is going to power elements of Bing Search. Google last week rushed to make an announcement around Bard, which is their ChatGPT equivalent that they announced last week. Now it seems like Microsoft is going to start embedding elements of ChatGPT into some of the Office products. So it seems like it's going to go into Outlook and Word and PowerPoint. And so this is clearly what is the topic de jour in Silicon Valley. So I thought it made sense to have on two of the people that have been working most on this internally. So I'll let them introduce

• 1:29Logan Bartlett?

  themselves here now. But I am joined by Jason Warner, who is a managing director with me and works on our growth fund, as well as Jacob Effron, who is a partner here at Redpoint from our growth fund as well. So two of the people internally that know the most about this stuff and thought it made for an interesting conversation. So thank you both for joining.

• 1:50Speaker 01

  Thanks for having us. Thanks for having us. Can't believe it's 52 episodes. I'm just, you know.

• 1:54Logan Bartlett?

  I know. I can, by the way. When you have to record this, it's, yeah. It's 52 episodes in, and it took 52 to really pivot to Redpoint propaganda here. So thank you guys for doing that.

• 2:08Speaker 02

  Yeah, we appreciate you warming everyone up for us.

• 2:11Logan Bartlett?

  Yeah, exactly. Fifty-one episodes of No Redpoint Branding, and then fifty-two. Here on out, it's just gonna be pure shilling Redpoint propaganda. No, but these guys do know a lot of this stuff. They're actually gonna launch their own AI podcast, I guess with this episode, right? Unsupervised Learning should be out by the time you're listening to this. So we will link that in the show notes, but exciting to dive into all this stuff. So Jason, you have experience in AI and in particular with OpenAI and ChatGPT, one of the first use cases, I guess, that actually has entered the mainstream. So maybe can you give a little bit of your background and a primer into why you're credible enough to have on this illustrious podcast?

• 2:56Speaker 01

  Sure, sure, sure. So for those who don't know me, my name's Jason Warner. Prior to working with these guys here at Redpoint, I was the CTO at GitHub for four years. So two years pre-Microsoft acquisition and two years post-Microsoft acquisition. Prior to that, I was VP of Engineering at Heroku, which is now a platform as a service owned by Salesforce, so high-tech. And before that, Canonical, the people who make Ubuntu Linux, so at the center of the open source world. Going back to GitHub, you know, things at GitHub that we worked on, you know, actions and packages and advanced security. But, you know, one of the final things that we incubated in the office of the CTO with a bunch of really smart folks there just before I left was Copilot.

• 3:42Speaker 01

  And Copilot, you know, probably needs no introduction at this point, is a collaboration between GitHub, Microsoft, and OpenAI at the center of it. And so, you know, we'd been working on that for probably six months in the office, and then we announced it. And then it'd been about a year in incubation, or sorry, a year in beta, and now it's generally available for everybody to use.

• 4:05Logan Bartlett?

  And what does Copilot actually do for those of us that aren't familiar with it?

• 4:09Speaker 01

  What we had liked to think about Copilot being was it was going to be effectively an exoskeleton or an Iron Man suit for programmers. It wasn't supposed to take away. Programming, it was supposed to augment a programmer's ability to get stuff done. And so it is, it's a generative AI model. It's a generative AI model for code. And it's supposed to offer suggestions, hints, tips, things of that nature about what you're working on. And things that the model will come up with saying, hey, we think you're doing this. This looks like something that would be appropriate for you. Do you want to accept this or not?

• 4:42Logan Bartlett?

  Yeah, the way I think about it, and I don't know if this is a fair analogy, but yeah. In Google email responses, when it basically autofills or gives this suggestion or Google search, right? When you're typing something in and it autofills elements of that. It's basically that much more advanced for code. Is that, is that a fair comparison?

• 5:02Speaker 01

  Very fair. I mean, obviously it's going to be at a different level and complexity. And, you know, for, for reasons, we kind of picked like code function level, you know, method, full method body level. We're trying to, to complete things. So think of it as, hey, this next paragraph in the email replies. And we'll, we'll give you the full, the full paragraph there. So I love the Gmail stuff because I hate having to write, you know, looking forward to talking to you soon. And Gmail gets that off for me. Similarly, I don't think I could go back to work without Copilot these days.

• 5:29Logan Bartlett?

  And do, can you give any adoption stats? Has Microsoft reported anything along those lines of how many people are actually using it? And what it, what it means for productivity and all of that? Anything along those lines?

• 5:41Speaker 01

  Sure. So I have some, they're all the public ones. Now, obviously I sell friends at GitHub and I had, there's some private numbers, which I'm not going to reveal here in the private. But I'll just, we talked about, they talked about some of the public ones, which are about 1.2 million developers are using it at the moment. The about 40% of code that gets written, gets written automatically by Copilot in the, in the context of using it. And the, I'm not sure exactly how they quantify this, but it shows a greater than 50% time savings. And I, again, not sure how exactly they quantify that, but that's how they talk about it publicly. Now, that is trajectoryally kind of directionally in line with exactly what we were hoping to achieve out of this. A lot of people using it, but in reality, we're writing a lot of boilerplate or ceremonial code or getting a lot of this kind of like BS stuff of code out of the way.

• 6:25Speaker 01

  So people can kind of go to the creative elements of it and just saving buckets of time for folks.

• 6:30Logan Bartlett?

  Yeah, makes sense. And so this is really, I mean, the first commercial application that sort of existed in the wild of what OpenAI was able to do, right?

• 6:41Speaker 01

  They had GPT-3 out at the time. You could get API access to GPT-3. And in fact, the Codex model upon which, you know, Copilot is built is a derivative of GPT-3. So they trained GPT-3. We together trained Codex from GPT-3, and that's the new model.

• 6:58Logan Bartlett?

  Got it. Okay. Well, certainly qualified to talk about this stuff. And Jacob, maybe give a quick primer on what you've been doing over the course of the last couple of months, outside of being probably highest IQ internally, I think we had, and whatever, top of your class at Yale, whatever those things were that we-

• 7:15Speaker 01

  People can decide if that's a high bar after listening to-

• 7:18Logan Bartlett?

  I don't know how much exposure they've gotten to other people at Redpoint beyond me, but I assuredly feel confident that at least your SAT scores and GPA and all that were far better than mine. So the bar is at least higher than than me here. But Jacob, maybe give a little bit about what you've been doing.

• 7:34Speaker 02

  Yeah, so I spent a lot of my time with Jason and AI, you know, kind of always been fascinated by the potential of these models. Joined Redpoint from a big data and cancer care company called Flatiron Health, actually with a podcast favorite, Zach Weinberg is a founder, and then spent time previously working with an incubator called Four Catalyzer that was basically building algorithms on top of low-cost medical devices so that anyone could use them. And so, you know, alongside Jason, I mean, we've been meeting just tons of companies in the space, kind of developing our own internal perspective. And then, you know, one of the most interesting things we've been doing is just talking with a ton of different companies out there that are experimenting with these models and trying to get a sense of what they can build into their products. So companies like Canva and Descript and Neva and Sourcegraph and a whole host of companies just playing around with

• 8:17Speaker 02

  both the closed source and open source models to start incorporating them into their products.

• 8:21Logan Bartlett?

  Yeah. Zach Weinberg was a reference for Jacob's hire. So we will, we give him credit internally for for that. But yeah. All right, Jacob, so you've been leading a lot of the initiatives around this stuff. Can you explain what has happened here in the last year, two years that sort of makes like, makes it feel like this has come out of nowhere? I know it's been a slow build, but now it feels like it's ubiquitous and we have all the crypto VCs pivoting to AI. So why is this happening? Happening now, why does it feel like this came out of nowhere?

• 8:53Speaker 02

  Yeah, well, you've had a technological change and a cultural change. So I guess technologically, it all started in 2017 with the advent of transformers, which happened at Google. And I think the basic thing to understand around transformers is that they allowed basically you to throw massive amounts of raw data at a model and the model to process that with higher and higher scales of compute. And so before you were really limited in the amount of compute that a model could do because you had to go one word at a time in a language model. But now you could kind of, you could parallelize that across the entire model. And so what that enabled was you moved from this kind of previous generation of AI.

• 9:34Speaker 02

  If you remember, companies were so focused on how do we get the perfect clean data set for this very specific task, like let's label whether something's a dog or a car. And now you could just throw tons of raw data from the internet to do, in the case of GPT-3, predicting what the next word would happen in a sentence. And so, you know, what you basically saw was this technological progress, which kept furthering. And OpenAI was really at the forefront with GPT-3 and then this thing called InstructGPT, which made these models even better. And so all the while you had this technological progress, but it was sitting there for a while and people were kind of using it, but not really using it. And then in 2022, two things happened that made these things much more open.

• 10:17Speaker 02

  So the first thing is that Ahmad actually was really at the forefront of open sourcing and making available some of these image generation models. So if you've ever played around with Midjourney or Stable Diffusion, you know, these models came out in the summer of 2022 and just took off. I mean, if you look at the developer adoption of these models, it's insane. It's faster than any open source product we've ever seen before. And then the second thing that happened is, you know, OpenAI had API access, as Jason mentioned, to these GPT-3 models, but nobody, you know, some people were using them and they were kind of waiting for people to build different applications on top. And at a certain point, they said, huh, no one's really built a chatbot-like application that, why don't we just release that? And the funny thing is, if you talk to the OpenAI folks, they didn't think this was going to be some like

• 11:01Speaker 02

  Massive thing that everyone used. They thought, yeah, it's kind of a natural just extension of the work we've done. And I mean, anyone who's spent any time on Twitter or in the tech world, I mean, ChatGPT has just exploded, right? And so the model capabilities had been there, but the kind of interface to just, you know, type in a rap battle or a sonnet or whatever it is that folks are using these things for, you know, was newly made available. And so I think what happened is, you know, people just got to experiment with both these image generation models, these chat models, and it really kind of transformed folks' imagination around what you could do.

• 11:34Logan Bartlett?

  Jason, I want to talk about the, so there's, there's two components, right? The cultural element that helped democratize access and understanding. And I will say it's pretty mind blowing to say, to type in, hey, a dog in Rome that is orange, right? Or whatever it is and have-

• 11:52Speaker 01

  Always orange with you.

• 11:53Logan Bartlett?

  Yeah, yeah, always orange. A dog that's a Tennessee volunteer fan. And be able to have an image come back. So clearly that is groundbreaking from a cultural element. But on the technological side, I want to drill into this in, in a simple form for people to understand. So you're throwing all this data at, it, it, it used to be we needed clean data sets, right? And this is kind of the problem that exists in the infinite desire to get autonomous driving is, is classifying all the data and being able to react to these different things appropriately and just constant edge cases that you're trying to stamp out over and over again. With, with large language models, in terms of the actual technological shift that happened.

• 12:36Logan Bartlett?

  So now you're able to take all these different data sets and just, just point it at the infrastructure and let it figure out what a dog actually is, or let it figure out that this has some semblance of ancient Rome. How does that actually work?

• 12:53Speaker 01

  So there's a couple of things that we do have to remember about these things. One is that they're not, no matter how smart they seem, they're not actually humans. They're not actually thinking like humans. They're not actually doing those things. We have to, there, there are elements along the way that we as humans have to reinforce. And actually reinforcement learning is an element here. There's whole companies around labeling, which you just mentioned, this is a dog, this is a car. But one of the breakthroughs too, that Jacob already alluded to for OpenAI, was they, they took their GPT-3 model and they put reinforcement learning on it. And they basically said, hey, does this output make sense? Does this output make sense? So there is a chain, there's a sequence of events that happen.

• 13:35Speaker 01

  So the, the, the transformation on the, let's call it like this process of training these models. The transformation on the front side of this was just throw raw data at this and the model will chunk away. We've got enough of all these super large computers in the world. They're going to process this data. They're just going to chunk it and they're going to start throwing it in there and it's going to, these models are going to do these things. These are, it's super technical what they're doing. So let's not get into that. It's just they're doing their thing. On the backside, humans are reinforcing it, saying, yes, this makes sense to a human. No, this does not make sense to a human. And they take various cases through this and they kind of go back and forth across that. Now you tune those enough, you start to get stuff that's coherent.

• 14:16Speaker 01

  And the model itself then starts to say, well, this new generation that I'm trying to create for somebody, actually similar to the old generation, which a human or somebody reinforced by saying this is correct. Therefore, I'm going to assume that what I'm about to produce will be acceptable to that person.

• 14:33Logan Bartlett?

  So I understand the front end now, you don't need the structured data to make sense of all of this stuff. But on the back end, did OpenAI need to start with humans in the loop from a reinforcement learning standpoint, saying, actually, no, that looks like New York City and not ancient Rome, or that is a cat, not a dog. And now it's gotten to an escape velocity that it's, you're able to do that technologically without a human actually in the loop. And so it's almost learning on its own.

• 15:02Speaker 01

  So this is where theory and practice come into play. So theoretically, it would be possible to do no humans in the loop. It would just take a long time, lots of compute and like infinite resources. Practically speaking, humans have been in the loop. This is where the reinforcement comes in, reinforcement learning comes in. And there are humans who are saying, yes, this is acceptable to us, or this is interesting, or this is real, this is not. Eventually, what we might see is these systems learning from themselves over and over again. And that's what's interesting about OpenAI at the moment, because it has such strong usage with ChatGPT, is in fact, they can start to see which responses are accepted and send those back in to reinforce. Now, they can't do that everywhere. So that's why it's been publicly, publicly talked about that OpenAI is going to be hiring tons of

• 15:47Speaker 01

  X, Y, Z's to look at these models, lawyers, or, you know, programmers, or even graphic artists to reinforce these models. And they've been open about doing this in the past in other areas. So they still need humans in the loop for speed reasons.

• 16:02Logan Bartlett?

  But they now have users actually like helping them as well in the process, right? And so the analogy I would use is Google search, and they they have people trying the search again, essentially, or rejecting the search. And so it's learning on itself across all these different use cases, right? Be it lawyers, or customer service ticketing, or competitive intelligence, or all this, right? So it's starting to compound on itself because of the usage. Jacob, is that fair?

• 16:30Speaker 02

  Yeah, no. And I think, you know, to answer your original question, like, you know, one of the problems, when these models just came out of the box, and you had GPT-3, and you'd ask a question like, you know, did the Knicks win a championship? And you're predicting the next word. Sometimes the model would just keep the question going, right? Like, you know, it would say, oh, and, you know, was Allan Houston on the team? Like, it would, and that's obviously not the answer to your question. And so, you know, there was kind of a sequence of human feedback that OpenAI had to introduce, right? The first thing they did was literally have people answer the questions, you know, which much to your chagrin, is unfortunately a no for the last few decades. Then, you know, the the second thing they did was they said, okay, let's have the model generate five potential answers, and we'll have a human rank those answers. And then what they were able to

• 17:14Speaker 02

  do from that is basically train a reward model. So if you think about like, the old applications of AI, like chess, that's a really easy thing to have a reward model around. You either win the game, or you don't. So it's an easy way to train a model from scratch. But what they had to basically do for ChatGPT was create a similar reward model, so the model knew, you know, when is this a good output and when is it not? And so you basically upfront are doing a fair amount of that training. And you can imagine that as people build legal models or as they build medical models, they'll have to do a similar amount of training. What's a good answer look like? Okay, now give me a few answers. I'll tell you which one's good and then we'll give you your your legal reward model. But certainly it's an iterative process. So as you get more and more feedback of was that a good answer? Was that not a good answer?

• 17:57Speaker 02

  You can kind of use that as input to to improve things. And even you can see in the chat GPT interface, right? Even just a simple thumbs up or thumbs down is really helpful in feeding back information into these models.

• 18:07Logan Bartlett?

  So the two types of models that have been most mainstream and and we've seen talked about the most are large language models and image generation models. And stable diffusion was originally or stability AI was originally around the image generation side. I think they're working on their their chat GPT equivalent. But can you guys talk a little bit, Jacob, maybe? Just give us a lay of the land in terms of who the players are and what the use cases are across the board from the landscape standpoint?

• 18:42Speaker 02

  Yeah. And and so I think, you know, the big distinction in between a lot of the players today is kind of this distinction of closed source models and open source models. And what that basically means is in the closed source world, like in OpenAI or companies that folks might have heard of like Anthropic or Inflection, Character, Adept. Basically, you send your data over to those companies. You send your input. You say, here's what I want the model to do. And that company hosts the model and they give you the output. They say, okay, here's the the result from the model, which is very different than the open source world, which stability is in. And this company, Hugging Face, which actually is the first guest on our podcast. And in that world, you as a company, as Microsoft or Bank of America or the Mayo Clinic.

• 19:25Speaker 02

  You can actually download the underlying model yourself. So you have the weights, you know exactly how it works and you run it yourself. And so basically, we're seeing this kind of split in the ecosystem today of, you know, a bunch of folks building their own kind of closed source companies. And a lot of these folks, you know, spun out of the original three, OpenAI, DeepMind and Google Brain, or they were authors on that Transformers paper that we talked about earlier. So kind of this, this small community of about 100 folks that really know how to train these large models. They're, they're kind of starting on the closed source side. And then separately, you're seeing folks in the open source world, a ton of different people contributing models that are, you know, pretty good on the large language model and image generation side. So in terms of use cases, obviously, folks

• 20:08Speaker 02

  are familiar with ChatGPT and probably Midjourney. But what's really been interesting is, you know, folks starting to incorporate this into their products, right? So you have design software like Canva and kind of being able to dynamically generate both text and images, you know, in that design software from using these tools.

• 20:25Logan Bartlett?

  What's an example of that, by the way? Like, like in a very literal case, Logan's making a, a design within Canva. How do I, how do I use image generation or large language models?

• 20:36Speaker 02

  Yeah. So you might like, you know, you might be creating a flyer to promote, you know, your podcast. And it's kind of similar functionality to what you might see in Google Docs, right? They kind of suggest like, okay, based on what you're doing. So let's say you're creating a blog post teaser or social media post, you know, Canva itself will kind of suggest, hey, you know, given the context we have around what you're doing, here's text that would be particularly, you know, useful or helpful for you to promote that.

• 21:03Speaker 01

  We saw a teaser just recently too, of something which might be pretty visceral for people listening to this too, which is in Figma, that we saw a teaser for something that says, create a settings page. Now can create a new landing page for X. And it just, it just auto fills all of these things out for you.

• 21:22Logan Bartlett?

  And Figma is a collaborative design software bought by Adobe for $20 billion, right? And so in this case, You're designing the, you're, you're using, uh, what is that? Large language models or image generation? I guess it's image generation.

• 21:37Speaker 01

  I don't actually know what to use. They're, they're, you're probably using some image generation and combination of image generation, large language models behind the scenes to parse this and do it. I don't actually know the, the config.

• 21:47Logan Bartlett?

  And, and so it's saying, hey, let's bu- build a login page, right? If I wanted to start a, uh, a, a community in which all podcast listeners could talk to one another and you had to log into the website to get there, I could say, hey, build my landing page to sign in through whatever LinkedIn or Twitter so that you can now get access to the community forum. And I would just type in build landing page for authentication on website, right? And it would spit back at least a template that then I could, I could work with and build around.

• 22:20Logan Bartlett?

  Jason, a question, I guess, having been at the forefront of a bunch of different open source movements, both GitHub as well as, uh, as well as canonical. Why does this, the distinction Jacob kind of drew there was closed source versus open source? And I, I feel like there's, there's a two by two matrix we could kind of create of image generation, large language models, and then closed sourced, open source that kind of allows, uh, people to think about what products exist where, right? And OpenAI is, uh, or ChatGPT is closed source, large language model, right? Versus, uh, Stable Diffusion is open source image generation. But why, why does this matter?

• 23:04Logan Bartlett?

  Like, who, who, who gives a shit about whether like the average user, why do they care about open source versus closed source? And why is this like a, a fundamental debate in the community right now?

• 23:16Speaker 01

  So interestingly, in, uh, traditional open source, just whether it was software, it was more ethical and concerns and, and, uh, learning concerns. So, you know, the original movement was about, hey, code should be open. We should allow people to inspect it. Um, it makes for better security, all that sort of stuff.

• 23:35Logan Bartlett?

  And is this original, are you, are you talking like 1990s Linux?

• 23:38Speaker 01

  I'm talking 1990s.

• 23:39Logan Bartlett?

  Okay.

• 23:40Speaker 01

  Yeah. And, and no, no user actually really cares. Like, there's a very small subset of people who actually gave a shit if anything was open source versus closed source. They cared that it worked. They cared that it was useful and all that sort of stuff. Interestingly, here.

• 23:55Speaker 01

  The general public might actually care a little bit more about open versus closed. Because it has to do with their own data this time. So whether it's their, you know, whatever data that they're using, it, they might be sending to somebody who is going to amalgamate it. Now, I still believe that that's not really going to be a thing. Users are just going to care that something's useful. But ultimately, what the distinction is going to be is, again, you know, how these are trained, how it's making predictions, how it's, how it's using that data, and then who has access to it. And again, you know, we're projecting into the future with this next statement here, but it's when you talk about who has access to these, these models and who gets to use them.

• 24:36Speaker 01

  The world's going to look really weird in unpredictable ways in about five years for folks. And I think that open and closed are going to matter a lot more in this case here. There's going to be a lot more people who want to use open models. Weirdly, take them open models and close them and use it and train them on their private data. And then they have exclusive access to that.

• 24:56Logan Bartlett?

  And so back in the day, like when, when, when you were talking about Linux, right? And or Red Hat or whatever it was. We were talking about running servers and stuff within banks, for example. And at the end of the day, as an end customer, I don't really give a shit about what's powering the bank's infrastructure. Uh, and if, if that compute, the, the, uh, bank itself had access to the source code. Like, I don't really care as long as the website's up or the app's working. Like, who cares what, what that actually looks like? And so, sure, there were philosophical debates that I know you've, you, you had to have, uh, quite a bit of over the last decade

• 25:37Logan Bartlett?

  about where, uh, you know, the, the, the, the power and lock-in of open source versus closed source and all of that. But now today, this is actually much more applicable because we're seeing manifestations of how actually data is, is being learned upon and what the data sets are. And so an example that I think people maybe got exposure to, but there, someone posted that you could write a, A sonnet or a poem or something about Joe Biden, but you couldn't about Donald Trump, right? That was a real example that that and OpenAI didn't exactly know why that was

• 26:16Logan Bartlett?

  the case. And so instead of having huge numbers of people able to look at the source code and figure out what the what data it had learned on and why that that existed, OpenAI basically had to go do the investigation to figure out what data set is it learning on? Why is that prevented? And so there are these use cases of all these things that you're learning from that might impact what the actual output is. Is that a fair characterization?

• 26:46Speaker 01

  It's fair. And there's these are the most complex of complex systems, unfortunately, too. Fortunately, they're going to do amazing things. But unfortunately, it's no answer is ever going to be simple in them. But I'm of two minds here, given having lived through open source for so long and having so many scars from it. One is, I hope people care about their data. But I think we've proven, you know, society generally has a little bit of care, but not a lot of care about their data.

• 27:11Logan Bartlett?

  I think people philosophically care about data, but when it comes push come to shove, I saw an interesting article from the New York Times that was like, why do ads suck now? Or something? It's like, oh, why do ads suck now? I have I have an idea. I have one idea. You remember that app tracking transparency thing that everyone was freaking out about data privacy? Like, now that's why all ads suck is because we can't collect the same level of data and all of that. So it's interesting, like, how people divorce the concept. Of course, I care about my data. Like, I own my data. It's about me. And then it's like, well, why am I getting these shitty ads? It's, you know, reconciling those two things in your head are kind of hard to apply. But

• 27:48Speaker 01

  It is. And I think for the majority case of all people, one, they're not going to understand what's actually happening. And it's not a statement on intelligence or anything like that. What I'm really saying is like, it's just, it's a system that they don't really think about on a regular basis. They see it as a as an interaction with an X entity. No big deal. The other is ultimately, when push comes to shove, again, users, I think, care about experience. They care about UI. They care about, does this thing work? As you mentioned about the the bank, I think that will continue to be true in the future. Now, however, I can foresee a very specific case where this is going to matter, when these language models actually start Make, I don't want to say making decisions because that's not what they're effectively doing. But when they're choosing a path for something to go forward, we're going to have to understand how it's choosing those things. We're going to have to be able to inspect those.

• 28:30Speaker 01

  There's a transparency element here, which I suspect that some entity somewhere is going to demand of these things. And then possibly us are going to ask.

• 28:40Logan Bartlett?

  It's honestly, we're taking all the issues we see with Twitter and shadow banning and like distribution and all of this stuff, right? What is hate speech? And it's going to be exponential, like the problems that exist coming out of this comparatively.

• 28:58Speaker 02

  Totally. I think it's going to be like a next level philosophical debate, you know, relative to anything around open and closed source in the past, right? Because on the one hand, you've got folks at OpenAI saying, these models are too dangerous to just let them out into the wild and let anybody use them. And then you've got people like Ahmad, who I think have really compellingly said, what? So you don't think people in Africa and India and other folks who otherwise wouldn't get access to these models, you don't think they should be involved in decision making around how we use these? But you know, these models, right now, we talked about these reward models, right? And how basically these models were trained to predict the next word in sentences, right? And that's led to responses. But it's, you know, you can imagine with these open source models, people could train that reward model to do anything.

• 29:43Speaker 02

  You could train that reward model to hack into systems, to do all sorts of more nefarious things. And so I think as a society, this is going to be like a huge debate over the next few years.

• 29:54Speaker 01

  My view is that we're, it's a false economy between open and closed sorts. It's going to be about decision transparency and things of that nature. Because you do imagine, my wife is a physician. Thank God she's never had to go before a panel, but doctors have to stand before panels and talk about some of their decisions in the worst possible cases of what happened? Walk us through this. Could you imagine a system where we've got many models involved that are actually sorting out patient?

• 30:24Speaker 02

  Patient care. What what are we going to demand a society from these models and the providers of these models? To talk about in those cases where something goes incredibly wrong. How are we going to have that conversation? And it's not a simple open source, close source conversation. It's it's a it's a lot bigger and we're just frankly, I don't know if we're prepared to have that conversation just yet.

• 30:45Logan Bartlett?

  So as we go through the landscape, and I want to get into the slippery slope element of it, and then the utopian element of this, because I think there's there's, you know, there's definitely elements of both here, like like almost any technology, we make the world better and also worse at the same time, right? So I want to talk about both both of those, both of those elements of this. But Jacob, can you give us a little bit of a lay of the land of you rattled off a bunch of names and and people there? Can you can you talk through the different companies that are at the foundational level? And then maybe we talk about some of the applications that are being built up from there?

• 31:25Speaker 02

  Yeah. And so basically, you know, a lot of the headlines you're seeing around these big fundraising rounds right now are, you know, companies that are building, you know, their own version of these large language models or image generation models. Increasingly, folks focused on video and and taking action in software and robotics. And so, you know, basically, as I said earlier, it's kind of the small community of about a hundred, 200 folks, many of whom were the authors of the original architectures or worked on GPT-3. And it feels like every day we see a different pair of these teams, you know, spinning out and starting a new company. And so there's probably, you know, five, six really prominent ones right now. OpenAI, DeepMind and Google were obviously the original players.

• 32:08Speaker 02

  Then you had kind of a lot of the folks that worked on GPT-3 spin out and start a company called Anthropic, which is which has raised a bunch of money and has a model called Claude, which is is pretty, if you've ever played around with it, it's probably the most comparable thing to to chat GPT out there. But there's other companies like Cohere, AI21, and then there's kind of like fun variations on these things. So there's a company called Character. If folks haven't checked out, it is one of these foundational. Nation model companies, but the way you actually interact with it is you chat with like Bowser or Mario or Elon Musk and that's their kind of like consumer application of just gathering a bunch of data of how people want to interact with these with these large language models. And so you're seeing different flavors of these spinning out, but we're seeing a bunch.

• 32:53Speaker 02

  And I think, you know, the reason that we're seeing that is I think there's this feeling in the ecosystem that, look, you know, these, these basic models are going to form the basis of so many applications going forward and they're going to have advantages that really just only compound from now, right? I think folks think, look, there's going to be a limited set of talent that's going to be able to come up with a set of innovations that furthers these models. You want to be backing some of that talent. It's going to take tremendous amounts of compute. How many companies are going to be able to raise the round sizes? We've been talking about, and it's only going to get further as these models get bigger. And then as we've been talking about throughout this episode, there's going to be a lot of data that comes back from having these models in the wild that's going to be really important

• 33:36Speaker 02

  in making these models better. And so I think there's this feeling now among folks that, hey, if you want to have an investment in the foundation model space, you better make it soon because these advantages are only going to continue to compound. And then in terms of the application side, you know, I think that's still a little bit more nascent, right? If you think about, I mean, everyone loves to throw analogies to mobile and cloud and all these other transformations, but you know, the reality was a lot of the, you know, the most prominent kind of technologies enabled by mobile, think like a DoorDash or an Uber, you know, it didn't happen until five, six years after the advent of the iPhone, right? And so I think you're seeing a lot of folks play around with applications. We've seen, you know, obviously I think the marketing copy applications, the code generation applications

• 34:20Speaker 02

  are really some of the first most prominent ones. But going forward, I think, you know, there's, it's hard to imagine a vertical that's not going to be touched by, by these products. And, and maybe that looks like existing incumbents just incorporating this into their existing tools, or maybe it looks like brand new startups. And I think, you know, certainly it's not hard to imagine something on the consumer side emerging and leveraging a lot of these technologies. You've seen things like Lensa really explode in popularity and that's just, you know, generating a few profile pictures.

• 34:48Speaker 02

  And on the enterprise side, it's been just a ton of folks experimenting. I feel like literally every time we talk with a product manager or a CEO of a company, they have some team of five, six folks messing around with these models and seeing how they can put them in the product.

• 35:04Logan Bartlett?

  Jacob, you touched on a bunch of different names there. But so OpenAI, I think people are familiar with. I think their last round, whatever that is, I mean, it started as a nonprofit and then it's evolved into like a for-profit capped earning thing. But whatever that is, it's the $29 billion, I think, was their last valuation. We talked about Google DeepMind. So that was an acquisition by Google in 2014, I think for, I don't know, $500 billion, something like that as well. You talked about Anthropic. That's a group that spun out of OpenAI and they're building their own model. Touched on a few others, but these are all the foundational players that are kind of going about building these different elements.

• 35:44Logan Bartlett?

  Obviously, we stable diffusion and stability AI are building these fundamental large language models or the image generation side. So these are going to be kind of the underpinning infrastructure that empowers. This is the analogy of iPhone or Android, right, to some extent. And then from there, there's going to be the applications and use cases built on top. And you touched on a few of those. But Jason, as one of the first architects or people that actually brought this to commercialization in some way with GitHub Copilot, what are some of the things that have gotten you excited from a use case standpoint that you've seen out there?

• 36:28Logan Bartlett?

  Maybe you referenced your wife as a physician, something that you've seen in those areas that people should be excited about?

• 36:35Speaker 01

  Well, I think these.

• 36:38Speaker 01

  This technology has obviously a broad reaching application in the future. So we could pontificate about all the different ways, but medicine obviously is a place where I think this could help. Just, you know, even diagnoses or understanding MRIs and all that sort of stuff. We've already seen some of that emerge, but it'll only get better and better and better.

• 36:56Logan Bartlett?

  And what's an example, like in terms of a real use case, how would that actually work? Maybe Jacob, since you, since you spend a lot of your time in healthcare, like what is an actual use case within healthcare in that regard that gets you excited?

• 37:08Speaker 02

  So I mean, if you think about the dream, it would be, you know, having, you know, basically a triage bot that could, you know, you've got a complaint, you're worried like, hey, am I, you know, got a little bit of a fever, you know, these symptoms, is this something I should be worried about? And, you know, you could basically chat with a doctor bot that could let you know, you know, hey, is this something that needs to be escalated or not? You know, long term, maybe that could even just be your doctor. I think the reality is it's going to take a long time as a society for us to get comfortable with that. And actually, the near term use cases are probably much more around, you know, making spaces that maybe traditionally we didn't think quite worked. They might actually work with these new models. So in healthcare, a great example is

• 37:52Speaker 02

  you've seen a lot of companies do, you know, want to transcribe notes from doctor visits, right? If you think about any doctor you know, they absolutely hate that like every time they see a patient, they have to then spend 15 minutes in Epic, like writing a long note to about the visit. And, you know, there hasn't quite been good enough technology to automate the notes from those visits. But I think with the advent of models like Whisper from OpenAI and a few others, you really could actually in the very near term, start to see products here that just automatically generate notes in a doctor's style, maybe even generate the billing codes from that or the next steps. And it's kind of, if you think about the sequencing that we'll see within medicine or law, it's going to start with, you know, slightly lower stakes tasks.

• 38:35Speaker 02

  Obviously, writing a note about a patient is very different than me giving, you know, hey, don't worry about your fever, you're fine. And I think that's probably the sequencing we'll see on the use case side. But if you dream the dream, I mean, why not have kind of the, you know, encode what the top doctors do into some of these models? And then you're basically just massively expanding access to the Cleveland Clinic doctor or the Mayo Clinic doctor by, you know, providing access to these models.

• 39:00Logan Bartlett?

  Yeah, I'm thinking about that use case of if if I'm a doctor and I'm talking into whatever my product, reading the the notes are from the meeting, I could just talk in real time stream of consciousness and then say summarize this and it could be distilled down into the bullet point version of it, which gets pinned to the the electronic health record. And so at the end of the day, you would still have the raw notes. If if there was a mistake made, you would be able to reference the recording or whatever it is, but you'd at least have a summary. So people wouldn't have to go through the the full, the totality of all the notes to make sense of it.

• 39:38Speaker 02

  And if you want to imagine it a little further, I mean, one thing these models are very good at is kind of reframing texts in ways that are like educationally appropriate for different audiences. So imagine you have that visit note and then you're referring that patient to another doctor. So you take that visit note and put it in the context that that doctor would understand. But then also you want to communicate back to the patient that, you know, maybe isn't super sophisticated. Maybe you want to communicate to that patient's kid or their parent. And you basically can take that core information and reframe it in a bunch of different ways automatically for whoever your end audiences.

• 40:13Speaker 01

  One of the things I think that we're going to see really early, and so what we call these things are generative models at the moment, a lot of these, the image and the chat GPTs are generative models. I think we're going to see a lot more of this kind of like bullshit boilerplate stuff removed from the systems. And so I think about legal documents. Now, you're never going to replace lawyers, but you could replace teams of people who have to like come up with these hundred page docs. You could just come up with them rather easily and then have a subset of people who say, yes, we want this clause. No, that clause is wrong. We kind of want to change it this way. And that's if you effectively think about what Copilot is, it's a that's what we talked about. It is. It's an Iron Man suit for the programmer. We were trying to take away some of the ceremonial elements of programming, which are really annoying, like Java as a language is incredibly verbose.

• 41:03Speaker 01

  It takes you a while just to type out the characters. We're trying to get people out of typing out the characters essentially.

• 41:11Logan Bartlett?

  It's interesting, like, for a long time, we thought of AI that it was going to be blue collar, hourly jobs that were going to get automated. But actually, what we're seeing here is, is it's not the highest level of creativity or expertise, but it's, it's the, maybe the people that work for elements of the highest level of creativity or expertise, right? And so if you're a Cleveland clinic, or a Mayo clinic, or a hospital for special surgery doctor, your job's not going away anytime soon. But the, the people that work for you to write your notes, or summarize them, or those things, like, these are kind of the, the low stakes jobs that we're kind of seeing these

• 41:55Logan Bartlett?

  applications brought to today and likely in the near future. Is that fair?

• 41:59Speaker 02

  Yeah, I mean, what I love about the space is, you know, as these models get better, it's really hard to predict what they will and won't be able to do well, right? I think before GPT-3 was trained, I'm not sure it was obvious to folks, like, oh, they're going to be great, you know, on the code generation side, and maybe not so good on, on some of the blue collar stuff. I kind of liken it to, you know, the particle accelerator, right? That they had at CERN where they were doing, you know, new experiments, and no one had any idea, no one had ever put that much energy into throwing atoms at each other. Who knows what's going to happen? It's like, people are going to throw billions of dollars of compute at these models in the next year or two, and we can kind of extrapolate what might happen. But the reality is, there's going to be all sorts of things that come out of those models in, in a few years that we're going to be surprised by.

• 42:42Speaker 02

  And people are still, even with GPT-3, are still finding new things these models do, right? Like, even the OpenAI folks don't know everything these models can and can't do. And so I think it's funny, you have, obviously, the point in time of, like, taking a look at what is, you know, a Claude or GPT-3 really good at, and what are they not so good at? But the pace at which it moves is just so fascinating, that I think, you know, every year or two, we'll have to reevaluate.

• 43:03Logan Bartlett?

  There's no use case coming for podcasting yet, is there?

• 43:06Speaker 02

  You know, not, not yet, but we'll, we'll see.

• 43:09Logan Bartlett?

  It's like a defensible business to be in.

• 43:11Speaker 02

  When it was a cartoon avatars, I think you were in trouble.

• 43:14Logan Bartlett?

  Yeah, that's right, that's right. I was, I was much more exposed on that flank.

• 43:17Speaker 02

  I think what's interesting about what could possibly

• 43:20Speaker 01

  Go away in the future is it's never going to be, it doesn't feel like at the moment is going to be the creativity or the deep, deep expertise. So if you're an architect or a doctor or a lawyer who is a domain expert and you say, no, that we're going to do these. Because you have to verify a bunch of these things. We're not going to trust these models to make the ultimate decision or the ultimate output for a long time. In high stakes endeavors. It's just not going to be the way that we're, the world's going to go for a little while. But there are a lot of elements of all of these industries, which are basically middle of that, like they're, they're, they're not. You know, blue collar, but they're effectively blue, blue collar versions of white collar jobs. Those were, I think you're alluding to, and I do think that those are going to, those folks are going to have to adapt and those jobs are going to be the first that probably.

• 44:06Speaker 01

  Get taken up by these models.

• 44:07Logan Bartlett?

  Yeah, they're, they're, they're commoditized in some way that they could be replaced by another person, right? It's sort of like how the likelihood that your job could have been outsourced 25 years ago or something like that. Now those jobs are the ones that in the nearish term are going to face exposure. But now we've talked about the, I, I, I think whenever we talk about job loss, there's obviously the, the negative implications of, of that. From a societal standpoint as well, like if large swaths of, of legal clerks get automated out of their job.

• 44:46Logan Bartlett?

  Or whatever the case may be, there's going to be negative externalities that come along with that social unrest and need for retraining or finding other jobs. And I, I think we believe that generally, I mean, over the course of human history, as these things, as more productivity gains happen. People are able to repurpose their skills and find new opportunities. And ultimately this should increase the productivity and output of, of society. Now that's a very abstract zoomed out consideration. At the near term.

• 45:24Logan Bartlett?

  We need to be empathetic that people are losing jobs and there needs to be elements of, of social safety nets or whatever there are that are associated with this. And we're also seeing this happen in such a, like this isn't generational changes that is coming to the industrial revolution. This is happening. In six months, a year, three years from now. So as you guys think about it, I'm not going to ask you to totally pontificate on some of the political implications from this. But Jason, maybe as you think about the negative elements that that might occur, both with the models themselves, as well as lower elements of the workforce being commoditized

• 46:09Logan Bartlett?

  away. How did you think about that with GitHub Copilot and the implications associated with that? Did you guys receive pushback from lower level engineers saying, hey, you know, we're going to lose jobs? Like, what was that that you guys saw at Microsoft GitHub at the time?

• 46:26Speaker 01

  At the time, no, because people kind of were looking to the future. But I think with with some distance in the beta, and even before we released it to the public, people were worried about us trying to replace programmers. And our intent, again, was never to replace the humans. It was to augment them. But, you know, in fullness of time, everybody's job likely does get automated to some degree. And, you know, there's got to be some recognition that even for programmers or for like, investors, a lot of our jobs could change over the next couple of years, and we would have to adapt. Now, I think your point, and this is where I hang out a lot, and, you know, when I'm trying to think about what's going to happen here, is the pace at which this is going to happen

• 47:09Speaker 01

  is going to be mind boggling. And we've already seen that. And from a societal perspective, how good are we at adapting to rapid change, especially rapid disruptive change? And, you know, we're not ready, we're not really ready for that, in my opinion. Like, we're not ready for, from a tech perspective, us in tech, who are the ones working on this, aren't ready for it, let alone other places where they're not, they don't even work at our pace. And I have a good friend, who's one of the original authors of that attention paper, Ilya, who's one of the co-founders of Near Crypto, which was originally Near AI. And I was asking him his opinion on this recently. He said, the only thing I'd be working on in AI at the moment, is I'd be working on how to, how to put safety in place

• 47:52Speaker 01

  for here. Because we're about to see an explosion of stuff that's going to happen. And we have to understand provenance, we have to understand safety, we have to understand, basically, garbage content that's going to be created. We have to understand, like, genuine human connection, and what's at the root of it. And we're just, we're not ready for it. We're not ready for a machine that's going to try to hack the US, the IRS. We're not ready for any of those things.

• 48:14Logan Bartlett?

  Regardless of where you are on the political spectrum or aisle, I'm not sure anyone would dispute the fact that it might not be the most functional body that moves at the quickest pace with any authority or expertise in a lot of the areas that they operate in. I feel like that's probably a pretty safe statement to make. Is this going to fall on individual countries to figure out what their policies are around this? How much of this? You mentioned earlier, Jason, some of the people within OpenAI that actually themselves are nervous

• 48:56Logan Bartlett?

  about the power of some of these things. And so is this ultimately on the companies themselves? Is this on the government? Is this on the end users? Jacob, how do you think about who exists within this and who we need leadership from to help make these very thorny political, socially impactful decisions?

• 49:20Speaker 02

  Well, first, let me say I'm honored to be participating in the timeless tradition of VCs opining on political issues that maybe they shouldn't be. So I'd say the reality is it's moving so fast that I think it's completely unclear what government bodies are going to be involved in regulating this, right? There's, I think, a lot of the government is really much more focused on the national security aspects of the race, the US versus China, and who's going to have both the hardware and then actual model capabilities in a military or cyber or information conflict. And so I think one way this could play out is actually,

• 50:03Speaker 02

  it's almost reminiscent of the crypto world, where I think for a while it felt like it wasn't exactly clear that regulators had any idea what's going on. And then you have something happen like FTX or something that really gets people's attention, that really impacts consumers, and that's really the catalyst to have a lot more on the regulatory side. But I'd say, in the conversations we've had, it's pretty nascent in how exactly this stuff's going to be regulated at a national or international level.

• 50:28Speaker 01

  Only thing to add in this would be that this is going to be interesting from a bunch of different perspectives, but consider this. Consider whether or not OpenAI's model is available, who is available to. Because they're effectively time slicing compute. Who gets to, who gets to ask it questions in the future? You know, are they going to allow the same thing as, you know, Iran can't ask it questions, all that sort of stuff. But what happens when Russia or China or some other entity has their own model? Well, how, what are we going to do in the world? All that sort of stuff. The, the, the simple answer here is that these are probably the most complex things that we have in tech at the moment.

• 51:08Speaker 01

  And there's, I don't, I don't really trust the ability for many people to really understand them and their implications. Not that they can't, but there's, there's some nuance here. So I do believe it has to be a diverse group of people, a lot of tech folks around the table, advising policy and lawmakers and all that sort of stuff. And I do believe the ecosystem around this has to come up with some safety measures. And I do believe that we have to push that.

• 51:30Logan Bartlett?

  Jacob, you made the analogy to crypto here and sort of the way FTX evolved, or FTX became kind of a watershed moment for people actually increasingly saying, we have to fix something here. And I feel like, while, while there's examples of things like Axie Infinity and people losing their life savings around that and FTX and people losing money, largely institutional people, but definitely some retail as well, losing, losing money. There's definitely sad situations that occurred from the crypto fraud and all of the, the stuff that occurred from even hacking and stolen Bitcoin and all of that.

• 52:13Logan Bartlett?

  But at the end of the day, it was losing money, right? That, that was sort of like the elements of what occurred. And I'm sure there's, there's anecdotes about people being so desolate with what happened that, that there was loss of life as well, right? But for the most part, it was financially oriented. Here, we're talking much more about who knows where all the implications exist. And, and we're talking about much more psychological elements of, of models. And we've seen the studies from Facebook and, and Instagram and about what little tweaks to their algorithms actually does to human psychology and the number of people you can impact around that stuff. And so as, as you think about the, the comparison there,

• 52:56Logan Bartlett?

  do you, do you agree with Jason's assertion that this is going to take? Tech people kind of being proactive and working and thinking through this stuff? Or does the government need to catch up and make sense of this? Or are we basically anointing more kings to make these broad sweeping decisions for us? Much in the same way we have over the last couple of years with Mark Zuckerberg and the folks over at Google and potentially Microsoft and Amazon and all of that?

• 53:29Speaker 02

  Yeah, I mean, I think philosophically, you know, I think I probably fall more in the Ahmad camp of, you know, this is something that, you know, because it's so transformational and a profound, you know, change is something that it does need to be regulated outside of these tech companies. And I think, you know, in some senses, that aspect is also similar to crypto, where I think, you know, the opening eyes of the world would be pretty happy with like sensible regulation and they're asking for, you know, sensible regulation around how these things are. I don't think, you know, Sam Altman wants to be the one deciding, the only person deciding how this is and isn't used. And so I think, you know, there's probably going to be a collaboration between, you know, the folks at the cutting edge of these models and the governments that are actually regulating them.

• 54:12Logan Bartlett?

  In terms of the comparison to crypto as well, if we take this one extension further, we've seen quite a, we saw quite a run up of crypto over the course of the last two years. And, and I think both of you at different points were at least curious about what was going on in the ecosystem where I was maybe a little bit more of the internal negative Nancy about this, some of those things.

• 54:37Logan Bartlett?

  Where are the comparisons that you guys think? People have definitely been drawing extrapolations saying, well, this is just another hype cycle similar to crypto. And obviously I've even made jokes here on this episode about crypto VCs pivoting to AI. Where is there truth in that? And maybe where does that fall short as a comparison in your mind, Jason?

• 55:01Speaker 01

  So I think there's obviously truth to the fact that investors are pivoting from crypto to this all the time. See entire firms change the the the brandage from crypto to AI. So that's the thing. But I actually think that there is some similarity. If you look at this at crypto and AI, a lot of the same people who are interested in one are interested in the other. And I think that there's, you know, kind of like space age sci fi aspects of that. And there's algorithms behind the scenes and there's a lot of math and all that sort of stuff. And they're kind of like generally, you know, PhD ish, kind of really mathematical, really kind of nerdy, proofish type of people. And they both attract that sort of thing. Crypto, because of the nature of it being effectively money, also attracted the biggest scammers in the world.

• 55:46Speaker 01

  And that I think is a major difference. Now, we're seeing that same sort of thing in AI at a different level, which is like, if you go to LinkedIn right now or whatever, you can't get away from the and chat GPT will make you this amount of money by blah, blah, blah. So the scammers on crypto side of the fence, which were the pump and dumpers are trying to use this to create bullshit content and, you know, all that sort of stuff over over here. But it's effectively different in the way they're engaging with them.

• 56:11Logan Bartlett?

  So less less direct grifters that are monetizing in the ecosystem, but equal hype.

• 56:17Speaker 02

  Totally. And I think like two things you see from that is, you know, like in any hype cycle, there, there's just, there's going to be just, you know, if you were a founder and you weren't totally sure what you wanted to build in 2021, you were like, I'm going to go build a crypto company. And I think similarly, like, there's going to be just this wave of like, tons and tons of people building pretty light, you know, gen AI companies. It's like the the dot XYZ to dot AI transition. And then the second thing I think you see in a hype cycle is these like self referential funding rounds where like, nobody really knows how to price these basic like seed stage like deals. But, you know, in the crypto world, you saw, well, if, you know, if Solana was worth this, then, okay, all these other companies might be, you know, worth some fraction of that. And it's all just in comparison to what these other, you know, potentially valid or not

• 57:02Speaker 02

  valid funding. are. And similarly, in the AI world, it's like, who knows how to value these foundation model companies? It's OpenAI was valued at $29 billion. So let's, you know, some fraction of that is what these other things are worth. But I think the big difference is, I mean, you've seen the largest tech companies move pretty quickly into saying, this is where we're betting the farm, you know, whether Microsoft.

• 57:23Speaker 01

  Actual use cases. We see actual, real, non-self-referential, self-referential usage of AI.

• 57:30Speaker 02

  If you had Zach on again, I think you'd get a much better response to some of the use case questions.

• 57:34Logan Bartlett?

  No, I, I, I agree with that. And that's why I am, uh, I'm more interested in sort of diving into the elements of, of this than I ever was with, uh, with crypto. And you're right. I mean, the reaction that you're seeing from Microsoft and, and Google in this regard is just, it's moving at light speed compared to, uh, I, I don't even know with crypto. I guess Facebook sort of had a crypto effort, but that was mostly it.

• 58:02Speaker 01

  Yeah. Facebook and Twitter is and whatever's of the world with the entities and stuff. But this one is interesting because, because it's images and text in this generation. And let's forget the image models and the whisper models and all that sort of stuff for the moment. But you can get the average person to all of a sudden imagine a world where this is useful. And

• 58:18Logan Bartlett?

  Imagine a world, Zach Weinberg. Uh, yeah, but this, you actually can't imagine the world, right?

• 58:24Speaker 01

  And you could see it and you can see, oh, well, I, it's not as easy to use at the moment, but I could generate like, you know, I, I just told my wife over the weekend, hey, let's get all three of our kids to give us an outline of the story. We're going to go into chat GPT and we're going to go into mid journey. We're going to actually create these stories. Now it's a pain in the butt at the moment to do this because you got to do, but at the end of the day, you actually have a book that we're going to get published for each one of the kids. And it's not hard to go do. But we're still talking about like the first real use case for 99% of all crypto.

• 58:54Logan Bartlett?

  So we, we've covered a lot of ground here, talking about a bunch of different, uh, things and the potential implications from a societal standpoint, from a, uh, you know, some of the use cases that exist here. I, I guess if you're in a, a founder that's existing outside of the AI landscape, right? Or if you're an executive and you're working at a business that exists outside of AI, right? And this isn't directly, uh, relevant to you today. What would you say to those folks and what should they be thinking about to either future-proof themselves from some of the implications that are coming down? Or just exploratory things that could occur to their industry in the next six months,

• 59:39Logan Bartlett?

  five years, 10 years, whatever it is? Jacob, what would you say to those people?

• 59:43Speaker 02

  Yeah, I mean, I think the biggest thing I'd say is just, you know, following kind of what others are building. I think, you know, the pace at which the space moves, I feel like on Twitter on a week-to-week basis, there's, you know, just new things that people are coming up with. Like last week and over the last few weeks, all the rage has been, you know, providing context to these large language models. And how do you kind of get specific responses that you can cite or tie back to your own data? And I think that like, if we're in just the idea generation phase and getting the creative juices flowing, I think just having a sense of what other people are building, because obviously some of that's going to be, you know, somewhat applicable to a business that folks have and some of it won't. But the pace at which the, if you take any snapshot at a moment in time, you'd really

• 1:00:26Speaker 02

  miss out on, I think, just the rapid pace at which these capabilities are improving. And so, you know, I don't think we've seen, you know, it's not like Bank of America has suddenly just transformed, you know, the business and is using all these models at scale. But people are at least, you know, getting the creativity flowing. And I think the reality is, you know, when folks actually decide they want to do this, there's so many well-funded companies that are popping up to help companies actually implement these models, that I think, you know, it's probably less of a daunting task than you might think. I mean, one thing I'm always struck by is Jason's team that started with Copilot was six people. And that's like the most used product in the space. And so I think the actual, the bar to, you know, starting to experiment with this or

• 1:01:09Speaker 02

  to put it in kind of like innocuous parts of the product is actually kind of, is much lower than people might think. And then, you know, if you, if you just look at it now and don't look for a year, you'd just be overwhelmed with kind of the pace at which the use cases and the ways people use this is changing.

• 1:01:25Speaker 01

  I think there's two things I would say to an executive that are looking at this. One is understand if you're trying to take an offensive or a defensive posture in your business. If you're taking a defensive posture, just understand what's possible, exactly what Jacob said here. Just understand the space, look for elements where to implement it. If you're taking an offensive posture, you really need to have a team. That's building some of these things, experimenting with it and really pushing internally the whole culture around, hey, this is going to be possible here to go do. And as Jacob mentioned, you don't need a lot of people if you're not building your own models. And if your business is going to be built around data in the future, you probably have to understand the models too. Because I think in the future, if you're a heavy data business, that's going to be your moat. You're going to have a model at some point or multiple models and that's going to be your moat. So you have to start to understand what that's going to look like.

• 1:02:08Logan Bartlett?

  Do you think every company of some scale, maybe as defined by 10 people plus, 15 people plus, 50 people plus, should have someone internally, if not singularly dedicated to this, at least as the resident expert in this? Or is it still too early for that to be productive?

• 1:02:28Speaker 01

  I don't think it's too early. I think this is one of those where people should have some somebody or some set of people looking at this for sure. And I absolutely think that in the future, all businesses are going to have elements of this. Whether or not they have their own models, but if they're calling out to others who are doing these things, there's going to be elements of this in all businesses. So you might as well get a jumpstart on this at the moment. Now, again, yeah, like probably under 10, I don't know. You've got to, you, post-product market fit, let's call it that way. Post-product market fit, if you're building a consumer or enterprise type company, you've started to have some traction here, you should probably be understanding exactly how this is going to work. But if you're, you know, a large entity, you know, I'm sitting in San Francisco, you're a large entity that does not have a team of people looking at this

• 1:03:11Speaker 01

  and probably some of the brightest people in your company looking at this, you're behind the ball already.

• 1:03:15Logan Bartlett?

  Where would you actually start? Where would you recommend outside of this podcast episode, outside of the unsupervised learning podcast that you guys are starting? Like where would you recommend, hey, I'm a founder listening to this and I'm going to task one person on my team to at least go investigate the implications for our industry and start poking around? Where do you point that person to or where do they start?

• 1:03:36Speaker 02

  Yeah, so a few things, you know, one is, I mean, OpenAI is just a great place to, in a light touch way, experiment with these models. And they've actually, we can link to it in the show notes. They have this thing called a cookbook, which is basically just a bunch of examples of how folks are using these models. And so, as you're just trying to get a lay of the land of what is possible with these models, what can you do? That's a great resource to, to use. And then on the slightly more sophisticated side, you know, there's this whole trend of people kind of chaining models together or taking different inputs from the real world and adding those to models. And, you know, that, there's really a community that's kind of coalesced around developers that are building here, around a few companies, but one, you know, specifically is called Langchain.

• 1:04:18Speaker 02

  And, you know, on their GitHub page, they have a ton of examples too of just how are people using these models? And what are different things you can do with them? And I think that, you know, is probably the right first step, right? Just getting a sense of the different things you can do with these models.

• 1:04:30Speaker 01

  I, I think that the easiest place to go look is break this down into a couple of subcategories. One is the generation models. And so just go to ChatGPT, go to OpenAI, go to DALL-E, go to Midjourney, go to Stability, go to anything that's going to generate something for you and just start playing with it. The other is fundamentally, if you want to understand what's going on under the hood, go to Hugging Face and follow the tutorials, Google Andre Karpathy's What is GPT-3? And we could, again, put those in the show notes here, but just understand what's happening here, OpenAI's. And then if you really want to go, and I don't recommend anyone to go do this, but you know, there's a bunch of papers that you're going to read about heading towards AGI and all that sort of stuff. I don't recommend anyone go down that path until they've done the first two, probably for a year.

• 1:05:13Speaker 01

  But really, understanding those things is going to be that way. But just, just play with the stuff. It's available to us. That's the amazing part is you can literally spend an hour on ChatGPT and all, your mind will be blown.

• 1:05:23Logan Bartlett?

  Jason, having, having implemented one of these, one of the use cases that is most prevalent today and most widely used. Any lessons learned or anything, let's say that those people go down the rabbit hole and end up exploring what you both just referenced? Are there any lessons that you would draw on from the GitHub Copilot experience that you would, or some of the things that you've seen talking to a bunch of the companies building applications in this space?

• 1:05:49Speaker 01

  Yes, and some of those are simple, some of those are nuanced, but the simple are you can get going really quickly. So start experimenting. You can literally just, there are API calls back and forth, so understand what that looks like. And I think you'll be amazed what you can do really quickly. And it'll be fun, you'll be invigorated, that's great. In the nuanced camp, it gets dicey to understand who owns what, you know? So like in GitHub's case, one of the things I've had to agonize over was what is our relationship going to be with OpenAI? And obviously, Microsoft was involved in GitHub had the open source code data. So what were we going to do? How were we going to do it?

• 1:06:31Speaker 01

  And you know, probably an in-depth conversation for another episode, but that was a very interesting, convoluted, very deep, legalese conversation.

• 1:06:40Speaker 02

  No, totally. I mean, I think, you know, as I mentioned, we kind of put together this piece we'll link to, talking to a bunch of companies and how they're building on top of these models. And a few things I thought were particularly interesting is, you know, you talk to the Canva product team, they did a ton of user interviews, and they had a few options, right? They could, you could imagine a world in which you say, you know, here's what I want, and here's the design I want, and the model just generates the design. And what they found actually is, that felt pretty alienating to folks. Like, people wanted to have, they said, I don't want the AI to like, just do the whole thing for me. I want, you know, to work hand in hand with it. I want feedback from the models about what I'm doing, but I don't just want it all done for me. And actually, you know, Ahmad Iqbal, who leads product for Magic, right? He likened to this earlier case study

• 1:07:22Speaker 02

  where, back in the day, I think Betty Crocker had like, released this cake, that all you had to do was add water. And it was almost so easy, and too easy for the end person, that it didn't sell. And then when they added that you had to crack an egg and put that in, suddenly it was like, adopted much more. And so I think that's like, driven a lot of how they've thought about, you know, look, you can't just remove the person entirely from the equation. You know, we talked to the folks at Ada Support, a kind of a support company that works with a bunch of enterprises. And one thing that was interesting there is, you know, they started and they said, well, what if we used AI for your, you know, to talk to your customers? And people were like, whoa, whoa, whoa, we're not ready for that. And so what they found was like, what's kind of the lowest hanging fruit we could use to actually like get these models in practice?

• 1:08:04Speaker 02

  And they said, look, you've got unauthenticated users that you know, no idea who they are. And they're coming to your FAQ page just to learn more. What if we put like an AI powered support bot there? Cause it's going to be better than the status quo. We're not going to replace like your human call center or supports agents. But like, what if we just got you comfortable there? And what's interesting, I think, is like, that's a really interesting way to start getting enterprises comfortable with, oh yeah, these models aren't so bad. In like a much lower stakes environment. And so I figure, and it's funny, there's so many of these lessons because people are just finding all sorts of ways to, uh, to get enterprises comfortable, to get around the limitations of some of these models, the context windows, the prompt engineering you have to do. And I feel like there's just going to be an explosion of learnings that these early adopters are going to start sharing with, with other folks.

• 1:08:47Logan Bartlett?

  Okay. Well, I mean, this has been super helpful, kind of going into all this stuff. And I appreciate both, both of you all coming on. Any, any parting thoughts, Jason, maybe anything that you would leave people with before we wrap?

• 1:08:58Speaker 01

  No, I would say that I think that the simplest way for me to close out this episode is say, hey, this is a real thing. It's going to get super weird from here and it's going to happen fast. And of all things, I'm going to recommend, there's a book by Henry Kissinger. Yes, that Kissinger and Eric Schmidt, former Google, Google CEO called, uh, The Age of AI and Our Human Future, or something along those lines. And it's worth a read if you're into this. It's not about the tech behind it at all. It's just kind of the implications and what might come. And it's fascinating for that perspective.

• 1:09:34Logan Bartlett?

  Awesome. Well, thank you both for coming on. I appreciate you doing this and yeah, we'll, we'll, we'll link all this stuff in the show notes so people can actually follow and keep up with everything we talked about here.

• 1:09:44Speaker 01

  Thanks, Ben.

• 1:09:45Speaker 02

  Awesome. Thanks for having us on.

• 1:09:51Logan Bartlett?

  So that'll do it for the 52nd episode of the Logan Bartlett Show. Thank you to my partners, Jacob and Jason for joining. Thank you to Justin and Rashad for their efforts with this episode. And thank you, everyone, for listening. A brief plug here to please like, subscribe, share, all of that good stuff. Leave a review. We spear you advertising here. You don't have to listen through me droning on and doing some ad read. And our only ask is that you promote the podcast for us so that we can continue to grow and keep doing this. So thank you, everyone, for listening in and we will see you back here next week for the 53rd episode of the Logan Bartlett Show. Have a good weekend.