How do we go from search engines to answer engines? With Perplexity AI’s Aravind Srinivas and Denis Yarats

With advances in machine learning, the way we search for information online will never be the same. This week on the No Priors podcast, we dive into a startup that aims to be the most trustworthy place to search for information online. Perplexity.ai is a search engine that provides answers to questions in a conversational way and hints at what the future of search might look like. Aravind Srinivas is a Co-founder and CEO of Perplexity. He is a former research scientist at Open AI and completed his PhD in computer science at University of California Berkeley. Denis Yarats is a Co-Founder and Perplexity’s CTO. He has a background in machine learning, having worked as a Research Scientist at Facebook AI Research and a machine learning engineer at Quora. No Priors is now on YouTube! Subscribe to the channel on YouTube and like this episode. Show Links: Aravind Srinivas on Google Scholar Denis Yarats on Google Scholar Perplexity AI Perplexity AI Discord AI Chatbots Are Coming to Search Engines. Can You Trust Them? - Scientific American Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @AravSrinivas | @denisyarats Show Notes: [1:46] - How Perplexity AI iterates quickly and how the company has changed over time [5:46] - Approach to hiring and building a fast-paced team [10:43] - Why you don’t need AI pedigree to transition to work or research AI [14:01] - Challenges when transitioning from AI research to running a company as CEO & CTO [16:50] - Why Perplexity only shows answers it can cite [19:33] - How Perplexity approaches reinforcement learning [20:49] - Trustworthiness and if an answer engine needs a personality [23:05] - Why answer engines will become their own market segment [26:38] - Implications of “the era of fewer clicks” on publishers and advertisers [30:20] - Monetization strategy [33:20] - Advice for those deciding between academia or startups

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  So for factual accuracy, our first step towards that was making sure you can only say stuff that you can cite. It's not just that we want to retrofit citations into a chatbot. That's not what perplexity is. In fact, it's more like a citation first service.

• 0:27Sarah Guo?

  This is the No Priors podcast. I'm Sarah Guo.

• 0:30Speaker 02

  I'm Elad Gil.

• 0:31Sarah Guo?

  We invest, advise, and help start technology companies.

• 0:34Speaker 02

  In this podcast, we're talking with the leading founders and researchers in AI about the biggest questions.

• 0:47Speaker 02

  With advances in machine learning, the way we search for information online will never be the same. We're back again to talk about the future of search. This week on the No Priors podcast, I'm excited to introduce our next guest from perplexity AI. Perplexity is a search engine that provides answers to questions in a conversational way and hints at what the future of search might look like. Aravind Srinivas is co-founder and CEO of perplexity. He's a former research scientist at OpenAI and completed his PhD in computer science at UC Berkeley. Denis Jarrett is a co-founder and perplexity's CTO. He has a background in machine learning, having worked as a research scientist at Facebook AI and also as a machine learning engineer at Quora. Aravind and Dennis, welcome to the podcast.

• 1:29Speaker 00

  Thank you for having us here.

• 1:30Speaker 02

  Thanks. Hi, thanks so much for joining. So the two of you alongside Andy Kowninski created perplexity around August or so of 2022. Aravind, do you want to give us a little bit of a sense of why you started this company and what the core thesis of perplexity is?

• 1:46Speaker 00

  Yeah, sure. Actually, Elad, you're our first ever investor who offered to invest in us. So those were the founding days. In fact, I remember the first ever idea we talked about in Noe Valley where we're sitting in the open space opposite Martha and I was telling you, oh, it'd be cool to have a visual search engine. The only way to disrupt Google was to not do text-based search, but to actually do it from camera pixels. And you were like, this is not going to work. You need to think about distribution and search was always the core. Motivation for me and Dennis and many others at the company. We were just bouncing around ideas and then I slid open air on the time and then we left. And Dennis also was still at Meta.

• 2:29Speaker 00

  Then he also left and Andy came in to help us sort of incorporate and get the company rolling. So that's sort of how the company started. The space of L and this was exciting. Generative, generative models were really exciting. And in general, we were motivated about search, whether it be a general search or vertical search. And we were bouncing around several different ideas. One of the ideas that you gave us was the working on text to SQL. And like, we were pretty excited about that and started prototyping ideas around that. And I think Dennis also like was hacking with us on building like a Jupiter notebook extension with like a copilot for every cell.

• 3:12Speaker 00

  And then we were trying it with SQL around databases. But it's all like a bunch of nonlinear pathways to eventually get to where we are right now.

• 3:21Speaker 02

  Yeah, absolutely. And hopefully I caveated whatever feedback I gave with, I'm probably wrong, but since I think I'm often wrong on directions.

• 3:29Speaker 00

  No, I think, I think whatever you said still applies. Search is tremendously a distribution game as much as a technology game.

• 3:36Speaker 02

  Yeah, I think one of the really impressive things about perplexity is the rate of iteration. And to your point, you've, you've gone through things like text to SQL, copilot for the next gen data stack. And I've always been impressed by how rapidly you've just been able to point in a direction, iterate really fast, prototype something, see if it's working and then move on to the next thing. And to your point, you always had search in the back of your mind. I remember even as you're prototyping these things, you were talking about indexing aspects of Twitter or other sort of data feeds and then providing search on top of them. How did you end up building a team that can iterate that rapidly as well as a culture of fast iteration? Like other specific things that, that you all do as a team to help reinforce that?

• 4:15Speaker 00

  Yeah, I'll I'll take the first part of this question and then also let Dennis answer this because he's a big part of why this is happening. We both are basically from an academic background. So in general, the culture and academia is to you have hundreds of ideas and you just need to try them out pretty quickly, run a lot of experiments really quickly and get the results and iterate. So we come from that background, both of us. And so so that's not really new to us. It's just that when it comes to trying out in products, it it's not just a result you get from running an experiment. You actually had to go to users and make them use it and talk to companies or customers or potential people in a company who will be using your product and get feedback.

• 4:56Speaker 00

  So there's that aspect of operational work that needs to be done to get results for experiments. And there's this aspect of quickly doing engineering to get it to a state where you can show it to people. So both of these things had to come together and that's why the company exists. And Dennis is incredibly good at engineering and recruiting. And so we found our other co-founder, Johnny Ho, who's also like a big reason why Perplexity operates this fast. And full credit to Dennis for helping recruit him. He was the world number one at competitive programming. It's like being Magnus Carlsen of programming. He got pretty excited about all the stuff that Dennis was showing him. So it's sort of like few people help you accelerate a lot and Johnny is like one big reason for that.

• 5:41Speaker 00

  And Dennis sort of continued doing that by getting more and more people into the fold.

• 5:46Speaker 02

  Are there specific things, Dennis, that you look for when you're hiring or when you're screening for great technical talent? Because I mean, one, the way one person put it to me is that you have a team that can build almost anything really fast. And so I'm curious, like, are there heuristics? So there are places that you look for people? Like, how do you think about hiring?

• 6:02Speaker 03

  Yeah, that's a that's a good question. I'm kind of like looking for the people who maybe have like this general interest in like applying this emerging technology like LLM, right? So like, a lot of people. Hear about it, maybe they don't have a specific background, but they're like very curious to learn about it and they wanna put like a lot of energy into it. So I feel like that's my number one indicator. So like personally, I would rather like get somebody who has this burning desire to work on these things rather than somebody who already has a lot of experience and not gonna put a lot of, or like as much effort as other people. I think another concept that we still use into this day, even though it's like takes a lot of time, it's we have kind of like this trial periods, I would say.

• 6:45Speaker 03

  So we basically get to know person a little bit better and once we see there is like good chance we might hire this person, we then ask person to sort of like work with us for like some time. We kind of like intend to keep our number of employees very small. I feel like it's much faster to operate. And because of that, each hiring decision is very important, you know, and we wanna like optimize for the best people. Uh, and that's why we kind of like run this trial process where, you know, we basically making very sure that we wanna work with this person. So I think that that's been helpful.

• 7:20Sarah Guo?

  Is there anywhere you've been surprised as you've made hires where you feel like, you know, the signal was wrong or the trial surprised you?

• 7:28Speaker 03

  Yeah, there has been few exceptions. Obviously, I think that's the part of it, but I feel like it's definitely, at least like comparing to my prior experience, it's definitely smaller chance where you're gonna get surprised. Normal interviews that, you know, big companies run, you know, you have like four or five meetings, like for forty five minutes each, and then you basically make decision after that. Sometimes it works, sometimes it doesn't. But I feel like the way we do things, it's gives us like much more confidence that we're gonna make right decision. And I, I think it's obviously it's very useful for us to get the signal, but I also feel like for candidates, it also makes sense to, uh, it's useful to make better decision as well, right? So they can understand, do they wanna work on things like we do?

• 8:14Speaker 03

  Do they wanna work with the base that we do? I think like one important thing for us and like many candidates, sort of like. Don't wanna maybe do this is the energy we put in into perplexity. It's kind of like work-life balance, maybe not the ideal. But that's the only way to sort of like beat competition and sort of like iterate very fast and do great things. So that's why we kind of like need to have this alignment at the beginning.

• 8:42Speaker 00

  Yeah, if you don't have a clear idea of like, you know, which product you're going to build or which market you're going after, and you still wanna be giving yourself a shot at success. Uh, iteration speed is the only thing that you can hope for. And so we decided, okay, like until we actually figure out our business and product, we'll make sure that this is non-negotiable, like the people should iterate really quickly with us.

• 9:07Speaker 02

  Yeah, I feel like the only real advantage that a startup has relative to incumbent is speed. Right? Because incumbent has more people, they have more money, they have more distribution, they have more product. And so really, the only thing you have is speed. In the early days of mobile, you know, I was working at Google and I I really helped get that team up and running. And then I started a company that Twitter bought and at Twitter, there was this weird meme internally that the only people they should hire for mobile were people with prior mobile experience. So I thought was a really dumb idea, right? It was more, you just find great engineers and they'll figure out how to write for iOS and stuff like that. How do you think about that in the in the ML world? Because there is this sort of dogma right now, I feel, that people believe that only somebody who's trained in LLM before can work on an LLM centered company or things like that. So I'm just sort of curious, you know, how you think about that.

• 9:53Speaker 00

  Yeah, actually, my thinking about this was already pretty clear because of having seen how OpenAI went about this. If you look at the people who did GPT-3, eventually they went on to start Anthropic. All of them are basically from physics background, the CEO Dario is a physics PhD and Jared Kaplan is a physics professor. He wrote the scaling loss paper. So they basically, OpenAI really succeeded in bringing these extremely talented physics people who wanted into machine learning and they came into the sort of language models and scaling. And that paid off well for them. And similarly, their whole engineering team, if you look at an engineering team, it's just Dropbox and Stripe people.

• 10:34Speaker 00

  All like software engineers, solid people who can build infrastructure front end. And now they're doing AI work. So it's always been clear to me that in order to work in AI, both research and product, you do not need to already have been in AI. And that's being shown clearly with Johnny Ho, our co-founder. He was not in AI. He was a competitive programmer, a trader. He'd worked at Cora for a year. But he's as good as anybody can get in picking up new things. So the other thing also is that L L M's are sort of in this weird territory where the people who use the L L M's for building stuff understand it better than the people who actually did gradient descent and trained these models. Like, you could find a PhD student at Stanford or Berkeley who would know a lot about

• 11:20Speaker 00

  how to train the model, but they might not be the best person to build a product with it.

• 11:24Speaker 03

  Yeah, I want to like quickly add to this point. So I was at early days at Facebook AI Research in Menlo Park, the office, right? And at that time, and that that was honestly one of my reasons to do PhD. Is there was like kind of this like very exclusive culture. So if you like, you don't, you don't have a PhD, you're not going to be a research scientist. And I didn't like that too much. So that's why I decided to do PhD later on. But uh turns out through my experience, the best research scientist, also very good engineers, like very good engineers. And we've noticed like through DeepMind and OpenAI is just like the the companies that made the most progress over like last six years or five years were companies where they're like extremely good engineers.

• 12:09Speaker 03

  And it's kind of like, I didn't like from the beginning, this, I guess, view from like academics that's just like, if you engineer, you're probably not, you know, either like smart enough or you're not going to do like great things. But turns out it's actually the other way around. So that was also, I think, like motivation. And I feel like you don't need to be, you know, this like very impressive, like academic with PhD to do great things.

• 12:32Speaker 00

  Yeah, this also goes back to the thing Dennis said earlier, that you want to find the people who really want to get into AI. Uh, rather than who are already in AI. And every, every company that's gotten big has done this in their early days. Including Google, like they got a lot of systems people compiler, Jeff Dean was a compilers and systems person. And then they, they got, and Urs Horslow is like a systems professor. They got all these amazing people and they told them, hey, you know, guys, like we're, we're having the most interesting computer science problems to solve here. And we can scale it up. So come work on search. So it's not like you have to go, the few people they hired from information retrieval or search background, but most of the celebrity researchers that they have right now are just people who wanted to

• 13:16Speaker 00

  get into that space for the first time.

• 13:18Sarah Guo?

  It's funny because now the, now the desired pedigree isn't the PhD from whatever, you know, academic lab because the industry labs filled with physicists and engineers and people not necessarily from the domain made the most progress. And so if you want somebody who's worked on, I, I hear the argument, you don't actually need it. Like you want people who are really smart and motivated. But a lot of companies are looking for somebody who has, you know, experience with X billion parameter training runs and those people come from now, OpenAI, DeepMind and such. So it's quite, quite funny, very quickly how the pedigree has changed. What's been the area of steepest learning curve with both of you coming from these like research engineering backgrounds?

• 14:01Speaker 00

  For me, it's like how to run company. That's mostly what I'm doing here. I'm not doing much the core engineering, tennis test that. So that was not easy at all, but I've had the opportunity to learn from many good people, including you out here. So if at all, there's an easy way to do it, it's like getting advice, rapid advice from people. The other thing is also like when you're making a mistake, like being super brutally honest with yourself and listening to feedback and quickly course correcting it. That was also like very new to me.

• 14:35Speaker 03

  Yeah, I guess for me, probably it was, you know, building a team, kind of like organizing everything in terms of like who does what, how to prioritize things, what to work on. I think like being a small team, it's even more important to identify this one thing that you want to work on and like put all your focus on it. Early on, we were like had this, sometimes made these mistakes that we're trying to go after like several things. That's why. Even though we're like six months or like seven months old company, we actually built so many things. We had like, you know, Twitter, we had like Salesforce integration, we have like HubSpot integration, like many other things that we never like released.

• 15:15Speaker 03

  One of the interesting thing is just like you, you have to have very precise focus and just like go after it.

• 15:23Speaker 00

  Yeah, also as leader, if you wanna lead the company and if you cannot do everything, the people will lose faith in you, right? They think you don't, you don't really have any clarity and that's why you're making them do one new thing every week. So you, you have more responsibility to sort of get it right and think more clearly. And even if you're wrong, like do one thing wrong and at a time and course correct rather than doing five things at once and seeing like which one wins. It's, it's difficult for us coming from the academic background for this particular thing because in academia, you're basically taught to hedge. You have like one first author project and like three or four co-author projects. And like one of them might become a big hit and might change your career.

• 16:03Speaker 00

  Whereas that's not how you should do startups, like you really have to focus and iterate multiple times on one thing or few things. So that took us a while to quickly learn.

• 16:13Speaker 02

  Yeah, makes a lot of sense. I think everybody goes through very similar paths as they start a company. You know, one thing that I think is really interesting right now is we're, we're at this point where consumers are really starting to wake up to how machine learning and AI is changing search. And to your point, you all were thinking about this actually before ChatGPT and before the Sydney News and before Bing integrated, all these things. So you were quite early to realizing that.

• 16:34Speaker 00

  Yeah.

• 16:35Speaker 02

  You know, this is gonna be a really core piece of search. Perplexity's mission, I, I believe, is to create the world's most trusted information service. How do you think about important product points around factual accuracy, bias in presenting aggregated information for users and things like that?

• 16:50Speaker 00

  Yeah, so I guess you're, you're also from a PhD background, right? So when you write your first paper, the thing your advisors teach you is you only have to write things that you can actually cite. Anything else that you write in the paper is your opinion, not, not a scientific fact. And so that sort of stuck with us pretty closely. And that's sort of why we did the first version where it's citation powered search. So for factual accuracy, we, our first step towards that was making sure you can only say stuff that you can cite. This is a pretty subtle point here. It's not just that we want to retrofit citations into a chatbot. That's not what perplexity is. In fact, it's more like a citation-first service that it'll never say anything that it cannot cite.

• 17:35Speaker 00

  So if people have tried to play with it as if it's like ChatGPT where like, tell me who are you or like things like that. And even for those questions, it would still go to a search engine, pull up stuff and come back with an answer. It's not going to say, I'm perplexity, I'm like a bot design, how are you doing or something like that. This is because of our obsession about factual accuracy, like even if it doesn't have a personality or or character in it, we don't care. We want, we only care about the other thing, which is obsession about truth and accuracy. Yet, the second point you mentioned about aggregation of things, when you mash up multiple sources together, you might end up hallucinating. Like, for example, if there's multiple people with the name Elon Gill, and like one of them is a venture capitalist and now some others like a doctor.

• 18:17Speaker 00

  Or let's say, even if it goes to your own LinkedIn and things like, oh, the Elon Gill who did a PhD in biology is different Elon Gill from the venture capitalist, because someone might think that, right? It's pretty unusual background. So then that it might end up coming up with some entertaining or funny summaries that collate different sources together. We still haven't thought of a proper fix to this, but one thing is obviously as language models keep getting better, they're going to understand these things, these subtleties even better, and we're already seeing that. And the second thing is, we are giving users the power to remove sources that they think are irrelevant, just like how you can curate sources in Wikipedia. So we're sort of working towards this accuracy and the bias issues step by step at a time, but I I feel like it'll take more iterations to get this truly correct.

• 19:01Speaker 00

  And I also don't think one LLM will just magically solve this problem. You need to build an end-to-end platform where users can correct the mistakes of an LLM. And that also means you need to design the platform where the incentives are right for the user, because this could also be used in the other way where users can use it to hide information. So we haven't really thought through all these issues thoroughly, but we are committed to sort of figuring these things out over time.

• 19:26Speaker 02

  That makes sense. I guess in addition to that, or maybe related, you've done an impressive amount of research in reinforcement learning. What's unique about the way that perplexity uses reinforcement learning and how does it tie into these plans?

• 19:35Speaker 00

  We like RLHF, like reinforcement learning from human feedback, where we use the contractors to, we collect feedback from the users on whether they like the summaries, the completions or not, and like we use contractors to do the some ratings themselves. And these days, even LLMs can be prompted to do the work the contractors do. Anthropic's written a paper on that. So all these things are getting really very efficient to do. So that's sort of how we have been thinking about reinforcement learning right now. But we haven't gone beyond that to think of like agents and browsers and things like that. We'll probably focus more on the first part for the next at least six months to a year.

• 20:12Speaker 03

  One add on this aspect, full blown, like RLHF is, you know, definitely something we're going to look into that. But there is like several many steps that you can have in between that significantly can increase your quality. So for example, I mean, even using something like a rejection sampling, you know, like discriminator on top, it's kind of, you want to like shift, you know, maybe you have several samples from your LLM and then you can, can rank some of them using different model, pick only those. It is in a sense, kind of like one step of reinforcement learning, but it helps a lot. It's very effective.

• 20:48Sarah Guo?

  You guys have talked about how trustworthiness is more important to you than like, I don't know, personality, the ability to play with a bot. Like, do you believe in chat as an interface? Like, where's the line between chat and search, given perplexity does support, for example, like follow-ups and things that are more conversational?

• 21:04Speaker 00

  We think chat UI is the future. People are using it pretty heavily. At the same time, if you can try to get the answer right in the first attempt, you should, right? Like, you have a responsibility to save people's time. It's not like Google doesn't. Do some kind of chat implicitly, like if you go to Google, you always get related questions, follow-ups, people also ask for and things like that. It's a sort of implicitly making you click on it and, you know, like you get a follow-up question that you sort of get an experience without the chat UI. So I feel like it's more like whether it makes sense for the particular experience you want to provide for your end user. And in our case, it does. Often you do not get the answer you want in the first attempt and

• 21:47Speaker 00

  you shouldn't feel the burden to get it also. Like sometimes you could ask a question by just asking for the keyword and then you might realize like, okay, this is actually what I wanted to ask for. I've seen this in live when people use it for perplexity, but they just couldn't know how to phrase this question the first time. So they use multiple attempts to get to the right question. So there are things like that, whereas the questions get more complex. The chat UI makes a lot more sense than Google's UI. But for like obvious questions, like if you just wanted to know whether it's gonna rain in Bay Area the next one week, why do you need to keep asking more, right? Like you just get the answer the first attempt. And we want to support both these experiences in perplexity.

• 22:26Sarah Guo?

  More broadly, I'll ask a dangerous question, but what do you, what do you think the future of search looks like? Right? Five years plus out. Do we still have monolithic horizontal providers if the players change? Do we get more embedded apps? Like contextual search as a feature in different places? Are there agents that do things for you? Like, what do you think it looks like?

• 22:46Speaker 00

  I think there's this phrase that's becoming popular. I think Karpathy was the first one who tweeted it and Satya Nadella is also using it. It's called an answer engine instead of a search engine that directly tries to answer your question instead of providing you a bunch of links or just snippets from the first link. So we believe in that, like perplexity is the first conversational answer engine, truly the first, I think nobody built it before us. I believe answer engines will become its own segment, market segment. Like, just like you have a default search engine, there'll be a default answer engine over time. If these things really work and the burden of getting ranking and search right will reduce in the sense answer engines can do more heavy lifting than search engines. As these things get really good and like.

• 23:31Speaker 00

  Way fewer hallucinations and even even if they do hallucinate, people can still go and click on these links. They will eventually prefer this experience over the regular like ten links or twenty links UI that Google has. So that that's something I'm pretty confident about and and I think the sort of asking follow up questions will become more of the norm. The number of queries in perplexity that go to at least one follow up has been increasing ever since we released the chat UI. So that that will keep going up. People will get used to the sort of experience where they're encouraged to ask one more question. And they're okay with not getting the answer right away. So that that'll happen. The third thing is like actions. People will be more deliberate in what they search for and try to execute an action on top of the

• 24:16Speaker 00

  search results they consumed. So that's definitely likely to happen. It's already happening. If you go to Google and book a flight, you just like fly from SF to Seattle. You just directly click on the book button. So that's going to happen more frequently in the chat UI too. And this will become an assistant more than just a search or answer engine. And I also think the fourth thing is there will be some sort of like much fewer traffic to the actual content site like very few links need to be consumed. In perplexity, in fact, like we only we don't even cite more than five links. It's it's a deliberate decision. Like a lot of people ask this, can you add ten links or twenty links? Can you just show all the links together? You put the summary at the top, but you also put all the ten links, the usual, I want both.

• 24:58Speaker 00

  It's a decision just we just made that, no, no, that's not the right experience for you. You actually need to feel the difference here. So we only made only three to four links or like, in fact, the first version shipped it like three to four links, I think. And like fifty word summaries and we expanded the summary more. So I think we just have fewer tabs open. We'll only open the tabs that we really need so that all this sort of behavioral change in the consumer is likely to happen, whether it be through us or Google itself doing these changes, it's unclear, but this is where it looks like it's trending towards.

• 25:31Speaker 02

  When do you think we're going to have that transition from almost like a pull versus push model, right? Because I think right now you go and you ask for certain information and you may ask for it repetitively, you know, I've been checking the weather every day. Versus a world where you have agents that are effectively understanding your intent ongoing and providing you with an information and it's sort of a push based way. How far away of a world is that? Is that a year away, five years away, 10 years away?

• 25:56Speaker 03

  I feel like we can do it now. In fact, like Google now was an attempt to do that. If you keep checking for scores of a favorite football club, it'll automatically give you a push notification for their next latest match that you might have missed. They tried some of these things already. So I feel like we can do these things even better now with language models. So yeah, I think it'll happen in a year, more than five years.

• 26:19Speaker 02

  And do you think it's going to be fragmented in terms of each? And I know this is all uncertain, right? It's kind of predicting the future is never correct. But I'm just curious how you think about, is there going to be like a agent for your Google Drive or an agent for your GitHub or an agent for your email? Or is it just sort of consolidated eventually into one central service?

• 26:37Speaker 03

  I mean, it's getting pretty clear, you know, with Google. The first few results, like at least like my pattern of using Google, I see like first two, three results, I basically skip most of the time because it's here. Or like some ads and that's like not ideal experience, obviously for the user. And it's also kind of like, why it might be very hard for Google to fully launch this system? Like Answer Engine, because it just like breaks monetization strategy for them. As Aravind mentioned, you know, there's going to be like fewer clicks. And Google wants you to click on those things, right? So that's how they make money. Basically, I think there has to be this like new paradigm where you kind of like you get your answer quicker.

• 27:22Speaker 03

  But then maybe you can monetize it better on it. Maybe now like you can help user to make a decision faster and then you can show like much more targeted and accurate ad. So then you know, like if user clicks it, it's going to be, maybe they're going to buy something or whatever. So there is going to be a few clicks, but each click is going to be more expensive. And I feel like overall user experience has to be better because you just, you just get what you want. You don't have to do any extra effort.

• 27:51Sarah Guo?

  In this era of fewer clicks, right? If the summaries in chat is just giving you the answer, how does that impact the relationship of search or answer engines with publishers? Right? Does that remove the incentive to publish information on the internet? Does it become more adversarial? How do you guys think about that?

• 28:09Speaker 00

  Probably not. I feel like, so it's sort of like whether you cite a paper or not, sort of thing, right? Like you would cite a paper if the paper was really good. It's actually going to bring back the whole concept of page rank even more. But the concept of page rank was inspired by academic citations, from what I've read. Like a very important paper tends to get cited. So when you're in the sort of citation-based search interface now, the better content a publisher has, the more likely it'll get cited by an LLM. Unless humans figure out answer engine optimization or LLM optimization, which I hopefully they don't invest effort into. But in general, it's unlikely that it's as easy as SEO with just keywords,

• 28:54Speaker 00

  because LLMs are going to be much smarter in understanding relevance to a query. So I think it's just going to incentivize people to publish higher quality content in order to get cited by an LLM-powered answer, like Substack or things like that try to do. Like you want to own your content and you publish it and make sure it's high quality and you have your own like set of subscribers. So people put a lot of effort into that, more than writing tweets. So something like that is likely to happen with this interface too. But it's unclear exactly how to make all this monetized at scale, like, you know, the click-based ad engine that Google has. And I think it's a super interesting problem. And it's amazing that many companies are trying this at the same time. So even if one company figures this out,

• 29:38Speaker 00

  like others can also like benefit from it.

• 29:41Speaker 02

  If you look at a lot of the biggest consumer services, they took a while to figure out monetization. So for example, Google's first attempt was literally to sell search results. So they got paid per thousand search queries that they'd syndicate to others. And then they built out an enterprise appliance, a literal piece of hardware that you'd install at enterprises to do search inside the enterprise. And then eventually they came up with ads and really realized that that was their future path for monetization. And then similarly, you look at YouTube and people said that would never make money and Facebook would never make money and all these things would never make money. And then, of course, they monetize eventually. How are you thinking about monetization? Or is it simply too early? And it's more about just getting market share? And then, you know, at that point, you can iterate on the right model.

• 30:20Speaker 00

  Yeah, we have multiple thoughts about this. Obviously, just like Google was paid to sell the search results for a thousand queries like. I mean Bing has APIs like that too, and people asked, at least more than thousands of people have emailed us. Or messaged us in various different ways to ask for perplexity as an API. And we haven't done that yet, but that's an obvious monetization strategy. And the other thing is obviously prosumerization of this, where like we're already sort of beginning to see our Chrome extension. Pick up rapidly, heading towards hundred thousand users. And extensions like Grammarly have the sort of free version and the prosumer version, which has more features in them.

• 31:02Speaker 00

  So there are ways to do that through the browser extension as a productivity assistant sort of thing. We already see it's like some kind of search pilot. Every time you're on a site, you can you can ask it to do things for you. And then there's the whole, as we keep getting more and more traffic onto our site. Like say hundreds of millions of people come to our site at one point eventually. That becomes a ripe ground for serving ads. But we need to not make the mistake that Google did of combining ads into the core search product itself. And figure out an alternative pathway like Facebook did. And and that might work out better for us. Subscription based search has been tried by other companies and that's something that ChatGPT is also trying. So.

• 31:46Speaker 00

  We we don't know yet if it's high margin enough. And so if a bigger behemoth like Google or Bing. Just put out the same or like even eighty percent as good as you for free, then you're never gonna make it as a subscription product. So we are likely to stay away from that pathway, but we don't know yet. And the final piece is like, if perplexity becomes like something that a lot of people want to use for their own internal data. Their links or their bookmarks or their company. And if we can make it easy for them to build that and become more like a platform which everybody can use, then. That's likely to lead to monetization too. So there are like so many different rollouts possible here that we don't know yet which one we'll actually go for.

• 32:27Speaker 00

  But in the in the short term, we are more focused on growing the product, the users, the traffic. In fact, improving the whole experience like this, I feel like Google and Microsoft will pretty much do the same thing we have right now. As well as us. So we, as and as we discussed in the first part of the podcast, like. We need to operate with more velocity, ship more things and stay ahead of them in terms of the core value of the product itself.

• 32:54Speaker 02

  Yeah, that makes a ton of sense. And I think, you know, to the point before, there's probably lots of paths to monetize once you have a lot of usage. And so it's more just, you know, figuring out what's native and natural to the product. You've mentioned earlier that one of the things you learned from academia is fast iteration. And I feel like most academics I've worked with are almost the opposite, you know, like I actually feel like there's a lot of pre-planning and a lot of discussion and there's, there's lots of a bias to action. And so I've been very impressed by the bias to action you all have. What advice do you have for researchers who are now deciding between an academic path or joining a company, research role, or starting a company? What advice would you give them, given that you've gone through a recent transition that's similar?

• 33:33Speaker 00

  Yeah. Firstly, we both share, Peter Beal is also Dennis's thesis advisor and my advisor. And he's a pretty different academic from the others. He pushes people to get results pretty fast. So that's a reason why we both are like that. And Dennis also worked in industry where you can't operate that slow. So for the advice to academic researchers, I feel like it's super hard to be an academic researcher right now, especially a PhD student is one of the worst jobs you can go for in a time when AI is so hot and so highly paid and you can do a startup or join a startup. You have to sort of give up all that and the buzz every single day you see on Twitter or other news journals and still focus on

• 34:15Speaker 00

  trying to build a future. It's more of a mental thing, I would say, more than like picking the right problem, like even having the composure and the maturity to sort of stay poor and like work on hard problems is super hard right now. Very few people in the world can have the ability to do that. To be very honest, I only came to the United States for doing a PhD because there was no other way for me to come to the United States. Like I couldn't take a loan for a master's or something like that. So PhD is fully funded and like sponsored. So that was the biggest reason I actually wanted to come here, more than doing a PhD. If I had the ability to get a job in industry, I'd have gone for that. So there's other reasons you might want to do a PhD. But okay, if

• 34:56Speaker 00

  all these things are sort of not a problem for you and you still want to do something, I think it's best to look for alternatives to the transformer, alternatives to like language models, that sort of radical directions than.

• 35:10Speaker 00

  Trying to improve them because there's so much incentive for the existing companies to do that. A lot of people think OpenAI has no incentive to improve GPT, the core architecture or some, or like the model or something that's far from true, like even if they have a lot of money, they would want to make it more efficient and train even bigger models that make better use of compute. So I feel like it's best for people to sort of look at places that are kind of controversial or radical. And, and that would mean even questioning Transformer itself, which is actually one of the best research problems to work on. That's what I would work on if I were to do a PhD, I would work on trying to be, write the next Transformer paper.

• 35:53Speaker 03

  Fully agree with this. I think I would probably want to also add, at least like from my experience, I think it's best to not go to PhD right after undergrad and kind of like spend at least a couple of years in industry. I think that goes to my point to me, like the best researchers are those who can, who are also very good engineers. Like you can get this like valuable experience of being, you know, a good engineer and then it's, it's basically going to propel your PhD. You can do things faster and especially, you know, like now AI becomes a little bit more like engineering than it used to be, right? So it's, it's just like this skills are essential.

• 36:34Speaker 03

  Either you, you won't be able, especially if you want to work on like LLMs or like large scale stuff, you have to be a very good engineer. But uh, yeah, if you still want to stay on like very academic side where you just like think like very deep ideas, then I agree with Aravind, you probably want to completely ignore LLMs and just do something very radical.

• 36:55Speaker 00

  There are some other ways to do stuff that's still useful and pretty different. Like I think Stanford has some students doing this, like state space models and flash attention. Uh, there's some really good papers like that coming out and flash attention is already being used to improve the efficiency of LLMs. So you can sort of do such things. Or you can go work on video or video generation, stuff like that. That's just still out there.

• 37:20Speaker 02

  That makes a lot of sense. Well, Dennis Aravind, this was a great conversation and it's really exciting to see all the progress you're making on an important area. So thank you so much for joining us today.

• 37:27Speaker 00

  Thank you. Thank you for having us here.

• 37:29Speaker 03

  Thank you for having us.

• 37:33Sarah Guo?

  Thank you for listening to this week's episode of No Priors.

• 37:36Speaker 02

  Follow No Priors for new guests each week and let us know online what you think and who in AI you want to hear from.

• 37:41Sarah Guo?

  You can keep in touch with me and Conviction by following at Sarah Normas.

• 37:45Speaker 02

  You can follow me on Twitter at @AladGill. Thanks for listening.

• 37:51Sarah Guo?

  No Priors is produced in partnership with Podpeople. Special thanks to our team: Synthil Galdea and Pranav Ruddy, and the production team at Podpeople; Alex Vicmanis, Matt Saab, Amy Michado, Ashton Carter, Danielle Roth, Carter Wogan, and Billy Libby; also our parents, our children, the Academy, and Open Google Soft AI, the future employer of all of mankind.