NeurIPS 2023 Recap — Top Startups

We are running an end of year listener survey https://docs.google.com/forms/d/e/1FAIpQLSeCg-mQiox_Si5do-1ZIrVg9hPe5IFMjc39gfHdSp3-UaAPDg/viewform! Please let us know any feedback you have, what episodes resonated with you, and guest requests for 2024! Survey link here https://docs.google.com/forms/d/e/1FAIpQLSeCg-mQiox_Si5do-1ZIrVg9hPe5IFMjc39gfHdSp3-UaAPDg/viewform. We can’t think of a more Latent-Space-y way to end 2023 than with a mega episode featuring many old and new friends recapping their biggest news, achievements, and themes and memes of the year! We previously covered the Best Papers of NeurIPS 2023 https://www.latent.space/p/neurips-2023-papers, but the other part of NeurIPS being an industry friendly conference is all the startups that show up to hire and promote their latest and greatest products and papers! As a startup-friendly podcast, we of course were ready with our mics to talk to everyone we could track down. In lieu of an extended preamble, we encourage you to listen and click through all the interviews and show notes, all of which have been curated to match the references mentioned in the episode. Timestamps & Show Notes * [00:01:26] Jonathan Frankle - Chief Scientist, MosaicML/Databricks * see also the Mosaic/MPT-7B episode https://www.latent.space/p/mosaic-mpt-7b * $1.3B MosaicML x Databricks acquisition https://twitter.com/alighodsi/status/1673300587419701249 * [00:22:11] Lin Qiao - CEO, Fireworks AI * Fireworks Mixtral https://blog.fireworks.ai/mixtral-8x7b-on-fireworks-faster-cheaper-even-before-the-official-release-ae024b5ad602 * [00:38:24] Aman Sanger - CEO, Anysphere (Cursor) * see also the Cursor episode https://www.latent.space/p/cursor * $8m seed from OpenAI https://techcrunch.com/2023/10/11/anysphere-raises-8m-from-openai-to-build-an-ai-powered-ide/ * Tweet: Request-level memory-based KV caching https://x.com/amanrsanger/status/1728502445184274710?s=20 * Tweet: GPT-4 grading and Trueskill ratings for rerankers https://twitter.com/amanrsanger/status/1732145826963828997 * [00:51:14] Aravind Srinivas - CEO, Perplexity * 1m app installs on iOS https://x.com/AravSrinivas/status/1734103154302824681?s=20 and Android https://x.com/AravSrinivas/status/1729686690296476037?s=20 * pplx-online api 7b and 70b models https://x.com/AravSrinivas/status/1732799979260199171?s=20 * Shaan Puri/Paul Graham Fierce Nerds story https://mixtape.swyx.io/episodes/creator-fierce-nerds * [01:04:26] Will Bryk - CEO, Metaphor https://x.com/DXTipsHQ/status/1725664223009894428?s=20 * “Andrew Huberman may have singlehandedly ruined the SF social scene https://twitter.com/wangzjeff/status/1692971635513430140” * [01:12:49] Jeremy Howard - CEO, Answer.ai * see also the End of Finetuning episode https://www.latent.space/p/fastai#details * Jeremy’s podcast with Tanishq Abraham https://twitter.com/jeremyphoward/status/1626743024897372160, Jess Leao https://www.decibel.vc/articles/welcoming-jess-leao-as-decibels-newest-partner * Announcing Answer.ai with $10m from Decibel VC https://x.com/jeremyphoward/status/1734606378331951318?s=20 * Laundry Buddy https://twitter.com/search?q=from%3Ajeremyphoward%20laundry%20buddy&src=typed_query&f=top, Nov 2023 AI Meme of the Month https://www.latent.space/p/nov-2023 * [01:37:13] Joel Hestness https://www.linkedin.com/in/joelhestness - Principal Scientist, Cerebras * CerebrasGPT https://www.cerebras.net/blog/cerebras-gpt-a-family-of-open-compute-efficient-large-language-models/, all the Cerebras papers we discussed https://www.cerebras.net/events/neurips23/ * [01:56:34] Jason Corso - CEO, Voxel51 * Open Source FiftyOne project https://github.com/voxel51/fiftyone * CVPR Survival Guide https://www.linkedin.com/posts/jason-corso_cvpr-2023-survival-guide-voxel51-activity-7067695162604281856-citq * [02:02:39] Brandon Duderstadt - CEO, Nomic.ai * GPT4All https://github.com/nomic-ai/gpt4all, Atlas https://docs.nomic.ai/, Demo https://www.youtube.com/watch?v=3fQaUql5z3k * [02:12:39] Luca Antiga - CTO, Lightning.ai * Pytorch Lightning https://github.com/Lightning-AI/pytorch-lightning, Lightning Studios https://lightning.ai/studios, LitGPT https://github.com/Lightning-AI/lit-gpt * [02:29:46] Jay Alammar - Engineering Fellow, Cohere * The Illustrated Transformer http://jalammar.github.io/illustrated-transformer/ This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:06Speaker 00

  Hello, hello. This is swyx back again with part two of our New Earth's coverage. This time, we're going to cover startups. And it's a special episode because this is the last episode of 2023. We are definitely looking back at the year with rose-colored glasses. This has been a fantastic year. We only started this podcast in February, and it's grown so much. Thanks to all of you who've listened and give feedback and shared it with your friends. And we actually managed to invite a few of our former guests back on the pod together with some new friends and probably some new voices that you're going to be hearing next year. So this is not a hard-hitting interview series. You know, it's not going to, it's not that kind of interview. It's not that kind of podcast where we try to go too deep. We, we, today, we're just going to go broad.

• 0:49Speaker 00

  And we're just going to check in on a bunch of startups that we like and monitor and we're present in Europe. So first up is John Frankel of Mosaic ML. We last talked to him in May for the MPT7B episode. That's episode 13. And I have to say, that was one of the best performing episodes of the whole year. So you're welcome to go back and listen to that if you missed it. And since then, they were bought by Databricks for $1.3 billion. And actually, during the interview, they were in the process of getting acquired. They just couldn't say anything about it. But it's definitely one of the biggest AI news of the year. And you can listen to what it's like or what's going through John's mind back then, as well as now, today, six months later. Hey, Jonathan, welcome back to the pod.

• 1:28Speaker 01

  Thank you so much. This is an interesting place to have the pod under the overpass of Interstate, whatever it is.

• 1:33Speaker 00

  Yeah, Interstate, whatever, in the city of New Orleans. Yeah, it's really good to see you. Since you were last on the pod, Mosaic got acquired.

• 1:41Speaker 01

  Yeah, thank you. I think you really deserve all the credit for this.

• 1:44Speaker 00

  No, you guys were sitting on that news. And we didn't know what was going to happen. But I did, I did come away from your interview with a very, very high impression of like, you guys are in a perfect place, perfect time. And it makes a lot of sense to join forces with Databricks.

• 1:58Speaker 01

  Yeah, they're kind of, I mean, I will say, we really didn't want to get acquired.

• 2:03Speaker 00

  You did not?

• 2:04Speaker 01

  We didn't. I mean, we, we loved being independent. Like, we loved doing our own thing.

• 2:08Speaker 00

  Sure.

• 2:08Speaker 01

  Um, but this just made too much sense, like. You know, they do data, we do LLMs, we both do enterprises, we're all a bunch of academics. Like, it was just kind of, we couldn't think of a better match, and so it just, we kind of came to the conclusion like, "Okay, I guess we can't not do this, like, it's too perfect."

• 2:26Speaker 00

  Yeah, yeah. And you've done a bunch of other podcasts on the acquisition, so I don't, we don't need to retread, I'll send people that way. Just like, what's new in Mosaic world?

• 2:34Speaker 01

  In Mosaic world, honestly, like, we're just cooking. I think we've been a little quiet lately.

• 2:39Speaker 00

  Yeah.

• 2:39Speaker 01

  Or at least, we look quiet from the outside. It is certainly not that we haven't been busy, and it's certainly not that, you know, we're not doing cool stuff. Part of it is that, you know, getting acquired, there's a bit of administrative involved, you know, we had to go through new employee orientation, get health insurance, you know, meet our amazing new colleagues. Part of it is, like, you know, the field has moved toward bigger stuff, and we've moved toward bigger stuff. So, I think we'll have some exciting stuff to talk about soon, but my philosophy is always, like, speak through the work.

• 3:05Speaker 00

  Yeah.

• 3:05Speaker 01

  So I don't want to hype, I don't want to, like, get people excited, you know, you'll see the work and you judge for yourself.

• 3:10Speaker 00

  Yeah. You talk about the industry moving towards bigger stuff. What trends are notable to you in the, let's say, second half of this year?

• 3:16Speaker 01

  Everybody's figured out how to build LLMs. Like, it's no longer a coveted skill of, you know, a handful of people, but now, we've all become LLM builders. The field has kind of narrowed an aperture again, and, you know, in yesteryear when we were all figuring out how to train ImageNet, you know, now we're all figuring out how to build really big, really powerful models.

• 3:33Speaker 00

  Yeah.

• 3:33Speaker 01

  And, like, that's now just an assumed skill. The rest is kind of, what do you do with that skill? How do you build a product? How do you differentiate? What cool thing can you do that's different from everybody else? That's going to determine, kind of, you know, what 2024 is going to be like.

• 3:47Speaker 00

  Yeah. I guess, like, a lot of people are banking on multimodal being, like, well, 2024 being the year of multimodal LLMs. I feel like that's a little bit too broad a brush. I don't know, like, what's valuable in that front?

• 3:59Speaker 01

  I mean, it, so multimodal is going to be a huge deal. Like, it's, but it's already a huge deal. Like, we have multimodal models.

• 4:05Speaker 00

  The Lava paper author I also interviewed on this pod.

• 4:07Speaker 01

  Yeah, like, Lava's amazing. Like, you know, I've been playing with it a bunch personally. It's awesome. And we've got Bard, and we've got Gemini, and we've got GPT-4V, and, you know, I'm sure there are going to be plenty more where that came from. I think the question is, as with all good things, you know, cool promise is different than, like, delivering value.

• 4:24Speaker 00

  Yeah.

• 4:25Speaker 01

  And I'm really curious, like, you know, what do people genuinely do this, with this, in real production settings? And the settings that will actually pay off the huge investment that's made to build these multimodal models.

• 4:34Speaker 00

  Right.

• 4:35Speaker 01

  I'm also kind of curious, like, are we going to start to see some big, open-source multimodal models? Like, You know, we've got lava, it's moving in the right direction. But like, is somebody gonna, you know, build something that looks a lot like GPT four V? Or something on that trajectory and kind of start another arms race in that direction? Like, it'll be interesting to see. I'm I'm honestly pretty curious and I'm I'm watching with bated breath for what everybody does.

• 4:55Speaker 00

  Yeah, well, I think in our chat earlier today, you said, you know, we kind of live in a diverse world where like every company has kind of found its niche, maybe? If you want to go through that.

• 5:06Speaker 01

  Yeah, yeah, I'm kind of like, I think there are, you know, they're the optimistic and pessimistic scenarios for where we go.

• 5:12Speaker 00

  Okay.

• 5:13Speaker 01

  Like, you know, I don't know, I kind of think there's a boring scenario where everybody basically is building these giant L L M's and maybe language. You know, image models or what have you, and they're all kind of the same. It's just you've got the Google version and the OpenAI version and the Amazon version. And it almost feels like cloud providers in some sense, like, you know, what distinguishes AWS from G C P? It's kind of, you know, where you are.

• 5:33Speaker 00

  Lightly different consoles and.

• 5:35Speaker 01

  Yeah, it's like different interface, maybe you prefer one or maybe like you've been using one for a while and like you're used to it. Or your I T person really likes this one because, you know, they used to work at that company or what have you. Um, that would be a pretty boring world, but I I think that's unlikely to be the case. I'm kind of, I'm looking at like, you know, all the cool stuff coming out of Gemini. Um, you know, all the cool stuff coming out of OpenAI and then like I'm looking at Adobe, like.

• 5:58Speaker 00

  Firefly, really?

• 5:59Speaker 01

  Firefly, they're building like a different model with a creative perspective. Like, I'm kind of looking at this and going, maybe we'll have a wide diversity of models and. Everybody will be building models like just by virtue of the fact that we need so much data to build any of these models. Everybody's gonna play to their strengths and, you know, use every resource they have at their disposal and Google has, you know. You know, they have YouTube, I don't know if they're using it, but like, that's a cool resource. Um, OpenAI has put a ton of energy into text data. Adobe like gets creative people and like there are a few other companies where that came from. So I'm kind of like, I'm honestly curious if we're gonna just see like really different models for different people. And I don't know, that's a pretty cool world to live in, like.

• 6:37Speaker 00

  That is great.

• 6:38Speaker 01

  We won't see this arms race, we'll just kind of see like diversity.

• 6:41Speaker 00

  Yeah, and we shouldn't forget Bloomberg, which tees Bloomberg G P T, but that's a source of significant tokens, the financial world.

• 6:47Speaker 01

  Yeah, yeah, like it's, I mean, my whole business on the Mosaic and Databricks side is, you know, helping people leverage the data they have. So I'm kind of, I'm excited about a world of diversity because, you know, it's. Not only do we have like these crazy diverse foundation models at the larger scales.

• 7:01Speaker 00

  Yeah.

• 7:01Speaker 01

  But everybody embraces whatever they have, like our friends at Replit do a code model and, you know. I don't know, Bloomberg does another finance model and like somebody does a healthcare model. And like everybody draws in their strengths and that's a cool world.

• 7:14Speaker 00

  Are you bullish every company training on a model? Oh, sorry. That's that's a stupid question to ask you.

• 7:20Speaker 01

  I I mean, I I think, you know, I'll give you the honest answer because I think it's, you know, the the business mosaic answer is, oh, yeah, I'm super bullish. Everybody should train their own model. Come do it on Databricks right now.

• 7:31Speaker 00

  Like, you should start from a base model that everyone shares, right? Like, that's kind of.

• 7:34Speaker 01

  Maybe, or like you you work your way up. I think it's like there's a journey with playing with any of these models that may or may not end with training your own, depending on where you go on that journey. Like, you start by playing with an API and maybe you do some retrieval and maybe you do some fine tuning and then maybe you build your own model.

• 7:51Speaker 00

  Yeah.

• 7:51Speaker 01

  But like, it's a journey and I think there's a destination many people will get to that involves training their own model.

• 7:56Speaker 00

  Yeah, totally. What other trends are going on that you're that you're liking or seeing or hating?

• 8:01Speaker 01

  Honestly, you know, maybe this gets to the question of like, you know, overall impressions of NeurIPS. Like, I thought this was a pretty garden variety NeurIPS in some sense, which feels weird to say in the age of, you know, ChatGPT and everything else that's happened in the past year.

• 8:15Speaker 00

  Yeah.

• 8:15Speaker 01

  But this felt like the most normal conference I've had since like 2019. Um, you know, I mean, we've had a pandemic in between and everything, but like the past couple of years, I actually internally at Mosaic, I always do a long write-up of every conference and the trends I see.

• 8:28Speaker 00

  Okay.

• 8:29Speaker 01

  Like some public, some that are more relevant to what we're doing. And like a lot of the write-ups I've done over the past year or two have been like all about like the unease. Sometimes it was just like my write-up for ICML 2022 was all about people capitulating to scale and the five stages of grief and, you know, how different people were responding, academics, people at Google Brain, back when it existed, you know, all that stuff. And it almost looks quaint to think about that it was insightful to say, people have capitulated to scale in this day and age where, you know. You know, tens of billions of parameters looks mundane.

• 9:05Speaker 00

  Yeah.

• 9:06Speaker 01

  But this kind of felt like, okay, the academics are trying to find their way forward. It's no longer just kind of coping and ignoring, but like trying to find their way forward. The industry folks are doing their thing. A lot more people keeping secrets than used to, but it's still like, you know, a lot of people also aren't keeping secrets and can talk about what they're doing still.

• 9:23Speaker 00

  Yeah.

• 9:24Speaker 01

  So it kind of felt like, you know, equilibrium. I don't know how long it'll last, but this was a lot less of a frantic and stressful conference than I think I'm used to, at least in the past couple of years. You know, I'm in a new role in some sense. I'm on the business side now. I'm on the industry side and I'm trying to find my own path. But I felt like a lot of us have changed roles in some sense as the past couple of years have, you know, have taken place and everybody's moved around and figured out what they want to do. But we've all kind of found our place at this point. I feel like, you know, we may be in different places, but the ecosystem, the community has kind of sustained with, you know, a bunch of new PhD students and all that good stuff.

• 9:59Speaker 00

  Yeah.

• 10:00Speaker 01

  Like it's kind of, you know, I don't know, it's nature healing in some sense from the insanity of the past couple of years. And a reminder that, you know, we're all kind of small pieces in a much bigger, you know, ecosystem and community.

• 10:11Speaker 00

  Yeah, and it's still growing though. Apparently the latest stats was something like fifteen thousand attendees.

• 10:17Speaker 01

  Oh, my God. Oh, my God. I will say one big difference, you know, in the time right before the pandemic, deep learning was getting so popular, the conferences would sell out the day registration open. Like as a student, you'd have to rush to register or you wouldn't even get to go. That I don't think is happening anymore.

• 10:33Speaker 00

  This year is easier, yeah. And there was some live streaming stuff, you know, so.

• 10:36Speaker 01

  Yeah, but it's kind of interesting that like, I guess we've adjusted to the huge capacity and everything that's, you know, going on. But it's, you know, even so with it getting bigger, it didn't feel that different, to be honest. Maybe it's just that I joined the community when things were already big.

• 10:50Speaker 00

  Yeah.

• 10:51Speaker 01

  But like, you know, there were some journalists here, some VCs here, but that's always been the case.

• 10:54Speaker 00

  Yeah, it's always been the case. You always have, you know, overrated, underrated papers. We'll maybe save the overrated stuff for later, but any underrated stuff that you want to highlight from this year, it doesn't have to be at the conference, but just want to remind you for underrated papers that people should pay attention to.

• 11:08Speaker 01

  I'm going to flip this a different way.

• 11:10Speaker 00

  Okay.

• 11:10Speaker 01

  Because I'm not a fan of overrated or underrated and I'm not like, I'm not a fan of passing judgment on stuff. I just don't like. Far be it for me, I like, one of my big gripes is like, we shouldn't have best paper awards. Like, and I say that having gotten one back in the day, so I feel like I have the ability to say that not just out of bitterness, but out of like recognition that it's dumb.

• 11:29Speaker 00

  Sure.

• 11:30Speaker 01

  But.

• 11:30Speaker 00

  Test of time, though. Test of time is great.

• 11:32Speaker 01

  Test of time is awesome.

• 11:33Speaker 00

  Yeah.

• 11:34Speaker 01

  You know, and you know, I look forward to everybody using lottery tickets in 2029. No, if you're working on lottery tickets, you know, there's a lot of other cool stuff out there. But I think it's really, I'll turn that question into like, what areas should academics be thinking about? I don't know, what would I work on as a PhD student right now? Or what would I recommend a student work on? And all the biggest questions in the field come down to how you measure and how you evaluate. Those are just such fundamental questions. Until we know how to measure things, until we know how to evaluate anything, you can't really even do any science. We don't know what we're even talking about.

• 12:05Speaker 00

  Yeah.

• 12:06Speaker 01

  And so I'm also thinking a lot about like synthetic data. Can we generate useful evaluation sets for all the little properties we want to find about an LLM? Creating data sets is really hard, but a model can help us do that. So I'm kind of curious, like, you know, can we bootstrap the evaluation process with synthetic data? Figure out good ways to help ourselves build good data sets and then, you know, from there, maybe we can start to really take a bite out of the evaluation questions and get moving on the actual science of understanding what's going on with these LLMs. All that seems very academically viable. None of those require huge amounts of compute. They require creativity, ingenuity, but that's in abundance in academia, even when compute isn't.

• 12:44Speaker 00

  Yeah, I would say that that's actually one thing I've had a big delta on for this year.

• 12:48Speaker 01

  Yeah, tell me more, I'm curious.

• 12:50Speaker 00

  Synthetic data, I always thought it was, you're just kind of sampling from a known distribution anyway, that you know is imperfect and doesn't match human preferences.

• 12:58Speaker 00

  And it's Kanjun from Embu that actually changed my mind on this.

• 13:01Speaker 01

  Oh, tell me more. That's a, that is a smart person you're talking to.

• 13:04Speaker 00

  She's like, you actually don't want to match human preferences. You want to, you want to spike it in different ways and useful ways. And so you want to synthesize data in useful ways that don't necessarily match human, human preferences. And once she said that, I was like, oh, okay, I think I'm actually sold on this as a viable practice.

• 13:20Speaker 01

  I would actually make a completely different argument, but she's right. So I'm probably going to make a wrong argument now because Kanjun is pretty much always right. And, you know, when she disagrees with me, it means I'm wrong, but. You know, the way that I look at it is synthetic data is not about like, it's not about relying solely on the model. Like, we as computer scientists love the idea that once you automate something, you fully automate it.

• 13:41Speaker 00

  Okay.

• 13:42Speaker 01

  It's really about like, how do you reduce the amount of work necessary to create something that's truly useful? And so synthetic data is not about can we like whip up a dataset automatically and then make a model better. It's about how can you use human time most effectively? And maybe labeling data or creating a dataset from scratch is not the most effective use of human time. Maybe it's curating a dataset that a model generated, you know, to pick the examples you like most and edit a few of them. When I think about the millions of different small properties of LLMs we want to study, like, in some sense, the unit tests of LLMs that we want to develop, you know, that's going to require a bunch of tiny eval sets on specific, really niche things. It's really hard for a human to just write from scratch. Nobody has the time or patience for that. If a model can help you do it and you can curate,

• 14:26Speaker 01

  you don't end up in a full feedback loop. You have a human there, but you're just making better use of your mind.

• 14:31Speaker 00

  Yeah, makes sense. I would just observe that this sounds like weak labeling. And I talked to Raja Hibi from Human Loop, who actually pivoted away from weak labeling.

• 14:40Speaker 01

  Interesting, tell me more.

• 14:41Speaker 00

  I don't know, I just think it might, it might have just been too early. I'm still a believer.

• 14:45Speaker 01

  This is, this is the thing about all of deep learning, like, you never know whether you're too early, and too early is often six months too early. It's no longer the like, you know, Yoshua Vencio and everybody being 20 years too early.

• 14:57Speaker 00

  Ashwin Huber.

• 14:58Speaker 01

  Ashwin Huber, of course, we have to salute, you know, Ashwin Huber as well. It's not like being 20 years too early. It's like, you might be six months too early and some crazy thing is going to happen.

• 15:08Speaker 00

  Oh, God.

• 15:09Speaker 01

  Or like, something will finally click.

• 15:10Speaker 00

  Yeah.

• 15:11Speaker 01

  And there goes that.

• 15:12Speaker 00

  Yeah, yeah, totally. Cool. We're almost at probably your destination. The workshops tomorrow, you said are like kind of the highlights for you for near reps?

• 15:19Speaker 01

  Yeah, yeah.

• 15:19Speaker 00

  What should be my workshop strategy? You know, I don't, I haven't, I picked out a few, but.

• 15:23Speaker 01

  Oh, wander.

• 15:24Speaker 00

  How do you do in Europe as well, basically?

• 15:26Speaker 01

  Wander and go to a lot of the poster sessions. Like, the talks at workshops are always great, but you know, often, honestly, the workshops are pretty eclectic in terms of talks. You try your best as a workshop organizer to put together a coherent program, but you know, presenters are gonna do what presenters are gonna do. And you can't really stop that. But instead, you know, I love the poster sessions, because like, you get students who are working on like, really crazy creative stuff that isn't even ready for the conference yet. Like, you're actually seeing things that have not been put out on Twitter yet. And that's such a nice change from NEREFs, where all of conference papers have been out for months, if not longer.

• 15:59Speaker 00

  Oh wait, I observed the opposite. Things that have been on Twitter for like, forever are now out of date, and there are posters because that's how long it takes to submit a paper.

• 16:06Speaker 01

  Yeah, yeah. But for the workshop poster sessions, it's the workshop poster sessions that are awesome. Because you're truly seeing stuff that was created this fall, may not be in art yet, nobody's talked about it, probably makes no sense yet, but may evolve into something really cool.

• 16:21Speaker 00

  Interesting.

• 16:22Speaker 01

  And so, and you also like, there's not as much competition to talk to the people, you can just kinda chill. So I love to like wander from poster session to poster session throughout the workshops. Because like, that's my favorite part. I don't know, I can hear, you know, somewhat important people talk anytime. But it's like, talking to the people and seeing, like, getting a glimpse of what might be ahead. You know, being able to say like, "Oh my gosh, I remember seeing the poster for this paper that a year later becomes very important." And like, kind of asking yourself, you know, "Is this nonsense, or is this brilliant?" And like, not actually knowing the answer, having 50 million people on Twitter having told you the answer. That's kind of, I don't know, it's fun. It takes me back to like, what the conferences were like for me, you know, when I was early in my career.

• 17:02Speaker 01

  Like, you know, it was just kinda some random people coming and chatting with me, and you never really knew what was important and what wasn't, but it was all kinda cool and fun.

• 17:09Speaker 00

  You use a formula hypothesis and, you know, search that way. Yeah, so I'm looking forward to tomorrow. If you find anything interesting, just let me know and I'll go interview with them. I've been recording sessions with poster presenters all the time. And I wanted to expose people who don't come to NEREFs, like, that this is what goes on. And there's so much I found, so much talent. That that does a lot of work that you don't hear about them online because they're just not online or they just don't have the the reach that you know, and I do.

• 17:34Speaker 01

  Yeah.

• 17:34Speaker 00

  So like, I want to give them that reach.

• 17:36Speaker 01

  Yeah, I think there's like, you know, I'll say two things kind of to close up. One is kind of that like, I feel like there's now so much hype attached to NeurIPS and ICL and ICML just by virtue of the hype that's attached to the field.

• 17:48Speaker 00

  Yeah.

• 17:48Speaker 01

  I don't know, this like, feels pretty mundane and boring to me, like, it's a, it's really cool, but it's also just, you know, it's just a bunch of academics, like, walking around having boring conversations, getting coffee and like, pretending to party. I, I definitely, my experience of

• 18:01Speaker 00

  Pretending to party, I love it.

• 18:02Speaker 01

  No, I'll say that, you know, I'll tell you

• 18:04Speaker 00

  It's true, it's so true.

• 18:05Speaker 01

  My experience of NeurIPS last year, like, I don't know, these conferences have a reputation of being over the top with industry parties and things like that. And my impression was that was probably true in 2017. Like, that year is known as the NeurIPS that broke NeurIPS. Um, for various reasons, I wasn't there at that time, that was before I was even in the field. But my experience last year, especially post pandemic, was a whole generation of students had like, heard stories and these stories had been built up in their minds. And they were trying to live out the fantasy of what they thought NeurIPS had been like. So these very boring happy hours, people tried to turn into ragers and it was hilarious. It was just adorable in some sense. Um, so you know, it's worth remembering like, you know, there's the fantasy and there's the reality and the reality is, you know, it's a boring industry conference where people are, or academic conference with some industry component where people are trying to make money

• 18:50Speaker 01

  and convince people to look at their posters and get a few citations and

• 18:53Speaker 00

  Lots of hiring, lots of hiring.

• 18:55Speaker 01

  Lots of hiring.

• 18:56Speaker 00

  Lots of hiring.

• 18:56Speaker 01

  Um, I think things have really settled into a new normal. And, you know, with all the hype and all the craziness over the past couple of years, people feel like everything is just exploding and changing all the time. Like, you see those LinkedIn posts of, "Everything has just changed."

• 19:09Speaker 00

  I hate those.

• 19:10Speaker 01

  I, I hate those so much. I hate LinkedIn. Um, if anyone is a LinkedIn influencer, I hate you.

• 19:16Speaker 01

  But, you know, it's kind of like, this felt like, okay, like, maybe there's a steady state again, maybe we can all catch our breath a bit. And it, it kind of felt like after a pandemic, after all the technical development that's happened in the past couple of years, like, It's nice. We can chill. Like, we can kind of breathe a little bit. And there's something really nice about that.

• 19:33Speaker 00

  Yeah, love that. Well, it's so nice to have you on again and chat and catch up.

• 19:37Speaker 01

  Thank you so much. It's good to see you.

• 19:40Speaker 00

  In case it wasn't obvious, that was not up to the usual standards of our recordings because that was a walking interview. I was carrying these portable mics all over NeurIPS. And really, the only way to schedule podcast interviews with people, especially busy people like John, at NeurIPS is to show up with a portable mic, shove it in their face, and talk to them. And that's what the majority of the hip podcast conversations are for this episode because that's the only way I can like, I see someone, grab someone, "Do you have fifteen minutes?" and talk through something. That's what happens. And that's how we schedule interviews with a whole bunch of people that we would not get otherwise. NeurIPS is too chaotic to schedule anything else otherwise. One takeaway from John's interview, which I want to highlight, apart from the whole, it's the new normal conversation,

• 20:22Speaker 00

  it's the focus on synthetic data generation. This is a recurring theme that is continually coming up from my conversations with literally everybody in this space. And how do you do it right? How do you do it with the blessing of OpenAI? ByteDance was recently banned from OpenAI because they were considered to be distilling from GPT-4, which is not allowed under the Terms of Service. I've heard that they're not the only company that is accused of or being thought of or rumored to be doing that. Probably the right approach is something that looks like DeepMind's approach, which on Monday of NeurIPS published a paper called "Beyond Human Data: Scaling Self-Training for Problem-Solving Language Models." And the concept is honestly not that complicated. For the domains of math and for coding,

• 21:05Speaker 00

  they were able to computer-generate data for training on. And they found that when training POM 2 on that synthetically-generated data improved their results and performance on the benchmarks for those relevant domains. It makes sense that, you know, we can scale beyond human data on those dimensions. That's the trivially easy stuff. And the question is, how do you scale beyond the verifiably correct? If you listen to part one of our NeurIPS coverage, we talked about DPO, which is more efficient usage of existing information. So it's not exactly using synthetic information. But just as Sneak peek of 2024. We've actually already recorded an episode with Nathan Lambert, now of the Allen Institute,

• 21:44Speaker 00

  on RLHF and RLAIIF. And I think those approaches might scale beyond just the narrow domains of math and code. So next up is someone who's new to the pod, but not new to me. I've talked with Lynn from Fireworks a bunch over the past few months, and they've definitely blown up in the inference space. So in some sense, you can think of Fireworks as a competitor to Together AI or Replicate, or any other sort of inference-serving platform that you might think about. But they have a really good team, and they've been doing some very good work with Ms. Ral. Lynn and her team have an amazing track record, which you hear about in the interview, and their customer list is pretty stellar, too. So it's worth checking out and checking in on the inference business with Lynn Chao from Fireworks AI.

• 22:25Speaker 00

  Rewind, we can do all that, because this will be edited. Okay, so who are you and what is Fireworks?

• 22:31Speaker 02

  Hey, Sean. We started Fireworks last year, and me and a few founding engineers, we have been working at Meta on building Air Platform and specific PyTorch for five years. When we started PyTorch, it was a framework for researchers, and we took the mission to build one framework for both production and research, and streamline research-production transition, operating PyTorch at huge scale for Meta and for the industry. So by the time we left last year, it is running more than 5 trillion inference per day across 50 data centers for Meta. And we feel like this is a great impact we have landed. But when we look at the industry, it's really, really behind.

• 23:16Speaker 02

  And we founded Fireworks to really bring this expertise to help industry adopt AI in the festive way, adopt the state of our best research into production in a very streamlined way. And why Fireworks the name? Because PyTorch holds fire, and we want the fire to be everywhere. That's why we'll come up with our name, Fireworks.

• 23:42Speaker 00

  Nice, nice. Well, there's also Lightning, and Lightning Labs is kind of a spin-off of that effort.

• 23:49Speaker 02

  Right, right.

• 23:50Speaker 00

  And basically, I think there are multiple teams working on like better inference for PyTorch. Could you elaborate on like, how do you see the landscape of sort of inference-as-a-service companies? I don't know if you consider yourself that, like, infrastructure companies in general, I guess.

• 24:05Speaker 02

  Right, so.

• 24:08Speaker 02

  I think when we think about inference optimization, there are different angles, right? I still think like PyTorch team, when I was there and now, you know, the PyTorch team, they are still doing a great job pushing for PyTorch performance optimization across training and inference through the PyTorch compile project. The goal here is to, hey, keep the simple PyTorch programming API, which is really good for researchers, and then take the heavy lifting of doing optimization in automatic way. But then, because PyTorch team to support and sustain a broad community,

• 24:50Speaker 02

  so the workload is much more diversified when they think about optimization. And here, at Fireworks, we take the same philosophy. We want to keep the simple API of PyTorch programming language, and take the heavy lifting of optimization, but more specific target at industry verticals, right? For example, when we started company, we started from ranking recommendation. And we have a product around that. And then later on, our customer we engage with, they're asking us, hey, can we help on GNI? Because all the GNI models are PyTorch models. It's bigger, it's more complex, it's even harder to operate and optimize.

• 25:34Speaker 02

  So then we start a vertical on GNI across large language model and image generation and other modality as well. But because we focus on verticals, so we can't afford to take a much more specialized optimization approach. And that is complementary to PyTorch compile, where PyTorch is driving for a broader audience. So that's where we are. And I would say, because of our PyTorch expertise, we are the best when it comes to performance optimization across the following areas, right? The performance for GNI models are pretty complicated because there's no one bottleneck on system resource consumption point of view.

• 26:16Speaker 02

  The bottleneck can scatter across CPU to GPU communication, the compute to solve memory bandwidth, and many other things.

• 26:25Speaker 02

  So we develop a very special scaling algorithm that allow us to tackle those bottleneck independently instead of blending them together. So that's very unique thing we are doing. The second is we build custom kernels across attentions, especially multi query attention, Matmul, or reduce. And those customer kernels outperform anything in the industry. Yeah, we also do many adaptive technology that just when we run the inference, it performance will get better. The more you run the workload, same workload,

• 27:09Speaker 02

  it will start to adapt to the workload and become better and better. So across all this, and that enable us to be in the leading position of GNN inference provider.

• 27:21Speaker 00

  Just to give people a mental image, obviously, they can go to the website. You have a self-serve option that people can try out. You mostly have a library of existing popular open source models. You just started creating your own models, which we can talk about. I didn't know that. That's super exciting. You actually recently enabled, you enabled Mixtral in one day after the release by reverse engineering the code?

• 27:44Speaker 02

  That's right.

• 27:45Speaker 00

  What's the high level of that?

• 27:45Speaker 02

  Yeah, so I think we did that twice, right? The first time when Mistral 7B got released the same day. They released in the morning, and then in the afternoon, we launched Mistral 7B. I were the first to get work.

• 27:59Speaker 00

  And this is basically like they release weights, but no code. And then you have to implement code by guessing the...

• 28:03Speaker 02

  Right. For Mixtral, that happened last week. They only released the weights, and there's no code. And then I think it's really fun for us, right? So because...

• 28:18Speaker 02

  Thanks to our, the technology we develop over time, we actually build a slew of componentized libraries that enabling new models is not every time built from scratch.

• 28:36Speaker 02

  So because those, all these models share similar kind of model architecture underneath with different components. And, and that's why kind of we have the velocity of the speed. But it was actually fun to hack it.

• 28:51Speaker 02

  Dima, he goes by Dmitry Zhukov.

• 28:54Speaker 00

  You're a CTO.

• 28:55Speaker 02

  Yeah, our CTO. He basically took the Llama model and tried to retrofit to the Mistral weights and it worked. It worked, we were like thrilled, oh, it's actually working pretty well. But on top of that, it was just a base model. It's not an instruct tune model. It's not really usable for chat. And then overnight we tuned a chat model and deployed to pull bots and used by many other users already at high scale. And the feedback is really, really good. Of course, now we switch to Mistral Insta as the official version, but we still keep getting users feedback. Our overnight tuned chat model sometimes even perform better.

• 29:38Speaker 00

  Wow.

• 29:38Speaker 02

  So, yeah, so that's what we do when it comes to the velocity of quality and velocity to high speed. We are the best company in the industry.

• 29:50Speaker 00

  Yeah, mentioning speed, I should also mention that a lot of AI engineers listening on the podcast would be familiar with the Versal AI Playground, which you are the primary provider for, right? I mean, that's the one that's most visible because they name you. But I don't know if there's any other that you serve that you can name as you're the sort of inference provider.

• 30:06Speaker 02

  Here's just kind of very highly selective list of the customer.

• 30:10Speaker 00

  Not excessive.

• 30:10Speaker 02

  Yeah, we, we get the marketing rights. So we already served home. And they're doing really good PowerPoint generation. If you haven't used that, please try it out. It's really cool.

• 30:22Speaker 00

  Yeah, I used it for my keynote for my conference.

• 30:24Speaker 02

  Oh, that's fantastic.

• 30:25Speaker 00

  Yeah, it's it's basically I use like a magic trackpad to to serve the tome. And then obviously, whenever I need to generate images, I actually generate it from inside of Tome. So I was using fireworks without knowing it.

• 30:35Speaker 02

  That's fantastic. We also serve the co co pilot kind of application. For example, Sourcegraph released Cody.

• 30:44Speaker 00

  And we by this time, by the time this releases, we'll release the our episode with Sourcegraph and Stevie Agui.

• 30:49Speaker 02

  Oh, that's great.

• 30:50Speaker 00

  Yeah, we recorded one. We're good friends.

• 30:52Speaker 02

  We also are the inference backend provider for Pole. That is a very popular chatbot and and Pole is building.

• 31:00Speaker 00

  Wait, doesn't it just Anthropic or GPT?

• 31:03Speaker 02

  At the beginning.

• 31:04Speaker 00

  Okay, now they have their own models.

• 31:06Speaker 02

  They are going big on open source models.

• 31:08Speaker 00

  I see.

• 31:09Speaker 02

  To provide a variety of different solving, different.

• 31:14Speaker 00

  Okay.

• 31:14Speaker 02

  Experiences, bring different experiences and a much better performance. And of course, from their point of view, like cost efficient. There are many other big enterprises, for example, with DoorDash, and they're using us.

• 31:27Speaker 00

  Did they say for what?

• 31:28Speaker 02

  Yeah, so we we actually, yeah, we release ranking recommendation stack with them to power their main business. Because when you go to their website, there are a lot of ranking recommendations happening, including ads and kind of

• 31:46Speaker 02

  restaurant search recommendations and so on.

• 31:48Speaker 00

  One thing I wonder about is for something like a DoorDash, and you know, I'm I'm a bit newer to Rexus in general. Shouldn't those be pre-computed? Like, why does it have to be fast or live? It shouldn't, it doesn't have to be live, right?

• 32:00Speaker 02

  Actually, there are a lot of dynamism, right? Because your your personal preference may change, right? It's also a quickly learning. And their distribution channel, their participating restaurant may change, their menu may change. There are a lot of dynamism in the matching criteria here. And as I worked at Meta for a long time, to actually do highly adaptive ranking recommendation, personalized ranking recommendation, yield the best performance when it comes to the relevance and revenue.

• 32:36Speaker 00

  Yeah, I'm just asking like offline versus online. I don't know how sensitive this is to latency requirements.

• 32:43Speaker 02

  Oh, yeah, yeah. I know. So. A lot of time, most people like, of course, at, you know, big companies, people do online training. But for those enterprises, I haven't seen the need to go online training yet. So usually, training is offline and then, but it's periodic. You have to refresh with new information and then you launch and deploy periodically. Yeah.

• 33:10Speaker 00

  Okay, and so I teased this earlier. I didn't know that you had your own models that you're also training. So you just released a clean Lava.

• 33:19Speaker 02

  Yeah.

• 33:19Speaker 00

  What's the story behind that?

• 33:20Speaker 02

  Right, so I think everyone knows like GPTV and the kind of the space of multimodality, right? I think as I talked about in one of the interview when I was at Meta for PyTorch, at the end, the moderator asked me, hey, what I think is future? My answer is multimodality. Because we lived in the whole world that it has so many different modalities across image, audio, text, video, and so many other things. And that is the mix of our world and the real world experience. So yeah, we really, we really think multimodality will be a very important aspect. So and we take the very popular Lava model from Microsoft,

• 34:06Speaker 02

  but it's kind of, it has the kind of GPT4 training data. So we replace that with our own training data and make sure it's commercially usable. Yeah, we're super excited about this.

• 34:19Speaker 00

  Yeah, I mean, it sounds like you'll be exploring more models as well and just putting all your platform. And you're the fastest way to access them. We're here in Europe, you're talking to a lot of industry folks. Any other top of mind conversations that you're just hearing a lot that may be surprising to people?

• 34:33Speaker 02

  So I mostly talk with many startups that is emerging. So number one, it's really refreshing to me that, but not surprising, that there's so much product innovation that's happening across the board. So much energy there, but a lot of those are built on top of GNI. Of course, it's not surprising, but it's kind of validating fundamentally, innovative technology can reboot a huge part of the industry. So that's really, really refreshing. The second is, I think there are

• 35:14Speaker 02

  A lot more hey, how we think about working together, right? How we build a bigger, more interesting product for broader audience together? I think those conversations is very, very interesting to me.

• 35:30Speaker 00

  Yeah, yeah. Okay, very cool. And you're also here to hire or recruit?

• 35:33Speaker 02

  Oh, yeah, absolutely.

• 35:35Speaker 00

  Who are you looking for? What's the profile?

• 35:36Speaker 02

  Yeah, we are, we're definitely growing very fast as a company. We are looking for system engineers. Hey, we want to, we are, we already have a rock solid infrastructure but we are scaling it quickly and aggressively. So anyone with cloud infrastructure experience move really fast, join us. We are also looking for researchers who has a lot of experience and understanding data a lot, understanding quality a lot, can get to kind of quickly help our customer get to high quality. And whether through training our own models or fine-tuning the models

• 36:20Speaker 02

  and the building task-specific fine-tuning services. Those are the areas we are pushing really aggressive on. And of course, we are hiring across the board of go-to-market people, all the way from marketing, solution architects, sales rep, and so on.

• 36:37Speaker 00

  Yeah, yeah. It seems like you're scaling very quickly. Thanks for coming on.

• 36:41Speaker 02

  Oh, thank you for having me. Cool.

• 36:44Speaker 00

  When I first met Fireworks, I was very impressed by your team. But since then, I've been more impressed by the execution. And my guess is that this will not be the only time that you'll hear about them on the Lanespace Pod. So far in organizing and editing this podcast, I've been trying to bias towards reintroducing previous guests of the pod as a form of, you know, end-of-year check-in episode with friends, right? But so many of them actually mentioned Fireworks. You'll see later with Cursor and Perplexity that I had to put Fireworks first just because that many people have interacted with them, used them, and loved them or compete with them. I think it's a really interesting open question as to how much mote any one inference or commodity infrastructure provider can have. The people who are not in the business say there's no mote and the people who are in the business like Lynn see tons of mote in the software that they write, which obviously

• 37:29Speaker 00

  is proprietary to them. It's also interesting to see them start training and releasing their own models. And Fireworks released a lava variant, which we previously covered in our previous New Earths episode as one of the best papers of 2023. So I highly encourage you to check out that conversation with Altean if you're interested. So I say all that to preface the conversation that we're going to have with the next two guests. The first is a return guest, which is Aman Sanger from Cursor.SO. We had them on in August to talk about their amazing rise to power as the AI-first code editor. They've definitely exploded all over my timeline. And at the time of the interview, I myself was a VS Code, Codee, Codium, Copilot, Codium fan. And since then, I've actually switched my own workflow over to Cursor because of the better workflow that they provide. But still, there's a lot of open questions around their business.

• 38:14Speaker 00

  Just like Mosaic, during our podcast interview, they were actually sitting on a fundraise and they recently announced their fundraise with OpenAI. So let's check in on Cursor. Okay, cool. So I'm back with Aman. Hey.

• 38:27Speaker 03

  Hey, how's it going?

• 38:28Speaker 00

  Hard to catch you. You're a difficult man to find.

• 38:30Speaker 03

  I guess so.

• 38:31Speaker 00

  You've been exploring Europe and you also announced your fundraise since our last episode.

• 38:34Speaker 03

  Yeah, so we raised $8 million from OpenAI. They've been a fantastic partner and I think it was a great decision.

• 38:40Speaker 00

  Yeah. OpenAI use you themselves.

• 38:42Speaker 03

  Yes, we have a lot of OpenAI users and we're growing pretty fast inside Dior. The thing that we like to say is like Cursor is the means by which research happens faster, right? Like as we make programming happen faster and faster, as we make programmers much more efficient, we're making researchers more efficient. And the bottleneck for research is really just implementation. If you can come up with an idea and then actually have the code, have the experiment all written for you immediately, research will just happen much faster. And so that's the goal that we're working towards and I think, you know, we're Tiny bit of the way there with a lot of OpenAI users.

• 39:18Speaker 00

  Yeah. What's the funniest or most interesting sort of feedback you get from OpenAI people versus regular coders? Like, do they prompt differently because they work at OpenAI?

• 39:26Speaker 03

  So they actually probably have less feedback than some of our other users who are less familiar with language models. Because they know what the deficiencies are. They kind of know what's going on underneath the hood.

• 39:34Speaker 00

  Yeah, you can probably give them interesting input on what people are trying and failing with.

• 39:39Speaker 03

  Yeah, that's true. We do give them a lot of feedback on a lot of their early alphas and whatnot.

• 39:44Speaker 00

  And so you've been tearing up the Twitters recently, putting in some effort. What are your sort of top messages that have been really resonating with people?

• 39:51Speaker 03

  I was a big fan of the KV caching tweet. It's surprising that not too many people, it seemed like not too many people knew about this before.

• 40:00Speaker 00

  Yeah. So when people learn about transformers, it's actually not in the documented literature and the academic side of things that KV caching is a common industry practice.

• 40:12Speaker 03

  Yeah.

• 40:12Speaker 00

  You only find out when you talk to industry people that.

• 40:14Speaker 03

  Yeah.

• 40:14Speaker 00

  It's KV cache.

• 40:15Speaker 03

  So like when you say KV cache, it's really confusing because the KV cache, like the KV cache can be cached, right? It's like almost like a double caching. But the key idea here is, well, let's look at all the big closed model providers, right? They all have like these chat models. And with chats and with conversations, like the first end conversation messages are always fixed. And that means like the first, let's say, like end tokens are going to be fixed. And that means when I put the next token in, why do I need to redo all the work of recomputing the keys and values for those first end tokens?

• 40:47Speaker 00

  Yeah.

• 40:48Speaker 03

  And a standard inference trick for this is you take those keys and values and you move them from GPU RAM to CPU RAM.

• 40:57Speaker 00

  Yeah.

• 40:57Speaker 03

  You sort them then, sort them there for some period of time before they're evicted. And then if another request comes in with a matching prefix, the matching original conversation history. You just load those back into GPU RAM and you save a ton of time on compute. Your time to first token goes. And then because you're saving on compute, you can increase your throughput. And this is a trick that you don't really see in any of the open source.

• 41:19Speaker 00

  Yeah, so you don't see that, but people implemented on top of it.

• 41:23Speaker 03

  Yes. Well, I'm understanding like I think Together, for example, I think is implementing this.

• 41:28Speaker 00

  Yeah, and I've just talked to Lin Tao from Fireworks as well.

• 41:31Speaker 03

  Yeah.

• 41:31Speaker 00

  Doing that. So one of the interesting, I always assume that it's because of personalization. Like, hey, in my system prompt, I have today's date. I'm gonna have to update that once a day. Fine, like no big deal.

• 41:42Speaker 03

  Yeah.

• 41:42Speaker 00

  But maybe if people have more customized prompts. But you know, you said there's some kind of cache eviction policy where if there's like a 95% match, you use the cache.

• 41:53Speaker 03

  Yeah, like I don't know what the exact eviction policy would be. You could probably use like, assume you have, I don't know, like 100 gigabytes of space per device. Probably a lot more, actually. Probably up to a terabyte of CPU RAM per device or maybe per machine. You could just do something like least recently used. And then if you start to use up more space than exists on device, you just evict the least recently used request.

• 42:18Speaker 00

  You are a consumer mostly of the GPU4 API.

• 42:21Speaker 03

  Yes.

• 42:22Speaker 00

  They don't really expose this.

• 42:23Speaker 03

  They don't.

• 42:24Speaker 00

  Any API. How does this affect you?

• 42:26Speaker 03

  I think it's actually pretty important to understand what's going on underneath the hood to take like advantage of these things.

• 42:33Speaker 00

  Yeah.

• 42:33Speaker 03

  So like we use dedicated instances.

• 42:35Speaker 00

  Yeah.

• 42:36Speaker 03

  And.

• 42:37Speaker 00

  So they expose their capability to you.

• 42:39Speaker 03

  Like somewhat, but like the key thing is they expose very little, actually. And you can, I mean, yeah, but like the only way that you can really take advantage of this. And I kind of had another tweet about this is like you need to really understand what's going on underneath the hood. So like you can then plan for when memory utilization is spiking based on how many tokens you're currently using or how much memory the instance you can speculate is. Or when are you getting a lot of cache hits? So you don't expect to be using as much compute, which means you can then increase your throughput without worrying about things going latency spiking or things going down.

• 43:15Speaker 00

  Yeah. And I don't know if you've, I've taken this thought to quite an extreme level. Like you can use this to cache rag stuff, like rag results.

• 43:23Speaker 03

  Yeah.

• 43:24Speaker 00

  And just general prompts, right?

• 43:26Speaker 03

  You can, you can. So I did have another tweet about this where there's, no one's done this to the best of my knowledge. And I think this would be very, very hard to do.

• 43:35Speaker 03

  But you could technically cache the entirety of some corpus.

• 43:40Speaker 03

  In something like S3, if you have a model which has smaller sized keys and values. So this would be, instead of full multi-head attention, it could be something like grouped query attention, which is, I think, usually around 8x smaller, or even multiquery, which can be 64 to 256x smaller. And so then what that means is you can actually read the weights from blobs, or if you have everything like really optimized. You can read it into RAM a decent bit faster than it would actually take to compute, recompute the key to KV cache. I think that will be very tricky to implement. And I think there are actually not too many use cases where it would be useful. I think codebases, there's actually one where it could be.

• 44:23Speaker 00

  Yeah. My, my final like observation on this is OpenAI had the opportunity to offer caching to people with the assistance API. And again, they're charging you for the whole thing every single time you send a message to the assistance API.

• 44:38Speaker 00

  And like, isn't like, I, I find it like, is there, is there some explanation? Is there, is it just like a, we can do it, so we're gonna do it?

• 44:44Speaker 03

  I mean, it's tricky when you're not like using, I don't know what they're doing underneath the hood, but if you assume they're doing something like caching at a machine level, like

• 44:54Speaker 00

  So I, I assume, I assume they're not, they're serverless, right? So you have to load and load and that costs, causes a cold start. And that's a problem for, for them.

• 45:01Speaker 03

  Yeah. So it's like really trivial. When you have like server endpoints, server-based endpoints, or like dedicated instances.

• 45:08Speaker 00

  Right.

• 45:09Speaker 03

  It's probably quite tricky to get right. I, I mean, I, I'm not really confident as to like what their decision-making was there.

• 45:14Speaker 00

  Yeah.

• 45:15Speaker 03

  But I'd imagine it's much more difficult to get right.

• 45:17Speaker 00

  Got it. What was your second tweet that you, that we prepped?

• 45:20Speaker 03

  Uh, well, one of them that I, I thought was interesting was generating a treble dataset.

• 45:25Speaker 00

  Yes, synthetic data.

• 45:26Speaker 03

  Using synthetic data. I mean, the key thing here is, there's a lot of using synthetic data to, like the outputs of models to actually train weaker models. And so a lot of people have done this with GPT-4 outputs. This is actually, I think, that requires like, I guess, the claim that you can train on GPT-4 and outputs and, and you'll still get like pretty good models out of that.

• 45:47Speaker 00

  Yeah.

• 45:48Speaker 03

  Yeah, which, which seems reasonable, but we're actually relying on a weaker claim. Because all we're doing is, um, I mean, people can check out the tweet to see it in more detail. But GPT-4 is quite good at this task of ordering. For like four candidate documents given a query as to the relevance of the query. Right. That's like there've been papers that show this like listwise re-ranking and it works really well. So if you do that for enough documents and you do it in an efficient way, which we kind of use a variant of Elo called Prue-Skill to do. You can then get a really high quality re-ranking dataset, a really high quality ordering over, let's say, a hundred candidate documents.

• 46:29Speaker 03

  Given some query. So we use GPT four kind of in the loop for doing a bunch of different synthetic data stuff. This is this is one of them. And I feel like more people should be doing it for this kind of stuff.

• 46:40Speaker 00

  Yeah, yeah, I I think people are exploring synthetic data a lot at the at the back half of this year for choosing like models as judges. Models as synthetic data generators.

• 46:52Speaker 03

  Yeah, I think a model's as judges is like almost certainly going to work. If you like use chat, it's a very easy task. I think this is a very easy task.

• 47:00Speaker 00

  This is how we do RL AIF.

• 47:02Speaker 03

  Yeah, yeah, though it's interesting, RL AIF, I was looking at that paper again. And it seemed to really be good for if you if you look at it compared to RLHF, it was it helped with harmlessness. It didn't. I I I don't believe it actually helped in helpfulness.

• 47:17Speaker 00

  No, but it it helps to achieve the Pareto optimal trade off, which is no decline in the other two.

• 47:22Speaker 03

  I think if you compare it to RLHF, it was it was pretty neck and neck. I don't think there's a statistically significant difference with with helpfulness at least. But it is interesting like RL AIF is just effectively getting better at censoring the model. Rather than improving its like almost like capabilities, right? It's it's helpfulness while RLHF like like it'll do it as well as RLHF, but it doesn't offer anything additional there, which kind of makes sense to me.

• 47:46Speaker 00

  Yeah. First impression is the nearest.

• 47:49Speaker 03

  I mean, very interesting. Lots of very smart people. I've had lots of very interesting conversations. I'll probably be back next year.

• 47:56Speaker 00

  I was kind of lukewarm on it coming in because everyone goes like, oh, it's a big conference. Like it's it's hard to navigate and all that. But then you like run into a few papers, people, authors that are interesting. And then like you're here, like a bunch of other people I want to meet are all here. Like it's a nice way to. Uh get everyone in one place and just

• 48:11Speaker 03

  Yeah.

• 48:11Speaker 00

  Catch up on everything. The house parties are fun. Uh yesterday was just a lot of parties.

• 48:16Speaker 03

  Yeah.

• 48:17Speaker 00

  And yeah, I don't know. It's it's to me, it's to me, it's a very overwhelming, but um I think the more uh uh exposures or epochs that you have on New Earths, the the better. Um and I'm basically trying to doing this audio experience to try to bring people in because there's many people who will just never come.

• 48:31Speaker 03

  Yeah.

• 48:31Speaker 00

  But they should get a sense of what's going on here. Like, I find there are people here who you've never heard of on Twitter. They're not on Twitter. They just know more than

• 48:39Speaker 03

  Yeah.

• 48:39Speaker 00

  Because they've just done the work.

• 48:40Speaker 03

  Exactly.

• 48:40Speaker 00

  They read every everything.

• 48:43Speaker 00

  Have you seen the data comms uh paper?

• 48:44Speaker 03

  Uh.

• 48:45Speaker 00

  I'll I'll I'll walk you over and show you, but I was very impressed by their work. Uh like, um these people, they they just come out of nowhere and once a year they do this.

• 48:52Speaker 03

  Yeah.

• 48:53Speaker 00

  And this is the place to find them. So that's why I'm here.

• 48:55Speaker 03

  Yeah, I mean, I completely agree.

• 48:57Speaker 00

  Yeah.

• 48:57Speaker 03

  Right. There's rarely such a good congregation of like very good researchers, right?

• 49:02Speaker 00

  Yeah. Are you trying to hire them? Uh let's make a hiring call.

• 49:05Speaker 03

  Yeah, I mean, look, I think right now we're very small, very strong team.

• 49:10Speaker 00

  We were five last time.

• 49:11Speaker 03

  Uh, yeah, so we are seven now. Um only six engineers though.

• 49:16Speaker 00

  Yeah.

• 49:16Speaker 03

  So very small.

• 49:17Speaker 00

  You're more millions than people.

• 49:20Speaker 03

  Yeah. Um, look, we're a very small train team and we're looking to grow the team. We're looking to grow it like very carefully and slowly because I think a lot of Uh companies fall to the pitfall of hiring too quickly.

• 49:31Speaker 00

  Yes.

• 49:31Speaker 03

  Um, so yeah, we're really looking for fantastic people. We're we're seeing like Um incredible traction, incredible growth. There's a lot more really interesting problems to tackle and Um people should ta- check out our blog post on that because I think like it's very exciting.

• 49:46Speaker 00

  The fundraising post?

• 49:47Speaker 03

  Uh, yeah, there's a fundraising post and then we kind of link there. It's there's a problems post. If you go to any sphere dot post slash problems twenty twenty three.

• 49:56Speaker 03

  There's lots of interesting work to do.

• 49:57Speaker 00

  Yeah.

• 49:58Speaker 03

  Um, and I think we have a really good chance of being the team that can crack code gen. So it's a really exciting space. I think you'd be joining a very small, strong team. And so, yeah, if you're interested in working with us at Cursor, we'd love to talk. Uh you could just reach out to Amon at cursor dot S H.

• 50:13Speaker 00

  Nice. S H. Okay.

• 50:14Speaker 03

  Yeah, well.

• 50:15Speaker 00

  I thought it was so.

• 50:16Speaker 03

  We might try to get dot A I or dot com. We'll see. We'll see.

• 50:19Speaker 00

  Yeah. Uh, cool. Well, thanks for jumping on.

• 50:21Speaker 03

  Yeah, for sure. Thanks for having me.

• 50:23Speaker 00

  So there again, you see one of the topics that I highlighted from my conversation with John Frankel, which is why I put it at the start, which is synthetic data generation in all its glory. And for Aman and Cursor, they're particularly interested in LLMs as rankers or LLMs as judges. And that seems to be generally a more blessed way than directly distilling the output of LLMs. And you can look out for our episode with Nathan in 2024 to go deeper on that. Another founder that recently raised that is the talk of the AI community, particularly with Guillermo Rausch and Toby Lutka recently endorsing the product, is Aravind Srinivas or Perplexity AI, which started off being, "Maybe we will construct SQL queries for you."

• 51:06Speaker 00

  Then they went to, "Maybe we'll construct SQL queries on our Twitter script for you." And now they've blown up as a potential Google replacement, which is a huge increase in ambition, but they have the web app and the mobile apps to prove it. So here's Arvind with Perplexity.

• 51:22Speaker 00

  And so, congrats on all your successive Perplexity. The two most recent accomplishments which I have seen at least on my feed is, one, you hit a million people on your mobile app. That's huge.

• 51:32Speaker 04

  On both platforms, Android and iOS independently.

• 51:35Speaker 00

  Is that because of your slick video editing skills?

• 51:38Speaker 04

  Actually, we have a good brand marketing designer. But I mean, more than everything else, I think the app's really good, fast. We spent a lot of time on it. In fact, our first rollout of the app was not the great. It was slow, it used to crash, users complained, and we listened to that and like recruited a good mobile team, much faster, more reliable.

• 52:00Speaker 00

  Any technical decisions that drove that? Like, is it React Native that's slow?

• 52:04Speaker 04

  It's all native. There's no, we're not like on one common React stack. And the reason to do that is that's the only way to make the apps feel fast, right? And I believe ChatGPT also does this. Like, they don't use React Native.

• 52:17Speaker 00

  And then the other accomplishment is PPLX online, which you have, which you're showing on the screen here.

• 52:21Speaker 04

  Yeah.

• 52:21Speaker 00

  What are the headline people, things that people should know if they haven't heard of PPLX online?

• 52:25Speaker 04

  Well, it's like the only L M API that has no knowledge cut off.

• 52:29Speaker 00

  Yeah.

• 52:29Speaker 04

  So if you're a developer and you just wanna prototype products that need information from the web or like has no knowledge cut off, this is the only way to do that. And super fast, pretty accurate. You have two versions, a seven B and a seventy B. So seven B is super fast, seven B is like a little slower, but also like better quality. And we plan to bring it up in the context of the Mixtral M O E as well. That's been recently released.

• 52:50Speaker 00

  Yeah, I think you've been pretty transparent that they are fine tuned to Lamatu.

• 52:53Speaker 04

  That's right. We are not in the business of pre-training.

• 52:56Speaker 00

  But like, what do you fine tune for between Lamatu and what do you have?

• 52:59Speaker 04

  Yeah, we fine tune for like summarization, the ability to take a bunch of sources and accurately give you a nice summary.

• 53:05Speaker 00

  And you are, I think, the only provider right now with online access or whatever.

• 53:09Speaker 04

  Yeah, that's right.

• 53:10Speaker 00

  But also like Grok has access to Twitter.

• 53:13Speaker 04

  Yeah.

• 53:14Speaker 00

  Uh, which you don't have, and they don't have, they'll release an API at some point.

• 53:16Speaker 04

  If they release, like, we'll be happy to, I mean, if they release, they'll be happy to use it, right?

• 53:20Speaker 00

  Yeah, yeah.

• 53:20Speaker 04

  Our goal is to just give accurate answers on the web.

• 53:23Speaker 00

  Yeah.

• 53:23Speaker 04

  And Twitter is just one part of the web. Their vision is like Twitter is everything app. We believe like that's the information out there that exists outside of Twitter that's also super valuable. In fact, like you can even make an argument that information outside Twitter may even be a lot more valuable than information within Twitter. Because most of the links that get shared on Twitter are all from outside anyway.

• 53:45Speaker 00

  Yes.

• 53:46Speaker 04

  So it's only like, what do you miss out on? It's like a specific person's opinion. And usually like journalists pick on that and like write web articles. So it's all gonna diffuse, right? Good ideas usually diffuse to the rest of the web. So we're not really missing out much.

• 54:00Speaker 00

  It's a, it's a different source of data.

• 54:01Speaker 04

  Yeah, it's a different source of data and like, also it's all about like, what, what do you want? Are you, are you, is your source a citation, like already highly curated human artifact? Or is it like some tweet? These are all like questions worth asking.

• 54:13Speaker 00

  One, one thing that you do show off, so I was watching you demo just now, you have sentence by sentence citations.

• 54:18Speaker 04

  That's right, yeah.

• 54:18Speaker 00

  That's a design choice.

• 54:20Speaker 04

  Yeah.

• 54:20Speaker 00

  Because realistically, your source articles actually overlap.

• 54:24Speaker 04

  That's right.

• 54:25Speaker 00

  The full paragraph. So why did you choose to impose sentence by sentence?

• 54:29Speaker 04

  That's how we write papers. I'm, I'm an academic. Every sentence you write in a paper needs to have a corresponding citation.

• 54:35Speaker 00

  As, as a user, it can be confusing. Like, when I click that link.

• 54:38Speaker 04

  Yeah.

• 54:38Speaker 00

  Maybe it's like the third paragraph and they're like.

• 54:40Speaker 04

  That's right. We can do better and like exactly navigating you to the right part of the link, but we're looking into all that.

• 54:46Speaker 00

  Yeah, of course. I mean, I do see you as like a search engine first with a very good language model team.

• 54:51Speaker 04

  That's right, yeah.

• 54:52Speaker 00

  Right?

• 54:52Speaker 04

  Yeah, answer engine. I would call it answer engine.

• 54:54Speaker 00

  Answer engine.

• 54:55Speaker 04

  Yeah.

• 54:55Speaker 00

  You are doing a really good job of that. I also noticed in your PPLX blog post that you also talked about the Fresh LLM paper.

• 55:02Speaker 04

  That's right.

• 55:02Speaker 00

  Maybe, could you introduce that? And like, did you talk to the authors? Are they here in Erip's? Like, what?

• 55:07Speaker 04

  I did not talk to the authors. Like, you know, it's not like we took a lot of inspiration from it.

• 55:13Speaker 00

  Okay.

• 55:13Speaker 04

  But it made sense to like, you know, attribute the citation to them and.

• 55:19Speaker 00

  Yeah, the intellectual backgrounds.

• 55:21Speaker 04

  Yeah.

• 55:21Speaker 00

  What do you look for at Erip's, a conference like this?

• 55:23Speaker 04

  Oh, we are, we are here for recruiting good, strong researchers to join our, our team. Especially if they're more focused on like shipping models to a search product, it's used by millions of people.

• 55:34Speaker 00

  Awesome. We'll, we'll talk about your sort of hiring call to action in a bit. I'm also interested in like labs, like perplexity labs.

• 55:40Speaker 04

  Yeah.

• 55:40Speaker 00

  It seems like a place for you guys to experiment with serving models.

• 55:43Speaker 04

  That's right. Yeah, we, like, everybody thinks like, you know, you start as a wrapper and then one magic day, you just switch over from three point five or to like, you know, like your own model. That's not how it works in practice. Your GPUs crash or like your nodes are not working or like Kubernetes doesn't work as expected and like, requests are like not having the throughput required. You, you optimize for latency, but then you are like worse on throughput. So you're not able to handle spike requests. So all these things can happen, right? So you only know about these if you start small and like, serve a playground where people come and test your own infrastructure and see how it holds up. And then take the lessons from there and use it to serve it on production, right?

• 56:28Speaker 04

  So labs are sort of our playground for testing open source models and our in-house models that have been fine tuned for factual accuracy and helpfulness. And it's like a nice way to, for people to test open source models if they're like curious about it. Especially if they think about it as alternatives to chat GPT. And then it's also a nice way for us to like battle test our infrastructure. Same thing goes to the API, like it's not like I believe these APIs are gonna take over GPT three point five APIs or something. But it's a nice way for developers who want an alternative to like explore, especially those who want to use faster small models. Like the seven B models and and it's also a good way for us to know like how we can handle like search requests and things like that.

• 57:09Speaker 00

  Yeah, I mean, so like I want to push back on this, like you said your playground is a way to battle test.

• 57:14Speaker 04

  Yeah.

• 57:14Speaker 00

  But I think you you probably get orders of magnitude more traffic.

• 57:18Speaker 04

  That's right.

• 57:18Speaker 00

  On your main app.

• 57:19Speaker 04

  Yeah.

• 57:19Speaker 00

  Than your side app.

• 57:20Speaker 04

  Look, we can't just directly ship to the main app, right? And you can never simulate.

• 57:24Speaker 00

  It's like your staging environment.

• 57:25Speaker 04

  Yeah, it's like a staging environment. But not just that, it's not just meant to be a staging for, I don't want to like downplay the importance of labs. Labs is sort of one of the fewest places on the internet today for you to go and explore and compare different open source models. And it also tells the user how fast our inference is. We give you all the metrics like tokens per second, the time to first token. It's also a very transparent way to communicate the speed of our infrastructure, which helps us also like recruit good talent for infrastructure.

• 57:52Speaker 00

  Yeah, but you're pretty, I think you're pretty opinionated that you are an app company first.

• 57:56Speaker 04

  Yeah, we are.

• 57:57Speaker 00

  Not an infra company.

• 57:58Speaker 04

  That's right.

• 57:58Speaker 00

  You just happen to have.

• 57:59Speaker 04

  We're not, we're not competing with Together AI or.

• 58:02Speaker 00

  Fireworks.

• 58:02Speaker 04

  Fireworks or like Octo ML.

• 58:04Speaker 00

  Yeah.

• 58:04Speaker 04

  You know, there are like too many of them, actually, honestly.

• 58:08Speaker 00

  What do you think they need to do to win as an objective?

• 58:10Speaker 04

  I think they should raise insane amount of capital and subsidize the cost so much and capture the market. Or it's basically going to be impossible because you're all offering the same thing, more or less. And Nvidia is basically commoditizing it, right? Like with TRD LM and like Megatron and things like that. So most people's stacks are going to get standardized. So then why am I paying you? I'm paying you for the GPUs then.

• 58:33Speaker 00

  Right.

• 58:33Speaker 04

  That's the game you can only pay it, like it's an economy of scale thing.

• 58:37Speaker 00

  Which you're also buying your own GPUs and running your own stack.

• 58:40Speaker 04

  That's right, but we care about buying GPUs to serve our own product more than helping other people serve their products.

• 58:45Speaker 00

  Yeah, what what have you learned being like a, I don't know, I I feel like you're both an infra CEO and a app application sort of product CEO. How do you balance that?

• 58:53Speaker 04

  Yeah, it's difficult, but you know, like you, one thing exists in service of the other, right?

• 58:57Speaker 00

  Yeah.

• 58:58Speaker 04

  Infrastructure exists in service of the product. You always have to remember that. For some people, product exists in service of the infrastructure. Uh, that's not how we are.

• 59:06Speaker 00

  What does perplexity become a year, two years from now?

• 59:08Speaker 04

  Hopefully, like a lot more people start using it as a Google replacement.

• 59:12Speaker 00

  I see. You're already, uh, I read some stats somewhere, you're ten percent of Bing traffic?

• 59:15Speaker 04

  I don't know about that, but uh.

• 59:17Speaker 00

  Some, someone was measuring like a third party, like similar type of thing.

• 59:20Speaker 04

  Yeah, maybe, maybe for, actually for Bing Chat, we might be even, even further ahead.

• 59:26Speaker 00

  Okay.

• 59:27Speaker 04

  Like, there's just perplexity versus Bing Chat, not Bing dot com. Which is crazy, given that they have so much distribution, right?

• 59:34Speaker 00

  Oh, yeah.

• 59:34Speaker 04

  And marketing power.

• 59:35Speaker 00

  But you are more AI native than they are?

• 59:37Speaker 04

  That's right.

• 59:38Speaker 00

  In a sense?

• 59:38Speaker 04

  That's right.

• 59:39Speaker 00

  You are a different search index, like you, you're, you are, you have your own crawlers.

• 59:42Speaker 04

  Yeah, we have our own crawlers and indexes, yeah.

• 59:43Speaker 00

  So like, if I don't want Bing, then I use your, your stuff and maybe you're, you.

• 59:47Speaker 04

  That's right, yeah, that's right.

• 59:48Speaker 00

  That's cool. Okay, hiring, what are you looking to hire? What people, what should people demonstrate when joining you? I think you have a very strong perspective on the kind of culture that you're building.

• 59:55Speaker 04

  Yeah, I mean, we work pretty hard and like, we wanna get stuff done fast. Uh, so if you enjoy like fast shipping cycles and.

• 1:00:03Speaker 00

  Can you give illustrations? Like, what do you mean by that?

• 1:00:06Speaker 04

  You know, every, every two weeks, like, we have some announcements we make. So we work on very clear, precise projects that have like clear deliverables. And we kind of constantly wanna keep improving the product. So as a machine learning research engineer, if you're excited about like training models and shipping them to production for such a useful use case like consumer search and, and, and wanna do it at the same velocity as like, as, as us, like a startup rather than a big company that has to wait for several months to get something to production. That's a unique spot like to be in, right? And you also wanna be part of a growing exponential rather than something that's trying to defend its territory, right?

• 1:00:43Speaker 00

  Like, defend its territory?

• 1:00:45Speaker 04

  Yeah, like Google. Google's defending it, yeah, yeah. So, they're attacked.

• 1:00:49Speaker 00

  Yeah.

• 1:00:49Speaker 04

  So you wanna be in an attack.

• 1:00:50Speaker 00

  Have you heard, like, what does Google say about you? Like, are they interested in buying you?

• 1:00:54Speaker 04

  I think they're, I think they're being pretty appreciative and respectful of the product, right?

• 1:00:58Speaker 00

  Yeah. But like, SGE is not great, for some reason.

• 1:01:02Speaker 04

  Yeah. I, I, I, I saw, by the way, I, I, I don't think Google people are not talented, like, they're incredibly, probably the more talented than we are. I think it's just that, I think it's just that their incentives are not clear. And they might have to cannibalize their own business model to like.

• 1:01:18Speaker 00

  Uh, this is the classic innovator's dilemma, right?

• 1:01:20Speaker 04

  Yeah, exactly.

• 1:01:21Speaker 00

  They, they have a cash cow and.

• 1:01:22Speaker 04

  That's right.

• 1:01:23Speaker 00

  To preserve that. Uh, you will, you don't have ads, but you're, you're serving subscriptions.

• 1:01:28Speaker 04

  Yeah.

• 1:01:28Speaker 00

  And that's the main business model for now.

• 1:01:30Speaker 04

  As of today, yeah.

• 1:01:30Speaker 00

  Yeah. Um, that's it. Well, thank you very much. All the people that we talked to so far, and some of the best founders I know, whether or not they're in AI, are fierce nerds. And Arvind definitely reminds me of the fierce nerds concept. But I don't think I'm the best person to tell that story. Maybe I'll tag in Sean Purry.

• 1:01:48Speaker 05

  Yeah, have you ever read that Paul Graham blog post called Fierce Nerds?

• 1:01:52Speaker 00

  No, what is it?

• 1:01:53Speaker 05

  It's an amazing post. I'm gonna read you a couple pieces of it, but it's one of those like Paul Graham, I think is somebody said this earlier, they go, what's that guy Andrew Tate? They sent some tweet that was really funny was Paul Graham was my Andrew Tate. Like growing up, which is like, so funny. So it's such a funny, it's such a deep cut joke, but if you get it, you're like, it just hits the spot. All right, so he wrote this post and he goes, most people think of nerds as quiet, you know, sort of like diffident people, right? Just sort of like, you know, passive. And in most social situations, they are, they're, they're quiet and you know, they're not the star quarterback in the middle of the gym, right? They're kind of a fish out of water in a bunch of different things. He goes, but this is an illusion because when, that only happens when non-nerds observe them. Cause they're observing them in non-nerdy situations.

• 1:02:34Speaker 05

  So you see a nerd at prom, you just see them as a quiet sort of passive nerd. There's no alpha in them. But in fact, some nerds are quite fierce. Fierce nerds are a small but interesting group. They are extremely competitive, more competitive, I would say, than competitive non-nerds. Because the competition is more personal to them. Partly because they're not emotionally mature and they, they distance themselves from it. But also because there's less randomness in the types of competition that they engage in. Therefore, they're justified in making it more personal.

• 1:03:06Speaker 00

  I'll cut it off there. That's a clip from the My First Million podcast. And that's a story about how Dharmesh Shah, the HubSpot CTO, is a fierce nerd. And I really like that concept because one, it helps to validate that nerds can also win. And why nerds can sometimes win more than regular people. And obviously for more, you can read that Paul Graham essay. But I think Arvind is a fierce nerd and I think Perplexity is a fierce nerd company. They do have competition though. It's not like Perplexity is the only company going after Google, not the only company going after search. One of my favorite parts in, in compiling these ensemble episodes is juxtaposing two competitors next to each other. Or people who disagree have different worldviews. Like you just heard Perplexity, you just heard Arvind dunk on all the infrastructure companies,

• 1:03:50Speaker 00

  including Fireworks, which we just had on. Now, I'm not the right person to tell you who's right and who's wrong, but I know for a fact that they cannot all be right. And that's what's fascinating. That's what makes a market. So next in full disclosure is a personal friend of mine. It's Will Brick from Metaphor Systems. Metaphor launched end of 2022 with an AI search engine narrative as well. But their approach is more of a pre-trained LLM research engine as opposed to Arvind's answer engine. There are all very minor differences in the end. At the end of the day, people want to punch in a query and get results. And Metaphor's approach is different. They are going after the infrastructure play rather than the application plus infrastructure play. And it's just nice to contrast them together. And I'll leave the conclusions to you. What is Metaphor?

• 1:04:32Speaker 06

  Metaphor is a search engine over the internet, but it's better than Google at handling complex queries.

• 1:04:38Speaker 00

  Okay, why is that?

• 1:04:40Speaker 06

  Why is that? Because we train an algorithm from scratch, a search algorithm from scratch, to handle complex queries, basically. It's a totally different algorithm, yeah.

• 1:04:47Speaker 00

  Why are you at NERUs?

• 1:04:49Speaker 06

  I'm at NERUs because we want to learn about all the cool things people are working on and also because we want to hire some crazy good researchers to help, you know, build the future of search.

• 1:04:57Speaker 00

  Metaphor has a search engine. That's what you launched with last year. And then you also released an API. And I've actually been using the API. Like it's actually really good for augmenting LLMs of search. I don't know how much to which you want to lean being in an app versus an infrastructure company.

• 1:05:11Speaker 06

  Yeah, so we're leaning towards the search infrastructure. So we really see ourselves as like, we want people to build applications on top of us. We see the future as like everyone will use LLMs as the interface to everything. And we want to be powering the search to underlie that. I think we want people to build really cool UIs on top of our search. But the hard part and the thing that we're focusing on is really good search results.

• 1:05:30Speaker 00

  Yeah. Can you give examples? Like you have some like really cool examples, like tweets and books and PDFs and stuff.

• 1:05:36Speaker 06

  People really get excited about like researchers working on something similar to them in the Bay Area or something like that. People have actually met-

• 1:05:42Speaker 00

  Oh yeah, yeah. Competitive intel research as well.

• 1:05:45Speaker 06

  People have met people in real life based on searches because the results are so high quality and they're not, you know, SEO spammed in any way. It's just like exactly what you're asking for. That, you know, it's cool to see that like digital information to real world interaction thing happen.

• 1:06:00Speaker 00

  I actually also interviewed Aravind from Perplexity, who I feel like is also in that sort of search domain, but he has, he's, he's less focused on search infrastructure. He's more focused on just being a search engine. I don't know if you like compared yourself to Perplexity in that way.

• 1:06:15Speaker 06

  Yeah, I know. We get asked this a lot. I mean, Perplexity is doing a great job at combining LLMs, you know, with search results and that makes, you know, that it, that does make for a better search engine. That is the future of the, the user interaction. But we're just like more focused on the search results themselves and really trying to handle the queries that, you know, Google Bing are not good at.

• 1:06:33Speaker 00

  Yeah.

• 1:06:33Speaker 06

  And so, I mean, we, we want people to build LLM style interactions on top of our thing as well.

• 1:06:38Speaker 00

  Wait, so you say Google Bing are not good at. Do you think that people will use you in complement to Google and Bing? Or do you just completely replace that?

• 1:06:46Speaker 06

  At least in the beginning. Like we're, we're going to be used in places where Google and Bing don't work well. So I mean, if, if your application wants to know the weather or wants to know like that Taylor Swift song, basically if your application knows the right keywords to search with, then sure, Google and Bing are going to be fine for you.

• 1:07:00Speaker 00

  Yeah.

• 1:07:00Speaker 06

  But if you want to, you know, make these complex, almost metaphorical queries with natural language, uh, which are really the most powerful ones, then you should be using metaphor.

• 1:07:09Speaker 00

  Yeah, yeah. I was actually walking from your, uh, we were walking from your sort of sushi party that you just had. It's like a recruiting event.

• 1:07:15Speaker 06

  I hope the food was good.

• 1:07:16Speaker 00

  It was pretty good. It's weird. I like, I love, love me a little bit of sushi. And I was actually talking to people about your sort of auto prompting feature. Cause a, a lot of people, I was, there was someone from Mid Journey there and they were saying how DALL-E 3 also does sort of auto prompting or rewriting of the prompts.

• 1:07:31Speaker 06

  Yeah.

• 1:07:31Speaker 00

  Is there art to auto prompting? How do you feel that? How do you feel about your auto prompting feature basically?

• 1:07:35Speaker 06

  Yeah, auto prompt is like we convert, we use ChatGPT basically to convert the queries that are come into the search engine into queries that are formatted, formatted for metaphors models. Because metaphor is trained to predict, predict links given text. So the model really, like the best way to prompt Metaphor is to search in a way where a link naturally follows, which can be confusing. So we have this auto prompt that converts into the right format. You could kind of think of Metaphor as in the same state as like what GPT-3 was in. I don't know if you guys remember, but, or if you remember.

• 1:08:04Speaker 00

  It's not instruction tuned, yeah.

• 1:08:05Speaker 06

  Yeah, it's like, you know, two years ago, GPT-3 was autocomplete, so you had to like prompt it in order to get the best output from it. It had a lot of power, but it just had this weird user interface. Metaphor is in a similar situation. The problem is when you RLHF, you like, and we've tried this, like, it does reduce the power of the model. And like, it's just okay to, to keep, because like, often we're using this auto prompt, like it's okay to keep this model the way it is, requiring this autocomplete type of search.

• 1:08:30Speaker 00

  And, yeah, would you call yourself a search LLM? Like, very, very long ago, the original pitch for Metaphor that I heard from you guys was, you're an LLM that predicts links instead of, instead of tokens.

• 1:08:41Speaker 06

  Oh, well, an LLM is, yeah, I mean, LLM is like, uh, like, it's modeling like, uh, yeah, usually like language. And we're not really, we're not exactly generating the links. We're, we're, we search over an index.

• 1:08:50Speaker 00

  They're not hallucinated at all, right? They're actually from an index.

• 1:08:52Speaker 06

  Yeah, I wouldn't call it a search LLM. It's more like, uh, really a search engine. You might even think of it as a research engine. There are a lot of different ways we're, we're trying to explain. I mean, I think we're like using terms that were developed in an old era for a new type of thing. So we might have to invent new words, uh, or wait until they are created.

• 1:09:09Speaker 00

  Yeah, yeah. What else should people know about Metaphor in general? Like, what are the interesting work that you guys are doing?

• 1:09:14Speaker 06

  I think just like the vision is super exciting, and I think people don't realize how exciting the vision is. Uh, basically, the vision is to solve search. What does that mean? It means no matter how complex the query, we should, Metaphor should be able to handle it. So we're talking like, you know, like, AI researchers, similar to you, who are in the Bay Area, who've worked on Rust before, who, you know, went to so-and-so college, who would be a great candidate for, for this startup, whatever it is, like, we should be able to handle it. And like, and language models are powerful enough to understand language, you know, at the level of a human. So you should theoretically be able to make a system like this. It's just a matter of how fast can it be? And we want to make these things like, do all those complex queries really fast. And imagine, imagine if you could do this, imagine if this was, was like possible.

• 1:09:54Speaker 06

  And then you combine that with like, you know, GPT-4, GPT-5, and that's how we want our customers to combine us with, you know, combine us with GPT-4, GPT-5. Suddenly now you have the ability to literally answer any information query, no matter how complex. Like, the entire world's knowledge is at your fingertips. That's like insane. Like, we, we've basically become all-knowing. You know, omnipotent.

• 1:10:13Speaker 00

  Yeah, that's.

• 1:10:14Speaker 06

  Omnistoned.

• 1:10:16Speaker 00

  Omniscient and then omnipotent because knowledge is power, right?

• 1:10:18Speaker 06

  I skipped a step.

• 1:10:19Speaker 00

  No, no, yeah, I can do that sort of Q E D proof of why omniscient equals omnipotent. I am very excited about you guys, you know, I've seen you grow literally from your living room. And it's definitely not over. What's it like having Mimi celebrity CTO who keeps tweeting viral shit?

• 1:10:38Speaker 06

  I mean, I love it. Like Jeff, Jeff literally just goes, Jeff has figured out Twitter. He just knows how to go viral because he has really good takes. And we often throw up a party in response to his viral tweets.

• 1:10:47Speaker 00

  So you want to talk about the Andrew Huberman party?

• 1:10:50Speaker 06

  Okay, so he had a tweet that was like Andrew Huberman has single handedly like destroyed the SF social scene. Cause like everyone, whatever is like sober at parties and go with him early. And so of course we had an anti Huberman party where we know everyone stayed late and we had like, you know, a bunch of beer and like everyone.

• 1:11:04Speaker 00

  Well, my favorite was all over the apartment that we had the party in. You plastered like quotes from Andrew Huberman about how alcohol's bad for you.

• 1:11:11Speaker 06

  Alcohol will destroy your brain and like all these things. I look, I mean, everything in balance, right? Like we don't, we should have fun in life, but also, you know, be safe and everything. But and then he had another tweet about, you know, how, how he was gonna go on a date, but the girl ghosted him and that allowed him to black folks on coding that night. So of course we had to have like a ghosted in SF party where everyone came to code together. Cause you're already gonna be ghosted on Friday night, you might as well code together while you're at it, so.

• 1:11:35Speaker 00

  Yeah, I love that part of the social scene. I think metaphor is also really driving that somehow. So congrats for all you do and it's just nice to check in with you. I've personally been enjoying the metaphor approach to L L M search APIs. I've often said this in context of the capabilities of G P Ts. So if you think about it, what are the capabilities of chat G P T as it is today? As well as G P Ts as announced on Dev Day, right? There's the L L M base layer, but then you tack on three core capabilities on top of it, right? One is retrieval and generation where you upload files and then you do rag on it. And second is a code interpreter where you do generate code in a sandbox and then you run code and you correct code. And finally you execute it.

• 1:12:18Speaker 00

  And third is you have a search feature. And so we have a bunch of companies competing for the rag functionality. You can check our episodes mutually with Harrison of Langchain and Jerry of Lama Index this year. There's a bunch of companies competing for the code interpreter capability. There's obviously Replit, but then abstractly, there's also Deno and Val Town. And anyone who runs code is in that game, basically. But what is surprisingly uncontested is open web search. And so far, I think it's perplexity of metaphor that are leading the pack in their different approaches. One, the PPLX API is an integrated LLM plus search API. And then two is metaphor, which is search only. And you kind of bring your own LLMs.

• 1:13:00Speaker 00

  For our next guest, we're actually going to go over to our last return guest, which is one of our most recent hits, which is Jeremy Howard, previously of Fast AI, but now of Answer AI. It seems that all people want is answers and Jeremy doesn't have them, but he has questions.

• 1:13:14Speaker 07

  Outside of the Decibel event, recording.

• 1:13:17Speaker 00

  I realized I had to be the interviewer and I was like, I should probably should not have.

• 1:13:20Speaker 08

  And I had to pick a wine and Sean told me, pick the most expensive one.

• 1:13:23Speaker 00

  Yeah, it's on Decibel anyway.

• 1:13:25Speaker 08

  Because Decibel is paying for it. So the one I'm having is from a $160 bottle and it's really good.

• 1:13:29Speaker 07

  And I did the same.

• 1:13:31Speaker 09

  And I'm not having any wine.

• 1:13:33Speaker 00

  It's just too young to drink.

• 1:13:34Speaker 09

  Yes.

• 1:13:35Speaker 00

  Could we go around and identify voices for people listening? Maybe Tanishq, you want to start?

• 1:13:39Speaker 09

  Sure. My name is Tanishq Abraham. I am the CEO of Medarc, which is a medical AI research organization. I also work as a research director at Stability AI. And I've been collaborating with Jeremy Howard for more than a year, like a few years, a couple of years, maybe. And he's also the president of Medarc and he's been heavily involved in my venture as well, so.

• 1:13:59Speaker 00

  And you have a podcast together, which I really enjoyed.

• 1:14:01Speaker 09

  Oh, yeah, yes. Jeremy had me on his podcast, which is a.

• 1:14:04Speaker 00

  Your first and only episode of.

• 1:14:06Speaker 08

  Yeah. It turns out that maintaining a podcast is hard.

• 1:14:11Speaker 00

  Ah, it's easy. Just shove microphones in front of people's faces.

• 1:14:15Speaker 08

  So I'm Jeremy Howard. This is my voice. And as of today, I'm Jeremy Howard of answer.ai.

• 1:14:22Speaker 00

  Yes, and repeat guest on InSpace. Your last episode did really well in terms of the number of views, so.

• 1:14:28Speaker 08

  Yeah, you guys are good interviewers.

• 1:14:30Speaker 00

  Well, also you drop a lot of spice, which is what we like as podcasters. We also have Jess Leao on for the first time. Hey.

• 1:14:36Speaker 07

  Yes, hello. I'm Jess Leao and I'm a partner at Decibel. Excited to be here. Excited to be providing the wine also.

• 1:14:41Speaker 00

  Standing in for Alessio. Oh, so good. Alessio dished us tonight, right? So you're the better replacement.

• 1:14:47Speaker 07

  Yeah, it's good because in a previous conference, Alessio was wearing my badge and replacing me. So now I can be Alessio for today.

• 1:14:55Speaker 00

  Well, you just worked.

• 1:14:55Speaker 07

  A shorter version of Alessio, basically.

• 1:14:59Speaker 00

  So today was the answer AI announcement, maybe you wanted to cover that? Just like, what should people know about it?

• 1:15:05Speaker 08

  What should people know about it? Oh, I don't know, man.

• 1:15:08Speaker 00

  You went from Fast AI to the dark side now?

• 1:15:11Speaker 08

  No, it's not at all.

• 1:15:12Speaker 07

  It is the light side.

• 1:15:13Speaker 08

  It is actually, it is the light side. Fast AI, look, I spent the last week in San Francisco, and the amount of love I received for Fast AI was overwhelming. I couldn't believe how many people told me it changed their life, you know?

• 1:15:30Speaker 08

  Which is, which is just amazing, but I, I have to say, it's actually the time to be rejuvenated, you know? The mission is the same, bring AI to as many people as possible. But now, we can't do it on the back of my bank account. I've been paying for everything, well, and my wife, you know? We can't afford it anymore.

• 1:15:49Speaker 00

  You've had donations and stuff.

• 1:15:50Speaker 08

  No, no, no, nothing.

• 1:15:51Speaker 00

  Actually, you were very steadfastly against donations, I remember this.

• 1:15:55Speaker 08

  Yeah, no donations, no revenue of any kind, totally independent. But now, you know, I think we can do a better job by having a bank account with money in it. So, thank you Jess for sending us money.

• 1:16:10Speaker 07

  We're happy to provide.

• 1:16:11Speaker 00

  What is it like when someone like Jeremy comes and like goes, you know, we need, we need a bank account?

• 1:16:16Speaker 07

  You know, there, there are some people that you go through a pitch, and then there's some people that you email and you start prepping the wire, and I would say that Jeremy fell into the ladder.

• 1:16:26Speaker 08

  Oh no, I didn't even ask for this money. I was just, I was just gonna have a chat with Alessio to get some advice.

• 1:16:31Speaker 07

  Yeah.

• 1:16:32Speaker 08

  And then Jess turned up, and Jess's other partner, John, turned up, and I was like, what are you guys doing here? And they're like, oh, we'd like to give you money. So, I was like, oh, okay, so that was good. They have good taste, right?

• 1:16:45Speaker 07

  Yeah.

• 1:16:45Speaker 00

  I've talked to you a bit, especially at the modular conference, which I'm wearing the badge of.

• 1:16:50Speaker 08

  Nice hoodie, yeah.

• 1:16:51Speaker 00

  Yeah, the hoodie is really nice. So, so you're interested in fine tuning? You're interested in fundamental research? What, like, could you list out the main areas of interest, maybe?

• 1:17:00Speaker 08

  I mean, basically, the interest is in making... AI as useful and valuable as possible.

• 1:17:07Speaker 00

  Yeah.

• 1:17:08Speaker 08

  That's how we make it like as accessible, you know, as accessible as possible, as widely used as possible. Help as many possible as people as we can with this technology, right? So, how do we do that? It needs to be cheaper, it needs to be faster, it needs to be easier to use, and it needs to be like more integrated into people's like day-to-day lives, into the stuff that they do. This is like hard, you know? And so, in the end, I guess I was inspired by Thomas Edison's invention factory in the late nineteenth century, where they had the same situation. They were like, "Oh, look, electricity's been invented. Okay, what do we do with this? Uh, it's a source of power. Um, I don't know."

• 1:17:53Speaker 08

  And they're like, "Oh, let's create the record player and the light bulb and the refrigerator." And, you know, it's like recognizing that now you have electricity, you can make all these things. That's hard. It requires really smart researchers who deeply understand the underlying technology, recognize like, "Oh, there are some gaps here, but they could be filled if we like use this kind of different kind of filament or whatever." And so you actually need like deep technical experts who will also have the like curiosity and playfulness and spontaneity to like think like, "Oh, what if the world had this new thing in it? I wonder if we could put that thing in the world now that we have AI."

• 1:18:37Speaker 00

  Yeah. You were very complimentary of like the open source. So, we last met at the open source meetup as well. We met so many times. Um, and you, you're very complimentary of like their approach towards just trying things, like model stacking, for example. Uh, is that the kind of people that you're looking to collaborate with?

• 1:18:54Speaker 08

  I think partly, um, you know, I'm deeply involved in the open source community and I wanna continue to do that, you know? They, all the best, um, kind of.

• 1:19:09Speaker 08

  Models outside of your kind of OpenAI and stuff are all created by the open source community at the moment through just trying crazy things.

• 1:19:21Speaker 08

  But it'll be a mix, you know. I also want to work really closely with the best academics in the world, you know. And I also want to collaborate with the people in parts of the world we've never even heard of, who never get a chance because nobody gave them a chance.

• 1:19:38Speaker 00

  Yeah.

• 1:19:38Speaker 08

  And, you know, so one of the things we're going to be doing a lot of is like recruiting in really weird ways, you know, to find those people who are underappreciated and.

• 1:19:50Speaker 00

  Would it be like a challenge, like a Kaggle type challenge?

• 1:19:52Speaker 08

  Yeah, like Kaggley kind of things and, you know, you know, physically find ways, you know, all through like open source bounties and stuff like that, like basically give people an opportunity to show that they can do amazing shit that nobody else can do.

• 1:20:09Speaker 00

  Yeah.

• 1:20:10Speaker 08

  Doesn't matter how old they are or where they live or what color their skin is or whatever, you know.

• 1:20:15Speaker 09

  Yeah, I think what the Fast AI community has shown is that a lot of people who don't have a traditional background that are really talented people. And I think, yeah, it's great that that was there for, that the Fast AI community was there and that Jeremy continues to highlight that those talents as well.

• 1:20:31Speaker 08

  So let me give props to Tanish as an example, right? So Tanish is the CEO of a research lab, which I'm the president of, and he, how old are you, Tanish?

• 1:20:43Speaker 09

  I'm 20 years old, which is why I'm not drinking the wine.

• 1:20:45Speaker 00

  You don't drink at this wine bar.

• 1:20:47Speaker 08

  You know, so like Tanish is a great example of somebody that most people wouldn't hire as a CEO, but why the hell not? Like, he finished, he finished high school 10 years ago. He finished high school at 10, you know, he had his first degree at, what, 14? Like, he's somebody who's.

• 1:21:04Speaker 00

  I mean, that's somewhat, that's somewhat like he, he went after the traditional accreditation, uh, the, the pieces of paper that you would pursue to show yourself as qualified. So in a way, he's, he's part of that, uh, status quo.

• 1:21:19Speaker 08

  In a way, but you know, unfortunately, people are ageist.

• 1:21:24Speaker 00

  Yes, they are.

• 1:21:25Speaker 08

  So.

• 1:21:26Speaker 09

  And I'll also note that I never actually did a computer, computer science degree or anything like this. My start with AI was actually through the Fast AI course.

• 1:21:33Speaker 00

  Yeah.

• 1:21:34Speaker 09

  Um, so yeah, and so it's been a long journey since then. Yeah.

• 1:21:37Speaker 00

  What, what would you ask him about Answer?

• 1:21:40Speaker 09

  Cause I, I already know a lot of what's going on at the company.

• 1:21:43Speaker 00

  What, what is he not saying? Is he too humble to say?

• 1:21:45Speaker 09

  I think what he's not saying is already, you know, has a great team of, of researchers that, you know, we, there are already two researchers that are at Answer AI that are amazing researchers that I've had the chance to also interact with over the past, maybe year, a year or so closely. And also just more generally, I'm, I'm, I'm looking forward to seeing what Answer does. And I'm really excited to continue to collaborate with Jeremy. I think, uh, this will be even better for me, like I'm selfishly, I'm very excited because I think it'll be better for me. Uh, you know, to work closely with Jeremy as well. Um, even though, you know, he's in his own research lab, but I think the, the collaborations that will come out of this will be, will just be amazing. So that's what I'm excited for.

• 1:22:27Speaker 08

  Me too.

• 1:22:28Speaker 00

  And Jeremy, last time you were on the podcast, you said that, you know, one of the most consistent pieces of advice that you always give is that people just need to show up, follow through, do the work, um, that stuff. Obviously, Tanish did that.

• 1:22:38Speaker 08

  Yeah. So Tanish is one of those rare people, right?

• 1:22:40Speaker 00

  But like what, like, I feel like Tanish is more special than that. Like what, what else did he do really well?

• 1:22:45Speaker 08

  Yeah.

• 1:22:46Speaker 00

  Is.

• 1:22:46Speaker 08

  So, I mean,

• 1:22:49Speaker 08

  God, how old were you when I first came across you? Like 15 or something, maybe?

• 1:22:53Speaker 00

  Wait, what? It's been so long?

• 1:22:54Speaker 08

  Yeah.

• 1:22:55Speaker 00

  Cause he only took Fast AI a year and a half ago.

• 1:22:57Speaker 08

  No, no, no. He was a Fast AI student back then.

• 1:22:59Speaker 00

  Okay.

• 1:23:00Speaker 08

  And, you know, he, he kind of got on the forum, helped answer questions, you know, asked interesting questions of his own.

• 1:23:11Speaker 08

  To stick with that for five years, that's tenacity, you know? And the last course we did was the hardest course we've ever had. It was the diffusion course. It was the first ever stable diffusion course. And none of us knew what the hell was going on. And, you know, he was the one who slogged through the math, figured out what the hell all those Greek letters were saying, and did the first math of stable diffusion video that, as far as I know, that ever existed. Um, we did that with Wasim, right? Along with Wasim. So, you know, he's like, he slogs through difficult shit.

• 1:23:54Speaker 08

  And the thing that I noticed now is like, you know, Tanishq's kind of famous, or was kind of famous, as a child prodigy.

• 1:24:01Speaker 00

  Yes, you did a TED Talk when you were 14.

• 1:24:03Speaker 08

  He was on Child Genius, he did a TED Talk.

• 1:24:05Speaker 09

  I was nine when I did it.

• 1:24:06Speaker 08

  He was nine, okay.

• 1:24:08Speaker 00

  Okay.

• 1:24:08Speaker 08

  So I kind of thought like, oh, things are easy for child prodigies. You know, they're so smart that they just, it's easy. And I'm like, oh, no. Actually, Tanishq's nearly as dumb as me. And so he just works really, he just works really hard. And he's like, he's, and I'm like, Tanishq, what does this mean? He's like, I don't know. Like, oh, okay, we better figure it out. And so that's been interesting to see that, like, actually, child prodigies have to work really, really hard as well, you know? That's part of what makes them a child prodigy is that they're tenacious and they don't give up even over five years.

• 1:24:47Speaker 00

  Does it look that way to you? Is that what you?

• 1:24:49Speaker 09

  Yeah, I think so. And I think, again, part of it...

• 1:24:51Speaker 08

  You agree you're nearly as dumb as me?

• 1:24:53Speaker 09

  I, no. Say it again for the pod. I think Jeremy's trying to trick me here. But I think the Fast and the Furious community has been so friendly that it's been a really pleasant experience to stay with that community. And I think that has also enabled my tenacity because I enjoy being in that community so much. So that's why I've stuck around in that community for so long. So without that, without the community that Jeremy has built, there's, I don't think there's any way I would...

• 1:25:20Speaker 00

  It supports you, yeah. I had the same with Free Co-Cab.

• 1:25:23Speaker 09

  So, you know, I think a lot of it has to do with...

• 1:25:26Speaker 08

  I'm gonna cry.

• 1:25:28Speaker 09

  I think a lot of it has to do with building good communities. And Jeremy has done a really good job of doing that. And it's actually a lot of hard work to build a good community and to nurture and grow that community. And, uh, you know, I've, I've been in many communities and I've kind of observed, you know, how different communities in the AI field have grown. And Fast AI is still, is like one of the best communities that I've had a chance to be a part of. So, you know, again, props to Jeremy for, for doing that as well.

• 1:25:52Speaker 00

  I'm so embarrassed right now. I, I want to give you the perspective, you've been an AI investor for a while.

• 1:25:57Speaker 07

  Yeah.

• 1:25:57Speaker 00

  And like, how do you, like, how do you view this community and this moment here?

• 1:26:01Speaker 07

  The, the one thing I will say to the conversation that we just, we're just having that I think is awesome is...

• 1:26:06Speaker 00

  We can move here a little bit.

• 1:26:07Speaker 07

  Yeah, there's so that people keep coming and drinking more wine.

• 1:26:10Speaker 00

  It's great, it's a mobile studio.

• 1:26:11Speaker 07

  Yeah, we're truly, yeah, mobile studio, middle of New Orleans, let's go. Um, one of my favorite heuristics as an investor is distance traveled rather than just your, um, rather than just like, what, what do I see today in your resume or whatnot? Um, because, um, I think if you just go by a certain pedigree or credential or whatnot, you miss a lot of people who have traveled a really big distance, who didn't have advantages or opportunities, or came from different places, or not from the US. Like, you name all the different, um, you know, all the different lists. And I always try to look for those kinds of people because they're the ones that are always pushing the frontier and like really run through walls. And I think this conversation is a good example of that, right?

• 1:26:53Speaker 00

  I mean, no one has a longer distance traveled than Jeremy.

• 1:26:55Speaker 07

  A hundred percent. Well, literally, and, and, and in the sense...

• 1:26:59Speaker 00

  Literally from Australia, yes, yes.

• 1:27:01Speaker 07

  And, and when we were, and I think when we were meeting last week, you were talking about this a little bit around looking for engineers and people at places that's not necessarily where everyone else would be looking at, but that has yielded some of the best, like deepest relationships you've had, right?

• 1:27:16Speaker 08

  Oh, absolutely. I mean, companies turn resources into valuable products and services, right? Like, what are the resources that we, we suck in? It's like, it's people and GPUs, you know?

• 1:27:28Speaker 00

  And money.

• 1:27:28Speaker 08

  And it's like, and, and, and well, we really need the money to get those GPUs and the, and the people, right? And like, the, the GPUs are, you know, reasonably like here you can replace one with another, no, no worries. But so it's actually the, the competitive advantage, the thing that makes you different is the people. So...

• 1:27:50Speaker 08

  This is the most important thing for us to achieve our mission is to build this team, you know, to build this really special team. And I, you know, I think the way to do that, and the way I've always built teams is to say, is to look at people and say like, okay, where is this person now? And what would it have taken them to get there? You know, like, so if, if somebody's like, you know, was kicked out of high school, you know, because they were dyslexic, or because somebody was like, grew up in the mountains of Bangladesh and didn't have a PC until they were sixteen, or, you know, somebody fought against, you know, uh, you know, a, a woman who

• 1:28:32Speaker 08

  grew up in an environment which he had to fight against like institutionalized sexism, or whatever, it's like, these are the people to me, I just kind of feel like, okay, this person's gone from like negative forty-three up to ninety-nine.

• 1:28:46Speaker 00

  Yes, overcome a lot.

• 1:28:47Speaker 08

  That's a kick-ass amazing person, where somebody who's gone from like ninety-eight to ninety-nine is like, okay, still cool. But they're probably not the people who are gonna like change the world.

• 1:28:59Speaker 00

  Yeah.

• 1:28:59Speaker 08

  And so, we wanna be a small team, where like literally every person in it is somebody who can change the world. And the nice thing is, when you're in a small team like that, it's just really enjoyable, because everybody's like, just really great to be around, you know, really inspiring. And so, yeah, that's why we're kind of looking for these extremely special individuals.

• 1:29:28Speaker 00

  Yeah, cool. So that's a hiring call explicitly, you know, if, if anyone's listening who fits that profile and really wants to work with you, uh, they should reach out, right?

• 1:29:37Speaker 08

  Yes.

• 1:29:38Speaker 00

  On, on, uh, and now we have a website to send people to. So I was gonna wrap it up with just overall Europe's tips, right? Like, what is it like to be at Europe's this year if you've been here before? And also, like, what's your best tip for doing Europe's right? Anyone can take it.

• 1:29:55Speaker 09

  I guess I'll start. This is my second Europe, so maybe I don't have a lot of experience with it. But, I mean, I've been enjoying it a lot so far. For me, I think it's about networking with people, and that's the best part of Europe's. Because at the end of the day, AI moves so fast that half of these papers are already kind of outdated. Like, you know, we've already seen like-

• 1:30:18Speaker 08

  They were written months ago, right?

• 1:30:19Speaker 09

  Yeah, yeah.

• 1:30:20Speaker 00

  In order to get here, they had to be reviewed.

• 1:30:22Speaker 09

  Exactly. So, you know, we're already seeing the second version or the third version of a lot of these models already. And, you know, so, I mean, it's, for me-

• 1:30:30Speaker 00

  So archive is all you need?

• 1:30:31Speaker 09

  Archive is all you need, I guess, yeah. So for me, the value comes out of talking with people and meeting with people and networking. And that's why, you know, we're coming to events like these to network and, you know, make these connections. And, you know, I actually meet a lot of collaborators and other researchers at all these conferences.

• 1:30:48Speaker 08

  And just to be clear, when you're saying networking, like, it's not like networking in that sense of like getting ahead. It's a kind of a really nerdy kind of networking. So like earlier, Tanishq and I were at another reception where it's like, oh, there's Albert Gu, he's the guy that like two days ago released the Mamba paper. And we got to him and say like, oh, you know, let's, we had a conversation about state space models and why he's using that, and what he thinks the opportunities and limitations are, or is there still room for attention? And like, so when we say networking, you know, we mean like-

• 1:31:18Speaker 07

  Really meaningful conversations, yeah.

• 1:31:19Speaker 08

  -geeking out on deep conversations about people's academic areas of interest.

• 1:31:23Speaker 00

  Yeah. I always follow up the question of like, okay, like, what's your name, where you work? And then what are your interests? And then we kind of go from there.

• 1:31:30Speaker 08

  Yeah, just like, what paper did you write last, or?

• 1:31:33Speaker 07

  You know, I will say one thing. So even though the posters, there are a bunch that truly you go by, and even the people presenting are like, yeah, this is kind of outdated. The one, the one hack that's really fun is a lot of those people are also already working on the next thing, and they can give you sort of an early preview of something that actually is not in archive yet. And so that, I actually have always, my favorite parts of the conference are actually just walking around the poster session, shaking hands with people who are presenting, and learning about what they're most excited about, what they're working on, what are some of the new things. So I find that really fun. And also, in my case, since I'm a VC, my best tip is throw an event with a lot of good wine. Let the people come.

• 1:32:13Speaker 00

  Excellent. Jeremy, you have any tips?

• 1:32:16Speaker 08

  I mean, like Tanish, this is only my second Europe's. But my opinion, quite a few conferences in general, and my tip, number one tip for all conferences is don't go to any sessions.

• 1:32:26Speaker 00

  Yeah, just stay outside and talk.

• 1:32:28Speaker 08

  Whatever they're saying, very, very slowly, and they're probably not an expert at verbal communication either. You can probably get the better version by just reading the damn paper that they're reading out to you. So don't bother with that. So like, yeah, hang outside, you know, in the hallway. Look on the app to see who else is around and reach out to them and like, try and like, find a group of six or so interesting people to go and like, check out the, you know, local Louisiana sausage special outlet with whatever.

• 1:32:59Speaker 07

  Reception hopping.

• 1:33:00Speaker 08

  Yeah, reception hopping. This is our fourth reception tonight.

• 1:33:03Speaker 00

  Oh my god.

• 1:33:04Speaker 07

  Fourth and best, right, Jeremy?

• 1:33:07Speaker 08

  This is why we left, this is why we came to this one last, so we can hang out here.

• 1:33:10Speaker 07

  Let's go.

• 1:33:11Speaker 08

  Until the wine's finished.

• 1:33:13Speaker 00

  So, so a lot of people hate on the official Europe's conference app, Hoover. But I kind of like it because of one thing, people can organize their own meetups and list it here.

• 1:33:21Speaker 07

  It's awesome. It's awesome.

• 1:33:22Speaker 00

  It's actually really good.

• 1:33:23Speaker 07

  Yeah, so I'm Brazilian and there's a Brazil, like, little chat, and it's so fun. Everyone's talking in Portuguese, talking all the time. They're sharing all the things that, and these are people talking about actually like interesting concepts in Portuguese. So it's, it's actually really fun. I love the app.

• 1:33:38Speaker 08

  I didn't even know you're Brazilian, so I learned so quickly.

• 1:33:40Speaker 00

  Yeah, Liao with a little.

• 1:33:42Speaker 07

  Liao, yeah, my, my accent kind of like trips people. And it also trips people when I say something incorrectly and you can't really tell, but I'm like really Brazilian.

• 1:33:50Speaker 00

  Yeah, well, we should, we should do a steakhouse the next time.

• 1:33:53Speaker 07

  Yeah.

• 1:33:53Speaker 08

  Oh, yes, please.

• 1:33:54Speaker 00

  Yeah, that's, that's one of those dinners.

• 1:33:56Speaker 07

  Done, done, Jeremy.

• 1:33:57Speaker 00

  Churros Cariños, right?

• 1:33:58Speaker 07

  Yeah, Churros Cariño.

• 1:33:58Speaker 00

  Exactly. My, my favorite was, there was a meetup for people who are interested in sushi. That, that was, that was the meetup.

• 1:34:04Speaker 07

  I love it, yeah.

• 1:34:05Speaker 00

  There's nothing but you learning about it.

• 1:34:06Speaker 07

  So at ICML, it was really fun. There was one meetup that I went to that was just like swimming in the morning because it wasn't Hawaii. It was actually kind of awesome. And then people were like actually discussing like super legit topics in the ocean.

• 1:34:17Speaker 00

  I'm actually kind of sad I missed out on ICML, but like, it felt indulgent to go to Hawaii for that.

• 1:34:21Speaker 07

  Yeah.

• 1:34:22Speaker 00

  Okay, well, I just wanted to bring it to a close. The, the last thing I was gonna say is, Jeremy, I don't know if you know, I picked your meme as the best meme of November 2023. It was laundry buddy.

• 1:34:35Speaker 00

  So, what's up with Laundry Buddy? Why do you hate it so much? What did it do to you?

• 1:34:41Speaker 08

  No, it did nothing to me.

• 1:34:44Speaker 00

  For people who are out of the loop, what did you do?

• 1:34:46Speaker 08

  I couldn't have walked it back more.

• 1:34:49Speaker 07

  Jeremy did walk it back on Twitter.

• 1:34:51Speaker 08

  You're really gonna make me revisit my shame?

• 1:34:54Speaker 00

  I just think it's a fun story.

• 1:34:55Speaker 08

  Okay, just for your show, I'm gonna revisit my shame.

• 1:34:57Speaker 00

  Some people don't know.

• 1:34:58Speaker 08

  I made a bold claim that Laundry Buddy was not the peak of OpenAI's path to societally beneficial artificial general intelligence. I was wrong.

• 1:35:13Speaker 08

  It is, in fact, very much on that path. It is well-loved.

• 1:35:20Speaker 08

  To be able to know that the world's best artificial intelligence can help you figure out how to sort out your whites and your colors, whether to use powder or pods, and what to do, you know, if you get a stain and you don't have laundry nearby. It's special, it's important, and it's a part of my life that I will never want to be without.

• 1:35:46Speaker 00

  I love that, so the ChatGPT now has an official Twitter account, and they even got in on the Laundry Buddy meme, which is amazing to me.

• 1:35:53Speaker 08

  I actually spent a couple of hours this morning hanging out with Boris Power from OpenAI, who, you know, was in there batting for Laundry Buddy from the start.

• 1:36:04Speaker 00

  Wait, there's an anti and pro Laundry Buddy?

• 1:36:07Speaker 08

  No, I mean, he was just a particularly strong enthusiast. He had the grace to not even bring it up, unlike you.

• 1:36:16Speaker 00

  I had to, it was so funny. I cracked up so much. It was great. Well, thanks for chatting. I'll return you back to your evenings.

• 1:36:23Speaker 08

  May your clothes be well-wandered. Thanks for having us.

• 1:36:26Speaker 07

  Cheers.

• 1:36:29Speaker 00

  That was Jeremy Howard, together with Tanishq Abraham and Jess Leao. Tanishq and Jeremy recorded a podcast separately, so if you want to learn more about Tanishq, he's done long-form interviews in more detail than I can cover because it's a lot of biomedical stuff, and that's one of the areas that we are not very knowledgeable on. And for Jess Leao, she was an investor in Mosaic, is one of the newest partners at Decibol, and led the round in Answer AI. Next, we're going to go to some people on the show floor of the NeurIPS Expo. They're not people I had prior relationships with, but they're still doing interesting work nonetheless. And the first is, we're going to check in with Cerebras, which is not only producing giant, massive GPUs, but also publishing interesting research. So here's my conversation with Joel Hesness, Principal Research Scientist at Cerebras Systems.

• 1:37:12Speaker 10

  That started working about a year ago. We started building out multi-box systems so that we could do cluster-level training, so larger-scale models. And so this last year, we've just been showing off what it's capable of. So, early this year, we started with our Cerebras GPT models that showed

• 1:37:34Speaker 00

  Yes.

• 1:37:34Speaker 10

  compute-optimal scaling for, so Chinchilla-style scaling, but it's open source. All those models we released is open source. Based on that work, we got attention of a few different groups. One of them was the Open Tensor Foundation, and they came to us and said, "Hey, we want a great 3 billion parameter model that does..." So it's something that's easy to deploy, like in a laptop or something, and we wanted to do very general language capabilities, long sequence length. And so we trained the BTLM language model for that. Concurrently with that, we also had an engagement that started up with Group 42 in the United Arab Emirates, so that's this poster, Core 42.

• 1:38:20Speaker 10

  They had interest to train large Arabic language models. So the first demos that we did for them were just Arabic models, but then they said, "Let's do multilingual Arabic and English." So we've been training the Jace 13 billion and 30 billion parameter models this year.

• 1:38:40Speaker 10

  We've released both of those publicly. The first version of the thirty billion just came out. And the the quality of that model is in Arabic is better than any other public models currently. And then in English, it's competitive with models like Falcon forty B. So we're on a good track there. More releases to come through Core forty two. We're excited to have that be open source and to contribute to the community there.

• 1:39:09Speaker 00

  Yeah.

• 1:39:10Speaker 10

  And then.

• 1:39:11Speaker 00

  Yeah, anecdotally, I mean, since we're already chatting, so might as well keep going. But the UAE also notably has the Falcon or T I I.

• 1:39:19Speaker 10

  Yes.

• 1:39:20Speaker 00

  Institute. Are they related? Are they are they competing with each other? What's going on?

• 1:39:24Speaker 10

  Initially, there was a little bit of competition. They're they're funded by different, different people, different groups. So but there is a countrywide effort going on in the United Arab Emirates to to consolidate a lot of their AI efforts.

• 1:39:37Speaker 00

  Yeah.

• 1:39:38Speaker 10

  And so that's why we're seeing very impressive and and good pushes towards let's make it open. Let's collaborate some more. And so there might be opportunities in the future for us to coordinate directly with T I I. And we have looked at things like their their data sets like refined web. So there has been some exchange.

• 1:39:58Speaker 00

  Yeah, with the macro data refinement process that I don't know if you know, it was a reference to an Apple TV show. Severance, anyway.

• 1:40:06Speaker 10

  Interesting.

• 1:40:06Speaker 00

  It's my fun, my fun fact. A little bit editor's note, the T I I Institute people were actually there at NeurIPS presenting a poster on Refine Web, the data set that they did for Falcon one eighty B and forty B. So I asked them about the name. My last question is about the name. Is it from Apple? Is it is it from Severance? Yes. So what's the story? What's the? No, it's just like, basically, in the end, we had someone look at the data every now and then, like, go through the thing. And that's like looking at the scary numbers. So, you know, this was the macro. Nobody comments about this. I know. I was like, wait, I saw this in Severance. Yeah, I know. Right? Like, I was like, this is a good joke because it's exactly what you do when you do filtering. Exactly. If you haven't seen Severance, it's a great show. It's on Apple TV, great watch for the holidays, pretty short, and it's interesting.

• 1:40:51Speaker 00

  I guess you can call it AI related now. But it's it's cool that, well, so one of the things I often get asked about because we have listeners in a lot of different countries, should every country have their own model? You know.

• 1:41:03Speaker 10

  I think this is a really tough question because um the the volume of data in different languages is a it's power law zip law distributed, so the number of low resource languages is massive. It's we're talking over a hundred languages that are low resource. You just have too few tokens to do uh to do a lot with in in the language modeling context. So it's much harder to deal with those. Now there uh we've actually seen a few different techniques at NeurIPS uh that are targeting those sorts of settings. And they're doing things like train a uh base language model in English. And then do transfer process where you co-train with both languages.

• 1:41:45Speaker 00

  That makes a lot of sense.

• 1:41:46Speaker 10

  It makes a lot of sense. That's in that setting, you want to get the knowledge representation from one language and then try to adapt the style or grammar syntax, I guess, the easier part.

• 1:41:57Speaker 00

  Yeah.

• 1:41:58Speaker 10

  Um in Arabic, we're in a sort of medium resource language. There, I think it makes more sense to try to mix two languages if you want to do multilingual. And then it helps you do things like translation.

• 1:42:09Speaker 00

  Okay.

• 1:42:10Speaker 10

  And then uh higher resource languages, so if you're talking uh European languages, French, Spanish, English, uh German. Those I think you can do probably um from scratch in those languages. And uh probably pretty easy to do multilinguality also.

• 1:42:28Speaker 00

  Yeah.

• 1:42:29Speaker 10

  So yeah, it's it's a definitely a very interesting, open direction we're uh we're pushing for. Um in fact, I'd maybe reference uh we have a multilingual workshop on Friday. Uh where we're we've invited a bunch of groups to come and give talks about their experiences with training different language models.

• 1:42:50Speaker 00

  Cool, uh well, people can check out the authors. I'm sure this is published and findable online.

• 1:42:55Speaker 10

  Yes.

• 1:42:55Speaker 00

  Um, cool. What, what other, so, um, we should probably get to intros a little bit. Uh, I mean, we're already recording. Uh, who are you and what do you work on and what is your team work on?

• 1:43:04Speaker 10

  So my name is Joel Hesness. I'm a principal research scientist at Cerebras Systems and, uh, I'm the, the lead of our core machine learning group. Uh, so I've helped us bring up our foundation language models first and helped kind of set some of the direction for expanding outward from there. So, uh, we started by expanding out a lot on, uh, the language, the common language functionality. And, uh, now we're expanding into other places where transformer language, uh, models can be used. So targeting things like, uh, multimodal, uh, and other, uh, workloads that are similar.

• 1:43:41Speaker 00

  Okay.

• 1:43:42Speaker 10

  So, um, a lot of our effort has been bringing this up and coordinating with the broader Cerebras organization to do, uh, lower these applications down, get them compiled to run at efficiency on our hardware. Uh, so there's been a lot of performance optimization, making sure numerics are correct for, for training large models, making sure things train stably, things like that.

• 1:44:05Speaker 00

  Yeah.

• 1:44:06Speaker 10

  So yeah, we're, we're focusing on scaling out right now, uh, getting much larger clusters. Uh, we've got, we've sold a couple already and, uh,

• 1:44:15Speaker 00

  To G42.

• 1:44:15Speaker 10

  G42.

• 1:44:16Speaker 00

  Yeah.

• 1:44:17Speaker 10

  Uh, yeah, exciting things to come there, I think.

• 1:44:20Speaker 00

  Exciting things to come. So we're going to cover some of the other, uh, posters that you have here, but one thing I guess I, people are very unfamiliar with anything but Nvidia. Uh, what should people know when working with a Cerebras chip like?

• 1:44:32Speaker 10

  Sure. Yeah. I think maybe, maybe people might be familiar with our, our wafer.

• 1:44:37Speaker 00

  Yeah.

• 1:44:37Speaker 10

  So Cerebras uses a full wafer for our processor instead of cutting the wafer apart into pieces. Um, if you cut it apart, you end up packaging it into a bunch of different cards and then you package those into a box.

• 1:44:49Speaker 00

  And you have to network them. Yeah.

• 1:44:50Speaker 10

  And then you have to network them all together with a bunch of extra software. That's very complicated for large scale applications. And so, uh, instead of doing that, we leave it together on a single wafer.

• 1:45:00Speaker 00

  Got it.

• 1:45:01Speaker 10

  That single wafer goes in a single big box. That's the performance is roughly equivalent, our CS2 box is roughly equivalent to maybe 20 A100 GPUs.

• 1:45:13Speaker 10

  And you can program it like running on a single GPU, so it's just much easier to use.

• 1:45:18Speaker 00

  Nice. And is it cost effective as well? I assume it is because you're saving a whole bunch of overhead.

• 1:45:24Speaker 10

  Right. So we aim, so the, the manufacturing process, it has a lot lower costs because we don't have to deal with as many moving parts. Um, the fewer points of failure, reliability is quite good.

• 1:45:37Speaker 00

  Yeah.

• 1:45:37Speaker 10

  And we try to, uh, we aim to be a price performance, uh, comparable to GPU systems.

• 1:45:45Speaker 00

  Cool. Awesome. That's the hardware stuff. We're also going to talk about the streaming things in a bit. But, uh, yeah, I'd love to, whatever you want to pick next as, as one of your works for this year.

• 1:45:54Speaker 10

  Just give a, an overview of some of our research directions. So our, our hardware is, uh, it has native support for completely unstructured sparsity. Uh, what that means is we can send in, uh, say if we're using the weight streaming mode, which I mentioned, the, a weight that comes in, we can do a vector multiply with some activations. Uh, so you can use that in your, uh, matrix multiplies on the wafer, but you can do that on a per weight basis.

• 1:46:24Speaker 00

  You don't need to load the whole thing at once.

• 1:46:25Speaker 10

  You don't need to load the whole thing to do, uh, matrix multiply. So what that means is we can do unstructured sparsity, just send in the weights that you actually want to use in the matrix multiply and you can get, uh, a sparse matrix multiply.

• 1:46:37Speaker 00

  Isn't the decision for, like, this is the argument, classic argument against that kind of sparsity is that the decision actually takes longer than just doing the math anyway. Like the, the branching, the, the sort of Turing complete branching.

• 1:46:49Speaker 10

  That's a, that's a, yeah. So, uh, part of the approach that we're using is a, a weight sparse approach, which means the sparsity is in our, in the model itself. And so then while you're training that, you'd prefer those weights to be, uh, the same sparsity structure for a while.

• 1:47:06Speaker 00

  Okay.

• 1:47:07Speaker 10

  So there are techniques that train.

• 1:47:08Speaker 00

  Some kind of constraints, some regularization thing.

• 1:47:10Speaker 10

  Right. Yeah. So, so, uh, the, the early works in this are things like the lottery ticket hypothesis, where you'd find the, yeah.

• 1:47:19Speaker 00

  John Frank, John Frank is like, uh, 10 feet from us. Yeah.

• 1:47:21Speaker 10

  And, uh, there you find the mask, uh, by, by doing some heavy duty training and then you rewind and retrain the model from scratch.

• 1:47:30Speaker 00

  Yeah.

• 1:47:31Speaker 10

  Now that's, uh, static sparse, so that you have the same weight sparsity all the way throughout. That works great on our hardware. We have, however, added a bunch of new functionality that's sort of beta in our recent release that allows you to change the sparsity throughout training. And so that's something that's being used in recent research works like the rigging the lottery ticket hypothesis work, so wriggle, and then another one called SET, a different approach to deciding how to change the sparsity. But those updates happen infrequently enough that it doesn't harm the performance on our hardware.

• 1:48:10Speaker 00

  That's cool. Awesome. So this is, sparse IFT is the paper that you published?

• 1:48:18Speaker 10

  Yes. So our sparse IFT work looks at different ways that you can swap out layers for sparse versions.

• 1:48:26Speaker 00

  Yeah.

• 1:48:27Speaker 10

  Using the same flops that might be able to get you better representation capability. So if you have pressure in your representation that's in your activations, for instance, let's widen the layer and sparsify it to give the model more activations. You can store more in those activations. Those end up staying dense. So our results here show that we can get something like a 2 to 3X performance improvement at 75% sparse, or you could flip it around and you can get, for the same flops, a better model by sometimes 3 to 5%.

• 1:49:03Speaker 00

  That's probably, budget-wise, that's, I guess you're choosing between pre-training and inference, just like many people, like what you're optimizing for?

• 1:49:10Speaker 10

  Yes.

• 1:49:11Speaker 00

  That's great. Awesome. And what else are you leading?

• 1:49:14Speaker 10

  So I'm also working on the, on some of the pre-training efforts that we're doing that look at things like gradient noise to estimate good batch sizing and make sure that we're getting, making efficient use of the compute. So there are techniques, so we have a poster, the Efficient and Approximate Per-Example Gradient Norms paper. This, this is, yes. So this is at the, we have this published at the Want workshop with NeurIPS. And the basic idea is... Uh gradient norm calculations are uh typically if you wanted to do the uh gradient norm calculation, you'd want to aggregate all the gradients together and then calculate the norm.

• 1:49:59Speaker 00

  Uh huh.

• 1:49:59Speaker 10

  And you do that over your batch.

• 1:50:01Speaker 00

  Yeah.

• 1:50:01Speaker 10

  So that's it's helpful if you want to measure some training dynamics. But if you want to look at something like critical batch size uh to understand is how well is my model training in terms of efficiency? You actually want to have sub batches. You want to understand the grad norms of the sub batches also. If you use that and then the large batch grad norm, you can calculate noise statistics, like signal to noise maybe. Um if you use this technique that was defined by uh one of my teammates, Gavia, we can uh we can do an approximation that allows us to take some run some statistics over activations and run some statistics over the the delta gradient values coming back.

• 1:50:46Speaker 10

  And then you can take a dot product, uh an element wise product of those now is much more compute efficient to calculate for each example, this is an approximation of the uh the gradient for that sample. And then you can arbitrarily kind of combine those back together to get estimates of uh gradient noise.

• 1:51:08Speaker 00

  Okay.

• 1:51:09Speaker 10

  So uh this this is something where we we improve the we improve the

• 1:51:14Speaker 00

  For cerebral GPT.

• 1:51:15Speaker 10

  Compute requirements. Uh we we use this in in a few different contexts currently, but uh it's it improves the compute requirement for this from uh for for high dimensional tensors from the dimension of the tensor down to linear

• 1:51:30Speaker 00

  Yeah.

• 1:51:31Speaker 10

  Uh linear time computation.

• 1:51:33Speaker 00

  Nice. And uh do you is there like um um I forget what this what is this called? It's kind of like an annealing curve or something where you use this technique at the start to initialize and then and then eventually you you sort of wean yourself off it.

• 1:51:49Speaker 10

  So if you, so this is something you do want to track throughout training?

• 1:51:53Speaker 00

  Yeah.

• 1:51:54Speaker 10

  Especially if you're, if you're doing like phased training, or if you're changing the data distribution or something, it's really helpful to have these statistics to decide, is my, am I using an appropriate batch size that I'm getting good generalization with the new data? It helps you set learning rates and things. So this is something you want to track throughout training. It's, it gives you an estimate of how big the batch size could be.

• 1:52:20Speaker 00

  Yeah, excellent. Very cool.

• 1:52:25Speaker 00

  Um, any, one more?

• 1:52:27Speaker 10

  Sure. So then, given that we have a sparse accelerator, we're also looking at applications where you can deploy sparse models. And part of our work is figuring out how to find those sparse models that you use in a deployment setting. And so we have other work that's related to like the sparse GPT work that's been recently released, where we, we do some pruning after dense pre-training, and we do some re-training to, to get the capabilities of the model back.

• 1:53:04Speaker 00

  Backup.

• 1:53:04Speaker 10

  Before you would put it in deployment.

• 1:53:06Speaker 00

  How much of it can you get back?

• 1:53:07Speaker 10

  Actually, I'm not totally familiar. This was work from, from my team members. I know we can do, so for, for large, very large language models that have, have not been trained on a huge number of tokens, you can do easily upwards of 50% sparsity and fully recover the, like the upstream losses from this re-training. So this is, this is a really big next step challenge for a lot of the organizations that we work with. They're interested, now I have, they're able to pre-train a very large model with the hardware. Now they're interested in figuring out how to deploy it in an efficient manner. So we're working with a few different groups on this.

• 1:53:48Speaker 10

  So we're working with Qualcomm and another group called Neural Magic that does inference for these large models.

• 1:53:58Speaker 00

  Yep. Amazing. Um, I was gonna ask if you need the same data set to retrain, but it looks like you train on the pile, so I guess, I guess that's a no.

• 1:54:08Speaker 10

  Yes, you can actually shift here. Um, the, obviously, different data distribution means you have to be a little bit careful about how you do the retraining. So I think there are a few different things we've learned about. Different learning rate warmups, different, uh, learning rate levels, I guess.

• 1:54:25Speaker 00

  Yeah.

• 1:54:26Speaker 10

  Because, uh, if you're doing a big distribution shift, you wanna allow the model to shift a little bit, and so you wanna slightly higher learning rate.

• 1:54:34Speaker 00

  But like, for example, you prune, you prune Llama 2, and we don't know what the original data set was.

• 1:54:39Speaker 10

  Yeah, I mean, well, so we, we kind of know that Llama 2 is a little bit similar to something like Slim Pajama

• 1:54:45Speaker 00

  Okay.

• 1:54:45Speaker 10

  and Llama 1, but yeah, it's, it is definitely a different data set. Um, we, we do know that PyL and Slim Pajama have a fair bit of overlap in, in some things, but it is definitely a different distribution, yeah. Um, so this is a, a lot of work that our Applied ML team, uh, our Applied ML team is working on. We're, we're expanding that team currently, by the way, so Cerebras is hiring for anybody who's, who's interested in listening. You can check out our website, cerebras.net/join-us, uh, if you'd like to check it out.

• 1:55:19Speaker 00

  Yeah.

• 1:55:20Speaker 10

  Send us, uh, send us your resume and we'll, uh, we'll take a look.

• 1:55:23Speaker 00

  Yeah, thanks for spending some time with us. Um, but before we go, what's one NeurIPS tip, uh, that you wanna give to people if they're attending NeurIPS? Like, you know, how do you do NeurIPS right?

• 1:55:33Speaker 10

  How do you do NeurIPS right? Well, so, uh, it's grown roughly 5X in the time that I've been attending NeurIPS, so, uh, it gets more overwhelming every year, so pace yourself and... I like, uh, I, I like that they've kind of backed off a bit on the talks and in favor of poster sessions. Like, just, you gotta go wander around, you gotta talk to people.

• 1:55:56Speaker 00

  Yeah.

• 1:55:57Speaker 10

  You gotta check out posters and, and kind of let stuff sink in and, and ask questions, so, yeah.

• 1:56:03Speaker 00

  Yeah, excellent. Well, thanks so much for your time.

• 1:56:05Speaker 10

  Definitely, thanks.

• 1:56:06Speaker 00

  Thank you. I think Cerebras is doing very interesting work here. Most people know them for their hardware, but I think they're doing very interesting work on the software and LLM trading side. And I'd be interested to have them on again in 2024. So next, we're gonna go walk down the floor to Voxel 51, which is not a company I've actually come across before, but it seems to be an interesting pair together with the next guest as well. So this is another one of those situations where I get to put two competitors next to each other and let you decide as to how they differ and how they talk about themselves.

• 1:56:36Speaker 11

  Sure, my name is Jason Corso. I'm the co-founder and chief scientist at Voxel. I'm also on the faculty of EAKS and robotics at the University of Michigan. So Voxel 51 is a spin out of my lab. We make a toolkit for AI engineers that sits on top of things like PyTorch and TensorFlow. And I think of it like a model and data set debugger, right? The key problem that we face is not that we can go download data sets and then train models on them, or even with foundation models, go pull one off the shelf and then expect it to work exactly the way you want. The problem is really the code development of a data set to then go and actually use one of those models or train or fine tune your own model. So 51 lets you represent the data that you're using or building alongside your models

• 1:57:21Speaker 11

  in a way that is extensible, visualizable and flexible so that you can write single Python lines of, like simple, single lines of code in Python to do queries of your data sets and your models. Like, show me the corner cases where Model A is outperforming Model B, and it's outdoors. Or show me, you know, intersections in my BDD data. Or let me visualize my embeddings that are either just vision or point cloud based or multimodal. And then visually interact with them with lassoing on the 3D embedding.

• 1:57:52Speaker 00

  Is the concept of active learning still in vogue? Or is it like not cool these days?

• 1:57:57Speaker 11

  Well, I mean, so 51 is a pretty flexible ecosystem of capabilities. The heart of it really is that data-centric data model of unstructured data. So we support images, video and point clouds. You can, in fact, there's a blog that one of my colleagues at Voxel 51 wrote maybe like a month ago on how to implement an active learning workflow on top of 51.

• 1:58:19Speaker 00

  It seems like it will lend itself easily.

• 1:58:21Speaker 11

  Yeah, exactly. It's plausible. I mean, the challenge with active learning is, you know, will just more data help or do you need to write more data?

• 1:58:27Speaker 00

  Of course.

• 1:58:28Speaker 11

  And I think that's kind of a, you know, it's, you know, that's the question, I think. Right? Yeah.

• 1:58:32Speaker 00

  Is it primarily a vision that you work on or it's just anything?

• 1:58:35Speaker 11

  Yeah, so my my experience is in computer vision, mostly video understanding and image imaging problems. So that's where we got started. However, the software is pretty flexible, so you can add your own data type. Like, you know, we're considering adding audio, adding text, IOT, you know, like, like temporal signals. But right now, it's images, video and point clouds.

• 1:58:55Speaker 00

  I've often heard it said that, you know, the best researchers and the best engineers are really the people who get their hands dirty in the data sets.

• 1:59:01Speaker 11

  Oh, yeah, you have to get your hands dirty. And this is so in some sense, the whole company exists because I was worried no one was getting their hands dirty enough.

• 1:59:09Speaker 00

  Yeah.

• 1:59:09Speaker 11

  Right? Like, they were just expecting to take a data set, take a model, and then train it once, and then out pops like your usable thing.

• 1:59:15Speaker 00

  Yeah.

• 1:59:15Speaker 11

  No, that's not the way it works, right? It's a hard. This is a hard problem and building intuition, building a comfort or like an ability to take a 10 million sample data set and find like the 1,000 samples that are giving you this problem here.

• 1:59:28Speaker 00

  Yeah.

• 1:59:28Speaker 11

  Is hard to do. And that's what 51 really lets you do.

• 1:59:30Speaker 00

  Yeah, yeah. Why it's a why the name, actually, I have to ask.

• 1:59:33Speaker 11

  Well, we had 50 bad ideas.

• 1:59:36Speaker 00

  And this is the first, the one that was like actually good.

• 1:59:38Speaker 11

  Well, that's, that's the way we say now. But the, the actual original way we got started as a company was as a video understanding as a service platform. And so that's why. So the voxel and the name is in the space time volume of pixels.

• 1:59:52Speaker 00

  Yep.

• 1:59:52Speaker 11

  You know, and 51 was just to elicit ideas of area 51.

• 1:59:55Speaker 00

  Yeah.

• 1:59:56Speaker 11

  Right? Like, can you find the right voxel? Is it there? That kind of thing. We've subsequently way pivoted away from that, as, as most startups will do at some point in their, in their journey.

• 2:00:04Speaker 00

  Yeah, it makes the domain easy to buy.

• 2:00:05Speaker 11

  Sure. Exactly.

• 2:00:07Speaker 00

  So anything else people should know about your, your platform, like top use cases, top customers that you always brag about?

• 2:00:12Speaker 11

  Sure. Well, I mean, it is open source, right? So as long as you have the key, three key assumptions, local data, one user, one machine.

• 2:00:19Speaker 00

  Yep.

• 2:00:19Speaker 11

  There's no limitation on the machine learning that you can do with 51. When you want to violate one of those assumptions, like work on a team or work in the cloud or whatever, then we have an enterprise product that you would talk to us to purchase, basically. And that's kind of like a Google Drive layer on top of the open source one.

• 2:00:34Speaker 00

  Very reasonable.

• 2:00:35Speaker 11

  Yeah, the only, I mean, we, we sell to, a lot of companies do use it. I'm not gonna name them here, but you can go to the website, there's a logo wall of those we can name.

• 2:00:44Speaker 00

  Yep.

• 2:00:45Speaker 11

  But it'd be great if you're listening to give us a GitHub star.

• 2:00:48Speaker 00

  Yep.

• 2:00:48Speaker 11

  That's our, like, we're here at NeurIPS to get users to get stars.

• 2:00:52Speaker 00

  Stars for swag.

• 2:00:53Speaker 11

  Stars for swag, you got it.

• 2:00:54Speaker 00

  Excellent. You published a guide to doing CVPR right.

• 2:00:58Speaker 11

  I did.

• 2:00:59Speaker 00

  And we're here at NeurIPS. What would be your guide for doing NeurIPS right?

• 2:01:02Speaker 11

  So how to do NeurIPS right? I think there's some key things of doing large conferences right. One is like, don't expect to do too much per day.

• 2:01:07Speaker 00

  Yeah.

• 2:01:08Speaker 11

  Right? So what I've always done, even when conferences were like a quarter of this size or less, like, for any one day, identify five to 10 papers in the morning that I just want to go, want to understand for that day, right? So then, then I will make sure, though, to spend time with that poster presenter or at the oral talk. To me, that's the key. And then, at the end of that day, I do tend to write a summary for my own brain, my own notes of what I did, like, what the key points were from those papers.

• 2:01:36Speaker 00

  Yeah.

• 2:01:36Speaker 11

  That's definitely one, one winning strategy for a big conference like this.

• 2:01:39Speaker 00

  All right. Any other advice for people building, you know, any papers that you're excited for this year?

• 2:01:44Speaker 11

  Well, I mean, advice, I don't know, if you don't know your data, then you don't know what you're doing, is the way I would probably say it. And indeed, like, getting close to your data is part of the model-building process, right? Like, just to say it again, like, I think of it as a co-development process of datasets and models.

• 2:02:03Speaker 00

  Yeah.

• 2:02:03Speaker 11

  Not of a model-training problem.

• 2:02:04Speaker 00

  Yeah. I actually had a really interesting chat with someone from Cerebrus, actually, where he talked about how they were doing evals on their loss per region on a dataset as they're training their large language models, so that they could increase the exposure on a specific subdomain if they saw that specifically, like, loss was not progressing as well in that particular subdomain. So it's kind of like online training and, like, watching their models evolve while they're training.

• 2:02:28Speaker 11

  Yeah, even on, I guess it sounds like on specific subsets of the data.

• 2:02:31Speaker 00

  Yes.

• 2:02:31Speaker 11

  Which is really important.

• 2:02:32Speaker 00

  Cool. Well, thanks so much for your-

• 2:02:33Speaker 11

  Thanks very much. Nice to chat with you, Sean.

• 2:02:35Speaker 00

  Coming from data engineering is pretty interesting to see this space develop. It's interesting also that a lot of them emphasize open source, which we'll see with the next speaker, which is Brandon from Nomic. Who are you and what's Nomic?

• 2:02:47Speaker 12

  Yeah, hey, everyone. My name is Brandon AI. I'm a co-founder and CEO of Nomic. Nomic is a company that does many things, but we have two main products right now. One of them is GPT for all, which is an open source ecosystem of low resource language models. So it lets you do things like run, you know, Mistral 7B, fine tune on Open Orca on a MacBook or, you know, some esoteric GPU, things like this. The second product is a tool called Atlas. It lets you explore massive unstructured data sets in your web browser. Since we're here at NeurIPS, a lot of people seem to respond to calling it massive clickable T-SNE as a service.

• 2:03:21Speaker 00

  Yes. I was actually thinking, is it T-SNE or UMAP?

• 2:03:24Speaker 12

  Yeah, so it turns out if you squint closely enough, they're the same algorithm up to a choice of low dimensional kernel. So we optimize the T-SNE objective function. One of our pieces of IP is we have the world's fastest optimizer for it. So if you take, say, the Nvidia Rapids UMAP implementation, which is kind of the fastest version of this in the wild, off the shelf and run it on Wikipedia on the biggest machine on AWS, it's going to take you a couple of days to actually get that map. And we can do it in about four hours.

• 2:03:50Speaker 00

  Excellent.

• 2:03:51Speaker 12

  Yeah, it lets you make the maps part of your iterative daily workflow as opposed to having to wait a week to get them.

• 2:03:55Speaker 00

  Nice. We'll throw a video on this on the show notes, but maybe you could sort of narratively show what you're showing. Like you showed a TikTok example and a Twitter example, right? So these are really for visualizing massive multimodal data sets?

• 2:04:07Speaker 12

  Yeah, so the fundamental thesis behind the tool is that the shape of data that people have has fundamentally changed as a result of generative. Instead of having these like big Excel spreadsheets of tabular things, you know, have vectors plus metadata. And we need to rethink visualization, you know, and the implications of that for the visualization stack. You are kind of seeing at the database layer, they're starting to penetrate with vector DBs and stuff, but I think there's going to be radical kind of implications for that change all the way up the stack. And so you can use it on, you know, getting back to your original question, Twitter data, TikTok data, images, sounds, text, anything that you can stuff into a vector, which is pretty much anything these days, you can map it, you can understand.

• 2:04:44Speaker 00

  Yeah. Can I bring my own custom embeddings and see the impact of that?

• 2:04:48Speaker 12

  You can. So there's two ways to get data into the platform. One way is bring your own embeddings. And then you just pip install Nomic from Nomic import atlas and then atlas dot map embeddings. You supply your embeddings, you supply metadata on top of them. And then a couple of minutes later, you'll get a web link back to a map where you can click on it and fly around it. If you just have raw data, we have a bunch of out of the box embedders that we develop and we work with partners to develop that you can use to to map it out the box as well.

• 2:05:12Speaker 00

  Yeah, and this is not open source, but GPT for all is.

• 2:05:16Speaker 12

  So there are aspects of the platform that are open source. The entire thing runs on a graphics engine that we developed called DeepScatter. It's the only tool out there that can render a billion point scatter plots in a web browser. And to do that, you have to, again, kind of fundamentally rethink how graphics in the browser works in the ground up. That is available source, but unfortunately is not fully open source.

• 2:05:33Speaker 00

  It's okay. Yeah, you don't have to apologize for anything.

• 2:05:36Speaker 12

  I do have to. You know, I wish we could open source everything, but like we are unfortunately subject to capitalism. And so we cannot, but in the limit, I would love to open source everything.

• 2:05:45Speaker 00

  I also maybe heard you in another introduction talk about this as like looker for language models. Like what like elaborate more about that like?

• 2:05:51Speaker 12

  Yeah.

• 2:05:52Speaker 00

  Are you? Do you have a query language? Like what what are you thinking about as the overall vision?

• 2:05:56Speaker 12

  Yeah, so I want to bring it back to the analogy of like the new shape of data disrupting the stack, right? So the first place we see it hitting is at the database layer. Things, you know, we see vector databases, there's a million of them nowadays. I think that that change is going to propagate all the way up the stack. And we are interested in, you know, what happens to the BI analytics, you know, visualization layer. And so really what we're thinking of this as is sort of like a tableau for unstructured data or a looker or Power BI or something like this. Where we've built the entire visualization system with embeddings as a first class citizen. And so that enables a lot of different actions. Some are already in the platform, some I can't tease yet, unfortunately. But having embeddings as a first class primitive enables a lot of like very, very useful things that you're not going to be able to get unless you have it.

• 2:06:38Speaker 00

  What do people use Atlas for? Like just maybe list out some more use cases that might not be obvious from people just thinking about visualization.

• 2:06:46Speaker 12

  Yes, we'll start with the most technical and we'll go the least technical. A lot of ML engineers use it to understand and evaluate their models and training data.

• 2:06:53Speaker 00

  Yeah.

• 2:06:53Speaker 12

  So we just did some work with Hugging Face on their Obelix dataset, which they use to train their Idefix model, doing some evaluation and training data analysis, looking at what areas of their.

• 2:07:03Speaker 00

  We actually interviewed those guys. I was in Paris and I talked to Leo and.

• 2:07:06Speaker 12

  And Victor.

• 2:07:07Speaker 00

  Yeah.

• 2:07:07Speaker 12

  Yeah, those guys are sick.

• 2:07:08Speaker 00

  Yeah.

• 2:07:09Speaker 12

  But yeah, so we worked with them on this and we discovered, you know, a couple of things in their training data. They should have like actually cleaned out of it. There was like a bunch of, you know, end of sentence tokens to be replaced that made it through stuff like this. Some really garbage content.

• 2:07:21Speaker 00

  Anomaly detection or is that up to people to code themselves?

• 2:07:24Speaker 12

  Yeah, so the anomalies usually manifest as like the little moons on the outside of the map.

• 2:07:28Speaker 00

  Sure, okay.

• 2:07:29Speaker 12

  And then you can just like hit them with the little lasso tool and stuff like this. But one of the things about the hugging face map that I found fascinating was because we supply like a topic model out of the box, you can look at things like are there topics where the loss tends to like cluster together? And for the hugging face model, there was this high loss mode in the poetry topic, which I thought was super interesting. And so I've got two theories for it. One is that poetry includes the distinct subversion of like common linguistic patterns. And so of course, language models will be bad at it. But the more perhaps optimistic theory is that poetry captures something that's fundamentally human that the machines have not grasped yet.

• 2:08:05Speaker 12

  The pragmatic version, I think, is probably what's happening, but I like to be optimistic, so.

• 2:08:09Speaker 00

  Idefix is a visual data set and you were also.

• 2:08:12Speaker 12

  Yeah.

• 2:08:12Speaker 00

  Okay, so they have poetry in there.

• 2:08:14Speaker 12

  Yep.

• 2:08:14Speaker 00

  Interesting.

• 2:08:15Speaker 12

  It's sort of interleaved webpages of like it'll be an image and then some poetry.

• 2:08:18Speaker 00

  Right, so that's the more technical side.

• 2:08:20Speaker 12

  And then coming down to the less technical side, you know, a lot of our customer base at this point is like consulting type companies. And they find the product really useful for connecting domain experts with large data sets. So generally what will happen is you'll have these domain experts, be it like a doctor or someone in regulation, someone with subject matter expertise, that'll be handed this massive set of documents from a client and be like, I don't even know where to start. I don't even know what's in this. And so a couple of the consulting partners we work with actually now have a KPI that's like timed to Atlas, where it's like how quickly from the data set hitting the company does it get to Atlas? So that we can send an analyst the map and they can start to explore it. And so we're really excited about enabling sort of traditionally non-technical people to explore and analyze these massive data sets with this no-code interface.

• 2:09:04Speaker 00

  You know what you should do? You should hook up with Google. Doesn't Google have a big set of like publicly available data sets?

• 2:09:09Speaker 12

  Yeah, so we've actually done a couple of collaborations with Google Cloud on some of those data sets. We can maybe link the blog posts or something.

• 2:09:15Speaker 00

  Sure, yeah, yeah. Okay, awesome. Just in Europe in general, you've been here a number of years. What do you look for when you come to Europe? Any tips that you have for people coming to Europe?

• 2:09:22Speaker 12

  Oh, that's a good one. Yeah, um, big tip is just like if you see someone cool, like they're probably nice, so chase them down and like have them talk to you.

• 2:09:31Speaker 00

  Shove a microphone in their face.

• 2:09:32Speaker 12

  Yeah, yeah. No, I love it. But it was like my second nerves or something. I saw Oriol Vinyals walk by. And he had just done like the um Starcraft stuff. Um and I was like, okay, this guy is sick. He's doing some really cutting edge stuff. So I like ran up and asked him for life advice. And he was so down to earth and like chatted with me for a bunch of time about like modeling and life and you know how to think about my career and stuff. And so like, yeah, if you see a hero, like shoot your shot.

• 2:09:57Speaker 00

  Yeah, yeah, very, very, very cool. Um, any papers that you're keen on this year or like maybe really affected you in previous years?

• 2:10:05Speaker 12

  Oh, that's a good one too. Um, this year, I think Q-Lora is here, which I think is like a very, very interesting.

• 2:10:10Speaker 00

  Tomorrow.

• 2:10:11Speaker 12

  Yeah, yeah, yeah. It's a very interesting um set of implications for like the low resource world.

• 2:10:16Speaker 00

  Can you elaborate?

• 2:10:16Speaker 12

  Yeah. So one of the things we think a lot about at Nomic is the accessibility of AI technology. And one of the things that's become very clear to us, and I think everyone this year is like, there's the GPU rich and the GPU poor. And so I think methods that make it so that anyone in the world can interact with this technology like Q-Lora are just like so, so, so valuable. And so I think any research into like low resource training of models and low resource deployment of models is just going to be so good for everybody, especially like the open source community. I really love to see it.

• 2:10:45Speaker 00

  Yeah. Uh, you just reminded me. So talking about, we forgot to talk about GPT for all.

• 2:10:49Speaker 12

  Yeah.

• 2:10:50Speaker 00

  Very, very uh early win, I think, in the overall space of things. But now, more recently in my mind, uh Lama CPP has come out to be its own platform.

• 2:10:58Speaker 12

  Yep.

• 2:10:59Speaker 00

  Uh, all Lama is emerging as like a thing like, there's a bunch of ways in which people run models locally. How should people think about GPT for all in context of all that?

• 2:11:07Speaker 12

  Yeah. So one thing that a lot of people don't realize is that a lot of the core contributors to Lama CPP actually work at Nomic.

• 2:11:13Speaker 00

  Yeah.

• 2:11:13Speaker 12

  And so I guess the the operating advice here is just like play nice with open source, right? Like GPT for all is this thing that's going to be free forever for our community. We're going to keep trying to improve it as you know, our Discord recommends and as people call for it. But you know, if we can do things like go and you know, contribute to other open source projects that are high impact, we're going to, right? And so the hope here is that like, you know, as economic pressures apply, like open source stays collaborative is really, really the goal for us, I think.

• 2:11:41Speaker 00

  Okay, cool. Well, that's it. Any other last words? What are you looking for? How do people find you?

• 2:11:46Speaker 12

  Yeah, you can follow us on Twitter at @gnomic_AI. You can also find our website, gnomic.ai.

• 2:11:52Speaker 00

  Hiring engineers, researchers?

• 2:11:54Speaker 12

  Yeah, we're always looking for super interesting people. Yeah, come chat about interesting things in our Discord, really. You can find our website and stuff. But really, the best way to get involved is like make some apps, do some open source work. Like, a lot of the people that we hired in this last kind of spree of hiring were like big open source contributors. And so like, yeah, just give back to the community and then, you know, we'll try and find you and boost you.

• 2:12:17Speaker 00

  Yeah, awesome. Well, thanks so much for your time.

• 2:12:18Speaker 12

  Yeah, take care.

• 2:12:20Speaker 00

  I think the way that Gnomics embracing and supporting open source AI is encouraging, and I think more companies should learn from that. But they're definitely far from the only open source AI company out there. Lightning AI is one of the oldest, I guess, if you can call that old, in the space. And I happened to catch Luca, the CTO, at their booth. And at Europe, they were there to launch Lightning Studio, which is their new development environment. Hey Luca, welcome. Good to see that you guys are launching a new product today.

• 2:12:44Speaker 13

  Yeah, sure. It's super exciting. It's the result of many months, if not years, of work and realizations.

• 2:12:51Speaker 00

  So maybe let's establish a baseline. Most people will have heard of PyTorch Lightning. What was the evolution to Lightning AI?

• 2:12:56Speaker 13

  Yeah, so PyTorch Lightning is a very healthy, has a very healthy community of people using it. We are 5.5 downloads, about 80 million downloads in, sorry, 5.5 million downloads, of course, per month. About 80 million downloads in total. And it's one of the frameworks that comes from the era of traditional quote-unquote "deep learning" that is one of the main actors in the gen AI space. Because, for example, stable diffusion was trained using PyTorch Lightning, a bunch of models. PyTorch Lightning powers Nemo from Nvidia.

• 2:13:30Speaker 00

  Yeah, they're a custom chip design language model.

• 2:13:33Speaker 13

  Yeah, so basically, PyTorch Lightning has evolved and grown into gen AI. And with the release of 2.0, 2.1, we've tried to make it better and better for use cases in which you have very large models and you have a hard time not going out of memory. So, and to distribute it with, PyTorch Lightning has always been very...

• 2:14:00Speaker 13

  Focus on distributed trainings, one of the things that it did the best. But when models get very, very large, I think that's where we improved a lot this year. We also launched Fabric, Lightning Fabric, which is a, it's a framework, it's a companion framework to PyTorch Lightning, where you get all the constituents of the Lightning trainer, but now you can write your own training loops. So for people doing very optimized stuff, very bespoke, I don't know, you know, collect calls, they want to place them where they want, they want to fully own the training loop, or they're doing stuff like reinforcement learnings where it's not the traditional training loop,

• 2:14:41Speaker 13

  you can still do it with a trainer, but it's a bit, a bit more difficult. Then Fabric lets you just write your four loops. But we'll still abstract away strategies, precision plugins, the login, the aggregation of metrics and all this stuff. I like to think about these frameworks as frameworks that reduce the surface area for mistakes. Because mistakes nowadays, well, a few years ago-

• 2:15:07Speaker 00

  Costs a lot of money.

• 2:15:08Speaker 13

  Exactly, right? They cost a lot of time to a PhD student. Right now, they cost a lot of money, so you don't want to make too many mistakes there. And Torch Matrix is a third project that we have that is very healthy and is powering a lot of the metric computation. Again, you don't want to compute accuracy and aggregate it across a multi-machine job in a wrong way, right? Because you'll get wrong indications and it's really easy to do it incorrectly.

• 2:15:37Speaker 00

  Yeah.

• 2:15:38Speaker 13

  And this year, we started doing-

• 2:15:41Speaker 00

  Yeah, as you mentioned, these are mostly open source.

• 2:15:43Speaker 13

  Yeah, these are all hundred percent open source.

• 2:15:45Speaker 00

  I think Fabric in particular was pretty popular.

• 2:15:47Speaker 13

  Yeah, yeah. So Fabric has powered also our language model repositories, LitLama and LitGPT. Basically, back when Llama was originally released, I, and me, of course, and-

• 2:16:04Speaker 00

  The weights were leaked. The weights were leaked.

• 2:16:07Speaker 13

  Yeah, the weights, yeah, exactly, exactly. But at some point, there was a model- Being published by Meta as well, it was a GPL license, so we didn't really like that. And so we say, why don't we take NanoGPT, because I was working with NanoGPT at the time, and turn it into Llama. And that started the whole thing of, you know, minimal implementation, single file. You have everything there, you have no layers to go through to understand how your layers are. And that became something that it became very popular within many organizations. So it's still very popular. So the LLM efficiency challenge, the starter kit had LIDGPT in it.

• 2:16:50Speaker 13

  And LIDGPT today supports many models, many different models. But it's very easy to get to the bottom of your, of the implementation of every single thing.

• 2:16:58Speaker 00

  Yeah.

• 2:16:59Speaker 13

  So it's very hackable.

• 2:16:59Speaker 00

  Yeah.

• 2:17:00Speaker 13

  It's one of my philosophies.

• 2:17:01Speaker 00

  File.

• 2:17:01Speaker 13

  Make it hackable because before you make it fast.

• 2:17:04Speaker 00

  Yeah.

• 2:17:04Speaker 13

  Right? Because more people can contribute to it and we have contributors being very successful. There have been initiatives of models being pre-trained using that, like TinyLama. And 360AI, I think, a few days ago came out and they said they used LIDLama to pre-train their seven billion parameter model. So it's, it's great. And a lot of those learnings went back into fabric and back into PyTorch Lightning.

• 2:17:31Speaker 00

  Yeah.

• 2:17:32Speaker 13

  And this is how we're kind of growing organically towards supporting AI use cases.

• 2:17:36Speaker 00

  There's an example of one of those learnings from those outside, outside usage of LIDGPT. Oh, LIDLama, I guess.

• 2:17:44Speaker 13

  Sorry, can you say?

• 2:17:45Speaker 00

  What's an example of one of those learnings that you got from like 360 contributing back?

• 2:17:51Speaker 13

  Well, 360 is very, is very young.

• 2:17:53Speaker 00

  Vanilla.

• 2:17:53Speaker 13

  In the sense that we, we just learned, I think, the day before yesterday that they used us.

• 2:17:58Speaker 00

  So it's great.

• 2:17:59Speaker 13

  So it's great. We're very happy about that. From TinyLama, they, they did some optimizations on top of our code.

• 2:18:07Speaker 13

  And they trained a 1.1 billion parameter model on three trillion tokens. I think they're still doing that. I don't think they're, they're done. And then some of the improvements that they made and we upstreamed it to our, like for example,

• 2:18:25Speaker 13

  I think chunk cross entropy, some kernels that they were using.

• 2:18:30Speaker 13

  And then we were happy to see that even our dataset that we optimized because it chunks your data and it can stream very quickly work for them. So it's kind of a mutual thing that we're doing. And also, all the quantization support, for example, right now Fabric and PyTorch Lightning support bits and bytes natively. And it's basically one of the few solutions where you can use quantization on any kind of model and not just the model that the original authors decided to support. Yeah, it's kind of flexible. But here today, I think the main thing we're doing today is launching our platform.

• 2:19:11Speaker 00

  Yeah, you just launched Studio today.

• 2:19:13Speaker 13

  Yeah, exactly. Livi Studio, again, is a result of many months and years of work. It basically makes you build AI at scale, but it feels like it's your laptop. So to me, it's kind of the first time I've seen a platform not leaking the abstraction of orchestration on the cloud and so on. Literally, there's nothing to learn.

• 2:19:36Speaker 00

  You put VS Code in the browser and then you add all the...

• 2:19:40Speaker 13

  You can even connect from your local VS Code and code there. You have the whole machine, like it's a whole machine.

• 2:19:45Speaker 00

  Yeah, it's a cloud development environment.

• 2:19:47Speaker 13

  Exactly.

• 2:19:48Speaker 00

  Okay.

• 2:19:49Speaker 13

  And it's built around reproducible environment.

• 2:19:51Speaker 00

  Similar to like code spaces.

• 2:19:52Speaker 13

  Exactly. But when you go in there, it's not that you need to build your Docker container. You just go in there, you present it with a machine, you can start working immediately. If you pip install something and then you decide to switch instance type, your dependencies will carry over. Or if you decide to duplicate my Studio, everything that I set up on that Studio, from the environment to the data, the code, the checkpoints eventually that I put there, you will find them. And so you will spend zero time setting up your environment.

• 2:20:22Speaker 00

  So are you snapshotting memory? How does this work?

• 2:20:24Speaker 13

  Well, that's secret sauce.

• 2:20:26Speaker 00

  Secret sauce? You're not using containers, you said?

• 2:20:29Speaker 13

  Yeah, well, I mean, we do like what if if you think about it, then it it all it's not like

• 2:20:36Speaker 00

  Yeah, it's not, okay.

• 2:20:37Speaker 13

  Fundamentally, but it's very complicated to actually get the perfect experience out of it.

• 2:20:42Speaker 00

  Like, maybe this is a thousand. Design constraints, what what are you optimizing for?

• 2:20:47Speaker 13

  We're optimizing for velocity, so we don't want people to spend time thinking about things they shouldn't think about.

• 2:20:54Speaker 00

  Yeah.

• 2:20:55Speaker 13

  Like when you're on, you're coding on a machine and you now want four GPUs, you should just be able to get four GPUs and keep working, right?

• 2:21:03Speaker 00

  Yeah.

• 2:21:03Speaker 13

  Without thinking about, oh, now I need to go to a console, spin things up, for my environment, attach drives, like, these are all things you shouldn't think about. And again, goes back to uh limiting the surface area for mistakes, right? Because you can do what you're good at and not do what you shouldn't mess with.

• 2:21:22Speaker 00

  Uh, it's like the fabric philosophy that's expanded to the dev environment.

• 2:21:26Speaker 13

  Exactly, exactly. Uh, yeah, we're very excited. You can do small things like in Colab, except that your data is persistent and uh you can switch off and switch on and everything will be there.

• 2:21:39Speaker 00

  Yeah.

• 2:21:39Speaker 13

  Um, or you can even train large language models.

• 2:21:43Speaker 00

  Yeah. Uh, what what are the the larger customers doing? Uh, you know, what are you, what are you doing for them? Because I feel like this might be targeted towards the smaller customers.

• 2:21:52Speaker 13

  No, actually, we run with, we we work with very big financial institutions. Uh, and we're actually pre-training models ourselves, so the scale at which you can operate is is pretty large. It's not like, it looks like something that you can do small stuff with, which is true.

• 2:22:12Speaker 00

  Yeah.

• 2:22:12Speaker 13

  It's super smooth there, but if you need to launch a job on a hundred GPUs, you can just do it, provided that you have the machines. But we manage reservations, so we can, we can target reservations, or you can attach your own cloud account. And negotiate your quotas with your cloud provider and we'll just orchestrate on your cloud account.

• 2:22:31Speaker 00

  Yeah. Any cloud providers you would shout out as particularly, I mean, people know the big three clouds, but

• 2:22:36Speaker 13

  Yeah.

• 2:22:36Speaker 00

  Any other providers that you would shout out as very good partners to work with so far?

• 2:22:40Speaker 13

  Right now, we're being focusing on AWS.

• 2:22:42Speaker 00

  Yeah.

• 2:22:42Speaker 13

  We'll expand, of course, because.

• 2:22:45Speaker 00

  Yeah, everyone needs everyone else.

• 2:22:46Speaker 13

  Yeah, exactly.

• 2:22:47Speaker 00

  Apparently Oracle's doing very well.

• 2:22:49Speaker 13

  Yeah, yeah. We talked to Oracle. We talked to most of the cloud providers out there. To us, it's more a matter of sequencing.

• 2:22:56Speaker 00

  Yeah.

• 2:22:57Speaker 13

  We have a very good relationship, of course, with AWS right now. They've been supporting us for the launch and so on. But surely we'll get into getting the best machines for for customers.

• 2:23:09Speaker 00

  Yeah.

• 2:23:10Speaker 13

  And in the near future, we'll also support on-prem clusters in terms of orchestration, like Slurm as an orchestrator or as a scheduler.

• 2:23:19Speaker 00

  People have very mixed feelings about Slurm.

• 2:23:21Speaker 13

  Well, yeah, but in this case, you now have to deal with it, right?

• 2:23:23Speaker 00

  Yes.

• 2:23:24Speaker 13

  We take away the pain and you still can orchestrate on top of that. It's still not out, but it will it will come in a.

• 2:23:35Speaker 13

  Yeah, in the near future.

• 2:23:36Speaker 00

  Yeah.

• 2:23:36Speaker 13

  We're already doing that with with some companies.

• 2:23:39Speaker 00

  Yeah. So I want to talk about that workshop that you're doing on Friday.

• 2:23:42Speaker 13

  Yeah.

• 2:23:42Speaker 00

  Uh, the efficiency challenge.

• 2:23:44Speaker 13

  Yep.

• 2:23:44Speaker 00

  Was it motivated by a paper? I saw I saw like a cramming paper that like what what's the maximum you can do with one day of compute, something like that?

• 2:23:51Speaker 13

  Yeah, so we noticed because they, Mark Starfim and and the other organizers ended up choosing Lgpt as one of the models for StarKit and we were happy about it, of course.

• 2:24:05Speaker 00

  Yeah.

• 2:24:05Speaker 13

  And so we say, yeah, what we can do together. And we ended up, and we really like the principle. So we believe you know, smaller models can empower, you know, people a lot, getting control and understanding how to extract value from AI.

• 2:24:24Speaker 13

  And so I think there's a dire need of. Consolidation, getting smaller, getting more efficiency and getting the result you want in the shortest time as possible. And that, that's how the velocity will increase and how eventually open source will will get there on par, if not beyond what's available in the closed source world. And so we are fully supportive of that. The way we ended up contributing is we maintained a public leaderboard. And it was a nice experience because we integrated with Discord. There was, there is a Discord channel. This is for the efficiency challenge Discord? Yeah, for, exactly, the efficiency challenge Discord.

• 2:25:09Speaker 13

  And we set up an agent that was running on a few of our machines. And people could submit to, through a DM to the bot, so that the bot would then spin out a work, a job, run things in a queue, get back the results from evaluation. And then essentially get a ranking on where they were. Yeah. And that, I think, helped a lot motivating people to compete against each other. Yeah. But in a very constructive way. And to be honest, in the first month, it's been very, very bumpy with that. Like, it was all new infrastructure and we were doing kind of it in spare time.

• 2:25:52Speaker 13

  So it wasn't the best of, of the experience. So together with the community that wasn't there, they helped us figure out like what was not going well. And I think at the end, we had like more than a thousand submissions. That's amazing. That were successful. Yeah. Like many more submissions that didn't complete was a submission problems. Yeah. Like user code problems, that there were more than a dozen submissions that were actually fully evaluated. Yeah. On that leaderboard. Did you, so?

• 2:26:22Speaker 00

  The challenge is over, but I don't know if you've done the analysis on anything to learn from the winning entries?

• 2:26:28Speaker 13

  So we've been, the rest of the organizers, yes, they have put a lot of effort in the next three weeks, four weeks, to reevaluate everything, run the first ones from scratch, and they've done an amazing job. And some of the code that we wrote for the public leaderboard ended up being part of this evaluation infrastructure. I was very, very busy with the launch, so I didn't participate there, so I'm super curious on Friday.

• 2:26:59Speaker 00

  Yeah, yeah, okay.

• 2:27:00Speaker 13

  What will be the tales from the winners?

• 2:27:06Speaker 00

  Yes.

• 2:27:07Speaker 13

  But I must say that all the community has been super nice, they were super constructive. I remember when Mistral first came out, there was a huge thread of people just getting in there, analyzing it, trying to find it. It was so much energy that we definitely want to push it forward and we'll create public studios with evaluation frameworks on them. And we want to enable this kind of...

• 2:27:30Speaker 00

  So studios are shareable, of course?

• 2:27:32Speaker 13

  Yes.

• 2:27:32Speaker 00

  Yes, that makes sense.

• 2:27:33Speaker 13

  Not only shareable between you and me, but also community-wide.

• 2:27:36Speaker 00

  Yeah, yeah, yeah, yeah, yeah.

• 2:27:37Speaker 13

  So there will be a lot of things, you can go in there, use your pre-credits to just run the evaluation on your model, you can do that.

• 2:27:44Speaker 00

  Yeah, great. So last question, I've been asking everybody this. You've been coming to NeurIPS for many years, what are your NeurIPS tips?

• 2:27:53Speaker 13

  Oh wow, yeah, go to posters, which I cannot do, but...

• 2:27:58Speaker 00

  But, okay, so I've been to like, you know, there's been three poster sessions so far. The popular ones are just crowded, there's just no way.

• 2:28:05Speaker 13

  Yeah, I think there's not, yeah, I don't like to go to popular ones.

• 2:28:08Speaker 00

  Yeah, okay, the less popular ones, and just talk to them.

• 2:28:11Speaker 13

  So I had so many super engaging conversations, even in topics like, even if something is not apparently something that you should focus on.

• 2:28:21Speaker 00

  Yeah.

• 2:28:22Speaker 13

  Your brain will oxygenate itself a lot, and typically after these conferences, I always come back with a... I had full of ideas and so I would say get enriched as much as you can by interacting with people, having very honest conversations with them.

• 2:28:38Speaker 00

  Yeah.

• 2:28:40Speaker 00

  I had a, I had an off-the-record conversation with one of the presenters who said like, "Yeah, I don't, like, this paper I'm presenting, I don't believe in it." I was like, "Wow, that's really honest."

• 2:28:51Speaker 00

  Because they already submitted it months ago, right? And since then, the world has moved on.

• 2:28:55Speaker 13

  Yeah, well, that's part of the, you know, the struggle. I don't know how it must be being a postdoc or a PhD student or even a master's student nowadays in AI. It must be like so stressful.

• 2:29:13Speaker 13

  Back in the day was a lot easier.

• 2:29:15Speaker 00

  You know, the prizes have got bigger.

• 2:29:16Speaker 13

  Yeah, for sure.

• 2:29:18Speaker 00

  Okay, well, thank you so much for your time and congratulations on your launch.

• 2:29:21Speaker 13

  Yeah, I think we'll, you know, meet each other on the platform maybe.

• 2:29:25Speaker 00

  Yes, I definitely will try it out.

• 2:29:27Speaker 13

  Thanks a lot. Bye.

• 2:29:28Speaker 00

  A few of my AI engineer and ML engineer friends checked out Lightning Studio and they were pretty impressed. So I'm personally interested to check it out next year. But last and not least, I want to give the mic to Jay Alammar of Cohere and LLM University, but more importantly, of the Illustrated Transformer and is now writing a new book. We're here with Jay Alammar, educator of many things. I've learned so much from you. I literally, it's one of those moments where at NeurIPS, you just kind of see someone walking and I'm like, is that Jay? And then I had to get your attention a few times. But it's so nice to finally meet you.

• 2:30:00Speaker 14

  It's great to meet you and great to be here and sort of meet all kinds of brilliant folks. I've watched your stuff and sort of been watching the revolution and how you're helping sort of crystallize people's thinking about this new domain of AI engineering. And I think the title is very helpful as categorizing that class.

• 2:30:20Speaker 00

  Yeah, trying to do for my audience what you do for just general ML education, which is, I think, something that you've really done an incredible job of.

• 2:30:28Speaker 14

  Yeah, no, that's wonderful. It's what the community needs, definitely. As machine learning and AI sort of goes out of research and goes into industry.

• 2:30:37Speaker 00

  Exactly. It's a different persona, different background. The kind of people, one of the reasons I'm doing this recording is like the kind of people that follow my stuff don't come here.

• 2:30:45Speaker 14

  Yeah.

• 2:30:46Speaker 00

  And maybe they shouldn't, right? Like this is some of this is like too in-depth.

• 2:30:50Speaker 14

  True.

• 2:30:50Speaker 00

  But I'm curious, like, you know, you've been to many NeurIPS. Like, what is your general take of the vibe? What are people talking about? What's top of mind?

• 2:30:56Speaker 14

  There's a lot of LLMs that's interesting to see.

• 2:30:59Speaker 00

  Suddenly, a lot of interest, right?

• 2:31:01Speaker 14

  Yes, that is, let's say, maybe possibly a new development in NERUs. That's the area that's growing. And a couple of interesting keywords or groups or directions are diffusion, even diffusion for text models, that's

• 2:31:17Speaker 00

  There was a paper on that yesterday.

• 2:31:18Speaker 14

  Yeah.

• 2:31:19Speaker 00

  I'm not too, what's the point of diffusion for text? Don't people want to stream things out?

• 2:31:24Speaker 14

  Well, I mean, if you think of, like, on the application side, autoregressive generation has some problems. So if you, the model makes a mistake with token five, you're stuck with that problem.

• 2:31:34Speaker 00

  Yes, that's what Tree of Thought solves, which the Tree of Thought guy was here.

• 2:31:38Speaker 14

  Yeah, it's always, yeah, it's one, let's say, one avenue.

• 2:31:40Speaker 00

  Yes.

• 2:31:41Speaker 14

  But it's like, maybe if you, if the model does not fall within, in a mistake in that way, you can unlock new sort of different applications. But also all the image generation stuff, that's really where.

• 2:31:54Speaker 00

  Make a plug, I actually had a, so there's a lot of house parties that happen after NERUs, which is fantastic. I ran into a guy from Mid Journey for the first time. They have this new storytelling section, and they are actually exploring text diffusion because of storytelling, because you have to generate a coherent story, just like you would an image.

• 2:32:10Speaker 14

  True, true.

• 2:32:11Speaker 00

  So I would buy that as a use case.

• 2:32:13Speaker 14

  That is fascinating. And then with the agent stuff, like, if you're interested in the future of agents, which is, there's a lot of the reasoning stuff. The reasoning research in NERUs will most likely sort of inform the upcoming, what's gonna happen in agents next. So chain of thought, tree of thought, that domain of research, for me, is very fascinating, because it's gonna be applied very quickly.

• 2:32:36Speaker 00

  Yeah.

• 2:32:36Speaker 14

  The React paper comes out, it's in LangChain, everybody's sort of using it. Everybody has a, has a sense of what agents are, but that really shows you the potential of what they're gonna be in the future. We're still in early days on agents.

• 2:32:49Speaker 00

  Any other, like, top of mind sessions? Did you go to the Chris Ray run this morning? I thought that was pretty cool.

• 2:32:54Speaker 14

  I have, I have that one and a couple of the other sort of keynotes. I'll be rewatching them.

• 2:32:59Speaker 00

  Yeah.

• 2:32:59Speaker 14

  But mostly, I'm just talking with people, recording video. That's been my, and sort of trying to orient myself. It's an overwhelming amount of content and people and posters and talks. And so I've been, yeah, looking at visualizations of, you know, these are the papers at NERUs, these are the ones that could be interesting to you.

• 2:33:17Speaker 00

  Yeah, people have published, like, T S N E things of them, and it's good, but like, it's not as good as just kind of seeing the vibes. And I actually think the conference organizers do a good job of curating, like, what the You know, oral session papers should be.

• 2:33:29Speaker 14

  True.

• 2:33:30Speaker 00

  Like, I generally found them, like, generally very insightful. I just found out about data comp from one of the oral sessions.

• 2:33:35Speaker 14

  Okay.

• 2:33:35Speaker 00

  I don't know if you've seen them.

• 2:33:36Speaker 14

  No.

• 2:33:36Speaker 00

  They're effectively a new ImageNet.

• 2:33:38Speaker 14

  Oh, nice.

• 2:33:39Speaker 00

  Which is like, oh, that's cool.

• 2:33:40Speaker 14

  Yeah.

• 2:33:41Speaker 00

  New benchmark. And, yeah, I mean, like, it's, to me, I'm taking it all in. So it's impressive how many people do so much work and you've never heard of them.

• 2:33:49Speaker 14

  That's true. That's true. Yeah.

• 2:33:51Speaker 00

  And they're conversant in all the techniques, all the papers, all the stuff that you see online.

• 2:33:54Speaker 14

  Yeah.

• 2:33:55Speaker 00

  They're just not online. And they just do research quietly and then, once a year, they show up here.

• 2:33:59Speaker 14

  Yeah, and sometimes you meet somebody here and you're like, and they would mention, they worked on that other paper. And it's a paper that you're very familiar with. And then you go into their Google Scholar or something and you're like, I've been reading this person's work for years, but the name never really sort of specifically popped up until you meet them in person. So that's why it's, yeah, it's definitely an interesting experience.

• 2:34:22Speaker 00

  Yeah, no particular order, but have you had those, like, any underrated person you would call out as like, hey, everyone should pay more attention to the work that this person's doing?

• 2:34:29Speaker 14

  One thing that comes across, which is why workshops are good. And we can, we can get into that sort of, like, there's David Bao's work on interpretability and editing language models and editing their knowledge was, was one thing that sort of really stood out to me after I've, you know, met David and sort of heard about, about Bao's work.

• 2:34:49Speaker 00

  Editing by editing weights?

• 2:34:50Speaker 14

  Yeah, they have a method of editing the model, exactly. Yes.

• 2:34:53Speaker 00

  There's a, is this the one where they played Go?

• 2:34:55Speaker 14

  This is Rome.

• 2:34:56Speaker 00

  And they flipped a...

• 2:34:57Speaker 14

  This is where they convince a model using that method that the Eiffel Tower is in Rome and not in Paris. And then they have subsequent methods of, let's say, if you make a hundred edits like that, the model degrades. So they have subsequent work on, okay, this is a better method to do many more of that. But also things like, I don't know if you've seen, like, low jet lens and sort of where in the model is this token being suggested? Like, is that at layer one or is that at layer five or that layer, that localization is, is, is interesting work.

• 2:35:30Speaker 00

  Yeah. So you do all these interviews on your YouTube? We'll send people there. Is this part of your work at Koh here or?

• 2:35:35Speaker 14

  A little bit, yes.

• 2:35:36Speaker 00

  How does this, what is your deal?

• 2:35:37Speaker 14

  Yes. That's true.

• 2:35:40Speaker 14

  So these, a bunch of them go on the Coher YouTube channel.

• 2:35:43Speaker 00

  Yeah.

• 2:35:44Speaker 14

  Um, and the Coher socials as well. Uh, so yeah, my work at, at Coher, I get to learn in public, basically.

• 2:35:50Speaker 00

  I love that.

• 2:35:50Speaker 14

  So Coher builds language models for embeddings, re-ranking and generation. And through selling them, I get to see what, how industry is solving problems with them. And that, to me, is very fascinating. So to see the technology coming out of research and then how it goes into industry. Uh, and how people use them, how people sort of need to be educated on, on the best ways of, of using them. That, that view to me is, uh, something I'm, I'm lucky to have.

• 2:36:14Speaker 00

  Yeah, yeah, it's a good job to, to get out, to be honest. If you love that stuff, you might as well get paid to do it. You probably don't notice, but I actually have written a book on learning in public and I am a big advocate of getting developers and engineers to, to learn in public.

• 2:36:25Speaker 14

  Well, you do it so well, so yeah.

• 2:36:27Speaker 00

  Yeah, this is, this is my way of doing it. Um, well, the final piece is, uh, you know, you, you've written a lot of foundational work on like transformers. A lot of people are talking about, uh, the state space models and what happens after the transformers. Um, do you have personal views on that?

• 2:36:39Speaker 14

  Not yet, not yet. I'm on the lookout, so yes, there are always new ideas that, you know, there's maybe poster number five hundred and two here that nobody paid attention to.

• 2:36:47Speaker 00

  Yes.

• 2:36:47Speaker 14

  Maybe in six months, we'll, we'll see that, oh, it crushes everything else on, uh, so that is always something you can never sort of expect.

• 2:36:56Speaker 00

  Yeah, my fa- favorite fact about the transformers paper is it, it itself was not accepted as, it was like a poster only paper, right? I don't know the story behind that.

• 2:37:04Speaker 14

  It was a big deal for machine translation.

• 2:37:06Speaker 00

  Yes.

• 2:37:06Speaker 14

  But it's like, okay, yeah, there's a, there's a cool.

• 2:37:08Speaker 00

  It's one of many, right?

• 2:37:09Speaker 14

  Yeah.

• 2:37:09Speaker 00

  Yeah, we already have BERT.

• 2:37:10Speaker 14

  One new attention method. We had Badano attention and we had Long attention and like now we have also, you know, one more. But then BERT comes out and it's like, okay, this is more than translation. Then GPT comes out and it's like, oh, it's a new generation tech.

• 2:37:22Speaker 00

  I'm still missing a good survey paper on everything that happened since Attention is All You Need. Like the, the evolution towards the modern decoder only paradigm.

• 2:37:30Speaker 14

  Okay.

• 2:37:30Speaker 00

  And I, I feel like someone needs to write that.

• 2:37:34Speaker 00

  Everyone's too busy inventing new things to stop and write like what happened.

• 2:37:37Speaker 14

  Because it's a massive thing and there are a few people who, because there's a lot of work on different kinds of attention and different, that, for the transformer specifically, how to improve it for this problem, for that problem. But one thing that I'm doing is rewriting the Illustrated Transformer with the ideas that have stood the test of time since then. So it's like, six years after, which ideas? So people are using rope, uh, flash attention, uh, and then yeah, rope and alibi, let's say positional encodings, localized attention. So some ideas that are, people are continuing to use over and over. Multi, multi query and group multi query, yes.

• 2:38:15Speaker 00

  And then sliding window?

• 2:38:17Speaker 14

  Not yet. So it's in Mistral, but I, you know, I need, we maybe need to see it in more work.

• 2:38:21Speaker 00

  My conspiracy theory about Mistral is that, so the Mistral paper heavily features sliding window attention and everyone is like bullshit, like come on.

• 2:38:28Speaker 14

  I mean, because you see it, you saw that for two years after the transformer, everybody was proposing new ideas. And if you put this, uh, in the transformer, it does better on this. But then what stands the test of time? The vanilla transformer really stood the test of time, uh, and did better than even a lot of these enhanced, quote unquote, enhancements. But these ones, let's say, stood the test of time. So these, this rewriting is gonna be, you know, part of the book I'm sort of currently rewriting.

• 2:38:54Speaker 00

  Are you writing a book?

• 2:38:55Speaker 14

  Yes. Writing a book for O'Reilly called, uh, Hands-on Large Language Models, including this as eight chapters. So if you want an updated Illustrated Transformer, that's gonna be a part of it.

• 2:39:05Speaker 00

  When you launch your book, you should come on and do a full episode with us.

• 2:39:08Speaker 14

  Amazing.

• 2:39:08Speaker 00

  Yeah, and, yeah, exactly. And then, uh, just general Europe tips, you know, as a, as an attendee, like how, if people are coming for the first time, what would you advise them to do?

• 2:39:15Speaker 14

  I really love the visualization by, I posted about this, by, by Hendrik Strobel, uh, and Ben Hoover of the TSCNI of all the papers, but also it's clustered. So if you're interested in language models, that is clustered.

• 2:39:29Speaker 00

  I use that for my planning.

• 2:39:30Speaker 14

  It's, it's so useful. Like, these things are, are absolutely incredible. I got to meet Hendrik and they have so, a lot of very interesting ideas there. It helps you sort of, um, orient yourself. And I've, I've also seen work kind of like it, but take, where you can do semantic search, um, on, so you can say, you know, Agent papers and it doesn't need to match the actual keywords. With Go here, we have a demo on RAG on Europe's papers as well. So you can ask a question, you're like, okay, I'm interested in LLM efficiency. It'll say, okay, this paper, this paper, this paper. And it's retrieval augmented sort of generation. So these are the three tools, but I think we need a lot more of these tools to make sense of.

• 2:40:09Speaker 00

  Yeah, I need it for the meetups too. You know, in the, in your conference app, there's all these like meetups for very specific things.

• 2:40:14Speaker 14

  That's true.

• 2:40:15Speaker 00

  Uh, I started one for, uh, for Singaporeans because I'm a Singaporean in tech. And, uh, yeah, there's just a, there's a bunch of very, very specific like running meetups. Like nothing to do with tech specifically, but like, you know, this is also a social event, right? Like that you're.

• 2:40:27Speaker 14

  Okay, yeah. You wouldn't happen to be at MNLP?

• 2:40:30Speaker 00

  No.

• 2:40:30Speaker 14

  Because some people did that because it was like last week and some people went to MNLP in Singapore.

• 2:40:36Speaker 00

  Uh huh.

• 2:40:36Speaker 14

  And then flew back here.

• 2:40:38Speaker 00

  Yeah, it's a tough call. Uh, yeah, I'm not gonna do that.

• 2:40:40Speaker 14

  That's rough.

• 2:40:41Speaker 00

  Yeah.

• 2:40:41Speaker 14

  That's rough.

• 2:40:42Speaker 00

  Well, thanks very much. It's a pleasure to have you on, a pleasure to meet in person.

• 2:40:45Speaker 14

  So good to meet, love your work.

• 2:40:47Speaker 00

  Thank you.

• 2:40:47Speaker 14

  Keep, keep doing it.

• 2:40:48Speaker 00

  Any, any calls to action for people if, while you're here?

• 2:40:50Speaker 14

  Well, I'm J L Amar on, on Twitter and YouTube and we have LLM University. LLM dot university, like I collaborate with Luis and Mior Amar to educate about.

• 2:41:01Speaker 00

  Some of the best YouTube, like very short, but like very comprehensive, authoritative.

• 2:41:06Speaker 14

  Very lucky to collaborate with these, with these, with these folks.

• 2:41:08Speaker 00

  Yeah.

• 2:41:10Speaker 14

  It's incredible, but yeah, thanks.

• 2:41:11Speaker 00

  Yeah, thanks for doing all that.

• 2:41:12Speaker 14

  With you.

• 2:41:12Speaker 00

  Thank you.

• 2:41:13Speaker 14

  Appreciate it.

• 2:41:14Speaker 00

  Okay, and that's it for our New York's coverage and for Latent Space Pod in 2023. We are still doing a listener survey, so if you are listening through here, you're definitely a big fan. We definitely want to hear from you, what you like about the podcast, what you want to hear for 2024. We've got a couple of really good episodes already recorded for the start of 2024. So we're gonna start the year strong and come up to the one year anniversary of Latent Space. So thanks for all your support. Have a wonderful end of the year and we'll see you soon. DJ, hit the outro.