Open Source AI is AI we can Trust — with Soumith Chintala of Meta AI

Speaker CFPs https://docs.google.com/forms/d/e/1FAIpQLScc-47zw-tWjYbhAkwTeLy_-MQW3L-3uwtaVnEzudrEZcQ7bg/viewform?usp=sf_link and Sponsor Guides mailto:[email protected] are now available for AIE World’s Fair — join us on June 25-27 for the biggest AI Engineer conference of 2024 https://twitter.com/aiDotEngineer/status/1754929063993737721! Soumith Chintala https://twitter.com/soumithchintala/ needs no introduction in the ML world — his insights are incredibly accessible across Twitter https://twitter.com/soumithchintala, LinkedIn https://www.linkedin.com/in/soumith/, podcasts https://www.listennotes.com/search/?q=soumith%20chintala&sort_by_date=0&scope=episode&offset=0&language=Any%20language&len_min=0, and conference talks https://www.youtube.com/results?search_query=soumith+chintala (in this pod we’ll assume you’ll have caught up on the History of PyTorch pod from last year https://www.listennotes.com/podcasts/the-gradient/soumith-chintala-pytorch-SLhENkyKZa5/ and cover different topics). He’s well known as the creator of PyTorch, but he's more broadly the Engineering Lead on AI Infra, PyTorch, and Generative AI at Meta. Soumith was one of the earliest supporters https://twitter.com/soumithchintala/status/1671267150101721090 of Latent Space (and more recently AI News https://x.com/soumithchintala/status/1764853209498034537?s=20), and we were overjoyed to catch up with him on his latest SF visit for a braindump of the latest AI topics, reactions to some of our past guests, and why Open Source AI is personally so important to him. Life in the GPU-Rich Lane Back in January, Zuck went on Instagram to announce their GPU wealth: by the end of 2024, Meta will have 350k H100s. By adding all their GPU clusters, you'd get to 600k H100-equivalents of compute. At FP16 precision, that's ~1,200,000 PFLOPS. If we used George Hotz's (previous guest https://www.latent.space/p/geohot!) "Person of Compute" https://geohot.github.io/blog/jekyll/update/2023/04/26/a-person-of-compute.html measure, Meta now has 60k humans of compute in their clusters. Occasionally we get glimpses into the GPU-rich life; on a recent ThursdAI chat https://sub.thursdai.news/p/thursdai-feb-15-2024-openai-changes, swyx prompted PaLM tech lead Yi Tay to write down what he missed most from Google, and he commented https://x.com/swyx/status/1765173028084936744?s=20 that UL2 20B was trained by accidentally leaving the training job running for a month, because hardware failures are so rare in Google. Meta AI’s Epic LLM Run Before Llama broke the internet, Meta released an open source LLM in May 2022, OPT-175B https://ai.meta.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b/, which was notable for how “open” it was - right down to the logbook https://news.ycombinator.com/item?id=31260665! They used only 16 NVIDIA V100 GPUs and Soumith agrees that, with hindsight, it was likely under-trained for its parameter size. In Feb 2023 (pre Latent Space pod), Llama was released https://ai.meta.com/blog/large-language-model-llama-meta-ai/, with a 7B version trained on 1T tokens alongside 65B and 33B versions trained on 1.4T tokens. The Llama authors included Guillaume Lample and Timothée Lacroix, who went on to start Mistral https://mistral.ai/. July 2023 was Llama2 time https://about.fb.com/news/2023/07/llama-2/ (which we covered https://www.latent.space/p/llama2#details!): 3 model sizes, 7B, 13B, and 70B, all trained on 2T tokens. The three models accounted for a grand total of 3,311,616 GPU hours for all pre-training work. CodeLlama https://ai.meta.com/blog/code-llama-large-language-model-coding/ followed shortly after, a fine-tune of Llama2 specifically focused on code generation use cases. The family had models in the 7B, 13B, 34B, and 70B size, all trained with 500B extra tokens of code and code-related data, except for 70B which is trained on 1T. All of this on top of other open sourced models like Segment Anything (one of our early hits!) https://www.latent.space/p/segment-anything-roboflow, Detectron, Detectron 2, DensePose, and Seamless, and in one year, Meta transformed from a company people made fun of for its “metaverse” investments to one of the key players in the AI landscape and its stock has almost tripled since (about $830B in market value created in the past year). Why Open Source AI The obvious question is why Meta would spend hundreds of millions on its AI efforts and then release them for free. Zuck has addressed this in public statements https://twitter.com/jeremyphoward/status/1753200925458735110: But for Soumith, the motivation is even more personal: “I'm irrationally interested in open source. I think open source has that fundamental way to distribute opportunity in a way that is very powerful. Like, I grew up in India… And knowledge was very centralized, but I saw that evolution of knowledge slowly getting decentralized. And that ended up helping me learn quicker and faster for like zero dollars. And I think that was a strong reason why I ended up where I am. So like that, like the open source side of things, I always push regardless of like what I get paid for, like I think I would do that as a passion project on the side… …I think at a fundamental level, the most beneficial value of open source is that you make the distribution to be very wide. It's just available with no friction and people can do transformative things in a way that's very accessible. Maybe it's open source, but it has a commercial license and I'm a student in India. I don't care about the license. I just don't even understand the license. But like the fact that I can use it and do something with it is very transformative to me… …Like, okay, I again always go back to like I'm a student in India with no money. What is my accessibility to any of these closed source models? At some scale I have to pay money. That makes it a non-starter and stuff. And there's also the control issue: I strongly believe if you want human aligned AI, you want all humans to give feedback. And you want all humans to have access to that technology in the first place. And I actually have seen, living in New York, whenever I come to Silicon Valley, I see a different cultural bubble. We like the way Soumith put it last year https://x.com/soumithchintala/status/1733310147743662107?s=20: Closed AI “rate-limits against people's imaginations and needs”! What It Takes For Open Source AI to Win However Soumith doesn’t think Open Source will simply win by popular demand. There is a tremendous coordination problem with the decentralized nature of the open source AI development right now: nobody is collecting the valuable human feedback in the way that OpenAI or Midjourney are doing. “Open source in general always has a coordination problem. If there's a vertically integrated provider with more resources, they will just be better coordinated than open source. And so now open source has to figure out how to have coordinated benefits. And the reason you want coordinated benefits is because these models are getting better based on human feedback. And if you see with open source models, like if you go to the /r/localllama subreddit, like there's so many variations of models that are being produced from, say, Nous research. I mean, like there's like so many variations built by so many people. And one common theme is they're all using these fine-tuning or human preferences datasets that are very limited and they're not sufficiently diverse. And you look at the other side, say front-ends like Oobabooga or like Hugging Chat or Ollama, they don't really have feedback buttons. All the people using all these front-ends, they probably want to give feedback, but there's no way for them to give feedback… So we're just losing all of this feedback. Maybe open source models are being as used as GPT is at this point in like all kinds of, in a very fragmented way, like in aggregate all the open source models together are probably being used as much as GPT is, maybe close to that. But the amount of feedback that is driving back into the open source ecosystem is like negligible, maybe less than 1% of like the usage. So I think like some, like the blueprint here I think is you'd want someone to create a sinkhole for the feedback… I think if we do that, if that actually happens, I think that probably has a real chance of the open source models having a runaway effect against OpenAI, I think like there's a clear chance we can take at truly winning open source.” If you’re working on solving open source coordination, please get in touch https://twitter.com/swyx! Show Notes * Soumith Chintal https://twitter.com/soumithchintalaa Twitter * History of PyTorch episode on Gradient Podcast https://thegradientpub.substack.com/p/soumith-chintala-pytorch * The Llama Ecosystem https://ai.meta.com/blog/llama-2-updates-connect-2023/ * Apple's MLX https://github.com/ml-explore/mlx * Neural ODEs (Ordinary Differential Equations) https://arxiv.org/abs/1806.07366 * AlphaGo https://www.youtube.com/watch?v=WXuK6gekU1Y * LMSys arena https://chat.lmsys.org/ * Dan Pink's "Drive" https://www.danpink.com/books/drive/ * Robotics projects: * Dobb-E https://dobb-e.com/ * OK Robot https://ok-robot.github.io/ * Yann LeCun https://twitter.com/ylecun * Yangqing Jia of Lepton AI https://twitter.com/jiayq * Ed Catmull https://en.wikipedia.org/wiki/Edwin_Catmull * George Hotz on Latent Space https://www.latent.space/p/geohot * Chris Lattner on Latent Space https://www.latent.space/p/modular * Guillaume Lample https://twitter.com/guillaumelample?lang=en * Yannic Kilcher https://twitter.com/ykilcher of OpenAssistant * LMSys https://chat.lmsys.org/?leaderboard * Alex Atallah https://twitter.com/xanderatallah of OpenRouter * Carlo Sferrazza's 3D tactile research https://sferrazza.cc/ * Alex Wiltschko of Osmo https://www.osmo.ai/ * Tangent by Alex Wiltschko https://github.com/google/tangent * Lerrel Pinto https://www.lerrelpinto.com/ - Robotics Timestamps * [00:00:00] Introductions * [00:00:51] Extrinsic vs Intrinsic Success * [00:02:40] Importance of Open Source and Its Impact * [00:03:46] PyTorch vs TinyGrad * [00:08:33] Why PyTorch is the Switzerland of frameworks * [00:10:27] Modular's Mojo + PyTorch? * [00:13:32] PyTorch vs Apple's MLX * [00:16:27] FAIR / PyTorch Alumni * [00:18:50] How can AI inference providers differentiate? * [00:21:41] How to build good benchmarks and learnings from AnyScale's * [00:25:28] Most interesting unexplored ideas * [00:28:18] What people get wrong about synthetic data * [00:35:57] Meta AI's evolution * [00:38:42] How do you allocate 600,000 GPUs? * [00:42:05] Even the GPU Rich are GPU Poor * [00:47:31] Meta's MTIA silicon * [00:50:09] Why we need open source * [00:59:00] Open source's coordination problem for feedback gathering * [01:08:59] Beyond text generation * [01:15:37] Osmo and the Future of Smell Recognition Technology Transcript Alessio [00:00:00]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO in residence at Decibel Partners https://decibel.vc/, and I'm joined by my co-host Swyx, founder of Smol AI https://smol.ai/. Swyx [00:00:15]: Hey, and today we have in the studio Soumith Chintala, welcome. Soumith [00:00:17]: Thanks for having me. Swyx [00:00:18]: On one of your rare visits from New York where you live. You got your start in computer vision at NYU with Yann LeCun. That was a very fortuitous start. I was actually listening to your interview on the Gradient podcast. So if people want to know more about the history of Soumith, history of PyTorch, they can go to that podcast. We won't spend that much time there, but I just was marveling at your luck, or I don't know if it's your luck or your drive to find AI early and then find the right quality mentor because I guess Yan really sort of introduced you to that world. Soumith [00:00:51]: Yeah, I think you're talking about extrinsic success, right? A lot of people just have drive to do things that they think is fun, and a lot of those things might or might not be extrinsically perceived as good and successful. I think I just happened to like something that is now one of the coolest things in the world or whatever. But if I happen, the first thing I tried to become was a 3D VFX artist, and I was really interested in doing that, but I turned out to be very bad at it. So I ended up not doing that further. But even if I was good at that, whatever, and I ended up going down that path, I probably would have been equally happy. It's just like maybe like the perception of, oh, is this person successful or not might be different. I think like after a baseline, like your happiness is probably more correlated with your intrinsic stuff. Swyx [00:01:44]: Yes. I think Dan Pink has this book on drive that I often refer to about the power of intrinsic motivation versus extrinsic and how long extrinsic lasts. It's not very long at all. But anyway, now you are an investor in Runway, so in a way you're working on VFX. Yes. Soumith [00:02:01]: I mean, in a very convoluted way. Swyx [00:02:03]: It reminds me of Ed Catmull. I don't know if you guys know, but he actually tried to become an animator in his early years and failed or didn't get accepted by Disney and then went and created Pixar and then got bought by Disney and created Toy Story. So you joined Facebook in 2014 and eventually became a creator and maintainer of PyTorch. And there's this long story there you can refer to on the gradient. I think maybe people don't know that you also involved in more sort of hardware and cluster decision affair. And we can dive into more details there because we're all about hardware this month. Yeah. And then finally, I don't know what else, like what else should people know about you on a personal side or professional side? Soumith [00:02:40]: I think open source is definitely a big passion of mine and probably forms a little bit of my identity at this point. I'm irrationally interested in open source. I think open source has that fundamental way to distribute opportunity in a way that is very powerful. Like, I grew up in India. I didn't have internet for a while. In college, actually, I didn't have internet except for GPRS or whatever. And knowledge was very centralized, but I saw that evolution of knowledge slowly getting decentralized. And that ended up helping me learn quicker and faster for zero dollars. And I think that was a strong reason why I ended up where I am. So the open source side of things, I always push regardless of what I get paid for, like I think I would do that as a passion project on the side. Swyx [00:03:35]: Yeah, that's wonderful. Well, we'll talk about the challenges as well that open source has, open models versus closed models. Maybe you want to touch a little bit on PyTorch before we move on to the sort of Meta AI in general. PyTorch vs Tinygrad tradeoffs Alessio [00:03:46]: Yeah, we kind of touched on PyTorch in a lot of episodes. So we had George Hotz from TinyGrad. He called PyTorch a CISC and TinyGrad a RISC. I would love to get your thoughts on PyTorch design direction as far as, I know you talk a lot about kind of having a happy path to start with and then making complexity hidden away but then available to the end user. One of the things that George mentioned is I think you have like 250 primitive operators in PyTorch, I think TinyGrad is four. So how do you think about some of the learnings that maybe he's going to run into that you already had in the past seven, eight years almost of running PyTorch? Soumith [00:04:24]: Yeah, I think there's different models here, but I think it's two different models that people generally start with. Either they go like, I have a grand vision and I'm going to build a giant system that achieves this grand vision and maybe one is super feature complete or whatever. Or other people say they will get incrementally ambitious, right? And they say, oh, we'll start with something simple and then we'll slowly layer out complexity in a way that optimally applies Huffman coding or whatever. Like where the density of users are and what they're using, I would want to keep it in the easy, happy path and where the more niche advanced use cases, I'll still want people to try them, but they need to take additional frictional steps. George, I think just like we started with PyTorch, George started with the incrementally ambitious thing. I remember TinyGrad used to be, like we would be limited to a thousand lines of code and I think now it's at 5,000. So I think there is no real magic to which why PyTorch has the kind of complexity. I think it's probably partly necessitated and partly because we built with the technology available under us at that time, PyTorch is like 190,000 lines of code or something at this point. I think if you had to rewrite it, we would probably think about ways to rewrite it in a vastly simplified way for sure. But a lot of that complexity comes from the fact that in a very simple, explainable way, you have memory hierarchies. You have CPU has three levels of caches and then you have DRAM and SSD and then you have network. Similarly, GPU has several levels of memory and then you have different levels of network hierarchies, NVLink plus InfiniBand or Rocky or something like that, right? And the way the flops are available on your hardware, they are available in a certain way and your computation is in a certain way and you have to retrofit your computation onto both the memory hierarchy and like the flops available. When you're doing this, it is actually a fairly hard mathematical problem to do this setup, like you find the optimal thing. And finding the optimal thing is, what is optimal depends on the input variables themselves. So like, okay, what is the shape of your input tensors and what is the operation you're trying to do and various things like that. Finding that optimal configuration and writing it down in code is not the same for every input configuration you have. Like for example, just as the shape of the tensors change, let's say you have three input tensors into a Sparstar product or something like that. The shape of each of these input tensors will vastly change how you do this optimally placing this operation onto the hardware in a way that will get you maximal throughput. So a lot of our complexity comes from writing out hundreds of configurations for each single PyTorch operator and templatizing these things and symbolically generating the final CUDA code or CPU code. There's no way to avoid it because mathematically we haven't found symbolic ways to do this that also keep compile time near zero. You can write a very simple framework, but then you also should be willing to eat the long compile time. So if searching for that optimal performance at runtime, but that's the trade off. There's no, like, I don't think unless we have great breakthroughs George's vision is achievable, he should be thinking about a narrower problem such as I'm only going to make this for work for self-driving car connets or I'm only going to make this work for LLM transformers of the llama style. Like if you start narrowing the problem down, you can make a vastly simpler framework. But if you don't, if you need the generality to power all of the AI research that is happening and keep zero compile time and in all these other factors, I think it's not easy to avoid the complexity. Pytorch vs Mojo Alessio [00:08:33]: That's interesting. And we kind of touched on this with Chris Lattner when he was on the podcast. If you think about frameworks, they have the model target. They have the hardware target. They have different things to think about. He mentioned when he was at Google, TensorFlow trying to be optimized to make TPUs go brr, you know, and go as fast. I think George is trying to make especially AMD stack be better than ROCm. How come PyTorch has been such as Switzerland versus just making Meta hardware go brr? Soumith [00:09:00]: First, Meta is not in the business of selling hardware. Meta is not in the business of cloud compute. The way Meta thinks about funding PyTorch is we're funding it because it's net good for Meta to fund PyTorch because PyTorch has become a standard and a big open source project. And generally it gives us a timeline edge. It gives us leverage and all that within our own work. So why is PyTorch more of a Switzerland rather than being opinionated? I think the way we think about it is not in terms of Switzerland or not. We actually the way we articulate it to all hardware vendors and software vendors and all who come to us being we want to build a backend in core for PyTorch and ship it by default is we just only look at our user side of things. Like if users are using a particular piece of hardware, then we want to support it. We very much don't want to king make the hardware side of things. So as the MacBooks have GPUs and as that stuff started getting increasingly interesting, we pushed Apple to push some engineers and work on the NPS support and we spend significant time from Meta funded engineers on that as well because a lot of people are using the Apple GPUs and there's demand. So we kind of mostly look at it from the demand side. We never look at it from like oh which hardware should we start taking opinions on. Swyx [00:10:27]: Is there a future in which, because Mojo or Modular Mojo is kind of a superset of Python, is there a future in which PyTorch might use Mojo features optionally? Soumith [00:10:36]: I think it depends on how well integrated it is into the Python ecosystem. So if Mojo is like a pip install and it's readily available and users feel like they can use Mojo so smoothly within their workflows in a way that just is low friction, we would definitely look into that. Like in the same way PyTorch now depends on Triton, OpenAI Triton, and we never had a conversation that was like huh, that's like a dependency. Should we just build a Triton of our own or should we use Triton? It almost doesn't, like those conversations don't really come up for us. The conversations are more well does Triton have 10,000 dependencies and is it hard to install? We almost don't look at these things from a strategic leverage point of view. We look at these things from a user experience point of view, like is it easy to install? Is it smoothly integrated and does it give enough benefits for us to start depending on it? If so, yeah, we should consider it. That's how we think about it. Swyx [00:11:37]: You're inclusive by default as long as it meets the minimum bar of, yeah, but like maybe I phrased it wrongly. Maybe it's more like what problems would you look to solve that you have right now? Soumith [00:11:48]: I think it depends on what problems Mojo will be useful at. Swyx [00:11:52]: Mainly a performance pitch, some amount of cross compiling pitch. Soumith [00:11:56]: Yeah, I think the performance pitch for Mojo was like, we're going to be performant even if you have a lot of custom stuff, you're going to write arbitrary custom things and we will be performant. And that value proposition is not clear to us from the PyTorch side to consider it for PyTorch. So PyTorch, it's actually not 250 operators, it's like a thousand operators. PyTorch exposes about a thousand operators and people kind of write their ideas in the thousand operators of PyTorch. Mojo is like, well, maybe it's okay to completely sidestep those thousand operators of PyTorch and just write it in a more natural form. Just write raw Python, write for loops or whatever, right? So from the consideration of how do we intersect PyTorch with Mojo, I can see one use case where you have custom stuff for some parts of your program, but mostly it's PyTorch. And so we can probably figure out how to make it easier for say Torch.compile to smoothly also consume Mojo subgraphs and like, you know, the interoperability being actually usable, that I think is valuable. But Mojo as a fundamental front end would be replacing PyTorch, not augmenting PyTorch. So in that sense, I don't see a synergy in more deeply integrating Mojo. Pytorch vs MLX Swyx [00:13:21]: So call out to Mojo whenever they have written something in Mojo and there's some performance related thing going on. And then since you mentioned Apple, what should people think of PyTorch versus MLX? Soumith [00:13:32]: I mean, MLX is early and I know the folks well, Ani used to work at FAIR and I used to chat with him all the time. He used to be based out of New York as well. The way I think about MLX is that MLX is specialized for Apple right now. It has a happy path because it's defined its product in a narrow way. At some point MLX either says we will only be supporting Apple and we will just focus on enabling, you know, there's a framework if you use your MacBook, but once you like go server side or whatever, that's not my problem and I don't care. For MLS, it enters like the server side set of things as well. Like one of these two things will happen, right? If the first thing will happen, like MLX's overall addressable market will be small, but it probably do well within that addressable market. If it enters the second phase, they're going to run into all the same complexities that we have to deal with. They will not have any magic wand and they will have more complex work to do. They probably wouldn't be able to move as fast. Swyx [00:14:44]: Like having to deal with distributed compute? Soumith [00:14:48]: Distributed, NVIDIA and AMD GPUs, like just like having a generalization of the concept of a backend, how they treat compilation with plus overheads. Right now they're deeply assumed like the whole NPS graph thing. So they need to think about all these additional things if they end up expanding onto the server side and they'll probably build something like PyTorch as well, right? Like eventually that's where it will land. And I think there they will kind of fail on the lack of differentiation. Like it wouldn't be obvious to people why they would want to use it. Swyx [00:15:24]: I mean, there are some cloud companies offering M1 and M2 chips on servers. I feel like it might be interesting for Apple to pursue that market, but it's not their core strength. Soumith [00:15:33]: Yeah. If Apple can figure out their interconnect story, maybe, like then it can become a thing. Swyx [00:15:40]: Honestly, that's more interesting than the cars. Yes. Soumith [00:15:43]: I think the moat that NVIDIA has right now, I feel is that they have the interconnect that no one else has, like AMD GPUs are pretty good. I'm sure there's various silicon that is not bad at all, but the interconnect, like NVLink is uniquely awesome. I'm sure the other hardware providers are working on it, but- Swyx [00:16:04]: I feel like when you say it's uniquely awesome, you have some appreciation of it that the rest of us don't. I mean, the rest of us just like, you know, we hear marketing lines, but what do you mean when you say NVIDIA is very good at networking? Obviously they made the acquisition maybe like 15 years ago. Soumith [00:16:15]: Just the bandwidth it offers and the latency it offers. I mean, TPUs also have a good interconnect, but you can't buy them. So you have to go to Google to use it. PyTorch Mafia Alessio [00:16:27]: Who are some of the other FAIR PyTorch alumni that are building cool companies? I know you have Fireworks AI, Lightning AI, Lepton, and Yangqing, you knew since college when he was building Coffee? Soumith [00:16:40]: Yeah, so Yangqing and I used to be framework rivals, PyTorch, I mean, we were all a very small close-knit community back then. Caffe, Torch, Theano, Chainer, Keras, various frameworks. I mean, it used to be more like 20 frameworks. I can't remember all the names. CCV by Liu Liu, who is also based out of SF. And I would actually like, you know, one of the ways it was interesting is you went into the framework guts and saw if someone wrote their own convolution kernel or they were just copying someone else's. There were four or five convolution kernels that were unique and interesting. There was one from this guy out of Russia, I forgot the name, but I remembered who was awesome enough to have written their own kernel. And at some point there, I built out these benchmarks called ConNet benchmarks. They're just benchmarking all the convolution kernels that are available at that time. It hilariously became big enough that at that time AI was getting important, but not important enough that industrial strength players came in to do these kinds of benchmarking and standardization. Like we have MLPerf today. So a lot of the startups were using ConNet benchmarks in their pitch decks as like, oh, you know, on ConNet benchmarks, this is how we fare, so you should fund us. I remember Nirvana actually was at the top of the pack because Scott Gray wrote amazingly fast convolution kernels at that time. Very interesting, but separate times. But to answer your question, Alessio, I think mainly Lepton, Fireworks are the two most obvious ones, but I'm sure the fingerprints are a lot wider. They're just people who worked within the PyTorch Cafe2 cohort of things and now end up at various other places. Swyx [00:18:50]: I think as a, both as an investor and a people looking to build on top of their services, it's a uncomfortable slash like, I don't know what I don't know pitch. Because I've met Yang Tsing and I've met Lin Chao. Yeah, I've met these folks and they're like, you know, we are deep in the PyTorch ecosystem and we serve billions of inferences a day or whatever at Facebook and now we can do it for you. And I'm like, okay, that's great. Like, what should I be wary of or cautious of when these things happen? Because I'm like, obviously this experience is extremely powerful and valuable. I just don't know what I don't know. Like, what should people know about like these sort of new inference as a service companies? Soumith [00:19:32]: I think at that point you would be investing in them for their expertise of one kind. So if they've been at a large company, but they've been doing amazing work, you would be thinking about it as what these people bring to the table is that they're really good at like GPU programming or understanding the complexity of serving models once it hits a certain scale. You know, various expertise like from the infra and AI and GPUs point of view. What you would obviously want to figure out is whether their understanding of the external markets is clear, whether they know and understand how to think about running a business, understanding how to be disciplined about making money or, you know, various things like that. Swyx [00:20:23]: Maybe I'll put it like, actually I will de-emphasize the investing bit and just more as a potential customer. Oh, okay. Like, it's more okay, you know, you have PyTorch gods, of course. Like, what else should I know? Soumith [00:20:37]: I mean, I would not care about who's building something. If I'm trying to be a customer, I would care about whether... Swyx [00:20:44]: Benchmarks. Soumith [00:20:44]: Yeah, I use it and it's usability and reliability and speed, right? Swyx [00:20:51]: Quality as well. Soumith [00:20:51]: Yeah, if someone from some random unknown place came to me and say, user stuff is great. Like, and I have the bandwidth, I probably will give it a shot. And if it turns out to be great, like I'll just use it. Benchmark drama Swyx [00:21:07]: Okay, great. And then maybe one more thing about benchmarks, since we already brought it up and you brought up Confident Benchmarks. There was some recent drama around AnyScale. AnyScale released their own benchmarks and obviously they look great on their own benchmarks, but maybe didn't give the other... I feel there are two lines of criticism. One, which is they didn't test some apples for apples on the kind of endpoints that the other providers, that they are competitors with, on their benchmarks and that is due diligence baseline. And then the second would be more just optimizing for the right thing. You had some commentary on it. I'll just kind of let you riff. Soumith [00:21:41]: Yeah, I mean, in summary, basically my criticism of that was AnyScale built these benchmarks for end users to just understand what they should pick, right? And that's a very good thing to do. I think what they didn't do a good job of is give that end user a full understanding of what they should pick. Like they just gave them a very narrow slice of understanding. I think they just gave them latency numbers and that's not sufficient, right? You need to understand your total cost of ownership at some reasonable scale. Not oh, one API call is one cent, but a thousand API calls are 10 cents. Like people can misprice to cheat on those benchmarks. So you want to understand, okay, like how much is it going to cost me if I actually subscribe to you and do like a million API calls a month or something? And then you want to understand the latency and reliability, not just from one call you made, but an aggregate of calls you've made over several various times of the day and times of the week. And the nature of the workloads, is it just some generic single paragraph that you're sending that is cashable? Or is it like testing of real world workload? I think that kind of rigor, like in presenting that benchmark wasn't there. It was a much more narrow sliver of what should have been a good benchmark. That was my main criticism. And I'm pretty sure if before they released it, they showed it to their other stakeholders who would be caring about this benchmark because they are present in it, they would have easily just pointed out these gaps. And I think they didn't do that and they just released it. So I think those were the two main criticisms. I think they were fair and Robert took it well. Swyx [00:23:40]: And he took it very well. And we'll have him on at some point and we'll discuss it. But I think it's important for, I think the market being maturing enough that people start caring and competing on these kinds of things means that we need to establish what best practice is because otherwise everyone's going to play dirty. Soumith [00:23:55]: Yeah, absolutely. My view of the LLM inference market in general is that it's the laundromat model. Like the margins are going to drive down towards the bare minimum. It's going to be all kinds of arbitrage between how much you can get the hardware for and then how much you sell the API and how much latency your customers are willing to let go. You need to figure out how to squeeze your margins. Like what is your unique thing here? Like I think Together and Fireworks and all these people are trying to build some faster CUDA kernels and faster, you know, hardware kernels in general. But those modes only last for a month or two. These ideas quickly propagate. Swyx [00:24:38]: Even if they're not published? Soumith [00:24:39]: Even if they're not published, the idea space is small. So even if they're not published, the discovery rate is going to be pretty high. It's not like we're talking about a combinatorial thing that is really large. You're talking about Llama style LLM models. And we're going to beat those to death on a few different hardware SKUs, right? Like it's not even we have a huge diversity of hardware you're going to aim to run it on. Now when you have such a narrow problem and you have a lot of people working on it, the rate at which these ideas are going to get figured out is going to be pretty rapid. Swyx [00:25:15]: Is it a standard bag of tricks? Like the standard one that I know of is, you know, fusing operators and- Soumith [00:25:22]: Yeah, it's the standard bag of tricks on figuring out how to improve your memory bandwidth and all that, yeah. Alessio [00:25:28]: Any ideas instead of things that are not being beaten to death that people should be paying more attention to? Novel PyTorch Applications Swyx [00:25:34]: One thing I was like, you know, you have a thousand operators, right? Like what's the most interesting usage of PyTorch that you're seeing maybe outside of this little bubble? Soumith [00:25:41]: So PyTorch, it's very interesting and scary at the same time, but basically it's used in a lot of exotic ways, like from the ML angle, what kind of models are being built? And you get all the way from state-based models and all of these things to stuff nth order differentiable models, like neural ODEs and stuff like that. I think there's one set of interestingness factor from the ML side of things. And then there's the other set of interesting factor from the applications point of view. It's used in Mars Rover simulations, to drug discovery, to Tesla cars. And there's a huge diversity of applications in which it is used. So in terms of the most interesting application side of things, I think I'm scared at how many interesting things that are also very critical and really important it is used in. I think the scariest was when I went to visit CERN at some point and they said they were using PyTorch and they were using GANs at the same time for particle physics research. And I was scared more about the fact that they were using GANs than they were using PyTorch, because at that time I was a researcher focusing on GANs. But the diversity is probably the most interesting. How many different things it is being used in. I think that's the most interesting to me from the applications perspective. From the models perspective, I think I've seen a lot of them. Like the really interesting ones to me are where we're starting to combine search and symbolic stuff with differentiable models, like the whole AlphaGo style models is one example. And then I think we're attempting to do it for LLMs as well, with various reward models and search. I mean, I don't think PyTorch is being used in this, but the whole alpha geometry thing was interesting because again, it's an example of combining the symbolic models with the gradient based ones. But there are stuff like alpha geometry that PyTorch is used at, especially when you intersect biology and chemistry with ML. In those areas, you want stronger guarantees on the output. So yeah, maybe from the ML side, those things to me are very interesting right now. Swyx [00:28:03]: Yeah. People are very excited about the alpha geometry thing. And it's kind of like, for me, it's theoretical. It's great. You can solve some Olympia questions. I'm not sure how to make that bridge over into the real world applications, but I'm sure people smarter than me will figure it out. Synthetic Data vs Symbolic Models Soumith [00:28:18]: Let me give you an example of it. You know how the whole thing about synthetic data will be the next rage in LLMs is a thing? Swyx [00:28:27]: Already is a rage. Soumith [00:28:28]: Which I think is fairly misplaced in how people perceive it. People think synthetic data is some kind of magic wand that you wave and it's going to be amazing. Synthetic data is useful in neural networks right now because we as humans have figured out a bunch of symbolic models of the world or made up certain symbolic models because of human innate biases. So we've figured out how to ground particle physics in a 30 parameter model. And it's just very hard to compute as in it takes a lot of flops to compute, but it only has 30 parameters or so. I mean, I'm not a physics expert, but it's a very low rank model. We built mathematics as a field that basically is very low rank. Language, a deep understanding of language, like the whole syntactic parse trees and just understanding how language can be broken down and into a formal symbolism is something that we figured out. So we basically as humans have accumulated all this knowledge on these subjects, either synthetic, we created those subjects in our heads, or we grounded some real world phenomenon into a set of symbols. But we haven't figured out how to teach neural networks symbolic world models directly. The only way we have to teach them is generating a bunch of inputs and outputs and gradient dissenting over them. So in areas where we have the symbolic models and we need to teach all the knowledge we have that is better encoded in the symbolic models, what we're doing is we're generating a bunch of synthetic data, a bunch of input output pairs, and then giving that to the neural network and asking it to learn the same thing that we already have a better low rank model of in gradient descent in a much more over-parameterized way. Outside of this, like where we don't have good symbolic models, like synthetic data obviously doesn't make any sense. So synthetic data is not a magic wand where it'll work in all cases in every case or whatever. It's just where we as humans already have good symbolic models off. We need to impart that knowledge to neural networks and we figured out the synthetic data is a vehicle to impart this knowledge to. So, but people, because maybe they don't know enough about synthetic data as a notion, but they hear, you know, the next wave of data revolution is synthetic data. They think it's some kind of magic where we just create a bunch of random data somehow. They don't think about how, and then they think that's just a revolution. And I think that's maybe a gap in understanding most people have in this hype cycle. Swyx [00:31:23]: Yeah, well, it's a relatively new concept, so. Oh, there's two more that I'll put in front of you and then you can see what you respond. One is, you know, I have this joke that it's, you know, it's only synthetic data if it's from the Mistral region of France, otherwise it's just a sparkling distillation, which is what news research is doing. Like they're distilling GPT-4 by creating synthetic data from GPT-4, creating mock textbooks inspired by Phi 2 and then fine tuning open source models like Llama. And so I don't know, I mean, I think that's, should we call that synthetic data? Should we call it something else? I don't know. Soumith [00:31:57]: Yeah, I mean, the outputs of LLMs, are they synthetic data? They probably are, but I think it depends on the goal you have. If your goal is you're creating synthetic data with the goal of trying to distill GPT-4's superiority into another model, I guess you can call it synthetic data, but it also feels like disingenuous because your goal is I need to copy the behavior of GPT-4 and- Swyx [00:32:25]: It's also not just behavior, but data set. So I've often thought of this as data set washing. Like you need one model at the top of the chain, you know, unnamed French company that has that, you know, makes a model that has all the data in it that we don't know where it's from, but it's open source, hey, and then we distill from that and it's great. To be fair, they also use larger models as judges for preference ranking, right? So that is, I think, a very, very accepted use of synthetic. Soumith [00:32:53]: Correct. I think it's a very interesting time where we don't really have good social models of what is acceptable depending on how many bits of information you use from someone else, right? It's like, okay, you use one bit. Is that okay? Yeah, let's accept it to be okay. Okay, what about if you use 20 bits? Is that okay? I don't know. What if you use 200 bits? I don't think we as society have ever been in this conundrum where we have to be like, where is the boundary of copyright or where is the boundary of socially accepted understanding of copying someone else? We haven't been tested this mathematically before, Swyx [00:33:38]: in my opinion. Whether it's transformative use. Yes. So yeah, I think this New York Times opening eye case is gonna go to the Supreme Court and we'll have to decide it because I think we never had to deal with it before. And then finally, for synthetic data, the thing that I'm personally exploring is solving this great stark paradigm difference between rag and fine tuning, where you can kind of create synthetic data off of your retrieved documents and then fine tune on that. That's kind of synthetic. All you need is variation or diversity of samples for you to fine tune on. And then you can fine tune new knowledge into your model. I don't know if you've seen that as a direction for synthetic data. Soumith [00:34:13]: I think you're basically trying to, what you're doing is you're saying, well, language, I know how to parametrize language to an extent. And I need to teach my model variations of this input data so that it's resilient or invariant to language uses of that data. Swyx [00:34:32]: Yeah, it doesn't overfit on the wrong source documents. Soumith [00:34:33]: So I think that's 100% synthetic. You understand, the key is you create variations of your documents and you know how to do that because you have a symbolic model or like some implicit symbolic model of language. Swyx [00:34:48]: Okay. Alessio [00:34:49]: Do you think the issue with symbolic models is just the architecture of the language models that we're building? I think maybe the thing that people grasp is the inability of transformers to deal with numbers because of the tokenizer. Is it a fundamental issue there too? And do you see alternative architectures that will be better with symbolic understanding? Soumith [00:35:09]: I am not sure if it's a fundamental issue or not. I think we just don't understand transformers enough. I don't even mean transformers as an architecture. I mean the use of transformers today, like combining the tokenizer and transformers and the dynamics of training, when you show math heavy questions versus not. I don't have a good calibration of whether I know the answer or not. I, you know, there's common criticisms that are, you know, transformers will just fail at X. But then when you scale them up to sufficient scale, they actually don't fail at that X. I think there's this entire subfield where they're trying to figure out these answers called like the science of deep learning or something. So we'll get to know more. I don't know the answer. Meta AI and Llama 2/3 Swyx [00:35:57]: Got it. Let's touch a little bit on just Meta AI and you know, stuff that's going on there. Maybe, I don't know how deeply you're personally involved in it, but you're our first guest with Meta AI, which is really fantastic. And Llama 1 was, you know, you are such a believer in open source. Llama 1 was more or less the real breakthrough in open source AI. The most interesting thing for us covering on this, in this podcast was the death of Chinchilla, as people say. Any interesting insights there around the scaling models for open source models or smaller models or whatever that design decision was when you guys were doing it? Soumith [00:36:31]: So Llama 1 was Guillaume Lample and team. There was OPT before, which I think I'm also very proud of because we bridged the gap in understanding of how complex it is to train these models to the world. Like until then, no one really in gory detail published. Swyx [00:36:50]: The logs. Soumith [00:36:51]: Yeah. Like, why is it complex? And everyone says, oh, it's complex. But no one really talked about why it's complex. I think OPT was cool. Swyx [00:37:02]: I met Susan and she's very, very outspoken. Yeah. Soumith [00:37:05]: We probably, I think, didn't train it for long enough, right? That's kind of obvious in retrospect. Swyx [00:37:12]: For a 175B. Yeah. You trained it according to Chinchilla at the time or? Soumith [00:37:17]: I can't remember the details, but I think it's a commonly held belief at this point that if we trained OPT longer, it would actually end up being better. Llama 1, I think, was Guillaume Lample and team Guillaume is fantastic and went on to build Mistral. I wasn't too involved in that side of things. So I don't know what you're asking me, which is how did they think about scaling loss and all of that? Llama 2, I was more closely involved in. I helped them a reasonable amount with their infrastructure needs and stuff. And Llama 2, I think, was more like, let's get to the evolution. At that point, we kind of understood what we were missing from the industry's understanding of LLMs. And we needed more data and we needed more to train the models for longer. And we made, I think, a few tweaks to the architecture and we scaled up more. And that was Llama 2. I think Llama 2, you can think of it as after Guillaume left, the team kind of rebuilt their muscle around Llama 2. And Hugo, I think, who's the first author is fantastic. And I think he did play a reasonable big role in Llama 1 as well. Soumith [00:38:35]: And he overlaps between Llama 1 and 2. So in Llama 3, obviously, hopefully, it'll be awesome. Alessio [00:38:42]: Just one question on Llama 2, and then we'll try and fish Llama 3 spoilers out of you. In the Llama 2 paper, the loss curves of the 34 and 70B parameter, they still seem kind of steep. Like they could go lower. How, from an infrastructure level, how do you allocate resources? Could they have just gone longer or were you just, hey, this is all the GPUs that we can burn and let's just move on to Llama 3 and then make that one better? Soumith [00:39:07]: Instead of answering specifically about that Llama 2 situation or whatever, I'll tell you how we think about things. Generally, we're, I mean, Mark really is some numbers, right? Swyx [00:39:20]: So let's cite those things again. All I remember is like 600K GPUs. Soumith [00:39:24]: That is by the end of this year and 600K H100 equivalents. With 250K H100s, including all of our other GPU or accelerator stuff, it would be 600-and-something-K aggregate capacity. Swyx [00:39:38]: That's a lot of GPUs. Soumith [00:39:39]: We'll talk about that separately. But the way we think about it is we have a train of models, right? Llama 1, 2, 3, 4. And we have a bunch of GPUs. I don't think we're short of GPUs. Like- Swyx [00:39:54]: Yeah, no, I wouldn't say so. Yeah, so it's all a matter of time. Soumith [00:39:56]: I think time is the biggest bottleneck. It's like, when do you stop training the previous one and when do you start training the next one? And how do you make those decisions? The data, do you have net new data, better clean data for the next one in a way that it's not worth really focusing on the previous one? It's just a standard iterative product. You're like, when is the iPhone 1? When do you start working on iPhone 2? Where is the iPhone? And so on, right? So mostly the considerations are time and generation, rather than GPUs, in my opinion. Alessio [00:40:31]: So one of the things with the scaling loss, like Chinchilla is optimal to balance training and inference costs. I think at Meta's scale, you would rather pay a lot more maybe at training and then save on inference. How do you think about that from infrastructure perspective? I think in your tweet, you say you can try and guess on like how we're using these GPUs. Can you just give people a bit of understanding? It's like, because I've already seen a lot of VCs say, Llama 3 has been trained on 600,000 GPUs and that's obviously not true, I'm sure. How do you allocate between the research, FAIR and the Llama training, the inference on Instagram suggestions that get me to scroll, like AI-generated stickers on WhatsApp and all of that? Soumith [00:41:11]: Yeah, we haven't talked about any of this publicly, but as a broad stroke, it's like how we would allocate resources of any other kinds at any company. You run a VC portfolio, how do you allocate your investments between different companies or whatever? You kind of make various trade-offs and you kind of decide, should I invest in this project or this other project, or how much should I invest in this project? It's very much a zero sum of trade-offs. And it also comes into play, how are your clusters configured, like overall, what you can fit of what size and what cluster and so on. So broadly, there's no magic sauce here. I mean, I think the details would add more spice, but also wouldn't add more understanding. It's just gonna be like, oh, okay, I mean, this looks like they just think about this as I would normally do. Alessio [00:42:05]: So even the GPU rich run through the same struggles of having to decide where to allocate things. Soumith [00:42:11]: Yeah, I mean, at some point I forgot who said it, but you kind of fit your models to the amount of compute you have. If you don't have enough compute, you figure out how to make do with smaller models. But no one as of today, I think would feel like they have enough compute. I don't think I've heard any company within the AI space be like, oh yeah, like we feel like we have sufficient compute and we couldn't have done better. So that conversation, I don't think I've heard from any of my friends at other companies. Eleuther Swyx [00:42:47]: Stella from Eleuther sometimes says that because she has a lot of donated compute. She's trying to put it to interesting uses, but for some reason she's decided to stop making large models. Soumith [00:42:57]: I mean, that's a cool, high conviction opinion that might pay out. Swyx [00:43:01]: Why? Soumith [00:43:02]: I mean, she's taking a path that most people don't care to take about in this climate and she probably will have very differentiated ideas. I mean, think about the correlation of ideas in AI right now. It's so bad, right? So everyone's fighting for the same pie. In some weird sense, that's partly why I don't really directly work on LLMs. I used to do image models and stuff and I actually stopped doing GANs because GANs were getting so hot that I didn't have any calibration of whether my work would be useful or not because, oh yeah, someone else did the same thing you did. It's like, there's so much to do, I don't understand why I need to fight for the same pie. So I think Stella's decision is very smart. Making Bets Alessio [00:43:53]: And how do you reconcile that with how we started the discussion about intrinsic versus extrinsic kind of like accomplishment or success? How should people think about that especially when they're doing a PhD or early in their career? I think in Europe, I walked through a lot of the posters and whatnot, there seems to be mode collapse in a way in the research, a lot of people working on the same things. Is it worth for a PhD to not take a bet on something that is maybe not as interesting just because of funding and visibility and whatnot? Or yeah, what suggestions would you give? Soumith [00:44:28]: I think there's a baseline level of compatibility you need to have with the field. Basically, you need to figure out if you will get paid enough to eat, right? Like whatever reasonable normal lifestyle you want to have as a baseline. So you at least have to pick a problem within the neighborhood of fundable. Like you wouldn't wanna be doing something so obscure that people are like, I don't know, like you can work on it. Swyx [00:44:59]: Would a limit on fundability, I'm just observing something like three months of compute, right? That's the top line, that's the like max that you can spend on any one project. Soumith [00:45:09]: But like, I think that's very ill specified, like how much compute, right? I think that the notion of fundability is broader. It's more like, hey, are these family of models within the acceptable set of, you're not crazy or something, right? Even something like neural or DS, which is a very boundary pushing thing or states-based models or whatever. Like all of these things I think are still in fundable territory. When you're talking about, I'm gonna do one of the neuromorphic models and then apply image classification to them or something, then it becomes a bit questionable. Again, it depends on your motivation. Maybe if you're a neuroscientist, it actually is feasible. But if you're an AI engineer, like the audience of these podcasts, then it's more questionable. The way I think about it is, you need to figure out how you can be in the baseline level of fundability just so that you can just live. And then after that, really focus on intrinsic motivation and depends on your strengths, like how you can play to your strengths and your interests at the same time. Like I try to look at a bunch of ideas that are interesting to me, but also try to play to my strengths. I'm not gonna go work on theoretical ML. I'm interested in it, but when I want to work on something like that, I try to partner with someone who is actually a good theoretical ML person and see if I actually have any value to provide. And if they think I do, then I come in. So I think you'd want to find that intersection of ideas you like, and that also play to your strengths. And I'd go from there. Everything else, like actually finding extrinsic success and all of that, I think is the way I think about it is like somewhat immaterial. When you're talking about building ecosystems and stuff, slightly different considerations come into play, but that's a different conversation. Swyx [00:47:06]: We're gonna pivot a little bit to just talking about open source AI. But one more thing I wanted to establish for Meta is this 600K number, just kind of rounding out the discussion, that's for all Meta. So including your own inference needs, right? It's not just about training. Soumith [00:47:19]: It's gonna be the number in our data centers for all of Meta, yeah. Swyx [00:47:23]: Yeah, so there's a decent amount of workload serving Facebook and Instagram and whatever. And then is there interest in like your own hardware? MTIA Soumith [00:47:31]: We already talked about our own hardware. It's called MTIA. Our own silicon, I think we've even showed the standard photograph of you holding the chip that doesn't work. Like as in the chip that you basically just get like- Swyx [00:47:51]: As a test, right? Soumith [00:47:52]: Yeah, a test chip or whatever. So we are working on our silicon and we'll probably talk more about it when the time is right, but- Swyx [00:48:00]: Like what gaps do you have that the market doesn't offer? Soumith [00:48:04]: Okay, I mean, this is easy to answer. So basically, remember how I told you about there's this memory hierarchy and like sweet spots and all of that? Fundamentally, when you build a hardware, you make it general enough that a wide set of customers and a wide set of workloads can use it effectively while trying to get the maximum level of performance they can. The more specialized you make the chip, the more hardware efficient it's going to be, the more power efficient it's gonna be, the more easier it's going to be to find the software, like the kernel's right to just map that one or two workloads to that hardware and so on. So it's pretty well understood across the industry that if you have a sufficiently large volume, enough workload, you can specialize it and get some efficiency gains, like power gains and so on. So the way you can think about everyone building, every large company building silicon, I think a bunch of the other large companies are building their own silicon as well, is they, each large company has a sufficient enough set of verticalized workloads that can be specialized that have a pattern to them that say a more generic accelerator like an NVIDIA or an AMD GPU does not exploit. So there is some level of power efficiency that you're leaving on the table by not exploiting that. And you have sufficient scale and you have sufficient forecasted stability that those workloads will exist in the same form, that it's worth spending the time to build out a chip to exploit that sweet spot. Like obviously something like this is only useful if you hit a certain scale and that your forecasted prediction of those kind of workloads being in the same kind of specializable exploitable way is true. So yeah, that's why we're building our own chips. Swyx [00:50:08]: Awesome. Open Source AI Alessio [00:50:09]: Yeah, I know we've been talking a lot on a lot of different topics and going back to open source, you had a very good tweet. You said that a single company's closed source effort rate limits against people's imaginations and needs. How do you think about all the impact that some of the Meta AI work in open source has been doing and maybe directions of the whole open source AI space? Soumith [00:50:32]: Yeah, in general, I think first, I think it's worth talking about this in terms of open and not just open source, because like with the whole notion of model weights, no one even knows what source means for these things. But just for the discussion, when I say open source, you can assume it's just I'm talking about open. And then there's the whole notion of licensing and all that, commercial, non-commercial, commercial with clauses and all that. I think at a fundamental level, the most benefited value of open source is that you make the distribution to be very wide. It's just available with no friction and people can do transformative things in a way that's very accessible. Maybe it's open source, but it has a commercial license and I'm a student in India. I don't care about the license. I just don't even understand the license. But like the fact that I can use it and do something with it is very transformative to me. Like I got this thing in a very accessible way. And then it's various degrees, right? And then if it's open source, but it's actually a commercial license, then a lot of companies are gonna benefit from gaining value that they didn't previously have, that they maybe had to pay a closed source company for it. So open source is just a very interesting tool that you can use in various ways. So there's, again, two kinds of open source. One is some large company doing a lot of work and then open sourcing it. And that kind of effort is not really feasible by say a band of volunteers doing it the same way. So there's both a capital and operational expenditure that the large company just decided to ignore and give it away to the world for some benefits of some kind. They're not as tangible as direct revenue. So in that part, Meta has been doing incredibly good things. They fund a huge amount of the PyTorch development. They've open sourced Llama and those family of models and several other fairly transformative projects. FICE is one, Segment Anything, Detectron, Detectron 2. Dense Pose. I mean, it's- Swyx [00:52:52]: Seamless. Yeah, seamless. Soumith [00:52:53]: Like it's just the list is so long that we're not gonna cover. So I think Meta comes into that category where we spend a lot of CapEx and OpEx and we have a high talent density of great AI people and we open our stuff. And the thesis for that, I remember when FAIR was started, the common thing was like, wait, why would Meta wanna start a open AI lab? Like what exactly is a benefit from a commercial perspective? And for then the thesis was very simple. It was AI is currently rate limiting Meta's ability to do things. Our ability to build various product integrations, moderation, various other factors. Like AI was the limiting factor and we just wanted AI to advance more and we didn't care if the IP of the AI was uniquely in our possession or not. However the field advances, that accelerates Meta's ability to build a better product. So we just built an open AI lab and we said, if this helps accelerate the progress of AI, that's strictly great for us. But very easy, rational, right? Still the same to a large extent with the Llama stuff. And it's the same values, but the argument, it's a bit more nuanced. And then there's a second kind of open source, which is, oh, we built this project, nights and weekends and we're very smart people and we open sourced it and then we built a community around it. This is the Linux kernel and various software projects like that. So I think about open source, like both of these things being beneficial and both of these things being different. They're different and beneficial in their own ways. The second one is really useful when there's an active arbitrage to be done. If someone's not really looking at a particular space because it's not commercially viable or whatever, like a band of volunteers can just coordinate online and do something and then make that happen. And that's great. Open Source LLMs I wanna cover a little bit about open source LLMs maybe. So open source LLMs have been very interesting because I think we were trending towards an increase in open source in AI from 2010 all the way to 2017 or something. Like where more and more pressure within the community was to open source their stuff so that their methods and stuff get adopted. And then the LLMs revolution kind of took the opposite effect OpenAI stopped open sourcing their stuff and DeepMind kind of didn't, like all the other cloud and all these other providers, they didn't open source their stuff. And it was not good in the sense that first science done in isolation probably will just form its own bubble where people believe their own b******t or whatever. So there's that problem. And then there was the other problem which was the accessibility part. Like, okay, I again always go back to I'm a student in India with no money. What is my accessibility to any of these closers models? At some scale I have to pay money. That makes it a non-starter and stuff. And there's also the control thing. I strongly believe if you want human aligned stuff, you want all humans to give feedback. And you want all humans to have access to that technology in the first place. And I actually have seen, living in New York, whenever I come to Silicon Valley, I see a different cultural bubble. Like all the friends I hang out with talk about some random thing like Dyson Spheres or whatever, that's a thing. And most of the world doesn't know or care about any of this stuff. It's definitely a bubble and bubbles can form very easily. And when you make a lot of decisions because you're in a bubble, they're probably not globally optimal decisions. So I think open source, the distribution of open source powers a certain kind of non-falsifiability that I think is very important. I think on the open source models, like it's going great in the fact that LoRa I think came out of the necessity of open source models needing to be fine-tunable in some way. Yeah, and I think DPO also came out of the academic open source side of things. So do any of the closed source labs, did any of them already have LoRa or DPO internally? Maybe, but that does not advance humanity in any way. It advances some companies probability of doing the winner takes all that I talked about earlier in the podcast. Open Source and Trust I don't know, it just feels fundamentally good. Like when people try to, you know, people are like, well, what are the ways in which it is not okay? I find most of these arguments, and this might be a little controversial, but I find a lot of arguments based on whether closed source models are safer or open source models are safer very much related to what kind of culture they grew up in, what kind of society they grew up in. If they grew up in a society that they trusted, then I think they take the closed source argument. And if they grew up in a society that they couldn't trust, where the norm was that you didn't trust your government, obviously it's corrupt or whatever, then I think the open source argument is what they take. I think there's a deep connection to like people's innate biases from their childhood and their trust in society and governmental aspects that push them towards one opinion or the other. And I'm definitely in the camp of open source is definitely going to actually have better outcomes for society. Closed source to me just means that centralization of power, which, you know, is really hard to trust. So I think it's going well in so many ways that we're actively disaggregating the centralization of power to just two or three providers. We are, I think, benefiting from so many people using these models in so many ways that aren't allowed by, say, Silicon Valley left-wing tropes. Like some of these things are good or bad, but they're not culturally accepted universally in the world. So those are things worth thinking about. And I think open source is not winning in certain ways. Like these are all the things in which like, as I mentioned, it's actually being very good and beneficial and winning. Feedback to solve the Open Source Coordination problem I think one of the ways in which it's not winning, at some point I should write a long-form post about this, is I think it has a classic coordination problem. I mean, open source in general always has a coordination problem. If there's a vertically integrated provider with more resources, they will just be better coordinated than open source. And so now open source has to figure out how to have coordinated benefits. And the reason you want coordinated benefits is because these models are getting better based on human feedback. And if you see with open source models, if you go to Reddit, local llama, subreddit, like there's so many variations of models that are being produced from, say, nose research. I mean, there's so many variations built by so many people. And one common theme is they're all using these fine-tuning or human preferences datasets that are very limited. And like someone published them somewhere and they're not sufficiently diverse. And you look at the other side, say front-ends like Uba or Hugging Chat or Ollama, they don't really have like feedback buttons. Like all the people using all these front-ends, they probably want to give feedback, but there's no way for them to give feedback. So these models are being built, they're being arbitrarily measured, and then they are being deployed into all these open source front-ends or like apps that are closed source, they're serving open source models. And these front-ends don't have, they are not exposing the ability to give feedback. So we're just losing all of this feedback. Maybe open source models are being as used as GPT is at this point in like all kinds of, in a very fragmented way, in aggregate all the open source models together are probably being used as much as GPT is, maybe close to that. But the amount of feedback that is driving back into the open source ecosystem is negligible, maybe less than 1% of the usage. So I think the blueprint here I think is you'd want someone to create a sinkhole for the feedback, some centralized sinkhole, maybe Hugging Face or someone just funds like, okay, I will make available a call to log a string along with a bit of information of positive or negative or something that. And then you would want to send pull requests to all the open source front-ends like Ooba and all being like, hey, we're just integrating a feedback UI and then work with the closed source people as also being like, look, it doesn't cost you anything, just have a button. And then the sinkhole will have a bunch of this data coming in. And then I think a bunch of open source researchers should figure out how to filter their feedback into only the high quality one. I'm sure it will be exploited by spam bots or whatever, right? Like, this is the perfect way to inject your advertising product into the next. So there needs to be some level of that, that in the same way, I'm sure all the closed providers are doing today, like OpenAI, Claude, the feedback that comes in, I'm sure they are figuring out if that's legit or not. That kind of data filtering needs to be done. And that loop has to be set up. And this requires that central sinkhole and that data cleaning effort both to be there. They're not there right now. They're not there right now, I think for capital reasons, but also for coordination reasons. Okay, if that central sinkhole is there, who's gonna go coordinate all of this integration across all of these open source front ends. But I think if we do that, if that actually happens, I think that probably has a real chance of the open source models having a runaway effect against OpenAI with their current daily active users. Probably doesn't have a chance against Google because you know, Google has Android and Chrome and Gmail and Google Docs and everything, you know? So people just use that a lot. But like, I think there's a clear chance we can take at truly winning open source. AGI Alessio [01:04:00]: Do you think this feedback is helpful to make open source models better or to get to like open source AGI? Because in a way like OpenAI's goal is to get to AGI, right? So versus I think in open source, we're more focused on personal better usage or like commercial better usage. Soumith [01:04:17]: Yeah, I think that's a good question. But I think, I actually don't think people have a good understanding of AGI. And I don't mean definition level. I mean, people are like, okay, we're gonna, AGI means it's powering 40% of world economic output or something like that, right? But what does that mean? So do you think electricity is powering 40% of world economic output or is it not? Like generally the notion of powering X percent of economic output is not defined well at all for me to understand how to know when we got to AGI or how to measure whether we're getting AGI. Like, you know, you can look at it in terms of intelligence or task automation or whatever. I think that's what we are doing right now. We're basically integrating like the current set of AI technologies into so many real world use cases where we find value that if some new version of AI comes in, we can find, we can be like, ah, this helps me more. In that sense, I think the whole process of how we think we got to AGI will be continuous and not discontinuous like how I think the question is posed. So I think the open source thing will be very much in line with getting to AGI because open source has that natural selection effect. Like if a better open source model comes, really no one says, ha, I don't want to use it because there are ecosystem effect, I'm logged into my ecosystem or, I don't know if I like the models, you know, whatever. It's just a very pure direct thing. So if there's a better model that comes out, then it will be used. So I definitely think it has a good chance of achieving how I would think about it as a continuous path to what we might define as AGI. OpenAssistant vs LMSys vs OpenRouter Swyx [01:06:18]: For the listeners, I would actually mention a couple other maybe related notes on just this very interesting concept of feedback sinkhole for open source to really catch up in terms of the overall Google versus OpenAI debate. Open Assistant was led by Yannick Kilcher who recently ended his effort. I think the criticism there was like the kind of people that go to a specific website to give feedback is not representative of real world usage. And that's why the models trained on Open Assistant didn't really seem like they have caught on in the open source world. The two leading candidates in my mind are LMSYS out of UC Berkeley who have the LMSYS arena, which is being touted as one of the only ways, only reliable benchmarks anymore. I kind of call them non-parametric benchmarks because there's nothing to cheat on it except for ELO. And then the other one is OpenRouter, which is Alex Atala's thing. I don't know if you've talked to any of these people. Soumith [01:07:11]: I obviously know all of the efforts that you talked about. I haven't talked to them directly about this yet. But the way I think about it is the way these models are going to be used is always going to be way more distributed than centralized. Like, which is the power of the open source movement. Like the UI within which these models are going to be used is going to be decentralized. These models are going to be integrated into hundreds and thousands of projects and products and all of that. And I think that is important to recognize. Like the LMSYS leaderboard is the best thing we have right now to understand whether a model is better or not versus another model. But it's also biased in only having a sliver of view into how people actually use these models. Like the people who actually end up coming to the LMSYS leaderboard and then using a model only use it for certain things. Like GitHub Copilot style usage is not captured in say LMSYS things. And so many other styles, like the character AI style things is not captured in LMSYS. Swyx [01:08:19]: Which OpenRouter could do. They don't do it right now, but. Soumith [01:08:22]: Yeah, so my point is like the way these models are going to be used is going to be always a large surface area. And I think we need to figure out how to provide the infrastructure to integrate with all these like ways in which it's being used. Even if you get the top hundred front ends that the model, like open source models are used through to subscribe to the sinkhole. I think that's already a substantial thing. I think thinking one or two things will by themselves get a lot of data I think is not going to happen. Swyx [01:08:58]: Yeah, fair enough. Other Modalities Alessio [01:08:59]: Before we let you go, can we do just a quick beyond text segment? So you're an investor in Runway, which is a beta generation. You're an investor in One X, which is a humanoid assistant. Osmo, which is focused on using AI for smell recognition and synthesis. You advise a bunch of robotics projects at NYU. Swyx [01:09:19]: Maybe. And he builds his own home robot. Yeah, exactly. Alessio [01:09:22]: On a more, yeah, maybe open editing. What are the things that you're most excited about beyond text generation and kind of the more mundane usage? Soumith [01:09:30]: Yeah, I mean, in general, I have more things I'm generally excited about than I can possibly do. Investing is one way to try to clear those urges. I'm generally excited about robotics being a possibility, home robotics being five to seven years away into commercialization. I think it's not next year or two years from now, but five to seven years from now, I think a lot more robotics companies might pop out. There's not a good consensus on whether hardware is a bottleneck or AI is a bottleneck in robotics right now. My view is actually hardware is still the bottleneck and AI is also a little bit of bottleneck, but I don't think there's any obvious breakthroughs we need. I think it's just work. So I'm generally excited about robotics. I spend a lot of personal time. I spend every Wednesday afternoon at NYU working with Lerrel Pinto and team and just getting towards my home robot that just does my dishes and stuff. Swyx [01:10:38]: What's the status of it? Like what does it do for you now? Soumith [01:10:41]: As of today, we just deployed a couple of months ago, we deployed our home robotics stuff into several tens of New York City homes and tried to make it do a bunch of tasks. And we're basically starting to build out a framework that gets to a certain level of robustness on fairly simple tasks, like picking this cup and putting it somewhere else or taking a few pieces of cloth on the ground and put it somewhere else or open your microwave and various baseline tasks that with low sample complexity. So I think one of the things people don't spend a lot of time in robotics is the user experience, which I think in the research I do at NYU, we spend a huge amount of time on. I think the key there is sample complexity has to be really low. A lot of the current robotics research, if you see they're like, oh yeah, we collected 50 demos and now it's able to do this task or we collected 300 demos or the number of samples you need for this thing to do the task is really high. So we're focusing a lot on, you show it two or three times and that's sufficient for it to actually do the task, but it comes with less generalization, right? Like there's some initial conditions that have to be true for it to do the task. So we're making progress. That's very interesting in general, the space. I don't think people in this space have settled on the hardware, like how the hardware looks like for it to be truly useful in the home or whatever, or the UX or the like AI, ML stuff needed to make it sample efficient and all of that. But I think lots of work is happening in the field. Alessio [01:12:28]: Yeah, one of my friends, Carlo at Berkeley, he worked on a project called M3L, which is two CNNs, one for tactile feedback and one for image. When you say hardware, is it running all these things on the edge or is it just like the actual servos and the- Soumith [01:12:45]: By hardware, I mean the actual servos, like the motors, servos, even the sensors. I think we have incredible vision that's still it's so much better compared to in the field of view and in resolution compared to any of the cameras we can buy. We have, our skin is all available touch sensing and we have some of the most efficient, some of the most high capacity motors that can lift large loads in the dexterity of a hand and stuff. So in terms of hardware, I mean in terms of those capabilities, we haven't figured out how to do a lot of this stuff. I mean, Tesla has been making incredible progress. One X, I think announced their new thing that looks incredible. Some of the other companies figure and others are doing great work. But we're really not anywhere close to the hardware that we feel like we need. And there's obviously the other thing I want to call out is a lot of what people show works, but has to be fixed all the time. And like, that's the other thing we are incredible at. Like we don't need any maintenance or the maintenance is part of us. If you buy a product, electronics product of any kind, you buy a PS5, you don't say, oh yeah, my PS5 breaks every six days and I have to do some reasonable amount of work on it. But that's robotics. Like if it's not industrial robotics where it's very controlled and specialized or whatever, you're talking about reliability in those ranges. So I think people don't talk about the reliability thing enough. Like what I mean, we're going to enter the commercialization phase. I mean, we're going to start thinking about, okay, now we have this thing and we need to figure out how to get reliability high enough to deploy it into homes and just sell it to people and Best Buy or something. So that's the other factor that we have to make a lot of progress on. Swyx [01:14:44]: I just realized that Google has a play in this with Palm E and stuff and OpenAI obviously has a long history of doing this stuff. Is there anything at Meta? No robotics stuff in Meta? Soumith [01:14:55]: We have a small robotics program at Meta out of FAIR. I actually used to do it at FAIR a little bit before I moved into Infra and focused on my Meta time on a lot of other infrastructural stuff. So yeah, Meta's robotics program is a lot smaller. Swyx [01:15:10]: Seems like it would be a personal computing. Soumith [01:15:14]: You could think of it as like, Meta has a ridiculously large device strategy, right? Like, you know, this is how our reality labs stuff. You know, we're going at it from VR and AR and, you know, we showcase a lot of that stuff. I think for Meta, the robot is not as important as like the physical device. Physical devices kind of stuff. Osmo - smell AI Swyx [01:15:37]: Yeah, for sure. Yeah. Okay, I want to touch on Osmo a bit because very unusual company to the stuff that we normally discuss, not robotics, sense of smell. The original pitch I heard from the founder, maybe you can correct me, is that he realized that you can smell cancer. Yeah. Is that intuitive? Is that what you get? Or is that the potential that you see? Soumith [01:15:56]: The very interesting reason I invested in Osmo is because Alex Wiltschko, the founder of Osmo, before PyTorch, there was Torch. And Alex Wiltschko actually worked on Torch. He's actually a frameworks guy. Like, you know, he built this thing called Tangent from Google, another autodiff framework and stuff. I know him from that side of things. And then, he is a neurobiologist by training. He just happens to also love, neural networks and hacking on those frameworks. So incredibly smart guy, one of the smartest people I know. So when he was going in this direction, I thought it was incredible that smell is something that we haven't even started to scrape in terms of digitization. When we think about audio or images or video, they're so advanced. So we have the concept of color spaces. We have the concept of frequency spectrums. Like, you know, we figured out how ears process, like, frequencies in mouse spectrum or whatever logarithmically scaled. Images for RGB, YUV. We have so many different kinds of parameterizations. We have formalized these two senses ridiculously well. Touch and smell, nada. We're where we were with images in, say, in 1920 or maybe even the 1800s, right? That's where we're at. And Alex has this incredible vision of, like, having a smell sensor just eventually just be part of your daily life. Like, as of today, you don't really think about when you're watching an Instagram reel or something, huh, I also would love to know what it smelled like, you know, when you're watching a reel of a food or something. You don't, because we really haven't, as a society, got that muscle to even understand what a smell sensor can do. I think the more near-term effects are obviously going to be around things that provide more obvious utility in the short term, like maybe smelling cancer or repelling mosquitoes better, or, you know, stuff like that. Swyx [01:18:12]: More recently, he's been talking about categorizing perfumes, obviously. Yeah, exactly. That's a market that you can pursue. Soumith [01:18:17]: Yeah, like, I mean, think about how you can customize a perfume to your own liking in the same way you can customize a shoe or something, right? I think all the near-term stuff, I think if he's able to figure out a near-term value for it, they, as a company, can sustain themselves to then eventually try to make progress on the long term, which is really in uncharted territory. Like, think about it, 50 years from now, it would be pretty obvious to kids of the generation to just, like, I was going to say scroll a reel on their phone, and maybe phones wouldn't be there. Swyx [01:18:58]: They're just on their glasses, they're watching something. Soumith [01:18:58]: Yeah, I think VR would be. And then, like, they immediately get a smell sense of that remote experience as well. We haven't really progressed enough in that dimension, and I think they have a chance to do it. Alessio [01:19:13]: Awesome, I mean, we touched on a lot of things. Anything, we're missing anything you want to direct people to, or? Swyx [01:19:19]: Yeah, call to action. Yeah. Call for research, call for startups. Soumith [01:19:22]: I don't really have a lot of calls to action, because usually I think people should be intrinsically, like, figuring it out. Swyx [01:19:29]: That's a good look inside yourself. Yeah. That's good. Alessio [01:19:33]: Awesome, thank you so much for coming on. Swyx [01:19:35]: Yeah, for sure. This was great. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Alessio Fanelli?

  Hey everyone, welcome to the Late in Space Podcast. This is Alessio, partner and CTO in residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Small AI.

• 0:15Speaker 01

  Hey, and today we have in the studio Subhiv Chintala, welcome.

• 0:17Speaker 02

  Thanks for having me.

• 0:18Speaker 01

  On one of your rare visits from New York, where you live. You got your start in computer vision at NYU with Yan Le Cun. That was a very fortuitous start. I was actually listening to your interview on the Gradient Podcast. So if people want to know more about the history of Subhiv, history of PyTorch, they can go to that podcast. We won't spend that much time there. But I just was marveling at your luck. Or I don't know if it's your luck or your drive to find AI early and then find the right quality mentor. Because I guess Yan really introduced you to that world.

• 0:51Speaker 02

  Yeah, I think you're talking about extrinsic success, right? Like, a lot of people just have drive to do things that they think is fun. And a lot of those things might or might not be extrinsically perceived as good and successful. I think I just happen to like something that is now like one of the coolest things in the world or whatever. But if I happen, you know, the first thing I tried to become was a 3D VFX artist. And I was really interested in doing that, but I turned out to be very bad at it. So I ended up not doing that further. But even if I was good at that, whatever, and I ended up going down that path, I probably would have been equally happy. It's just like, maybe like the perception of,

• 1:34Speaker 02

  oh, is this person successful or not, might be different. I think like after a baseline, like your happiness is probably more correlated with your intrinsic stuff.

• 1:44Speaker 01

  Yes, I think Dan Pink has this book on drive that I often refer to about the power of intrinsic motivation versus extrinsic, and how long extrinsic lasts. It's not very long at all. But anyway, now you are an investor in Runway. So in a way, you're working on VFX.

• 2:00Speaker 02

  Yes, I mean, in a very convoluted way.

• 2:03Speaker 01

  It reminds me of Ed Catmull. I don't know if you guys know, but you know, he actually tried to become an animator in his early years and failed, or didn't get accepted by Disney. And then went and created Pixar and then got bought by Disney. Create a Toy Story.

• 2:17Speaker 02

  Yeah.

• 2:18Speaker 01

  So you joined Facebook in 2014 and eventually became creator maintainer of PyTorch. And there's a long story there you can refer to on the gradient. I think maybe people don't know that you also involved in more sort of hardware and cluster decision affair. And we can dive into more details there because we're all about hardware this month. Yeah. And then finally, I don't know what else, like what else should people know about you on the personal side or professional side?

• 2:40Speaker 02

  I think open source is definitely like a big passion of mine and probably forms a little bit of my identity at this point. I'm irrationally interested in open source. I think open source has that fundamental way to distribute opportunity in a way that is very powerful. Like I grew up in India. I didn't have internet for a while. And in college, actually, I didn't have internet except for like GPRS or whatever. Like and like knowledge was, knowledge was very centralized, but like I saw that evolution of knowledge slowly getting decentralized. And that ended up helping me learn quicker and faster for like $0. And I think that was a strong reason why I ended up where I am.

• 3:25Speaker 02

  So like that, like the open source set of things, I always push regardless of like what I get paid for. Like I think I would do that as a passion project on the side.

• 3:35Speaker 01

  Yeah, that's wonderful. We'll talk about the challenges as well that open source has, open models versus closed models. But maybe you want to talk, touch a little bit on PyTorch before we move on to the sort of meta AI in general.

• 3:45Alessio Fanelli?

  Yeah, we kind of touched on PyTorch in a lot of episodes. So we had George Hots from Tiny Grant. He called PyTorch a Sisk and Tiny Grant a Risk. I would love to get your thoughts on PyTorch design direction as far as, I know you talk a lot about kind of having a happy path to start with. And then making complexity hidden away, but then available to the end user. One of the things that George mentioned is I think you have like 250 primitive operators in PyTorch. I think Tiny Grant is four. So how do you think about some of the learnings that maybe he's gonna run into that you already had in the past seven, eight years almost of running PyTorch?

• 4:24Speaker 02

  Yeah, I think there's different models here, but like I think it's two different models that people generally start with. Either they go like, I have a grand vision and I'm gonna build like a giant system that achieves this grand vision. And maybe one is or like, you know, super feature complete, whatever. Or other people say they will get incrementally ambitious, right? And they say, oh, we'll start with something simple and then we'll slowly layer out complexity in a way that optimally applies Huffman coding or whatever. Like, you know, where the density of users are and what they're using, I would want to keep it in the like easy, happy path. And where the more niche, advanced use cases, I'll still want people to try them, but they need to take additional frictional steps.

• 5:07Speaker 02

  George, I think, just like we started with PyTorch, George started with like the incrementally ambitious thing. I remember TinyGrad used to be like we would be limited to a thousand lines of code. And I think now it's like a 5,000. So I think there is no real like magic to which why PyTorch has a kind of complexity. I think it's like probably partly necessitated and partly because we built with the technology available under us at that time. PyTorch is like 190,000 lines of code or something at this point. I think if you had to rewrite it, we would probably think about ways to rewrite it in like a vastly simplified way for sure.

• 5:48Speaker 02

  But a lot of that complexity comes from the fact that in a very simple, explainable way, you have memory hierarchies. You have CPU has like three levels of caches and then you have DRAM and SSD. And then you have network. Similarly, GPU has several levels of memory. And then you have like different levels of network hierarchies. NVLink plus like InfiniBand or Rocky or something like that, right? And the way the flops are available on your hardware, they are available in a certain way and your computation is in a certain way. And you have to retrofit your computation onto both the memory hierarchy and like the flops available.

• 6:30Speaker 02

  When you're doing this, it is actually like a fairly hard mathematical problem to do this setup, like you find the optimal thing. And finding the optimal thing is like, what is optimal depends on like the input variables themselves. So like, okay, like what is the shape of your input tensors? And like, what is the operation you're trying to do? And like various things like that. Finding that optimal configuration and writing it down in code is not the same for every input configuration you have. Like, for example, just as the shape of the tensors change, let's say you have three input tensors into like a sparse dot product or something like that. The shape of each of these input tensors will vastly change how you do this optimally placing

• 7:14Speaker 02

  this operation onto the hardware in a way that will get you maximal throughput. So a lot of our complexity comes from writing out like hundreds of configurations for each single PyTorch operator and templatizing these things and like symbolically like generating the the final CUDA code or like CPU code. There's no way to avoid it because mathematically we haven't found symbolic ways to do this that also keep compile time near zero. You can write a very simple framework, but then you also should be willing to eat the long compile times of like searching for that optimal performance at runtime. But that's the trade off.

• 7:53Speaker 02

  There's no like, I don't think like unless we have like great breakthroughs like George's vision is achievable. Like, or like he should be thinking about a narrower problem such as I'm only gonna make this for like work for self-driving car convnets. Or like I'm only gonna make this work for like LLM transformers of the llama style. Like if you start narrowing the problem down, you can make a vastly simpler framework. But if you don't, if you need the generality to power all of the AI research that is happening and keep like zero compile time and in all these other factors. I think it's not easy to avoid the complexity.

• 8:32Alessio Fanelli?

  That's interesting. And we kind of touched on this with Chris Lerner when he was on the podcast. If you think about frameworks, they have the model target, they have the hardware target, they have different things to think about. He mentioned when he was at Google, TensorFlow trying to be optimized to make TPUs go burr, you know, and go as fast. I think George is trying to make, especially AMD stack, be better than Rockem. How come PyTorch has been such a Switzerland versus just making meta hardware go burr?

• 9:01Speaker 02

  First, Meta is not in the business of selling hardware. Meta is not in the business of cloud compute. The way Meta thinks about funding PyTorch is we're funding it because it's not good for Meta to fund PyTorch. Because PyTorch has become a standard and a big open source project. And generally, it gives us a timeline edge, it gives us like, whereas like leverage, like, and all that within our own work. So why is PyTorch like more of a Switzerland rather than being a Pidginated? Like, I think the way we think about it is not in terms of Switzerland or not. We actually, the way like we articulated to all hardware vendors and software vendors and all who come to us being like, we want to build a backend in core for PyTorch and ship it by default is like,

• 9:45Speaker 02

  we just only look at our user side of things. Like if users are using a particular piece of hardware, then we want to support it. We very much don't want to king make the hardware side of things. So as like the MacBooks have GPUs and as that stuff started getting increasingly interesting, we pushed Apple to push some engineers and work on the MPS support. And we spent significant time from like Meta-funded engineers on that as well. Because a lot of people are using the Apple GPUs and there's demand. So like, we kind of mostly look at it from the demand side. We never look at it from like, oh, which hardware should we start taking opinions on?

• 10:27Speaker 01

  Is there a future in which, because Mojo or modulus Mojo is kind of a superset of Python, is there a future in which PyTorch might use Mojo features optionally?

• 10:36Speaker 02

  I think it depends on how well-integrated it is into the Python ecosystem. So if Mojo is like a pip install and it's readily available and users feel like they can use Mojo so smoothly within their workflows in a way that just is slow friction, we would definitely look into that. Like in the same way, like PyTorch now depends on Triton, like OpenAI Triton. And we never had a conversation that was like, huh, that's like a dependency. Should we just build the Triton of our own? Or should we like use Triton? Like it almost doesn't, like those conversations don't really come up. For us, it's the conversations are more like, well, does Triton have like 10,000 dependencies?

• 11:20Speaker 02

  And is it hard to install? Like we almost don't look at these things from like a strategic leverage point of view. We look at these things from like a user experience point of view. Like, is it easy to install? Is it like smoothly integrated? And does it give enough benefits for us to like start depending on it? So yeah, we should consider. That's how we think about it.

• 11:37Speaker 01

  You're inclusive by default, as long as it meets like the the minimum bar of, yeah. But like, maybe I phrased it wrongly. Maybe it's more like, okay, like what problems would you look to solve?

• 11:46Speaker 02

  Okay.

• 11:46Speaker 01

  That you have right now?

• 11:48Speaker 02

  I think it depends on what problems Mojo will be useful at.

• 11:51Speaker 01

  Mainly the performance pitch, some amount of cross-compiling pitch.

• 11:55Speaker 02

  Yeah, I think like the performance pitch for Mojo was like, we're going to be performant even if you have like a lot of custom stuff. Like you can write arbitrary custom things and like we will be performant. And that value proposition is not clear to us from the PyTorch side to consider it for PyTorch. So PyTorch, it's actually not 250 operators, like 1,000 operators. PyTorch exposes about 1,000 operators and people kind of write their ideas in the 1,000 operators of PyTorch. Mojo is like, well, maybe like it's okay to completely sidestep those like 1,000 operators of PyTorch and just write it in a more natural form. Just write like raw Python, like write for loops or whatever, right?

• 12:38Speaker 02

  So from the consideration of how do we intersect PyTorch with Mojo? Like I can see one use case where you're like, You have custom stuff for some parts of your program, but mostly it's PyTorch. And so we can probably figure out how to like make it easier for say torch compile to like smoothly also consume Mojo subgraphs. And like, you know, the interoperability being actually usable. That I think is valuable, but like Mojo as a fundamental front end would be replacing PyTorch, not like augmenting PyTorch. So in that sense, I don't see a synergy in more deeply like integrating Mojo.

• 13:21Speaker 01

  So call out to Mojo whenever they have written something in Mojo and there's some performance related thing going on. And then since you mentioned Apple, what should people think of PyTorch versus MLX?

• 13:32Speaker 02

  I mean, MLX is early and I know the folks well. Ani used to work at Fair and I chatted, I used to chat with him all the time. He used to be based out of New York as well. The way I think about MLX is that MLX is specialized for Apple right now. It has a happy path because it's defined its product in a narrow way. At some point, MLX either says we will only be supporting Apple and we will just focus on enabling, you know, this is a framework if you use your MacBook, but once you like go server side or whatever, that's not my problem and I don't care.

• 14:13Speaker 02

  Or MLX, it enters like the server side side of things as well. Like it, one of these two things will happen, right? If the first thing will happen, like MLX's overall addressable market will be small, but it'd probably do well within that addressable market. If it enters the second phase, they're going to run into all the same complexities that we have to deal with. They will not have any magic one and they will have.

• 14:40Speaker 02

  More complex work to do, they probably wouldn't be able to move as fast.

• 14:45Speaker 01

  Like having to deal with distributed compute?

• 14:47Speaker 02

  Distributed Nvidia and AMD GPUs, like just like having a genericization of the concept of a backend, how they treat compilation with plus overheads. Right now, they deeply assume like the whole MPS graph thing. So they need to think about all these additional things. If they end up expanding onto the server side, then they'll probably build something like PyTorch as well, right? Like eventually, tell where it will land. And I think there, they will kind of fail on the like lack of differentiation. Like it wouldn't be obvious to people why they would want to use it.

• 15:24Speaker 01

  I mean, there are some cloud companies offering M1 and M2 chips on servers. I feel like it might be interesting for Apple to pursue that market, but it's not their core.

• 15:33Speaker 02

  Yeah, I mean, if Apple can figure out their interconnect story, maybe like then it can become a thing.

• 15:39Speaker 01

  Honestly, that's more interesting than the cars.

• 15:41Speaker 02

  Yes. I think like, I mean, the mode that Nvidia has right now, I feel like is that they have the interconnect that no one else has. Like AMD GPUs are pretty good. I'm sure there is very silicon that is not bad at all. But like the interconnect, like NVLink is uniquely awesome. I'm sure the other hardware providers are working on it, but.

• 16:03Speaker 01

  I feel like when you say it's uniquely awesome, you have some appreciation of it that the rest of us don't. I mean, the rest of us just like, you know, we hear marketing lines, but what do you mean when you say Nvidia is very good at networking? Obviously, they made the acquisition maybe like 15 years ago.

• 16:16Speaker 02

  Just like the bandwidth it offers and the latency it offers. I mean, like TPUs also have a good interconnect, but you can't buy them. So you have to go to Google to use it.

• 16:27Alessio Fanelli?

  Who are some of the other fair PyTorch alumni that are building cool companies? I know you have Fireworks AI, Lightning AI, Leptin, and Yankeng you knew since college when you was building Coffee.

• 16:40Speaker 02

  Yeah, so Yanchi and I used to be framework rivals, like K8, Torch. I mean, we were all a very small, closed net community back then. Cafe Torch, Tiano, Chainer, Karas, various frameworks. I mean, it used to be like more like 20 frameworks. I can't remember all the names. CCB by Liu Liu, who is also based out of SF. And I would actually like, you know, one of the ways it was interesting is like, you went into the framework guts and saw if someone wrote their own convolution kernel or they like were just copying someone else's. There were like four or five convolution kernels that were like unique and interesting.

• 17:25Speaker 02

  There is one from this guy out of Russia. I forgot the name. But like I remembered who was awesome enough to have like written their own kernel. And at some point there like I built out these these benchmarks called ConvNet benchmarks. They're just benchmarking all the convolution kernels that are available at that time. And it hilariously became big enough that at that time AI was getting like important, but not important enough that industrial strength players came in to do these kind of benchmarking and standardization like we have MLPerf today.

• 18:04Speaker 02

  So a lot of the startups were using ConvNet benchmarks in their pitch decks as like, oh, you know, on ConvNet benchmarks, like this is how we fare. So you should fund us. I remember Nirvana actually was at the top of the pack because Scott Gray wrote like amazingly fast convolution kernels at that time or interesting but separate times. But to answer your question, Alessio, I think mainly Lepton, Fireworks are the two most obvious ones, but I'm sure the fingerprints are a lot wider. They're just.

• 18:42Speaker 02

  People who worked within the PyTorch cafe to cohort of things and now end up at various other places.

• 18:50Speaker 01

  I think as a, both as an investor and a people looking to build on top of their services. It's a uncomfortable slash like, I don't know what I don't know pitch. Because I've met Yan Ting and I've met.

• 19:04Speaker 02

  Lin Chao.

• 19:04Speaker 01

  Yeah, I've, you know, I've met, I've met these folks and they're like, you know, we, we are deep in the PyTorch ecosystem and we serve billions of inferences a day or whatever at, at, at Facebook and now we can do it for you. And I'm like, okay, that's, that's like, great. Like, what should I be wary of or cautious of when, when, when these things happen? Because I'm like, obviously this, this experience is extremely powerful and valuable. I just don't know what I don't know. Like, what, what should people know about like these sort of new inference as a service companies?

• 19:32Speaker 02

  I think at that point, you would be investing in them for their expertise of one kind. So if they've been at a large company, but they've been doing amazing work, you would be thinking about it as like, okay, like what these people bring to the table is that they're really good at like GPU programming or understanding the complexity of serving models once it hits a certain scale, like, you know, various expertise like from the infra and like AI and GPUs point of view. What you would obviously want to figure out is like whether their understanding of the external market is clear,

• 20:12Speaker 02

  whether they know and understand how to think about running a business, like understanding how to be disciplined about making money or, you know, various things like that.

• 20:23Speaker 01

  Maybe I'll, I'll put it like, it's, I actually, I will de-emphasize the investing bit and just more as a potential customer.

• 20:29Speaker 02

  Oh, okay.

• 20:29Speaker 01

  Like, it's more like, okay, like, you know, you're PyTorch gods, of course. Like, what, what, what should I, what else should I know?

• 20:36Speaker 02

  I mean, I would not care about who's building something if I'm trying to be a customer. I would care about whether.

• 20:44Speaker 01

  Benchmark.

• 20:44Speaker 02

  Yeah, I'm, I, I, I use it and it's like usability and reliability and speed, right?

• 20:51Speaker 01

  Quality as well.

• 20:51Speaker 02

  Yeah, I, if someone from some random unknown place came to me and say, use our stuff, it's great, like, and I have the bandwidth, I probably will give it a shot. And if it turns out to be great, like, I'll just use it.

• 21:07Speaker 01

  Okay, great. And then maybe one more thing about benchmarks, since we already brought it up. And you brought up Confine benchmarks. There was recent, some recent drama around AnyScale. The AnyScale released their own benchmarks, and obviously they look great on their own benchmarks, but maybe didn't give the other. I feel like there are two lines of criticism. One, which is they didn't test sort of apples for apples on the kind of endpoints that the other providers, that they are competitors with, you know, on their benchmarks, and that is due diligence baseline. And then the second would be more just like optimizing for the right thing. You had some commentary on it, I'll just kind of let you riff.

• 21:41Speaker 02

  Yeah, I mean, in summary, basically my criticism of that was AnyScale built these benchmarks for end users to just understand what they should pick, right? And that's a very good thing to do. I think what they didn't do a good job of is give that end user a full understanding of what they should pick. Like they just gave them like a very narrow slice of understanding. I think they just gave them latency numbers, and that's not sufficient, right? Like you need to understand your total cost of ownership at some reasonable scale. Not like, oh, like one API call is like one cent, but like a thousand API calls are like 10 cents.

• 22:25Speaker 02

  Or like, you know, like people can misprice to cheat on those benchmarks. So you want to understand, okay, like how much is it gonna cost me if I actually subscribe to you and do like a million API calls a month or something? And then you want to understand the latency and reliability, not just from like one call you made, but like an aggregate of calls you made over several various times of the day and times of the week. And the nature of the workflows, like it's like, is it just like some generic single paragraph that you're sending that is cacheable? Or like, is it like testing of real work, right, real world workload? I think that kind of rigor, like in presenting that benchmark wasn't there.

• 23:08Speaker 02

  It was a much more narrow sliver of what should have been a good benchmark. That was my main criticism. And I'm pretty sure if before they released it, they showed it to their like other stakeholders who would be caring about this benchmark because they are present in it. They would have easily just pointed out these gaps. And I think they didn't do that and they just like released it. So I think those were the two main criticisms. I think they were fair and Robert took it well.

• 23:40Speaker 01

  He took it very well. Yeah, I will have him on at some point and we'll discuss it. But I think it's important for, I think the market being maturing enough that people start caring and competing on these kinds of things.

• 23:49Speaker 02

  Yeah.

• 23:50Speaker 01

  Means that we need to establish what best practice is because otherwise everyone's going to play dirty.

• 23:54Speaker 02

  Yeah, absolutely. And my view of the LLM inference market in general is that it's like the laundromat model. Like the margins are going to drive down towards the bare minimum. Like it's going to be all kinds of arbitrage between how much you can get the hardware for and then how much you sell the API. And how much latency your customers are willing to let go. Like you need to figure out how to squeeze your margins. Like what is your unique thing here? Like I think like Together and Fireworks and all these people are trying to build some faster CUDA kernels and faster like, you know, hardware kernels in general. But those modes only last for a month or two. Like these ideas quickly propagate.

• 24:38Speaker 01

  Even if they're not published?

• 24:39Speaker 02

  Even if they're not published, like the idea space is small. So even if they're not published, the discovery rate is going to be pretty high. It's not like we're talking about a combinatorial thing that is really large. You're talking about like Llama style LLM models and we're going to beat those to death. Like on like a few different hardware SKUs, right? Like it's not even like we have a huge diversity of hardware you're going to aim to run it on. Now, when you have such a narrow problem and you have a lot of people working on it, like the rate at which these ideas are going to get figured out is going to be pretty rapid.

• 25:15Speaker 01

  Is it like a standard bag of tricks? Like the standard one that I know of is, you know, fusing operators and.

• 25:20Speaker 02

  Yeah, it's the standard bag of tricks on like figuring out how to like improve your memory bandwidth and all that.

• 25:27Alessio Fanelli?

  Any ideas instead of things that are not being beaten to death that people should be paying more attention to?

• 25:34Speaker 01

  One thing I was like, you know, you have a thousand operators, right? Like, what's the most interesting usage of PyTorch that you're seeing maybe outside of this little bubble?

• 25:41Speaker 02

  So PyTorch is very interesting and scary at the same time, but basically it's used in a lot of exotic ways, like from the ML angle, like, okay, like what kind of models are being built? And you get all the way from like state space models and all these things to like stuff that like nth order differentiable models, like neural ODEs and stuff like that. I think like there's one set of interestingness factor from like the ML side of things. And then there's the other set of interesting factor from the applications point of view. It's used in Mars rover simulations to drug discovery to Tesla cars.

• 26:23Speaker 02

  And there's a huge diversity of like applications in which it is used in. So in terms of most interesting application side of things, I think I'm scared at how many interesting things that are also very critical and really important it is used in. I think the scariest was when I went to visit CERN at some point and they said they were using PyTorch and they were using GANs at the same time for like particle physics research. And I was scared more about the fact that they were using GANs and they were using PyTorch. Because at that time I was like a researcher focusing on GANs. But the diversity is probably the most interesting, how many different things it is being used in. I think that's the most interesting to me from the applications perspective.

• 27:08Speaker 02

  From the models perspective, I think I've seen a lot of them. Like the really interesting ones to me are where we're starting to combine search and symbolic stuff with differentiable models, like the whole AlphaGo style models is one example. And then I think we're attempting to do it for L L Ns as well with like various reward models and then search. I mean, I don't think PyTorch is being used in this, but like the whole alpha geometry thing was interesting because again, it's an example of combining the symbolic models with the gradient based ones. But there are stuff like alpha geometry that PyTorch has used that, especially when you intersect biology and chemistry with ML. Like in those

• 27:51Speaker 02

  areas, you you want stronger guarantees on the output. So yeah, maybe from the ML side, those things to me are very interesting right now.

• 28:03Speaker 01

  Yeah. People are very excited about the alpha geometry thing. And it's kind of like, for me, it's theoretical. It's, you know, it's great. You can solve some Olympiade questions. I'm not sure how to make that bridge over into the real world applications, but I'm sure people smarter than you will figure it out.

• 28:17Speaker 02

  Let me give you an example of it. You know how like the whole thing about synthetic data will be the next rage in LLMs is a thing?

• 28:27Speaker 01

  It already is a rage.

• 28:28Speaker 02

  Which I think is fairly misplaced in how people perceive it. People think synthetic data is some kind of magic wand that you wave and it's gonna be amazing. Synthetic data is useful in neural networks right now because we as humans have figured out a bunch of symbolic models of the world or made up certain symbolic models because of human innate biases. So we've figured out how to ground particle physics in a 30 parameter model. And it's just very hard to compute, as in like it's like it takes a lot of flops to compute, but it only has 30 parameters or so. I mean, I'm not a physics expert,

• 29:13Speaker 02

  but like it's a very low rank model. We built mathematics as a as a field that basically is very low rank. Language, like a deep understanding of language, like the whole syntactic parse trees and like just understanding how language can be broken down into a formal symbolism is something that that we figured out. So we basically, as humans, have accumulated all this knowledge on these subjects, either synthetic, I mean, we created those subjects, you know, in our heads, or like we've grounded some real world phenomenon into a set of symbols. But we haven't figured out how to teach neural networks symbolic world models directly.

• 29:58Speaker 02

  The only way we have to teach them is generating a bunch of inputs and outputs and gradient dissenting over them. So in areas where we have the symbolic models and we need to teach all the knowledge we have that is better encoded in the symbolic models, what we're doing is we're generating a bunch of synthetic data, a bunch of input-output pairs, and then giving that to the neural network and asking it to learn the same thing that we already have a better low-ranked model of in gradient descent in a much more over parameterized way. Outside of this, like where we don't have good symbolic models, like synthetic data obviously like doesn't make any sense. So synthetic data is not a magic wand where

• 30:41Speaker 02

  it'll work in all cases in every case, you know, whatever. It's just where we as humans already have good symbolic models are. We need to give impart that knowledge to neural networks and we figured out the synthetic data is a vehicle to like impart this knowledge to. So but people, because maybe they they don't know enough about synthetic data as a notion, but like they hear like, you know, the next wave of data revolution is synthetic data. They think it's some kind of magic where we just just create a bunch of random data somehow. They don't think about how. And then they think like that's just the revolution. And I think that's maybe like a gap in understanding most people have in this hype cycle.

• 31:23Speaker 01

  Yeah, well, it's a relatively new concept. So yeah, oh, there's two more that I'll put in front of you and then see if you see what you respond. One is, you know, I have this joke that it's, you know, it's only synthetic data if it's from the Mistral region of France. Otherwise, it's a sparkling distillation, which is which is what news research is doing. Like they're distilling GPT-4 by creating synthetic data from GPT-4, creating mock textbooks inspired by phi two, and then fine tuning open source models like like Llama.

• 31:50Speaker 02

  Yeah.

• 31:50Speaker 01

  And so I don't know, I mean, I think that's, I, it, should we call that synthetic data? Should we call it something else? I don't know.

• 31:56Speaker 02

  I, I, yeah, I mean, the outputs of L L Ms, are they synthetic data? They probably are, but I think it depends on the goal you have. If your goal is like, you're creating synthetic data with the goal of trying to distill G P D four's superiority into another model. I guess you can call it synthetic data, but it also feels like disingenuous because your goal is like, I need to like copy the behavior of G P D four and.

• 32:25Speaker 01

  It's also not just the behavior, but data set. So I, I've often thought of this as data set washing. Like you need one model at the top of the chain.

• 32:33Speaker 02

  Yeah.

• 32:33Speaker 01

  You know, unnamed French company that has, that, you know, makes a model that has all the data and that we don't know where it's from, but it's open source. Hey, and then we distill from that.

• 32:41Speaker 02

  Yeah.

• 32:42Speaker 01

  It's great.

• 32:43Speaker 02

  Yeah.

• 32:44Speaker 01

  To be fair, they also use larger models as judges for preference ranking, right? Like, so that is, I think, in a very, very accepted use of synthetic.

• 32:53Speaker 02

  Correct. I think it's a very interesting time where we don't really have good social models of what is acceptable. Depending on how many bits of information you use from someone else, right? It's like, okay, you use like one bit. Is that okay? Yeah, that's accepted to be okay. Okay, what about if you use like 20 bits? Is that okay? But I don't know. What if you use like 200 bits? Like, I don't think we as society have ever been in this conundrum where we have to be like, where is the boundary of copyright? Or where is the boundary of socially accepted understanding of copying someone else?

• 33:34Speaker 01

  Yeah.

• 33:35Speaker 02

  Like, we haven't been tested this mathematically before, in my opinion.

• 33:38Speaker 01

  Yeah, and whether it's transformative use.

• 33:40Speaker 02

  Yes.

• 33:41Speaker 01

  So, yeah, I, I think this New York Times opening eye case is gonna go to the Supreme Court and we'll have to decide it because.

• 33:47Speaker 02

  I think it'll be very interesting.

• 33:48Speaker 01

  Never had to deal with it before. And then finally, for synthetic data, the, the thing that I'm personally exploring is solving this very stark paradigm difference between RAG and fine tuning, where you can kind of create synthetic data off of your retrieved documents.

• 33:59Speaker 02

  Yeah.

• 33:59Speaker 01

  And then fine tune on that. That's kind of synthetic. All you need is variation or diversity of, of samples for you to fine tune on. And then you can find new knowledge into your, your model.

• 34:09Speaker 02

  Yeah.

• 34:09Speaker 01

  I don't know if you've seen that as a direction for synthetic data.

• 34:13Speaker 02

  I think you're basically trying to, what you're doing is you're saying, well, language, I know how to parameterize language to an extent. And I need to teach my model variations of this input data so that it's resilient or invariant to the, to language uses of that data.

• 34:32Speaker 01

  Yeah, it doesn't overfit on like a

• 34:33Speaker 02

  Yeah, so I think that's a hundred percent like synthetic, right? You understand, like, the key is like, you create variations of your documents. And you know how to do that because you have a symbolic model or like some implicit symbolic model of language.

• 34:48Alessio Fanelli?

  Do you think the issue with symbolic models is just the architecture of the language models that we're building? I think like maybe the thing that people grasp is like the inability of transformers to deal with numbers because of the tokenizer. Is it a fundamental issue there too? And do you see alternative architectures that will be better with symbolic understanding?

• 35:09Speaker 02

  I am not sure if it's a fundamental issue or not. I think we just don't understand transformers enough. I don't even mean transformers as an architecture. I mean, like, the use of transformers today, like combining the tokenizer and transformers and the dynamics of training, like when you, when you show math-heavy questions versus not. I don't have a good calibration of whether I know the answer or not. I, you know, there's common criticisms that are like, well, you know, transformers will just fail at X, but then when you scale them up to sufficient scale, they actually don't fail at that X. I think this is entire subfield where they're trying to figure out these answers called like the science of deep learning or something.

• 35:53Speaker 02

  So we'll get to know where I don't know the answer.

• 35:57Speaker 01

  Let's touch a little bit on just meta AI and, you know, stuff that's going on there. Maybe, I don't know how deeply you're personally involved in it, but you're a first guest in meta AI, which is really, really fantastic. And Lama one was, you know, you are such a believer in open source. Lama one was more or less like the real breakthrough in, in open source AI. The most interesting thing for us on covering on this, in this podcast was the death of chinchilla, as people say. Any, any interesting insights there around like the scaling models for, for open source models or smaller models or whatever that, that design decision was when, when you guys were doing it?

• 36:31Speaker 02

  So Lama 1 was GEM, Lompo and team. There was OPT before, which I think I'm also very proud of. Because we bridged the gap in understanding of how complex it is to train these models to the world. Like, until then, no one really, in gory detail, published.

• 36:50Speaker 01

  The logs.

• 36:51Speaker 02

  Yeah, like, why is it complex? And everyone says like, oh, it's complex. But no one really talked about why it's complex. I think OPT was cool.

• 37:02Speaker 01

  Also, I met Susan and she's very, very outspoken.

• 37:04Speaker 02

  Yeah. We probably, I think, didn't train it for long enough, right? Like, you know, that's, that's kind of obvious in retrospect.

• 37:12Speaker 01

  For a 175B?

• 37:13Speaker 02

  Yeah.

• 37:14Speaker 01

  You trained it according to Chinchilla at the time?

• 37:17Speaker 02

  I can't remember the details, but I think it's a commonly held belief at this point that like, well, if we trained OPT longer, it would actually end up being better. Lama 1, I think, was, yeah, GEM, Lompo and team. GEM is fantastic and went on to build Mistral. I wasn't too involved in that side of things, so I don't know what you're asking me, which is like, well, like, how did they think about scaling loss and all of that? Lama 2, I was more closely involved in. I helped them a reasonable amount with like their infrastructure needs and stuff. And Lama 2, I think, was more like, let's get to the evolution.

• 37:59Speaker 02

  At that point, we kind of understood what we were missing from the industry's understanding of LLMs. And we needed more data and we needed more to train the models for longer. And we made, I think, a few tweaks to the architecture and we scaled up more. And like, that is Lama 2. I think Llama 2, you can think of it as like, after GEM left, the team kind of rebuilt their muscle around Llama 2. And Hugo, I think, who's the first author. It's fantastic. And I think he did play a reasonable big role in Lama 1 as well. And he overlaps with Lama 1 and 2. So Llama 3, obviously, hopefully, will be awesome.

• 38:42Alessio Fanelli?

  Just one question on Lama 2, and then we'll try and fish Llama 3's spoiler side of you. In the Llama 2 paper, the loss curves of the 34 and 70B parameter, they still seem kind of steep. Like, they could go lower. How, from an infrastructure level, how do you allocate resources? Like, could they have just gone longer? Or were you just like, hey, this is all the GPUs that we can burn? And let's just move on to Lama 3 and then make that one better?

• 39:07Speaker 02

  Instead of answering specifically about like that Llama 2 situation or whatever, I'll tell you like how we think about things. Generally, we're, I mean, Mark released some numbers, right?

• 39:20Speaker 01

  So let's cite those things again. All I remember is like 600K GPUs.

• 39:24Speaker 02

  That is by the end of this year, and 600K H100 equivalents. With 250K H100s, including all the other GPU or accelerator stuff, it would be 600 and something K aggregate capacity. That's a lot of GPUs. We'll talk about that separately. But the way we think about it is we have a train of models, right? Lama 1, 2, 3, 4. And we have a bunch of GPUs. I don't think we're short of GPUs, like.

• 39:53Alessio Fanelli?

  Yeah, no, I wouldn't say so.

• 39:54Speaker 02

  Yeah, so it's all a matter of time. I think time is the biggest bottleneck. It's like, when do you stop training the previous one? And when do you start training the next one? And how do you make those decisions? The data, do you have net new data, better clean data for the next one? In a way that it's not worth like really focusing on the previous one. It's just a standard iterative product. You're like, when is the iPhone 1? When like, when do you start working iPhone 2? Where is the iPhone? Like so on, right? So mostly, the considerations are time and generation rather than GPUs, in my opinion.

• 40:31Alessio Fanelli?

  So one other thing with the scaling laws, like Chinchilla is like optimal to balance training and inference costs. I think at meta scale, you would rather pay a lot more, maybe a training and then save on inference. How do you think about that from infrastructure perspective? I think in your tweet, you say you can try and guess on like how we're using these GPUs. Can you just give people a bit of understanding? It's like, because I've already seen a lot of VCs say, Llama3 is being trained on 600,000 GPUs, and that's obviously not true, I'm sure. How do you allocate between the research, like FAIR and the Llama training, the inference on Instagram suggestions that get me to scroll, like AI-generated stickers on WhatsApp and all that?

• 41:10Speaker 02

  Yeah, we haven't talked about any of this publicly, but like as a broad stroke, it's like how we would allocate resources of any other kinds at any company. You run like a VC portfolio, like how do you allocate your investments between different companies or whatever? You kind of make various trade-offs and you kind of decide, should I invest in this project or this other project, or how much should I invest in this project? It's very much like a zero sum of trade-offs. And it also comes into play like how your like clusters configured, like overall, like, you know, what you can fit of what size and what cluster and so on. So broadly, there's no magic sauce here. Like, I mean, I think the details would add

• 41:53Speaker 02

  more spice, but also wouldn't add more understanding. It's just gonna be like, oh, okay, I mean, this looks like they just think about this as I would normally do.

• 42:05Alessio Fanelli?

  So even the GPU-rich run through the same struggles of having to decide where to allocate things?

• 42:11Speaker 02

  Yeah, I mean, like at some point, I forgot who said it, but it's like you kind of fit your models to the amount of compute you have. If you don't have enough compute, you figure out how to make to that small of models. But like, no one, as of today, I think, would feel like they have enough compute. I don't think I have heard any company within the AI space be like, oh, yeah, like, we feel like we have sufficient compute and we couldn't have done better. So it's like that that conversation, I don't think I've heard from any of my friends at other.

• 42:46Speaker 01

  Stella from Alouther sometimes says that because she has a lot of donated compute.

• 42:51Speaker 02

  Yeah.

• 42:51Speaker 01

  And she's trying to put it to interesting uses, but for some reason, she's decided to stop making large models.

• 42:57Speaker 02

  I mean, that's a cool, high conviction opinion that might pay out.

• 43:01Speaker 01

  Why?

• 43:02Speaker 02

  I mean, she's taking a path that most people don't care to take about in this climate. And she probably will have very differentiated ideas. I mean, think about the correlation of ideas in AI right now. It's so bad, right? Like, so everyone's fighting for the same pie. In some weird sense, like, that's partly why I don't really directly work on LLMs. I used to be a gen, like, I used to do image models and stuff. And I actually stopped doing GANs because GANs were getting so hot that I didn't have any calibration of whether, like, my work would be useful or not. Because, oh, yeah, like someone else did the same thing you did. It's like, there's so much to do.

• 43:47Speaker 02

  I don't understand why I need to, like, fight for the same pie. So like, you know, I think Stella's decision is very smart.

• 43:53Alessio Fanelli?

  And how do you reconcile that with how we started the discussion about intrinsic versus extrinsic, kind of like accomplishment?

• 44:01Speaker 02

  Yeah.

• 44:02Alessio Fanelli?

  Success? How should people think about that when, especially when they're doing a PhD or like early in their career? I think in Europe, I walked through a lot of the posters and whatnot. There seems to be multicolored in a way in the research, a lot of people working on the same things. Is it worth for like a PhD to not take a bet on something that is like maybe not as interesting? You know, just because of funding and, you know, visibility and whatnot? Or, yeah, what suggestions would you give?

• 44:28Speaker 02

  I think there's a baseline level of compatibility you need to have with the field. Basically, you need to figure out if you will get paid enough to eat, right? Like, and like whatever reasonable, normal lifestyle you want to have as a baseline. So you at least have to pick a problem within the neighborhood of like fundable. Like, you wouldn't want to be doing something so obscure that people are like, ah, I don't know. Like, you can work on it.

• 44:59Speaker 01

  Would a limit on fundability, I'm just like observing.

• 45:04Speaker 01

  Something like three months of compute, right? That's the top line. That's the like max that you can spend on any one project.

• 45:09Speaker 02

  But like, I, I think that's very ill specified, like how much compute, right? I think the, the notion of fundability is broader. It's more like, hey, are these family of models within the acceptable set of you're not crazy or something, right? Like even something like neural O D E is, which is a very like boundary pushing thing or like state space models or whatever, like all of these things I think are still in fundable territory. When you're talking about, I'm gonna do one of the neuromorphic models and then apply image classification to them or something, then it becomes like a bit questionable. Again, it depends on your motivation. Maybe if you're a neuroscientist, it actually is feasible.

• 45:52Speaker 02

  But if you're like a A I engineer, like the audience of these podcasts, then you know, it's more questionable. The way I think about it is like you, you need to figure out how you can be in the baseline level of fundability, just so that you can just live. And then after that, really focus on intrinsic motivation and depends on your strengths, like how you can play to your strengths and your interests at the same time. Like I try to look at a bunch of ideas that are interesting to me, but also try to play to my strengths. I'm not gonna go work on theoretical M L. I'm interested in it, but when I want to work on something that I try to partner with someone who is actually a good like theoretical M L person and see if I actually have any

• 46:37Speaker 02

  value to provide and if they think I do, then I come in. So I think you'd want to find that intersection of ideas you like and that also play to your, your strengths. And I'd go from there. Everything else, like actually finding extrinsic success and all of that, I think is, the way I think about it is like somewhat immaterial. When you're talking about building ecosystems and stuff, like slightly different considerations come into play, but that's a different conversation.

• 47:05Speaker 01

  Yeah, we're gonna pivot a little bit to just talk about open source A I. But one, one more thing I wanted to establish for meta is like this 600 K number, just kind of rounding out the discussion. That's for all meta, so including your own inference needs, right? It's not just training.

• 47:19Speaker 02

  It's gonna be the number in our data centers for all of meta.

• 47:23Speaker 01

  Yeah, so like, you know, there, there's a decent amount of workload serving Facebook and Instagram and, you know, whatever. And then is there interest in like your own hardware?

• 47:31Speaker 02

  We already talked about our own hardware. It's called MTIA. Our own silicon, I think we've even showed like the standard photograph of you holding like the chip that doesn't work. Like as in the chip that you basically just get like.

• 47:51Speaker 01

  As a test.

• 47:52Speaker 02

  Yeah, a test chip or whatever. So we are working on our silicon and we'll probably talk more about it when the time is right, but.

• 48:00Speaker 01

  Like what gaps do you have that, you know, the market doesn't offer?

• 48:04Speaker 02

  Okay, I mean, this is easy to answer. So basically, remember how I told you about this memory hierarchy and like sweet spots and all of that? Fundamentally, like when you build a hardware, like you, you make it general enough that a wide set of customers and a wide set of workloads can use it effectively while trying to get the maximum level of performance they can. The more specialized you make the chip, the more hardware efficient it's going to be, the more power efficient it's going to be, the more easier it's going to be to find like the software, like the kernel's right to just map one, that one or two workloads to that hardware and so on.

• 48:44Speaker 02

  So it's pretty well understood across the industry that if you have a sufficiently large volume enough workload, you can specialize it and get some efficiency gains, like power gains and so on. So the way you can think about everyone building, every large company building silicon, like I think a bunch of the other large companies are building their own silicon as well, is they, each large company has a sufficient enough set of verticalized workloads that can be specialized, that have a pattern to them, that say a more generic accelerator like an Nvidia or an AMD GPU does not exploit. So there is some level of.

• 49:29Speaker 02

  Power efficiency that you're leaving on the table by not exploiting that. And you have sufficient skill and you have sufficient forecasted stability that those workloads will exist in the same form, that it's worth spending the time to build out a chip to exploit that sweet spot. Like, obviously, something like this is only useful if you hit a certain scale and that your forecasted prediction of those kind of workloads being in the same kind of specializable, exploitable way is true. So yeah, that's why we're building our own chips.

• 50:08Alessio Fanelli?

  Awesome. Yeah, I know we've been talking a lot on a lot of different topics. And going back to open source, you had a very good tweet. You said that a single company's closed source effort raid limits against people's imaginations and needs. How do you think about that? How do you think about all the impact that some of the meta AI work in open source has been doing and maybe the reactions of the whole open source AI space?

• 50:32Speaker 02

  Yeah. In general, I think first, I think it's worth talking about this in terms of open and not just open source, because like with the whole notion of model weights, no one even knows what source means for these things. But just for the discussion, when I say open source, you can assume it's just I'm talking about open. And then there's the whole notion of licensing and all that, like, you know, what happens? Commercial, non-commercial, commercial with clauses and all that. I think like at a fundamental level, the most benefited value of open source is that you make the distribution to be very wide. Like, it's just available with no friction and like you can, people can do transformative things in a way that's very accessible.

• 51:17Speaker 02

  Like, maybe like it's open source, but it has a commercial license and I'm a student like in India. I don't care about the license. I just don't even understand the license. But like the fact that I can use it and do something with it is very transformative to me. Like, I got this thing in a very accessible way. And then like, so it varies degrees, right? Like, and then like, if it's open source, but it's like actually like a commercial license, then a lot of companies are going to benefit from like gaining value that they didn't previously have, that they maybe had to pay a closed source company for it. So open source is just a very interesting tool that you can use in various ways. So there's again two kinds of open source.

• 52:02Speaker 02

  One is like some large company doing a lot of work and then open sourcing it. And that kind of effort is not really feasible by say like a band of volunteers doing it the same way. So there is both a capital and operational expenditure that the large company just decided to ignore and give it away to the world for some benefits of some kind. They're not as tangible as like direct revenue. So in that part, Meta has been doing incredibly good things. They've funded a huge amount of the PyTorch development. They've open sourced Llama and those family of models. And several other fairly transformative projects. Fis is one.

• 52:46Speaker 02

  Segment Anything, Detektion, Detektion 2, Denspose. I mean, it's Seamless? Yeah, seamless. Like, it's just like, the list is so long that, you know, we're not gonna cover. So like, I think Meta comes into that category where like, we spend a lot of CapEx on OpEx and we have a high talent density of great AI people and we open our stuff. And the thesis for that, I remember when Fairway started, the common thing was like, wait, why would Meta wanna start a open AI lab? Like, what exactly is the benefit, like, from a commercial perspective? And for then, like, the thesis was very simple. It was like, AI is currently rate limiting Meta's ability to do things.

• 53:31Speaker 02

  Our ability to build various product integrations, moderation, various other factors. Like, AI was the limiting factor. And we just wanted AI to advance more. And we didn't care if the IP of the AI was uniquely in our possession or not. For us, like, however the field advances, that accelerates like Meta's ability to build a better product. So we just built like an open AI lab and we said, if this helps accelerate the progress of AI, that's strictly great for us. But like very easy rational, right? Still the same to a large extent with like the llama stuff. And it's it's the same values, but like, you know, the argument it's a it's a bit more nuanced.

• 54:13Speaker 02

  And then there's the second kind of open source, which is, oh, you know, we built this project nights and weekends. And we're very smart people and we open sourced it. And then we built a community around it. This is like the Linux kernel and various software projects like that. So I think about open source, like both of these things being beneficial and both of these things being different. They're different and beneficial in their own ways. The second one is really useful when there is an active arbitrage to be done. If someone's not really looking at a particular space, uh, because it's not commercially viable or whatever, like a band of volunteers can just coordinate online and do something and then make that happen. And that's great.

• 54:57Speaker 02

  I want to cover a little bit about like open source L L Ms, maybe. So open source L L Ms have been very interesting because I think we were trending towards a an increase in open source in A I from twenty ten all the way to like twenty seventeen or something. Like where more and more pressure within the community was to open source their stuff so that their methods and stuff get adopted. And then the L L Ms revolution kind of took the opposite effect. Open A I stopped open sourcing their stuff and DeepMind kind of, you know, then like, you know, all the other cloud and all these other like providers, they they didn't open source their stuff.

• 55:37Speaker 02

  And it was not good in the sense that first like science done in isolation probably will just form its own bubble where like people believe their own bullshit or whatever, right? So there's there is that problem. And then there was the other problem, which was the accessibility part like, okay. I again always go back to like, I'm a student in India with no money. What is my accessibility to any of these closer models? At some scale, I have to pay money. That makes it a nonstarter and stuff. And there is also the control thing. I strongly believe if you want human-aligned stuff, you want all humans to give feedback.

• 56:20Speaker 02

  And you want all humans to have access to that technology in the first place. And I actually have seen, because, you know, living in New York, whenever I come to Silicon Valley, I see a different cultural bubble. Like, all the friends I hang out with talk about some random thing, like Dyson spheres or whatever, you know, that's a thing. And most of the world doesn't know or care about any of this stuff. Like, it's like, definitely like a bubble. And bubbles can form very easily. And when you make a lot of decisions because you're in a bubble, they're probably not globally optimal decisions. So I think like open source, the distribution of open source, powers a certain kind of non-falsifiability that I think is very important.

• 57:03Speaker 02

  I think on the open source models, like, it's going great in the fact that Laura, I think, came out of the necessity of open source models needing to be fine-tunable in some way. Yeah.

• 57:19Speaker 01

  Without, yeah, without a large piece.

• 57:21Speaker 02

  And I think DPO also came out of the academic open source side of things. So do any of the closed source labs, or did any of them already have Laura or DPO internally? Maybe, but like, that does not advance humanity in any way. It advances like some companies' probability of doing the winner takes all that I talked about earlier at the podcast. I don't know, it just feels fundamentally good. Like, when people try to, you know, people are like, well, like, what are the ways in which it is not okay? I find most of these arguments, like, and this, this might be a little controversial, but like, I find a lot of arguments based on whether like closed source models are safer or open source models are safer,

• 58:06Speaker 02

  very much related to whether, what kind of culture they grew up in, what kind of society they grew up in. If they grew up in a society that they trusted, then I think they take the closed source argument. And if they grew up in a society that they couldn't trust, where the norm was that you, ah, you didn't trust your government, obviously, like it's corrupt or whatever. Then I think like the open source argument is what they take. I think there's a deep connection to like people's innate biases from their childhood and their trust in society and governmental aspects that pushed them towards one opinion or the other. And I'm definitely in the camp of open source is definitely going to actually have better outcomes for society.

• 58:50Speaker 02

  Close source to me just means that centralization of power, which, you know, is really hard to trust. So I think it's going well in so many ways. That we're actively disaggregating the centralization of power to just like two or three providers. We are, I think, benefiting from like so many people using these models in so many ways that aren't allowed by like, say, like Silicon Valley left wing tropes. Like, some of these things are good or bad, but like, they're not culturally accepted universally in the world. So those are the things worth thinking about. And I think open source is not winning in certain ways. Like, these are all the things in which, like, as I mentioned, it's actually being very good and beneficial and winning.

• 59:35Speaker 02

  I think one of the ways in which it's not winning, at some point, I should write a long form post about this, is I think it has a classic coordination problem. I mean, open source in general always has a coordination problem. If there's a vertically integrated provider with more resources, they will just be better coordinated than open source. And so now open source has to figure out how to have coordinated benefits. And the reason you want coordinated benefits is because these models are getting better based on human feedback. And if you see with with open source models, like, if you go to like, Reddit, local llama subreddit, like there's so many variations of models that are being produced

• 1:00:18Speaker 02

  from say nose research to, I mean, like, there's like so many like variations built by so many people. And one common theme is they're all using these fine tuning or human preferences data sets that are very limited and like someone published them somewhere and like they're they're they're not sufficiently diverse. And you you look at the other side, like say front ends like Uber or like Hugging Chat or uh Ollama. They don't really have like feedback buttons, like all the people using all these front ends. They probably want to give feedback, but there is no way for them to give feedback. So these models are being built, they're being arbitrarily measured,

• 1:01:02Speaker 02

  and then they are being deployed into all these open source front ends or like apps that are closed source, they're serving open source models. And these front ends don't have, they are not exposing the ability to give feedback. So we're just losing all of this feedback. Maybe open source models are being as used as GPT is at this point in like all kinds of, in a very fragmented way. Like in aggregate, all the open source models together are probably being used as much as GPT is, maybe, you know, close to that. But the amount of feedback that is driving back into the open source ecosystem is like negligible. Maybe less than one percent of like the usage. So I think like some, like the the blueprint here, I think, is

• 1:01:47Speaker 02

  you'd want someone to create a sync hole for the feedback, some centralized sync hole, like maybe Hugging Face or someone just funds like, okay, like I will make available a call to log a string along with like, you know, a bit of information of positive or negative or something like that. And then you would want to send pull requests to all the like open source front ends like Uber and all being like, hey, we're just integrating like a feedback UI. And then work with like the closed source people is also being like, look, it doesn't cost you anything, just like have a button. And then the sync hole will have a bunch of this data coming in. And then I think a bunch of open source researchers should figure out how to filter their feedback into only the like high quality one.

• 1:02:30Speaker 02

  I'm sure like it'll be explored by spam bots or whatever, right? Like, this is like the perfect way to inject your advertising product into like the next.

• 1:02:38Speaker 01

  My Coca-Cola now.

• 1:02:40Speaker 02

  So there needs to be some level of that, that in the same way, I'm sure like, like all the close providers are doing today, like OpenAI, Claude, like with the, like the feedback that comes in, I'm sure they are figuring out if that's legit or not. That kind of data filtering needs to be done. And that loop has to be set up. And this requires that central sinkhole and that like data cleaning effort both to be like there. They're not there right now. They're not there right now. I think for capital reasons, but also for coordination reasons. Okay. If that central sinkhole is there, who's going to go coordinate all of this integration across all of these like open source front ends. But I think if we do that,

• 1:03:25Speaker 02

  if that actually happens, I think that probably has a real chance of the open source models having a runaway effect against OpenAI with their current like daily active users, rumored. Probably doesn't have a chance against Google because you know, Google has Android and Chrome and Gmail and Google Docs and everything, you know, so people just use that a lot. But like, I think like, there's a clear chance we can take at truly winning open source.

• 1:04:00Alessio Fanelli?

  Do you think this feedback is helpful to make open source models better or to get to like open source AGI? Because in, in a way like OpenAI's goal is to get to AGI, right? So versus, I think in open source, we're more focused on personal better usage or like commercial?

• 1:04:16Speaker 02

  Yeah, I think that's a good question, but I think like, I actually don't think people have a good understanding of AGI. And I don't mean definition level. I mean, people are like, okay, we're gonna, AGI means it's powering forty percent of world economic output or some, some, something like that, right? But what does that mean? So do you think electricity is powering forty percent of world economic output? Or is it not? Like, generally, the notion of like powering X percent of economic output is not defined well at all for me to understand like how to know when we got to AGI or how to measure whether we're getting AGI.

• 1:05:00Speaker 02

  Like, you know, you can look at it in terms of intelligence or task automation, whatever. I think that's what we are doing right now. We're basically integrating like the current set of AI technologies into so many real world use cases where we find value that if some new version of AI comes in, we can find, like, we can be like, ah, this helps me more. In that sense, I think like the whole process of like how we think we got to AGI will be continuous and not discontinuous like how I think the question is posed. So I think the open source thing will be very much in line with getting to AGI because open source has that natural selection effect.

• 1:05:45Speaker 02

  Like, if a better open source model comes, really, no one says, huh, I don't want to use it because there are ecosystem effect, I'm logged into my ecosystem. Or like, I don't know if I like the models, you know, whatever. It's just a very pure, direct thing. So if there's a better model that comes out, then it will be used. So I definitely think it has a good chance of achieving how I would think about as a continuous path to what we might define as AGI.

• 1:06:18Speaker 01

  For the listeners, I would actually mention a couple other maybe related notes on just this very interesting concept of feedback sync for open source to really catch up in terms of the overall Google versus OpenAI debate. Open Assistant was led by Yannick Kultur, who recently ended his effort. I think the criticism there was like the kind of people that go to a specific website to give feedback is not representative of real world usage. And that's why the models trained on Open Assistant didn't really seem like they have caught on in the open source world. The two leading candidates in my mind are LMCS out of UC Berkeley, who have the LMCS arena, which, you know, is being touted as one of the only ways, only reliable benchmarks anymore. I kind of call them non-parametric benchmarks because there's nothing to cheat on it except for.

• 1:07:04Speaker 01

  ELO. And then the other one is Open Router, which is Alex Atala's thing. I don't know if you've talked to any of these people.

• 1:07:11Speaker 02

  I obviously know all of the efforts that you talked about. I haven't talked to them directly about this yet, but the way I think about it is the way these models are going to be used is always going to be way more distributed than centralized. Like, which is the power of the open source movement, like the UI within which these models are going to be used is going to be decentralized. Like, it's, these models are going to be integrated into like hundreds and thousands of projects and products and all of that, right? And I think that is important to recognize. Like, the LNS leaderboard is the best thing we have right now to understand whether a model is better or not versus another model.

• 1:07:54Speaker 02

  But it's also biased and only having a sliver of view into how people actually use these models. Like, the people who actually end up coming to the LNS leaderboard and then using a model only use it for certain things, like GitHub Copilot style usage is not captured in, say, like LMS things. And so many other styles, like the character AI style things is not captured in.

• 1:08:19Speaker 01

  Which Open Router could do. They don't do it right now, but.

• 1:08:22Speaker 02

  Yeah, so like, I think, like, yeah, my, my point is like, the way these models are going to be used is going to be always a large surface area, and I think we need to figure out how to provide the infrastructure to integrate with all these, like, ways in which it's being used. Even if you get like the top hundred front ends that the model, like, open source models are used through, subscribe to like the sync hole. I think that's already like a substantial thing. I think like thinking one or two things built by themselves get a lot of data, I think is not gonna happen.

• 1:08:58Speaker 01

  Yeah, fair enough.

• 1:08:59Alessio Fanelli?

  Before we let you go, can we do just a quick beyond text segment? So you're an investor in Broadway, which is a video generation. You're an investor in One X, which is a humanoid assistant. Osmo, which is focused on using AI for smell recognition and synthesis. You advise a bunch of robotics projects at NYU.

• 1:09:19Speaker 01

  And he builds his own home robot.

• 1:09:21Alessio Fanelli?

  Yeah, exactly. On a more, yeah, maybe you open another thing. What are like the things that you're most excited about beyond like text generation and kind of the more mundane usage?

• 1:09:30Speaker 02

  Yeah, I mean, in general, I have more things I'm generally excited about than I can possibly do. Investing is one way to try to clear those urges. I'm generally excited about robotics being a possibility, home robotics being like five to seven years away into commercialization. I think like it's not like next year or two years from now, but like in five to seven years from now, I think like a lot more robotics companies might pop out. There's not a good consensus on whether hardware is a bottleneck or AI is a bottleneck in robotics right now. My view is actually hardware is still the bottleneck.

• 1:10:14Speaker 02

  And AI is also a little bit of bottleneck, but like I don't think there's any like obvious breakthroughs we need. I think it just works. So I'm generally excited about robotics. I spend a lot of personal time. I spend like every Wednesday afternoon at NYU, working with Laurel Pinto and team. And just getting towards my like home robot that just does my dishes and stuff.

• 1:10:38Speaker 01

  What's the status of it? Like, what does it do for you now?

• 1:10:41Speaker 02

  As of today, we just deployed, a couple of months ago, we deployed our home robotic stuff into like several tens of New York City homes and like tried to make it do a bunch of tasks. And we're basically starting to build out a framework that gets to a certain level of robustness on fairly simple tasks, like, you know, picking this cup and putting it somewhere else, or like taking a few pieces of cloth on the ground and put it somewhere else. Or open your microwave, like various like baseline tasks like that.

• 1:11:18Speaker 02

  With low sample complexity. So I, I think one of the things people don't spend their time in robotics is like the user experience, which I think we, in the research I do at NYU, we spend a huge amount of time on. I think the key there is sample complexity has to be really low. A lot of the current robotics research, if you see, they're like, oh yeah, we collected like 50 demos and now it's able to do this task, or we collected like 300 demos. Or like the number of samples you need for this thing to do the task is really high. So we're focusing a lot on, you show it like two or three times and that's sufficient for it to actually like do the task. But it comes with like less generalization, right? Like you, there's some initial conditions that have to be true for it to do the task.

• 1:12:03Speaker 02

  So we're making progress. That's very interesting in general, the space. I don't think people in the space have settled on the hardware, like how the hardware looks like for it to be truly useful in the home or whatever, or the UX, or the like AI ML stuff needed to make it sample efficient and all of that. But I think like lots of work is happening in the field.

• 1:12:28Alessio Fanelli?

  Yeah, one of my friends, Carlo at Berkeley, he worked on a project called M3L, which is two CNNs, one for tactile feedback and one for image. When you say hardware, is it running all these things on the edge? Or is it just like the actual servos and?

• 1:12:44Speaker 02

  Yeah, by hardware, I mean like the actual like servos, like the motors, servos, even like the sensors. I think we have incredible vision that's still like, it's so much better compared to, in the field of view and in resolution compared to any of the cameras we can buy. We have, our skin is like all available touch sensing. And we have like some of the most efficient, you know, some of the most high capacity motors that can lift large loads, you know, in, in like the dexterity of a hand and stuff. So in terms of hardware, I mean, like in terms of those capabilities, like, you know, we haven't figured out how to do a lot of this stuff.

• 1:13:28Speaker 02

  I mean, Tesla has been making incredible progress. One X, I think announced their new thing that looks incredible. Some of the other companies figure and like others are doing great work. But we're really not anywhere close to like the hardware that we feel like we need. And there's obviously the other thing I want to call out is a lot of what people show works, but like has to be fixed all the time. And like, that's the other thing we we are incredible at, like, we don't need any maintenance or like the maintenance is part of us. If you buy a product, electronics product of any kind, you you buy a PS five, you don't say, oh, yeah, my PS five breaks like every six days and I have to like do

• 1:14:09Speaker 02

  some reasonable amount of work on it. But like, that's robotics, like if it's not industrial robotics where it's very controlled and specialized or whatever, like, you're talking about reliability like in those ranges. So I think people don't talk about the reliability thing enough. Like, what I mean, like, we're gonna enter the commercialization phase. I mean, like, we're gonna start thinking about, okay, now we have this thing and we need to figure out how to get reliability high enough to deploy it into homes and like just sell it to people and like Best Buy or something. So that's the other factor that that's we have to make a lot of progress on.

• 1:14:44Speaker 01

  I just realized that Google has a play in this with like Palm E and stuff and OpenAI obviously has a long history of doing this stuff. Is there anything at Meta? No robotic stuff in Meta?

• 1:14:55Speaker 02

  We have a small robotics program at Meta out of FAIR. I actually used to do it at FAIR a little bit before I moved into infra and focused on my my Meta time on a lot of like other infrastructural stuff. So yeah, Meta's robotics program is a lot smaller.

• 1:15:10Speaker 01

  Seems like it would be a

• 1:15:12Speaker 02

  I think like

• 1:15:12Speaker 01

  personal computing.

• 1:15:14Speaker 02

  You could think of it as like Meta has a ridiculously large device strategy, right? Like, you know, this is how our reality labs stuff. Like, you know, we're going at it from VR and AR and, you know, we showcase a lot of that stuff. I think for Meta, the robot is not as important as like the

• 1:15:34Speaker 01

  Screen real estate.

• 1:15:35Speaker 02

  Physical devices kind of stuff. Yeah, for sure.

• 1:15:37Speaker 01

  Yeah. Okay, I want to touch on Osmo a bit.

• 1:15:40Speaker 02

  Yeah.

• 1:15:40Speaker 01

  Because very unusual company to the stuff that we normally discuss, not robotics, sense of smell. The original pitch I heard from the founder, maybe you can correct me, is that you realize that you can smell cancer.

• 1:15:51Speaker 02

  Yeah.

• 1:15:52Speaker 01

  Is that intuitive? Is that what you get?

• 1:15:54Speaker 02

  Yeah, I mean, first, like, the very interesting reason I invested in Osmo is because Alex Wielsko, the founder of Osmo, before PyTorch, there was Torch. And Alex Wielsko actually worked on Torch. He's actually like a frameworks guy, like, you know, he built this thing called Tangent from Google, like another like alternative framework and stuff. Like, so I know him from that side of things. And then, like, I also, like, he is a neurobiologist by training. He just happens to also love like neural networks and like hacking on those frameworks. So incredibly smart guy, one of the smartest people I know. So when he was going in this direction,

• 1:16:34Speaker 02

  I thought it was incredible that like smell is something that we haven't even started to scrape in terms of digitization. When we think about audio or images or video, they're like so advanced. So we have the concept of color spaces. We have the concept of like frequency spectrums. Like, you know, we figured out how ears process like frequencies in Mel's spectrum or whatever, like logarithmically scaled. Images for like RGB, YUV, like we have so many different kinds of parameterizations. We have formalized these two senses ridiculously well.

• 1:17:13Speaker 02

  Touch and smell nada, very like where we were with images in, say, in nineteen, twenty, or maybe even the eighteen hundreds, right? That's where we're at. And Alex has this incredible vision of like having a smell sensor just eventually just be part of your daily life. Like, as of today, you don't really think about.

• 1:17:38Speaker 02

  Like when you're watching an Instagram reel or something, huh? Like, I also would love to know what it smelled like. And if you're watching a reel of a food or something. You don't, because we really haven't, as a society, got that muscle to even understand what a smell sensor can do. I think the more near-term effects are obviously going to be around things that provide more obvious utility in the short term, like maybe smelling cancer or like repelling mosquitoes better, or you know, stuff like that.

• 1:18:12Speaker 01

  More recently, he's been talking about like categorizing perfumes.

• 1:18:15Speaker 02

  Yeah, exactly.

• 1:18:16Speaker 01

  That's a market that you can pursue.

• 1:18:17Speaker 02

  Yeah, like, I mean, think about how you can customize a perfume to your own liking in the same way you can customize a shoe or something, right? So that like, that's, I think, all the near-term stuff. I think if he's able to figure out a near-term value for it, they, as a company, can sustain themselves to then eventually like try to make progress on the long term, which is really an uncharted territory. Like, think about it, 50 years from now, it would be pretty obvious to like kids of that generation to just like, I was gonna say scroll a reel on their phone. And maybe phones wouldn't be there. They're just like, you know, on their glasses, they're watching something.

• 1:18:59Speaker 01

  I think VR would be.

• 1:19:00Speaker 02

  And then like they immediately get like a smell sense of that remote experience as well. Like, we haven't really progressed enough in that dimension. And I think they have a chance to do it.

• 1:19:12Alessio Fanelli?

  Awesome. I mean, we touched on a lot of things. Anything we're missing, anything you wanna direct people to or?

• 1:19:19Speaker 01

  Yeah, call to action.

• 1:19:20Alessio Fanelli?

  Yeah.

• 1:19:20Speaker 01

  Call for research, call for startups.

• 1:19:22Speaker 02

  I don't really have a lot of calls to action, because usually I think internet, like people should be intrinsically like figuring out.

• 1:19:29Speaker 01

  That's a good look inside yourself.

• 1:19:31Speaker 02

  Yeah.

• 1:19:32Alessio Fanelli?

  That's good. Awesome. Thank you so much for coming on.

• 1:19:35Speaker 02

  Yeah, for sure.

• 1:19:36Speaker 01

  Thanks a bit.