#496 – FFmpeg: The Incredible Technology Behind Video on the Internet

Jean-Baptiste Kempf is lead developer of VLC and president of VideoLAN. Kieran Kunhya is a longtime FFmpeg contributor, codec engineer, and the person behind the now-infamous FFmpeg account on X. Thank you for listening ❤ Check out our sponsors: https://lexfridman.com/sponsors/ep496-sc See below for timestamps, transcript, and to give feedback, submit questions, contact Lex, etc. Transcript: https://lexfridman.com/ffmpeg-transcript CONTACT LEX: Feedback – give feedback to Lex: https://lexfridman.com/survey AMA – submit questions, videos or call-in: https://lexfridman.com/ama Hiring – join our team: https://lexfridman.com/hiring Other – other ways to get in touch: https://lexfridman.com/contact EPISODE LINKS: FFmpeg on X: https://x.com/FFmpeg FFmpeg: https://ffmpeg.org/ VideoLAN (VLC): https://www.videolan.org/ VideoLAN on X: https://x.com/videolan Jean-Baptiste’s Website: https://jbkempf.com/ Jean-Baptiste’s LinkedIn: https://www.linkedin.com/in/jbkempf/ Jean-Baptiste’s GitHub: https://github.com/jbkempf Kieran’s X: https://x.com/kierank_ Kieran’s LinkedIn: https://bit.ly/3OORhmC Kieran’s GitHub: https://github.com/kierank SPONSORS: To support this podcast, check out our sponsors & get discounts: Larridin: Measure AI adoption in your business. Go to https://larridin.com https://lexfridman.com/s/larridin-ep496-sc Blitzy: AI agent for large enterprise codebases. Go to https://blitzy.com/lex https://lexfridman.com/s/blitzy-ep496-sc BetterHelp: Online therapy and counseling. Go to https://betterhelp.com/lex https://lexfridman.com/s/betterhelp-ep496-sc Fin: AI agent for customer service. Go to https://fin.ai/lex https://lexfridman.com/s/fin-ep496-sc LMNT: Zero-sugar electrolyte drink mix. Go to https://drinkLMNT.com/lex https://lexfridman.com/s/lmnt-ep496-sc Perplexity: AI-powered answer engine. Go to https://perplexity.ai/ https://lexfridman.com/s/perplexity-ep496-sc OUTLINE: (00:00) – Introduction (03:00) – Sponsors, Comments, and Reflections (10:48) – Weirdest things VLC opens (15:12) – How video playback works (24:33) – Video codecs and containers (35:20) – FFmpeg explained (56:20) – Linus Torvalds (1:00:59) – Turning down millions to keep VLC ad-free (1:15:17) – FFmpeg & Google drama (1:34:31) – FFmpeg developers (1:41:08) – VLC and FFmpeg (1:45:42) – History of FFmpeg (1:48:59) – Reverse engineering codecs (2:02:14) – FFmpeg testing (2:06:21) – Assembly code (handwritten) (2:30:39) – Rust programming language (2:39:55) – FFmpeg and Libav fork (2:48:17) – Open source burnout (2:56:04) – x264 and internet video (3:09:20) – Video compression basics (3:16:17) – CIA and fake VLC (3:26:52) – Ultra low latency streaming (3:44:20) – AV2 codec and video patents (3:54:12) – VLC backdoors (4:04:27) – Video archiving (4:11:04) – Future of FFmpeg and VLC

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Lex Fridman?

  The following is a conversation all about FFmpeg and VLC with Jean Baptiste Kempf and Kieran Cunha. FFmpeg is an open source software system that is the invisible backbone behind YouTube, Netflix, Chrome, VLC, Discord, and basically every platform that touches video or audio on the internet. It can decode, encode, transcode, stream, and play almost any video or audio format ever created. To me, it is one of the most incredible software systems ever developed, and it's all done by volunteers. VLC is also a legendary piece of software.

• 0:45Lex Fridman?

  It is an open source media player that plays basically anything you throw at it, any format, any platform, no ads, no tracking. It has been downloaded over 6 billion times. And again, for me, it has been one of my favorite pieces of software ever, with the most legendary logo, which I, of course, had to honor in this conversation by wearing the VLC Traffic Cone hat the whole time. So again, above all else, thank you to the incredible volunteer engineers who put their heart and soul into this code that has been used and loved by billions of people. Thank you.

• 1:26Lex Fridman?

  And about the two great engineers and human beings I'm talking to in this episode, Jean Baptiste is the president of Videoland and is a key figure behind VLC and FFmpeg. Kieran is a longtime codec engineer, FFmpeg contributor, and the man behind the now infamous FFmpeg account on Twitter, X, that I recommend everybody follow for the memes and for the unapologetic celebration of open source and great low-level software engineering.

• 2:03Lex Fridman?

  Let me also say that it's inspiring and humbling that so much of modern civilization rests on software built by people who are not chasing fame or money, but are obsessed with the craft of engineering. We live in a world where billions of people consume video every day without ever thinking about the invisible machinery underneath it. But that machinery matters. Open source infrastructure matters. It is one of the great examples of human beings quietly collaborating across borders to build something useful, durable, and elegant for the rest of us. And so this conversation is not just about codex and media pipelines.

• 2:46Lex Fridman?

  It is also about the deeper spirit of engineering and generosity that makes projects like FFmpeg possible. Again, I can never say it enough. Thank you.

• 3:00Lex Fridman?

  And now a quick sheet second mention of each sponsor. Check them out in the description or at lexfridman.com/sponsors. It is, in fact, the best way to support this podcast. We got Leredin for understanding how AI is used in your business. Blitzy for code generation and large code bases. BetterHelp for mental health. Finn for customer service AI agents. Element for electrolytes and complexity for curiosity-driven knowledge exploration. Choose wisely, my friends. And now on to the full lotteries. I try to make them interesting, but if you skip, please still check out the sponsors. I enjoy their stuff. Maybe you will too. To get in touch with me for whatever reason, go to lexfridman.com/contact. All right, let's go.

• 3:45Lex Fridman?

  This episode is brought to you by Leredin, a platform that helps organizations understand how AI is being used across their business and what it's doing for productivity and performance. There really is a transformation happening at the individual developer level. Many people switch from writing, let's say 50%, 40%, 30% of their code where the rest is written by AI. They switch to basically where it's 0% of code written by hand via so-called agentic engineering. And so we could see that in the individual developers. Now, the question is, when you scale that to two, three, four, five, 100 developers, what does that look like?

• 4:27Lex Fridman?

  What does that look like from the perspective of a company that's trying to actually ship products? And trying to coordinate teams so they can build and collaborate together? How is AI being used to increase the productivity of the individual contributors and team seen as a whole? That's what Leredin does. If AI is part of your organization, now is the moment to get control of it. Head to Leredin.com to book a demo and to start maximizing impact from AI. This episode is also brought to you by Blitzy, an AI-powered autonomous software development platform. Built for large, complex code bases, huge number of cooperative agents, really optimized for huge code bases.

• 5:12Lex Fridman?

  Optimized for scaling speed when you're talking about a very large number of agents working together. And that's the big, interesting question. When you have a huge number of agents, a huge company, huge code base, how do you then, seen at the big picture code base level, have the growth and development, the evolution of that code base where a very large percentage of that code base is continuously worked on autonomously? The question is, when you have a large code base that already delivers value, that already sells stuff,

• 5:50Lex Fridman?

  that already has huge number of customers, how do you then use agentic engineering to continue adding features? Continue improving, continue in the usual kind of development with the testing, with the security, all that kind of stuff? How do you do that without messing stuff up, without filling up your code base with AI slop? And nevertheless, doing it so for the most part autonomously, not fully autonomously, semi-autonomously, but majority of the code is written autonomously. That's what Blitzy specializes in. The future of autonomous software development is here. Learn more or speak to a member of their team at blitzy.com/lex. That's blitzy.com/lex.

• 6:32Lex Fridman?

  This episode is also brought to you by BetterHelp, spelled H E L P, help. Moving away from AI to the human. The human mind is still, to this day, out of reach of our understanding from the perspective of creating intelligence. There are so many intricate psychological complexities to the mind, which I think attributes to what makes humans incredibly special. But those complexities get all tangled up in ways that are counterproductive. And so they need to be untangled. And I'm a big fan of talk therapy as a set of tools, as a methodology

• 7:14Lex Fridman?

  for untangling the complexities of the human mind. The easy, discreet, affordable way of doing that is BetterHelp. That's why I keep recommending it. It's a really good way to take your first steps. If you haven't done talk therapy, get a licensed professional therapist in under 48 hours. BetterHelp makes it super easy. Check them out at betterhelp.com/lex and save in your first month. That's betterhelp.com/lex. This episode is also brought to you by Finn, the number one AI agent for customer service. As I mentioned already about humans, humans are complicated. And customer service is ultimately about looking at each individual human and really listening.

• 7:57Lex Fridman?

  So I try to do with the podcast to truly listen to each individual person, whether we're talking about a super technical. Topic, or we're talking about the big questions of the human condition. And sometimes when you're talking about the details of the technical topic underneath there, you can feel the presence of the big picture questions of the human condition. All of those complexities peek out of the shallow surface interactions that at first glance, you might think customer service interaction is. But really, we're trying to solve the big problems of the human condition and specific to that individual person.

• 8:39Lex Fridman?

  So customer service is a really hard problem, but it's a really important one. And that's what Finn specializes in, how to leverage, how to use, how to utilize AI in the task of customer service. Go to fin.ai/lex to learn more about transforming your customer service and scaling your support team. That's fin.ai/lex. This episode is also brought to you by Element, my daily zero sugar and delicious electrolyte mix. I'm traveling to parts of the world that at first feel, at first experience are so foreign that they border on the uncomfortable. All new things can be uncomfortable.

• 9:23Lex Fridman?

  And for me, Element on the health side, psychological side, are a source of comfort. They're a reminder of home. It's a place where I'm healthiest. It's a place where I have everything sort of in line. I'm exercising, running every day, whether that's jiu-jitsu or running or lifting. I am on a good diet. I'm fasting a lot. And for all of that, you need the electrolytes that Element provides, sodium, potassium, magnesium. Really, if you get the electrolytes right, everything else is a little bit easier. You don't get the headaches. You don't get just the weird iffy feeling when you're fasting. Favorite flavor, as always, I should mention, because I love it so much, is watermelon salt. Get a free ACOM sample pack with any purchase.

• 10:08Lex Fridman?

  Try it at drinkelement.com/lex.

• 10:14Lex Fridman?

  This is the Lex Freedom Podcast. To support it, please check out our sponsors in the description, where you can also find links to contact me, ask questions, give feedback, and so on. And now, dear friends, here's Jean Baptiste Kemf and Kiran Kuniya.

• 10:48Lex Fridman?

  So the legend goes, VLC can open everything. What's the weirdest thing that you know that it can open?

• 10:55Speaker 01

  You know, there is a ton of people who are using VLC to record VHS videos, right? Like, it's just like you plug it with a capture card and you can basically record VHS video.

• 11:05Lex Fridman?

  Well, how does that work?

• 11:06Speaker 01

  Basically, it's, you know, those type of capture card where you can put a Peritelle in, or RGA, and you put that, and actually VLC can play those type of cards. And there is a module which allows to control directly some of those VCR camcorders. We support DVD audios lately, right? We spent the summer working on DVD audio support. And like, there is no, no one's making any DVD audio support. There is a custom encryption schemes.

• 11:31Speaker 02

  What about Lucasfilm?

• 11:33Speaker 01

  Oh, yeah. And there is, of course, all the weird codecs, game codecs, supported by FFmpeg.

• 11:38Speaker 02

  The one Star Wars video game, the first 10-second opening sequence, someone has gone and implemented that and make sure that's bit exact on one disc that existed at one time of one little sequence in the game.

• 11:49Speaker 01

  And then, funnily, there was a, at one Videoland conference, we made a competition to make the weirdest and most horrible file ever and see if VLC could play it.

• 11:59Lex Fridman?

  What did it end up being? What's the file?

• 12:01Speaker 01

  It was an MKV file made by Dirac, which each of the frame was changing resolution, aspect ratio, rotation, and it was like...

• 12:12Lex Fridman?

  Did it work?

• 12:13Speaker 01

  Yes. And there was another one where the whole video was actually animated subtitled, right? SSA, right?

• 12:20Lex Fridman?

  Yeah.

• 12:22Speaker 01

  This one was, so each, each frame was a black frame, but on top of that, there was a subtitle that was animated for each frame.

• 12:29Speaker 02

  There was a file that's a valid zip and a valid MP3 at the same time, or something like that, so.

• 12:33Speaker 01

  So, yeah, we'd made a competition of stupid files.

• 12:36Lex Fridman?

  And it worked. It opened all of the stupid files.

• 12:39Speaker 01

  Yes.

• 12:40Lex Fridman?

  By the way...

• 12:41Lex Fridman?

  For people who are not familiar, I am wearing a hat. Would it be fair to say this is the best, worst logo of all time, the cone?

• 12:49Speaker 01

  Yeah, by far, right? The logo of VLC is so iconic, right? Like, we are a team with a small number of people and the icon is known everywhere. I go to middle of nowhere in India or in China, people know the cone, right? And 25% of the website traffic that comes to our main website is cone player, right? So so many people don't know VLC, right? They know the cone player.

• 13:12Lex Fridman?

  That's the thing that Google forced, cone player.

• 13:14Speaker 01

  Yeah, they go on Google and they put cone player and they download VLC, right? So that's iconic. And once we tried to change it as a joke, right? We said it was going to be a type of caterpillar construction. And we said that during April 1st. We had around 10,000 emails saying, no, don't change the logo and so on, right? So it's so iconic, right? It's so distinctive, right? If you want to do a video player, you're going to put a play button on the TV, right? And that's a YouTube, YouTube logo, right? It's an original. This one is orange, right? It's very bright and and it's weird.

• 13:50Lex Fridman?

  And it's ridiculous and it's absurd and it's hilarious. It becomes meme and meme becomes culture.

• 13:54Speaker 01

  And you keep it and you know about it. And you know that in 20 years, like you still have going to have the cones and remember, oh, yeah, that was a video player.

• 14:02Lex Fridman?

  Yeah, and we'll talk about, you know, the the mission of FFmpeg being a kind of the archival aspect of it. So you could think about 1,000 years from now, we'll have all these videos that only VLC can open. Humans, human civilization has already destroyed itself multiple times. And the only thing that will remain is this like, you know, the cockroaches will be crawling around. It'll be the VLC logo with some of the archival footage that VLC can open. And the aliens will show up and they'll press play and they'll get to see all this.

• 14:31Speaker 01

  Really, really hope so, right? But there is also so many memes where people say, well, I'm sure I can put a pancake inside my DVD drive and VLC will play it.

• 14:38Lex Fridman?

  Can they?

• 14:39Speaker 01

  No, we tried, it doesn't. But we actually have a video of us trying that, didn't work.

• 14:44Lex Fridman?

  A codec for physical reality? I don't know what that would even look like.

• 14:47Speaker 01

  There was a guy who did that, right? He he he printed a small cone, right? Like the the ones we distribute as goodies. And inside he put an RFID chip, which was his way of playing a movie, right? And so he put this on RFID player. And when he put that, it was playing like The Last Star Wars and so on. So instead of having like DVD boxes, he had like. VLC cones all around and you plug that and there was like physical objects.

• 15:12Lex Fridman?

  So the thing that we're talking about is everything around video codecs, video encoding, video decoding, video streaming, video player client, that I'm wearing on my head, the entire ecosystem enabling free media. We'll talk about FFmpeg, we'll talk about video LAN, VLC and all the other incredible video technology that is used probably by billions of people. So JB, you're the lead developer behind the legendary VLC player. Kieran, amongst many other things, you're the lead developer behind the legendary FFmpeg handle on Twitter. And both of you have spicy opinions, I would say. So today I want to talk about FFmpeg and VLC.

• 15:58Lex Fridman?

  For context, for people who are not aware, and I'm sure basically everybody listening to this have used these two technologies probably regularly without knowing it. So FFmpeg analyzes basically most video on the internet, including YouTube, Netflix, Chrome, Firefox, of course VLC and countless other video platforms. It is estimated that over 90% of video processing workflows online and offline involve FFmpeg. VLC has been downloaded at least 6.5 billion times. But likely that number, because it's impossible to really count the number, is much higher than that.

• 16:41Lex Fridman?

  Virtually any operating system supports virtually any media format.

• 16:48Lex Fridman?

  The limitation being you can't open pancakes. So can we just lay out some of the basics to help people understand what's involved in all of this? So when we press play. On a video player like VLC, what happens? What, how does it go from the file or the stream to the pixels on the screen and the sound on the speaker? What are the big stages to be aware of?

• 17:14Speaker 01

  So there are several stages, right? The first stage is to get from an address, right? Which is like a type of URL to give you a byte of streams, right? So this would be, for example, HTTP file DVD, right? To give the pass to the media and give you a stream of data.

• 17:32Speaker 02

  The stream needs to be cut up by what's known as the container, the demultiplexer or demux. We'll try and keep the jargon light throughout this, but it needs to go and start demarcating video and audio frames. So it just gets data from the operating system blocks at a time and needs to start cutting these frames up into compressed data. It then needs to start doing simple parsing of the video frames, mainly to figure out whether that codec is GPU decodable or needs to fall back to software. We're very sort of used to assuming the GPU will play all of these things, there'll be hardware acceleration. I think it's up to 45% of files are not GPU decodable. So these need to be probed, they need to be detected. There can be variants of a given codec, some of which are decodable on the GPU.

• 18:17Speaker 02

  Different vendors of GPU might have different capabilities. So those need to be detected. So if it's GPU capable, you pass it through to the GPU black box. So now if there's a software fallback, that means in the beginning is to first do de-entropy coding. So removing the mathematical coding of the bit stream. So this uses capabilities such as Huffman coding or arithmetic coding to actually decompress the mathematical layer of the bit stream. We then need to start reading the syntax elements for intra-predictions. So intra-predictions are like still images of the video, so your iframes.

• 18:52Speaker 01

  So this works and operates in the spatial domain. So you do your interprediction spatial domain. You have a residual because your prediction isn't quite matching that of reality. So you've made a prediction, but then there's a little bit left and that's what's known as the residual. This is stored in the frequency domain and these are quantized to decompan their space. We then need to do the inverse transform to bring them back to the spatial domain and apply these residuals.

• 19:18Lex Fridman?

  So a lot of the process of the decoding is this thing is compressed.

• 19:22Speaker 01

  Yes.

• 19:23Lex Fridman?

  Yes. And you have to predict the highest quality thing that's supposed to go there. iframe is the best representation you have spatially.

• 19:31Speaker 01

  Yes.

• 19:32Lex Fridman?

  And then you, and then there's a lot of temporal compression that can happen depending on the codec. And then you're predicting, you're predicting what the reality that was captured in this rawest form.

• 19:42Speaker 01

  Yeah, because what, what people don't realize is that the compression on video and audio is one hundred times, right? Like people don't realize how compressed we, we do, right? For audio, you move, you're compressed by when you go from normal audio to MP3, you're compressed by 10 times, right? When, when you move to video, you need one hundred times, 200 times, right? So you need to remove all the details, but that you don't care about. Because all the compressions that we do, and that's very important, people forget about that, is to be viewed by humans, right? So all the codecs either for audio mimic basically how your ear works, right? And, and a lot of things about like the, the, the response on the ear and same for, for your eyes, right?

• 20:25Speaker 01

  And, and so for example, on video, we don't work on RGB, right? Everyone expects to work in RGB. We don't, right? We move to YUV, which is basically one is luminance, brightness, and the other are colors. And this matches your eyes where inside your eyes, you have the cones and the buttons, right? With some of them look on brightness and more on the other on colors, right? So we need to compress a lot and so we need to degrade. But in order to degrade, we need to match the human perception. And this is why it's so difficult. And then we need to use the maximum power, mathematical power, very complex technologies.

• 21:02Speaker 01

  To the frequency domain, as Kiran said, we do a ton of dequantizing and in order to get the best compression, but it still looks good.

• 21:11Lex Fridman?

  You're trying to compress in order to maximize the highest quality thing for human perception.

• 21:17Speaker 01

  That is correct, and that is correct, and this is very important, right? Compression is not like a zip, right? A zip, you have data in, you get data out, right? And you try with all the zip compression to arrive with the limit. Here, we are degrading the signal, right? And so we need to degrade both the audio and the video signal in the best way possible. And we can do that, but it involves first a lot of theoretical knowledge about how it works, the eye works. But a lot of mathematical change, a lot of mathematical tricks, right? For example, when you move to RGB and you go to YUV, for example, what we do very often is that we scale down the resolution of the color compared to the brightness.

• 22:01Speaker 01

  And most of the time, and just this without compression, it divides the size by two. But most people don't see it, right? And so on and so on, right? And then you go to very complex mathematical change. So of course Fourier transform, which the factor are not Fourier transform. They are like discrete cosine transform, but that's the same idea. So frequency domain, we split the video by blocks, right? So that's why when it's wrongly decoded, you see those blocks and badly encoded, you see those blocks and so on. To arrive to compression states that are insanely high, right? And each generation of the codec is like thirty percent less for the same quality, right?

• 22:43Speaker 01

  And this requires amount of power, of computational power that are huge.

• 22:48Speaker 02

  No, you should elaborate. It's thirty percent better, but an order of magnitude, perhaps, perhaps even two orders of magnitude more compression power. That's the big difference.

• 22:58Lex Fridman?

  What do you mean by compression power?

• 23:00Speaker 02

  Sorry, CPU power to achieve that level of compression.

• 23:02Lex Fridman?

  Oh, yeah. Saying you have to be able to leverage the CPU and sometimes GPU, like you mentioned. And then we should mention that a lot of this programming is done at the lowest possible stack, whether it's C and of course, as the legendary Twitter handle reemphasizes over and over, a lot of assembly.

• 23:22Speaker 01

  So what happens globally is that you have an address, right? Which gives you with the operating system, a stream of bytes, a stream of data, right? And this is the first step. And the second step arise with demaxing, where you're going to separate audio, video, subtitle in type of different tracks. And then on each of those tracks, you're going to decompress them, decode them, either audio with an audio codec, video to video codec, and subtitle to subtitle codec. And once you've decompressed those type of things, you have raw images, raw, and then you're going to talk with your graphic card in your screen and display that. And same for the audio, you're going to talk to your audio card, which then is going to go in analog to your audio speakers.

• 24:03Speaker 02

  And everything we've just said in the past couple of minutes, every sentence is someone's lifetime's work. There are books about every sentence. So the level of complexity, in many cases, is inordinate. You know, it's, it's, every sentence has thousands of people working on this in, in industry as a whole, books written about it. So there's a lot of detail, there's a lot of subtleties, there's a lot of both academic and practical realities, both of which matter.

• 24:33Lex Fridman?

  We mentioned codecs, but I don't think you mentioned containers. So what, what's the actual containers for some of the stuff we're talking about? So people are familiar with MP4, MOV, MKV. So anyway, what are containers versus the thing that goes inside?

• 24:53Speaker 01

  So the container is what we call also the maxer, right? When I said demaxing, it means decontonizing, right? So actually, if you look, max, multiplexer and demax, duplexer, right? Max and demax are those. And same, a codec is actually coder, decoder, right? And so containers are this collection of multiple tracks, right? So it's what normal people call the file format, but it's a bit more subtle than that. But the most known one, of course, is MP4. But when I started, it was AVI, right? AVI was the, the video format from, from Microsoft. And Move MOV, which became MP4, was a format from Apple.

• 25:37Speaker 01

  In the open source community, one of the person that is still active on video line is called Steve Loam and started this Matroska format, which is like a bit more complex and more feature proof. And there are so many others.

• 25:51Lex Fridman?

  So, I mean, it's a pretty common thing and maybe it'll even happen in this conversation that people confuse container and the codec, right? So confuse MP4 and H.264, for example. Is that a horrible violation?

• 26:04Speaker 01

  No, it's not, because technically, the name of H.264 is MP4 part 10. Because MP4 is actually a meta specification, which has several things in it, right? There is the part two, so there is like audio codecs, right? AAC, the factor with MP4 audio, something. There is actually several video codecs, right? Inside the MP4 specification. One of them is MP4 part 10, called also AVC, called also H.264, right? So it's completely the fault of the industry to make things difficult to understand. So that's very difficult, so that people then don't understand

• 26:45Speaker 01

  why sometimes you talk about MP4 part 10, where you mean H.264, and why it's not MP4.

• 26:51Lex Fridman?

  So you can technically shove in all kinds of different codecs inside containers and horrible so.

• 26:57Speaker 02

  But broadly speaking, though, MP4 is understood to generally be H.264 plus AAC audio. Ninety-nine percent of the time, that's that and that. The rest are de minimis, the small effects, you know, edge effects really compared to that. So it's not the end of the world, that there are people who do get annoyed by that. But also in reality, something like VLC, just to point out, the file may say dot MP4, but it may be something completely different. And that's one of the challenges both FFmpeg and VLC have is. The real world is a completely different place to a three-letter file format.

• 27:30Speaker 01

  And this is very important to say, right? Like, for example, in VLC and FAVIP, we discard the file format, right? We we look into the file to understand what's in it. Because so many people, like, they say, oh, it's a video, which must be MP4, but technically it's an MOV or maybe it's a MKV, right? So we analyze in real time everything that we have and we don't trust the the the format.

• 27:55Lex Fridman?

  So what information does the fact that it's dot MP4 give you?

• 27:58Speaker 01

  It helps, right? It gives you a hint, right? Just like, oh, it it's finished by dot MP4. I will start first by opening, probing it with the MP4 container demixer to see, well, it should be that. But I don't trust it. And if I'm lost, I say, OK, maybe I'm going to try. So it bumps the priority of the module.

• 28:19Lex Fridman?

  So how do you get to, just to take a bit of a tangent there? You know, the dumb thing is if you try the MP4, but it turns out it's a different codec than you would have expected. Uh, most players just break there.

• 28:34Speaker 01

  Yes.

• 28:35Speaker 02

  Yes.

• 28:35Lex Fridman?

  So how do you not break? There's just a philosophically, I'm sure there's a bunch of stumbling blocks along the way where you it's easy to just break and stop, freak out, that's it. How does VLC not?

• 28:47Speaker 01

  This is why VLC is popular. Um, but the reason is because actually VLC was is just a client of a streaming solution called Videoland from from from a very long time ago, from the late 90s. And when you're playing video, which are on UDP, right, in network, they might be damaged, right? So you don't trust your inputs. And this is very important into the security is that you don't trust your inputs. So everything in VLC is prepared to um work with broken files. And it's a philosophical idea from the beginning and everything is engineered into that. And and it's a culture, right?

• 29:26Speaker 01

  And so, for example, and VLC became very popular on that because a long time ago, when people were pirating content, um, which they do a lot less today.

• 29:36Lex Fridman?

  And none of us ever have.

• 29:37Speaker 01

  No, of course not. Um, the metadata to play some files like AVI is at at the end of the file, right? And when you're downloading, you don't have that, right? So VLC was just like, hey, this file is broken, but I'm still going to try to interpret it. And this was very useful.

• 29:54Lex Fridman?

  We hinted at the awesomeness of the various different stages. We hinted at the awesomeness of codecs, the depth and the richness and the complexity of everything involved there. What, let's try to define, what is a video codec? What, what's involved there? What, what does it mean to compress something? You already started to hint at it, but can, can we elaborate a little bit more?

• 30:14Speaker 02

  So there's a huge amount of redundancy in any video, both spatial and temporal. And the point of any video codec is to remove this redundant data, use mathematical properties as part of this reduction process. So more often than not, using several orders of magnitude more compute to compress, because that's more costly versus, both costly, both financially and in CPU resources, versus the decompression. So it's asymmetric in that respect. Often the case because compression is done once, but there could be lots of viewers of another file. So to take that information and compress it by a hundred X, two hundred X, removing redundant information and using mathematical properties to make that small, but also have properties such as error resilience. So as, as JB suggested,

• 31:00Speaker 02

  VLC in the beginning was, was used to play UDP network feeds and UDP network feeds lose packets. And so some of the design goals of a codec is also to be recoverable. You, you need to actually be able to join the stream. It's not necessarily a file. You need to join, get on the decoding process and start decoding.

• 31:19Speaker 01

  And, and to give a more image to, to, to, to people who are not familiar, right? Like when you're going to see any type of movie, right? You're going to see the camera is going to pan, right? And, and, and travel and you realize that, for example, all the background is the same from, for like a minute, right? Or thirty seconds, right? So you can reuse the cloud that you see on the background. You can reuse that from a frame to another, right? And so it's gets, the more, the more memory you have, the more power, the more comparisons you can make, right? And so the more compressed you can be. And most of the modern codecs are basically doing that.

• 31:57Lex Fridman?

  So just to make it even more explicit. So what is video? Video is a bunch of pixels off an RGB.

• 32:07Lex Fridman?

  You have three values and you have a grid of pixels and you have, let's say, twenty four, thirty or sixty of frames a second. And you just have all these pixels repeating and showing different stuff thirty times a second. And so the question, the philosophical, the technical question is, how can I compress all of that? Store all of that at one hundred X?

• 32:33Speaker 01

  One thousand X, right?

• 32:34Lex Fridman?

  One thousand X.

• 32:35Speaker 01

  The target is one thousand X, right?

• 32:37Lex Fridman?

  And the goal is, when you say redundancy, what is redundant meaning? Stuff at best that humans wouldn't notice if it was missing.

• 32:48Speaker 01

  So, for example, you have a picture of a cloud, right? And from the next frame, there's still going to be the same cloud. So it's redundant. You could just put it once and not do it, right? Or you have a, a, a black background behind me, for example. The black is the same on the whole picture, right? So you can say, well, you know, in this picture, take the pixels that you have on the top left and the one on the top right. I'm not going to give the value. I'm just going to tell you it's the same at the top left. And then you can say for frame one, um, reuse something from the previous frame or the previous, previous frame and so on and so on, right? So you could, basically, it's unlimited, but then it's limited in terms of memory or in terms of compute power.

• 33:28Speaker 01

  Because, for example, if you need to compare pixels on two hundred frames in the past on four K resolutions, it's a huge amount of compute.

• 33:39Lex Fridman?

  And then when you're showing it, you have to do the decompress of all of that. So is it the codec, the, has the encoding and the decoding is a, is a coupled process that you're developing?

• 33:50Speaker 01

  Exactly, right? And those are two different, um, uh, trade offs, right? Are you going to compress more? But then it might be more difficult to, to, to decode. Are you going to comp, to make it a codec that is more complex to encode and easier to decode? Are you going to make a codec that is easier to encode because you need to be fast? But then the, the client side, the player is going to spend more time. That's why you have so many different type of codecs is that it's not always easy. And to make it even more complex, modern codecs like AV1, AV2, or VVC are actually not codecs. They are a collection of tools, right? There are multiple tools, multiple codecs in the same codec to, depending on the image, get the more compression.

• 34:34Speaker 02

  So just to elaborate, codecs like AV1, VVC have a much wide, have a wide audience. It could be a screen share content, it could be video, it could be animation. All of these require different coding tools. So what happens these days is a collection of tools are put in and called AV1 and called AV2, called VVC to allow for different use cases. So you may be on Zoom and sharing your PowerPoint and then you need to show the audience a video that codec needs to start changing its toolset depending on the content to compress in a different way.

• 35:12Lex Fridman?

  And like you said, there's a bunch of incredible engineers behind each part of that, each part of the tools that make up AV1, for example.

• 35:18Speaker 02

  Sure.

• 35:19Lex Fridman?

  Uh, so we've kind of danced around it. We talked about VLC, the logo, the hat. Uh, let's talk about FFmpeg. What, what is FFmpeg exactly?

• 35:30Speaker 01

  FFmpeg is basically the low-level libraries for codec to compressions and decompressions, maxes and demaxes and filters. It's, the core is this, and then you have several tools which allow you to create a type of pipeline to process any type of video files. Uh, and it's used as a library absolutely inside everything from VLC to Chrome to your smart TVs to basically any video that you see online, um, you usually use FFmpeg. And FFmpeg in it has all those type of tools and sometimes depend on other libraries like X264, uh, libvpx and others, right?

• 36:13Speaker 01

  So it's really now the de facto, um, tool to process images.

• 36:19Speaker 02

  From a philosophical level, I think it's incredible that your home videos, your, your grandmother's home videos and trillion-dollar corporations effectively are on a level playing field using the same technology stack.

• 36:33Speaker 01

  It's it wouldn't be a surprise, you know, these big companies just have three thousand line FFmpeg commands. There are some that use the API, but there are some that just have long command lines.

• 36:44Lex Fridman?

  So, yeah, there's a bunch of tools, like literally command line tool FFmpeg, of course, FF probe, there's libraries, libv codec, libv format, libv filter, but the FFmpeg on the command line is like legendary. Because you can cut, there's so many parameters, you can customize everything to help.

• 37:06Speaker 01

  It's a language, it's an actual language.

• 37:07Lex Fridman?

  It's an actual, yeah, you could think of it as a programming language.

• 37:10Speaker 01

  Yeah, of course, I'm sure, because so most of the people, they're going to take FFmpeg, file in, file out, and specify the format, right? But you can, we've seen thousands of characters and we've seen also like people like doing programming, generation of command lines to to to make FFmpeg. There is a ton of people who are using AI to generate common lines for FFmpeg, because you have no idea what it is, but you can specify so many filters, right, on command line, right? So FFmpeg is this collection of toolbox for multimedia processing that everyone, everyone uses. And everyone that is watching your videos are also using, right? You you're on YouTube, well, it's FFmpeg on on the client side.

• 37:54Speaker 01

  Well, your server side, on the server side, the client side is probably Chrome. Well, you're using FFmpeg also. And and you're using OBS to record. Well, it's FFmpeg, right? You're using a ton of important, like big box, professional boxes. Well, it's very possible that inside some part of FFmpeg is running.

• 38:10Lex Fridman?

  I mean, there's like so many, just to give people an idea, like I I use FFmpeg a lot on on everything, just trivial stuff like, take a video, add an intro video and an outro video and fade one into the other, like, what is it called, dip to black, like where it dips and then shows the next video and does the same thing with audio. There's like a cross dissolve of the audio, it's a quiet, it quiets the audio and makes it loud again. And then there's a bunch of stuff like showing the captions on screen hard, like baking the captions in, you can customize the font. You can do all kinds of layering of audio and video.

• 38:53Lex Fridman?

  There's a million things and of course, all of that works like magically with basically any codec. Like anything you could shove in on the audio and the video side, it works.

• 39:05Speaker 01

  But it's like, if you, if you look at, for example, you can do things that you would do with Adobe After Effects in Common Line on FFmpeg, right? It's, and it's very interesting because for example, for, for, for images, there is not such tool. There is a few tools, but not with the breadth of FFmpeg.

• 39:23Lex Fridman?

  So Image Magic has a similar kind of.

• 39:26Speaker 01

  Yes, but you will not do some filters, complex filters. You don't have the equivalent of Photoshop in Common Line, right? But you, for video, you have FFmpeg in Common Line.

• 39:35Lex Fridman?

  Yeah, it's incredible. I mean, it's like, and it's an example of a thing when a bunch of great people get together and they get a vision and they stick by that vision for many years, which is incredible.

• 39:46Speaker 01

  And the vision behind, and the same for VLC and FFmpeg is that we make every, everything that is very complex easy to use for the normal people, for everyone, right? Our goal is to make something that is insanely complex technically and make it easy to use, right? And, and people, you use VLC, they drop a file, they don't realize how complex the file is, but they, they play it. Or, or people put any type of thing inside FFmpeg with complex filters and it just works like magically, right? And people, and this is our mission, right? Make very complex things.

• 40:21Speaker 02

  We wouldn't be here and you wouldn't be here if this required, you know, a traditional television studio setup. It's tools like FFmpeg that democratized this, the, the podcast and streaming revolution, the YouTube revolution was caused, you know, FFmpeg was a big player in that because it, it democratized this technology that was once in the, in the nineties, for example, you needed equipment that cost hundreds of thousands of dollars to do compression. It was the size of a car. And now everybody has that at almost an exact level playing field. And that's something that's so remarkable.

• 40:55Lex Fridman?

  It gave voice to a lot of people. And we, just to clarify, we say you, you wouldn't be here, not the human, but the podcast.

• 41:02Speaker 02

  The podcast, sorry, you, you as the, sorry.

• 41:04Lex Fridman?

  VLC did not have anything to do on a biological level, creating me as a human.

• 41:09Speaker 01

  You realize also, everything move from text to images and images to video. But look, look at social networks. Video is everywhere. It's the most powerful medium there is, right? And, and when you see shorts and, and reels and TikTok, right? It's amazingly powerful to, to, to give. Video is amazing for that, right? But the complexity is important.

• 41:33Lex Fridman?

  This is what people don't realize. I mean, this is really, it gave power to the individual all across the world. It's real freedom. And I think, I can't believe it, but we still haven't mentioned the actual obvious thing for people who are not familiar, which it's open source. And there's a open source community of users and developers behind it. So it's really, it's a movement. So like, we'll talk a bunch in a bunch of different ways about the community behind it. But can you speak to the open source element? So when we say, what is FFmpeg? It's an open source project.

• 42:10Speaker 01

  Yeah, so FFmpeg, VLC, X264, Video Land, everything we do is fully open source. And for the people who don't understand how open source is, my usual analogy is about a chocolate cheesecake, right? Usually for you, when you want to buy your cheesecake, you go to a bakery, they give you the cheesecake. The other one, way of having a cheesecake is have your grandma give you a recipe of how to make that. When we do open source, we give you the chocolate cake and we give you the recipe to actually remake the same cake. But at the same times, tell you how to build the oven and also how you're allowed to modify the recipe. And we sell it to someone else.

• 42:48Lex Fridman?

  Yeah.

• 42:48Speaker 01

  And this is, because software is just a very long recipe of small instruction. Computers are not very clever. They go very, very fast. So a normal program has tens of billions of instruction instead of the tens when you have your chocolate recipe. So. A lot of the software industry was about selling software like where you just have like the final cheesecake. In open source, we make, we give you everything. And that managed to get a lot of people work together, right? Because then you decide that you're going to make the best program, the best recipe for video. And you create communities in FFmpeg since the beginning of FFmpeg. Probably two thousand to three thousand people have contributed from the beginning, right?

• 43:32Speaker 01

  And then it's exactly like the Linux kernel, right? The Linux kernel has probably ten thousand people contributing everywhere. And they get together, mostly online, right? So they virtually get together to create the best tool for something. And on FFmpeg and VLC is just like, well, this codec doesn't work. So I'm going to work on the codec and I'm going to add the support for this file inside FFmpeg. So it will be beneficial to everyone. Because again, we work for the greater good. We work for everyone. And that is what open source is.

• 44:07Lex Fridman?

  And we should mention, depending on the licensing, you could probably build a billion dollar, maybe even a trillion dollar company around this as a rapper.

• 44:20Speaker 01

  Well, yes, people do. People do, right? There was a lot of problems with mostly a cloud providers who are basically running some open source tools in the cloud and just give you the API to access to that. And there was a lot of databases like Mongo or Elastic who changed their license in order to avoid those type of scenarios.

• 44:44Speaker 02

  This is a question we get a lot in FFmpeg is why don't you do that? And you can't. We have thousands of contributors, some of whom aren't even alive anymore. You would need all of their agreement to do that. And JB will go maybe a bit later and talk about how challenging that process was in VLC to do the relicensing.

• 45:02Speaker 01

  The license is a social contract in terms of who saw the factor. Of the community. The community does not agree on much beside the license. People go around, discuss around because of the license. And that also allow those license fork, right? Sometimes the community splits, but it's possible because of the license then to merge back. And we've seen that so many times, right? GCC and EGCC in the past. We have seen, for example, all the web browsers, right? They started as web, like K H M L, which becomes WebKit and then which becomes Blink, right? So, open source license is like the core of the community.

• 45:43Speaker 01

  And people are coming from all over the world, very different type of religion, political borders. They work in the same way on a project to solve a specific problem. And the specific problem we're working on is to make multimedia easy for everyone.

• 46:03Lex Fridman?

  Looking it up on perplexity here, looking at the different open source licenses. Most major open source licenses fall into two buckets. Permissive, very few conditions, and copy left. Share alike requirements for derivatives. Below is a brief, practical summary of the main ones you'll see in the wild. MIT license, BSD, ISC, Apache, GNU GPL, GNU A GPL. Where's LGPL? Yeah, LGPL. Let's see, there's the Mozilla Public License, there's Eclipse Public License. It goes on. There's a lot of variety. I mean, I think the really popular ones is MIT, GPL, LGPL.

• 46:47Speaker 01

  And BSD.

• 46:48Lex Fridman?

  And BSD, Apache, sometimes you'll see. Unlicensed, that's an option. Attempts to dedicate code to the public domain with a fallback permissive license.

• 46:56Speaker 01

  There are many licenses for many different thing. What people don't understand that public domain is something that doesn't exist worldwide, right? So it's, all the open source licensing use the copyright law, right? The international copyright law in order to give rights on how you use the software or how you modify. It's de facto a copyright license contract that you give to the end user or to the developer. And so you have like the first one, which are basically very permissive MIT BSD. You give the code and basically you do whatever you want, right? You take it, you want, you modify, you do what you want. And this is popular for JavaScript and the type of BSD operating system.

• 47:42Lex Fridman?

  So some of them, one of the parameters is whether they require attribution, meaning if you use the code you have to say.

• 47:48Speaker 01

  Yes, so in those type of permissive license, some you need to say if you use it, which is called attribution, and some you don't. And then there is the other part of license which are copy left, where you need to give back to the community your modifications. And with different string attachment, some weak copy left license, like the Mozilla public license, to some which are a bit stronger, like GPL, or even very strong like AGPL. So all of those are different type of licensing that depends on what your goal are and how you want to structure your community, which is why I spoke about social contract, because this is very important to understand.

• 48:30Speaker 01

  FFmpeg and VLC are mostly GPL or LGPL. The Linux kernel is GPL, but Android is Apache. A ton of JavaScript framework that are using are mostly MIT. All the BSD kernels, OpenBSD, NetBSD, of course BSD. And so it's philosophical change on how you want people to contribute back, basically.

• 48:57Lex Fridman?

  So there's, I think you talked about that you've moved at one point from GPL to LGPL on certain parts of the project. What can you describe the difference between the two and what does it take to move to, I guess, a more permissive? So that direction is more permissive, LGPL is more permissive than GPL.

• 49:16Speaker 01

  Yeah, so you have to realize that you can always go from more permissive to less permissive, right? Because of course, those license are basically statement. So if you restrict, you can always restrict more, right? So in a GPL project, you can take MIT code, but you cannot do the opposite, right? Because they are more constrained to match. Indeed, in fact, I changed the. Core of libVLC, which is the engine of VLC.

• 49:46Lex Fridman?

  Mm hmm.

• 49:46Speaker 01

  From um GPL to LGPL. Uh and there were two reasons uh to do that. The first one is that so people can use um the VLC engine, libVLC, into third-party applications. So a lot of applications which are playing video on your phone or on your tablet are actually VLC engine in it, which is calling FFmpeg in it.

• 50:07Lex Fridman?

  Yeah.

• 50:08Speaker 01

  Um so that was one of the way to create uh one of the company I created, which is doing consulting and integration of those application where you integrate VLC into third-party solutions. Um like inside game engines or stuff like that. With GPL, you couldn't do that because that means you needed to open source everything. And those are for a lot of like commercial companies who don't want that.

• 50:30Lex Fridman?

  So you can create a company with LGPL, you can create a company around it. You can do a commercial thing, you don't have to open sources. So that's a big, big leap.

• 50:38Speaker 02

  So you could play video in your game.

• 50:40Speaker 01

  Yes.

• 50:40Speaker 02

  The problem, the problem is I'm a game developer and I want to play some videos.

• 50:44Speaker 01

  Mm hmm.

• 50:45Speaker 02

  And I don't want to be forced to open source the entire game just to play those videos. So that's where the consulting business, the libVLC LGPL.

• 50:52Speaker 01

  Mm hmm.

• 50:53Speaker 02

  Allows you to do that. The LGPL, the library GPL, as it used to be known, allows you to do that.

• 50:57Speaker 01

  And FFmpeg is the same. It force, LGPL forces you to give back what you change on this component, this library, which is why it's library GPL. And so you can use FFmpeg as LGPL into like any type of application, even non-open source. But you need to give back the modification you did on FFmpeg, same on libVLC.

• 51:19Lex Fridman?

  Is it limiting from an open source perspective to go GPL? Because if you, if your library, if your code is GPL, it means you're not, you, you, you're basically discouraging companies.

• 51:33Lex Fridman?

  From building a business around it, right? Is that fair to say?

• 51:36Speaker 01

  It depends on, it depends on the company. But the company whose business model requires the source, the application to be closed source. Yes, it's limited. So that's why, for example, I moved to LGPL. The second reason is a bit more obscure is that the terms of condition of the app store, the Apple App Store for iOS makes it very complex to have GPL application on it, while it's easier to have LGPL applications on it. So VLC on Windows and on Mac and on Linux is GPL. The core is LGPL. But on iOS, the iPhone version and the Apple TV version is a type of different license called the MPL. And yes, I went and changed the license and it was a long story.

• 52:21Lex Fridman?

  Yeah, so I think basically to change the license, you have to contact all the contributors.

• 52:25Speaker 01

  Yes, it's very important to understand that open source projects are what we call in the US copyright law, joint work or in civil law, collective works or collaborative works. Is that you work all together in terms of the same goal and then it creates one software, which is one release. But the copyright is kept by all the individuals. Some open source project don't do that, the forced copyright assignment. But this is not what we do with communities. So everyone has basically copyright on what they changed. And this copyright stays even if at the end your, your contribution was deleted because the new contribution was based on your previous one, right?

• 53:07Speaker 01

  So if you want to properly re-license, you need to find all the contributors. And at that time, I had to contact more than 350 people. And sometimes, well, they are just an email, right? So it's, you need to actually track down. I actually like traveled to someplace to go somewhere that I was like, sorry, that I had found online to see a To go to their job and say, well, you licensed that. Can you, do you want to change from GPL to LGB? Most of the times, they don't even care. They wanted to help you. But also, it brought me to very complex situation. I arrived to the work of a person who was a factory worker. And I said, well, I need to you to sign that. Because it was his son who died, who actually wrote the code, right?

• 53:52Speaker 01

  So I had to explain all those type of open source meaning. And no, I was not a company trying to rip out the two lines or five lines that that guy did. But was useful and the whole community agreed on that. And he had no idea I was a factory worker. This and I was a lot younger, right? Like it was fourteen years ago and like, like I was almost in tears, right? It's we very difficult, right? We are talking about life of people and he explaining. And we went to talk about the photo of this guy, right? So it's important to do it right and to do it correctly. But yes, that means tracking down everything because every contribution works. There are some projects who don't respect that and we do relicensing a bit like aggressively.

• 54:36Speaker 01

  But as I said, it destroyed the whole heart of the community because it's. We only agree on this, on this license, so that's important.

• 54:44Speaker 02

  I would emphasize the community is such a wide ranging group of people. There's people in a Syrian war zone with electricity, part time. There's, there's all people from all walks of life, rich, poor, young, old.

• 55:00Speaker 02

  So it's quite remarkable to get, you know, a group of people aligned on something. And that's an achievement in itself.

• 55:08Lex Fridman?

  Yeah, it's incredible. And a lot of them are introverts. So you coming, coming to find them and getting them and getting them to answer an email. It might be quite, quite difficult.

• 55:18Speaker 01

  Most of us are introverts, right? You, you need to be more precise. You are extremely introverts, extremely, extremely introverts and introverts, right? It's just like a whole spectrum of different people. It doesn't matter. The important is, is your code good? Is your code great? Is your technology great? We care about excellent code. We don't care who you are. Sorry, it's just like, we have no idea to check. We cannot check, right? Like maybe you're a dog. I don't care, right? I don't care where you come from. I need to look at your code. And this is important because people don't understand that. And they come to the community and send them some patches and they get rejected. And they don't like that because. I mean, you're just like, sorry, it's not up to our standards.

• 56:00Speaker 01

  Oh, yeah, but I'm engineer at this very large company in Italy, in Germany, in the US. We don't care. We care about the quality of your code, because this is what defines our community. And which means that we have a lot of people who contribute who are some very different backgrounds and very introvert, sure, but that's okay, right?

• 56:20Lex Fridman?

  So one of the legends of the community is, of course, Linus Torvalds, who created Linux and is a longtime maintainer of the Linux kernel. As the legend goes, he can be pretty harsh on this meritocratic process of reviewing the code and saying it's not good enough. Can you just speak to the legend of Linus Torvalds?

• 56:42Speaker 01

  Linus is one of a kind, right? And I would even go and say that what he did on Git is more interesting than what he did on the Linux kernel. He's very harsh, but what people don't see is usually when he's harsh to it's people who are maintainer of part of the kernel, right? So they know him, right? So he's not very harsh like that to everyone. The thing is, what he created in his room is basically powering every server online, right? Even at Microsoft Cloud called Azure, I'm quite sure seventy, eighty percent of the servers are running Linux. All your Android phones are running Linux. What he did with the power of a pull source, sure, is amazing.

• 57:26Speaker 01

  And yes, the quality of the Linux kernel is very high, and yes, it's difficult, but we cannot compromise on that. We cannot compromise on quality. Because in the end, and you have to understand that, is the core community of VLC is five people. The core community of FFmpeg is ten to fifteen. And we are the ones who are going to maintain your code, right? Because one thousand contributors in the timeline and just ten states, one percent chance that someone comes and stays, one percent. So you will have change of job, change of wife, you have children, you have accidents in life, you're going to change jobs, whatever. You're not going to come back. It's most likely.

• 58:10Speaker 01

  So we are the one going to maintain your code. It needs to be maintainable. It needs to be excellent.

• 58:18Speaker 01

  And yes, sometimes that means that you need to rework your work because it was good, but it's not excellent. And we need excellence because we have very few to maintain something that is critical for the whole.

• 58:28Lex Fridman?

  But we should also mention that there's some spiciness, some harshness to the language that's sometimes used when you're keeping this high bar of excellence. Is there something to say to that?

• 58:40Speaker 01

  It's true, right? It's also the fact that, for example, what we're doing is low level, it's extremely technical, you get into this community, the tone gets very like a type of, it's a subculture, right? So people who arrive from the external are basically not known to the subculture. Most of those people around FFmpeg and VLC, we do Videoland Dev Days, VDD every year. They are so fun in real life and they love it. But it's true that you're online and sometimes like the tone, you don't realize how it is. But that's okay.

• 59:14Lex Fridman?

  It's a culture. I mean, you get this in the gaming culture, there's pretty harsh and tense the way people communicate. And it's, everyone understands that the way you show love and respect just looks different in different communities. Sometimes people, it depends. If it's a book club, usually people are going to be much sweeter. If it's an open source project that's very high stakes and used by millions of people.

• 59:37Speaker 01

  But it's very not often insults that you see, for example, in the gaming, right? So Linux tone is a bit unusual even for the open source community. It's more like, it's more harsh on the result, saying, no, this is not good, this is crap. Those type of things that you will see.

• 59:53Lex Fridman?

  Try not to make it about the person, make it about the code.

• 59:55Speaker 01

  Yes.

• 59:56Speaker 02

  It's very matter of fact. And I think you've got to look at it in terms of, you know, the famous, FFmpeg is developed almost entirely by volunteers and that's true. And you've got to imagine someone's done a hard day's work at their day job, they come home.

• 1:00:09Speaker 02

  You know, terse-ness might be a thing, you know, and that's not something to take personally. You're tired, you're busy, but you still care about this open source stuff. But you may not be able to explain and handhold someone on every subtle detail.

• 1:00:23Speaker 01

  And also, you have to realize that most people don't speak English as a native language. And this is especially for for open source projects like FFmpeg and VLC, which are mostly centered out of Europe. Sometimes, like, people who are from the US or just like are very not happy about the tone. But most of the time, it's also like, they don't know better, right? It's difficult. The language is, English is a difficult language. There are so many subtleties and tones and so on that you don't have, right? So often, it's also difficult in those type of community about like different cultures and languages.

• 1:00:59Lex Fridman?

  So as the legend goes, JB, you repeatedly turned down millions of dollars to keep VLC open source free for everyone, without ads. So take me through the reasoning behind that decision of leaving millions of dollars on the table.

• 1:01:19Speaker 01

  Yeah, that's like almost a meme, right, on Reddit?

• 1:01:22Lex Fridman?

  There literally is a meme on Reddit.

• 1:01:24Speaker 01

  Nine Gag and, yeah, yeah, I see there's you.

• 1:01:29Lex Fridman?

  Looking like a wizard in the VLC hat on Reddit. This is JB, the creator of VLC Media Player. He refused tens of millions of dollars in order to keep VLC ads free. Thanks, Jean Baptiste Kempf. You can even summon him on Reddit.

• 1:01:46Speaker 01

  Yeah, and usually, if you see, right, it's usually like people tag me, right? And then there is me, and then like I say, good morning, I got 24K upvotes, which is great, right? My karma on Reddit is amazing, at least on that account.

• 1:02:00Speaker 01

  So the question is, needs to be answered first, what is a story about VLC, right? Because. Yes, this is true. I refuse dozens of millions of dollars. Yes, several times. Yes, I could be a multimillionaire and be somewhere on the beach. But I did not do it because I thought it was not moral and it was not the right thing to do. And this is very important for myself is to be like, I work for the greater good. I work for people and I don't want, it's not just by myself. But the reason is also because I, I did not feel that I'm completely legitimate to do that. And let me explain you why. V L T story is a very weird story.

• 1:02:42Speaker 01

  In France, we have university and we have a type of top colleges. And those top of excellency schools are engineering school, business schools, and basically lawyers and medical, right? But they're outside of university. And in order to enter those, you spend two years working like crazy mass physics to enter those best engineering school. One of the school is called the Ecole Centrale Paris. It has changed name since, but it was called the Ecole Centrale Paris. And because it was central, they had to move it because it was too small after the World War Two. And, and they moved it, they wanted to move it to the central of France in a place called Clermont-Ferrand. And the alumni decided that this was not okay, right?

• 1:03:25Speaker 01

  It is the school that Eiffel, right, the one who did the Eiffel Tower attended to, right? So they said, no, no, we are amazing grade school. We cannot do that. And so they bought a piece of land south of Paris, very near Paris. And it was a campus managed by a nonprofit of the alumnis, okay? Because of that, everything on the campus was managed by student. The university did nothing, right? So radio, TV, supermarket, library, defining who was going into which rooms. Everything was managed by the student. That's amazing. That's an amazing experiment. That it all didn't go to hell quickly. It somehow flourished.

• 1:04:08Speaker 01

  It worked great and I learned so much in my life doing those side activities, right? Because you're twenty two and you need to run your campus, else you don't have electricity, right? So you care about that, right? But anyway, in the eighties, they did a full experiment of deploying a network, mostly sponsored by IBM and 3com, which was a token ring network. So token ring is something that probably almost no one knows about anymore. It's a networking technology where you don't have routers, right? Everyone is linked, it's like really a ring. And when you want to send a message, you talk to your neighbor, who's going to put the message to the next one, who's going to put the things to the next one. In terms of ring.

• 1:04:51Speaker 01

  The issue with token ring is of course is that it's very slow, because every computer on the network needs to open the message, see if it's okay. Is it for me? No, it's not. And then send it back like a token, which is traveling around the ring.

• 1:05:06Speaker 01

  In the eighties, you're doing some telnet and sending mails as university. That's okay, right? But starts the nineties. In the nineties and start video games. And when you have high latency in video games, basically you die, right? So in nineteen ninety four, nineteen ninety five, around Doom and Duke Nukem coming around, they want a faster network. So the students go and see the university and say, you know what? We want a faster network. We need to work, which should also play video games. And the university tells them that basically, oh, I'm sorry, we cannot help you. Because you understand, the campus is not ours. You manage it, so do something. And you should see some basically partners of the university and basically go away.

• 1:05:50Speaker 01

  And they go and they actually go and see the CIO of Bouygues, which is a large French company and who's doing some TVs in France. And he says, well, you know what? The future of video is satellite. Well, today we know it's not, but at least it was a good idea in nineteen ninety five, the first of satellite dish. And he says that instead of having like one satellite dish and a big decoder for each of the students, which are one thousand five hundred, what about you build like you put a? Enormous dish and only one decoder and you send the video directly on the network. And that required a very fast network. Today, it's obvious, but at the time was like the first to do video streaming.

• 1:06:34Speaker 01

  So they build this project, which was called Network 2000. And of course, we are in the nineties, right? Everything is futuristic is called 2000.

• 1:06:43Lex Fridman?

  Yeah, 2000, yeah.

• 1:06:45Speaker 01

  And so they do the Network 2000 project. It's completely hacked. It crashes after forty five seconds. That's okay. Demo is forty seconds. It leaks memory. That's okay. They put sixty four megabyte of RAM instead of the eight or sixteen you have. And the demo should have stopped there. And that was the Network 2000 project by the students.

• 1:07:02Lex Fridman?

  What was the format of the video that they had to work with?

• 1:07:04Speaker 01

  MPAC two, because satellite is MPAC two TS for transport, MPAC two video and MPAC two audio at that time. And the project should have stopped there. Everyone was happy. They had like amazing ATM network at one hundred and fifty five, fifty five megabits per seconds. They had probably one of the best network in Europe at that time. And they stopped the project. Six months or a year later, two students arrive and say, well, you know what? Maybe other people care about video streamed on a local network and they create the VideoLAN, Project VideoLAN. And one of them is called Christophe Mathieu, that is a good friend of both Kiran and me. And they start the project. It's not even open source yet.

• 1:07:45Speaker 01

  And they spend around three years to get the school to agree to make it open source. Because the university wanted to get, because of the IP and copyright of the students, wanted to basically monetize these MPAC two decoders.

• 1:08:00Lex Fridman?

  Just to be clear, so what's the main application streaming on a local network?

• 1:08:03Speaker 01

  It was streaming on a local network.

• 1:08:05Lex Fridman?

  By the way, that's just like to state the obvious. This is before YouTube. This is before.

• 1:08:09Speaker 01

  Ten years before YouTube. You have a Pentium sixty or seventy five, right? You, you, you, the main machine was forty eight, eighty six DX at thirty three megahertz, right?

• 1:08:17Speaker 02

  Bear in mind, television was the main form of video at the time. You could get new channels in the nineties, having even one new channel when you, you grew up with four channels, having a fifth or a sixth was a big deal. And so having this satellite service with, you know. Even hundreds of channels was so groundbreaking.

• 1:08:34Speaker 01

  Especially because this is university where you had ton of different nationalities, right? So there was a ton of people who wanted, so in the end, they had like several dishes on different type of satellite, right? Because for example, a lot of people were coming from the Maghreb or the Middle East and they, so they went to different type of, uh, of satellites. Anyway, the solution were great and they started the Videoland project. The Videoland project has several and some are completely crazy solutions, like one how to create multicast on a unicast network. But let's not come to that, it's too, too complex. But Videoland client part is what became VLC.

• 1:09:15Speaker 01

  Actually, they basically strong-armed the university to force it to open source because university did not understand that. And in two thousand and one, it's still early, but basically, yes, the university agreed early two thousand and one to make it open source. I joined the project in two thousand and three because that's when I joined the university. But the first thing is I'm not the one who created VLC because actually no one did, right?

• 1:09:39Lex Fridman?

  Just kind of naturally emerged from the Videoland project. And we should mention that, like, again, you, you, you said it, just to make it clear. Videoland is what it became was at the time is a set of technologies around video. The VLC, what you called the client, that's the thing that most normies.

• 1:10:00Speaker 01

  That is correct.

• 1:10:01Lex Fridman?

  Think of like as the thing, which is like the thing that pops up when you click on video and you play it.

• 1:10:06Speaker 01

  So I arrived in two thousand and three and then I will create the open source, um, nonprofit organization called Videoland. And I took everything out of the university to create it, a nonprofit project and some things sustainable. It's, yes, it's true that I spent more time than anyone on VLC and Videoland, that is sure. But it's a continuity of a previous project, Videoland, the student project, which is a continuity of the network two thousand project, which is a continuity of that and that.

• 1:10:35Lex Fridman?

  I'm sure there's moments along the way there, you were thinking of like, what is the future of this from the open source perspective? Because as, as the internet is blowing up and there is companies, I mean, for people who don't remember, like there's companies making huge amounts of money.

• 1:10:52Speaker 01

  And I can tell you that in two thousand and five, the project should have died. Um, and I made it to continue the project. At some point, we were only two active developers. And I thought it was great technology and was useful and it will be useful. And, and, and I made that my, my, my life and my, my time. And I made that grow from a few hundreds of thousands of users, millions of users to what we have now, which is probably billions of version of V L C around the world and used everywhere. So that's a bit the story of V L C. There is ton of very funny story around that.

• 1:11:33Speaker 01

  Many people from around the world working on it, like you said, in Syria or middle of nowhere in India. But along the way, I got several offers which were either to bundle toolbars, right? You remember those horrible toolbars, which were basically spyware? Or changing your web browser or your search engine? Or even like advertisement inside V L C? And I didn't like that, right? I am, and, and people don't understand that is not, I'm not against money, right? I'm very happy to make money. I created several startups and, and one, I hope that is going to work very well. It's a fact that I believe that you need to win money ethically. There is the right way of doing that.

• 1:12:18Speaker 01

  And, and doing sneaky advertisement or stealing data is not the correct way, right? For example, if Netflix arrived at some point and say, well, we want to put Netflix inside V L C, probably the story would have been different, right? But they didn't. The only people who came to us were shady ads company. And if I do that, right, I would have a ton of money, right? And then three years later, project is gone, right? Someone forks it and, and something else happens.

• 1:12:44Lex Fridman?

  It's not even necessarily ads or any of that. It's the shadiness of the dishonesty of the, so you had a good radar. You had a good threshold of like, no, this compromises the spirit of what this is supposed to represent.

• 1:12:58Speaker 01

  But also it's, it's for me, right? I'm, I'm like very selfishly, I need to go to bed at night and be happy about what I've done, right? Maybe it's my upbringing, um, uh, maybe it's my parents' fault or, or whatever, right? But I believe there is right and wrong, right? And this was the right decision at the time. It still is. I want to be proud of what I've been doing and. Like if I had sold out, I would have betrayed so many other people's work.

• 1:13:27Lex Fridman?

  Yeah, well, I should say, me and most of the internet, thank you for that decision. It's inspiring for others. I think that they're pushing the open source movement forward. That it's okay to do these kinds of huge sacrifices if you believe it's right. And I think in that case, it was right and it was the reason that VLC became as successful as it was. Because it's an embodiment, it's a symbol of like, you know, freedom and what the open source community can create.

• 1:13:59Speaker 01

  Yeah, and be a service for so many people around the world, and this is important.

• 1:14:03Speaker 02

  We should emphasize, in the 2000s, it was really normal to download a program and it secretly installs some spyware.

• 1:14:08Lex Fridman?

  Yeah.

• 1:14:09Speaker 02

  It was buried in very faint text or in the license text box that nobody reads that at the bottom. Oh, I will be installing this toolbar and changing all these things. And it was, it was very common to have to, you know, you install a program to do something at the time of any sort.

• 1:14:24Lex Fridman?

  To put yourself in the mind of a developer at that time, I think it's very easy. To everybody listening to this, it's very easy at that time to convince yourself to take a few thousand dollars. A few thousand dollars to do it. To say no to much more money takes guts and takes vision.

• 1:14:46Speaker 01

  The last offer I had was obscene. And they say, yeah, but imagine with all that money, you could build something new, open source, right? It was like, the mind trick was, it was difficult. But for me, it was just like, no, this doesn't work like that, or this is not the right thing, so I don't do it.

• 1:15:07Speaker 01

  And again, right, it's not that I don't like money or whatever, it's just like, it wasn't right.

• 1:15:14Lex Fridman?

  Well, once again, thank you from me and from the rest of the internet. Let me talk a little bit more about the open source movement, about the fact that, as you say, over and over and over and over, FFmpeg is, and many open source projects are built by volunteers. So there's a bit of drama recently, Karen, on the interwebs, on Twitter. You have a spicy style on Twitter that I think articulates and celebrates all the incredible developers and development and the code, especially assembly that's involved in building some of these codecs and building some of this incredible technology. But that brings us to the bit of a debacle that happened.

• 1:15:58Lex Fridman?

  Tell me the full saga of what happened with the Google security engineers.

• 1:16:03Speaker 02

  Just to be clear, Google are one of the biggest supporters of open source out there. They have been for a long time. It's just I think some things kind of went a bit overboard this time. So FFmpeg itself, and this is not like a secret, it's on the homepage, you know, the, it processes untrusted data. There can be security issues when you parse untrusted data, that's very normal. But recently what changed was Google started using AI to create security reports on an open source project, FFmpeg. Volunteers had to deal with that. They did, they provided very limited funding and they even went to the media first announcing how good their AI was before the issues could be fixed.

• 1:16:42Lex Fridman?

  And this is in the public forum?

• 1:16:44Speaker 02

  Yeah.

• 1:16:44Lex Fridman?

  So reporting an issue, using AI to find an issue in the code, which is a security vulnerability and then reporting that publicly before you're able to fix it.

• 1:16:53Speaker 02

  Yeah, it's announcing how good their AI is, that they provided a standard 90 day industry deadline

• 1:17:01Speaker 02

  without really understanding the nature of volunteer driven development. In addition, this vulnerability was on an obscure 1990s game codec.

• 1:17:13Speaker 02

  The way, and let's look at it from their standpoint to begin with, let's, you know.

• 1:17:17Lex Fridman?

  Yeah, can you steal me on their case?

• 1:17:19Speaker 02

  Yeah, sure. They have substantial resources working on the security of open source projects that, you know, are ubiquitous and they've used, you know, a lot of compute to do that and very expensive and very capable security researchers to do that. And that's their viewpoint is they are contributing by doing that. But I think that's where. Opinions differ. Um. It it opened up a lot of interesting fishes, uh, I would say. Um. It does seem that there's a portion of the security community that um look at themselves a bit like building architects that never have to go to site. You know, going to site is something that is a little bit beneath them, the actual day to day construction.

• 1:18:02Speaker 02

  They're there to do their security things and it's someone else's problem.

• 1:18:07Speaker 02

  The security industry also kind of has a very aggressive tone towards things. The the language they use is extremely aggressive. They use very strong language like you will get popped. Um. And to to Joe Public, get popped. You know, it means something quite bad. Um. For them, it means to get hacked. The way I would look at it personally is a little bit like the padlock on your home. Um.

• 1:18:30Speaker 02

  Not everyone, that a padlock on your home, or you know, the lock on your home is there to to protect against the capabilities of of what it's there to protect. It's not there to protect nuclear secrets. It's not there to protect Fort Knox and it could be looked at that they're using A I at a level of scale to go and pick those locks and then say, hey, your locks, your locks not secure. You need to deal with this. Whereas actually, they're the ones with resources to be able to to fix this. But that seems to not be something either they'll contribute to in terms of patches or in terms of financially. And the scale of A I is kind of the issue. The

• 1:19:11Speaker 02

  the bug reports are very wordy. They're very, very, it's almost a denial of service by A I generated bug reports on very niche codecs. Um. And the other issue the security community has is everything is marked high priority. You're going to, you know, this is the most important thing in the world and you need to deal with this. High, high, high vulnerable, scary, scary, scary on a game codec used on one disc in nineteen ninety three.

• 1:19:36Lex Fridman?

  Yeah.

• 1:19:37Speaker 02

  And that that's where the dichotomy lies. Going around telling everyone that their padlock's not safe. Well, that's a hobby project of somebody. The the safety of that codec is consummate to what that person thinks. It's their hobby. It's good that they're security analyzing it, but it doesn't need a big scary warning. This is a critical vulnerability. Um. May also may recently also um see that um there was another quote unquote vulnerability. It wasn't at Google in this case, but um. A filter could overflow and have an integer overflow and one of your pixels could be the wrong color. And this was marked high 7.5 severity in red.

• 1:20:18Speaker 02

  And at some point, the security industry needs to realize you can't keep crying wolf like this because this just leads to people, you know, the equivalent thereof of putting password stickers on their PC, you know, you can't just keep crying wolf every day. And I appreciate, you know, that's their modus operandi is to create as much scared and fear, but from the Google standpoint, at the end of the day, they need to contribute either financially or with patches. Google uses FFmpeg and a scale probably you or I couldn't even contemplate. Millions of CPU calls. And yes, they contribute in areas mostly regarding their own products, so VP9, AV1. But in a wider sense,

• 1:21:04Speaker 02

  there's a disproportionate level of contribution. Yes, they fund students. Yes, they fund summer of code.

• 1:21:10Speaker 02

  And I think, so Alex Strange is a former FFmpeg developer, I think posting in a personal capacity.

• 1:21:15Lex Fridman?

  So he posted about security engineers on Hacker News. His post reads, "The problem with security reports in general is security people are rampant self-promoters." In parentheses, Linus once called them something worse. Imagine you're a humble volunteer open source developer. If a security researcher finds a bug in your code, they're going to make up a cute name for it. Start a website with a logo. Google is going to give them a million dollar bounty. They're going to go to DEFCON and get a prize and I assume some kind of secret security people orgy where everyone is dressed like they're in the matrix.

• 1:22:00Lex Fridman?

  Nobody is going to do any of this for you when you fix it.

• 1:22:07Lex Fridman?

  Basically commenting on the sort of the incentives for the different people involved and misaligned.

• 1:22:14Speaker 01

  The problem here is Is a disproportion of means on discovery compared to patching it, right? And this is the biggest issue, right? Um, and after that debacle, Google did some changes.

• 1:22:27Speaker 02

  They are now starting to send patches, which is.

• 1:22:30Speaker 01

  And they also now have reward tools for fixing issues. So it it has changed a bit because of that debacle. So it's good, right? But we've seen, and we talked about Google, but we have seen like some, some other large companies saying, oh, you need to fix this bug because it's critical in our product.

• 1:22:46Lex Fridman?

  Can you explain the XZ fiasco? The FFmpeg tweet reads. The XZ fiasco has shown how a dependence on unpaid volunteers can cause major problems. Trillion dollar corporations expect free and urgent support from volunteers. Microsoft, Microsoft Teams posted on a bug tracker full of volunteers that their issue is high priority.

• 1:23:15Lex Fridman?

  After politely requesting a support contract from Microsoft for long term maintenance, they offered a one time payment of a few thousand dollars instead. This is unacceptable. We didn't make it up. This is what Microsoft, Microsoft Teams actually did. And then they, you give the image and the details and all that kind of stuff, showing that these trillion dollar companies are not giving much money, not giving much support.

• 1:23:37Speaker 02

  They think, they think an open source project is a traditional vendor that they have an SLA. They think a public bug tracker is actually, you know, a third party vendor's Jira, where you can do all of these things. It's not, it's there to report bugs. I think the thing that made this particularly heinous was the name dropping of Microsoft, the name dropping, this is a visible product. If this was, if this was just a general bug report, I think that would have made it a lot better.

• 1:24:04Lex Fridman?

  Yeah, so they literally said like, this is a big deal because a lot of people are using it in Microsoft. I wonder what happens psychologically. So I think what happens in these companies, maybe you can correct me, is they, you're right. They just think of FFmpeg as like a vendor that Microsoft surely is paying a huge amount of money to. They kind of assume that in their interaction and nobody anywhere on the stack is going like, wait a minute. Shouldn't we be giving like millions of dollars to FFmpeg?

• 1:24:38Speaker 01

  And this is a very big problem in large, like we're talking about some companies, but it's the same everywhere, right? A lot of those companies, like the, when we talk to that person, right? He was just like a manager on one project in Microsoft Teams, right? He had like never really discussed with open source community. He had no idea, right? It was like, and, but the problem is that usually there is what we call OSPOS, right? Open Source Program Offices in those type of companies. And they are the ones who are supposed to discuss with open source vendors or open source communities. But like, they often don't explain that correctly internally, right? And here is just like, we are not your supplier.

• 1:25:23Speaker 01

  If you want me to be a supplier, I'm very happy, right? I will send you a contract and SLAs. Like I created five companies who are doing that around open source projects, so that's okay.

• 1:25:32Lex Fridman?

  We should say that some, some of the spicy tweets that Karen, you're behind and some of the debacle produced results. Positive results.

• 1:25:42Speaker 02

  Donations have increased substantially. They are still not enough to cover even a single full-time developer. But on both a, you know, awareness level and a technical level, there's substantially more technical awareness and sort of awareness of the importance of FFmpeg as a result. As a result of X and what's happened, I can say, you know, it, it, it solves its purpose. People realize the level of importance FFmpeg has.

• 1:26:08Speaker 01

  And on video landing, it's the same, right? Like, for example, a very simple example.

• 1:26:14Speaker 01

  For more than a year, we couldn't update VLC on Android because of a bug on the Play Store, on Android Play Store, right? The only way we got someone to answer was to put a very spicy, as you say, tweet saying that we were going to stop distributing VLC for Android, right? And we have around 100 million people using that. And now then, someone from Android actually came and discussed to us, right? We had the same issue with with Microsoft or like saying that we were going to stop distributing VLC on the Windows Store.

• 1:26:52Speaker 01

  And unfortunately, we are so small that the only very strong power we have to solve those issues is blaming on social network because it's nobles and now they listen to us. But those large companies often have difficulty talking to us, like for example VLC, right? Is probably one of the top 10 software used on Windows. I am not part of Microsoft ISV programs, right? I don't have a point of contact at Microsoft, right? Well, I'm sure any other software, Adobe, Spotify, has a point of contact. I don't have that, right? So raising awareness works.

• 1:27:35Speaker 01

  It's sometimes very spicy, lots of drama. X and Twitter are okay for that, but it's efficient.

• 1:27:43Lex Fridman?

  So everybody listening to this should go follow FFmpeg on Twitter on X. Follow Videoland on Twitter on X. Go donate, donate to FFmpeg.

• 1:27:58Speaker 02

  Thank you, Lex. Over the years, several years, you've been a supporter of, you know, FFmpeg and Videoland on X, you know, giving us shout outs, appreciating, you know, what we do.

• 1:28:08Lex Fridman?

  FFmpeg for life.

• 1:28:10Speaker 01

  And for example, like Team Sweeney, Carmack and a few others, like very high level people have raised also the awareness on on our X accounts and that helped a lot also.

• 1:28:22Speaker 02

  Carpathia.

• 1:28:22Speaker 01

  Carpathia.

• 1:28:24Lex Fridman?

  Yeah, I mean, also, you know, outside of the fact that so many people use it, it's so impactful on the world. It's also a great representation of a great open source project. Like the value of assembly and C and making sure that like you take. Programming seriously for real world systems.

• 1:28:43Speaker 02

  It's not just that, we'll talk about assembly later, I'm sure, because that's its whole topic in itself. But it's also celebrating people like Andreas Reinhart, who do maintenance. It's, I believe unpaid, I believe as a volunteer, he's doing massive refactorings. Andreas Reinhart and Anton Kernov rewriting FFmpeg dot C with threading. Celebrating those guys, celebrating the untold labor that's gone into this. That actually doesn't change anything from the user's standpoint. The files are exactly the same, but wow, the, the, the aeroplane has been rebuilt whilst it's in the air.

• 1:29:16Lex Fridman?

  Christian Garcia said, as a teenager running this account, referring to the FFmpeg account. And you responded, teenagers have written more assembly in FFmpeg than Google engineers. But also just pointing out that there's a lot of incredible contributors who are teenagers.

• 1:29:32Speaker 02

  Like JB said, we don't care who you are, where you're from, what you do. Teenagers have written thousands of lines of assembly over the years. Give a shout out back in the days to Daniel Kang. So also highlighting the work of people like Ruikai Peng, this is a 16 year old, some of his first contributions to FFmpeg. Actually doing and putting some of these quote unquote, security researchers to shame by actually finding issues and fixing them and being 16. There's no barriers, there's no barriers to you have to study on at college under this person and understand these. It's, if you can learn C, and let's be honest, it's from, it's from the K and R book, learn C. You can learn assembly, we'll talk about that maybe a bit later.

• 1:30:16Speaker 02

  You can contribute to world-class technologies.

• 1:30:19Speaker 01

  In VLC, one of the oldest contributors is called Felix. He's the one doing everything on Mac and iOS. He's starting, working on VLC, he was 16. We had a guy called Edward Wong, who used to be a Google Summer of Code student, who stayed for three years around Videoland. He was 14, right? And, and part of Google Summer of Code and Google Coding, which were programs where basically we have students or high school. We wrote a ton of assembly for X two six four and for VLC and for FFmpeg, right? So everyone can contribute.

• 1:30:54Speaker 02

  And he also did a good job because he didn't play the alarmist CVE heist. Create a CVE, which is like a public exposure of security and do these big scary red 7.5 priority.

• 1:31:08Speaker 02

  He just fixed an issue in Git after three days and just fixed it. He didn't need to go and play a big security drama about it. And I think I posted, you know, the kids are all right. Whereas there's, there's, there's, you know, there is a, I'm not saying all security people do this, but there is a portion of the security community, as Alex said, that likes to hype themselves up by creating drama. They would have happily raised, this is a high priority CVE eight point zero or whatever, on a, on an issue that actually was in Git, it wasn't even in a release, it was in development, and three days later was fixed.

• 1:31:40Lex Fridman?

  Well, I just wanna put a little bit of love out there, even to the bigger community.

• 1:31:47Lex Fridman?

  Much love and respect to Google engineers. Like you said, they're some of the best software engineers in the world. And they do contribute a lot, even on the security front. And also, you know, I'm a big fan of Theo, much love to Theo. He was part of this debacle and drama a little bit. I think when you just zoom out on the grand arc of human history, the drama contributed positively to everybody involved, donations went up. It brought more attention to the topic, allowed everybody to bicker in a way that ultimately got them to figure out what FANPG is all about.

• 1:32:24Speaker 02

  So the way, the way we looked at this is like, it's a rap battle at the end of the day. No, but it is, we say stuff, we say stuff.

• 1:32:32Lex Fridman?

  Yeah.

• 1:32:32Speaker 02

  But we can, we can leave it on X is a perfect place for, you know, international rap battle. You say stuff, I say stuff about your mama, but it doesn't mean, you know, I'm gonna have an actual personal issue with her.

• 1:32:41Lex Fridman?

  Yeah.

• 1:32:42Speaker 02

  And that's what it looks like. The Theo situation, you know, JB can maybe expand, went a little bit too far and there was a little, but, you know, it's just a bit of fun, it's just a bit of rap battle, it's a bit, it's WWE. You know, everyone's having a bit of fun on X, it doesn't need to be taken seriously. You know, the teenagers thing, you know, that, so that, that guy was a Google employee saying, hey, you know, there are other ways to run an open source business, you know, go and, I just have a bit of fun, you know? That's what the point of this account is. And furthermore, if you can teach people about the ways of open source projects, assembly, et cetera, by doing that, I think there's a lot to be offered here. It's not dunking on people for dunking sake, it's showing actually, the story that I think X learnt is, these are not big corporate open source projects, this is not Kubernetes where there's,

• 1:33:27Speaker 02

  you know, hundreds, maybe thousands of people paid to develop this stuff. These are just people in their basements in their spare time. And if you can address that topic in a fun and entertaining way, I think that that that's the good thing and that's that's the value of X and then the reach we have.

• 1:33:41Speaker 01

  And and to be honest, right, like even on at Google, Google is

• 1:33:47Speaker 01

  one entity, but so many different people, right? And and some and there is a ton of Google engineer we work with

• 1:33:55Speaker 01

  all the time and even like Google from YouTube to Chrome to Chrome Media to the rest of Google. Those are very different type of entities, but what we do is efficient. And for example, for for Theo, right, it went a bit too far. I had him like I calm everyone down. I had him on the phone. We say, okay, like this goes too far and so on. But in the end, yeah, it's a hard battle, but it's positive for the project. It like the awareness we have on open source and and I mean true open source from communities, right? Not is increased dramatically in the last two years and this is useful.

• 1:34:32Lex Fridman?

  What do you think motivates all the incredible contributors that we've been talking about? Like, what's the what's the engine? It's so interesting to see. Like you said, they're sitting in the basement. What's the driver? What's the engine there?

• 1:34:42Speaker 01

  There are many drivers, but weirdly, the main one is that what we do in multimedia plays videos and video is cool, right? And and for example, we have so many people in the community who arrive because they loved watching anime, right? And and this is like the advice when people ask me, what should I work on in open source? How do I start? And my question, my answer is always the same. Work on something you love. I am working on V L C because I love movies, right? And I love watching the same movies over and over, even if my wife hates me when I do that, right? But because it's interesting, right? Because it's a topic that you like, right? The first, that's the first thing where people come to usually to V L C and F F M back. The second thing is that.

• 1:35:28Speaker 01

  Technically, we, because we search for excellence, this is the best school ever, right? This is the best school ever of programming. If you're good in C, in FFmpeg, if you know how to write assembly, are you sure you're going to be one of the best programmers ever? Even if you're working on writing TypeScript, because this is the most amazing thing to do. And you will like have to get reviews by some of the most seasoned programmer ever, who are going to look at every part of your code and tell you why it's not great. It's like we are the best teachers that you've ever had in programming, right?

• 1:36:05Speaker 02

  Andrew Kelly started Zig, he was an FFmpeg developer and started Zig after FFmpeg school. I mean, it's the place to learn so many aspects of programming in the real world, in a thing used by billions of people. You have nowhere to hide, you have to be open and honest about your flaws and how you can learn and be better.

• 1:36:25Speaker 01

  And what is also interesting in multimedia is that you have 16 milliseconds to display a frame. It's not like a game engine where you can basically slow down and wait a frame. Like, so it's, you need to be good, right? There is no choice, else you don't have your video. And because of how codecs, if you miss a frame, you're going to destroy the look of the video, right? So you need to be good, you need to be perfect to have the right thing. But also is that it's not just pure programming in the mathematical fence, right? A lot of people don't understand, but um in order to program correctly on the open source multimedia community, you need to understand how computer works.

• 1:37:07Speaker 01

  And when you write assembly, you need to understand about CPU pipelining, right? You need to understand how S I M D works, how the ILU works, right? You need to understand how I O works, right? And this is what I think that is missing to a lot of engineers and software engineers today, is understanding what we call computer architecture. And like seriously, like some of the debates is like, should we use this assembly call or this one? And people say, well, no, it's going to be like three cycle on this type of CPU and this one. And it has massive impact on the output, right?

• 1:37:40Speaker 02

  We should expand, FFmpeg is probably one of the biggest CPU users in the world. That's probably, it's probably running as we speak, easily a hundred, order of magnitude, a hundred billion, maybe even a billion CPUs as we speak. So every instruction matters, there's not.

• 1:37:58Speaker 02

  The the the impact, at least in terms of CPU, is massive for everything that we do.

• 1:38:04Speaker 01

  So first you come because it's an interesting subject, then you stay because it's excellent. And in the end, you're very proud of it because it's on the end of everyone. Like so many people like, oh, I'm working for whatever consulting company and I'm doing some

• 1:38:23Speaker 01

  portal to download invoices for your PG&E. Wow, great. Like so many jobs are like that. You're not going to to tell that to your grandma. But if you go to see your grandma and say, I do this so that you can play video on your laptop. They understand, and this is very important, right? Because you're working on VLC, FFmpeg, X two six four. It's in the end of hundreds of million of people and you have an impact. And so you can be proud of yourself. And so I think that in addition to great, doing a great resume, all those things are why people contribute.

• 1:38:54Speaker 02

  Yeah, those are side effects. My favorite quote on this topic is John Collins, and he said, the world is a museum of passion projects. You know, everything out there is is a passion project and open source multimedia and open source in general. You can just do that so much faster. There's such a faster network effect, you know. I can open a cafe and that can be my passion project, but I have to get building codes. I have to build a building. I have to find a thing, uh, find a location. I have to do all the, you know, all sorts of things. Well, in the software world, that passion project can be, can move quickly. It can be amplified by the network effect. And that amplification can be more than the the the sum of the sum of the parts. You know, you can be, you can find people interested

• 1:39:36Speaker 02

  in extremely obscure things and have a network effect and make something that is truly amazing.

• 1:39:44Lex Fridman?

  And on that topic of passion projects, Tim Sweeney actually said in a reply to a tweet that was complimenting JB. He said, quote, many things in the world only happen because an awesome person decides to do it. This is the case with VLC. And that speaks to something interesting to me. It does seem that a small number of people, sometimes one person, can create.

• 1:40:10Lex Fridman?

  Something incredible in the software world. You said this over and over and over. I think JavaScript is an incredible thing created by initially a single person. Some of the programming languages like Python and C and Java, like this one person has this vision, has this design and brings it sometimes over a weekend is the initial spark.

• 1:40:31Speaker 01

  Yes, Linus built Git in two weeks.

• 1:40:34Speaker 02

  Yeah.

• 1:40:35Speaker 01

  Wow.

• 1:40:35Lex Fridman?

  Yeah, it changed the world. It, I mean, it really changed the world.

• 1:40:38Speaker 02

  Linus' passion for, hey, uploading this trouble to an FTP, like, deal with it.

• 1:40:42Speaker 01

  But for me, it's not just in software, right? And I believe in individuals that are going to change the world, right? And it's with a good, as you said, vision, right? I want to do that. It is useful, it will be useful. And whether it's going to like build train or cars or rockets or something like, I, I believe people who believe in themselves and have a vision can have a huge impact for humanity.

• 1:41:09Lex Fridman?

  Let's actually zoom out before we zoom back in. We'll just keep going up and down the, the stack. So, uh, you know, we've been talking back and forth, VLC and FFmpeg. Karen, you said that FFmpeg and Videoland VLC coexist. And there's no central point of importance. It's a kind of what you call the binary star system.

• 1:41:30Speaker 02

  Yeah.

• 1:41:31Lex Fridman?

  They succeed because of each other. Can you explain the difference, how they interact? What is the, are they competitors?

• 1:41:39Speaker 02

  I don't, I don't think they're competitors. I think, I think the simple answer is, the short answer before I go into detail is VLC is to FFmpeg as Android is to Linux. So they depend on each other, but they, they coexist because of each other. So they are a binary star system is the analogy I used.

• 1:41:56Lex Fridman?

  By the way, I feel horrible that I just recently learned that Alpha Centauri, the closest star system to us, is a triple star system.

• 1:42:03Speaker 01

  And, and when you start doing the physics, it's a nightmare, right?

• 1:42:06Speaker 02

  Yeah.

• 1:42:06Speaker 01

  But, but like.

• 1:42:07Lex Fridman?

  Hence the three body problem.

• 1:42:09Speaker 02

  But anyway. So a lot of FFmpeg pipelines involve the X264 project, which is a Videoland project. I would, I would put a finger in the air that's, I'd say 80% plus of those pipelines are dependent on a Videoland project. VLC, obviously, as we've discussed, a Videoland project uses FFmpeg, gives it reach, exposure to weird files, historically used some donation money to fund FFmpeg development. And we'll talk a bit maybe about some of the reverse engineering later.

• 1:42:40Speaker 01

  So it's a binary star system. They they work and feed off each other. Many of the developers are shared. There's no central location. It's a virtuous cycle working together.

• 1:42:49Lex Fridman?

  And we should mention that X two six four is the encoder for H two six four video standard. So H two six four is the standard.

• 1:42:57Speaker 01

  Yes.

• 1:42:58Lex Fridman?

  X two six.

• 1:42:59Speaker 01

  Is the implementation, open source implementation of the standard.

• 1:43:02Lex Fridman?

  That's used by basically everybody for everything.

• 1:43:04Speaker 01

  Yeah.

• 1:43:05Lex Fridman?

  It's that that is the main driver of this. When you think of an MP four file that has H two six four codec in it.

• 1:43:12Speaker 01

  If it if it came from a software environment, like a data center or somewhere, the chances are it was created with X two six four.

• 1:43:18Lex Fridman?

  And that's under the flag of uh Videoland.

• 1:43:22Speaker 01

  That's a Videoland project. So in the Videoland graphic, it sits in the Videoland world.

• 1:43:26Lex Fridman?

  And Videoland has a there's a bunch of stuff in it. Go to the Videoland website. There's a bunch of icons.

• 1:43:34Speaker 01

  Like if you look, there are so many libraries, right?

• 1:43:37Lex Fridman?

  Lib DVD CSS.

• 1:43:40Speaker 01

  Yes.

• 1:43:40Lex Fridman?

  Lib DVD NAV, Lib DVD PSI, Lib VLC, of course, VLC Unity, Lib.

• 1:43:48Speaker 01

  Lib Blu-ray.

• 1:43:51Lex Fridman?

  Yeah.

• 1:43:51Speaker 01

  And there is so many more, right? Lately lately, the David project that we might talk about is the last project from Videoland. It's everywhere, right? And we do, we have a Lib Special Audio lately that we announced.

• 1:44:04Lex Fridman?

  Check ISM.

• 1:44:04Speaker 01

  Check ISM, which is like an insane project, but amazing. Um so and X two six four is one of those Videoland projects. And my opinion, for example, is that X two six four was is the most amazing encoder ever designed. And this helped the adoption of FFmpeg. A lot of people and large companies went through FFmpeg because they wanted to use X two six four and X two six four increased the popularity on FFmpeg. But also VLC had its popularity because it's played so many files that were done by FFmpeg, right? So it's it's many projects that are intertwined and work together.

• 1:44:45Speaker 02

  Yeah, unfortunately, there's a there's a thing on X where um VLC is mentioned and there's people. A quick reminder that it's FFmpeg inside doing the actual work and that's, like I said, it's not. That that's not the case, that we work together.

• 1:44:59Speaker 01

  And to give you an idea, right, when I compiled VLC for Windows, I compiled around sixteen million lines of code, right? One million of those are inside the VLC repository and FFmpeg in total is probably two, around two, right? But so it means that so many dependencies are outside. And if you also look at FFmpeg per se, FFmpeg also is integrating third-party libraries like X264, but Libopus and so many others, right? So we all depend on each other.

• 1:45:28Lex Fridman?

  Yeah, that's why I was hoping to do this episode as we are doing that just kind of joins FFmpeg and VLC. Cause it's really, it's really two, two of the same, like you said, binary star system and we're all just orbiting it. Can we give a shout out to some of the people along the way? We didn't really quite talk about the history of FFmpeg. So maybe can you tell me about Fabrice? Can you tell me about Michael Niedemeyer? Can you tell me about some of the key figures here?

• 1:45:59Speaker 02

  About the eras of FFmpeg, because there's key eras and key people that made this possible. Fabrice Boulard, as you mentioned, creating the concept. And then probably in the 2000s era, I would call the era, eras tall of FFmpeg is the 2000s era was Michael Niedemeyer. So key things he got done was exhaustive support for DivX and Xvid at the time. And all sorts of weird variants of what's known as MPEG four part two. So this predates the MPEG four part 10 that we used to. So this was 2000s era video codecs where there were flavor after flavor of weird, weird decoders. At the time in the 2000s, you needed a new player to play every different type of file format.

• 1:46:44Speaker 02

  So there was Windows Media Player to play Windows media formats. There was RealPlayer to play Real Media Formats. And those were the other, the other key thing in FFmpeg at the time were native decoders for those. I actually do remember being a teenager, I must have been figuring out there was this one player that could play, could decode these files without having separate bloated players. Cause at the time when you downloaded RealPlayer, there was a ton of other stuff in there, a ton of ads, a ton of other things. And just having a simple library that was fast led to that. And then I think 2008 was, 2008 onwards.

• 1:47:21Speaker 02

  Was a big change because that's when H two six four got its maturity and I think something hopefully we'll talk about a bit more. This was the beginning of high definition video. So H two six four was the key decoder of that. So I'd call that the late two thousand and two thousand and tens and that's when the big reverse engineers came along and really did astonishing work. The beginning was a single player that could play Xvid Divex, Windows Media and RealPlayer was already in a massive achievement in itself without codec packs, without weird stuff you had to download that had weird ads and weird spyware.

• 1:47:56Speaker 01

  VLC one dot zero was out on those times, two thousand, two thousand nine, two thousand ten.

• 1:48:02Speaker 02

  Yeah.

• 1:48:02Speaker 01

  And this is like where it exploded.

• 1:48:05Lex Fridman?

  Yeah, without codec packs, it just works across all these different.

• 1:48:10Speaker 01

  The fact, so it's just like all the codecs packs are FFmpeg inside VLC, plus we have other modules for all the type of.

• 1:48:16Speaker 02

  But at the time that wasn't is they were weird. In the two thousands, they were weird codec packs with DLLs coming from this place.

• 1:48:23Speaker 01

  With a lot of spyware.

• 1:48:24Speaker 02

  With spyware, with you know what, it wasn't reliable, you didn't know and having a single player that was open source or single playback module slash player that could do this that was open source. But I think the thing to emphasize is this task in the two thousands that Michael did was was Sisyphe and it was really, the number of edge cases are poor beyond comprehension in terms of you could have a Chinese CCTV system that did one weird variant of MPEG four, part two, what's known as MPEG four ASP and that was a weird variant and you had to fix that without breaking everybody else. Times a million.

• 1:48:58Lex Fridman?

  So that's a, so you said that that's where a lot of the reverse engineering was happening.

• 1:49:02Speaker 02

  It started in the two thousands with the Windows Media stuff because that was proprietary. It started with the Real Media, so with Benjamin Larsen.

• 1:49:09Speaker 01

  Kostya Shiskov.

• 1:49:10Speaker 02

  Kostya Shiskov, that era, those were the key, that was the key groundwork. And then in in the twenty teens was kind of the Paul Mahol Kostya era. Building, doing, doing some of the most difficult codecs. JB maybe can talk about go to meeting four and go to meeting five and.

• 1:49:26Lex Fridman?

  What? What's the go to meeting?

• 1:49:28Speaker 01

  So, so, like, let's talk about this amazing Ukrainian guy called Kostya. Who was at the time, living in Germany and who was in love with Sweden, right? And the guy was the most, he is like, like, like, a lot of the people in the community are very clever. He is one of those who are like borderline geniuses, right? He was able to reverse engineering extremely complex codecs. And he does that and we do a bit of engineers with Kiran, but clearly not at this level.

• 1:50:03Speaker 02

  No, no.

• 1:50:04Speaker 01

  Um, he reverse engineered binary blobs, which are twenty megabytes.

• 1:50:09Speaker 02

  Yeah, so just for reference, uh, um, one megabyte binary blob to reverse engineer is probably order of magnitude, a month of work. And this guy is doing twenty, thirty megabyte blobs. Maybe we'll talk about that in a minute about the subtleties of how you do that. But this guy is doing it for very difficult and very obscure codecs.

• 1:50:26Speaker 01

  And, and did that for fun, right?

• 1:50:28Speaker 02

  Yeah.

• 1:50:28Speaker 01

  And, um, so go to meeting, uh, was a big problem with V L C because that's what like the number half, number one uh, feature request for a long time. So I put a bounty and, and the guy at some point said, okay, JB, I'm going to do it. And in a matter of two months, and then he explained how he did it. He was just like, oh, I looked at the code. Like this looked like a, a, a D C Ts that I used to see on W M V and so on. He did that. And the funniest part is that the code is written is a ton of jokes. And there is, is, there is a ton of JB, right? My name and, and Kemph, um, and Kemph and Kostya, uh, jokes inside the code. The code is beautiful, right?

• 1:51:10Lex Fridman?

  So one of these, I, I, I want to comment is, uh, I've got a chance to speak to some of the developers, some of this, this, this assembly language level, uh, uh, people. And they all always make everything sound like it's kind of easy. There's a, a kind of humility. Because maybe just the level of what's required to do this stuff is so high that everything else seems easy, I guess, is the lesson to take away from that.

• 1:51:36Speaker 01

  So in the community, like some of the most impressive people are the ones doing reverse engineering and the other ones doing the assembly force, right? And both of those type of people are amazing. X two six four, for example, became amazing because of a guy called Loren Merit, who is was from University of Washington, I think.

• 1:51:57Speaker 02

  At the time, yeah.

• 1:51:58Speaker 01

  And who was like, who made everything great and fast during a ton of assembly.

• 1:52:05Speaker 01

  Yeah, so this is like the the the golden era, I guess, where so many things got done.

• 1:52:10Speaker 02

  If you look at Costia, for example, he looked at the world as a binary specification. He didn't need documentation or anything. I have a binary and I can figure all of this out. And he regularly used the phrase binary specification. Ah, you know, it's not a problem. And he went and he would go away and he would come back and he would do interesting stuff.

• 1:52:28Lex Fridman?

  Can you actually speak to the details or any add color and texture to what it takes to reverse engineer a blob?

• 1:52:35Speaker 02

  Yeah, so so let's look at go to meeting, for example, is a good one because I record a meeting on go to meeting, for example. How do I play it back without needing this go to meeting? Player, they may not even be a player. I may I may need to send a recording of a meeting to someone that doesn't have a player or whatever. So first of all, there's a ton of other stuff there. There's an actual video conferencing client you need to go and find. It may be easy, it may not be easy to find the actual module doing the decompression.

• 1:53:04Speaker 02

  You need a way to actually dump the YUV data from the module. So often it involves opening in a disassembler, trying to guess where the hooks are to incorporate that module and run that module natively to decode a sample file. So figure out where this module is doing the decoding process and and find a way to hook in and output the raw YUV data because you will need that as a point of comparison for when you actually do the reverse engineering because you'll need to be bit exact or in some cases, close to bit exact. And then you you open up your disassembler, use a lot of intuition to go and figure out, you know, where the DCT is, where's entropy coding? There is a kind of.

• 1:53:48Speaker 02

  Not a rulebook, but there's always a pattern of some sort. For example, go to meeting, you know, it will be a lot of screen codec tools. There's also different variants. So often, I think it's what's go to meeting four, five.

• 1:53:59Speaker 01

  Two, three, four, I think.

• 1:54:00Speaker 02

  Two, three, four.

• 1:54:01Lex Fridman?

  Yeah. So as you mentioned here, going to perplexity, go to meeting uses its own proprietary codec for older recorded sessions historically stored in WMV files that require a special decoder to play properly on Windows. Without this decoder installed Windows media player and some editors cannot decode the video track. So you may only hear audio or see a black screen. Boy, do I remember that. But this is reverse engineering that.

• 1:54:29Speaker 01

  This is key, right? Because the go to meeting is something that not many people know anymore, right? Where you know about Zoom and Teams and so on, but like now let's fast forward 10 years, 15 years. And like this is a go to meeting dot X Z for Windows 32 bits, right? Which is like, oh yeah, but I'm on Android, I'm on an iPad, I'm somewhere else, right? How are you going to do that? I'm going to be on RIS5 on ARM. Those are blocked, but there are tons of files we need to support for the future. And this is why those type of work are

• 1:55:01Speaker 01

  exceptionally useful for humanity.

• 1:55:04Lex Fridman?

  I just have to say though, that reverse engineering process is mind blowing. It's crazy. It's like, it's a kind of like, you know, I've been reading a lot and interview archaeologists. I mean, you, you just have so little signal. Yes, yes, you know, over time, you get so much experience, you understand the structure of the original code, so you can kind of start inferring basics. But you're like, like, like, like archaeologists with a little brush, trying to reconstruct the entire human civilization.

• 1:55:35Speaker 01

  Kiran is too humble, but Kiran has done some reverse engineering.

• 1:55:37Speaker 02

  Of Cineform, yeah, at the time.

• 1:55:39Lex Fridman?

  Cineform, nice.

• 1:55:40Speaker 02

  Yeah, at the time before, actually led to the open sourcing of that work. So in parallel to doing the binary side, you obviously have samples. In many cases, you don't have many samples. So you have to figure out what all the different flavors are and you may have a Cineform, for example, is actually a collection of different approaches and toolkits within that codec, because often it, it grows naturally. And, and the hard part is finding the sample that gets you kind of. Somewhere to start without having to implement ten different other things. So start there. I think thankfully at the time I found a sample by pure chance that had a lot of flat blocks. It was animation. So that really helped a lot because it wasn't using particularly complex coding tools, et cetera. And you could kind of get somewhere and then and then build up and build up until

• 1:56:26Speaker 02

  you figure, hey, here's a few bits here. I missed this. I missed this this if branch that it does and go, oh.

• 1:56:30Lex Fridman?

  So when we say samples, you mean sample videos and then and then you're tracking, trying to infer like, what is this code doing?

• 1:56:37Speaker 02

  Yes.

• 1:56:38Lex Fridman?

  By observing the sample and then looking at what at the at the machine.

• 1:56:43Speaker 02

  The machine code saying, I have this byte is six, take this branch. And then a different sample. Oh. That's nuts, man. That is nuts.

• 1:56:51Speaker 01

  So you see, this is not, then you go to things like go to meeting. It's like.

• 1:56:56Speaker 02

  Mine was easy, right?

• 1:56:57Speaker 01

  Imagine.

• 1:56:57Lex Fridman?

  Yeah, right.

• 1:56:58Speaker 01

  Two order of magnitude of more complexity, a guy alone, somewhere in Germany, doing that.

• 1:57:06Lex Fridman?

  Yeah.

• 1:57:06Speaker 01

  And for a long time you work, you're in a black box. Because the decoder for a long time, because there is so many steps from the entropy decoding, the intra prediction, the motion prediction, the IDCT and so on. For a long time, you don't see anything, right? So you're debugging purely in memory.

• 1:57:22Speaker 02

  Debugging guesswork and you may have the buffer that the coefficients are stored in completely wrong. And so you may be going down a complete rabbit hole, thinking it's this and then, oh damn, that's not, that's that's something else and.

• 1:57:34Speaker 01

  And you're doing that on binaries that are tens of megabytes, millions of instructions, right?

• 1:57:40Speaker 02

  So you're stepping through the debugger, like one by one, you know, instruction by instruction going, hey, this instruction changes this, this does this.

• 1:57:47Speaker 01

  Pausing the program on the CPU level.

• 1:57:49Speaker 02

  Pausing it there on the CPU level, watching what's going on, trying to figure out.

• 1:57:52Speaker 01

  Sometimes you need to like be in a VM. So yes, you can pause the VM.

• 1:57:55Speaker 02

  Yeah, pause the VM, dump the memory because it could, some of the codecs could have encryption. There could be um like a DRM on there. So you need to dump the memory from a virtual machine.

• 1:58:05Speaker 01

  Like when I joined Adecol Central Paris in two thousand and three, John Lesch Johansson basically broke the DVD specification and created DCSS, showed us how he was breaking a DRM which was MP4 fair play from Apple. What he did on his laptop, and I was young, I was twenty one, was just like blind boing because he was basically debugging Windows in inside a type of VM with like, wow.

• 1:58:30Lex Fridman?

  It's incredible. It's mind blowing and inspiring. Does it get like from your experience and from what you've seen in the community, does it get discouraging? Does it get?

• 1:58:39Speaker 02

  People help you. People send you samples. People are keen. Sometimes you don't have access to an encoder, so this is even more difficult because you just, you just ask and you have to ask for samples. I remember Videoland used to tweet for samples at one stage, hey, I need this obscure sample and.

• 1:58:55Speaker 01

  For a long time I was, oh, I need this codec and I need this codec.

• 1:58:58Speaker 02

  And and if you were really lucky, you would find like, if you were on, if you were unlucky, you'd get like one or two, you'd get nothing or you'd get one or two. And then they would, sometimes you'd find a gold mine. It's like, yeah, my company has a hundred thousand of these files because we depended on it for some reason. Um, and so those are the, those are kind of the best if, if, because then they can test bit exactness across the huge range of coding tools.

• 1:59:18Lex Fridman?

  Can you explain bit exactness?

• 1:59:20Speaker 02

  Bit exactness, so um, most, but not all video codecs, certainly from about the two thousands onwards, have a bit exact definition. So every implementation must produce exactly the same bits, bit for bits, in exactly the same data that comes out of a decoder.

• 1:59:39Lex Fridman?

  For like a large number of samples?

• 1:59:41Speaker 02

  For a given sample. So Lex's implementation, JB's implementation and my implementation of H two six four must match bit exactly. That wasn't the case in the nineties of MPEG two, probably fair to say one of the biggest mistakes the video industry made. And I think people who were in the room in ninety two, I don't know, most of both of us were in diapers, I suspect, but um, have acknowledged, um, I would give a shout out to Yuri Resnick. He's acknowledged that was one of the big mistakes of the era.

• 2:00:08Lex Fridman?

  And you're saying the encoder is needed to be able to run tests and then the, the, the bit exactness. I mean, that's a nice thing to guarantee. Like, there's a parallel sort of development here on the way the web browser works, which is, you know, takes HTML and displays it and there's no bit exactness there across the different engines.

• 2:00:28Speaker 02

  I would, I would point out, actually, FFmpeg is unique in the sense that it's, it has been a winner takes all scenario. You have browsers is a good analogy because it has to parse a lot of different content and render it in a particular way, like a decoder. But there still are multiple browser engines. There's Firefox's one, there's Chrome's one, there's a few Japanese ones that are pretty decent. That's not been the case in multimedia in general across a wide range of codecs. FFmpeg is kind of one it all, I suppose, in a sense because of, because of the fact that you can get every new codec added is actually worth more than the value of that codec itself because it makes the whole thing better.

• 2:01:05Lex Fridman?

  Man, this is really cool. Going to perplexity, Yuri Resnick is a multimedia and signal processing researcher, got his PhD in computer science from Kyiv University with over 150 papers and more than 80 granted US patents, contributor to major multimedia standards, including issue 6.4, MPEG4, AVC, H.265, MPEG4, ALS, G.718.

• 2:01:30Speaker 02

  G.718, telco stuff.

• 2:01:31Lex Fridman?

  Telco. And so he was more connected to companies.

• 2:01:34Speaker 01

  Real audio, real video, right?

• 2:01:35Lex Fridman?

  Yeah.

• 2:01:36Speaker 01

  Oh, yeah. Very important at that time.

• 2:01:38Lex Fridman?

  Zemcator, Brightcove, Contexts. Man, I need to hang out with Yuri. He's legit.

• 2:01:43Speaker 01

  Yeah. And he's like one of the most nice person ever, right? Like, like, for example, for my, for my startup that I'm doing right now called Kyber, right? I met Yuri because I met him every year at the Mile High Video Conference, which is in Denver. And he would, he gave me like so much good ideas and good things. He's like really amazing person.

• 2:02:05Speaker 02

  He tells, he tells us how, how, you know, how great it is to be, you know, even know us. And I was just like, you know, you look at that and I think it's, I think it's the other way around, Yuri.

• 2:02:14Lex Fridman?

  That reminds me of a thing that you mentioned to me about fate testing and like the insanely rigorous process that's used to test everything that's incorporated into into FFmpeg. Can you take me through the testing process?

• 2:02:27Speaker 02

  Yeah, so FFmpeg has a system called Fate, FFmpeg Automated Testing Environment. Because FFmpeg runs on so many different OSs and can be compiled with so many different compilers, there's been a crazy number of configurations. So you can see the absurd combination of compiler variants, operating system variants, instruction sets. You can see at the top Mac OS has tons of different variants because it has iOS, it has TVOS.

• 2:02:55Lex Fridman?

  I'm looking at a page fate.ffmpeg.org, eighty-one minutes ago, seventy-six minutes ago, looking at the different architectures, the operating systems, different compilers, Apple clang version.

• 2:03:10Speaker 02

  Combinations are crazy.

• 2:03:11Lex Fridman?

  The combination is insane.

• 2:03:13Speaker 02

  These are all run by volunteers. So these are all volunteer systems. The ones at the top, for example, the Macs, I host in my office, for example, host all sorts of different stuff. Other people host other things. So it's really there to make sure, because FFmpeg does quite complex C code, for example, you do have missed compilations. So the compiler will sometimes compile C code incorrectly, for example. This happens once in a while.

• 2:03:38Lex Fridman?

  Oh, there's like, there's a log of all the compilations.

• 2:03:41Speaker 02

  Yeah, log of all the compilations, all the tests. I think one of the other ones will show all the tests passing.

• 2:03:45Speaker 01

  If you click, you can see all the tests back, all tests successful.

• 2:03:50Speaker 02

  In logs test, yeah. So you see all those tests are passing

• 2:03:55Speaker 02

  of all the different codecs, all the different filter transformations, all the, the level of scale is quite crazy. On all the combinations, it's not just a matrix at this point. It's like a pivot table of different combinations.

• 2:04:08Lex Fridman?

  That's nuts.

• 2:04:10Speaker 02

  And it's a key part of what we do because you, you may be able to test something locally, you make a change, but actually that breaks GCC version eleven on Mac or something like that, and you're able to then fix that. We also have missed compilation. So the C code, sometimes the compiler can have a bug in it where it creates the wrong output. And that can have quite a big effect sometimes on a video, on a video because of the way frames have dependencies, even a small change in the output. Can can cascade to actually quite big glitches.

• 2:04:40Speaker 01

  You see power PC, you see risk.

• 2:04:43Speaker 02

  There was power PC, there was risk, there was weird stuff in the past, like D E C Alpha.

• 2:04:47Speaker 01

  You see Visual Studio, different version of Clang.

• 2:04:51Speaker 02

  Intel compiler, Apple client, you name it.

• 2:04:53Lex Fridman?

  What are some of the pain points? Like, maybe, do you have emotional triggers? Uh, maybe nightmares about a particular operating system, a particular container, codec, combination?

• 2:05:06Speaker 02

  For for me, it's really easy because, uh, so I I have a day job, um, my company builds. The company I started builds, um, equipment for broadcasting sports matches between TV stadiums and studios, for example. We have to work with ten bit video and ten bit video has, um, a set of challenges that you can't process ten bit data natively on a CPU. So that means you have to stick it in, um, sixteen bits. So that means you have six wasted bits. So there's different packing formats to actually pack the data more efficiently, because when you send that over a network, you lose, because you need to, you need to save that forty percent. For example, on PCI Express, you may only have bus bandwidth to do that.

• 2:05:46Speaker 02

  And so I think internally, we have about, so some are, some are industry ones and some are internal to our own hardware that we build. We have a, I think, a five by five or six by six matrix of every single format to every single other format conversion. In fact, one, one of them I sent you and they're all written in handwritten assembly. And they're all written and they all support different CPU generations. So this is really traumatic, handling all these different combinations, times a million.

• 2:06:15Speaker 01

  By the way, the company you're talking about is Open Broadcast Systems.

• 2:06:17Speaker 02

  No, no relation to the Free OBS streaming service.

• 2:06:20Speaker 01

  Yeah, yeah.

• 2:06:21Speaker 02

  But JB and I have started companies broadly speaking around the FFmpeg VLC ethos. So that's really low level work. So, so in most companies, this, this wouldn't be written in assembly. It would be accepted that C is, C is fast. Um, as you can see from that, C is not fast.

• 2:06:38Lex Fridman?

  Uh, so here it says sixty two times faster than C.

• 2:06:43Speaker 02

  Yeah, so it's taking those, the ethos of doing low level programming, real time programming,

• 2:06:52Speaker 02

  and take, using that for commercial applications. And JB and I have started companies around that, in many cases, hiring developers from the open source community.

• 2:07:01Speaker 02

  To use that ethos and so that that's a great example of of some of the things we're doing. In most companies, it would be, oh, I'll write this in C and it's fast and we're done. But actually, you can get a lot better.

• 2:07:12Speaker 01

  For me, like, some of the headaches we have is around some OS that are difficult to support, right? Because um uh if you look at VLC and thanks to FATE and FFmpeg, we run on the last version of VLC runs on Windows XP and still run there and runs on Windows 11. We work on macOS 10.7 to the latest macOS, whatever it is, right, 26.

• 2:07:44Speaker 01

  We work on iOS since iOS 9, where we are actually iOS 26, right? We support um we support many types of Linuxes, BSD, Solaris. The last version still runs on OS 2, right? Like, there is maybe 10 users of OS 2 in the world and one of them is maintaining VLC. Then you realize that this very small team around VLC and using FFmpeg codecs and and all the other ones support more OSes than Microsoft or Google or Apple. And they have infinite amount of uh of uh power and resources. But for example, the worst is iOS.

• 2:08:24Speaker 01

  For in order to build on iOS 9, we need to do some very clever mixing of several version of the Xcode IDE and SDK from Apple, from several version and do a type of Frankenstein version of that so that we can still support iOS 9, which is not supported at all by the the compiler of uh of uh Apple in order to still run on ARM 32 um on iOS 9. And you've seen on FATE that it was still supporting iOS 9, right? So so um my headaches are mostly related to the the support of so many OSes. And it's important because like we receive so many people saying, hey, thank you.

• 2:09:05Speaker 01

  I still have my iPad 2 to watch movies and it still works on iOS 9, right? And it's also an impact of like not forcing people to buy new hardware when it works fine. If you optimize it correctly, which brings us to what we were saying about assembly, it's also fighting like. The fact that you need to buy something new nonstop uh while you could optimize more, which is a lost art.

• 2:09:31Lex Fridman?

  You gotta tell me about this lost art or this uh the the carriers of the flame of assembly. What uh what is what is assembly? Why is it beautiful? Why is it challenging? How does it work?

• 2:09:47Speaker 02

  So when you write assembly code, you write this using the instructions the actual processor is using directly. So most of the time you would write in a language, let's take C is a good example. The compiler would use that to create um assembly language and machine code instructions for you based off your C code. And there's a specific flavor of assembly that we use in FFmpeg. That's called S I M D, S I M D, Single Instruction Multiple Data. So this means, for example, say, say I want to add five to a number in Scala assembly. So this is what's known as you work on an individual element. So I wanna have a number, I have a number ten and I want to add five. I use the add instruction and I add five to ten and I get fifteen.

• 2:10:32Speaker 02

  With S I M D, with S I M D, I can have a whole vector of sixteen different numbers. It could all be different. If I want to add five to that, I can I can run one instruction. And that one instruction sums all sixteen elements. And that, as you can imagine, lends itself very well to video. Video uh video is, you know, pixel grid. So I can perform operations on multiple pixels at the same time. The key thing that we do differently in FFmpeg is we don't use any abstractions or any major abstractions on top of that. So there's a part of the world that uses what's known as intrinsics. So these are C functions that behave very similarly, but not quite the same to writing assembly by hand.

• 2:11:15Speaker 02

  So the registers that data i uh is stored in. On the CPU, the compiler allocates those for you. And so the key thing to understand was when we write CMD is we have a 10x, not percentage, 10x to 50x speed improvement, that that function is 62x.

• 2:11:34Lex Fridman?

  That's nuts.

• 2:11:35Speaker 02

  On the FFmpeg account, as you know, posts and tweets a lot about that to try and say, hey, we're doing this stuff.

• 2:11:41Lex Fridman?

  You are a person who sees the beauty in assembly, but it's also extremely useful for these kinds of application to actually significantly outperform even C, which is crazy.

• 2:11:53Speaker 01

  It is necessary, right? Because like, one of the projects that we need to talk about is called David, right? So David is a decoder for the format that was done by Alliance for Open Media, which is a video decoder called AV1.

• 2:12:11Lex Fridman?

  So for people who don't know, we've been talking about H.264. AV1 is another hugely popular standard and codec that is increasingly taking over the Internet.

• 2:12:25Speaker 01

  And when this format was launched, many people said, especially even from the Alliance for Open Media, right, which is Google, Netflix, Amazon, Mozilla, say, well, this format is so complex, it must be done in hardware to do decoding, right? And well, I arrived with a few other people, mostly Ronald, Henrik and Martin, who said, we need to have an extremely good software decoder because it's going to take time to have hardware. And so we wrote this project, which is beyond insane. We are talking about 30,000 lines of C, but 240,000 lines of handwritten assembly, right?

• 2:13:09Lex Fridman?

  Handwritten assembly, 240,000 lines.

• 2:13:15Lex Fridman?

  That's incredible. I mean, some of the stuff we're talking about is probably the biggest assembly code bases.

• 2:13:22Speaker 01

  To give you an idea, and Karen can correct me, but I think the FFmpeg has 100,000 lines of assembly for all the codecs.

• 2:13:30Lex Fridman?

  All codecs.

• 2:13:30Speaker 01

  And just this one has 240,000. It's a video and project, of course. And it is optimized at the maximum because the motto when we're starting the project is every cycle matters, right? Every cycle matters because David is used in VLC and in some software AV1 playback stacks. We are talking about probably 3 billion devices which are going to decode video nonstop. Because for example, 30% of the video from Netflix are now in AV1, 50% of YouTube, right? So, and you often don't have a hardware decoder because not many devices have a hardware decoder.

• 2:14:11Speaker 01

  And with David, we realized that when one or two cores, you were able to decode 720p correctly. So it is like literally incredible, right?

• 2:14:22Lex Fridman?

  That's David. Look at that. Yeah, so this is another spicy tweet from you. This is what peak video codecs should look like. 79.9% assembly, 90.6% C and 0.5% other.

• 2:14:38Speaker 01

  And what's incredible is with those tweets, which is factual, people get crazy. They are unhappy, right?

• 2:14:46Lex Fridman?

  For the last two years, they go crazy. No, Intrusion6 is fine. The compiler is... Oh, they go.

• 2:14:50Speaker 01

  You can optimize your compiler, auto vectorization, it's your fault, you don't understand. And we've tried that forever, right?

• 2:14:57Lex Fridman?

  For two years and two years later, showing hundreds of examples of handwritten assembly. No, no, no, you're doing it wrong. The compiler can do this. So we should actually just articulate a little clearer. So the intuition there from the software engineering folks, when you have code like, okay, let's just take an example, C++. There's a compiler. That's doing a lot of the optimization.

• 2:15:18Speaker 02

  Yes.

• 2:15:19Lex Fridman?

  And the presumption is if you have a good enough compiler, if you continue to improve the compiler, you're going to generate code.

• 2:15:26Speaker 02

  Yeah.

• 2:15:26Lex Fridman?

  That can perform like optimal performance. You cannot possibly beat it. And you're consistently challenging that thought that if you do.

• 2:15:34Speaker 02

  By orders of magnitude.

• 2:15:35Lex Fridman?

  By orders of magnitude, handcrafted assembly can outperform C.

• 2:15:40Speaker 01

  The two things that they tell us is, yeah, but modern compilers have auto vectorization, right? Because S I M D that we're doing is vectorization. And like, it's not even close, right? It's not even close, right? It's not like five percent, 10% slower. It's multiple times slower.

• 2:15:56Lex Fridman?

  So can we, I don't know if you can say something philosophically, because there's a lot of, there's a lot of great software engineers, great engineers, great machine learning people. Karpati will listen to this and say, what's the intuition he's supposed to get from this? What are we supposed to?

• 2:16:10Speaker 02

  Karpati learnt assembly because of the tweets, by the way. I just, he's like, oh, I think this is a move.

• 2:16:14Lex Fridman?

  Let me figure out what's happening.

• 2:16:16Speaker 02

  And you know, the way he documents his work and so.

• 2:16:18Speaker 01

  Philosophically, what's important to realize is that we passed the time where hardware was going so much faster, right? We have the end of the Moore law. We have limitation for, for AI, for memory. You need to go down in the stack and optimize more to get more power from what you have. Because our requests for power, CPU power, GPU power are exploding while the hardware is not exploding in speed, right? So what people do is that they add more cores, right? But that's basically like, at some point, you can add 250 cores, right? So what we do is to take every inch of the machine.

• 2:16:59Speaker 02

  Not just that, not just that. We abuse the machine. We go and use, we use the machine in ways that the, that the creator didn't expect. Sometimes we use an instruction that's completely unrelated to what we do. We use a cryptography instruction in video processing to do nothing related.

• 2:17:14Speaker 01

  And one of other things that we do, for example, in David, which is a bit crazy, is that we don't use the function. Calling convention from the operating system.

• 2:17:26Speaker 02

  We should explain that.

• 2:17:27Speaker 01

  That is extremely complex, but basically usually when you do move from one function in in code to another, there is a way to save the registry, the state of the CPU to enter another function. And this is like standard.

• 2:17:41Speaker 02

  It's a bit complex. I would I would simplify this a bit. So. So David does things to abuse the calling convention. You could define the calling convention as. I've written a function and I want to call another function. How is the data shared between the functions? Because there's a convention, what's known as a calling convention. And what David does for optimal reasons is. Create its own calling convention sometimes. So if I want to call Lex Fridman's library, we got we've got to agree on a convention so that I can share data with you in the assembly language space. And one of the challenges in assembly is every operating, well, not every operating system, but there are at least four that I can think of on x86. Linux 32 bit, Windows 32 bit, Windows 64, Linux 64.

• 2:18:24Speaker 02

  They all have their own calling conventions. And so one of the amazing things Lauren Merritt did, who we talked about before, was create a very lightweight abstraction layer. So you could write your assembly code once and it handled all the calling convention stuff for you.

• 2:18:38Speaker 02

  Which is always a problem because you had to manage four different variants. But David takes this even further. For speed reasons, it does its own calling convention. Within itself. To bypass the kind of rules, the rules of the rules of sort of of functions and say, okay, actually, I'm going to call a function this way. Because I know it's within my library.

• 2:18:58Speaker 01

  Does it have to be special to every single operating system?

• 2:19:01Speaker 02

  Well, if it's custom, no, but the the the challenge is. In general, yes, and in terms of in terms of each instruction set. So the thing to also emphasize is. We do this on every instruction set. So every instruction set has its own handwritten assembly. Which is even more crazy. And that that that has um. That matrix has got bigger in recent years because of RISC-V, because of ARM 64, because of the new SVE, there's SME. x86 has AVX 512, AVX. So we do runtime processor detection. We see what the machine FFmpeg is running on or David's running on is capable of. Because you could be on a laptop from 2008 where this isn't there. Runtime detection, we set function pointers accordingly. And then from then on, off you go.

• 2:19:46Lex Fridman?

  Or you could be on a machine with verse five.

• 2:19:49Speaker 01

  Yes, and in all that, we don't even respect the calling convention of the operating system in order to be faster, because we know that we are going to be called from within our binary, so we can share data without saving all the registry in the common way, because that can lead to loading and saving registry on the L1 and L2 CPU and gets us faster. So that's why I said that understanding CPU architecture, computer architecture is key. And this is also why it's handwritten. I don't know anyone. I've never heard any other project than David doing that. This is what Kiran calls it, calls it an art, right? It is an art.

• 2:20:27Speaker 02

  I think, I think in a, in a mass world, there isn't something on billions of devices. I, I know there are some specialist industries. I know in high frequency trading, they take this really seriously where they're receiving feeds from a market and they need to react within X number of microseconds. And so the instructions matter, but that's not a mass, you know, mass produced thing that's on a billion devices, that's hyper specialized, running on hyper specialized hardware. We're running on all hardware from.

• 2:20:52Lex Fridman?

  Sorry to linger on it, but like, that's a really counterintuitive, almost like revolutionary

• 2:20:59Lex Fridman?

  idea here, that there is a huge amount of value to assembly. Like, what are we supposed to take away from that? Like what, you know, there's a bunch of people listening to this. They're basically like, sorry, from myself included, you know, I, I programmed for many, many years in C, C++ going up the standards of C++ found a love of C++ even meta programming and so on, and then transition more and more because of machine learning about 15 years ago to Python. And so like, for me in this Python world, JavaScript world, now vibe coding where I'm just using natural language, sitting in my jacuzzi, drinking a drink and just talking to the computer. Like like record stops, why is the value to go back all the way down to the low level?

• 2:21:44Speaker 01

  Because you can get more power per dollar invested, right? And sometimes it's going to be a problem that is limited by your hardware. A good analogy is what you see in quantization in L L M's, right? And people are doing, oh, I'm going to do that in F P eight or F P F P four or some some crazy things like Microsoft fear who did in one dot five. Because you're constrained by memory, because you're constrained by the machine you can run. Because at some point we are doing real time. And I believe this is going to happen on A I inference also is that at some point you need to get faster and you cannot always get harder, more powerful hardware, right?

• 2:22:26Speaker 01

  So you need to analyze code and see where like where is the mission critical? Where is the things that are called nonstops? And for example, David is a good example. It's going to be run billions of hours per day.

• 2:22:41Lex Fridman?

  Yeah.

• 2:22:41Speaker 01

  That makes sense. It doesn't make sense to be on the glue of F F M P C L I. It makes sense over there.

• 2:22:48Lex Fridman?

  Yeah, this has to do also, we'll talk about it more, but your new effort in your company, Kyber, is doing that kind of thing for ultra low latency. So the slogan being every millisecond counts. So when you actually extremely highly constrained in some dimension.

• 2:23:06Speaker 01

  We are also arriving at a point where we've done

• 2:23:10Speaker 01

  so many great things, but the hardware is getting back to us, right? Because cost is increasing because we need more power. And so you're limited by either your C P U, your R A M or your networking. And you need to optimize. And this is where value is going to be, especially because like doing A I is going to help do the programming of like business, right? And so the core thing that you will not be able to vibe code are optimization for the hardware to be as fast as is possible.

• 2:23:43Lex Fridman?

  I'd love to talk to you about who and how should learn assembly. But first, I think we need a bathroom break. Quick 10 second thank you to our sponsors. Check them out in the description. It really is the best way to support this podcast. Go to lexfridman.com/sponsors. And now back to the episode. All right, and we're back. There's this nice repo with the assembly lessons. First of all, do you think developers should learn how to program in assembly? And how would you go about learning it? What is this ASM-dash lessons?

• 2:24:21Speaker 02

  So I personally wasn't happy with the way assembly is taught in books and online. Cause it's very grammar focused and you don't in general learn a language from

• 2:24:33Speaker 02

  learning the grammar and the structure. You learn a language by asking someone what their name is and you start from there. And you go and solve real problems that you have when you want to communicate. You don't, you don't learn sentence structure. This is the interrogative and the adverb. And all the assembly books seem to be doing like going through every instruction, even ones that aren't really relevant, explaining what they all do and how they actually doesn't really change much. So, and the other problem that we have in our community is assembly is taught sort of hand to hand, like person to person, like blacksmithing one by one. That's the only logical sort of analogy. And that doesn't really scale online. It doesn't do other things. So this, this, this, I started a set of assembly lessons in the, in the way it's done in FFmpeg, which is a little bit different to the way

• 2:25:19Speaker 02

  assembly in general for, I don't know, I'm trying to think the other good big use case of assembly is in embedded devices in really, really low power, cheap devices. And that's completely different to what we're doing here. I think it would be good if you could highlight the requirements, which are quite simple. It's high school mathematics and C. And actually not even C, really, really it's pointers. To emphasize, yes, we've talked about how brilliant this stuff is, but high schoolers like Daniel Kang have written.

• 2:25:46Speaker 02

  Assembly and FFmpeg, I think there's been contributions because of these lessons. So it's really about trying to to get this dying art to continue because we've shown it's possible with David to produce something amazing. There's still a lot of codecs in FFmpeg that are only maybe partially assembly, um, assembly optimized. And so it really, it really starts with basics and continues, explains a lot of the jargon, a lot of the syntax. It doesn't really try and explain to you, you know, interrupt handlers and interrupt instructions and all of these different jump targets. It actually makes this really vector focused.

• 2:26:21Lex Fridman?

  And describes all kinds of registers, general purpose registers, vector registers. Uh, really nice examples. This is cool.

• 2:26:31Speaker 02

  It's a classic, it's a classic example of FMP, but um, some of this assembly language is is really beautiful and I, and I think it's beautiful because it's kind of like flying a Spitfire. It's really aviation at its purest, but also pushing the aircraft beyond what the designer thought was possible. So we're abusing, for example, sometimes cryptography instructions to do certain things. And there's a level of beauty in art where it's really you and the processor. There's, there's, there's nothing in between. It's you and the, the joystick of the cockpit and you, and you, you move that joystick and it's physically connected to the ailerons. And you can push that plane beyond what it can normally do. And there's a level of,

• 2:27:13Speaker 02

  yeah, beauty and amazingness to go there, but I don't think the sort of person by person assembly that someone taught me and I've taught multiple people is gonna work long run because of the particular flavor and the way that we do it.

• 2:27:29Lex Fridman?

  It's literally, uh, no, I should, I was gonna say wizards handing it down. I realize I look like a wizard wearing this hat. But you're basically just like the sages, the wise sages handing down the craft. Can, can I ask about L L M's? Like, can they help?

• 2:27:45Speaker 02

  They had more of an understanding than I expected, but they are still, I've asked it questions and it still goes and starts hallucinating, not hallucinating, but making modifications. And then I go, is it bit exact? No, fix it. And then it just goes and does the same thing. And it's going, it, there isn't the corpus of information like Stack Overflow to work on.

• 2:28:05Speaker 01

  There is not enough data to train on. And this is the biggest issue. I started my career actually doing some assembly for Ethereum, right? So Ethereum is a dead processor type, right? Which was done by Intel in HP a long time ago when they wanted to do 64 bits. Well, they lost and then we got AMD who did it. AMD 64, which became x64. But Ethereum was extremely interesting in the sense that those were processors who had a ton of computing power to do floats, FMAs, which is similar to what we need now for LLMs, right? And you could pack three operations per line that could be loaded.

• 2:28:50Speaker 01

  So basically you had an output of basically 6 billion of operation per second. But the bus, the memory bus only allowed 1.5, right? So your CPU was four times faster. So you had to do crazy things to to pack things in memory, reuse the register. And those type of semantics, no language could do that, right? So like I have the Ethereum programming book because Intel did amazing books. But that's exactly what Kieran says. If you don't know what you're going to do, it's impossible to read, right? It's a ton of jargon and so on. While those lessons are amazing because they are targeted to a real problem and you can do it yourself.

• 2:29:34Speaker 02

  People have, people have their patches and it's now I studied your lessons and here's my first changes.

• 2:29:39Speaker 01

  That's amazing. And part of that in the lessons is a framework called x86 Inc. Written by Loren when he was working on x264. And it allows you to do more things about that to create a type of like not caring too much about different calling convention. And we had a lot of students who gave code to x264 using that a long time ago, right? So it's really doable and I believe it's necessary to understand assembly language, even if you don't do it much. To understand what's going on inside your computer and that will make you a better programmer. And I assure you that because doing that, you will understand some of the

• 2:30:23Speaker 01

  architecture of the memory inside your your computer, right? Understanding register L one, L two, L three, RAMs, SSDs, disk and so on. Which are very important because then you have a good programming culture that will make you a better programmer.

• 2:30:40Lex Fridman?

  What do you think about the Rust programming language? Because that's a bit of a meme.

• 2:30:44Speaker 01

  We have very different opinions with Kiran.

• 2:30:46Speaker 02

  I think it's valuable what they're doing in terms of memory safety as a concept.

• 2:30:50Lex Fridman?

  Can it achieve this, some of the speed up that assembly achieves?

• 2:30:54Speaker 02

  Not assembly by hand, no. I think that that's a given, C potentially, but I see it very. It has a very big Esperanto vibe about it. It's like, we're gonna solve this and we're doing this in a particular way.

• 2:31:07Lex Fridman?

  Meaning it's a bit too utopian?

• 2:31:09Speaker 02

  There's a lot of focus on the self-importance rather than solving real-world problems. It reminds me of the Sinclair C5. Sir Clive Sinclair of Sinclair Computers built a car. And he said, oh, everyone will be traveling around in in one of these electric cars. And it was, um. Rust reminds me of that where I think the community doesn't, the community doesn't quite understand that in order to get people to move, you have to build something that's as good as, if not better than what you have now. Yes, people are doing Rust rewrites, but if they're if they only, if they only do eighty-five, ninety percent of the feature set of what what we need, like things like call util, that last one percent takes ninety-nine percent of the time.

• 2:31:53Speaker 02

  To to use Elon's famous quote, prototypes are easy, like this kind of stuff is easy, but this to get a real electric car, you have to make a car as good as, if not better than what we have now. And Rust isn't in that stage yet. I think we, I don't think anyone would object to seeing Rust code in FFmpeg. But it needs to work as well and support the same unit testing as everything else. It needs to be flawless. It can't just randomly break. They can't just randomly break ABI when they want to. It needs to. It needs to have, I think, more, I think it still has only one compiler implementation.

• 2:32:24Lex Fridman?

  Yes.

• 2:32:24Speaker 02

  So. It's got to be as good as if not better. And saying, hey, here's my utopia of memory safety isn't enough, even though we probably all agree that that's the goal.

• 2:32:37Speaker 01

  So I've done a ton of Rust. And the two major topics I had was adding Rust modules inside VLC. One of the reasons VLC got popular, and which was one of the main architectural decision, is that VLC is a very small core and a ton of modules, right? And so you can write modules in C, in C++, in Objective C, and anything that is basically interoperable with C. And so we did some Rust modules. And so I have experience on that, and I wrote some of it. And also, like, my new startup called Kyber is an open source project mainly done in Rust.

• 2:33:16Speaker 01

  What Rust is extremely good in the sense that it's a better C++ that cares about memory and allows you to do things about memory ownership that no one else can do so far. However, it's great when you start a new project from scratch and you do everything in Rust. But it's very not good when you interrupt with the existing part. And some part of the Rust community believes that they need to rewrite everything, and everything will be better with Rust. And the answer is like, no. Like, I'm almost always, in all my years of being engineer, manager, CTO of startup and so on, don't rewrite, right?

• 2:34:00Lex Fridman?

  That's the initial instinct for a lot of people when they show up to a codebase, probably before LLMs, is like, probably because they don't understand the wisdom of the way things have been done in the past. They say, well, we need to rewrite it. Hence why there's a thousand JavaScript frameworks.

• 2:34:19Speaker 01

  But the reason is the following. And this is very important to understand. It is an order of magnitude easier to write code than read code. Yeah. And you see that also with L L M. They can write code. The analyzing is a lot more difficult. And so when you arrive and when you arrive to a very complex piece of code, right? You you don't understand it, right? Because it's so much more effort to understand the code from someone else because you don't have the thought process. And often I joke about some some languages, mostly Pearl, for example, which has very complex syntax.

• 2:34:59Speaker 01

  And imagine I am at my maximum intellectual efficiency in programming, right? And I write the best code ever. I will not be able to understanding myself six months later, right? Because reading code is more difficult. So very often you arrive, you don't understand all the wisdom, all the business logic, the reasons that were done, that is maybe not documented. And you say, well, I'm going to write it. And the thing is, no, you don't, right? Because that's as Kiran said, right? I'm going to rewrite core utilities in Rust. And then, of course, you arrive very quickly at eighty percent. Then ninety percent takes a bit more time and then you got the last ones, right? On the other side, right? So for new projects, it's great.

• 2:35:43Speaker 01

  Everything related to parsing files, networking, because of the memory checker, borrow checker, it's amazing and there is nothing else. To answer a bit differently for us. Imagine I take a piece of software like David or X two six four, right? Which has a ton of runtime in assembly, right? I rewrite the C part in Rust, right? So it's more secure, yes. But then you arrive into the assembly and you can jump anywhere in the memory because we are doing handwritten assembly. So even if I rewrite the C part in Rust for security reason, you you break all the security when you rewrite handwritten assembly because we can jump anywhere.

• 2:36:29Speaker 01

  So in my opinion, we need to do something that is secure assembly, right? So which is compile time, check the assembly, which is similar to the check assembly project that we're doing on on David and X two six four with Videoland is. To start instrumenting your assembly at compile time to check that it's not jumping anywhere in the memory, because else you might rewrite a part of C in Rust. But if you want to have the same performances, you're going to have inline assembly and so you destroy your whole security model.

• 2:37:02Speaker 01

  So that's a bit what I think about Rust.

• 2:37:04Speaker 02

  I just want, I would say on a personal level, I'm so in awe about Assembly, I actually,

• 2:37:11Speaker 02

  once in a while, it never gets old, the speed improvements to show 62X. There are months, on a personal level, I run our internal test suite at work and just see, I'm still in awe of the gains we have.

• 2:37:23Lex Fridman?

  There's a source of joy and happiness for programming for different reasons. But I think one of the greatest happinesses is in optimization of code. And it sounds like you're like at the cutting edge of that.

• 2:37:38Speaker 02

  Whoa, that was cool.

• 2:37:38Speaker 01

  In the community, I want to speak about two people who are wizards of Assembly, right? The two of them are actually working, living in North South Europe, Sweden and Finland. And Henrik Gramner knows so much about Intel x86 Assembly that when we ask questions at Intel about things, they tell, like, why are you asking us, Intel? You have Henrik, Henrik knows better. He knows all the cycles of almost all the S I M D instruction by all the CPU generation. Oh, yes, this is a P4, this is a Nehalem, this is a Core 2, et cetera.

• 2:38:21Speaker 01

  That person is like the best person on Assembly in the world. And he's the nicest person that you've seen, like, very, he arrives, you don't see, he's amazing. And the other one is, is called Martin, Martin Storsio. And he's doing mostly the same on ARM, right? So Neon, right? And iPhones and Androids and so on. And he codes in Assembly on his phone.

• 2:38:52Speaker 01

  Editing it with the crappy keyboard, like virtual keyboard you have, while watching his kids play in the playground, right? Like, like, this is just like wizard level. So those two people are like.

• 2:39:07Lex Fridman?

  Yeah. So apart, when you're programming assembly at that high level, a part of that is knowing the architecture that you program out. Yes.

• 2:39:16Speaker 02

  ARM in particular, yes.

• 2:39:17Lex Fridman?

  ARM in particular. But X, X is, I mean, these are complicated architectures, right?

• 2:39:22Speaker 02

  Yeah, but ARM, um, in some ways, more X eighty six with out of order execution. It's not so bad. ARM, you really need to understand all the different generations of ARM processor because they're all different. There's A seventy two, et cetera, et cetera. And there's the Apple variant, there's this variant, there's that. And you need to write code that works efficiently on all of them. X eighty six, well, broadly speaking, you have Intel AMD, you have sub variants. But generally speaking, there's something fast is going to remain fast on all of the variants. Whereas in ARM, it's a completely much more complicated ballgame.

• 2:39:55Lex Fridman?

  We're taking a nonlinear journey through history here, but we were talking about Michael Niedermayer. And I wanted to ask about this. For a time, there was a split in FFmpeg and a libAV.

• 2:40:13Speaker 01

  Yes, so, um, in open source projects, um, sometimes you disagree, right?

• 2:40:23Lex Fridman?

  You have such a nice way of putting it, yeah.

• 2:40:25Speaker 01

  And the good thing is because of the license, you're allowed to basically do your own, right? And this is normal and this has happened all the time, right? At the point, there was a GCC at the time of GCC two and EGCS, which became then GCC three, right? There is what we call KHTML with WebKit with Blink. It is a sane process. And also like when I want to do a new feature today in VLC, I fork, I do my thing on my own and then I merge back to the community. So there was a split in the open source community on FFmpeg, which become libAV and FFmpeg. And after a few years, well, the community merged back and people moved on. It's a bit.

• 2:41:07Speaker 01

  Drama that is normal in open source community, but folks are even they're important because they they they change the status quo of a community.

• 2:41:19Speaker 01

  Not talking about FFmpeg and any baby here, but the GCC fork made GCC a ton better. Because the some people wanted to change the architecture fundamentally to make it faster. And of course, it's always question of people and so on. But in the end, you realize that FFmpeg today is better than it was before the fork. And now, well, we're back all together, right? And I spent a lot of time, and Kian can say, in the in the community. It's not often, to be honest, very well explained, because a ton of the reasons are not very public. But I think that's that's normal and that's good.

• 2:42:02Lex Fridman?

  Yeah, I mean, you're making it sound really nice, but there there is bad. There's pretty heated battles inside open source projects. I mean, it is a very passionate community and you are kind of in a distributed way have to define the direction of things. So here, looking at perplexity, FFmpeg and libav split in 2011 mainly over project governance, leadership style and development processes, not because of a fundamental technical disagreement. FFmpeg effectively absorbed libav's work while libav withered and most distributions and developers moved back to FFmpeg. Yeah, that was a that was a weird that was a weird from a user's perspective, that was a weird experience. Because, you know, I'm a Linux user, so, you know, whether it's the Bonto and so on.

• 2:42:44Lex Fridman?

  All of a sudden, I think for for for a little bit, Bonto, I feel like, if I remember correctly, switched to libav.

• 2:42:52Speaker 01

  Twelve, fourteen, something like that.

• 2:42:53Lex Fridman?

  Yeah. And then they switched back. I was like, what is happening? So on the sort of you you get to feel the ripple effects of the different internal debates that are happening.

• 2:43:06Speaker 02

  To be fair, on Apple, when you type GCC, you get C lang, like they they did something like that as well, so.

• 2:43:11Speaker 01

  Yeah, so to me, it's like the fork was like heated drama, but most of the development from libav was merged back into FFmpeg, right? So de facto, FFmpeg got a a superset around libav. And so that gave the user, because in the end, we work the user for the users, a larger set of features and a ton of things that were. Discussed, for example, the debate on reviews and on how we push are something that now is completely settled in FFmpeg and is following mostly what everyone in the committee agrees, right? So the facto, everyone who was active on the baby came back in work on FFmpeg

• 2:43:54Speaker 01

  because the disagreements were fixed and in the end FFmpeg is stronger than it was before, right? And I know people love drama, but.

• 2:44:05Lex Fridman?

  Well, my main concern, I understand, and I think looking at the long history, it's all for the good. But I do, I am concerned because there's so few humans that are critical to the success of open source projects that I have seen it be a psychological toll on folks. And, you know, sometimes at least the burnout. So you have these incredible people that are at the core of open source projects. There is a moment that happens because like, what is the motivation of doing it? Ultimately, it's because you're passionate about it and it makes you happy. Then at a certain point, you wake up and it's like, this been a bit too much heat from the drama.

• 2:44:47Lex Fridman?

  So like, at the, at the project level, the project continues and often flourishes. But sometimes there's these individual humans, they're just like, I've had enough.

• 2:44:58Speaker 01

  Yeah, but it's not just about folks, right? So it's very, what you, what you are referring to is one of the most challenging and most interesting part of open source today is maintainer's burnout, right? And AI is a problem because of that. And Daniel Steinberg, which is the maintainer of Kearl, who's probably one of the best promoter of open source in the world. He's, by the way, a member of the European Open Source Academy with me. So I'm very like humbled to be on the same community as him, right? He's against what he called AI slop, right? Because it gives a ton of fake reports or bad reports, bad patches.

• 2:45:42Speaker 01

  And then a lot of maintainers have a lot of burden to maintain the software. And this is straining the the the mind of open source developers much more than folks. And for example, the XZ fiasco was because there was one guy maintaining it and he got basically hammered by two attackers who were asking him questions nonstop at weird times at night to block him. And at some point, he got fed up and says, okay, I can't do that and gave the commit access to the attacker. So burnout in open source community is some things that exist. But mostly it's about maintaining things, right?

• 2:46:24Lex Fridman?

  No, for sure. I wonder how we we help that because those people are so important. The the human beings are so important, the core of these projects.

• 2:46:31Speaker 01

  So for example, now I am maintaining a ton of multimedia and non multimedia library as maintainer. Because the maintainers got fed up, right? Some on video and some outside of video land. Because it's sometimes you need a tough skins, right? Because you get like, it's not really attacks, but oh, this is not working, this is not working. And you feel it personally. And this is also why resources or the the Google fiasco is was a problem, right? They don't realize that in the end you have, you know, it's like the same graph where you see like everything. And it's just like one random open source project that is maintaining the whole internet, you see the one, right?

• 2:47:14Lex Fridman?

  Yeah, this is the meme. I mean, it applies to a lot of open source projects. But this is the all modern digital multimedia infrastructure. And then that thing at the very bottom that everything relies on is FFmpeg. That's true. And then there's usually, you know, a handful of folks that are maintaining that.

• 2:47:32Speaker 01

  And FFmpeg or VLC, right? You have a community of 10, 15 core developers are not the worst open source project. XZ, which is even in more installations, is one person, right? There is one guy.

• 2:47:45Speaker 02

  LibXML is.

• 2:47:46Speaker 01

  Yeah, LibXML, right? There was a big stop. No one is maintaining LibXML anymore, which is like the only library that is able to pass XML everywhere.

• 2:47:54Speaker 02

  All the crazy edge cases of XML under ridiculous circumstances. And they get attacked by security researchers because there's one other crazy edge case that they haven't thought of. I think, yeah, but. The body of knowledge to actually resolve that is massive.

• 2:48:09Speaker 01

  There is one guy maintaining all the time zones for everyone who is in the middle of, I think, was it Nebraska? Or South Dakota? Like, the mental health of the open source maintainers is something that large corporations don't care or don't see, right? It's just like, oh, yeah, I'm just doing an open source report and so on.

• 2:48:29Lex Fridman?

  Some of it is financial, but some of it, and people should definitely support open source financially all across the board. But some of it is also like spiritual on a basic human level. There's something that happens, like with this image of FFmpeg and so much of the internet depending on it, where people almost like talk down to the folks who are carrying these projects forward and maintaining it.

• 2:48:53Speaker 02

  In the security community, they certainly did. That was one of, that was one of the things I think that argument came out is, there was, there was a portion of the security community that, no, these guys write crap code. They need to fix their crap code. I'm like, no, no, no, no, this is a guy's hobby project. You've, you've have a security bot that's gone and found some AI generated stuff. That guy didn't write crap code. It's just an edge case to the ninety nine point nine nine nine nine nine percentile. He didn't think about because it's his hobby project decoding Star Wars games.

• 2:49:23Lex Fridman?

  Forget the hobby project aspect of it. It's, it's just hard work and it's, it's beautiful. And there's like the right approach there is to celebrate people for doing incredible, incredible work. It's just, it's just incredible that humans step up, not getting really paid at first or maybe ever. And then they're doing it out of the love of it. And we need to, like, human civilization runs on people like that. We need to celebrate them.

• 2:49:48Speaker 01

  To give you an idea, I received death threats on, on video line, right? And.

• 2:49:53Lex Fridman?

  You mentioned that to me, like, what, what is, what is behind that?

• 2:49:56Speaker 01

  So that must be. What, two thousand nine, two thousand ten, right? Apple is moving from Power PC to Core Duo. That's probably in two thousand six. And by two thousand nine or two thousand ten, I decided that we are not going to do new versions of VLC for Power PC. At that time, like VLC, we were close to the number one dot zero release. We were four of us, right? Like, just like, no, this is not possible. So I received a death threat with some powder in it, right? It, remember, there was some anthrax threats at that time, right? And it was because I had taken the decision to not maintain the Power PC port anymore.

• 2:50:39Speaker 01

  And of course, it wasn't anthrax, of course, it was some type of flaw and so on. But I received that as a, with a letter of like, uh, you, you're a piece of shit, you should die, uh, Power PC forever and so on. And it was two thousand nine or two thousand ten, right? I was, I was, uh, I was young, I was just like, why? What did I do, right?

• 2:51:00Lex Fridman?

  Yeah, they can break your spirit. It's like, why?

• 2:51:02Speaker 01

  My mother freaked out, right? We had to go see the police and so on. And now, like, I, I, I'm going to say that I'm quite happy that this happened at that time. It, it fooled me a lot, right? I am, I, I can see, I can take a lot of, uh, of hate on me. I'm, I'm okay with it, right?

• 2:51:20Lex Fridman?

  It sucks that that's part of reality, because all the people that love VLC, all the people that love FFmpeg, like me, you know, I, I legitimately,

• 2:51:32Lex Fridman?

  hundred, probably thousands of times in my life, had a smile on my face because FFmpeg made me happy. Period. And how many times did I get a chance to say that? Zero, until I realized there's a Twitter account. And every once in a while, I'm like messaging it.

• 2:51:48Speaker 01

  One of the things I like on the Reddit meme about me, which I don't like this meme for a lot of reasons, but. And someone says, oh, JB is on Reddit, which I am, right? And I say, and say hello, right? And then I got so many people who say, oh, thank you for V L C. And like, I take pictures and then I shared that to, to, to the Signal, to I R C. Uh, yes, we use I R C on, on.

• 2:52:10Lex Fridman?

  I, I, I saw it as a quick tangent. You mentioned I R C is like Slack for old people. So you still use I R C?

• 2:52:16Speaker 01

  Of course.

• 2:52:16Speaker 02

  Yeah, I have it on my phone as well.

• 2:52:18Speaker 01

  Of course.

• 2:52:18Speaker 02

  Every day.

• 2:52:19Speaker 01

  Works fine.

• 2:52:20Lex Fridman?

  Wow, it works fine, huh?

• 2:52:21Speaker 01

  Works fine, yes.

• 2:52:21Lex Fridman?

  You have the power with the crank.

• 2:52:23Speaker 02

  No, but there's, there's no.

• 2:52:25Lex Fridman?

  There's AOL, there's social media.

• 2:52:26Speaker 02

  There's no ads, there's no tracking, there's nothing, like it's a.

• 2:52:28Speaker 01

  The, the, the biggest issue, to be honest, right, compared to Slack is that it doesn't have threads. That's annoying. It doesn't have emojis for reaction. Sometimes it would be nice.

• 2:52:37Speaker 02

  V three, huh?

• 2:52:38Speaker 01

  Yes, V three, but no one does it and you cannot edit your messages.

• 2:52:41Speaker 02

  Yeah.

• 2:52:42Speaker 01

  Right? And the rest, it works perfectly fine.

• 2:52:44Lex Fridman?

  Wait, how do you communicate without emojis?

• 2:52:46Speaker 01

  Well, that's, that's why I said it's for old people.

• 2:52:48Lex Fridman?

  Old people, alright.

• 2:52:50Speaker 01

  And we know, we know emojis, we like, you know, the columns and that shit.

• 2:52:53Lex Fridman?

  Yeah, exactly.

• 2:52:54Speaker 01

  Parentheses, right?

• 2:52:54Lex Fridman?

  Old school. So anyway, hey, you communicate on I R C. What were you even talking about?

• 2:52:59Speaker 01

  Yeah, we were talking about that thread and, and, but having people thanking you. And sometimes they got people who send me a message and, and, oh, thank you for V L C. And I always answer because I want to, to, to validate the fact that you need to thank the, the, the open source community.

• 2:53:16Lex Fridman?

  Yeah, please, everybody listening to this, celebrate, celebrate F F M P X, celebrate V L C, celebrate all the incredible, uh, open source projects, Linux, everything. There's so many, there's so many, and, and you know what? I mean, even outside of open source, just celebrate companies that create software that you use a lot and love.

• 2:53:39Speaker 02

  Celebrate human endeavor, celebrate the human effort to not just build something that's okay.

• 2:53:44Speaker 01

  Yeah.

• 2:53:44Speaker 02

  Build something that is damn good.

• 2:53:47Speaker 01

  Yeah, this is important, right? Like, because as we said, right, we work for technic, we do something very complex for.

• 2:53:55Speaker 01

  The normal people, like, we want our excellence in tech to be useful for everyone. And this is why, like, this is why we work, right? This is why I wake up in the morning is because I want people to use our stuff. Because it's making everyone's life easier.

• 2:54:11Speaker 02

  Want to solve hard problems. Work on something interesting, work on some interesting technical challenges.

• 2:54:15Speaker 01

  We are engineers, we love to build things, right? When I was young, like, very early, I knew I wanted to be engineer. I wanted to do cars, right? Maybe at some point I will go back to cars, right? But this is like, we want to build things that are cool and useful. And they need to be challenging, right? Because you want your brain to turn on.

• 2:54:34Lex Fridman?

  When the two of you first fall in love with programming, with building, with engineering?

• 2:54:39Speaker 01

  When is the first time you program, Kiran?

• 2:54:42Speaker 02

  Microsoft Q Basic, as on Windows three point one and Windows ninety five, Microsoft Q Basic.

• 2:54:47Lex Fridman?

  Wow, what did you build?

• 2:54:49Speaker 02

  Uh, like a multiplication, just counting loops, like ten, twenty, thirty, forty.

• 2:54:53Lex Fridman?

  Nice.

• 2:54:54Speaker 02

  Uh, then I thought I could do everything after that. I wanted, I jumped from doing that to, I want to create a soccer, no, a football soccer video game. And I drew all the, I drew everything out, I was like, I'm gonna do it. And I didn't quite grasp that actually, uh, I think grasp actually is a massive piece of work to jump from basic and drawing some pictures to a video game, but there we go.

• 2:55:12Speaker 01

  Uh, I think I did also basics and then Turbo Pascal when I was, yeah, um, at end of elementary school. But mostly the first time I actually did some serious programming was the first year of, um, you call that middle school, when you're eleven? Um, I was, uh, I lived in Italy for a year in Florence and it was amazing year. And like, the maths teacher told us to work in a programming language called Logo, where you had a turtle that was designing things on the screen and you would turn left and right.

• 2:55:53Speaker 01

  And in the end, we used that to do, uh, very complex programming, because of course you could do things. And, and this changed, like, as I knew I was, I wanted to do things with computers and program.

• 2:56:04Lex Fridman?

  I don't think we quite talked about X two six four properly. We talked about David. Can we return, backtrack a little bit to X two six four, this thing that powers basically all of the video on the internet? So, uh, can you tell me the story?

• 2:56:21Lex Fridman?

  Of X two six four and Karen, you're actually a contributor to X two six.

• 2:56:25Speaker 02

  So X two six four is a video encoder for the H two six four video standard. It dominates internet video, but also other areas such as Blu-ray discs. And Blu-ray discs are interesting because the people that make them really want the highest quality. And there's some really cool high-end films that have been encoded broadcasting and all sorts of other areas. X two six four was a big step change. Cause it kind of happened at the right time as well. A lot of development took place when HD video was coming out. Intel Core T and N nehle M CPUs were getting fast. You could do real-time video, but the most important thing was a key sort of focus on visual metrics.

• 2:57:05Speaker 02

  So industry and academia for twenty years before was obsessed with mathematical metrics. So what's known as peak signal to noise ratio, so mean squared error, logarithm of mean squared error. And that led to tons of issues because mean squared error leads to blurring. Cause you actually want to, you want to minimize, you want to add a little bit of error to everything to reduce the mean squared error as opposed to having a big error. And that led to loads and loads of blurring. So but hobbyists bucked that trend. It was for their own personal videos, mostly anime. So there were two, there were two things they did differently. And there's a big iterative feedback loop with the community. They did some stuff differently. Two, two big things, psychovisual rate distortion.

• 2:57:45Speaker 02

  So using block energy, trying to compensate for human perception when making decisions.

• 2:57:51Lex Fridman?

  So the psychovisual distortion, that's the critical thing. That's the thing. I mean, it's kind of revolutionary, like that we can like rethink. Don't, don't make it like this kind of theoretic thing of compression. Make it all about.

• 2:58:09Speaker 02

  Being pleasing visually to the eye.

• 2:58:10Lex Fridman?

  Yeah, yeah. So compressing in a way that loses the least amount of information for the stuff that matters for us humans.

• 2:58:17Speaker 02

  Yes, exactly. As opposed to what industry, some parts of industry are still obsessed by this, which is mathematical numbers that don't look good in reality. And then adaptive quantization was the other big one where it was biasing bits against complex areas and redistributing them to less complex areas like grass. Grass has some high frequencies, but it's kind of. It's less complex overall compared to more complicated things. And this came around by um Park Joy. So Park Joy was really the canonical sample that was the running around in the park. Yeah, so this guy was really the um.

• 2:58:55Speaker 02

  So this this was created by Swedish television in um the beginning of H D and it was done on film and it was no expense spared in terms of production quality and it was given away for free. This was really and this is the sample really that sorts the men from the boys in terms of it has so many challenges with the trees, with the water, with the grass, with the motion, with the I don't think there's there's still been any any public test sequence as good as that these days.

• 2:59:24Lex Fridman?

  So for people who are just listening, we're looking at a bunch of humans running along a river. You have the reflection, a lot of really high information textures everywhere, the leaves and the lighting playing with the leaves and all of this.

• 2:59:40Speaker 02

  You could show clearly that encoders with high PSNR

• 2:59:43Speaker 01

  Will blur everything.

• 2:59:44Speaker 02

  Will blur everything. And you could see actually I could turn on psychovision stuff. I could turn off the data quantization and it would just look so much better. But your your metrics and these metrics are at the time what at the time were considered so holy.

• 2:59:56Speaker 01

  Yeah.

• 2:59:57Speaker 02

  These are the holy metrics that are untouchable. PSNR is the most important thing.

• 3:00:01Lex Fridman?

  Uh, can you speak to how do you measure psychovision stuff? Like, how do you turn how pleasing a compression is for a human eye into a number? Is that even possible?

• 3:00:12Speaker 02

  That's what that's what Netflix have been trying to do with V math. They said they've used a machine learning model.

• 3:00:17Lex Fridman?

  That's the more recent thing, but back in when Exorcist was being developed, that's by I.

• 3:00:23Speaker 02

  It was by I. It was developers on their laptops. So it's not like even with big companies with professional screens or anything. It's and that was actually one of the goals, which was I don't the developers at the time, Lauren Merritt in particular, is I don't want to test this on a thirty thousand dollar screen. It's I want this to look good on someone's laptop at home.

• 3:00:40Lex Fridman?

  Yeah, brilliant.

• 3:00:41Speaker 01

  And there is another sample, which is um a a sample that is a planet Earth's killer sample that I absolutely love. And you are going to see why, right? Um, it's a ton of birds, right? Flying and and the more it goes, the more there are birds and at at the end, right? It's almost like. You have millions of birds. It's the most complex thing ever to encode, right? And well, you're watching it on YouTube and you see how bad the YouTube encoding is actually, right? Um, and this is like phenomenal to optimize and get um perfect quality in a constant bit rate.

• 3:01:24Speaker 01

  Um, there was a lot of optimization, mostly by Loren also um on anime, right? For a long time, anime was very badly encoded because there was a ton of bending, right? And you see those um those issue and there was a ton of things. Um, so X two six four is like, and today it's still the reference to any encoder, new encoder, AV one, AV two, VVC, HVC, everyone compares to X two six four.

• 3:01:51Speaker 02

  One of my favorite films, um Cinema Paradiso, another engineer who created the Blu-ray, and he showed me the comparisons of X two six four versus others and uh that it's completely different. And I think a bunch, a bunch of guys in the Blu-ray world started using X two six four. Um, I think the big one was Chris Anderson from Warner Brothers. He did the whole French box set with that's a quite like a thing a person on the street actually watches and wants to look good. And so they kind of took a risk in their jobs doing that because they're in a big company. That big company can buy whatever they want. And they said, no, no, no, I want to use this free and open source thing so that things look good for my, my customers and build the best. And to this day, I personally still try and avoid watching the most cinematic films on streaming services

• 3:02:34Speaker 02

  and buy the physical discs because they look, they look good without even having to buy an expensive TV. I think that's the key thing.

• 3:02:41Speaker 01

  And X two six four is yet another example of open source project. It was started by Laurent Emmer when he was at Décolle Centrale Paris where VLC was born. And then you got a generation of people like Loren, like Jason, like uh Mance, like uh so many.

• 3:02:56Speaker 02

  Henrik.

• 3:02:57Speaker 01

  Henrik, uh and this is Anton, and this is where the assembly thing that we use now on FMPEG David and so on was born, right? So X two six four is like amazing project with people who are really all over the world. And I think most of them never met each other.

• 3:03:12Lex Fridman?

  But all of them, according to Karen, are a large percentage love anime. There's several things I've never got into and one of them is anime and I need to.

• 3:03:21Speaker 01

  I watch anime so much, especially at the time, like at the time it was like a lot of anime content doesn't exist commercially, right? We are before Crunchyroll, right? So what happens is usually people who love anime, who take some things, some DVDs in Japan and rip them because there is no commercial offering and some of the people who are what we call fansubbers are basically translating themselves to make subtitles, right? And at that time, you download completely illegally. It was the only way to do that, right? And so all of that was handcrafted and it fits the open source community, right? Because they needed tools to encode, to do fansubbing, right?

• 3:04:06Speaker 01

  One of the most amazing open source projects for subtitles is called AEG Sub and it's a subtitle, it's done for anime, for South Asian, in Japanese languages.

• 3:04:19Speaker 02

  There are weird textures in anime that I don't think you get in real life content. I think that was a key one, which was optimizing these weird textures that you get because anime is not done in a normal fashion.

• 3:04:30Speaker 01

  Yeah, the way you produce is not, you mostly produce it like on screens, right? Since a bit of time and you have all those gradients, right? In colors because they are very easy to produce digitally, very complex to produce in real life. And the subtitles also are very complex because you need to have often the Japanese and then you need to have the diacritics, right? The what we call the oribi, right? Which is the hiragana and the katakana for the kanji. And then because of course you, so that you have the official subtitling, but you also need the English subtitles or the French subtitles because you want to learn that, right? And there is so many things crazy on subtitles and we had like crazy samples on subtitles that we've seen all around.

• 3:05:14Speaker 01

  So this is an important part of the culture, but also because there was no official offering. There was no way of doing that.

• 3:05:25Lex Fridman?

  Can you speak to the difference in H264 and AV1 and then X264? And David, this is this big step. Can you help people understand are some of the streaming sites moving more towards that direction of AV1?

• 3:05:39Speaker 01

  Let's be honest, all of those codecs since MPAC 2.

• 3:05:45Speaker 01

  Video are the same concepts. The same concept about inverse transform, about intra prediction, motion composite, entropy coding. All of them. However, each generation gives you a bump between twenty five and fifty percent more compression for the same quality.

• 3:06:08Speaker 01

  And so you had the MPAC two, you had the Divx area, you have H two six four, which was like changing, right? H two six four improved so much. And then you had more, right? You had HVC, you had VP nine at the same time of HVC. VP nine is a bit similar to HVC in terms of quality compression, but it's royalty free. Because in multimedia, there is ton of patents and the licensing after H two six four became out of hand, right? And could cost hundreds of millions of dollars per year. So it made no sense. So Google did this VP nine and the alliance for open media did this new codec called AV one.

• 3:06:49Speaker 01

  So you can imagine that AV one saves between forty and sixty percent less bandwidth than H two six four for for the same quality, visual quality.

• 3:07:01Speaker 02

  At a given bit rate.

• 3:07:02Speaker 01

  At a given bit rate, right? So that's really like you increase the quality. Either you you set the bit rate and you increase the quality or you set the quality and you decrease your bit rate. But because now you move from from SD to HD and HD to four K and four K to four K HDR, like you increasing the size by like two factor two, three, four, right? So you need to have better compression to keep it in terms of something that is manageable.

• 3:07:29Speaker 02

  It's more coding tools, more bigger blocks, lots more subdivisions in each block. It's just exponentially more complex.

• 3:07:36Speaker 01

  It's more complex because the the encoder needs to search more possibilities, right? So you, for example, one of the things that is easy to understand is To predict a block, a color block to another, you have directions, right? You can go left, right, bottom, up. And then in terms of like the other quadrants, right? And what I call north, northeast, northwest and so on, right? But that's eight directions. Then you can do more direction. You can do sixteen or sixty-nine or one hundred and twenty-eight, right? You can, and every time your encoder is going to spend more time to see, oh, well, these blocks is exactly this one. And those type of tools that you can bring.

• 3:08:18Speaker 01

  And the encoder needs to check which of the tools are going to compress you better. And so I guess that AV1 encoding is two order of magnitude more than H264 in terms of CPU cycle, right? Order of magnitude, right?

• 3:08:34Speaker 02

  Yeah. And as we discussed, CPUs are not getting faster. You're just throwing more cores at the problem.

• 3:08:38Speaker 01

  But also, it's a fact that you encode once and you have hundreds of millions of users, right? So for example, YouTube, a very good example. YouTube encodes almost everything in H264, but the popular video gets encoded in AV1. Because it costs more, of course, to encode, but you encode once and you send that to millions, right? So it's a trade-off between encoding time and complexity and CPU usage on the server side and on the client side. Because at the end, if you're distributing a video to hundreds of thousands of people and the size is half of the other, then it's better, it's better for your batteries, better for your modem, et cetera, et cetera.

• 3:09:19Lex Fridman?

  So we can lay out, let's say, the top five codec container combos would be H264 inside MP4 containers, AV1 inside MP4 WebM containers, ProRes for nonlinear editing.

• 3:09:40Lex Fridman?

  Uh, inside MOV containers. So for people who don't know, I guess ProRes is.

• 3:09:45Speaker 02

  It's Apple's codec for editing, originally for Final Cut Pro. And it's designed to be faster decode, faster seek, because an editor will need to move very quickly. So it's a different use case to the distribution element.

• 3:09:58Lex Fridman?

  There's no or very minimal temporal compression.

• 3:10:01Speaker 02

  There's none yet. There's none in ProRes. So you can cut, so you can do cuts.

• 3:10:05Speaker 01

  This is what we call intra-only codecs, right? So I'm going to explain quickly what is IPB frames.

• 3:10:12Lex Fridman?

  Yes, please.

• 3:10:13Speaker 01

  So I-frames often key-frames, but is complete frames. It's like an image. It's a JPEG, right? You have, you can start, you see everything, right? Um, and then you, the next image can be a P-frame, which is a predicted frame. So you take some part of the previous image saying, well, I need the block five and seven and forty-two. And you replace it and then you just give the extra information, right? But that means that in order to decode this P-frame, you need to have access to previous I-frame, right? And then, of course, you have more complex one, which are B-frames, which are B-predicted.

• 3:10:54Speaker 01

  Um, frames which can be depend on different type of frames, some in the past, some in the future. Um, and so ProRes is an intra-only codec. For the people who can see, this is a very good one, right? So I-frames are complete frames. Um, uh, P-frame basically, uh, depend only on I-frame and B-frames can depend on in front.

• 3:11:17Lex Fridman?

  In this, uh, GOP group of pictures, I think the default for actually FFmpeg for H. six four

• 3:11:28Lex Fridman?

  is, uh, like two hundred fifty frames, something like this.

• 3:11:32Speaker 01

  Yes.

• 3:11:32Lex Fridman?

  And to me, it's just, it's like magic, like you could predict, you could have a complete frame every.

• 3:11:39Speaker 01

  Several seconds, that means.

• 3:11:41Lex Fridman?

  Several seconds. And then you could still, you could have this chain of predictions you make. And the fact that you can, the fact that somebody like me can, can use FFmpeg to compress something and not notice that the result still plays back smoothly is, is like magic.

• 3:11:56Speaker 01

  You can even have. And we use that in tons on Kiber is what we call intra-refresh, where basically it's, there is no iframe.

• 3:12:07Speaker 02

  You have one at the beginning and never send an iframe.

• 3:12:10Lex Fridman?

  How does that work?

• 3:12:11Speaker 02

  You build up an iframe gradually across as the stream continues.

• 3:12:15Lex Fridman?

  Ah, so you refresh certain parts of the image.

• 3:12:18Speaker 01

  But so you never have an iframe. This is intra-refresh that we use, right? But for me, for me, the biggest mind blown when I started was the B-frames.

• 3:12:27Lex Fridman?

  Yeah.

• 3:12:28Speaker 01

  B-frames means be predicted frames can depend on frames that are coming in the future.

• 3:12:36Speaker 01

  That means that in order to decode this B-frame, you need to wait for the next frame that is dependent, buffer that decode that one so that you can decode the B-frame, right? So the, the, the way you decode the frame, the decoding order is not the same as the display order, right? That means the encoder needs to be very clever and decide that, well, you know, I'm going to depend on things like in the future. So this is like mind blowing.

• 3:13:05Lex Fridman?

  Yeah.

• 3:13:06Speaker 02

  The fact it works so smoothly every day is, is kind of miraculous in some ways. It, it works so, you can have a stream that works across the world on their decoder versus one in the US versus one here of different manufacturers and they produce bit for bit exactly the same material. That's quite remarkable and do quite complex things and getting more and more complex and still be bit exact. There's a lot of work that goes into that.

• 3:13:31Lex Fridman?

  There's a lot of knobs you can control in this whole process. There's a lot of really fascinating parameters that I've gotten to know more and more over the years that FFmpeg gives you complete access to. Maybe you could speak to some of them. So first of all, like obviously we can lower the resolution, we can lower the frame rate, we can use different kinds of codecs as we mentioned from H. six four to AV one. There's ways to tune the trade off between bitrate and quality as we've kind of spoken to, you know, you could do constant bitrate, you can do constant quality, say RSCQ, QP. We can do the longer or shorter group of pictures GOP that we mentioned. I mean, all that kind of stuff. It's crazy. Number of B-frames.

• 3:14:13Speaker 01

  Yeah, what is crazy is that. A ton of people's job is to optimize those parameters, right? A ton of people that you see at YouTube, at Netflix, at Meta and so on, they're not writing codecs, they're just like finding the right parameters for the file they have, for the format they have, right? Because like something that is for a movie or something that is user-generated content from your phone or a screen recording or something that you're going to video edit, you don't want the same things. And there are thousands of people whose job is just to optimize all that.

• 3:14:50Lex Fridman?

  Yeah, they're wizards. Hats off to them. Uh, YouTube, like to deliver all the streaming sites, actually, to deliver at scale. And like YouTube is really magical because it's not just doing like what Netflix it does, which is one way, like broadcasting type thing. It's also has to upload videos from all the places. So they're also doing encoding at scale. For videos that are going to be watched by like five people and it still has to deliver them like at a moment's notice, no delay, nothing, no late, very minimal latency. And and also serve it in all different resolutions.

• 3:15:33Lex Fridman?

  Like YouTube is basically the web version of VLC.

• 3:15:39Speaker 01

  Well, actually, it's funny because like Google Video, which was something they did before they acquired YouTube, was actually using the VLC plugin so that you could run VLC inside the web browser using the ActiveX plugin. And so it worked in Internet Explorer and you were actually running VLC inside your browser, which is funny because today we have the opposite, where we have VLC Web Assembly, where we compile all VLC NFF MPEG to decode, to run VLC in type of, inside the JavaScript virtual machine with Web Assembly.

• 3:16:17Lex Fridman?

  Okay, there's this legendary story that you pointed me to, that it was discovered via WikiLeaks release of Vault 7 documents the CIA was using a modified version of VLC to basically try and trick people, what, to steal their data?

• 3:16:36Speaker 01

  Yes, exactly.

• 3:16:36Lex Fridman?

  So can you explain what the heck happened?

• 3:16:40Speaker 01

  So this was a surprise, right? Because at some point WikiLeaks mentioned some documents. There were a few ones with something related to blurries and VLC. But the most interesting one was the CIA Vault 7, which, if I understand correctly, was the CIA had like a custom version of VLC, where they had a specific plugin. Yeah, exactly. This is like, we had to write a press release on that.

• 3:17:07Lex Fridman?

  Video Land wrote a press release saying, "The only safe source for getting VLC Media Player is the official Video Land website." I mean, I suppose that's a security vulnerability for basically any pieces of open source software. Somebody can trick you.

• 3:17:22Speaker 01

  To download in a fake website or targeted advertisement. That was a targeted advertisement to watch a specific file you need to watch with this custom version of VLC. And it was the normal binaries of VLC, except they added one DLL. I think it was PS API dot DLL, which was basically reading your document folder, encrypting that and sending that. And the thing is, this is very clever, to be honest, because once you're watching a movie, you're going to do that for two hours and you're not going to touch your computer. And sometimes it's normal because it's HD that your fans are going up and say, and there is turn off the CPU usage because you're using VLC, right? That's normal.

• 3:18:05Speaker 01

  But the thing is, what you don't see is that actually a powered version of VLC that is used by CIA. We had exactly the same problem with Chinese hackers that were targeting Indian people. And that got VLC banned from India until I had to fight in courts in India. The Indian government to unban VLC. They didn't use VLC, they took just one DLL because we signed the DLL correctly. And they used that DLL to do another program. So you had a VLC.exe and was calling libVLC, but it was calling it into a fake one. And they used that to target.

• 3:18:48Speaker 01

  There is not much we can do actually to block those type of hacks.

• 3:18:52Lex Fridman?

  Yeah, I think people should, for all open source software, for all software in general, people should pay attention where they download the thing.

• 3:18:59Speaker 01

  Yes, because that means that they were not downloading it from our website.

• 3:19:03Lex Fridman?

  Do the search engines help you?

• 3:19:05Speaker 01

  No, they don't.

• 3:19:05Lex Fridman?

  Just to clarify, because you can, you know, to prevent threats from people manipulating SEO to get up there on the links.

• 3:19:12Speaker 01

  Absolutely not, right? We have a big issue for like more than 10 years. He said there is a fake version of VLC in Germany that was reported for now for 12 years. And Google basically decides to not, they know what's in it. But the binary is too big for their virus analyzer to analyze it. And so, well, if you're in Germany, you can go to a website that is a fake version of VLC with a custom installer. And it's very popular in Germany because their website is in German and Google's mentioned that before Videoland. And the weirdest thing is that it doesn't do anything on your machine for three weeks. Because that's how they they do the detection.

• 3:19:54Speaker 01

  And after three weeks, there is a small program that is a service that is installed at the same time. That wakes up after three weeks and it starts downloading spyware and adware. And Google knows about it, they've decided not to do anything. The guys use dark SEO in Germany to to to do that at some point.

• 3:20:12Speaker 01

  And this is very damaging, right? Because one of the things that they are downloading is actually something that is replacing your ads inside your machine, right?

• 3:20:21Lex Fridman?

  It's actually quite surprisingly effective. Whoever is doing it with Twitter and X, with X, I'll get emails about your X account has been hacked. And however they phrase it. It gets me to like, at least click on the email, not to follow the thing. And then you're like, man, whatever they're doing with the psychology to try to trick you, they're quite good.

• 3:20:45Speaker 01

  There is a security version of VLC, right? You receive an email saying, hey, there is a security version. He updates on VLC, he think about updating right now because it can hack your computer. You come, it's a website that looks decent and you download, it's a new version of VLC. Great, you don't know, a month later, you're hacked, you have no idea, you're part of a botnet.

• 3:21:03Lex Fridman?

  Yeah. So make sure wherever you're downloading stuff, it's legitimate. I'm part of the botnet. Speaking of which, so you've mentioned that VLC sandbox is something you're working on. And it's actually something quite challenging. Why is it important? Why is it hard?

• 3:21:21Speaker 01

  So VLC is a core with around 500 plugins, right? One of them is FFmpeg, but we have, we support so many other formats. We support new protocols, we support new filters, we support weird architectures. And in this release of VLC, you have modules that are going to call your drivers, right? Mostly the hardware decoders, which are going to call your Intel, your Nvidia, your AMD driver. And all calling FFmpeg, right? And there might be a security issue. There might be a security issue in the shader, there might be a security issue in VLC, in FFmpeg, that is going to basically crash.

• 3:22:05Speaker 01

  The issue is that you're running VLC, like every other program, like Adobe, right? You're running it on your machine and it has access to all your documents, right? So the idea is to be sure that you do a sandbox so that we can protect from ourselves. Because inside the VLC process is running some code that is not even ours. Either it's open source for other projects that we integrate in VLC, or it's your GPU driver or something that is provided by someone else inside. And so when we crash, we want to not allow people to do bad things, right? Because one of the common way of hacking people is to crash a program, very often done with a web browser, very often done with PDF files.

• 3:22:50Speaker 01

  Less often with multimedia, but that could happen. And when you crash, you launch something on the on the machine of the person. Could be a ransomware, could be a botnet, right? So, security of desktop application is important. On mobile, it's a bit different because most of the mobile applications are running on inside their own sandbox. But for VLC, we could run it inside one sandbox. But the problem is that we need access to so many things that is basically we we would do. We would have all the permissions, right? And so if you have a sandbox and you put some holes everywhere, it defeats the purpose, right? So what we are trying to do and we're actually doing is splitting VLC into several processes.

• 3:23:33Speaker 01

  One is decoding, one is demuxing, one is filters, and all of them run into their own sandbox. So that the whole VLC, a part of VLC crash, like Chrome crashes on some on some tab, right? It crashes a crash, but it did not crash the whole program. And this is what we're trying to do. And it's difficult because it's a sandbox that needs to sustain gigabits per seconds of of mem copies. Now, it's not a website which is five megabytes or 10 megabytes. We're talking about hundreds of megabits per second. So this is why it is quite challenging. And this is a research topic that we we are working on in order to have multimedia player that is secure.

• 3:24:16Lex Fridman?

  This is all the kind of stuff you have to think about when millions of people are using. You you mentioned something somewhere where like all the different features of VLC. When you have that many people using it, somebody will use every single feature and they will tell you about it.

• 3:24:33Speaker 01

  Best feature in VLC is called the puzzle filter, right? So you click the puzzle filter and it transforms your video into a jigsaw puzzle, right? And you can click. And move the pieces, right?

• 3:24:46Lex Fridman?

  Yep.

• 3:24:47Speaker 01

  And it's very, very useful when you're watching a French movie, right? You're bored and because it's like, like very long things or, or a love triangle, right? We've seen that so many times, right? But, but you need to watch it because someone, your wife or told you to do that, or, or your, your boyfriend told you to do that. Right? So you're doing that, right?

• 3:25:05Lex Fridman?

  Yeah.

• 3:25:05Speaker 01

  And you can click and move the pieces around.

• 3:25:08Lex Fridman?

  Yeah.

• 3:25:08Speaker 01

  It's absolutely useless, right? Like, who cares about that? First, it was done by a, a maths teacher in high school in south of France to teach his students about Bezier curves, which is something that everyone should know about, right? It's very useful. But the code was clean, so it got in VLC. It was merged in two thousand and ten. Five years later, I received an email saying, hello JB, I have a problem with VLC. The puzzle is too simple. And I was just like, what? And yes, the puzzle was in the UI maximum by sixteen by sixteen, right? Only two hundred and fifty-six pieces. And he says, I'm sorry, but in a movie, I love puzzles. This is too, too simple, right? So there is a comment of me, you can take it online,

• 3:25:52Speaker 01

  which is JB changing that the dimensions are two hundred and fifty-six by two hundred and fifty-six. But my point is, so many use features are used by a few people, right? There is a way to watch VLC movies in common line without any UI, right?

• 3:26:10Lex Fridman?

  I saw that you can do ASCII.

• 3:26:12Speaker 01

  ASCII art. Is it useful? Very useful. Imagine you're debugging, imagine you're debugging a multicast network, right? You have thousands, very complex, very complex networking stack, right? You can SSH to all of the routers and put VLC on it with no UI. And you're going to see whether it's black or it's not black, right? So you see if, or it's all green or not all green, right? So you can see, right? People don't realize there is so many things in VLC that are useful. Um, and they are, they have users because once you have hundreds of million of users, you have people who use every feature.

• 3:26:53Lex Fridman?

  I would love to sort of zoom in and, and, and talk a little bit more about the distinction between kind of downloading a file and watching it offline versus streaming. So the, the complexities, the challenges of streaming. Is there something we could say about what it takes to? Uh, stream files, because we've been talking about codecs. And I think a lot of that implies encoding and decoding

• 3:27:21Lex Fridman?

  without the having to communicate over the network.

• 3:27:24Speaker 01

  Sure, sure.

• 3:27:25Lex Fridman?

  So can you elaborate like what's required to do over network stuff?

• 3:27:29Speaker 01

  Yeah, but it is less complex than it seems compared to everything that we've talked about. Especially because the most complex thing is not about streaming in terms of streaming services, but it was was what was done to actually broadcast through satellites. Because in in in most of the modern broadcasting services, you can pause and you can go on. But when you're sending live streaming, whether it's broadcast or live for streaming services, which are live, this is much more difficult because you need to encode in real time. When you go on a satellite, you have a specific size of the link, right? You cannot have a burst of bandwidth even for a second, right?

• 3:28:13Speaker 01

  Because you don't have the space for that in your your total file. However, there is different types of challenges, which are interesting challenges. But I think they are less complex than the one we've seen with late 90s and early 2000s about broadcasting and streaming through satellite.

• 3:28:30Lex Fridman?

  They're different. There are control systems challenges, whereas whereas some are more mathematical. I think that's the difference.

• 3:28:36Speaker 01

  In the streaming world, what you have is called what we call adaptive streaming. Because the difficulty, and it's not really a video problem. It's mostly a CDN problem is that you might have too many people watching the same thing at the same time. And it's congestion of the network, right? So your player has difficulty downloading things fast enough to play them. So what happens is that locally, the player is going to read a lower resolution of it. But There are some very clever algorithms to do that, but most of it is quite basics, to be honest.

• 3:29:12Lex Fridman?

  Even on the buffering side is pretty basic.

• 3:29:15Speaker 01

  Yeah, you you you you start to download a a segment, what we call a segment, and then you you time, right? And if you if it takes more than fifty percent of the time to download the segments, you go down to, right? And the difficulty is more about when do you go up in bandwidth in in quality? But this is not very complex to do. When you encode, you're going to encode seven resolutions, right? And and you're going to give the bitrate. The difficulty is to have your encoder give the same bitrate, but it's not as strict as it it used to be, so.

• 3:29:47Lex Fridman?

  Uh, probably YouTube has to figure out how the human psychology side of that, like, how pissed off do you get when it's like very low bitrate? And uh, how long should it wait before it increases the bitrate even though the connection is better? Because maybe the the changes in the bitrate is what like affects you psychologically.

• 3:30:11Speaker 02

  I I I think actually the the interesting one is the audio.

• 3:30:13Lex Fridman?

  That's true.

• 3:30:14Speaker 02

  The you can kind of notice when they move from um full fat AAC to the um, there are compressed versions of AAC that use spectral band replication. You can kind of see it goes a bit tinny and that up and down is very jarring. The video side is a lot smoother and there's less notice. It's really the audio you can you can definitely you can definitely feel it from when it's moved you from a different audio profile to one or the other. I don't know, we're surprisingly tolerant at skipping audio glitches. I I'm surprised people I know who are not video engineers, how tolerant they are, how tolerant they are to watching sports at thirty FPS, for example, whereas it should really be sixty. The world is a lot more tolerant to that, but audio people are very, there's it's an immediate feedback mechanism of all.

• 3:30:57Speaker 01

  If you hear a glitch, you realize it directly.

• 3:31:00Lex Fridman?

  Yeah, I get to fully realize that, I suppose. But one of the things I'm afraid of when I listen to audio more and more, that I get to notice every single tiny detail. And that you can over obsess when uh people people in general are able to kind of. Kind of blur their consumption, they can, they can look past certain imperfections.

• 3:31:21Speaker 01

  But then, when you combine like um an event that is, for example, a sport event, there is probably going through satellite or somewhere else and goes to a central place for encoding. And then you need to encode this older resolution in real time, you don't have time for QA. You need to push that to CDNs, you need to add probably DRM uh for protection. You need to have that over a ton of different um devices, then yes, it is complex. Um but and also like you're in the web browser or in very much different devices that you use for t- television, where you had like a defined setup box or cable box that, that you know where you control end to end.

• 3:32:05Speaker 01

  Um so it's a challenge, but it's less, I, I think the networking part uh while you agree to have ten, twenty seconds of latency, I don't think this is very difficult.

• 3:32:18Lex Fridman?

  Speaking of networking and latency, so your new effort, as we mentioned, is Kyber, which is uh aimed at ultra-low latency. As you say, every millisecond counts and you're applying that to remote control machines like robots, drones, computers. Can you tell me about it?

• 3:32:38Speaker 01

  Sure, um if you start from where we used to be, right? You used to use FFmpeg to encode files, right? And then we used FFmpeg and VLC to encode in streaming services, right? And then you need to go lower and lower. And the question was where up to where we can, can we go? Um and this question is very important because there are many use cases where you need to be fast. And it's when you have. Feedback interaction, right? We are not just listening to something. You're actually controlling it, right? Because and that's the biggest difference that compared to what we've done so far is that I I need video to have a feedback on something that is happening in life.

• 3:33:21Speaker 01

  Whether it's a drone flying, whether it's controlling a humanoid robots from distance, whether it's controlling a rover, whether it's playing a video game in in the cloud gaming, because this is what I did on a previous job, right? I was CTO of a cloud gaming startup. And this is a very interesting topic because you push to the limits, the network. You need to be to care not about the quality like we've done on video. And we've talked about with X two six four. You care about latency because a milliseconds is meaningful when you're controlling a car, right? Well, you we've you've seen you've used Waymos, right?

• 3:34:04Speaker 01

  When Waymos don't work, and that happens even if one percent of the time, there is someone that is basically a remote controlling that. And this is exactly the stuff that we're building. It's a really SDK platform to do end to end control of machines.

• 3:34:23Lex Fridman?

  So the this comes up quite a lot in a lot of different contexts in robotics. So obviously teleoperation, teleop is becoming more and more important. And including for training robots via machine learning.

• 3:34:38Speaker 01

  Yes, and what we do is a bit different from everyone else is that we take only one socket, one connection, which is a quick protocol based on UDP, which is interesting because it's done for low latency. It doesn't have two of the what we call the TCP end offline problem and HTTP end offline problem. It's offered by default, but on the same wire, we send multiple streams like multiple track. We send audio, we send video, but we also send the comments, right? Mouse, keyboard, gamepad and so on. And we do that while maintaining coherence, right? Synchronization, because what people don't realize is that all the clocks actually drift. And when you're controlling a robot, a robot is going to have like two cameras,

• 3:35:23Speaker 01

  five cameras, 10 cameras, a ton of captors, GPS and so on. And if you want to train correctly your robotic AI model, you need to have all those. That are in sync and currents. And what we've done, and it's all the stuff that we learn on V L C in broadcast in real time, and M P E C T S that Kiran knows well, is that we account for clock drifting. And so when I record a Kiberstream, a robot, I am sure that it's going to be predictive in the way you played back. And so when you're going to do recording and training of your A I model, you need to be sure that every time you retrain based on the data, the data is going to stay coherent. And clocks actually drift.

• 3:36:07Speaker 01

  Like, the existing solution works with one camera. Once you're going to have five or six, it's more complex.

• 3:36:13Lex Fridman?

  So you want to make sure that the visual snapshot perfectly matches the time it actually happened.

• 3:36:21Speaker 01

  Exactly. And also, if you're going to control, right? I do something on robot. I need to be sure that it is actually happening at that precise time, right? And so we have on the server, which would be a robot, a time of like re timestamping mechanism accounting for clock drift for that, right? So that's one of the use case of Kiber to control robots. I also think like remote drones, remote, whether it's defense or non-defense, remote cars, remote submarines. There is many places in industry or remote surgery where the expert cannot go everywhere the machine is because it's dangerous or it's too costly, right?

• 3:37:01Speaker 01

  So you you allow people to have machines next to you, right? The goal of Kiber is to make distance disappear. Because it's either projection of skills or projection of power, right? So imagine, we are all like within the meta Reba and everyone else, right? You need to stream there, right? Because you're not going to run anything over there, right? So you need G P U power that's on a cloud, on a phone to stream that. And so all of these use cases needs to be not about extremely low latency, but real time latency for video. And so that means you need, we're toying with the encoders so that the encoders encode a frame in four milliseconds.

• 3:37:41Speaker 01

  And and Kiran, with his company, also goes under those type of latency. Because you need to optimize at max the

• 3:37:54Speaker 01

  Local latency, right? Because it's decoder, the encoder and so on. Because this time is going to be added to your networking time. So and it's not just about low latency. It's also about like reliability. We do clever things like forward error correction, right? So forward error correction is you over transmit a bit of data, right? A few percent. And while over transmit, you're allowed to lose some packets. Because all of that is very difficult over a internet network, where you're going to do things very far away. And if you check that all packets are delivered, you add a ton of latency.

• 3:38:37Speaker 01

  If you don't want latency, what we do is that we over transmit some data that you can reconstruct on the client side when there is things that are broken, right? So and we, a few, a few days, weeks ago, we were doing the demo around Las Vegas for the CES about, we had a rover that is fully three D printed. It's very simple. It's a car, right? It's a small car with a telescopic arm. And it was actually Control from France, right? And the video was with a webcam and a very small server, right? A small, a small PCB was basically running and send that to someone that is on the other side of the planet. And so there is so many use cases.

• 3:39:21Speaker 01

  You can also think about having AI who are going to control many drones and so on. And technically, we need to be amazing in video. We need to be amazing at networking. We need to care about any milliseconds in networking, in encoding time, in decoding time. And also you need to integrate very low level.

• 3:39:39Lex Fridman?

  So sync everything together well. But how, like, what kind of latency can you get to? Like, why, when you say milliseconds, what, what's the goal?

• 3:39:47Speaker 01

  So my goal is four milliseconds glass to glass latency.

• 3:39:51Lex Fridman?

  What's glass to glass mean?

• 3:39:52Speaker 01

  So it's easy, right? You have a computer which is running a program, right? Probably a video game. And this one is actually running, right? It could be, it's an example of a robot, right? And you have the replicate that is done through the network. And you want, if you take a 1000 Hertz camera, you can take a picture and you want that to be at four milliseconds. Four milliseconds means 240 Hertz.

• 3:40:19Speaker 01

  Right? Yes, not. So far, we we achieve seven milliseconds from a Windows to Windows or Windows to Mac. And if you look in the timing, most, there is around three dot five milliseconds inside the Nvidia hardware encoder and around two milliseconds on the Intel decoder, right? So like the encoder plus the decoder is already six milliseconds, right? So in order to go down, we'll need either to have some other type of codecs

• 3:40:51Speaker 01

  or some better encoder that are faster. But four milliseconds is would be the grail. That's pretty nuts. I love it, though. I I don't think anyone's ever achieved that, right? That's fast. You can achieve that with custom hardware, with SDI, with professional hardware. But I want that to work over the internet. I want that to work with any robot where you're going to have a small Jetson Nano in it or or N one fifty, right? I want that because there is going to be millions of robots. Or drones are just rolling robots or flying robots or swimming robots, right? It's just you, a machine that you control. And in order, either you need to teleoperate them

• 3:41:36Speaker 01

  or when everything will be fully autonomous, you need to tele-observe them, right? You need to check what's happening. Yeah. And in my view, in the future, like all those remote cars will be tele-observer-observed by an AI model, which is just going to say, well, everything is go good. And when it's not good, say, hey, there is a problem. And then you have an operator, right? And this is going to be about safety, right? When you have your humanoid taking care of your grandma or my grandma, I want to be sure that everything goes well and I'm not in those type of horrible scenarios where the robot is dangerous. Or when I'm driving, I I want like the car to to stop when it should stop. And if needed, someone takes care of that, right? And so there is so many case scenarios about real time.

• 3:42:21Speaker 01

  And so the goal of Kyber is to make real time. Control of machine, distance disappear.

• 3:42:28Lex Fridman?

  It's incredible. And some of the same technologies, some of the same ideas we've been talking about is is connected to what you're doing.

• 3:42:35Speaker 01

  And for for me, it's amazingly challenging, right? Because I would say that on video, I'm doing okay. But networking, I have so much more to learn, right? It's about like congestion protocols, bit rate adaptation in real time. But it's it's quite funny. And and so I created this project and and we we are foundries in the US, of course. But it's open source, right? This is important, right? Like, we've not said that, right? But everything on Kyber is open source.

• 3:43:03Lex Fridman?

  How do you make money?

• 3:43:04Speaker 01

  It's a dual license, commercial and AGPL, right? You remember what you said about about licenses? Basically, if you want to use Kyber in your product, you must have your full product open source. If you want to use this amazing technology, but not open source, you pay the commercial license, right? So the small people or the the hobbyists and the very small guys who want to do that, they can use the technology. They build something that is open source and cool. And if you're a large company, you're going to have the support, all the IP, the right to modification and so on. So, yeah, it's really cool. And and and also, I'm building robots, and I love that, right? Like, the the rover we have is 3D printed.

• 3:43:48Speaker 01

  We are finishing a demo where it's an actual wing, right? Like a type of drone wings that is also fully 3D printed. We are trying to do a sailboat that is 3D printed. And and and we'll work on some humanities. Of course, they are not going to be very good robots, right? It's not our job, but we're here for everyone to make robots. Cool.

• 3:44:10Lex Fridman?

  You're talking to the right guy. I love robots. There's a bunch of them upstairs. And teleop is going to be really, really important, especially as the number of robots goes across the world. So 100%. Let's talk about the future of.

• 3:44:24Lex Fridman?

  Multimedia FFmpeg, VLC, but some of the codecs, we didn't really mention AV2. So can we just lay out what is AV2? What is the hope for it? What is H.265, H.266?

• 3:44:38Speaker 01

  So AV1 is this codec that is done by the Alliance for Open Media, right? Where there is Google, Netflix, Amazon, Apple, Videoland, where we try to make a royalty-free, very good codec, right? And now it's being deployed. But actually, the codec was finished in 2018. But a codec takes years to be used in wide scenarios, right? So AV2 is the next generation of this codec. It's 30% better, right? So if you keep the same quality, you got 30% bandwidth reduction compared to AV1.

• 3:45:16Lex Fridman?

  What's the connection with the David and AV2?

• 3:45:19Speaker 01

  We are going to do a David 2, right? That I call David because deux is two in French.

• 3:45:26Lex Fridman?

  Well done.

• 3:45:26Speaker 01

  And you have to know that David is an actual, what we call recursive acronym, right? Because it means D, David, is an AV1 decoder, right?

• 3:45:36Lex Fridman?

  Oh, nice, nice. I didn't even think of that. And people should know that David is spelled with a one.

• 3:45:42Speaker 01

  Yes, it's, and so David 2.

• 3:45:45Lex Fridman?

  It's gonna be spelled with a two.

• 3:45:46Speaker 01

  It's going to be D A V two D. Sorry, I don't know how you pronounce that. And again, we did a demo at the CES of VLC running the first demo of AV2.

• 3:45:57Lex Fridman?

  So can you clarify to me the specification of AV2? And then the encoding and the decoding?

• 3:46:05Speaker 01

  Sure. So the specification is like the document who explains how the codec is supposed to work, right?

• 3:46:12Lex Fridman?

  And that's really AV2.

• 3:46:14Speaker 01

  That is AV2, like H.264.

• 3:46:16Lex Fridman?

  Yeah.

• 3:46:17Speaker 01

  Right? Then you have an encoder. The current encoder is called AVM. And there will probably be other encoders, probably one called SVT AV2. And those are the encoder. The same way X.264 is an encoder to H.264. The same way that X.265 is an encoder for the H.265 codec. And the decoders for AV1 is David. The decoder for AV2 is David 2. The decoder for H.264 is FF, H.264, inside FFmpeg.

• 3:46:53Speaker 01

  The decoder for HEVC is FFHEVC inside FFmpeg. And there is a next generation codec from the MPEG world after H.264, H.265. There is one that is called H.266, also known as VVC.

• 3:47:12Lex Fridman?

  So HEVC is H.265, VVC is H.266. Why is H.266 super sexy and so much better?

• 3:47:23Speaker 01

  So the question often we have is why are there two names? Because most of the time, it is a conjunct work from the ISO world and the ITU, which is the International Telecommunication Union.

• 3:47:37Lex Fridman?

  These are these two regulatory bodies.

• 3:47:39Speaker 02

  One is a private entity and one is the United Nations.

• 3:47:41Lex Fridman?

  Which one is the private?

• 3:47:42Speaker 02

  ISO is private.

• 3:47:44Speaker 01

  In theory, H.264 is MPEG four part 10, H dot two six four slash AVC.

• 3:47:55Speaker 01

  And this is a full name.

• 3:47:56Lex Fridman?

  Nice.

• 3:47:57Speaker 02

  So it's the concatenation of the ISO name and the ITU name.

• 3:48:00Lex Fridman?

  Yeah.

• 3:48:01Speaker 02

  Even though they work together, this is, this is politics, historical, you know?

• 3:48:06Speaker 01

  And for HEVC, it's MPEG eight, H.265, HEVC.

• 3:48:11Lex Fridman?

  Got it.

• 3:48:12Speaker 01

  And there is H.266, which is also named VVC.

• 3:48:15Lex Fridman?

  Is there a high level thing to say about the improvement?

• 3:48:18Speaker 01

  Thirty percent each generation is a best summary.

• 3:48:22Lex Fridman?

  This is true both for the AV codecs and the H.264, five, six.

• 3:48:27Speaker 01

  So the professionals who are listening to us are going to kill us because they say, no, it's thirty five percent, forty five percent, fifty, blah, blah, blah. But globally, you need to know that HEVC is thirty percent better than, than H.264. H.266 is thirty percent better than H.265 because there is so many cases and so many scenarios. For example, there are cases, especially for screen recording, where the gains are.

• 3:48:52Speaker 01

  Humongous because you arrive, you have the right tool that is done for that. And so for a specific video, a new generation is going to give you seventy percent gain or eighty percent gain. Yeah. But there used to be a ton more codecs, but now the two main codecs for transmission are the H two six four, H two six five, H two six six. And the other is AV one, AV two. And I guess the the major difference would be the cost of encoding. Yes, and the royalty of the patents. And this is the reasons why you see the AV version of codecs is because they try to be as royalty free, which means no cost for the patents as much as possible.

• 3:49:32Speaker 01

  Because what you need to know, and we've not talked about that so far, is that multimedia is what we call a patent minefield. There is two places where it's you have the most patents. It's everything related to three G, four G, five G, RF and multimedia. Because it's very mathematical and you can get great gains and so on. So Google and Meta and Netflix wanted something where it was royalty free. There are people who say that they have patents outside, but they are fringe patents, right? So it's mostly true that is patent free.

• 3:50:06Speaker 02

  Or you should extend patent checking was done as part of the standardization process in AV one, AV two, whereas patents are not even discussed in the MPeg world. Patents are off topic completely.

• 3:50:21Speaker 01

  Can you educate me at the patents side? So usually, so MPeg does a format, right? And then there is, everyone comes around and say, well, I have all those patents or formats. And they do usually a union called, was called MPeg LA, MPeg Licensing Association. And you put all the, all your patents in and then you ask everyone who's using this format to pay for it. Wait, can you elaborate? What does it mean to have a patent of a codec? Why is there many patents? Uh, imagine I'm doing something where I'm going to, instead of doing blocks which are square, I'm going to do rectangles, right? Oh, so every idea. Yes. Somebody patents it. Yes.

• 3:51:02Lex Fridman?

  Oh, man.

• 3:51:04Speaker 01

  Yes.

• 3:51:04Lex Fridman?

  Oh, man. People and the, how many lawyers are?

• 3:51:09Speaker 02

  I mean, it pays for a lot of lawyers, right?

• 3:51:11Speaker 01

  The biggest issue is not the following, right? Because at the time of H two six four, the patents were, let's call it like sane. But there was, there was so much money in that, that for H V C, a lot, there were a ton of things that were pushed inside the specification, which are not useful in ninety nine dot nine percent of the time. But so just one could add a patent on it. And so it became that for H V C licensing, there was MPeg L A plus another patent pool called H V C Advance, plus, um, I think was Nokia was outside of the patent.

• 3:51:47Speaker 02

  Yeah, a few of them are outside and some other one.

• 3:51:49Speaker 01

  And so it was impossible to license, right? And I think that several months ago, H P decided that they were going to remove support from H V C in their Windows laptops, because the, the cost was increasing of those patents. And it arrived where a point where, and there was uncapped pad. And so for YouTube or Netflix, we could talk about hundreds of millions of dollars of licensing for patents per year. And they said, you know what, as one hundred millions per year, you know, I could create my own codec. And this is what they did. And so that's why we have the Open Media Alliance, Alliance for Open Media, where we are part of. That is, that created A V one and creates A V two.

• 3:52:32Speaker 01

  Uh, we create also audio codecs. But yes, so the main difference would be that. And because you need to work around the patents or go do some things that are not patented, a lot of things are different, right? The basic things that were done in MPeg two thirty years ago are caught out of patents. But so for example, there is things like a, a golden frame, a S frame, or, or, or different type of.

• 3:52:59Lex Fridman?

  These are all patented ideas.

• 3:53:01Speaker 02

  Yeah, no, I, I can't believe it's not butter. I can't believe it's not a B frame. I mean, I mean, it's kind of what it is. In some ways, it's like a.

• 3:53:08Lex Fridman?

  Oh, so it's a different variant of a beef.

• 3:53:10Speaker 02

  Yeah, that's to try and sidestep things like that.

• 3:53:13Speaker 01

  And so you need to have double creativity, right? Creativity in terms of being more efficient, but creativity of being sure that you don't infringe existing patents. And so, for example, VVC is has all the patents of each VVC plus new ones, right? It's why AV2 tries to be as royalty free as possible.

• 3:53:33Lex Fridman?

  To what degree does FFmpeg and VLC have to think about this kind of stuff?

• 3:53:37Speaker 01

  We don't. And one of the reasons why VLC was in France is that France rejects software patents. So most of those patents are illegal in France. Because I once made the compute, the calculus that if I had to pay all the licensing fee for VLC, I needed to pay more than 200 euros per user, right? It's the same in dollars. But most of those patents are invalid in Europe because those are called, it's basically mathematical patents or idea patents, and they are not valid in Europe.

• 3:54:12Lex Fridman?

  Uh, let me just at a high level, just out of curiosity. So the meme online and the interwebs on X and Twitter and so on. And my own, I have friends in Europe.

• 3:54:26Lex Fridman?

  The the sense is that Europe is not friendly to entrepreneurship. They overregulate, there's too much bureaucracy and so on.

• 3:54:35Lex Fridman?

  Is is there any anything positive to say? Is there hope for entrepreneurship?

• 3:54:39Speaker 01

  Yes.

• 3:54:39Lex Fridman?

  In the future of Europe? Is Europe over? From a tech perspective?

• 3:54:45Speaker 02

  Just just look at the two of us, right? It's notable that there's two people from the European continent on this podcast talking about video. It's fair to say the community has waited heavily.

• 3:54:55Speaker 01

  What you probably don't see yet is that there is a new generation of entrepreneurs in Europe and mostly in France. UK has done it since a long time because, well, it's more Anglo-Saxon type of business. Look at business. But especially like what happened in France. And of course, sometimes a bit overdone with everything called French tech. But today, most of the people who come on the market wants to create startups. Fifteen years ago, it wasn't the case. Everyone wanted to work on big companies. Because when you failed in France, for example, twenty years ago, fifteen years ago, and you destroy your company, which is normal for startup, right?

• 3:55:38Speaker 01

  You were not allowed to create a new company, right? There was a lot of stigma. This stigma is gone. There is so many things happening on AI in France and so on, right? So there is, sure, over regulations. I know that, right? I'm a entrepreneur. But it has some good things also.

• 3:55:58Lex Fridman?

  I mean, is there some paralyzing aspects? If I look at the case of somebody I've become close with, Paul Durov, you know, he was blamed directly by the French government for the kind of things his quote platform was hosting. I could see the same kind of stuff. Basically, just as an example, V L C being blamed for the kind of videos that people are watching.

• 3:56:26Speaker 01

  Tried, right? Like we had, we like, we, we had issues like, we like.

• 3:56:32Lex Fridman?

  I mean, that's the pressure that people worry about. Because if you have to think about that kind of stuff when you're kind of just obsessed about.

• 3:56:38Speaker 01

  No, you don't think about it. And that's, that's okay, right?

• 3:56:42Lex Fridman?

  But what if they come in? What if they show up and?

• 3:56:44Speaker 01

  There is no office. Fideland doesn't have office.

• 3:56:46Lex Fridman?

  I mean, this was happened with Paul, they arrested him, right? So arrested him for particular videos or, or particular content that's being shared on the platform.

• 3:56:55Speaker 01

  Sure, I don't have any platform. Everything is on the client side.

• 3:56:58Lex Fridman?

  Yeah, but they're, they can still arrest you.

• 3:57:00Speaker 01

  On what ground? I'm not sharing anything. I'm not, the content doesn't go through my, my stuff.

• 3:57:04Lex Fridman?

  For sure, but it's still lawyer fees. That's the problem.

• 3:57:08Speaker 01

  Yes, that's correct.

• 3:57:08Lex Fridman?

  It's paperwork. So like, actually, if you had infinite trillions of dollars. You would win easily because you're on the right side. But the thing is, there is a degree to which they suffocate you with paperwork. Yes, that's the downside of bureaucracy through paperwork, through process.

• 3:57:28Speaker 01

  Yes.

• 3:57:28Lex Fridman?

  You know, it's the Kafkaesque thing.

• 3:57:30Speaker 01

  You have to realize that one of the good things, for example, in France or most of Europe is that the answering to a court order does not make you bankrupt, right? It's not like in the US where it can actually bankrupt you, right? There is, the way the law system works is that, like, I receive lawyers' letters every week, right? And I can tell you that the cost of lawyer fees for Vidalan is less than $10,000 per year, right? Right? So that's not really scary.

• 3:58:06Lex Fridman?

  I mean, similar with Pavel, the intelligence agencies try to like, say, can you put a backdoor on V L C?

• 3:58:12Speaker 01

  Yes, two of them.

• 3:58:14Lex Fridman?

  What do you say?

• 3:58:15Speaker 01

  No. I was a lot less polite.

• 3:58:18Lex Fridman?

  I see you, yeah, yeah. You're basically saying, hell, no.

• 3:58:22Speaker 01

  Like, if we had to compromise our software, we would shut it down. This is clear.

• 3:58:26Lex Fridman?

  And what's the definition of compromise? Like, allowing a government to do a backdoor?

• 3:58:32Speaker 01

  There is no code that gets into V L C that we don't control. And the way we compile V L C, you would call me completely paranoid. Like, we compile on boxes that are offline, where we start by compiling the compiler. We do everything offline on places that have never been connected to the internet. We, the way we do signing, there is double signature, especially because, for example, we've seen, and we believe it's a governmental agency that is not from the Western world to try to push a fake binary into our own servers, and that scared us a lot. And Vidalan is open source. How can you kill it? Like, I moved to where, right?

• 3:59:17Speaker 01

  I moved to Malta, I moved to, I don't know, Cayman Islands, and I changed the domain name and I start again, right? Like... VLC is a tool. It's a tool that is going to help people doing things. We're not a platform. And for patents, well, I'm sorry, but most of the patents, like you shouldn't be able to patent math and matrices. Like, this is wrong.

• 3:59:45Lex Fridman?

  Does VLC ever censor the kind of videos it can play and not based on the content of the video?

• 3:59:52Speaker 01

  No, never. We never do that. Because VLC is completely offline, doesn't talk to any server. So we don't know anything that you're using the software for.

• 4:00:02Lex Fridman?

  So again, there's no government that can say, you know, like the French government come in and say, we don't want, I think anime is destructive to society. We don't want any anime, not allowed to be.

• 4:00:15Speaker 01

  No, they cannot, they cannot do that. And also what they tried is to say, hey, I want to know if that person watched that type of video. And the answer is like, no idea.

• 4:00:24Lex Fridman?

  So no on that too. So for surveillance, no.

• 4:00:27Speaker 01

  No, no, because the only infrastructure we have is a downloading infrastructure. There is no telemetry in VLC.

• 4:00:33Speaker 02

  It would be difficult because of the international nature. It would be difficult for you to incorporate that code because there would be someone in the UK and someone in Germany and someone in the US as part of video and who'd be able to see that. It would be extremely difficult.

• 4:00:46Speaker 01

  The only thing that we can do, which happened, is like, we had the issue, we had the case with some police in the US who said, we have a murder case, right? And the file is distracted or doesn't play in that version of VLC. Could you help us, right? We never have access to the video. It's like a normal support, right?

• 4:01:03Lex Fridman?

  Oh, it's really about playing the file?

• 4:01:04Speaker 01

  Yes, and like, I remember in the middle of the Afghan war, right? I received an email from someone in the army. And I don't remember the grace, right? It was just like, we have a big issue with the latest version of VLC because it doesn't play correctly the file on an RTSP server that we have where there is all the movies. And he says VLC is very important for the morale on the troop on the ground, right? Because at night, I think it might be boring, right? So they have a collection of videos to watch or movies over there, right? And, and of course. I did an update and I broke some support of RTSP, right? So I gave them another version just for them, right? Because it was important and because VLC is completely open source.

• 4:01:47Speaker 01

  I think it is allowed on the US Army laptops, right? Because I guess someone in the in the US military actually looked at it and say, well, okay, this is okay, right? And the way we document how we process, that was okay, right? So the only way we we work with authorities is to help them doing support.

• 4:02:06Lex Fridman?

  That's amazing. That's really me. That's amazing story. Yeah.

• 4:02:09Speaker 01

  We don't see anything happening on how people use VLC and this is strong.

• 4:02:13Lex Fridman?

  Do you feel the stress of this? So first of all, millions of people using it. Second of all, military using it, maybe sometimes pressure from governments. Does that, does that, that's a, this is a small team, right?

• 4:02:29Speaker 01

  Yeah.

• 4:02:29Lex Fridman?

  How big is VLC, like the core contributor, how many?

• 4:02:33Speaker 01

  Six, eight?

• 4:02:36Speaker 01

  But and and and everything legally is on me. Everything that is legal is on me.

• 4:02:41Lex Fridman?

  You're not stressed about this?

• 4:02:43Speaker 01

  I used to stress about that a lot.

• 4:02:45Lex Fridman?

  Yeah.

• 4:02:45Speaker 01

  But the thing is, we're doing what we can for everyone, for the greater good. We we work that we make some extremely complex technology easy for everyone. We are tool and every tool is going to be used for great things and for bad things, right? You you cannot blame a tool, uh, I think. And and this is like very important for us. I used to be a lot in a lot of stress. I'm not anymore, right?

• 4:03:13Lex Fridman?

  What's the secret to your Zen? I mean, over and over in the chats I've had with you in in the conversation today about every even tense topic, you're very Zen. What's the, what's the source of Zen?

• 4:03:27Speaker 01

  I have a way of thinking about what is the worst case scenario, always, right? And the answer is, at the end, if I take like a chess player, right? In the end, am I dead? Yes or no? Right? And I do that nonstop, right? And that's also how I do my startups, right? Is that I'm here to get something great. What is the worst case? It got bankrupt. That's life. A company lives, a company dies. That's okay, right? Like, and so my moral way is always like, am I dying in the end? Am I hurting someone? If I'm, this answer is no, then too bad, right? Like, oh, some lawyers are going to be unhappy. What are they going to do? Take all the money of Videoland?

• 4:04:12Speaker 01

  Wow, they're going to have fifty grand. Amazing, right? What are they going to do with that? The code, the source code is out there. It's not stoppable. Also, because what we do is good and it's done for everyone.

• 4:04:26Lex Fridman?

  That's beautiful. Karen, you said that there's an active archiving preservation community? I think that's super fascinating. You wrote that they're stretching budget, but they see the extreme importance of FFmpeg as a Rosetta Stone. So that multimedia can be played a thousand years from now. I mean, that's a beautiful way to see FFmpeg VLC as a tool for preserving visual knowledge.

• 4:04:54Speaker 02

  Yes, so one of the coolest communities in open source multimedia, mainly led by someone called Dave Rice, I'll give him a shout out, I think, from City University of New York, is the archiving community. They've done so much stuff that they value that they value open source. One, because yes, they lack budgets, but two, they see the fact that archiving video is important for the world. And but being able to play that is a big problem. Um famously in the UK, there was a something called the the new doomsday book. And they they archived lots of stuff on BBC microcomputers. Within ten to fifteen years, no one had the right software to play that. I think it was twenty years or something like that. And someone had to go and reverse engineer this. And that was like twenty years. Imagine that in a thousand years.

• 4:05:39Speaker 02

  I think one of the the great things about FFmpeg is it's written in C. C is the closest to mathematics you're probably gonna get, the closest to logic.

• 4:05:47Speaker 01

  Do you think in one thousand years we'll still have C compilers?

• 4:05:50Speaker 02

  Yes, we have languages that exist that haven't changed too much. We have mathematic mathematical notation that exists. Um it will be like Latin, C will be like Latin. It will be a thing that you learn from the past, but it will still be usable in certain contexts. And so the archiving community are really great practically. They they again, limited funds. They funded the development of the FFV one codec. So that's a lossless codec. So for so the the archiving community is really scared about the act of compression losing things. And this could this is they have a fair point in this, you know?

• 4:06:20Speaker 01

  Yeah.

• 4:06:21Speaker 02

  If they compress too hard, it could change the view of the material. There could be something slightly different here and there. So they're really concerned about things need to be not just compressed well, but lossless and be fast. And and so they worked with FFmpeg to develop a whole new codec designed for fast software based encoding. They're really concerned about resilience. So they if they're storing on tapes or other hard disks, I lose some bits. I need to recover quickly. I can't lose a whole GOP because I've lost a bit, something like that.

• 4:06:54Speaker 02

  So they're a really great bunch of people. They funded GPU encoding in FFmpeg um to make FFV one encode faster. And and it's really about preserving the world's multimedia heritage in a way that's usable. And there's there's a lot of great teams and a lot of archival groups across the world who've who've chosen FFmpeg and FFV one as their archiving solution.

• 4:07:17Speaker 02

  And they can really provide us also super specialist advice. They can explain, ah, in the nineteen fifty's colorimetry was done like this on this certain type of tape. And so there is this special case that you need to handle and you'll never get this anywhere else.

• 4:07:33Speaker 01

  You see, they know things on video that we don't.

• 4:07:37Speaker 02

  Yes.

• 4:07:37Speaker 01

  Like every time I talk to with a Dave.

• 4:07:39Speaker 02

  Dave Rice.

• 4:07:40Speaker 01

  Or the people from the British.

• 4:07:42Speaker 02

  British film.

• 4:07:42Speaker 01

  Film. It's just like every time I just learn something new and I've been doing video for twenty years. They they have especially on colorimetry and colors.

• 4:07:52Speaker 02

  Storage, these other things.

• 4:07:53Lex Fridman?

  I mean, they have a deep, deep appreciation of the content itself, of the video itself. And like, especially when you're thinking of lossless. They're terrified of losing something essential about the thing. And in so doing, they're deeply understanding the thing that is to be preserved, which you sometimes might not be thinking about when you're obsessing about the actual technology of the encoding and so on.

• 4:08:16Speaker 01

  And when you enter the habit hole of film scanners, right? So you take those, those things to make to digital life. It's like a huge topic that like would take another five hours of podcasts just on that topic.

• 4:08:32Speaker 02

  And there's a lot of film that needs to be archived. Film is degrading. It's maybe not stored in the right environment. The other thing is they can, what they also do is because it's open source, they give this away. Their workflows to countries who can't afford to have archiving institutions where archiving is done by volunteers. It's done by the things they go and teach, you know, in India, they teach children to do, to do FFM pick commands. They're really great. They're really, ah, they're really the model community, the model ethos of what we're trying to achieve. They are such a great bunch of people, so interested in participating and being part of something much bigger. Because they realize the work they're doing in a thousand years is going to tell a lot. You know, in a thousand years, we may be drowning in AI slot.

• 4:09:15Speaker 02

  This stuff needs to be important and archived. Well, what was, what was life like?

• 4:09:20Lex Fridman?

  Yeah, it feels like capturing the twentieth century and the twenty first century is essential. Because it feels like a transition point where we went from scarcity.

• 4:09:30Lex Fridman?

  Of data to slop.

• 4:09:33Speaker 02

  Yeah.

• 4:09:33Lex Fridman?

  Oceans of slop. And that transition point is good to archive.

• 4:09:37Speaker 02

  It's important.

• 4:09:38Speaker 01

  People don't realize we are losing today a ton of films. There is a ton of things from the thirties, from the forties and the fifties that were. There is no value.

• 4:09:50Speaker 02

  And tape seventies and eighties, there's tape and there's not enough tape heads in the world.

• 4:09:54Speaker 01

  To read all the tapes.

• 4:09:55Speaker 02

  So they have to decide what they want to archive and throw away the rest of the tapes. There's huge moral hazard, I guess, for want of a better phrase, around this topic because. This is a digital record of human history and they have to make decisions.

• 4:10:10Speaker 02

  That and there's digital stewardship, I suppose, for want to, I've I made that phrase up, that's not a real phrase. Um. To make sure the world can have this information in in something that's playable by everybody, not. Playable on some device that doesn't exist anymore.

• 4:10:26Lex Fridman?

  And then there's like, realistically speaking, there's a needle in a haystack where there's a lot of value in in archiving all that footage and then. Over time, finding the gems that we don't know are there.

• 4:10:38Speaker 02

  Hey, there was something in that corner that we just didn't, you know. And that that would have been compressed away because some little thing, oh, wow, there's something there and and that's. They've made sure that it's lossless, they can prove mathematically that it's lossless, they can. Run different trade offs for if there's a bit for, they lose a bit, a single bit flips, I can make sure that I only lose a portion of a given frame. We can do error recover, they can do error recovery on previous frames, they could do all sorts of different things.

• 4:11:05Lex Fridman?

  Do you think V L C and F F M P E G will be here a hundred years from now?

• 4:11:10Speaker 01

  F M P E G, yes.

• 4:11:11Speaker 02

  Yep.

• 4:11:11Speaker 01

  V L C, maybe.

• 4:11:14Lex Fridman?

  What's the future of, where where is F F M P E G going, where is V L C going? Like in the next, if you think about like five years, ten years, twenty years.

• 4:11:23Speaker 01

  Five years, ten years is easy, the question is after that, right?

• 4:11:28Speaker 01

  The question is, do we arrive at something called holograms, right?

• 4:11:32Lex Fridman?

  Yeah, so will VLC and FFmpeg expand to whatever multimedia?

• 4:11:40Speaker 01

  Yes.

• 4:11:40Lex Fridman?

  So multimedia might become, sorry for the pothead expansion of topic, but you know, if you look at something like Neuralink with brain-computer interfaces, it's very possible that we start to consume what multimedia means is whatever codec, whatever data that our brain wants to consume through the brain-computer interfaces. That's one. And virtual reality, of course.

• 4:12:05Speaker 01

  You will have VLC for Neuralink.

• 4:12:08Speaker 02

  And you'll have FFmpeg-I input format human brain.

• 4:12:11Lex Fridman?

  There's going to be codecs for the brain.

• 4:12:14Speaker 02

  Sure, 100%.

• 4:12:16Speaker 01

  Of course.

• 4:12:16Speaker 02

  Yeah, to compress neural information.

• 4:12:18Speaker 01

  Today, there is like, there are new codecs for, for example, what we call Point Cloud, right? Or volumetric videos, right? There is a ton of research on what we call RGBD, right? The codecs for depth that is useful for robotics and for 3D things. There is a ton of codecs for compression of 3D elements.

• 4:12:35Speaker 02

  Compression for astronomy.

• 4:12:36Speaker 01

  For example, on VLC, we also have already a VR and XR version of VLC. And also on Kiber, right? We talk about Kiber. On Kiber, we also like do streaming of XR content on, for the glasses who cannot have enough power or Intuit, the Apple Vision or the Quest. So we already work on streaming 3D XR, interactive, low latency. There is something called volumetric video, Point Cloud videos. So it's not stopping. And yes, at some point, it will manage 3D data inside VLC and FFmpeg, right? It's obvious.

• 4:13:10Lex Fridman?

  So that's where it is moving, like the community is open.

• 4:13:13Speaker 01

  Not everyone in the community sees that, but like, as Kiran and I, we are entrepreneurs. We know where it's going. We see that, right?

• 4:13:22Lex Fridman?

  So I suppose that there is a tension probably inside FFmpeg. It's like, hey, listen, folks, we're really good at doing video and audio.

• 4:13:33Lex Fridman?

  So like, why expand? Like, let's do the thing we're really good at doing.

• 4:13:38Speaker 01

  In order to answer that question, we need to answer the definition of what is multimedia.

• 4:13:43Lex Fridman?

  Yeah.

• 4:13:44Speaker 01

  And multimedia is a digital representation of several streams for the human senses. And we will do that, right? So imagine there is now a way to not have a mic, but have an odor sensor and a diffuser of odors. It will get into FFmpeg.

• 4:14:07Lex Fridman?

  So your DMUX.

• 4:14:08Speaker 01

  Yes, yes. Of course, your DMUX has a new track type that is basically odors, right? And you already have.

• 4:14:15Lex Fridman?

  Smell, touch.

• 4:14:16Speaker 02

  It's like audio, you have a left and right nose track, you have a left and right audio pads. Easy.

• 4:14:20Speaker 01

  Yes, of course.

• 4:14:21Lex Fridman?

  The stereo smell.

• 4:14:22Speaker 02

  Stereo smell, yeah.

• 4:14:23Speaker 01

  So in VLC, for example, we already have a plugin for APTIC. It's mostly for what we called 4D cinemas, right? You know, those ones on hydraulic, I don't know how you say that. Hydraulic arms. Arms. And where everything is moving, like you have in theme parks, right? And there is a data feed synchronized, which is basically transporting this information.

• 4:14:45Lex Fridman?

  Is there yet a standard for that?

• 4:14:46Speaker 01

  There are many standards, right?

• 4:14:49Lex Fridman?

  You make me so happy.

• 4:14:50Speaker 01

  And so, of course, like, we have a plugin, which is not in the normal version of VLC, that is basically transporting those type of movements, which is physical movements, which is APTIC movements, right? It is a human sense, so it will get in.

• 4:15:06Lex Fridman?

  That's such an exciting future. What was it? I mean, it's a small community of developers. How do you pull that off? Like, if you're a contributor to FFmpeg or VLC, it feels stressful. Like, just looking on Twitter, it's like, it's a huge amount of work to make it work on all these different operating systems. An incredible effort.

• 4:15:30Speaker 01

  No, see it in the other direction. We are not the contributors, we are the maintainers, right? So we maintain for everyone, meaning that. For example, every year, there is around one hundred and fifty people who contribute to VLC and maybe three hundred on FFmpeg, right? Our goal at the small team is to get all the contribution in. So if there is more usage, there will be more contributions. And those people will do the right module, the new formats and so on. We care about the architecture of VLC, the architecture of FFmpeg, right? Now we're doing things in VLC, which is a spatial audio, right? We did the demo not long ago.

• 4:16:13Speaker 01

  There was changes needed on the architecture. And we did the first spatial audio module. When it's going to add the second one, it's going to be easy. Or the third one is going to be easy, right? Our goal is going to be the same for others or haptic, right? We need to work the architecture so that modules can be added to add future capabilities. So yes, we are going, we are multimedia frameworks. So that's not just audio and video. It's everything that is timed and represents something that you can sense. And if it's brainwaves, it's going to be brainwaves.

• 4:16:49Speaker 02

  I think that's inevitable, sorry.

• 4:16:51Lex Fridman?

  I love this on so many fronts because so FFmpeg and VLC are pushing companies and pushing the world to standardize. So for example, to standardize brainwaves. So standardized, it would push, like I hope Neuralink comes up with a standard for multimedia via brain-computer interfaces or for robots with haptic.

• 4:17:17Speaker 01

  By experience, what happens is always the same, right? You start, it's a new topic. There is like five different standards because everyone starts to do this. The hype goes down because every time the hype goes down, then people start to say, well, you know what? We need to do a standard. People, because two or three companies, usually not the leader, but the two or three followers do a standard. And then we implement the standard and and then it's the end of the curve. It starts to be more paper.

• 4:17:45Lex Fridman?

  And then the leader is kind of pressured into it because it is better to do a standard.

• 4:17:49Speaker 01

  Example, 3D audio, right?

• 4:17:51Lex Fridman?

  Yeah.

• 4:17:51Speaker 01

  Six or seven years ago, it was everything about 3D. You go, you had the cardboard on Android. You had two, two audio formats. They're all dead, right? And now it's coming back with actual use cases and we learn from the mistakes of the past standard. So it will be the same everywhere.

• 4:18:09Lex Fridman?

  And not try to avoid closed. I, I saw somewhere you, you didn't have too many nice things to say about Dolby.

• 4:18:16Speaker 01

  No, I don't.

• 4:18:17Lex Fridman?

  What is, uh, can you educate, uh, me on why, where they went? What, what, what'd they do bad that made you mad?

• 4:18:26Speaker 01

  It's used to be an amazing company doing tons of great things with amazing engineers. They define what sound was. And now it's mostly like lawyers and licensing things.

• 4:18:40Lex Fridman?

  Oh, so they're, yeah, they're, they're, they're closing stuff off. They're trying to get money on the licensing.

• 4:18:44Speaker 01

  They don't innovate as much as they did and so on. It's a bit like, I'm sorry to say, right? Like HP, right?

• 4:18:51Speaker 01

  Not true.

• 4:18:53Lex Fridman?

  Oh, since we talked about Twitter a bunch in a bunch of different contexts, do you have a, do you have a favorite? Your and least favorite, most embarrassing tweet on either video land or FFmpeg Twitters?

• 4:19:06Speaker 02

  The two, my two favorites are Talk is Cheap, Send Patches. I think that, that embodies a lot of the stuff doesn't get, as, as we've talked about, stuff doesn't get built unless someone does it. It doesn't just appear from the ether. Uh, the other one that I like is, um, FFmpeg, nothing is beyond our reach. I think that comes from a US military satellite patch where I think they, they invented some kind of, um, monitoring system that could see the whole world and this was released.

• 4:19:31Speaker 01

  Wasn't there something where FFmpeg was running on a rover on Mars also?

• 4:19:35Speaker 02

  Yeah, so FFmpeg is used by the Mars rover, uh, the Mars 2020 rover to compress, uh, pictures. And they really wanted, um, they wrote a paper about it and they really wanted to use as much commercial off the shelf technology as possible. Um, FFmpeg runs on Mars, so we are. We are a multi-planetary open source library.

• 4:19:52Lex Fridman?

  Nice.

• 4:19:54Speaker 01

  Very often, we've seen tweets for people using VLC in weird places. A lot of the people doing Formula Ones are in all the paddocks that use VLC to play the live feed. We've seen the European Space Agency, we've seen SpaceX like monitoring the launches with VLC. And like, it's like fills you with joy, right?

• 4:20:18Lex Fridman?

  I've seen Particle Accelerator.

• 4:20:20Speaker 01

  Oh, yeah, yeah. We had one of the most amazing thing that I went for was to go to the CERN at the LHC because they were using VLC to monitor all the captures on the ring. Because the ring is 27 kilometers. And so they had some analog cameras and they were using some of the capture cards to go to analog to VLC. So VLC could stream on their multicast network for the whole CERN to access to that. And like, I visited that in 2010 with Laurent and, and, and like, we fixed their issue in an hour or something like that, right? Because there was some parameters, maybe not well documented at the time.

• 4:21:04Speaker 01

  And he said, okay, for the whole day, what do you want to do? And we visited everything, like things where with antimatter and, and, and colliders and so on. And that was like one of the most amazing day of my, my physics background.

• 4:21:19Lex Fridman?

  Yes, you use like everywhere. Any, any tweets, Karen, you regret?

• 4:21:25Speaker 02

  Tweets of regret?

• 4:21:27Lex Fridman?

  Or is it like that? Well, how's the French song go? Regret nothing?

• 4:21:30Speaker 02

  Je ne regret rien.

• 4:21:31Speaker 01

  Yes, that's very important for me, right? Don't regret.

• 4:21:36Lex Fridman?

  Anything?

• 4:21:37Speaker 01

  No, it's because regrets are a tax on your mind, right? So learn from your mistakes, but don't regret. Because you've done it. So except if you have a time machine to go back in time, don't regret, right? It's going to just tax your brain. Learn from your mistake, sure, don't regret.

• 4:21:56Lex Fridman?

  It's like, it reminds me, it's beautiful. It's a tax on your brain. It reminds me of the Johnny Depp quote I saw where he was saying, hate, you know, I don't hate. Hate is a very expensive emotion.

• 4:22:09Lex Fridman?

  Are you comparing me to Johnny Depp? Because that would be your first one. Well, gentlemen, like I said, I'm eternally grateful for the software that the two of you and the bigger community have been part of building with FFmpeg and VLC and everything else. I'm eternally grateful for the spicy tweets, never stop. And I'm grateful that you would talk with me today and give me this sexy hat. I feel like a wizard. I feel special. And I feel special to get a chance to talk and celebrate the piece of software that brought me so much joy over the years. So, thank you for everything and thank you for talking today. Thank you for having me. Thank you so much.

• 4:22:51Lex Fridman?

  Thanks for listening to this conversation with John Baptiste Kempf and Kiran Kurnia. To support this podcast, please check out our sponsors in the description, where you can also find links to contact me, ask questions, give feedback and so on. And now, let me leave you with some words from the legendary Linus Torvalds. Most good programmers do programming, not because they expect to get paid or get edulation by the public, but because it is fun to program. Thank you for listening and I hope to see you next time.