Can AI replace the camera? with Joshua Xu from HeyGen

AI video generation models still have a long way to go when it comes to making compelling and complex videos but the HeyGen team are well on their way to streamlining the video creation process by using a combination of language, video, and voice models to create videos featuring personalized avatars, b-roll, and dialogue. This week on No Priors, Joshua Xu the co-founder and CEO of HeyGen, joins Sarah and Elad to discuss how the HeyGen team broke down the elements of a video and built or found models to use for each one, the commercial applications for these AI videos, and how they’re safeguarding against deep fakes. Links from episode: HeyGen McDonald’s commercial Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @joshua_xu_ Show Notes: (0:00) Introduction (3:08) Applications of AI content creation (5:49) Best use cases for Hey Gen (7:34) Building for quality in AI video generation (11:17) The models powering HeyGen (14:49) Research approach (16:39) Safeguarding against deep fakes (18:31) How AI video generation will change video creation (24:02) Challenges in building the model (26:29) HeyGen team and company

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Sarah Guo?

  Welcome Joshua. We're so excited to have you here today. How are you?

• 0:09Speaker 01

  Hey Sarah, I'm so excited to be here. Thanks for having me today.

• 0:14Sarah Guo?

  It's our pleasure. Let's get started. Welcome to the Huberman Lab Podcast, where we discuss science and science-based tools for everyday life. I'm Sarah Guo and I'm a professor of neurobiology and ophthalmology at the School of Medicine.

• 0:28Speaker 01

  Wait Sarah, I'm so confused. What's going on here?

• 0:30Speaker 02

  Is this thing on?

• 0:31Sarah Guo?

  Today, we're here to discuss how AI can benefit your health and what medicinal properties the technology holds.

• 0:38Speaker 01

  Sarah, I'm so lost. Isn't this the No Priors Podcast where you interview technology superstars like Gary Tan and Alexander Wang?

• 0:47Sarah Guo?

  No, that's only for humans.

• 0:49Speaker 03

  We're really excited to have you. Welcome Joshua.

• 0:51Speaker 01

  Yeah, excited to be here. Thank you for having me.

• 0:53Speaker 03

  So let's start with a little bit of backstory. You started this company, HeyGen, and it's had this amazing growth trajectory and is being used by millions of people now. What's the story of starting the company?

• 1:06Speaker 01

  Yeah, sure. So yeah, hello everyone. My name is Joshua, co-founder and CEO of HeyGen. We founded the company roughly three and a half years ago and before that I was working at Snapchat for about six and a half years there. I started robotics at Carnegie Mellon and joined Snap back in 2014 there. I initially worked on machine learning in Snapchat ads as ranking and recommendation. Then I spent my last two years at Snap working on AI cameras. So you know Snap leveraged a lot of AI technology to enhance the camera experience. If you look at, you know, 2018 Snapchat released a baby filter and Disney style filter. That was the first time I saw a computer can actually create and generate something

• 1:50Speaker 01

  that does not exist in the world. I was just so fascinated by the technology back then and I had the feeling that that would potentially change the way how people create the content. So you know Snapchat is a camera company and everybody created content through the mobile camera. But we wanted to replace the camera because we think AI can create the content and AI could become the new camera. And that's how we got started with HeyGen and our mission is to making visual storytelling accessible to all.

• 2:20Speaker 03

  I love it. The greatest minds of our generation, you know, inspired by, you know, your face is a cute kitten or whatever. What does replacing the camera mean to you? Like, why do we need to do that? I use my camera a lot.

• 2:33Speaker 01

  I kind of grew up my career in the whole, you know, mobile camera space, where we work on a lot of the software and technology to enable people to feel uncomfortable and make it easier for people to create content through the mobile camera. But, you know, there's still lots of people are not able to create good content using the camera today. And we felt that if we can replace the camera, that means we can remove the barrier for visual storytelling, for visual content creation. And that would help us to step ahead in terms of the whole content creation space.

• 3:07Speaker 04

  What are some of the areas that you think, you know, the technology that you developed is applied to? Because I think you've started with different forms of like virtual avatars, so that you can take a video of yourself and then turn it into a into an avatar that you can then feed text to. It can speak in your voice. It can do all sorts of really interesting things for different areas. How did you both decide to start with avatars? And then where do you think the main applications are?

• 3:29Speaker 01

  When we initially started the company, we tried to like disassemble the whole video production process. This is really about camera and then editing. So camera is more about A role, which represent the human spokesperson, the avatar piece. Editing is more about B role, adding, you know, different assets, voiceover, music, transition, animation, stuff like that. So editing, we just learn from customer that editing is not that expensive because it's pretty standard service. But camera is super expensive. And imagine, you know, as a CEO of a company, he wants to record something. We probably need to schedule that ahead of, you know, two weeks of time. We need to bring in the camera crew, have a studio to actually record it.

• 4:14Speaker 01

  And even for two minutes of footage, sometimes we need to record it for 20 minutes because people need to remember the script. And that's the piece that blocking a lot of a business create new content. So that's how we get started from, you know, trying to replace that piece of the process and making avatar to replace the camera for the video production.

• 4:34Speaker 04

  Where do you think that goes in the future? So, you know, people are already using Agen for all sorts of different application areas in terms of, you know, marketing and sales and. In some cases, like internal webinars or learning or other things, I'm a little bit curious, like, is the eventual form of this, you know, everybody has somebody who steps in for them for their Zooms? Or is it used for entertainment purposes? Or how do you kind of view the evolution of this sort of technology over time?

• 5:01Speaker 01

  Yeah, I would say there's many possibility out there. I think what we are tackling the problem so far, it is the, you know, entry point of the content creation, where all the content started with the camera. And then we would have people doing a lot of editing after that. You know, we can clearly see a path where people can already assemble all this generative footage and apply the AI editing to assemble the final video. And again, you know, if we push forward into technology, making the performance much better, I think we will be able to create experience like generative is video in a streaming way.

• 5:42Speaker 01

  And that actually will be potentially replace a lot of the, you know, real time conversation we have today, especially with the GPT-4O and with all this multi-modal real time streaming technology all together.

• 5:56Speaker 03

  Okay, we're still in asynchronous video creation land 2024. How do people use HeyGen today? Like, what are the favorite use cases you have?

• 6:05Speaker 01

  I would categorize the use, the use case of HeyGen into three: create, localize, and personalized. And, you know, people can select, you know, the cast from a library from our avatar or create their own digital twin and just like select a template or type the script and generate a video. This works the best for product spinner, how to videos, learning development, and some self-enablement training content. We can also take the existing video that localized that into a hundred, more than 175 different languages and dialects. And, you know, in this way, we can help customers to really localize their content into local languages.

• 6:48Speaker 01

  And last but not least, people can also use HeyGen to personalize the video messaging at scale. So I think there is a many, many very creative use case on HeyGen today. We are a very horizontal platform. I would say one of my favorite use case is probably the recent launch with McDonald's. And McDonald's launched a sweet campaign where they can allow people to send a message to a family member in the different languages.

• 7:17Speaker 05

  I love him so much. I would run out of words to express my love for him.

• 7:21Speaker 01

  You know, I just want to call it out, you know, AI is for everyone, grandma and grandchildren alike.

• 7:26Speaker 03

  Yeah, that's really cool. I mean, like, that's a big brand in a public, like, consumer-facing use case. How do you think about, like, the quality of HeyGen today? It like, you know, I would have thought of that as like sort of tip of the pyramid in terms of quality. And, you know, how can you tell when the avatars are good enough and not?

• 7:48Speaker 01

  Yeah, so I would say quality has always been, you know, the number one priority of the product and business and technology, I would say. You know, I always have a frameworks like this. There's an invisible line of a quality, you know, let's say that threshold's the 90. Anything below 90 essentially is unusable for the customers because we cannot really replace the real-life production process they have. We really, really focus on making the video generation quality that go above that threshold. And I think especially for Avatar today, it is above that. So we can really helping people to replace the real camera

• 8:31Speaker 01

  and unleash a lot of creativity process that help people to scale the content production there. And, you know, obviously, you know, there's much more room to improve. For example, generating the full-body avatar, being able to bring all sorts of element into a video. Yeah, we're in the process of that.

• 8:50Speaker 03

  What are you most excited about in terms of like what's next or new releases you guys have coming?

• 8:55Speaker 01

  I think there's many things very excited going on in our technology and product roadmap. I think particularly I'm very excited for the full body generation of Avatar. Historically, all the Avatar technology has been focused on the upper body. It's really hard to generate a gesture and the body emotion. But a lot of academic research has proven that this is very possible now. And we just need to like basically take that into the last mile. And another thing I would say, something I'm very excited about the streaming Avatar, especially with the latest release on GPT-4O, really, really help to improve the performance of the real-time interaction with text and voice.

• 9:36Speaker 01

  And Hejin Avatar could become a visualization layer for all those applications.

• 9:42Speaker 03

  Obviously, you need like full gesture control and movement to get to any video of any kind. But what do customers want to do in terms of full body motion today? Like you had like a demo of walking in, you know, the last couple of months.

• 9:57Speaker 01

  The way how we look at it is that there is a spectrum of the quality requirement

• 10:05Speaker 01

  laying out on different use cases, right? Let's start from the left side of the spectrum. It's the learning development content, educational content. It's more like one-to-many broadcasting, talking about educational training content. The quality there is lower because the Avatar can be more still, more professional. But if on the right side of the spectrum, we call that as like the high-end, you know, marketing content, really dynamic. You know, the one example would be the ads creative. And people ship very, very dynamic content on ads. And because that can really help to improve the ROI of a content, making it more engaging.

• 10:48Speaker 01

  I think making the full body, enabling that full body rendering, would be able to help us to bring the avatar, to bring the video into the next level of engaging and authentic. And that would help to unlock a lot of use cases in a broader case of marketing and sales.

• 11:03Speaker 04

  Newscast or other things, to your point, they often have the shot of the people walking and talking is like a standard can shot. And there's like these standard things that they use that if you had full body, you could provide for all sorts of application areas. I guess related to that, what is the technology that you folks are using today? You mentioned some things like GPT-4O, but you've also built your own models in-house. Like, how do you think about the technology stack that you're using? And how does that have to evolve in order to be able to do full body or other new things?

• 11:29Speaker 01

  There's a three model, right? Text, voice and video. So we work with OpenAI, ChatGPT on the text generation side. Obviously also serves like the brain of the orchestration engine that we built internally. And we work with, you know, OpenAI and Infini Lab on the voice engine. But we build the entire video stack in-house, including after creation, video rendering and B-roll generation. So I think over time, I think the whole technology chain has been moving towards a direction. A lot of all these things will be chained together. The multimodal model, multimedia all get to into one single model. One of the challenges I want to call out for the full body generation is actually,

• 12:15Speaker 01

  how do you actually connect that voice into, together with, you know, gesture motion? And that's actually something will be unlocked by actually getting the voice model and the video model training together. So that it can sort of like build a connection underlying the model as well. And that has been historically really, really hard because we have to like train the TTS model on one hand, and then feed that TTS model outcome into a video model. And this is pretty hard to build that connection. But with multimodal model training, that's very possible.

• 12:49Speaker 03

  Obviously, Sora is not available to developers and end users today, but there are like world-class text-to-video generation models that are generic, not avatars. How, how does this technology differ from something like Sora?

• 13:02Speaker 01

  When we initially started Hagen, we want to help the business solve the video creation problem. What does a business looking for? They're looking for quality, they're looking for control, they're looking for consistency. Right, so when we try to look back, okay, this is North Star. How can we get there? What's the technical path to get us there? This essentially probably potential two path. One is the test to image, the Sora, where we try to generate the entire thing from end to end. And so you get the entire video at once. And the other approach is that what we believe in at Haagen's that we try to disassemble the whole video into different components. Largely will be A roll and B roll.

• 13:46Speaker 01

  B roll represent all different kinds of elements like voiceover, music, transition. A roll spin the avatar. And we try to tackle this component one by one. And then we build an orchestration engine around that to assemble that final video together. We felt that this technical path is more capable to deliver the quality, you know, the control and the consistency that the brands looking for. Because for example, there's some stuff we should probably should not try to generate. It's the logo and the fonts. That needs to be very accurate. And not to mention that we also need to be able to learn about, especially in the business context, we need to learn about the brand style, the color mapping, et cetera, from the customers.

• 14:29Speaker 01

  And I think the second approach would give us more feasibility and capability to build that system around it. And in fact, we actually see Sora as our partner because we can, we are able to integrate that as one of the component, you know, generator. And then feed that into our orchestration engine for the business application.

• 14:50Speaker 03

  How do you think about what research, you know, if you just focus on like components of the experience, in particular, the video stack being the thing that you really want to own and be state of the art in at Haagen? How do you approach like new capabilities from a research perspective? Is it, you know, look at what's available in academia? Look at the problems customers give you sort of de novo?

• 15:13Speaker 01

  I would say it's a combination. I would add one more thing is that we need deeply understand the limitation around the model and try to find the connection between what is the customer looking for? What's capable with the technology? Like when we really try to look at it, all the AI model had some sort of a limitation. And I think the key question is that in order to deliver a great product experience for the customer is that how do we design a product around it? So that we can try to avoid the limitation of the model, but help to amplify the strength of the model. And that's something that's really important to find the new area that unlocked a new creation experience.

• 15:57Speaker 01

  But one example would be when we look at, you know, the video translation technology. It's a, you know, it's a whole new way to translate a content compared to traditional dubbing. We preserve the user, the natural voice and their facial expression. But if you look at really underlying the model, what enabled that video rendering? It is actually a lip sync model, right? But we kind of figure out a way to combine all this together, together with the voice as well as the translation with ChatGPT and build a great experience around it. And sort of like we are creating a whole new experience for localized their video and content.

• 16:36Speaker 03

  So there's lots of like great McDonald's like exciting commercial applications. I think a lot of people also think deep fakes are really scary. Like and the ability to, you know, abuse somebody's, you know, likeness or voice is scary. How do you think about? Safety, election safety, abuse.

• 16:56Speaker 01

  First of all, we do not allow any political or election content on our platform today. Hejin's policies strictly prohibit those creation of unauthorized content, and we take abuse of the platform seriously. So we have our security safeguard include very advanced user verification, include live video consent, dynamic verbal passcode, and rapid human review in the back of all the alcohol being created on the platform. Trust and safety is critical to our business, and we are actively partnering across the industry, continue developing the tools and best practices to combat misinformation and AI safety. And we actually build the safety as part of the design.

• 17:41Speaker 01

  If you look at a lot of the application process on Hejin, and we pay all this safety concern and safety guard on every single step of the creation process as well.

• 17:53Speaker 04

  It makes a lot of sense. I guess it's kind of interesting because if you think about it, at least from the positive version of this, and you talked about how you try to protect against the negative. The positive version is you're running for office and you should be able to send a personalized message to each voter, literally into their inbox with a short video clip of you talking to them specifically, or talking to issues that they specifically care about, or things like that. And so you could imagine using this technology in the future for actually hyper-personalized political campaigning. And as long as you can avoid some of the deep fake side of it, then obviously it could actually be quite valuable. How do you think this ability to really generate

• 18:35Speaker 04

  large-scale, differentiated, personalized, etc. content of individuals talking, how does this kind of generation change how people make or use video in general?

• 18:46Speaker 01

  You know, if people can generate very engaging and authentic video content, they will basically create more videos and use video more for their business to grow their business. And we live in a video-first role where every business want to create more videos. I think the bottleneck today in the industry is that video is just very expensive to make and it takes like weeks or months to make a video. I think it would fundamentally change a lot of way how people are thinking about how to grow the business, how to do the communication, how to do the marketing and sales. So I do think there's a huge possibility that we can create and generate

• 19:32Speaker 01

  a very high degree of personalized video, especially with the full-body avatar that being able to deliver a very dynamic and high-quality content out there. So I want to give you one example is that I think a lot of AI generation is not only about, obviously, you know, cost-saving and time-saving is one aspect of the value prop. But it's actually we are seeing a lot of customer using that to unlock new use cases and being able to do something they were not able to do it before. I think that's the key job in point of a lot of a business outcome today.

• 20:07Speaker 04

  How do you think about it in the context of real-time versus asynchronous? It feels like a lot of these technologies are focused right now on asynchronous use cases. And that's true as well of just pure text-to-speech models. When do you think we move to any sort of real-time or close-to-real-time video avatars and sort of the uses of that?

• 20:24Speaker 01

  I look at it in two ways. Like one is that the real-time application of the avatar, like even now it's possible. I think people can already experience that on HiGen. We are making a new update that can make it even faster. So it can potentially become, let's say, the virtual, you know, AI SDR, virtual support that help to take customer calls or provide support, right? And, you know...

• 20:54Speaker 01

  I think the technology has been always like developing like this trend. Two years from now, it would not be crazy to look at a lot of avatar generation. As synchronization pipeline will become real time streaming capable. And I also see the world is moving towards a way that we can probably generate the entire video in real time as well in the future. Let's say five years from now. I have an opinion like, you know, generative image is still image. But generative video is not a video. It is a new format. What I meant by that is, you know, when we really look at video, we look at it as an MP4 file. Right? So it is immutable.

• 21:35Speaker 01

  Like, for example, if you and I are on Instagram, we probably get recommended by different, um, two different, two different ads. But also we are recommended from the same business, we are looking at the same MP4 file. But it does not need to be the same. Let's say if maybe I like avocado, I should be watching an ad with Coca-Cola and avocado. And showing the, uh, you know, the, the, the new story about Coca-Cola with me. And you like something else, you could be looking at something else. And this is not possible today because making videos expensive. But this could be very possible, let's say we can actually, you know, real time generating the video ads that you like according to your user attribute.

• 22:18Speaker 01

  That will potentially become a new format. You know, when we really look at today's video player, it corresponded to only one the MP4 file. And it, it doesn't need to be true, uh, doesn't need to be like that. That video player can actually take in a lot of, uh, you know, user attribute and generate something in real time to match what's the best way to deliver their content to customers. Yeah?

• 22:45Speaker 03

  Yeah, I think it's, you know, one interesting analogy would just be like if you think about, you know, YouTube as one of the largest learning devices in the world today, like, it is static immutable video for everyone, but it's pretty clear Bloom studies and, and everything else that like personalized education is going to be the path that is more effective. If people want to learn by video, but it's very hard, it's too expensive to make that video personalized. This feels like a, you know, opportunity for a very different educational future too.

• 23:20Speaker 01

  Yeah, and, and one of the use case we have seen from customers that, you know, Pubsys Group, they generate more than a hundred thousand videos, a thank you video to send to all the employee globally. And in localized into different languages personalized with a name and they are, they, what they like about, about it, you know, when they join the company and stuff like that. And historically, that is actually only deliver with one video, right? So they may be the CEO, the executive team hop on into a camera and we call something, you know, saying thank you for the, you know, 2023. But now that message and communication can really personalize at a very big scale.

• 24:01Speaker 04

  So one thing you mentioned is the various aspects of research that you're doing in terms of building your own video models, as well as using third-party APIs. What's been difficult or hard from a research perspective?

• 24:09Speaker 01

  Unlike a lot of other model, I think building video model,

• 24:16Speaker 01

  you know, being able to integrate aesthetics into the AI models pretty hard. So, you know, video generation is not only about solving a mathematical problem. It's actually about creating something the customer love and appreciate. So essentially a model with a lower optimized cost function doesn't mean it actually produce a better visual outcome. So I guess that is the piece that making it really hard to evaluate, but also really important to deliver the mile, the last mile of the value for the customer. And you know generally evaluation is also hard. We have to rely on in product signal, for example, A B test to know which model is actually better. Because you know, only the customer can be the judge for that.

• 25:01Speaker 01

  And this process generally is just not differentiable from a mathematical standpoint. We kind of have to form a system, build a system around it. And be able to feed back those data into our model training so that we can continuously to improve.

• 25:15Speaker 04

  Did this approach come to you because of your work at Snapchat, working in consumer products? Or is it something that you had to come up with in the context of HeyGen itself?

• 25:23Speaker 01

  I would say it's very similar, especially on when we work on the camera software. So how do we know whether this parameter was better or the other one was better? And I think we can definitely come up with some, you know, very objective matches about, hey, you know, lightening score, this is lighting score, this is resolution. But there's many things we figure out, hey, a better resolution, I mean, high resolution doesn't mean it's a better image quality for customers. If you look at iPhone, it does not have the best resolution always compared to a lot of other phones. But it does produce the image that most people like about using iPhone to capture the image.

• 26:06Speaker 01

  And yeah, there's a very similar lessons out there we learn from in early days of Snap. Yeah.

• 26:12Speaker 03

  What can you say about how big HeyGen is today?

• 26:15Speaker 01

  We are a little bit over forty people, but we are serving over forty thousand paying customers on the platform today. And I think what's so interesting about our customer is that these are not the typical AI early adopters. These are mainstream companies from European manufacturers to small business, to global nonprofits, to Fortune 500 companies. Which is space the problem we are solving.

• 26:43Speaker 04

  Given the other thousand customers per employee, which is an incredibly impressive metric. Are there specific key roles that you're hiring for or other things that maybe members of our audience may want to apply for?

• 26:53Speaker 01

  Sure, yeah, we're hiring across different teams, basically product, design, engineering, AI research and go to market. Yeah.

• 27:01Speaker 03

  This has been a great conversation. Thanks, Joshua.

• 27:03Speaker 04

  Thanks so much.

• 27:04Speaker 01

  Yeah, thank you. Thank you for having me.

• 27:07Sarah Guo?

  Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.