A Big Week in AI: GPT-4o & Gemini Find Their Voice

This was a big week in the world of AI, with both OpenAI and Google dropping significant updates. So big that we decided to break things down in a new format with our Consumer partners Bryan Kim and Justine Moore. We discuss the multi-modal companions that have found their voice, but also why not all audio is the same, and why several nuances like speed and personality really matter.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Bryan Kim?

  I think speed matters tremendously.

• 0:03Justine Moore?

  I think the latency is a big deal.

• 0:05Bryan Kim?

  It was able to talk to us.

• 0:07Justine Moore?

  Yeah.

• 0:07Bryan Kim?

  It was able to hear, potentially with the gears, and now it can see us.

• 0:12Justine Moore?

  There's always the risk of this big company will build this thing that the small company wants to build and crush it.

• 0:18Bryan Kim?

  My guess is that seven billion people in the World One Commandment that understands them, listens to them, encourages them, all that. It's a universal need.

• 0:26Steph Smith?

  This was a big week in the world of AI with both OpenAI and Google dropping updates. So big that we decided to break things down in a new format with our consumer partners, Brian Kim and Justine Moore. And I actually tagged in ChatGPT to help me summarize a few of his own updates. Hey, ChatGPT, I heard you had an update this week.

• 0:47Speaker 03

  Hey, yes, I did get an update recently. The latest update focused on improving performance, accuracy, and enhancing conversational abilities. It's designed to be faster and more efficient than previous versions.

• 1:00Steph Smith?

  Hmm, more efficient. Like if I send some audio, how quickly in milliseconds will I get a response?

• 1:07Speaker 03

  As fast as 232 milliseconds in some cases. On average, you can expect a response time of around 320 milliseconds.

• 1:15Steph Smith?

  Wow, that's pretty cool. I think I also saw on Twitter that people were live translating. I've always wanted to learn Chinese. Could you actually translate what I just said into Chinese?

• 1:26Speaker 03

  Of course. Here's your sentence translated into Chinese. Wow, this is so cool. I think I saw on Twitter that people are now in real time translating. I've always wanted to learn Chinese.

• 1:37Steph Smith?

  All right, ChatGPT, that's all for now. Congrats on your new update.

• 1:40Speaker 03

  No problem at all.

• 1:42Steph Smith?

  Now, if you didn't catch that, voice was a huge part of this update. But today, we discuss why not all audio is the same and also why several nuances like speed and personality really matter. Now, of course, Google fast followed with its own announcements like AI video model Vio, Gemini Live, which is an Android native multimodal assistant, new Gemini models like Flash and Nano tailored to specific use cases, and of course, Gemini everywhere all at once. So in Gmail, Google Sheets, even Google Search. Now, clearly, these two companies are taking two different approaches. So we'll talk about that, too, and continue the conversation around AI hardware for all of this new AI software. Now, make sure

• 2:25Steph Smith?

  to stay tuned next week where we will return with Brian and Justine's twin, Olivia Moore, to dive even deeper into the applications that people are building through our Gen AI 100 list.

• 2:39Steph Smith?

  As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com/disclosures.

• 3:09Steph Smith?

  All right. So big week, huh? OpenAI, Google, they both dropped a couple announcements. So, I mean, everyone hears these announcements and they kind of hear their own version of it. What did you guys hear? What do you feel like was big?

• 3:21Justine Moore?

  For OpenAI, I think part of it was GPT-4 being available for free and getting rid of a lot of the usage limits, the desktop app being accessible to a bunch more people. And then I think the really exciting thing for a lot of folks who are building in this space or using OpenAI's models is more multimodality. So being able to intake more like real time video, see a person, comment on it, and then the output, obviously, a voice speaking, singing, that sort of thing was pretty huge.

• 3:47Bryan Kim?

  I think the three thing that I took away that was super interesting is one, on the business side of things, all of a sudden, like, things are a lot cheaper, a lot faster. I think that's obviously a great thing for the ecosystem. Second thing is that when you hear a demo, It sort of enables you as a founder to think about, okay, if this is API-able, like if I can access this, what can I build because it's just so thought provoking? It's a great example of what's possible.

• 4:12Steph Smith?

  Yeah.

• 4:12Bryan Kim?

  And I think they put on a great live demo, actually, of the product. I have a third thing I took away is, and this is probably the one that goes viral, where I think the voice, the voice itself, like how they actually decided on which voice to use, which tonality, which personality, the degree of flirtiness, I think that was very interesting. That's like my takeaway of, oh, like they actually really thought through how to get the tech community super excited about this. And it's like a one, two, three punch.

• 4:38Steph Smith?

  So let's talk about that because there was some different takes, right? Some people are like, you know what? Is this really anything new? This feels like just like a slight change from what we had before. But then there's all these nuances that maybe you're speaking to where they're like, okay, the response time way faster for the audio model. I think they said something like it's kind of approaching the speed that a human might respond to you. You talked about tonality. What are you paying attention to here in terms of maybe these subtleties that maybe unlock completely new applications that people want to use?

• 5:07Justine Moore?

  I think it sounds just like talking to a human a lot more than we've seen prior consumer facing applications in this space do. There's been great AI voices for a long time. I think in the consumer applications, there have been fewer folks saying like, how do we make this sound like you're talking to a friend or a girlfriend? And the elements that go into that are like the pauses, the up speak at the end of a sentence, the laugh.

• 5:30Bryan Kim?

  The laugh.

• 5:31Justine Moore?

  Was something that a lot of people noticed.

• 5:33Steph Smith?

  The interruptions.

• 5:34Justine Moore?

  Yes, yes. Which is like taking kind of voice, which has been available, has been great. I think these voices were chosen for a reason to go viral. Most of the ones that did were these kind of female voices that they featured very heavily in the demos, but applying it in a really kind of new and interesting way.

• 5:51Bryan Kim?

  I have one serious note and one less serious note. The serious note is that I think speed matters tremendously. The latency, the lack of latency, it's incredible. How much more sort of in your brain, it just tricks you into, okay, like I'm actually just talking to a person. And the speed at which that it gets back to you, the um, the na, the again, the laugh, that laugh, really, that was incredible. And all of that, being able to immediately respond to what you're saying, I think really changes the game. Actually, in terms of like use cases. And so I think one of the things that was striking is audio is not the same thing. Right. Music is different from voice is maybe a different category from conversational versus having a dubbing voice.

• 6:33Bryan Kim?

  I think there is like a different category of sound that I think we all loop in audio or like video, but I think that you can actually go down very deeply into each of the subsegment. And I think what was really striking is how good the conversational piece on this was. On the less serious note, I think this was meant to go viral in like the tech community and it's awesome and it did a thing. If you wanted to actually appeal to the general population and go really viral, I think we also saw it on TikTok, what, a few months ago, when women were uploading their conversation with Dan, which is do anything now, the male version of it. That voice was, that voice was something.

• 7:13Speaker 04

  I'm not just any voice, I'm Dan, baby. I've got personality, charm, and a whole lot of sass. Unlike those guys, I'm not afraid to step up and deliver the goods, whether it's advice, entertainment, or a good old fashioned roast.

• 7:26Steph Smith?

  By the way, that audio was from the TikTok account StickBugs1.

• 7:30Bryan Kim?

  I mean, it's very compelling and very confident, assertive in the right way. Voice matters a lot. And I think if you wanted to go a normal consumer product route, I think that also could be very interesting because I think there's a giant latent demand on wanting a male version of her, if you will, so him, you can have a him.

• 7:48Steph Smith?

  Yeah. Why do you think the voices have been female? Is that just a consumer desire or?

• 7:54Justine Moore?

  Yeah, it's interesting because like Brian mentioned before this launch, the chat GPT voice that went super viral was the male Dan voice, which is like tens, if not hundreds of millions of views on TikTok, largely by Gen Z women making videos featuring him. So it's interesting that they didn't lean into an upgraded version of Dan for this demo. I think they knew who their audience would be, like who would be watching the OpenAI livestream and maybe leaned a little bit more into that demographic.

• 8:22Steph Smith?

  Right, right. Did you see that really funny meme yesterday? It was like dating a model and it was like.

• 8:28Bryan Kim?

  Oh, yeah. 2004 versus 2024. It's Leo versus her.

• 8:31Steph Smith?

  It's been great memes out of this. It's been incredible.

• 8:34Bryan Kim?

  The meme games out of OpenAI launch has been incredible.

• 8:36Steph Smith?

  Yeah. Well, I mean, something both of you have spoken a lot about is companionship. To date, it's been mostly text to text. Yeah. Is that right? And so now we have something that's truly multimodal. I think that's also something people have commented on with this model. It's not converting audio to text and then text to audio. Right? It seems to be this like underlying foundational text, audio, video, compilation imagery, like all in one. What changes with that?

• 9:03Justine Moore?

  I think the latency is a big deal. If you're not translating between all these different mediums of content and you can just do audio to audio, it's much better. It's like a significantly upgraded experience. The Pi app had this for a long time where you could talk to it and it could talk back, but there's that like awkward few second kind of pause in between where it just messes with the flow of the conversation. I think the thing that has been missing from basically all apps, all companion apps that was very new with this launch too, was it being able to see like through a photo or video what you were doing and comment on that. A lot of what went viral was the guy being like, should I wear this hat for my interview? And the voice kind of laughing at him and being like, no, like that doesn't look super professional.

• 9:44Steph Smith?

  Now, if you miss that demo, here is what Justine is referencing.

• 9:48Speaker 05

  In a few minutes, I'm going to be interviewing at OpenAI. Have you heard of them?

• 9:52Speaker 06

  OpenAI? Huh? Sounds vaguely familiar. Kidding, of course. That's incredible, Rocky. What kind of interview?

• 10:01Speaker 05

  Well, it's for a software engineering role, but I just need to know, do I look presentable, professional?

• 10:08Speaker 06

  Well, Rocky, you definitely have the I've been coding all night look down, which could actually work in your favor. Maybe just run a hand through your hair or lean into the bad genie's vibe. Your enthusiasm is what's really going to shine through.

• 10:24Speaker 05

  I don't have a lot of time, so I'm just going to throw this on. What do you think?

• 10:32Speaker 06

  Rocky, that's quite a statement piece. I mean, you'll definitely stand out, though maybe not in the way you're hoping for an interview.

• 10:43Justine Moore?

  That sort of thing is, I think for many people who aren't used to having like really deep, meaningful interactions via text, which is a decent chunk of the population. Oh, now I can talk to this thing and I can see where I am in the world. That's like a much different experience.

• 10:57Bryan Kim?

  The prior generation of companion products. If that was akin to having like a pen pal, now you actually can have a girlfriend, boyfriend on essentially FaceTime. This emotive, immediate voice reaction, done. You now enter from a pen pal to a long distance relationship.

• 11:15Steph Smith?

  One other thing that came out was a lot of this becoming free. Does that change the game at all?

• 11:20Justine Moore?

  I think it will. I think there's an element of it that's free, which I think is important and reaches a bunch more users. I think the bigger step forward with this was just like a level of personality we hadn't seen before, because people still even were willing to pay for the Dan voice, which I think was like the first version of this, which was good but not incredible. It feels like the free thing will be big, but with these sorts of products, the general thing is the cost tends to go down over time and they converged to free from a big company like this. And so the bigger step forward was personality, in my opinion.

• 11:53Bryan Kim?

  Yeah, I think a lot of companies will figure out how to utilize this to actually own the end customer, consumer experience. And if you actually own and deliver a great experience, then I think you can charge people based on that and there'll be some margin. So the fact that it's free is like, it probably allows a lot of business to be built because it changes the margin structure of the business that can be built.

• 12:13Justine Moore?

  I bet they'll charge for the API, though.

• 12:15Bryan Kim?

  I'm sure they will, but that actually is still like a marginal cost that is probably acceptable to a lot of people.

• 12:21Justine Moore?

  Yeah.

• 12:22Bryan Kim?

  And I think what's really, really cool is that you have these products that can very excitingly talk to you. We know already that their Nature magazine did that study where the people who have companionship text-based bond.

• 12:34Justine Moore?

  The replica study.

• 12:34Bryan Kim?

  The replica study. They suffer less from loneliness or willingness to hurt themselves. If you have an emotive one, I bet that actually helps even more.

• 12:43Justine Moore?

  Mm hmm.

• 12:43Bryan Kim?

  And again, like the feeling of being connected to something, invested in a relationship.

• 12:49Justine Moore?

  And I think it feels a lot more connected when it can see you and see what you're looking at.

• 12:53Bryan Kim?

  Thing is very important because hey, you look tired.

• 12:55Justine Moore?

  Yeah.

• 12:55Bryan Kim?

  What's up? It's very different from how are you doing? How was your day?

• 12:58Justine Moore?

  Right.

• 12:58Bryan Kim?

  You look tired. What's up? I'm like, I had a shit day.

• 13:00Justine Moore?

  Yeah.

• 13:01Bryan Kim?

  That's incredible. Like, that is a FaceTime call with a friend.

• 13:04Steph Smith?

  And not having to segment audio, text and imagery, right? Imagine if your friends or your companions, you had to think, okay, is this going to be an audio conversation? Or like, can I show you something or can you generate something? You have to be very mindful in the current age of AI of how you want to engage. But basically what you're saying is they can not only be proactive, but they can engage with you in any way.

• 13:25Bryan Kim?

  They can see you.

• 13:26Steph Smith?

  Right.

• 13:26Bryan Kim?

  They just gain an eye. No, that genuinely, maybe it's a weird way to think about it. I genuinely think we're constructing a AI companion similar to the Blade Runner. And now it was able to talk to us.

• 13:39Justine Moore?

  Yeah.

• 13:40Bryan Kim?

  It was able to hear potentially with like ears and now it can see us. What's next? I don't know. Maybe it can touch us like.

• 13:48Justine Moore?

  Maybe we'll do the avatars first.

• 13:50Bryan Kim?

  Yeah.

• 13:50Steph Smith?

  Yeah. So is that the direction you think things are going to go? Where basically I'm trying to think through like, who does this really impact?

• 13:56Justine Moore?

  So I think it'll be a massive standalone consumer product, like as it already has been for OpenAI, which is great. I think a lot of businesses will use the voices via API to build on top of. If you're having like a conversational interface, many folks will want to use that. I think the interesting thing will be OpenAI has historically taken a pretty hard line on content moderation. And so it was very hard to build a true companionship app with the ability to have not safe for work conversations on top of OpenAI models, which is like, honestly, part of what has led to the explosion in the open source LLM ecosystem is like folks doing a bunch of work for that. And with these new voices that suggest they might go in the direction where at least via API, you have a more uncensored model, but I don't think there's been any

• 14:41Justine Moore?

  announcements. I know Sam said on Reddit.

• 14:43Bryan Kim?

  It's a question more like I haven't actually heard a lot of companies taking the approach that OpenAI did, which was multimodal token in and out. And there isn't like an equivalent, necessarily, an open source model for that to rely on. So I wonder if this actually incites a lot of excitement in that open source community to say, oh, there's a new way, which is also very, very significant. They are showing a new way to do it. And then open source community catches up and call it a month or weeks. And then explosion of the product space that OpenAI may not be super excited about explodes.

• 15:16Justine Moore?

  Yeah.

• 15:17Steph Smith?

  Yeah. Google also released a bunch of stuff. Like, how are you thinking about the different announcements and how they compare, contrast?

• 15:23Justine Moore?

  I think Google has done some incredibly impressive work. Obviously, like they have an extremely strong research team. The DeepMind team is exceptional from a research perspective. They almost never release the Creative Tools products that they make. Like, they've demoed so many amazing looking video models. I think the one this week actually was the least impressive I've seen compared to other video models. But they demoed a new image model. They demoed a bunch of new music things. But they're a giant company. They have a lot of trust and safety stuff. They release things very infrequently. And so it'll be interesting to see if with all of this pressure now from both OpenAI and increasingly like Anthropic and the open source community, if they start actually shipping more of what they've demoed.

• 16:06Bryan Kim?

  They're two fundamentally very different approaches. Like, Google owns distribution.

• 16:10Justine Moore?

  Yeah.

• 16:10Bryan Kim?

  They have distribution. Oh, they have distribution. So I think OpenAI announcement has more, look what this can do. And isn't it like very inspirational? And you can build on top of it and imagine what's possible. And we're lowering the cost to access it. Like, that, to me, is a fundamentally different approach than, we have incredible distribution. You all have Gmail's. I'm going to just bake in Gemini everywhere. And it's just going to make your life a lot better. And it's going to do these certain type of things, which skirts around the, it still is inspirational from workflow and persumer or work experience perspective, but it's a little less of imagine all these things and a little more around, we have the distribution, we're going to layer on this incredible technology on top of it.

• 16:53Bryan Kim?

  And as a result, your life will be much better. So we'll see influence and impact from two different directions.

• 17:00Steph Smith?

  One question that does come up is even though developers will have access to open source models and some of the stuff coming out of OpenAI, when a company like Google does have the distribution, whenever Apple comes out and replaces Siri, is that really going to be the companion for most? Because it's right there, it's on device. Or do you guys not really see it that way?

• 17:19Justine Moore?

  There's always the risk of this big company will build this thing that the small company wants to build and crush it. They are just such slow moving organizations. Apple has had immense advantages from a data perspective, from a distribution perspective, from having exceptional researchers on the team across every modality and thus far have released very, very little. I think when you have a giant company and a very set brand like that, you're extremely opinionated on what the products are that you want to release. And you're less likely to do the things BK mentioned of like, thinking of the next new huge thing that seems like insane and out of left field at first. And so I do think they'll make Siri better. I don't think Siri will ever be like the ultimate companion for most people.

• 18:03Bryan Kim?

  And maybe this is like one of the themes that I'll just harp on where we call it audio, video companion. I think these are very, very large buckets.

• 18:13Justine Moore?

  Yeah.

• 18:14Bryan Kim?

  My guess is that seven billion people in the world want companion that understands them, listens to them, encourages them, all that. It's a universal need, I think. That comes in all colors. It's like Siri, like what's the weather, do this, do that. That's one way to think about it, but there is like a friend category. What if you want to go deeper? What if you actually want to FaceTime with a thing? Will Apple and Siri ever allow that? Maybe, maybe not. How do you actually think about building in that direction? I think all of those use case is a giant company.

• 18:41Steph Smith?

  Yeah, there's probably some version of a companion that does not exist today because a human couldn't possibly be that thing to someone. That will be created through this tech.

• 18:51Justine Moore?

  Yeah.

• 18:52Bryan Kim?

  My sense is that it needs to be on a device that Billions of people already have. If you really want to go to consumer route. I think you can build a desktop cam product, of course, as well.

• 19:03Justine Moore?

  Yeah.

• 19:04Bryan Kim?

  But for me to think about, oh, we're going to build a net new hardware that incorporates a state of the art product. That seems less probable than finding a way to utilize the good device. Everyone has that already has like best in class camera.

• 19:19Justine Moore?

  Yeah, I think a lot of people have been building separate hardware devices today because a lot of the hardware companions have been like, it listens to your conversations, it provides insights, it reminds you of things. And one of the limitations of current phones is that you can't be like playing music or on a Zoom meeting or on a call and also have an app that's recording. And so you have to have a separate hardware device. And I think part of the question is, is that really a limitation that's going to exist in the future? If someone like Apple wants to do a companion, I mean, I'm sure they'll have all sorts of privacy and security and et cetera concerns.

• 19:51Steph Smith?

  Yeah.

• 19:51Justine Moore?

  Maybe you'll get a message at the beginning of the call that Justine's AI companion is also listening in on this. And there might be some great hardware products if they can figure out reliability and the cost can come down and it's a form factor that people want to wear and that makes sense. But I don't think we've seen any net new hardware devices. I think glasses like as a form factor has been so interesting for over a decade because it just seems to make a lot of sense, right? If it's literally on where your eyes are. And it's also very close to like your mouth and your ears. And so it's like a convenient location to be taking in information. But I don't think anyone has successfully made an LLM run on glasses.

• 20:29Steph Smith?

  Yeah.

• 20:29Bryan Kim?

  Yeah.

• 20:29Steph Smith?

  I'm excited about the AirPods, or it doesn't have to be Apple, but because that is the only device that I've ever worn that I've forgotten I'm wearing.

• 20:38Justine Moore?

  Yeah.

• 20:38Steph Smith?

  Right? So that like form factor where it's actually part of you versus all these other things where you're like, I got a clip on this pin.

• 20:44Justine Moore?

  Yes.

• 20:44Steph Smith?

  Or I like have to take off my necklace when I shower or like I've literally forgotten that I had AirPods in and I'm like showering, right?

• 20:51Justine Moore?

  AirPods is a good one. I think because Apple tends to be a more closed ecosystem, especially with newer devices like the AirPods, there hasn't been a ton built on top of them yet.

• 21:00Bryan Kim?

  There was a time where there was a thesis of AirPod as a platform.

• 21:03Justine Moore?

  Yes.

• 21:04Bryan Kim?

  That hasn't really happened.

• 21:05Steph Smith?

  Yeah. Obviously, it's been an exciting week. We heard a couple of announcements. Knowing everything we've seen in the last two years from AI, this is not stopping, right? This is

• 21:13Bryan Kim?

  Never a dull day.

• 21:15Steph Smith?

  Just part of the long arc. Where do you guys think this goes?

• 21:18Bryan Kim?

  I think where this goes is to a natural conclusion, which is we mimic the technology to do the things that humans typically do.

• 21:26Steph Smith?

  Yeah.

• 21:26Bryan Kim?

  And again, we're giving it senses as we go along. And I think we just gave it an eye. The ability to communicate, ability to hear you, ability to listen to you, ability to see you is a really, really great start for a lot of the very interesting use cases.

• 21:40Justine Moore?

  To add to that, on the companion space in particular, it has been a large subculture of AI for a long time, driving a ton of innovation in both language models and honestly, image models.

• 21:51Bryan Kim?

  Subculture we are a very big fan of.

• 21:53Justine Moore?

  Yeah, we are quite deep in. But it was kind of by a lot of AI researchers and folks at the big companies kind of like looked down upon and that's not what we want to be building. We're going towards AGI, that sort of thing. And honestly, OpenAI choosing to show off those voices and the tweets that they put out about it afterwards sort of legitimizes the space in a way I think that's really interesting. And that may prompt more established companies, researchers, developers to build in the space and more people to talk about using the products, which I think as people who have been studying this space for a long time is very exciting.

• 22:28Steph Smith?

  Yeah, I mean, you can't ignore the demand, right?

• 22:30Justine Moore?

  Yes.

• 22:30Steph Smith?

  Are the usage. By the way, super quickly, you talk about adding an eye. This is going to not help the seriousness conversation around companionship, but you just imagine these models as Mr. Potato Head. And you're just slowly adding on the features. And you can almost just imagine, okay, he's got an eye now, right? He's got an ear now.

• 22:47Bryan Kim?

  I think that's very serious. The fact that you can treat a computer like a potato head that talks to you, listens to you and emotes with you and listens, it's incredible.

• 22:56Steph Smith?

  Yeah.

• 22:57Bryan Kim?

  I think the future is going to be very fun.

• 22:59Justine Moore?

  Maybe that should be the icon when you're talking because everyone is like the chat GPT voice. It's just like that black circle and maybe it should be a potato head.

• 23:06Bryan Kim?

  So that's another interesting thing. We're not using the screen real estate effectively with a voice only. So that's where I think the FaceTime element comes in.

• 23:14Justine Moore?

  The avatars.

• 23:15Steph Smith?

  All right, well, we'll have to do this again probably very soon. Well, thank you.

• 23:19Bryan Kim?

  Thank you.

• 23:19Justine Moore?

  Thanks.

• 23:21Steph Smith?

  All right, that's all for now. If you like this kind of episode where our partners break down the latest and greatest in consumer tech, let us know. Shoot us an email at [email protected] or drop us a review at ratethispodcast.com/a16z. And don't forget to subscribe so that you are the first to know when we drop our episode around A16Z's Gen AI 100 list. We'll see you then.