Udio & the age of multi-modal AI

2024 promises to be the year of multi-modal AI, and we are already seeing some amazing things. In this “fully connected” episode, Chris and Daniel explore the new Udio product/service for generating music. Then they dig into the differences between recent multi-modal efforts and more “traditional” ways of combining data modalities. Sponsors: Fly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Changelog News https://changelog.com/news – A podcast+newsletter combo that’s brief, entertaining & always on-point. Subscribe today https://changelog.com/news. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Udio https://www.udio.com/CLIP https://openai.com/research/clipBridgeTower https://arxiv.org/abs/2206.08657LLaVA https://llava-vl.github.io/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another fully connected episode of the Practical AI Podcast. In these fully connected episodes, Chris and I keep you fully connected with everything that's happening in the AI world, the news, the trends, the new models, all the good stuff, and talk through some things that will hopefully level up your machine learning game. I'm Daniel Whitenack. I am founder and CEO at Prediction Guard, and I'm joined, as always, by my co-host, Chris Benson, who is a principal AI research engineer at Lockheed Martin. How you doing, Chris?

• 1:18Chris Benson?

  Doing great today, Daniel. I don't know how we're gonna pick what to talk about. There's so much stuff coming out right now.

• 1:23Daniel Whitenack?

  There's a lot, mostly kind of new. Well, there's always new models, I guess, but it did seem like a big week with, I think, new GPT for Turbo, new Gemini. It's really hard for me to keep track of the numbers and parameter counts and all that, but I know that new Gemini had 1.5 something. I forget the different numbers.

• 1:46Chris Benson?

  Yep.

• 1:46Daniel Whitenack?

  And then new Mistral, new, I think it's a different mixture of experts. Top of the open LM leaderboard, we've got Udio, which I was just, so I've been at some startup accelerator stuff and then at conferences, events, and I'm like, it seems like I go into one of these events, and at the end of the day, People just say, oh, did you hear about X? And I'm like, no, I was, I haven't had my laptop open. Something else has happened, apparently.

• 2:18Chris Benson?

  For the last two hours, I haven't had my laptop and I've missed, I was doing work, you know?

• 2:23Daniel Whitenack?

  Yeah. I don't know if you've seen the trend. I think this was a prediction for 2024, which I think was a well-informed prediction for 2024 from many different people. And I think we talked about this in our own discussions of 2024 around multimodality of AI in 2024, whereas in 2023, you kind of saw this explosion of, in many cases, text to text AI, meaning I put in a text prompt and I get some text back. Now we're seeing an explosion of multiple modalities of data input and/or output

• 3:04Daniel Whitenack?

  to these models. And that's mostly what I'm seeing. Is that consistent with your view as well, Chris?

• 3:10Chris Benson?

  Totally. I was thinking about that in a moment for us to brag. I tell you what, we've actually been fairly good with our predictions the last few years.

• 3:18Daniel Whitenack?

  Who knows? Maybe we're actually setting the, it's a self fulfilling prophecy. It's just everyone's listening to practical AI and they're making our predictions real. That must be what it is, I'm sure.

• 3:30Chris Benson?

  You know that all of the, you know, like over at OpenAI, they're just listening to us and they're going, okay, that's what we need to go work on.

• 3:37Daniel Whitenack?

  It's a lot of pressure.

• 3:38Chris Benson?

  Yeah. And I'll tell you what, it's a good thing. We're steering the entire industry by ourselves, right?

• 3:42Daniel Whitenack?

  Yeah, which is, uh huh. Yeah, it's a lot of pressure, but it's good.

• 3:45Chris Benson?

  Sure. No pressure at all, man.

• 3:47Daniel Whitenack?

  I do want to talk about multimodality today, but I've just got to share with you some of this Udio stuff, Chris, which I think, well, Udio, Udio, I don't know if anyone knows how to say this yet. I was saying Udio, maybe it's Udio, U D I O, I believe.

• 4:06Chris Benson?

  I'm not going to hazard a guess till I hear somebody else do it.

• 4:10Daniel Whitenack?

  Yeah, yeah. It's coming out of beta. I think that there were some leaks of some of what they were doing before. But essentially, if you go to the This website, you can sign up for an account. They have it marked as beta. So I'm not sure exactly where this is going necessarily product wise. But what you see, at least in its beta form, is essentially a space where you can put in a text prompt. It kind of reminded me of almost like a clip drop or something. Some of these image generation platforms where you can kind of pre select some elements of the prompt.

• 4:49Daniel Whitenack?

  And the goal would be to completely generate a coherent and compelling song or piece of music or composition. It's essentially a music generator. So we've seen a little bit of this in the past, right? And in the past, we've kind of heard things generated like kind of dreamy ambient things. And, and maybe useful for kind of backing YouTube videos or something like that, but not really compelling music in and of itself. And I think what's interesting about this U D O is that it generates both this sort of compelling music,

• 5:32Daniel Whitenack?

  but also lyrics and also synthesized voices singing the lyrics all together in one. So Chris, before, while you were setting up your studio there to record the podcast, I was busy on U D O figuring out what, what is there. Now, there's a couple of really interesting ones that I listened to. And I've preloaded a couple in here for us to listen to and to give our audience a little bit of a sense of the audio.

• 6:02Chris Benson?

  Absolutely.

• 6:03Daniel Whitenack?

  Because this is an audio podcast, so you know, what better format? So one of these, which I found really intriguing was.

• 6:13Daniel Whitenack?

  Dune, the Broadway musical.

• 6:16Chris Benson?

  And I would go to that, by the way, just to make it very clear, I'm standing in line to buy tickets, so to speak.

• 6:21Daniel Whitenack?

  Well, the music has been generated for this, and I'll cite Bobby B., so Bobby B. on Udio, and he's created Dune, the Broadway musical. So just to give a sense of people, the prompt that went in to this, to create it, it says teen pop show tunes, film soundtrack, uplifting, playful, female vocalist, happy. Anyway, so you get a sense of, kind of similar to a, I guess, like an image generation prompt, where you're saying like, high resolution, Unreal Engine, like this sort of stuff, to give it some stylistic guidance. But I've got this preloaded in here, Chris.

• 7:06Daniel Whitenack?

  Everything that you're gonna hear is AI-generated. So let's listen to Song of Arrakis from Dune, the Broadway musical.

• 7:15Chris Benson?

  Oh, you got my attention, man.

• 7:37Daniel Whitenack?

  What do you think?

• 7:38Chris Benson?

  Move over Wicked, move over Les Miserables, all, just, you know, I saw Hamilton recently, move over Hamilton.

• 7:46Daniel Whitenack?

  Yes, yes.

• 7:46Chris Benson?

  We're all about Dune the Musical now, that's it.

• 7:48Daniel Whitenack?

  Exactly, yeah, so good. And even the lyrics, you know, "Eyes bright blue and hair jet black, you should see him ride on a sandworm's back." I mean, that's, that's good, right there.

• 8:00Chris Benson?

  I like the fact that the music actually deviated far from the darkness of Dune, you know, the perpetual darkness of the theme. That was fun, that was great.

• 8:11Daniel Whitenack?

  All right, ready for more?

• 8:12Chris Benson?

  Yeah, wow, yeah.

• 8:14Daniel Whitenack?

  So I tried out my own. Of course, I had to try out my own.

• 8:18Chris Benson?

  Of course.

• 8:18Daniel Whitenack?

  So my prompt that I put in, and I only experimented with this, so I'm sure you can do much better. But I put in a song about two podcast hosts trying to navigate the wild and crazy world of AI in the style of pop rock.

• 8:36Chris Benson?

  Practical AI, the musical?

• 8:38Daniel Whitenack?

  We really appreciate you, Breakmaster Cylinder, the mysterious Breakmaster Cylinder behind our theme music for the show. But this is what U D O can do. And I selected specifically to have it pop rock and to auto generate the lyrics. So I didn't put in any lyrics. So I have two options for you, Chris. I have two selections that you can see which one you like better. Here's the first.

• 9:36Daniel Whitenack?

  All right, selection one. Thoughts?

• 9:38Chris Benson?

  I like that one.

• 9:41Chris Benson?

  You just transported me from like, you know, 53, which is what I'm at now, all the way back to like 16 in the eighties, you know, late eighties, that I was all about that. That was good.

• 9:51Daniel Whitenack?

  Okay, cool. Yeah, I love it. Okay, that one was, they also generated a title. It's interesting. I don't know how much of this, you know, how many models are at play under the hood here and how they're coordinated. I'm guessing maybe there's some that generate the lyrics and some that generate the title. And then somehow that's merged together in a music generation because obviously the voice and the lyrics have to be coordinated somehow. I at least didn't see a lot of underlying explanation of what's going on here, but pretty interesting. And that was generated, I would say, in 30 seconds or something. I don't know, not that long, right? So let's take a listen to number two. This one was titled Digital Odyssey.

• 10:40Speaker 03

  Both host out and deep, my friend. Oh yeah. Oh yeah. Voices through the digital tide.

• 10:52Speaker 03

  They're learning as the codes blend. Got theories, theories. Everyone must hear a journey through the AI sphere.

• 11:07Daniel Whitenack?

  There you go. Even with a little bit of a guitar type solo there at the end.

• 11:11Chris Benson?

  I know it was good. I like, I liked the first one better. The first one felt like I was, you know, like I was a kid again right there, but, but I like both. And, uh, yeah, this is good. I could just spend all day generating music now that I may do that, actually.

• 11:25Daniel Whitenack?

  I might have just taken up your Saturday.

• 11:27Chris Benson?

  Oh my God. My wife has all these chores planned for me because we're recording here late on a Saturday morning and I'm, I may get myself in trouble by, by, yes.

• 11:36Daniel Whitenack?

  Okay. Well, there you go. So U D O, um, check it out. Super cool stuff. I think this does bring up some really interesting challenges, issues, uh, struggles, excitement also, of course, joy in hearing Dune the Musical. But it is super interesting. I even thought about this when I was going through here. Well, you know, Bobby B created Dune, the Broadway musical, and I just downloaded and I'm doing it what I want with it, which I guess is playing it on my podcast. So Bobby B, I, I hope you're okay with that. Now technically, technically this is machine generated.

• 12:16Daniel Whitenack?

  So at least as far as I understand, in the current US legal system, such a thing would not be copyrightable. Sorry, Bobby.

• 12:29Chris Benson?

  Sorry Bobby.

• 12:30Daniel Whitenack?

  I'm not giving legal advice here, obviously, and not a lawyer. But that's my understanding from our previous conversations. But what's interesting is, I think similar to these AI-generated art things that, you know, were put into art competitions and won, right? There could be, now in my case, I just put in a simple prompt that, you know, generated something in 30 seconds. There could be some really deep thought put into how to construct this prompt and the various kind of, I think Bobby's prompt was much better constructed than mine.

• 13:10Daniel Whitenack?

  And also you can upload your own lyrics into this to add a level of creativity. So there's really an open question here of how much human creativity is actually a big portion of this generation? And will the established laws and legal entities eventually recognize the creativity that's put into prompting these sorts of systems? Just like there was a time when there was a question of whether or not if you took a picture with a camera, right? You just click the button.

• 13:46Chris Benson?

  Yeah.

• 13:47Daniel Whitenack?

  Now photographers out there are gonna get really mad at me because I think they would recognize it's way more than clicking a button, right? There is a whole lot to photography and that's, I think, why it's been accepted as an art. But people argued at one point, you know, hey, you click that button, it's machine-generated, you can't have a copyright for that. But eventually that, those laws changed. So I wonder, Chris, if you have any thoughts about if or when that might change in these cases?

• 14:17Chris Benson?

  Yeah, I mean, I think it will because I guess if you look at these stream of AI advancements that we've covered over the years. There's a sense of inevitability that when these things come out, they catch on and they become popular and then they become the norm. And then eventually, as we keep seeing, the laws gradually catch up over time. And things like Udio, if I'm pronouncing the name right, is going to be typical. It won't just be them, there'll be others as well. And so I know that, you know, by way of example of that inevitability, we have a Spotify account for our family. And, you know, we're listening to the traditional way of streaming music historically.

• 14:58Chris Benson?

  And one of the things I do on that is I really like to explore new genres and new types of music that I don't know. And I'm always trying to think, how do I get to that?

• 15:06Daniel Whitenack?

  Yeah.

• 15:06Chris Benson?

  But I'm very likely to use something like Udio to prompt what I'm feeling, what I'm thinking, and try to explore new music that way. Because I don't really care, as a user, whether or not it's an artist that's human or an AI model that generated, if it sounds good to me. And so I think that sense of inevitability will bring about the change over time.

• 15:28Daniel Whitenack?

  Yeah, I think personally, I think right now even it's a gray area where, especially if you're going back and forth, like you're trying a prompt and then maybe you're modifying the lyrics. And if there's some sort of back and forth, that definitely gets into a little bit of a gray area where how much, even of the generated stuff, ignoring the creativity and the prompt, how much of the generated stuff is actually machine generated versus human post-edited, for example?

• 15:57Chris Benson?

  That's right.

• 15:58Daniel Whitenack?

  So, yeah, I think that even now, that's a bit of a gray area. But then my personal thought is eventually this will be more recognized as a creative pursuit, but, you know, we'll see.

• 16:11Chris Benson?

  You know, this new inroads into music through this AI model, with this being so far the most interesting that I've seen. This probably will really scare the music industry, you know, because this is taking it to a whole different level. And there probably will be a lot of lobbying, a lot of lawsuits. You know, we saw this past year, you know, actors going on strike because of AI-based video and the creation of characters or the representation potentially of live people. And I think we'll see some form of that here. This is a process we're going to go through over and over again. And I was talking to a good friend just the other day about this and life ahead and stuff and how to do this.

• 16:54Chris Benson?

  And I said, the smart people will align themselves with these capabilities. You know, it's not about whether it's a good future or a bad future or whatever from perspective, but it's an inevitable future. If I could give advice as a non-lawyer and non-professional musician in the music industry, but someone observing this, I would say, find a way to get on board with it and make it work for you quickly because it's not going away.

• 17:22Chris Benson?

  Yeah.

• 17:22Daniel Whitenack?

  Yeah, very true. And I do also wonder, of course, those things that we just heard were completely AI-generated, but it's interesting to me that maybe a creative person who is, and there are many that are embracing some of these things, like musicians, could actually iterate very, very quickly on different ideas, putting their own voice to backing music or getting prompted with lyrics that aren't quite so good as what they would like, but gives them a creative starting point and, you know, really explore spaces that they might not have explored before. So that might be cool to see as well, that kind of human UDO teaming.

• 18:05Chris Benson?

  Yeah, I agree. And another thing that I think will become inevitable, it's, you know, this, so here's a startup idea for folks, is with all the advancements over the last few years and kind of emotional recognition from models and understanding if you combine a capability like this and you choose to opt in, which there's privacy concerns, obviously, with a service that also is monitoring yourself, you know, and maybe, maybe the data is only available to you, but can generate content that is exactly specific to what you're dealing with in life and when it, when you need to pick me up. Not only does it find the right music, but it finds the right lyrics for the situation

• 18:48Chris Benson?

  and stuff. And so there's a lot of interesting psychological considerations here that could be both good or bad, obviously. But I think that's pretty fascinating. I'm wondering if I can find a service in a few years that will do that. And it follows me through the day and I keep the content private to me in my account, but it gives me to pick me up when I want. That's what I'm looking for, for whoever is going to go out and do that in the world.

• 19:13Daniel Whitenack?

  Personal soundtrack and narration and vibe.

• 19:17Chris Benson?

  My life.

• 19:18Daniel Whitenack?

  Yeah.

• 19:35Speaker 00

  This is a changelog news break. Youtuber Internet of Bugs posted a lengthy breakdown exposing Devon's creators, cognition labs, for falsifying claims about their world's first AI software engineer. Devon was pitched as a fully autonomous software developer, and one of the more impressive demos showed it completing and getting paid for freelance jobs on Upwork. Sound too good to be true? It did to Internet of Bugs, who says, quote, "I broke down the Devon Upwork video frame by frame, and here I show what Devon was supposed to do, what it actually managed to do instead, and how bad a job of that

• 20:16Speaker 00

  it did. On the whole, that's not surprising, given the current state of generative AI, and I wouldn't be bothering to debunk it, except, one, the company lied about what Devon could do in the video description, and, two, a lot of people uncritically parroted the lie all over the internet, and, three, that caused a lot of non-technical people to believe that AI might replace programmers soon." End quote. Devon really did garner a lot of attention, also known as money, because of that demo. We talked about it on our shows with a healthy amount of skepticism, I think. But I'm thankful their claims have been debunked, and I hope we all give cognition labs the side eye from here on out. Exaggerating your development capabilities? Maybe Devon really is human after all.

• 21:00Speaker 00

  You just heard one of our five top stories from Monday's Change Log News. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com/news to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com/news.

• 21:27Daniel Whitenack?

  A lot of these things, like I say, are moving into this multimodal sphere. And it might be worth just kind of looking back a little bit at how we got to where we're at in terms of multimodal functionality, sort of how that gradually has changed over time from NLP and speech to multimodal models that we're seeing now. I think one good way to, if we kind of step back and look at it on a holistic or historical standpoint, you kind of started out with modes of data processing that were maybe separated,

• 22:12Daniel Whitenack?

  but often tied together in a sort of chained way. We didn't really think about it chaining at the time, right? But you had speech synthesis models, for example, that were really specifically trained to only do text to speech, right? And in some cases, even that was broken up into sub models of like a vocoder and other types of models. And you had text to text models. You had maybe computer vision models that would process images to do object recognition or even videos in certain cases or frames of videos. But all of these were specializations.

• 22:56Daniel Whitenack?

  So the whole idea of there being computer vision is right as a specialization is that I am specializing in models that process this mode. Of data, right? And speech technology, the discipline of speech technology is a discipline of really focusing on processing either speech inputs or speech outputs. And then NLP quote unquote had special models that would take in text and maybe classify or detect entities or do machine translation or or these sorts of things. So we kind of that historically was kind of how the field was developing. And if we skip kind of the

• 23:40Daniel Whitenack?

  middle portion and come back to it, now we've gotten to a point where there's seemingly these large foundation models that are able to take in multiple inputs at the same time of multiple modes. So for example, an image and text in the same input paired together and, you know, answer questions. So this would be like what we see with GPT vision or we see a text prompt or even an video input like we have seen with Gemini recently where you can import a whole video and ask for a summary of all the visual components and and that sort of thing. That's kind of from my end, how I view the the bookends.

• 24:25Daniel Whitenack?

  Is that also from your standpoint, any comments on that, Chris, in terms of how we've progressed from one end to the other of that?

• 24:32Chris Benson?

  I'll pivot slightly in response to that and say, as you were describing that, it really resonated with me on with something else that I've been thinking in that development. And that is, I consume a lot of content through audiobooks. Anytime I'm kind of on autopilot, you know, driving or mowing the lawn or doing any kind of thing, I'm listening to audiobooks for learning purposes, mostly at kind of a double speed, as fast as my brain can process it because I like to consume as much as I can.

• 25:03Daniel Whitenack?

  I can't do that. My mind is too slow.

• 25:05Chris Benson?

  No, you get used to it after a while. But it's just to get the information in. And I just went through a really Pulitzer Prize winning book through audio called An Immense World by Ed Yong, which is fascinating and I highly recommend it to anybody that wants to do it. But it is all about the way we and all animals and not only humans, but all animals in their unique ways perceive the world through their senses and how vastly different those are. And the theme that came up to me throughout that was how multimodal everything about humans are. The way that we learn, our experiences are all multimodal. We don't have just vision and just audio and just text. We're taking it all in at the same time.

• 25:49Chris Benson?

  I think this progression that we've seen in terms of moving into multimodal this year has been really fascinating in terms of coming in to really how we take in information and how we learn. And I think going back to Udio today and seeing what they're doing and looking at the other multimodal capabilities that we've been learning. It feels like we're finally getting to some, I know we keep saying this. It's always kind of cool at the moment when the new thing comes out. But it feels like it's really aligning with what it means to be human as well. Ironically, that's the background thought process I had as you were going through that.

• 26:21Daniel Whitenack?

  Yeah, there's these scenarios in which knowledge is definite. Like we process knowledge across multiple modes of data inputs. And certain things are not all, you know, many things are not all represented in text or in any given mode. And I think you've seen this already kind of utility over this with things like GPT Vision, which is a kind of visual instruction tuned model. And maybe that's something to share with the audience. If you're not familiar, there's kind of this, the music generation stuff. Maybe that's a little bit newer, but there's kind of this ongoing work

• 27:04Daniel Whitenack?

  in visual instruction tuning. And this would be the type of model in which you would have an image input and maybe a text prompt. And traditionally, like I remember, I think there's even some of these models still that are quite popular to use on, for example, AWS Texttract, for example, is a OCR system, but you can also do visual question answering. Now, it used to be you had a specific model architecture for visual question answering. It was a research topic in and of itself. There was a specialized model and this kind of illustrates some of the progression that we've had.

• 27:49Daniel Whitenack?

  There was a very specific discipline around visual question answering and very specific models that could do those things and they advanced. But then Recently, you've got what has begun being termed visual instruction tuning for models, where the models are actually similar foundation models to what people are using for other modes. So for example, if we look at the lava model, L L A V A, so not llama, but lava, maybe a bit hard to distinguish in the audio, that's a open source manifestation

• 28:29Daniel Whitenack?

  of the GPT vision system, or similar functionality to that. And if we look at how that operates, it actually is built off of, and this is kind of, we talk about this a lot on the podcast, Chris, where you're always sort of building on the shoulders of giants and a lot of what's come before, even though some of these functionalities seem to pop up out of nowhere. But there are kind of previous signals. And Chris, I don't know if you remember, we, I think, had an episode where we talked about CLIP.

• 29:01Chris Benson?

  Yes.

• 29:01Daniel Whitenack?

  Which was a multimodal way to embed both text and images.

• 29:07Chris Benson?

  Yep.

• 29:08Daniel Whitenack?

  Something developed by OpenAI.

• 29:09Chris Benson?

  Contrast of language image pre-training is what CLIP is from OpenAI.

• 29:14Daniel Whitenack?

  Correct, which thankfully is open to everyone back in the days when OpenAI was open, and we can still use it. But CLIP allows you to embed an image or text in a similar embedding space, which means you're converting an image or a piece of text into a set of numbers. And if you compare those sets of numbers in that vector space, you can actually find things that are semantically similar by the distance between those vectors, which is interesting and kind of makes immediate sense if you're doing text to text things, like the semantics of one piece of text or the meaning of one piece of text could be

• 29:58Daniel Whitenack?

  similar to the meaning of another piece of text. It's very intriguing, though, if you make this multimodal and say, a nice sunset on a beach in Florida, and then you have an image. Of a sunset somewhere on a beach. And then you have an image of a car driving through New York. And then you have an image of a spaceship in outer space. And you could actually find which of those images is semantically similar to a text input. Right. So that's kind of the clip way of embedding things. And then on the other side, you have large language models, right? Which can take a text prompt, reason over that text prompt, even though it's not really reasoning.

• 30:42Daniel Whitenack?

  It's just auto completing, but we can think of, you know, functionally, it takes in that prompt and outputs some output related to the question that's input, the query, the instruction, that sort of thing. So what they've done with Lava, which has been around for some time and people have built different types of lava models and sort of its own family in and of itself, is paired the CLIP style embedding model, or a visual encoding system, with a large language model and then created this text and image input.

• 31:21Daniel Whitenack?

  So if you look at the architecture of what they do, what happens is they have an image input that goes through the vision encoder, for example, CLIP, that produces an embedding. They have a language model like Llama that accepts a language instruction or text input, and that creates an internal hidden representation embedding. And the first thing they do is they train a projection matrix for the vision encoder.

• 31:52Chris Benson?

  And can you talk about what a, real quick, what a projection matrix is?

• 31:56Daniel Whitenack?

  Yeah, yeah. So the language model produces an embedding, embedded representation of the text. The vision model creates an embedded representation of the image. But these two are different model architectures and the embeddings can't be directly compared one to another because one's Llama and one's Clip, even though they functionally produce embeddings. So the projection matrix is a sort of translation of the output of the vision encoder, the QIPT model into a space in which it's concatenated or combined together in some way with the output of the Llama model. And that is a trained projection such that it accomplishes the end tasks that you're

• 32:40Daniel Whitenack?

  training for, like a visual question answering or reasoning over an image, that sort of thing. So that's the initial pre-training is finding that projection matrix. And the interesting thing here is actually this, it's a combination of models, which is intriguing, right? Because you can always update Llama to the next cool thing like Gemma, or you could always update Clip to the next cool thing like Bridge Tower from Intel and combine them in really interesting ways and do this retraining. And then people then fine tune these models based on data sets that they've created for specific tasks, like we've seen with language models. So there might be a science question and answer for reasoning over science images, right?

• 33:25Daniel Whitenack?

  Or sort of visual chat in a specific domain. So to give people a sense of the functionality of this type of model, if you haven't played around with GPT Vision or something like that, one of the examples on the Lava paper site, which I find interesting is there's like a meme image of a world map, but it's made out of chicken nuggets. And so it looks like a world, but it's made out of chicken nuggets. So the picture is there and then the user input, the text input, along with the image input is can you explain this meme in detail? So there's some element of the question that's needed to answer that question because

• 34:08Daniel Whitenack?

  you're saying this is a meme, you're asking for specific details, right?

• 34:14Daniel Whitenack?

  And then you definitely need the visual content to answer that question. Otherwise, you would just hallucinate something about a meme.

• 34:22Chris Benson?

  Sure.

• 34:23Daniel Whitenack?

  So the lava answer is the meme in this image is a creative and humorous take on food with a focus on chicken nuggets. As the center of the universe, the meme begins blah blah and it essentially explains the humor, which is maybe not the best way to make something more humorous.

• 34:40Chris Benson?

  Yeah, a little bit dry there.

• 34:42Daniel Whitenack?

  Yeah, but you can think of other cases where you would need sort of visual input and text input to create an answer, right? Like if you say, what was the guy who raised his right hand in this video wearing, right? That necessarily like as a human, we would process that both from the text input standpoint and the visual content standpoint. And so I think it's really interesting, this sort of exploration of not just chained models from multiple modes together. Like we've seen in the past, kind of in history, where you have a speech model, you have a computer vision model, you have a language model.

• 35:22Daniel Whitenack?

  And you can chain them together in interesting ways, but this joint encoding, this joint processing of multiple modes of data at the same time is actually required for some of the types of reasoning that we might want to augment or automate from a standpoint of how we process information as as humans. So yeah, I would recommend that people kind of look into this lava model. It's open, there's multiple, like I say, it's sort of a family or a style of doing things. So there's a bunch of examples of that on Hugging Face and demos that you can actually try out. And try kind of an open version of what you get with GPT vision.

• 36:04Chris Benson?

  It sounds good. I've just been struck through our entire conversation, going back to what I mentioned earlier about how how close this is in terms of matching how we as humans process, you know, as you as you took us through the merging of the modalities a few minutes ago, and me having just, I'm actually in the middle of a, I think it's called The Great Courses, and I'm about the best brain, and it's talking about how exactly that happens in our brain to convert it into a form which is a chemical, you know, electrical in nature for our brains to actually operate on, since we don't actually see and smell. And so it's just fascinating that while the underlying AI models are not the way the brain operates, that kind of the

• 36:49Chris Benson?

  modalities are starting to merge in that way. It's really neat. So, thank you very much for taking us through that understanding of how multimodality works in a practical sense. You lived up to practical AI in all ways there.

• 37:05Daniel Whitenack?

  Well, this episode has a little bit for everybody where you get a fun Broadway song, and then we also talk about projection matrices. So, there you go. Something for everyone. I enjoyed it, Chris. And yeah, I think this will be a trend that we continue seeing throughout this year. So if you haven't got hands on and tried a little bit of this multimodal stuff, whether you go to Udio and try to create a song, or you go to ChatGPT and try to use GPT Vision or Gemini and process a video, or download the Lava model and try to run some multimodal queries, that's the best way to sort of get an intuition for how these things behave and what's possible. We'd really encourage you to get get hands on. So, homework

• 37:49Daniel Whitenack?

  assignment between now and next episode, I guess.

• 37:53Chris Benson?

  Absolutely.

• 37:54Daniel Whitenack?

  All right. It's been fun, Chris. We'll talk to you soon.

• 37:56Chris Benson?

  Take care, Daniel.

• 38:05Speaker 00

  All right, that is Practical AI for this week. Subscribe now if you haven't already. Head to practicalai.fm for all the ways and join our free Slack team where you can hang out with Daniel, Chris, and the entire change log community. Sign up today at practicalai.fm/community. Thanks again to our partners at fly.io, to our beat freaking residents, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.