E6: The Computer Vision Revolution with Junnan Li and Dongxu Li of BLIP and BLIP2

As recently as January 2021, the challenge of "interpreting what is going on in a photograph" was considered "nowhere near solved." Today's guests Junnan Li and Dongxu Li changed that with their publication and open-sourcing of BLIP, which delivered state-of-the-art performance on image captioning and other vision-language tasks. BLIP became the #18 most-cited AI paper of 2022, and now Junnan and Dongxu are back with BLIP-2, this time showing how small models can harness the power of existing foundation models to do multi-modal tasks. We talked to Junnan and Dongxu about their research and how they see the trend toward connector models shaping the future. We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. Timestamps: (00:00) Preview (01:17) Sponsor (01:35) Intro (05:50) Convergence of AI techniques (07:33) Evolution of BLIP to BLIP-2 (08:12) How BLIP-2 unlocked multimodal functionality (12:43) The size, training dynamics, and optimization function of BLIP (20:15) Practical/Business applications of BLIP (29:43) Efficiency of BLIP-2 compared to other models (41:52) Two-stage pre-training (47:11) Architecture of Blip-2’s connector model (58:52) Language models as the executive function of the brain (01:07:32) Vision for an ultimate multimodal system and democratized pre-training for models (01:12:59) Useful AI tools in these researchers’ day-to-day (01:14:56) Upcoming projects SPONSORS: SHOPIFY | OMNEKY Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive *Thank you Omneky for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. Twitter: @CogRev_Podcast @LiJunnan0409 (Junnan Li) @DongxuLi_(Dongxu Li) @labenz (Nathan) @eriktorenberg (Erik) Join 1000's of subscribers of our Substack: https://cognitiverevolution.substack.com Websites: Episode transcript at Cognitivervolution.ai huggingface.co/spaces/Salesforce/BLIP huggingface.co/spaces/Salesforce/BLIP2 Show Notes: - Original BLIP demo - BLIP 2 demo - BLIP is the #18 most highly-cited paper in AI - Image captioning comparison tool - Understanding images with AI - for use in language models and image generation - Image Aesthetics - Product & Model Reviews This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  But in a way, it's not that dissimilar from how we see, right? Like, we have our eyes, the eyes kind of take in raw light and turn that into a signal, and that signal goes through the nerve and finally gets back to the back of the brain. And by that point, it's not that interpretable either, right? It doesn't necessarily correspond to language. But then there's some further connector that like turns that visual data into something that I can understand as language, or at least understand and then articulate as language. So it feels like there is something kind of analogous taking shape in the AI world.

• 0:36Speaker 01

  Imagine you are human, you grow up learning only knowledge, and now one day you open your eyes, you don't really know how to interpret what you see. So that's what we are going to do here, to kind of build this bridge between these two modalities.

• 0:54Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Eric Torenberg.

• 1:17Speaker 02

  Omniki uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omniki so much that I invested in it, and I recommend you use it, too. Use Cogrev to get a 10% discount.

• 1:35Nathan Labenz?

  Today's episode was a fun one for me. Researchers Junnan Li and Dongxu Li, both of Salesforce Research's Singapore office, have co-authored some of the most practically useful computer vision papers of the last year. As recently as January 2021, the challenge of using AI to interpret what is going on in a photograph was considered to be nowhere near solved. But just a year later, Junnan and Dongxu changed all that by publishing and open sourcing BLIP, a family of pre-trained models that delivered state-of-the-art performance on image captioning, visual question answering, and image text matching.

• 2:15Nathan Labenz?

  For Waymark, my company, Blip was a godsend. Suddenly, we had a reliable way to understand the contents of users' images, allowing us to make useful image suggestions for the very first time. This was something we had worked toward for years. Unusually in today's AI landscape, Blip has held the title of Best Image Captioner for over a year, ultimately becoming the 18th most cited AI paper of 2022. And more recently, just as worthy rivals to Blip started to come online, Junnan and Dongxu changed the game again with Blip 2. As an aside, for a funny moment in cognitive revolution history, you can listen back to episode 1,

• 2:57Nathan Labenz?

  in which Su Hail tells me about the release of Blip 2 live on the show, forcing me to clear my calendar for the rest of the afternoon to go check it out. Now, Blip 2 uses a different approach, which I think may ultimately prove even more influential. Rather than training a large model end-to-end, this time, they trained a much smaller model that connects a frozen vision model to a frozen language model. This strategy has several benefits. First, because it injects semantic visual information into the language model's latent space, you can now have an open-ended dialogue about an image in which the language model shows remarkably detailed and nuanced understanding. Second, because the connector model is so much smaller,

• 3:42Nathan Labenz?

  training time and cost are dramatically reduced. Blip 2 was trained on just a single A100 machine in less than 10 days, making it easy to upgrade the system as new and more powerful language models become available. You can even use your own fine-tuned language models as well. Small connector models like Blip 2 show just how much potential remains to be drawn out of today's large language models, and seem likely to play an important role in the great implementation of multimodal AI across society. One note for listeners: both Junnan and Dongxu are native Chinese speakers, and while both are perfectly fluent in English, audience members listening at 2X speed might benefit from watching this episode on YouTube,

• 4:27Nathan Labenz?

  where we've also included subtitles for your convenience. Now enjoy our conversation with Junnan Li and Dongxu Li. Junnan Li and Dongxu Li, welcome to the Cognitive Revolution.

• 4:40Speaker 01

  Thanks. Excited to be here. Thanks for having us.

• 4:44Nathan Labenz?

  So you guys have really done some outstanding work in the subdomain of AI known as computer vision. I guess I would even ask you guys right off the bat, like, do you think of yourselves as computer vision specialists? Or now with Blip 2 and your work kind of branching into multimodal and models being involved, do you think of yourselves even as kind of beyond computer vision? I'd love to hear how you think about where you fit into the broader AI landscape.

• 5:14Speaker 01

  Yeah, I think with how AI has been developing, I think the the boundary between different fields is kind of blurry now. So I think both Dongxu and I, we started as computer vision people during our PhD, but gradually like it doesn't require like too many expertise really to move to another domain. But that's why we started with explore other like language domain and also other domains. And try to see how can we also be involved to build better AI models.

• 5:44Speaker 03

  You know, he's just trying to be humble. He actually has a lot of expertise.

• 5:50Nathan Labenz?

  But there is definitely this kind of great convergence phenomenon. Transformers kind of increasingly working for everything. It is amazing to see how quickly people can move from what used to be one subfield to another and just how how much the same techniques are really starting to work across these domains. So I find that super fascinating. You guys have also done something that is pretty rare in today's AI landscape, which is that you put out a model almost a year ago now, right? If not, maybe a full year ago was the original release of Blip. And that continued to be state of the art or really neck and neck with one or two other models for state of the art

• 6:36Nathan Labenz?

  captioning, image captioning model, all the way up until you released your most recent paper, which is Blip 2, which obviously supersedes Blip 1. And I think now is safe to say is kind of the state of the art, both captioning model.

• 6:53Nathan Labenz?

  Question answering model as it pertains to images and really starting to unlock like longer dialogues and true understanding of images beyond kind of bite sized captions that we've seen in the past. So I think that's really an amazing accomplishment. In today's AI world, we don't see too many things that can hold the top of the leaderboard for a year. Typically, it's more like a month. Sometimes it's more like a week. But you guys put something out there that really in in today's AI context, a year I would call enduring work. So I want to compliment you on that. But I also want to dig into both kind of the original blip and blip two and understand like, are you guys got into these projects, how they work, how you train them? And we really do want to go pretty deep on

• 7:35Nathan Labenz?

  that. So maybe let's start with the original blip. Tell us about kind of the origin of that project and how it all came together.

• 7:46Speaker 01

  Sure, sure. Thanks for the compliment, by the way. So I think actually before blip, we have a previous one or two paper in this vision language domain. So for me, actually, my first paper is this ALBAM. I'm not sure if you have heard, but it's also been accepted at the Europe. And that is kind of when we started to explore this view and feel like there's something we can do here. So in the ALBAM, it's kind of at the same time as CLIP from Open here was published, we published ALBAM short after that. And in that paper, we found out that the idea was that we want to build this kind of multi model encoder

• 8:30Speaker 01

  that can understand both image and text. So CLIP is kind of a uni model encoder, right? You have the image encoder, you have text encoder, a computer similarity. And what we found is that we can build on top of that kind of contrast learning approach to have another this fusion encoder, we encode both image and text together. And we find that achieves quite good performance on some task that requires understanding of both image and text because of this fusion mechanism. So that's actually where we started. And then we find out that this kind of encoder architecture is good at some understanding

• 9:10Speaker 01

  task, but it's not really that good at generation task, inverted text generation task, like captioning or BQA that requires some open end generation. So in the BEEF paper, actually, the architecture, we heavily inherit our L-BIF architecture, but we made some change such that the encoder can also function as a decoder. So we propose this, what we call a mixture of encoder and decoder model, which basically like a single model with shared parameter, but you can switch from either decoder or decoder. So that's where the architecture kind of, uh, uh, comes from.

• 9:52Speaker 01

  And then we also find out that, uh, in order to train this model, right, you need a lot of data from this web domain, like image text pairs. And, uh, for, for previous method like CLIP, you just want to do contrastive learning, right? You want to learn encoder. Then it's fine that you have a lot of noise, you know, dataset like mine, those big data set have a lot of noise. If you just want to do contrastive learning, to do, uh, repetition learning, uh, the noise is fine most of the time. But if you want to do image captioning, this noise can really be quite harmful, uh, because this language model also is more like a finer grain version than the contrastive loss.

• 10:32Speaker 01

  So, so that's where another kind of contribution of our BEEF was that we kind of bootstrap this dataset, uh, so that, uh, it has synthetic generated captions and also we use this filter to remove the noisy captions. Uh, and that's basically these two pieces, the architecture perspective and the dataset perspective together makes BEEF work.

• 11:01Nathan Labenz?

  You filtered, or did you, how did you remove the noise from the dataset? Some of the captions that actually describe the image have a high similarity score with their, the corresponding text encoding, but then it's just such a big dataset, there's so much noise in there, got a lot of things that are just like, what an awesome day. And that doesn't really line up with what's in the image, so that. And add noise to the system. It sounds like in your case to create a captioner, this was working against you. So did you just go through the giant data set and just kind of filter out things with low similarity score and work from that?

• 11:39Speaker 01

  Yeah, exactly like what you described. We just go through all the examples and filter out those that are not aligned with the image. And I think it's a perfect point that in some case that noise could be good. Like I mentioned in contrast, we're learning stage noise could be good because really you are just learning one single vector. You are not really trying to generate the text. But in the more like captioning space, let's say you give an image, you want to generate text. If the text is totally irrelevant to the image and the model could learn something that that we are now trying to learn. It learns, it doesn't capture anything from the image. It tries to hallucinate some stuff out of the, uh, basically there's no context.

• 12:22Speaker 01

  So I think that's something we try to avoid.

• 12:25Nathan Labenz?

  Remind me, how big is the original BLIP model? It's like one to two gigabytes downloaded, if I recall correctly. It's not huge, but it's. So yeah, tell me, tell me about the size and kind of the training dynamics. I want to contrast that ultimately to the BLIP two successor.

• 12:43Speaker 01

  Yeah, I think that's actually a beauty part about BLIP two is that if you compare the number of trainable parameters. So there's trainable minions that you are using backpropagation to optimize the gradient during this pre-training. Actually, BLIP is larger than BLIP two in that trainable parameter, the count for that. BLIP have a few hundred minions, but BLIP two only have less than 200 minions. So the reason is, I think in BLIP, we are training everything end to end. So including this vision encoder, this language model, everything we train end to end. But in BLIP two, we have this frozen image encoder, which we don't have data at all. And we have this frozen large language model, which is very large.

• 13:27Speaker 01

  It has can be a few billion parameter. But because we keep it frozen during pre-training, it's actually, it costs very little computation cost. So if you just compare how much time and GPU we need to take during this pre-training, BLIP two is actually cheaper than BLIP. Because we use this already off the shelf available models are built by other amazing research teams.

• 13:52Nathan Labenz?

  On the on the original blip, is the what are you optimizing for there? Are you just optimizing for generating the exact caption like token by token? Or is there some more kind of abstracted or like semantic loss function that you're optimizing against?

• 14:11Speaker 01

  So actually, this blip, right, is inheritor of this ALBAP paper. So the our optimization function, we actually, if you look at ALBAP, BLIP and BLIP2, we have kind of this foundational laws, which are very similar. So there are these three laws that we use. One is the contrastive learning, like it's the same as what CLIP use to learn better reputations and align this image and text orders.

• 14:41Nathan Labenz?

  Can you unpack the contrastive learning a little bit better? I mean, I think most people will be familiar with CLIP. And I think they'll understand it largely as kind of the thing that somehow, some way, stable diffusion somehow is downstream of, and so it's important. I don't think people have a great sense in general of what exactly the insight there was that created that possibility. So give us a little bit more on that before we go on to the next two papers.

• 15:15Speaker 01

  Sure, sure. So this term contrastive learning actually already is before CLIP is from one like computer vision field. I would say that so basically, the idea is that you want to learn some kind of reputations for your data, such that if your data is similar, like in semantic meanings, they should have similar reputation. And when we consider this for image and text domain, what it does is that you have this positive pairs. So called positive pairs is what you collected from the web is you have an image and text and they are they are correlated. And you want to train the model such that their reputations for this positive pair

• 16:01Speaker 01

  is more similar to each other compared to the similarity for negative pairs. Those negative pairs are basically random sampled image and text pairs that don't correlate to each other. And so what this loss does is it trains this image and text encoder. They they don't interact with each other until the last stage. So they they will extract this image and text features individually. Uh, that's why we also call them like uni model encoders, because they don't interact uh with each other. After you extract the features, the final stage is you compute their similarity. So that's where they interact. And it's very simple mechanism.

• 16:43Speaker 01

  You just use the dot product, which measures the cosine similarity of this normalized embeddings. And you try to maximize the embeddings, the similarity of the embeddings from this positive pair. So that's kind of contrastive learning. How it works is is quite the simple mechanism. Uh, and in the end, what you'll get is you get this very good image and text encoders that can produce good reputations that represents what they their semantic meanings are. That's why like the V I T of Klib and text encoder of Klib has been really successful applied to different data tasks, because it captured the semantic meanings of those data.

• 17:25Nathan Labenz?

  Cool. Go, let's go back to the history then.

• 17:28Speaker 01

  Yeah. So uh, so the first loss, right, is this contrastive loss that gives you good uni model encoders. But then if you want to do more finer grain interaction between the image and text, you need more than just a dot product. You need some parameterized mechanism to interact. So that's where we have this cross attention, uh, where you. Have the text encoder that can cross attend to the image encoder. So it's kind of like consider a T five encoder decoder architecture where the encoder is your V I T. And your decoder is your text model. Uh so in the term, uh in in while we consider image captioning laws.

• 18:12Speaker 01

  Basically, this text decoder will cross attend to the image encoder and generate the the text tokens. So that's basically our second laws, which is the standard image captioning laws. Uh it's just a not not language model laws, but conditioned on on image.

• 18:34Speaker 01

  And our third laws, right, is is what we call image text matching laws. So the purpose of this laws is that we want to learn even finer grain similarity. Or or kind of matching between the image and text through this cross attention mechanism. So so you can't really expect a single vector to capture all the finer grain details of one image, right? Because image is worse, something worse, so there are too many ways to describe this image. So so uh this a single vector is very concentrated and good rotation. But if you really want to find a grain, you need this cross attention.

• 19:15Speaker 01

  So so for this image text matching laws is kind of a binary classification task. Where we give the model a image and text pair, right? And we ask the model through this cross attention after this computation. To tell me whether this is a matched uh a pair or not matched pair. So by doing this, we can further enhance this alignment between these two modalities. Uh so this is basically what the image text matching loss does. And in our experiment, we find that these three laws complement each other. Being that they have this uh multitask learning kind of uh objective will enhance the final performance. For each individual loss uh of of and objective.

• 20:00Speaker 01

  So this work that through different experiment, that's why we we fix those three laws as kind of the standard laws losing. And I think uh many other papers also kind of now will adopt this three laws or something similar.

• 20:16Nathan Labenz?

  One of the things that I think Blip. Has notably done better than any other model that I've tried is handling logos. Like can almost read the logos. A lot of times it like kind of fudges the words from the logo. So help us understand that. Like as a, you know, just as a user, I see this really interesting behavior. Other captioning models really struggle with logos and you may know that. Bleep does quite well. How did you manage to do that? Was that a matter of like the training data or was there, is there a technical reason that that happens? And then also, when I do see something, if it's like the Coca-Cola logo, it'll just know, okay, that's the Salesforce logo, it's going to know that's the Salesforce logo.

• 20:59Nathan Labenz?

  But when I have these long tail, small business logos, it's like probably never seen them before. I imagine most of them are not in the training set. And then you'll see these things where it might be like Torenberg Plumbing. And instead it'll say like Torrestein Plumbing, right? It'll be like almost the right name, but it'll kind of just flip it a little bit on like the last couple of letters and the last token. And I've always kind of wondered like, what is happening there? You know, where it clearly can sort of recognize the letters, but it's not doing like exactly an OCR type mechanism, obviously. And so I'm always kind of confused as to how it ends up just being a little bit off in those scenarios. But I'd love to understand that.

• 21:42Speaker 01

  Yeah, I think that's quite a fascinating phenomenon. I really didn't really observe this before. But I think in terms of why Bleep can understand logos, I would say it's mostly from the pre-training data that we use. And we are well the first to scale up to this IAN dataset.

• 22:03Speaker 01

  Yeah, I think we, once after LINE released this 400 million dataset, we started to use it, which in the captioning model. And so there are a lot of logos in that data. So I would imagine Bleep can learn such information, web scale data. And of course, this data are quite noisy, right? So it's likely that some spelling error can lead to a wrong recognition of the logo text. Uh, but I would still say that it is not a perfect OCR model. Because it's not really designed to do OCR. And the the V I T itself, even though powerful, is not really. At least compared to the best OCR architecture, it's not the same mechanism, right?

• 22:47Speaker 01

  So OCR have this detect detect first and then you uh really zoom in, right? To each individual letters and you recognize. Uh, V I T is more like a holistic view of the image and you're trying to recognize more, maybe the most salient part of the text. So that's why if the text is small, uh, it may fail. And another reason like you mentioned is if it's not a common name that appears frequently in the training data. The model may not really know that name, so it lacks the prior information to spell the correct uh name. Uh, and I'm not sure, maybe Bleep Tool will be better at this because we are using an early knowledge model that have more knowledge about words.

• 23:29Speaker 01

  So maybe it knows more companies, so there's stronger prior to to give you the correct information.

• 23:36Speaker 03

  Yeah, I I think if I want to add something to that, like. Uh, I think that's the emerging uh advantage of this contrastive learning in the context of multimodal data. If you look at the uh some of the earlier models, I mean, uh, before Albe and the uh Glee. Uh, many models are actually using some off-the-shelf detectors to provide uh object level uh labels. But because of the usage of these uh object detectors, they won't be able to necessarily take into account this logo information. Because if you look at these detectors, uh.

• 24:19Speaker 03

  They they usually don't don't try to detect the logos, like the boxes, tables, but not the logos. I think one emerging property, since ever the contrastive learning was introduced by Blip and Alves, is that you don't need this off-the-shelf detectors anymore. And this contrastive loss, they really very well align the captions, each text tokens in the caption to the corresponding region in the image. And because of that, this text, if you see some text in the logo, that really aligns to the text appearing in the in the caption.

• 25:03Speaker 03

  So that gives kind of the alignment on this logo thing and the text words. And I think it is, there is no OCR loss for that, but we also have some other two members who try to adapt Blip to the OCR context, which turns out to work pretty well. And that really demonstrates this foundation model are critical to serve for general general purposes for the model model understanding. Without too much, you know, task-specific designs or task-specific,

• 25:42Speaker 03

  which is just a little bit adjustment, this foundation model could be used for different scenarios applications, which is correct.

• 25:52Nathan Labenz?

  So with the original Blip, you said it's end-to-end training. There's, is it correct to say that there is no knowledge before that end-to-end training? Like the the only text that Blip ever has seen is the image captions that are associated with the images? There's no other text pre-training or anything like that that's being built on top of? Do I have that right?

• 26:17Speaker 03

  Actually, we do use a pre-trained BERT model to initialize this text encoder. So of course, BERT now is not considered to to be the best language model, but still, if you consider its size, it has some decent text understanding capabilities. So that allows us to initialize with a model that already knows text. Then we start to train it on this caption to enhance its capability.

• 26:44Nathan Labenz?

  What was the total training time like for the original BLIP model?

• 26:51Speaker 01

  Yeah, so if I recall correctly, we used this 32 800 GPUs. And the training takes about, for the largest model, I believe, takes about a week to finish.

• 27:06Nathan Labenz?

  I was under the impression it was even longer. That's actually pretty reasonably efficient. How many cycles did you guys go through in the process of doing that research? Like, did you run that week-long training process five times, 50 times, 500 times?

• 27:24Nathan Labenz?

  We all see the one end product, but how many, you know, earlier versions of it were there that ultimately got tossed out?

• 27:31Speaker 01

  Yeah, so actually, your research really start with not the largest model and relatively like a smaller size and smaller training data. So we can iterate faster and maybe we have a few days. We know how the training is going and we can adjust. And because we already have this L-BAB work to kind of give us a solid this pre-training loss. We didn't really make any adjustments about the noises because we are quite certain it would work well. And we made some adjustments to the model architecture because we have this one model that can do both decoder and encoder, right? We do do some ablations on that.

• 28:16Speaker 01

  I think in total, maybe less than 100 trials, maybe 10s or 20s trials. Until we finalize the final model architecture and the pre-training strategy.

• 28:32Nathan Labenz?

  And what would you say were the biggest things that you learned or adjusted during that process? Is it like a sort of learning rate schedule or other hyperparameters or something else?

• 28:47Speaker 01

  Yeah, actually, uh, we didn't really have a single thing that gave us a significant boost. So I would say mostly the model is robust to different changes. So the, uh, we do observe some kind of instability in training if we increase the learning rate uh, too much, so the loss may go to none sometimes, uh, for mysterious reasons. So that's why we decided to kind of keep the learning rate a bit lower. And in terms of other hyperparameter like the batch size, we just feed feed in the largest batch that we can feed within the thirty two GPUs. And also the data and the pre-training loss, we didn't really make too much changes.

• 29:30Speaker 01

  We back tweaked the architecture a little bit, but it's more like there is a trade-off between your efficiency and your performance, and we find a sweet point, uh, that give us the best performance while still being efficient.

• 29:43Nathan Labenz?

  So this, it's thirty two GPUs and like seven days. So that is essentially, whatever, two hundred-ish GPU days. That's a significant amount of compute.

• 29:59Speaker 01

  Yeah, ju- just to jump in, so like i- if you, there are, I think, tons of models that spend way longer hours than that. So even, uh, some of the earlier models, they, although they are not as flexible, uh, they are way longer hours. I think that's also, uh, shows that this model is, our architecture is quite efficient to capture this model. Um, this two hundred, uh, GPU days, uh, is indeed a lot, uh, uh, I mean, uh.

• 30:43Speaker 03

  If you just think about it, it's awesome. And if you look at K and Align, especially this ones on Google.

• 30:53Speaker 03

  They maybe turns on like a tens of thousands of GPU hours, I would say, at least. And I think at least tens of times more training data than BLIP. While saying that, K still able to achieve comparable, I would say in most of cases, even better performance. I think that really demonstrate that it's important to make good choices on architecture and training strategies. In addition to proper scaling in terms of devices.

• 31:32Nathan Labenz?

  I don't know if you guys know the answer to this question, but I looked it up just today. How, if you don't know, I'd love to hear what you would guess is the number of times that the original BLIP model has been downloaded from Hugging Face in the last month. Just the last month.

• 31:52Speaker 03

  I'm not too sure, like, because I think BLIP was uploaded to Transformer not long ago, right? Transformer, the Hugging Face team integrated BLIP like half a year or maybe less. I would guess 1,000, maybe 3,000, I would say.

• 32:11Nathan Labenz?

  From just the last month, mind you, just under 20,000 downloads of the original.

• 32:19Speaker 03

  Which model was that? Action model or?

• 32:22Nathan Labenz?

  That's a good question. Let me see.

• 32:24Speaker 03

  Yeah, I just want to say that, yeah, we sometimes also take a look at the statistics and we feel quite excited about it. And we also learn quite a lot from how people are using, how people are using it. And we see a lot of different application scenarios that.

• 32:44Speaker 01

  We wasn't actually expecting, but it was amazing. And yeah, I just really appreciate the community effort and feedback. I think that really also helped us a lot in developing a better idea of what the model is doing and where we are going to in the future.

• 33:04Nathan Labenz?

  Just to answer your question, it is the BLIP image captioning base model that has been downloaded exactly, according to Hugging Face, 18,976 times in the last month.

• 33:16Speaker 01

  That's amazing.

• 33:17Nathan Labenz?

  I'd love to hear some of those stories of unexpected use cases. And maybe you could also just tell, give us like a little bit of guidelines for anyone who's thinking, okay, how do I get in on this? Action, I would say, by the way, too, for Waymark, we do not fine tune the model. We just use the base, the kind of web scale data that you trained it on. Base works really well for us. I'd love to help the audience understand, like, what would it take for them to do a fine tuning in terms of? Data set, any gotchas that you guys have observed? Compute resources that they would need to go into that?

• 33:53Speaker 01

  Yeah, yeah, I think that's definitely something I also want to say is that if you want to fine tune BLIP, we have the support in our LAVI's library. And so we spent a lot of time and effort last year to build this library that really offers you not only the very convenient inference using our models, but also training and benchmarking. So we set up this framework where you can do training of our pre-trained models, fine tune them quite easily using your custom data set. So all you need to do is maybe you prepare the data set in a certain format.

• 34:36Speaker 01

  And then you write some configuration files. I follow the default ones, but if you want to change some hyperparameters, you can also override them. And then you call our trainer master to train the model. And in terms of compute, I think it depends. Like, of course, if you have a lot of GPUs, that's great, but if not. It takes longer to train, but for the base model, like a few, one or two GPUs can still run the training. It just may take longer, you may need to do some gradient accumulations stuff. And I think any size of dataset would work.

• 35:16Speaker 01

  Of course, if you have only a tens of, uh, hundreds of captions, maybe. Uh, I I'm not sure how much effect it will have, but as I have hundreds of captions, I would say that's worth to try to fine tune. Uh, so that's kind of, uh, the strategy that, uh, I I I would suggest to do the BIP fine tuning. And for BIP tool, we also provide this fine tuning, uh, in the, in the same library. Uh, but that would maybe require a bit more computing resource because we use these large language models. But even we keep them frozen, uh, I think for the, uh, our smallest, this OPT two had, uh, two point seven billion.

• 35:59Speaker 01

  You still need a decent amount of GPU memory. Uh, I would say, I think a V hundred should be enough to, to, to run this fine tuning, uh, using the smallest BIP tool. Uh, model that, that, that, that will provide. Uh, but for BIP tool, I think most of the time you don't really need fine tuning. Because, uh, it's quite generalizable to different scenarios. Uh, and I've seen several, several use case, uh, uh, for you, you, using this captioning. Uh, I think last year we saw this quite interesting use case where they, they generate captions for this Pokemon images. And fine tune a stable diffusion to generate the Pokemon based on the, the text.

• 36:43Speaker 01

  So that's one, uh, interesting use case I've seen before. And recently I find there's another demo that use captions to do this image search. So that's also something we have been, uh, exploring is these texts are kind of a, a concentrated representation of your image. And it's human interpretable, right? Uh, and, and if you just try to translate every image to text, the amount of like information is, is concentrated to this very small. Uh, uh, literally text tokens, right? They occupy very small space compared to, to, to the original image. And you can use all these existing techniques like sentence embeddings to do very fast.

• 37:29Speaker 01

  similarity search across a wide database. And that gives you an alternative way to do this image to image search. And also even image to text search.

• 37:43Speaker 03

  Yeah, if I would to add something to how you want to use the the captioning model. I'll say first, I strongly recommend to take a look at the Bleep Bleep two model. Uh, we have tried that one in our uh some uh some of other uh recent experiments. And we find that the captions from Bleep two are significantly better than the Bleep. Um, this was happening there, I think it also depends kind of on your use case.

• 38:22Speaker 03

  Hopefully, the the model, the the captioning model released there was fun to um Coco caption. And the the effect of that, you will say, you you kind of observe that the description sometimes tends to be a little bit generic in a sense that it kind of lose the very um specific uh and names uh kind of stuff. If you look at the Bleep two, we have multiple versions of the captioning model release. And some of them are just pre-trained on the Lion and the other web data set. And if you use that model, that actually gives you very concrete and

• 39:08Speaker 03

  I would say customized names of the object. It can recognize, for example, the uh car makes, uh all these logos and kind of stuff. Which uh could provide out more useful information if you want to have a really finer grain of information uh there and really try the uh web pre uh web pre-trained a Bleep two version first before you actually go into the.

• 39:38Speaker 01

  And tuning face, which is more expensive. And usually, I would say the the two pre-trim version is probably strong enough for a lot of a lot of these applications.

• 39:53Nathan Labenz?

  You kind of anticipated another question I had with the car makes. So I'm in Detroit, Michigan. The auto industry now here is in our blood. One of the challenges that car makers have a lot of times is identifying their cars down to like the model. And then they've got the trim, which is like the specific package that they have. So do you think it would be, let's assume, and they have lots of data, right? Plenty of shots of all these cars. Do you think it's feasible to get to the point if you had a significant scale data set where you could get accurate down to, you know, the the model of the car and even like the details of the car?

• 40:39Nathan Labenz?

  Have you seen people push the performance to that? Like very high level in a in a narrow range with Blip?

• 40:46Speaker 01

  I think it's definitely possible. If you have good enough, then large even they said. I don't see any reason that you cannot do it. I think currently in terms of more like research perspective, we we we don't really have access to those finer grain data set. But we are trying to improve the tool on certain domains that could be more widely applicable. Like those you mentioned, we try to improve our OCR or try to improve some other capabilities by like specific tuning on the individual domains. So I would say that's definitely some way the a good way to improve the performance on specific downstream task.

• 41:33Nathan Labenz?

  Cool. Well, let's, I mean, we've kind of gone back and forth a little bit between Blip and Blip 2. And it's funny, because I'm just such a Blip stan that I want to talk about it from all these different angles. But your new thing is Blip 2, and that has superseded, as you said, even the original as more general and has this. I think the most fascinating part about it is the fact that it uses and really connects these pre-trained vision or image and then pre-trained language models and kind of helps them to work together. My gut says that this is going to be a big trend, right?

• 42:14Nathan Labenz?

  Because we're seeing the proliferation of the language models and as well as the image models, but especially language models are really going wild right now. And yet, you know, the 200 days, as much as that's not the biggest thing that's ever been done, it's prohibitive for most projects, right? Most people cannot get to that, if only because the cycle time is just going to be too slow and they just don't have the calendar time to figure all that out, right? That's, that's a lot of research. Now, Blip 2 also a lot of research, but the training time, because it's a connector model, because it's so many fewer parameters, I believe is down to 10 days on a single machine.

• 42:58Nathan Labenz?

  And I assume that could be paralyzed down to even shorter runtimes. So I really want to unpack a little bit this connector model. Tell us, like, where did the idea for the connection style model come from? I really want to get your vision for what that's going to look like over the next couple of years.

• 43:18Speaker 01

  Yeah, I think what you mentioned is a very insightful point. And that's something actually we try to push in the big two paper. Because initially, we find that there's all these amazing vision models, right? You have people just dedicating to pre-train better vision models with self-supervised learning or contrast learning. And you have another group of NLP researchers trying to push the boundary of this language models, right? Have instruction tuning and all these GPT style models. But for vision language domain, people are still doing pre-training from scratch. So that's kind of puzzling for me. Is why can't you just bring the available progress together and you do some kind of connection

• 44:02Speaker 01

  that can have a very flexible way to combine different models and efficient way so that you can harvest from this progress from the individual field. So that's kind of our motivation. I think in terms of the connector module itself, the architecture, we are heavily inspired a few from a few previous work because we use this kind of query technique to extract features. We have this cross attention. So a process inherited from Blee Blee's previous Blee architecture. We also inspire from models that the data, which is kind of the first model that uses query, one of the first to use this query to extract features.

• 44:48Speaker 01

  And also Flamingo is involved the previous work that also uses kind of query mechanism. But I think what we find different is not really the the architecture itself. So this connector module, there are different ways to build it. I think we choose one of the most efficient way so that you don't need to change anything about a language model, you just plug in as kind of prompt tuning. So so but I would like to highlight that the reason we make it work is because of the our pre-training strategy. And this is really something unique from from Blee too, I think, because we have this two-stage pre-training strategy. So what we did is that we first connect this connector to the vision model and do

• 45:32Speaker 01

  pre-training so that this connector is very well aligned with the vision model. So we can understand the vision information very well in terms of how the text can correlate to to to the image features. And then only after this stage, we plug in the language model. And adapt this connector so they can work as a bridge between this vision model and language model. And what we find in our paper is that if you remove the first stage, you just do a connection between these two models and you do this kind of image captioning loss. Then the performance becomes much, much worse and there will be, you know, like catastrophic forgetting,

• 46:14Speaker 01

  which was widely observed in previous papers like Domingo that just use this kind of generational of the, uh, of the star. So, so I think the reason is that we need this connector to have a good understanding first before it learns how to uh, teach the language model to generate. Because this language model, they are really large, they, they, they are prone to overfitting and they, they don't really have any understanding about the image. So that's why like this pre-training strategy, I would say is the most useful techniques that we try to propose. And this also applicable to other multimodal domains, right?

• 46:52Speaker 01

  You just uh, need this connector, you, you connect with the first module first, you do some pre-training, then you connect to the second module and you do some pre-training. I think that's quite a generic way to do it and we do hope that this can be applicable to other domains and power other applications.

• 47:11Nathan Labenz?

  So one of the things I think is most fascinating about it is your connector model, which really, I mean, Blip 2, when you use it is an ensemble, right? You have the image part, which is frozen, you have the language part frozen, then you have the connection in the middle, which is what you've trained. It is predicting embeddings direct, that get injected directly into the language model, correct? It's bypassing the text encoding and just going straight to the embedded text layer of the language model.

• 47:45Nathan Labenz?

  So that in and of itself is kind of like eye opening thing for me. It also creates some like discomfort for me in the sense that obviously at this in February of 2023, I don't think we're facing imminent danger from models like Blip 2. But I sort of extrapolate this trend out a little bit and I start to think, boy, you could really connect a lot of different sensors to language models in this way. And then you could really start to cobble together, not just, let's say, bimodal, but like a truly multimodal systems.

• 48:29Nathan Labenz?

  And those can start to do all kinds of things. But what is a little bit concerning to me is the the lack of understanding of exactly what is being injected into the language model. Like, how is this understanding happening, right? It becomes ultimately pretty inscrutable. So I wonder what you think about that. And also was curious about whether you have any way to figure out if those embeddings that it is predicting are like, you know, can you back port those to text? Is there a way to sort of understand, like, in a human legible way, what exactly is being injected into the language model?

• 49:12Speaker 01

  Yeah, yeah. And that's a great question. I think this kind of injecting embeddings has been there for a while from this prompt tuning kind of technique, right? Now you have this soft prompt, which are basically embeddings. And you learn those prompts and you prepend them to your text input and give this to the language model and. Somehow, it can guide the language model to predict certain things. Like you can fine tune the prompts to to guide the language model for a certain downstream task and it can give you better performance. And people have tried to interpret what this prompt means. And I think so far, the current conclusion is not really interpretable.

• 49:55Speaker 01

  Uh, it's kind of like a black box. Uh, what this soft prompt really captures. Because, uh, like I said, the language model is so big, right? There are so many hidden like kind of reputations that can guide it towards certain things. Uh, so in terms of this soft prompt, I would say BLIP2 is similar. That we are trying to provide prompts that can embed vision information. So, uh, we don't really know what exactly are these vision information. Uh, but they are reputations of the image that the language model can make use of. And and and I think why we are sure that they must be reputation of the image

• 50:38Speaker 01

  comes back to our first stage pre-training. Because in our first pre-training, we are using these contrast rules. We're using this image text mention rules. So that from this pre-training objective, we can be certain that this connector, like we call it a Q-former, is learning the most representative feature from the image. So it's kind of like a feature extractor, right? You extract good features from a frozen this image encoder. So we are certain of that because we we know these are good features that represent the image well. Then we are then that's why we are confident that if you put this to a language model, then it's most likely will teach the language model about the image rather than something else.

• 51:25Nathan Labenz?

  Yeah, I wonder if we can give folks even a little bit better intuition for this.

• 51:33Nathan Labenz?

  I mean, as you said earlier, a picture is worth a thousand words. And it is fascinating to me that, especially with relatively little compute, right? One machine for 10 days is kind of the total thing, that you can figure out a way to predict these injectable embeddings into this space, which was originally created by embedding text and which is interpreted as if it were text by the language model. And the loss is ultimately what is based on what comes out the other end of the language model. And that that all still works.

• 52:17Nathan Labenz?

  It's like, there is this sort of invisible, kind of dark space within the language embedding space that language itself cannot access, but which this model can learn to access in such a way that it is still immediately interpretable by the language model itself.

• 52:44Speaker 01

  It's definitely worth more research to really find out the working mechanism. I just want to mention some of our previous efforts. So we do have some previous paper that where we try to directly map the image to text, like human interpretable text focus. And we give this text as input to the language model and see what it can do. And we find that it's quite good. I say we generate captions and then we give these captions as the language model to the language model as context to answer questions or do some other task. It can perform well, but there are some limitations. And the major limitation is that those captions. They are, they, it's hard to represent everything of our image.

• 53:28Speaker 01

  So we need to first find relevant captions. Relevant meaning that it's relevant to my task. If I want to ask specific question, then I need this caption to be relevant to this specific question. So that I know the language model can make use of this. So and secondly, we need to generate a lot of captions. Like one caption is not enough. We need maybe generate twenty, thirty, or even more to hopefully capture more information about the image. So, so that's why we changed to this paradigm where we inject embeddings. Because each embedding is itself a vector. And I think it's seventy, uh, seven, seven hundred and sixty eight dimensional vector. And we have thirty two of these vectors.

• 54:11Speaker 01

  And this actually can capture quite a lot of information. Uh, because if you consider image as itself is just two hundred by two hundred pixels. And now we kind of try, uh, try to transfer the knowledge into these embedding vectors. How do we make sure this are interpretable by the language model, right? If we just randomly generate these vectors and we give the language model and we ask the language model to generate the text. It's likely there are thousands of ways the language model can interpret image and, and generate some text. So, so that's most previous approach are using, like they just train with this language model and all. At the end, right?

• 54:52Speaker 01

  So that this training signal is purely coming from the language model's output and back propagate to this connector. Uh, and there are a lot of ways for this connector to basically cheat. So it can cheat, does that during training, it can guide the language model to produce certain output. But it cannot really generalize. It doesn't really understand the image. Right? It's just cheating because it can change its own output and adapt to the language model. So again, that's why we need this first stage to make sure this connector itself has really good understanding of the image. And that's why during our two, three pre-training, right? Our first stage actually takes longer time. So we take maybe six days to pre-train the first stage.

• 55:37Speaker 01

  And the second stage, while we plug in the language model, we only need maybe two days. So that's the drastic differ from previous approach. And this also means that after we pre-train this connector first, we can plug in different language model and it doesn't take much time to adapt. So because the connector itself already have good understanding of the image, right? It's hard for itself to like, because they try to find the shortcut. So you, uh, deep learning models always try to learn the most easy solutions to, to certain problem. If there's a shortcut, it, it will find it. And so, and we, by making it to understand the image first, it's, it's shortcut is, the, the, these obvious solutions will disappear because it can already

• 56:23Speaker 01

  like overfeed, right? It's basically the easier solution for, for the connector is now I need to understand the image because it already knows the image. So that's become easier solution. So that's why I think it works well in our case. And we don't really just relying on the language model to teach the connector. I will kind of pre-teach it first so that it can teach the language model instead.

• 56:47Nathan Labenz?

  That's really interesting and definitely gives me a better understanding than I had just from reading the paper. So thank you. Have you gone as far as to try to find? Cause you, you might think, okay, what's the closest I could get if I just had text to the embeddings that are predicted by the connector model? Have you tried to figure that out? And if so, like, are you able to get at all close? Or is it just kind of like a totally different universe that the, that the connector? Or let's probably better analogy be totally different language that the connector model is speaking.

• 57:23Speaker 01

  Uh, yeah, I would say it's quite difficult to map the output of the connector to certain text tokens. Uh, because like you said, this, this language model space, input space is so huge, right? This, uh, I don't think it's really interpretable by human language. But I do think there is a lot of information there. Uh, maybe not in the form of human interpretable, but definitely interpretable that up by the language model. And I do see a lot of research work can be done in that space to try to make it less of a black box for us.

• 57:59Nathan Labenz?

  I kind of think of language models increasingly, especially with the rise of these connector models as kind of like the executive function of.

• 58:11Nathan Labenz?

  These kind of expanding ensembles that increasingly are going to be fairly general, I think, and can kind of do a lot of different things. In a way, it's not that dissimilar from how we see, right? Like, we have our eyes, the eyes kind of take in raw light and turn that into a signal, and that signal goes through the nerve and finally gets back to the back of the brain. And by that point, it's not that interpretable either, right? It doesn't necessarily correspond to language. But then there's some further connector that like turns that visual data into something that I can understand as language, or at least understand and then articulate as language. So it feels like there is something kind of analogous

• 58:55Nathan Labenz?

  taking shape in the AI world right now, where the the language model feels like kind of a center. And I'm not big on analogies. I honestly am very suspicious of analogies. So tell me, I want to hear, I want to hear why you think this analogy is wrong, or where you think it falls short. But it does, it feels intuitively like there's this kind of analogy between kind of

• 59:18Nathan Labenz?

  myself, or my kind of conscious awareness and my narrative, you know, center. And then obviously, the eyes and the image models are analogous, and you've kind of now created the circuits that connect the two. Do you like that analogy? Do you think that that is missing the mark? How is that falling short?

• 59:42Speaker 01

  I think that's a very good analogy, to be honest. I think the reason why these knowledge model are so powerful now is because they are pre-trained basically on the entire web, right? So the amount of knowledge and information is so much on the web. And in particular, right, these texts, they are very concentrated information. I would say with the current kind of deep learning model and transformers, it's easy to learn from text than from images, because the images are raw pixel, right? It takes, it takes some processing to really understand what's going on. But these texts, they are very, like, one word can encompass

• 1:00:23Speaker 01

  a lot of the information in that. I think that's why the knowledge model have been very successful in learning this kind of word knowledge. And have conversations, right, answer questions and do all this kind of amazing task. So, so, so, so, so I do see that this is kind of a, kind of a brain, but in, in the current status, at least this language model can be considered as the, the, the central piece that holds all the knowledge. And, and then what we are trying to do is that I will bridge this, right, knowledge to some other modalities and make a process other modalities information. Uh, thi- this could be hard because these language models, they have never seen those data, right?

• 1:01:08Speaker 01

  Like, imagine you are human, you grow up learning only knowledge, and now one day you open your eyes, you don't really know how to, how to interpret what, what you see. So, so that's what we are trying to do here, uh, to kind of build this bridge be- between these two modalities. And, and, and I do see there's a potential that by adding additional region or other modalities, this language model itself can also improve. So, we are giving it more informations to learn from. And, for now, we are keeping the language model frozen because mostly our data, this image and text data is not so great, uh, that in the sense that the image is good,

• 1:01:53Speaker 01

  but the text corresponding to this data are limited. And so, so let's say we have better quality text, even like, uh, better paired image and text. Potentially, I do see there is a way we can teach the language model to further improve from this additional data as kind of a new knowledge it can have.

• 1:02:17Nathan Labenz?

  And I can see a pretty direct path to that. I mean, you, you already have, but just combining a couple of ideas from your, your last few papers. Slicing, one thing we're going to try, we haven't done this quite yet because you just sent me the paper the other day, but with Waymark, we're going to at least start to experiment a little bit with slicing images into just pieces, like slice them into four square, four rectangles, or nine rectangles, or whatever, caption each of those, then maybe use a language model to try to synthesize all those captions into kind of one overarching caption. Seen some of that stuff with video as well, where you take a frame every second or whatever, and then caption all of those, and then use a language model on top of that,

• 1:03:01Nathan Labenz?

  and you can kind of synthesize the narrative. And again, this is all just frozen stuff, right? I'm talking like, I think I did that with text DaVinci 2. You know, here's a bunch of captions, tell me what must be going on in this video. And that works off the shelf. So it seems like there's, we're now entering kind of a phase two where it would probably take a significant amount of compute to do this, but I would expect that if you went back through like, the Lion dataset, for example, and revisited either some of those noisy images, or even some of the better ones that just may have short captions, and kind of ran a process like that, you could probably enrich the dataset quite a bit, and end up with a dataset that you could then do, go back to the end to end training

• 1:03:46Nathan Labenz?

  with. So is that kind of the direction that you guys are headed next? Am I picking up where you're going?

• 1:03:54Speaker 01

  Exactly. I think actually this has been done, not exactly like what you described, but it has been done. So like I said, in our previous paper, we already kind of generated synthetic captions on these nine images. We didn't really slice them into different crops. That's definitely something we would have done, but I think due to this efficiency and speed concern, like nine, the asset is so huge, right? We just randomly sample captions for each image. But I think after the paper, the Lion team themselves, they actually have this version called Lion. Uh, Cincinca caption or Coco caption data set, right?

• 1:04:37Speaker 01

  Where they use this split model to generate uh captions. Uh, and they make sure the captions are higher quality. So they do some random sampling and they even do some paraphrasing. Uh, and that data set, in my opinion, is quite good in terms of the quality. Uh, it may lack some diversity because it's generated by just one model, one bleep model. So if you compare to the web data, it may lack some diversity. But that's definitely a much higher quality that the tanks are more aligned with what the the image is about.

• 1:05:11Nathan Labenz?

  Yeah. Do you think that these techniques will work for other modalities? Like when I think of video, for example, or sound, the one thing that jumps out immediately is like there's a time dimension to those that complicate things quite a bit. Um, with video, I can more easily imagine just like down sampling and kind of running things that way. With sound, it feels like it would be a bit of a bigger leap. But ultimately, I also kind of imagine that it would be going to be smart enough to figure that out. Do you anticipate that similar techniques will work across like all these different modalities?

• 1:05:48Speaker 01

  Yeah, I I would say so. Like for videos, like I said, it's quite actually quite straightforward to just adapt it to to video input. You just like down sample certain frames, right? To add a time step uh position encoding to each frame and keep this to the to the encoder. So that they can encode all all the frames in and keep track of of their relative positions in the time dimension. And for sound, I'm not really expert in sound, but uh I would say you can do a similar thing, right? You have this sound encoders that can encode this uh uh wave signals. And then you have this large model that can understand language, right?

• 1:06:30Speaker 01

  You try to bridge those two using some our technique. Uh, I would say it would definitely work.

• 1:06:38Nathan Labenz?

  So ultimately, this feels like kind of one path to something like AGI, right? I think obviously people have very different. Things in their mind when they talk about AGI, but take a sort of ChatGPT or a Claude and maybe like a next generation of those and equip them with all of these connector models where they have kind of the ability now to understand visual data and the ability to understand sound data. And then you can also imagine potentially connector models coming out the other side too, right? Like if you strip off that kind of last layer that turns all the logits into token into a single token prediction and you sort of take that like last probability distribution

• 1:07:24Nathan Labenz?

  and feed that into another connector model. Like it doesn't seem like a huge stretch that you could get action or sort of motion predictions out of that. So is that is that like the macro vision that you guys are working toward? Do you see yourselves as kind of building toward this sort of giant, ultimately like this colossus of a giant ensemble that ultimately can be sort of a super powerful system?

• 1:07:47Speaker 01

  Yeah, our vision is to build this kind of ultimate multi-model kind of system where you can do a lot of things, not only image, you can also maybe do coding and do actions and understand other modalities data. I think that's something we are trying to build. And I would definitely love to have the larger like ChatGPT model to train with, but unfortunately that's not available. So I think I still source research, like what we are trying to push is we want first of all, open source all the models we have. So I think almost a hundred percent of our research will be delivered to the community with open source code and models.

• 1:08:30Speaker 01

  And we are also trying to kind of democratize this AI and the pre-training in particular because they are so

• 1:08:39Speaker 01

  Costly, like in particular those language models, right? They are so costly to train. I think most practitioners won't have the resource to train them. So that's why we're trying to propose these techniques that it's more like a methodology rather than a model that you can, in a more convenient way, an efficient way to make use of these available models. And you can add on top of our method, you can add some other strategies, like you have this efficient adapter module that you may want to try. So, so I think it opens up a door for others to really make use of these large models,

• 1:09:20Speaker 01

  because we can really adapt them, but keep them frozen during our training.

• 1:09:26Nathan Labenz?

  What should we be on the lookout for, from you guys next?

• 1:09:29Speaker 01

  Yeah, we got some exciting news. Dongxu will have a new model coming out soon. It's about text to image generation. So, yeah, I feel that could be quite exciting. And also on the Splint tool, we are working on the next version, which hopefully will be even stronger than the current model.

• 1:09:54Nathan Labenz?

  It is amazing the pace of research right now. Do you guys feel like everything you do is working? Like, are there a lot of failures that you're like not publishing? Or is it basically like all the projects are working?

• 1:10:07Speaker 01

  There are definitely failures. So I think our kind of strategy is we focus on a direction first. So a topic and direction that we feel like it will be impactful to the community. And then we work on that. So we can meet some failures in a way, but because we are confident that this will work in the end, we will continue to push for it. And, and most of the time, eventually, we'll, we'll meet our target. I, I think a lot of these failures in research, yeah.

• 1:10:45Speaker 03

  They're not necessarily just winners, they succeed at the different degrees. So like if you want to get AGI, you don't get it immediately after one grand research project. But it takes a lot of uh uh iterative efforts. So we uh we succeed in a sense that we uh believe we are making steady progress towards that goal. And we also want to take some risks uh in the meanwhile, so that we can uh always explore something that we are curious about and not just uh

• 1:11:27Speaker 03

  uh and also to tell people something we discover uh not just uh to you know uh pushing up the metrics and benchmarking results. We just really want to share the insights and discoveries so that ours can also benefit from uh our findings. And this is um to to do that is also important that we ensure our methods and codes and things are all open source. And I think we are a um we really benefit a lot from the open source community from others. Uh especially this split home model, we leverage we leverage this open source vision and the language models.

• 1:12:12Speaker 03

  And we really want to say that the their fit this community feedback and community efforts are really important to uh pushing the development of AGI and we are also quite uh committed uh to that.

• 1:12:28Nathan Labenz?

  What are your guys' first languages and what do you speak in the like office environment? Is it all English at work or is it a mix of things?

• 1:12:38Speaker 03

  Yeah, we our team come from quite diverse uh uh countries like Dongxu and I, we are from China. We also have team members here like from Singapore, Vietnam, India. So at office, we speak English, that's kind of the official language. But yeah, personally, Dongxu and I, we we we we will speak Chinese.

• 1:12:59Nathan Labenz?

  Being able to speak at a high level on technical topics in another language, I think it's impressive. I guess before too long, we might have some AI babblefish that could kind of sort everything out in real time. But for now, I'm relying on on your English ability, obviously. How has AI affected how you guys work on a day to day basis? Are there any tools that have made an impact on your daily workflow?

• 1:13:24Speaker 01

  Yeah, Donshu maybe can speak to that. He's a heavy Copilot user.

• 1:13:32Speaker 03

  I think there are a lot of things happening behind the scene that we may not be directly aware. I'm not sure that this search result on, for example, Google, a lot of AI is going on behind. So I think I Google a lot of things every day on Stack Overflow. That's the that's the main thing I rely on coding. I think in terms of that, having improved the search efficiency using this AI is powerful. And recently, I also I also gave $1 to Microsoft every month for the Copilot subscription.

• 1:14:14Speaker 03

  And sometimes they gave me a bug, but it was taking some days to figure out, especially for machine learning experiments. But for most of the time, for like, for example, while them is they this boilerplate code, it really helped a lot. And it's a code comments, testing cases, unit tests, it really saved me a lot of time. And that, yeah, I really, sometimes I don't, I don't just, I may not feeling of it. But if I disable that plugin, I feel like my life quality has worsened a lot.

• 1:14:57Speaker 03

  I really have to keep paying for the subscription. And yeah, and when I look at this, because I'm right now, I'm working on this text to image generation.

• 1:15:12Speaker 03

  Project and I keep an eye on this forum, on Reddit and all this in face community. And there are a lot of this very surprising and impressive image generation result every day. And that really blows my mind that I also learned a lot from there. It's not just about entertaining, but sometimes also get to know people. What are the most interesting things to people? Yeah, that's also called amazing.

• 1:15:51Nathan Labenz?

  Cool, thank you. I'm also a big Copilot fan, so I can totally relate to that. And a big Hugging Face fan. So we're definitely sharing tools. That's from 12 time zones away.

• 1:16:03Speaker 03

  Yeah, we we have some in-house code generation models. And hopefully I don't need to pay that subscription.

• 1:16:10Nathan Labenz?

  So. I'm sure you're aware of how Replit used their Ghostwriter model was a distillation, I believe, from the Salesforce code gen method. So if you want to, I don't know if you've used the the Replit Ghostwriter, but it's also very good. And they also now have a chat mode. Hard to say, I would both Copilot and Replit, I think, are advancing pretty quickly. It's amazing that Replit's not that big of a company. And they are, I would say, keeping pace with Copilot, but certainly got a big jump out of the gate by being able to build on the Salesforce code gen. So if you haven't checked that out already, I definitely recommend it as well.

• 1:16:52Nathan Labenz?

  You guys are doing, I think, some really useful work with the Blip set of models. As I said at the top, much more enduring than almost anything else that we see in AI right now. So that's awesome. You have this kind of understanding of how this work fits into a bigger picture of kind of an ensemble strategy for AGI. But zooming out even a little bit further, kind of thinking about society, thinking about the the change that we're all about to see over the next however many years as AI kind of goes from a research agenda to an applied reality in life. What are your biggest hopes?

• 1:17:35Nathan Labenz?

  And also, what are your biggest fears for what AI could mean for all of us?

• 1:17:42Speaker 01

  That's a big question.

• 1:17:46Speaker 01

  I think my biggest hope is, I think it's getting close, right? Uh, like, I'm surprised by the advancement every day. So, I think it is, the progress is even beyond what I can hope for now. It's developing so fast, uh, with all this generative AI. Like, I think this year or for the past two years, it has been really, really growing fast. I, I don't really fear it in the term, because, uh, like, is, uh, literally speaking, because we are the one fundamentally who build these models.

• 1:18:30Speaker 01

  So, I would say, maybe there is, uh, the public has a different opinion on this AGI, but as a researcher, I would still consider them to be quite artificial and, and even still be a lot of times. Uh, uh, uh, and I know it is not, it's far from like sentient or, or perfect because fundamentally it's just a big neural network. It is far from what we expect from humans. So, so I was looking forward to that day to, to come in the future.

• 1:19:06Speaker 03

  Yeah, I think in, in the meantime, now while we are innovating, uh, we also, um, pay attention to kind of related ethic issues on these foundational models. And we have a ethical team who help us to review our use cases and especially on some of the demos we are making. And we, I, I think we have seen a couple of examples of this, um, uh, kind of threatening user large language model interactions, uh, in some of these recent, uh, large language chatbot. And we want to make sure that eventually we are responsible with, uh,

• 1:19:51Speaker 03

  our, our model is responsible for, uh, what it is actually producing. It's understanding what it's doing. Uh not just to abuse it to the cases that harmful meaning innocence to the human. And we do have a says that a lot and uh our ethics teams, um they work very hard to ensure that our delivery and the development is uh safe and also uh production ready when we want to develop uh deploy that.

• 1:20:25Speaker 01

  And then just to add on top of that, we we this external AI is also one of our major focus and we do have a library called OmniX AI that is all about how do you interpret models predictions uh who make really safe and uh like interpretable decisions based on those AI models.

• 1:20:47Nathan Labenz?

  Awesome. Well, Junnan Li, Dongxu Li, thank you guys very much for joining us on the Cognitive Revolution.

• 1:20:55Speaker 01

  Thank you. It's a pleasure. Thanks a lot.

• 1:20:59Speaker 02

  Omnikie uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omnikie so much that I invested in it, and I recommend you use it too. Use Cogrev to get a 10% discount.