Teaching AI to See: A Technical Deep-Dive on Vision Language Models with Will Hardman of Veratai

In this episode of The Cognitive Revolution, Nathan hosts Will Hardman, founder of AI advisory firm Veritai, for a comprehensive technical survey of vision language models (VLMs). We explore the evolution of VLMs from early vision transformers to state-of-the-art architectures like InternVL and Llama3V, examining key innovations and architectural decisions. Join us for an in-depth discussion covering multimodality in AI systems, evaluation frameworks, and practical applications with one of the field's leading experts. Here's to the link to one of the most comprehensive reference documents for VLMs prepared by Will Hardman: https://dust-mailbox-c73.notion.site/Vision-Language-Models-11b675d75dd480af994cc474a754bb26 Help shape our show by taking our quick listener survey at https://bit.ly/TurpentinePulse SPONSORS: Oracle Cloud Infrastructure (OCI): Oracle's next-generation cloud platform delivers blazing-fast AI and ML performance with 50% less for compute and 80% less for outbound networking compared to other cloud providers. OCI powers industry leaders like Vodafone and Thomson Reuters with secure infrastructure and application development capabilities. New U. S. customers can get their cloud bill cut in half by switching to OCI before March 31, 2024 at https://oracle.com/cognitive 80,000 Hours: 80,000 Hours is dedicated to helping you find a fulfilling career that makes a difference. With nearly a decade of research, they offer in-depth material on AI risks, AI policy, and AI safety research. Explore their articles, career reviews, and a podcast featuring experts like Anthropic CEO Dario Amodei. Everything is free, including their Career Guide. Visit https://80000hours.org/cognitiverevolution to start making a meaningful impact today. CHAPTERS: (00:00:00) Teaser (00:00:55) About the Episode (00:05:45) Introduction (00:09:16) VLM Use Cases (00:13:47) Vision Transformers (Part 1) (00:17:48) Sponsors: Oracle Cloud Infrastructure (OCI) (00:19:00) Vision Transformers (Part 2) (00:24:58) OpenAI's CLIP Model (00:33:44) DeepMind's Flamingo (Part 1) (00:33:44) Sponsors: 80,000 Hours (00:35:17) DeepMind's Flamingo (Part 2) (00:48:29) Instruction Tuning with LAVA (01:09:25) MMMU Benchmark (01:14:42) Pre-training with QNVL (01:32:13) InternVL Model Series (01:52:33) Cross-Attention vs. Self-Attention (02:14:33) Hybrid Architectures (02:31:08) Early vs. Late Fusion (02:34:50) VQA and DocVQA Benchmarks (02:40:08) The Blink Benchmark (03:05:37) Generative Pre-training (03:15:26) Multimodal Generation (03:37:00) Frontier Labs & Benchmarks (03:47:45) Conclusion (03:53:28) Outro SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://www.linkedin.com/in/nathanlabenz/ Youtube: https://www.youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Is multimodal understanding in AI important on the path towards AGI? It's not entirely clear that it is, but some people argue that it is. So one reason that one might want to research these things is to see if by integrating information from different modalities, you obtain another kind of transformational leap in the ability of a system to understand the world and to reason about it. I would say, in inverted commas, similarly to the way we do. For open source researchers, like the last few months have really seen the arrival of these huge interleave data sets, which has kind of really jumped the pre-training data set size that's available.

• 0:42Speaker 00

  I'm kind of amazed that the percever resampler works because it feels to me just like tipping the image into a blender, pressing on, and then somehow when it's finished training, the important features are retained and still there for you.

• 0:55Nathan Labenz?

  Hello, Happy New Year and welcome back to The Cognitive Revolution. Today, I'm excited to share an in-depth technical survey covering just about everything you need to know about vision language models, and by extension, how multimodality in AI systems currently tends to work in general. My guest, Will Hardman, is founder of AI advisory firm Veritae, and he's produced an exceptionally detailed overview of how VLMs have evolved, from early vision transformers to CLIP's pioneering alignment work to today's state-of-the-art architectures like Intern VL and Llama 3 V. Will examine key architectural decisions like the choice and trade-offs between cross-attention and self-attention approaches, techniques for handling high-resolution images and documents,

• 1:40Nathan Labenz?

  and how evaluation frameworks like MMMU and Blink are revealing both the remarkable progress and the remaining limitations in these systems. Along the way, we dig deep into the technical innovations that have driven progress, from Flamingo's percever resampler, which reduces the number of visual tokens, to a fixed dimensionality for efficient cross-attention, to Intern VL's dynamic high-resolution strategy that segments images into 448x448 tiles while still maintaining global context. We also explore how different teams have approached instruction tuning, from Lava's synthetic data generation to the multi-stage pre-training approach pioneered by the Chinese research team behind Quen VL.

• 2:22Nathan Labenz?

  Our hope is that this episode gives anyone who isn't already deep in the VLM literature a much better understanding of both how these models work and also how to apply them effectively in the context of application development. Will spent an estimated 40 hours preparing for this episode, and his detailed outline, which is available in the show notes, is probably the most comprehensive reference we've ever shared on this feed. While I have not worked personally with Will outside of the creation of this podcast, the technical depth and attention to detail that he demonstrated in what for him is an extracurricular project was truly outstanding. So if you're looking for AI advisory services and you want someone who truly understands the technology in depth on its own terms, I would definitely encourage you to check

• 3:05Nathan Labenz?

  out Will and the team at Veritae. Looking ahead, I would love to do more of these in-depth technical surveys, but I really need partners to make them great. There are so many crucial areas that deserve this kind of treatment, and I just don't have time to go as far in-depth as I'd need to to do them on my own. A few topic areas that are of particular interest to me right now include, first, recent advances in distributed training. These could democratize access to frontier model development, but also pose fundamental challenges to compute-based governance schemes. Next, what should we make of the recent progress from the Chinese AI ecosystem? Are they catching up by training on Western model outputs, or are they developing truly novel capabilities of their own?

• 3:47Nathan Labenz?

  There's not a strong consensus here, but there's arguably no question more important for US policymakers as we enter 2025. I'm also really interested in biological inspirations for neural network architectures or any comparative analysis of human and artificial neural network characteristics. The episode that we did with AE Studio stands out as one of my favorites of 2024, and I would love to have a more comprehensive understanding of what we collectively know about this space.

• 4:15Nathan Labenz?

  I'm similarly interested in the state of the art when it comes to using language models as judge or otherwise evaluating model performance on tasks where there's no single right answer. This is a problem that we face daily at Waymark and which seems likely to have important implications for how well reinforcement learning approaches will work and scale in hard to evaluate domains. Finally, for now, I would love to catch up on the latest advances in vector databases and RAG architectures. I've honestly been somewhat disillusioned with embedding based RAG strategies recently, and I've been recommending flash everything as the default relevance filtering strategy for a while now. But I do wonder, what might I be missing? In any case, the success of our previous survey style episodes, including our AI Revolution

• 4:59Nathan Labenz?

  in Biology episodes with Amelie Schreiber and our Data Data Everywhere Enough for AGI episode with Nick Gannon, suggest that people find these detailed overviews to be a helpful way to catch up on important AI subfields. So if you have or are keen to develop deep expertise in an area that you think our audience would benefit from understanding better, please do reach out. I'm open minded about possible topics and very interested to hear what you might propose. You can contact us as always via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. And of course, I'm more than happy to give you the chance to plug your product or services as part of your appearance on the show. Now, I hope you enjoy my conversation with Will Hardman, AI advisor at Veritae, about

• 5:42Nathan Labenz?

  all aspects of vision language models. Will Hardman, AI advisor at Veritae and AI scout on all things vision language models. Welcome to the Cognitive Revolution.

• 5:54Speaker 00

  Thanks, Nathan. Great to be here.

• 5:57Nathan Labenz?

  Yeah, I'm excited about this. We've talked about this for a few months now, and you have put a real Herculean labor into A very deep dive into all of the techniques, data sets, different strategies, variations on making vision language models work. I think this is going to be a really interesting crash course and overview on all that. And I think it's something that I know that I want and need, and I think a lot of people will really benefit from getting the sort of fast forwarded version of all the research you've done. So thanks for putting all the legwork in upfront to make this happen. And basically, what I want to do today is just kind of give you the floor,

• 6:39Nathan Labenz?

  let you take us through everything that you have found to be important in vision language models. And I'll certainly have my questions along the way as we go. But I'm excited for this.

• 6:51Speaker 00

  Cool. Yeah. So this would have been a lot easier to compile, just to feel that stayed still for five minutes, and the leaderboards hadn't jiggled around every day, and new papers hadn't come out every week, making me think, hmm, we should probably include this. But we're kind of at a checkpoint in time. So it's worth saying we're recording on December 20th, 2024. We're still not the end of the opening AI 12 days of Christmas. So something may change tomorrow. Paper may be released tomorrow. So this is a kind of a point in time view of vision language models. And I guess it's a deep dive, if you're coming from the perspective of someone who's interested in AI, but not super familiar with vision language models. But if we're talking

• 7:34Speaker 00

  about vision language models specifically, it's definitely not a deep, deep dive into the research, because it's huge, and there's so much going on, and some of it's so complicated, and so much we could cover. So what else we could do is just stick to like a few things. So firstly, let's look at some of the most important architectures and some of the trends in research, and we'll illustrate these through some of the most notable models from the last couple of years. And these will be models that anyone who's working or building this space is quite likely to encounter. And then we'll talk a little bit, I mean, touch briefly on the key data sets and some of the benchmarks. And one benchmark in particular, we'll explore in a bit more depth because it's really interesting. And then I guess we'll talk a bit about recent attempts at what we

• 8:19Speaker 00

  call true multimodality. So a vision language model is really reading kind of audio, sorry, images and text inputs and then reasoning about them. But true multimodality would be generating images as well. So we'll kind of come on to that at the end. And then kind of we'll finish up, I think, just by taking a kind of, as of today, snapshot, what's best in class across some of the key benchmarks and where do I go and get it? And what can I do with it? Does that sound good?

• 8:46Nathan Labenz?

  That's good. Yeah, I'm already taking away that you're not classifying me as a truly multimodal entity in as much as I can't produce image outputs. So the, talk about the bar rising quickly. I think I'm already outclassed by what you're calling the true multimodal models.

• 9:04Speaker 00

  You mean, you can't draw?

• 9:06Nathan Labenz?

  Not very well. Not, not well enough that I would, I would not see an API anytime soon. That's for sure.

• 9:12Speaker 00

  Yeah. So in that case, I'm like you. I mean, I can just about doodle. That's about it. Because just to start off, like all good research reviews with a motivation section, like, why do we care? Because obviously, there's lots of interesting use cases for very lens. It was really interesting recently when you had the team from Google, who were talking about the new Gemini APIs. One of the things they said was loads of people are building large language models, relatively few are building right now with visual language models. And that's, they think, going to be a growth area next year. So that's, that's kind of cool. There's, there's loads of kind of use cases. The obvious ones like medical assistance, you know, being able to look at image modalities as well as patient history and then say things about patient that might be useful to the clinician.

• 9:57Speaker 00

  But I mean, other use cases, you know, content filtering, for example, and knowing what is in an image and text. For example, if you were looking at a social media platform and you're trying to screen out images or content of concern, indexing large quantities of archival material or like product catalogs or something like that. We've got in both visual components and text components. You want to better understand, you know, what is this product, given I can see it and given I've got some information about it. But I've also seen applications, for example, in insurance where people are having photos of cars. For example, there's a description of what's supposed to have happened to the car, it might be in some damage. And the question is, can I actually see the damage in the image? Does it kind of reflect what the person is saying on the report? Yes or no?

• 10:40Speaker 00

  There's various use cases, but I think beyond that, there's kind of two other reasons we might be interested in vision language models. One of them is that in building VLMs, what you're learning to do is integrate two modalities. They start off very separate and somehow you're going to reason over both of them. And then if you can find the recipes for doing this right, then into the future you can think about, okay, can I integrate audio, touch data, LiDAR data and other modalities? And if you think about like, for example, robotics of the future, just think about the number of kind of different sensory modalities that you'd need to have a robot cook a meal, you know, it's got to handle everything, see everything.

• 11:21Speaker 00

  You can think about VLMs as being the first step to learning how to do this, so we can integrate a lot more for the future. So that's kind of a longer term thing. And the second one is a bit more of a philosophical question is, is multimodal understanding in an AI important on the path towards AGI? It's not entirely clear that it is, but some people argue that it is. So one reason that one might want to research these things is to see if by integrating information from different modalities, you obtain another kind of transformational leap in the ability of a system to understand the world and to reason about it. I would say, in inverted commas, similarly to the way we do. We know that it loses things the same way we do.

• 12:07Speaker 00

  There you go. And I guess there's arguments against, you know, is it important would be to say, well, look, frontier language models show lots of evidence of high level abstraction, world models, sophisticated reasoning. There's no obvious ceiling in performance as of today. Maybe a grounded multimodal understanding of the world is not that important for achieving AGI. But we'll kind of explore this. Maybe there are some little bits of evidence we'll come across today, which might point us towards one or the other view here.

• 12:40Nathan Labenz?

  Yeah, I mean, I would be very surprised if we end up, I mean, it just seems like imagine yourself unable to see, right? It's like, it would certainly be a major hurdle to have to get over. And my guess is, and maybe we'll, you know, shine some light, so to speak, on this question as we go. But my guess is we'll never really answer that philosophical question of like, could we have built an AGI that isn't multimodal? Because, you know, to state the most obvious spoiler in the history of the world, like there has been a lot of progress. And it seems like if nothing else, it will be the path of least resistance. Like multimodal is clearly going to work.

• 13:21Nathan Labenz?

  The details are to be unpacked, but it seems like maybe, you know, the sort of philosophical crowd will continue to say, well, we might have been able to do it without multimodality, or it would have been impossible. But, you know, it seems like in the end, this is going to be the norm and these things are going to walk among us. Probably if I had to guess, you know, sooner rather than later.

• 13:43Speaker 00

  There's always sooner rather than later, isn't there, in this world? So I suppose before we dive into the first vision language model that we'll cover, there's probably two important little prefaces we ought to do. One, we ought to talk about vision transformers for a moment. And then we ought to talk about the clip models from OpenAI. And the reason is that both of these are going to crop up again and again. So let's just kind of refresh our memories as to what they are. And then we'll kind of dive into the VR lens themselves. So the vision transformer itself, so I'm going to assume we're all familiar with a language model transformer, especially the decoder architecture. The canonical paper here is called "An Image Is Worth 16 by 16 Words",

• 14:25Speaker 00

  which is from Google, about four years ago now, 2020, I think. And previous to this, like most vision models have been based on convolutional neural networks. So they've basically been stacking convolutional filters to extract more and more kind of global features from the images. And the question that the Google team asked was, well, could we use the transformer recipe to build something that understands images? And so the recipe is, let's see, quite straightforward. You take an image and you divide it into non-overlapping patches like this, okay? You then linearize the patches and you have a linear embedding, which basically converts them all into tokens, okay?

• 15:08Speaker 00

  So now we have just a sequence of visual tokens through a learned embedding. And then we feed these patches, one by one, into a transformer encoder and we use full attention across it. So every little image patch can pay attention to every other image patch in the image, okay? And this is very similar kind of in thinking to how a model like BERT is trained, because what we do is they stick a classification token, they prepended to the sequence. And the training objective is, can you classify the image that you see into one of a large number of categories? You take the classification vector out the end and that's what you use to figure out if you've got the right classification. So very, very simple recipe going on there.

• 15:53Speaker 00

  And the key finding was that if you make these things big enough, these vision transformers, the transformer architecture does beat the convolutional neural networks of the day. And so that makes it a very kind of useful building block.

• 16:07Speaker 00

  There's just a couple of things we ought to take away from the design of the vision transformer, which is also called a VIT. So I'll probably use the word VIT throughout. The first is that the image resolution is going to be fixed by design. So in the original vision transformer, it was 224 pixels square. Okay, so everything has to be that size when we feed it in, then we get fixed number of patches coming out. For the original training, they would stick, like I said, this classification token in. When we come to talk about the vision language models later, normal practice is to take the entire sequence of hidden states from the transformer out and use that as your encoded image. Okay, so we don't just take the classification vector, we take everything.

• 16:50Speaker 00

  And that means you could get quite a lot of vision tokens out of such a model. So if you started with 224 times 224 is your image size, and your patches were 16 times 16, that's the back of the envelope math, so you're going to get 196 visual tokens out at the end, which can be quite a lot. Okay, so that's the other thing to note.

• 17:12Speaker 00

  And I guess the third thing is, and this is just a convenience, when we talk about vision transformers, you'll hear them describe like VIT H16, for example. And this just tells you something about the kind of dimensionality of the vision transformer. So the VIT is a vision transformer. The H in that stands for huge, and we just have to know that huge stands for about 600 million parameters. And the 16 tells us the patch size. So that's what we're patching our images up into. So if I use that later on, and here you'll know what I mean, if I say VIT G16, for example. It's a giant one, which is even bigger than huge. There we go. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 17:54Nathan Labenz?

  So a couple of little follow-ups there, just to make sure I understand this correctly. One, just to contrast the image attention pattern versus the language model attention pattern.

• 18:10Nathan Labenz?

  At least the one that we're most familiar with, which is like a look back only pattern, right? In language, the attention in the image context is generally all to all, right? So there's no, there's not like a sense of ordering or the sort of, you know, obviously language unfolds token by token. So that's one kind of fundamental difference. It's just that this is more of a, which obviously reflects the modality itself, right? The image is a snap of a scene in time and that is all on par with each other in the way that it's being processed all to all. The other thing that I wanted to dig in on just

• 18:49Nathan Labenz?

  a little bit more is how the tokenization happens. In language, we have these tokenizers, which sort of tried to figure out like, what's the sort of optimal way to, and it's interesting that those are typically not part of the end to end training, right? You sort of have this, this separate sort of bolted on system that, you know, people have tried, kind of hated and have tried to get rid of for a long time. And maybe there's some signs that that could be about to happen. It's interesting. I was just reading last night a paper from Meta that's about a more dynamic way to batch text as opposed to with these fixed tokens that are predefined. But, you know, if anybody hasn't seen this, you can go to the OpenAI

• 19:32Nathan Labenz?

  Tokenizer and just paste in your text and it will immediately chunk it into bits of text and color code them. And you can see what all the tokens are. So that's a vocabulary of, I think now up to like 100,000 different little bits of text that text is broken down into before it is translated into numeric form and then processed. And that translation of this token to this vector representation is something that at runtime is fixed. And there's basically as many, people often refer to this as one hot encoding, where there's as many possible input vectors as there are tokens. Yeah.

• 20:16Nathan Labenz?

  How, that's a bit different now in this case, right? For images, my understanding, if I'm understanding correctly, is there's not like a fixed vocabulary size of

• 20:29Speaker 00

  That's

• 20:29Nathan Labenz?

  Of possible tokens, right?

• 20:30Speaker 00

  That's correct. So we're going to use the term throughout Mercer's, we're going to use the term tokens quite loosely. Because as he correctly says, you know, text tokens can be mapped back to text through a code book. You've literally got, you know, 80,000 codes, you look it up and you get your byte pair or whatever it is at the end. The same is not the case with visual tokens. They exist on a continuum, as you pointed out. So to go from your little patch, which is really just a, you know, a matrix, you've got a few dimensions and channels in there. You're simply going to pass that through a matrix, which is going to generate the vector that you want to stick in to the transformer. And it's learnable, that transformation is learnable, but the important thing is that your tokens are going to come out on a continuum.

• 21:16Speaker 00

  And they don't need to be quantized at this point. There's nothing in the transformer architecture that says tokens have to be quantized to a code book. You can still run the attention mechanism, even if your tokens exist on a continuum like this.

• 21:30Nathan Labenz?

  Okay.

• 21:31Speaker 00

  Note that because we're training, like you said, the visual, the vision transformer with a classification objective. We don't actually have to decode anything at the end. So it doesn't matter.

• 21:43Nathan Labenz?

  I'll then, I'll save my next question I have for as we get a little deeper into the journey here. I think the last thing that's worth just reflecting on for a second is just how small the images are that are being processed. I've done a little bit of this, not recently, because these days we have these foundation models where I can just throw basically anything into it. I think

• 22:03Speaker 00

  Yep.

• 22:03Nathan Labenz?

  There are some hard limits that I've run into occasionally of my images like north of 10 megabytes or whatever. But typically just throw it in there, they handle it. You as a developer don't really have to worry about it. With earlier generations of models, you did have to do this sort of pre-processing where you would take your image. It was your responsibility as a user of the model that somebody open sourced for your convenience to take your image and

• 22:30Nathan Labenz?

  Basically morph it, or I guess resize it is probably the right term into the required size that the model could handle. And it's presumably just because everything was smaller back then and compute resources were more limited and the results weren't so spectacular in general. Two twenty four by two twenty four, even until fairly recently, and even with like the OpenAI kind of small mode, whatever their sort of low res mode is, it is remarkable how much performance can come out of these like very small images, even when they're dramatically shrunk and they're often like significantly distorted because your original image might not have even been a square, but you're just kind of like, whatever, I'm

• 23:14Nathan Labenz?

  just gonna make it a square. I don't care if things are smooshed. I don't care whatever happens. That's what we're going to work with. And it's amazing to me how well that actually works. And I think these days that's getting liberalized for sure because it's not all low res on like the OpenAI API, but it is remarkable how far that can go.

• 23:33Speaker 00

  Yeah. Without wanting to spoil the big reveal, of course, they're not compressing everything to two two four by two two four and using that as the image inputs. There's some much more sophisticated and smarter things going on, at least with the leading visual language models. So we'll see how they do it in a bit.

• 23:51Nathan Labenz?

  Okay, cool. Well, let's carry on. That's a great start.

• 23:54Speaker 00

  Cool. So that was the vision transformer. And I think the other thing is that we should introduce right now is the clip model from OpenAI, because that's again going to be really fundamental. And the paper here is from back in 2021, and they call it learning transferable visual models for natural language supervision. And CLIP itself stands for contrastive language image pre-training. So it's a canonical model in the field, and it's a really nice introductory one to study for how we align image and text encodings. Okay. So the idea is you start with a vision encoder, which could be a vision transformer, and a text encoder, which could be an encoder only transformer. And you've got a large

• 24:37Speaker 00

  data set of images with their captions that have been scraped from the web. Okay. So the process is to jointly train both encoders, so they're generating embedding vectors, both for the text and for the images, such that if you take an image and its caption, the two vectors are going to have very high cosine similarity for an image and its caption. But if I take two random captions in the image, they will have low similarity. And the way this is done is you simply pass the image through the vision transformer, the text through the text transformer. You then add just a linear projection to get them kind of the same conforming dimensionality.

• 25:21Speaker 00

  And then then use this contrastive loss function. And then what that does is it says, suppose I've got a batch with like N image caption pairs in it. I know within that that I've got N true pairs and I've got N squared minus N bad pairs, okay? So you set the loss function up to say penalize any dissimilarity between my true pairs and likewise penalize any similarity between the non-pairs within the batch. And that's a contrastive loss function. It's basically bringing things that are supposed to be the same close together and pushing the vector representations apart for things which are not the same. So that was how the clip model worked.

• 26:05Speaker 00

  And one nice thing about this is that once you've trained it, I mean, obviously you can use clip itself for things like image search. But you can also just take the trained vision transformer out of it and use it downstream. And that's really nice because what you've done is you've already in some senses kind of trained it to embed things similarly to a language model. So therefore, if we were going to put that into a vision language model, it should only be a small step away from being aligned to whatever we're going to use in that model. So there's the thinking there. And that is why, as we kind of go through to the rest of today, we're going to see that they very often, the research will start with a clip.

• 26:46Speaker 00

  It gets them kind of two thirds of the way there. So I wasn't going to do a deep dive into the clip model, but just to say that, you know, it is a vision transformer. That's how it's trained using this contrastive loss objective. And we can now use it downstream.

• 27:02Nathan Labenz?

  Yeah, I remember using this one in the early days of Waymark small business video creation. We were at the point where we could get the fine tuned GBT three to write a somewhat decent script, you know, certainly things have improved a lot since then. But then you had the challenge of, okay, now that we have this narrative that we want to put together for this random small business, and we've also found all these images off the web. What should we choose out of this bag of images, right? And at the time, you know, you had sort of prior to clip, you had like classic, you know, most of this stuff was done on

• 27:45Nathan Labenz?

  finite sort of pre-established data sets, right? I mean, and one of the big advances of clip was that they moved from your like ImageNet type thing where you have a certain canonical set of images with a certain fixed set of classifications. And the game used to be about how, you know, comparing architectures, basically, right? Can I come up with an architecture that on this standard data set does a better job than everybody who came before me in some way, shape or form there? You know, where I can claim to have state of the art and I'm great. That didn't do much at all for us with an application like Waymark because it was, you know, maybe could have made it work, but it would have been very noisy because it would have been like, well, the image that I have of this, you know, business, whatever it may be,

• 28:31Nathan Labenz?

  potentially is not even well represented at all by any of the classes in an ImageNet set of classes. I think it's a thousand classes that they have in ImageNet. And there were smaller ones before that that had fewer classes. So.

• 28:45Nathan Labenz?

  This was the first moment, as far as I know, where they went, let's forget competing on these standard data sets. What people really want is to understand anything that they might be looking at. And the web scale data is out there and enough of these images are captioned. And by the way, when you really got into clip, tons of noise in the caption data, you know, typically if you just ask yourself like, how do people caption images? You know, it's lots of different ways, you know, sometimes with jokes, sometimes with just a straightforward description of what's in it, sometimes with a line from a poem, you know, I mean, it's a tremendous amount of noise in that original data set. We found in using it

• 29:29Nathan Labenz?

  that it was pretty good. If you said, okay, I want an image of this, which of the images from this set of possible images most closely matches that query, we could get a pretty effective sort. But we'd see all sorts of artifacts and weirdnesses. Like sometimes if the text that we were, you know, let's say we were, for example, doing a video for a pizza restaurant, there's a pizza restaurant in Detroit that I've always kind of used as one of my go tos. If you put the word pizza in as your text query and run that through the text side and then put all the images through the vision side and, you know, each one now is represented by the vector. And as you

• 30:11Nathan Labenz?

  said, like if they're similar, you should have a high cosine similarity. So basically literally multiply these vectors together, take the sum and sort by that. That was the procedure. But sometimes what you would find is if the word pizza in text was in the image, then that would, you know, pop to the top. And so you'd have all these sort of weird things to deal with. I think largely as a reflection of the fact that this was just extremely noisy web scale data that at the time, they didn't really even necessarily have a great way to clean, right? Because it's like, all this technology is kind of being created out of nothing, you know, these days, you would say, well, why do you filter those, you know, images that have captioned image content mismatch or that seem nonsensical?

• 30:58Nathan Labenz?

  And I think the answer at the time was basically, we have no way to do that. So we just have to try to throw as much data into this as we can and hope that some signal comes out, and it did. We did find also that, I promise I won't do this every single step, but I have memories of clip.

• 31:15Nathan Labenz?

  Aesthetics were basically totally unrepresented. And I guess this would be because a few of the image captions found online are like a beautiful picture of X or, you know, an unattractive picture of X. But we wanted that because small businesses have very wide ranging quality of pictures. Sometimes you get user generated content stuff, most of their Facebook, sometimes it's professional. The difference matters a ton, right? They do not want some of these ugly images that might be associated with them online in their marketing. But how could you tell? There was no real aesthetic signal in clip. It was all content, but not quality. So it's crazy. That wasn't all that long ago.

• 32:01Speaker 00

  No, we're still three years, four years ago. And now we're going to jump to two years ago. And then everything else we talk about is going to be in the last two years. And you're right, that one of the stories that's going to unfold is we kind of cover a few of the BLMs today, is this increasing obsession with filtering data for quality, for both the pre-training and for subsequent by training stages. And that is to kind of get rid of this problem of noisy data, which really does seem to hurt BLMs in particular.

• 32:31Nathan Labenz?

  Cool. Well, I think that's enough memory lane down the clip alley for me. So let's keep going.

• 32:38Speaker 00

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 32:43Speaker 00

  Okay, so we're going to jump forwards then to 2022. And a model that I've heard described as the GPT-3 moment for vision language models, and that is DeepMind's Flamingo model. A number of really interesting innovations in this. And so it's worth covering this in a bit more depth. And also, it's the first example we'll see of actually how the LMS constructed. Okay. So the basic pattern, and I'm going to see this through all of the models we cover, is that you're going to encode the two modalities, that is text and images separately. So you use a text tokenizer and again you use an image encoder, normally a vision transformer. We're then going to select a language model, which is called the backbone.

• 33:28Speaker 00

  And the backbone is going to be the thing that does all the reasoning of both the text and the images. And that just leaves us with a question of how we connect the two. So for example, in the flamingo model from DeepMind, they chose, they actually looked at a vision transformer and a convolutional neural network. Everything I've looked at since has been just using a vision transformer. And then they used a Chinchilla language model. Okay, so to connect the two things together, it decided to freeze a language model and then introduce cross-attention layers sandwiched between, I think, every fourth transformer block in the Chinchilla language model. Okay, so the idea of the cross-attention layers is that they're going to look up information that we got from the vision transformer.

• 34:14Speaker 00

  And there's, immediately upon thinking about it, you might see a couple of challenges that need to be resolved for this to work. So the first is, if we've got an input, say, that's got more than one image in it, then obviously when we've encoded those, we're going to have like a variable number of image tokens generated. And if we're using a cross-attention mechanism, so similar to self-attention, other than the keys and the values are not coming from the text with the coding, they've got to come from the image, then that's a problem because obviously the dimensionality of the cross-encoder mechanism is fixed. So we need a fixed number of visual tokens generated. So what if we've got two, three, four images, or we've got different sizes of images?

• 34:59Speaker 00

  And the second question is, we can sometimes get a lot of visual tokens. So if we want to train really efficiently, is there a way to like reduce the number of visual tokens that we're actually going to attend to? That would just make training the model a lot easier. Okay, cross-attention layers don't need to be quite too deep. And so the way that the DeepMind team solved this is, I think, very smart. And they used something called the perceptron sampler, which might actually have been their innovation, I think. So percever resampler is like a separate model that's going to be introduced. So not all cross attention layers, we're introducing a separate fresh model here. It's going to look at the visual tokens. And it's designed to select out or sample, resample, the most important visual information

• 35:43Speaker 00

  that the vision transformer has encoded. But you want to do this in such a way that if you have a very, very, very long sequence of vision tokens that have come from one or more images, what you don't want to do within your percever resampler, which has its own attention mechanism, what you don't want to do is compute the kind of all to all sequence times sequence attention matrix, because that can be very big. Okay. So the innovation here is that in a normal attention mechanism, you've got your queries, keys and values, right? And they all come from the context that you're decoding. All right. So the idea here is that the explosion comes from the fact that the queries need to be

• 36:26Speaker 00

  multiplied by the keys at some point, and that creates this all to all attention matrix, which is size of the sequence length squared. And it is, of course, the bane of transformers. So what they did in the percever model is they said, well, what if we jettison this need to come up with queries based on the context that we've just read? What if we instead have like a small number of fixed queries? And I say fix, I mean learnable. So they're just latent vectors that can be learned at training time. And they selected for Fermentio, they selected this to be 64, 64 query vectors, which can be learned at training time. So in the percever resampler model, you're going to look at all of the visual tokens that have come in, your query is going to be a size 64.

• 37:10Speaker 00

  So the actual query key matrix that you calculate is now sequence length times 64, which is way, way, way smaller. Okay. The beauty of that is, once you've finished the attention calculation in the perceptron sampler, you've got something that's 64 times whatever your hidden dimension is. So it could be 7, 6, 8, for example. So again, you get something very small out of it. And the perceptron sampler model is a module that's essentially this. And it does it in a number of stages. But it's essentially just using these learnable keys. And the nice thing is about the end, we know the size that the visual tokens are going to be at the end. 64, 10, 7, 6, 8, for example. Which means that we can now define a cross attention layer, which is always going to

• 37:55Speaker 00

  read visual tokens of that size that have come out of the vision transformer. Is that making sense?

• 38:02Nathan Labenz?

  Yeah, and this happens after the initial vision transformer layers, right? So we've got still the all to all attention happening on the vision side of the, it's kind of too, I can remember the chinchilla diagram and even more so, I can remember the bowl of yarn soup that I went around showing to people at parties in spring of 2022 and into summer, when I was like, look at this, this exists now. So you've got basically, again, very similar to clip, and this will be a pretty common theme, although there are some exceptions too, but images being processed through one

• 38:45Nathan Labenz?

  kind of main model, text being processed through another main model, these things are often totally frozen at the time that we now want to figure out a way to fuse them. And the way to fuse them is two parts here. One is a cross attention, but then the second is figuring out how to essentially make life easier for the cross attention and the kind of main language model that's going to carry things on by finding a way to standardize the sequence length for the image. So toward the end of the image processing with its full attention, then there's this sort of adapter that says, okay, regardless of how big the image was or how many tokens,

• 39:30Nathan Labenz?

  whatever, it's always going to be output from this step at 64 tokens. And therefore, the cross attention can always be the same size and everything downstream of that can get simpler. I think this is a really... Interesting thing. I mean, I probably didn't have the sophistication at the time to appreciate it, but it's a good indicator of just how

• 39:56Nathan Labenz?

  malleable all of these latent spaces are and how I sometimes say like, everything is isomorphic to everything else, which I think probably doesn't actually mean anything, but it's, you know, the intuition that I'm trying to express there is just like, I've seen so many of these things at this point where one space is bridged over to another space or, you know, reformed or, you know, you almost see this even just with the libraries that are used. I'm thinking of all the Python libraries that allow you to like reshape matrices.

• 40:27Speaker 00

  Yeah.

• 40:27Nathan Labenz?

  It's almost like those matrix reshapings are, you know, you don't think of those as like semantic, but when you scale them up to this sort of thing, a very similar thing starts to happen where you're like, okay, I've got all these, you know, all the all, but what I really need is just a sequence and it needs to be fixed length. And, okay, I'm just going to train a thing that no matter what, it's going to output a fixed length sequence and hope it works. And, I mean, not everything works. I'm told by the researchers that there are many failures along the way, but it sure seems like almost everything of this sort kind of works. And I think that's a really striking reality. I remember seeing this one and there were,

• 41:12Nathan Labenz?

  there was Blin2, which was another one that really brought it home for me just because of how few new parameters had to be trained. I don't know if you have those numbers like right off the top here, but it's a strong contrast between the amount of parameters and pre-training on both the text and image side versus how small these connectors can be in many cases.

• 41:36Speaker 00

  Yeah. So one of the teams from Hugging Face did a comparison and we'll talk about it in a bit more detail later. They said in a seven billion parameter language model class, if you add the cross-attention layers in, that's about 25% of the parameters now need training. Whereas they said, if you're going to do it the other way, which is just a simple projection, that's about 10%. So you do introduce a lot more parameters using this kind of cross-attention mechanism that we're talking about. But it's still, it's small compared to retraining the whole language model. And that's actually one of its benefits is that you can literally freeze the vision transform if you want, you can freeze the language model and just train the cross attention parameters and the perceivory sampler. And like you, I'm kind of amazed that the

• 42:21Speaker 00

  perceivory sampler works because it feels to me just like tipping the image into a blender, pressing on, and then somehow when it's finished training, the important features are retained and still there for you. It feels to me like they would have all been mixed up, but guess it goes to show intuition, at least for me, doesn't really work when it comes to these like very sophisticated mechanisms.

• 42:45Speaker 00

  Yeah, okay. Cool. So let's just say a little bit more then about, so we've got the perceivory sampler, we've got the language model, and then to train them, you simply need to switch on your next token prediction, training objective, and start training the newly lationalized layers in the language model. And that's not too difficult to do. And what's happening is you're replacing where your images would have been in the text prompts that you're giving to the language model during its training, that's prompting it to look up the outputs from the perceivory sampler. So these are not tokens that get scored, for example, when you're decoding, but it's prompting it saying, when I see this token, it's okay, I need to go and look up something in the perceivory sampler,

• 43:29Speaker 00

  and that's going to give me enough context and know what the next token should be, because it's where the image would have been, my input text. And in terms of the training data sets, like to see how this actually works in practice, they obviously used images paired with their old texts, which is the same that's used in CLIP. But that's not enough. And one of the contributions that the flamingo team made was to realize the real importance of what's called interleaved data. So interleaved data is basically data scraped from the web, like HTML documents, where you've got images, I think, got text in the document. And they know by looking at the document object model when they scraped the website, they know kind of roughly what order these things are supposed to appear in.

• 44:12Speaker 00

  That interlead data sets that you can then pass through your Decoza and then you can say, you know, when you get to an image, look up the image, you know, now I'm going to keep on producing the text. They found that it was really, really important for actually maintaining all of the kind of downstream performance metrics. So interlead data turns out to be super, super important for vision language models. Now was one of the kind of major findings of the Flamingo paper. So in terms of how they evaluated this thing once they, once they'd done the training, it was visual question answering and OCR and captioning. Those are kind of the key tasks they looked at. And they discovered firstly, it works quite well in a few shot setting. So the, the

• 44:55Speaker 00

  train was very compute efficient because you don't need to touch anything in the language model and the newly initialized parameters. And yet the output was still very competitive with very much more focused task specific models. So they had a model that could do several things and it could be competitive with all the task specific ones. And that's why it was a really kind of, I think foundational model in visual, vision language models is because you have this kind of several tasks you could perform and you have this one training recipe strongly based on the transformer training recipe. And the basic pattern of using cross attention layers has been used by other teams since, but it's not the only way of doing it. And we'll come on to the other way in just a second.

• 45:38Nathan Labenz?

  They never released Flamingo to the public, did they? I don't remember ever having a chance to actually use it. I just remember-

• 45:48Speaker 00

  No, I don't remember. I don't think they did. Unlike what we'll come on to in a second and, and most of what we'll cover today.

• 45:56Nathan Labenz?

  Yeah, that was before Google was dancing, I guess.

• 45:58Speaker 00

  Hmm. So that's the basic training recipe we just covered for a cross attention model.

• 46:05Speaker 00

  One of the things that makes large language models so good at what they do is the instruction tuning. And that's proved to be one of the big unlocks. And in the Flamingo recipe there, it's not kind of clear, you know, there was no specific instruction tuning step. And the difficulty there was it's just really difficult to come up with an instruction tuning data set, right? There weren't such things around. There are now. And that kind of this how to do this is the contribution of the next model that we're going to look at, which is the LAVA model, which stands for the Large Language and Vision Assistant. And this is a 2023 vintage model. So we've already jumped on

• 46:46Speaker 00

  like another year. So the original LAVA model comes from a mixed team from some people from Microsoft and then from academic institutions. And it's the first in kind of a long series of LAVA models, all of which are kind of been based on this same recipe. The big innovation here is instruction tuning and how they built the instruction tuning data set. They start with a key observation that the generative vision language models that existed at the time they they built this could only follow a relatively limited range of user instructions. So they could do some captioning for you. They can answer some basic questions about images, but they couldn't do anything like the range of tasks that a language model

• 47:27Speaker 00

  could do once it had been instruction tuned. And one of the reasons they put this down to was just looking at the big data sets that that we used to train models. You have the interleaved text and images from crawled web documents. And you also had these big captioning data sets. We'll talk about a couple of them in a second. But relatively few of them were actually containing like lots of task oriented labels for the images. And that was just missing. So the question is, how can you build such a thing that would work for a vision language model? And should probably talk a little bit about the basic architecture of LAVA, because it actually looks different to the Flamingo model. We talked about that Flamingo described it as being a cross encoder model, because they've

• 48:11Speaker 00

  introduced these new cross encoder layers. The LAVA team chose actually a simpler approach. And this is pioneered, I think, by the Salesforce team that built Blit. So we can call this the auto regressive architecture, rather than the cross attention architecture.

• 48:28Speaker 00

  And the idea is that you're going to take your vision tokens, which have been processed by a vision transformer. Even better if it's from a clip vision transformer, because that's already been aligned in the past to a language model. And then you're going to train a simple projection matrix and inject the tokens directly into the decoder stream. Okay, so no cross attention needed here. You have a simple projection matrix, and then you're going to mix them in. Well, they're in the original Lava architecture, they kind of pre-pended all of the text tokens that were going into the language model backbone with all of the vision tokens. So your kind of training sequences would be a bunch of vision tokens, text, and you'd then be attending to everything to the left.

• 49:13Speaker 00

  Okay. So that's the auto-regressive architecture. It seems a lot simpler than the cross attention one. But it's got maybe one or two downsides to it. The first is that if we generate a long sequence of vision tokens from the vision transformer, we've literally now got to unroll all of them in the decoder, right? So they're all going to be part of the attention mechanism in the decoder. All right. So that's kind of one disadvantage, perhaps. The second is that if we've got this projection matrix, as was used in, and I think they used a simple, like a linear layer in the Lava model, that gives you many fewer parameters in introducing a big cross attention mechanism.

• 49:55Speaker 00

  So you've now got to learn all the alignments using those parameters. And if you want to do anything further, if that's not enough for you, you've got to unfreeze the language model backbone, and you've got to start mucking around with the attention mechanism. So one of the downsides there, as we know, is if we start fine tuning, you know, the language model and the attention mechanism, it's very easy to kind of suffer catastrophic forgetting on some of the tasks that the language model was fine tuned for in the first place. This is one of the downsides of using this auto-regressive architecture that just uses self-attention.

• 50:29Speaker 00

  But that's what they chose to do with Lava.

• 50:34Nathan Labenz?

  Yeah, you mentioned Blip. We did it. One of my first episodes, actually, of the podcast was with the authors of Blip 2. And I remember this sort of, you know, moment of understanding where I was like, wow. So they are somehow converting these images into text embedding space and treating the images as if they were text from that point on. The rest of the model doesn't even know that it has been fed anything other than text because, you know, it's frozen, right? So I mean, it's been trained to handle text. It can only handle text.

• 51:18Nathan Labenz?

  But now somebody's figured out how to represent these images as text in the text embedding space. And one of the really interesting things about that was how you are accessing parts of text embedding space that text itself can never get to, right? It's similar a little bit to what we talked about earlier where the, you have the one hot token level text encoding, but these image projections into text embedding space are not bound by that. At least they weren't in the Blip one. I don't think they would have been in this Lava one either. And so you're just accessing, you realize that like the space of possible inputs

• 52:00Nathan Labenz?

  is actually hugely bigger than the space that the actual 100,000 token vocabulary routinely accesses. And that, again, just kind of magically seems to work. It's like one of these, you know, divine benevolence things where you're telling me we trained an entire language model on nothing but this 100,000 tokens.

• 52:27Nathan Labenz?

  And now it's gonna be just, we're gonna bypass, you know, that layer and just directly go into embedding space with whatever is learned by this projection process. And the language model is still gonna handle that normally? And the answer is yes. It's kind of amazing.

• 52:47Speaker 00

  It's quite an amazing result. I mean, as a simple projection matrix, taking takes your vision tokens, lets you mix them in. I mean, and in other models arbitrarily mix them in with the text tokens. And it doesn't destroy the performance of the transformer. Doesn't seem to need complete retraining of the attention mechanism. Like that simple projection matrix is enough to get it to work with the visual tokens, which I think is quite an amazing finding.

• 53:17Nathan Labenz?

  Yeah, no doubt. I mean, I don't like to do analogies all that much, but I'm just, I'm trying to sort of imagine like, what, is there any, you know, sort of analogous challenge that we could pose to a human? I mean, we're obviously natively multimodal, but to think about putting something into word space that is not words, it's like kind of going directly into thought space, I guess. And yeah, that was, there were a few of these moments. For me, I think one of the reasons that I got so obsessed as early as I did, not having had like a, you know, an academic background in the subject was seeing how the same-ish architecture was working

• 54:00Nathan Labenz?

  across modalities and then starting to see these bridges where it was like, boy, if you can just take two frozen things and kind of mash them up in a couple of different ways, and they all kind of seem to work, then that means we are headed for a world where all of this is going to be integrated and it's going to be quite something to see when that happens. So these were for me like very leading indicator moments of just how much was almost certainly going to be possible. It was like, if the language model could do that and not break, then I think, you know, we've got to expect that there's going to be just a lot more Frankensteining to come, much of which, you know, might be weird in all kinds of ways, but a lot of it's gonna end up being viable.

• 54:46Speaker 00

  Yeah, absolutely. So we said we'd talk a little bit about how they did the instruction tuning with the lava model. And as this is, I guess, what the main contribution of the paper was, we've talked about what the auto aggressive architecture looks like. So the way they did this is actually really, really smart. So they started with images from a dataset called COCO, which was produced by Microsoft, I believe, back in 2014. And the idea behind that, there's about 200,000 images in the COCO dataset. You've got images, and then you've got descriptions with bounding boxes in them describing what's in different areas in the image. And so that actually appears in the text, you know, so in this

• 55:29Speaker 00

  area, region, and then this thing, region is this thing. And the idea is that it's to teach visual grounding for vision models. So what the lava team did, which is very smart, is they used a strong model, I think they used GPT-4, not a vision model, just a language model, and a very careful and clever set of Q-shot prompting templates. And they asked the GPT-4, can you generate a conversation between a questioner and a vision assistant framed as though the assistant can see the image, although GPT-4 cannot see the image? Okay. So, for example, given a description of an image, which might say,

• 56:13Speaker 00

  here's a bunch of people standing around a vehicle, there's luggage on the floor, and then some bounding boxes, here's a piece of luggage, here's a bicycle sitting at the side, here's a person, and so in the bounding boxes. So read that GPT-4, and now come up with a question for me, okay, as though you could see the image. So the question might be, what type of vehicle is in the image? And that's not easy for GPT-4 to do, because the image, the caption will say what kind of vehicle is in. But then you can think of more intelligent questions. For example, what is the thing to the left of the car? And we know from the bounding box that there's a thing, you know, the thing to the left of the car is a bicycle. All right. So not, not even needing to see the image, you can

• 56:56Speaker 00

  come up with that question, just by simply computing the bounding, you know, where the bounding box is. Or you could ask a more in-depth reasoning question like, what challenges do the people around a car face? So the people are loading luggage into the car. All right, so if you just saw the image, you would have to infer all of this stuff from the image. So the really smart thing here is getting these kind of conversational dialogues built up by a strong model like G P T four. And then you've actually got all of this data you can use for instruction tuning. So you're going to ask the questions and you've also got the model answers. Because G P T four knows what the model answer should be. Remember, it came up with a question and the answer using all the information it could get from the descriptions and from the bounding boxes.

• 57:42Speaker 00

  Okay. And that's the kind of the genius here is that when they built the Lala model, they froze the vision transformer. And then they updated the language model they were using and obviously the projection matrix, which is going to align the two. And they only did the the pre-training on six hundred thousand pairs from a captioned images dataset. And they used a very kind of simple form of their instruction question answering. So it was something like, here's a question that G P T four has come up with about the image based on the caption. Then you show the image. And then what you want a language model to do is finish with the answer, which would be, you know, the answer to the question that came up with. And then for the fine tuning, that's when you use this really kind of complicated,

• 58:26Speaker 00

  more sophisticated instruction tuning dataset they came up with, which includes like multi-tone conversations, questions about the images, asking what's in particular regions of the images, asking these reasoning questions again. And they were able to generate one hundred and fifty thousand of these things using this process. And so they used that as the instruction tuning dataset.

• 58:49Speaker 00

  And when they evaluated it at the end, they found that it was outperforming all of the other vision language models at the time, complex reasoning tasks of images. And they found it was also slightly better at conversational tasks than anything else. So what they showed was that putting all this effort into being really smart about generating like augmentations for the instruction tuning dataset yields a much smarter language model at the end that could complete a much wider kind of variety of downstream tasks. So just because you generated them all. And so I think that was a really like smart innovation. And in the subsequent lava models, the same kind of recipe for generating instruction tuning data has been followed,

• 59:33Speaker 00

  although the models have got a lot more sophisticated. And the latest one, which is, I think for a team at ByteDance, is called lava one vision. And that's sitting really, really well on the M M N U leaderboard, which we'll talk about in a moment. So it shows the recipe is still like strong a year and a half later. And that the class of models is really competitive.

• 59:58Nathan Labenz?

  Yeah, it gives you a sense too for why the leading developers seem to think they'll be able to get around whatever data walls may be naturally occurring, because these, you know, these results were achieved with largely synthetic data, at least when it comes to the final margin of actually getting the thing to behave and like be a useful, you know, image assistant. So, yeah, this next one isn't so interesting. I don't know. Well, maybe we'll skip it. But at the time in summer 2022, I was fine tuning the text Da Vinci oh oh two series of models. And they never actually released the fine tuning

• 1:00:42Nathan Labenz?

  there. I think it wasn't very efficient. And they had other things they wanted to do with their resources. And G P T four was was cooking, although I didn't know that at the time. But we had this kind of preview access to that thing. And I was using it to process images in a similar way. One challenge I remember we had was, we wanted to be able to take a video and look at, or try to answer the question, how much, well, first of all, what text is on the screen, but also, how long is that text on the screen? And for an arbitrary video, you could like,

• 1:01:20Nathan Labenz?

  Take a still out of it and use OCR and that's what we did, but then it was like, okay, well now I've got 900 frames in a 30 second video. What is the right frequency to take these stills? And then OCR all of them and then come through and have the language model process. That data was so gnarly. But that was one of the first things where I had this experience of feeling like, geez. I could go try to get human annotators to do this. And create some data set and train on that. But actually, what I'm finding myself drawn to do, and I think this is like, actually still a pretty winning recipe these days for many things. Is 10. I was like, I'll do 10 myself.

• 1:02:04Nathan Labenz?

  Then we'll fine tune on that. We'll have the language model do the next 100. And then, you know, hopefully most of those will be right. The ones that are wrong will correct, then we'll fine tune again. And in that way, kind of bootstrapping into something that could do this like very gnarly task that humans were not very good at. You know, very unfamiliar data, very sort of not, not the kind of thing we're evolved to really handle. You could do it if you really stapled your pants to the chair, but it was tough. And it would have been really tough to hire out with any sort of quality. It was even tough for me to sit there and do the 10 or do inspect the next 100.

• 1:02:42Nathan Labenz?

  But that was definitely a moment where I was like, this is going to be transformative because now we're hitting the point where with the the tools available, I can actually bootstrap my way into a pretty fundamentally new capability. That, and I can do it in a way that's like way faster and way more affordable than having to go out and like actually hire human annotators. And that was just like, hmm, if I'm figuring this out, I bet a lot of people are figuring this out and there's going to be a lot of new. In my case, it was narrow. I wasn't trying to create a general purpose thing.

• 1:03:22Speaker 00

  Yeah.

• 1:03:22Nathan Labenz?

  Even then, I was like, it's so, you know, if I can create a new capability in a few days of work as like a hybrid programmer, annotator, bootstrap my way into this sort of thing. If I can do that in just a couple of days, like, that's how long it would take me to like, set up, probably, probably would have taken longer to set up like decent infrastructure to collect the data and figure out where, what platform are we using to have the annotations done and who do we trust to do this? And how do we like validate that they're actually doing it versus, you know, just filling in whatever and not caring?

• 1:03:56Nathan Labenz?

  And it was like, boy, I can, I can bootstrap my way into that with just a day or two's worth of work. In many cases, once I kind of got the hang of it. It was another one of these moments where it's like, yeah, this is going to happen across a very, very wide range of tasks.

• 1:04:11Speaker 00

  Yeah. And I think this, this, this idea of using AI models to generate your synthetic data for, you know, training the next generation. Is we see it all over the place with the vision language models. So there are examples where generating synthetic OCR data, for example, in order to train them to be able to read corrupted images and know what the text is. It's kind of a classic one. And I've seen generating latex documents as well. So you generate the latex document and so you know the text you started with. Synthetically generated the latex and now you've got a pair. And so this, you know, trying to grow the data sets, and this is always very difficult, especially in the realm of

• 1:04:52Speaker 00

  vision language models, because you can't just deal with a single modality of data. And it has got to be a correspondence between the vision components and the language components. And the correspondence has got to be good. Like, as you said back at the beginning, when we were talking about click, it's like when there was a lot of noise in the data. It was only, there's only so far you can go with the quality of the model at the end. So one of the challenges and something that lots of the teams spend their time doing is thinking about how can I create much larger, really high quality, pre-training data sets where I kind of know I've got a good correspondence between the visual things that I want the model to learn and the language.

• 1:05:31Speaker 00

  So I think, you know, the lava recipe, and by the way, you can get the instruction tuning data set from hugging face, you can see exactly how it's built, it's there. I think it's one of the really smart way, a really creative way to think about how you build out a really sophisticated data set. And then it has this dramatic effect when you actually use it for instruction chain on the capability, the model to act as an assistant, you know, can do all of these other things that that previous vision language models couldn't do.

• 1:06:01Nathan Labenz?

  Well, we're nowhere near the end, so I didn't expect this to be such a nostalgia fest. That'll be less because I think as we get closer to the present day, honestly, just the amount of time I've spent wrangling these like weird idiosyncrasies has dropped pretty dramatically as the foundation models have gotten better for an application developer like me. I've been very content to leave these sorts of weird things in the past and we're, you know, in the next couple of models that you're going to get to through this history, we start to hit the point where it's like, it's just starting to work. So let's keep going and I'll probably be doing less of this, I remember when stuff as we get to the present day.

• 1:06:48Speaker 00

  So good, it comes with age, Nathan.

• 1:06:51Nathan Labenz?

  Time flies, that's for sure.

• 1:06:54Speaker 00

  So I just mentioned, like at the end, I said that, you know, the latest lava model was called lava one vision. And I said it ranks very creditably on this, this MMU benchmark. So we should probably say a word or two about what that is. So MMU, it stands for massive multidiscipline multimodal understanding, which is something of a mouthful. And I guess like easiest way to think about it is it's, it's like the multimodal version of NMLU. Okay. So I think it's probably the, the most interesting and relevant vision language model benchmark for just understanding how smart your VLM is and how much reasoning it can do.

• 1:07:38Speaker 00

  So it's designed to measure explicitly three skills. So perception, what can it see in the image? Knowledge, what does it know about what the image is showing? And reasoning, so what can it infer from the image? And the way it was compiled is by a bunch of students, of course, from various disciplines and subjects at university, drawing questions from online sources, from textbooks, and even from lecture materials. And the idea is that each question that they find has to require like an expert level understanding from the domain in question. You need to have that to answer the question. So they built up about 11,000 of these questions, and they ranged over about 30 different subjects.

• 1:08:23Speaker 00

  So that includes things like history, medicine, electronics, market research, music. And some of them also require sort of mathematical reasoning to solve. For example, some of the problems involve Fourier transforms. Each question says something like, you know, here is an image showing harmonic intervals in the musical score. Which one of these? And you've got like four musical scores. And which one of these is constructed incorrectly? And then it's going to give four options. So it's always comes in fours, you know, in a row. And so the language model has just got to select who's A, B, C, or D that's incorrect. So it's given the multi-choice questions. So quite diverse, a lot required of the models to answer them.

• 1:09:08Speaker 00

  And when they released the benchmark in November 2023, the top scoring model was GPT-4V. And that got about 55% on the benchmark. So like a random guessing would get you 25% on this. The top open source model was one of the Lava models. Like we just discussed, there's the second model in the Lava series. And that got 34%. One interesting development since then, I guess, is that O1, as you might not be surprised to know, is now topping the leaderboard. So in just over a year, the leader score has jumped to 78% at the moment.

• 1:09:49Speaker 00

  So O1 is sitting a full eight points clear of the runner-up on this. But you can see that benchmark is, I wouldn't say crushed yet, but a lot has been done on it in the last kind of 18 months.

• 1:10:01Nathan Labenz?

  By the time we get done with this recording, we might find a new leader at the top.

• 1:10:05Speaker 00

  That is the risk because it's actually O1 preview that is right at the top there. We haven't got, you know, who knows what's been done to O1 preview since it was previewed and now it's O1.

• 1:10:18Speaker 00

  But yeah, it could be beaten. Also, we don't have scores for Gemini 2.0. And we don't have scores for the new Claude model, I think. Actually, you might revise this because I think I might have them further down, actually.

• 1:10:34Nathan Labenz?

  Yeah, I'm looking at their leaderboard page right now. I don't know how often they are updating it. They do have O1. They do have a 3.5 Sonnet. Not clear if this is 3.5 Sonnet original or 3.5 Sonnet new.

• 1:10:46Speaker 00

  Original 3.5 Sonnet, yeah. I actually, I actually think I've, I've managed to find all the scores for those models, actually. So I can tell you what they are later on. But anyway, yeah, O1 is standing top. So what's interesting about that is when the benchmark was first released, the team who produced it actually looked at asking text-only GPT-4 to answer all of the questions. What they would do is they'd extract, you know, any text they needed from the images via OCR or they get the Lava model to caption the images. And they basically give a text-only model, you know, the captioning that was, that was extracted from the image and the question, then ask how it did.

• 1:11:27Speaker 00

  And they report a score from GPT-4 of 34% on the benchmark. And that, I guess, just highlights the fact it's above 25%, highlights the important role that reasoning will play in answering the questions. So reasoning is really important. And the rest of the over 34% and up to whatever the models get, you can put down to their smart interpretation of the visual tokens and their ability to reason over them. So just a quick introduction to the MMMU leaderboard. We'll come back to it because from now on, we're going to score every model on it, see how it does. Shall we crack on?

• 1:12:01Nathan Labenz?

  Let's go.

• 1:12:02Speaker 00

  Okay, so I guess the next thing to talk about is pre-training with vision language models. And what's been learned about that? So what do people do these days? And the model we'll use for examining the pre-training recipes is the Qwen-VL series, which is from the team Alibaba. So this series of models has two of them, Qwen-VL and Qwen-2-VL, 2023 and 2024, respectively.

• 1:12:28Speaker 00

  So they're self retention models, they follow the self retention auto regressive architecture. And they're using Quen's language model, Quen LM, I think it's called, as the backbone. And they're using a vision transformer as the encoder. Okay. And to connect the two, they're using a single cross attention layer. So they're not adding many cross attention layers to the language model. They're using a single kind of standalone cross attention layer, which similar to the perceivore sampler, has got learnable queries. And so what they're solving for there is if you've got an arbitrary number of visual tokens, can we compress it down to a smaller number? And then can we inject them into the language model decoder?

• 1:13:11Speaker 00

  This is really about, in this situation, this is really about using that mechanism just to reduce the number of vision tokens that go into the decoder. Do you remember we said in the auto regressive architecture, you've got to unroll all of those tokens? They become part of the attention that the decoder's got to compute. So if we minimize them, that's better. So that's what they use for connecting the two. If we talk about the training now for the Quen BL language model, the innovation here is that they're actually going to break the pre-training into three stages. So rather than just doing general pre-training, what they learned and what everyone's done subsequently

• 1:13:51Speaker 00

  is that actually you want to break your pre-training into two stages and then do your supervised fine tuning. So in the first phase of pre-training for the VLM, they're taking their image caption data sets and they're taking their interleave data. And so large quantities of data. They are training the vision transformer and they're training the connector module. But they're freezing the language model. Okay. And in the pre-training, they're having all the images resized to 224 by 224. So they're using the kind of natural resolution of the vision transformer. So presumably this design, so you've got, you know, not that much detail in the images,

• 1:14:36Speaker 00

  but you're able to process a lot of them. And the vision transformer is of course small compared to the language model. So it's okay to unfreeze that. And the cross attention module, again, small compared to the language models, okay to unfreeze that. So that's the first pre-training they do. So the innovation is to add in a multitask pre-training step here. And the idea is that for the second phase, you're going to unfreeze the whole language model. Okay, you're now going to allow the images to be a larger size as well. So in the first training step, they were 224 by 224. And now they're going to be sized at 448 by 448. And the vision transformer is fixed, of course.

• 1:15:19Speaker 00

  So this, what this means is we're going to split each of the images into four tiles like this. Okay, so we've got many more visual tokens coming in to the second phase of pre-training. And by multitask, they're also going to be adding in synthetic OCR data. And then they're going to create a visual grounding data set as well. And so the idea here is to have a data set where they've constructed lots of bounding boxes and references for objects in the image, and they're trying to do this at scale with a pipeline. It's going to build out this kind of second pre-training data set at scale with bounding boxes in it, with references in the text, what's the bounding boxes are pointing at. And then they go add in, again, visual question answering data sets.

• 1:16:02Speaker 00

  By the time they started training the Qwen, the Qwen-VL model, there are a number of kind of visual question answering or document question answering benchmarks and fine-tuning data sets that have been released. And so what they did is just, and some of these are of considerable size, in 10, 50, 100,000 images in each case. So they actually added these to the pre-training data set. So now they're representing much greater task diversity in the pre-training mix, slimming down the size of it somewhat. They're also adding in text-only data. And the idea here is because they're going to unfreeze the language model, they're going to be mucking around with the attention mechanism now. And so they need to keep that text-only data in there to kind of preserve text-only performance.

• 1:16:46Speaker 00

  So the multitask pre-training, it's bigger than fine-tuning, it's smaller dataset than the original pre-training dataset. And we'll see this theme again, that because the multimodal dataset is scarcer, like pre-training like this, first on your low-quality data, then on your higher-quality data, and then finally taking a much smaller supervised fine-tuning dataset, which involves a lot more manual augmentation of the images and using a self-instruct process to come up with decent prompts, is the way to do it. And that actually, for the Qwen-VL team, produced a really, really strong vision language model at the end.

• 1:17:26Speaker 00

  Qwen-2, which is the latest one in the series, is sitting just behind the leading models from OpenAI, Anthropic and Google on the MMU, but it's sitting above pretty much all other language models. The small models are all open-sourced. The 72B class, which is their leader model, is available behind an API, but they do say in a paper that they plan to release it at some point. Okay. But I thought that's, it's an interesting model because they really broke down this pre-training process into two parts. They mixed in much more diverse data into the second multitask pre-training, and had a really dramatic effect on the capabilities of the language model at the end.

• 1:18:10Nathan Labenz?

  So let me just try to summarize the narrative of the training of a model like this. You start with a language model, then you throw your sort of high-volume but low-quality or mixed-quality data at it, and that is the vision language pre-training step, which comes after the original language pre-training step. And the main purpose of that is to sort of get these things on the same page, right? Like, we at least need to sort of...

• 1:18:52Nathan Labenz?

  Sort of bring the latent spaces together. Once that's working, then you say, okay, now these things are, and I guess, why wouldn't you want to change the language model? At first, because basically, I guess you have a language model that you're really reasonably happy with. And if you're doing back propagation through the language model. You're like, well, I don't necessarily know what's going to change there, and I don't want to be. Changing things and potentially losing capability. When I, I kind of know like what I need to change, what I need to change is the part that is. Mapping the images into the language latent space. So do that first, get to a decent. Place.

• 1:19:34Nathan Labenz?

  Then open up training of the full model. Now it's like, okay, you guys are generally working well together. Now you're, all the sort of constraints are off. Now we're going to. Back propagate through the entire thing. And the, I think the interesting detail, a couple of interesting details there, but one was definitely. The need to continue to mix in standard text only stuff, because again, you don't want to. Be over indexing on this one particular task that won't necessarily. I mean, what's nice about these things is they're super general, right? So you don't have to give them an image. You still want them to be able to work. As normal. So you got to mix that. Text only. Data in as you do this phase, but this is where they really sort of.

• 1:20:18Nathan Labenz?

  Cohere or anneal or whatever is the right word into. A single system that's all been end to end trained together. And then. Naturally, the last step is the data set's not given, and that's certainly been a very. Common theme outside of vision two is just that, you know, the sort of. Preference data or instruct data on which a lot of the frontier models. Are trained is highly proprietary technology that.

• 1:20:45Nathan Labenz?

  Is expensive to generate. From a recent episode that I did with Nathan Lambert, who studies this post training stuff deeply. One of the major takeaways from that conversation was him saying, in the absence of a frontier model like a GPT-4 that we use to generate the data that we then turn around and use for instruction tuning, we would have no way to get the volume of that quality of data that we would need to do this. I've probably referred back to that on like half of the episodes since because it's I think a really interesting data point for where things are going in terms of like, as they start to clamp down on reasoning.

• 1:21:28Nathan Labenz?

  Of course, we've now also got Google's thinking model that is sharing its reasoning. So and Chinese ones too that are sharing their reasoning. So the final chapter by no means is written, but it's really interesting to see how that instruct that final phase of training, the data that powers that has to be so high quality and is so expensive to get and therefore so valuable that it's typically not released meta for all their, you know, openness is also not releasing that kind of data. And so people are currently left to like, try to generate it with GPT-4 if they want to do something in the or whatever in the wild. But what is the future of that? Is that still going to be a viable option? Maybe with Gemini, you know, showing their reasoning traces, but you know, maybe not.

• 1:22:13Nathan Labenz?

  We'll see kind of how the the dynamics evolve. But yeah, I think that's good. And it is remarkable how I always marvel to at like, Chinese companies are not far behind, you know, I think. I don't know if you have a perspective on this.

• 1:22:28Speaker 00

  They needed in the LMS, in fact. Yeah.

• 1:22:33Nathan Labenz?

  Yeah, well, I don't know if now is the time where you want to maybe do this a little bit later, but I'm interested to hear your thoughts on leaders versus fast follows versus, you know, certainly. I think we in the West, you know, a couple of very broad terms, the broad we and the broad West seem to be, in my view, kind of overconfident about just how much of a lead we quote unquote have relative to Chinese researchers. Even on some of these earlier papers that we've been discussing, you know, there's an awful lot of Chinese names on papers that are coming out of Western institutions. So that's a whole other dynamic here. But yeah, we can either put a pin and come back to this, but I'm interested in sort of how you handicap

• 1:23:16Nathan Labenz?

  this field and and sort of how you think like, how we should even determine, you know, I mean, the Chinese companies are definitely more open, but it seems like typically, I would have said your OpenAI's and Google seem to get there a little bit first, but they're not open. So if you were to say they're the leaders in open models, I would say, yeah, that seems pretty, yeah, pretty apt leader overall, including the proprietary ones. I don't know if you would, you know, say the same or see it differently. But I'm interested in how you think about that head to head.

• 1:23:48Speaker 00

  So we can answer that now. And then I'll offer a couple of comments to your observation about the high quality data in pre-training. So the next model we're going to look at is the Intern VL model. And again, this comes from an organization called the OpenGV Lab, which is based in Shanghai University. Okay. And it's probably the leading open source model. And the story behind the Intern VL series is it's been one about how do you scale these things up? How do you make them bigger and bigger and train efficiently? And they're looking at efficient training, and they're also looking at larger models.

• 1:24:28Speaker 00

  And one thing I think is quite interesting is that all of the top open source VLMs are kind of hovering around the 70 to 80 billion parameter mark, term of size.

• 1:24:41Speaker 00

  The top proprietary models, we don't know, right? We don't know how big they are, but bigger than that, right? Maybe an order of magnitude bigger than that. The latest model that's the OpenGV lab released, they looked at how efficiently they could train. And they said they ended up training on 120 billion tokens, I think, of mixed data. All right. And then they said this was very efficient because as compared to the Qwen-VL model, which we just looked at, the latest Qwen-2VL, it's a train on 1.5 trillion tokens. Okay. Shortly after they released the model, they released a data set which has 1.5 trillion tokens in it. So it's one of the largest out there.

• 1:25:25Speaker 00

  So you can see already that they've gone to a 72 billion parameter model class. They trained on 120 billion tokens to get there. The data set they just released is 10 times that size. So they can see, you know, my guess would be they're planning to scale up very quickly and get to a much larger model since they already have the top open source model and it's pretty competitive with everything else out there. I'm really interested to see what they come up with. And this, of course, is, you know, it's a team from a university, which is super impressive without the enormous funding behind it that some of the front end labs have.

• 1:26:03Nathan Labenz?

  Now, when you say top visual language model, is that an MMMU?

• 1:26:08Speaker 00

  Yeah, so, sorry, I'm being lazy. I'm talking about just MMMU, which is my benchmark of benchmark. We should probably talk a bit later about a couple of the other benchmarks, which are really interesting, but MMMU is kind of the big one in the same way that MMLU might be the default benchmark you show when looking at language models. Yeah.

• 1:26:31Nathan Labenz?

  Yeah, okay, cool. My kind of rough understanding and not necessarily rule of thumb, but. I remember Llama 2 was trained on 10 trillion tokens. And I've always kind of rounded to say it seems like the multimodality comes at like another 10% cost, like it's another 1 trillion-ish tokens typically. Do you think that's like a good intuition? I mean, this Intern VL one seems to be notably significantly less than that. It's like two orders of magnitude difference between the original full pre-training and the image portion of training. But what would you say is kind of the normal?

• 1:27:15Nathan Labenz?

  Is this in fact an outlier in terms of how small the dataset is?

• 1:27:19Speaker 00

  I think there's not a normal because what's happening is the dataset is getting much bigger now. So let's say at the start of this year, so the biggest data sets, multimodal data set you could train on would be the Leon, right? Which is 5.8 billion images with text captions. It's been filtered for quality, so there should be high quality captions and they're all extracted from the common crawl. And so there's a German organization that produced this, I believe. And that, I think, is the largest kind of image caption dataset that's publicly available. Okay. But just in the last few months, we've seen an interleaved dataset called Mint, which has been released by, it's a multi-contributed team, but Salesforce were behind it.

• 1:28:04Speaker 00

  So that's a trillion tokens, that one. And it includes not just HTML, but it includes PDFs, archived papers, things like that. And the OpenGV Lab, who are the team that reduced the Intern VL model, have just released a dataset they call Omnicorpus, which is again, another interleaved dataset. It's 2.2 billion documents sourced from common crawl dumps. They say it's got 8 billion images in it and 1.6 trillion text tokens. So those are just much bigger than, you know, anything that was previously available. So I I would say you're probably about right now by thinking a trillion tokens is about right.

• 1:28:47Speaker 00

  Organizations like Meta do have access, you know, and when we talk about the Lama 3D, we see that that is Russia, you have a token count used for division, division model there. But for open source researchers, like the last few months have really seen the arrival of these huge interleaved data sets, which has kind of really jumped the pre-training data set size that's available.

• 1:29:12Nathan Labenz?

  Scaling's a hell of a drug.

• 1:29:15Speaker 00

  Exactly. Okay, so we've looked at the Qwen-VL model and we've looked at the multi-part pre-training stage, where we can use our lower quality data set at the point where we're training the vision transformer and the cross-attention layer. And we don't want to muck around with a language model at that point, because you know, that's been heavily trained in the past. It's doing great, thanks. We don't want to mess that up. So we don't want to use low-quality images and captions and be training that thing. So we just trained the vision transformer and the cross-encoder. When we come to the multi-task pre-training, we've got a lot more attention and care into the data set that now we're going to allow ourselves to train the language model.

• 1:29:58Speaker 00

  And that, I think, was just in response to the reflection you made earlier about, you know, why do this? Or the question about why do this in multiple stages like this? Yeah, it's going up the quality scale, down the size, and unfreezing world parameters as we go. And that recipe just works really well for the vision language models. I guess the last one for now that I wanted to cover is the InternVL series from the OpenGV Lab. And this is a team from Shanghai University in China. So one of the challenges that's facing particularly open source researchers building vision language models is the training cost, okay? Particularly when you have to take a large model, unfreeze the parameters, and find

• 1:30:43Speaker 00

  and fine-tune it, or do continued pre-training because you're now introducing this extra modality.

• 1:30:49Speaker 00

  So in a series of three models, or four models, actually, they've looked at some ways that they can improve the capabilities of the open source language models. And one of their papers was in fact called "How Far Are We's GPT-4V Closing the Gap?" So that was for their 1.5 model. But I'll just touch on each of them, kind of one after the other, because I think each of them introduces something really interesting to the picture and tells us something about where vision language models are going. So their first VLM, which is called InternVL, they observed that everything today, we've been using a vision transformer that's been relatively small

• 1:31:33Speaker 00

  and it's also been pre-trained separately from the language model that it'll eventually connect it to. So if we started, for example, with a clip net, so one that had been used in the clip model, it's of course been aligned to a language model encoder, but it's a different language model to the one the decoder will eventually want to connect it to.

• 1:31:56Speaker 00

  So what they did is they started with a fresh vision transformer and they scaled it up. So there's a 6 billion parameters, which is large for a vision transformer, right? So the VIT-H, a bit huge, has around 600 million parameters in it. And I think the biggest of the standard vision transformers is like the VIT-G, and that's about 1.8 billion parameters. So this again is several times that size. And what they do is they conduct a fresh, kind of contrasted, pre-training of the model. They use a frozen Llama 7B, like the original Llama, as a decoder, okay? And what they do is they take, in the same way you would train a clip model,

• 1:32:42Speaker 00

  you take images and text pairs and you start to feed both of them into each, you know, the text into the text model, the images into the vision transformer. You then take the hidden states from your Llama. You can then take an average over that embedding, that's going to be how you embed the text. And then at the end of the vision transformer, you're going to take the states from that and you're going to, for example, pull that somehow. And then exactly the same deal with the clip model. You're going to do a contrasted pre-training step, right? So you're going to try and get the correct caption pairs to be similar, the others to be far apart. So not so different like to to clip several years back, it's just we're using a decoder now.

• 1:33:23Speaker 00

  And it's a decoder, it's a Llama 7B, so it's going to be much more similar to the language models we then want to connect it to in our VLM. And we also just scaled up the size of the vision transformer. Okay. So that was their first innovation here. And what they showed is that this leads to a really high quality and really well aligned vision transformer. And what they could do is they can actually forget about the llama model that they used for the contrasted pre-training. And they can connect it to a totally different language model when they build a VLM and it works really, really well. Okay, the first innovation, just train the vision transformers from scratch. Using something much more similar to the language model they're eventually going to connect it to when you're doing the contrasted pre-training.

• 1:34:08Speaker 00

  Cool. That was the first inter V L. The second one, the 1.5 model. They looked at one of the well, image resolution. That was their, the big question there. So we talked way back at the beginning about, you know, images don't just come in 224 by 224 squares, you know? Or 448 by 448 squares, they come in many different kinds of resolution. So how can we try and, you know, if we've got a higher resolution image, how do we try and get more out of it? So what they did is they developed this strategy they called dynamic high resolution. Okay. What this is, it takes an image, it could be of an arbitrarily high aspect ratio resolution.

• 1:34:49Speaker 00

  And they segmented into tiles of a fixed size, 448 by 448. Okay. And the number of tiles that they use are going to be based on the aspect ratio and the resolution of the image. Okay, so they try and match the tiling configurations, 4 by 4, 4 by 2, 1 by 2. They try and match that to the image itself in its natural resolution. Okay. Then going to encode each of the patches from the larger image separately. And then concatenate a thumbnail of the entire image to the end of the sequence. So now you've got your thumbnail and you've got all of your patches of high resolution. You know, great big long set of visual tokens.

• 1:35:34Speaker 00

  In that point, they've got too many visual tokens. And so they use something called a pixel shuffle, which is just a strategy for trying to compress that down. And the way they do that is if you imagine you've got a patch from a visual tensor of an image, it's got a width, the height and the depth. One of the things you can do, obviously, the number of tokens you're going to get out of that at the end is determined by the width and the height of the patch. So what they can do instead is they can just resize that. So in other words, there's more going to the depth dimension of the tensor. And then you're able to split it up into fewer patches. And that's just called the pixel shuffle strategy.

• 1:36:13Speaker 00

  And because this dynamic high resolution ends up with so many tokens, that was the way to squeeze it down. So you use the pixel shuffle strategy. And like quite a few of the latest V L Ls have done the same thing. So Quen 2 V L does the same thing, does now dynamic high resolution. But this is quite an interesting innovation. So now you can essentially process images at whatever the natural resolution is, simply by generating more tokens. And of course, if you're using a self retention or decoder only architecture, you don't worry about whether you're putting in 100 visual tokens or 1,000, except for the expense of having to unroll them in the decoder. There's no kind of haves.

• 1:36:57Speaker 00

  You don't need to have a perceivable sampler that somehow squishes them all down to the same dimensionality at the end. So that's kind of the innovation from the one point tried model for internal V L. And

• 1:37:12Nathan Labenz?

  I was doing, I was searching for something and I would just wanted to flag that while we don't know too much about how a G P T four V works, we can look at pricing and that gives us a pretty good indicator that it is probably very similar under the hood. Because you can go even just on the OpenAI pricing page and click the low resolution checkbox. And then what you see is 75 tokens are your base number of tokens. For any image, that's the minimum you can get.

• 1:37:47Nathan Labenz?

  That would seem to correspond to the thumbnail in this last scheme. And then if you uncheck the low resolution box and start to increase the height and width, what you see is that it starts with one tile. And the max, I believe a tile can be is like 512 by 512.

• 1:38:07Nathan Labenz?

  Once you get over, either the height or width gets over 512, even if the other dimension stays small, now you're into two tiles. And you can still continue to get charged those 75 base tokens. So it seems like they are probably doing something under the hood where it's like, yep, you're always going to have that full image as a single thing in low res. And then the question is, how many higher resolution tiles are you going to have? That's going to depend on how big the images that you feed us. And basically you see this exact scheme reflected in the OpenAI pricing structure. So interesting. Yeah. I wanted to, this is probably on me because I was a little bit confirming that as you were describing it.

• 1:38:49Nathan Labenz?

  But can you tell me the pixel shuffle thing again? I didn't quite grok it.

• 1:38:56Speaker 00

  Yeah. So the explanation I gave was the best to my knowledge because I didn't delve into this too much. So, but basically you have a height and a width and a number of channels in like any representation of an image. And the height and the width determine the number of tokens you're going to get out at the end. As the idea is, can I just reshape the tensor such that there's more in more in the depth channel? And then I'm going to unlock a few tokens at the end.

• 1:39:27Nathan Labenz?

  Hmm.

• 1:39:29Nathan Labenz?

  So in other words, if my,

• 1:39:33Nathan Labenz?

  if I go back to the OpenAI pricing thing, if I create a tall but narrow image.

• 1:39:42Nathan Labenz?

  If it is, if a, if a single tile can be up to five twelve by five twelve, and I have a five thirteen by one hundred and fifty image, that can fit into that space. But in its natural orientation, it doesn't. And basically what you're describing here sounds like a way of sort of reshaping that so that it kind of fits into the space, which I would assume under the hood with data augmentation and kind of, I mean, there's a ton of that kind of stuff, right? Where sort of weird, I mean, the, from the first, right, of just putting everything into the same two twenty-four by two twenty-four. There's a long history of these sort of like

• 1:40:22Nathan Labenz?

  programmatic manipulations of images to put it in some form, which might be quite strange from a human visual system perspective, but which ultimately probably makes the AI more robust. And in this case also, you know, demonstrates that you can save tokens, save money, save compute all at the same time.

• 1:40:44Speaker 00

  Yeah, I mean, as best I understand, it's, it's about just reshaping the tensors themselves for the, that you've encoded. So you're actually shifting stuff from the X and Y dimension to the depth dimension. And I, it felt to me when I was reading about it, and it was like similar to what goes on in like a U-Net, you know, where you're actually just reshaping the tensors again and again, making them deeper and narrower. Do you like, you're not actually losing information, you're just kind of adjusting where it is. So you come out with something very different, the different dimensions, but you haven't actually, you know, if you multiply the dimensions out, it's still the same size.

• 1:41:24Speaker 00

  I guess the last of the intern VL models to look at is the latest one, which is two point five. And so, you know, we've looked at how we, how we use dynamic high resolution. So that was in one point five, looked at how we scale up the vision transfer itself in the original one point zero model. And also do the contrast of pre-training using a much bigger language model. So the, the third thing that they looked at in, in, of note, in the last two point five. Is using what they call a progressive scaling strategy. Okay, so the idea is to try and train efficiently on a large number of tokens.

• 1:42:05Speaker 00

  And what they figured out is, because they built several classes of model, they built seven billion parameter version, and then like a mid-sized one, and then a seventy-eight billion parameter one at the end. What they did is they aligned the vision transformer to the smaller language model backbone. So they do the training process with a vision transformer and a smaller language model. And then they swap the smaller one out, they introduce the bigger, you know, the next grade up language model, and then continue the training. And then they swap that out, and they bring in the biggest one, and continue the pre-training. And what they find by doing this, so progressively increasing the size of the language model backbone during pre-training,

• 1:42:47Speaker 00

  is that they they reach convergence like quite early on in the training process with a smaller language model. Okay. And at that point, the vision model, vision transformer, has learned a lot of what it needs to learn in being aligned to this class of decoder language model. So they can swap that out, and they can put the next figure one in, and they can continue the training. And it turns out it's much more efficient to do this than it is to start with a large language model, start with the large vision decoder, and try to align them both at scale from scratch. And in the paper, they reported that this strategy of what they call progressive scaling, they reckon they read one hundred and twenty billion tokens during the pre-training phase of the two point five model.

• 1:43:33Speaker 00

  And that's their nearest peer competitor, which is the Qwen two V L language model. I had to process about one point four trillion tokens to kind of reach the same complexity loss. So they reckon it's a very, very much more efficient way to do it. And so there you go. That's kind of three strategies to use. Dynamic high resolution to how we process the images. How we're training the much bigger vision transformer. And then this kind of progressive scaling strategy. And this seems to be a winning recipe. Because if we look at the M M U leaderboard at the moment. The Intern V L two point five, the seventy eight billion parameter class. Is sitting just behind O one.

• 1:44:16Speaker 00

  And that means it's beating G P T four O, the legend from May this year, it's on the leaderboard. The original three point five sonnets and Gemini one point five pro on the M M M U. So it's an extremely successful recipe. It also seems to be, by the way, on a visual question answering and O C R bench as well, which two other important kind of benchmarks.

• 1:44:41Nathan Labenz?

  That data, scale of data thing is really interesting because it's not just that it is saving compute by starting with a smaller model. It's actually also that it's using something like one twelfth of the actual data set size. So you're saving on two dimensions. And yeah, that's really interesting. That reminds me of an episode, which was one of my favorites that we did with a couple of guys from Microsoft. On Tiny L M. Which was language models of this, I think of like

• 1:45:17Nathan Labenz?

  tens of millions of parameters, like really small, trained exclusively on these like short kind of kid's story type documents. And they looked at what do these things learn and in what order? And it was kind of like, you could see that they were learning things like parts of speech and, you know, really kind of structural elements of language first. And then gradually started to seem to have an understanding of like nouns and, you know, what was what? And then started to become coherent. And at sort of the far end of their process, they started to see like what I remember as micro reasoning skills.

• 1:46:02Nathan Labenz?

  Which was like, you know, the farthest they pushed this. And again, these are very, very small models, but you could get to the point where it was like, you know, Sally doesn't like soup. So, you know, Jimmy offered her.

• 1:46:19Nathan Labenz?

  Blank, right? And the earlier models would just put soup in again because it was like, well, soup appeared once, you know, soup is probably likely again.

• 1:46:27Speaker 00

  Yeah.

• 1:46:27Nathan Labenz?

  And at the end of their training, they would start to see these micro reasoning skills. It was like, well, it's got to be something else besides soup, given, you know, the full context of what has come. I don't know that they looked at like a progressive scaling from those really small models up, but do you have any intuition for why? I guess what I'm, what I'm kind of struggling for, and maybe there is no good answer right now, but it seems like we're saying something here that is like, the small model is actually learning faster. The small model is more sample efficient.

• 1:47:08Speaker 00

  Yeah. So let's imagine what we're trying to teach the models here, right? So the same way you just described with the small language models. Okay. So there are progressively more complicated understanding and modes of reasoning that you can learn as training continues, right? And we can imagine that larger models can have more capacity to learn more of these things. But a larger model, of course, has many more free parameters during training. So if you have a lot of free parameters, it's going to take you longer to find the right kind of basins in the gradient descent that actually represents good capabilities,

• 1:47:51Speaker 00

  reasoning capabilities and understanding capabilities. So the idea here, and I think this is why this is working, is that if you start with a smaller language model, you start aligning your vision transformer to that, there is going to be some cap on the degree of complexity of the tasks it can undertake. But because you've got fewer free parameters in the training setup, you're going to find kind of good solutions sooner. And at that point, you can then use the larger of your two language models. You now got more free parameters, but you've already found a reasonable place in the search space that you're starting from. And so you just continue the pre-training from there, right? Now you've got more capability,

• 1:48:34Speaker 00

  potential capability in the larger connected model, that you're already starting from a good place, right? So you don't have to search as widely. So that would be my intuition as to why does progressive scaling strategy works?

• 1:48:51Nathan Labenz?

  Yeah, I mean, that is quite interesting. There was a recent claim out of one of the Chinese companies. I forget which one it was exactly, but they basically said that they had trained a roughly frontier class model at like, single-digit compute requirements of what they believe the leading developers in the West, quote unquote, the West had, the big model at the West had used. And this sort of thing, you know, could be a really interesting way that that is happening. So, yeah, that's a striking data point. I mean, the idea that you can do it with under 10%. Again, it's not just that the parameters are fewer and you save compute that way,

• 1:49:32Nathan Labenz?

  but it's a compounding savings because you're also using far fewer data points. This would probably be relative to, you know, if they did quint to full size for 1.4 trillion tokens, and then this progressive thing only took 120 billion tokens, it would be like this low to mid-single-digit compute. So, yeah, that's definitely really, it's worth a ponder.

• 1:50:00Speaker 00

  Yeah, so we've come now to the InternVL 2.5, which is the top open source model out there in the MMU benchmark. And you may have noticed that since we talked about Flamingo, everything else was talked about has been based on this auto-regressive self-attention architecture. And so one thing you may be wondering is, okay, does that mean this cross-attention architecture is dead? And the answer is no, even though most teams have opted for this kind of auto-regressive architecture, just to be different, maybe, when the Lama 3 vision models were released earlier this year, that was based, the Lama 3 V was based on cross-attention model.

• 1:50:45Speaker 00

  So they used a vision transformer, VIT H14, should I know what that is now? It's a 600-million parameter vision transformer. But they, I think they also introduced some new cross-attention layers actually into the vision transformer itself.

• 1:51:03Speaker 00

  So they made some modifications to it. The report given, like the report for the Lama 3D model, goes into a bit of detail about some of the modifications were made, but doesn't give you quite the same level of detail that you might get in some of the other papers covered. So we know that, for example, they modified the vision transformer. And we know also when they did, they used some very large pre-training data sets. They did the same things a lot of other teams did, adding kind of machine-generated OCRs where a lot of effort's spent improving the safety filtering, desensitizing, deduplicating, and quality filtering the multimodal data sets that they use. So that was a real big thing there.

• 1:51:47Speaker 00

  They also did some synthetic augmentation of their pre-training data sets, so it's kind of the same way we've seen the QuenVL team do. So this is like adding synthetic captions, generating kind of tabular data, generating kind of latex documents. So adding a lot more of this stuff in at scale into the pre-training. And then when they do train the model itself, the Lama 3 model, they added new cross-attention layers into it. And they then froze the rest of the Lama 3 model. So they only trained the cross-attention layers and the vision transformer. And that is both during kind of pre-training and during supervised fine-tuning. And they also did DPO at the end as well,

• 1:52:31Speaker 00

  which I think is the first time that I've come across DPO being used in the vision language models. But it's interesting that I've also seen it again recently. So it seems to be more of a thing now. So one of the advantages, I guess, of this is that if you're just training the cross-attention layers and you're not doing a full fine-tune of the Lama 3 model, you're preserving all of the capabilities of Lama 3 and you're not risking degrading them by introducing the vision component. And this might be why they decided to do this with cross-attention rather than using the self-attention. You know, if we're doing the auto-aggressive architecture, we've really just got that projection matrix to play with, you know?

• 1:53:14Speaker 00

  And once, once we've trained that as best we can to do the alignment of the text and the images, if we want to keep improving the model, we basically got to unfreeze the attention mechanism and train that in the decoder. Well, they didn't have to do this because they were using quite a large, like a 90 billion parameter Lama 3, 25% of those parameters roughly being introduced for their fresh cross-attention layers, training on those, that's a lot of free parameters. So they just, they just did that when they, when they trained the model. And the Lama 3.2, the 90 billion parameter version of that is the second-placed open source model on the M M M O at the moment.

• 1:53:56Speaker 00

  So it really shows that, so just behind the internal BL 2.5 is the one that we talked about. So it kind of just goes to show that it actually really doesn't matter. Or, you know, you can build frontier, open source, vision language models using either of those two recipes. And that was what they, what they showed.

• 1:54:18Nathan Labenz?

  Is there any performance difference or any sort of practical? I mean, I hear you on the like, and having seen just a tiny bit of Lama development from the inside, not really the inside, but I participated in a little safety review project for Lama 3. And I mean, it just, it was a lot of people moving a lot of different directions is kind of how I would summarize what was going on there. You know, it's amazing on some level that the whole thing comes together. I suppose that's probably always the case, but out of OpenAI, out of Anthropic, you see these sort of small focused teams, or at least that's the perception from the outside. The, you know, internals of Meta were just like, you know, felt like 10 different projects going on at once.

• 1:55:03Nathan Labenz?

  So I could easily understand and interpret this as being a reflection of just how much more sprawling the organization is and multiple different goals. And maybe also thinking, geez, you know, not everybody wants or needs vision, you know, we'll, let's create that modularity for the open source community that's going to use this downstream as well. All those things I get, is there anything that people should have in mind about? The different architectures leading to different results? Or does it really seem to be just kind of either way works? And as long as you do a good job, you really can't tell the difference after the fact?

• 1:55:39Speaker 00

  Yeah, so there are some, there are some differences in that. There are some ways in which the architectures leads to, to different results. Obviously, in the kind of on the design side, I mean, introducing the cross attention blocks adds more three parameters to the vision language model than just introducing a simple kind of projection matrix and going down the ultra aggressive route. So with a larger model like Lama 3, there are probably enough, I guess they figure there's enough three parameters introduced by the cross attention blocks that we can train our kind of vision language modality alignment and get a really good result just by training on the newly introduced blocks.

• 1:56:23Speaker 00

  Whereas, as I've said before, like the downside of using the kind of self attention architecture is that if you are, you know, if you've gotten the best alignment you can, just out of whatever MLP you use to connect the vision tokens to the language model, then you've got to unfreeze the language model itself. And at that point, you've got to worry about degrading of the capabilities. So it could be that the decision was taken with like Lama 3, like it's a big model. We, we want to preserve all of the capabilities of our language model. So let's not muck around with a large model. Let's introduce these new cross attention layers and let's just see if we can align it for the vision language tasks.

• 1:57:08Speaker 00

  And if we do so, is it good? And the answer is, yeah, it's really good. Provided you put a lot of attention into, you know, cleaning and curating the data set, which they did. And they used a lot of tokens as well, trillion tokens or so in the training budget. So which, you know, if you've got a lot of parameters to tune that you just introduced, kind of makes sense. So the recipe works. There is, you know, some indication that the cross attention model is not so good for things like OCR and maybe also not for some forms of multimodal reasoning. For OCR, the kind of the The reason for that might be because you need to introduce these, you've got these

• 1:57:50Speaker 00

  new cross attention layers. And so you've got to use something a bit like a procedural resampler or something else to kind of fix the size of your vision tokens so that you can actually look them up. And for this reason, something like a procedural resampler is doing a bit of shuffling of the visual tokens at quite a fine grain. And this might be why it's affecting OCR performance. So a few teams have noted this, kind of cross attention, the attention architecture is not quite so good when it comes to these, these tasks that require really kind of fine grained understanding of small errands of the documents. Or, so that's kind of one downside. I've also read some authors speculating

• 1:58:32Speaker 00

  that multimodal understanding, so like reasoning, is actually better in the decoder only, the self attention architecture as well. And this might, you know, is not really know why, but it's just, it seems that putting the vision tokens into this, mixing them up in the same sequence with the text tokens in a decoder only architecture, the attention mechanism just finds it easier to kind of reason over that. But again, nobody really knows why. These are just two kind of minor findings. And I think it's probably enough to steer people in general over towards the kind of decoder only self retention architecture, which I think is probably going to be the one that wins out. Then as caveat, which we'll come on to a second, might not quite be the case.

• 1:59:20Nathan Labenz?

  The only other thing I can offer at the moment is, I think one of the better AI podcast episodes of the year was from Late in Space where Swix had Yi Tay previously and now once again of Google, but with a sort of middle period where he was involved with Reca, I hope I'm saying that right. And his take was basically that.

• 1:59:44Nathan Labenz?

  Some of these architectures are a legacy of the fact that they were originally different teams. And that, you know, again, probably plays out much more at like a Microsoft or a Meta or a Google where they had for a long time, you know, an organizational architecture or hierarchy or whatever that had people focusing on different modalities before the great unification of all architectures showed up. And some of that organization persists even when, you know, there, there now is a sort of unification of, or at least potential unification of the architecture. And so, yeah, you can maybe see some of these echoes where it's like, the language model team, you know, is moving on to the next thing. And now you're

• 2:00:26Nathan Labenz?

  kind of taking the baton on this. And, you know, this, this architecture is friendly to that. But it does seem hard to imagine, and I know we've got, you know, joint pre-training coming up, not too much deeper into the agenda. It does seem like at some point, the bitter lesson has to come for this, right? I mean, there's no, no escaping that forever, presumably.

• 2:00:48Speaker 00

  Yeah, it does feel that, you know, the Lama 3D model does feel very different to everything else that's come out of meta or fair in the last couple of years. Like I say, they're focused a lot on kind of like early fusion architectures, which we'll talk about in a minute. This does feel like a bit of an odd one out, going to the kind of cross attention approach. And not just nodding out, given meta's research, but a bit of an odd one out, getting like, you know, all of the models that have come out of the last year or so, where they've really been tending towards the auto-regressive architecture.

• 2:01:24Speaker 00

  But like I say, it could just be a feature of wanting to preserve the performance of the Lama 3 kind of on language modeling tasks, just by introducing cross attention layers and not having to muck around. With the, you know, the models already perfectly well fine tuned self attention mechanism. Yeah, okay, cool. So we've covered a handful of what I think are some of the more significant models. And by no means have we covered all the significant visual language models. But we've looked a bit at, you know, the importance of some of the things that have been learned along the way. The importance of interleaved data at scale. The importance of data augmentation in the pre-training mix.

• 2:02:07Speaker 00

  And then staging your pre-training data as well. So staging a pre-training. So you start simple, gradually unfreeze more and more parameters of the combined model. Adding in all this augmented data of high qualities, you go. We've looked at that. We've looked at some of these mechanisms for processing high resolution images. So tiling things and then adding a little thumbnail at the end. Looked at the progressive scaling. So that is aligning your vision transformer and your language model backbone with a smaller language model. And then once you're reaching kind of plateauing in your training run, switching it out for a larger one and carrying on. And we've also looked at the importance of task diversity in instruction tuning, which was shown by the LabA team.

• 2:02:52Speaker 00

  And just how important that is to getting your vision assistant at the end to be able to complete a wide array of vision language tasks. And where we're at now is there's this kind of these two architectures, this self-attention or auto-regressive architecture. Where we're just injecting the token, the visual tokens directly into the decoder stream along with the text tokens. And then we've got this kind of cross-attention architecture where we're actually injecting fresh cross-attention layers into our language model. And then we're using those to kind of look at the encoded visual tokens.

• 2:03:29Speaker 00

  And a couple of recent research teams have tried to do a more systematic comparison of these two architectures and said, okay, which is best? And one of them was a team from Hugging Face. And they've built a series of a couple of models which they call IDEFICS, I D E F I C S. And although their two papers were building these models, really what they were doing, I think the team was doing is exploring what makes vision language models work well. And that was really the thrust of their research. So what they've done, what the Hugging Face team have done in their two paper series,

• 2:04:11Speaker 00

  is try to explore what happens if you take a particular decoder and you take a particular vision transformer and then you try out both of the architectures, okay? And see what happens, yeah? So you've got the same training data. In other words, you're just keeping the experimental conditions the same and just changing how you connect the two and looking at what happens. And so some of the key findings from their research, well, first off, is that if you freeze the language model and you only train the newly initialized parameters, so these would be the cross-attention layers, or they would be the projection matrix if you're using the autoregressive architecture, then the cross-attention architecture works a lot

• 2:04:55Speaker 00

  better, like gives you better results. That's perhaps not surprising, since if you bring in some new cross-attention layers, you've got more parameters there to play with than if you just have a simple projection in the autoregressive architecture. But then they say, so when they try to update the language model backbone, the autoregressive architecture can perform much better. So one of the things they noted in, I think it's the second of their papers, is if you try to do a full update of the attention mechanism in the language models, they discovered they had training instability, so it's really, really hard to get the training to work. So they just switched to using low-rank adapters and there's no issues.

• 2:05:39Speaker 00

  They were able to update the language model attention mechanisms and in that case, the autoregressive architecture was performing much better. So that's interesting. Perhaps unsurprisingly, if they increase the size of the vision transformer or they increase the size of the language model, both of those lead to a better vision language model. But if you had to pick one for a fixed parameter count, they say you get more bang for your buck by increasing the size of the language model component than you do increasing the size of the vision transformer. So there you go. One other finding, and we have kind of referenced this a lot. If you do add a perceivory sampler, which you can add in both architectures,

• 2:06:22Speaker 00

  something like that is necessary in the cross-attention architecture because you've got to fix the size of the lookup vectors that you're performing cross-attention on. But you can also have something like that in the auto-aggressive architecture. And the reason you might want to do that is if you want to reduce the number of vision tokens that were actually being unrolled in the decoder. So you could do it for both. And they obviously find that if you introduce a perceivory sampler, it does speed up training, but it doesn't necessarily have a positive impact on performance in the end. So that's interesting as well.

• 2:07:00Speaker 00

  And finally, they show that if you use the interleaved image-text documents, which the Flamingo team and other teams have found to be super important. They performed an ablation where they left those interleaved documents out, and they found it had a dramatic effect on performance of the model at the end. And that, in particular, adding interleaved image-text documents like this seems to really benefit few-shot learning at the end. Those are some of the conclusions from their Idefect series of papers. And they're not the only ones to have a look at this. There's a recent model called NVLM that came out from Nvidia a few months ago.

• 2:07:45Speaker 00

  And there, what they did exactly the same thing. So they trained three different variants of vision language models. Well, actually, two initially, two variants initially. They used a common backbone, so they used the language model from Quen. So that's the same one that's used in the QuenVL series. And the vision encoder, they used the Intern vision encoder. So we talked about that earlier. That's the vision encoder of 6 billion parameters, so entirely new, kind of trained from the ground up along with a decoder. So they trained this at a larger scale with a decoder transformer to see if they get better performance from the vision transformer. So those are the two components that are used by Nvidia. So.

• 2:08:30Speaker 00

  They denote the two architectures, D for the decoder-only version and X for the cross-attention version, and they compare them both. And they discover after training for a certain number of flops, that the decoder-only version has got the best multimodal understanding, the best reasoning, reasoning over images, and the best OCR performance. But they also note that the cross-attention version was much more efficient to train. The reason for that, and we've kind of mentioned this earlier, is because you have to unroll the full sequence of image tokens in your decoder and apply the attention mechanism across all of them if you're training auto-aggressively. So it's perhaps unsurprisingly that you get lowest training throughput. So that's interesting.

• 2:09:16Speaker 00

  They report that the perceiva resampler affects OCR performance. So negatively, negatively affects it. And the thinking here is the perceiva resampler, it's the way it does resampling, it's probably shuffling some of the information in the fine-grained kind of spatial dimension. And that does seem to be hurting tasks like OCR, which require a very kind of high-resolution view of the image. And that's their hypothesis to what's going on there.

• 2:09:48Speaker 00

  So I mentioned they did three architectures. And this is because the Nvidia team then looked at, well, what if we did a hybrid of the two? So what if we had cross-attention and self-attention? So the idea here is trying to make it more efficient to train the model. And that is by removing all of these kind of high-resolution image tokens from needing to be unrolled in the decoder. So all of the high-resolution image tokens are now going to be presented to the models through the cross-attention mechanism. But they take the thumbnail, so remember if we're dealing with high-resolution images, we typically tile the image. encode all the tiles to make the stream of tokens, but then also include a thumbnail,

• 2:10:30Speaker 00

  an overview of the image as well. So they do inject that into the decoder stream. So this is their hybrid architecture. So it gives the self-attention mechanism of the language model a copy of the image to kind of reason over. So remember this finding that the decoder-only architectures seem to have better multimodal understanding and reasoning. It seems to be something about co-locating the image tokens and the text tokens in the decoder stream that works really well. So still got that, but the decoder is then also able to look up via cross-attention the high-resolution tiles from the image when it needs to. So that's kind of, you might think of that as a compromise because it's

• 2:11:14Speaker 00

  clearly more efficient to train. And when I look through the results, you can see that the decoder-only version, it does beat the hybrid version on OCR tasks still, but the gap is smaller, as you might expect, than between the cross-attention and the decoder version. It beats it on chart understanding slightly. But what's really interesting is that the hybrid version actually beats both the decoder-only version and the cross-attention version on the MMMU validation set. Only slightly, but it does beat them both. So I think this is a really interesting kind of approach, this hybrid architecture. And it wouldn't surprise me if we saw this explored more in the future.

• 2:11:59Nathan Labenz?

  So is there any way in which it's like strictly better? I guess I'm understanding that my super-high-level summary of all of that would be, it seems like the decoder approach where the images go in right alongside the text at the beginning and get the full workup is the best performing. The cross-attention is a lot more efficient, but has some relative weaknesses and some that are like particularly idiosyncratic. But it's easier to do. A hybrid tries to get the best of both worlds, but if I'm like...

• 2:12:41Nathan Labenz?

  OpenAI or DeepMind and I'm like trying to make the best thing I can make.

• 2:12:48Nathan Labenz?

  Is there any argument or result here that would suggest that this hybrid approach has a claim on, you know, in any way being the best thing you could make? Or is it only appealing because it has the efficiency advantages?

• 2:13:05Speaker 00

  I would say it's too early to tell. So the decoder-only architecture that Nvidia put together, the D model, does win over the other two variants on, for example, chat understanding and OCR tasks. But on the validation split of the MMMU benchmark, the hybrid version beats both the cross-attention and the decoder-only version by about a percentage point, which is interesting, right? Now, there aren't any other hybrid models out there that I'm aware of. So it's unclear yet whether this is generally a better approach.

• 2:13:47Speaker 00

  But in this particular instance, on that one benchmark, that was their best performing model. And indeed, right now, if you look at the MMMU benchmark, you'll find that we've got the Intern VL model, which is the top open source model. We've got the Llama 3.2 vision model sitting right underneath it. And you'll find this on the MMMU leaderboard. You'll find the NVLM hybrid architecture sitting just below those two in terms of the open source models. So I think it's a really interesting direction. From my perspective, the jury's still out as to how else you could vary this architecture, what might be the best.

• 2:14:31Speaker 00

  So no convincing evidence that decoder-only is hands down the best way to go. I think this is an interesting data point suggesting maybe there's more to the story.

• 2:14:41Nathan Labenz?

  Yeah.

• 2:14:43Nathan Labenz?

  Interesting. It does. I'm always amazed by how simple the transformer is. You know, that's just a recurring point of amazement where I'm like the tangled mess that is my own brain and all the feedback loops and everything, but basically you can get as far as we've got with none of that. So it, I guess, you know, I on one hand have sort of a strong prior that that'll just keep being the best because it's been the best for a while. But then there's another part of me that's like, surely a more complicated architecture can work better, or we would have presumably evolved in a simpler direction ourselves. So yeah, I don't know which one of those should dominate, but presumably the more complicated, I mean, the more I think about it, it seems like

• 2:15:28Nathan Labenz?

  the more complicated architecture, just given how many possible versions of that there are, not to say this specific thing that has been tried here, but in general, it seems like more complicated architectures have to be better in some way, shape or form. You got to find them though, and you got to make them perform it on the compute. So those are huge advantages, obviously, but the current remarkably simple architectures have. But yeah, that's cool. One other thing I was laughing at, I think you noticed at the very beginning of this section was the paper from Hugging Face was called What Matters When Building Vision Language Models? And I would submit naming does matter somewhat. They came up with I D E F I C S one and two, as you had to spell it out. What is that? I defects?

• 2:16:12Nathan Labenz?

  I do whatever, whatever that is, it's not popping off the page to me. And I think it's gonna have a hard time in the jungle of models out there of standing out without a little bit of a catchier name. So that would be, as a reviewer who never actually read the full paper, I would say at the title, you know, there, there's a question posed that they could perhaps answered a little bit better with a better named model. But that's just a funny reality across all of AI right now. Everybody feel like the O three skipping O two is just in some sense, like totally insane. And in some sense is kind of perfect for the moment we're in. So they're certainly in, you know, good company for naming their models in a strange way.

• 2:16:58Speaker 00

  Yeah, I mean, there may also be trademarking issues with O two.

• 2:17:03Nathan Labenz?

  Yeah, I heard that. Who's got O2? I guess maybe they can't trademark it just because O2 is such a common thing.

• 2:17:11Speaker 00

  Called O2.

• 2:17:12Nathan Labenz?

  Oh, interesting.

• 2:17:13Speaker 00

  Which could be, be one of the reasons.

• 2:17:17Nathan Labenz?

  I was totally unaware of that. I was thinking that maybe because it's oxygen, you can't trademark something so commonplace. But I guess it maybe is running the other way. It is hilarious that they just introduced L1 and now we're on O3 and there is no O2. But yeah, somehow it does feel appropriate at the same time.

• 2:17:37Speaker 00

  Yeah. So there's two other things I wanted to pull out from the Nvidia paper in ascending order of interest for me. The first is a really interesting result halfway through where they looked at some of the leading open source multimodal language models, the VLMs really. One's a Lava model, one's an InternVL model. And they noticed that if you apply text only benchmarks, if you run text only benchmarks, I don't remember which benchmarks they actually ran against the models. But you, oh yeah, no, they ran MMLU, Math, Human eval, a couple of others.

• 2:18:16Speaker 00

  They found that in the VLM models, there had been a drop in text only benchmark performance as compared to the language model backbone originally. So for example, in the Lava one vision model, whichever language model decoder they used originally, they knew what the benchmarks were across all these scores. And then it just repeated it with the vision language model and they found a drop. And they found this drop consistently amongst all the ones they tested, except for the Llama 3 series. And if you'll recall, when we mentioned the Llama 3V series, they introduced all these cross attention layers and freshly trained them.

• 2:18:56Speaker 00

  But they froze the rest of the Llama language model. So they didn't suffer that degrading. However, to kind of counter this, the Nvidia team spent a long time building what they considered to be a really high quality text-only supervised fine-tuning dataset. And as a consequence of this, what they saw was an improvement on all of the language-only benchmarks from the NVLM series of models. That is, as compared to what the decoder they used was scoring before. So that's very interesting, right? An improvement in the text-only performance after vision language training.

• 2:19:37Speaker 00

  Possibly this could be down to just seeing the, you know, there's an additional large training data, pre-training dataset it's seen, which it hasn't seen before. But, you know, one does wonder whether there is something about the interleaving of the two different modalities now, which is somehow causing the transformer or the overall model to be able to reason better.

• 2:20:02Speaker 00

  So I think that's a very interesting finding. And it will kind of be backed up by the fact that they saw this improvement particularly in maths. And they note that in their multimodal fine-tuning dataset, they had an awful lot of kind of maths questions in there, like geometry questions, for example. And they think it improved the model's ability to do numerical and mathematical reasoning, even in text-only form, text-only benchmarks, even though this additional training data came in the form of images.

• 2:20:38Speaker 00

  So I think that's a very interesting finding.

• 2:20:40Nathan Labenz?

  Yeah, let me make sure I have that clear because there's, there's two things there. One is it usually, the text-only performance usually degrades if you just do image datasets without maintaining text-only data in the mix.

• 2:20:57Speaker 00

  Yeah.

• 2:20:57Nathan Labenz?

  So they did that. And then in this finding where they see an improvement in the text-only math encoding benchmarks, that is, that is the same thing where there are.

• 2:21:14Nathan Labenz?

  Continuing to have some text only data in that mix.

• 2:21:18Speaker 00

  Yeah, the way to think about this is if you perform supervised fine tuning provision language model, you'll see a degradation in text only tasks. And we know this from large language models themselves, you know, if you do an additional round of fine tuning on a particular task and you want to preserve the capabilities of other tasks, you have to mix in fine tuning data just to preserve that. They're not the first team, and this is not the first model where text only data has been included in the supervised fine tuning, but they seem to really go to town on this and draw it out in their paper as something they really paid attention to, having a large text only fine tuning data set. As we mentioned, the Llama 3D models saw no degradation, but then they froze

• 2:22:00Speaker 00

  the language model backbone. And the NBLM models actually saw an increase in their text only performance, and they saw this particularly on the maths data sets. And so their conclusion is the introduction of mathematical questions in image format has improved the model's mathematical reasoning overall. With a result back on text only maths questions, it now gets higher scores than it did before, than the language model backbone did before.

• 2:22:27Nathan Labenz?

  I kind of recalls the legendary room tweet of shape rotators and word cells. It's like the language model has, with the addition of the vision modality, has perhaps gained a shape rotator capability that it didn't have when it was text only.

• 2:22:45Speaker 00

  Yeah.

• 2:22:46Nathan Labenz?

  It certainly makes sense. I mean, if you had never seen a drawing of a triangle and only, you know, and I'm kind of imagining your SAT type math problems here. If you never saw that, you know, any of those diagrams and were forced to do it purely through text tokens, that would be weird, you know, there, I think there's a clear reason that we do that for ourselves. So it is interesting to see a similar.

• 2:23:11Nathan Labenz?

  Pattern popping up here.

• 2:23:13Speaker 00

  Yeah, it's it's perhaps not surprising. It's just really interesting to see it actually kind of verified in the research. And it brings me back to one of the questions that we posed at the very start, which was, you know, to what degree is multimodal understanding important for achieving the levels of intelligence, let's say AGI, that we want in the future? Well, here to me is like a small data point that is suggesting, look, there are real benefits to doing this. It doesn't say it's necessary. It does say it helps. So I think that's the interesting implication of the result.

• 2:23:48Nathan Labenz?

  Yeah. Yeah, that's definitely worth a little meditation. I mean, you can imagine that that can go a lot of different directions. That that seems like it will be a huge trend. And we're already seeing, of course, like more modalities beyond just image being added, video, audio, et cetera, et cetera. I've often wondered just how far that can go. Like, are we going to see natural language models that also get trained on like biological sequence data? Because that's a whole other track that I've been quite obsessed with recently, where there's been all this training on sequence data. And in a way, it's really cool that there's not natural language there, because I think it sheds really interesting

• 2:24:33Nathan Labenz?

  light on how things are working when it's like, it's picking this up from, you know, all the all the things that it's learning about proteins and so on. It's it's learning those from raw data in a way that is not mediated by human understanding in a lot of those scenarios. And I think that, for me, has been extremely clarifying when it comes to questions of can these things learn new concepts on their own that humans don't know? It's very hard to determine that through natural language models, because everything's kind of out there somewhere. And, you know, what's interpolation versus genuine out of distribution, whatever. But when you see these higher order concepts emerging from

• 2:25:17Nathan Labenz?

  pure DNA sequences or pure, you know, amino acid sequences, it seems like, okay, there's something really there that is pretty undeniable. And then I wonder, you know, does do all these things sort of converge? I mean, it seems like the global Maximum probably is a model that, and this may not be economical, economical, there may be all sorts of reasons why it doesn't happen in the immediate future. But it seems like the global maximum has got to be something that is just literally trained on everything, you know, and has the text and the image and all the way out to like weather data and just has this like very robust sort of all to all understanding.

• 2:25:59Nathan Labenz?

  And yeah, this is just like one, one little data point that suggests that that's true, but I feel like big picture, it's hard to see how that wouldn't ultimately be the case.

• 2:26:09Speaker 00

  Yeah, a part of the story, we kind of said at the beginning that, you know, vision language models are really interesting route into understanding how you actually exploit the relationships between two different modalities to improve reasoning. And some of the story we've told here is about what has been done over the last couple of years to get better and better at doing this. And I don't think we're at the end of the road yet. We've seen some interesting ideas so far, particularly this like this hybrid architecture from Nvidia, which suggests there may be more droplets you can squeeze from the lemon in terms of getting kind of more, more efficient cross transfer of information between the modalities.

• 2:26:50Speaker 00

  So really interesting, and it's been a tremendous amount of progress in two years in this one area.

• 2:26:57Nathan Labenz?

  Yeah, this cross attention thing would also be

• 2:27:01Nathan Labenz?

  probably the way that it happens if you were going to try to integrate a protein language model or whatever with a natural language model. It seems like you could do the end to end thing, but for starters, you would probably grab two that work and try to make them talk to each other. You know, somewhere in the middle layers where all the representations are already there and you have like a lot of confidence that you're working with.

• 2:27:32Nathan Labenz?

  You know, things that have their own understanding respectively and just trying to bridge those in a useful way, especially, you know, you can imagine too, as you have like lots of these modalities, it'd be an interesting question to, to try to figure out, like, even if the, and it's not even really that the, the end to end thing is strictly best, because the hybrid one is, is really competitive. But I can also see just lots of reasons that you might, for convenience, right? And we talked a little bit earlier about like, how to some degree, these architectures are legacies of team structures. And, you know, if you break that out now over lots of different modalities, and you've got like a whole, you know, different universe of people working on biological models, then,

• 2:28:15Nathan Labenz?

  you know, it might be really hard to redo everything from scratch, or get the data mix right, or all those sorts of things could be really hard. But if you have things that are working specialists across these domains, then I could see the cross attention approach being a really natural way to bridge them without having to kind of go back to the drawing board as much.

• 2:28:35Speaker 00

  Yeah. So a term we haven't introduced yet in this conversation, but which we could introduce now, is early fusion versus late fusion. And so what this is describing is, at what point in your information processing architecture, do the modalities come together? And in everything we've discussed so far, the answer is, we've got two separate encoders, right? We've got a vision transformer, and we've got, I guess, a tokenizer for the language model. And we're kind of, so we're, we're encoding the two modalities separately, and then we're fusing them in the architecture. But we'll discuss something in a second, where we're looking at early fusion, which is, okay, can we get one thing to encode them

• 2:29:20Speaker 00

  both, such that the modalities are aligned right at the start of their journey through the model? And I don't want to speak out of tanks. I'm not a neuroscientist, but my bet would be if you asked a neuroscientist, how does the brain work? The answer would be there are early fusion and late fusion and probably multiple integration points for different modalities of data through the information processing pathway. And it may well be that the situation that AI eventually finds itself in is that you similarly have these kind of multiple points of fusion. We've already seen with the N V L M hybrid architecture. You know, we've got two points of fusion there. We've got a cross cross model, cross attention look up. And we've got an image thumbnail being added to the decoder stream. So we've still already seen the first example of that.

• 2:30:06Speaker 00

  So one of the trends I would expect to see is that we find, you know, we have early and late fusion going on in future architectures. But I'm definitely not smart enough to say what they look like.

• 2:30:19Nathan Labenz?

  Yeah, when when people talk about early and late fusion, to what degree is this a statement about the pre-training process versus the architecture? I've always been a little bit confused about that because there's sort of,

• 2:30:37Nathan Labenz?

  you know, in the hybrid, the early fusion and the later fusion are still with like

• 2:30:45Nathan Labenz?

  separately trained, separately pre-trained modules, right, that are kind of learning each modality. And then it's like there's a question of do I want to inject that image data at the beginning of the transformer or do the cross attention thing in the middle? But then I don't know if it's like

• 2:31:05Nathan Labenz?

  feels like you could sort of make a two by two matrix of these perhaps where you could have like, you know, I don't know about the fourth box of that matrix, but joint pre-training is like another thing. And I'm kind of trying to untangle that concept from the early and late fusion.

• 2:31:25Speaker 00

  Yeah. So first off, I don't think there is a settled and canonical

• 2:31:31Speaker 00

  split in architectures where you can say, you know, late fusion means this, early fusion means this. Let's think of it more as a continuum. I would say everything we've looked at so far is late fusion. And that includes clip because we are encoding both modalities separately and then aligning them.

• 2:31:51Speaker 00

  And so for that reason, they are later than what you could consider very, very early fusion, which would be, we have one thing that encodes both our text and our vision at the same time. So we have one token space, if you like, right from the get go.

• 2:32:07Nathan Labenz?

  Right.

• 2:32:07Speaker 00

  And we will see an example of that before we finish today.

• 2:32:11Nathan Labenz?

  Okay, cool. Well, let's keep rolling.

• 2:32:14Speaker 00

  Yeah. So I just wanted to say a couple of words on this before we move on to a really interesting benchmark. On a couple of the benchmarks that are very important at the moment, that are also have been turned into or are being used, the training splits in these benchmarks are being used as part of the fine tuning data sets and lots of the models we see. And so the first one is the VQA benchmark. This is a reasonable sized data set. I think there's 250,000 images in it overall. I don't know what the train validation split is. But there were images taken from COCO, so there's common objects in context and possibly from other sources too.

• 2:32:58Speaker 00

  But the idea is that each image is associated with an open ended question about the image, which was generated by Amazon Mechanical Turk workers. And each question is supposed to require an understanding, both of the vision, the language and some common sense answers. And then you're given some, you know, if you're executing the benchmark, you're given multiple questions for image and multiple answer options per image. So they reckon there's about a million questions in the data set overall. This has become very important, as I said, as part of the fine tuning split. And so just to give you kind of put it in context, let's get an example from the VQA data set.

• 2:33:36Speaker 00

  So a photo of a woman who has got a pair of bananas grouping down from her upper lip like a moustache. And one of the questions is what color are her eyes? And so I guess the expectation is that the VLM or the vision model is going to lock on to the yellow in the middle of the screen. And answer yellow, which is not the right answer. But then it also asks, what is the moustache made of? So again, this is requiring it to know where a moustache sits on the face, what kind of shape a moustache has. And then what objects are performing that role in this image?

• 2:34:12Speaker 00

  So that's kind of an example from the VQA dataset. So common sense and reasoning over images, very different notes to what's in MMMU. So if one was looking for reasoning over images, as you, you know, that's the thing you wanted out of your VLM. The MMMU benchmark is great. If what you were looking for is kind of an understanding of common objects, their relationships, you know, what's going on in an image. The VQA benchmark might be what you're looking at, the visual question answering. There is also a kind of variant of this called the DOCVQA benchmark. Again, it's got a large training set, so it's often found in the fine tuning mix.

• 2:34:56Speaker 00

  That's about 50,000 questions that have been generated over about 12,000 images. And these have been extracted from a dataset described as industry documents. Okay, so this is things like PDF scans of documents from the workplaces, charts, graphs, like tables of data, invoices, business infographics and handwritten notes, that kind of thing. And the tasks in the benchmark are to isolate and report kind of precise spans of text from the images that answer a question. So for example, it could be, what is the number on this invoice? And then you followed by a PDF scan of an invoice. Okay.

• 2:35:37Speaker 00

  So this is an interesting benchmark and because this is the kind of thing that a lot of people want to use vision language models for. Right, is processing scans of documents. And if that's the use case that one really cares about, then performance on the DOCVQA benchmark is the one to look at.

• 2:35:57Speaker 00

  And just as a final word on fine tuning and instruction tuning datasets. Uh, the Hugging Face team of the unpronounceable I D F X model fame have bundled 50 of these fine tuning datasets up together.

• 2:36:11Nathan Labenz?

  What's better name this time?

• 2:36:12Speaker 00

  Yeah, they've called it the cauldron. Which is a great name. So the cauldron is obviously available on their platform, and it's probably the easiest way to acquire a good fine tuning data set. And the reason I mention this is because we've talked earlier many times about how augmentation of images in data sets has been really key for actually learning the alignment between the modalities. If I was starting with a task which was going to require a vision language model, and I was struggling a little bit to get, you know, the performance I wanted, one of the things I would do is look at the cauldron. I would find a task that seems to be similar to the one that I'm doing, and I would look at the augmentations and the prompt structure

• 2:36:57Speaker 00

  from that particular task data set. And I would ask, is there any way I can do augmentations on my own images? Or can I restructure my prompts such that it looks like this data set? Because the, whether it's just an inference time or whether you're going to build your own SFT data set, this would probably be the most informative and useful way to go about it. So that's a cheap code, in my view. Yeah, that's cool. It's a good, good tip. Shall we linger on the Blink benchmark for a minute? Because this is fascinating. So we talked about MMMU. I've just mentioned the VQA and DocVQA benchmarks, which I think those are super important.

• 2:37:41Speaker 00

  Blink is a really interesting one that was produced by a mixed academic and team from Alan AI earlier this year, July, I think. So Blink contains about just under 4,000 multiple choice questions, and it ranges over 14 common perceptual tasks. And these are tasks that they say humans can solve in a blink, but which should be difficult for VLMs. Across these 14 tasks, like human performance is in the kind of mid 90%. And because they're all multiple choice questions, random guessing gets you just over a third. Alright?

• 2:38:19Speaker 00

  The thinking behind the Blink benchmark is that lots of the questions on MMMU are actually about reasoning and the authors described them as they almost reduce to a dense captioning task, right? Can you can you just extract a dense description of what you see in the image here? Another way of saying that is if you were to replace the image with like a rich description of what's in the image, a language model should still be able to answer a lot of the questions. And indeed, when we mentioned MMMU, that was actually one of the baselines that the MMMU team created was to caption with a lava model. And then they saw that you could do much better than random guessing, just from the captioning.

• 2:39:04Speaker 00

  So the interpretation of this that the Blink team made is that a lot of what MMMU is testing is reasoning, okay? And that less emphasis has been placed on the classic visual perception capabilities. And one piece of evidence for this could be that if you look at the jump in MMMU performance between GPT-4O and GPT-401, there's been a huge jump. And if that's all attributable to reasoning, it's suggesting that, you know, that is a lot of what the benchmark is measuring. So probably should talk a little bit about or like introduce some of these categories, you know, what what do the Blink authors think is meant by perceptual reasoning? So I'll just cover a few of the categories.

• 2:39:48Speaker 00

  First, a couple that the VLM seems to do really well on, and then a few where they seem to do really poorly. Okay? With me so far?

• 2:40:00Nathan Labenz?

  Yeah, I like this because this definitely calls to mind some of the challenges that we've had with aesthetics, which I mentioned earlier as well. But, you know, again, in the Waymark context, it's not enough just to know the content of an image. We want to make our users look good and

• 2:40:20Nathan Labenz?

  The early versions of this were basically, first, they were not dense captions. They were very sort of not very informative captions. You know, you'd get like in the earliest version that there was once upon a time when the Microsoft API was the best. And then Blip became the best for us for captioning. But it was still, you know, this is two years ago, we're back to the beginning of the outline. You know, so sparse, then we got these denser captioners. Those were much better for at least getting the relevant images content wise. But still no signal for a while on what actually looks good. And that has definitely notably changed.

• 2:41:02Nathan Labenz?

  And I think these subtasks within this blink dataset that you're going to take us through are a really interesting way to interrogate how exactly they have changed and kind of reverse engineered to some extent, too. Like what the developers have done to add these capabilities when you look at these challenges. So, yeah, I think it's a

• 2:41:25Nathan Labenz?

  very interesting kind of look behind the curtain.

• 2:41:27Speaker 00

  Yeah. Well, if we peel back the curtain then, on the best solved task, blink task is what they call the art art style. Okay.

• 2:41:38Speaker 00

  At the time of recording, the best data that I can find is that the model that solves this the best is GPT four O. Okay. We don't have, I think, four O one results in blink. Neither do we have the latest Gemini or Sonnet, I believe. So take this with a little pinch of salt when I say what was the best, but it gives you an understanding of, you know, as of June this year, what did the rankings look like? So the art task, the idea is you've got three paintings, for example, right? But you've got ones from a prompt, which is a sketch. And then you've got maybe two others from two other art styles.

• 2:42:17Speaker 00

  And the question is, which of the two, numbers two and three, seem to match the image in the prompt in terms of visual style? Okay. And for a human, this is very easy. Like one can look at this and say, oh, this is you input a sketch. There seems to be some kind of Renaissance painting, the second one. And then the third one, I mean, I'm not quite sure what the style is, but it looks a bit like a sketch to me. So I'm going to say it's the second one. And I know nothing about art and yet I can solve that very quickly. Foro does really well on this, gets about eighty-three percent and is the top scorer model. Humans are at ninety-five percent and random guessing would give you fifty. Okay, so art style seems to be sold, I would say reasonably well, by the current generation of of VLMs.

• 2:43:03Speaker 00

  Again, another similar one is is what they call visual similarity, which is actually sold rather well. So here you might get a series of two photographs and then a reference photo. And the question is, you know, which of the two photos is most similar? And in the paper, they show like two waterfalls and then another waterfall. And one of them, you know, one of the reference image is taken from the same perspective of one of the images. And the other image of waterfall is taken from a different perspective. So I immediately pair the correct one in my head. Humans indeed get like close to ninety-seven percent on this. And here, GPT-4 Turbo was the winning model at eighty percent. So those are two of the well solved ones. What I think is really interesting is the worst solved blink tasks.

• 2:43:48Speaker 00

  So of these fourteen categories, which ones seem to have the biggest gap at the moment? One of those is the IQ test. So if you've seen the kind of images you get in IQ test, you know, the the example given in in the blink paper is one where you've got a simple diagram, you know, with a sequence of shapes in it. And then you're being asked which sequence uh seems to match this, you know. So I've got three, three in a sequence and then can you complete the next sequence? Uh, pick an image of these four, which seems to complete it in the same way the first sequence is completed. And so that's if you've seen IQ tests, you've seen lots of variants in this before. Humans get this right roughly four fifths of the time.

• 2:44:30Speaker 00

  Okay, random guessing in this particular area, the benchmark is four options, so twenty-five percent. And GPT-4 Turbo was the best performing model with thirty-two point six seven percent. So not a lot greater than random guessing.

• 2:44:46Speaker 00

  When I look at this blink task, I am reminded a lot of the R-KGI questions. And I don't know if that's a lazy analogy or an ill-informed analogy on my part. But I seem to solve R-KGI challenges in much the same way I solve these IQ tests. Am I doing, you know, guided program search based on perceptual priors? Because that kind of makes sense as an explanation as to why I can do it in a blink and the language model is struggling.

• 2:45:16Speaker 00

  So I think it's a really interesting one. And I think, you know, my interpretation is that this really backs up a lot of what Francois Charlet says in how he designed the RKGI benchmark.

• 2:45:31Speaker 00

  Why he thinks it's difficult.

• 2:45:35Nathan Labenz?

  Yeah, maybe just two interjections from my experience. One, on the even the well solved tasks, I would say if you're actually going to do something like this in an app, the way you ask definitely still matters if only for avoiding unnecessary refusals. Like I found that GPT four O and Claude, you know, through five, probably similarly, really doesn't want to tell you you're ugly. So and that's, you know, presumably a reflection of its reinforcement, learning from human feedback and it's, you know, Claude's virtue ethics, right? It's it's trying to be a good friend to you. So it doesn't, it sugarcoats sometimes. If you

• 2:46:17Nathan Labenz?

  say like, you know, is this image an attractive image? For example, it will sort of often hedge and be like, well, it's sort of in the eye of the beholder. So there's a couple of different ways we've found to get around that. Kind of similar in some ways to like, you know, if you want a model to talk to you about a medical question and you're like, you know, getting the sort of, I'm not your doctor. I always get around that by saying, I'm preparing for a conversation with my doctor and I want to be as informed as possible. And then it will basically let its guard down and help you. And in these like image aesthetics questions, we found if we just say the tersest, you know, rate this image one to five beauty or whatever, it will sometimes balk at it. But if we say, is this an image that a small

• 2:47:02Nathan Labenz?

  business would be proud to put forward in their marketing, then you are much better able to get a result. And I've also seen kind of similar things with even just pairwise type thing. So, you know, took a couple selfies of myself and my kid. And then, you know, one of them I made the kind of a whatever contorted face that was like clearly not a great picture. The other one was much more normal. And Just saying like, is this a good picture? It really doesn't want to shoot you super straight. But if I put both in and I said, which of these should I send to my wife? Then it will say, you know, I think you're better off with a second one or whatever. So play around with that stuff. If you are trying to get it to work, like everything else, you know, prompt engineering is definitely

• 2:47:44Nathan Labenz?

  still a bit of a thing, especially if you're kind of asking it to be a judge. You know, it's comfortable judging in some ways and it's really not comfortable judging in other ways, not to anthropomorphize it too much, but that intuition definitely can be helpful. On the R K G I thing, I was also just really struck, you know, back when that was dominating the discourse not too long ago, by how weak the, even the best models were when it came to simply describing the ARC images. So just took a few screenshots right off their website. But then I didn't ask it to solve the problem. My first thing was just, can you see what this is? Can you describe it, you know? And it was very not good at that, actually. Yeah.

• 2:48:29Nathan Labenz?

  Not good at even the most basic stuff of like, how many squares? You know, what is the dimension of the grid? I was really amazed by how weak that was. And, you know, presumably that just reflects kind of a lack of that sort of thing in the training data. But it's still pretty surprising because it's like pretty good at reading tables and, you know, they can do OCR reasonably well. And you would think that it would be able to like count the squares. But that was really weak. It was interesting to see too, just in the last couple of days with the O three results that as far as I know, they weren't doing any image and literally just presenting,

• 2:49:13Nathan Labenz?

  which is, you know, the actual underlying data beneath ARC is literally just arrays of arrays, right? And they literally just give you numbers and all of the sort of color representations that we've seen. On top of that are basically a gloss for humans to make it easier for us. Cause we're obviously good at colors and stuff and recognizing these shapes when they're like in contrasting colors. That plays to our strengths. You don't have to do that, you know, for the ARC thing. And it turns out that O three seems to not be doing that and is literally just reasoning over text tokens. Yeah.

• 2:49:48Speaker 00

  Yeah.

• 2:49:49Nathan Labenz?

  What's your theory? Would you give any other sort of deeper theory on why these IQ tests are so poorly solved?

• 2:50:01Speaker 00

  Uh, I am not smart enough to be a theoretician, but I've just noticed, like, to me, it just, it reminds me-

• 2:50:07Nathan Labenz?

  Speculation. Let's call it a speculation.

• 2:50:08Speaker 00

  Yeah, it reminds me so much-

• 2:50:09Nathan Labenz?

  You're smart enough to be a speculator.

• 2:50:11Speaker 00

  It reminds me so much of R K G I. The whole point behind the blink benchmark is that humans can solve this, like, in a snap, right? So there's something about the kind of the perceptual priors or the perceptual features that we extract as a natural part of looking at the image that means we can very quickly answer these IQ test questions. And it seems that, like, understanding of images in V L M's just doesn't work the same way. So maybe it hasn't got the same kind of perceptual scaffolding that we do. And therefore, I can answer it very, very quickly because I don't need to do an exhaustive

• 2:50:53Speaker 00

  program search. I've got some priors which can help me zoom in on the correct rotation, if you like, of the image in the IQ challenge, therefore select the right answer. Otherwise, I have to do some kind of exhaustive reasoning over the possibilities, as, you know, Shallow would say, a program search. So, you know, the job of priors in this case is to constrain your search space, right? So you zoom in on the correct answer very, very quickly. So my interpretation would be, yeah, that they're probably not understanding the kind of perceptual elements of the image in the same way that you or I would. And that doesn't mean that you can't do very well on a benchmark

• 2:51:32Speaker 00

  like this just by adding reasoning. And my guess would be that, you know, four oh, four oh three, given a lot of compute time, would do a lot better on this than four oh, which is what has been measured. And simply because it can reason over the different possibilities. That would be

• 2:51:52Speaker 00

  But then again, you know, whereas with the RKGI benchmark, you're presented with essentially a matrix, you know, an array. That's the input actually comes in as an array. And so you can kind of reason over that as a series of text tokens. Whereas the VLM has got to process the image in the blink challenge. And so it's not quite clear that you get such a clean decomposition of the IQ test image into tokens, which you can manipulate through reasoning in the same way. But I'm guess going to guess that reasoning plays a role in solving this and that you can essentially like brute force your way through problems in the blink data set, perhaps by adding reasoning, but it just doesn't feel like the most efficient way to do it.

• 2:52:36Speaker 00

  And I think that's probably the point that the authors are trying to demonstrate.

• 2:52:41Nathan Labenz?

  Yeah, I mean, you're not given any, in this case, you're given only the image, right? So that's a notable difference from the RKGI where you are given something that is text token representable and the option, you have the option to, you know, use images too, but you don't have to do that here. You have to confront the image and make sense of it. Is there anything in this recent line of work? I mean, again, I don't know a lot about the human visual system and I also don't know a ton about old convolutional networks, but my general sense of the human system is that

• 2:53:23Nathan Labenz?

  we sort of gradually, as the information is being more and more processed through layers of the brain, that we're going from like very simple, you know, edge detection type things and angle detection and all these sort of like features, basically, up to more and more semantic features as we get to higher levels. And that definitely has been demonstrated in certain vision systems in the past.

• 2:53:54Speaker 00

  They're labeled, yeah, that way in the human brain.

• 2:53:59Nathan Labenz?

  Do you think that there's something about this, this sort of vision transformer that's like not doing that? I haven't seen, seems like that work was a little older and the, you know, obviously the paradigm has shifted. And convolutional networks also can have like much more engineered features where you can put these kind of specific priors into the structure itself of the convolutional,

• 2:54:26Nathan Labenz?

  whatever, processing for, for lack of the right precise term.

• 2:54:31Speaker 00

  That's absolutely right. And it was pointed out in fact by the IDFX team that, you know, much more work has been put into looking at language models than has been put into looking at the actual architecture of vision transformers. And you're actually right. And one of the, one of the things about a vision transformer is that those inductive priors that we use to code into CNNs are missing. And in fact, that's why at smaller scales, probably, CNNs actually do better. And this was when we first introduced the vision transformer. That was one of the findings from the Google team originally. Transformer takes over, you know, as the scale increases, but it's smaller scales of size of model and smaller data sets.

• 2:55:17Speaker 00

  The CNNs do rather better. So there is an open question, I think, as to what should we be looking at in terms of the architecture of vision transformers itself? Another observation that has been made, and I can't recall where it's from, is that a lot of vision transformers are now trained using these contrastive learning objectives. And could this be weaker than perhaps some other way of training? Because, you know, when we train a transformer, we do it a different way. We have a generative pre-training recipe, which is extremely effective. So could there be such a thing for a vision transformer? And that's such a great question that we might actually answer it in 10 minutes' time.

• 2:55:59Nathan Labenz?

  Can't wait. Let's keep going.

• 2:56:02Speaker 00

  So I'll just touch on one other example of, you know, a task that's very poorly solved by current generation of vision language models in the Blink benchmark. And that's called the relative reflectance task. So here, the idea is that you give the VLM a picture and you put a couple of markers on the picture for two different areas and you ask which point has the darkest surface color or are the colors about the same? So you always get three answers. So random guessing would get you thirty-three percent. So in the paper, they show an image of a hotel bedroom and it's got the kind of cream colored headboard for the bed and a white wall behind it. But because of the impact of the light shining in through the window, you know, the pixels from the cream colored headboard are actually

• 2:56:46Speaker 00

  a bit lighter than the ones on the wall. However, just looking at the image, I can tell you that the headboard is cream colored and the wall is going to be white. And so I know the headboard is darker. And I can answer that in a blink, as they say, because my brain is doing some adaption to images and is accounting for the effect that the light is going to have and where it's coming from and where it's shining. So these are some of the perceptual priors I'm bringing to the problem. And it turns out that the VLMs have a really hard time solving this. Humans are kind of ninety-five percent of the right answers and the VLMs, the top forming one was a lava model at the time they tested, and it was doing a shade under forty percent on this benchmark. So just another kind of example that they might not be doing things the same way we are.

• 2:57:31Speaker 00

  I also think that we should point out perhaps the most interesting observation from the Blink paper in my book. And I don't know if explicitly mention this, but if you just look at the performance tables of the different models they tested, you can see they tested GPT-4V and Turbo and 4O. And you can see that 4O has improved in a number of the tasks, but it's actually regressed in a number of other tasks, which is really interesting. So for example, there's been a significant regression in counting. 4V was solving the counting tasks at about sixty percent. 4O, they measure at forty-nine percent. So it regressed in its ability to count objects in the images.

• 2:58:17Speaker 00

  It's really not clear like why this is. Is this an artifact of distillation? Is this an artifact of like fine tuning? Not sure.

• 2:58:28Speaker 00

  But it's a result.

• 2:58:32Nathan Labenz?

  Yeah, on this, I can only say these things are weird beasts and find similar things in other areas too. I mean, not so much on like major benchmarks. I don't know if this would necessarily count as a major benchmark. It's not usually, you know, one of the ones that they would report in kind of the headline table. So they don't let those typically degrade, but certainly every time there's a new model, somebody's got to complain with it, right? With the super wide range of use cases that are out there. Yeah, I wonder if they're even measuring this kind of thing. I mean, they measure a lot, but are they specifically like looking at the Jigsaw score on the blink? And maybe, maybe almost by definition not, right? Because this paper came after some of these models, right? So.

• 2:59:17Speaker 00

  Yeah, June, I think, June, I think this year.

• 2:59:20Nathan Labenz?

  Yeah, interesting.

• 2:59:22Speaker 00

  But counting is a really interesting one because on, on the surface, it feels like a really simple task to count the number of objects in an image.

• 2:59:29Nathan Labenz?

  Yeah.

• 2:59:30Speaker 00

  And, you know, across the blink counting task, it's one of the kind of, it's in the middle of the pack in terms of how well it's solved. And they measured the Lava 1.6 model, 34 billion parameter variant, leading the pack of models with 66% success rate. Human score is like near 98%. And generally, just a four choice question. So the random choice baseline is 25%. And I actually found another data point, a paper called the Effectiveness Assessment of Recent Large Vision Language Models, again, from June this year, which was finding that a number of open source vision language models, including the latest

• 3:00:12Speaker 00

  Lava at that time, 1.5, were outperforming GPT-4V on counting tasks. So there's this kind of a second data when they weren't using the blink data set for this, they had another kind of methodology for this. Very interesting to ask like why? And there's a team from DeepMind did a bit of work earlier this year. They were actually looking at getting diffusion models to produce the correct number of instances of an object in a picture. But as part of doing this, they did a scan across some of the commonly used image caption pre-training data sets. And they found that there are some captions in there which do denote the numbers of objects in the images, but they said they're very, very scarce.

• 3:00:53Speaker 00

  And actually, possibly not enough to learn how to bind the number correctly to the appropriate features they're extracting from the image. And this might be one of the problems, just the scarcity of this kind of task in the pre-training data set. So it does make me wonder that, you know, a lot of times earlier we've seen the story of improving the ability of tasks, task-specific capability of a vision language model has been a consequence of augmenting a data set, you know, in order to be able to train it at modest scale on that particular task. And maybe that's the way forwards with things like counting. You know, we just, we just need an augmented data set

• 3:01:36Speaker 00

  to be added into the kind of the mix here, and that might improve things. But you know, not all is lost if you're trying to train a VLM to do counting. If you look at the kind of Anthropic's notebook collection of kind of best practices for Claude, they show you how to use good prompting techniques in order to make the Claude think through the counting task and decompose the image and then ask what it can see. And that does work a lot better. But again, it's showing you that the fact that, you know, reasoning can compensate for some of these perceptual deficiencies. But the perceptual deficiencies are there. Yeah, okay, cool. So a couple of minutes ago, you were asking about, you know, could we do, you know, is the vision transformer part of the problem here in, in

• 3:02:21Speaker 00

  learning this kind of rich representation of the images? And you asked about, you know, could, has jettisoning the inductive prize that we brought to CNNs been, you know, a problem here?

• 3:02:36Speaker 00

  And I said, well, you know, the other thing that I've heard people mention or read experts mention is that it's the contrastive learning objective that you train a vision transformer with that might be part of the problem. And actually, a team at Apple earlier this year took a look at this very question. And they said, can we do multimodal pre-training of a vision encoder in a different way, i.e. not using a contrastive learning objective?

• 3:03:04Speaker 00

  So what they did is they took a vanilla vision transformer, okay? And they asked if they can change the recipe for pre-training it such that they use a kind of a generative training objective. So I'll explain the setup and then we'll see how it works, how they do it. So you start with a vision transformer and a transformer decoder, both trained from scratch. And it's trained on a large mix of image caption pairs. The captioning is a mixture of old text, the scrapes from the web, and synthetic captions, so for example, generated by a Lava model, something similar.

• 3:03:46Speaker 00

  So the data is prepared by taking image patch tokens, so these are small snippets from the image, and text tokens, and always presenting them in, so you've got your image tokens first and then your text tokens, okay? Simple tokenizer in place. Training is done using prefix attention. So prefix attention is where you basically randomly mask a subset of the visual tokens, which are all appearing first. It could be all of them are masked, or it could be, you know, a small number of them are masked. And then the decoder has to generate the rest of the sequence, which will include generating the missing visual tokens, remember these are soft tokens, they're not from a

• 3:04:31Speaker 00

  codebook, followed by all the text tokens. And at this point, it's using kind of standard left-to-right attention masking, okay?

• 3:04:41Speaker 00

  The loss is only calculated though over the non prefix tokens. Because obviously you're going to feed in the masked ones first. So it's going to compute the loss over the the production of visual tokens full of by text tokens. The idea here is you're doing generative pre-training. So you're asking the decoder to generate the missing image tokens. But you can't use the same loss function for text and image. Okay, because all the text tokens come from a code book. So you can do a standard cross entropy loss for the text tokens. For the image tokens, they very simply just use a mean squared error loss. So the decoder generates an image token. It's seen, you know, some fraction of the image already.

• 3:05:25Speaker 00

  It generates what it thinks is the next one for the next patch in the sequence. And they simply compare it to the real, you know, the real token and then just compute the MSE loss. Okay. So that's a decoder only training, a generative training recipe for the vision transformer. And they're basically training the decoder and the VIT from scratch in one step using this pair of objectives. Recall that the idea here is to train a new like vision transformer. And so what they demonstrate in the paper is that they can jettison the transformer decoder that they've trained as part of this recipe. And then connect their new vision transformer trained in this way to, and they, I think they used a llama three model

• 3:06:09Speaker 00

  to create a vision language model. They connect the two in an also aggressive fashion using a simple MLP. And then they train it on the lava supervised fine tuning mixture, which we talked about earlier. And this means they can do a nice ablation. They can compare the the power of the vision language model with a vision transformer created using this generative pre-training recipe with kind of a drop in, for example, a vision transformer of the same size, but trained on contrasted learning objective. And what they find is that they see improvements in all of the VLM benchmarks that they tested against. But particularly in captioning and in visual question answering.

• 3:06:50Speaker 00

  So those are the two they see very significant improvements in. So this is really interesting. You know, one of the questions was, is it the contrastive learning objective that is limiting the the power of the vision transformer element of the recipe? Answer seems to be yes. Switching to the well proven kind of generative pre-training recipe that we know works in language models, works really well for the vision transformer as well. What I would be really interested to see is whether this AIMV two vision encoder from Apple being injected into a vision language model has an effect on blink performance. I don't know if it will or won't, but I think it'd be really interesting to know if it did.

• 3:07:34Speaker 00

  I think what we're going to see is more experimentation in the next year or two in this space. I think we're going to see the vision transformer come under scrutiny and we're going to see more really smart ways of trying to adapt and enhance it, maybe even revisit the way it works, reintroduce some of the inductive priors that we lost.

• 3:07:53Nathan Labenz?

  Yeah, so can we just linger for a second more on the difference between the contrast of training and this generative training?

• 3:08:02Speaker 00

  Hmm.

• 3:08:05Nathan Labenz?

  I think I got it, but give it to me one more time and make sure I, uh, because this seems like a pretty important.

• 3:08:11Speaker 00

  Yeah.

• 3:08:12Nathan Labenz?

  Conceptual distinction.

• 3:08:13Speaker 00

  Training on a contrastive objective, which pretty much everything we've looked at before now, apart from the very original vision transformer, has been done with, is you want to encode your image, you encode your caption or your paired text. Okay, you've got two encoded vectors at this point. And in a contrastive learning setup, you look across your batch and you say, I've got this one true pair within my batch and I want the cosine similarity of the two embeddings to be high. And I want the cosine similarity of all the non-true pairs in that batch to be low. And the contrastive learning objective kind of forces the embedding space, you know, distorts it, if you like, such that you get that result.

• 3:08:58Speaker 00

  So you'll always get, you know, an image paired closely in the embedding space with its relevant caption with similar text.

• 3:09:05Speaker 00

  For this objective, we're using a generative pre-training objective here. And so what's happening instead is we're pre-pending each example with a load of visual tokens. We're masking a bunch of them, okay? And then we're saying, so in other words, we're not computing the loss amongst the first, let's say, three quarters division tokens. And then we're simply asking a decoder to decode the rest of the visual tokens followed by the text tokens. Okay, we're jointly training the vision transformer and the decoder at the same time. And we're just measuring its success in reconstructing the visual tokens by the MSC between what the true token was, next token, and what it predicted.

• 3:09:50Speaker 00

  And then we're evaluating its performance on text in the same way. So what it should be learning to do is to attend. So the caption, the text that appears afterwards, it should be learning to attend to the vision, the image earlier. Because it's using this decoding strategy. But what they found was if you put all of the image tokens first and then the text, a lot more of the onus is on trying to learn the remaining unmasked visual tokens, tokens from the image. And this appears to make a much stronger vision encoder than doing it the other way around.

• 3:10:26Nathan Labenz?

  Now, that original vision transformer, it was just trained purely on images and that was just filling in masked tokens as well?

• 3:10:37Speaker 00

  Yeah, the very first vision transformer was actually trained as a classifier.

• 3:10:42Nathan Labenz?

  Oh, classifier, that's right.

• 3:10:43Speaker 00

  And I believe on ImageNet. And so, again, the output sequence contains a classification token. And they simply took that and then with a simple linear projection, we're training it to predict which of one of a large number of categories the image belonged to. That was how the very, the very first vision transformer was trained. But all of the language, all the VLMs we've looked at have basically used a contrastively trained vision transformer. And we talked about how that was done when we looked at Clip.

• 3:11:15Nathan Labenz?

  Yeah, yeah. So with this, with this Apple approach, would you call this early fusion or late fusion? Because I'm looking at the diagram in the paper and it's like, there is still this sort of cross entropy loss. And there's like a different encoder for the vision component. So if you think of the fusion question as being like where in the architecture it happens, it seems more late. But given that they're all being pre-trained, you know, together at the same time, in some sense, that seems like quite early.

• 3:11:48Speaker 00

  Yeah, I, I would agree with you. It seems like a bit of both, doesn't it?

• 3:11:53Speaker 00

  I'm going to call it late because there's still a vision transformer that's separate from the language model it's connected to. But the alignment between the two has happened a lot earlier.

• 3:12:03Nathan Labenz?

  Yeah, okay. So yeah. I like it. I don't think I have anything else on the Apple one. And I see we've got the, I wouldn't call it the final frontier, but the next frontier is next. I think these, these papers are really interesting too. So yeah.

• 3:12:18Speaker 00

  I was wondering about just skipping over the chameleon one and just talked about the transfusion paper.

• 3:12:27Speaker 00

  And the reason I was thinking that is because I think

• 3:12:31Speaker 00

  transfusion is just so much better, like in terms of not only its capabilities, but also its training efficiency. It feels like this is just great. And it's the way that DeepSeek built their model as well. I feel like this is the recipe. But um.

• 3:12:49Nathan Labenz?

  I trust your judgment, whatever you think best.

• 3:12:52Speaker 00

  Yeah, so and then we just, we finish up with just like looking at the frontier labs offerings, we'll just go through that. Sounds good. I'll try not to derail us too much. No, don't worry, we've, we've, yeah, we've got plenty of time in hand. Okay. So we've mainly focused today on vision language models, which have been looking at understanding images. And to some extent, video, we haven't really mentioned video today, but a lot of the models that we've described can handle videos because a sequence of frames from a video is not so different to a sequence of pictures. And so, you know, a lot of what you get for, you get for free, basically.

• 3:13:37Speaker 00

  There's a bit of John's video, whether you consider it a separate modality or not, it's kind of up to you. There are some video specific benchmarks. There's a video MME benchmark that you can look at, see how well the different models handle sequences of videos. But what we haven't talked about is, you know, what's, what's been missing for everything so far is the ability to generate images. We've really looked at image understanding. Obviously, the simplest way to have a V L M or an A I in general, generate an image is to have it generate a prompt and then to hand that prompt off to a diffusion model and have the diffusion model generate the image. And indeed, if we look at the frontier labs today and the services they offer, this is what Gemini and G P T four O are doing

• 3:14:22Speaker 00

  with Image N three and Dali respectively. And we just put behind that in parentheses. Actually, it seems from the original G P T four O announcement earlier this year that it is a true multimodal model and it can directly generate image outputs. I mean, the O stands for Omni in it, so it can generate other modalities too. But this capability is not yet being released. It's been promised but not released due to safety and infrastructural concerns, I think. It's what OpenAI said. But it's interesting nonetheless, and the question is, how might this be working? And there's actually been quite a lot of work come out of, like notably, uh, FAIR, Meta,

• 3:15:05Speaker 00

  over the last year or two, looking at kind of true multimodal, multimodal models. And if you're interested, kind of the sequence of papers to look at is CM3, which is, they called a causal masked multimodal model of the internet. And this was followed by a model called CM3 Leon, which is pronounced chameleon, uh, which was then helpfully followed by a model called chameleon, spelled chameleon, not the same thing. But this was all part of Meta's exploration of kind of true multimodal models that can both input images and text and generate them both. But I thought the one I wanted to focus on today was, was the latest in the sequence from Meta, and it's called Transfusion.

• 3:15:50Speaker 00

  And in the Transfusion recipe, they're actually looking at going beyond simply just images and text, but it introduces what I think is the most promising recipe for multimodal generation as well as multimodal understanding. Okay, so if we focus on the image and text part of this here. So the approach is to pre-train a single transformer on an even mix of image and text data, okay? But they're going to use during pre-training a different training objective for each of them. Okay, so

• 3:16:31Speaker 00

  previous papers in kind of this series, or previous models in this series from Meta, when attempting this, had actually quantized the image tokens before they entered into the model. Now, it's probably just worth saying very quickly, what does that mean? How does it work? So we've talked before about how, you know, text tokens obviously can be looked up backwards in the codebook. Okay, so they're, they're quantized, right? But we've used the term image token very loosely because we actually know they exist on a continuum, right? But they don't have to. So one way to quantize image tokens is using a vector quantization, um, method.

• 3:17:16Speaker 00

  If you come across the VQGAN architecture, this is like one way it's done. So the idea is you take an image, you pass it through a variational kind of autoencoder layer to get some latent vectors. And you then have like a learnable codebook. So you can imagine this is if you've ever studied how K-means clustering or something works, you're kind of learning where your cluster of centroids are in your vector space as part of your training recipe. Okay, so the idea is you then quantize to the closest of your quantized vectors in this kind of latent space. Maybe you've got 10,000 of them. So you encode your image token, quantize it, and then you need decoder again,

• 3:17:59Speaker 00

  using the variational autoencoder decoder part. And now you just perform a number of metrics to see, okay, I've decoded my image again, having encoded, quantized it, decoded it. And now I'm going to compare it to the original image. And then run it just a number of metrics, you know, calculate a number of losses over that to see how close I was to my original image. And I keep changing my codebook, you know, my learnable codebook, until I'm getting images that look very close to what I put in. And now I've got a codebook that I can use to quantize any image tokens that I see. So that's kind of how they were doing it previously. And they actually jettisoned this approach completely for Transfusion. And they said, you know, we're not actually going to quantize any of our image tokens at all.

• 3:18:43Speaker 00

  We're simply going to pass them through a variational autoencoder encoder part. And this kind of turns them into like latent patches, if you like. They're going to then pass it through either a multi-layer perceptron or through a U-Net down sampling block. This gets them their latent vectors. And they're simply going to use those as tokens and inject them into their transformer. Okay, so they're kind of continuous. And then the text tokens are going to be handled in the usual way. So they're going to be turned through a tokenizer into their vectors. And now they're going to feed this stuff into the transformer. And the idea is that you're training using a next token prediction. But you're going to do different losses, okay, different losses for text.

• 3:19:28Speaker 00

  So text region is kind of easy. We're going to handle this with a simple linear layer and use a cross entree. The same way we always train with a transformer. But when we detect that we're outputting image tokens, we're going to process it through the corresponding U net up path. And then through the variational auto coded decoder to actually start generating an image. And then we're going to use the diffusion loss objective there to actually train, train that part. So we've got two different things going on here. And transfusion really is, if you actually look at the architecture diagram, it is like a latent diffusion model that's been pulled in half. It's got this kind of transformer in the middle.

• 3:20:12Speaker 00

  And then you've got both sides of, if you look at a stable diffusion architecture, for example, it's like that. Pulled apart with a transformer stuck in the middle with the text being handled in a new way.

• 3:20:24Speaker 00

  The very important thing here is, is how you do the attention masking when you train it. So the training objective, they use a causal attention masking for all the text. So if you're decoding a text token, doing the next token prediction, you can look at everything to the left. For images, if you're generating image token, you've got bidirectional attention. So you can look at all of the other tokens in the image when you're reconstructing one. This means every patch can attend to every other patch within the same image. But you can only attend to text or patches of other images that appeared previously in the sequence. Does that make sense?

• 3:21:03Nathan Labenz?

  I think so.

• 3:21:05Speaker 00

  So more complicated masking to set up. Look, obviously more complicated decoding regime. You need to know whether you're decoding text or you're decoding images. You've got the kind of U net and VAE structure around it. You've properly here got an amalgamation of the two architectures. It sounds very complicated, like very complicated to set up, very complicated to make work. But what they discovered when reporting transfusion results is really interesting. Compared to the previous way of doing it, so actually quantizing all of your image tokens in their previous chameleon series. They say that they're producing images of a similar quality.

• 3:21:47Speaker 00

  After only training for about a third as many flops, which is like truly impressive. So that really suggests they're on to something here.

• 3:21:55Speaker 00

  In the previous series of papers, the Chameleon Papers, they described having real difficulty actually getting the pre-training to work stably. And they had to induce a number of kind of what they called architectural innovations, which feel like kind of sticking plasters and changes to get it to train properly. Nothing like that was reported in the Transfusion paper, suggesting they had a kind of a smoother time of it.

• 3:22:18Speaker 00

  And then very interestingly, they found that on just text to text tasks, this new recipe was matching kind of the training losses that they saw in their previous series, the Chameleon series, at half the flops. So they were getting not only better images being produced, but also better text. So it sounds like it's a really efficient recipe, just rather complicated to set up.

• 3:22:40Speaker 00

  And not much more to say about it, other than, you know, this, this recipe, I think, was very similar to the one followed by the DeepSeek team. So that's kind of another recent model. It's fairly new, so late 2024, the Transfusion paper. But this feels like this is something we're going to see a lot more exploration of these kind of hybrid architectures in the very near future. And the team did some early experiments to ask if they can actually adapt this same recipe across kind of new combinations of modality. Can they look at audio, for example? They did some small scale experiments in the paper, and they suggest the recipe will work for those as well.

• 3:23:24Nathan Labenz?

  One thing I'm not very clear on is why ever quantize the image tokens in the first place?

• 3:23:33Nathan Labenz?

  I recall reading about that some time ago, and I was like, well, why? It just seems strange.

• 3:23:40Speaker 00

  Because if, if you do that, then you can simply train a single decoder transformer to produce both images and text tokens. You've just got a larger code book or two codebooks. And that means you can use a much simpler training objectives. You can, for example, look at the cross, cross entropy for both of them. You don't need all this other machinery in there to do the auto encoding and then the U net down sampling and then up sampling. It's You know, I'm surprised that when they mention that, you know, they achieve the same training losses at half or a third as many of the flops. I thought it was really interesting because it sounds to me like with all that other machinery,

• 3:24:21Speaker 00

  that actually there's it's going to be much more compute intensive to build one of these hybrid architectures like home transfusion. So I'm assuming that is all factored in when they talk about the losses seen at the number of flops is it is a more complicated model.

• 3:24:40Nathan Labenz?

  I'm scanning through both of these papers and it is remarkable on the chameleon one, which is the one that has the discrete or quantized image tokens. It is remarkable that the image outputs do look pretty good. Like you would think that, and does it say quickly how big the vocabulary size is of this?

• 3:25:07Speaker 00

  They say in the paper.

• 3:25:11Nathan Labenz?

  They did. It's not that big. It's only code book size of 8,192.

• 3:25:20Nathan Labenz?

  And the 512 by 512 image gets broken into 1,024 discrete tokens, which means, you know, they, yeah, that's, that's weird. I don't know. That's like, you've got 250,000 pixels, right? 500 times 500, you've got four zeros with a 25 in front of it. So 250,000 pixels, a thousand discrete tokens. So you've got 250 pixels per token.

• 3:26:01Speaker 00

  Yeah.

• 3:26:02Nathan Labenz?

  In this chameleon thing.

• 3:26:04Speaker 00

  Which being by 16 patches then by the look of it.

• 3:26:10Speaker 00

  So if you take 512 by the way 16, you get 32, square that. You get 1024. So 16 by 16 patch. So when you think about it, 16 by 16 pixels. And and then you've got 8,000 tokens to cover that.

• 3:26:31Speaker 00

  How many different variations of a 16 by 16 patch do I need?

• 3:26:35Nathan Labenz?

  I would still think a lot more. I don't know. It just seems like there's, you know, each pixel is three colors, you know, with a range of 256 values. So just the number of possible colors for a single pixel is 256 cubed, right? And just from that alone, I would think, boy, 8,000 tokens to represent a 16 by 16 little patch would just does not seem like enough. But I mean, it's hard to argue with the results. They the images do look

• 3:27:15Nathan Labenz?

  remarkably natural. I'm quite surprised that that works. I would I would have intuited that it would have been just much like more artifacty, that you would be able to see these like, you know,

• 3:27:28Nathan Labenz?

  where these tokens, you know, come together. Yeah, I would expect you would see the seams of this and generally speaking, I don't see them. That's quite, that's quite crazy.

• 3:27:40Speaker 00

  Yeah. They certainly reported for transfusion that they did some evaluation benchmarks and compared it to DALL-E 2 and Stable Diffusion XL. And they found that they were outperforming both of those models. And that they were reaching Llama 1 performance on text-only tasks. So that was their 7 billion parameter transfusion model.

• 3:28:05Speaker 00

  Again, and I don't know if this interpretation is warranted from this, but it's really interesting to see such like fantastic performance. I know these are, you know, it's more recent architecture has been a lot of tips and tricks they're using in training it, which weren't used for SDXL or DALL-E 2. One also wonders, you know, what role has this mixed modality learning played in improving the ability of the model to generate images? I'm not sure, but it's, it's an interesting question, given that we've certainly seen it happen the other way around.

• 3:28:42Nathan Labenz?

  Yeah, the image editing on this is really impressive too. I'm now switching over to scroll through the Transfusion paper. And I think the demos of this that are most compelling to me are basically instruct style editing. There was an image, uh, there was a model not, geez, when was this? I guess it was almost two years ago now. Instruct pics to pics was the one that basically you could give an image and give a command, and it would attempt to edit that image according to your command. And I did one of an ultrasound image of my son, who's now a year and a half, and said, make it look like a newborn baby instead of an ultrasound. And it kind of did that, not, not too bad.

• 3:29:28Nathan Labenz?

  It was a funny experiment anyway. And yet, those were limited, to say the least, in terms of what you could do and how good the quality was and how much fidelity it, it actually had to your instructions. It was, you know, kind of all over the map. Here, I'm like, man, this is exactly what we need for the Waymark application, because we're talking very precise, local edits that are.

• 3:29:58Nathan Labenz?

  Not changing the overall composition, not making the image feel like it's a totally different image, but doing the sort of precise cleanup that you might see from an actual pro image editor doing it in a Photoshop or what have you.

• 3:30:15Speaker 00

  Yeah.

• 3:30:16Nathan Labenz?

  So things like, you know, a couple of examples here that stand out to me most. One is change the graffiti on the, on the side of a truck into calligraphy writing. And the before and after is just amazing. I mean, you see the one, you know, before is like tagged, of course, with graffiti. And then it's just like, man, it's perfectly in situ calligraphy that's been printed onto the side of the truck.

• 3:30:37Speaker 00

  Yeah.

• 3:30:38Nathan Labenz?

  Examples of removing things, you know, replace one object with another, change the color of this thing. This is pretty.

• 3:30:45Speaker 00

  Yeah.

• 3:30:45Nathan Labenz?

  Yeah, it is pretty impressive. This has never been released, right? Was it, do you know why they haven't released this?

• 3:30:51Speaker 00

  No, to, to my knowledge, none of the, the series has been released and I'm not sure why. And one does wonder what image sources they used for the different models. They do say, at least in some of the papers, they use only kind of open source or publicly available image sets. I don't think that's true for all of them. I'm sure for everything they use. They may also, their text to text, they may also have some in-house text to text kind of data sets that they used, which for whatever reason they don't want to release, but I'm not a hundred percent sure. But you're right, it certainly beats like the Spaghettis web, country UI canvases that I end up building to do image editing, just be able to write it in a single sentence.

• 3:31:37Nathan Labenz?

  Yeah, you can really see the, the future here, I think, and in some of the GPT-4O demos as well. It's funny, people ask us all the time about with Waymark application.

• 3:31:50Nathan Labenz?

  Do you use stable diffusion or DALL-E or whatever? And we've actually found that those things are not very useful for our users because they aren't realistic to, you know, small business wants to present itself in a, you know, positive light, but in like a realistic light. They don't want it to feel like, wait, this is nothing like what I saw on TV when you actually show up. So the lack of control and the difficulty of grounding the purely generative models has been a real challenge. And you of course can do image prompting, but a lot of times those also haven't worked super great. They sort of noise your input image and then take it in kind of a different direction.

• 3:32:31Nathan Labenz?

  And it's like, oh, I actually wanted something that was more like local specific, like maintain the integrity of this, but change it in this one, you know, very specific way. And that has been hard to do. I also can imagine how there's other techniques now that are, that are popping up for this too, but character consistency has always been a real challenge.

• 3:32:51Speaker 00

  Yeah.

• 3:32:52Nathan Labenz?

  For people that are trying, just trying to create original content. The way Mark's creative team has done some really cool stuff. We did an episode, I hope we have another one soon, but we did an episode on basically a short film that they made with all DALL-E 2 images. This has been a while. Now there's a part two that uses newer models and all sorts of new techniques. Character consistency was a huge problem in those early models. Scene consistency was another huge problem. They, I mean, they came up with elaborate prompting techniques and all sorts of ways to try to get around that. And I think did, you know, a really remarkable job with what they had at the time. But when I look at this, I'm just like, man, a lot of it falls out of this very quickly where you can just say, you know, change this, do this, put this guy in a different scene.

• 3:33:37Nathan Labenz?

  You know, next thing you know, you're kind of off to the races on a lot of the different things that you want to do that have been hard. So yeah, presumably that'll be coming at some point from an API provider, GPT-4O or otherwise, or maybe they'll finally get around to releasing this. But it is, um, I noticed too, Lily Yu, who's a former guest, I had her on to talk about Megabyte. It was called basically a byte level transformer. And interesting to see her name pop up on another one of these kind of continuous space transformer projects.

• 3:34:15Speaker 00

  Yeah.

• 3:34:18Nathan Labenz?

  Cool. So this has been a fantastic deep dive and walkthrough. Where does this leave us now?

• 3:34:27Speaker 00

  Well, maybe we could just wrap up just with a quick summary of what the frontier labs are offering and what's winning on the different benchmarks. So I found I had to compile this data from a number of different sources because it wasn't all available all in one place. So I've done the best I can. But if one is building with one of these models and you know that you want something that's well on MMMU or for example, Doc VQA, so that's extracting information from images. Or maybe even if you're interested about what does well on Blink, I tried to compile this into a simple table. And I can just walk you through now what I think is winning in each case.

• 3:35:11Nathan Labenz?

  Cool. Yeah, this is really useful.

• 3:35:14Speaker 00

  So what I looked at, I looked at the Grok 2 beta. I looked at the latest version of Claude Sonnet 3.5 new. Gemini 1.5 Pro because I was unable to find results yet for Gemini 2. 401 preview, I could only get one result for, and that was the MMMU. But 40, I could get results for all of them from. And then just a couple of open source models as well. Okay, so we look at leaderboard here and we say, okay, what is currently doing best on MMMU? If I really cared about reasoning over images. And 401 is standing head and shoulders above everything else with 78% there.

• 3:35:57Speaker 00

  And if I wanted to look at what was next down from that, we would find Claude 3.5 Sonnet new and the Intern VL 2.5 model, which is currently available, I believe, through an API. We find those down at 70 and a bit percent each. So Gemini 1.5 Pro is coming a few percentage points below those two. So that's kind of where we're sitting on MMMU at the moment. Be interested to see what Gemini 2 does there. But the Grok 2 beta is sitting at 66% on MMMU.

• 3:36:32Speaker 00

  And if we're

• 3:36:34Nathan Labenz?

  One quick note on that too, because this intern, I keep forgetting as we've gone through this, who made this model? The OpenGV Lab, which is, yeah. So it's another, I mean, I think we touched on this a little bit earlier too, but it's another good reminder that the Chinese models, the Chinese labs are not super far behind. They are, in this case,

• 3:36:59Nathan Labenz?

  right there, head to head with Anthropic five months earlier.

• 3:37:04Speaker 00

  The Lava One Vision model is from ByteDance, the Qwen model is from Alibaba.

• 3:37:11Speaker 00

  Yeah, and they all perform very well.

• 3:37:14Nathan Labenz?

  And this is not, yeah, the reason I mentioned the five month earlier thing too, is because sometimes people will be like, oh, well, they're, you know, doing that, they're starting with, you know, an American thing, or they're, you know, they're starting with Llama and not acknowledging it, or they're, you know, trading on the outputs of the American models, whatever the sort of rationalization often is, and I'm sure sometimes that stuff is happening. But here, it is still, as of now, and I guess we've got, you know, caveats around like O3 and whatever, and full O1, where not all the data points are available yet, but it is striking that this Shanghai AI group has

• 3:37:54Nathan Labenz?

  three tenths of a percent lower than Claude 3.5 Sonnet knew, a full five months ahead of time. So they're definitely not, definitely not training on Claude outputs to achieve that. And that is higher than GPT-4O by just a point, but still, it's like, I think it's, it is very much worth keeping in mind that the gap here is like basically zero. And arguably, you could even say, and

• 3:38:23Nathan Labenz?

  if you squint at it, you could even say the Chinese labs are maybe a little bit ahead, but I would say.

• 3:38:29Speaker 00

  Yeah.

• 3:38:29Nathan Labenz?

  Overall rounding, you should probably say they're about the same.

• 3:38:32Speaker 00

  Yeah, we also note as well that the parameter counts in their models, all around the 70 to 80 billion parameter mark. And presumably they're going to build larger and larger, and presumably they'll get better and better as a result.

• 3:38:47Speaker 00

  But, you know, lots of what we talked about earlier in the podcast about how fast the training recipes and the data sets are developing. You know, it probably goes to account for a lot of this, just the more recent the model, like they get better every single time.

• 3:39:02Speaker 00

  Because people are learning so much about how to effectively do this.

• 3:39:06Speaker 00

  I think it's also interesting to look at the DOC VQA scores here. You know, all of the models that I mentioned are doing above 90% on that benchmark. The best I can find is, again, is the Qwen-2VL, QwenVL2 model, which is currently doing 96.5%. So that's open source, proprietary model. I don't have results for the new Claude 3.5 or for 401 on DOC VQA, so that's just to point that out. But that model is doing better, for example, than GROK2, it's doing better than Gemini 1.5 and GPT-4O on DOC VQA.

• 3:39:44Speaker 00

  So kind of leading there. And Blink is very interesting because we haven't got results for everything here. But kind of at the top of the leaderboard, we've got 4O, which has about 63.2%. I have seen a result that's higher than that, but I've seen three results for it. Two of them are at 63.2% on Blink, so I'm going to go with that one. The OpenGV Labs InternVL 2.5 is clocking in at 63.8, so doing ever so slightly better. Gemini, 61%, and Claude 3.5 Sonic New, that's 56.5%. So those are the best and most up to date scores I can find on those three benchmarks.

• 3:40:30Speaker 00

  It's also worth just looking at the mini class here. So here I've taken kind of the flash and mini versions of some of these models. And they're obviously a little bit lower down most of the benchmarks, they're smaller models, but it's interesting to know if I was to pick one up, what would I use? If we look at M M M U, we've got results for Gemini 2.0 flash, which is clocking in at 70.7%. And that makes it stand head or shoulders above everything else in the mini class, which I'm including 40 mini, the intern V L eight billion parameter. I haven't got result for 401 mini. Grok 2 mini is at 63.2%. So best bet, if you want kind of reasoning

• 3:41:14Speaker 00

  over images, would seem to be Gemini 2.0 flash at the moment. Doc V Q A got very few results here, but the intern V L 2.5 eight billion parameter model is clocking in at 95.1%, which seems pretty darn good to me. And interesting enough, that's exactly the same result as it gets in its full 78 billion parameter form. Blink, we're just going to finish on Blink, the mini class of models. The best score I can find, and I couldn't get one for Grok, for Gemini 2.0, or for 401 mini, but I could find it for G T 40 mini, and that's clocking in at 51.9%, which means that the intern

• 3:41:59Speaker 00

  V L 2.5 model is getting slightly better at 54.8%. The surprise winner coming in from left field in the mini class is Microsoft's Phi, or Phi, depending how you want to pronounce it, 3.5 vision model, which is a four billion parameter class model and scoring 58.3% on the Blink benchmark. Not doing so well on M M M U, doing really well on Blink, which is really interesting. So I combed the technical report for the V vision model.

• 3:42:33Speaker 00

  And just a couple of interesting details from it and not enough to make conclusions on. For a four billion parameter class model, they used half a trillion pre-training tokens from the mixed pre-training data set. And that's a big data set for a small model. We think about the Intern VL 2.5 model, they said they trained on 128 billion tokens. The Qwen 2 model, 1.5 trillion tokens. But those are all many times the size. So it's a fairly large training data set for a small model.

• 3:43:05Speaker 00

  The other really interesting thing was that they mentioned their SFT data set, which they described as a combination of data sets from a significant component of which was built in-house by Microsoft. Thirty-three billion tokens. So this is instruction fine-tuning data. That's an extremely large data set, especially for a small model. I mean, I can't can't recall seeing one that large anywhere else in the research. And they also mentioned performing DPO, which only a couple of the other models like Alma 3V explicitly mention a DPO step.

• 3:43:42Speaker 00

  So not sure why it does so incredibly well, but anecdotally, I have seen plenty of commentary online with people saying, wow, the V3.5 model is really good at visual understanding. So really interesting, perhaps something we should have dwelt on a bit more in the podcast. And that is about everything that we wanted to cover today. I mean, we've kind of told the story of the last couple of years of VLMs. What are we going to see in the future? I think we'll see a lot more of these kind of true multimodal models following the Transusion recipe. I do expect to see the scale of parameter count of open source VLMs increase further, particularly now as we know about this kind of progressive upsizing of the language model backbones work so well.

• 3:44:28Speaker 00

  We should expect to see more innovation, either pre-training of vision transformers, or maybe even the replacement at some point. And a continuation of this kind of production of new fine-tuning data sets that contain programmatic or human augmentations. So I would, for example, expect to see new fine-tuning data sets there.

• 3:44:52Speaker 00

  What we haven't seen too much of as well is, we just mentioned DPO at the very end there, not too much exploration of the role that alignment post-training can play in vision language models. So I expect to see that explored as well. And that concludes not-quite-so-whistle-stopped tour through the last two years of vision language models.

• 3:45:12Nathan Labenz?

  Amazing. Well, the depth of research that you put in to make this possible is outstanding and definitely much appreciated. And I learned a lot from it. So as you know, that's how I tend to score these episodes for myself. And I come away definitely with a much better understanding of

• 3:45:32Nathan Labenz?

  the various options and strengths and weaknesses, and even a few prompting techniques along the way. How does this relate to what you typically do? And maybe in the last couple minutes, just tell us a little bit about like, your normal work and the sort of stuff you do commercially?

• 3:45:49Speaker 00

  Yes. A lot of this is relevant to some of the work that I'm doing. So Veritae is the small consultancy that my colleagues and I set up a couple of years back. We do a lot of AI strategy work. So before that, we used to do a lot of data science and analytics strategy in my background. And so we've moved on to thinking about how do you develop and build an AI strategy? And especially if you're an SME. So I think that's our sweet spot. Although what we do does work, I think, in larger organizations, if you're kind of working at the department level. So we do a lot of that. But we also do a lot of prototyping work for people. So proof of concepts. You have some ideas, you know, that in your strategy,

• 3:46:32Speaker 00

  things you want to follow, things you want to, things the company wants to try out. So we have a very kind of structured way of performing these experiments cheaply and finding out what's easy and what's difficult. Because as you probably know, a lot of working with AI today is, you know, it's your mileage may vary. So we kind of know what the best practice is. But it's very hard to know from the outset whether you're going to get great results, given the data or compute constraints that your client may have. And so we've kind of developed this rapid experimentation methodology. But in some of the domains we're working in, and we do some in medicine, particularly looking at medical terminologies and how they get used by language models and how language models can work with them. But also, and I've spent a lot of time looking at open source intelligence for a few clients recently.

• 3:47:15Speaker 00

  And there, a lot of kind of the data that we're trying to interpret is multimodal in nature. So there's lots of trying to understand, you know, how does this image seem to correspond with this claim that's being made? You know, can we infer who might be in the image or what the image is about, given that there's some surrounding text or context we understand? And so in that, you know, it's in that domain that we've certainly been working with some of the V L Ns here.

• 3:47:43Nathan Labenz?

  Yeah, I hear a couple

• 3:47:46Nathan Labenz?

  sort of reflections, probably have some common experiences that we've had. I don't do a ton of this sort of stuff, but occasionally, you know, mostly I've done it just for my own company, which I've mentioned a dozen times already. But occasionally I'll take on a project for somebody that asks me for help with something. And it's an interesting juxtaposition often of like, and I'm sure you've found various versions of this. A lot of times you can like answer somebody's question really quite quickly. And I'm always reminded of Tyler Cowen's answer for when he's asked how long it took him to read a given book. Because he's a famously fast reader. People ask him that question and he always answers with his age. And I think this

• 3:48:31Nathan Labenz?

  tour through visual language models is a good reflection of how much depth and sort of obsessive quality of research has to go into the ability to then turn around quickly and be like, I think I know what to do here for your random situation. And I think we can put together a proof of concept on that pretty quickly. But that's because you've done this like extensive exploration of everything that's out there and have a really already fine tuned intuition for which direction to go.

• 3:49:04Speaker 00

  Yeah, the way we do strategy projects relies on this a lot as well. It's like you have to come with a prepared mind. You have to come having been immersed and soaked up everything that's happening. So that when you see the, you know, when people say this is the problem that exists here, you can pattern match it to something. And, you know, I am sure like you have got a like a big archive, search of an archive, like my own personal kind of rag rag index that I maintain at home of just everything that I ever come across, papers, newsletters, sub stacks, whatever it is, sits in there. And I just try and pattern match it so that I have some grasp of, okay, this is how we're going to, you know, suggest the options for how you might solve this problem, A, B, or C.

• 3:49:47Speaker 00

  And yeah, but you do need to be steeped in it, I think, in order to then spot the opportunities to use the technique, the technology, the model, whatever it is, when you actually see it in practice.

• 3:50:02Nathan Labenz?

  Yeah, cool. You want to tell us where we can find you and the company online?

• 3:50:07Speaker 00

  Yeah, you can find the company website at veritai, which is V E R A T A I dot co.uk, veritai.co.uk. Or you can find me on LinkedIn, Will Hardman, and I am writing a fair bit at the moment about AI strategy. And next year you'll be writing about various other things and maybe about vision language models as well.

• 3:50:30Nathan Labenz?

  Cool. Well, I'll be sure to connect with you there and encourage the audience to do the same. This has been a fantastic walkthrough of visual language models. I know a lot of work has gone into it, but if you want to tackle another topic like this, I would love to do it. But now I will say thank you for this one. And officially, Will Hardman from Veritai, thank you for being part of the cognitive revolution.

• 3:50:52Speaker 00

  Thanks for having me on, Nathan.