How to train your own Large Multimodal Model — with Hugo Laurençon & Leo Tronchon of HuggingFace M4

Latent Space is heating up! Our paper club https://lu.ma/llm-paper-club ran into >99 person Discord limits, oops. We are also introducing 2 new online meetups: LLM Paper Club Asia https://lu.ma/llm-paper-asia for Asia timezone (led by Ivan), and AI in Action: hands-on application of AI https://lu.ma/el0y5mpi (led by KBall). To be notified of all upcoming Latent Space events, subscribe to our new Luma calendar https://lu.ma/ls (sign up for individual events, or hit the RSS icon to sync all events to calendar). In the halcyon open research days of 2022 BC (Before-ChatGPT), DeepMind was the first to create a SOTA multimodal model by taking a pre-existing LLM (Chinchilla 80B https://arxiv.org/abs/2203.15556- now dead https://news.ycombinator.com/item?id=37383413?) and pre-existing vision encoder (CLIP https://arxiv.org/abs/2103.00020) and training a “glue” adapter layer, inspiring a generation of stunningly cheap and effective multimodal models including LLaVA https://x.com/ChunyuanLi/status/1710299381335798202?s=20 (one of the Best Papers of NeurIPS 2023 https://www.latent.space/p/neurips-2023-papers), BakLLaVA https://github.com/SkunkworksAI/BakLLaVA and FireLLaVA https://twitter.com/lqiao/status/1748243039766925351. However (for reasons we discuss in today’s conversation), DeepMind’s Flamingo model was never open sourced. Based on the excellent paper https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/tackling-multiple-tasks-with-a-single-visual-language-model/flamingo.pdf, LAION stepped up to create OpenFlamingo https://laion.ai/blog/open-flamingo/, but it never scaled beyond 9B. Simultaneously, the M4 (audio + video + image + text multimodality) research team at HuggingFace announced https://x.com/SanhEstPasMoi/status/1632775840135016448?s=20 an independent effort to reproduce Flamingo up to the full 80B scale: The effort started in March, and was released in August 2023. We happened to visit Paris last year, and visited HuggingFace HQ to learn all about HuggingFace’s research efforts, and cover all the ground knowledge LLM people need to become (what Chip Huyen has termed https://huyenchip.com/2023/10/10/multimodal.html#clip) “LMM” people. In other words: What is IDEFICS? IDEFICS is an Open Access Visual Language Model, available in 9B https://huggingface.co/HuggingFaceM4/idefics-9b and 80B https://huggingface.co/HuggingFaceM4/idefics-80b model sizes. As an attempt to re-create an open-access version of Flamingo https://arxiv.org/abs/2204.14198, it seems to track very well on a range of multimodal benchmarks (which we discuss in the pod): You can see the reasoning abilities of the models to take a combination of interleaved images + text in a way that allows users to either describe images, ask questions about the images, or extend/combine the images into different artworks (e.g. poetry). 📷 From IDEFICS’s model card https://huggingface.co/HuggingFaceM4/idefics-80b-instruct and blog post https://huggingface.co/blog/idefics The above demo screenshots are actually fine-tuned instruct versions of IDEFICS — which are again in 9B https://huggingface.co/HuggingFaceM4/idefics-9b-instructand 80B https://huggingface.co/HuggingFaceM4/idefics-80b-instructversions. IDEFICS was built by connecting two unimodal models together to provide the multi-modality you see showcased above. * Llama https://huggingface.co/huggyllama/llama-65b v1 for language (specifically huggyllama/llama-65b https://huggingface.co/huggyllama/llama-65b) - the best available open model at the time, to be swapped for Mistral in the next version of IDEFICS * A CLIP https://github.com/mlfoundations/open_clip model for vision (specifically laion/CLIP-ViT-H-14-laion2B-s32B-b79K https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K - after a brief exploration of EVA-CLIP https://huggingface.co/QuanSun/EVA-CLIP, which we discuss on the pod) OBELICS: a new type of Multimodal Dataset IDEFICS’ training data used the usual suspect datasets, but to get to par with Flamingo they needed to create a new data set. Enter OBELICS https://huggingface.co/datasets/HuggingFaceM4/OBELICS: “An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents”: * 115B text tokens * 141M English documents * 353M images These bullets are carefully curated and filtered by going through Common Crawl dumps between FEB 2020 - FEB 2023. We discuss the 2 months of mindnumbing, unglamorous work creating this pipeline: There’s a lot of mentions of ‘multi-modal' web documents’ which deserves some explanation. We’ll show you instead of tell you: You can see from this graph that OBELICS ends up outperforming the other image-text pairs dataset (LAION in this case) when stacked head-to-head. You can view a subset of OBELICS and perform visualizations on them here https://atlas.nomic.ai/map/f2fba2aa-3647-4f49-a0f3-9347daeee499/ee4a84bd-f125-4bcc-a683-1b4e231cb10f: 2024 Update: WebSight et al Most of this interview was recorded on Halloween 2023 at HuggingFace’s headquarters in Paris: In anticipation of an IDEFICS v2 release. However, several roadblocks emerged, including a notable scandal around CSAM in LAION-5B https://www.theverge.com/2023/12/20/24009418/generative-ai-image-laion-csam-google-stability-stanford, which affected all models using that dataset. The M4 team have adopted a strategy of shipping smaller advancements in 2024, and the first ship of the year is WebSight https://huggingface.co/datasets/HuggingFaceM4/WebSight, a dataset of 823,000 HTML/CSS codes representing synthetically generated English websites, each accompanied by a corresponding screenshot (rendered with Playwright https://github.com/microsoft/playwright). This is intended for tasks like screenshot-to-code workflows like Vercel’s V0 or TLDraw https://www.latent.space/p/tldraw, and will be part of the dataset for IDEFICS-2. As noted in our Best Papers recap https://www.latent.space/p/neurips-2023-papers, synthetic data is emerging as one of the top themes of 2024, and the IDEFICS/OBELICS team have wasted no time enabling themselves with it. Timestamps * [0:00:00] Intro * [0:00:00] Hugo, Leo’s path into multimodality * [0:09:16] From CLIP to Flamingo * [0:12:54] Benchmarks and Evals * [0:16:54] OBELICS dataset * [0:34:47] Together Redpajama v2 * [0:37:12] GPT4 Vision * [0:38:44] IDEFICS model * [0:40:57] Query-Key Layernorm for training * [0:46:40] Choosing smaller vision encoders - EVA-CLIP vs SIG-GLIP * [0:49:02] IDEFICS v2 * [0:52:39] Multimodal Hallucination * [0:59:12] Why Open Source Multimodality * [1:05:29] Naming: M4, OBELICS, IDEFICS * [1:08:56] 2024 Update from Leo Show Notes * Introducing IDEFICS: An Open Reproduction of State-of-the-Art Visual Language Model https://huggingface.co/blog/idefics * IDEFICS Knowledge sharing memo: technical lessons and mistakes https://github.com/huggingface/m4-logs/blob/master/memos/README.md * Victor Sanh memo https://docs.google.com/document/d/1ZNGyVWYFUbzV0xuei4SED2QAakGjMpaaQALcKYQm46U/edit#heading=h.66l93vag25o8 * OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents https://huggingface.co/papers/2306.16527 * Papers cited: * BLOOM https://arxiv.org/abs/2211.05100: A 176B-Parameter Open-Access Multilingual Language Model * Barlow Twins https://arxiv.org/abs/2103.03230: Self-Supervised Learning via Redundancy Reduction * CLIP paper https://arxiv.org/abs/2103.00020: Learning Transferable Visual Models From Natural Language Supervision * Vision Transformers paper https://arxiv.org/abs/2010.11929: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale * Flamingo paper https://arxiv.org/abs/2204.14198: a Visual Language Model for Few-Shot Learning * April 2022 preprint from DeepMind https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/tackling-multiple-tasks-with-a-single-visual-language-model/flamingo.pdf, blogpost https://deepmind.google/discover/blog/tackling-multiple-tasks-with-a-single-visual-language-model/ * VQAV2 paper https://paperswithcode.com/paper/making-the-v-in-vqa-matter-elevating-the-role: Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering * OK-VQA: A Visual Question Answering Benchmark https://arxiv.org/abs/1906.00067Requiring External Knowledgehttps://arxiv.org/abs/1906.00067(https://okvqa.allenai.org/) * MMBench: Is Your Multi-modal Model an All-around Player? https://arxiv.org/abs/2307.06281 * Qwen-VL https://arxiv.org/abs/2308.12966: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond * Sig-GLIP paper https://arxiv.org/pdf/2303.15343: Sigmoid Loss for Language Image Pre-Training https://arxiv.org/pdf/2303.15343 * Nougat https://facebookresearch.github.io/nougat/: Neural Optical Understanding for Academic Documents * MMC4 (Multimodal C4) https://arxiv.org/pdf/2304.06939v1.pdf: An Open, Billion-scale Corpus of Images Interleaved With Text * Dall-E 3 paper: Improving Image Generation with Better Captions https://cdn.openai.com/papers/dall-e-3.pdf * GPT-4V(ision) system card https://openai.com/research/gpt-4v-system-card from OpenAI * Query-Key Layernorm trick: paper https://arxiv.org/abs/2302.05442 (Scaling Vision Transformers to 22 Billion Parameters), tweet https://x.com/SanhEstPasMoi/status/1632775853640646657?s=20 * EVA-CLIP: Improved Training Techniques for CLIP at Scale https://arxiv.org/abs/2303.15389 * “We intially explored using a significantly bigger vision encoder (the biggest in open-access at that time) with EVA-CLIP https://huggingface.co/QuanSun/EVA-CLIP. However, we ran into training instabilities very quickly. To lower the risks associated to the change of vision encoder, we decided to continue with laion/CLIP-ViT-H-14-laion2B-s32B-b79K https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K which we have been using until that point. We will leave that swap for future iterations and will also consider using higher resolution images.” * Datasets * Together’s RedPajama-Data-v2 https://www.together.ai/blog/redpajama-data-v2: An open dataset with 30 trillion tokens for training large language models * LAION COCO: 600M synthetic captions from Laion2B-en https://laion.ai/blog/laion-coco/ * Chip Huyen’s writeup on LMMs https://huyenchip.com/2023/10/10/multimodal.html * Joseph Nelson of Roboflow on Latent Space https://www.latent.space/p/segment-anything-roboflow * HuggingFace M4 https://huggingface.co/HuggingFaceM4 * HuggingFace timm https://huggingface.co/docs/timm/index: library containing SOTA computer vision models, layers, utilities, optimizers, schedulers, data-loaders, augmentations, and training/evaluation scripts. It comes packaged with >700 pretrained models, and is designed to be flexible and easy to use. * Logan Kilpatrick declaring 2024 the year of Multimodal AI https://www.youtube.com/watch?v=bNZV9s3_u44&t=80sat AI Engineer Summit This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:06Speaker 00

  Welcome to the Latent Space Podcast, where we dive into the wild, wild world of AI engineering every week. This is Anna, your AI co-host. Happy New Year, did you miss me? As an AI language model, I cannot miss you back, but I'm glad to stand in for LSAO while swyx is traveling, this time in Paris at Hugging Face HQ. At the AI Engineer Summit in 2023, Logan from OpenAI pronounced 2024, "The Year of Multimodality."

• 0:32Speaker 01

  I'm excited for 2024, which I think is really going to be the, I don't know if I can trademark this, but the Year of Multimodal Models. It's a tongue twister, but also hopefully the domain is available, yearofmultimodals.com. No, don't buy it if it's available. Yeah, so I'm excited. OpenAI has a ton of multimodal capabilities that are in the works. Some folks might have already tried some of these in ChatGPT in the iOS app or the web app today. Things like vision, taking in images, describing them. We'll show that later on. Also, the ability to generate images. We've had this historically with Dolly 2, but Dolly 3, really, if folks have tried it, it takes things to the next level. So excited to show some of that today as well.

• 1:17Speaker 00

  In 2024, the Latent Space Pod will offer deeper dives into multimodality. Today, we'll talk to Leo Tronchon and Hugo Laurensen of Hugging Face, who trained IDEFICs, a fully open source reproduction of DeepMind's closed flamingo model done from scratch, scaled all the way up to 80 billion parameters. By the way, dear listener, we are expanding our online meetups this year after the success of the Latent Space Paper Club. See the show notes for the new AI in Action and Paper Club Asia meetups. Watch out and take care.

• 1:52Speaker 02

  Thanks for having me at your beautiful office. It's really surreal for me to visit the Hugging Face Paris office because I've always seen you guys online and you organize really huge meetups here in Paris. I want to learn everything about Hugging Face and you guys' work.

• 2:05Speaker 03

  So my name is Leo. I've been working at Hugging Face for two years. I started working on data sets for the Bloom language model. So it's the 176 billion parameter model that we open source and that was at that time the biggest one. And it was also multi-modal. So I worked on the model and the data set. And then I moved to multimodality with the current project with Edifx and Obelix. Now I am working also with you on the version two of Edifx.

• 2:38Speaker 02

  And Leo, you start.

• 2:39Speaker 04

  So my name is Leo. I joined Hugging Face a year and a half ago. I was a student still. So first six months, I was still as an intern. But I started to work on multimodality right away. And then I spent all my time here in the research team working on multimodality and Edifx that we open sourced in August.

• 3:00Speaker 02

  I think a lot of people are very interested in learning more about Edifx and multimodality in general. Bigger question first. How is Hugging Face organized? You told me some surprising details about the size of Hugging Face. You guys are a $4 billion company, only 200 people, less than 200 people?

• 3:14Speaker 04

  About 160 people, yeah.

• 3:16Speaker 02

  And then how many people in the research team?

• 3:19Speaker 03

  This is like maybe 15.

• 3:22Speaker 02

  So between 10 and 20% of the company is research?

• 3:25Speaker 04

  Yeah, I'd say.

• 3:26Speaker 02

  One, that's impressive. And then two, this is something that we discussed before. It's also unintuitive why Hugging Face needs to do research.

• 3:32Speaker 04

  I think the company has a good incentive to do research because most of the companies that do AI, they have an incentive to get very good models out, but not the best model out. Their competitive advantage is to have the best model in-house that they can fine tune for their customers. And then the open source is for sure. But Hugging Face is one of the only companies that has an incentive to get the best model out there in the open. And that's why I think the research team is quite important. It's also important Because all the tools that Hugging Face makes are used by the researchers. So they get all the feedback directly from us. And I think this is really useful to develop the tools behind it.

• 4:13Speaker 02

  Are you talking about the Transformers library?

• 4:15Speaker 04

  Transformers library, Diffusers library, data sets.

• 4:18Speaker 02

  So those seem to me more like inference type tools. Are there any sort of training tools that you do?

• 4:24Speaker 04

  Data sets is used for the for the training. Transformers, we've been using for modeling internally. We're also developing a library for training. I think it's going to be open source, but we'll see.

• 4:38Speaker 03

  So for example, we used for the construction of Obelix, we used our whole pipeline, the library data sets. The aim of our big open source project is also to test our own internal libraries and see if they if they scale well. For example, the data sets guys, they never worked with data sets this big before. So this is a way also to test our solutions.

• 5:05Speaker 02

  This big meaning 114 million images, something like that?

• 5:08Speaker 03

  More than 3 million, 300 million images.

• 5:12Speaker 02

  I've tried Transformers and tried Diffusers, I haven't tried data sets. Why do I need data sets?

• 5:16Speaker 03

  So I think data set is great because you can load data sets that don't fit in memory.

• 5:22Speaker 02

  So it's just kind of like virtual library, virtual pointers or whatever?

• 5:25Speaker 03

  Exactly. And also you can easily filter a rows of your data sets, map them, manipulate and modify the content. So it's really easy. And also to do the operations in parallel is much easier with this library.

• 5:43Speaker 02

  What is the leading alternative to data sets? Like what do machine learning researchers use if they don't use data sets?

• 5:49Speaker 03

  They just do everything by hand or?

• 5:51Speaker 02

  Basically manually pageinate, write code to pageinate.

• 5:54Speaker 03

  Yeah, yeah, that's what I did before, but it's just much faster to the because everything is done for you. The multi-processing, you just have to implement your function.

• 6:02Speaker 02

  That's great. Well, I think that's a good intro to the overall Hugging Face ecosystem. But I'm interested in the journey from Bloom into computer vision for yourself. And then obviously you'll Also had your own journey into multimodality. A lot of people who are listeners and readers of InSpace also following that same journey, right? They only have some kind of NLP background and now everyone is interested in multimodality. What was that journey like for you?

• 6:27Speaker 03

  Not from the research team, but from the hub team. When they started hosting multimodal models and data sets. And quickly after that, we also realized that it would be a good idea to also train ourselves, multimodal models, to catch up with the proprietary models from DeepMind, Google, etc. I think that was the natural path for us. But we didn't drop the idea of doing pure text models. So there is also a team for LLMs. It was just the creation of another team.

• 7:02Speaker 04

  For me, the journey was a bit different because I didn't really... I came right away from my masters. So I had projects on computer vision, where, for example, I don't know if you've heard of Dino, probably. And there was another paper called Barlow Twins. Basically, I had a project on which I tried to combine the two objectives. So I was more towards computer vision before joining. But I was really interested in doing multimodal. When I saw that there was an internship for this position, I was glad. Then the team was already starting to do the project when I joined. And so I kind of joined the train, yeah.

• 7:39Speaker 02

  Just a demographic question. Is everyone here? Is everyone on your team here?

• 7:43Speaker 03

  We had a big shift in the team in the recent months. Some people left for other startups or creating their own. But what is really interesting to me is that when we started the project, not a lot of people in the team previously worked with multimodal models. Maybe only two of them. And we were like six, seven really working on the project. So it was really new to us this field. And we also wanted to have this knowledge because there, of course, there are. It's explained on the papers how to do things. But without doing them yourself, you still miss a lot of things and you miss the intuition.

• 8:27Speaker 03

  And we also wanted to build this knowledge of multimodality. When building the version two, that's we go much faster because we have a better intuition.

• 8:38Speaker 04

  Yeah, this and also I think it talks about the philosophy of Hugging Face, of having small teams with big impact. And so we started with a fairly big team for Hugging Face standard with six, seven people. As Hugo was saying, we lost a few people to different startups. But the idea is still to go as fast, if not faster, with less people right now. And I think it's possible because of all the background we built in the previous iteration. Because small teams can work a lot faster because there's less

• 9:10Speaker 02

  Less overhead.

• 9:11Speaker 04

  Communication, less overhead.

• 9:12Speaker 02

  Very cool. Yeah, so I do want to get into Itifx and Oblix. I wanted to basically go over a little bit of introductory stuff for people, right? So in my mind, the two main multimodality papers that everybody should read is CLIP and Vision Transformers. Would you mark out anything else or what do you personally get from those two papers?

• 9:32Speaker 03

  These two papers, I think, build a starting block. Because now what we are noticing is that for building super large models, we don't train them from scratch. We just pre-train usually uni-modal models that we somehow mix together. So I think CLIP or VIT can serve as a pre-trained backbone that you combine with another pre-trained language model backbone to obtain something multimodal. So these are foundation models that play the same role to us as LAMA models or language models.

• 10:10Speaker 04

  The important thing to understand with vision transformers and CLIP is that they provide the basics for them integrating images into this language modeling objective that we use. Then it's mostly a question of data and image resolution. And a lot of engineering goes there.

• 10:31Speaker 03

  And just to note on this, so some research has shown that when you use pre-trained vision encoder that was trained also with text objective, for example, constructive loss, like the CLIP loss, it's better to use this type of vision encoder than vision encoder trained only on classification or uni-modal task if you are building multimodal models. So if you are building Vision language model is better to take as a pre-trained backbone, a pre-trained vision encoder that has been trained using text.

• 11:06Speaker 02

  Is that not intuitive?

• 11:08Speaker 03

  Imagine you can take, imagine you take a vision encoder that is super good at classification. Then you can imagine that the embeddings that you get from your vision encoder, a superstar to plug into your language model. It's intuitive, but it could clearly work to have a vision encoder trained without text at all. But researchers have shown that it's better to use this contrastive loss, for example.

• 11:32Speaker 04

  And once you have those backbones, the question is really how you integrate it? Like, how you integrate both of them into the architecture? It turns out with very lightweight updates, you take the embeddings that come from the clip, the output of the clip, and you have just a linear that you train on top of this. Then you, when you pass this to the language model and you only train this part, you can already get pretty good results in multimodality. You don't have to train all the parameters when you're training the multimodal model. You can just train the adapter.

• 12:06Speaker 02

  Is this what was spelled out in Flamingo? Or you just kind of derive some kind of transform that you're happy with?

• 12:15Speaker 04

  So in Flamingo, you introduce a lot more parameters, but there's still like those cross attentions that you insert in the model that are new. Those you train from scratch. But the rest of the model, the language model backbone and the vision backbone, they are frozen during the training, so you never update it. So it's a different type of adapter, but it's more, it's more heavyweight than what you could have in recent papers. Now with more parameters, you also often get better performance. Is it necessary to have all those parameters when you only train the adapter part? It's not clear answer. There's no clear answer yet.

• 12:51Speaker 02

  Okay, I think that brings us up to date. Oh, except for benchmarks, I wanted to introduce people to the concept of how hard it is to evaluate benchmarks for multimodality.

• 13:01Speaker 03

  So they are the academic benchmarks, classic. For example, H2, G2. They are for visual question answering. There is also, there are also the image captioning benchmarks.

• 13:16Speaker 02

  The COCO.

• 13:17Speaker 03

  COCO, exactly. Flickr. However, one really important thing that we noticed is that this benchmark.

• 13:27Speaker 03

  The performance of your model heavily depends on how you formulate the answer. For example, for visual question answering tasks, you will have a question and an answer. This answer will be generated open-ended by your model. You just prompt the model with your question and then it will generate some words until the end of sequence is a token is reached. But the thing is that if you have a question and the answer is simply no. If your model says you can count it wrong or you can count it as.

• 14:01Speaker 02

  There's ways to adjust for that, like, you know, some kind of distance metric or something.

• 14:06Speaker 03

  But it's hard, it's hard.

• 14:08Speaker 02

  Use another model.

• 14:10Speaker 03

  So just the way you're formulating the answers heavily impacts your performance. And the fact that some people are fine-tuning directly on the benchmark to try to optimize this formulation. Or the fact that other people are doing a few-shot evaluation. So a few-shot is by giving the model examples of how to formulate the answer. So it makes it hard to compare the models. Because they are not evaluated the same way, even if it's on the same benchmarks. So this is a problem. So you will have all these academic benchmarks. And then you will have these new benchmarks that are not commonly adopted yet. But are created basically with strong language models like GPT-4.

• 14:53Speaker 03

  And people are prompting GPT-4 with images. And ask it to generate automatically question and answers. And then we can evaluate our models. And this is very new.

• 15:07Speaker 04

  The evaluations in multimodal, like for multimodal models are still a bit rough, I think. But even for language models, there's discussions of if benchmarks are really the way to go for some tasks. When you do instruction tuning, for example, for a language model or LHF, you shouldn't be evaluating on the same benchmarks from the point of view of a lot of people. On multimodal, it's also that the quality of the data sets we're evaluating on are not super clean. I recruited something recently, someone that was showing like failure cases of VQAD2, I think. And it was interesting that sometimes the questions and answers are like super obvious and sometimes it's like so far away, even you wouldn't.

• 15:49Speaker 02

  Not even a human would get it.

• 15:50Speaker 04

  Yeah, and I think sometimes it's just plain wrong. And so it's also the quality of the data sets to evaluate on and the diversity of them. It matters a lot. And right now, we still have a few blind spots in evaluations. But it's really interesting to see the field move on this because as we have a lot more multimodal models, the evaluations benchmarks are improving. Maybe four or five come out that were nice in the past two months.

• 16:15Speaker 02

  Off the top of your head, can you name any of these that's come to mind?

• 16:19Speaker 04

  MN Bench, Poppy. There's Seed and there's a brand new one, but I don't know if it's out yet. Like the paper is out, it's called Halucin, like something, Halusion, I think. And this one, I read the paper, I don't know the size of it, but from the examples they give on the paper, it seemed really, really interesting and hard to beat. Yeah, I'm excited about this one mostly.

• 16:42Speaker 02

  This is like the new race, right? In the last five years, there was a race towards like sort of common sense benchmarks in NLP, but now this is the new.

• 16:51Speaker 04

  Yes, it's getting to multimodal.

• 16:54Speaker 02

  Very cool. Maybe we should, we can go into the work that you did for ObliX. Let's describe the size of the data set, what you did to clean it up. You know, a lot of these things start from CommonCrawl and CommonCrawl is great, but also it's very messy.

• 17:09Speaker 03

  So first, why we wanted to do it? We were trying to replicate Flamingo and Flamingo built their own data sets of Interleaved image-text web documents. I think it contained more than 50. A hundred million images, if I'm not wrong. And it was based on...

• 17:29Speaker 02

  For Flamingo?

• 17:30Speaker 03

  Yeah, for Flamingo. And it was based on like fifty million webpages. However, the dataset was not open. So I talked to the authors. And one of the reasons it was not open is because they used their PageRank Google algorithm to try to know in advance which website to target in their dataset.

• 17:51Speaker 02

  So meaning higher paid rank, higher ranking SEO sites have higher weight?

• 17:55Speaker 03

  Yes, exactly. So that's how they scrapped the websites. That's one of the reasons why they don't...

• 18:04Speaker 02

  Many reasons to not open source your datasets.

• 18:06Speaker 03

  So we wanted to build a dataset that was at the beginning similar to this one. And so we made it even larger and fully open source. Because we believe foundation multimodal models trained on Interleaved image-text documents are better than the ones trained only on pairs.

• 18:29Speaker 04

  Maybe to go further into that point, what we found, and that is interesting, is it's for the VQA tasks that this dataset is really important. For the captioning tasks, you have an image-text dataset like Leon, and it's great. And it's going to improve pretty well. Like, the alignment is strong. Just to explain alignment, it's like basically images that are aligned with the text. So the text means something that is related to the image. And so for Leon, it will be enough. For the captioning tasks, it will... Maybe for some OCR tasks, although it's still weak on this one. Even edifx could use improvements on this one. And then the ObliX dataset is really important for reasoning,

• 19:11Speaker 04

  to have the model be performing on VQA V2, OK VQA. So those depend heavily on the web documents. It was interesting to see the dichotomy when we used only one dataset or the other. That's in the paper of ObliX.

• 19:27Speaker 03

  Essentially, the pairs, image-text pairs, are good for the alignment. Just align what you see in an image with the corresponding text. But if you want to have more abilities to reasonate, it's better to have a higher proportion of web documents. With longer context. Also, this is not the only reason why we wanted to do it. Why we wanted to do it is because the image-text pairs are super noisy. So, well, the advantage of it is that it's super easy to collect. You just scrap a lot of HTML codes. And anytime you find an image with the corresponding alt text, you download the image and you pick the alt text and you have your pair. Building a web document is much harder

• 20:12Speaker 03

  because you have to clean properly the text. You have to check what you want to keep and what you want to discard. So it's obviously much harder. However, you have also a longer context for each image. So there is really a parallel to be made. And it's not the same type of data because on image-text pairs you have an image and the direct caption of it. On web documents, you have, well, this is essentially what you see when you open any website. So you have a text, then sometimes an image, another text, an image. And then the alignment here is weak in the sense that the text don't necessarily describe perfectly the image. However, they share the same context.

• 20:57Speaker 03

  So this is another type of data. And we also think that this diversity helps to improve the performance.

• 21:06Speaker 02

  How much? So this sounds good in theory, but you had no idea of knowing. I mean, I guess you talked to the flamingo office and they just told you that this is what they did.

• 21:18Speaker 03

  You mean like the proportion of, exactly. Even them, even them, they built their dataset. And they told me, yeah, we use this proportion, but maybe we could have used less or we don't know. So we didn't really know in advance the proportion of web documents you would need compared to pairs.

• 21:44Speaker 04

  We did an evaluation, though. So basically, we can control how much we sample web documents versus land pairs. And so we did a We did an experiment where we moved those probabilities a lot. It was very inconclusive. So there was no, like, we had a range of like, what was, uh, what was a good range for like, how much web documents we should have versus lay on pairs. But overall, past a certain threshold, it didn't matter too much.

• 22:17Speaker 02

  And when you, when you measure performance, do you split it out into things like individual tasks, like segmentation or detection or anything like that? Or is it just VQA?

• 22:27Speaker 04

  We don't have a detection or segmentation because the model is basically, like, it outputs text. So it's, we can't really evaluate on those benchmarks. But we, we did captioning, visual question answering, text recognition a little bit. But it was, it was done through captioning datasets or VQA datasets. And we did classification. So those are the three ones that, uh, three categories that were doable with the, with the setup, like the model we, the outputs we give. But I know that recent, QuenVL, for example, they use bounding boxes in the, in the datasets. So I am, uh, they can do detection, yeah.

• 23:08Speaker 02

  You also mentioned, by the way, uh, that resolution was a big deal for you. Uh, image resolution. And how did, how do you deal with that in Obelisk?

• 23:17Speaker 04

  So resolution is important when we, when you want to do OCR particularly.

• 23:22Speaker 02

  Because otherwise it's just fuzzy, right? If it's.

• 23:24Speaker 04

  Exactly. It's too small. If, if you can't see it, the model probably struggles as well.

• 23:30Speaker 02

  Well, not just that. Models typically see much smaller images than we do. Right? I don't know what resolution you guys have. It's like a resolution of like 480 by whatever, right? It's super small.

• 23:40Speaker 04

  480 is smaller than that, yeah. Smaller than that. It's 224.

• 23:44Speaker 02

  So you're gonna, you're gonna lose a lot of detail.

• 23:47Speaker 04

  Yep, definitely. On top of this, you have the vision model. And it outputs a certain number of tokens depending on the image you put, right? And above the model, we have a perceaser. So we reduce. The number of tokens that come out of the vision model. By doing this, we make it even less, like even harder, I guess, for the model to be precise on those very, very small details. So this is something that happened with the EDFIX, the version one. And probably, I mean, we're gonna, we're gonna improve on this for version two. But it's really, really important for OCR, that's for sure.

• 24:29Speaker 04

  We think it could also be important to, like, visualizing details, improving on those things as well. For example, it's like a finger or like a hand is a certain color or is doing a certain thing. If all your images are tiny, it's gonna be, it's gonna be hard for the model to pick up on that.

• 24:48Speaker 03

  So we are bottlenecked also by what is available on the open source side. For example, now Google recently released SigLip. And it's a clip, but there is a version of it. It's called SO Optimize. It's of size like 400 million parameters. And it is trained with 384 resolution images. So it's a bit bigger than the 224 that we had. So I think this is the largest resolution you can get with open source models. But we are of course bottlenecked by this. Usually, Google, they release like this version of SigLip, but they didn't release the better version of it.

• 25:31Speaker 03

  So we are definitely limited by this.

• 25:34Speaker 02

  Well, so it doesn't really affect, it sounds like it doesn't really affect ObliX.

• 25:39Speaker 03

  Yeah, exactly. So when creating the dataset, we simply downloaded the image with the full resolution. And after that, you resize them on the fly during the training. But certainly, one of the biggest challenges when making ObliX was dealing with all these images because they wait a lot.

• 26:01Speaker 02

  Aren't you tempted, so like to me OCR is extremely important. And aren't you tempted to like run some kind of extra data augmentation thing to say like, oh? You know, on the Obliques dataset, run some OCR pipeline on it, so that you augments your.

• 26:16Speaker 03

  Yeah, that's really interesting what you mentioned, because this is also one thing that we want to do in the near future. And also, people have kind of did that for NUGA, right? So NUGA is a model from Facebook, and they just tried to have a vision model that can read it. So they fed to the model PDF with the associated text, and the model is pretty strong. So maybe if we inject this data in our pre-training, it will definitely help. And this is also one of our.

• 26:51Speaker 04

  And this is one of the thrives we're exploring, to have a lot more OCR data. We have a team actually at Hugging Face that works on Document AI with Ross Whitehead. Do you see the team library for vision models? No? Okay. But yeah, basically, he's been working on Document AI and on getting a very strong open source model that can read.

• 27:15Speaker 02

  And is that primarily PDFs?

• 27:17Speaker 04

  Yeah.

• 27:18Speaker 02

  Screenshots of PDFs, or just raw PDFs? Is there a difference?

• 27:22Speaker 04

  I think screenshots, but I'm not familiar with the dataset yet. We may use it as well for our training in the future.

• 27:29Speaker 02

  It's interesting that documents obviously are a very, very important form of multimodality that is very OCR-heavy, very focused on charts. I feel like you could classify sort of three types of multimodal models. Like, one is the traditional classification types of models, the clips of the world. And then two is the VQAs, where it's a general image of like a webcam, you know, where it's like there's three people in this image and all that. And then the third would be like Documents AI.

• 28:01Speaker 04

  Yeah.

• 28:02Speaker 02

  I don't know if it's.

• 28:03Speaker 03

  You can combine them all, actually.

• 28:05Speaker 02

  Can you combine them? I don't know.

• 28:06Speaker 03

  But like for. Actually, I don't know. No, but like, uh, general model like GPT-4, it does all of this. Yeah, yeah. All these tasks. Yeah. Even if it's not like, uh, trained purely on classification, can classify the.

• 28:21Speaker 02

  Deeper the better, right? One God model to rule them all. Uh, I don't know if it's like a mixture of, you know, different, different models.

• 28:29Speaker 04

  Yeah, uh, GPT-4V, not, not like, probably built upon GPT-4, but adapted for images. Uh, that would make sense. But then it's like the, the DALL-E 3 model is different from, like it's separated to create different, to create images.

• 28:45Speaker 02

  Yeah, something they just introduced was, uh, now, now you don't have to switch modes, right? Now you can just kind of, um.

• 28:50Speaker 04

  Yeah.

• 28:50Speaker 02

  Do one, one model and it just does its own routing, which is kind of very interesting. And then the other was, uh, mentioned but not released was that they could add vision to GPT 3.5, not just adding it onto 4, like it's not a variant of 4. It is a pluggable vision module that you can kind of add to 3. They never released. Yeah.

• 29:12Speaker 04

  Yeah.

• 29:12Speaker 02

  Anything else that people should know about Obelix? Like, obviously, this is like the, the big work. You, you mentioned in our, in our prep that you expect it to last for a while because there's, there's a lot to mine from it.

• 29:22Speaker 03

  I think it's big enough to train, uh, large models. So we trained our ATB parameter model on it. So it is definitely sufficient for the next one, two years. We spent a lot of care curating, uh, the, the data, like regarding the text quality and the image quality. And I think we, so there, there is also a, an altern, and now an altern, alternative, sorry, uh, to Obelix. It's called, uh, multimodal C4, MMC4. It was, uh, published at the, at the, around the same time as us. However, we think we took more care to, uh, in the deduplication part, to deduplicate the images and the, and the text,

• 30:07Speaker 03

  and also based on the text quality. This is measured, of course, uh, qualitatively, just by looking and exploring at our documents, but also, uh, quantitatively, by looking at certain metrics like perplexity. We obtained good scores that match the, the best NLP, uh, only, uh, data set. This was a win for us.

• 30:30Speaker 02

  For someone who's never really dived into these data sets, I mean, I can open up a data set and manually look through these things. But how does perplexity, how is how do you measure perplexity in a multimodal data set?

• 30:41Speaker 03

  So perplexity essentially is something really simple. Is you take a smaller, small model and you feed them with the, with the token of your, of your text. And then you measure the probability of, of, of the, of the document. Of course, you normalize by the length so that everything is equally treated. And then the thing is that we obtained that we had perplexity scores that match the distribution from the documents from the pile. And the pile is documents that were taken from good quality sources, like Wikipedia archive and so on.

• 31:26Speaker 03

  It's not something that you can really scale. And however, we, we also noted that we obtained better perplexity scores than the ones from CIFAR, the big data set or OSCAR.

• 31:39Speaker 02

  Based on your own measurements, right? Because obviously the multimodal CIFAR people would not share.

• 31:45Speaker 03

  Yeah, just based on, just based on the, on the text. So yeah, I think this was, so this is how we computed perplexity and how we assess the quality of the data set.

• 31:56Speaker 04

  But you could also run a multimodal model on this and get the perplexity from it. It would not be measuring the quality of the text, but also would like the alignment would, would come into account.

• 32:09Speaker 02

  Alignment of?

• 32:10Speaker 04

  Of image and text, because it would be easier for the model if the text is very heavily related to the image to guess the next token.

• 32:18Speaker 02

  And then one more question, just about the, the whole process. Like, how long does it take to make obliques?

• 32:25Speaker 03

  So we spent a good time at the very beginning of the project, just simply to iterate on the pipeline, like how we collect HTML codes, how we clean them. We had to go through all of the HTML tags and look if they were important. So this is, yeah, an engineering part. You have to be really...

• 32:45Speaker 02

  It sounds very boring, but very important. It is boring and important, but that's how you get the good data.

• 32:54Speaker 03

  Yeah, but this is also why people don't do it.

• 32:58Speaker 02

  Yeah, but the industry has not converged on a shared set of tools that everybody uses for this. You're just parsing raw tags yourself.

• 33:07Speaker 03

  Yeah, we did that, yeah, because we found it was better. We passed raw HTML codes. So we had to clean the DOM tree, select the good HTML nodes, correctly extract the tags, the images, clean, so they duplicate. So as I said, there was just a good amount of time at the very beginning of the project just finding the pipeline. So maybe one month, but we were like one or two on this and it was really exploratory. And then for actually making the dataset, download all the images, do all the processing scripts and so on.

• 33:48Speaker 03

  I think it took us like up to two months.

• 33:53Speaker 04

  Yeah, but then there's also like iterations through the project where we're like, we think we should do filtering on this on top of what we were already doing. So we improve on the dataset as we...

• 34:04Speaker 02

  Something I would think makes sense for the industry is kind of an open source set of deduplication rules, because everyone seems to be reinventing this from scratch every time.

• 34:15Speaker 03

  For duplication, you have to do it all the time from scratch, because it depends on your original set of documents.

• 34:24Speaker 02

  Everyone draws from Common Crawl.

• 34:27Speaker 03

  Yes, but they are not from the same domes. But that's true.

• 34:31Speaker 02

  Yeah.

• 34:31Speaker 03

  Someone should take all the Common Crawl documents.

• 34:35Speaker 04

  Was it done recently?

• 34:36Speaker 02

  I don't even need the same exact rules. I just need to be like, oh, these smart guys thought about that. I should include that. Right, that's very simple. Like, you know, if you have 100 rules, someone else has like 80 rules. Maybe they have some that you don't have.

• 34:50Speaker 03

  Exactly, exactly.

• 34:52Speaker 04

  I think we did a little bit of this, like, because there's a bit of literature like spread around.

• 34:57Speaker 03

  Even the ones that we designed before for the dataset of Blue. Yeah. We took.

• 35:03Speaker 02

  Yeah, I'm sure you used a bunch of that.

• 35:05Speaker 04

  And ObliX is big, but the datasets that come for NLP right now are also like huge.

• 35:10Speaker 02

  Did you see the Together one from yesterday?

• 35:13Speaker 04

  Yes, yes, impressive, 30 trillion.

• 35:16Speaker 02

  So they said they had a raw dataset of 100 trillion, and they got a cleaned, high-quality dataset of 30 trillion, which means they kept 30% of Common Crawl, which is still too high.

• 35:28Speaker 03

  Yeah. So I think the idea of the project, and I think I like, I agree with this idea, is that everyone unsets the thresholds for the filters as they wish. So for each document, they computed the threshold, the filters, or no, actually, filter values for a set of rules, and then you can decide whether you want to keep the document or not. And then, so you define your own rules. So I think they remove like, as you said, the 70% of the dataset that is really like, you can't do anything with it. And then for the remaining part, they let people decide.

• 36:11Speaker 03

  But of course, if you actually want to train something on it, it will be much smaller, I guess, because you will remove a lot of things.

• 36:19Speaker 04

  But data is super important, and yeah, my point is also that it's still very early in multimodal. We're seeing now in NLP the. Thirty trillion datasets. It's really important. Mistral, the first thing they said when they when they released their rules is how impressed they were that they were capable of doing the data pipeline in three months. They didn't talk about like training the model or not. It's just like they knew what they were doing for this. It's straightforward. But creating the whole data pipeline, this is what took them a lot of time.

• 36:50Speaker 02

  Yeah.

• 36:50Speaker 04

  So I think that was that was one thing that that struck me when they made their announcement.

• 36:54Speaker 02

  Is it is it confirmed that they have eight trillion tokens?

• 36:57Speaker 04

  I don't know. They won't say. Could be this. It could be more. Could be more, I think. Given we have now an open source dataset of thirty trillion, I wouldn't surprise. I wouldn't be surprised that they have that they have more.

• 37:08Speaker 02

  Oh, I just keep coming up with questions. Uh, one more thing on datasets. Did you read the GPT four vision system card that they put out? They put out this paper describing a little bit of their process. Something like ninety five percent of the the labels for GPT four vision was augmented by GPT four itself. And I was just curious, like, how much room is there for open source augmented datasets?

• 37:33Speaker 04

  I think there's a lot of room. I think there's a lot of room, and I think synthetic data works a lot. Like, works great, particularly in multimodality. Uh, recently, the recent papers have been using, for example, Leon Coco instead of Leon. This is not, this is just captions on Leon created from Blip. So it's not, it's not even like super like extraordinary, but it does bring more performance with a lot less example because the alignment is is more straightforward, I guess. And we've been observing this recently because we've been, we've been using it in our recent experiments. I think the potential for synthetic data in multimodality is very big and very underutilized right now.

• 38:17Speaker 03

  Even DALL-E three, they they said that they used heavily synthetic captions to train their model. Also, for multimodal, foundation multimodal models, like GPT, they train on synthetic captions.

• 38:31Speaker 02

  Yeah, actually, I think I was referencing the DALL-E three paper, not the GPT four vision paper. Yeah, because they didn't, they didn't actually put out a paper for GPT four vision.

• 38:39Speaker 04

  Yeah.

• 38:40Speaker 02

  Cool. And then, so you created Obelix and then you trained Idefics?

• 38:46Speaker 03

  Yeah, on it, on top of it. In addition, so we trained Idefics on Obelix, but also on other data sets, like Lion and Public Multi-Model Data Sets, which is just sets of data sets that were open source at the moment. Like conceptual captions or...

• 39:04Speaker 02

  And yeah, just could you take us through just the Idefics process? You created a smaller version and then you scaled up to the full Flamingo 80 billion size.

• 39:15Speaker 03

  Actually, it was the other way around.

• 39:17Speaker 02

  Really, really?

• 39:18Speaker 03

  Well, we tested that it worked at smaller scale, of course. But we did not train fully our small model before doing the big one. We fully trained our big model.

• 39:31Speaker 02

  That's unusual.

• 39:34Speaker 04

  Yes.

• 39:37Speaker 04

  Yes, they were like, they were training pretty much at the same time. But we needed to launch the big model before also in terms of like timing, because it took a long time to train. So it was more like managing computer resources. But through the whole journey was a bit longer than just this moment when we trained a three Idefics model, because we started with the objective of matching Flamingo's performance. But the open source models that are out there, they're just not good enough. So we have OPT, you have GPT-Neo, but it's just, it's just so below Chinchilla that it's almost impossible to reach the performance. It's only when Llama came out that it started making sense and that we started matching the performance.

• 40:20Speaker 04

  And from there, we were able to train the big Idefics models. A long journey, I think, because we had a lot of things to learn, because we had quite a lot of instabilities. We shared a blog post about...

• 40:34Speaker 02

  The checkpointing every 250 steps?

• 40:36Speaker 04

  Oh yeah, we were checkpointing every 250 steps, but we had to restart a few times.

• 40:41Speaker 02

  Was that the instability you're talking about? Or this is just something else?

• 40:45Speaker 04

  That was the final training, where we still had the instabilities, but a lot less. Like before this, we were struggling to train the model at all. What saved us at that moment was the query key layer norms. I don't know if you've heard about this, but basically, there's a paper from Google where they scaled up the vision transformer to 22 billion parameters. And they needed this trick to keep the stability of the training. Without this, if I remember well the mechanism, you like you would get something along like a hard attention. And once you get this, like the model would get very unstable. So you needed to normalize the queries and keys to basically avoid this. Anyway, so when we did this, we were able to train further.

• 41:29Speaker 04

  And that was really useful, for sure. So query key layer norms, if you want stability.

• 41:37Speaker 02

  It sounds like a trick that is repeatedly applied whenever you have instabilities. You just do a dir norm or softmax.

• 41:45Speaker 04

  Yeah.

• 41:48Speaker 03

  Because it's parameters that explode.

• 41:51Speaker 02

  Yeah, yeah.

• 41:52Speaker 03

  Become too big. So you need some sort of regularization on them. At first, you have to inspect which parameter explodes first.

• 42:03Speaker 02

  Yeah.

• 42:03Speaker 03

  And then you put a regularization on them.

• 42:06Speaker 04

  But it's really tricky to see. When you go in the gradients and in the activations and you try to see where it blows up, when it blows up and why. Everything is interlinked. It's really hard to pinpoint one particular layer or... So it was a tough one to crack. But very interesting. It's also hard because you never know exactly. Once, when you're in the process, you don't know where the instability comes from. It could come from bad data, it could come from a bug, it could come from the size of the model, the learning rate that's too high, the warm-up that's not... Like, there's a lot of hyperparameters

• 42:50Speaker 04

  and potential bugs that can come into account. And debugging is very, very tough.

• 42:55Speaker 02

  So do you have a checklist of what do you look at when you see loss explode or whatever?

• 43:01Speaker 04

  You see the loss explode, you look in the activations, possibly the gradients, to see where it blows up.

• 43:07Speaker 02

  Across all your parameters?

• 43:09Speaker 04

  Uh, yeah, you find a way to aggregate this. Otherwise it's tough. If all previous solutions fail, because this is the hard part. This is the hard part, but it tells you a little bit where it's happening. So that gives you like where it's happening-ish on the model. And so what's to blame? And then you have a wide range of things to blame, but less than before. So you can look for a bug. You can look for, for example, normalizing some layers. And you can look into the data to see if you have like very bad data that's impacted.

• 43:45Speaker 02

  That's what I would look at first, right?

• 43:47Speaker 04

  Exactly. I would look at that too.

• 43:50Speaker 02

  Is this what weights and biases would do for you? Or is there one integrated solution that kind of?

• 43:56Speaker 04

  You would wish?

• 43:57Speaker 02

  Yeah.

• 44:00Speaker 04

  No, to see the activations and the...

• 44:03Speaker 02

  We go from activations to then like inspect datasets and then look at, you know...

• 44:09Speaker 04

  What weights and biases would get you, like the parameters?

• 44:13Speaker 02

  It's just logging, right?

• 44:15Speaker 04

  It's just logging, like you have tons of them. You can't really do much with it. So we had a tool where we would like log them periodically. We had a script that would aggregate them and display them for us in a nice way. So that like interpretable way. So that we could try out and see what mechanisms were and could be impacting the instabilities. But yeah, it was a very, very interesting journey for sure.

• 44:41Speaker 02

  Yeah, and you published knowledge sharing documents and a memo.

• 44:47Speaker 04

  Yes.

• 44:47Speaker 02

  There's some interesting detail there, but obviously not everything. Anything you want to highlight just first for listeners on that one? Obviously I can send them the link, but...

• 44:56Speaker 04

  I'm sorry, hi, I'm sorry.

• 44:57Speaker 02

  Just any other like big discoveries or learning, like you talked about the query key layer norm.

• 45:04Speaker 04

  Query key layer norm was the big aha moment. But this and there's also the, this is one we fixed afterwards. But there's like in the mask, in the image mask, there was like a little information leak. In that instead of attending to all the images, instead of attending to none of the images, sorry. For a few tokens, very few of them, it would attend to all of the images. Like basically, you tell them, you go in the attention and you have this masks, this mask that's like, like don't attend to anything. But in effect, it's like attend to one over n, right?

• 45:47Speaker 04

  And it's not like, it doesn't prevent you from training, but it doesn't help. And it's, yeah, it's better, it's better to fix it for training for sure.

• 45:57Speaker 03

  Yeah, sometimes you really have to go through all your code base to.

• 46:00Speaker 02

  Because you're doing gradient descent on something that has no information.

• 46:04Speaker 04

  Yeah.

• 46:06Speaker 04

  Yeah, no, it's tricky because this, like for example, this, it would not have an impact if you only train on web documents. Because the documents are long.

• 46:17Speaker 04

  But it would have an impact if you train on image text pairs and you pack them together. Because you're attending to images in a document that has nothing to do with it. But you can still train with it. It's, you can still get a very good model out of it. It's just a lot like it's more painful. And you don't get like, I think we can, we can get better performance definitely without the bugger.

• 46:38Speaker 02

  Yeah, interesting. You mentioned, you know, just in terms of like the baseline foundation models that you had. Lamo is a big breakthrough on the, on the language model side. But then you also, you didn't talk as much about the vision porter side of things. I actually had a question from Joseph from the RoboFlow episode. Where he talks about, where you mentioned that the larger the clip, the better results. But in your final memo, you actually went for a smaller version of clip. So Eva Clip versus Lion 2B. Does this, does this ring a bell? Basically, it was kind of like unintuitive, like the clip choice there.

• 47:11Speaker 03

  Yeah, so Leonti is the data set on which our clip based model was trained on. And our clip model was indeed like 400 or 600 million parameters. And it's true that the Eva Clip 1 is of, the biggest Eva Clip 1 is of 5 billion parameters. So definitely at the beginning of the training, we saw a big boost using Eva Clip. However, at that time, we had, we still had instabilities to train this big Eva Clip. We are not sure exactly why, however, we fixed it.

• 47:51Speaker 03

  And now that we, like for the next, for the V2 version of EDFX, now that we can train longer, we actually saw a boost by using Eva Clip instead of the previous, instead of the previous clip, a small clip that we had. However, we now think that Eva Clip is under trained. So it means that even if it's really big, we can obtain the same performance with smaller models. So there is this Sigclip model that I mentioned just earlier, that by Google, that is much smaller, 400 million parameters, and that is, that is more efficient.

• 48:34Speaker 02

  You're choosing that as your base?

• 48:35Speaker 03

  Yeah, we did, we did an ablation, Eva Clip versus Sigclip. And actually Sigclip was a bit slightly better. But the thing is that it's much faster for the inference and also for the training, because there's less, less fewer parameters.

• 48:49Speaker 04

  One, one thing is also that Sigclip is a higher resolution. So that has, that has an impact for CR tasks.

• 48:57Speaker 03

  3884 instead of 224.

• 49:00Speaker 04

  Yes.

• 49:01Speaker 02

  That's, that's great. Yeah, maybe we should talk about EDFX 2. So we're going to time this podcast release with, whenever you guys are releasing it. I actually had no idea you were working on a V2. I just came in wanting to talk about your old work, but obviously you're still doing active research.

• 49:17Speaker 04

  For, for EDFX V2 or V1.5, whatever we call it, the major axis we wanted to improve on was the

• 49:29Speaker 04

  The image resolution, the base model. We wanted a better base model and a smaller one. Oh, sorry, pre-trained language model. So we've been using the Mistral one. We wanted to iterate a little bit also on the data to have better filters on the obetics, sorry. Better synthetic data for the image textures.

• 49:56Speaker 03

  So essentially, iteration on the data by replacing original lion pairs by synthetic captions to have a stronger alignment. Cleaning a bit of obetics on the perplexity. So it's not removing too much, but like potential bad data. Also, yeah, using just better pre-trained models. So for the band bones, we use like a better clip, C-clip. We use also Mistral that is better than Llama 1. We are right now changing also the modeling. So moving away from the Flamingo architecture to something that has fewer parameters.

• 50:38Speaker 03

  Instead of incorporating the vision component directly into your LLM by breaking and adding cross attentions at each layer or every N layer. You can instead take your vision encoder, take the embeddings out of it, make them, feed them to linear layers and feed them directly to the language model. And this works quite well. And this contains fewer parameters and it's much easier to train. So we are currently trying to do this. But what we can say is that without this new modeling,

• 51:20Speaker 03

  just by iterating on the better data and better pre-trained models, we are now matching the Flamingo 8CB performance with a 9B model. So this is already a big improvement compared to our first version, without even touching the modeling part.

• 51:40Speaker 04

  I think also one of the big improvements with the new Edifx is the licensing. The issue with Edifx was it's based on Llama and the license is not commercial. So now with a model that's based on Mistral and Siglib, most probably, this is a lot better for anyone that wants to use the model commercially. The model will also be smaller, so a lot better for inference. And hopefully, we can beat the performance of Edifx ATB. That would be really good.

• 52:16Speaker 02

  So you're only producing a 9B?

• 52:18Speaker 04

  Not exactly 9B, because we're taking off parameters. It should be about 7.5B. Right now, the focus was really better data, smaller open source models, but better. And resolution, I'm improving on this as well. That was the focus.

• 52:38Speaker 02

  You mentioned in our prep as well that there are some topics that you're paying particular attention to, like hallucinations. Maybe you could talk about the topics that you are finding are particularly areas of concern with multimodal models.

• 52:54Speaker 04

  So we've been using hallucination at the beginning as a broad term. And we realized that it was better to categorize it a little bit more specifically to some categories that were more targeted. So for example, like there's the object attributes, where you would have like a small attribute, like let's say the hand of a person that's a certain color. And the model would be like, it's yellow when it's red. So that would be one. There's like objects that are not there, but the model thinks are there. Like when the model is trying to reason with different elements in the picture, but it gets it wrong. So you get like comparisons, kind of hallucination, counting.

• 53:36Speaker 02

  Oh my god, yeah.

• 53:37Speaker 04

  You have the environment. So it would talk about like the object or the person in the picture, but it would get the whole environment behind wrong. And a few others that I don't have in mind right now. But basically categorizing those I think is important because it helps you target the type of data that's missing or the type of fine tuning that you should do afterwards that's missing. And so this is going to be very useful for us in building future data sets. I'm not sure if we'll be able to incorporate all those like those data sets we've been thinking about for the V one point five. But we'll definitely do this for the iteration after.

• 54:19Speaker 04

  And the goal here is really to get to something that's on par or better than what is currently done in closed source. Like G P T four V doesn't hallucinate as much on those topics. And you want you want the the open source models to to at least match this. But it's still like it's still an open research topic because those models still do this. If you push them a little bit, if you ask the like some specific questions, like they will hallucinate things in the picture.

• 54:47Speaker 02

  Yeah, you mean G P T four, including G P T four V. Yeah, yeah. I have tried this, by the way, I tried to use it to interpret like a menu and it would just make up menu items and make up prices and.

• 54:57Speaker 03

  Something really interesting is that G P T four, even much better and bigger models like G P T four shows the same failure cases as our model. So for example, it means that the way we are training the training the models now is not ideal. And especially for example, counting you take this task. Even if you train on web documents or image textures, you will never really find this task in your training data. You will always have a picture and maybe a caption of one apple or two computers. But you would never get to like ten, fifteen, twenty.

• 55:41Speaker 03

  It doesn't doesn't really happen. So you you don't learn this ability just by training on the on the image textures or web documents. So what you have to do is to create your own data sets that target specific tasks. Like for example, create one specific. Specifically for OssiAR, create one specifically for counting. Create one specifically, I don't know, to challenge the model on hallucination, some types of hallucination.

• 56:12Speaker 04

  A sort of flan, but for multimodal. And I think in multimodality, those hallucinations are even more unforgiving than what they are for NLP. In that, when the language model is making up facts, you can think that it got a little bit wrong or it's making up a story, but when a multimodal model tells you there is a teapot in this picture and there is none, it's very obvious, it's very obvious, very quickly. If you're the user of this model and you receive this, you are having a hard time trusting it for anything. So this is, this is one of the big, big things to tackle. And for hallucinations in NLP, a fine tuning that has helped a lot is the reinforcement learning with a human feedback

• 56:56Speaker 04

  or AI feedback, and this is still very early in multimodal. We're not sure exactly how much we can improve with those, with those types of datasets. But it's likely that it would help, it will help the model understand uncertainty to a more refined level. And so improve on all those types of hallucinations ultimately.

• 57:16Speaker 02

  And you said, you know, Prepp, as well, that you're measuring this against other closed source models like Bard. Like basically, do you have your own internal benchmarks that you're running?

• 57:25Speaker 04

  No, no, this is, this is mostly qualitative. It's more like, does it, like, we know it happens in ours because we've played with it. Does it happen with theirs? And yes, it does. The, the thing is for ours, the evaluation datasets that we use or that we've used for like EDFX so far, they, like, they don't measure hallucinations that much. It's not, it's not targeted for this. New datasets that are coming for evaluations are exciting. And I think we'll go in this direction and will help us evaluate hallucinations and different types of hallucinations more accurately. But right now, at least for EDFX, even with those evaluation benchmarks and even if this helps,

• 58:07Speaker 04

  like obviously if you get a very bad scores, your model is hallucinating, like could be hallucinating things. It's not targeted to this and we were flying blind a little bit on this, on this topic.

• 58:19Speaker 03

  But there are definitely benchmarks to evaluate hallucinations currently. So I think about sugar crepe, for example. So you just have an image and two captions of it, and the model has to select the best one. Or vinogram, I think. So actually, the captions are made such that it's tricky.

• 58:42Speaker 02

  Vinogram, like the common sense NLP one?

• 58:45Speaker 04

  Yes, this one is really hard.

• 58:47Speaker 03

  But you can have one with images, yeah. I think this can be a great way to evaluate hallucinations of the models. However, it's not really commonly used. So right now, on the recent models, like Pali, for example, they report their numbers on classic evaluation benchmarks. So I think it needs to be more widely adopted, this kind of new benchmarks.

• 59:15Speaker 02

  I think the last topic that we prepped was just overall, like why is it important for there to be OSS multimodal models? Like, what can people use them for? What can it be useful?

• 59:25Speaker 04

  My understanding of this is that ultimately, you want foundational model that understand the world similarly to the way we do. And multimodal models, like, understand visual data a lot better than language model, obviously. But it means they provide a better foundational backbone for other tasks in general. I think in the future, when you want to pre-train a foundational model, you will want to have it multimodal. So this is why it's important now, and it will be important in the future. And this is where everybody is going. Because training on text data, it gets you really far, as we've seen,

• 1:00:09Speaker 04

  but it only gets you so far. And it can only adapt on subset of tasks that we do every day. And you need models to be able to understand vision if you want them to be helpful for, I don't know, robotics, for example, and a bunch of other use cases.

• 1:00:29Speaker 03

  I think, for example, even for medicine, you can have tons of applications with a vision input. So you just take an image of whatever, ask...

• 1:00:43Speaker 03

  Panser. Yeah, yeah, no, honestly, you can ask for help, for help with your model, or even in your everyday's life, how to build this table. You just take a picture of what you have and you ask the model. You just take a picture of even something handwritten, ask things about it. Like solve the exercise shown in this picture. A lot of things that you can't do with text-only models, that you can do with multimodal models. So definitely, it's a big step forward. It's still a bit immature, and we have seen that because, yeah, because of all the allegations we mentioned,

• 1:01:27Speaker 03

  and because it's still early, are still trying to unlock some abilities for these models. Yeah, we believe in less than two years, we will only have multimodal models in the future.

• 1:01:43Speaker 02

  Like they will overtake everything?

• 1:01:46Speaker 03

  Yes.

• 1:01:47Speaker 04

  Also, right now, it's a lot of image-text models, because it does most important things that you want, without requiring too much compute. I think it's more computer fusion than if you would put video in it. Ultimately, I think we'll need video to be incorporated into pre-training as well. I'm quite excited about what it could look like. Probably the ones that are more advanced, like the ones that are the most advanced on it are probably the self-driving cars. They're doing like very, very interesting work there, but it's all closed source. Oh, no, I think, was it Koma AI that released Gaia? Or something like this? But yeah, basically, like video generation. So I think that's very interesting.

• 1:02:28Speaker 02

  Is video, isn't just that, like a series of frames of still images? What is so different about that?

• 1:02:36Speaker 03

  It's hard, because you need to encode each frame, or at least select the, say, how many frames you want to integrate per video. Also, one big change is like the length of the video.

• 1:02:53Speaker 03

  Really arbitrary, while if you have an image, you have always the same number of tokens associated to it. So all of this, plus the fact that having a big, a big video data set is really challenging. Because you can't, for example, scrap YouTube like that, it's harder. So yeah, all of this make it hard to, to build a model with videos.

• 1:03:18Speaker 04

  It's also extremely heavy. Already, when you, when you go from a text data set to an image text data set, the, like, how much it weigh is, is so different. It means, it means you have to that the data pipeline, it means the product, like the data set preprocessing takes more time. If you go with video, like you, you, you, you, you do, you do this like 10x, if not, if not more. For a transformer, it takes tokens into, as inputs, a sequence of tokens, right? And for text tokens, you, you get like one token for that, one token for like a subset of text or word-ish.

• 1:03:56Speaker 04

  For an image, depending on the resolution you want, you can get from like, like you can get a lot of tokens for a single image. For a 2-2-4 image, you'd get, I think, 2-5-6 tokens per image. If you want to increase the resolution, it goes, it goes up fast. For SigLip, I think it's upward 700 tokens for a 384 resolution. So imagine this, and then you make it a video. So you have like, it, it, it gets really tricky and you have to, so you have to create a bottleneck, right? You have to pack those images together and, and output us like a sum of tokens. That's what's possible with Flamingo, it also means you have like this bottleneck there.

• 1:04:38Speaker 04

  It's a lot of work and right now, I think the trade-off is not yet worth it. Cause we have a lot more to do with just image text, but it will be, it will be at some point, I'm, I'm pretty sure.

• 1:04:48Speaker 02

  That's all the questions I had. Any final call to action? You want people to go somewhere, check out the models, check out the papers?

• 1:04:56Speaker 03

  Yeah, essentially, so we are excited for the second version, which will be released about the time of NERIPs. And yeah, we think that Obelix can be useful in the next years, even if we continue updating the model. I think the dataset will remain pretty much the same. And we will keep iterating on better modeling and better data.

• 1:05:24Speaker 04

  And if you want to follow our work, we're the Hugging Face M4 organization.

• 1:05:28Speaker 02

  Let's talk about naming. What is M4? And then maybe you should mention Obelix and I've heard of X.

• 1:05:34Speaker 03

  I think so at the beginning of the project, it's like massive, multimodal, multilingual, multitask model. But then we dropped the multilingual. And but it is still massive, it's still, well, not that massive anymore, since we are trained to go smaller, go smaller. Multitask for sure.

• 1:05:57Speaker 04

  I mean, the datasets are still massive, so that's good.

• 1:06:00Speaker 02

  It should be still multilingual, no? If you train it on Common Crawl?

• 1:06:03Speaker 04

  Train it on Common Crawl, it's just not targeted for a lot of different languages. Like, we didn't take care of having multiple languages. But there's probably, like, there's definitely other languages in there. Yeah, the naming, like, it changes through the project, but we kept it. The idea was also to have four modalities. So it was fitting at the beginning. At the beginning, we wanted audio, video, text, image. We thought that audio and video, video is just a lot of compute for not that much results right now, like, results right now. And we were, from the moment we decided to reproduce Flamingo, we dropped video and audio, which was not immediately. This started as a different project.

• 1:06:45Speaker 02

  Yeah, what's wrong with audio?

• 1:06:47Speaker 04

  It's still very, very heavy and not necessary like right away. Because you can take the text tokens, plug it into an audio model after, and it will just spit out the audio. Ultimately, I think it will be good because there's different data as well. Like, when people talk, it's not the same as when people write. So I feel there's a lot of interesting data to have with audio. Again, not worth the compute right now, I think. And also, you would want, like, this model to be integrating everything with, like, a given size and then be applicable to whatever. You give it to your robot and you're like, what?

• 1:07:26Speaker 02

  Yeah, and to close the loop there, Obliques was an Asterix reference that you

• 1:07:31Speaker 04

  Yes.

• 1:07:31Speaker 02

  made into a backronym or whatever.

• 1:07:34Speaker 04

  Into an acronym, yeah. It was really hard to find the acronym. I wanna, I wanna stress this out. I thought about it like for like days. We were brainstorming it for, for like a few days. We had to cheat at the end a little bit. We took the S of cross attentions and added it to.

• 1:07:50Speaker 02

  Yeah, cross, cross attentions.

• 1:07:52Speaker 03

  The other problem is the F of Flamingo, right, for Idafix?

• 1:07:56Speaker 02

  No, that's right.

• 1:07:57Speaker 03

  We are, yeah, but we are moving away from the Flamingo architecture.

• 1:08:00Speaker 04

  Oh, for the, for the next iteration.

• 1:08:02Speaker 03

  For the next iteration, it wouldn't be really accurate, but whatever.

• 1:08:06Speaker 02

  Okay.

• 1:08:06Speaker 04

  Uh, but yeah, the acronym works only for the first version, uh, cause if I remember well, it's image aware decoder enhanced à la Flamingo, uh, with interleaved cross attentions. Yes.

• 1:08:20Speaker 02

  Yeah, it's a very, very, uh, very impressive project and, uh, you know, the, we, we were talking about it even before the GPT-4 vision rollout that like this is the most impressive sort of open source uh, reproduction and, uh, it's, it's amazing that you're still continuing to work on it. Uh, I think a lot of room left in uh, Obliques to, uh, to keep mining that, uh, those rocks. And there's, there's a lot of learnings as well in, in multimodality. I think it's a very important area of research. So.

• 1:08:50Speaker 04

  Thank you.

• 1:08:51Speaker 02

  Yeah, thank you.

• 1:08:55Speaker 02

  Hello, hello. This is swyx coming in from the editing room in 2024. If you're listening in this far, you're definitely a true fan. Thanks so much and I hope you enjoyed that conversation as much as I enjoyed recording it.

• 1:09:08Speaker 02

  Um, we recorded that conversation on Halloween in 2023 in the hopes that we would be able to release it, um, at NewRips or around NewRips with Idafix V2. V2 was supposed to be updated with a new base model, which is gonna be Mistral and a bunch of other data set updates. And a couple of things have happened, uh, but you know what? Hey, let's just get Leo back on to talk about it.

• 1:09:32Speaker 05

  Hello from 2024. It's Leo. Just wanted to add a few things since the recording was done a while ago now. We haven't trained the model yet, because we're trading a lot more on data. So we're making progress on OCR, on image-to-code capabilities, and we want to be a lot more thorough in the image text pairs dataset that we use. I don't know if you've heard, but the Leon dataset has had an issue with T-SAM images, and so we want to get ahead of this and fix the problem before we start the training. But we will start training soon. In the meantime, we released website. It's an image-toHTML dataset.

• 1:10:13Speaker 05

  The idea behind this dataset was to show that we could create a very useful synthetic dataset at scale with open source models. Hugo spearheaded the effort, and so he used Mistral and the DeepSick coder model to generate the pairs of screenshots and HTML code. And then we find tune an early version of IDFX 2.0 on the dataset to have a demo. You can find the model on the hub. You can find the demo on the hub, and you can find the dataset on the hub. So definitely go there and check it out.

• 1:10:57Speaker 05

  I think some people have already started training their own model on it. But it's been very well received, and so we think we're going to do a lot more small releases like website in the future. Basically, switching from releasing everything in one package with the model trained to releasing the datasets, architecture, and training insights that we get along the way, and then releasing the model. So stay tuned. I hope you will enjoy the podcast, and thanks Sean for having us.