AI competitions & cloud resources

In this special episode, we interview some of the sponsors and teams from a recent case competition organized by Purdue University, Microsoft, INFORMS, and SIL International. 170+ teams from across the US and Canada participated in the competition, which challenged students to create AI-driven systems to caption images in three languages (Thai, Kyrgyz, and Hausa). Featuring: Matthew Lanham – Website http://matthewalanham.com, X https://x.com/MatthewALanhamMark Tabladillo – LinkedIn https://www.linkedin.com/in/marktab, X https://x.com/MarkTabNetDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Purdue University’s Krannert School of Business https://krannert.purdue.edu/masters/business-analytics-and-information-management/Master the basics of Azure: AI Fundamentals https://learn.microsoft.com/en-us/users/sandramarin/collections/zopanqdn7w1p1Azure Architecture Center https://learn.microsoft.com/en-us/azure/architecture/SIL International https://www.sil.org/The bloom-captioning dataset https://huggingface.co/datasets/sil-ai/bloom-captioningBooks “Applied Machine Learning and AI for Engineers” by Jeff Prosise https://www.amazon.com/dp/B0BM35KY4CUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Daniel Whitenack?

  Hi everyone, this is Daniel coming to you with a slightly different episode of Practical AI this week. Recently Purdue, Microsoft, informs and a few others put on a case competition, which included student teams from across the nation, around 170 something teams, all working on a shared task related to image captioning. This is a task where an image is input to a model and then the job of the model is to output a text caption corresponding to that image. And I had the privilege of getting to be one of the judges for this competition. And I took the opportunity to interview some of the sponsors and the participants in the challenge.

• 0:42Daniel Whitenack?

  Also, it was really fun because the competition used some of our data, the data from SIL International in the Bloom Captioning data set, which includes image captioning data for a lot of languages. But specifically this competition focused on image captioning in Thai, Haas and Kyrgyz. So I hope you enjoy the discussion of this challenge and here we go.

• 1:15Speaker 01

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive and accessible to everyone. Subscribe now if you haven't already, head to practicalai.fm for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out at fastly.com and to our friends at fly.io. We deploy our app servers close to our users and you can too. Learn more at fly.io.

• 1:51Daniel Whitenack?

  Welcome to a very special episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with S I L International. And this is a very special episode because I'm here at Purdue University, judging a really interesting case competition, Data Analytics for Good, that's sponsored by Purdue University, Microsoft, S I L International, my organization, and Informs. And I'm here with Matthew Lanham, who is the Academic Director of the M S Business Analytics and Information Management Program, or BAME Program, which we've had the privilege of getting to know each other over the past few years. And it's really cool to collaborate and judge this competition.

• 2:34Daniel Whitenack?

  Matthew, could you tell us a little bit about it and how it came about?

• 2:38Speaker 02

  Sure. Yeah, so as the Academic Director for the Business Analytics Program, my job is basically trying to make sure our students are involved with analytics and data science competitions. And over the last seven years since we've had this program, our students have won or placed in many of these national competitions. So we've got a really well-established brand and name out there. And we thought, hey, why don't we create our own national data analytics competition? And let's do something that's for good, not necessarily just focus on trying to make money.

• 3:09Daniel Whitenack?

  Yeah, yeah. So tell us a little bit about the actual problem that the students are working on, and maybe a little bit about like the mix of who is involved in the competition from across the nation.

• 3:20Speaker 02

  Sure. So the actual problem is sponsored by your company, S I L International. And basically, what they're trying to do is use natural language processing to do image captioning, which is not a trivial task by any means. So when we put this problem out there to the students, they're like, oh, my gosh, what is this? And the great thing is, is it's not something that you would see in a traditional N L P course, this kind of problem. And there's been a lot of great learning involved. Overall, we had 172 teams across the nation apply and register for the competition. And there were 36 universities that were represented. Two of those were outside the United States.

• 3:58Daniel Whitenack?

  Wow, that's, yeah, that's really great. Yeah, the competition, so this image captioning, it's been cool to see because recently, S I L put out this data set around image captioning. And it was convenient timing because about a week or so later, you reached out and said, hey, we're running this cool case competition. Do you have any cool data sets to work on? And that worked out really good. I think I've heard. Students, you know, all learning a lot about natural language processing, but also the world's languages. So when you try to do image captioning in Thai, for example, there's no spaces and you can't tokenize words with just like spaces. So just even realizing things like that has been, uh, has been quite interesting to see for students.

• 4:40Daniel Whitenack?

  And I know I've, we're sort of halfway through the day of judging at this point that we're recording, and I've already been surprised and encouraged by a lot of the, a lot of the solutions. One of the other sponsors of the, of the competition is INFORMS, which I'd love, you know, people to know a little bit more about what INFORMS is generally, because it is, you know, a vibrant and, and large community. Could you tell us a little bit about what that is?

• 5:05Speaker 02

  Absolutely. So the INFORMS stands for the Institute for Operations Research and Management Science. It is my favorite professional organization. So it's been around for, for many years. And, you know, we used to call it Operations Research and Management Science, but now we refer to a lot of this stuff that we've done for years as Analytics and Data Science. Within INFORMS, there's also a certification program called the INFORMS Certified Analytics Professional, or CAP. So I'm a CAP. I was one of the first CAPs many years ago. And basically the whole idea with the CAP is you don't have to be a, a technical person to get this CAP, or you don't have to, or you don't have to be just like the, the business person. It's really for everybody. And the whole idea is this, um, how it came about was

• 5:49Speaker 02

  the people at INFORMS worked with, uh, business professionals from all different areas of Analytics and Data Science, Operations Research, to identify what are the key kinds of tasks that you would do as a professional. And what they did is they came up with basically seven domains. Business problem framing is the first one, then analytical problem framing, knowing your data, methodology or approach selection, model building, deployment and lifecycle management. And if you guys, if the audience hears those kinds of things, they're probably thinking, hmm, that sounds a lot like CRIS-BM that we've all heard about, you know, in school or at some point following a process. And that was the thing is a lot of times when we're working on these problems, you gotta follow a process. And they kind of extended that CRIS-BM framework.

• 6:33Speaker 02

  And there's just a lot of tasks within there that we Hope people are aware of and think about when they try to develop solutions and practice.

• 6:41Daniel Whitenack?

  Yeah, and we've been utilizing that framework in the judging here at the competition, which has been really, really useful, I would say, to consider these different elements of the process. And maybe, how would you say, after working with teams in this process, you know, for years now, actually, like, thinking about these different elements, how do you think that kind of rounds out someone's view of like, an actual, solving an actual business problem with like AI or analytics or data science in ways that maybe is sometimes neglected in a lot of just sort of process when you step into a problem? What are the main areas that you think kind of stretch students? But then, as they go into the professional workspace, how do you think that sets them up for solving real-world problems?

• 7:26Speaker 02

  Great question, and this is why I just love Inform's cap and why we make our students follow the Inform's cap when they do projects with companies, is because you'll see a team that maybe they're the data science team, the real technical team, and they love to get into the nitty-gritty details, and there's absolutely nothing wrong with that. But at the same time, you need to know your audience, and I think just following those seven domains, those seven domains for Inform's cap is important, because before you even get into the nitty-gritty details of your problem, you need to be able to say, well, how does, what is the business problem here? And then, how do we frame it? What are the possibilities of framing it into an analytics problem? And then, so that's the front end of the thing, right? And then, you'll get in the middle part, which a lot of the stuff that we would talk about as data scientists, the data, the methodology, the model building, all the stuff that we really like to do.

• 8:11Speaker 02

  But then, the last part of it, the deployment and lifecycle management, that's so key, right? So that's when you get into architecting and make, developing the pipelines, all the stuff that I know you're an expert in, it's just so key. So basically, that's what the Inform's cap's doing, is to say, hey, let's lay all this out to make sure that when we architect a solution, we design a solution, we try to create a solution to our problem, that we've thought about all of these things, and we haven't missed anything along the way.

• 8:36Daniel Whitenack?

  Yeah, well, thank you so much again for helping organize this. Matthew, it's been, it's been a pleasure, and yeah, really just appreciate your work on this and also your work with Informs.

• 8:45Speaker 02

  Thank you, Dan.

• 8:46Speaker 03

  I want to share exactly how the funding happened.

• 8:51Speaker 03

  It was just about Azure and the SOX? I mean, what is this? From Microsoft?

• 8:57Speaker 02

  Oh, oh, yes. So from Microsoft. So that's actually a really important point. So one of the interesting things about this competition that's different than all other competitions that my students have participated in is we designed it where there's three phases. And Phase Two is where they actually work on the problem provided by Dan's company. But the Phase One is we wanted to actually provide some training on cloud services, which a lot of people in the industry, they know you've got to be familiar with these services if you're going to architect a solution and put it into practice. So Microsoft offers free training on their Azure AI. Basically, everything's free if you're a student. And they also offer students free practice exams and certification vouchers, which is amazing. So we told them, they said, you know,

• 9:42Speaker 02

  we would like, we could get a whole bunch of students to participate in your training events if we can kind of piggyback off of you. And they said, absolutely, we love this. So that's what the students did in the Phase One was they had some training from Microsoft professionals. Some of them even set for certifications. And then in Phase Two, the goal was to try to apply some of those web services for this particular problem. The last phase, Phase Three, is when they, the top teams that performed the best in the Kaggle competition would come on campus, present their solution, and then show how they could follow the INFORMS, the seven INFORMS CAP JTA's to architect their solution. That's how it all came about.

• 10:17Daniel Whitenack?

  Awesome, yeah. And that definitely brings us right into the Microsoft involvement with this competition, which also pleased to have with us Mark Tabladillo, who's a cloud architect with Microsoft. Yeah, thanks for being here and being part of the competition and also Microsoft's involvement in this. It's been interesting, as we've seen some presentations already even, to hear how students are sort of making that realization about, hey, I've been working on my laptop solving maybe like data science toy problems. In my courses or something like that, but I got to this problem. And even one of the student groups said, hey, I had to, I bought more RAM for my laptop

• 11:01Daniel Whitenack?

  to try to solve the problem, but then they're like, that's not doing it. So then they started thinking about cloud services. So how do you think, Mark, as, I guess my question is, as students and maybe people getting into the field are kind of making this realization around the resources required to solve actual like business problems. What are those ways in which they can kind of start dipping their toes into cloud and experiment with things to kind of expand their horizons in terms of what's possible without, you know, knowing about, maybe people don't know about Docker and Kubernetes and like all that stuff yet, but they want to start dipping into more resources. What's a good way for people to kind of get into that as you've seen these students kind of do that?

• 11:46Speaker 03

  Sure. And I think there's more resources available more than ever to do self-learning. And it was maybe 10 to 15 years ago where it was very common to go to a bookstore and find these big, thick books on Microsoft.

• 12:00Daniel Whitenack?

  With animals on the front.

• 12:01Speaker 03

  Yeah, maybe they, you know, they would be training books for technologies. And of course, the publishers are still out there. O'Reilly's still out there producing books. And I have a friend, by the way, who's coming out this month with his book on practical machine learning and AI, Jeff Prosise. And I'm so proud of him to write a new book. But the point is that so many things are online. And in the Microsoft ecosystem, there was a time when you had to pay to even get the proceedings from a conference. Like you didn't even go and you couldn't even get the recordings. Well, now Microsoft's making a lot of that available for free and in a way that people can find it.

• 12:40Speaker 03

  And so I think for the audience of this podcast, I would love to have them look at YouTube, what's available there. Microsoft's got a few channels of content on there and that's a good way to get started. Sometimes they're short sessions between five and 10 minutes. Sometimes they go to an hour. But that's definitely one way to get started.

• 13:02Daniel Whitenack?

  Yeah, and could you help us, I think it's good for people to kind of organize certain categories in their mind. I've heard students talk about like the Microsoft Azure kind of studio environment. And then there's other things like these like cognitive services and like managed AI services. So how do these things differ and like how how might they be used? Or how have you seen them being used either in this competition or other places?

• 13:29Speaker 03

  Okay, so the unifying thing is either, and you can pronounce either Azure or Azure, both correct, both correct.

• 13:35Daniel Whitenack?

  Good, good. It's good to have the definitive answer on that.

• 13:38Speaker 03

  This is the definitive answer for for all time. I tend to use Azure, but just because out of habit. You know, the unifying factor is Azure Active Directory. So that's the the main authentication path to going into an Azure subscription. And the subscription itself is, you know, think about it like a credit card. And there, you know, when you if you sign up for a subscription, you would have to put your credit card on there to, you know, pay the bills. Again, another free path, Microsoft offers a lot of things. We have free subscriptions and we even send our customers to go get them. And I even tell my customers, I say, you know, it does run out. So I say, well, go make a new email at outlook.com and then just make a new one.

• 14:21Speaker 03

  All right, we want you to get hands on because there's no substitute for experience. And that's even kind of the point of this competition. You know, you can study it in the book, you can do class exercises. But two things are true. First of all, putting into practice and working as a team. And that's what we're doing in this competition. So back to your earlier question, now that a team may all all join the same subscription. The Azure Machine Learning Studio is our flagship technology for machine learning. And it can run on regular CPU or GPU instances. We have regions available around the world. I happen to work in the federal space, so we also have specific clouds just for that use.

• 15:05Speaker 03

  And there's there's different sovereign clouds now. And Microsoft's now beginning to build out specialty clouds on top of our regular clouds. And also call out to for students, we have a lot of Promotional for students where they get things at discount rate and also nonprofits. Nonprofits will get special treatment inside the Azure ecosystem. But let me go back to the original question. All right, so there are two focuses that people will have. One is machine learning and it tends to be thought of now as building a model. That is the way kind of to think about the products. AI is the is the marketing term for all our technologies now. So if you go to the main website, everything is AI.

• 15:47Speaker 03

  However, inside the Microsoft technology, there are cognitive services and those are considered mostly APIs, the REST APIs, and they're already pre-trained models and they do certain things. And we've seen some of the teams here at the contest, they've been using maybe computer vision or text to image or image to text. You know, those types of technologies are already out there. And Microsoft's not unique. Other vendors have these and Microsoft is now behind the scenes supporting all open source technologies. Some come pre-built inside machine learning studio, like we have a certain version of a Python kernel that will run inside there.

• 16:29Speaker 03

  And another technology, okay, so going back to the Azure machine learning studio, we have doubled down on ML flow as the way we are organizing our workspace. So on the new version of the API, that is the path forward and that is open source.

• 16:45Daniel Whitenack?

  And could you just describe a little bit what that what that is, ML flow?

• 16:48Speaker 03

  ML flow, so it is a way to organize experiments and training and models and model deployment. And it has its own syntax in terms of vocabulary and API, you know, the way to work. And Microsoft's not alone in using ML flow. There's other vendors that use ML flow in their. Technology, but it is a way to organize kind of the technology and the assets. And Microsoft decided to make that native to our own API and how that works.

• 17:18Daniel Whitenack?

  Great, yeah. And I think one of the things, like as we've been in the presentations here today, I think when I was in grad school, you know, I programmed, but it was like, you know, I'd use MATLAB or Python or whatever, and I did some things. I had no concept of like how infrastructure worked in industry or like, you know, the whole thing about doing programming in academia is just like not, not always a parallel to industry. So like coming from the Microsoft perspective, I've found it really encouraging to see students, you know, saying things like object store or, you know, model registry or like these things and thinking through like the architecture.

• 17:59Daniel Whitenack?

  I know that's one of the things in the informs, like they emphasize is that sort of model deployment and model lifecycle management. So yeah, do you have any words of encouragement for maybe those, you know, listeners who are, you know, again, getting into this space or maybe they're students in terms of getting hands-on with actual infrastructure that people use in industry and how that benefits kind of, you know, your understanding of how to create value with, with what you're producing rather than just creating a cool model?

• 18:29Speaker 03

  Right. So let me call out a few things. Now, some people aren't so lucky to be even admitted to Purdue. And I, and if you were, I would certainly, you know, want to come to a program such as here and you can participate in all these cool events. But short of having that, you know, either undergraduate or graduate experience, Microsoft has, and this is what I believe is the front door. We have something called AI Business School. It is a series of courses that show how to tie in the value of AI in a business context. And a lot of the videos were done by our own leadership and they've shown how we've used AI inside the Microsoft business. Now, it's not intended to be a catalog of all possible ideas, but it does kind of

• 19:13Speaker 03

  cover the landscape of, you know, along the lines of the informs domains, it covers the landscape of, of here's, we have a challenge, here's how we're going to use data modeling and then put in production, and here's how we're evaluating use. And it just gets people started. So it's something I do recommend to my customers because people do have different roles. And even, you know, what I'm working, we're working on internally. We're now rethinking through who are the personas of people who touch data projects. And we talked about the domains, right? But we're going now to thinking about, alright, so who's that person? What does that person do? Are all modelers the same? We don't think so.

• 19:55Speaker 03

  So we're now beginning to think that because we're now serving a large internal community inside Microsoft in terms of our programming. So that's the first thing I think about is EAI Business School. And then also in terms of getting started, we have inside the, all our technologies, we have tutorials and samples to get started. Those sample data sets, program notebooks that run quickly. They don't take hours, but they, they show you a variety of things. They're guided toward, you know, specific outcomes. They, they began to get better. I mean, I've seen Microsoft examples and how they have grown in the last 15 years. And they're just getting better and better because they're getting better minds thinking about it. But I'll also call out, you know, Microsoft also is always looking for partners that want to share their stories.

• 20:40Speaker 03

  And we have a lot of case studies of companies doing things in different industries, whether it's for profit or nonprofit. And one, I'll call out, you know, one example, we work with the Metropolitan Museum of Art to digitize their entire holdings. And I don't know if they did 100%, but just they had the same challenge as a lot of art owners, and that is not all their collection is on display. And some researchers want to have access to those products. So that's an example of, you know, anytime we do work with major organizations, we put those ideas out there. But more practically, and this may even be helpful for students or users, we have what's called the Azure Architecture Center. And inside, we see many architectures, very similar.

• 21:25Speaker 03

  By the way, we're only looking at the top presentations, but we are seeing architectures presented. It's the type of thing that I do in my own work. And the architectures will be in there, the diagrams and also the case study of what it did and you know, kind of what the use case is. So it gives people, again, a catalog of different ideas of how do you use the different resources that are available. So between all that's a lot.

• 21:51Daniel Whitenack?

  Yeah, yeah. Thank you so much, Mark. We'll definitely link both to informs, what they're doing, to Purdue and the BAME program and to these resources from Microsoft in our show notes for the podcast. So make sure and check all those things out. Thank you again, Mark and Matthew, for what you're doing on this and looking forward to hearing the rest of the presentations.

• 22:12Speaker 03

  Thank you, Dan.

• 22:13Daniel Whitenack?

  Thanks.

• 22:22Daniel Whitenack?

  All right. Well, I'm here with the winning undergrad team from the competition, the image captioning competition, which is from Butler University. I've got Chris Stein, Andrea Marky and Aaron Pinner with us. So congratulations on winning the undergraduate portion of the competition.

• 22:40Speaker 04

  Yeah, thank you very much. We are pleased.

• 22:43Daniel Whitenack?

  Yeah. So your solution was really interesting, actually. All the undergraduate presentations, I was surprised they seemed like graduate student work to me. But tell us a little bit, so the task again was image captioning. So just tell us one highlight about maybe one of the challenges that you faced in the competition.

• 23:05Speaker 04

  So I think that one of the big challenge in this case was about the data set because you know, for certain languages, like the other languages that you know, we have to work on, the data set was kind of small and also there were not much variety into the pictures. So that was the biggest challenge. So we kind of overcome that challenge by either artificially augmenting the data set or adding new pictures to the data set.

• 23:38Daniel Whitenack?

  Great. Yeah, and specifically, part of the competition was thinking about different languages where maybe image captioning isn't supported. And I think one of the things I appreciated about your all's presentation as well was thinking through the business implications of something like this technology of image captioning that could enable new or expanded possibilities for local language communities that don't have this technology. Could one of you comment on maybe what you envision in terms of the impact something like this could make in terms of image captioning for a language where it's not supported yet?

• 24:14Speaker 05

  Yeah, so our idea was almost to make, create a web app or a mobile app that, you know, small businesses using like Kyrgyz or Thai or these small languages could go on this app and submit their photos there, right? So, you know, everyone's got a cellphone, you know, in all communities nowadays. And if they can utilize that cellphone to almost leverage it and, you know, upload those pictures right then there and get a caption, you know, that is handy for the small business and S I L and their mission. You know, a small business would want to use this because, you know, a lot of people are drawn to websites because of images. They click on images in Bing and Google. So if we can help small businesses, especially if they have a user base with a minority language, you know, that helps both S I L and the company. So really, you know, if there's a monetary win,

• 24:59Speaker 05

  but really we're, you know, helping the world in a way. So it's really neat. Yeah.

• 25:03Speaker 04

  Yeah. And also, sorry, if I can add something, like last night, we had a brief talk, you know, about, you know, the languages around the world. And so also, there is not only a business implication to this challenge, but we know that also we are losing a cultural heritage. Like you mentioned last night, that every two weeks, one language is lost forever, right? And so there is no way that we can keep those languages alive. So if this can also help make the world more open towards, you know, those small communities, you know, this is also good thing also for the world because it makes the world a more interesting place, right?

• 25:44Daniel Whitenack?

  Yeah, yeah. Awesome. And maybe a comment from Aaron. As far as this competition, maybe what's one of the highlights of something that, something that you learned throughout the competition that you view differently now, either in terms of the technical challenges and that side of things, or the business problem, or something that you'll carry with you throughout the rest of your work?

• 26:07Speaker 06

  Yeah, I think one thing I learned was just, I think the challenge really opened my eyes to like this problem that existed. I would have never thought about like using AI or machine learning in a way that like directly impacts languages. So I think that was definitely something I learned and was really interesting.

• 26:24Daniel Whitenack?

  Great. Well, congratulations again. Hope your travels back home are safe and yeah, congratulations. Hope to stay in contact.

• 26:33Speaker 04

  Yeah, thank you very much. Thank you.

• 26:43Daniel Whitenack?

  Okay, well, I'm with now the winning graduate team from the Purdue Using Analytics and Data Science for Good competition. This team is from Georgia Tech. Here I have with me Harsha, Varun, Ravi, and there was another team member, Sanchita, who couldn't make it to the competition here in person, but want to acknowledge her and her contribution. So congratulations, first of all. You all are the first out of, I think it was 170 something teams in this competition to come up with a with an image captioning model that performs well on three sort of diverse languages from around the world, Thai, Kyrgyz, and Hauser.

• 27:24Daniel Whitenack?

  So first off, congratulations. And I think one of the things that was really interesting to me about your all's solution is one, kind of looking to state of the art models like CLIP, which was something that featured in your solution, but then also using sort of a multi-stage approach where you actually determined if a caption existed already that you had in a database of captions, and then if it didn't exist, generating a caption. So could you, one of you describe a little bit about how you kind of eventually got to that solution, how you considered, you know, using CLIP and got to thinking about that direction?

• 28:05Speaker 07

  So I think the problem itself was quite challenging when you look into the dataset and actually see what the data is. You can see poems, you can see philosophical statements, moral statements, parts of stories. And if you want to predict these kind of statements, you need information about the previous part of the story or the further part of the story in order to even build a model. A zero-shot captioning model is where it's very difficult to achieve a good zero-shot captioning model for such kind of prediction tasks. So the next step that we thought was maybe we could do some sort of classification model. That was the original thought process, that we could select from a corpus of sentences, can we select a sentence that best matches this?

• 28:50Speaker 07

  And from there, we started researching, basically. And when we went through hugging face models, we found the clip model, and then we researched further, we found a multilingual clip model that could handle different languages. And it sort of went through that process. And when we actually used it, it was decent, I wouldn't say it was perfect, but it certainly improved our overall solution quite a bit.

• 29:12Daniel Whitenack?

  Yeah, yeah. So when you were thinking about this idea of looking to existing captions and using those when you could, how often, in the dataset that you were looking at, which is this Bloom dataset, how often did you have to generate image captioning, image captions, excuse me, versus maybe looking to a list of captions and using one that preexisted?

• 29:38Speaker 07

  So even in the training dataset where we had the images and all the captions that we needed, when we actually used the multilingual clip model, it was more about like 20, 30% that were matching, and we had a very low threshold by that itself. And we didn't want to lower the threshold because we didn't want to get more false positives in a way. And Basically, we we just decided on that threshold. We didn't do any optimization on that particularly. From there, when we actually used the model on the test set, we suddenly got a huge jump in the score. That was basically it. So we were covering about twenty to thirty percent of the images. Even when you had all the captions for the images, only twenty to thirty percent were actually matched by the multilingual clip model. All the other images went to the generator model.

• 30:20Daniel Whitenack?

  Yeah, and you've mentioned sort of clip, hugging face. These are all, you know, kind of the industry standard, state of the art sort of things as a team getting into this problem. What were the kind of challenges that you faced in terms of maybe even finding where to start? Or maybe it's computational challenges or other issues?

• 30:42Speaker 08

  So when we initially started with this dataset, we were like, we were stumped, honestly. Like, we hadn't even heard of a model that could generate like contextual information with as much depth as was required by the solution here. So like, as I said, we did our initial EDA with, say, Microsoft Azure, using their computational API and translator model. So like, we, when we actually used that, we thought, okay, these are reasonable guesses, like, okay, human would make these guesses. But we had to go deeper. So we had to like match, we had other ideas as well, like we thought of like clustering common images. Maybe they belong to the same same story or same piece, they're part of a single book or something like that.

• 31:25Speaker 08

  Yeah, so that's how we got started off. The EDA that we did helped a lot, like, understanding that they were poems and stuff mixed into the data helped us look for more deeper models that could generate context.

• 31:36Daniel Whitenack?

  Great. Yeah, yeah, thank you for that that info. So I think, you know, as you as you look forward to kind of, I mean, I know all of you will be going very far just with the innovations that you you've demonstrated here. I hope that maybe when you you own a billion dollar startups, you'll hire me to like sweep the floors in your in your startup or something. But how do you think kind of working on a on a solution like this from start to finish has influenced how you'll think about maybe AI or data science problems in the future? Any input?

• 32:12Speaker 09

  So the amount of good the AI can do to the real world we play out looked at lot of things the SL does it's actually improving a lot of language proficiency among the students and also increasing the educational rate among the people who are not studying so much so the amount of good the data or AI can do is will definitely influence our thoughts in future as well so it's like the kind of use cases all these things can have on the lives of the people are definitely going to stay in home and then we definitely try we will definitely try to contribute wherever we can by keeping this in mind so this is going to stay with us forever.

• 32:52Daniel Whitenack?

  Great, great. Well, thank you for your participation and congratulations again. I hope your travels are safe back home.

• 32:58Speaker 09

  Thank you.

• 33:09Speaker 01

  All right, that is our show for this week. If you dig it, don't forget to subscribe. Head to practicalai.fm for all the ways. And if Practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets, to fly.io for backing our dynamic requests, to Breakmaster Cylinder for the beats, and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.