OpenAI and Hugging Face tooling

The time has come! OpenAI’s API is now available with no waitlist. Chris and Daniel dig into the API and playground during this episode, and they also discuss some of the latest tool from Hugging Face (including new reinforcement learning environments). Finally, Daniel gives an update on how he is building out infrastructure for a new AI team. Sponsors: RudderStack https://rudderstack.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog – Smart customer data pipeline made for developers. RudderStack is the smart customer data pipeline. Connect your whole customer data stack. Warehouse-first, open source Segment alternative. Me, Myself, and AI https://sloanreview.mit.edu/audio-series/me-myself-and-ai/ – A podcast on artificial intelligence and business produced by MIT Sloan Management Review and Boston Consulting Group. Each episode, Sam Ransbotham and Sheervin Khodabandeh talk to AI leaders from organizations like Nasdaq, Spotify, Starbucks, and IKEA. Me, Myself, and AI is available wherever you get your podcasts. Just search Me, Myself, and AI. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: OpenAI’s API Now Available with No Waitlist https://openai.com/blog/api-no-waitlist/Hugging Face SnowballFight beta https://huggingface.co/spaces/ThomasSimonini/SnowballFightHugging Face Data Measurements Tool https://huggingface.co/blog/data-measurements-toolPandas Tutor https://pandastutor.com/ClearML https://clear.ml/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Daniel Whitenack?

  Reinforcement learning in general is not like a model. It's a framework in which you can train agents or models. And if you think of self-driving car or something like that, if you say, I'm going to go from point A to point B in a self-driving car, well, there's a whole bunch of routes you could take. It's not really that there's one perfect solution to that problem. It's more about the decisions you make along your route based on what you've done so far and the feedback that you're getting from your environment.

• 0:34Speaker 01

  Big thanks to our partners, Leno, Fastly, and Launch Darkly. We love Leno, they keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com. This episode is brought to you by our friends at RudderStack. And we're calling all data engineers to check out RudderStack Cloud and start building smart customer data pipelines. RudderStack is warehouse-first, no more silos. RudderStack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API-first. RudderStack can be easily integrated into your existing development processes.

• 1:18Speaker 01

  And because they're open source, you can see all their code, so you don't have to worry about vendor lock-in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. RudderStack is free, up to 500,000 events, and pricing scales transparently from there. Learn more and get started at rudderstack.com. Again, rudderstack.com. That's R-U-D-D-E-R-S-T-A-C-K.com.

• 1:52Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI.

• 2:18Daniel Whitenack?

  Well, welcome to another fully connected episode of Practical AI. This is where Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into some learning resources to help you level up your machine learning game. I'm Daniel Whitenack. I'm a data scientist at S I L International. I'm joined, as always, by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

• 2:49Chris Benson?

  I am very well, Daniel. Looking forward to diving into some of today's topics. And as you will explain to the audience, we get to put you on the hot seat just a little bit today.

• 2:59Daniel Whitenack?

  A little bit, yeah, yeah. Talk about some of the things in my life. But yeah, it's exciting. Lots of exciting things going on as we, as we wrap up what is the last weeks of 2021. That seems like it went, it went by fast. I don't know if it did for you.

• 3:16Chris Benson?

  It has gone by incredibly fast. The last two years, just craziness. It's just like, where did they, where did they go? But you know what? We got some new things happening here, some new things to talk about that you can guide us through.

• 3:29Daniel Whitenack?

  Yeah, so I think maybe one of the things that we can talk about, which we've talked about several times on the show and has been a sort of developing theme that we visit occasionally is GPT-3 and the OpenAI APIs. I don't know when you saw this, but recently I saw how OpenAI recently made their API available with no waitlist. So previously you had to sort of apply on a waitlist, they would approve you, and then you could use some of their models. And I think originally, even when we first talked about GPT-3, they released it and it

• 4:12Daniel Whitenack?

  was fairly closely guarded, I think mostly because of what they considered, you know, a safety process and making sure that people didn't misuse.

• 4:24Daniel Whitenack?

  The model in certain ways. And I think based on the blog posts where they talk about, you know, them opening up some of the availability. They emphasize a lot of these kind of safety features that they have put in place.

• 4:39Chris Benson?

  And that might be actually before we dive into it, that might be a good place to start is is why? For those who are not already very familiar with it. Why would this model that they've released need all of this careful vetting and slow rollout and such? Cause it's been quite a while since they did it. So you, you want to talk toward that a little bit?

• 5:00Daniel Whitenack?

  Yeah, I think it's probably, first of all, I'm not speaking for OpenAI. But I think in general, what people's thought process is around these types of models is that GPT three just kind of stepping back is a sort of large scale language model that enables a variety of natural language processing tasks to be performed. And some of those, for example, like natural language generation are very, performance is very impressive from GPT three. So one thought is, well, GPT three could be used to do sort of malicious things,

• 5:41Daniel Whitenack?

  like create a bunch of fake news type of stuff, misinformation, sort of distribute a lot of this kind of thing. As well, GPT three was trained on a, on a huge amount of data that, you know, was kind of crawled from the internet and other places. And I think the behavior of the model, because, you know, all of the biases and other things that exist in that kind of large corpus aren't totally probed out. So another question is like, how are biases showing up in this data? And, you know, if we just say, hey, everyone use this thing to generate text or integrate it in their applications.

• 6:21Daniel Whitenack?

  They might not take that sort of caution in mind if they're just kind of applying it wholesale across the board.

• 6:29Chris Benson?

  Yeah, and just to name at least the nine categories without all the detail. They note that they prohibit users from knowingly generating or allowing others to knowingly generate with the account. The following categories of content, there's nine. They are hate, harassment, violence, self harm, adult, political, spam, deception, and malware. And apparently they have spent a fair amount of effort putting the safeguards around this. So that it could do, it'll be interesting to see whether or not going forward, they're able to put safeguards around other models they release much faster. Now that they've kind of got some, the infrastructure in place.

• 7:11Chris Benson?

  So it might speed up the ability to get on to new models because this has been, I mean, how long has it been now? Maybe a year and a half?

• 7:18Daniel Whitenack?

  It's been quite some time. Yeah. So it's interesting. I mean, they put a lot of thought process into this. And I mean, they're to be commended for leading a lot of the thought around these areas. I think there's different views on how the community should go about addressing these things and whether it should be via a, an access controlled API or via open source code and models and data sets. Or, you know, however it should be, be approached, but they have put a lot of thought into this. They talk about in the blog post, how they are putting things in place to review applications.

• 7:59Daniel Whitenack?

  So applications of the model, applications of the AI before they go live. Monitoring those applications for misuse and, you know, supporting, better supporting applications as they scale and, and understand the, the effects of the technology. So very interesting developments here from OpenAI. Chris, have you logged in and tried anything with the OpenAI? Playground or anything like that?

• 8:27Chris Benson?

  I've just gotten an account now that they've opened it up. I am ready. So what I'm gonna look for you to help me get into this. I have my, I have my brand new account. I have the site that they send you to open where it says, "Welcome to OpenAI. Start with the basics." I'm looking at their examples. I have OpenAI Codex open. And by the way, for our listeners out there, this might be a good moment. If you don't already have the account, pause the podcast for a second. Go grab an account and follow along with us if you're not driving in the car or something. And I would love, I know, Daniel, you've been working with this for some time. If you could kind of guide us through a little bit about what to see, you know, I know you're a practitioner, you're not with OpenAI, but as someone who

• 9:09Chris Benson?

  has used it, if you can maybe give us a little boost on things that you've already learned, that would be fantastic.

• 9:15Daniel Whitenack?

  Yeah, so we can maybe just kind of look at some of the basic functionality that you can do here. And I mean, there's, you can do a lot of things. So this is by no means a complete introduction. But if you kind of log into the API interface, one really nice thing is that they do give a kind of big blocks that give you introduction and examples. So you can scroll through those. But they have this cool thing called the playground. So they have documentation examples, and then they have the playground. Ultimately, you can use the API in a variety of ways, including like a REST type interface. But they built this kind of playground to help you try out things and see like, what are

• 9:58Daniel Whitenack?

  the types of things that I could do and what's applicable to my specific use case. So if you, if you click on the playground, it kind of opens up this essentially a text box. The most basic of things that you could do is just start typing. You know, I could say, "Chris Benson is really cool."

• 10:22Chris Benson?

  Oh, boy.

• 10:23Daniel Whitenack?

  And knows a bunch.

• 10:24Chris Benson?

  I thought no deception, remember?

• 10:26Daniel Whitenack?

  About AI. Yeah, so Chris Benson is really cool and knows a bunch about AI. And then I could click generate, and it's going to start generating a bunch of text. So, so.

• 10:40Daniel Whitenack?

  So yeah, I don't know if you want to hear what it generated as a result of that, Chris. So I put in Chris Benson is really cool and knows a bunch of AI.

• 10:48Chris Benson?

  Oh.

• 10:48Daniel Whitenack?

  And then it started saying, it said, I think he's still a bit of a newbie, but he's learning fast. This is his first post. Enjoy it. And then it goes on, I've been playing with this new stuff called machine learning lately and I've found it rather fun. What I'm going to talk about today, so you can keep clicking that button and it's going to generate more and more of Chris Benson's new blog post about machine learning and how he's finding it rather fun.

• 11:15Chris Benson?

  Oh, boy. I was about to put yours in, but since you put such an unrealistic thing about me, I just put in, there was a guy named Chris who had nine dogs and I started with that. It took an interesting path. It says, he wanted to go for a drive in his new car, a 67 Ford Mustang. It was a nice car and he was proud of it. He asked his neighbor if he could borrow his dog for a few minutes. The neighbor said, sure, take him for a few minutes. Chris went to the garage. So that's what I had. So this is, this is fun. This is, I have a feeling this is going to turn into my favorite. Saturday night with a drink or two, not too many, of course, because we're responsible and no driving.

• 11:58Chris Benson?

  But yeah, this might be the new game with friends on a Saturday night thing to do after a beer or a glass of wine.

• 12:04Daniel Whitenack?

  Yeah, but I mean, you can see, and I think the interesting thing is that there's a number of things that you can tweak here, right? Along the right-hand side of the playground, you can change the different sort of engines that are available to do this, but then also kind of response length, you can change various kind of parameters, hyper parameters about what's going on and you can show probabilities or not. So there's more here that you can get. And a cool thing is to, similar to like, like if you were writing REST API calls or testing them in Postman, you can generate code to make that call.

• 12:45Daniel Whitenack?

  Here in this playground, it's similar. You can actually generate code. So if you click on view code, you can then see, hey, here's the Python code that will actually call the OpenAI API and get the similar response from the engine. And that way you can sort of integrate. So it's telling you how to integrate this sort of text completion. Into an application and you can, you know, go down and see how to do that with Python or just calling a REST API, which is pretty cool.

• 13:16Chris Benson?

  Yeah, or REST API or curl.

• 13:19Daniel Whitenack?

  Yeah.

• 13:19Chris Benson?

  So they give the JSON and the curl.

• 13:21Daniel Whitenack?

  Yeah, now the interesting, so this is cool. The text completion thing is cool, but I think the maybe more interesting thing, at least for our team and how we've looked at models like these is that they can be quickly adapted to a very specific task that you are more interested in than maybe just kind of general text completion. So if you look, if you're in the playground and you look at, there's a little dropdown called load a preset.

• 13:50Chris Benson?

  Okay.

• 13:51Daniel Whitenack?

  You could load, for example, a Q and A preset. That's the first one that pops up for me and you'll see what, what happens is the playground will sort of pre-fill a little bit of example data for the model. So it's giving a sort of pattern. You can kind of think about this like a very small amount of prompt data or a very kind of few-shot example type of thing where you're giving just a little bit of warm-up to the model to tell it, hey, this is the sort of thing that I want to generate, right? And so when you pull up the Q and A thing, for those that are listening, there's a Q colon

• 14:34Daniel Whitenack?

  and there's a question and then an A colon answer, right? Q colon question, A colon answer, Q colon question, you know, A colon answer. And, and it provides a bunch of these answers. So the model all of a sudden is realizing, oh, it wants me to generate things that are like Q colon some question and A give me the answer. And so now at the bottom, you can type in a, a new question. So we just recorded an episode about federated learning. So I'm curious if it knows what that is. So I'm going to say, what is federated learning? Question mark and I'll generate. And it says federated learning is a machine learning technique that allows

• 15:16Daniel Whitenack?

  a single machine to learn from multiple sources of data, which is actually quite relevant. How did it know how to do that? Well, it's trained on a whole bunch of text data from the internet and, you know, the world. Right? So at some point, it maybe knows something about that or has been prompted.

• 15:35Chris Benson?

  I just want to share with you what I put in while you were doing that.

• 15:37Daniel Whitenack?

  Sure.

• 15:37Chris Benson?

  I put in for question, what is GPT-3? And the answer was GPT-3 is a question answering system developed by IBM.

• 15:47Daniel Whitenack?

  Well, they should work on that.

• 15:48Chris Benson?

  They might.

• 15:48Daniel Whitenack?

  Maybe we should switch over to the IBM, IBM Watson API.

• 15:55Daniel Whitenack?

  This is a lot of fun. And actually, you could scroll through these. You'll actually see there's examples that they give for summarizing text or text to a command or parsing unstructured data or classification. And so it basically gives you an example of, hey, this is the way that you can sort of prompt GPT-3 in order to have it do a new sort of task for you. We've used this for kind of some data augmentation type things. So where we've wanted, wanted to generate data in a certain way for a purpose and actually use GPT-3 to kind of help us generate that data. And that's been very helpful for us.

• 16:39Daniel Whitenack?

  So that's kind of a, maybe a general rundown of what is GPT-3? And why you might want to check it out?

• 16:48Chris Benson?

  Yeah, the, just to call out some of the things in that dropdown. There's chat, there's Q and A, which we talked about. Grammatical standard English summarized for a second grader, text to command, English to French, parse unstructured data and classification. And then it has a more examples section, but yeah, it looks good. I'm looking forward to diving into this.

• 17:20Speaker 01

  This episode is brought to you by me, myself and AI. It's a podcast on artificial intelligence and business. It's produced by our friends at MIT Slow Management Review and Boston Consulting Group. And the question is, why do only 10% of companies succeed with artificial intelligence? That's the question they aim to answer with this podcast. Each episode, Sam Ransbottom and Shervin Kotabande talk to AI leaders from organizations like NASDAQ, Spotify, Starbucks and IKEA. And as you'll hear, the show discusses tough topics like bias in AI.

• 17:51Speaker 03

  We've built technological safeguards and prods. Some of it is actually having the technology prompt the human and say, hey, you're building a model. You have identified that maybe you don't want race as a variable in this model because it can introduce bias. But we see here you have a field that is zip code and zip code can be highly correlated with race. There's a very human element to this question, but to address it at scale, you actually need to automate the solution as well.

• 18:21Speaker 01

  All right, me, myself and AI is a collaboration between MIT Slow Management Review and Boston Consulting Group. It's available wherever you get your podcasts, just search me, myself and AI.

• 18:58Daniel Whitenack?

  Well, Chris, Hugging Face continues to be the darling of the AI world, does it not? It does. Do things all the time. Cool stuff. I've seen a couple of things come out from Hugging Face recently. So for those maybe listeners who are new to the AI community or aren't familiar, Hugging Face is an AI. Company and actually has a whole host of things that are quite relevant to AI development and research and application. One of those being a model hub where you can get models, actually of all types now. It started with natural language processing

• 19:40Daniel Whitenack?

  models, but now it has vision models and speech models. It has data sets that you can pull. It has spaces where you can host machine learning applications. It has an accelerated inference API where you can serve inferences from your models. And you can kind of think about this almost like, you know, people post their code to GitHub. People post their container, software containers to Docker Hub. You can post models and data sets to Hugging Faces Hub. And those can be public or private. And so you can version your models and data sets there and serve your models and data sets there for your company. So

• 20:18Daniel Whitenack?

  it's really kind of a, it's becoming a one stop shop for a whole bunch of really useful AI tooling. At least that's kind of how I'm starting to see it.

• 20:31Chris Benson?

  That's totally right. Hugging Face is one of the names in the AI world that everybody respects and everybody recognizes. They just keep doing innovative things that are cool and they're super user friendly. And so it's, they're kind of the go to, one of the go to's in this space that you always are going to be using.

• 20:50Daniel Whitenack?

  Yeah. And one of the things that came out recently, which is really exciting, is a first ML agents reinforcement learning environment on, on Hugging Face.

• 21:03Chris Benson?

  Called Snowball Fight.

• 21:05Daniel Whitenack?

  Oh, yeah, yeah. I mean, very relevant for the Christmas holiday season, right? Is Snowball Fight. Maybe people are in someplace where you don't have snow. You can now have a snowball fight on Hugging Face. What was your first impression when you saw, when you saw this, Chris?

• 21:22Chris Benson?

  I'm playing with it, right? I'm loading it up as we speak and it is cool. Good names, good graphics. And I am trying to load up the Snowball Fight demo here right now.

• 21:32Daniel Whitenack?

  Yeah, I was playing it a bunch earlier and had some fun. If you load into it, you can actually play the game interactively. So just to give people a sense of what we're looking at. You sort of load into it, it loads this interface where a game comes up. And that's kind of driven by Unity. And you can move around this little guy in a snowball sort of field and throw snowballs at another avatar on the other side. It's a lot of fun.

• 22:02Chris Benson?

  The music's playing in my earphones, just so that you know. So, you know, you have a soundtrack going.

• 22:07Daniel Whitenack?

  Festive.

• 22:08Chris Benson?

  Yeah. Oh, boy, this is fun. Okay, since nobody can see what I'm doing, I'll stop.

• 22:12Daniel Whitenack?

  Now we know what Chris is gonna do the rest of the day.

• 22:14Chris Benson?

  Oh, this is it. This is taking over. So don't let my boss know that I'm on to this.

• 22:19Daniel Whitenack?

  Well, one cool thing is, I think we do have scheduled to have Thomas from Hugging Face, who created this in an upcoming episode. So we'll, we'll dive into it more then, but I wanted to, to mention it here because I think it's cool that Hugging Face sort of made this transition. I mean, at first, really centered around chat. Interfaces and then more broadly, like open source NLP.

• 22:45Chris Benson?

  Yep.

• 22:45Daniel Whitenack?

  And then kind of into open source and kind of general purpose AI tooling. And, you know, hosting services. And now we see this reinforcement learning piece coming in where the goal is that within Hugging Face itself, you'll be able to kind of build and share reinforcement learning environments. Now, Chris, I know that we've talked about it a little bit in the past. Reinforcement learning in general is not like a model. It's a framework in which you can train agents or models. And if you think of self-driving car or something like that, if you say, I'm gonna go from

• 23:26Daniel Whitenack?

  point A to point B in a self-driving car, well, there's a whole bunch of routes you could take, right? And it's not really that there's one perfect solution to that problem. It's more about the decisions you make along your route based on what you've done so far and the feedback that you're getting from your environment. And so in order to train an agent to execute decisions in that environment or make actions in that environment, you need to have a sort of simulated environment that will allow an agent to navigate in that environment. Get feedback and rewards and then be trained accordingly to operate in that environment, in this case to win a snowball fight.

• 24:07Daniel Whitenack?

  So that's ultimately, although the game is interesting, that's ultimately the most interesting thing about this is that it provides a route towards training reinforcement learning agents in environments that are shared on Hugging Face.

• 24:23Chris Benson?

  It's a powerful approach. And at a previous company, back when deep reinforcement learning was still fairly new, we were using it for training robots on our team at this other place. And it was really good there. It's used in video games. It's used in the industry I'm in now in a non-specific way. It's used to move all sorts of things we call platforms, things that move around and do things. And that's how we get autonomy to work these days. I mean, it is a fantastic tool. There is one that I think worth calling out that was a DARPA. It's a public DARPA thing. And DARPA, it's an interesting place. It is the Defense Advanced Research Projects Agency.

• 25:03Chris Benson?

  And they, they do all sorts of government oriented and military oriented experimentation, very cutting edge. And about a year ago, they did something called Alpha Dogfight. And I know I've brought it up in the past, but they, in a simulated environment, they trained a model to be able to do dogfighting against other models and

• 25:25Daniel Whitenack?

  Like in planes.

• 25:26Chris Benson?

  Like as in airplanes. Right. I'm really glad you said that, just to be very, very clear.

• 25:31Daniel Whitenack?

  I didn't want people to think about like robot dogs fighting each other.

• 25:36Chris Benson?

  Yeah, we're talking about, think of the movie Top Gun. Okay. That kind of dogfight, like pilot, Tom Cruise, all that kind of stuff. Not the same plane though. And they trained them. And so, and at the end of it, this was not the Navy, which is what Top Gun is, but the weapons school, which is the equivalent of Top Gun in the air, in the United States Air Force, they had a weapons school instructor. This would be like the equivalent of the instructors at Top Gun go up against the model and the model demolished the instructor over and over and over again.

• 26:09Daniel Whitenack?

  Wow.

• 26:10Chris Benson?

  And so you're talking about one of the best fighter pilots in the world. Period, getting demolished by the simulated AI that was based on deep reinforcement learning, beating it in the simulator. They were using a real, I believe it was an F16 simulator. I watched the whole thing live and it was just amazing. So this is some pretty cool technology here and it's evolving rapidly.

• 26:31Daniel Whitenack?

  But I think one of the things that I've always wondered as related to reinforcement learning is like you see the power of that, but then you see how hard it is to create these environments in which you need to train the reinforcement learning agent. So in order to train a reinforcement learning agent, you kind of have two choices. You can either like train the agent in the real world scenario. Like if you're training something that's going to fly a plane, that's not very practical, right? Because you're going to crash a bunch of planes and maybe kill some civilians.

• 27:05Chris Benson?

  People don't like that.

• 27:06Daniel Whitenack?

  So you have to create this sort of simulated environment. However, creating that is actually, I mean, it's not my core competency in terms of like 3D environments and like Unity and games and simulation and all that stuff. So I think the idea that there could be a place in which these environments are created and shared more broadly on Hugging Face to enable kind of people to share things, modify them, update them, train agents on them. I think it's a really interesting concept because I wouldn't personally, it would be hard for me to know where to start in creating this sort of environment if I didn't have a good way to kind of jump off.

• 27:49Chris Benson?

  So to generalize on that a little bit, it's a great way of accomplishing this in any kind of environment or industry that where the, the cost of that training would be prohibitively expensive. You know, you mentioned about where you don't want to crash planes a thousand times and all the, the damage. Another area would be medicine. If you want to try new surgeries or alter surgeries and carry procedures forward and you don't want to kill patients in the process, this is an area where you can use deep reinforcement learning for that. There's so many areas out there that are just very expensive. To do that, and I don't mean just financially expensive, but you know, loss of life and things like that.

• 28:29Chris Benson?

  That it's you're seeing it all over the place. You're seeing it more and more. And I think simulation is going to become ever more a part of businesses getting done what they are organizations in general, getting done the things that they're trying to get done.

• 28:41Daniel Whitenack?

  So one more thing that I wanted to mention from hugging face just before we're we're moving on. They've had a couple of releases that are quite interesting. The other one I wanted to mention was what they're calling a data measurements tool, which is an open source project. This just was released as well, and you can kind of look through it. The data measurements tool, they say, is an interactive interface and open source library that lets data set creators and users automatically calculate metrics that are meaningful and useful for responsible data development. And so you can kind of, you know, this is new enough that I haven't

• 29:23Daniel Whitenack?

  dug into it, but I think this does certain things, everything from kind of basic things that you might expect from exploratory data analysis, like figuring out missing values and descriptive statistics, all the way to analyzing biases in data sets, maybe related to particular factors like gender or other things. And so I think ultimately what they're trying to do here is create a nice way for people that are maybe using data sets off of hugging faces, data sets hub, or creating new data sets to really understand a little bit more about them and document them a little bit better so that people aren't just kind of

• 30:03Daniel Whitenack?

  pulling whatever data set looks good without understanding the implications of that. So it's a pretty cool route that they're going, I think.

• 30:11Chris Benson?

  Even as we're talking, I'm looking through some of the like the graphs they have that do analysis on the data sets that they're offering. I mean, the one that I'm in right now is hate speech 18. And you can go explore things like that. So it's interesting in a in a bad way. Things I definitely wouldn't want my child, but it kind of gives you all these different ways of. Analyzing and measuring. So very, very cool technology, a tool that is long, long overdue now that it's here. I mean, all kudos to them for doing it. Wish I'd had this for a while. So it, you know, it's funny, Daniel, as we, as we look at some of these tools right now, it feels like this industry is, is maturing a little bit in, in terms of not just having-

• 30:54Daniel Whitenack?

  Yeah, we have nice tools now, right?

• 30:55Chris Benson?

  Yeah. Not just having the models, but having some of the tooling we need around it to make it safe and get to what you need to get to for a good output without some of the missteps.

• 31:22Speaker 01

  Earlier in the show, you heard a teaser from our friends behind the podcast, Me, Myself and AI, MIT Sloan Management Review and Boston Consulting Group came together to produce this awesome podcast. And every episode, host Sam and Sherwin talk to leaders that are engaged in the theory and the practice of AI.

• 31:39Speaker 04

  I remember one project we had, we were training a chatbot. And it turned out we used raw, you know, logs, all privacy assured and everything, but we used these logs that a customer had provided because they wanted to see if we could build a better model. And it turns out that the chat agent wasn't exactly speaking the way we'd want another human being to speak to us. And why? Because people get pretty upset when they're talking to customer support. And the language that they use isn't necessarily a language I think we would use with each other, you know, on this podcast.

• 32:13Speaker 01

  All right. Me, Myself and AI is a collaboration between MIT Sloan Management Review and Boston Consulting Group. It's available wherever you get your podcasts. Just search Me, Myself and AI.

• 32:42Daniel Whitenack?

  So Chris, you had just started to kind of talk about how we're getting to a point where there's a good number of tools that fulfill a lot of the needs that an AI researcher or data scientist. The needs are so diverse, everything from analyzing data sets to serving models to dealing with infrastructure and tracking things. And there's a lot of good tools out there now. And I know that our team, and maybe this is a follow up from a previous conversation that we had about building data teams. Our team at S I L, we've been in the sort of growth phase. We're building up a team that's doing NLP research and development. And we've gone through a

• 33:25Daniel Whitenack?

  kind of process of figuring out what tools work well for us and how to kind of plug all of these tools together. And it's, it's kind of taken a year to work through a lot of, a lot of those things. But I know there's a lot of things that we talked about on this podcast around ML Ops and GPU servers and tracking models and experiments and, you know, all of those things. And I thought it might be good to follow up on that conversation that we had before and talk a little bit about how some of these things might tie together in a real world environment, because we talked about a lot of them individually.

• 34:05Chris Benson?

  I really want to hear what you're doing at S I L. And the reason I say that is for those of us who have followed the show for a while, when we've had some of these other related conversations, I know we talked about some of the infrastructure that we had at Lockheed Martin, which was, you know, a very large organization and kind of how we approach things. But that's a different set of business drivers and a different set of constraints about how you go and evaluate such software. And so you're coming from a different organization, different size, different constraints, different budgets, all that kind of thing. I would love not only for you to share what you have done. And how you arrived at those, but kind of what some of those constraints were that you had to learn to live within and yet make it work.

• 34:48Daniel Whitenack?

  Yeah, for sure. So to give a little bit of context, we've now got probably a group of, depending on how you count them, probably 15 or so people between people on my team plus kind of academic collaborators, plus other, you know, close collaborators that are working on kind of a similar set of problems for NLP tasks. And how we sort of thought about this was, you know, we need a way for this team to do kind of a diverse set of experiments.

• 35:24Daniel Whitenack?

  So we're working on all sorts of things from machine translation to spoken language identification and speech-related problems to chat and dialogue. And so we need people to be able to use a whole variety of tooling. But also we want to create some standardization and centralization around how we're tracking experiments, how we're running jobs, and how we're sharing models one with another. And so we're kind of, you know, settling down on some of that. Part of that was thinking about, okay, where are we going to run training? Where are we going to run inference in our context?

• 36:04Daniel Whitenack?

  And how and where are we going to store and track models, data sets and code? So some of that's a little bit easier than others. I mean, code, we version in GitHub. That's pretty standard for everyone. But also we use Google Colab a lot because our organization, we use G Suite for all of our kind of, you know, docs and Drive and all of that. So we use Google Colab a lot. So if we think, okay, some people are going to have code that lives in GitHub. Some people are going to have code that's living in notebooks in Google Colab. Colab, of course, has some GPU resources, but in order to train some of these larger NLP and speech models,

• 36:46Daniel Whitenack?

  we need other more robust GPU resources. So we did end up getting an on-prem GPU server, which is sitting down in Dallas. But that brings up a new set of questions. So if we've got 10 to 15 people and sometimes more distributed sort of all over the US, but also in Europe, in Asia, basically all over the world. How do we get them all running things in a reasonable way on this on-prem server in Dallas? You might be saying HBC stuff. I know you worked in HBC stuff for some time.

• 37:22Chris Benson?

  I was gonna say a scheduler, you know, is one of those things in terms of getting the jobs lined up and stuff. How did you approach it? I'm curious what's different from what we, how we addressed it in it.

• 37:31Daniel Whitenack?

  So I think one of the things that we wanted to make sure was that our people running and supporting the server on the ground weren't really the ones that we're going to like, although they're installing the server, they're not really administrating the server. You know, we don't have a large kind of, you know, DevOps and engineering team behind our NLP team supporting us. So we wanted a kind of simple solution that we could work with on our distributed team. So we ended up using ClearML for this. So ClearML, we actually had a conversation on the podcast about a similar kind of tool, which is in the same vein as weights and biases. That's another very popular one.

• 38:13Daniel Whitenack?

  But ClearML allows you to have a dashboard where you track all your experiments and your runs.

• 38:19Chris Benson?

  Yep.

• 38:19Daniel Whitenack?

  And you can actually just by, so I could be running a Google Colab notebook, import the ClearML library and register that experiment in the ClearML dashboard. But I could also, from the ClearML dashboard, in queue a job on the GPU server that will also be registered on the same dashboard and will be sent to the GPU server in Dallas, sort of like a scheduler. It's not as sort of full-featured as like these, you know, really robust, big schedulers that are used on supercomputers. But it's enough for our team because we can say, well, We have this many queues on our GPUs and you can put things in there and they'll run in that order.

• 39:03Daniel Whitenack?

  We don't really need much more functionality than that. And all of that's registered and all of the input-output data is registered in a backing data store in S3. For us, we're running code locally on our laptops, code in Colab and code on the on-prem server. All of that is importing ClearML and all of those jobs are being registered in ClearML. And all of the input data and the output artifacts like model files are being stored in S3 in a version way. So we know what data was used to train this model. When it was created, we can look at the exact run and all of that type of stuff.

• 39:43Chris Benson?

  Without diving down like the alternatives that you didn't choose, because I don't want to do that to them. But kind of abstracting that a little bit, what's some of the reasoning that you that you chose ClearML? Or conversely, that you didn't choose a competitor from a capability standpoint? Or from a, you know, how it was satisfying the need? What were some of the things that made you arrive at this solution being the right one for your organization? That were not universal across all the solutions?

• 40:11Daniel Whitenack?

  Yeah, I think it was a combination of, one, just the simplicity of administrating the solution for not systems administrator people, although we got their help, right?

• 40:24Chris Benson?

  Sure.

• 40:24Daniel Whitenack?

  The majority of people running and operating this thing are data scientists and NLP people. So we needed something that we could support and not some thing that we would have to know a lot about HPC systems to support.

• 40:36Chris Benson?

  Yeah.

• 40:37Daniel Whitenack?

  So that was one thing and we needed a way to queue jobs. And the fact that, you know, we could also integrate that with our, you know, runs on Google Colab and all that was was really nice. Now, we do use also, so this is kind of for our training jobs. We do also use other solutions for kind of persistent data pipelines. We use Pacaderm for that, which actually allows you to kind of create and subscribe to data sources. And Pump those through to, you know, update data sets in a version sort of way. So we use that for other purposes, but ClearMell gave us that sort of experimentation for NLP research.

• 41:18Daniel Whitenack?

  And what we found is we can run our experiments there, but then we can also kind of, and this is where I wanted to kind of get with the connecting the pieces. So we're using notebooks, we're using GPU server, but then these models and data sets that we're creating in our experiments, we can upload those now to Hugging Face data sets hub, models hub. Those are versioned in that model hub. And then any inference we can do, we could either use the Hugging Face inference API, or we could call down the model using the Hugging Face libraries into Python code where we could serve that model in, you know, a custom application of some type.

• 42:01Chris Benson?

  Right.

• 42:02Daniel Whitenack?

  It provides a really then flexible route towards inference as well, because if you can now store your models in a sort of standardized way in a hub and have a standardized way of serving them, it lets you have kind of a consistency and efficiency around how you do that bit as well.

• 42:21Chris Benson?

  I'm curious about one of your constraints in general, aside from, because I understand the solution you've taken us through so far. Are you able to keep things kind of cloud based, or at least locally on that server and its immediate environment? Do you have any kind of edge considerations that you have to push to? Is that part of your requirement or not?

• 42:41Daniel Whitenack?

  So it depends on the project. So some of our projects, the end target for deploying these models is, is it some inference server in the cloud? So that's part of what we deploy to. And that could be, like I say, either a custom inference server that we've built, or something that we're deploying to an inference, hosted inference service. But we all also deploy to edge devices, particularly some of our speech solutions are running on edge devices. That does have other concerns like you're talking about, but the nice thing is if you've got your models, for the most part on our edge devices, they are connected to the internet.

• 43:21Daniel Whitenack?

  So we can ship a Docker container to those edge devices. And if we're, for example, downloading a model from Model Hub or S3, it can directly download that version of the model from there on at runtime. So we have a little bit of flexibility there on the edge devices because they are connected to the internet.

• 43:42Chris Benson?

  Gotcha. I'm curious if, and I know a while back you took us through some specs in a previous episode for GPU server that you had. Is this the same one that we're talking about or is this a different one?

• 43:55Daniel Whitenack?

  That's my one-off DIY build that you're talking about, the previous one, which was an interesting build and I still use it more for sort of the one-off things that I'm doing.

• 44:05Chris Benson?

  Yep.

• 44:05Daniel Whitenack?

  This server is, it's a rack mount unit with A100s in it. So A100 GPUs.

• 44:11Chris Benson?

  Yep.

• 44:12Daniel Whitenack?

  Which is another thing is nice about that solution is we can, the A100 GPUs have this Meg technology which lets you split up the GPU into sort of multiple virtual GPUs.

• 44:25Chris Benson?

  Yeah.

• 44:25Daniel Whitenack?

  Which is really great because we don't all the time, yeah, we don't all the time need to run like big jobs on, you know, whole combinations of our GPUs. We might need to run a whole bunch of small training jobs, right?

• 44:38Chris Benson?

  Yeah, you get a lot more utility.

• 44:40Daniel Whitenack?

  Yeah, exactly. We can split them up and sort of slice and dice them the way that makes sense for, you know, the season of research that we're in. I think that's a really beneficial thing. So yeah, I would highly recommend people look into that technology if they're, if they're able.

• 44:56Chris Benson?

  It is. That was my single favorite feature when the A100s came out was that ability to do that instead of having, you know, having to use them all. It was otherwise, once you had the A100s to look back, felt very inefficient in terms of how you were going into training. I'm curious, I want to go back, go and hit a really basic question that people are facing a lot. And that is, how did you determine your crossover for the Organization for when it needed to have an on-prem server versus when it could use cloud resources, whether that be Google Colab or AWS or any of the others that are out there. How did you make such a determination?

• 45:32Daniel Whitenack?

  Yeah, I think it was basically when we looked at for the year, kind of an estimate of this sort of scale of training that we would need to be doing for our models. And we realized that like we would be making back our cost on the GPU server with the amount of training that we're doing. I think it's partially that. So there was a break even point there. But then also it was when we realized kind of we could develop also some operational efficiencies by having people kind of centralized their jobs on this server in these queues by kind

• 46:13Daniel Whitenack?

  of combining all of these people's work rather than like this person over here spinning up a GPU server in the cloud and this person over here spinning it up and this person over here spinning it up and they're not utilizing all of those GPUs to their full extent, right? By utilizing like a more of like a job queue thing, then we could do that. Now you could also spin up a cloud server and implement like similar queues and such. But yeah, so there's a variety of options. There's also increasingly favorable options for running these things in the cloud. So it's still not a kind of story that's that's finished. I don't.

• 46:49Chris Benson?

  No, it's definitely evolving, but we are still at that point where there seems to be a crossover as you get more capable, sophisticated and you're running models on a more consistent basis and not just doing it for a short while, you know, each day kind of thing. And it's sitting there not running during those off hours. Once you get to that point, it definitely seems to pay in the current economics to go there, go that way.

• 47:12Daniel Whitenack?

  Yeah.

• 47:12Chris Benson?

  So as we turn to other, other things, do you have any, any learning resources we're sharing today?

• 47:19Daniel Whitenack?

  Yeah, I mean, I don't have that many, but I did want to point people to a quick thing that I saw, which I do think was really cool for people to explore, which is pandas tutor.com.

• 47:33Daniel Whitenack?

  So pandas tutor.com, if you go there, this is a way for you to visualize your panda, Python pandas data transformation, which is really cool. So for those that don't know, pandas is a way to construct data frames of sort of tabular data in Python. It's kind of ubiquitous in the data science AI world. It's very common to use, but it's so powerful. There's so many data transformations that you can do. Sometimes it's hard to visualize and strategize about like what your code's actually doing and some non-intuitive things can happen.

• 48:14Daniel Whitenack?

  From my perspective, a really cool thing that this is addressing is helping people gain that intuition about what certain transformations and pandas code does to data. And it does that in a very visual way.

• 48:28Chris Benson?

  It does, looks pretty cool. I'm looking through it now. Thank you for sharing this.

• 48:32Daniel Whitenack?

  Yeah, for sure. And you know, there's a, there's also a bunch of things going on in the month of December around like the advent of code and 27 days of JAX. I've seen that. So there's a whole bunch of kind of every day of December, trying to do some coding thing out there. So if you're into that, you could look up some of those things as they're always good learning experiences. But yeah, I wanted to share this pandas thing that I, that I ran across.

• 48:59Chris Benson?

  Well, thanks. You know, you've really taken us on a bit of a tour today between OpenAI and Hugging Face and then the, how you guys put together your current approach to training. So thank you for sharing that.

• 49:13Daniel Whitenack?

  Yeah, for sure. It's fun, Chris, and I hope you have a good rest of the week. I'm gonna put on my heavy coat and go through the cold back to my apartment because now it's winter. But yeah, appreciate the conversation. Looking forward to chatting next week.

• 49:27Chris Benson?

  Absolutely. Talk to you then.

• 49:32Speaker 05

  That's our show. Thanks for listening. For more like this, check out our master feed. It is all Changelog podcasts in one easy to consume place. Let your podcast app snag everything we produce and then pick and choose which ones to listen to. Subscribe today at changelog.com/master or just search for Changelog Master in your podcast app of choice, you'll find it. Special thanks to Brake Master Cylinder for providing our music and to our longtime sponsors, Fastly, Launch Darkly, and Linode. That's all for this week. We'll talk to you again next time.