Accelerating ML innovation at MLCommons

MLCommons launched in December 2020 as an open engineering consortium that seeks to accelerate machine learning innovation and broaden access to this critical technology for the public good. David Kanter, the executive director of MLCommons, joins us to discuss the launch and the ambitions of the organization. In particular we discuss the three pillars of the organization: Benchmarks and Metrics (e.g. MLPerf), Datasets and Models (e.g. People’s Speech), and Best Practices (e.g. MLCube). Sponsors: Code-ish by Heroku https://www.heroku.com/podcasts/codeish – A podcast from the team at Heroku, exploring code, technology, tools, tips, and the life of the developer. Check out episode 98 https://www.heroku.com/podcasts/codeish/98-the-ethical-side-of-deep-fakes and episode 99 https://www.heroku.com/podcasts/codeish/99-the-technical-side-of-deep-fakes for insights on the ethical and technical sides of deep fakes. Subscribe on Apple Podcasts https://itunes.apple.com/us/podcast/code-ish/id1457879893 and Spotify https://open.spotify.com/show/5VvkHRhd9E9SQf7i9HsPMd. Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Knowable https://www.knowable.fyi – Learn from the world’s best minds, anytime, anywhere, and at your own pace through audio. Get unlimited access to every Knowable audio course right now. Click here https://www.knowable.fyi to check it out and use code CHANGELOG for 20% off! Featuring: David Kanter – GitHub https://github.com/thekanter, X https://x.com/thekanterChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Learn more about People’s Speech https://mlcommons.org/en/peoples-speech/Get involved with the People’s Speech project https://mlcommons.org/en/groups/datasets/MLCube GitHub, including several different examples https://mlcommons.github.io/mlcube/MLCube Mailing list https://feeds.transistor.fm/[email protected] Training Benchmarks https://mlcommons.org/en/training-normal-07/MLPerf Training HPC Benchmarks https://mlcommons.org/en/training-hpc-07/MLPerf Inference Datacenter Benchmarks https://mlcommons.org/en/inference-datacenter-07/MLPerf Inference Edge Benchmarks https://mlcommons.org/en/inference-edge-07/MLPerf Inference Mobile Benchmarks https://mlcommons.org/en/inference-mobile-07/MLCommons on Twitter https://twitter.com/commons_mlUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This is, I think, about thirty or forty X bigger than anything that's publicly available today. And I think the key thing is, for speech-to-text models, you need about ten thousand hours to actually produce something that's functional. Now, I don't claim that this will give you a truly production-worthy model, but I think it can really lower the bar for a lot of people, and it's very exciting to me.

• 0:26Speaker 01

  Being with for Change Log is provided by Fastly. Learn more at fastly.com. Our feature flags are powered by LaunchDarkly. Check them out at launchdarkly.com. And we're hosted on Leno cloud servers. Get a hundred dollars in hosting credit at leno.com/changelog. Hey, friends. This episode of Practical AI is brought to you by Codish, a podcast from the team at Heroku that explores code, technology, tools, tips, and developer life. There's tons of great conversations on the Codish podcast, so I would encourage you to check it out and subscribe. But in particular, I wanted to bring to your attention two episodes, episode ninety-eight and ninety-nine, where Julian Duque explores the ethical and technical sides of deep fakes.

• 1:07Speaker 01

  The rise of manipulated pictures and videos and other forms of computer-generated media are able to cause uncertainty and doubt in what we see in here online. So how are we able to use these tools for good, if at all? Here's a sneak peek.

• 1:22Speaker 02

  Let's say we want to do a deep fake of my voice. And we train the model, and we have enough data and everything. This will be also able to imitate my accent, for example, like how I pronounce English and the strong pieces of my accent. Or is not there yet?

• 1:43Speaker 03

  It really depends. If there would be a person with similar accent on the input, then it would be fine, but it's kind of cheating. You can think it's cheating because we're reusing accent of a different person that's similar to your accent. But if it would be like an American native speaker or a British, a person with a British accent. Or like whatever other accent, then it will kind of be a mixture on the output. So we're not there yet in terms of converting accents. It's a little bit more difficult than we initially anticipated because like when we started the company, we thought it would be, you know, we'll kind of solve it in a year or something.

• 2:24Speaker 03

  But then it turned out that, whoa, no, we're here for much longer.

• 2:30Speaker 01

  Check these episodes out. Links are in the show notes to both episodes, or head to heroku.com/podcasts to listen and subscribe. Again, check the show notes for links or go to heroku.com/podcasts.

• 2:57Speaker 04

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter here at Practical AI FM.

• 3:23Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by my co-host, Chris Benson from Lockheed Martin. How are you doing, Chris?

• 3:39Chris Benson?

  I am doing very well. Happy New Year. How are you doing today?

• 3:42Daniel Whitenack?

  Happy New Year. Doing well. Got some snow on the ground. It definitely looks like winter up here in the Midwest. And yeah, it was a good New Year. Helped my wife with a few things with her business over the holiday. And also I ripped apart our bathroom in an attempt to remodel it. And I am mostly, so I'm on the rebuilding part of that. So it still has momentum, but it's not done yet to my wife's disappointment entering the New Year. But it will be done. I'm committed.

• 4:18Chris Benson?

  It sounds good.

• 4:19Daniel Whitenack?

  Yeah.

• 4:20Chris Benson?

  As we kind of swing, get back in the swing of things, today was my first day back into work after a little vacation time.

• 4:26Daniel Whitenack?

  Same here.

• 4:26Chris Benson?

  So I'm trying to like readjust my whole brain into actually having to be productive, which is a scary thing.

• 4:32Daniel Whitenack?

  It's absolute chaos coming back, right?

• 4:36Chris Benson?

  It is, but I took advantage. I'll tell you real quick. I took advantage of the holidays. I've started flying lessons.

• 4:42Daniel Whitenack?

  No way. That's super cool.

• 4:44Chris Benson?

  And over the holidays, I got in about a dozen flight hours, you know, getting close to solos. I'm sharing that with our audience. Now, if I don't finish this thing, everyone's going to hammer me.

• 4:53Daniel Whitenack?

  Yeah.

• 4:54Chris Benson?

  But I'm hopefully I'll eventually within a

• 4:56Daniel Whitenack?

  You'll be able to fly from Georgia up here to where I'm at and we can do some in-person recording.

• 5:02Chris Benson?

  There we go. Perfect. Exactly.

• 5:04Daniel Whitenack?

  Come, come every Monday.

• 5:06Chris Benson?

  But I think you scare me because I mean, I'm, I'm in the sunny south where it's a, a, a terror. It's a just a frigid 55 degrees Fahrenheit right now.

• 5:14Daniel Whitenack?

  You're all bundled up too. You got your jacket hoodie on.

• 5:19Chris Benson?

  I'm afraid to go north now.

• 5:22Daniel Whitenack?

  Well, with that.

• 5:24Chris Benson?

  Make that transition, Daniel. Go for it.

• 5:25Daniel Whitenack?

  Something completely different, which I'm really excited about actually. So there's a couple of things that got me really interested in this. When we got connected with this group, one is that this is a sort of, it's a new thing. It's a nonprofit thing, which is related to my world because I work for a nonprofit. But there's a crossover with some of the things I'm working on because they're working with speech data and are releasing some, some data for speech to text and other things. But I'm really excited today because we are joined by David Kanter, who is the executive director at ML comments. Welcome, David.

• 6:06Speaker 00

  Happy new year and it's a pleasure to be here.

• 6:08Daniel Whitenack?

  Yeah. Happy new year to you as well. It's great to have you with us as our first official recording of the new year.

• 6:14Chris Benson?

  First victim. Sorry. It's not really. Just joking.

• 6:19Speaker 00

  No, it's, it's actually great to be here, although it is kind of amusing. So I'm in, in San Francisco and you know, I think the weather is kind of very similar to, to that in Georgia. But you know, I'd note, we're the two wearing coats and the gentleman surrounded by snow is like, no, I've got a t-shirt. Full blown t-shirt, like looking relaxed, like might even have sandals on. I don't know.

• 6:41Daniel Whitenack?

  Yeah. Well, yeah. I mean, with Zoom, you don't even know if I have pants on, but uh,

• 6:47Chris Benson?

  Not a good image.

• 6:48Daniel Whitenack?

  Yeah, we'll leave that. Maybe just assume there.

• 6:52Chris Benson?

  He had to go there again.

• 6:53Speaker 00

  Interpolating you with pants on.

• 6:55Daniel Whitenack?

  Okay. That's good. That's good. That's probably a good interpolation. Anyway, would you just tell us a little bit about yourself? What you've done in the past and how you ended up doing what you're doing now with ML Commons?

• 7:08Speaker 00

  Yeah, absolutely. So, you know, actually, I'd say in some ways, my background in ML dates back to when I was in university at the University of Chicago, where I did math and economics. And, you know, a lot of econometric analysis is actually very similar to some forms of machine learning. But from there, I was very heavily involved in sort of computer architecture, microprocessor design. I had a startup for a few years. And then, sort of, when I shut that down, I ended up getting into a lot of consulting and ended up falling in with the predecessor to ML Commons, which was a collaborative benchmarking effort called MLPerf. And, you know, in a lot of ways, MLPerf brought together the community of machine learning

• 7:53Speaker 00

  and then sort of system architecture, computer engineering, and so forth. And then, you know, as you mentioned, we formed ML Commons, which is a nonprofit, a global engineering consortium to bring everyone together around our overall goals. And so, you know, I very much come from the technical side, again, on sort of the performance aspects. But over the last two to three years, I've really, really leveled up my understanding of machine learning quite considerably, both through my consulting and, you know, now my day job.

• 8:29Daniel Whitenack?

  Yeah, and MLPerf, I remember, so when did that come about? Because that's been around for some time, right? I remember seeing things about it.

• 8:38Speaker 00

  I want to say 2017 or 2018.

• 8:44Daniel Whitenack?

  Gotcha, yeah. So actually, it's interesting that MLPerf was a sort of predecessor of ML Commons. And now ML Commons is sort of, it's like the parent organization, is the way to put it, of MLPerf.

• 8:57Chris Benson?

  Was it the first, you know, thing inside the association?

• 9:00Speaker 00

  Yeah.

• 9:01Chris Benson?

  It was the Linux inside the Linux Foundation?

• 9:04Speaker 00

  That is a very good way to put it, yeah. And, I mean, in many respects, you know what? The genesis story is that, you know, our DNA is very much like a startup and everything kind of started off very informal handshake agreements, do things with consensus and move very fast. But one of the things we agreed on is, you know, that that will only work for so long, but we wanted to, you know, ultimately put this in the hands of, you know, an industry organization that was, you know, owned by none, beholden to all, so to speak. And so ML Commons is that container and but but the other thing is, you know, we knew that this was one leg of the tripod, right?

• 9:48Speaker 00

  And that we had other projects that we wanted to get done. And so, you know, it was very much the camel's nose under the tent, so to speak.

• 9:58Daniel Whitenack?

  Yeah, good metaphor. Yeah. That's actually a new one on me. I'm gonna have to start, uh, start using that one.

• 10:04Speaker 00

  Yeah.

• 10:05Chris Benson?

  So before we go too far, just for those who have not heard of ML perf, since we're kind of leading into you into the story with that, can you give a brief outline of what ML perf was and then kind of go on to talk about what the container is of ML commons and how it's structured and and then maybe we'll we'll get into some of the other things as well?

• 10:25Speaker 00

  Sure. So ML perf is a set of performance benchmarks to measure how fast you can train a neural network or do inference. And, you know, there's a very long history of this sort of benchmarking in computer systems. And so in a lot of ways, like the effort was, you know, sort of spiritually influenced by by those prior efforts. You know, organizations like SPAC and TPC that help to, you know, sort of bring together industry and academics and kind of give folks a common set of benchmarks. And what we really noticed early on is, you know, there's so much excitement around machine

• 11:06Speaker 00

  learning, around AI, but, you know, it was almost like the UN without their translation devices and people would be talking about performance in ways that were very much not apples to apples. And so the sort of the goal of the benchmarking side of things. Is to both provide a common set of metrics for what constitutes progress, right? A benchmark in some sense is a barometer on progress. And, you know, does it always point true north? You know, not exactly, but if it's off by five or six degrees, it's fine. You're still heading in the right direction. But it's also about giving everyone a common language for where you are in fact heading,

• 11:46Speaker 00

  whether it's sales and marketing or engineering or academic research or even folks in the government and sort of clarifying the field and, you know, providing a map and a barometer in some sense.

• 11:59Daniel Whitenack?

  Yeah, it seems kind of interesting to me to think a lot about this, this side of benchmarking. So I think where a lot of people, if you were to talk to a lot of like AI practitioners or ML practitioners and talk to them about like performance or leaderboards or something like that, they probably immediately think of like, oh, some list of models and they got like this accuracy on ImageNet or they got this blue score on this machine translation task or something like that. And it's really a measure of the performance of the output of the model, not of its performance on hardware.

• 12:40Daniel Whitenack?

  And actually, you kind of see as these leaderboards have been released, people put more and more effort in and actually the models at the very top might be very poorly performant on a, on a lot of, or maybe even like some lowly people like me might not be able to run them on any hardware that we own. So how has, you know, your conversations with people over time around ML perf and actual like performance on hardware overlapped or, or come into conflict with those other sort of conversations around like, oh, how accurate can we be and that sort of thing?

• 13:19Speaker 00

  Yeah, no, I mean, I think that's a very, very on point observation, right? Which is that, um, you know, what you've described is what I think of as accuracy, right? And that's a very, you know, common, common set of benchmarks and. You know, I think one of the actual sort of core motivations behind what we do is that there is actually a very foundational paper by one of our founders when he was at Baidu, Greg Dymos. And sort of the rough outline of the paper is that accuracy for ML follows a relatively simple equation, which is you get a lot of compute and a lot of data together. And as you crank those up, you will increase your accuracy.

• 14:04Speaker 00

  Right? And, and, and you've seen these papers from OpenAI and other people talking about, oh, our need for compute is, you know, doubling every six months or something like that. And so, you know, I think the good thing is that there are absolutely folks who are very cognizant about the importance of this. But it's also true that there's, you know, a large number of people for whom, you know, the performance is, is sort of a level below them. Right? And, you know, this goes back to, I think, sort of the goals of our organization, which is that when I look at machine learning today, it makes me think of what aeronautics was probably like around the turn of the 20th century. Right?

• 14:45Chris Benson?

  You're speaking my language now.

• 14:47Speaker 00

  Yeah, you know, you had dozens of people working on planes from the Wright Brothers to the folks in France who I can't remember their name, but it's all sort of custom and hand done. And, but then you look at where we are today, when you hop in a 787, and that is, you know, the process to make it, the output, just totally different. And to pivot to sort of the second part of your question about, you know, ML commons, you know, our goal is advancing innovation in machine learning to drive the whole industry forward. And so I, when I think about where we are today, there are a lot of digitally native companies that have tremendous AI capabilities. But then you look at how do we reach the rest of the world?

• 15:30Speaker 00

  How do we reach analog companies? You know, like someone like Goodyear, for example, or Macy's, you know, as opposed to Amazon, where I think of digitally native shopping. And so, you know, in order for that to happen, it can't be these handcrafted things. So you need a lot of standard components. And, you know, we have the, the beginnings of that. We have things like TensorFlow and PyTorch to help us along the way, but. You know, there's a well-proven recipe for industry growth. And so ML Commons is really built to address sort of three key things that we see there. Right. And one are these performance benchmarks and generally speaking, metrics.

• 16:15Speaker 00

  Right. And you know, if you think about the industrial revolution, you know, you want to make a precision airplane, you know, if you confuse your meters with inches, you end up with things like crashing the Mars lander, right? So, you know, we've got to get the measurement down. Then we need our sort of raw materials and you know, you've had folks on the show before like our colleagues at Mozilla and Unsplash or you referenced ImageNet, right? And this is the raw ingredient to AI and ML, right? Which is data sets and in fact, you know, I think in there, we're very much influenced by ImageNet, right? And they provided the resource that catalyzed this latest sort of revolution in machine learning, right?

• 16:59Speaker 00

  AlexNet and convolutional nets beat humans at image classification and sort of the rest is, you know, an adventure we're living out. And then sort of the third pillar of ML Commons is what I call best practices, right? Which is, you know, how do we ensure model portability? How do we get from where we are today, which is, you know, if you look at BERT, right? Which is just, you know, profoundly changed how everyone thinks about NLP. It was a paper and maybe you got some code, but it would take you probably a month to go from paper and code that works at Google on their internal infrastructure to I could run this on the compute resources that I might have at Lockheed Martin.

• 17:44Speaker 00

  And so if we can shorten that time down to a day or hours to allow reproducibility of these models, to allow portability, that would be a tremendous boon to the whole industry. So it's sort of, those are the, I think, the three things that we see. Really driving the industry forward and helping to bring the benefits of machine learning, whether it's, you know, translation, speech recognition, you know, self-driving cars, medical diagnostics to the whole world.

• 18:36Speaker 04

  Have you heard about Knowable? It is an awesome new platform for learning from the world's best minds anytime, anywhere, at your own pace, through audio. Learn about the performance benefits of a plant-based lifestyle from NBA All-Star Chris Paul, or how to launch a startup from Reddit co-founder Alexis Ohanian. There's even a 10 lesson course from astronaut Scott Kelly. Here's a sneak peek.

• 19:01Speaker 07

  We learned a lot up there, but what can you learn from a life in space? The answers might surprise you. In this Knowable course, I want to share some of the things I've learned that you might not expect. Lessons about leadership on a dark night on an aircraft carrier in the middle of a churning sea. Lessons about the fear you feel with 7 million pounds of thrust exploding underneath you. And most of all, there's an idea out there that astronauts are always perfect. Failure is not an option, right? That's why I want to take you through some of my life experiences to show you how that's just not true. I believe everyday, regular, human failure, if we handle it right, can be one of our

• 19:44Speaker 07

  greatest opportunities to learn, grow, and succeed.

• 19:48Speaker 04

  Knowable is accessible on your phone and on the web, and each audio course is broken out into individual lessons, usually around 15 minutes long. As a Change Log listener, you can get an annual membership to Knowable for 20% off. Get unlimited access to every Knowable audio course right now. Just download the Knowable app or visit knowable.fyi and use code CHANGELOG for that 20% discount. We put a link in your show notes for easy click-ins. Check out Knowable today and start learning from hundreds of top experts from around the world. Once again, that's knowable.fyi, code CHANGELOG.

• 20:35Daniel Whitenack?

  So David, I'm pretty curious about, I guess, the structure and origins of ML Commons in the sense that I'm kind of reading about this really exciting group of people that's joined together to help form ML Commons with initial founding board, including representatives from people like Alibaba and Facebook AI and Google, Intel. But then you've got people involved, researchers from academia and even startups. So could you kind of describe a little bit of the origin story, I guess, of how this organization came to be? And what was it like? Was it like these people from all of these different companies on

• 21:17Daniel Whitenack?

  a Zoom call and just saying, hey, this thing needs to exist. Remember that ML perf thing? Maybe we need to like create this other thing. Or has it been sort of forming and is it rooted in kind of deeper things like ML perf and that sort of thing?

• 21:33Speaker 00

  Yeah, so it really was, I think, you know, one of the things you tapped into is, you know, bringing people together and forming a really dynamic community. And, you know, ML perf, you know, really grew organically. And it did start, and I'd say, you know, of our founders, Peter Mattson from Google, who's the president, and probably talk to him almost every day, even when we're on vacation, you know, really kind of dreamed it up. And we got started with probably an equal mix of academics and industry folks like, you know, the folks you mentioned, Nvidia, et cetera. And that was sort of the, you know, I think we had 70 or 80 companies with representatives involved in ML perf.

• 22:15Speaker 00

  And, you know, sort of at the tail, the start of last year, we kind of got started on forming a nonprofit that would become ML Commons. And then actually in December is when we sort of formally launched. And so, you know, a lot of these companies, a lot of these researchers, you know, became sort of official members. And, you know, that is what got us started. But I think it was the excitement around the benchmarks and just having industry standard benchmarks and establishing a reputation for, you know, we're doing things that are fair to everyone, right? Startups, you know, respective of what country they're in, and useful, and seeing that, yeah, this is a community that people can trust and work with.

• 22:58Chris Benson?

  So as you talk about the community, one of the things I'm wondering is there are these other communities, like, you know, Google has built a community around TensorFlow. Nvidia has built a community around their hardware platform and the software that supports it. There's the PyTorch community. And as you've built out the ML Commons community, how do you go about, you know, interfacing with those other communities and the ecosystems that those communities have built? Because you essentially have multiple ecosystems that are all trying to work together for that total solution. So how does that look from your perspective?

• 23:32Speaker 00

  Yeah, so actually, I mean, one of the things that we're, you know, very blessed by is that, you know, a lot of the folks who are pioneering key pieces of infrastructure in ML are members. So, you know, TensorFlow and PyTorch, you know, we have representatives from Facebook and from Google. So, you know, when we run into something or have a question, you know, sometimes we're lucky enough to get the right people there in the call to start with. But, you know, it's pre-COVID days, a tap of the shoulder away, and, you know, now probably an instant message or Slack or Facebook message away. So I think that has actually been a tremendous strength of ours. But, you know, a lot of these communities are, you know, almost orthogonal to what we do. I mean, like, you know, our benchmarks,

• 24:15Speaker 00

  for example, are full system, right? So it incorporates software, systems, cloud, hardware, you know, compilers, sort of everything. But, you know, we do very much, you know, we're a global and open standard, right? And in some sense, you really have to be as a benchmarking organization because there's this intrinsic judging quality to it. Right? And, you know, just as this in the judicial system in the United States, right? You know, the judging organization, so to speak, must be above reproach, must be open and equal to all, right? And that is very much our goal. So You know, it's not that they're members per se, but you know, we want to work with everyone, right? We're all fellow travelers.

• 24:56Daniel Whitenack?

  Good way to put it. I like that a lot. So we've talked a lot about ML perf. Maybe we could dive into a couple of these other pillars that you mentioned. You mentioned data sets. I'm looking at your website now. You talk about data sets and models publicly available and can form the foundation of new capabilities. And specifically, I see you mentioned the speech data set, the world's largest public speech to text data set, which is maybe your first foray into this. So could you maybe describe your vision in the longer term for this data sets component of ML commons and then how you got started with this speech data set?

• 25:36Speaker 00

  Yeah, absolutely. So I think, you know, you've sort of hit the nail on the head, which is that, you know, we want to be providing and drive the industry forward with these data sets, you know? I think of our speech project as ImageNet for Speech in shorthand, right? That's sort of like the 30 second pitch to someone in ML. And, you know, as I look out, there's a lot of other areas that could use this sort of public data set. Now, my colleague Vijay Reddy, who's a professor at Harvard, he did a study with his graduate students. And one of the things he found is that even at companies that have huge amounts of internal data, like, you know, a Google or an Alibaba, most of their research

• 26:18Speaker 00

  studies are done with public data sets because that gives you a level playing field to assess accuracy, right? And you get reproducibility and all of these things. But we want to push forward with data sets in all these different areas because that will ultimately democratize access to this technology and thereby benefit everyone. So that's, that's sort of the vision. And as you say, right, the people's speech is our first data set. And right now, we actually, late last year, sort of started sampling it to partners. So to our members that have speech teams and, you know, if there's anyone out there with a speech team that's interested, you know, this is, I think it's about 10 years worth of audio.

• 27:01Daniel Whitenack?

  I'm raising my hand. No one can see that on Zoom, but I'll have to follow up afterwards.

• 27:06Speaker 00

  No, absolutely. We should totally talk.

• 27:09Speaker 00

  You know, and it's big. It's 10 terabytes of data. And actually, one of the pieces of feedback we got was, for many people, it's actually a little too big. So we need to, you know, slice it down into smaller chunks so that, you know, individuals can play with it, or, you know, a university. But we're getting it out to field test because, again, the hypothesis here is this is, I think, about 30 or 40x bigger than anything that's publicly available today. And I think the key thing is, for speech-to-text models, you need about 10,000 hours to actually produce something that's functional. Now, I don't claim that this will give you a truly production-worthy model, but I think it can really lower the

• 27:51Speaker 00

  bar for a lot of people. And it's very exciting to me. You know, there's two folks in my life who actually really benefit from speech technology. You know, one is my mother, who she had a stroke a few years back and has difficulty reading. But she uses speech-to-text in her phone all the time. And so, you know, that's great. She speaks English. And both English and Mandarin have very robust systems, but it doesn't take long before you get to other very commonly spoken languages, like Portuguese, where there's really almost no public data. But that's a language that's spoken by 300 million people, you know. So if we can help push those frontiers out, you know, I would love to be able to extend

• 28:35Speaker 00

  and play a role in extending speech technology to sort of, you know, the next several billion people, you know, this decade.

• 28:43Daniel Whitenack?

  So I'm kind of interested. I know, as I was kind of indicating while you were talking, I'm very interested in this because we have our own speech projects going on in our organization. And we're, you know, as our organization's mission is to extend, you know, benefits of these types of technologies and other things to local language communities. I'm super interested. But I know from experience, this is a lot of work putting these data sets together. So how does a sort of collaborative nonprofit, which is sort of, I imagine, kind of... Amorphous and changing a lot, you know, how do you like put the right pieces in place

• 29:25Daniel Whitenack?

  to make sure that this this can be accomplished at this scale?

• 29:30Speaker 00

  Yeah, well, so the good news is we are more official than three raccoons in a trench coat.

• 29:34Daniel Whitenack?

  Good, good.

• 29:35Chris Benson?

  I like that. I like, that's a good one.

• 29:37Speaker 00

  Yeah, I don't know if you guys have seen Bojack Horseman, but there's a great character Vincent Adultman, who is a take on that, who is literally three children inside a trench coat, who goes to work and does a business. You know, that's sort of my prototype.

• 29:51Daniel Whitenack?

  Yeah, yeah. And I know, like, for example, our nonprofit, there's a lot of like, you form a lot of partnerships and at some point, it's like, you know, it gets a little bit hard of like, where does like this entity end and this one start in terms of like what they're doing and like, these people are with this organization, but they're working on the like, you know, maybe there's people in Facebook or Google, but they're working with and on ML Commons projects, you know, how does that play out and how do you staff up to, you know, do this data set work?

• 30:26Speaker 00

  Yeah, so I think, you know, that kind of dives into, you know, some of what our organization provides, right, which is, so we do have a budget and we do use some of that for staff, you know, hire consultants and folks to help us out. We have a core of like very dedicated folks who are, you know, incredibly interested and focused on speech and, you know, a lot of those folks have kind of gravitated towards us, you know, and collaborating with others is absolutely part of this. But I do see some of the benefits that we have is that, you know, through the community that is ML Commons, we have access to tremendous, you know, compute resources, right? So, you know, if we want to start training a model on this 87,000 hours of speech, which we do, you know, we can go

• 31:10Speaker 00

  to some of the world's top cloud providers and, you know, work with them to to take advantage of the resources that they have and collaborate together. But, you know, it is true that as a volunteer project, one of the biggest challenges is, you know, sort of turnover. And this is true in graduate schools as well. And actually, this is one of the things that I think is sort of a lesson from ImageNet. That, you know, we're trying to build on and improve, which is the ImageNet folks did something amazing on a very tight budget. And we want to reproduce that. But we also think that, that sort of the techniques that go into these, building these data sets are durable and that by putting a lot of this expertise within an organization, we can drive down the cost over time. Right?

• 31:54Speaker 00

  Because there's all sorts of tricks of the trade that, you know, are very much embodied in people and may not get written down. And so when you have people, you know, rotating in and out, you kind of lose that. But by having an organization behind that where it's all written down and, you know, we have some continuity, you know, folks like myself and other leads, you know, I, I think there is a great opportunity to really build up, you know, momentum that is beyond a single project.

• 32:27Speaker 04

  ChangeLog++ is the best way for you to directly support Practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of Practical AI into the future. Simply follow the ChangeLog++ link in your show notes or point your favorite web browser to changelog.com/plusplus. Once again, that's changelog.com/plusplus.

• 33:20Chris Benson?

  So David, I'm kind of curious. I know that one of the pillars that you have that is around best practices is called ML Cube. I'm wondering if you could start us off by describing what that is, what its conventions are and such, and kind of give us a quick intro to that.

• 33:36Speaker 00

  Yeah, so that is a great question. ML Cube is really, I think of it as a set of conventions around ML models. And it's sort of, it's distinct from containerization, right? You know? Obviously, many folks are going to be familiar with Docker and so forth.

• 33:57Daniel Whitenack?

  Yep.

• 33:57Speaker 00

  But, you know, you can't take a Docker from inside Google containing BERT and expect that to run out of the box on AWS or, you know, systems that don't allow for containerization, right? So some of the organizations that we work with are supercomputer sites and some of those are classified supercomputers or portions of their infrastructure are classified. They have all sorts of rules about what can and cannot work.

• 34:25Daniel Whitenack?

  They do indeed.

• 34:26Speaker 00

  Yeah, right? I'm sure this is like, you know, every day for you, right? So how can we get a set of conventions in a way that will help you move models around transparently so you can pick it up from one set of infrastructure to another and just, you know, be up and running? And, you know, it might not give you the best performance, but it will actually run and allow you to reproduce things. And so that's kind of the goal of MLQube and we're getting, you know, we're always very interested in volunteers, folks to use it because ultimately what I would love to see is, you know, a lot of these very common models that people use as references, whether

• 35:04Speaker 00

  it's ResNet or BERT, to be packaged up in MLQube and so that, you know, you can trust that when you grab something from the model zoo, you know, it's just going to run wherever you are and, you know, you don't have to fight tooth and nail with whatever the underlying infrastructure is. And I think that would just make everyone's life, you know, a lot easier.

• 35:26Daniel Whitenack?

  Yeah, I definitely agree. Speaking after some, some, some interesting experiences even this last week, trying to get a number of random models to, to run, run locally. It's, it's difficult and it's, and people are eager because they see these like really awesome things coming out in research papers and oh, this happened at Google or this happened in OpenAI. And it seemed, you know, almost instantly it's up on GitHub, but then like actually running it and doing something useful with it is, is a whole nother game. So in terms of like, let's say I'm an AI practitioner and You know, you mentioned sort of Model Zoo and sort of portable models.

• 36:10Daniel Whitenack?

  What are you envisioning with MLCube? My workflow would be if I was either contributing some model to MLCube or some method? And also on the other side, so, you know, being a consumer of that and integrating it into my own work.

• 36:27Speaker 00

  Yeah, no, I mean, that is a perfect question. Actually, this is one of the things that, you know, it makes me very glad that we had certain deliverables in our launch, which was the MLCube team put together a great tutorial on how to publish a model and package it up with MLCube. And so I think we have a couple of examples up already. And like I would, you know, the beautiful future I'd love to see is where, you know, it's so low friction that everyone who's publishing research papers does it, right? And then, you know, the interaction with MLPerf is, you know, we have a bunch of models that we use to measure performance and those run on like a huge variety of hardware. So this is something that is in our own interest as well, right? And seeing cloud providers, you know, package things up.

• 37:10Speaker 00

  So I've seen more people package things up would be great in part because I think a lot of what this does is creates huge convenience on the consumer side, as you point out, right? So that, you know, you can grab things and just get them up and running a lot more quickly. Now, I think especially for research and experimentation, this is tremendously powerful. Because, you know, if you're, if you're doing things in production, you're going to really, really need to crank out optimizations. But I think even in production, this is actually potentially very powerful. If you think about, you know, sort of a scenario where you have a model that you sort of centrally developed, but then you need to like maybe tailor a little bit for hundreds of different sites.

• 37:55Speaker 00

  Right, and so this could be a convenient form to pass that model around. And so, you know, we have some tutorials on how to, you know, the sort of hello world equivalent. And I think we might have one with MNIST and then one with some other sort of more classic ML stuff, potentially BERT. I'd have to check my notes to see. For, you know, here's how you publish and then here's how you consume. But again, it's really about how do we remove all of these sources of friction from the folks who are going to drive the industry forward and cutting-edge research?

• 38:27Chris Benson?

  Does most of that happen inside the working groups that you have? Or are there working groups for some things and then you, you know, how do you divide out the productivity of what's coming out of the whole process and in?

• 38:39Speaker 00

  Yeah, no, I mean, you totally nailed it. We, you know, DNA-wise, I think we're, we have a lot in common with open source organizations. Like, you know, in terms of principles, like by default, everything is open, it's on GitHub. You know, we have working group meetings, you know, usually once a week. And then we used to have, I mean, when we started, it was monthly physical meetings in the Bay Area and then quarterly. And I think one of the things that I'm very much looking forward to is, you know, we are an international organization and one of my goals for 2020 was to start having some sort of regular meetings in Europe and in Asia. And obviously that, that's on the agenda for the second half of this year.

• 39:20Chris Benson?

  Some unexpected things came up along the way. Couldn't imagine why.

• 39:24Speaker 00

  Yeah, no idea.

• 39:26Daniel Whitenack?

  Hopefully, this is not too bold of a question, but I think people might find it interesting because they see certain organizations, like Chris mentioned, you know, Linux Foundation or maybe Cloud Native or whatever it is. So you mentioned you had a budget, you had staff. Do the companies that are sort of subscribing and contributing and are part of the organization, are they contributing financially to make it happen? Is that how it happens, you know, practically?

• 39:56Speaker 00

  It's exactly like NPR. We have some fantastic mugs.

• 40:00Chris Benson?

  Yeah, I bet so.

• 40:01Speaker 00

  Although it's actually jackets. You can't see it because this is a podcast, but I am wearing one with the ML Purf logo.

• 40:07Chris Benson?

  Ah, yeah.

• 40:08Speaker 00

  I see there.

• 40:08Chris Benson?

  Chris and I can vouch for the

• 40:10Speaker 00

  We can vouch for the

• 40:10Chris Benson?

  Vouch for the style.

• 40:11Speaker 00

  Yeah. Yeah. So, right, so the membership is is open to individuals and academics and that is free. But for for companies, it is a paid membership. Now, you know, I think when we started out, actually, one of the questions we asked ourselves was, do we really need to create another organization? And one of the things that we actually, we clearly came to the conclusion of yes. And I think the motivator behind that is that we are unlike, there are a lot of engineering organizations that exist, you know, Linux Foundation, Open Compute, but none of them are actually focused on AI. And then there are a lot of AI organizations, you know, like the Partnership on AI and so forth, but none of them are actually focused

• 40:54Speaker 00

  on collective engineering. Right? And so that's kind of the sweet spot that we want to focus on and do. And, you know, you know, you have marketing focused organizations and we'll leave that to them. You have policy focused organizations. And again, that's a very complicated area. And we really want to focus on building things. That's kind of what's in our DNA.

• 41:18Chris Benson?

  I have a quick follow up to that. Over time and and across many episodes, Daniel and I have chit chatted around the fact that as AI really matures and it's really along the longer path merging with software development as there's a set of tools that are becoming available and they're becoming easier over time and standardized over time. So how do you see your future with these? Obviously, you're kind of in your sweet spot, but your sweet spot as it merges over time closer to other sweet spots that other organizations are filling. How does that change the nature of where you all are operating, not just your own organization, but the others that you're working with? You have any sense of that? You know, five years out, 10 years out?

• 42:02Speaker 00

  Ooh, that's a really good question.

• 42:03Chris Benson?

  Or did I just throw a complete random?

• 42:05Speaker 00

  No, no, no, no, no. So I think, you know, realistically, the the benchmarking and the metrics, you know, probably stay the same. And that that value is very independent. I think that, again, the data set creation is very unique. To ML, right? Because there's a lot of things where when it comes to benchmarking and other stuff, you can fake the data set and that's commonly done. But like, you know, by definition, that does not work in ML. Right? And so I can, I only see the need for building data sets growing over time as ML pervades more and more things, right?

• 42:36Daniel Whitenack?

  That's a totally fair answer, yeah.

• 42:38Speaker 00

  And on the best practices, you know, I mean, in some sense, like, let's take this to the limit, right? Maybe one day, you know, ML will be as pedestrian as Excel is. And at that point, you know, the best practices will, you know, will have done its job. I mean, but you know, it'll be decades before we ever get there, right?

• 43:01Daniel Whitenack?

  Yeah.

• 43:02Speaker 00

  And, and also, I think, to the extent that I think of best practices, I think of it as how do we remove frictions? There will always be new things popping up, right? Today, you know, model portability is probably, and interoperability is probably number one as, as we've said, but, you know, I think there's a very long list and, and as we go along, we'll find more friction that we want to help remove. Until we get to the saying, right, you know, the, the demo that is truly indistinguishable from magic, right?

• 43:33Daniel Whitenack?

  So how do you, with this sort of very collaborative group and so, so many parties involved, do you know, sort of, right now, like, I'm just thinking of data sets, how do you choose what's the next data set, you know, that you're going to focus on and what area it is and that sort of thing? Is that driven out of sort of feedback from the community? Or is there a kind of roadmap from the, from the board or like a combination of the two? How's that work?

• 44:02Speaker 00

  No, that's a very good question. So I think it is very much driven by, you know, sort of the member community and the board and right, and that is. You know, one of the keys to any effort like this is, you know, we don't really want to be taking money away from our member companies, but we obviously have to keep the lights on and do, you know, there's a lot of functions that we are providing. And so, but part of the value of being a member is helping to drive that forward. And so there are, you know, other things that we've identified as being sort of opportunities, you know, in terms of data sets. And that is sort of a board decision, but you know, the way the board works is, you know, is this is not like a Roman emperor, right? This is, this is actually genuinely community driven. We listen to our members.

• 44:46Speaker 00

  And then, you know, there's also a bit of vision involved, right? And that, and you know, vision is not, you just listen to everyone and do what they say. It's, you know, you have to start with an end in mind, right? And, you know, by nature, a community will, you know, generally pull you to where their, their focus is. But, you know, one of the, some of the things that we happen to think are very important are things like AI in medicine, right? You know, that's clearly huge potential for impact on the whole world, you know, self-driving vehicles, you know, another one, right? Tens of thousands of people are killed or injured in automobile accidents. And if we could cut that down by an order of magnitude, that would just be, you know, a huge benefit to society.

• 45:27Speaker 00

  So, you know, it's kind of a combination and there's a bit of art to it because it does draw on sort of all three of those things, right? Broader community interest, what the board thinks, and then a bit of vision in terms of where researchers and academics and folks really, really, really on the cutting edge think there is opportunity.

• 45:45Chris Benson?

  So you've actually just totally transitioned me into my next question because you were starting to get into vision. And so as we wind up here, I'd really be interested, you, you mentioned AI and medicine, you mentioned self-driving, but you personally, as you were helming the organization, you're creating all these relationships, you're engaged in the technology that's driving this forward and the algorithms, just clean slate, any kind of answer you want. What is your vision for where things are going over the next five years or longer, 10 years, whatever you feel comfortable with? You know, what's the thing that when you are going to bed at night and laying down and before you fall asleep is kind of getting you excited and thinking about, at some point, we're going to hit that?

• 46:28Speaker 00

  I mean, honestly, it's just really empowering new innovations.

• 46:35Speaker 00

  That are going to benefit everyone built on ML and AI. Right. And I think the speech to text is a great example, right? I can see, you know, a lot of our data is in English initially. But, you know, step two is how do we go from English to a dozen or 100 or 1,000 other languages, 1,000 if we're being very bold, right? You know, and you can start thinking about how you might leverage transfer learning to get from a speech to text system that has a solid command of English to other languages. Right. But, you know, again, medicine, my father's a physician. He's retired now, but, you know, I just see huge amounts of potential there. And, you know, will we necessarily be building the pioneering innovations,

• 47:18Speaker 00

  things like BERT that really do drive the whole industry forward? No, but we're going to be filling out the gaps around that. And so I think just knowing that our data sets are going to help catalyze things going forward, that we're going to make everything more efficient. Like, those are the things that are really exciting to me. And it's a tide that lifts all ships. Right. You know, the name of the game is how do we build a better world in a bigger role for AI? Because ultimately, then all of our members benefit and everyone across the world, right? So I think those are the things that really excite me. And, you know, five years from now, you know, maybe there are other pillars to our organization that kind of come up. And one of the things that we like having is we have a sort of a research

• 48:02Speaker 00

  group that allows for sort of more rapid development of these ideas. So that, you know, things like the data sets, we can sort of quickly iterate on until we figure out that, hey, okay, yeah, this is a good idea. This is mature now. Let's go push some real weight behind it. Or, you know, that idea wasn't quite ready. And, you know, there's a lot of ideas that, you know, are great, but it might not be the right time for them. So I think just, you know, the three things we've charted should really keep us good for the next five years, maybe even longer. But Just the impact and then maybe some other new ideas as they come along. So that's for me, I think.

• 48:35Daniel Whitenack?

  Yeah, I think that's a very inspiring and good way to start out 2021, thinking about some of those ways in which AI can benefit those who it's not benefiting now and as well, all of us who are in the industry can benefit as well. So I really appreciate your perspective on that and excited about the things that ML Commons is doing. And we will definitely link to a bunch of different things in our show notes, including MLCube and the People's Speech Data Set and MLPerf and ML Commons in general. So definitely make sure and check those out, check out what they're doing and get involved if you're able to. But thank you so much, David.

• 49:17Daniel Whitenack?

  I really appreciate you joining us today and recording with us at the very start of 2021.

• 49:23Speaker 00

  Yeah, it's been my pleasure. No, I thank you for taking the time to talk with me and, you know, it's great, especially to connect to folks who, you know, are genuinely interested both on an intellectual level and on a sort of day-to-day level on, you know, the tasks and the mission that we're engaged in. I mean, yeah, there's a lot of exciting things going on in AI and ML. So maybe we can have a separate section where I get to ask you the questions on your thoughts on that.

• 49:47Daniel Whitenack?

  That sounds great. We'll definitely have to have a follow-up. I mean, with ML Commons being so embedded across the whole industry, I'm sure we'll be crossing paths and hope we can have another conversation.

• 49:58Speaker 00

  Absolutely.

• 49:59Daniel Whitenack?

  Thank you. Happy New Year.

• 50:01Speaker 00

  Happy New Year to you as well and everyone listening.

• 50:07Speaker 04

  Thank you for listening to Practical AI. If this is your first time, make sure you subscribe so you don't miss a thing. Head to practicalai.fm to subscribe or find us in Apple Podcasts, Spotify, or wherever you listen to podcasts. And if you get value from the show, please do share it with a friend or a colleague. We appreciate you spreading the word. Practical AI is hosted by Daniel Whitenack and Chris Benson. It's produced by Jared Santo and our music is provided by Breakmaster Cylinder. We are brought to you by some awesome sponsors. Shout out to Fastly, Linode, and Launch Darkly. That is our show. On the next episode, Daniel and Chris are joined by two fascinating folks from Aribell

• 50:49Speaker 04

  to talk about, get this, their AI-enabled electronic nose. Stay tuned for that one coming at you next week.