AutoML and AI at Google

We’re talking with Sherol Chen, a machine learning developer, about AI at Google and AutoML methods. Sherol explains how the various AI groups within Google work together and how AutoML fits into that puzzle. She also explains how to get started with AutoML step-by-step (this is “practical” AI after all). Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean now offers three managed databases https://www.digitalocean.com/products/managed-databases/ — PostgreSQL, MySQL, and Redis. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. Changelog Weekly https://changelog.com/weekly – We track, log, and contextualize what’s happening in software throughout the week and deliver it directly to your inbox on Sunday mornings. Head to changelog.com/weekly https://changelog.com/weekly to browse the archives, subscribe, and push the easy button on your continuing education. Brain Science https://changelog.com/brainscience – For the curious! Brain Science is our new podcast exploring the inner-workings of the human brain to understand behavior change, habit formation, mental health, and being human. It’s Brain Science applied — not just how does the brain work, but how do we apply what we know about the brain to transform our lives. Featuring: Sherol Chen – Website https://medium.com/eliza-effect, GitHub https://github.com/ffpaladin, LinkedIn https://www.linkedin.com/in/ffpaladin, X https://x.com/ffpaladinChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Google cloud https://cloud.google.com/Google AI https://ai.google/Tensorflow https://www.tensorflow.org/Quoc V. Le’s research https://ai.google/research/people/QuocLe/Cloud AutoML https://cloud.google.com/automl/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com slash change log.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front end web servers that need to be full duty CPU all day, every day. Then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co slash change log. Again, do.co slash change log.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slay with us around various topics of the show at change.com slash community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:46Daniel Whitenack?

  Welcome to Practical AI. I'm Daniel Whitenack, a data scientist with S I L International. And I'm here with my co-host, Chris Benson, who is a principal AI strategist at Lockheed Martin. How are you doing, Chris?

• 2:01Speaker 00

  Doing great. How's it going, Daniel?

• 2:03Daniel Whitenack?

  Uh, it's going really good. I, uh, I spent some vacation time doing a little bit of backpacking in, uh, in Minnesota. So, uh, just about the opposite from my day to day life, sitting in front of my computer screen. So it was, it was a really nice, uh, disconnected time. So, uh, happy to, happy to have that time. Um, do you, you have any, any good times over Labor Day?

• 2:26Chris Benson?

  Yeah, I had a good time with the family over Labor Day. We went to a local lake in the Atlanta area and did a little kind of a beach thing, which was nice. And I too am looking forward to a couple of weeks of vacation coming up. I'll be in the UK. My wife is British, so we spend a lot of time over there. And other than that, lots and lots of business travel, conferences, things like that.

• 2:46Daniel Whitenack?

  Yeah, definitely. You're keeping busy for sure. Well, today, we have Sheryl Chen on the show. She is an ML developer at Google. I met her recently over the summer. She's doing some really awesome things and has done some really quite a bit of different things while she's been at Google. But I'm really happy to have her on the show today to talk about some of the things that she's doing with with Google Cloud and some some other AI things. So, welcome, Sheryl.

• 3:15Speaker 03

  Hey, nice to meet you.

• 3:17Daniel Whitenack?

  Yeah. Well, could you just give us maybe just a little bit of background about, you know, how you got into computers and eventually AI and programming and ended up at Google and doing machine learning and all of these things?

• 3:34Speaker 03

  Yeah, of course. So yeah, so currently I work at Google, like you said, and my primary area of focus is auto machine learning, which has been this up and coming practice in machine learning, where you're basically given the the labeled data. And at the end of it, you get, you know, predictive model without having to really do very much machine learning in the process. And what brought me here, I guess, like that story he starts with my first computer, which I would actually say was the Nintendo, which kind of dates me, I guess.

• 4:13Daniel Whitenack?

  Nice. Nice. Like the original Nintendo.

• 4:15Speaker 03

  The original Nintendo was like the my first, you know, gateway into technology. And I just remember I have a lot of memories of just playing. Uh, you know, hours upon hours of games and being captivated by these virtual worlds, these pixel, very pixel-headed virtual worlds. And, you know, just, just like, not only was it just, you know, wondering how, how it was possible that we could have these, the input-output, all that feedback coming out of my TV, um, but, you know, thinking and dreaming of how I could make it better. And, uh, so naturally, you know, I just, I went to a high school that happened to have, you know, um, computer science, and I remember asking, you know, what is computer science?

• 4:59Speaker 03

  I, I, uh, you know, was in a classroom eventually of like, you know, four or five students that, you know, happened to have a particular interest. It wasn't like a very widely popular subject area. Then, of course, naturally, you go to college, and I think I'd forgotten that I loved video games, um, at some point, or like, that that was kind of my motivator. And so, uh, at some point, I think it was like, I was, and I was late, uh, late at to playing this game, but Final Fantasy VII. Nice. Really, uh, got me back into, uh, thinking about these worlds and the stories that they told. So then I, um, signed up for some summer research, and I did a research project looking at like, artificial intelligence and storytelling,

• 5:42Speaker 03

  which then led me to UC Santa Cruz, where I studied under Michael, Dr. Michael Mathias. Um, he was a leading expert in, um, interactive intelligent dramas, he would call them. So they're like, basically these, um, he created this game called Facade, which is this very lifelike, you know, reactive, um, natural language-driven, uh, interaction or game with, uh, within this story of this couple that you would go visit them and talk to them. And at the time, it was like, that was like ten years ago, it was like state of the art, you know, there hadn't been anything like that before, and... That really, really motivated me to go and pursue studies under him. So I went to do grad school there. And while I was in grad school, I thought I would be like an academic for the

• 6:26Speaker 03

  rest of my life. That I wanted to be a professor. And in some ways, I still do want to be a professor, but I, I was in Silicon Valley and I took an internship with Google because somebody asked me like, why not? And it totally changed the way I saw the world. I thought of impact in a totally different way. And so I started, like you said, at Google, I did a couple of different roles. So I was at, I was at YouTube, working on YouTube search as a software engineer, really looking at search algorithms and the ways that like little changes here and there could impact how people use the product. Then I wanted to be more publicly engaged. So I was doing developer advocacy at Google Cloud, working with the machine learning and AI tools. And that's really kind of where, like, that

• 7:11Speaker 03

  was also around the time that ML boom was happening, one of the big like, oh, my gosh, you know, look at all this, look at all this progress, look at all of the, the abilities that weren't possible before, enabled with machine learning. And that was probably like 2016. Yeah, it was around 2016. And then, then I ended up doing a rotation with Google Brain, which is one of the premier groups working in AI today, working on Project Magento, which does music and machine learning creativity. And, and then as, as I'm like digging deeper into these products, that led me to AutoML, which I explained at the beginning. But yeah, that's basically my journey from video games to working in ML at Google.

• 7:57Daniel Whitenack?

  That's amazing. Yeah, I, I'm interested in like, one of the things you said was like it kind of, you know, going to Google and experiencing that kind of reset your, your ideas about, about impact. Do you mean like, like in academia, you can have an impact because you can research new things and kind of release them into the world and in some sense? Is that the kind of impact you're thinking of? Or how do you view impact these days in terms of like the AI work that Google's doing?

• 8:25Speaker 03

  Yeah, I mean, I think you're right in that, like, what we do in academia is that we We write papers and these papers hopefully are cited by future papers and that people receive them well. And it helps, you know, continue to advance the state of the art. Um, and to me, that was like, is very romanticized in my mind, you know, like when I went to undergrad, I thought, you know, this is where everyone's just like talking about ideas and, and um, challenging one another to do better. And, you know, I found that in grad school more so. But then it was kind of all I knew, right? It's like when you're looking up to your professors, you're like, oh, I'm gonna become that someday.

• 9:07Speaker 03

  You think, in addition to all that, think about how you're gonna be a teacher, how you're gonna mentor people and how you're gonna grow a group that way. And so, you know, I'm almost embarrassed to tell this story, but I honestly didn't know, like, I didn't go into computer science thinking, I'm gonna be a software engineer, I'm gonna, you know, like, I wasn't influenced by the dot com stuff. I really had no idea. I wanted to make video games. Like, I would call up Nintendo's hotline, like, I think it's like one eight hundred, like, like, I, like, I had it, like, memorized, you know, like, two five five three seven oh oh. I don't even know if that's right. But it was something like that. And, and I would ask them, like, how do I make video games? That was how, like, detached I was to how kind of the non-academic world functioned.

• 9:51Speaker 03

  So when I went to Google, I was, you know, really excited because there's a whole new world that I didn't understand a whole lot about. One of the questions they asked me was, you know, this is really embarrassing, how many users did I think YouTube had? And I was like, okay, you know, like, what's a big number? For me, you know, in research, like, having a hundred users was like, that was like a lot of users, like, too many users. I remember the statistician telling me, like, a hundred users, that's like too much data, like, like, you, you collected too much.

• 10:25Daniel Whitenack?

  Yeah, I don't, I don't think what I coded in, in grad school ever had a hundred users, for sure.

• 10:31Speaker 03

  Yeah, yeah, yeah. So, so the number I gave, and sometimes I'll even, like, multiply my guess by a factor of ten because it's so embarrassing. I said, I was like, oh, like, ten thousand, like, I didn't even think about it. I said ten thousand, and the manager, like, laughed and then said, never mind. And then moved on.

• 10:50Daniel Whitenack?

  There's at least 10,000.

• 10:52Chris Benson?

  That's a good story.

• 10:53Speaker 03

  Yeah, no, this was and this was like just that was like how new I was to this, how little I knew about like the the product side of things, like the real world applications. Yeah, and sometimes when I tell the story, I'll say 100,000 because it's a little bit like, okay, like, you know, I could see that. But yeah, no, I literally said 10,000. So.

• 11:10Daniel Whitenack?

  Yeah. So like the scale of the impact of YouTube and like anything you would integrate in YouTube, including like whatever it is like, you know, whether it's AI related or not, is going to have an impact on a scale that's just totally outside or just totally different than what it is in academia, basically.

• 11:29Speaker 03

  Absolutely. Yeah.

• 11:31Chris Benson?

  So I gotta before I ask a question, I'm actually going to throw back all the way to the beginning when you were talking about your childhood in high school. And I think I might be slightly older than you because my first my first computer game was the original Atari system.

• 11:46Speaker 03

  Yeah.

• 11:46Chris Benson?

  And then when I learned how to when in high school, they were still teaching typing on typewriters.

• 11:53Speaker 03

  Hmm.

• 11:53Daniel Whitenack?

  Nice.

• 11:54Chris Benson?

  That was just coming to an end. I know, I know. So and we didn't have a computer in the school at the time. So I'm slightly older than you, but just barely.

• 12:02Daniel Whitenack?

  Yeah, I work with a guy. I just think this is cool. I work with a guy who met his wife messaging back and forth on the I think it was the University of Michigan like mainframe. So it was like there was only like there was no internet, right? So they like let students access like the University of Michigan mainframe. And they were like two of the students messaging on there. And they eventually, you know, talked and got married. I thought that was pretty great.

• 12:32Chris Benson?

  That might be the only mainframe romance story I've ever heard about in my life. So, okay, so I want to actually get a little clarification because, you know, we are all always hearing about everything AI from Google and there's all these different things. Google Brain, Magenta, TensorFlow, Google AI, Google Cloud, you name it. And so like, how how do these things? How do these different parts of Google that work in AI areas relate to each other? And, you know, are some of those the same as Google Brain and Google AI the same or not? And and how does TensorFlow, you know, as a tool fit into that ecosystem? And, you know, can you kind of just give us a sense of where all these pieces of AI fit in at Google and how they relate?

• 13:15Speaker 03

  Yeah, I would say that that's a really good question and I definitely have a pretty good insight, having worked with a lot of these teams. So like any, any big company, there's always like, there's like the research side and there's the product side, right? And so when you're, if you're, if you're looking to work in technology and you want to be a research scientist, then you know, you're looking at like being the leading expert in your area. So you're good at doing image recognition and you're like, you have all these publications. If you're looking to be like, on the product side, oftentimes they're looking for people who are like straight up builders, so engineers, they're looking at people who have business sense, they're looking at people who understand users, things like that. So they're, they're like, in some ways, their objectives are very different.

• 14:00Speaker 03

  One is looking at being published at NERIPs or ICLAR, these top machine learning conferences. And some are looking, and the other, on the other side of things, you're looking at people who um want to, you know, create a tool that has a lot of users. So at Google, you know, it's a big company, you know, like close to a hundred thousand employees. It's in the ballpark, I think, and it's all over the world, right? And so a lot of these projects are really like, it's really kind of an opportunity for people to see, you know, what really sticks, what's actually going to show traction. So you see a lot of like, organic emergence of different applications of technology, whether it's VR, AR, whether it's like, you know, home systems, like, like the Google

• 14:44Speaker 03

  Assistant, whether it is something like AI and ML. Uh, so like, what is the landscape? So like, I, I've heard people say like, you don't want the outside facing representation of your, um, of what you do to look like your organizational chart. So it doesn't, you don't want the team structure to be directly reflecting of like what it is that you're providing for people in the, you know, the public to, to use and to see and to, to experience. I would say that the way that they're shaping that, and a lot of this is like, I think, you know, just being good storytellers or being good marketers to kind of put the tools in places where people who need them will be able to get them the easiest. Um, so we have like, you know, Google Cloud, right?

• 15:28Speaker 03

  That's where they're moving towards providing business solutions, they're providing, um,

• 15:34Speaker 03

  Enterprise technologies, things like that. So things that will help your business. Then there's what we call Google AI. Um, a lot of Google AI, uh, like inside within Google, it's called research and machine intelligence. Um, it's basically a lot of the advan- like the less, um, like looking at how many users we have, but more looking at like, you know, is this, i- is the work that's coming out of here relevant to the state of the art? Or like advancing the state of the art? And so those are kind of two of the bigger, bigger organizations that we could, that dissect even further. But then outside of that, there's like one more group, which is, I would say like more of a collection of groups where you have like Google Assistant, um, you know, Google Photos.

• 16:20Speaker 03

  A lot of these Google products use machine learning and AI to create better features, to create, um, a better user experiences and things like that. So, uh, like, so Google Cloud, I think is pretty straightforward. If you go to the cloud website, you can kind of look at all the different offerings and things like that. I think Google Cloud AI and like Google Research are prob- is probably where it's kind of like, okay, what, where does TensorFlow fit and where's the AI, um, side of things? So I can kind of, I could talk about it from, um, I would say it was like, it was like, it was Google I O twenty seventeen. And that was when we moved to being an AI-first company.

• 16:56Chris Benson?

  Yep, I remember.

• 16:57Speaker 03

  Like Sundar was like AI-first. Um, I had just joined Google Brain at that time. And we were in a building that was like, you know, there's the, in, you know, Google's based in Mountain View. We have the Google Plex, like it was in like, you know, the main area, but not in the, the Google Plex, which is like a set of four, let's say forty, forty-one, forty-two, forty-three, four buildings.

• 17:21Speaker 03

  And so Google, like a lot of the, you know, Google Brain team, it was actually like across the street, right? So I joined, Sundar announces, um, we're like AI-first. And all of a sudden they're getting moved to the center of campus. And so I remember my desk was like, maybe like thirty feet away from like Sundar's desk. Um, because of that, that move and like, Of course, I never really saw Sundar and there was like a wall that separated us and you couldn't badge into that side of the building. So I like, I think we were, I'm just like looking at like the, the blueprints of the building. I was kind of like, oh, like, I'm like, pretty close to, we're probably breathing the same air or something.

• 18:00Speaker 03

  But I didn't get to see. Anyways, uh, that's just kind of give you a sense of like topologically, like how, how the feel of it is, right? Um, so TensorFlow, right? TensorFlow is like, basically is, is a, it dwells, it lives, it's like being developed and, um, and improved within the Google Brain team. Google Brain team is, was at the time run by this superstar engineer, Jeff Dean, that, you know, he's a celebrity inside and outside of Google. I believe like, you know, Andrew Ng and some other people also were responsible for starting this like, you know, Google Brain effort. And when we became AI first, it kind of got moved into, uh, you know, a more central location, uh, physically, but also, um, as far as focus goes.

• 18:45Speaker 03

  So then, Brain isn't the only thing in cloud AI though, right? Cause we have, uh, we have groups like machine perception and, um, Descartes, which, you know, have you, do you guys know Ray Kurzweil?

• 18:55Chris Benson?

  Yes, absolutely. Yeah, legend.

• 18:58Speaker 03

  Okay, absolutely. Yeah. Okay. You guys would, so like Ray Kurzweil has a team, right? And that team isn't a part of Brain, but he's also doing incredible work. So one of my favorite works that came out of Ray Kurzweil's group, uh, is the Talk to Books. And so, you know, Ray Kurzweil has these like, you know, amazing, fantastical, like, like, uh, aspirations for technology. And so his thought is like, you know, what if you, you needed to know something in a book? Like, what if you could just ask the book and say, hey, book, you know, tell me, you know, what's the meaning of life? And the book can just kind of spit out an answer. And so like, that was the kind of, like, like every research lead would like kind of bring in their own, uh, their own flavor.

• 19:37Chris Benson?

  You know, ironically, that's kind of how the rest of us think about just Google search in a lot of ways, you know, as saying, we're just going to go and find out what the answer is. So.

• 19:44Speaker 03

  Yeah.

• 19:44Chris Benson?

  It's just funny to hear it coming out of your mouth there.

• 19:46Speaker 03

  No, totally. And it's like, and so like in a team like that, like they're thinking, you know, what is the next evolution of search? Right now it's, it's just. It's been the kind of like this like keyword like pattern matching where you look for the exact pattern of characters that you've typed in. But what if there was more of like a semantic reasoning behind, you know, search and results like that that arise from those queries? So like when you talk to your Google Assistant, you have the ability to kind of, it's kind of trying to reason more about your intent behind what you're trying to look for. And so they've created this idea of like, you know, rather than keyword search, what about like semantic search? And they created this game, it's called Semantris, where you have like a list of

• 20:30Speaker 03

  of objects and then you're trying to like semantically match the meanings and you can knock out different words if you can get like a direct match or there's like a, you know, one degree separation between meanings. Oh, cool. So anyways, anecdotally, that's like something that I found really, really delightful.

• 20:48Speaker 03

  But, okay, so then Google Brain,

• 20:51Speaker 03

  run by Jeff Dean and Jeff Dean now, like, you know, JG was originally the the man in charge for research, but he has left and Jeff Dean now is running RMI and doing a lot of the research leadership. Google Brain, though, at the time and now focuses really on like, you know, how how machine learning can make the world better. And so there's a huge like medical side of it, right? So there are a lot of people looking at, you know, medical advancements or medical use cases for machine learning. There's, you know, Project Magenta, which is under kind of the Generative Models Group, where they're looking at how we can generate not only just music and art, but text.

• 21:32Speaker 03

  And one of the cool use cases of that is generating Wikipedia pages because you do have a large, well-curated data set from Wikipedia. And so how can you use that to generate text in intuitive ways? And yeah, robotics is another group. So then there's all these different topics, use cases. And I think the reason why Cloud AI came about was that, as you can tell, it's kind of a lot of, there's a lot going on under the hood and Cloud AI is kind of this landing spot, this portal where you can get the Cloud AI stuff, where you can get. Into what Magenta is doing, where you can see what kind of research is coming out of the group in a more user facing side of things, rather than if you're like Jeff Dean,

• 22:16Speaker 03

  overseeing the research arms and limbs.

• 22:33Speaker 04

  Hello there. This is Jared Santo, Managing Editor here at ChangeLog. The fact that you're listening to this means you are actively investing in your future in this industry. Things move fast and keeping up is hard work. Help us help you stay relevant by subscribing to ChangeLog Weekly. We track, log, and contextualize what's happening in software throughout the week and deliver it directly to your inbox on Sunday mornings. Head to change log.com/weekly to browse the archives, subscribe, and push the easy button on your continuing education. That's all from me. Once again, that's change log.com/weekly.

• 23:23Daniel Whitenack?

  So you've mentioned AutoML a couple of times, or Auto Machine Learning, and I think it would be great if we jump into a little bit of detail on that because it's definitely come up a few times on the show. But as we were discussing prior to starting the recording, we haven't really talked about it in any sort of detail. So I'm wondering if you could just describe AutoML generally, how it's different than how people are used to doing ML, maybe because some people might think ML is automated in some ways already in the sense that, oh, I can get a CSV right, and then I can go use random forest from scikit-learn and choose

• 24:08Daniel Whitenack?

  the column that's the labels, right? And it seems like it just happens for me, right? So how is AutoML different than that? And maybe also, why is it needed in certain scenarios?

• 24:24Speaker 03

  Yeah, so I love practical AI, right? Let's be practical. Let's think about-

• 24:30Speaker 03

  How um.

• 24:32Daniel Whitenack?

  That would be great.

• 24:33Speaker 03

  Yeah, yeah.

• 24:34Daniel Whitenack?

  Perfect.

• 24:35Speaker 03

  Yeah, I wanna be, let's now let's take this down the practical route. Um, so we have, uh, we have machine learning, we have AI. Uh, how does that impact the world? Like, do you like, can, like, how, how does it affect the everyday person, right? So a lot of people in the US, they use smartphones. And so now they're able to, um, now they're able to, you know, if they're traveling abroad, they could take a picture of a menu and get that translated on the spot. Or they don't even have to take a picture, like, just within the computer screen. They can translate from like Spanish to English. And a lot of this, this image recognition or language understanding comes from advanced, it's machine learning. So on the user side, there's already a lot of people using it.

• 25:18Speaker 03

  And what they found is that a lot of these tools are useful. Like, if you want to be able to use voice commands, um, you don't need to necessarily retrain. Um, you know, like, I don't even know how much data they use. But like, all the data that they require to kind of train that, that voice model, that voice speech to text model. They don't need to necessarily have to go through, uh, like every, every company that needs that doesn't need to go through and do that again and again, right? So, so Google, having, uh, worked on this technology and using it in its own products, ma- makes this available for people. So you can come in, um, you have an app that you wanna build, you wanna use like, you know,

• 25:58Speaker 03

  you wanna it to be voice activated, or you wanna be able to take images and figure out if they are, um, kid-friendly images. Like, there are these models that Google has pre-trained and that they work, and, and they, and Google is ensuring that these models work well. So, without any data, with just the desire to have this feature, you can already plug and play, um, certain products that Google have in machine learning, right? Anything with, a lot of stuff it requires, uh, anything that requires vision or audio processing or natural language. I think these are very common use cases. Um, and then on the other end, end of things, right? Like you were mentioning, you could, you could set up your own Python environment. You could be, you could be coding in TensorFlow. You could train your own model.

• 26:43Speaker 03

  You can, you could feature engineer on your own. You can hyperparameter on your own. You can look at the models and the precision and recall and assess how good these models are doing. And then you can do it all over again. And as data, as you know, maybe you have something running in production. You can kind of have all of that, you know, like all the control over every aspect of that. So that would be the other end. And of course, you know, enterprise cloud technologies, they provide the environments. You have Docker containers. You have virtual machines that can also satisfy that need. Um, what AutoML does is it focuses on the middle piece, or at least trying to figure out how you go from one end to the other. So on one end, you have pre-trained APIs where you have like basically no control over how that model is making decisions.

• 27:28Speaker 03

  And then you have like build it from scratch, like do it all yourself. For a lot of businesses, um, it's kind of hard to go sometimes from research directly to, you know, production. And so trying to find that middle point, you know, is what the, well, people working in auto machine learning are trying to figure out like, like things like hyperparameter tuning, right? Are we tuning these weights around to kind of get a feel? And people before have said, you know, this is a highly intuitive thing where you kind of get a sense of, you know, whether this number should be bigger or lower. Auto machine learning, one of the aspects is like trying to figure out how to just simply do that. With the tools that we have now, like it could just be a matter of thinking, you know, why, why?

• 28:10Speaker 03

  You know, you might be able to figure out, uh, you know, safe search or, uh, kid friendly images. It's something that, that, uh, pretty universal for the most part, but maybe not for every culture, right? You might want to label your own data as like, you know, this image isn't safe. You might, you may, maybe like it's not enough to know which pictures have cats in them. Like you can go to your Google Photos right now and do that, right? You can go into your Google Photos, type in cat pictures, get all the cat pictures. But what if you want to know what type of cat? If you want to know like, you know, different colors of cats. The specificity that these pre-trained more general use cases may not provide.

• 28:51Speaker 03

  And so AutoML in the in the most like general use case is basically giving you the opportunity to label your own data. And then feed that in. So you're getting customization without having to build everything up from scratch.

• 29:05Daniel Whitenack?

  Yeah. So am I correct in saying that kind of AutoML encompasses what could be a lot of things? So there's also like all this other sort of jargon that's that we're running across, right? Like meta learning, like learning to learn, and then there's like the transfer learning. Side of things which is taking like, you know, pre-trained models and either fine tuning on on additional small amount of data. Or maybe adding adding layers to a network or something like that. So is AutoML kind of an encompassing term that encompasses a lot of those things? Or is it or is it more specific or separate from those things?

• 29:45Speaker 03

  Yeah, as we as we know in like AIM ML jargon is like, like often it's very controversial sometimes. No, I would I would say so, like the people who are like, you know, learning to learn, like, these are all groups that. That for which AutoML is the product, like they draw upon the findings. And and the the answer isn't like, how do we make machine learning better, which is a research question? What it's really doing is it's trying to figure out how do we deliver the best prediction model? And so maybe that prediction model is like heavily rules based. And they basically figured out in like two if statements, you can like figure out like the best prediction for a certain problem, right?

• 30:24Speaker 03

  I think I think AutoML, like, the goal really is to make sure that you're providing the most accurate results. Given the data and the kind of outcomes that you're looking for.

• 30:34Daniel Whitenack?

  Yeah, so it's it's kind of a general term that I guess could encompass many different things. So it could, like you were mentioning hyperparameters, right? Like, there could be this sort of automated way of determining hyperparameters. I know there's like maybe more complicated things like figuring out like the right types of, you know, nodes or architecture of your neural network and that sort of thing. So I guess all of that would fit into AutoML. Would that be appropriate to say, I guess?

• 31:06Speaker 03

  Absolutely. Like, one of my, there's this researcher at Google Brain that works like, who I'm friends with. And so we joke a lot. And he's always, like, we joke that he like uses up like so much of the the computing for the whole company. Like, you know, he like, right, there's like a leaderboard for like, who uses the most computing for throughout all of Google. And one thing that I always joke about, I'm like, oh, yeah, like, how are those hyperparameters doing? Because it's kind of a, like, we're kind of like, minimalizing the kind of work that he's doing by joking about that. But I think it's, you know, hyperparameters tuning is just such a like, a fundamental, like, something that people who get their introduction to ML like, know about. And they know that

• 31:47Speaker 03

  this is this thing that's been highly, you know, human intuitive, intuited practice. And so it's kind of a this go to, you know, how would you automate machine learning type deal. But yeah, but no, whenever I joke about it, he's just like, you know, it's not, I didn't, I don't mean to, I only do it in jest to minimize it a little bit. But no, it's, it's a very like, common example to use. But yeah, like you said, it's like way more than that. It's the kind, what kind of algorithm are you being, what kind of model are you training? What kind of approach are you, are you, like, like even within like RNNs, there's already so many different ways of like, applying and using and deploying RNNs, things like that.

• 32:25Daniel Whitenack?

  And that that almost takes right into what I was about to, I was kind of had at the top of my mind, anyway. And that is, how is AutoML being used in production? And and what makes it important for the future of AI and ML? So, you know, if people are using it in some way out in the real world, where does that get a lead for them? Where is that going to take them?

• 32:44Speaker 03

  No, that's a good question. I was actually talking to one of the gaming partners. So like a gaming company. And I was, it's interesting, because on one hand, you think, you know, AutoML's Trying to take a lot of that in between the statistical work out of it, right? So I was talking to data science teams and I ultimately feel like AutoML's like purpose should really be able to to be able to extend the the data scientist's abilities and intuitions. It should be give them a sense of like which features are, you know, the most relevant, things like that. And sometimes I feel like a lot of this, there's like ML engineers, there's data scientists, they they work together and sometimes they're working separate.

• 33:29Speaker 03

  But I do think that like ultimately it's the data scientist that's kind of serving that AutoML role, especially for specifically structured data. So we're talking about tabular, you know, spreadsheet-like data. I feel like a lot of that work is the the human intuition has been coming from data scientists at large. And the question is, does AutoML meet the standard of the data scientist? Does it extend the capability of the data scientist? Does it give the data scientist who has been this expert like stewarding this area, does it give the the data scientist the ability to kind of do more or to get a better sense of like, you know, what direction should we go in in making these decisions?

• 34:10Speaker 03

  So, yeah, so I think that when we talk about these problems, we think like, you know, if you're a gaming company and you're looking at use cases for whether a player is going to continue, like whether a player is kind of, you know, plateauing and and really like the game is just not engaging anymore and what point that that is. And you're trying to make a prediction that after these actions, this this next thing's happening. How do you correct course so that you make something that is more satisfying to the user? Things like that, where like I think a data scientist would sit and kind of go through like here are the signals and here are the signs. I think AutoML would be like an additional like tool to be able to help arrive at those

• 34:54Speaker 03

  conclusions in a more efficient way.

• 34:57Daniel Whitenack?

  So there is like, I guess, some level of expertise that's needed to to use AutoML. Like, I was looking at one of the, I'm not sure where I was seeing this. I think this is one of the examples when I was looking at AutoML on the Google Cloud site. But, you know, my life right now, at least, is all about AI and natural language. And I think I was seeing, you know, one of the examples or use cases they were talking about was like, oh, you have this machine translation model, maybe that's that's pre-trained for some language pair, like English to German or whatever. But really, you need like some very specific types of vocabulary translated.

• 35:37Daniel Whitenack?

  Like, maybe it's like legal vocabulary, or maybe it's like some domain that's not covered well by the the data that the model was trained on. So you need to kind of get that model, you know, fine tuned a bit to that, to that, you know, vocabulary that's that's not sort of general vocabulary. So I'm assuming that still like, you know, to structure that data, to get it in like the language pairs that you need it in, to kind of understand and evaluate if your model's like doing well, even if you're not like setting all of the hyperparameters, or you're not like, you know, you know, coding all of the neural network architecture.

• 36:20Daniel Whitenack?

  There's still like a level of expertise that's required to set up that that experiment, right? And be able to know if it's it's working, is that right?

• 36:28Speaker 03

  Um, so I would say that it really, we were talking about tabular data earlier, right? I can kind of walk through a little bit of like how that would look. But I would say, to answer your question, that you really need to, I think the the expertise really comes in once the model is trained. Because from that, you kind of, you need to have kind of a sense of whether the model is doing well enough for you. So every, for every, you know, in production, every, every feature, every use case is going to be different. And sometimes, you know, you know, if you're 99% accurate, that's good. So for some of them, that's not good enough, right? But really, uh, I, I would say the expertise comes in with, um, with also once, if say

• 37:13Speaker 03

  the model needs to be better, um, or say some change needs to be made, to be able to understand like how to engineer or, or shape the data in a way that's gonna give you a better outcome. So there's, there's like, there's basically, there's two different ends of it, right? You could just, uh, trial and error it, you know, give it data, uh, run the prediction, uh, or train it and then run the predictions and then see how well it does. And if it's good enough from your sense, uh, I, I would say that, that's kind of the bare minimum. Uh, but I think, you know, to hold this kind of higher standards and to kind of, uh, oversee like, you know, what I said earlier, the state of the art, um, is doing, I, I do

• 37:52Speaker 03

  think it's important to have somebody with like data science or statistical understanding or machine learning understanding to be able to see whether, like to kind of just not let it run, run in the wild. And this is something that, um, that people talk about, you know, auto mail, this is like true democratization, because you're really just giving, you, all you have to do is just have the labeled data, show it lots of examples, and then it's gonna make the prediction. Um, I've talked to like, you know, the product manager, the product side of TensorFlow, you know, going back to the research side, and what they would sometimes say is, you know, is that truly democratization? Because at the end of the day, you're, you're relying on Google Cloud Auto Mail tables to make this decision, and you're controlling these two ends.

• 38:35Speaker 03

  You get to decide what data goes in, and you get to decide whether this prediction model goes into production. But like, you know, tense, like arguably TensorFlow, this open source tool for machine learning. Uh, you actually have is like ultimately giving you the like all the control and all the power. It's just the learning curve is much higher. So there there's the cost benefit of that. But uh, but yeah, I I would say that I could teach, you know, a thirteen year old how to use with no machine learning, understanding, somebody who's like, you know, understands how to point and click and load in files, um, how to train their own model and give them kind of an intuition of what that evaluation, how that should evaluate and how to test that out.

• 39:20Chris Benson?

  So one of the things I'm I'm wondering, um, I'm pretty interested. I know in in a few minutes, I'm going to ask you to kind of take us through kind of an example, if you would. But before we get to there, what are some of the disadvantages maybe for AutoML? You know, things like would it would interpret interpretability be one of those? And are there are there certain kinds of domains for which it might not be the most suitable thing? And, you know, and I guess since I'm talking about, you know, what what is it, you know, potentially not best suited for? What are some of the biggest research areas, the biggest open questions that you guys are are taking AutoML into? You know, so what's it? What what's it challenged on? And where do you where do you see it going in the future in terms of applicability?

• 40:01Speaker 03

  Yeah, yeah, I guess in the in the spirit of AI, it's like throw out throw out more buzzwords like humans in the loop and mixed initiative, co-creative systems.

• 40:11Daniel Whitenack?

  Differential privacy.

• 40:12Speaker 03

  Yeah. Um, so I like the I think the question I get asked most when I'm, you know, showing, working with like data scientists or showing teams, customers, partners, people who want to, you know, people who are already or want to use machine learning. When I show them these tools, you get kind of a range of questions. If somebody is like more on the data science, machine learning side, they'll say things like, like, what if I want to be able to get in there and like make some changes? I mean, it gives you like this tool and it and honestly, this tool change, it it launched AutoML Tables for tabular data, launched in April of this year. So it's changed a lot already since then. It's

• 40:56Speaker 03

  not been that long. So it's constantly changing. But um but well, right now, what is what is available to you is you can um Once the model is trained, you can call a rest API, rest API. You can um export the model as a tensor field package and download it onto your own machine and run it from there. There's things like that that you can do, right? But what we get asked often is like, well, can you give us more information? We want more analytics, we want more uh we want to see like really, you know, how well the features are doing in more detail. And that is like totally under the hood. You don't have access to that. You can't um, you know, say if you wanted to, you couldn't really tune the hyper parameters in this case. It's really just giving you uh the endpoints

• 41:39Speaker 03

  of machine learning from training to to um inference. Um yeah, so so I I would say that that's probably one of the drawbacks is that you're ultimately not getting the amount of control that you would if you were building your own model from scratch. And that question of control, or at least intuition, and you said interpretability and understanding why it's making these decisions. It's even more abstract. It's like like the it's almost like by design is trying to like reduce complexity by not showing you these things. But again, with like, you know, this concept of humans in the loop or understanding what kind of controls do we want? As we like, you know, the whole point of machine learning and it is almost like seen in research is the more humans are involved, the more they're gonna mess it up.

• 42:22Speaker 03

  But like like as we as we reintroduce like like working in like the team magenta, right? It was focused on project magenta was focused on creating generative tools, like extensions of humans. And so this concept of where does the human, where what kind of control the human wants is very important. Um, the problem is is that machine learning can get very complicated very quickly. So uh so either people are just overwhelmed with the, you know, with the burden of choice that they have and not really understanding how it all works together. Or it's like, we're kind of taking that away so that they can still make decisions without kind of taking for granted all the stuff that happens in between. That makes sense.

• 43:14Speaker 00

  Hey, guess what? Brain Science is officially launched. Episode number one is on the feed right now, so head to changelog.com/brain-science to listen, to subscribe, and to join us on this journey of exploring the human mind. Once again, changelog.com/brain-science, or search for Brain Science in your favorite podcast app.

• 43:51Daniel Whitenack?

  Okay, so let's say, so I've never used AutoML, other than just kind of browsing around the website and looking at some examples and that sort of thing in very general detail. So I was wondering if you could kind of just walk us through, like if I'm going to go implement my first AutoML application or workflow, what are kind of the steps I go through, at least with, you know, Google's AutoML tools?

• 44:24Speaker 03

  Yeah, yeah, of course. And yeah, you know, Google's not the only company that's offering these capabilities. But yeah, it definitely starts with what you've been saying earlier about how, you know, you have certain language, like say you're a German doctor and you need something, you need a tool that recognizes that context, or you're, you know, searching for a cat and food pictures is general, but maybe you're an automotive manufacturer and you need to, you have all this labeled data of car parts that isn't going to show up in the more, you know, general use cases of the pre-trained models. So what you need to do is you need to, you have this data set and you know this data set is good and

• 45:05Speaker 03

  accurate. What you would do, in this case, let's use like, I've talked a lot about tabular data, right? So we're using like spreadsheet data. Spreadsheet is some Thing that, you know, almost everybody uses today. But in this case, what feature we're looking to establish within like a spreadsheet is the ability to say, you know, select a column in a spreadsheet and say, given all these other columns, can you predict this, like what the value of this column is gonna be? And so, let's say it's like a, it's, it could be categorical, it could be numerical. Like, the example that I've been using a lot is, um, there's, you know, one of the open data sets that we have is like a banking, a banking data set

• 45:46Speaker 03

  that shows different types of, uh, banking customers and what they, like, you know, things from, you know, how often they deposit to like, you know, what their income is, to what kind of jobs they have. And then trying to infer which banking customer is most likely to have a, um, to turn on direct deposit. So they're gonna have their paychecks directly go to the bank. And so what you would do is you would find, first of all, you know, whether it's, you, we talked about CSV files. We've ta-, uh, of course, this is Google, so you could have your data in BigQuery and you just give it the BigQuery ID. You go to AutoML Tables and you would select, you would load in that, uh, that data set.

• 46:30Speaker 03

  So you would import it and, you know, given all the different, so it'll be labeled, it'll be either like, you know, categories or numbers or text. And from there, you have, uh, the opportunity to kind of look at the different column names. From the column names, you can then, you know, like, nullify or, um, select your features. Uh, in this case, they have things like age, education, balance, housing, loans, um, things like that. And from there, you look at each of the column types and you select one to be the target. In this case, we're gonna select like, direct deposit. So this is like a true false, like, does this person use direct deposit, do they not? Um, then there's o- other analytics you can look at, right?

• 47:15Speaker 03

  So it gives you the opportunity to see kind of the spread or whether there's any, um, features that, or any, um, columns that have missing values or invalid values, things like that, where you can kind of remove, and this is where a data scientist said that, you know, these kinds of tools actually are very useful because they often have data, they have li- like lines and lines and lines and lines of data. For which like some of the data might not even be valid. So being able to detect that automatically would be very useful for them. Um, from that point on, you just, you would just then train the model. And the selection you'd make there is how long you feel like training for. So you could say, I want to train usually like one hour, honestly, like for maybe like, these are approximations and it changes case by case, but like, you know, an example, right?

• 47:59Speaker 03

  For this case, I would say for 4,000 lines of, of input. Um, you could train for an hour and you'll get a decent result. You'll get like a, you know, 95% accuracy for predicting whether or not someone has direct deposit. Once it's trained, you select that threshold. If it finishes earlier, and of course it's going to detect things like overfitting, um, when it's like just kind of reaching diminishing returns. So it will terminate early as well, um, if it needs to. But you get, like at the end of that, you get like an evaluation. So you look at, um, you know, there's the recall, there's precision, there's, uh, looking at the true positive rates and the false positive rates. There's the accuracy. So in this case, this model that I, I'm just

• 48:42Speaker 03

  going to pull up one right now, uh, this model has accuracy of 91%, um, which is, you know, pretty good. Um, again, case by case, it's like, what is your use case? What is your threshold in production? Uh, and it's going to differ. It also show you feature importance, um, how the labels are doing and you can then, uh, basically what you could deploy it right then and there. You can just deploy it within AutoML and then it's, you know, do, you can run online predictions, you can run batch predictions, or you can export the model. So you can, um, like I said earlier, you know, export it, it'll be a TensorFlow package. You can run the model in a Docker container. You could take that Docker container,

• 49:23Speaker 03

  you can download it onto your local machine, or you can just, you know, not do any of that and just, uh, use REST API and be able to make predictions that way. And so that, that kind of gives you kind of the end to end, uh, you know, it's basically, once you load it in and you click train and the training completes, you're basically ready to, uh, you know, you, once you end and you have to deploy it as well, but you're ready for online predictions after those few steps.

• 49:51Chris Benson?

  So just to clarify, to make sure, I know like Daniel and I have come at things from more of a framework perspective and using TensorFlow or PyTorch or one of the other frameworks over time. In this case that you're talking about, you're really not doing that at all, right? Your AutoML is its own process and you're selecting the data from BigQuery or elsewhere, pulling it in and essentially producing a product at the end in the form of a model that you can go use. Is that fair that it is a parallel path versus an integrated path with whatever framework you're using?

• 50:24Speaker 03

  Absolutely. Yeah, it's you're loading the data, you click a button, it says train, and then an hour later, you have this model, which you kind of look at the stats for it, and then you're online after it's deployed. So it's like a few clicks. I mean, getting the data is non-trivial, I would say, and then making sure that the model is actually performing the way you want it to is also non-trivial. But yeah, it is like a few clicks away and then you have what you would normally do if you had like, you know, you had like maybe even like a TensorFlow notebook, right? You would still have to kind of, you still had the ability to like, you know, choose the type of approach you want to use and then, you know, change the values around a little bit. But in this case, you don't

• 51:08Speaker 03

  really have that. You get to select the target of which you want to be the, what you're predicting for. But that's about it.

• 51:15Daniel Whitenack?

  Yeah, I definitely see what you're saying about like the expertise that's needed because like if you're, if you're looking at a screen and you're getting, first of all, like, you know, a package out or a Docker container out, like you need to understand how to work with those things or like REST API. So there's like engineering expertise there, of course, but even if you're presented with the screen and it says like, oh, here's precision and recall and these different things, like, you know, there's, there has to be a level of understanding of those things to be able to understand, you know, if your model is really doing what you want it to do, like you said. It seems like there's like, if I'm thinking about my AI workflow or machine learning workflow, there's of

• 51:59Daniel Whitenack?

  course always the data side Of things, which is still going to be in some cases hard, like you're saying here, because like, you know, the data quality is going to influence things. There's the training process, which, you know, if you do it manually, might involve, like you were saying, hyperparameter tuning, like trying out a bunch of different architectures or models and that sort of thing, to where you get something that works, then you have to, you know, evaluate that and then put it into production. It seems like here, there's, you're like, you're really greasing the wheels around that training process side of things, where you don't have to think as much about the, the architecture, the specific model, the hyperparameters. And then you're also helping a bit in terms of like, you know, of course,

• 52:41Daniel Whitenack?

  like, productionizing that in the sense of making a REST API available and that sort of thing. So you can spend a lot more time thinking about the data and the, the type of model that you want out, rather than those sorts of architectures and, and other things.

• 52:58Speaker 03

  Yeah, that's totally, that's the idea, is to make that more accessible, to give people the opportunity to kind of use it at that level of abstraction. But like, I think, I think in some point, we're trying to look at like, you know, that full, the full sense of like, how much control and what kind of control we want to have in, as we create these models as well.

• 53:20Daniel Whitenack?

  Cool. Yeah. So to kind of wrap up here, I think this has been great to, I mean, I've learned a lot going through this. Me too. And definitely want to like, try out some of these things. So I guess ending, you know, to kind of end up with some useful things. I was wondering if you could just kind of recommend, like, so thinking about someone that's maybe wanting to try out these sorts of AutoML techniques, what the kind of easiest way for them to try out those things are, maybe there's tutorials or like, you know, notebooks, or I don't know what the, the interface is, to try those things. And then also, if people are interested in kind of learning generally about like,

• 54:02Daniel Whitenack?

  maybe the methods behind AutoML and like, working on those, you know, like, learning to learn things and hyperparameter, automated hyperparameter tuning, if there's any kind of specific places that, that you could point them to?

• 54:16Speaker 03

  So yeah, for people who want to know about like AutoML in general, I mean, the Google AI, the portal, they link to one of the groups. The researcher that, one of the researchers that I hear a lot of working in this area is Kwak Lai. He's a Google Brain Research Scientist that has a team of people working in auto machine learning. So, anything under him, any of his publications, I think would give you kind of an understanding of the state of the art um, within that area. Um, for people who wanna try it out, um, so if you go to cloud.google.com/autoML, it'll kind of bring you to, you know, all things auto ML that is available as tools for people to use.

• 55:01Speaker 03

  It gives you kind of a list of the different applications or verticals. Um, there's like, you know, site where you have auto ML vision, auto ML video intelligence. So this is again, you're providing your own label data. Um, there's language, so auto ML natural language, auto ML translation. And then structured data, which is what we talked about today, auto ML tables. There's a few like customer testimonies where you can see how it's being used. And um, and a little bit of a description and then you'll, you'll find, uh, if you go to structured data, there's a, there's a, there's a video narrated by me. I pulled together this video for uh, explaining kind of the basics of it. But if you wanna try it, there's a, again, you would choose from those categories.

• 55:41Speaker 03

  And so if you choose, you know, auto ML tables, um, there'll be the opportunity, it'll kind of pull up the UI in the cloud console. So of course you'll have to have like a Google Cloud platform account. But you do get some, you, you get, you could train for free for up to a certain amount to kind of try it out for yourself. And then actually the cost isn't very much. Um, I'm not on the sales side, so I actually don't know the exact numbers. But every time I've talked to like a solutions engineer and heard them quote, you know, it's like very low amounts of money once it actually, once you actually wanna get it going. But yeah, I do know that there's a free, there's amount that you get to use for free. So if you just wanna check it out, try it out yourself, um, it's pretty accessible.

• 56:22Speaker 03

  And, and even if not that, if you wanna even, you know, try some of the pre-trained APIs, like, you know, like you wanna upload a cat picture and see if it knows your cat is a cat. Um, you can just do that within the web. So if you just wanna like see how well these pre-trained ones work, they have like online demos like within the website as well.

• 56:42Daniel Whitenack?

  Awesome. Well, thanks for pointing those things out. We'll definitely share those in the show links. So make sure and check out the show links. We'll put those links there and you can try out AutoML. And definitely consider joining our community somehow. So you can go to changelog.com/community. And we have a Slack team where you can chat about AutoML and Practical AI and all those things. We also are active on Twitter and LinkedIn. So make sure you engage with us. Tell us what you're using AutoML for, or where you found it useful, or what research you're interested in. But thank you so much,

• 57:24Daniel Whitenack?

  Cheryl, for joining us. Really great conversation. It was really great to hear your story, hear about some of the things you're doing and the things Google is doing in AutoML. So thank you for taking time. Thanks a lot. Yeah, thank you.

• 57:40Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor. Go on iTunes, give us a rating. Go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you've got to do. Share the show with a friend if you enjoyed it. And bandwidth for Changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do hear Changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode cloud servers. Head to linode.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address. Get our weekly email, keeping you up to date with the news

• 58:25Speaker 00

  and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.