Learning to learn deep learning 📖

Chris and Daniel sit down to chat about some exciting new AI developments including wav2vec-u (an unsupervised speech recognition model) and meta-learning (a new book about “How To Learn Deep Learning And Thrive In The Digital World”). Along the way they discuss engineering skills for AI developers and strategies for launching AI initiatives in established companies. Sponsors: O'Reilly Media https://www.oreilly.com/changelog – Learn by doing — Python, data, AI, machine learning, Kubernetes, Docker, and more. Just open your browser and dive in. Learn more and keep your teams’ skills sharp at oreilly.com/changelog https://www.oreilly.com/changelogSnowplow Analytics https://snowplowanalytics.com/ – The behavioral data management platform powering your data journey. Capture and process high-quality behavioral data from all your platforms and products and deliver that data to your cloud destination of choice. Get started and experience Snowplow data for yourself at snowplowanalytics.com https://snowplowanalytics.comThe Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: High-performance speech recognition with no supervision at all https://ai.facebook.com/blog/wav2vec-unsupervised-speech-recognition-without-supervision/deeplearning.ai MLOps course https://www.deeplearning.ai/program/machine-learning-engineering-for-production-mlops/GitHub MLOps resources https://mlops.githubapp.com/Books “Cracking The Data Code” by Mike Bugembe https://www.amazon.com/dp/1781333335“Meta Learning” by Radek Osmulski https://rosmulski.gumroad.com/l/learn_machine_learningUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Chris Benson?

  This is really the core about how people build careers now. If you are thinking about how to get there, you have to build in constant lifelong learning into your process. And if you don't do that, you will fall behind. And as things are moving faster and faster forward, the ability to thrive in a digital world is really crucial to having it. Because anything you're learning right now in more of a formal context, if you're coming out of school, for instance, a lot of that will be obsolete. The basic math will be there, but the algorithms are going to change. How you're achieving problem solutions will change. And so you really have to be able to do that. I think that is a fundamental for being a digitally literate person in the job world these days.

• 0:45Speaker 01

  Big thanks to our partners, Lenovo, Fastly, and LaunchDarkly. We love Lenovo. They keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com. And get your feature flags powered by LaunchDarkly. Get a demo at launchdarkly.com.

• 1:04Speaker 01

  This episode is brought to you by our friends at O'Reilly. Many of you know O'Reilly for their animal textbooks and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos, and all their conference talks. Plus, you can learn by doing with live online training courses and virtual conferences, certification practice exams, and interactive sandboxes and scenarios to practice coding alongside what you're learning. They cover a ton of technology topics, machine learning, AI, programming languages, DevOps, data science, cloud, containers, security, and even soft skills like business management and presentation skills. You name it, it is all in there. If you need to keep your team or yourself up to speed on their tech skills,

• 1:48Speaker 01

  then check out O'Reilly's online learning platform. Learn more and keep your team's skills sharp at o'reilly.com/changelog. Again, o'reilly.com/changelog.

• 2:06Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:32Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by my co-host Chris Benson, who is a strategist with Lockheed Martin. How are you doing, Chris?

• 2:46Chris Benson?

  I am doing very well. How's it going today, Daniel?

• 2:49Daniel Whitenack?

  It's going good. For listeners who are in the US, you'll know that this last week was Memorial Day weekend, which is good. It was a nice, long weekend. It's beautiful weather here, but I'm feeling the amount of catch-up I need to do in this short week. I don't know about yourself.

• 3:02Chris Benson?

  No, they give us a holiday, but then the amount of work for that week never changes. So you just have to cram it all in.

• 3:08Daniel Whitenack?

  It's always the same, right? Yeah, I'm feeling it this week, but you know, lots of good stuff to work on, but I'll sleep well over the weekend, I'm sure, and have a nice Saturday to sleep in, hopefully. So.

• 3:20Chris Benson?

  There you go. Recovery time this weekend.

• 3:23Daniel Whitenack?

  Recovery time this weekend. One thing is, there's a lot of news and updates and AI stuff to catch up on. I know both you and I have probably been feeling that. Lots of things have been happening in the AI world. And so today on our episode, what we're going to do is we're just going to kick it back old school. And as our long-term listeners will know, sometimes Chris and I like to do these episodes where we don't really have a specific guest speaking about a topic, but we kind of talk about a variety of topics that we've seen in the AI ecosystem, we think are interesting, maybe of note, and discuss them here live. So you up for that, Chris?

• 4:04Chris Benson?

  I'm totally up for that. We have fun with these episodes. I enjoy them.

• 4:08Daniel Whitenack?

  Yeah.

• 4:08Chris Benson?

  This is a little free form for us.

• 4:10Daniel Whitenack?

  Exactly. Yeah. And I'm sure it shows to our listeners how ignorant we are in certain cases, but that's okay. Cause as everybody who's exploring this space, everybody has their sort of own little niche sphere of knowledge. And it seems like these other people are like very knowledgeable in all of AI, but they just have their own little niche of knowledge too. So it's good to sort of dip our toes into different areas. At least it is for me, I think.

• 4:34Chris Benson?

  You know, that is something that we talked about a little bit in the last episode for, for anybody that might've been listening was just the fact that there is too much stuff for everyone to know all of it. And so you kind of have to pick and choose where you're gonna dive in. And that might be something worth talking about today because you have to kind of make some choices. And it's interesting how the different choices people make affects how a team might come together and what the team's capabilities are and stuff. So there's a lot of ramifications here.

• 5:04Daniel Whitenack?

  Yeah, there's definitely the concept as well of like when you're putting together a data science team or an AI team or something like that. Actually, it's funny, we're actually bringing this up because we had one of these discussions internally, actually even this week is, hey, what are the advantages of having a sort of organizational, the separate, you know, data science and AI unit versus data science and AI people embedded in various teams throughout the organization? And also within those specific people, of course, there's a slide, sometimes people sort of want to hire in data science people or AI people to just do the sort of modeling and analysis stuff. And like, don't worry about the other things, just focus on that stuff and we'll handle the business

• 5:47Daniel Whitenack?

  logic and like pushing that out into products and all of that stuff. In other cases, people sort of very much like to, you know, there's even this term now, full stack data science where it's like, hey, I'm going to take care of doing this analysis piece, but I'm also going to figure out how to integrate it into a product. So there's just so many different parameters there, I think, in terms of how people put their teams together. And it makes it hard for a new person to figure out where they fit in that, I think.

• 6:14Chris Benson?

  It sure does. I mean, not only that, but it doesn't stand by itself in that, you know, AI, if you're thinking of AI as deep learning, which we tend to in these last few years, and it has to be, you know, you have the algorithms, you have the, the need to deploy it out to be useful to your, your user community. It integrates in with other software, it integrates in with the microservices and the APIs. And so you have a huge area of target surface of things that have to get done. And I have never seen two organizations do it the same way. And the organizations that I've worked with, it's like, everyone has their own way of doing it, which means there's not a normalized or standardized way to achieve that.

• 6:57Daniel Whitenack?

  Yeah, yeah. And of course, this is something, because we have a podcast or we're, you know, interacting with a lot of different groups, something that I'm asked a lot is, hey, where, you know, I'm wanting to get into this field, where do I need to focus? And oftentimes, my response to that is, well, let me ask you a question. Like, are you more interested in this sort of, you know, more engineering slant? Or are you just wanting to do sort of researchy type things and analysis and that sort of thing? And because those two could take a very different path. And probably if you're focusing on one, you're not going to be well-prepared for the other one. It's very rare that there's someone that sort of operates in both of those spaces.

• 7:37Chris Benson?

  Yeah, totally. It's a super hard challenge to figure out for someone new into it because I always ask people what they're already passionate about, or I ask them what kinds of things are they, are they reading and watching videos on that has captured their imagination? And then start building on that. And you can ask five different people and get five very, very different answers, which isn't necessarily a bad thing. You know, if they're each gravitating toward different areas that have to be addressed, then that's the makings for a team right there.

• 8:05Daniel Whitenack?

  Yeah. Maybe one thing I could suggest to promote my own biased opinion. I was thinking about this recently. I'm kind of involved to some respect over at Purdue University. Sometimes some of their students help my organization with some projects, like practicum-type projects, or I help their students in working on projects, or occasionally I teach some lectures. And I was putting together some material for a course. This time around, my thought process was really like, For these students, if I was hiring them in, what would be the sort of set of things I would want them to go through or be able to go through that would really give me confidence in their abilities to perform, you know, on my data science or AI team? And so the thing, and you know, you could disagree with this.

• 8:50Daniel Whitenack?

  Like you said, every team's different, but what I did was I said, okay, I proposed a certain problem. The problem was question answering. So, you know, a piece of text comes in, there's a question. You have a model that actually extracts the answer out of the text. And this is a well-known problem and there's models out there that do this. Because it's a short time that we have the students together, I was basically saying, hey, well, this model exists, so figure out how to use it for inference. And then I would like to see three different sort of manifestations of that model. One that I can have a REST API and, you know, send a request and get a response. One where there's a user interface where I type in my question and I type in my passage and get the answer.

• 9:33Daniel Whitenack?

  And one where I can use it in a batch way. So if I want to answer, I have a data set of 20,000, you know, 50,000, 100,000 questions and passages and I want to answer them all at once. What's a way that I can, you know, do that in a batch sense? And I think thinking through those different sort of manifestations of how a model could operate was really, I think it was actually helpful for me in terms of thinking through, you know, how to structure the material.

• 9:59Chris Benson?

  Yeah.

• 9:59Daniel Whitenack?

  But also it, you know, I thought it was a good way to think about things and maybe something I'd share on the podcast because as people are getting into things, they want to sort of show a project or something. And I would encourage people, you know, maybe it's worth, you create a model, but then you create these sort of different manifestations of the model and how it actually interacts with users or the end use case, something like that.

• 10:20Chris Benson?

  Yeah.

• 10:21Daniel Whitenack?

  Yeah.

• 10:22Chris Benson?

  So let me ask you a question about that. Was the model itself fundamentally designed differently? Or was it more of a software input-output thing?

• 10:31Daniel Whitenack?

  Yeah, so the model itself was basically the same in the three cases, although in a more batch sense.

• 10:39Chris Benson?

  I had a feeling you were going there.

• 10:40Daniel Whitenack?

  One thing, of course, you can do is operate on, you know, instead of pushing one input into the model and getting one output and doing that over and over, of course, you can push multiple in. And then you can also paralyze across the data set and all those sorts of things. So there is a maybe a different inference pattern. The model architecture itself wouldn't be fundamentally different. I think if there was more time, I would have loved to add a kind of upfront component, which would have been like, fine tune the model, take a pre-trained model, fine tune that model with just a little bit of data for a slightly different task or a slightly different domain, and then sort of make it available in these three different ways.

• 11:22Daniel Whitenack?

  I think, I mean, that's almost the sort of boot camp in and of itself. If you can sort of go through that process, I think you're really, you're well, you're at least well set up to talk the language and understand how people go about their AI and ML development. And, you know, I'm sure there's people out there that disagree with me.

• 11:43Chris Benson?

  Yeah, yeah, I would argue that that's almost more of an engineering thing than it is truly a modeling thing. Cause your model is not fundamentally changing. It's just how you're choosing to deploy it and integrate it in with the larger software environment that you're in.

• 11:55Daniel Whitenack?

  Yeah, and I would probably go more, I'm naturally more on that engineering side, probably in the way that I operate than I am and the sort of pure research or creating new model architectures side of things. So if someone wants to go then and like be the next team, developing the next cool model that is, you know, fundamentally, fundamentally different and that, you know, they spend five years on that to develop that model and really prove it out and probe all of the implications and all of that, that is also really needed. But it's the sort of different skill set around research and experimentation.

• 12:29Chris Benson?

  You know, you raise a really interesting, there's a dimension to this I'd like to draw. And tell me if you have a different sensation about this, but is I'm looking at the deep learning world over the last year or two compared to the years prior to that, that it seems like we were seeing lots of completely new architectures coming through, through the late teens, especially, you know, early and into the late teens. And then we kind of got into the twenties and we're really in an age of implementation. I don't know if that's because I'm in my bubble and I'm seeing tons and tons and tons of implementation, but not a whole lot of truly revolutionary new ideas coming to market. What is your

• 13:11Chris Benson?

  impression? Am I trapped in my little bubble? Or do you think that we're seeing more implementation than we are research right now because so many more people are getting into deep learning?

• 13:21Daniel Whitenack?

  It's a good question. I think part of it is maybe the maturity of the field in the sense that in maybe previous years, everything seemed a little bit more new because nothing was really normalized yet. It was all this sort of storm of ambiguity and new things. Whereas there's, you know, really developed workflows around transfer learning and, you know, even, you know, running really large scale jobs on multiple GPUs and other things. And all of that's sort of like people have developed workflows around that. So maybe that those pieces of it seem less new, but I don't know. I don't know if I would say that, you know, the progression of new models and architectures has slowed or increased.

• 14:06Daniel Whitenack?

  I don't know. There's definitely still some of that going on, but the perception may be different.

• 14:11Chris Benson?

  It feels evolutionary at the moment to me because I was really wondering this, knowing that we were going to have this conversation among ourselves. I was coming into it, thinking about it and going, when's the last time I saw a completely new architecture sweep through the market and, and really take hold? And, you know, like, you know, back when we were looking at, you know, the, the sweep of convolutional neural networks. And then we hit the age of, of NLP really hitting its stride, followed immediately by another big jump with transformers. And then we've seen reinforcement learning have some jumps in there as well. And I was thinking, hmm, when's the last time one of those big, big jumps leapt out at us? Can you think of what that might've been? What would you think of is the most recent big jump forward?

• 14:57Daniel Whitenack?

  I don't know. I'm also pretty stuck in my little NLP bubble, maybe. So obviously transformers and large-scale language models have been the thing. Your bread and butter. Yeah, definitely. That have really pushed the envelope forward. But that really has happened over the past few years. Yes. There is one thing I'd like to discuss in a bit that I think we can decide if it's new or not. In that sense of new, it's new, but yeah, I don't know. I definitely think that there's more people getting into the field. And because of the tooling and the maturity of the tooling, there's more people that are able to do very advanced and sophisticated things, whereas maybe that was restricted to a very elite before.

• 15:42Speaker 01

  This episode is brought to you by Snowplow Analytics. Snowplow is the behavioral data management platform for data teams. Maximize the value of your behavioral data using Snowplow Insights. A managed data platform that's built on leading open source tech leveraged by tens of thousands of users. Capture and process high-quality behavioral data from all your platforms and your products and deliver that data to your cloud destination of choice. When marketing needs to make data-informed decisions, when product needs next-level understanding, and when analytics needs rich and accurate data, Snowplow is the solution for data teams who want to manage the collection, processing, and warehousing of data across all their platforms and products.

• 16:24Speaker 01

  Get started and experience Snowplow data for yourself at snowplowanalytics.com. Again, snowplowanalytics.com.

• 16:45Daniel Whitenack?

  Chris, you were kind of bringing up this topic of like new architectures and like what is a big leap in terms of new kind of models and what is like building on the shoulders of things before. It's to some degree, I mean, everything's building on things from before. But one of the things that I've really Enjoyed sort of watching and have seen kind of have a lot of discussion on Twitter and in the circles that I'm in is this new model from Facebook Wave to Vec U. I don't know if you've seen this. It's a pretty interesting model. It's a speech recognition model. And the interesting thing, well, I guess it sort of stands on the shoulders of Wave

• 17:26Daniel Whitenack?

  to Vec, which was a previous model, but it is very interesting in that it is unsupervised. That's what the U stands for.

• 17:35Chris Benson?

  Okay.

• 17:35Daniel Whitenack?

  And I think what maybe shocked people or what got them really thinking is that this Wave to Vec U essentially learns how to create transcriptions of audio or speech using purely unpaired data or untranscribed data. So just putting audio in, which is kind of a weird thing if you think about it. Have you seen this or has it crossed your path at all?

• 18:00Chris Benson?

  I had seen the name, but I hadn't dug into it. And I've just, I'm looking at it while we're talking about it and it came out a couple of weeks ago. So it's still very new, if I'm reading this correctly.

• 18:09Daniel Whitenack?

  It's still fresh. You know, of course, speech recognition is very much a problem or often the problem with speech recognition is the fact that you don't have a lot of data. We've talked to people like Jeff Adams or others on this podcast who worked in speech technology for some long time. And a lot of times the problem there is, you know, you need a good amount of high-quality transcribed audio, like we're talking about thousands and thousands of hours of transcribed audio. And you need it sort of pretty high quality. You need a diversity maybe of speakers. You need a diversity of accents and gathering and curating all that data is incredibly labor intensive.

• 18:51Daniel Whitenack?

  And so this model, basically it operates in a slightly different way than previous models in that what it's trying to do is take in unlabeled speech audio, just audio, and generate the phone names corresponding to that audio. So phone names is a word that I, you know, I'm not a linguist. So I learned this after joining with SIL and they were talking, started talking about phones and I was confused about, you know. Phones, cell phones, what type of phones are we talking about? But phone names are phones. Here, we're talking about actual sounds. So phones are the sounds corresponding to the audio. And what you can do is you can actually try to create a model that takes in speech

• 19:36Daniel Whitenack?

  and generates phones from that or phone names. And then you could take those phone names and then map them to text. And it's that sort of generating from the audio to the phones that they did in an unsupervised way. And the way that they did this was actually in a sort of generative adversarial way. So they had audio coming in in one stream to a generator model that tried to generate the phones corresponding to that unlabeled data. And then over here, they had actually unlabeled phonemized text. So they took text that was already, you know, it was good text. It was like, you know, text that they knew was in the language that they're working with. They converted that to phones.

• 20:19Daniel Whitenack?

  And then they tried to see if a discriminator could tell the difference between the sort of, quote, "good phones" that were actually from actual text versus the phones that were generated from the generator model. And so it's interesting that they're working in this in two ways. They're working in this phone space, but then they're also using this idea of generative adversarial networks to allow them to solve this problem in an unsupervised way. So I don't know if this fits into, you know, after sort of talking through it, if this fits into fundamentally, you know, new, different, or just like incremental? I don't know.

• 20:56Chris Benson?

  I would say I don't know that it's a whole new category, but it's certainly an interesting step forward in kind of that spans the natural language processing and generative adversarial network world together. I'm kind of curious, could you talk about like a use case where you could see this being applied? Just to give context for it?

• 21:15Daniel Whitenack?

  Yeah, good question. So of course, this also stood out to me because I work with a lot of local languages. And oftentimes, you either have a very small amount of sort of transcribed audio for a language in one of these lower resource language, or you may have none. But oftentimes, in like, language survey and documentation and language preservation efforts, you do collect audio of the language, right? It's just not transcribed. And in some cases, maybe it's actually never been written down, right? Like, there's many languages out there, people might be surprised, that are still just oral languages. They've never even been written down in any type of script. So I think that although this might not be

• 22:00Daniel Whitenack?

  applied like, out of the box in that scenario, I think it works us much closer to where, you know, people working with these sort of endangered languages and very extremely low resource languages, who are able to gather audio, you know, of the language speakers still, they may have an advantage in terms of being able to actually work towards transcribing that audio more quickly than they would have before.

• 22:25Chris Benson?

  Fair enough. I'm really fascinated by the number of use cases. You know, we've just seen so many orders of magnitude exploding outward the last two or three years, in terms of where, you know, when we started this podcast, and we are closing in now on three years that we have been doing this.

• 22:43Daniel Whitenack?

  That's kind of crazy, right?

• 22:43Chris Benson?

  The landscape, it is a little bit crazy. I agree. The landscape since we started has changed dramatically in terms of how widespread these technologies and specifically these models being deployed has been. It was still a little bit unusual to find use cases in the marketplace where you had deployed deep learning models that were actually in production. They were kind of here and there, whereas now, it is everywhere. And really, it is going back to some conversations that we had some time back. We are seeing, you know, it very much just being part of software development, the idea of model-based engineering is just what engineering has become. It's not a separate thing anymore. It's not a call.

• 23:24Chris Benson?

  All out. And I think it is unlikely these days to build any significant software system without any thought into model deployment as a service or microservice that can be utilized in that, which brings us back to all of those folks out there who may either be trying to create a career in AI or deep learning, or however you want to call it, or even software and trying to figure out where exactly they fit in and how they should approach, as well as a lot of organizations. It's now that the cost has been driven down so substantially from the early days when you, you know, very early when the widespread large things were being done by the big name companies.

• 24:04Chris Benson?

  But now it's available to just about anyone. And all the major cloud providers have full tooling sets, you know, that cover not only the model creation and the model deployment, but the entire software stack with it integrated. So if you have somebody coming out of college right now, and maybe they have a computer science degree, maybe they don't, maybe they're in another area, but this has captured their imagination. How would they help their new employer get into this? In what different areas? How would you, if you were coming in, it's been a while since we talked about it from a complete newbie standpoint. And I'm just curious, how has that changed now in the last couple of years?

• 24:40Daniel Whitenack?

  So when you say that, do you mean like I am a data or AI person coming into a company that's not doing AI things? How do I go about creating that transformation? Or are you coming at it more from I'm coming out of college and I want to step into this world of AI type stuff?

• 24:58Chris Benson?

  I think we should divide them. Let's hit both questions for a few minutes. And I think it's worth asking because you and I have been doing this for such a long time. And it feels like we've been there and done that many times. But in my day job, outside of the podcast, I'm still having people come to me on a regular, you know, day to day basis and asking for mentorship and how do I do this? And it makes me realize how many people are still trying to find a path into it. That happens a lot.

• 25:24Daniel Whitenack?

  Yeah, I think there's actually, and maybe we can go in terms of learning resources for those actually trying to get into the field as an AI person here in a second. But I think that the first of those that you mentioned about like, How do you sort of seed or spawn AI work within an organization that's maybe not operating in that area currently? I think that's actually a very challenging problem. I would recommend, I don't, I don't know if you remember, we were talking about how long the podcast has been going on, but way, way, way back at the beginning in the first few episodes, I forget which of the first few episodes, we had Mike Bugembe on the show.

• 26:05Chris Benson?

  Oh yeah, I do remember that.

• 26:07Daniel Whitenack?

  Yeah. He's sort of an expert in, I mean, I think, I know he's an expert in this area. I don't know how you'd describe himself, but how I sort of think of him is really, really having amazing expertise in this sort of behavioral economics of creating this sort of change within an organization. He has a book called Cracking the Data Code. So that's maybe a resource out there for those looking to create more of that organizational change. But he has a formula that he follows, but I think that part of that really comes down to making sure that you have and develop more and more empathy for the people in your organization and the sort of pain points that they're feeling. The more that I think

• 26:51Daniel Whitenack?

  they can feel truly listened to and that you understand their pain points and why they go about the way that they're solving those problems now and the challenges that they're having, the better chance you have of sort of coming beside them and, you know, bringing them in very early into the process and helping them sort of shape what a solution looks like. Cause once you kind of, I think, get through a first cycle of this, and if it is successful, other people will sort of start asking and, you know, momentum will build. But if you kind of get that wrong the first time and you develop a really bad taste in people's mouth for like AI predictive things, then that's almost impossible to overcome, I think.

• 27:36Chris Benson?

  That's a really good point, and that was a really good answer, actually, to how to address that. One of the things that I'm seeing also is organizations trying to figure out how to kind of what we talked about in the very beginning a little bit, but how to address AI. Do they do it as a separate organization? Or do they, you know, they integrate it in with their data science or their software teams? And that can vary based on the domain expertise they have, what their employees are, what kind of business they're in, and also what technologies they're focused on. I think early on, I saw a lot of AI being broken out more and more as its own thing. And I think the people that drove that were very savvy on kind of internal marketing and getting budget for that.

• 28:20Chris Benson?

  And certainly, as we've seen it normalize over time a little bit, you're starting to see it roll back into more of the data science and DevOps, you know, side of things as well, since it's, you know, it's just another model type. Why, you know, divide up your model types into some are, you know, some happen to use GPUs and some don't. It seems a very artificial, an artificial thing.

• 28:42Daniel Whitenack?

  Yeah, and there's always going to be, I don't think there's any right, you know, perfect solution here, because there's going to be challenges either way. If you try to centralize your data science and AI team within an organization, the challenges you're going to face are those where you have to be very intentional about having those people, you know, reach out and make sure that they connect with the end user, understand their pain points, understand their challenges, and that they really have the tentacles out in the organization. Whereas if you have everybody distributed everywhere, you're probably going to deal with more like duplication type issue, duplication of work. How are they going to share resources? Maybe it's more efficient to have centralized infrastructure, but now it's more complicated

• 29:26Daniel Whitenack?

  because people are in all of these different organizational units and all of that sort of stuff. So I don't think any one particular model is perfect.

• 29:46Speaker 02

  We deserve a better internet and the Brave team has the recipe for bringing it to us. Start with Google Chrome, keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits, we don't need them. Mix in ad and tracker blocking by default, quick access to the Tor network for true private browsing, and an opt-in reward system so you can get paid to view privacy respecting ads. Then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give tipping a try on change.org.

• 30:31Daniel Whitenack?

  Well, as always on these fully connected shows, I think that our conversation has led us exactly to where, you know, we often like to go in these types of episodes and that's to learning resources for people, maybe new ones that have come out or things that have been updated. And we are talking about two different scenarios. One where, you know, the challenge is kind of having an organization be transformed into one that uses AI or data science type techniques and the other where maybe you're an individual and you're trying to break into an organization that's already using AI. Maybe on that latter point, I ran across something that I haven't gone through totally, but I very much want to and I've seen sort of a bit of buzz around.

• 31:15Daniel Whitenack?

  And this is this new book that is called Meta Learning. Have you, have you seen this? I haven't, no. Okay, so originally when I saw the book, I thought, oh, cool. It's a meta learning book and you know, of course, that's a bit of a loaded term in the deep learning AI type world because meta learning, we even have, I think, an episode with, I'm not sure, well, I know we talked to Sheryl Chin about AutoML. I think we talked to a couple of people about meta learning. Yep. Where this is really learning to learn, right? So you're, you have some method that maybe learns an architecture or learns an optimization technique or something like that. You're sort of learning to learn in some way. Well, that's not the kind of... Well, maybe that's part of the book. I don't know.

• 32:01Daniel Whitenack?

  But this sort of learning to learn book is really about how to learn deep learning. So it is meta learning. It's how to learn deep learning.

• 32:08Chris Benson?

  It's the human learning, not the model learning in this case.

• 32:11Daniel Whitenack?

  Yeah, the human learning, not a model learning how to do learning, but a human learning how to do deep learning is a nice title. Yeah, how to learn deep learning and thrive in the digital world. And sorry if I get the name wrong, Radek Osmulski. And it looks like a really nice book. And I think that the focus here, he talks about, I learned to program and do deep learning using online resources. Most of my income over the last two years has come from deep learning roles. How did that happen? If, you know, and he, he basically lays it out. How did he go from, you know, learning deep learning using online resources? And now he sort of makes his career from, or his career is doing deep learning type things.

• 32:56Daniel Whitenack?

  So this-

• 32:57Chris Benson?

  You know, and I think both of us sort of ended up in that same path. Most of our learning has come from self learning.

• 33:03Daniel Whitenack?

  I mean, certainly I'm grateful for the sort of math foundational things in my past and certain programming things, experience with doing experiments and that sort of thing in science. But I mean, physics isn't exactly deep learning. And I didn't really think about machine learning until I was in industry. So, and maybe you can remind people about sort of your status prior to working in AI.

• 33:27Chris Benson?

  Well, I was doing lots of software for years, but actually my, even that would be the same thing in that I learned how to do software fairly early in my career. And my degree is actually in finance. So it has nothing to do directly with any of these things that we have been talking about, whether it be software or deep learning or the things that come next. So I think the beauty of a book like this, as I'm looking through the webpage about it, is that this is really the core about how people build careers now. If you are thinking about how to get there, you have to build in constant lifelong learning into your process. And if you don't do that, you will fall behind. And I meet people all the time that are-

• 34:08Chris Benson?

  Doing both. I meet people that are constantly learning. I know you and I are definitely two examples of that. I have some other good friends, good colleagues that are constantly learning as things are moving forward and figuring out where and how they're going to dig into those ideas incrementally as they go. And then you see people that don't do that and they do fall behind. And as things are moving faster and faster forward, the ability to thrive in a digital world, going back to the book subtitle, is really crucial to having it because anything you're learning right now in more of a formal context, if you're coming out of school, for instance, a lot of that, you know, will be obsolete. The basic math will be there, but the algorithms are going to change. How you're achieving problem solutions will change. And so you really have to be able to do that.

• 34:52Chris Benson?

  I think that is a fundamental for being a digitally literate person in the job world these days.

• 34:58Daniel Whitenack?

  Yeah. And there's also a lot of what I've seen as maybe difficulty in parsing through. It's kind of a good thing and a bad thing. There's so much available online that you can use for your learning. But there's also so much available online.

• 35:14Chris Benson?

  For free or very inexpensively. Yeah.

• 35:16Daniel Whitenack?

  So like, how, how do you figure out what to focus on when there's so much you could focus on and in what order, right? That's often the missing piece. I feel like something like this, and again, I'm expecting that the content of this book at least goes into a little bit of this, although I haven't explored it in detail, but it sounds like from the purpose of this that part of it is, you know, kind of putting some framing around the resources that are online and helping people understand the different things that they might need to explore if they want to have this sort of role or, or be part of this kind of world. And that sort of framing can be almost the missing piece, right? Cause someone publishes a course over here on this topic and someone publishes a course on this topic. And then there's,

• 36:02Daniel Whitenack?

  you know, a thousand courses and there's a route through those courses that makes sense. But if you don't know anything about the topic in the first place, it's hard to determine what that route is. And then you sort of get all confused and maybe discouraged.

• 36:16Chris Benson?

  Yeah, I think a really good tactic, which actually comes straight out of the software world, is the concept of scratching your own itch, is that if you want to get into deep learning and into AI over time, as it continues to evolve, having a personally significant use case, something that you want to get done, and it doesn't have to be a day job thing, it can be something you're doing on your own, it could be doing something around the house, there's an infinite number of possibilities. And we've talked about these off and on over the three years of the show so far, at various times, in terms of things that we wanted to do. And having a use case and figuring out a path through that, and there can be an infinite number of paths through that, if your fascination is how

• 36:58Chris Benson?

  do I create the models, then you spend your time and learning resources on what are the possibilities. Somebody else may be a lot more interested in the engineering side of it, and they're really just keen on taking advantage of transfer learning, and finding a model that's already doing very similar to what you want, if not exactly what you want, and basically getting that into your context and deploying it out there. And those are very, very different paths that you might take to achieve essentially the same thing. But doing that also helps you find your own personal golden path on how you're going to come in to this AI world and be able to be productive and passionate about what you're doing. So really scratching your own itch is finding the things you want to do. If it seems a little intimidating, just try to dig in a little bit and try to figure

• 37:43Chris Benson?

  out what those parts would be, before you go, oh, this is too much, and break it down, and then continue to break each piece down and put them in the right order, and get there. I find that I have a habit of, the things that I've gravitated toward have usually been because of that, because there was a personal fire I had going on that particular thing.

• 38:02Daniel Whitenack?

  Yeah, the other interesting element of this sort of finding your path, and I don't know that this is explicitly detailed anywhere, maybe this would be a good resource that could be out there, but depending on where you want to end up in your career and who you're wanting to work with, the sort of stack that you're working with and It can be very different, as we've talked about. And certain of these courses online sort of assume just like a single lane, because it's all from one perspective in terms of the tech stack and the tooling as well. I'm thinking of, so I was looking up for some ML Ops resources recently.

• 38:44Chris Benson?

  And just by the way, for people who aren't familiar with that, ML Ops is kind of the machine learning version of DevOps, you know, development operations, getting your stuff out there so people can use it in a reproducible and standardized way. Keep going. I just wanted to throw that out for those who weren't used to it.

• 39:00Daniel Whitenack?

  Yep, very good. Yeah, so I was looking for, you know, some resources on this at different courses out there that teach ML Ops. And all of them actually that I was looking at seem really good. But for example, I was looking at the deeplearning.ai new ML Ops course. And I'm sort of looking through there and I think partly because I've been exposed to some of this stuff, I kind of deduced, and partly from, you know, who's, who's teaching it, you can sort of deduce, hey, this is a very TensorFlow Extended specific pipeline of tooling, right? Like they're working with this framework called TensorFlow Extended in terms of how they're doing both model serialization, optimization, deployment, sort of from end to end, this is like the world that they're, that they're living in,

• 39:45Daniel Whitenack?

  which makes sense for, for a lot of companies. But there's a lot of companies that have very drastically different opinions on that sort of pipeline. And another one I was looking at was from Elvis Sarvija. He publishes a lot of great content and he has a site called Machine Learning Ops, a collection of resources on how to facilitate machine learning ops with GitHub. And so this is like a very different, this is all about like, okay, I have code that does machine learning stuff. How can I automate the deployment and manage testing? And all of those things with things like GitHub Actions, which is GitHub's sort of CICD framework, continuous integration, continuous deployment, and how to like version things properly and all that stuff.

• 40:30Daniel Whitenack?

  And so, but then there's another sort of world where like ML Ops, like if you're in a certain organization, they've bought into like a platform like, you know, Domino Data Lab, Data Robot, these sort of things that handle a lot of those things as well, but at an enterprise level. So. Yeah, there's just these different, I don't know how to help people navigate through things like that because if they, if they choose any one of those courses, it's that that's good content, right? That they're going to learn a lot. But then if they walk into a company that's doing the other thing, you know, it's not that it's not relevant, but it's sort of, a lot of it's not going to be the same.

• 41:09Chris Benson?

  Yeah, there's that dichotomy that you are almost going to be required to buy in or invest in some sort of particular flow, an environment, if you will, an ecosystem. You have TensorFlow and PyTorch as two of the, the giant ecosystems and such, and there are others as well. And, you know, we've talked, we recently were talking about Apache on the show. And so at some point, you have to kind of say, okay, well, I have to pick something. And you're going to pick the one that either your company's doing, or that appeals to you, or you read an article, whatever. But it really helps to try to abstract back out a little bit about what it is that those things are trying to do. So you don't lock yourself in and be able to, and at some point, you need to go look at some of the other

• 41:54Chris Benson?

  ecosystems and try to map them to what you just learned in that first one. And that way, you have the ability to understand how you're doing ML Ops. It's not limited to a single ecosystem or community on how they're doing it.

• 42:07Daniel Whitenack?

  Yeah, I think that's a great point. I hope that this discussion, this sort of rambling discussion has been useful for people that are trying to navigate this road. We'll provide links to all of the things that we talked about in the show notes. We'd really encourage people to join our Slack community by going to changelog.com/community where you can share some of your favorite resources and talk about how you've been learning deep learning and other things online. But, um, yeah, that, that, uh, that recording went pretty quick, Chris.

• 42:36Chris Benson?

  It was a fast one.

• 42:37Daniel Whitenack?

  Yeah, I feel like there's a lot to talk about and catch up on, but, uh,

• 42:40Chris Benson?

  Well, we'll have more fully connected shows coming up.

• 42:42Daniel Whitenack?

  More in the future. Yeah. Thanks for the discussion. It was fun.

• 42:45Chris Benson?

  Thanks a lot, Daniel.

• 42:51Speaker 02

  Thank you for listening to Practical AI. We appreciate your time and your attention. If you enjoyed this episode, help us out by spreading the word. Think of a friend, think of a colleague, somebody who would benefit from listening to it, and send them a link. We'd really appreciate it. Practical AI is hosted by Chris Benson and Daniel Whitenack. It's produced by Jared Santo with music by Brake Master Cylinder. Thanks again to our sponsors, Fastly, Linode, and Launch Darkly. That's our show. We hope you enjoyed it, and we'll talk to you again next week.