AI adoption in the enterprise

At the recent O’Reilly AI Conference in New York City, Chris met up with O’Reilly Chief Data Scientist Ben Lorica, the Program Chair for Strata Data, the AI Conference, and TensorFlow World. O’Reilly’s ‘AI Adoption in the Enterprise https://www.oreilly.com/data/free/ai-adoption-in-the-enterprise.csp’ report had just been released, so naturally Ben and Chris wanted to do a deep dive into enterprise AI adoption to discuss strategy, execution, and implications. Sponsors: DigitalOcean https://do.co/changelog – Check out DigitalOcean’s dedicated vCPU Droplets with dedicated vCPU threads. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. strongDM https://www.strongdm.com/?utm_source=changelog – Manage access to any database, server, and environment. strongDM makes it easy for DevOps to enforce the controls InfoSec teams require. Discover.bot https://bit.ly/2OWMyw2 – A digital space for bot developers and enthusiasts of all skill levels to learn from one another, share stories, and move the bot conversation forward. Want to learn more about building bots? Get started with their Guide to Bot Building Frameworks https://bit.ly/2Vxk7qO. GitPrime https://www.pluralsight.com/product/flow – Download GitPrime’s 20 Patterns book https://resources.gitprime.com/books/20-patterns/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog(q1), a field guide to help engineering managers recognize achievement, spot bottlenecks, and debug development processes with data. Featuring: Ben Lorica – Website https://gradientflow.com/, LinkedIn https://www.linkedin.com/in/benlorica, X https://x.com/bigdataChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonShow Notes: O’Reilly’s AI Adoption in the Enterprise: How Companies Are Planning and Prioritizing AI Projects in Practice https://www.oreilly.com/data/free/ai-adoption-in-the-enterprise.cspO’Reilly AI Conference 2019 | NYC https://conferences.oreilly.com/artificial-intelligence/ai-nyUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com slash change log.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front end web servers that need to be full duty CPU all day, every day. Then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co slash change log. Again, do.co slash change log.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slay with us around various topics of the show at change.com slash community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:46Chris Benson?

  Welcome to another episode of the Practical AI Podcast. I'm your host, Chris Benson. I am the Chief AI Strategist at Lockheed Martin. Today I am recording live from the O'Reilly AI Conference in New York City. The date is Wednesday, April 17th, when we're doing the recording. And I have the great privilege today of having a conversation with Ben Lorica. And did I get your last name pronounced correctly? That's perfect, Chris. Okay. And Ben is the Chief Data Scientist for O'Reilly Media. He is also the Program Chair for the Strata Conference and this AI Conference. And it's a pleasure to have you on the show.

• 2:27Speaker 02

  Thanks for having me.

• 2:28Chris Benson?

  So, really hoping to cover a bunch of different topics and stuff, but I noticed, I wanted to start out with that I noticed that you had put out a publication through O'Reilly called AI Adoption in the Enterprise. It's an ebook that I know that our listeners can go and download, and we'll put a link to that in the show notes. And I was just wanting to see if you could kind of give us a little overview of the, of what you're covering and maybe do a couple of deep dives, enough to at least tease everyone on what you've, what you've hit in the book.

• 3:01Speaker 02

  So I think at a high level, we wanted to understand the state of adoption of AI, which for the most part, these days refers to machine learning technologies. And so, the first thing we did was we tried to get people to kind of self-describe their level of maturity. So, at a high level, so we, we consider people with mature practice to, to be companies that have certain number of years of having models in production. And then, and then on the lower end, we have companies who are just at the evaluation and exploring stage.

• 3:43Speaker 02

  So if you take these two buckets, you know, mature and exploring, so a couple of interesting things that jump out. One is a level of investments, planned level of investment. The people, the organizations with mature practice plan to invest a substantial amount of money compared to the ones who are still in the exploratory stage.

• 4:07Chris Benson?

  And do you think that's mainly just because they're still kind of convincing themselves, proving out the technology and its usefulness, or?

• 4:13Speaker 02

  Yeah, I think, I think so. But, so before I dive into that, so for us, that seems to indicate that maybe the gap between the leaders and the laggards may widen a little more so far as machine learning. So as far as what are some of the key bottlenecks that the respondents cited? So again, there's a distinction between those in the exploratory stage and those with mature practice. The ones with, in the exploratory stage cited problems identifying the right use cases.

• 4:47Speaker 02

  And and company culture, right, so convincing people to invest in AI technologies.

• 4:53Chris Benson?

  Sure.

• 4:53Speaker 02

  The ones who are who consider themselves more mature cite lack of data. So they may have an idea, but they haven't quite collected the right data to execute on that project and lack of skilled people.

• 5:08Chris Benson?

  Gotcha. And you're so are you saying that even the more mature ones are struggling with lack of data at this point in the survey?

• 5:15Speaker 02

  I think that I think that.

• 5:18Chris Benson?

  Or just the immature ones.

• 5:19Speaker 02

  Companies always, you know, so so once, once you start down the machine learning and AI path, you probably start generating ideas and use cases because you get gain more confidence.

• 5:32Chris Benson?

  Mm hmm.

• 5:32Speaker 02

  And and for the for many of these use cases, you may not have the right data yet, right?

• 5:39Chris Benson?

  Gotcha.

• 5:40Speaker 02

  You have to start generating the data and then execute.

• 5:43Chris Benson?

  So could you could you kind of just taking that a little farther, could you kind of describe maybe what a typical on the forefront, those who are kind of leading the way or making the investments, what some of those may look like based on what the survey results showed in terms of did you did you did you go into use cases in it at all or just talking about whether or not they were making the investment?

• 6:05Speaker 02

  So we didn't have them describe their use cases, so but I think at a high level, so the more advanced companies are probably using more deep learning these days, right?

• 6:19Chris Benson?

  Sure.

• 6:20Speaker 02

  So I think that companies who have existing machine learning products or applications have been evaluating how deep learning can either augment or replace their existing systems.

• 6:34Chris Benson?

  Sure.

• 6:34Speaker 02

  So this this applies not to kind of the traditional areas where you that you associate deep learning with, like computer vision or speech or even text, but problems that involve structured data like recommender systems and and time series forecasting. So I think companies are beginning to examine whether or not deep learning can play a role in in improving those systems. And then the really bleeding-edge companies, I think, are beginning to examine machine learning against live data, and now you're starting to enter the world of reinforcement learning.

• 7:15Speaker 02

  So I don't know if you were in the keynotes this morning, but Tony Gibbard from Netflix talked about their work in adding contextual bandits to their recommender systems. So now you're beginning to enter the world of reinforcement learning, which I think is super interesting. At this conference, we had a tutorial on an open-source project from UC Berkeley called Ray from Rice Lab. And it's a distributed computing framework that you can use for a lot of things because it's got a certain amount of flexibility.

• 7:55Speaker 02

  So people, so the Rice Lab team have written some libraries on top of it, including one for hyperparameter tuning. They have a library on top of it called Moden, which for people who are listening who are familiar with pandas in the Python world. So Moden is basically pandas on Ray, which means it runs faster on your laptop and it automatically scales a cluster if you need it to. All you need is to add one line of code. And then it turns out the most popular library on top of Ray is RLlib, which is the reinforcement learning library.

• 8:37Speaker 02

  And so what's nice about this is now reinforcement learning becomes a library that you can just use. If you're a developer, you don't need to write your own algorithms. So the experts can use Ray to write algorithms. And in fact, what they've designed Ray so that both users and reinforcement learning researchers

• 9:00Speaker 02

  Can find utility in it. So it's a great, it's a great new project to pay attention to.

• 9:05Chris Benson?

  No, that sounds great.

• 9:06Speaker 02

  Also, one thing, one thing I should add is that actually Adfinance, which is the largest unicorn company in the world, it's a financial services in China.

• 9:16Chris Benson?

  Mm hmm.

• 9:17Speaker 02

  I don't know the exact valuation, but I think it's over a hundred billion.

• 9:21Chris Benson?

  Wow, big.

• 9:22Speaker 02

  Yeah, and so they're, they're using Ray in production in multiple use cases. One of them is real time personalization recommendation, so.

• 9:33Chris Benson?

  Okay, well, you know, you, you talked also about some of the ones that were, that were just exploring still. And I know, I know in this book that you guys put out, you also talk about, you know, some of the things that are holding back adoption of AI. Can you kind of talk a little bit about that? Because actually, you know, I, I know that we have a lot of listeners in a lot, you know, some of them have been doing it for a while. But we also have listeners who are trying, one of the reasons they listen is to try to kind of figure their way into the space. What, what have you found on things that are holding back adoption of AI?

• 10:02Speaker 02

  So I said, the, the people in the valuation say, stage cite problems identifying the right use cases.

• 10:10Chris Benson?

  Yep.

• 10:11Speaker 02

  Which to me points to one thing, Chris, that I think sometimes we undervalue, which is the need to educate not just your developers and engineers about machine learning, but your organization, right? So your managers and decision makers. Think about when we started talking about big data and how data can drive decision making. Well, you had to educate your workforce about how to make decisions using data. So I think the same thing with machine learning and AI. There's a certain amount of education that needs to be done so that your organization is aware about what's possible, what are the limitations?

• 10:53Speaker 02

  And what are the requirements for some of, for the technologies that we have today?

• 11:00Speaker 02

  And then the second, the second main bottleneck they cite is related to this as well, which is basically just convincing the rest of the company, so company culture about investing in AI.

• 11:13Chris Benson?

  Sure.

• 11:14Speaker 02

  And so, one of the things that we found is that the companies that seem to have taken initial steps and succeeded in terms of using machine learning and AI, they tended to build on existing analytics infrastructure.

• 11:33Chris Benson?

  Just kind of iteratively moving it on.

• 11:35Speaker 02

  Yeah, so you have data that you were using for something else, you maybe start using it for machine learning and AI. Layer a bit of machine learning on top of your decision making.

• 11:47Chris Benson?

  So, so doing that iteratively in that way is probably kind of one of those success factors and that instead of starting something entirely new, you take an existing team.

• 11:56Speaker 02

  Yeah, you might wanna, you might tell yourself, oh, this computer vision is cool, let's do a project in computer vision, but then now you have to gather the data, develop kind of the expertise on how to store that data and so maybe you're better off starting with things you're already familiar with and, and the rest of the organization already appreciates whatever KPIs you have.

• 12:21Chris Benson?

  Sure.

• 12:21Speaker 02

  So maybe improve those KPIs by layering in this new technology.

• 12:25Chris Benson?

  I, I know, speaking from personal experience in industry, you know, getting, getting the data that you need. I have one, without naming the company, it was a company that I was working for and the, the CTO of the company has said, we have all, all the data you could ever want. The thing that I found in reality was it wasn't the right data for doing the kinds of projects that we wanted. And, and so, and then a lot of other companies simply don't have the data pipelining at all in position. Any, any thoughts toward what, what companies can do in terms of getting that kind of prerequisite work done so that they can get to productive machine learning afterwards?

• 13:06Speaker 02

  So, one of the things that I've been trying to socialize and evangelize is that if you wanna build an organization where you can have A sustainable machine learning practice. You can't ignore some of these foundational technologies that you described. Yeah. So things that you might find they're not. What does this have to do with AI, right? So like data integration and ETL. Absolutely. Data governance, data lineage.

• 13:38Speaker 02

  And then, and then, so that's, that's the, the data aspect of what you need to do. But then it turns out, machine learning, people are realizing requires some special tools for machine learning development, right? So one of the most popular open source projects over the last year is a project called ML Flow out of Databricks, which full disclosure, I'm an advisor to.

• 14:04Speaker 02

  It's a 10-month-old project and it's basically a project which has three components and you can use any of the components, but it's meant to facilitate ML development. Okay. Within 10 months, over 200 companies are using it. Oh, that's fantastic. They have contributors from over 40 companies. And then what they're finding is one of the most popular component of ML Flow is the component that helps you track and manage machine learning experiments.

• 14:37Speaker 02

  And so then, so then there's the whole tooling for helping you develop machine learning. But there, I think as, if you, if you look ahead, if you use more and more machine learning and machine learning becomes more and more important to your company, the models themselves will become kind of assets that you have to manage. Just like you have data and data assets and a chief data officer, data governance, data catalog will have to have tools for model governance, model operations, right? So monitoring, tracking, alerts, dashboards for different personas, right? So business users may have a dashboard for tracking models.

• 15:22Speaker 02

  The data engineers may have their own dashboard. The data scientists may have their own dashboard. But also just a catalog listing all of the models, their state, who built them, all these things.

• 15:41Speaker 00

  This episode is brought to you by StrongDM. StrongDM makes it easy for DevOps to enforce the controls InfoSec teams require, manage access to any database, server, and any environment. And in this segment, we're talking to Jim Mordko, VP of Engineering at Hearst. He's sharing how they're using StrongDM within their team of 90-plus engineers. It now takes them just 60 seconds to offboard a team member from a data source.

• 16:05Speaker 03

  We have an engineering team of somewhere in the area of 80 or 90 engineers. Because we've got so many services and many databases and so many developers, we need a reasonable way to manage access to them. It was a somewhat painful and, you know, labor-intensive process. Our DevOps team would literally have to manage every one of the permissions for everybody who wanted access. So StrongDM has been a real godsend in that area for us. Requests for access to specific databases were pretty much manual. Now, we've adopted StrongDM. It's something that you don't even know is there. Once it's installed, it just works. It's very simple. We've set up a multitude of data sources so that when somebody's on-boarded, we just give them access to StrongDM. It's pretty simple.

• 16:48Speaker 03

  Our DevOps team, they have a very minimal effort required to enable each data source to be connected to StrongDM. And then installing the client software is very, very simple and straightforward. You can use whatever client you want to to talk to the database. So there's really no training necessary.

• 17:03Speaker 00

  All right, if your team can benefit from nearly instant onboarding and offboarding that's fully SOC 2 compliant, head to StrongDM.com to learn more and request a free demo. Again, StrongDM.com.

• 17:29Chris Benson?

  To extend that a little bit, I've also seen that people, not only are they not necessarily ready for that as they're trying to get an operation up and running, but they'll also not have really thought their way through, how do you get the model back into a software stack and usable? Out there in a product service, you know, whatever your target environment is that you're going to get that model into and what's the process? And so not only is there a whole aggregate the data you need, get the right data, get it into the right form so that you can use it for training, but then afterwards, when you have a model that presumably you may be iterating on, having that feedback loop that goes, that not only places the model out there into the target, but also is pulling it back in.

• 18:14Chris Benson?

  Any advice on how people should be thinking about, about actually productizing their model, putting it into production?

• 18:21Speaker 02

  Yeah, that's an interesting question because traditionally data scientists have been somewhat not involved with deploying these models and these analytic products into production. And in fact, a couple of years ago, we started notice, noticing in the Bay Area, a new job role with the title machine learning engineer. And this role sits somewhere in between data science and data engineering and data ops. So the focus of this machine learning engineer is to productionize ML models.

• 19:04Speaker 02

  And so that means that they're stronger on the software engineering side and data engineering side, but they have enough data science knowledge to build some of the more routine models. And then, so we started hearing about this role a couple of years ago and then, and about a month ago, I, before our Strata Data Conference in San Francisco, I threw up a Twitter poll because I'd been hearing that data scientists were rebranding themselves somewhat into the machine learning engineer. Because a machine learning engineer, anecdotally, I think is higher compensated. So I, the poll question was clear and, and simple, which was if two years ago, you were

• 19:46Speaker 02

  describing yourself as a data scientist or using the title data scientist, what are you using today? And so I found over, over a third said they're now using the job title machine learning engineer. So now it might be the case that some of the data scientists have up scaled their software engineering skills and become machine learning engineers, but there might also be a cohort of them who are. Who have rebranded themselves. Now, the other thing too is that the tools for going from a model that's a prototype to production.

• 20:27Speaker 02

  So there are startups and companies trying to build tools to kind of blur that distinction. So that data scientists who are working on an internal data science platform where they can collaborate can take those models and deploy them into production systems. But traditionally, the production systems are run and managed by a different team. And, you know, I mean, data scientists don't normally wear pagers and get paged when something goes wrong.

• 21:02Chris Benson?

  That's true.

• 21:03Speaker 02

  So I think that, so whenever someone tells me that you don't need to make this distinction, I always ask them, so do your data scientists wear pagers?

• 21:12Chris Benson?

  That's pretty funny, actually. Yeah, so you've kind of gone into talking about how roles are changing. And so I'm going to ask you a little bit about skills and skill gaps. And I'll actually want to lead in a little bit. One of the things that I have noticed in recent years as I've been part of organizations that have started to turn that corner and try to set up their own AI operations and make those happen was that the, in a lot of cases, the data scientists that were already there had no experience or real understanding about deep learning as they were trying to ramp up. And that it was certainly a distinct skill set from things that they had done in the past. What, you know, speaking towards that and as well as the general set of skills that

• 21:57Chris Benson?

  it takes to make all this stuff happen in this space, what are you seeing? What are the gaps? What, how are people managing that?

• 22:04Speaker 02

  So first off, the job title data scientist has kind of become confusing to some people. So I will not name these companies, but they're in the Bay Area. One is in social media. One is in, one is in, one is in ride sharing, so I'm not going to name them. But inside these, inside many companies, the term data scientist

• 22:31Speaker 02

  Increasingly refers to two types of people, right? So one is a business analyst, business analytics, mostly does SQL type of work. And then the actual data scientist who does machine learning.

• 22:45Chris Benson?

  Yeah.

• 22:45Speaker 02

  But to us on the outside, when we see Ben is from company X, he's a data scientist. So I've complained to my friends, I said, it's confusing for us on the outside because if you guys have really two different types of personas, then you should give them two different titles, right? But I think the fact that data scientist is a hot title, so then they have to kind of incentivize their employees.

• 23:11Chris Benson?

  Yeah, one of the things, it's interesting. I think when people talk about the need for skills in this area and that they need more people to do it, I certainly sympathize with that. But I've also, I've come to where I, a perspective where I'll disagree with people who say there's not enough data scientists in the world, because I think that's fragmenting. I think this kind of catch-all position called data scientist that was once one thing, and now that we're moving into the space, is becoming a number of different, you know, specific roles that people are taking on in the future.

• 23:42Speaker 02

  Right, right, right. So, and then to answer your question about deep learning, right? So when I first started focusing on deep learning in 2013, so there weren't the open source libraries we have today that are well documented with easy to use examples that you can get started with. So it was mostly confined to a few research groups. You literally had to apprentice with one of these research groups because a lot of it, a lot of the knowledge was passed through oral tradition. So these days, of course, we have good libraries like TensorFlow and PyTorch and BigDL and a bunch of other libraries that have documentation

• 24:27Speaker 02

  and the researchers

• 24:30Speaker 02

  In the academic and industry labs tend to publish their papers and have code that you can start to play with. So there's some, so there's some notion of a running start.

• 24:44Chris Benson?

  You have, I was just gonna say, since I can say that for you, you have media companies like O'Reilly Media that are putting out great material on this to learn by.

• 24:52Speaker 02

  Right, exactly. And then, so then the question is, are those, is that material enough for companies? I think that to the extent that the pre-built models, pre-built architectures and pre-trained models apply to their domains, yes. So for example, if what you need is a speech to text tool, maybe many of the existing off-the-shelf or cloud services will be enough. But, you know, take a data type like text, right? So last year was a big, big year for natural language models and research.

• 25:36Chris Benson?

  Huge, yeah.

• 25:37Speaker 02

  But if you dig, if you drill down, right, so many of the models were published with code, some of the models are even pre-trained, so you can use them. But then they may not be quite tuned to your domain.

• 25:54Chris Benson?

  Sure.

• 25:55Speaker 02

  So for example, if you're in healthcare and you want to use one of these pre-trained models, well, even within, even within healthcare and medicine, different areas of specialization have very different lingo and shortcuts for how they communicate with each other, right?

• 26:10Chris Benson?

  Yes, that's true.

• 26:11Speaker 02

  So you might still need staff who are capable of doing some of this tuning and retraining and things like this.

• 26:21Chris Benson?

  So, and how does domain knowledge about that, since that's kind of what we're getting into at this point? How do you overcome that? My next question was going to be, how are organizations kind of using AI and all that? But you're kind of pointing out another one. You know, we've talked about several of the challenges on making all this stuff happen. And that is one, is being able to marry your domain knowledge in very specific areas to the people, the teams of people in the data science space, whatever your team is calling those, that are doing this, ML engineers. How do you make that domain knowledge transfer happen in an efficient way that keeps the business pushing forward?

• 27:02Speaker 02

  So this is part of the kind of the evolution of how some of these software systems are going to be built. If machine learning will play a role moving forward in many of these systems, then a lot of software development may start resembling ML development, which means, you know, gather data, train a model, evaluate the results, and then repeat, rinse and repeat. But then that might also mean consulting with domain experts who know what data might be useful.

• 27:40Speaker 02

  And actually, honestly, in many cases, data is not perfectly cleaned in the beginning. You have to kind of clean the data and prepare it. And again, that's where the domain experts might be helpful to you. So then, one trend that I am seeing is that in the case of data preparation and data cleaning, companies are starting to use tools that actually use machine learning. Because you have a set of domain experts, they can label a few examples, and then maybe a system will automatically kind of go through the rest of the data and try to extract similar

• 28:25Speaker 02

  examples.

• 28:28Speaker 02

  And so, yes, so I think that domain knowledge at least is going to be essential to the extent that we're not talking about general intelligence here. We're talking about very narrow and tuned systems that can help companies automate many, many very specific workflows. So another example I like to cite is robotic process automation. I don't know if you're familiar with this term, right? So I think it's a mistake to think that robotic process automation is something that will be directed from above, right?

• 29:10Speaker 02

  Because I think it's gotta be a ground from the ground up.

• 29:14Chris Benson?

  I think that it's a terribly name field, though. It's very confusing for most people to understand it.

• 29:20Speaker 02

  Because it's the those workers who are in the front lines who know which tasks are repetitive.

• 29:28Chris Benson?

  Absolutely.

• 29:29Speaker 02

  And if you explain to them enough about at a high level what the current technologies are capable of doing, they can help you identify which of these workflows lend themselves to automation or partial automation, right? Because maybe it won't be full automation, right?

• 29:45Chris Benson?

  So one of the things that I'm kind of extracting from your explanation is there's this kind of democratization of the technology as it's becoming widespread and finding many, many use cases, even within a given organization, but you're seeing it all over. I think, is it fair to say that this field, and I'm kind of talking about deep learning as it's finding more and more use cases, it's going to become somewhat synonymous with software development in the sense that as you have ML engineers become part of that team as a standard thing, it's no longer the cool new hotness that you're doing, but it's just an everyday thing down the road that you're really going to have.

• 30:28Chris Benson?

  Neural computing is really kind of the future versus the deterministic, you know, I have all my if, then, and case statements of the past, that neural computing will be part of many, many software stacks out there?

• 30:39Speaker 02

  I would say machine learning, because as you know, right now, of course, deep learning is very successful in many, many areas of

• 30:52Speaker 02

  That affect companies like computer vision, speech recognition and text. But, you know, if you follow the history of machine learning, there are, there are things that go in and out of fashion. Although, right now, we're not seeing anything coming close to challenging deep learning in a variety of tasks, right?

• 31:12Chris Benson?

  Sure, but that's probably, I agree with you.

• 31:14Speaker 02

  That's not a given. I think the workflow will probably remain the same. To the extent that machine learning is part of software development, that workflow will be the same.

• 31:24Chris Benson?

  It's becoming a part of everyday life that companies are using in production for all these areas.

• 31:29Speaker 02

  Yeah, and so actually, one of the things that we aspire for in this conference is to kind of, so we have, so we have, in this conference, the AI conference, the O'Reilly AI conference, we have a business summit. So we have content for decision makers and managers. So they know what other people are doing, so case studies. But also, give them high-level overviews of important topics through executive briefings. But we also have content for developers, who are not data experts, who just want to build, build things. But then also, you know, we wanna show people the bleeding edge.

• 32:13Speaker 02

  So we have researchers and machine learning experts. So one, one other area, I think that I've been trying to emphasize is this notion that when it comes to machine learning, I think companies are coming to realize that it's not a simple trying to optimize some business metric or some statistical metric, right? So there's other important considerations, which, over the last year, I've been giving these, I've been trying to give talks around this notion of managing risk. And I've been collecting a bunch of these other considerations, like fairness and bias, privacy and security, safety and reliability, explainability, right?

• 32:55Speaker 02

  So when, so if you take any one of these considerations and risks...

• 33:03Speaker 02

  And you imagine yourself as a company that has begun to use more and more machine learning. Then you start realizing, oh, I really need the foundational technologies, right? So for example, you look at security, right? So your machine learning model gets attacked by an adversary or starts behaving weirdly. Well, now you gotta retrace back. So I need tools that will allow me to go all the way back and audit. Maybe now we're talking about data governance, data lineage, right? So where did this data come from? And things like this. So a lot of these foundational technologies are not just important because you want

• 33:48Speaker 02

  to do more and more machine learning, but it also will allow you to manage risks that come with having a lot of machine learning.

• 34:04Speaker 00

  This episode is brought to you by discover dot bot. Learn everything there is to know about bots at discover dot bot slash practical AI. Discover the bot was built by Amazon Registry Services as an online community for bot creators and makers of all skill levels to learn from one another, to share stories, and they regularly publish guides and resources to answer questions like how to set up payments to your bot, how to stop shopping cart abandonment, what KPIs are worth measuring, how to write an engaging chat bot dialogue. You can even register dot bot domains there. Learn more and explore this huge library of bot resources at discover dot bot slash practical AI. Again, discover dot bot slash practical AI.

• 34:52Chris Benson?

  So referencing back to the ebook, I was noticing that you had some sections on building block technologies and data types. And I was noticing within that you kind of list kind of the respondents, what they were using. And some of them, you know, like supervised learning was right there at the top and deep learning. And then it kind of shows the usage of each. I noticed. Down here that reinforcement learning was still fairly low on the list and yet we're talking about it so much, you know, out there in conferences and talks, you know, we've really seen an enormous interest over the last year or so, maybe two years, in reinforcement learning. Do you see deep learning and reinforcement learning kind of together going forward? Do you think we'll see that rise up on the list?

• 35:35Chris Benson?

  Or what other, I'll leave it a little bit more open-ended. What other technologies are you seeing in the future as likely use cases?

• 35:42Speaker 02

  So I think that, I think that I would say reinforcement learning right now, as I mentioned earlier, the tools are improving and becoming more accessible, so that might let companies play around with it some more. And then I think over the next six to 12 months, we're gonna hear companies share what they've done, and so that's always inspiring, because it's one thing to hear that reinforcement learning is being used for AlphaGo or for a self-driving car, but if you hear an enterprise in your own industry using it, so that may cause you to take pause and try to understand,

• 36:27Speaker 02

  okay, so how did they do it? What did they use? And can we do it ourselves, right? So I think we need, we need companies to start talking about how they use reinforcement learning, and we're going to begin to see that. So as I mentioned earlier, right, so the RAY already has use cases in production, they're gonna start, the team from Berkeley RISE Lab is gonna start trying to convince some of these users to write blog posts, and some of them are enterprise users, so that might inspire people to jump in. But I think in the short term...

• 37:07Speaker 02

  More companies are probably gonna start playing around with deep learning because that seems to be at a point where you can really relate. If you're in a specific industry, chances are there's a company who's using it that you can really relate to.

• 37:23Chris Benson?

  So it sounds like you kind of mentioned that it went back into tools and having those use cases that companies are doing out there. So would it be fair to say that one of the reasons for deep learning that maybe things like reinforcement learning need and probably will get in the near future are having a set of tools out there that are accessible and easy to use so they can start experimenting, along with that probably data sets that are applicable to that?

• 37:49Speaker 02

  So with reinforcement learning, actually, usefully you need, usefully involves a simulation environment?

• 37:56Chris Benson?

  Yeah, that's true.

• 37:58Speaker 02

  It's more of an agent interacting with an environment and you have a reward function and you're trying to learn a policy, which is basically what to do given the certain settings of the environment. Right? So I think, so it's a combination of tools, right? So you have Ray, which I mentioned earlier, which will give you the RL, which will allow you to learn the policy. But you need to have a simulation environment in order to play around or the ability to simulate data, right?

• 38:35Speaker 02

  But I think the main thing is that what motivates companies is seeing their peers use something and seeing how much reward, so the reward function of their peers, right? So if you're in financial services and you hear Ant Finance... As use rate to do real time recommendations of users interacting and learning from live data, then you might be more motivated, right, as opposed to hearing about it being used for AlphaGo.

• 39:13Chris Benson?

  Gotcha. I wanted to actually also go back to something else you mentioned earlier. You were talking about risk in general, and there are different variations. There's bias, there's the ethical and moral considerations of how you're using data.

• 39:29Speaker 02

  Their security and adversaries and privacy.

• 39:33Chris Benson?

  Absolutely. The concept of AI safety, where how you use it in benevolent ways.

• 39:39Speaker 02

  Or not just benevolent ways. AI safety also refers to safe systems. So if you have AI in mission-critical situations, we don't have them yet, and I probably won't ride a plane yet, that uses completely relies just on deep learning and not control theory. But you can imagine deploying these systems in environments where they can kill people.

• 40:05Chris Benson?

  Sure.

• 40:06Speaker 02

  You need error bars and robustness and really the same types of discipline that a lot of these fields of engineering have had to go through and learn.

• 40:17Chris Benson?

  That's true. And for what it's worth, for the record, the FAA doesn't actually allow, they will not certify a neural network at this point because of the black box component. So that's one of the big challenges. Working for a company that's in aeronautical at Lockheed Martin, that's certainly a big challenge is being able to say, to pass all the certifications, you know, well enough both for ourselves and for FAA requirements.

• 40:42Speaker 02

  So actually, the notion of explainability is interesting because people think about it as in terms of You know, the need to understand what the block box is doing from a variety of angles, right? So one is, think about it from the end user's perspective, right? So if the end user can, has some level of understanding about what the system is doing, they're more likely to use it, probably more comfortable. But also, maybe their explainability might help you if things go wrong and you have to audit the system and go back and understand how you can improve it and things like this.

• 41:24Speaker 02

  But, so there's a bunch of startups working on explainability for machine learning and deep learning.

• 41:34Speaker 02

  And I've always tried to get them to think about it more ambitiously, which is, you know, don't aim your tools at the model builders and the data engineers, the machine learning engineers and data scientists in the backend and helping them understand how their model is working and therefore help them improve their model, which is great. But make yourself have a role for the end user, right? So if I have a machine learning product or a product in the enterprise that I want more people in the company to use, maybe there's an explainability widget that allows them to get more comfortable so that they end up using the system more, right?

• 42:17Chris Benson?

  Gotcha. And as we kind of wind up here, I wanted to ask kind of a very high in the sky, open-ended question and just kind of where, you have this opportunity to talk to so many different companies and get perspectives. I'd love it if you would just kind of share where you think, how you think this field is going to evolve over, you know, the relatively near future, over the next few years. I'm not asking for long-term prediction. But where do you think we're going? As a, kind of in an overview, as a summary, how do you see this field evolving?

• 42:49Speaker 02

  So I think so on the research side, so the researchers will continue to publish at a furious pace, right? So

• 42:58Speaker 02

  And a lot of that research is now in the open, usually with code using open source libraries. So people can take advantage of that research. On the enterprise side, I think companies will continue to struggle if they don't understand the limitations of these technologies and understand how they work and how these models get built. So I think there's a certain level of education that needs to take place across the company, not just the technologists.

• 43:41Speaker 02

  And then I think as companies become more comfortable with machine learning and AI, they'll realize they need to build some of these foundational pieces. But also, the industry needs to help companies by building better tools for machine learning development, model governance, model operations, and frankly, automation as well. So it's no surprise that one of the areas in technology where you're seeing a lot of automation is in data science and data engineering itself, right? Because these are the people who understand this technology, and what are they going to do? They're going to automate the things that they can automate.

• 44:24Chris Benson?

  Absolutely.

• 44:25Speaker 02

  So to the extent that you're studying data science today, don't be surprised if by the time you graduate, some of the things you studied have been automated.

• 44:38Chris Benson?

  Yes, I would agree with that. I think that process is accelerating, too.

• 44:41Speaker 02

  Right.

• 44:42Chris Benson?

  Well, thank you very much. It's been a great conversation with you. I know you have been, as I've watched you, you've been very busy through this conference as the program chair. So thanks for taking a few minutes to talk with me about this. And if viewers want to reach, sorry, listeners want to reach out to you, how can they access you? Are you out on social media or anything?

• 45:02Speaker 02

  Yeah, so my Twitter handle is impossible to remember. It's @bigdata.

• 45:06Chris Benson?

  That's a hard one.

• 45:07Speaker 02

  And then if you want to reach me on email, I have an impossibly hard-to-remember email address, [email protected].

• 45:17Chris Benson?

  Those are two fantastic handles there. So well, thank you very much, and I'll let you get back to the conference.

• 45:24Speaker 02

  Thank you.

• 45:27Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for change log is provided by Fastly. Learn more at fastly.com. And we catch our errors before our users do here at change law because of roll bar. Check them out at robot.com slash change log. And we're hosted on Leno cloud servers at the leno.com slash change log. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address, get our weekly email, keeping you up to date

• 46:12Speaker 00

  with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.

• 46:28Speaker 00

  Congratulations. You've listened all the way to the end of the show. And guess what? Got a little surprise for you. Here's a preview of Brain Science, our upcoming podcast coming out very soon. The easiest way to subscribe is to subscribe to our master feed at the changelog.com slash master. Get all of our podcasts in one single feed, plus some extras that only hit the master feed, including Brain Science. Brain Science is a podcast for the curious. We're exploring the inner workings of the human brain. So we can understand things like behavior change, habit formation, mental health, and this thing we call the human condition. It's hosted by myself, Adam Stokoviac, and Muriel Reese, a doctor in

• 47:09Speaker 00

  clinical psychology. It's Brain Science applied, not just how does the brain work, but how do we apply what we know about the brain to better our lives? Here we go.

• 47:20Speaker 00

  As humans, one of the things that separates us from any other animal out there is the fact that we have language, we have words. And we have super powerful words that truly change how we feel and how we make other people feel. If the words we say have so much potential to influence ourselves and the world around us, how do we begin to understand the power of words?

• 47:41Speaker 04

  So words really are the thing that separates us from all other animals. Because, right, sharks, bats, dogs, lizards, they don't talk. And this is really critical when it comes to managing our moods and our feelings. One of the things that I sort of talk about, or even I mentioned earlier about the way in which we file things in our mind according to feelings, this is exactly how we differentiate it too. Thinking about an example like with professional athletes, they, you might say that they get anxious, like before a race or before, you know, a run or a dive. But using that word, it's not really a threat, right?

• 48:26Speaker 04

  But their, their brain would be like, oh, I'm nervous and now I start this whole sequence of events in my body. Whereas if I just change the word to like, I'm anticipating, or I'm excited, it creates a different sort of rollout of emotions as well as physiological responses. I mean, I'm anxious about going to Disneyland is not usually what we say, right?

• 48:49Speaker 00

  I'm excited.

• 48:50Speaker 04

  Exactly, exactly. So it then puts a lid on or files things differently in our mind, which then changes how we feel about it. So in my field, in psychology, I would say, we always say, name it to tame it. The better I can name different feelings, the more I can tame whatever emotion that is. And so then I'm not really stuck living in this sort of mammal and reptile lane where I'm always just flipping my lid, I'm reactive, I'm angry, or I'm sad. But rather I can go, I recognize this is how I'm feeling, or like, I'm, I'm afraid of some other threat, like losing my job. And I can go, you know what?

• 49:32Speaker 04

  Here's the words I can use to talk to myself about that fear, so that I'm not just stuck feeling afraid of a possible threat, which has never occurred yet.

• 49:42Speaker 00

  You use this concept too to say customized thinking. I'm not sure I fully understand what you mean by customized thinking. What do you mean by that?

• 49:51Speaker 04

  Well, because we are human, we do have the power of choice, which is super powerful. Like, nobody has to tell you how you need to think or how you need to feel. Right? And like, your version of success might be very different than mine, which is going to impact my choices and the direction I'm headed. And so when you think about customized, right? I mean, you can customize a car. You can customize your order at a restaurant. Like, it really is tailored specifically to you and going, how do I want to think and how do I want to feel? One example I consider is I want to always, I want every day of the week to feel like I do on the weekend.

• 50:36Speaker 04

  Because to me, the weekend feels great. I'm with my family. I don't, I'm not sort of running things with such a tight timeline. And there's just a different sort of ethereal vibe to the weekend.

• 50:49Speaker 00

  Right.

• 50:50Speaker 04

  And I think, why does that only have to exist on the weekend?

• 50:54Speaker 00

  Yeah.

• 50:54Speaker 04

  I want that every day.

• 50:55Speaker 00

  Why is that? I want that every day, too.

• 50:59Speaker 04

  Well, and I think part of it is really our attitude and our expectations. I mean, there are legitimate threats all around us. But it doesn't help me do me or do my life any better if I am only focused on threats. So I want to practice changing the channel in my mind that says, hey, yeah, I see that potential job loss, but I also see I'm with my family right now. And right now, nobody can take sort of what I've been through and how I feel away from me. I'm in charge of how I feel. So I'm going to do things that actually contribute to feeling better.

• 51:38Speaker 00

  So how do we apply this name of the tamed idea to this model, then? Because maybe if you name the week, the weekend, can you change how you feel about it? Because that's really what it's about. Like, how do we take, you know, the labels we apply things to things, the names we give things, the words we use, the choices, what I think we might call nuance. I'm not really sure how you, how you put that into play with the power of words. But the difference between, like you said before, being anxious or being excited. You know, fundamentally, it's almost the same feeling, but you know, from a nuanced level, it's very different. You know, it's it's one direction or the other of excitement. You know, negative excitement potentially, or positive excitement. How do we apply that

• 52:23Speaker 00

  to customized thinking?

• 52:25Speaker 04

  Well, I think that's a great way to say it, Adam. I really like that nuance. Because what we're looking for, even as I talk about the different brains, we want a symphony. I mean, I'm not going to fire the woodwind section because I don't like a violin, right? So I don't want to fire a certain part of my brain like, you're not really helpful. I don't need to see that. But what we need is a sense of congruence. And so, sure, not every day of the week can feel exactly like the weekend. So I'm not going to say this is how I feel, but I have to actually believe it for it to impact my mind, my brain and my body in the way in which I desire it to. And so I might use

• 53:10Speaker 04

  the words like, I strive for every day to have a feeling that reminds me of exactly how I feel on the weekend. So that I don't lose sight that like every day really is a gift and I get to enjoy every day of my life to some degree. And so, another example might be, I'm living out in the Pacific Northwest, a lot of people have negative feelings about the weather. Imagine that. But so if someone were to say that they just need to learn to love it, that's going to create what we call cognitive dissonance. It doesn't fit. So it doesn't matter how much I'm like, oh, I do love the gray. I do love the clouds.

• 53:54Speaker 04

  It's not going to jive with me. And so it won't stick. So instead, I can say, I love the way in which the rain creates the green. And in the summer, when it is green, it is amazing.

• 54:08Speaker 00

  This idea of learning to live with it, though, get over it. Uh, it is what it is. Like, there's so many phrases we use to say just that, like, just learn to live with it. What is it called again?

• 54:19Speaker 04

  Cognitive dissonance.

• 54:20Speaker 00

  And what does that mean when you play it out?

• 54:23Speaker 04

  It doesn't go together. So that if you're like, oh, just, just do it. You just need to get over it. Like, that really isn't helpful either because your body is giving you a signal and, and your brain is telling you, I don't like this sensation. I don't like how this feel. I mean, a lot of people will say, oh, I just hate the gray and the gray is just overwhelming. And so we have to go, well, what's my emotional buy-in? Like, what, what do I like? How does that even allow me to enjoy something else? And so I'm gonna look at going, you know what? I really like that I get to wear warm clothes or I really do love my coffee because it's for such a long time.

• 55:07Speaker 04

  It's gray and rainy. I want to be inside by a fire drinking my coffee.

• 55:10Speaker 00

  Right.

• 55:11Speaker 04

  And so how can I look for going, you know what? If I do these things I, I might not want to do, I do get some more of what I do want to do. And so it's really almost like a bartering system in your brain of saying, if you do this thing you don't like, you get this thing you, you do like. Or, you know, I know you don't have to make yourself do this thing. Unless you can see a way in which it actually benefits you or speaks to you emotionally. Everything Adam really has to have this emotional buy-in. And if there's no good emotion, no, really, the primary neuro, neurochemical in our brain is dopamine for feeling good.

• 55:55Speaker 04

  I don't get some hit of dopamine, my brain's gonna be like, it's not worth it and I'm not gonna do it. Period.

• 56:05Speaker 00

  That's a preview of brain science. If you love where we're going with this, send us an email to get on the list to be notified the very moment this show gets released. Email us at [email protected]. In the subject line, put in all caps, "Brain Science" with a couple bangs if you're really excited. You can also subscribe to our master feed to get all of our shows in one single feed. Head to change law.com/master or search in your podcast app for change all master, you'll find it. Subscribe, get all of our shows and even those that only hit the master feed. Again, change.com/master.