Data management, regulation, the future of AI

Matthew Carroll and Andrew Burt of Immuta talked with Daniel and Chris about data management for AI, how data regulation will impact AI, and schooled them on the finer points of the General Data Protection Regulation (GDPR). Sponsors: Hired https://hired.com/practicalai – Salary and benefits upfront? Yes please. Our listeners get a double hiring bonus of $600! Or, refer a friend and get a check for $1,337 when they accept a job. On Hired companies send you offers with salary, benefits, and even equity upfront. You are in full control of the process. Learn more at hired.com/practicalai https://hired.com/practicalai. Rollbar https://rollbar.com/changelog – We catch our errors before our users do because of Rollbar. Resolve errors in minutes, and deploy your code with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Linode https://linode.com/changelog – Our cloud server of choice. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2018. Start your server - head to linode.com/changelog https://linode.com/changelogFeaturing: Matthew Carroll – Website https://www.immuta.com/, X https://x.com/mcarroll_Andrew Burt – Website http://www.andrewdburt.com/, X https://x.com/AndBurtChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Introducing the world’s first data management platform for data science. - Immuta https://www.immuta.com/General Data Protection Regulation - Wikipedia https://en.wikipedia.org/wiki/General_Data_Protection_RegulationUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Beta for change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at change law because of Rollbar. Check them out at rollbar.com and we're hosted on Linode servers. Head to linode.com slash change log. This episode of Practical AI is brought to you by hired. One thing people hate doing is searching for a new job. It's so painful to search through open positions on every job board under the sun. The process to find a new job is such a mess. If only there was an easier way. Well, I'm here to tell you there is. Our friends at hired have made it so that companies send you offers with salary, benefits, and even equity upfront. All you have to do is answer a few questions to showcase who you are and what type of job you're looking for.

• 0:45Speaker 00

  They work with more than 6,000 companies from startups to large publicly traded companies in 14 major tech hubs in North America and Europe. You get to see all of your interview requests. You can accept, reject, or make changes to their offer even before you talk with anyone. And it's totally free. This isn't going to cost you anything. It's not like you have to go there and spend money to get this opportunity. And if you get a job through hire, they're even going to give you a bonus. Normally it's $300, but because you're a listener of Practical AI, it's $600 instead. Even if you're not looking for a job, you can refer a friend and hire will send you a check for $1,337 when they accept the job. As you can see, hired makes it too easy. Get started at hired.com slash Practical AI.

• 1:38Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slack with us around various topics of the show at change.com slash community. Follow us on Twitter, we're at Practical AI FM. And now onto the show.

• 2:03Daniel Whitenack?

  So Chris, are you terrified of the GDPR?

• 2:09Chris Benson?

  I am loving the GDPR from my standpoint, trying to learn more about it, but I think though it might be imperfect, it's about the about time we finally start addressing this in a public manner.

• 2:22Daniel Whitenack?

  Sounds great. Well, I brought Matt and Andrew from Amuta onto our show today. Welcome, Matt and Andrew.

• 2:29Speaker 03

  Thanks so much.

• 2:30Speaker 04

  Thanks for having us.

• 2:31Chris Benson?

  Welcome. Thanks for joining.

• 2:33Daniel Whitenack?

  Yeah, so I met Matt and Andrew back, I think in the spring of 2017. We ran into each other at a bunch of different conferences and I realized that these guys have pretty much all knowledge around, you know, AI and regulation and data and privacy. And I was just learning a ton from them. So I think it's great to have them here to discuss some things around AI and how it should be regulated, how it is being regulated, what are what the trends are there. So.

• 3:03Chris Benson?

  I have so many questions for them.

• 3:05Daniel Whitenack?

  I know, I know. This is, this is gonna be, this is gonna be great. So just to start out, Matt, why don't you give us a little bit of a personal intro?

• 3:13Speaker 04

  Yeah, sure. So, you know, by trade chemist, went into the US government, deployed to Iraq, Afghanistan, fell in love with technology enabling that mission. And so leaving the government, went and started a services company, sold the services company, and then eventually got drawn right back into the government around the problem of how do we make the law and data science work together so where we can solve problems at the speed of the business, but still maintain ethical and legal controls around our data. And so that kind of led in 2015 to the creation of Amuta. And so to date, I'm the CEO of the company and we're,

• 3:55Speaker 04

  you know, tiny, going on 37 people, but are growing really fast and it's a great market. And, you know, just excited to be part of it.

• 4:05Daniel Whitenack?

  Yeah, I imagine that all the hype around GDPR and other things is definitely not hurting your business.

• 4:11Speaker 04

  No, certainly not, quite the opposite. But I think the question really, and it would be great to talk about today, is what does it actually mean? I think that's really from a practical perspective. I think sometimes we get a little ahead of ourselves.

• 4:23Daniel Whitenack?

  Yep. Awesome. And Andrew, I believe is a lawyer. Is that correct?

• 4:28Speaker 03

  Yes, guilty as charged.

• 4:31Daniel Whitenack?

  Well, tell us a little bit about how you how you fit into this story.

• 4:35Speaker 03

  Yeah, so I'm my title at Amuta is Chief Privacy Officer and Legal Engineer. And the legal engineer part, I think, is particularly relevant for today's discussion. And basically, my charge at Amuta is to think about how law and data science overlap, and to think about what types of requirements law is placing on data science activities that we can embed within the platform. And so my background is I've been involved at Yale Information Society project for a while. I'm a lawyer. I spent some time working for the FBI Cyber Division, really kind of understanding this conflict between compliance demands and the legal burden associated with data, and all

• 5:20Speaker 03

  of the really new, really important projects that are going on in the data science community. And so the goal is to figure out, how can we think about legal requirements and risk management in a new way?

• 5:33Daniel Whitenack?

  Awesome. Well, it sounds sounds perfect for this discussion. Are you the only law engineer in the world? Are there more of those people?

• 5:41Speaker 03

  Legal engineer.

• 5:42Daniel Whitenack?

  Is it law engineer?

• 5:43Speaker 03

  Legal engineer.

• 5:44Daniel Whitenack?

  Legal engineer.

• 5:45Speaker 03

  So that's a hard question to answer. I don't know any others, but that doesn't mean they don't exist. And there's this kind of like funky history around even the term legal engineering. It actually was coined, I think, in the late 1980s by some folks at Stanford. So people have been thinking about what it would mean to like embed laws within technology for a really long time. So instead, you know, you're a lawyer, instead of writing a memo, you write code. And people have been thinking about that. But I think really in the last few years with, you know, the intersection between data science and regulation, I think that we've seen a real need for this field of legal engineering.

• 6:27Speaker 03

  But I can neither confirm nor deny that I am the only legal engineer.

• 6:32Daniel Whitenack?

  All right, well, thanks for those intros. One of you guys, maybe Matt, tell us a little bit about, you mentioned Amuta. What is Amuta? What are you guys trying to accomplish? What does Amuta do?

• 6:44Speaker 04

  Yeah, so, long story short is, Amuta is a data management platform for data science. And the creation of Amuta was really at the intersection of, we have these three users that make up data science operations. We have data owners, you know, and they control the data. You get data scientists who want to use the data to do something with it and to provide insight. And then we've got these governance and legal teams that need to oversee that process. And the problem is, is that, you know, what we found, all of these three parties are kind of acting as antagonists to one another. It's a very human process, lots of meetings, very slow. And so there wasn't a way to really provide each one of them with a single digital platform

• 7:27Speaker 04

  where they can all work symbiotically, even if they don't know it. And so we had to create Amuta, because what we felt was, data management for application development is very different than data management for data science operations. And there was a massive gap. So if, you know, other than the few companies in the world, like Google, or, you know, maybe even from the government side, NSA, where you have thousands and thousands of people that can write code, that can request ad hoc access to all these distributed databases, right? There's all these silos of data, and they can just write ad hoc code to that to get access to it. Very few organizations in the world can do that. And the most successful, you know, can. But then there's all these others, and they don't have people that can write code.

• 8:12Speaker 04

  So they have, they built applications. And based on predicting the types of questions their analysts would want to ask of the data, they'd throw BI tools up, they would throw their own custom apps on top. And so their version of data science was much more complicated, as they brought data scientists in, who need to ask ad hoc questions across all these silos. They're managing their keys separately. So they've got hundreds of keys to get access to these databases. They're making lots of copies of the data and bringing it local. So they're potentially breaking rules by doing that. And so it was very cumbersome and slow. So Amuda, what we did was is we built a, think of us as a data control plane, where we can connect to any stored data in any storage system.

• 8:55Speaker 04

  We can then virtually expose that data in a read-only fashion to BI tools, data science platforms, IDEs, through generic access patterns versus custom APIs. So like File System, SQL, Spark, Dope, and connect to any of this disparate data through a single connection. And then finally, what we did was, and I think the most valuable thing that Amuda provides right now is, we then built an interface for lawyers to be able to implement rules on this data and dynamically enforce those rules as people ask the questions. And so what that allows us to do is, data owners can virtually expose their data in a catalog. Data scientists can bring any tool to bear, connect it to that data. And governance teams and general counsel can actually implement

• 9:39Speaker 04

  the law as it changes on that data without impacting any of the other parties. And so the idea is, is that we can actually streamline the process by which people can get access to data, so they can connect to it. They can control the data, depending on the current state of law or where they are, what region they're in. But then we can also prove compliance. And some of that isn't just, you know, your user rights, but it's also, and I'm sure Andrew will talk about this later, it's, but why? Like, for instance, like with the Facebook, you know, data leaks or scandal, if you will. It's not just that people use the data. They legally could use the data, but it was around, but why are you using it? And is it for nefarious purposes? So this concept of

• 10:24Speaker 03

  Compliance and understanding who's using the data, what data is it and why? And that's a muta in a nutshell.

• 10:31Chris Benson?

  That's pretty fascinating. Yeah. So you guys are, having thought about this for the last few years, are probably way ahead of us in terms of thinking about kind of where regulation around AI and data is at this point and kind of what the lay of the landscape is. Could you talk a little bit in the context, especially with GDPR finally being a reality for everyone and where in general, regulation is going? And maybe even a bit about what GDPR is kind of getting right or wrong or could fix. Give us a lay of the landscape and maybe what we should see in the next months and years?

• 11:09Daniel Whitenack?

  Yeah, and we should probably say that the GDPR is, these guys are going to correct me if I'm wrong, the General Data Protection Regulation. Is that the right, is that the right thing of the acronym?

• 11:20Speaker 03

  Bingo, yes.

• 11:21Daniel Whitenack?

  Awesome. Yeah, so it's an, it's an EU thing, right?

• 11:25Speaker 03

  Yes, it applies to, so I'm happy to take this one. So the GDPR basically applies to any EU data that could be considered personally identifying. And so the standard in practice is was the data generated in the EU and could you use it to figure out anything about a human being? You know, could I identify your name? Could I identify where you live? IP addresses are widely considered to be identifying under the law. So if you're a data scientist and you're working with data that comes from the EU, the answer is that in practice, this applies to you.

• 12:06Speaker 03

  And let me get to, so I'm happy to talk about the AI issue. That's a, those are really big questions. So, you know, jump in or follow up. But I think in a nutshell, every government in the world is realizing that the power of AI is new, it's a big deal, and they're talking about what to do about it. And so on the one hand, governments like France recently and the UK are saying, you know, we need to have an explicit strategy. To promote this new technology. And then, on the other hand, regulators like the ones that enforce the GDPR are saying,

• 12:49Speaker 03

  "We need to control this," kind of like a "not so fast." And so the GDPR is really the first, I think, major regulation that's been implemented that is explicitly or has parts of it that are explicitly focused on AI. And so, in general, the way it's going to impact, really like, machine learning models is that there are different types of requirements for explainability. You might have heard to it as a right to explainability, which in my own opinion, I think, is a little bit too much. But the basic idea is that when you're using models that are deployed autonomously, or that might be inherently opaque, or as some call them, you know, "black box" models,

• 13:29Speaker 03

  the people who are subject to these decisions, whose data is being processed by these models, have basic rights. And so those rights are, they should be able to understand, you know, how and why their data is being used by the model. They should be able to opt out, or to say, "I don't want a model to be making a decision, let's say, for a credit score. Instead, I want human review." So those are the types of requirements that the GDPR puts into place.

• 13:56Chris Benson?

  So you alluded a moment ago about, you know, you thought that in that one instance, they had gone a little bit too far, too much. And I really love your personal opinion on, you know, what do you think GDPR has gotten right, where it could stand a little improvement, and maybe even speculate a little bit about where you think regulation outside the EU and/or future versions of the EU might go? How will the US respond? And just, you know, what are some of your thoughts there?

• 14:24Speaker 03

  Yeah, so the way, so the Europeans and folks like us in the US, we really approach regulating technology. Very differently. In the US, we tend to want specific regulations focused on specific problems. That's why we don't have one national regulation that just covers all data. And in the EU, they very much like the opposite approach, which is, they'll call it a principles-based approach to regulation. And so they want overarching rules, and the downside is that those rules come at the cost of very, very steep ambiguity and vagueness. And so I think there's a lot of good stuff in the GDPR, and I think the intent is wonderful. I think

• 15:09Speaker 03

  the intent of trying to mandate certain levels of fairness and automated decision-making, that's wonderful. But when the rubber meets the road, I think it's going to be very hard for a lot of data science and of programs that are heavily investing in machine learning. It's going to be very hard. There's going to be a lot of fine-tuning as to what some of these specific provisions actually mean. How specific does the explanation of the model actually have to be? And in particular, when is it not specific enough? When is it vague enough? And when is an explanation not detailed enough that someone should actually be penalized? So those are really hard questions. Legal departments everywhere are focusing

• 15:52Speaker 03

  on them. And we're going to start to see how that regulation is enforced, and I think we'll start to learn from it. I would say that I think one of the strongest points, and this is something we've been really focused on at Imuta, one of the strongest points of the GDPR, in some senses, I think the genius of it, is that it bets really heavily on purpose-based restrictions on data. Most regulations around data are focused on data at collection time, and a lot of that means that the emphasis is on consent. So I consent at collection time to give you X, Y, and Z of my data. And that's the traditional

• 16:32Speaker 03

  approach to data. What the GDPR recognizes is that in addition to that, you need to be able to understand and restrict how that data is going to be. Used as it's being generated. And so that's a really new type of concept. And frankly, I think we generate so much data. It is frankly impossible for us as consumers and individuals to understand what types of insights we're giving up as we generate this data, which is, you know, the beauty of data science. And so placing restrictions on that data's use as it's being generated. I think is really like, I think that is going to be the future of of regulation for data.

• 17:11Daniel Whitenack?

  Yeah, that's, that's super interesting. And I guess it kind of, it kind of leads me into, into my next question. I was really trying, like while you were talking, I was thinking in my own mind, like, like, holy crap, how do I make my models explainable? And I don't know if you've had the same thought. It's a realization I have every once in a while, Chris. I don't know if it's come to you yet as well, but I kind of freak out a little bit when I think about that. Cause I remember, you know, like in my first, first position, I like wrote all of these PowerPoint. Well, it was Google slides, I guess, but presentations on like how my models were working and trying to explain it to just my own team. And it was like incredibly difficult, but it sounds like what you're,

• 17:55Daniel Whitenack?

  at least partially what you're saying in the, in the near term, a lot of the focus is going to be on how your data was, was kind of processed through the pipeline to what end? Not necessarily, you know, explaining a deep neural network to, you know, some, some random person is, am I getting the right sense there or is that, or is that wrong?

• 18:16Speaker 03

  Well, sadly, the answer to both of those questions is, is yes. And it's, it's yes, in the sense that the GDPR is-

• 18:26Daniel Whitenack?

  You're not making me sleep any better.

• 18:27Speaker 03

  Yeah, no, I'm, I'm, I'm sorry not to be a, not to bring easy news. The GDPR has a huge compliance burden. There's no, there's no kind of sidestepping that. Data being used for any purpose, like that needs to be documented. You're not going to be able to use EU data at scale in a data science shop without, you know, a plan for how you got the data, the legal basis for that data. You know, what you're going to do for it. At the same time, there are also requirements on- The types of models you use, or I should say, there are explainability requirements surrounding those models.

• 19:08Speaker 03

  So it's not that you're going to have to be able to, you know, explain the weighting on every single neuron in a neural net, but you are going to have to be able to say, here is, in general, how the model is working, here's where it's getting data from, here is, you know, here are the reasons why it's being used. So it's not the type of explainability, I think, that might have you, you know, waking up in the middle of the night in a cold sweat, but still, you're going to have to be able to provide very basic information about the models that you're using.

• 19:44Speaker 04

  Yeah, and I think the reason that you are concerned is that there's really no frameworks to automate this. So there's this now massive legal burden on the data scientist, who is typically not a lawyer, to expound on, you know, why they made certain decisions, what data was used, who potentially was in that data, for what purpose. And so it's not just about data provenance any longer, but it's also what types of activities were taken to ensure that there was proper ethical curation of the data itself before the model is being trained? And then it's what are the guardrails put in place to ensure that you're controlling

• 20:28Speaker 04

  the model as it's put into production? And these are things that, you know, historically, we've had guardrails in place through software, right? Through SAS, they've always had these guardrails in place, and, you know, other types of software kind of automated a lot of these controls. And now, the problem is, is that the ability for anyone to be a data scientist and use the data and use open source tools, they just don't carry the rigor that's required. And I think that it's not just the GDPR, but we're seeing now, even in the US, California is looking at enacting legislation around data privacy. This is something people care about because what they're afraid of is the open source

• 21:10Speaker 04

  community in large organizations like Google and, you know, Microsoft and Apple are making it easier and easier for anyone to be able to design these models because they're incentivized to buy up small, little AI companies. And so we need to put rigor into these, into guardrails around these pieces of software. And so... It's tough and it's hard and people aren't educated around the law. So that's what makes it even scarier.

• 21:36Daniel Whitenack?

  Yeah, and I think that like one of the things I'm hearing from you, Matt, and also in your kind of when you're talking about what you're trying to accomplish with the Muta. I mean, this spans a lot of different areas, right? All the way from data curation to training of models to deploying of models, to building APIs in which models are deployed and with which they interact. There's a lot of different kind of teams involved in this whole process, right?

• 22:05Speaker 04

  Yeah, and the challenge is that they're traditionally, if you look at most of these organizations, because they're mostly Global 2000 organizations that are really trying to accomplish this at scale, it's these teams aren't sitting next to each other, right? So they don't have the benefit of through osmosis to coordinate effects, right? A lot of them are operating on, you know, five to 10 projects because there's not enough data scientists. There's definitely not enough counsel that's caught up in the space. So they're trying to get smart on, you know, how does each project function? What are they trying to build and why? So it's all very slow and cumbersome. And the worst part about it all is, I think the worst case scenario is,

• 22:48Speaker 04

  you know, you build the most fantastic algorithm that can really change the way a legacy business is operating, right? Like, that's the whole point of this, is we can do things better and faster. And then the whole thing has to be brought down because they have no insight to how it's working. And the whole business is then crippled by this. And so that they don't want to make the investment in these advanced technologies and because they don't have the legal understanding or the internal engineering to be able to do this in an ethical manner. And that's the fear right now. And that's what the GDPR is for, I think, is I don't think they want to come around and, you know, destroy business. But I do think what they want to say is, listen, there's an ethical way

• 23:32Speaker 04

  and there's this like any engineering process, there's got to be a governance end of it as well. So, you know, hey, Global 2000, if you're going to use EU data, this is how you're going to do it.

• 23:43Chris Benson?

  Well, I'll tell you what, you guys have really had an impact on me in this podcast already. I started off the show saying, hey, I'm excited about GDPR. And I think part of that is as a, as just a consumer out there and, you know, using many services, I am conscious of privacy, but when you, as you've educated us on, on just how, what the burden really is that we're going to all be thinking about and the, and the lack of tools to do that. A little bit worried, I was wondering, would you mind delving into some of the maybe specific industries where you see impact in maybe healthcare or transportation or others, where you think that they're going to have very specific problems to contend with?

• 24:21Speaker 03

  Yeah, so happily, Matt, do you want to, is there any part of that question you want to take?

• 24:25Chris Benson?

  Go ahead, Andrew.

• 24:26Speaker 03

  Okay. So I think, I mean, so to start with, I think one of the most fascinating things about being a data scientist is that the work that you do can apply to so many different industries and the same techniques can be incredibly powerful across healthcare or finance. So I think GDPR in general, I think is, is meant to be as broad and as far reaching as possible. So if you are an organization that really cares about data and is using data, the GDPR applies, you know, directly to you. I think what we are seeing from, you know, the, the global 2000 perspective is I think

• 25:06Speaker 03

  healthcare and life sciences, finance, transportation in particular, I think all of those groups, and then, and then obviously advertising and, and marketing organizations. I think those groups are the first to really get, kind of get hit by the GDPR or to take it seriously. But, I mean, I think the bottom line with the GDPR is unlike in the US where we try to do sector specific regulation where, you know, we have an FDA and the FDA only regulates, you know, specific drugs and products. This is really meant to touch everyone.

• 25:40Speaker 04

  Yeah, and I, to expound on that, I think the, the point of the GDPR, again, it's, it's not designed to be punitive to, to business generally, right? I mean, we want the global economy to scale and it's, it's all, it's in our best interest for that to occur. I think that the focus though is they want to put teeth around a regulation, hence the, you know, 4% of your annual revenue or 20 million euros, whichever is larger, is they, they want to show people, listen, there is a massive slippery slope. You know, when we start to use people's data at scale. And I think that there's a couple of core pieces to this is one is we've never had so many people in the world that are producing data than ever before.

• 26:23Speaker 04

  It's only going to increase as we, as internet proliferates and 5G proliferates throughout the world. And so I think, you know, that the goal here is this is a time in humanity where we can say, listen, we've got to put controls around this because every industry is going to be impacted. Everyone wants to build algorithms. Everyone wants to operate faster and humans yearn for instantaneous gratification of all their apps. I mean, voice is the new cool thing, right? Like, very few people are now wanting to type in their search. They just, you know, use their assistant on their phone, right? And so there's all these personal items that are being introduced to physical devices.

• 27:08Speaker 04

  And there's algorithms behind them and no one really knows how or why it's being used. And I think that's like generally the concern is that's every industry. It's every business in the world, small, mid to large businesses. But I think the global 2000 are going to be impacted the most because, well, let's just be candid. They have the most to lose and the data scientists in those organizations are now carrying the largest amount of risk.

• 27:33Daniel Whitenack?

  Yeah, no, that, that, that makes, that makes total sense. I, I think, I, I definitely, as Chris has, has mentioned, I've appreciated like the, uh, kind of the, the candor and the, the insight that, that we're getting kind of in, in the trends that, that you guys have been, been following and, and examining. I think as a data scientist or AI ML engineer or whatever I, whatever I am, what I'm thinking is, you know, as, as me or as Chris, who's, who's a chief scientist or as a software developer of, of AI apps, what are kind of some practicalities as far as, okay, we, we get that this is a big deal. Give us some good news. What, what can we do?

• 28:18Daniel Whitenack?

  What can, what are some kind of like initial practical steps that would help us kind of move in the right direction of, you know, being responsible and how we deal with people's data, even if we're not in the, even if we're not in the EU? What are kind of some, some first steps that we can take?

• 28:34Speaker 04

  Well, Andrew, go ahead. I think, go ahead, and I think one of the big steps here is probably best.

• 28:42Speaker 03

  Yeah, so happily. So I think one of the biggest takeaways is that what makes for good governance actually makes for good data science. So basically, good governance, I think, translates into good data science. And the reason is-

• 29:03Daniel Whitenack?

  That's great to hear.

• 29:04Speaker 03

  Yeah, exactly. Yeah, so it should be, I think, reassuring in the long run. I think a good way to think about the GDPR is a little bit of a paradigm shift, which is right now, a lot of data scientists end up operating in a little bit of a vacuum, where they say, kind of, here's a project, give me the data, and then I'll get back to you, and I'll tell, you know, I'll play around with it, and I'll tell you what we can do. And that is good, and kind of one-off interactions. It just does not scale. And so the ways to-

• 29:35Daniel Whitenack?

  Yeah, you've summarized well so many years of my life, in which-

• 29:40Speaker 04

  Me too.

• 29:42Daniel Whitenack?

  I used to, I used to frequently refer to myself as the data monkey, rather than a data scientist, because I felt like that was more appropriate, because of the way I operated.

• 29:52Speaker 03

  Yeah, because you're just kind of bouncing around and climbing things, and, you know, the view looks good from here-

• 29:59Daniel Whitenack?

  Yeah.

• 29:59Speaker 03

  -climb a different view.

• 30:00Daniel Whitenack?

  Yeah, no flinging of poo. So that was good. But in other ways, yes.

• 30:05Speaker 03

  I was gonna, I was gonna ask, but thank you for clarifying. So I guess the really big takeaway is that if you're gonna do data science at scale, and you're gonna run a program, you need to really have a sense of what data is available, how do I get access to that data as a data scientist? You need to have some way of documenting what it is you want to do with it, and what you've done with it, if someone else gets added to the project, or if you leave. These are just like, kind of, basic organizational measures to make sure that you can collaborate, and that the documentation is there. And so if you split that up, what that really looks like is, as soon as a project begins,

• 30:48Speaker 03

  understanding kind of key objectives that you want to achieve, key objectives that you want to avoid. Another way of stating that is legal liability. And so that's where lawyers can come in and say- Make sure you don't do this, or if you do this, make sure you're going to mitigate it in some way. So that ends up being really important and then documenting it so people can be added and subtracted from that project. And then there's very specific ways that you can examine the data you're going to use and control, and Matt used the word guardrails. And so kind of setting up some guardrails to think about potential biases embedded in the data, things you can do to try to compensate for it. And as we all know, all data sets are biased.

• 31:32Speaker 03

  So the question is really just trying to prioritize what it is that you're trying to avoid. So I could go on here for a while. I think I mentioned to both of you before we started recording that Amute is going to be releasing a white paper with an organization called the Future of Privacy Forum. And it's literally going to be a white paper designed to be a practical guide for managing risk in deploying machine learning models. And so it's meant to speak to both data scientists and lawyers and to give some real kind of depth to some of these suggestions.

• 32:05Chris Benson?

  And we'll definitely include that in the show notes so that as it becomes available, listeners are able to find it easily.

• 32:11Speaker 04

  To deep dive into there, I think as Andrew said, good governance leads to good data science and they kind of go hand in hand. I think the first thing is, this is one of the things I learned in the government, is lawyers are actually there to help you. They're not there to slow you down. And I think in a lot of these, especially if you're a data scientist that's working in a big organization, starting to align yourself into the governance organization and asking for support early and often is really key. Understanding that most large organizations have gone through some sort of semantic context around their data, right? They know what metadata they have and generally where it is and what are the rules around that?

• 32:55Speaker 04

  And starting to understand risk levels, right? And so bringing those people in and working in your programs, it seems so simple, but yet when we talk to clients, this is not occurring, right? It's on a one-off. Kind of relationship where every so often they ask.

• 33:10Chris Benson?

  Yeah, I've never seen that.

• 33:11Speaker 04

  Yeah, and it's unfortunate because, you know, the thing is that I learned in the government is is they will very, very easily relieve a lot of risk off of you, right? And then you could start looking at it differently, right? It's the risk isn't on you. You've now brought in others to where you've gone to council, you've gotten review. And you can start looking at that and it doesn't need to be technical, right? A lot of times it's just like, hey, I'm using this data, I'm using this data, I'm using this data and I'm bringing it together. You know, where do you think my potential risk is here? Are there any regulations where, you know, maybe it's not PI, but this is considered sensitive data or not? Should we be doing masking on this data or not? You know, and sometimes just the general rules change when you start bringing data together, right, and mashing it up.

• 33:56Speaker 04

  And so I think early and often bringing governance in and having a good working relationship, potentially even creating a working group to where you review different types of projects and potential risks as part of your development cycle.

• 34:08Chris Benson?

  That's a fantastic idea.

• 34:10Speaker 04

  Yeah, and then I think once you get to that point, then the second thing is, is it really comes down to what kind of data are you using? I think a lot of times we just tend to copy all data and then figure out what data we want to use, right? We figure out like how we want to analyze a problem and look at, then look at a bunch of data sets and bring it in. And I think at times we could do a much better job data engineering in the sense of planning it out and looking at it in a generalistic pattern. Meaning if I'm, I'm going to build an algorithm, right, to look at x-rays. Do I really need all images with all the organs in there if I'm just looking at bones?

• 34:53Speaker 04

  Like, is there any value of trying to automate a boxer fracture? For any of you that had teenager angst, a boxer fracture is when you punch the wall. It's almost, or you punch someone's face. Hopefully you just punched the wall because you were angry, right, at your parents or something. And it's a very easy break, you know, and I've been on the wrong end of the wall a couple of times. And so. You know.

• 35:18Daniel Whitenack?

  We won't go into that story.

• 35:19Speaker 04

  Yeah, we'll avoid that here. But but the point is incredibly easy for an algorithm to to look at your your hand and look at the x ray and see, yep, that bone is fractured in such an angle. That is a boxer fracture. And here's the prognosis and and here's what we can do for you. Right? Very easy. But you probably don't need to see all the tissue. You don't need to see all these other things, you know, in there where potentially there's like, oh, well, there's classification and there's this, there's that. Well, why do you need that information? Does my insurer and does my insurance company get that information or not? Um, so these are the leading issues that we see where there's derived information out of the model itself that could potentially be misused against you.

• 36:03Speaker 04

  And so only provide the data that's really necessary, like really plan this out and think it out. Use, you know, kind of mind mapping, right? Like, I only need bone to this. Why do I have any other data in there? Why don't I remove out tissue out of the image itself? How do I isolate the bone? So like that, those are the types of things that I I don't think people really think through when they're going to their models. Because they're, I I think they have a, which is great. They're super positive about why they're building something. And it's just like they're looking at the upside, right? Which is, hey, I'm going to be able to solve this problem for radiologists and they can focus on really complex issues, not realizing that there's a potential nefarious use of the derived results of that model itself.

• 36:47Speaker 04

  So how do you keep that out? And then building that exercise into data engineering is just as critical as the the data cleansing and the data preparation that goes into the model, you know, and feature building itself.

• 37:01Daniel Whitenack?

  Yeah, that's that's super interesting. I know, and I don't know, I I maybe you guys have have seen this trend too, but I think we've kind of gotten into this trend of like, kind of pawning off a lot of the intuition around the features in our models into kind of deeper, more complex models that kind of figure it out on on their own, right? And we never really go back and say, well, these features or this data that we put in isn't. Really isn't really necessary. Why are we using it? It's both, you know, causing us potential, you know, compliance issues, but also it's just making things harder because it's more data and all of that. So, yeah, I don't know. That's definitely one trend I've seen.

• 37:41Speaker 04

  Yeah, more data isn't better. That's what I've always kind of despised about big data is just because you have a lot of data, it doesn't mean you should use it, right? The point is, and I think this is just generally good data science, actually just generally good science, right? Is you're usually on a mission to solve or answer a question or solve a problem, right? And then work backwards. What do you need for that? But I think just throwing more and more data at a model for it to figure out, you know, intra-model potential patterns and useful features out of the data itself isn't necessarily useful holistically to the consumer. It's useful to you, not necessarily the consumer. And I think that's the challenge is that we have to take into account

• 38:26Speaker 04

  who's in the data just as much as the problem we're trying to solve.

• 38:29Chris Benson?

  So it almost sounds like that maybe in if you look at the AI space versus more traditional data science, like, you know, just analytics or ETL or visualization. As you talk about feature engineering and the fact that, you know, more data isn't always the right way to go. Is that a particular concern that I guess we're going to see in the AI space going forward? Is given the fact that we're used to throwing so much data at our models and letting the neural network figure out which features matter. Is that something that AI practitioners need to be particularly concerned about, do you think? Or is it really just the same level as the others?

• 39:09Speaker 04

  No, I'll let Andrew, you know, chime in on this, but I, we have to stop being lazy. Lazy is bad. You know, we're engineers for a reason, right? There's a process and laziness leads to.

• 39:24Speaker 04

  Bad actions and there are bad actors out there. And as we move to a world, which I think is fantastic, where machines can provide the necessary intelligence to augment human decision making. It it's on us collectively to ensure that we hold ourselves to the highest standards. And just randomly throwing a bunch of data in there, just because we are able to collect it, process it and make decisions on it, doesn't mean it's the the ethically right decision. And I think what I'm concerned about is, and maybe this is just a broader theme, is people are really afraid of the Amazons of the world, right? I mean, the inertia behind an organization like that.

• 40:04Speaker 04

  And I think people are willing to take shortcuts to try to catch up through augmenting business operations through machine intelligence. So whether that's AI or it's just like a simple linear regression, right? They're they're trying to automate as much as they can because they feel like that's the only way they can compete against an Amazon. And so I think that's going to lower the barrier of entry to deploy into production. And that's my biggest fear, honestly. It's I'm not worried about Skynet. I'm I'm really not. I'm more worried about we're going to start making bad decisions without understanding the potential repercussions to the direct consumer. Not that AI is going to take over, but rather our AI is incorrect and all of a sudden we're not providing mortgages to a subset of Detroit.

• 40:49Speaker 04

  Or insurance premiums are going up to for, you know, the 30 to 60,000 per year, you know, socioeconomic sector, right? Is that we tend to, based on the data we have, isolate and polarize. And that's based on bad, you know, data governance, in my opinion.

• 41:09Chris Benson?

  So you might summarize it as bias versus, you know, Skynet is.

• 41:13Speaker 04

  Yeah, I mean, I think inadvertent bias, right? There's always bias in our data. I don't think that's ever going to be, I mean, otherwise there's no statistical relevance, right? But I think bias for the wrong reasons without us knowing it is potentially increased based on the the more data we throw it because it's not possible for humans to run through all of the data. And I just don't think that algorithms aren't good at looking at risk where humans are. And so we need a good way to quantify the risk based on the type of model we're using, right? To the type of policies and and the existence of of regulations and law on that data. And the potential negative outcome of the algorithm itself. And we need to merge that together to really think about quantifying risk in a different way.

• 41:57Speaker 04

  The days of parameterization around data to quantify risk are over. The data is too big and vast and complicated. And so it's really outcome-based decision-making is really our future in the AI space. And it all starts with good governance and understanding what debt is going in and why we're using it. And based on that, choosing the right models to attack the problem.

• 42:21Speaker 03

  Yeah, and let me just add in, I think what Matt said is 100% correct. And one of the reasons why I think more data is not just kind of blanket always better is I am worried about both bias and any potential failures. And I think what we're looking at is a world in, let's say, you know, two to four, five years, something like that, midterm future where no one fully understands where all the models that are deployed have gotten their data from. And so if there's a problem, it's going to be really hard to identify exactly why that problem occurred.

• 43:04Speaker 03

  And so I think the type of like tech debt that we're looking at when we're in a world where machine learning is something we're really relying on is gonna, it's gonna change the paradigm. It's gonna mean that we need to do a lot of this governance and risk management upfront. Otherwise, we simply won't be able to fully understand failures when they start to emerge.

• 43:30Daniel Whitenack?

  Yeah, this is great. I don't know, I've thought for a long time, I mean, software engineering and software engineers have had a long time to kind of come up with their, their standards and process around responsible software engineering. And we haven't really done that with AI and data science. And I think a lot of what you guys are saying is, is super relevant to that discussion and, you know, are great takeaways. I mean, be ethical, don't be lazy with your, your models and your data. Talk to your lawyers early. Right, write docs, which everyone should be, should be doing anyway. But docs with regards to, to explainability and think about and quantify fairness on the, on the outcomes and policies that you're trying to enforce in your outcomes.

• 44:14Daniel Whitenack?

  I think all of these are just, just super, super helpful. I know we, we only have, you know, a little bit of time here to discuss all these things. I think you guys have done a great job at, at giving us kind of a crash course into, you know, regulation around AI and practical steps we can take. Where can listeners follow up? You mentioned that there's this, this white paper that you guys are coming out with. I'm super excited to, to read that. You mentioned some institute that was associated with, with that as well. What, what are some other places if, if, you know, our listeners who are out there in the trenches, developing models and, and developing software, where can they, where can they start out there to try to get some, get some more, you know, practical info about this that they'll be able

• 44:57Daniel Whitenack?

  to consume and, and hopefully bring back to their, to their teams and their discussions?

• 45:02Speaker 03

  Yeah, so there's not, sadly, there's not a lot out there. And that's one of the reasons why we wrote this white paper and why frankly, we're so enthusiastic about it. Because I think there's this real need for guidance and, you know, practical guidance. And it's really hard to find. So that's going to be released, the plan right now is mid-June. The organization we're co-releasing it with is the Future of Privacy Forum. And that'll be on the Amoeba website. We're giving a talk at STRATA New York on basically exactly this, on practical ways to govern machine learning. And that'll be, I believe that's in early September. Matt, are there other places you'd recommend that listeners go to get more info?

• 45:45Speaker 04

  No, I mean, I'll kind of second that. There's not a lot on the open web. There's not a lot of formal reading out there. What I would say is this, though, in each domain, there is an abundance of proper governance frameworks, specifically to verticals, different regulations around management API. And I would say, look internally, you know, for those who aren't, you know, individuals or Small companies but reside in large companies. I think the place to start is meeting with your internal governance team. Ask what type of documentation exists out there. And then also talk to application teams as to, you know, what has their documentation process been?

• 46:26Speaker 04

  What is their standards as to how they work with governance teams and with the internal data stewards and potentially even internal data protection authorities? I think the data protection officer is becoming a real title now where people actually, that is their job. So I would say look internal first. Externally, of course, you know, there's this thing called Google and you can search it. But yeah, Andrew's right, it's a shame, but there isn't a lot out there to really get started.

• 46:53Daniel Whitenack?

  All right. Yeah, well, appreciate that. Definitely come see these guys at Strata. Check out their white paper. I know that I've learned a ton from them already and hope to, uh, hope to see you again at a conference soon and, um, and maybe, uh, maybe go out afterwards and avoid those, uh, those boxer fractures for sure. Um, but, uh, but appreciate, appreciate your guys' wisdom here. It's been, it's been super helpful for me. Thank you so much.

• 47:23Speaker 04

  Thanks, gentlemen. I appreciate it.

• 47:25Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes and give us a rating. Go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before our users do here at changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Leno cloud servers. Head to leno.com/changelog, check them out, support this show. This episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com.

• 48:08Speaker 00

  When you go there, pop in your email address, get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.