Explaining AI explainability

The CEO of Darwin AI, Sheldon Fernandez, joins Daniel to discuss generative synthesis and its connection to explainability. You might have heard of AutoML and meta-learning. Well, generative synthesis tackles similar problems from a different angle and results in compact, explainable networks. This episode is fascinating and very timely. Sponsors: Linode https://linode.com/changelog – Our cloud of choice and the home of Changelog.com. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2019 OR changelog2020. To learn more and get started head to linode.com/changelog https://linode.com/changelog. The Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Sheldon Fernandez – LinkedIn https://www.linkedin.com/in/sheldonfernandez, X https://x.com/sheldonfffDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Darwin AI https://www.darwinai.comFermiNets https://arxiv.org/pdf/1809.05989.pdf - a paper that introduces the idea of generative synthesis. COVID-Net https://www.technologyreview.com/2020/03/24/950356/coronavirus-neural-network-can-help-spot-covid-19-in-chest-x-ray-pneumonia/ - Darwin AI’s COVID-related computer vision model. NLP highlights podcast episode on Behavioral Testing https://soundcloud.com/nlp-highlights/114-behavioral-testing-of-nlp-models-with-marco-tulio-ribeiro - behavioral testsSheldon’s explainability primer on medium https://medium.com/@sheldon.fernandez/dark-ai-and-the-promise-of-explainability-part-i-a6b35009a88cUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I'm a Microsoft guy. So I remember BizTalk. I don't know if you remember that product was supposed to do away with all coders. And like, you know, then you encounter it and like one asymmetry in the data or one irregular use case blows the whole system up. So we have enough experience to know, look, we're giving you a tool that takes away the arduous elements of deep learning, but you still apply your creativity and understanding that gets you there a lot faster. And like, I think that's going to be with us for some time.

• 0:26Speaker 01

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com/Changelog.

• 0:44Speaker 01

  Linode makes cloud computing simple, affordable, and accessible. Whether you're working on a personal project or managing your enterprise's infrastructure, Linode has the pricing, support, and scale you need to take your ideas to the next level. We trust Linode because they keep it fast and they keep it simple. Check them out at linode.com/Changelog.

• 1:09Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We're at Practical AI FM.

• 1:35Daniel Whitenack?

  Well, welcome to another episode of Practical AI. My name is Daniel Whitenack. I'm a data scientist with SIL International. And normally I'm joined by my co-host Chris Benson, who is Principal AI Strategist at Lockheed Martin. But this week is kind of weird in a couple of respects. So Chris is is out dealing with a personal thing, which I totally understand, and he'll be back next week. But also, we're all just kind of, at least if you're in the US or watching on from afar, it's kind of a crazy time right now. Yeah, there's a lot of people struggling and suffering and experiencing a lot of hardship,

• 2:15Daniel Whitenack?

  whether that be from the sort of police brutality that's happened, or the looting, or even just the ongoing struggle from COVID. It's definitely a hard time right now. But there's important issues to talk about, actually not unrelated to this. We know as AI practitioners that a lot of how our models behave is driven by the data that we put in. And because we're often gathering data from a biased world, then often our models end up being biased and not fair. And so this is a real problem. So people like to talk about the sort of sentient problem or singularity that they're afraid of AI taking over the world in that way.

• 2:59Daniel Whitenack?

  But I think in our immediate terms, you know, these sorts of problems of bias and fairness and understanding why that happens, the explainability around our AI models is even more so important because of the things that are happening in our world and because of all of these things that are driving the data that we're using. So today, I'm really excited to have an expert on this topic with us, and not only an expert on the topic of explainability and building things related to explainability, but also the CEO of a really innovative company doing a bunch of great things. The company is Darwin AI. And today I have with me Sheldon Fernandez, who's CEO of Darwin AI. Welcome, Sheldon.

• 3:43Speaker 00

  Thank you for having me today. Appreciate it.

• 3:45Daniel Whitenack?

  Yeah, definitely. So as we get started into this topic, I'd love to just hear a little bit about your background and how you got into the AI world and eventually found your way into Darwin AI.

• 3:58Speaker 00

  Yeah, it's quite the interesting story. So I went to the University of Waterloo here in Canada. So I'm right now speaking to you from Toronto, Canada, home of the NBA champion Toronto Raptors, and will be champions for a little bit longer, it looks like.

• 4:11Daniel Whitenack?

  And Waterloo is kind of a tech hub, right?

• 4:13Speaker 00

  Correct.

• 4:14Daniel Whitenack?

  So for Americans who might not be like a little ignorant of Canadian things, that's like a big tech hub, right?

• 4:21Speaker 00

  Correct. It's kind of like the MIT of the United States. So it's a very engineering focused heavy school and a lot of our tech innovation comes from Waterloo. There's U of T, there's Montreal, but like, of course, I'm going to be biased and say Waterloo is by far the best.

• 4:33Daniel Whitenack?

  Of course.

• 4:34Speaker 00

  So I went to Waterloo, did computer engineering, started a consulting company, enterprise software consulting company in 2001 and grew that to a size of 700. And we were acquired in 2017 by a company called Avanon. They're co-owned by Microsoft and Accenture. So my job, and I was the CTO of that company, was to bring emerging technologies to the enterprise about two or three years before the enterprise was ready to use them. And so when DeepMind accomplished what they did with AlphaGo, you might remember that in 2016. I remember really paying attention to that and thinking this was significant. I had followed computer chess, if you remember, for many years.

• 5:13Daniel Whitenack?

  Yeah.

• 5:14Speaker 00

  And you know, it was a very sad day for me when Deep Blue beat Kasparov. I was very downcast and my mom thought it was a girl and when she found out it was that Deep Blue beat Kasparov, I think she was half proud, but also half worried that she'd never get grandkids, right?

• 5:28Daniel Whitenack?

  Right, right.

• 5:29Speaker 00

  But the thing with Go, the game of Go wasn't supposed to be conquered by machines till 2030 or even 2050. So when DeepMind did that, I remember thinking, this is significant and how did they do this? And then really getting into deep learning and doing the Geoffrey Hinton deep learning course. And then when we got acquired, I had been speaking about deep learning. A mutual friend said, go have a conversation with this academic team at Waterloo. It's a special team and you'll have just a wonderful conversation. And I did. And you know, they just had incredible technology and I was just supposed to advise them. I wasn't supposed to start another venture. I had just finished a 17 year journey and was gonna take time off and drive my wife to work and watch The Price is Right and all the wonderful things you do when you're retired. But this team was just too special.

• 6:13Daniel Whitenack?

  Not to be.

• 6:14Speaker 00

  Yeah, right?

• 6:15Daniel Whitenack?

  Yeah.

• 6:16Speaker 00

  So I started advising them and then the talks got more serious and I'm like, okay, I have to do this. And so, you know, we formalized the company in 2017 and got our venture funding in 2018. And four months after that, my wife got pregnant with her first child.

• 6:29Daniel Whitenack?

  Oh, wow.

• 6:30Speaker 00

  So actually, I have two startups. I've got an artificial intelligence startup and a biological intelligence startup.

• 6:34Daniel Whitenack?

  Right, both on the same sort of timeline.

• 6:37Speaker 00

  Exactly, and they're both magical and exhausting in equal measure, but in different ways.

• 6:41Daniel Whitenack?

  Yeah.

• 6:42Speaker 00

  And so, yeah, it's just been an incredible ride. And, you know, our chief scientist, Professor Alexander Wong, is Canada's research chair in artificial intelligence. So this team just has an incredible amount of scholarship and innovation behind them. And so to work with them, to take the product to market, has just been incredibly exciting, especially given the use cases around AI and so forth. So that's a quick journey. And then, very quickly, although I did an engineering degree, I took some time off in my previous venture. I did a master's degree in theology.

• 7:12Daniel Whitenack?

  Oh, awesome.

• 7:13Speaker 00

  Which was just out of interest, and somehow that's significant now because of the ethics and the way we think about AI. So just a fascinating combination of things.

• 7:21Daniel Whitenack?

  Well, there is a number of interesting groups kind of exploring that connection. I know there's a group in Seattle.

• 7:27Speaker 00

  Okay.

• 7:28Daniel Whitenack?

  Also, I work for a faith-based, not value-of-human-life, or whether it's like the sort of things that we started talking about with bias and fairness and all.

• 7:36Speaker 00

  Yeah.

• 7:37Daniel Whitenack?

  Yeah, it's interesting to have those conversations. So interesting to hear that background as well.

• 7:42Speaker 00

  Yeah, for sure.

• 7:43Daniel Whitenack?

  As I was looking through the Darwin AI website and some of your work, I'll link to your website in our show notes, but I was looking specifically through the information about the platform, but also this page you have about research. And it seemed like there were a few kind of themes popping out that were really focus areas, one of those being kind of edge computing and running AI at the edge. So I see things like edge SegNet, which is a compact network for segmentation, edge SpeechNet for speech recognition at the edge. I also saw a theme of generative machines and generating networks in some way. And then, of course, what we talked about a little bit at the beginning, which

• 8:28Daniel Whitenack?

  was related to explainability. So I was wondering if you could kind of give a little background on how those themes came up, maybe starting with the edge case.

• 8:39Speaker 00

  Yeah.

• 8:40Daniel Whitenack?

  Why are people concerned about AI at the edge now? Why did that seem like a sort of direction you wanted to put some focus into?

• 8:48Speaker 00

  Yeah, so let me bring this together and talk about how the core IP was formed because it kind of informs those three areas, right?

• 8:53Daniel Whitenack?

  Yeah, that'd be great.

• 8:55Speaker 00

  So our academics have been working with deep learning for about a decade, right? So well before it even entered the consciousness of the enterprise, our academics knew what this was, they were familiar with the machinery, and they were actually doing it for their own research. And they found it to be terribly difficult to develop deep learning neural networks. And they said, "Look, as academics, this is difficult. We can only imagine how hard this is going to be when non-academics encounter this."

• 9:24Daniel Whitenack?

  And was that difficulty mostly related to like computational infrastructure difficulty? Or was it like the sort of complication around the tooling? Or sort of the theory behind like what's best to do when? Or what was the difficulty mainly focused around?

• 9:42Speaker 00

  So there were three difficulties, right?

• 9:44Daniel Whitenack?

  Yeah.

• 9:45Speaker 00

  The first was, they said, "You need an incredible level of skill to develop these things. The toolsets are immature, but the mathematical background you need is significant and a barrier." The second, as you say, is the computational overhead in running these networks. So our professor often jokes, he originally did this work for some of his scholarship, and he didn't have the funding to pay for hundreds of thousands of dollars in Azure or GCP. So he had to invent a technique to make it quicker. And then the third was, it was so painstaking to do this because you had no understanding of how these networks came to this conclusion. It was like debugging a program without the source code, right?

• 10:26Daniel Whitenack?

  Right, right.

• 10:27Speaker 00

  So those were the three problems they encountered. So they invented scholarship initially for their own purposes to address those problems.

• 10:35Daniel Whitenack?

  To scratch their own itch, essentially.

• 10:37Speaker 00

  Exactly. And so they termed it generative synthesis. And the way it works is, they use other machine learning techniques to probe and understand a neural net. They develop a very sophisticated mathematical understanding of the neural network, and then they use a generative approach to generate an entirely new family of neural nets that is a lot more compact than the original, as good as the original and can be explained, right?

• 11:02Daniel Whitenack?

  Yeah, so as you're kind of describing that, I'm starting to get like vibes of like some of the meta learning, auto ML sort of things. And I know there's a lot of people interested in this. This seems like a sort of unique flavor of it, I guess, at least. Do you see those as kind of these generative techniques that you're talking about and like stuff that people might call auto ML or meta learning? Do you see those as in the same family?

• 11:29Speaker 00

  I'd say they're in the same family. They're analogous to what we do, but but different, right? So auto ML does a, you know, a search across a vast search space and gives you something that it thinks is appropriate.

• 11:41Daniel Whitenack?

  Which in itself is computationally.

• 11:43Speaker 00

  Exactly. Right? So Google will give it to you for free, but you got to do it in GCP, and that's where they get you, right?

• 11:49Daniel Whitenack?

  Yeah.

• 11:49Speaker 00

  It's not free. Whereas we will look at your data and then synthesize a new network from scratch. It's a lot more granular in terms of how we do it. But it is like conceptually similar. And so, yeah, that's the process that these guys invented. And then, of course, we asked the question when we started the business, okay, what's the commercial potential? Like, what do you do with this?

• 12:09Daniel Whitenack?

  Right.

• 12:10Speaker 00

  And one of the first ones, and this addresses your first question, was the edge-based scenario, right? What do you do when you need to deploy deep learning to a GPU or CPU? You don't have, you know, three or four servers to run it. And so that was kind of the first place and crevice that we found when we started thinking about this tech.

• 12:27Daniel Whitenack?

  Yeah. And what have you seen now that you've kind of explored that space a bit, are working with clients and and people that are doing things at the edge? What do you see as the sort of real world driving factor of people wanting to deploy AI at the edge? Because from my perspective, I hear different things. I hear like on the one side, like privacy is the main issue, which I could definitely, you know, see that if if data's not leaving the device.

• 12:54Speaker 00

  Yeah.

• 12:54Daniel Whitenack?

  But then there's also like, you know, your device and maybe you're running at a farm, or maybe you're running at a factory where the connectivity is not that great. So it's a connectivity to the cloud thing.

• 13:06Speaker 00

  Yeah.

• 13:07Daniel Whitenack?

  Or maybe, and then I hear like a third set of things, which is like latency, right? So you got like your your algorithm at the edge, and it's really fast, and you don't have to like wait for things to come back from the cloud or wherever. So from your perspective with clients, what are you seeing as sort of the driving factor there?

• 13:23Speaker 00

  So it depends on the vertical, is what I would say, right? So when you're talking autonomous vehicles, or aerospace, or defense.

• 13:32Speaker 00

  They can't afford a trip to the round server. One, because of latency reasons, if a car needs to make a decision or a drone needs to make a decision in milliseconds, you can't depend on that round trip and they so they have to do things at the edge. So that's the predominant, you know, motivation that I see in those verticals. In consumer electronics and health, privacy is probably the more predominant factor, right? If you have a watch and that watch is using deep learning to do some kind of analysis on your heartbeat or detect COVID or whatever it is. You know, you as a consumer don't want that data shared with some central location that is going to aggregate and monetize that data. And so in that situation, privacy is the motivating factor to do it entirely on the edge.

• 14:14Speaker 00

  So it does differ depending on the vertical that you're dealing with, yeah.

• 14:18Daniel Whitenack?

  Interesting. And just to get a sense, if we're talking, and this is another, maybe a point of confusion that I see a lot of times is like, when we're talking about AI at the edge, what is the edge? Is the edge, you know, a mobile device? Is it a Raspberry Pi with a camera in your house? Is it like a legit computer, but it's like at the edge in a manufacturing place?

• 14:41Speaker 00

  Yeah.

• 14:42Daniel Whitenack?

  What are you seeing trends that way? I know there's a lot of, even Nvidia came out with their new architecture. They have like the edge specific card that they're talking about, so.

• 14:53Speaker 00

  Yeah.

• 14:54Daniel Whitenack?

  And maybe what do you see as the trend of the focus in the future? Or maybe it depends on the vertical again.

• 14:59Speaker 00

  Yeah, it's a, it's a great question. I mean, now I think we use the edge colloquially to describe a scenario where all the processing is done on device, whatever that might mean, with no processing done in the cloud, right? And so that could be a super powerful GPU in a car. One of the autonomous trucking companies, I know they have eight GPUs in the truck, right? And so is that edge? I mean, there's more compute power there than Deep Blue had in 1997.

• 15:27Daniel Whitenack?

  Yeah, or that I ever use for training.

• 15:29Speaker 00

  Or that you ever use for whatever, right? So it is an evolving term. I do think we generally mean on some kind of device or something that is autonomous and self-contained and is not being done on the server, right? So a satellite, a drone, your hand, not a computer in the traditional sense is the way we think of it.

• 15:50Daniel Whitenack?

  Gotcha. And in these cases, what is the biggest concern or hurdle when you're getting to the edge? So you talk about like compact networks, you know, obviously for a thing that is like lower power, you know, if we're thinking about like a small computer or like a small like smart camera or something like that, that's going to be low power. It's not going to have that much power and storage space and RAM and all of that stuff. So is the compactness, is it dual purpose, both for the like getting it on the device and the efficiency? Or what's the blocker there?

• 16:27Speaker 00

  So again, it depends on vertical.

• 16:30Daniel Whitenack?

  Yeah.

• 16:31Speaker 00

  In the case of, for example, defense, they have pretty powerful, you know, devices already that are outfitted on whatever it is they're trying to do. So it's efficiency on those devices. How many concurrent systems can I run with this hardware that I've already agreed is going to be on the device? In autonomous vehicles, it's I need my perception network to recognize the scene in front of me in 10 milliseconds. And I therefore need it to be really fast on this hardware. Right? With consumer electronics, it's a bit different. It's look, accuracy is important, but it's not as mission critical as, you know, finding a child in a scene when a car is driving.

• 17:12Daniel Whitenack?

  Right.

• 17:13Speaker 00

  You know, when you're talking to Siri, hey, you get your last name wrong. Okay, just re-say it again. In that case, it's performance accuracy on the device is usually the predominant factor. So again, it differs, right? Depending on your use case.

• 17:34Speaker 03

  I'm Jared Santo, GoTime's producer and a loyal listener of the show. This is the podcast for diverse discussions from around the Go community. GoTime's panel hosts special guests like Kelsey Hightower.

• 17:47Speaker 00

  And sometimes you can leverage your cloud provider and make margins on top. That's just good business. But when we're at the helm making the decision, we're like, yo, forget good business. I'm about to deploy Kafka to process 25 messages a year.

• 18:03Speaker 04

  It's nerd pride, right?

• 18:05Speaker 03

  Picks the brains of the Go team at Google.

• 18:09Speaker 05

  You don't get a good design by just grabbing features from other languages and gluing them together. Instead, we tried to build a coherent model for the language where all the pieces worked in concert.

• 18:19Speaker 03

  Shares their expertise from years in the industry.

• 18:22Speaker 06

  Don't expect to get it right from the start. You'll almost definitely get it wrong. You'll almost definitely have to go back and change some things. So yeah, I think it goes back to what Peter said at the start, which is just make your code, write your code in a way that is easy to change. And then just don't be afraid to change it.

• 18:36Speaker 03

  And has an absolute riot along the way.

• 18:39Speaker 07

  Yeah, you know that little small voice in your head that tells you not to say things? What, what is that? How do you get one?

• 18:48Speaker 07

  You want one of those? Is it like an in-app purchase?

• 18:51Speaker 03

  It is Go time. Please select a recent episode, give it a listen and subscribe today. We'd love to have you with us.

• 19:09Daniel Whitenack?

  Okay, so I would love to maybe dive into this generative model technology a little bit more. You mentioned that there's this sort of, I think what you call an inquisitor model that kind of studies something. I'm not sure if I'm clear, you know, exactly the process that that goes through. So I guess from a practical perspective, when we're talking about this, is this a case where like, I still am using like the same types of models, like I have a convolutional net or I have a recurrent neural network or whatever it is? And I have another model that's performing this function? Or is everything happening together in the same sort of different model of some type?

• 19:53Speaker 00

  Right. So I mean, it's happening underneath the hood with this technology that uses this inquisitor generated pair. You're giving it a neural network and you're getting a number of neural networks generated that are more compact and work against your data with usually the same accuracy and are faster.

• 20:11Daniel Whitenack?

  Gotcha.

• 20:12Speaker 00

  So like the internals of how it works, it's interesting to academics and we've issued papers on this without giving away the core IP. But really, you're giving me a neural network, which is a graph, and you're getting a much more compact graph as an output. And then our platform will provide the explainability and so forth. Yeah.

• 20:28Daniel Whitenack?

  Gotcha. It would still be up to, let's say, if I'm an AI practitioner and I'm using this type of technology, it may still be up to me to determine, like, hey, here's a computer vision problem. I'm going to train a convolutional neural network on this data. But then afterwards, I'm going to provide it to the system and get a better architecture out. Is it kind of two-stage like that? Or can you do like everything in one shot?

• 20:55Speaker 00

  So you can do it in one shot. You can choose a popular public reference model in our, we're adding this feature to the platform. You can say, look, I have a computer vision problem. I don't know what the best thing is. Is it Inception? Is it ResNet? And we will take a public model and produce a really optimized version for you against your data. Or if you're a more intermediate or advanced user, you might already have a network. You might have already trained it, not done all that pre-work. And you're just going to give that to the platform and say, give me the best version of this against my data. So it can work in either way, depending on where you are in the process. Yeah.

• 21:28Daniel Whitenack?

  Yeah, personally, I kind of like this way of thinking about it. Because oftentimes, when I talk to people about AutoML or meta-learning or something like that, it seems like the end goal of where people want to get is like, I just have data. And then like, whatever sophisticated system I have figures out everything for me.

• 21:48Speaker 00

  Yeah.

• 21:49Daniel Whitenack?

  Maybe a possibility in certain cases that have been very well studied. But I also know just from my own experience that every use case I come up with, it's like weird in some way that just doesn't match like something.

• 22:04Speaker 00

  That's our big thing. Like, we are proponents of human-machine collaboration.

• 22:08Daniel Whitenack?

  Yeah.

• 22:08Speaker 00

  Right? You need a human in the loop to couple the laborious intelligence of AI with your own intuition as a human being. And that's not going away anytime soon, right? I mean, how often, I've been in the technology field for 25 years. I, so many times I've heard you, you don't have, you don't ever have to code again, right? There's all these two, and then like-

• 22:30Daniel Whitenack?

  Software engineers are going to automate away software engineers.

• 22:34Speaker 00

  Right? And like you hear that and you roll your eyes because you've seen it so many times. You know, I'm a Microsoft guy, so I remember BizTalk. I don't know if you remember that product was supposed to do away with all coders. And like, you know, then you encounter it and like one asymmetry in the data or one irregular use case blows the whole system up. So we have enough experience to know, look, we're giving you a tool that takes away the arduous elements of deep learning. But you still apply your creativity and understanding that gets you there a lot faster. And like, I think that's going to be with us for some time.

• 23:04Daniel Whitenack?

  Yeah. Do you think that, I know one of the things, I forget where I saw this. I think maybe it was at a TensorFlow Dev Summit when they were talking about AutoML.

• 23:13Speaker 00

  Okay.

• 23:14Daniel Whitenack?

  That one of the things that they saw as interesting in this process is not so much automating away everything, but just learning new architectures that they wouldn't have guessed prior.

• 23:26Speaker 00

  Right.

• 23:27Daniel Whitenack?

  Is that something that you found in doing this different, it's a different approach, but you are still generating sort of new graphs, like you say. In looking at those new graphs and those new architectures, have surprising things come out from that? In terms of like what's actually needed for, to solve certain problems or?

• 23:45Speaker 00

  That's a great question and one I would have to ask our like deep researchers.

• 23:49Daniel Whitenack?

  Yeah.

• 23:50Speaker 00

  Do they look at the new architecture and does that give them an idea? The fact of the matter is like very few people are designing networks from the ground up.

• 23:59Daniel Whitenack?

  Yeah.

• 23:59Speaker 00

  It's like, you know, like the big five basically do it because they've got the intellectual horsepower to do it now.

• 24:04Daniel Whitenack?

  Yeah.

• 24:04Speaker 00

  Where we do have insights though, and maybe we'll get to this, is the explainability piece of why certain things are being made. Like that is intriguing and that teaches you things that just never would have, would have occurred to you before.

• 24:16Daniel Whitenack?

  Yeah. So I totally agree with you. I, I often, when I teach classes, I say like, most of AI and practice is not sort of. Like drawing networks on the chalkboard and like starting with a blank chalkboard and then going. It's more like cooking in the sense that you get a recipe and then you have to bring your ingredients to it, your data to it, and you might have to change the recipe a bit because you don't have these ingredients or those.

• 24:43Speaker 00

  That's a great analogy. I'm going to steal that.

• 24:45Daniel Whitenack?

  Okay, please do.

• 24:46Speaker 00

  I like that a lot, but I will give you credit when I use it. Yeah.

• 24:49Daniel Whitenack?

  It sounds good. Yeah, I at least I can contribute something to the AI community that way.

• 24:54Speaker 00

  Exactly.

• 24:55Daniel Whitenack?

  Since we've kind of got there naturally, let's talk a bit more about the explainability piece. And maybe we can actually start at a at a higher level there as well and talk about like,

• 25:06Daniel Whitenack?

  let's say in the absence of the things that that you're doing and your team is doing. And actually, you know, many other teams are exploring explainability things. For those that are maybe newer to AI or are in a company and exploring AI and are concerned about it, what are the main sort of reasons why we need explainability? And then at what level do we need explainability? Like, you know, because like, for example, with GDPR, when people are talking about, oh, give an explanation for how how you process people's data. Well, like, I don't always know why my network did something. And I think it would be infinitely hard to describe everything.

• 25:47Daniel Whitenack?

  So what are the main challenges and what expectations can we have for explaining?

• 25:53Speaker 00

  Okay, so great question. So first of all, the fundamental problem with machine learning and deep learning is you are essentially saying to these systems, here's a bunch of data. You infer your own rules as to how you're going to make a decision against this data. And all I care about is the results, right? And the reason machine learning, and you know, this is so powerful, is it is great at, you know, characterizing situations where the rules cannot be codified in human terms. And that's why it's great. And the example I often give is, you know, giving a neural network a picture of something, of a lion or a tiger and saying, hey, classify this, you know? And this was a profoundly difficult problem in computer graphics before neural networks. Something my son can do now at 18 months was impossible, right?

• 26:38Speaker 00

  So it's wonderful that we can do that, but the explainability problem exists because we don't know how the neural network is orienting itself internally with its weights and so forth to reach that conclusion, right?

• 26:50Daniel Whitenack?

  Yeah.

• 26:51Speaker 00

  And so the problem with explainability is if you don't understand how a decision is being made, you don't understand where it will fail. And if you don't understand where it'll fail, there are all these edge cases that are lurking in the network with potentially catastrophic consequences. So a very practical example I can give you, in the early days of Darwin, we worked with a autonomous vehicle company, and I've used this example in my writing a little bit, where their car, the perception network in their car, or the AI in their car, would turn left with increasing statistical frequency when the color of the sky was a certain shade of purple. Right? So just pause on that. Like, you and I know that the color of the sky never influences the way you turn.

• 27:35Speaker 00

  I mean, maybe if there's a volcano on the right or something, right?

• 27:38Daniel Whitenack?

  Yeah.

• 27:39Speaker 00

  But that was mystifying to them. And so without explainability, they couldn't understand, what are the drivers here? It's what we call a nonsensical correlation. And so we were able to help them debug that they had done the training for the turning scenario in the Nevada desert when the color of the sky was this shade of purple, and that was the correlation the car had made, right? But in order to understand those nonsensical correlations, in order to identify the edge cases, you need to have some insight into why the neural network is doing what it's doing. And so that is why explainability is so important, is to make more robust networks and give the data scientist and the deep learning developer tools to make those more robust networks.

• 28:21Daniel Whitenack?

  Yeah, I think you really summarized that well. And I think some of this is, like, it's beginning to be on my mind so much more as I develop. I know just last week, just to shout out one of my favorite podcasts, which is the NLP Highlights podcast from Allen Institute of AI.

• 28:42Daniel Whitenack?

  His name's Marco Ribiro on there, talking, he's from Microsoft, and he was talking about behavioral testing of NLP models. And he basically used a bunch of commercially available systems in his paper, like from Microsoft and Google and Amazon that they sell, for example, for like sentiment analysis. And he did some very kind of like, so what he called minimum functionality tests that were not based on like a training set that they use to train the model, but just were like, the minimum functionality you'd expect from a sentiment analysis. Like, can it get, "I don't like food"?

• 29:16Speaker 00

  Right.

• 29:17Daniel Whitenack?

  The sentiment of that.

• 29:18Speaker 00

  Yeah.

• 29:19Daniel Whitenack?

  But then what he did is he made perturbations on that, like changing, you know, "I don't like burritos" to "I don't like oranges." So like, simple perturbations that should not change the sentiment from like, negative to positive.

• 29:33Speaker 00

  Right.

• 29:33Daniel Whitenack?

  Or changing like, "I like the US" to "I like Turkey."

• 29:38Speaker 00

  Right.

• 29:38Daniel Whitenack?

  And then seeing if that changed. And he actually found these like, huge percentage of failures in these commercially available systems for these kind of minimum functionality tests.

• 29:48Speaker 00

  Right.

• 29:49Daniel Whitenack?

  And some of those things were tied to the way things were represented in the data, like, Turkey was represented in a very negative light in a lot of the data that was trained on this model. So, I totally agree with you. In that case, like, this thing's already deployed, right? And you have this existing problem in the system.

• 30:08Speaker 00

  Yeah.

• 30:09Daniel Whitenack?

  And you don't know it until you hit it.

• 30:11Speaker 00

  Yeah.

• 30:11Daniel Whitenack?

  Whereas you're saying, like, if you're developing, you should ideally know where you're gonna maybe hit some of those pitfalls, or at least understand why you hit those, so you can make the system better.

• 30:21Speaker 00

  And so sometimes, the system will get it right, but for the wrong reasons. Right? So, a very popular example I've heard is, there was a neural network that was trained to detect horses, recognize horses, and it performed admirably well. But what they didn't realize was, apparently, this was news to me, many professional pictures of horses are copyrighted. So, it was actually looking at the copyright symbol at the bottom of the picture, and that was a tell to it. But of course, so what you do then, is you say, "Oh, okay, let's remove the copyright from the picture, and let it organically or naturally look at the features of a horse to detect things." So that is what you're trying to do, is...

• 31:02Speaker 00

  Align how it's triggering on data for decision making with your own intuition that yes, this makes sense, right? Now, sometimes you will actually learn from explainability. The neural network will teach you something about explainability. So our professor does a lot of work with neural network and detection of disease. And so actually, we've been in the news because when Corona became a serious thing here in Canada, we released an open source network called COVIDnet that detects Corona using chest X-rays. But his previous work was detecting lung cancer with CT scans. And we show this example where the neural network was looking at the walls of the lung, which had never occurred to radiologists, apparently. I'm probably oversimplifying a little.

• 31:46Speaker 00

  But they actually started looking at them thinking, huh, what maybe here can we learn from what the neural network is looking at? So explainability-

• 31:54Daniel Whitenack?

  That's really interesting.

• 31:55Speaker 00

  Yeah, so there's a second benefit in that sometimes, not often, but occasionally, it will actually teach the subject matter expert about a new way of thinking about the domain.

• 32:12Speaker 03

  We deserve a better internet and the brave team has the recipe for bringing it to us. Start with Google Chrome, keep the extensions, the dev tools and the rendering engine that make Chrome great, rip out the Google bits, we don't need them, mix in ad and tracker blocking by default, quick access to the Tor network for true private browsing and an opt-in reward system so you can get paid to view privacy respecting ads, then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give tipping a try on changelog.com.

• 32:54Daniel Whitenack?

  So as we're thinking about this explainability piece, I'm kind of curious, we've kind of motivated, I guess, why explainability and some of the pitfalls that people can fall into and also this sort of dual benefit of also learning in some cases from explainability. But I'm curious, like, from a practical perspective, as I'm using this system and I'm learning about my network, what is this sort of range of things that I can learn since, you know, there could be so many different- Types of like features that could be contributing to something. So like when I'm using the system, what sort of feedback am I getting? Is it like, you know, these portions of the network are doing something weird or interesting? Or is it more having to do with the data like this?

• 33:40Daniel Whitenack?

  Like you say, the segment of the data is important for this prediction? Or what's the sort of range of things?

• 33:46Speaker 00

  Yeah, it's a great question. So we really asked the question, how do we surface explainable insights that are most useful to developers? And what we saw was very few of them really go down to the architectural level and tweak individual weights and so forth. It happens, but like in a very small minority of cases, most of them want explainability against data. Why is the network doing what it's doing against this data set or this family of data? So that's what we surface in the platform, right? You know, when we detected a lion being a lion, what were we looking at predominantly? When the network got it wrong and different from the human labeler, what did it get wrong? When you remove the predominant, you know, inputs that the our explainability algorithm detects are important.

• 34:31Speaker 00

  When you remove those, how does the prediction change? So that's kind of the data that we surface that we find really accelerates the deep learning development process.

• 34:41Daniel Whitenack?

  Gotcha. How do you balance the sort of range of data that people are dealing with? Is it a matter of kind of starting to specialize in a few different types of data like text and images? And then like moving on past that, it just in terms of of product development. I mean, it's got to be a burden to sort of think about all of these weird scenarios.

• 35:04Speaker 00

  Yeah, so it's a great question. So right now we've our explainability focuses on things you can represent visually. Object detection, object classification. Your question's a good one. How do you surface explainability for natural language translation? Or something that's inherently non-visual? So we're doing the visual stuff first because it's easier to surface. And then eventually we'll get to some of the other stuff.

• 35:27Daniel Whitenack?

  Yeah, I know that there's some interesting attempts out there. Actually, we had a guy on the podcast who was talking about recurrent units in a neural network and how those behave in terms of the memory or what they pay attention to previously or forward in a text sequence and visualizing that sort of thing. And that was really interesting. I'll link that one in the show notes. So maybe there's some things that are possible there. I could definitely see it's, that in itself is a research topic, almost.

• 36:00Speaker 00

  Yeah.

• 36:02Daniel Whitenack?

  Yeah, yeah, that's interesting. So in terms of the system, how do you make decisions about like, do you provide? And I'm kind of curious about this, just from like an AI startup perspective.

• 36:15Speaker 00

  Yeah.

• 36:15Daniel Whitenack?

  Cause there's a lot of people out there trying to do different things and some kind of like, well, you know, our system bolts on to TensorFlow or has a TensorFlow backend or PyTorch or whatever.

• 36:26Speaker 00

  Yeah.

• 36:27Daniel Whitenack?

  Are you kind of providing a self-service portal for people to do things where like the frameworks and that sort of thing are transparent and they're really just kind of importing models and that sort of thing? Or is it a kind of augmentation to their existing workflow?

• 36:43Speaker 00

  Yeah, it's the latter. So we build on top of TensorFlow right now. You give us a TensorFlow model and the data, we do what we do and then you'll get a TensorFlow output. It's not SaaS.

• 36:54Daniel Whitenack?

  Yeah.

• 36:54Speaker 00

  And so that was a big learning for us when we started Darwin. We wanted it to be SaaS because then we wouldn't have to expose anything. But the enterprise was quickly saying, we're not sharing our models, number one, and we're not sharing our data. So like, you know, disabuse yourself of that fantasy right now. So yeah, so we provide an enterprise platform that sits on top of TensorFlow. We're adding support for PyTorch later this year. Then, you know, that way your workflow doesn't have to change considerably. You just use the parts of the tool that you, that you find are useful.

• 37:23Daniel Whitenack?

  Awesome. And how big is the team now? Or is it quickly growing or what's the?

• 37:27Speaker 00

  Uh, it's about 25.

• 37:29Daniel Whitenack?

  Okay.

• 37:29Speaker 00

  So we're, you know, smaller team in Waterloo. I live in Toronto and was commuting before COVID. But as the commercial traction increases and people start doing it, I imagine we'll be growing in the months ahead.

• 37:40Daniel Whitenack?

  Yeah, and I'm always curious for organizations like yours, you know, if I go to your research page and look at all the great things that you've submitted, or if you look at a company like Hugging Face or others where it's still a relatively small team, but it's like there's a product and there's also like this great research coming out. Like, I don't know how you do it all. Is that like strategic partnership with the university as well?

• 38:07Speaker 00

  Yeah, great question. So two of our co-founders are professors at Waterloo.

• 38:12Daniel Whitenack?

  Okay.

• 38:12Speaker 00

  We're in the unique position of having an organic academic connection with the university. And part of their university responsibility is to publish scholarship and to publish, you know, academic papers and so forth. It's a natural part of our identity. It requires discipline to get the balance right because research is good and academics love to research things. But if you're a startup, you also need to channel that research into your product.

• 38:40Daniel Whitenack?

  Right.

• 38:41Speaker 00

  While at the same time, giving your academics the latitude to explore things for the next, you know, the next gen stuff. So requires discipline. We're hopefully getting it right, but it's something you're constantly thinking about.

• 38:52Daniel Whitenack?

  Yeah, yeah, for sure. I mean, you're doing great work. I'm really happy to see a lot of it. From your perspective, maybe whether that's with edge computing, AI at the edge, or maybe it's with explainability, what are you excited about for the future in terms of like things coming down the road or things that you want to explore that seem like really interesting areas? What are some of those things that excite you personally?

• 39:19Speaker 00

  Yeah, I think one of the big things is really seeing significant deep learning use cases realized in the next, you know, couple years. You know, there's this concept of the adjacent possible that you may have heard about, which is what can you do with your technology, given where the world is at? And sometimes at Darwin, you know, we were maybe a year or two early when we started, you know, and so the industry is now catching up and grappling with the problems that our academics knew they would have 10 years ago. And so what excites me is now that they're actually doing it, now that, you know, Lockheed is trying to do AI and implement robust AI at the edge, the problems we foresaw are ones our toolset helps with.

• 40:02Speaker 00

  So it's seeing how, you know, those use cases play out and just knowing what deep learning can do. And the number of different areas where it can be used. I mean, healthcare, to cite an obvious example, you know, the, the amount of interest we've gotten because that vertical is really looking seriously at experimental technology to create a vaccine is incredible, right? Digital learning, like so many things. I think just the general applicability of it is what fascinates me. I say sometimes it sort of reminds me of the internet in the early nineties. And I don't know if you remember that, but like I was, you know, a teenager in, in high school. And like, we didn't think in 1992 or 93 that these little signals going over telephone wires with modems

• 40:47Speaker 00

  would reimagine the world and reimagine industries. And to me, it's the same thing with artificial intelligence, only greater. So the potential there is, is incredible for me.

• 40:56Daniel Whitenack?

  Yeah, that's really good to hear. And I, I could definitely see what you're saying, even looking at conferences that I attended, like three, four years ago and kind of leading up until now.

• 41:10Speaker 00

  Yeah.

• 41:10Daniel Whitenack?

  It seemed like, of course, there was a lot of focus on like, you know, we did this cool thing with a model. We processed this much data with our big data platform or whatever.

• 41:20Speaker 00

  Yeah.

• 41:20Daniel Whitenack?

  And now I think, you know, I see kind of two huge things, almost in some cases dominating discussion at conferences now, which is like the explainability and fairness and bias piece. And then there's the, um, like, how am I going to manage this now? Like, it's a legit piece of our software stack.

• 41:41Speaker 00

  Yeah.

• 41:42Daniel Whitenack?

  And we've decided to buy in. So like, how do, how do we actually like integrate AI? And, and so you see a bunch of, you know, platforms and tools and, and deployment systems and, and all of those things out there. So, yeah, I, I think, you know, that's really exciting for sure from my perspective, because of course this is practical AI. So, you know.

• 42:05Speaker 00

  Think about the first days when the very first developers wrote in Notepad or the, the text editor. And they just wrote program code and compiled and that was it. And then as it got more sophisticated, we had IDEs and source control and QA and like,

• 42:19Daniel Whitenack?

  Yeah.

• 42:19Speaker 00

  you know, we're speaking at ML Ops, uh, DevOps, you know, virtual event later. And like, yeah, the enterprise is now like, oh, how do we manage this? How do you version models? How do you version data? Like, you know, so it shows you. The seriousness with which organizations are taking this, right? And the tooling needs to be there for it.

• 42:36Daniel Whitenack?

  Yeah, and that it's it's not going anywhere for sure.

• 42:39Speaker 00

  Yeah.

• 42:40Daniel Whitenack?

  I don't know, I feel like, and I've heard people refer to this, that there's kind of going to be this AI layer in the software stack now.

• 42:48Speaker 00

  Right.

• 42:49Daniel Whitenack?

  And just like if you're a software engineer and you don't know anything at all about C I C D. Probably you need to learn a little bit about, at least to where you can interact with that system, not that you have to like spin up your own or whatever. But like you're going to be interacting with this in a similar way, like, okay, maybe you don't need to be an AI expert, but you're going to need to interact with these systems and they're going to be part of our lives, so.

• 43:15Speaker 00

  Exactly.

• 43:15Daniel Whitenack?

  Yeah, for sure. Well, it's been really good to chat with you. Where can people find out more about Darwin AI? And what sort of tips would you give people if maybe they're convinced now that the explainability is important? What kind of tips would you give them for maybe either getting started with it with your system? Or maybe just learning about explainability in general and the topic out there? Any suggestions?

• 43:44Speaker 00

  Yeah, so we actually wrote a pretty lengthy explainability primer on Medium, which goes over the problem, which goes over traditional techniques. And so if you follow us on our Twitter, Darwin AI or our LinkedIn page, we post this all. And so that's a really good place to start because we go over, you know, why the problem exists, what the common techniques are, what our technique is, our scholarship around this. So I think that that's a really good, you know, starting point for people that are grappling with this and thinking about, you know, addressing the problem. Yeah.

• 44:16Daniel Whitenack?

  Yeah, great. And in light of, you know, a lot of the things going on in our nation and otherwise too, I definitely recommend. One thing that I've found really useful out there in terms of the sort of exposing, maybe on the data side, like the biases and the fairness in your data, there's a great toolkit from IBM, IBM Fairness 360.

• 44:35Speaker 00

  Okay.

• 44:35Daniel Whitenack?

  Even if you don't use their toolkit, you can learn a lot about like the various ways people are looking. At bias and data and other things like that. So just for our listeners who might be interested on that topic. But really appreciate you joining us today, Sheldon. It's been a really great and timely conversation and really excited to see what comes out of Darwin AI and how the platform progresses. And I hope we can meet and chat at either a virtual or a real conference or something sometime.

• 45:05Speaker 00

  Yeah, no, thank you for having me. And, you know, as you alluded to at the beginning of the program, there's we're in a period of real challenge as a species. So, you know, Godspeed to everybody who's listening. And, you know, there's a, there's a phrase I told my team this morning that I'm fond of in Martin Luther King, you know, the, the arc of the moral universe is long, but it bends towards justice. I believe that, and it's sometimes hard to see.

• 45:28Daniel Whitenack?

  Yeah.

• 45:28Speaker 00

  And we regress a little bit, but I do believe that's true. So, uh, you know, um, let's just all stay strong and united.

• 45:35Daniel Whitenack?

  Yeah, thank you for that. And thank you for joining.

• 45:37Speaker 00

  Thank you so much.

• 45:42Speaker 03

  Thank you for listening to Practical AI. We appreciate your time and your attention. Word of mouth is the number one way people find new podcasts. If Practical AI has helped you on your AI journey, please do tell a friend. Hey, they'll thank you later. Special thanks to Brake Master Cylinder for the beats and to our awesome partners for their support. Shout out to Fastly, Linode, and Rollbar. If you and your organization would benefit by speaking directly to the AI community, you should sponsor Practical AI. Podcast advertising is highly effective and we would love to work with you. Head to changelog.com/sponsor to learn more. That's all for now. We'll talk to you again next week.