The AI doc will see you now

Elad Walach of Aidoc joins Chris to talk about the use of AI for medical imaging interpretation. Starting with the world’s largest annotated training data set of medical images, Aidoc is the radiologist’s best friend, helping the doctor to interpret imagery faster, more accurately, and improving the imaging workflow along the way. Elad’s vision for the transformative future of AI in medicine clearly soothes Chris’s concern about managing his aging body in the years to come. ;-) Sponsors: Code-ish by Heroku https://www.heroku.com/podcasts/codeish – A podcast from the team at Heroku, exploring code, technology, tools, tips, and the life of the developer. Check out episode 98 https://www.heroku.com/podcasts/codeish/98-the-ethical-side-of-deep-fakes and episode 99 https://www.heroku.com/podcasts/codeish/99-the-technical-side-of-deep-fakes for insights on the ethical and technical sides of deep fakes. Subscribe on Apple Podcasts https://itunes.apple.com/us/podcast/code-ish/id1457879893 and Spotify https://open.spotify.com/show/5VvkHRhd9E9SQf7i9HsPMd. The Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Knowable https://www.knowable.fyi – Learn from the world’s best minds, anytime, anywhere, and at your own pace through audio. Get unlimited access to every Knowable audio course right now. Click here https://www.knowable.fyi to check it out and use code CHANGELOG for 20% off! Featuring: Elad Walach – LinkedIn https://www.linkedin.com/in/elad-walach, X https://x.com/elad_walachChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonShow Notes: Aidoc | Website https://www.aidoc.comAidoc Medical | LinkedIn https://www.linkedin.com/company/aidocAidoc | Twitter https://twitter.com/aidocmedUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  When people think about massive amounts of data, they just say, oh, I just want as many images as possible. There's a saying by my CDO that I love. He says, guys, what I want is not to have the most images. I want to actually train most effectively on the least amounts of images if I can do that. And how do you do that? By selecting the right cases for optimization. So in medicine in general, annotation costs are sky high because you need physicians for that. So you really want to pick your balance and the case selection methodology, even before saying about the biggest, which is something that I'm proud of, but he's saying like you shouldn't be proud of that. I would ideally want to have the smallest annotated imaging database, but the most effective one. So the case selection, how do you pick the right cases to optimize the learning performance as much as possible?

• 0:46Speaker 01

  Being with for change log is provided by Fastly. Learn more at fastly.com. Our feature flags are powered by Launch Darkly. Check them out at launchdarkly.com. And we're hosted on Leno cloud servers. Get $100 in hosting credit at leno.com/changelog. Hey friends, this episode of Practical AI is brought to you by Codish, a podcast from the team at Heroku that explores code, technology, tools, tips, and developer life. There's tons of great conversations on the Codish podcast, so I would encourage you to check it out and subscribe. But in particular, I wanted to bring to your attention two episodes. Episode 98 and 99, where Hulian Duque explores the ethical and technical sides of deep fakes. The rise of manipulated pictures and videos and other forms of computer generated media

• 1:32Speaker 01

  are able to cause uncertainty and doubt in what we see in here online. So how are we able to use these tools for good, if at all? Here's a sneak peek.

• 1:41Speaker 02

  Let's say we want to do a deep fake of my voice and we train the model and we have enough data and everything. This will be also able to imitate my accent, for example, like how I pronounce English and the strong pieces of my accent or is not there yet.

• 2:02Speaker 03

  It really depends. If there would be a person with similar accent on the input, then it would be fine, but it's kind of cheating. You can think it's cheating because we're reusing accent of a different person that's similar to your accent. But if it would be like an American native speaker or a British, a person with a British accent or like whatever other accent, then

• 2:27Speaker 03

  It will kind of be a mixture on the output. So we're not there yet in terms of converting accents. It's a little bit more difficult than we initially anticipated because like when we started the company, we thought it would be, you know, we'll kind of solve it in a year or something. But then it turned out that, whoa, no, we're here for much longer.

• 2:49Speaker 01

  Check these episodes out. Links are in the show notes to both episodes or head to heroku.com/podcasts to listen and subscribe. Again, check the show notes for links or go to heroku.com/podcasts.

• 3:15Speaker 04

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We're at Practical AI FM.

• 3:42Chris Benson?

  Welcome to another edition of Practical AI. We are the podcast that makes AI practical, productive, and accessible to everyone. My name is Chris Benson. I am a principal emerging technology strategist at Lockheed Martin. I can't say my own title. And unfortunately, Daniel Whitenack, my illustrious co-host, is not able to join today. We'll miss him, but I am pretty excited because we have an interesting show ahead. With me today, I have Elad Walak, who is CEO of ADOC, joining me today and looking forward to learning all about ADOC and that industry. Welcome to the show, Elad.

• 4:18Speaker 00

  Yeah, thank you, Chris, for having me.

• 4:21Chris Benson?

  So I guess to get us started, if you would kind of let us know about yourself, who you are, how you got to the point where you're at, and a couple of moments when we know you a little bit better, I'll ask about your company, but looking forward to finding out kind of how you arrived where you're at right now before we actually get to the topic at hand.

• 4:38Speaker 00

  Yeah. So again, thanks for having me. And I'm really excited about this Practical AI podcast because I come from the AI world. So my background is, I'm an Israeli. So everyone in Israel, we all serve in the military. I've had the pleasure of being in what's considered the most elite technology unit at the Ministry of Defense, a unit called Talpiot. And as part of that, I headed the AI at the Israeli Air Force.

• 5:05Chris Benson?

  Okay.

• 5:06Speaker 00

  So no healthcare, no knowledge about CP imaging or X-ray. And to be honest, I didn't even know what's the difference when, when we started back, back in the day. But what we're really passionate about is utilizing the skills we learn, which is basically applying artificial intelligence to real operational problems, solving real world issues to the healthcare space.

• 5:28Chris Benson?

  So before you even dive on, you started with something so cool that I don't want to let that get by. So when you talk about that unit with the Israeli Defense Force, could you tell us whatever you can a little bit about that? Cause I've just never had someone say that before. It intrigued me. And then I want to keep going with your own history. But you can't start with something like that without talking a little bit about, about what it does.

• 5:50Speaker 00

  Yeah. So I'll, I'll say what I can.

• 5:53Chris Benson?

  Okay. Fair enough. Say what you can. And I get that there's a point where you can't. I work for a defense contractor. So I understand that.

• 5:59Speaker 00

  Exactly. So I'll start with the, the grand vision of the Talpiot program. So in Israeli, there was a big war called the Yom Kippur War. And Israel was technologically surprised. And that was the key. So Israel decided strategically, we need to always have the technology on top. And the idea was let's take 18 year olds, right? When people are in their prime time, let's make them like the innovators of the Israeli Ministry of Defense. So let's take the brightest 30, 40 people every year, screen them very seriously, and give them a lot of benefits, but allow them, I would say, to serve their country. So when I was 18 year old, I was screened for that program. And as part of that, I

• 6:41Speaker 00

  had to sign up for a decade at the Ministry of Defense. That's a service you have to perform. But the benefit is that they basically teach you everything there is to know about the military. So we kind of visit all units and they give you this very rigorous training in both technology and leadership. And then they catapult you to leadership positions across the Ministry of Defense. And I was very fortunate to be at the Air Force in a time where AI was just starting to pick up in terms of applications, you know, computer vision, NLP, and what can be that. And it's just so much fun to look at the academy and all the innovation and trying to apply them for, you know, real projects that, you know, help save lives.

• 7:21Chris Benson?

  Sounds good. Okay. Thank you for at least kind of, kind of getting us going on that. That sounds like a pretty cool start. Pretty cool way of getting into this. So pick up now that I've rudely interrupted you. Go ahead and pick up and tell us about yourself, you know, from that point.

• 7:34Speaker 00

  Yeah. So when I finished the service after 10 years, I've made my two co-founders, which were both part of that program as well. One headed the department for the Special Forces, one had been in kind of the Israeli NSA. And we were just very passionate about the healthcare space. We, we saw that there was this massive pain point with increasing amounts of data. We can talk more about that. But we're just, we needed to know how we can use the, the tools that we got through our service to, to build a company. And we're passionate about healthcare space, basically for two big reasons. The first is that it's really a deep tech area, right? So we didn't want to compete in, you know, three guys in a garage, right? We wanted like to build something that is, you know, higher barriers,

• 8:17Speaker 00

  very defensible. And the other reason is more an emotional one. We just wanted to feel that direct connection. I think there are a lot of companies bring value, but But to be honest, I love the fact that, you know, once a day, we get a text message from a physician that tells us, hey, guys, you just helped save save somebody's life. And I think that's what we were looking for when we started the journey all those years back.

• 8:38Chris Benson?

  Okay, I guess if you could tell us a little bit about what it is in terms of the company that you've started, what the vision is for that, you know, where are you going, you know, at a high level, tell me what you're doing.

• 8:50Speaker 00

  Yeah, so we are utilizing artificial intelligence for radiology. Radiology, for those who don't know, is the science and art of interpreting medical images, CTs, X-ray, MRIs. And in recent years, we've seen a massive overflow of information. You have cheaper scanners, you have higher resolution, you have less radiation, you do this more and more. And the problem is that each and one of those images has to be interpreted by an expert physician called a radiologist.

• 9:21Chris Benson?

  Yep.

• 9:21Speaker 00

  And unfortunately, they are stagnant and have been stagnant for you in terms of the number of radiologists. And the situation today is almost absurd. Mayo Clinic did research showing that today they have about two seconds to read an image. Think about it, a human has to look at an image, massive image, we're talking big images, two seconds to detect if something is wrong with you.

• 9:41Chris Benson?

  Right.

• 9:42Speaker 00

  Obviously, it's an error prone process. Obviously, it's very difficult and obviously we're starting to get delays. And because radiology is so critical to all of healthcare, you know, now you do, you know, even for STO, you get radiology, right? So every delay, every error has massive downstream implications. And we see it impacts, you know, stay and errors and patient lives at the end of the day. So yeah, pause here if, see if that makes sense.

• 10:05Chris Benson?

  It does make sense. It makes perfect sense and it's it's interesting. Without specifics, I have heard about AI being used to do radiology in terms of the interpretation. You said the one thing, though, that I'm just curious about. It's a totally a side note, but I can't help but ask. You mentioned that the radiologist has like a couple of seconds to make that. And is it just too costly, you know, given that they're doing rounds and they have many patients or something to spend the time? But, you know, they're just expected to obviously see it quickly. But I'm just kind of curious what's driving the metric about how little time they have to make that? Given the fact that that human in the loop right there as they're looking at that image has a huge downstream, you know, the cost to getting it wrong.

• 10:48Chris Benson?

  Is it just too few radiologists for too many patients in general?

• 10:52Speaker 00

  Exactly. It's it's kind of supply of demand of images. And another big factor that comes in is more data per image. So if images were small, so like say a CT exam, in the past, they had, you know, fifty images, something like that.

• 11:05Chris Benson?

  Right.

• 11:06Speaker 00

  Today, we're talking about thousands for a single patient, for a single exam. Uh, so that obviously factors in as well, right?

• 11:12Chris Benson?

  I don't think I realized that. That's I just learned something. I was thinking, you know, when when I'm in and they throw two or three up, you know, and all that, but they they have a whole lot more than that to deal with today is what you're telling me.

• 11:22Speaker 00

  Yeah, the analogy I I heard from one of our customers, I love that. It's like, uh, phone cameras, right? So first of all, you have much higher resolution. Right? And you have all the different filters now, and you need to look at all that. And the other thing, you know, everybody's doing imaging today, you know, you have selfies all around. So CP images are like, you know, are like phone cameras. Now everybody's doing selfies and now you have a lot more resolution.

• 11:45Chris Benson?

  Got you. It sounds like a good problem to scale with automation. That's that's what I have to say about that.

• 11:50Speaker 00

  Yes, yes, or augmentation.

• 11:53Chris Benson?

  Fair enough. So why don't you tell us a little bit about that augmentation? What what is the business model that you're addressing here? How do you fit into that process?

• 12:00Speaker 00

  So basically, our AI is a safety net to radiologists. So the AI constantly runs in the background. Nobody needs to activate it, uh, or anything like that. And searches for certain type of findings that require prioritization. So the big problem we tried to solve, or at least the area we started, we started with, is critical care, because you have so many patients and, you know, the big, the big thing, if you have more data and less people, is that those patients can get treated much later. Uh, and our AI runs in the background and searches for just critical findings. Stroke, bleed, pulmonary embolism, spine fracture. And if we find something, we basically alert the whole care team, the radiologist, to their existence.

• 12:42Speaker 00

  To the presence of a critical finding and this can shorten the time to treatment significantly. We're talking about, you know, even in ED settings where time is really fast. We're talking about an hour or two, but for elective care, it could be sometimes, you know, days of reduction in terms of time to treatment.

• 12:58Chris Benson?

  So what, what does that look like? What, what is the relationship? So you're doing the interpretation at this point with the AI. What does that look like?

• 13:07Speaker 00

  We are basically flagging cases for the radiologist. Still the human is doing the interpretation. We're just helping him screen through the data, see where areas the radiologist should focus on first. And let me give you an example of what it means without us or with us. So let's say you have, one example is the emergency department. So typical emergency department, especially if night is very busy, you have dozens of patients coming in and you really don't know which of them are really critical or not. So we have a patient, let's say fell down a few stairs, coming to the ED. They're even doing CT imaging for that patient, trying to see if that patient has a brain bleed. But unfortunately today, just because of the overload, that patient could wait hours before anybody interprets the exam.

• 13:51Speaker 00

  That's the key bottleneck, looking at the exam and seeing if there's a bleed. So today it could wait, you know, a couple of hours and we're talking about very time sensitive pathology, right? Time is brain, you might, you know, we all heard that. So we can basically reduce that time because we say, we look at the image, we say, hey, there is a brain bleed here, let's alert the radiologist. You should definitely take a look at that patient, even though that patient is 15 year list, that should be number one. Take a look right now and get the patient the treatment they need.

• 14:24Chris Benson?

  It's, it's interesting, I just to, to throw in, as you, as you're describing this, I have a very specific context that I'm applying it. I have a stepdaughter who is a fourth year med school student and she is right now working in an emergency room at a hospital near us and that, that's what she does. And so she relays a lot of her experience, you know, because I have no background in that, that she likes to tell me these things. So it's very interesting. I'm envisioning how this fits into her world as she's been describing. To me over time. And she has talked about the fact that getting the image results can be a big delay in the ER when they're trying to move quickly, especially on a like a Saturday night or something when things are coming in at a very fast pace. So I guess you so you

• 15:07Chris Benson?

  still have the radiologist directly involved in the process.

• 15:10Speaker 00

  Yeah.

• 15:10Chris Benson?

  Is it essentially just speeding up their process to draw their attention directly to what they need to focus on?

• 15:17Speaker 00

  Yeah.

• 15:17Chris Benson?

  Speed that up?

• 15:18Speaker 00

  Exactly.

• 15:19Chris Benson?

  Okay, but you're anticipating, at least at this point in time, the radiologist is always the key to the picture still along with the technology that you're providing. The two are partnered essentially.

• 15:30Speaker 00

  Exactly. Exactly. And I think that's key, especially this day and age. And right now, I think radiologists have to be involved. It's A, because it's really necessary, because at the end of the day, I wouldn't want to have AI interpreting my my full images. And you know, I'll add one more thought to that. The AI is really good. I trained him to be the best brainly detector I can. But we have thousands of diseases that we can have. And where radiologists, where humans are really good at, is that comprehensive picture, right? They know how to aggregate information. They know how to look to this. Something here looks wrong, right? That's, that's where humans are very good at. So we want to, we're not replacing them in any way, shape or form. We're just providing this augmentation. So they could

• 16:15Speaker 00

  focus, you know, if there is a critical finding, you know, let's focus that on that right away. And let's make sure that we treat the patient very fast for that downstream impact.

• 16:35Speaker 04

  We deserve a better internet and the brave team has the recipe for bringing it to us. Start with Google Chrome, keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits. We don't need them. Mix in ad and tracker blocking by default. Quick access to the Tor network for true private browsing and an opt-in reward system so you can get paid to view privacy respecting ads. Then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give tipping a try on changelog.com.

• 17:17Chris Benson?

  Okay, so that's really fascinating. We've kind of talked about how your technology fits in to that, in this case, the ER with the radiologist having to interpret that. Before we dive into the technology side, could you dive us kind of specifically through the process, maybe describe a little bit about the radiologist, we've talked about the radiologist trying to interpret and having to do that very, very quickly. How is the radiologist actually interfacing with your technology? What does that feel like to the radiologist? And how does that change their process? If you could describe that. And then after that, I'd love to dive into the specifics of the AI, whatever you're willing to share, to understand some of the cool underlying models that you're using to make all this work.

• 17:59Speaker 00

  Yeah, so I'll start with an anecdote. When we started the company, we had a conversation with a radiologist, a very known neuroradiologist. And we talked with him about the workflow. And he told us, like, guys, look, radiologists are so bombarded with information. You have so little screen real estate. You really can't make us press buttons and click things. It has to be seamless. So, back to it all, again, it's so great insight, so no more than one button click for the use of the product. He told me, no, I don't think you've got it. No button clicks for the use of the product. It has to be fully seamless, automatically appear whenever I want it.

• 18:43Speaker 00

  And that was kind of the challenge we faced when we started the company, because they really are, imagine that environment of the person that has 50 patients waiting for critical, like getting phone calls, having to do a lot of these. You really can't disrupt anything. So what we wanted to create is a very seamless workflow for them. So it would appear automatically in their own systems. They're using a worklist, so it would just appear as a flag in their worklist, just like a small pop-up. Things that are really easy for them.

• 19:13Speaker 00

  The challenge was, however, you know, we thought, well, great, great idea. But the problem was that in healthcare, there are so many legacy systems. Right? To imagine a hospital using systems, sometimes, you know, 10 years old systems.

• 19:26Chris Benson?

  I noticed that every time I do have to go to, you know, doctor's office or hospital or something, it feels like a step back in time if you're into technology.

• 19:34Speaker 00

  Oh, exactly, exactly. And trust me, it's really no fun to integrate into all of those legacy.

• 19:40Chris Benson?

  I could imagine.

• 19:42Speaker 00

  So that's basically the challenge we had and a big focus for the company is actually, you know, we're an AI company, but half of my engineering is on creating a very seamless workflow. And what we did, we basically created this very cool integration engine that allows to integrate directly into those different systems. So whenever radiology gets up in the morning, he starts seeing our results popping up within their own workflow, the same system we're using. So if they were using a specific work list in a certain viewer, uh, we'll just start popping up within their existing systems to kind of highlight the findings. So what they do is they just do whatever they were used to doing to the state. So opening a work list and then going, guys, we're just small flag appearing there saying, hey, you know,

• 20:27Speaker 00

  this is a critical case, small, small like priority flag, look at that first, very, very seamless.

• 20:32Chris Benson?

  To accomplish that, so it sounds like you're integrating not only at a software level, but also at a, at a hardware level to some degree. Or am I misunderstanding that? Cause you have this hot new AI that you're trying to integrate with this 10, 15 year old system that the doctors are used to using. How does that look? How does that merge? You're talking about kind of moving into the workflow, but how do you affect that?

• 20:54Speaker 00

  Exactly. So the key is, it is, it's still in the software level, but it's in a lower level, you know, you have to really understand the different communication protocols and how they communicate between the systems. And there are some patterns that emerge once you kind of get understanding that. But if you do it naively and just try to integrate, you know, vendor by vendor, you really, you know, you can't, it is just impossible. It's, it's finding those ways that are very scalable and that work very well across all systems.

• 21:20Chris Benson?

  So you're basically kind of using networking stacks and software to try to integrate as best you can with these older systems so that you get the most seamless workflow you can.

• 21:29Speaker 00

  Exactly.

• 21:30Chris Benson?

  You're talking about really old stuff blending with very, very new, modern stuff to affect a user experience that's adding value to that. How does that change for the radiologist? They now have presumably, you know, they have the image that they would have looked at anyway. What are you doing in that workflow that changes it for the radiologist in terms of how their, I'm using the word use case, I guess, but how their workflow seems from their perspective? What's different for them?

• 21:57Speaker 00

  From their perspective, it's almost unnoticed, I would say. And that I think that's the beauty of it, right? Don't want to disrupt too much. We want to just create value in a seamless way. So we actually had a customer that they didn't even show the alerts. They configured that in a way that automatically updates their prioritization stack. So they just use this as a feed and they decided just to change the prioritization. But typically for a customer, they get up in the morning, you know, they come to the regular work list where they have participations, they just see flags of our results. And if they hover on a flag, so they just in their normal workplace, they see like this orange ADOC icons.

• 22:32Chris Benson?

  Yeah.

• 22:33Speaker 00

  If they hover on that, we could have showed them a preview image of what the AI detected and then they can basically choose that case. If they see the, you know, look at a preview image, looks like a prioritized case, then they open the case and then diagnose it.

• 22:47Chris Benson?

  Gotcha. So to turn it a little bit away from the practitioner using it now and talking a little bit about the underlying technology, I suspect we have a lot of people in the audience that are just craving to hear about what kind of, you know, which models you use or their CNNs, is there NLP? There's probably all of this, I'm guessing.

• 23:04Speaker 00

  Yeah.

• 23:04Chris Benson?

  If you could take us through a little bit of kind of high level architecture about what you're using and why, and maybe some of the choices that you had to make along the way. I realize that you have trade secrets and stuff, but I'm hoping that you can tell us as much as you can without giving us, giving the secret sauce completely away, maybe a tiny taste of secret sauce.

• 23:24Speaker 00

  Yeah. So the guiding principle for us was that, you know,

• 23:30Speaker 00

  Theoretically, AI, you could build any AI, just any fully connected layer to detect whatever you want, right? Sure. But that doesn't really work that way. You know, if you want to create an effective system, you have to create a, you know, you have to steer the learning process in a way that is most efficient to learning that specific domain. So the question we ask ourselves is how to train AI most effectively to solve the specific problems of medical imaging? And what are those? I'll just give a few examples and then I can talk about what elements are we using? So the first issue that you get when you go to medical imaging is data scarcity.

• 24:09Speaker 00

  And it's not one of the, so you know, you think about ImageNet, you know, you think millions of images annotated, very easy, you know, you do a mechanical torque or something, you can outsource people flagging. It's not the same for radiology. We have what we believe is the biggest annotated database in the world right now and and we have hundreds of thousands of images, like that, that's basically, we're talking about roughly order of magnitude of hundreds of thousands, a million of studies, each contains hundreds of images, right? But so it's tens of millions of slices of images, but a lot of the imaging is 3D stacked, so you have one patient with 500 images. So that's one big challenge that we had to tackle. The second is class imbalance, right? So you have, look at the

• 24:54Speaker 00

  small patient image is like a million pixels, something like that. And a disease can be on, you know, a three by three by three, we're talking eight pixels out of a million. And that's a positive example, right? If I'm talking air force lingo, I would say so, if I'm creating an algorithm to detect, you know, bunkers, it's like I would give you map of the world and tell me, hey, there's a bunker in here, right? Somewhere. Go and have fun. So it's really trying to handle this really class imbalance where you have all these normal areas. That's like a big other challenge. And the third challenge for us was that humans have a lot more information oftentimes that the system is accessible to, and I'm talking about the clinical context.

• 25:37Speaker 00

  So radiologists, they cheat in the sense that they look at that historical exam, they look at patient data, and it helps them kind of focus on what they really want to do. And that's another thing that we really wanted to focus on. And those are just three examples of how we basically built the underlying architecture to support those pain points. So basically what we're using, it's not totally different. It's convolutional neural network on the medical imaging. But what we really innovated is on the training process to handle the class imbalance on some of the specific layers in the network to try and replicate as much as possible how a physician would look at the image. And we added a lot of clinical information from other imaging, so from other sources,

• 26:22Speaker 00

  other clinical sources for the neural network to have as much information as possible when doing, and doing it. So before I dive any deeper, I want to pause here and see if you have any questions on that.

• 26:34Chris Benson?

  No, that's very interesting. I'm just curious, when you're looking at the world of CNN architectures out there, what are some of the architectures that you like in general? And I'm not asking how you're using them in specific ways, but what is interesting? Because, and I ask with a reason, Daniel and I often talk about the fact that a couple of years ago, you know, we were seeing all the evolution of CNNs, but we've heard more recently more about NLP and, you know, transformers over the last year or two. And if you're looking at it from our perspective as doing interviews, we're hearing more about this other side of AI. I love the fact that you're taking us back into the CNNs, and we haven't really heard how they've evolved as much from guests lately. I'd love to find out, like, what do you think is fantastic out there in the CNN world

• 27:18Chris Benson?

  in terms of useful stuff? Are there things that you think of are old and you wouldn't mess with them at this point, having used them a lot in the last couple of years? Just wherever your perspective is.

• 27:28Speaker 00

  Yeah, it's a great question, and I would say that It's funny when, you know, when we submit to the FDA even, so every one of our products has to be clear about the FDA.

• 27:38Chris Benson?

  Yep.

• 27:39Speaker 00

  They ask us what architectures we're using as well, so it's important to them. And I always found it kind of, I wouldn't say funny, but, but when, when we look at, at our domain, we typically don't just, you know, download architecture from, you know, I'm not downloading, you know, a Google Net or ResNet or something and, and Google Meta as is, we're taking elements and we're experimenting with them, but it's relatively like, I knew, I can't say it's even similar to anything, you know, anything specific, because it's just so adapted to our domain. You know, we have 3D data, we have the, the, the, the blank information. Uh, we do love attention, that's, I think is a big one for us. Uh, again, because of the class imbalance and all that, so that's, that's a big thing.

• 28:20Speaker 00

  And the other thing we really like is, uh, is tile transfer networks for data augmentation. So anything around data augmentation or data scarce elements, I love those as well. So it's like, if I can pick like two elements that I, I love and we use often, I would say those, but it's really hard for me to say like, this is an architecture I would, I would just use because it's so, so different. Like they pick a few elements and you really experiment a lot.

• 28:44Chris Benson?

  Gotcha. And that was a great explanation for kind of how you're seeing that. I, I like that. It really sounds like you're finding the competitive advantage there in terms of customizing a number of different things in a way that's unique to your business and which, you mentioned barriers to entry earlier, and I would imagine that creates substantial barriers to entry. You also talked about the world's largest annotated data set for this. Are you able to tell us kind of how you developed that, where that came from? You know, at what point did you see the need for that and how did you approach that? And you could take it from the perspective of there are other people in other industries and other businesses out there that are trying to figure out how to solve it for their business. They know that they need the data, they're not sure how to do it.

• 29:25Chris Benson?

  I'd love to hear how you got to that point where you could affect the solution that you were striving for.

• 29:33Speaker 00

  Yeah, absolutely. I want to add one more point regarding the last point we discussed in terms of customizing the networks. There is a cost to it, by the way, I just want to make sure that I'm sharing that transparently and that if you are experimenting, you need to have a very flexible experimentation infrastructure. And we've actually invested a lot in developing our cloud experimentation, but even like the small challenges of how do you run a thousand experiments a day and track those? And you do it on spot instead and recover experiments if they fail? Like, that was a massive undertaking. Right now, I believe we're in a much better spot than we were and we can run thousands of experiments every day, which is fun as hell, you know, just have an idea and go crazy with it.

• 30:17Speaker 00

  Even me as a CEO, I can, you know, just ask them to implement like some, hey, you know, we did a new ABC, let's try. But it's massive. It's one of those challenges we don't often hear about is the engineering of doing very efficient, scalable experimentation. Regarding the annotation, so as you can imagine, a lot of it is secret sauce, they'll try and get the high-level elements. So I want to share two things. The first is that when people think about massive amounts of data, they just say, oh, I just want as many images as possible. There's a saying by my CDO that I love, he says like, guys, what I want is not to have the most images, I want to actually train most effectively on the least amount of images, if I can do that. So, and how do you do that?

• 31:00Speaker 00

  By selecting the right cases for optimization. So in medicine in general, annotation costs are sky-high because you need physicians for that. So you really want to pick your balance. And the case selection methodology, even before saying about the biggest, which is something that, no, I'm proud of, but he's saying like, you shouldn't be proud of that. I would ideally want to have the smallest annotated imaging database, but the most effective one. So the case selection, how do you pick the right cases to optimize the learning performance as much as possible? And again, let's take, brain bleed, I think is a great, great example. Am I taking a thousand pictures of patients with massive brain bleed, which are obvious? Or am I taking maybe the only 50, but those 50 are those with like subtle brain bleeds

• 31:43Speaker 00

  that you can really train and improve the system with? So I would tackle that challenge. If more, then I would tackle the challenge of annotating very efficiently. And regarding efficient annotation, I would just say that the key we found is having everything very tight. So is outsource as much as possible, that the connectivity between the data teams, and the AI teams, and the annotators, and like the fact that we control all the bits and pieces, the annotation, selection, everything, like everyone knows their parts and they can communicate, that was very efficient. I heard about Google doing something for, I believe it was for Dermatology Yads, for eye images, and they had like a thousand-page manual for their annotators.

• 32:26Speaker 00

  Okay, that for me is ineffective, right? And the problem was the annotators were outsourced, so just like going through all the information and helping them inefficient. What we believe is key is having like very high interactivity, a lot of feedback loops, a lot of, you know, AI people, "I want this," and data people, "Hey, this is tough for me to annotate. Do you really need that?" And all of those small bits and pieces, we find, really help accelerate the process. And I won't talk too much about how exactly we solve it, but I would say that's, that's a big one.

• 33:04Speaker 04

  Have you heard about Knowable? It is an awesome new platform for learning from the world's best minds, anytime, anywhere, at your own pace, through audio. Learn about the performance benefits of a plant-based lifestyle from NBA All-Star Chris Paul, or how to launch a startup from Reddit co-founder Alexis Ohanian. There's even a 10-lesson course from astronaut Scott Kelly. Here's a sneak peek.

• 33:29Speaker 06

  We learned a lot up there, but what can you learn from a life in space? The answers might surprise you. In this Knowable course, I want to share some of the things I've learned that you might not expect. Lessons about leadership on a dark night on an aircraft carrier in the middle of a churning sea? Lessons about the fear you feel with seven million pounds of thrust exploding underneath you?

• 33:54Speaker 06

  And most of all, there's an idea out there that astronauts are always perfect. Failure is not an option, right? That's why I want to take you through some of my life experiences to show you how that's just not true. I believe everyday, regular, human failure, if we handle it right, can be one of our greatest opportunities to learn, grow, and succeed.

• 34:15Speaker 04

  Knowable is accessible on your phone and on the web, and each audio course is broken out into individual lessons, usually around 15 minutes long. As a Change Log listener, you can get an annual membership to Knowable for 20% off. Get unlimited access to every Knowable audio course right now. Just download the Knowable app or visit knowable.fyi and use code CHANGELOG for that 20% discount. We put a link in your show notes for easy click-ins. Check out Knowable today and start learning from hundreds of top experts from around the world. Once again, that's knowable.fyi, code CHANGELOG.

• 35:05Chris Benson?

  So you mentioned something a moment ago that I'd like to go back to, and that's you talked about kind of the ability to do your experiments. You referenced your own workflow internally, the infrastructure that you had, and I'd really love to hear kind of how you've approached that because, you know, people tend to talk about the AI. It's, you know, it's the sexy thing to talk about and what you're doing, but they miss the fact that it has to fit into a larger software process that can take you all the way from experimentation and figuring out where you're going, and then integration into your other software that it has to, you know, because it's a model has to sit inside a larger, a larger set of components that are software and eventually has to be

• 35:49Chris Benson?

  deployed out to be usable in the real world. And so how have you approached that? You know, there are everything from cloud to buying your own equipment to being, you know, which workflow, which vendor, which open source you're choosing. How do you approach these challenges, these considerations that you have to make, considering that each one has a fairly substantial impact if you, if you don't get it right?

• 36:14Speaker 00

  Yeah, so I think the first big element that is worth to note is team structure. I think that was actually the first consideration to take into account and our head of AI, Yidan, did an amazing job. He actually, you know, interviewed a few companies, how they build it, you know, how Google does it, how other people do it. And we figured out a structure where the AI algorithm engineers are actually a small part of the team. And we have, so I'll go through the evolution we had. So we started with having algorithm engineers just focused on algorithms, but they weren't really focused on deploying and anything like that. We then figured out that was a mistake. One, people that can take the product end to end and take full ownership and not like do cross because things get left and it's really inefficient.

• 36:59Speaker 00

  So we brought people in that are both very sharp algorithmically, but you know, know how to project manage, know how to deploy, know the software side. They're not afraid, you know, to have their hands dirty with, you know, implementation. Then we kind of started figuring out ways, but that's not also not very scalable. We need people that are even better at software and infrastructure. So we created like a whole AI infrastructure team. We have a ratio of about two to one. So two infrastructure folks for every one algorithm engineer. And then they started working on the infrastructure and it's exactly the thing you mentioned. There are so many of those challenges. Do you scale? How do you scale? This is the cloud experimentation, experimentation infrastructure and deployment infrastructure and monitoring infrastructure. Also very important how do you monitor accuracy in production?

• 37:42Speaker 00

  And then, then we, I would say that's kind of how, how the team is built today and we're scaling it the same way. We were always big fans of the cloud. We started on prem, which was, I would say, I don't know if it was a mistake back then. We didn't have as many experiments, but for now it's for sure. We can't do it on prem because it's just so, so much experiments. So if you really want to scale fast. You really have to solve the cloud. And it's even like stupid quote unquote challenges. Like one of our challenges that we ran out of spotting sensors for GPUs. That's a challenge apparently. And you know, solving that and how do you like make sure that you work on multiple regions and how can you potentially transfer experiments between?

• 38:26Speaker 00

  So all that, all the small things in making it tight. I think it's just like respecting the engineering side. That was a big, big thing that I learned. Just make sure that you know, that's probably a bigger part of making it work in practice versus like science.

• 38:40Chris Benson?

  Gotcha. There's so many different ecosystems. Each of the cloud providers has their own ecosystem. Nvidia with GPUs has their own ecosystem and workflow. There are a number of tools that, that are cross ecosystem in terms of, you know, like TensorFlow will work in all these different places. As well as, as PyTorch and others. How do you buy in to, or how do you evaluate a particular ecosystem or, or a combination of them to make that bet to go down that road? Any thoughts, any guidance that you can provide us on that?

• 39:11Speaker 00

  For sure. First of all, we took the path of AWS and we're very happy with it.

• 39:16Chris Benson?

  Sure.

• 39:16Speaker 00

  And the main reason we chose AWS was that it is, in medicine, we, we say standard of care. So that's why we'd say the, the leader probably at this point in time still. And that makes it very easy to hire, you know, DevOps people that know how to do it. Like all the new tools go on that platform. So that was for us a big consideration. And the other is just your cost. So when I, you know, when we pick, those are the two biggest ones, like innovation speed and, and cost and can we create the architecture? The other big element in terms of consideration is, is who is available and can help you work through those challenges. So again, I think there are several companies that were really accessible, but we really enjoyed the interaction with AWS.

• 39:59Speaker 00

  We had all the technical resources to help us think through the challenges. And I think what, what I learned as, you know, as someone that Was technical but not technical on the cloud side. I learned there are so many different elements, so many ways to create your architecture that you really want to have a partner to help you think through those challenges.

• 40:17Chris Benson?

  So that was super cool about how you're trying to evaluate the software. I love hearing how people make those hard choices. And clearly you have a kind of an approach to how you're evaluating the possibilities out there that has worked very well for your organization. So that really, as we look forward, you know, there are so many things I want to ask you and kind of wherever your thoughts are taking you on this. We have the growing world of AI with all the tools on that. You're in an industry that we know is in the process of being revolutionized and probably in the very early stages of this technology. And there are so many choices that you have to make and both as a leader of your organization

• 40:59Chris Benson?

  in terms of where you're taking the group and how to evaluate a really complex landscape of what AI is in the world that we're at and all the tooling. Where do you see the world going at this point? You know, if we're looking over the next few years in the relative short term and maybe even farther when we're talking about where we might arrive. And, you know, obviously nobody can read the future accurately, but I'd love to hear kind of where you think the world might be going down the road.

• 41:27Speaker 00

  Yeah, it's a fascinating topic and we have to, because we're in such a fast moving industry, we really have to think about it all the time. And I'll start with taking us back into the evolution of AI this past two years, just to understand what are we even talking about in the speed. AI in healthcare and in medicine has been in this frozen period for a few years. So if you would go three years back, not talking a decade, three years back, really not know, almost no big clearances, you know, want to deployment. Two years back, we're talking about 0.01 market share of AI. Like we're talking two years ago. And today, you know, only us, we have hundreds of hospitals and, you know, the leading names

• 42:11Speaker 00

  that, you know, you would know are already utilizing AI. So we're talking about massive adoption in two years. And, you know, thinking ahead for the next few years, I think we're going to see really, probably everywhere, you're going to see one AI or another, which I find fascinating. I love being in an industry that is so, you know, fast moving, kind of like we broke that barrier and then it's starting to evolve. But I think that right now AI is in that really first age of healthcare AI. And AI today is doing, I would call it workflow augmentation. So AI right now is only helping to the humans where they either way would do, but better. So in my case, it helps radiologists detect, you know, pulmonary embolism faster.

• 42:56Speaker 00

  Okay, so that's assistance. But if you ask me three, four years from now, AI is going to start going into the predictive space. And what do we mean by that? I mean, seeing things that humans cannot see with their naked eye, right? So if today you can detect stroke at point A, you know, if AI could detect it in A minus two, Alzheimer, early detection of cancer. And I think that's where really the promise of AI becomes really exciting, where you're creating new diagnostic capabilities. It's like initially, you need to do a CT to even look at the bodies and understand. This stage, right now, we're talking about the next phase of that, seeing, you know, patterns that no one of us could predict. And I think that's the exciting next evolution for AI.

• 43:40Speaker 00

  I think it's very challenging for that. You need very high levels of trust. You need to have mountains of data. We're talking about, we're just more than there is today. But I think once you get that, that's going to be, that's the holy grail we're all aiming for. And I think it's not that far away. We're talking three to four years where I'm going to start seeing things and really impacting in a way that is really impossible by humans today.

• 44:03Chris Benson?

  That's really interesting to me. I'm 50. I don't mind telling people that. I'm kind of getting to that point where I'm thinking as a guy who doesn't care much about medicine in terms of my own body so much, I'm just doing my thing, getting out there, exercising. But, and I'm starting to think now about, whoo, I'm at an age where this is starting to matter.

• 44:22Speaker 00

  Yeah.

• 44:22Chris Benson?

  And so it's really good to hear that as I move farther into my 50s, that those kind of new diagnostics are coming out because I am, I'm counting on you to keep me healthy, buddy. So anyway, thank you very, very much for coming on to the show. It's been a fascinating conversation. And like I said, for my own selfish purpose in terms of keeping myself healthy and also in the perspective of hearing these ER stories and hearing how you're changing that whole picture and changing how fast people can get care. I appreciate your work and what your organization does. And thanks for coming on the show.

• 44:57Speaker 00

  Thank you so much.

• 45:02Speaker 04

  Thank you for listening to Practical AI. If this is your first time, make sure you subscribe so you don't miss a thing. Head to practicalai.fm to subscribe or find us in Apple Podcasts, Spotify, or wherever you listen to podcasts. And if you get value from the show, please do share it with a friend or a colleague. We appreciate you spreading the word. Practical AI is hosted by Daniel Whiteneck and Chris Benson. It's produced by Jared Santo and our music is provided by Breakmaster Cylinder. We are brought to you by some awesome sponsors. Shout out to Fastly, Linode, and Launch Darkly. That is our show. We hope you enjoyed it, and we'll talk to you again next week.