UBER and Intel’s Machine Learning platforms

We recently met up with Cormac Brick (Intel) and Mike Del Balso (Uber) at O’Reilly AI in SF. As the director of machine intelligence in Intel’s Movidius group, Cormac is an expert in porting deep learning models to all sorts of embedded devices (cameras, robots, drones, etc.). He helped us understand some of the techniques for developing portable networks to maximize performance on different compute architectures. In our discussion with Mike, we talked about the ins and outs of Michelangelo, Uber’s machine learning platform, which he manages. He also described why it was necessary for Uber to build out a machine learning platform and some of the new features they are exploring. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean is simplicity at scale. Whether your business is running one virtual machine or ten thousand, DigitalOcean gets out of your way so your team can build, deploy, and scale faster and more efficiently. New accounts get $100 in credit to use in your first 60 days. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We catch our errors before our users do because of Rollbar. Resolve errors in minutes, and deploy your code with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Linode https://linode.com/changelog – Our cloud server of choice. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2018. Start your server - head to linode.com/changelog https://linode.com/changelogFeaturing: Cormac Brick – Website https://www.linkedin.com/in/cbrick/, X https://x.com/cormacbMike Del Balso – Website https://www.linkedin.com/in/michaeldelbalso/Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Intel’s Movidius Group https://www.movidius.com/OpenVINO Toolkit https://software.intel.com/en-us/openvino-toolkitngraph https://github.com/NervanaSystems/ngraphUber’s Michelangelo https://eng.uber.com/michelangelo/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

• 0:00Speaker 00

  Beta for change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at change law because of roll bar. Check them out at rollbar.com and we're hosted on Lino servers. Head to linode.com slash change log. This episode is brought to you by Digital Ocean. They now have CPU optimized droplets with dedicated hyper threads from best in class. Intel CPUs for all your machine learning and batch processing needs. You can easily spin up their one click machine learning and AI application image. This gives you immediate access to Python three, R, Jupiter notebook, TensorFlow, scikit and PyTorch. Use our special link to get a hundred dollar credit for Digital Ocean and try it today for free.

• 0:45Speaker 00

  At the do.co slash change log. Once again, do.co slash change log.

• 1:03Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence, practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show at change.com slash community, follow us on Twitter, we're at Practical AI FM. And now into the show.

• 1:29Speaker 01

  McCormick, thanks for joining me here at O'Reilly AI. It's great to have the chance to talk to you. I know you just got out of your talk a little bit earlier. You talked about portability and performance and embedded deep learning. Can we have both? So I want to dig into that a little bit more later, but first I'd love to hear. I know you work and help lead the mo videos group at Intel. And I'd love for you to just kind of let the audience know what mo videos is. If they haven't heard about it, what you're doing and what you're working on now.

• 1:55Speaker 02

  Yeah, hey, thanks, Daniel. And yeah, good to talk. I guess, yeah, my name is Cormac Brick. I lead kind of VPU architecture at Mavidius as part of Intel. VPU for us is a kind of visual processing unit, and that's the kind of the key engine we have in our kind of product line. So yeah, I kind of lead that architecture. And at Mavidius, we're very passionate about, you know, machine learning and computer vision at the edge. This is something we've been at for a long time, going back, you know, kind of five, six years, even before we were part of Intel. And we have kind of multiple products now in the field. And yeah, we've learnt a lot as of course, you know, as a result of all of that interaction with customers over the years. And yeah, the goal of, you know, the talk this morning was to was to really kind of reflect back some of that knowledge, as in,

• 2:38Speaker 02

  what have we learned about, you know, tuning neural networks from for for embedded silicon? And then also tuning embedded silicon for neural networks, right? To kind of just reflect back what some of the realities are when you go to take a network to the edge. What's kind of really required to make that run really, really well?

• 2:57Speaker 01

  Awesome. Yeah. So just to kind of dig into that a little bit deeper.

• 3:02Speaker 01

  When you're talking about, you know, customers that are tuning neural networks for the edge on things like VPU's, which you mentioned. What are some of the kind of customer use cases around this and people that have found a lot of value in going down that road?

• 3:15Speaker 02

  Yeah, sure. So we at Mavidius, like we have customers who are engaged heavily in things like digital security and kind of smart city type use cases, where really making more intelligent cameras. That's that's one big use case. We've also shipped a lot of products on drones. That's another use case. As well as a lot of things around, you know, robotics and smart devices and camera devices as well. So, you know, there's things like the Google Clips product that's on the market now that uses our kind of myriad two silicon. A lot of the DGI drones have used the myriad two silicon as well. And they have things like you can wave at the drone using your hands to control it and then put out your palm and the drone can land on the palm of your hand.

• 3:59Speaker 02

  So really, really compelling use cases that have been enabled through our silicon, through the use of, I guess, both vision and AI kind of working hand in hand.

• 4:07Speaker 01

  Awesome, yeah, and just to kind of confirm that, I was actually at GopherCon last week, and one of the keynotes, I think on the second day or something, they used a drone with a Myriad chip in it to do some facial recognition and all of that. It was some cool stuff. So let's kind of dive into a little bit more about what you talked about. Is there, in these types of use cases where you're wanting to run your neural network in a drone or in a camera or whatever it is, explain a little bit the tension between kind of portability and performance that we've seen in the past and the state of it now?

• 4:39Speaker 02

  Yeah, sure, so I guess what we've seen is a lot of, you know, if you go on Archive or if you go to NIPS or iCloud or these sort of conferences or CVPR, like leading academic vision conferences, what kind of find there that people are, there's a lot of work being done to kind of optimize neural networks for, for things like kind of ImageNet or MS COCO or kind of academic data sets. And that's awesome in terms of pushing the envelope of the field and, you know, advancing the science and it's moving super fast, right? So then, you know, typically when embedded engineers will start off a problem, they've access to that sort of research and these sort of models, and then they kind of want to do something that's going to work for them and their device, right? And one of the things that they would find is a lot of the models that are available out there

• 5:22Speaker 02

  were tuned on ImageNet, which is great at recognizing, you know, a thousand classes of images, you know, and it can differentiate one sort of whale from a different type of porpoise and this sort of stuff, right? Very, very fine-grained classification on specific tasks. Yeah, not so much in the real world, right? We have different problems to solve. So then in the real world, we may care about, hey, like my robot wants to be able to recognize, you know, 100 common objects found in the home or with this sort of, in this security camera, we want to be able to recognize these different types of objects that are happening. Yeah, so different problems. And often those problems are simpler than the 1000-class problem of an ImageNet. So one of the things we were talking about this morning is using techniques like model pruning and sparsification to,

• 6:07Speaker 02

  you know, if you're doing what we would call domain transfers, you go from your 1000-class problem, you know, let's say if you were taking ResNet-50, and you're now retraining that for your home robot, which wants to recognize 100 Hundred images, you'll find that you can get away with a much simpler network with a less representational capacity to solve that hundred image problem than the one you started off in the thousand image problem. So we were sharing some results and some techniques specifically around channel pruning, which is very, very powerful technique. When you are doing domain transfer to a simpler problem domain and also looking at techniques like sparsification, which is introducing more zeros into a neural network. Because that's great in terms of on platforms that support, you know, memory compression of neural network models.

• 6:49Speaker 02

  It'll enable those models to run much more faster in bandwidth limited devices, such as those typically found on the edge.

• 6:58Speaker 01

  Awesome. So in terms of like, let's say that I'm working on, you know, I'm working on one of these robotics problems or whatever it is. And I'm using a neural network and I want to pursue some of these methods to kind of prune it down or optimize it for that setting or for that architecture. What's kind of the process and the barriers that I would face as of now going into that? And what's kind of the state of the usability of these tools and that sort of thing?

• 7:24Speaker 02

  That's a great question. Cause for sure, we were presenting a lot of work this morning saying, hey, you know, we're able to take a network and do this sort of pruning and quantization and sparsification. And then go from eight bit weights to four bit weights and this sort of stuff. But, you know, straight up today, pretty non-trivial to repeat the results that we were kind of showing this morning, right? To bridge that gap, you know, working, we now work in Intel as part of the AI products group. As part of the Intel AI products group, there is an open source project called Distiller. It's one of the resources listed in my slides, I think on the final slide, and I believe they'll get posted to O'Reilly at some point.

• 7:59Speaker 01

  Yeah, we'll put them in the show links here as well.

• 8:01Speaker 02

  So yeah, there's a link to something on GitHub called Distiller. And there, one of the things we're doing is, you know, if you went back maybe kind of 12 months ago, you'd have found, oh, like this is an awesome quantization technique that somebody published. You know, some grad student kind of put this, you know, published a PyTorch fork or something with this, right? And then here's something else that was available in TensorFlow for quantization. And here is something else that was available in a different framework. What we were doing is really kind of. Taking all of those techniques that are available in a fairly fragmented way across the internet and trying to put them, you know, under one, under one roof in a way that's kind of a little bit easier to access. And that was the kind of the goal of the Disturbor project is really to, is to show that. And it's an ongoing project at Intel within AIPG to have this kind of set of tools.

• 8:46Speaker 02

  So they were available in, they're available in PyTorch. And, and that's great because PyTorch can export to Onyx, which, which is then widely available. But in addition to the work we're doing, it's entirely appropriate though to give a shout out to the work the TensorFlow team are doing. So there's under TensorFlow contrib, yeah, there is a bunch of useful tools there on both quantization and on pruning as well, right? And there's a pretty strong ecosystem there, also showing a variety of techniques.

• 9:12Speaker 01

  Okay, yeah, so it is at least to a point where I could, you know, get a model off of some repository, maybe in, in PyTorch or wherever, and, and have some tooling that's publicly available to, to prune that down for certain architectures.

• 9:26Speaker 02

  Yeah, yeah.

• 9:27Speaker 01

  What about prepping the model for certain, maybe specialized hardware? You mentioned like VPU's and I know there's a lot of other people pursuing things around, of course, GPUs, but also FPGAs and other things. What is kind of the state of the art, are these kind of pruning methods and all of that tied into that world? Or is that something totally separate?

• 9:46Speaker 02

  Yeah, and that's also a good question. And it was one of the kind of the goals of the talk today was to show that, hey, you know, here's kind of four key techniques that you can use that will work well on any hardware. And on some hardware will work extra well. But if you employ these techniques, you're not going to hurt your model's ability to run across a broad range of silicon, right? So and those techniques specifically are kind of model pruning, sparsification, using fewer, using, you know, quantizing a network to eight bits and then doing further quantization on weights to use kind of a lower bit depth, right? So if you employ those kind of four techniques, you will still have a model, you know, if you take a model and you represent it in Onyx or in, or in, or in TensorFlow,

• 10:28Speaker 02

  you'll still have a model that can work well on a wide variety of devices, but on some devices, it's going to work extra well, right? Because different silicon, well, and different. Different abilities to run quantized models, you know, at varying degrees of acceleration. And also different silicon will have varying degrees of, let's say, weight compression and technology. So, and even in extreme cases, you know, for sparsity, there's, there's some silicon out there that can process sparse networks directly and in an accelerated fashion, right? So, so against a variety of silicon, you can employ these four techniques and get really, really good results across a range of silicon and even better results in some silicon. So that was the core point, right? But to answer the second part of your question, in the final slide, we're making the point

• 11:11Speaker 02

  as well that, hey, if you, if you set out to have a, with a single network and you know the piece of silicon you're running on, absolutely, there's other techniques you can employ to really fit that piece of silicon as best as you can, to really make this one network shine on this combination of this network and this silicon. And there's been some very interesting work published on that in the last couple of months, and it's a pretty, pretty hot research topic now, is, is showing how to like using, you may be familiar with kind of AutoML, right? So being able to use that type of techniques to kind of, to refine a model or to learn a model that works really, really well on a particular version of silicon with these types of performance cliffs and trade-offs, yeah. So it's a, that's a pretty active area of, of research that's pretty interesting.

• 11:54Speaker 01

  Awesome, awesome. And I know that one of the things that I've appreciated as kind of like, as I'm hacking on things at home is that, you know, a lot of the stuff that you've come out with through Movidius makes it really easy to experiment with, you know, neural networks on, on a lot of different types of devices through like the neural compute stick and other things. I was wondering if you had any interesting stories or customer experiences that you've heard about of, of people enabling new sorts of things with these devices?

• 12:22Speaker 02

  Yeah, we really enjoyed the experience of launching the, the first version of the neural compute stick based on Miriand 2. And it was great to get out there and meet lots of developers. And also, you know, when we launched that, I guess it was kind of, we announced it some time before and we really launched it then at CVPR last year. Yeah, it was great to see what everybody was doing, but also to kind of show them, hey, you know, AI at the edge is possible, right? If you go back 15 months, people, you know, Or two years ago, people really associated AI with the cloud, right? So our first goal was to kind of, you know, break down those perceived barriers and for people to and for more people to be able to use AI and to see, hey, AI at the edge is possible, right? So that was our initial goal and it was a great experience, very enjoyable talking to all the developers.

• 13:04Speaker 02

  A couple of things we've seen, we've seen, we've seen people use this. One of the software ambassadors for Intel used this to do, to do a prototype kind of water filter. So kind of taking the guts of a microscope, putting that up to a camera into a Raspberry Pi with the Vidious Neural Compute Stick connected and being able to show that you could actually use this to detect water impurities. So to have an entirely offline water impurity detection device that could be used, you know, effectively like on premises, you know, at the edge with no cloud connection or anything like this. Super cool idea, right? And be able to show that that's possible. Equally, we have people putting them on a drone to detect sharks in the water.

• 13:47Speaker 02

  Also doing kind of prototype medical imaging to detect melanoma on skin. Also kind of driven by image classification. And yeah, so there, there are just a few things, but there's been a lot of other, you know, fun projects posted on GitHub and I don't have a link to our Model Zoo site and example site, but I can provide you with them for, for the blog page also.

• 14:07Speaker 01

  Awesome. Yeah, well, we'll make sure that gets in our show notes for sure. Yeah, well, I appreciate you taking time again. Kind of to wrap things up here, I was wondering, you know, from your perspective, since you've been working in this space for a while, what can we look forward to, you know, over the next couple of years with performing AI at the edge? What are you excited about and what do you think we'll see over the next couple of years?

• 14:27Speaker 02

  Yeah, I think, I think we're definitely going to see a lot more silicon become available, both from, both from the videos Intel, also from a bunch of competitors. And I think that's going to be really interesting as inference silicon, you know, there's kind of metrics business people would track, like the number of like ops per watt we can deliver or the number of ops per dollar we can deliver. And we'll expect both of those metrics to progress at a really, really fast pace over the next number of years. And if I look at what people are able to do with the first Vision of the neural compute stick with the capabilities that has. And while I can't disclose product roadmaps with some visibility of the type of things we're going to see in terms of the volume of compute we can, that various people can bring to market at much lower price points and much lower powerpoints.

• 15:12Speaker 02

  I'm really excited to see what's, how that's going to play out and the type of things people will do with that. I think it's going to be a very exciting space to watch in the next few years.

• 15:20Speaker 01

  Awesome. Well, thank you again for taking time and enjoy the rest of the conference.

• 15:23Speaker 02

  Thanks, Tyler.

• 15:33Speaker 03

  I'm Tim Smith and my show Away From Keyboard explores the human side of creative work. You'll hear stories, sometimes deeply personal, about the triumphs and struggles of doing what you love. Jumping off into the abyss is kind of my skill. And so I'm not saying that it's not scary. I'm saying that perhaps my skill is just not being able to estimate how scary it will be. New episodes premiere every other Wednesday. Find the show at changelog.com/afk or wherever you listen to podcasts.

• 16:19Speaker 01

  Well, thanks for joining us, Mike. It's great to chat with you and meet you here at O'Reilly AI. I've heard about Michelangelo, this ML platform that you guys have developed at Uber. And I'd love to hear a little bit more about it, but first, give us a little background of who you are or how you ended up where you are.

• 16:33Speaker 04

  Yeah, thanks. Happy to be here. Yeah, so I currently am the product lead for ML infrastructure at Uber. And that encompasses a lot of things, most notably the Michelangelo platform. A little bit of background on me is I'm an electrical engineer by training. And I, out of school, I worked at Google. And one of the kind of places I got my ML chops, so to speak, which is weird to say, is I worked on the ads team at Google, specifically the ads auction group. And I was the product manager for all of the ML. ML signals that go into the ads auction there. So these really like real time, high scale, super productionized ML systems that predict if you're gonna click an ad and if you're gonna,

• 17:17Speaker 04

  this ad's gonna be relevant and stuff like that. So that's kind of like where I learned how to do ML right and probably best in industry in terms of productionized machine learning. And then I, about three years ago, I joined Uber where we started the Michelangelo, which is not named after me in any way.

• 17:35Speaker 01

  That's a shame.

• 17:36Speaker 04

  Yeah, and people get that question all the time. We started the Michelangelo platform, which helps people, which helps data scientists and engineers across the company, build ML systems, kind of prototype, explore ML systems, build them and then deploy them into production and serve predictions at scale.

• 17:51Speaker 01

  Yeah, so why, if you're in a company that's trying to build up their AI presence within the company, why would they need a ML platform? Why isn't like Jupiter notebooks everywhere just fine for people?

• 18:04Speaker 04

  One of the things, so kind of like the state of Uber's ML stuff about three years ago was that a lot of people were trying to do that, right? So there was a lot of people, you know, grad students learn how to build their ML models in their grad school classes and whatever, and they have their own ways to do it. Everybody has their own, I use R, I use Python. And what we saw was that people were trying to, either trying to productionize like an R model and run an R runtime in production at high, at low latency, and which is just like very challenging and kind of people will cringe when they hear that today. Secondly, you would, you would see like teams that did have, data scientists that did have engineer support,

• 18:46Speaker 04

  they would build up these bespoke like towers of infrastructure at a per use case basis that would tend to be less well built just because they had lower resources, but like duplicate, duplicative of different, different pieces of infrastructure that people would build to serve these models in production across all the different ML use cases the company has. And then kind of the scariest is people just wouldn't get started at all because they wouldn't have a way to, some people wouldn't, wouldn't have a way to get their models into production. So, so we saw the opportunity to build a common platform to help people. Have a unified way to build models and to, and this is the trickiest part, put those same models that they prototyped on into production to make those predictions.

• 19:27Speaker 04

  And along the way, bring a lot of data science best practices, build into the system reproducibility, common analyses and all of that kind of like versioning and all that kind of good stuff. That is kind of like these data science best practices that aren't yet really well established. And you know, we have a lot of really well established software engineering best practices that everybody knows C I C D and you know, version control and stuff like that. That stuff's not as well appreciated in the data science community. And it's just because a lot of this work is new and it's not like these guys don't understand the importance of it, but it's just like the best processes and the best patterns for building this stuff have not yet, we have not really converged on those yet.

• 20:09Speaker 04

  So kind of spent a lot of effort to focus on what, where we think this stuff is going to go and to help build the tools to enable, to like empower data scientists to kind of do the, do the right thing from the beginning.

• 20:21Speaker 01

  Awesome. So how many people are using Michelangelo at Uber?

• 20:25Speaker 04

  That's really, that's really hard to say. I would say we probably have more than, so this platform supports machine learning use cases across the company. So everything from like fraud related things to predicting how long it's going to take a car to get to you, to even like Uber Eats, like ranking dishes in the Uber Eats app. All of the main M L stuff runs through this platform now, but this is just like an interesting kind of platform development challenge is, you know, we have a lot of people who like kind of use it. They're like, hey, I kind of want to build an M L thing and they dabble in, explore a couple of little models they want to make, but maybe they haven't, they never end up fully deploying that model to production.

• 21:04Speaker 01

  Right.

• 21:05Speaker 04

  And so it's kind of tricky to say like, how many actual use cases do you like, do you have on the system? We know it's well over a hundred, but you know, it's hard for us in the platform to say, is this something that this team is just using this as an experiment? Or is it like fully productionized and deployed across the whole company? And that's just like a, an area that we've just under invested in a little bit, but we think there's a lot more to do there.

• 21:28Speaker 01

  Is there, like, as you've seen people start to use the system, are there features of it that you thought that kind of surprised you in the sense of how people relied on them or things that people needed that you didn't expect that they would need or other things?

• 21:43Speaker 04

  Yeah, that's a really good question, and I've been reflecting on this a lot recently. And the, you know, I'm the product manager, so it's kind of my job. But, but, but the thing that I would say that kind of has gotten disproportionate adoption, given our, maybe even like, under investment into this, where we could have, we still could do a lot more in this space, but our users just adopted this overwhelmingly and they love it, is our feature store, which is part of the platform. And what that allows, so, you know, common problems for, for managing features related for ML workflows are that you have to clean your data and transform your data and combine it all, and also historically for into a training data set, so you can train your model.

• 22:24Speaker 04

  But then, once your model's created, how do you do all of those same transforms in the same way, the same preprocessing to that data in real time when you deploy your model? So there's kind of this like dual type of ETL that happens in different compute environments. It's really tricky and...

• 22:40Speaker 01

  Yeah, possibly on a variety of resources.

• 22:43Speaker 04

  Yeah, and I mean, we see a lot of like vendor solutions here, but I feel like we don't see anybody really tackling that kind of stuff. And I think it's partially because it's not sexy at all to work on that stuff, and also because it's just super hard to do properly. And we've provided some nice ways for people to define their feature transforms to the platform and then be confident that that, those transforms will happen consistently across both computer environments, you know, real time and offline. But I think the other interesting thing is, we saw, let's take the Uber Eats world, for example, they probably have more than 10 different models that they use to predict, to rank dishes and whatever they do. A lot of those models use the same kind of features.

• 23:26Speaker 04

  And before this feature store, data scientists didn't have any insight into, hey, other people that were working on similar problems, what kind of feature pipelines had they built? And then when this feature store came along... Now, when a data scientist wants to start a new model, they can just look and see what features exist that are relevant for me. Let me just like start including or start off, warm start my model exploration process with the X features that are most relevant to this problem from the beginning. So there's a whole new element of collaboration, visibility, feature sharing that was previously not there. I really don't see much solutions in that space in industry today either. So I think that's a really promising area.

• 24:07Speaker 01

  Cool, yeah, I look forward to hearing more about that. And definitely, if you publish anything about that, we'll be happy to post that on the show links here. Yeah, the other thing I was curious about, just from the fact that you know, you mentioned before that the incentives for data scientists are kind of different and not always aligned with producing, you know, production-ready models and all of those things. How do you, how do you build up a team to build an ML platform where really you kind of need a software engineering experience to be able to build something that's production-ready, but you need the knowledge and the expertise around machine learning to be able to understand, you know, what to build. So you're, it's going to be relevant to the people you're building it for.

• 24:48Speaker 04

  Yeah, so I think one of the nice things is that we've had a little bit of, the leadership in our organization has been a relatively forward-thinking to be willing to fund an ML, the development of an ML platform much earlier than I think is common in industry. And that's allowed us to get it wrong a couple of times before we got it right. But we feel like we really got it really right now. And there's like a tension between data scientists want this nimbleness and flexibility throughout their exploration and prototyping stages. And, and you know, if you think of any productionized system, it's super stable. And, and so how do you kind of accomplish both of those constraints? It's a challenge.

• 25:30Speaker 04

  And so what we, some of the design philosophy that we're taking and we're, you know, this is always developing is we're trying to allow data scientists to work within our system using the tools that are most relevant for them. So we'd love for them to work in. Jupiter notebooks and write all their models the way they normally would. We can provide some helpful APIs for them. For example, the feature store stuff to pull in their data. So they don't have to re implement a whole bunch of work that already exists in terms of like enterprise intelligence, you know, that's already been done. But after a certain point, when the kind of prototyping stage is complete through, if you think of like this machine learning lifecycle where it's like now I want to actually use this in in production and maybe it doesn't mean you're going to launch it to the whole

• 26:14Speaker 04

  company and it's going to be and you're done with the project. It could just be like, I want to experiment with this on live traffic. We focus on making it relatively low activation energy to take your prototype and transform it into something that can go into these productionized, well engineered, hardened systems that we can be confident will be stable from a systems perspective. And we still want to give data scientists the ability to monitor these models that are in production or not just, you know, systems issues like whatever applies to typical microservices, but also like the data science monitoring. How accurate is this model over time? Are there any model drifts, stuff like that? And so there's a story for data scientists throughout the lifecycle and a story for

• 26:57Speaker 04

  engineers throughout the lifecycle and the balances and the challenges like how do you balance between those at the different stages, taking into account all of the priorities for both stakeholders throughout?

• 27:07Speaker 01

  Awesome. Yeah, that that gives some great perspective. Well, to kind of end things out here, are there places online where people can find out more about what you guys have done and maybe also some things that you put out there that you might want to share?

• 27:19Speaker 04

  Yeah, that's a good question. We've published a blog post about Michelangelo and I think October 2017. And it's pretty easy if you just search Michelangelo ML platform on Uber, on Google, rather, you could find that. And we've published a lot of other pieces about related ML work we've done. And I think we're likely to, in the near future, open up the kimono a little bit more on Michelangelo. So stay tuned.

• 27:42Speaker 01

  Cool. Awesome. We'll look forward to that. Well, thanks for, uh, thanks for joining and enjoy the rest of the conference.

• 27:47Speaker 04

  Appreciate it.

• 27:51Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoy this show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for change log is provided by Fastly, learn more at fastly.com. And we catch our errors before our users do hear change log because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode cloud servers. Head to linode.com/changelog, check them out, support this show. This episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address, get our weekly email

• 28:36Speaker 00

  keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.