So you have an AI model, now what?

Fully Connected – a series where Chris and Daniel keep you up to date with everything that’s happening in the AI community. This week we discuss all things inference, which involves utilizing an already trained AI model and integrating it into the software stack. First, we focus on some new hardware from Amazon for inference and NVIDIA’s open sourcing of TensorRT for GPU-optimized inference. Then we talk about performing inference at the edge and in the browser with things like the recently announced ONNX JS. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean is simplicity at scale. Whether your business is running one virtual machine or ten thousand, DigitalOcean gets out of your way so your team can build, deploy, and scale faster and more efficiently. New accounts get $100 in credit to use in your first 60 days. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We catch our errors before our users do because of Rollbar. Resolve errors in minutes, and deploy your code with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Linode https://linode.com/changelog – Our cloud server of choice. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2018. Start your server - head to linode.com/changelog https://linode.com/changelogFeaturing: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: News: NVIDIA’s open sourcing of TensorRT https://news.developer.nvidia.com/nvidia-tensorrt-inference-server-now-open-source/Amazon launches a machine learning chip https://www.cnbc.com/2018/11/28/reuters-america-amazon-launches-machine-learning-chip-taking-on-nvidia-intel.htmlThe recently announced ONNX JS project https://github.com/Microsoft/onnxjsSnapdragon Neural Processing Engine SDK https://developer.qualcomm.com/docs/snpe/overview.htmlLearning resources: Rise of the model servers https://medium.com/@vikati/the-rise-of-the-model-servers-9395522b6c58TensorRT server tutorial https://devblogs.nvidia.com/nvidia-serves-deep-learning-inference/ONNX JS https://github.com/Microsoft/onnxjs on GitHubTensorFlow JS tutorials https://js.tensorflow.org/tutorials/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Beta for change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at change law because of roll bar. Check them out at rollbar.com and we're hosted on Lino servers. Head to linode.com slash change log. This episode is brought to you by Digital Ocean. They now have CPU optimized droplets with dedicated hyper threads from best in class. Intel CPUs for all your machine learning and batch processing needs. You can easily spin up their one click machine learning and AI application image. This gives you immediate access to Python three, R, Jupiter notebook, TensorFlow, side kit and PyTorch. Use our special link to get a hundred dollar credit for Digital Ocean and try it today for free.

• 0:45Speaker 00

  At the do.co slash change log. Once again, do.co slash change log.

• 1:03Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence, practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show at change.com slash community, follow us on Twitter, we're at Practical AI FM. And now into the show.

• 1:29Chris Benson?

  Hi there, this is Chris Benson and welcome to another fully connected episode of Practical AI, where Daniel and I will keep you fully connected with everything that's happening in the AI community. We take some time to discuss the latest AI news and we dig into learning resources to help you level up on your machine learning game. How's it going today, Daniel? Oh, it's going great. I'm excited about some of the news we got going on today. Yeah, it's, I love the format, the way we're diving into it. For those of you who may have listened to our last Fully Connected episode, I think it was, hopefully it was as good experience for you. We're definitely listening to your feedback, trying to shape the show to better serve your needs going forward.

• 2:08Daniel Whitenack?

  Yeah, and I think that there's, I mean, I've been talking to a couple of people this week. There's just so much going on. It's good to just have a chance to, for me personally, just to have a chance to talk through some of these things because there's so much going on, there's so many topics, there's so much jargon. To kind of try to put some of that into words is, I think, helpful and we're kind of learning along with everybody listening. So keep us, keep us honest and let us know what we get right or wrong as we're going through this stuff.

• 2:39Chris Benson?

  Yep. And if you haven't already, we hope you'll join us in our Slack community at changelog.com. And we have great feedback, great conversations that are happening there between the shows. We're also on LinkedIn, in a LinkedIn group, and we hope you'll join us on LinkedIn. You can just search for Practical AI.

• 2:56Daniel Whitenack?

  Awesome. Well, this week, as I was kind of going through and looking through Twitter and various news sources, one of the things that, or the themes that came up when I was looking through things, was really having to do with all the things that happen after we train our AI. So the question is, you know, we've trained an AI model, what next? So in your opinion, Chris, what, what happens next? What happens after you train an AI model? How, you know, what do you do? How is it useful?

• 3:27Chris Benson?

  Yeah, it's funny. Before I answer that, I'll just note that this is the side of things that we tend not to, to think about too much until we get there. You know, the courses that are out there are really focused on training and architecture and, you know, people will kind of say, okay, I've got it. And, but your model doesn't do any good until you deploy it into the real world and it's, it's useful for your customer, for your end user. I know that as I was learning my way up through it, through the field, this has been a bit of a challenge because the deployment environments and what you're targeting for deployment can be very different and the standards have been slow to arrive there. That's changing now, but it's definitely, as I started out before, some of these standard approaches were starting to come into being, every vendor was different and that was a real pain.

• 4:12Daniel Whitenack?

  Yeah. And for those of you that are new to some of this jargon too, what we're talking about here, you know, you can kind of think about this AI model as a sort of really complicated function that has a bunch of parameters in it. And so when we do training, we're using a whole lot of data through this training process to tune and tweak all of those parameters of our model. So we might have, you know, millions of these parameters that parameterize our AI model function to do something, you know, to, you know, transform an incoming image into an indication of objects in that image, for example. So the question is, you know, once we've gone through that process and set our parameters, now we have this function that can transform data.

• 4:55Daniel Whitenack?

  What do we do with it? So what are, what are some of the things that you've done after training? Or you've needed to do after training, or you've seen other people do after this kind of training process, Chris?

• 5:05Chris Benson?

  Well, honestly, a lot of it involves cooperating with other teams in midsize or larger companies. If you're in a small company, it may be just yourself, but you've got to, a model is only useful if you are able to integrate it into some software that's going to go out onto your target device where you're deploying. And, and that's a whole different set of skills.

• 5:26Daniel Whitenack?

  Yeah. So when you say integrate it, what, what is the integration? Or what are you integrating? Really?

• 5:33Chris Benson?

  So you would take a trained model and you have to put it into a software package. And therefore the model has to be in a form that's usable. And by usable, it means you have a trained neural network that is able to operate on the hardware and software environment that you need to put it in, in the end, and that it needs to be able to have access to the data that is going to be feeding through it for inferencing purposes, so that you're actually operating. And those are, there's a lot of stuff to think about there that your traditional data scientist may never have had to deal with before. There's a lot of software engineering, and maybe even systems engineering involved in trying to get it out there. And so I thought this was a great topic to go ahead and delve into and, and talk about what those pain points are.

• 6:18Daniel Whitenack?

  Yeah, so I'm glad you kind of brought up the software engineering side of things. And, you know, if you're, if you're trying to code some, you know, AI stuff, whether you're a software engineer or not, you probably know that, you know, this idea of functions or handlers or classes are part of, uh, part of software that we build. And so I think, you know, in my mind, what, what, as I'm kind of translating what you're saying, Chris, I'm thinking about in a web server that's serving a website or something, right? We might have a whole bunch of functions that do something like you give it a specific request and it gives you content back, maybe a picture or a video or just some HTML or JSON or something.

• 6:58Daniel Whitenack?

  And so in integrating AI into that, really, we're saying that at some point in those functions or classes or other things that are part of the software that's running in production in our company, somewhere in there, we're actually accessing this model that you've mentioned. And so it has to be in some form, like you said, to be accessed. And most of the time that's a trained form. In other words, we train our model and then we save it somehow. And then we load that saved or serialized model into one of these functions and then just execute the data transformation that it does. Like I said, from image to objects or something like that. And that process of utilizing the function is, is called inference.

• 7:41Daniel Whitenack?

  So with that, I don't know, did I miss anything there, Chris? Or any jargon that you think is, is relevant?

• 7:47Chris Benson?

  No, I think another word that you might use is to simplify things is just think of it as you need to wrap your model up as a software component. And just as your, whatever your software that you're deploying may have a number of components that, that make it up, the models are also components. They're components wrapped in whatever language you're deploying in. So it may be that while you're training your model in Python, in TensorFlow, or PyTorch, or whatever you're using, uh, it may be that you're deploying in C, your C++ or Java, or I know you and I love Go as well. And, and those may to where you're, you're, you're doing the inference thing as opposed to the training through that way. And that so, and you, you think of the model as a piece of that software component going forward. And, and it's part of deployment.

• 8:31Chris Benson?

  You think of all the things that surround software engineering and deployment go into that.

• 8:35Daniel Whitenack?

  Yeah. So when you've deployed models in this way, a lot of times, how, what's been the access pattern or how have people interacted with the model? I know for me, it's been a lot of times integrating the model into some sort of API. We can talk about a little bit more later as related to some of the news, but essentially just where it's integrated into kind of like a web service, where you would make a request for a prediction and get back a result. Have you seen other, other patterns? That's, that's the one I've seen most often, probably.

• 9:07Chris Benson?

  Yeah, it's always in the form, using it loosely as a service. If I've seen web services used most often on server side, where you may not be constrained by your connectivity and stuff. A lot of times though, if your deployment target is an IoT device or a mobile device, you, you still have an API, but it's really, it is operating as a function, you know, to use the, the phrase you were using earlier. That's, that's just the API, may not be a public API that your software component is using inside your, your group of software components that constitute your solution. It doesn't really matter, in my view, so long as that you are essentially following the best practices of the environment in which you're coding and what your, your deployment target is made up of, so.

• 9:49Daniel Whitenack?

  Makes sense, yes. That brings us right into really some of the news that is related to this, that, that came up this week. First, let's, let's kind of focus in on this inference service or servers bit of things. One of the things that I saw come out this week was an announcement from Nvidia that their Tensor RT inference server was now open source. So Tensor RT, I think it's been around a little bit, but this was the official announcement of the Tensor RT inference server officially as an open source project now. So this is, this is a project from Nvidia. And part of the goal, in my understanding of Tensor RT,

• 10:34Daniel Whitenack?

  is to perform these inferences that we've been talking about. So post-training your model, when you're actually utilizing your model, is to do that in a very, very optimized way, maybe on Certain specialized hardware, for example, on GPUs, which Nvidia, of course, is concerned with. So it was exciting to see this actually be open sourced and available for the community. It seems like there's a bunch of great stuff in there. It also includes examples of how developers could extend Tensor RT to do things like custom pre and post processing and integrate additional framework backend, so more than just TensorFlow, but like Caffe 2

• 11:16Daniel Whitenack?

  and others via the Onyx framework that we've talked about here quite a bit, which is pretty cool. So yeah, I was excited to see this. I know that you've utilized GPUs probably more than I have. Chris, have you ever tried to integrate the inference side of things on GPUs?

• 11:32Chris Benson?

  Yeah, it's, I know, working at some of the employers that I've had. And for our cases, we always have a project or service that we're supporting. We're always deploying. And so, you know, one of the great things about Tensor RT was really the first one that I got into in kind of at scale. And it does a number of optimizations to your model specific to deployment. So you're essentially taking your model and putting it through this process that Nvidia has, where it optimizes it for inference and then deploys it. And I'm not really surprised to see that Nvidia has open sourced their inference server, because they have, they've been leading the way in a lot of areas and forcing some of the other previous,

• 12:15Chris Benson?

  you know, giants like Intel to play catch up for a while. But now we're starting to see the market stabilize a little bit and seeing more than one player out there. And so if they want to continue to be the leader, open sourcing their Tensor RT technology is a very sensible thing to do to make it accessible. So I applaud the move on their part and wish they had done this earlier when we were first learning it. Because, you know, being open sourced now, we can, we can figure out what our problems are on our own a little bit better, obviously, by going through the source code and not having to worry as much about bugs that aren't documented, that kind of thing. So it's a great move on Nvidia's part.

• 12:51Daniel Whitenack?

  Yeah, and I mean, the, I guess, one thing to point out here, and correct me if I'm wrong, because I think you have more, more experience here, but it seems like Tensor RT, a lot of the focus is in optimization, not necessarily on the kind of setting up an API to access your

• 13:10Chris Benson?

  That's correct.

• 13:10Daniel Whitenack?

  model. Although I do see that, you know, they have this statement in the article about, you know, to help developers with their efforts, the Tensor inference server documentation includes various things, including, I think there is a tutorial in there that they've illustrated how to set up a REST API with Tensor RT, and we'll link that in, in the show notes, of course, but I think that's definitely a helpful thing because at some points I've seen a bunch of, it's hard for me, at least when I see a bunch of stuff about optimization, but then I still struggle with the integration part like we talked about initially. So I'm glad to see them at least have some, some examples in that regard.

• 13:51Chris Benson?

  Yeah, I think, I think Tensor RT started with, with those deployment optimizations, and that was kind of its foundation, but it's definitely provided more and more tools for developers and DevOps engineers to be able to get this out into the real world. And we're seeing a general push in industry to do that from these companies that are supporting, you know, with GPUs and other, and other technologies to get that out. So it's getting easier and easier to use these, and Tensor RT has definitely been a big part of that for Nvidia.

• 14:19Daniel Whitenack?

  Yeah, and speaking of running inference on specialized hardware, you were mentioning to me right before the show about something that you saw from Amazon, right?

• 14:31Chris Benson?

  Yeah, Amazon is like we've seen with other providers, they have announced that they are launching their own machine learning chip. It's not something they're planning to sell. They're going to be driving some of the servers in AWS this way. In the article that I was referencing, which was a CNBC article, they used the phrase taking on Nvidia and Intel, but I think to some degree, it's, it's them reducing their risk or dependency on specific vendors. I don't think we're going to see, you know, vendors out of AWS. Entirely anytime soon, but Amazon now not only is able, has more tools in the toolset in terms of chips that support these, this type of, of work, but also it gives them leverage

• 15:12Chris Benson?

  with those vendors in terms of the pricing they're going to go. So it's all good from my standpoint. In the, in that I'm hoping that this drives prices down. It gives them a little bit of leverage and Nvidia, Intel and Amazon all end up lowering prices. I hope it doesn't, I hope it doesn't take another path from that.

• 15:29Daniel Whitenack?

  Yeah. Let me know if you, if you think this is a good analogy because I'm, I'm not sure that it is, but you know, Google, like all the cloud providers now pretty much have GPU support, right? And I think most of those are Nvidia GPUs, but also Google has kind of developed this TPU architecture, right? Which is only available in Google Cloud. It seems like now Amazon is kind of doing maybe not the same type of play, but doing some sort of specialized hardware that's maybe only going to be available in AWS. Is that, do you think kind of a similar play or?

• 16:08Chris Benson?

  I do, I do. I think that, and you know, if we go back to the episode where we had Nvidia's chief scientist, Bill Dally on, and he, he schooled us all in, you know, GPUs versus TPUs and ASICs and such, and you know, all the different hardware possibilities here. He talked about kind of the rise of ASICs and you know, the TP, you could think of the TPU to paraphrase him is, is almost a lighter version. A GPU has a whole, a bunch more to it other than just doing the math necessary in a neural network. And so I think you're seeing these kind of very specific chips coming out with Amazon and with the, with the Google TPU and you know, the GPUs have that same capability, but they also have a whole bunch more.

• 16:50Chris Benson?

  But it seems to be that as people really focus on that specialization of doing the matrix mathematics, the matrix multiplication, it is, it's really kind of commoditizing the industry because they, instead of trying to recreate an entire GPU competitively, they're really focusing on this use case.

• 17:07Daniel Whitenack?

  Yeah, but it seems to me at least, and I, you know, I'm not a hardware expert, but it seems to me like all of these people are coming up with all of these different architectures, including, you know, Intel having the Movidius stuff and other people having specialized hardware. It seems like there's just a lot of kind of architectures to support now, and that does seem like a challenge. You know, maybe, maybe these projects like Onyx are a way to kind of mitigate that challenge because now we might want to train model and we do that, let's say, in PyTorch or TensorFlow, but we may want to deploy the inference on one of many different architectures.

• 17:51Daniel Whitenack?

  So I don't know, it seems like there needs to be a central point for standardizing our model artifacts. And I've at least had some success with Onyx in that respect. And so those aren't familiar, we've mentioned Onyx on the show a few times. It's the Open Neural Network Exchange format, which is a collaboration between a bunch of people, including Facebook and Microsoft and Amazon, I think. But it's still pretty rough. So in some respects, like if you're trying to, if you're trying to serialize a model from scikit-learn to Onyx, for example, there's a little, a few rough edges there, at least in my respective, or my history with, at least with the docs.

• 18:34Daniel Whitenack?

  But it is a really great ambitious project and I certainly hope that they succeed because I definitely see a lot of problems that could arise from trying to support all of these different architectures. Seems hard.

• 18:47Chris Benson?

  Yeah, I agree with you. And I think, I think Onyx was a fantastic first way of providing that, that commonality across these different technology platforms. And I think that there is still a lot of room, especially within the open source world, of producing other tools with, with a similar intent. Just as Onyx has provided us that common format, there may be a number of deployment tools that come out where, where a deployer can focus on learning that as a, as kind of a standards-based approach. Rather than all the individual stuff. I know that in a prior company, we were deploying to both Tensor RT and something that I'll bring up, which is the Snapdragon from Qualcomm.

• 19:29Chris Benson?

  While the workflows had similarities, they were completely different workflows that we had to learn and we had people on the team that kind of specialized in either approach and stuff. It would be really great if you could target one workflow that would work across vendors in that way.

• 19:44Daniel Whitenack?

  Yeah, abstract that away. So, right before, just a second ago, Chris, you mentioned that you had worked with this Snapdragon before, which I'll let you describe here in a second. But one of the other trends that I saw kind of in the news and updates in the world of AI this past week was some stuff having to do with running inference, running models in the browser, on mobile, on client devices and IoT devices. This kind of idea of pushing models out of, you know, always being run in the cloud, in some service in the cloud, into kind of more towards the quote unquote edge or the client devices. Is this a trend that you've been seeing as well?

• 20:29Chris Benson?

  Yeah, I think it's interesting. I think that you're seeing a lot of inferencing being pushed out to the edge. And I know that that has been specific use cases that I've dealt with have had to do with mobile devices that were kind of leveling up and getting a Snapdragon in them that we've deployed to and also IoT. And so I think, you know, the world that we're at right now, you have lots of mobile and IoT devices that are not nearly powerful enough. I think with the recognition that inferencing is being pushed to the edge, you're seeing a number of vendors starting to sign up with Snapdragon or similar types of technologies. Basically, low-power inferencing engines that can be deployed to inexpensive hardware on the edge with very limited computing resource.

• 21:14Chris Benson?

  And so I think you're going to see that type of thing all over the place. And I think that's a given at this point where your inferencing workload is distributed between the cloud and the edge as it makes sense. I think the big question now is whether or not there's enough use cases of doing actually training on the edge, whether or not that becomes a thing. I don't think that's really taken hold. There's certainly lots of conversations around it. But I haven't seen it personally in industry, you know, actually being deployed in a production sense.

• 21:46Daniel Whitenack?

  Yeah. So in the cases where you're talking about when you were using the Snapdragon thing, the neural processing engine, the motivation for pushing that inferencing out to a mobile, or it sounds like in your case, an IoT device, maybe a sensor or something like that, what was the motivation for that? Was it like connectivity? Was it efficiency or timing? Or what was the primary motivation?

• 22:11Chris Benson?

  Yeah, it really depends on the resource environment that you're deploying into, and also what the performance parameters are of your, of actually operating, you know, on whatever.

• 22:22Daniel Whitenack?

  So by, by resource environment, you mean the actual resources on the device that you're deploying to, or?

• 22:28Chris Benson?

  Yeah, well.

• 22:29Daniel Whitenack?

  The CPU or something?

• 22:31Chris Benson?

  Yeah. And there can be a number of cases. An example that I had personal experience in was in speech recognition and natural language processing, where you may need to, you don't have time, or you may not have an environment equipped with the right network connections to pass to the cloud and then pass back. There's latency involved in that. If you're in an environment where you simply don't have time for that, you know, a few two tenths of a second delay, or whatever it is that you're dealing with. In some cases, there are speech recognition technologies where the use case requires that you start processing before you're even done necessarily speaking a sentence. So you may be already having processed the first part of this sentence I'm saying right now before I finish this

• 23:13Chris Benson?

  second part. It may be that the latency issues get in the way. I've seen some very specific constraints around that in industry. And there may be some situations where you can go either way, where you can have it be cloud-based. But I think, I think as inferencing becomes easier and cheaper on the edge, you're going to see it more and more where instead of it being specifically a constraint, you're going to see where does it make sense to put this, you know, from a cost benefit analysis?

• 23:37Daniel Whitenack?

  Yeah. I'm thinking back to that, play our, way back at our episode three, where the, the team at Penn State was kind of deploying this app for African farmers that would classify plants. I'm guessing, I don't know, but I'm guessing that there's probably connectivity issues. For the devices when they put them out in the field, which is literally the field, like the farming field, right? In this case, so I imagine that like they can't necessarily rely on inferencing cloud environment because they simply just can't connect. So I think there's like this one issue of, you know, maybe just not being able to connect and having to run that on the device. But of course, there's issues with that.

• 24:20Daniel Whitenack?

  I remember them talking about inferencing, you know, really, if I remember right, kind of draining the battery of the device and that sort of thing. So there are, I know there are, you know, constraints here. I don't think, you know, you can totally just export everything right now to the, to these low power devices and expect things to work, work out great. But um, there is some encouraging signs. One interesting thing that I wanted to bring up, which I've seen referenced a few times this week. One in particular, I saw this release of the Onyx JS project from Microsoft, which is a project for running models and model related operations in your browser in JavaScript.

• 25:05Daniel Whitenack?

  So there's, there's a similar project TensorFlow JS, which is specific to TensorFlow. And I'm sure that there's other JavaScript frameworks out there. I'm not, I'm not a huge JavaScript person, but in my understanding, so in addition to these things that we've talked about in terms of connectivity and all of that, there's actually a huge privacy and data element to where you run inferencing. So it could be that when you run training, you run it in a very, you know, on a big beefy server in the cloud. And the reason why you do that is because you have to process a ton of data. Maybe you're processing 200 terabytes of data or something like that.

• 25:47Daniel Whitenack?

  But maybe that data, it doesn't include sensitive data or something. Maybe it's anonymized in some, some case. But then if you, if you transfer that model over and run it in someone's browser, and then you're running the inference in their browser, you may be processing their particular data like. You're processing the feed off of their webcam, for example, right? And if you're doing that, obviously, that could be very sensitive data. And so one thing you could do is transfer all of that data up into the cloud, do your inferencing in the cloud. But then you're essentially, you know, taking possession of all of that sensitive data. Whereas if you run the model actually in the browser and do the inferencing there,

• 26:31Daniel Whitenack?

  then the user's sensitive data actually just stays on their device. So you can kind of totally, maybe not totally, but you can avoid many of these kind of privacy and security related issues in terms of how and what data you're processing where.

• 26:48Chris Benson?

  Yeah, and there's other considerations like a while back in an episode, we were talking about the General Data Protection Regulation, GDPR in the European Union, which is actually, though it's only officially applied there, many organizations are applying it globally so they don't have to support multiple business approaches and processes. And it may very well be that by running, by doing the inferencing in your browser, for instance, instead of passing up to a cloud, you're able to fit within particular regulations in a given country where you're not actually moving the data. The model can be deployed widely, but the data has to stay where it is. And therefore, that might be the only option or one of the only options that you have short of having servers in every jurisdiction that you're going to operate in.

• 27:30Chris Benson?

  So there's a strong use case going forward from a regulatory standpoint for being able to just do it right there in the end user's browser and let them keep the data private. It never moves. It takes the whole regulatory concern, at least that aspect of it, out of the picture.

• 27:45Daniel Whitenack?

  Yeah, I think there are with everything that we've talked about before, and I guess everything related to this, there's always trade offs, right? It seems like I was talking to a friend of mine who is at a startup and part of their startup, you know, IP and really the secret sauce of what they're doing is in their machine learning model, right? But then if you take that model and then you push it out to someone's client device and run it in their browser, of course, there's always the opportunity for You're releasing that model out into the wild and people can just maybe just take it and you know, look at the view source in browser and figure out how to get your model and utilize

• 28:26Daniel Whitenack?

  it and all of that. So I know that he was concerned about about those risks, but it's probably, I don't know, in my mind, maybe the benefits outweigh the cost because in the same way, there have been a lot of papers that have shown even for doing inferencing in the cloud, if you're exposing some service that does inferencing for like image recognition or something like that, it only takes a certain number of requests to that API to be able to kind of mock or spoof that machine learning model and actually create a duplicate of it. So I guess there will always be those, you know, those trade offs, but there is kind of this transfer of the model to the client's device, which is probably has some trade offs

• 29:09Daniel Whitenack?

  there. But also, you know, these models aren't super small. And if you want to update them over time, maybe there are some, you know, storage or battery or other sorts of issues going on there. So I'll be interested to see, you know, how how people deal with those trade offs and what ends up becoming the the driving force there.

• 29:31Chris Benson?

  Yeah. And kind of to go back full circle, you know, that's when we talk about these deployment technologies such as Nvidia's tensor RT or the Snapdragon neural processing engine, which is called Snappy for short, those optimizations we made, they literally will change the architecture of the model that you've trained when you're deploying and they in, there's a number of techniques that they apply to optimize that. So that's part of that deployment of models out. I think the way I see it is it's great to have all these choices and options that are finally coming into into into being in the software engineering world. There have been, over the years, the evolution of software has given us many choices for client side and server side and how we're going to choose to distribute workloads and

• 30:16Chris Benson?

  such. And fortunately, we're seeing that same evolution happen fairly quickly. And there's already, you know, there's already a roadmap on that from the software engineering world. We're seeing That being applied to data science and to AI technology specifically fairly quickly at this point. You know, we're measuring it now in in weeks and months instead of years or even decades, the way it took in software engineering. So I think having different ways of deploying a given thing, a given model in the days ahead is gonna is gonna make allow us to best serve our customers in that way. So I choice is good.

• 30:49Daniel Whitenack?

  Yeah, and choice is good. I mean, in the sense of cost too, like you've already mentioned, if there's more more choices out there for this type of specialized hardware. You know, I know that this has been a big win for Intel's chips that are in drones and you can plug in via USB stick and stuff. It just allows people to do, you know, fun things really quickly with deep learning and also functional things that are really crucial to certain certain products. And so I think that you ultimately win as a consumer, right? I've kind of stopped. Well, part of me still wants to buy a big, you know, GPU workstation, which I probably will never do because I don't have all the money.

• 31:28Daniel Whitenack?

  But but the other side of me says, you know, well, at this point, it doesn't matter because I can get any sort of specialized hardware for doing this stuff in the cloud. And moreover, I can, you know, go and buy one of these chips that I can integrate into my Raspberry Pi or another fun device and just build some fun projects. And when I need more compute power, then I just spin up more more on the cloud. So yeah, I'm glad that I don't have to, you know, keep that saving going for a huge GPU machine that they'll sit in my in my office. Although it probably be good for heating.

• 32:04Chris Benson?

  Just through through employers, I've had I've had the the privilege of having access to DGX ones at this point DGX twos.

• 32:13Daniel Whitenack?

  And those are machines from Nvidia, right?

• 32:15Chris Benson?

  Yeah, those are supercomputers from Nvidia and and also the workstation, which is essentially half of a DGX one. At least that's what it was. The specs may have changed and they're those are all very, very expensive, but those are for training at scale, very, very complex models. And it's it's great to see. I think right now we're seeing so many players getting into the space. With ASICs and TPUs or the equivalent and such and and there's now choice in hardware. And that is really commoditizing the entire field. So I think I think it's becoming very reasonable to get into deep learning for small projects, the way we do in software engineering where you know, you might, you might go to work and have a primary, large scale project you're working on for your employer.

• 32:57Chris Benson?

  But then you'd come home at night and on weekends and work on on something that's really passion driven. And I think that is becoming more and more viable for data scientists who are really into deep learning and for software engineers who are getting into deep learning. So it's, I think, I think we'll continue to see that. I still think we're going to have incredibly expensive AI supercomputers. You know, the DGX two is substantially more powerful and more expensive than the DGX one was. We're seeing a breadth of what's available out there.

• 33:26Daniel Whitenack?

  Yeah, and kind of turning now from all of that news and great stuff about inference and hardware to some things that will help us as we build those, you know, passion projects or try to figure out how we can do inference at our, at our new, at our company or on their new project. We'll kind of turn now to the part of Fully Connected where we share some learning resources. In particular, we're going to share some with you today as related to this topic of inference. One of the ones that I really like, that I think if you're new to this whole idea of what happens after training my AI model, maybe you didn't know that there was something that happened after that.

• 34:09Daniel Whitenack?

  Maybe you didn't know about this whole idea of integrating models into APIs. This article, it's called Rise of the Model Servers, which sounds very scary, actually.

• 34:22Chris Benson?

  Sounds like a movie, doesn't it?

• 34:23Daniel Whitenack?

  It does. It should be made into a movie, but it's from, sorry if I mispronounced the name, but Alex Vikadi. And it's on Medium and it says, Rise of the Model Servers, New Tools for Deploying Machine Learning Models to Production. And I just found this to be a really good summary article in terms of first telling what a model server is, which we've kind of already discussed here, but she goes. Into a little bit more detail. And then she just goes through and gives you five different kind of common choices for this, which includes Tensor RT, which we already discussed, but it also includes something that I've used before, which is Model Server for Apache MXNet, includes TensorFlow Serving, Clipper, and Deep Detect.

• 35:04Daniel Whitenack?

  She goes through and talks about each one, but also gives you a link to the various repos and the papers that are relevant. So it's a good, it's a good jumping off point if you're new to this whole side of how to do inference or set up inference servers.

• 35:18Chris Benson?

  Yeah, there's another thing, just to note, is I know we've talked about Tensor RT. Nvidia has some, some great tutorials and references on their dev blogs. Nvidia, it's devlogs.nvidia.com that you can get into and learn about that. And, and since I also mentioned that Qualcomm Snapdragon and the snappy Snapdragon Neural Processing Engine, their SDK, which you can find at developer.kalcom.com, has a lot of good material on how that you can jump into that. So those are two vendor-specific sources that I know that I personally have used quite a lot over the last, over the last.

• 35:54Daniel Whitenack?

  Snappy. I didn't get that acronym until right now. I've never.

• 35:58Chris Benson?

  S N P E. Snappy.

• 36:00Daniel Whitenack?

  Ah, that's a good one. I mean, it's not immediately obvious to me, but, uh, but still a good play on their part. That's a, that's a catchy one.

• 36:08Chris Benson?

  Yep.

• 36:09Daniel Whitenack?

  The last thing I wanted to share was just, um, so as I mentioned, I'm a noob at JavaScript and a lot of things along with that. But if you've been, uh, kind of interested in that side of things about running AI in the browser, maybe you would just want to learn a little bit of JavaScript and want to learn a little bit of AI at the same time. There, we'll put these in the show links, of course, all of these links. But if you're interested in this Onyx JS project that was just released, they have some examples and demos and a demo site that's on their GitHub. And then also there's a link that we'll put there for the TensorFlow JS tutorials. So they have in these tutorials, kind of a natural progression.

• 36:56Daniel Whitenack?

  From core concepts, so they talk about, you know, the, the specific things such as tensors and operations and models and layers and how those are represented in JavaScript, all the way down to more complicated things like, you know, uh, doing, uh, synthetic data and webcam data, WebGL API layer for, for Keras, um, all of these sorts of things that, that might be a little bit more, um, little bit more advanced. So that, that will probably get you a little bit further at, at this point. So it's definitely something that I kind of want to explore a little bit. As I mentioned, I, I'm pretty new to that, that side of things.

• 37:35Chris Benson?

  Yeah, I've done JavaScript over the years, uh, but more focused probably like mode people, most people on, uh, mostly front-end development, uh, like Ember JS. And these days I use React and obviously you use Node JS for all sorts of stuff, whether you're coding or not, but, um, I haven't really delved into this, applying the JavaScript skills into, uh, into the deep learning world. So I think I really need to dive into this and see what it has and understand how it can fit into other things that I've done in JavaScript.

• 38:03Daniel Whitenack?

  Yeah. So I, I heard a talk once and, um, I'll have to remember who it was, but I remember the statement was that, you know, no one codes in JavaScript, but everybody codes in JavaScript. I think that was the, the, the statement. So, uh, yeah. So I think that brings us to the, to the end of our fully connected episode. So for all of you not JavaScript programmers, slash JavaScript programmers out there, appreciate, uh, you going through this, this journey and learning a little bit about inference with us. Like I say, we'll put all of these show, all these links in our show notes and would really appreciate you finding us on, uh, changelog.com/community, connecting with us on LinkedIn and hope to hear about, uh,

• 38:44Daniel Whitenack?

  all of the things that you're finding interesting in the world of AI right now. And, uh, Chris, we'll talk to you later.

• 38:50Chris Benson?

  Sounds good, Daniel. I'll talk to you later.

• 38:55Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed this show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for change log is provided by Fastly. Learn more at fastly.com. And we catch our airs before our users do here at change law because of roll bar. Check them out at rollbar.com slash change log. And we're hosted on Linode cloud servers. Head to linode.com slash change log. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address,

• 39:39Speaker 00

  get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.