The landscape of AI infrastructure

Being that this is “practical” AI, we decided that it would be good to take time to discuss various aspects of AI infrastructure. In this full-connected episode, we discuss our personal/local infrastructure along with trends in AI, including infra for training, serving, and data management. Sponsors: DigitalOcean https://do.co/changelog – Check out DigitalOcean’s dedicated vCPU Droplets with dedicated vCPU threads. Get started for free with a $100 credit. Learn more at do.co/changelog https://do.co/changelog. DataEngPodcast https://www.dataengineeringpodcast.com?utm_source=changelog&utm_medium=podcast&utm_campaign=practicalai – A podcast about data engineering and modern data infrastructure. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We move fast and fix things because of Rollbar. Resolve errors in minutes. Deploy with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Our locally installed stuff: Jupyter https://jupyter.org/Docker https://www.docker.com/Python https://www.python.org/Go https://golang.org/Postman https://www.getpostman.com/Where we see AI workflows running: AWS https://aws.amazon.com/GCP https://cloud.google.com/Azure https://azure.microsoft.com/en-us/Kubernetes https://kubernetes.io/ and KubeFlow https://www.kubeflow.org/On-prem workstations: NVIDIA https://www.nvidia.com/en-us/data-center/dgx-systems/Lambda Labs https://lambdalabs.com/System76 https://system76.com/Experimentation / model development: JupyterLab https://github.com/jupyterlab/jupyterlabGoogle Colaboratory https://colab.research.google.comAWS SageMaker https://aws.amazon.com/sagemaker/Data Science platforms: Domino https://www.dominodatalab.com/DataBricks https://databricks.com/DataRobot https://www.datarobot.com/H2O.ai https://www.h2o.ai/Pipelining and automation: Pachyderm https://pachyderm.io/Airflow https://airflow.apache.org/Luigi https://github.com/spotify/luigiModel optimization: OpenVino https://software.intel.com/en-us/openvino-toolkitTensorRT https://developer.nvidia.com/tensorrtTensorFlow Lite https://www.tensorflow.org/liteServing: MXNet https://mxnet.apache.org/TensorFlow serving https://www.tensorflow.org/tfx/guide/servingSeldon https://www.seldon.io/Monitoring/visibility: TensorBoard https://github.com/tensorflow/tensorboardNetron https://github.com/lutzroeder/netronKnock knock https://github.com/huggingface/knockknockPrometheus https://prometheus.io/ElasticSearch https://www.elastic.co/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com slash change log.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front end web servers that need to be full duty CPU all day, every day. Then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co slash change log. Again, do.co slash change log.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slide with us around various topics of the show at change.com slash community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:47Daniel Whitenack?

  Welcome to another fully connected episode of Practical AI, where we keep you fully connected with everything that's happening in the AI community. We're going to take some time to discuss some things related to the the recent topics in AI news. And we'll dig into a few learning resources that are related to those to help you level up your machine learning game. So I'm Daniel Whiteknecht, data scientist with S I L International, and I'm joined by my co-host, Chris Benson, who is a chief AI strategist with Lockheed Martin RMS APA Innovations. How you doing, Chris?

• 2:23Chris Benson?

  Doing great. How's it going, Daniel?

• 2:24Daniel Whitenack?

  It's going really well. I'm, uh, I'm sitting in a newly remodeled home office, so I'm, I'm pretty happy. We got some final painting done and set up my monitor and, uh, new desk and everything, so I'm feeling pretty good. What about you?

• 2:39Chris Benson?

  I'm relieved to be home. Um, I've been traveling the last couple of weeks and, uh, uh, hit Washington DC, New York, and, uh, I was just in Silicon Valley as we record this for, uh, Nvidia GPU, uh, Technology Conference. And so back, recorded a couple of things there. I know last week we had, uh, uh, we had a guest from there, and there's gonna be some more down the road, so, um, I'm really looking forward to today.

• 3:04Daniel Whitenack?

  Yeah, me too. And I think it's kind of, uh, ideal that I just went through all of my personal setup here in my home office this week, cause, uh, you had suggested that we talk about a certain topic that I know is really on a lot of people's minds as they get into this field, and as they kind of try to figure out what to focus on as they're learning things and how to build a team. So you wanna intro what we're gonna be talking about today?

• 3:31Chris Benson?

  Sure. Uh, so today, we're gonna be talking about a fairly broad topic that we're labeling AI infrastructure, um, which encompasses a whole lot of stuff. Uh, and the reason that I had suggested it was, uh, I have so many conversations with people who are trying to kind of get their, their own AI operations set up, both at a, at a personal level, just like you and me as, as data scientists, uh, working on stuff, but also at an organizational level, trying to figure out how their company needs to get everything stood up that they need there to do what they're doing. So we're gonna talk about kind of a lot of the ideas. Uh, it's a huge topic, so there's only so much we'll be able to cover, but, uh, hopefully we can kind of dive into some of that stuff today and have fun with it.

• 4:09Daniel Whitenack?

  Yeah, for sure. I know that there's a lot of, when I do trainings and other things, I always get a lot of questions about, "Oh, how should I do my personal setup? What do I need to buy to actually be an AI practitioner on that side of things?" So the personal infrastructure side. But then also, there's like so many choices of things out there as far as how you set up your workflow and all of that. So just a disclaimer as we kind of go into this conversation, we'll probably be primarily focusing on a lot of the things that we have personally interacted with, but we would love to hear some of the infrastructure or the frameworks or the setup that our listeners have, or maybe that we're missing, or definitely if we misrepresent anything.

• 4:52Daniel Whitenack?

  So definitely do that. Join our Slack community. You can do that at changelog.com/community or on our LinkedIn page and let us know, you know, what we're missing or what your personal setup looks like. But we'll try to, you know, as we go to conferences and like I said, as we do trainings and other things like that, I think we've seen a lot of what people are doing out there. So hopefully we can convey some of that today and give kind of a landscape of infrastructure.

• 5:20Chris Benson?

  Absolutely. I think, and there's just a, there's so many different ways to put together infrastructure. There's so much choice. This field has just absolutely exploded in the last couple of years. When we were first talking about machine learning back in the, you know, when we first got to know each other and stuff, there just weren't the plethora of options that we have at this point. So we'll try to sort through some of that today.

• 5:42Daniel Whitenack?

  Yeah, so let's kind of jump in maybe with a general question and think about like, as AI practitioners, how much time do we spend kind of doing development, you know, on our local setup or our local machine or laptop versus in a cloud or hosted environment or kind of specialized on-prem hardware? So what's your experience with that, Chris?

• 6:05Chris Benson?

  So I know different people who do different ways, but I really focus on using cloud or hosted environments most of the time. I have friends and colleagues that have bought their own home equipment, you know, in terms of the different Types of GPUs that are available and you know, they can plug in graphics cards and that kind of thing. But from where I'm coming from, I don't tend to be the guy who's always out buying the latest new thing constantly. And this field is moving so fast that I have kind of chosen to opt out of buying my own equipment since I would constantly be replacing it. Because the new shiny thing would be out there. So if it's a trivial toy little thing like, you know, for demos or for teaching people,

• 6:44Chris Benson?

  then I might do something on my MacBook, just using the CPU, but that it has to be, you know, truly a tiny thing for that to be the case. Almost any other time, I'm either going to a hosted environment or a cloud environment.

• 6:55Daniel Whitenack?

  Yeah, I think that kind of how it splits up for me, or at least has, you know, in the past couple of years is, I do a lot of kind of the initial work to test my code and ensure that it actually runs. So, you know, deal with a lot of those issues, maybe deal with some like data formatting or data preprocessing, or kind of looking at example data, making some example API calls, figuring out how to deal with that data in a Jupiter notebook. All of those sorts of like, kind of initial things, a lot of those I still do locally.

• 7:29Chris Benson?

  Actually, I do as well. I was only thinking in terms of training when I said that, so I should have been more clear.

• 7:34Daniel Whitenack?

  Yeah. So then, of course, like you said, you know, at a certain point, you know, you're limited locally, but also you need to scale things up. And, you know, we've, we've said a lot of times here that AI doesn't really do you any good. You know, if it just stays on your laptop, it has to get out there and be practical. So I like to make that jump. I think, you know, maybe a good way to put it is to make that jump to a production-like environment, whatever that's going to be, whether that's going to be somewhere in the cloud or on on-premise hardware or whatever, make that jump as soon as you reasonably can, without, you know, wasting much time. That's kind of my viewpoint on that.

• 8:11Chris Benson?

  Yeah, I would agree with that. I mean, there's so many, as you pointed out, in my brain, as I answered that last question, I was kind of jumping straight into training on a GPU or TPU. And in that case, I move it off my Mac pretty quick. But for the vast majority of the data prep, which is the most To the work, getting everything, you know, ready for training and, you know, pulling data in and massaging it and doing all the things you have to do so that it is ready for that. Most of that I do on my Mac unless we're talking about, in some cases, the data sets are simply too big and then I'll offload it to a server, not necessarily a GPU or a specialty, you know, something like a DGX, but to some other server just to crank away while I do other stuff.

• 8:51Daniel Whitenack?

  Yeah, and a lot of the stuff, you know, if you're gonna be running a training for a model for, you know, five hours or 12 hours or whatever it ends up being, it's just simply not practical to do that locally. But like you said, that's actually probably the proportionality wise, the smaller amount of things that that an AI practitioner would do. The majority of things are, you know, figuring out what data to use and figuring out what format it's in and then getting it, you know, engineering some features or trying out some certain things, making sure your code runs before you, you know, spend up for GPU time in the cloud or something like that.

• 9:31Chris Benson?

  Agree. It's funny that, you know, we like to think of ourselves as AI practitioners and yet the training piece of that, even though training itself may last quite a while as you're doing that, in the scheme of a project, it's a very small amount of time that you spend.

• 9:45Daniel Whitenack?

  Yep. So let's think about with that in mind, I guess, you know, one of the questions that I get a lot when I'm going around doing trainings and other things is hey, do I need to invest in some sort of GPU workstation for my home office or a really expensive laptop with a GPU or something that can be there in my office? Obviously, I think neither one of us have that situation, so maybe that's the answer. But I think if there's people out there that are wondering, that's not really necessary at this point. So I think you can use a little bit cheaper hardware to do your local development as long as you're able to connect to the APIs and

• 10:25Daniel Whitenack?

  the UIs and, you know, open up a terminal and connect to the instances that you're running elsewhere.

• 10:32Chris Benson?

  Yeah, I have a rule of thumb on that and the way I do it, so I have both, initially, you know, we've talked a little bit about this in previous. Things that we do at a personal level that we're interested in personally, separate from work. And then I have the work things. And because I work for Lockheed and have previously worked for other large companies. Since I've been in the AI space, I have resources there where they are dedicated equipment. In general, people say, hey, should I be buying a DGX workstation? Or should I buy some graphics cards? I really say it comes, there's a crossover point where it depends on how much you're training. If you have enough going in your operation to where you are really needing training cycles kind of around the clock.

• 11:13Chris Benson?

  Then, you know, and that's more than just personal projects, obviously, that's at work or a team of people.

• 11:18Daniel Whitenack?

  Yeah, that's going to be in a corporate setting.

• 11:20Chris Benson?

  Right. Then it can make sense to buy your own equipment because, you know, there's a big investment that you're making. But then you're utilizing that equipment constantly. And so that it makes sense, but for for most of us who are not doing that kind of around the clock operation. I think, you know, going with cloud providers is probably the way to go because you can just use what you need, pay for that bit. And then move on without continuing to. So, but if you were training around the clock. Then there's a crossover point where cloud providers can become more expensive than actually making that investment yourself.

• 11:52Daniel Whitenack?

  Yeah. And I mean, if you're, especially if you're looking for a job in AI. Or if you're getting into AI, or even once you have a job in AI. Most of the time, any of that specialized hardware would be purchased. It would be purchased by your company to enable things for your team.

• 12:08Chris Benson?

  Correct.

• 12:09Daniel Whitenack?

  You're never going to have to invest personally. At least you don't have to. I mean, you could if you really want to work on some crazy personal projects, but you don't have to do that. And so just to kind of be transparent, you know, as a data scientist. My personal infrastructure basically looks like, and by personal infrastructure, I mean, my local setup just looks like a MacBook. You know, without the goofy touch bar thing, because that's, that's weird. I have to have an escape key, I'm sorry.

• 12:37Chris Benson?

  There you go.

• 12:38Daniel Whitenack?

  You know, an external monitor, I get a nice keyboard. That's essential for me, that mechanical keyboard makes typing a joy. But then I don't really have a ton of stuff even installed locally. So I have the native Python or brew installed Python. So I don't use like Anaconda or any of these kind of loaded sort of package managers. Those can be really nice for a lot of people. I just don't find it as nice for me personally. But there is some advantage to that. I use a really simple, I just use a VEM ID and I have Jupiter and Docker installed locally. I use things like Postman for testing API calls and Go occasionally. You know, and Slack and Zoom and all the web conference stuff because I work remotely.

• 13:23Daniel Whitenack?

  So that's kind of what I prefer. Again, all of those are personal preferences. I know a lot of people that find a lot of value in kind of these environment managers like Anaconda or maybe other ways of doing things. That's just not what I do. But what does your, your personal setup look like, Chris?

• 13:39Chris Benson?

  Sadly, it's much like yours. So I won't go through everything, but yeah, I'm on a MacBook. Like you, I also like Ubuntu separately if I'm on a server. Standard kind of MacBook setup with an external monitor, keyboard, trackpad, nothing, nothing fancy for me. I also brew install Python as well. I've had trouble with Anaconda when I tried to change use cases around and it somehow would have start throwing errors. So I found that to be simplest. For deep learning, I'm always starting in a Jupiter notebook and hoping that it's successful enough to migrate later down the road out of that Jupiter notebook into code as a library. Docker a lot, I mean Docker is, I'm so glad Docker was, came along before the AI explosion

• 14:21Chris Benson?

  happened because utilizing it with containers has made the world of AI training and deployment so much easier. And then I know, like you, anyone who's listened to us knows we both love Go. I use Go as my default go-to language. I use Python for the data science things that tend to be Python specific. And I don't have a GPU. I actually, I have like a TX2 that I play around with and I'm about to get a Nano from Nvidia. But you know, those are. Mainly for my toy projects and stuff for any training I am going to typically if either whatever my company has to offer and we have stuff within Lockheed Martin that I can use for work. If it's on my own, I'm going to like AWS SageMaker and Google Colab and so that gives you a sense.

• 15:03Chris Benson?

  And we can talk in both about some of the those in more detail as we go forward here.

• 15:07Daniel Whitenack?

  Yeah, so I think moral of the story, I mean, you don't need a fancy computer, even a MacBook. We have those, but if you just have a cheaper notebook, that's fine as well. Cause you know, a lot of the things that you'll probably be doing are hosted. There's no need for that specialized hardware. There's probably is a lower limit to that. One time my, my wife borrowed my MacBook for a week and I tried to, to go for a week on a, on a Chromebook. And there was a lot of pain there, although I think that's getting better too. I think, you know, people like Kelsey Hightower and others, you know, develop on a Chromebook.

• 15:41Chris Benson?

  Yeah, I know. He loves it. I know from his Twitter feed, but I, I remember when you did that because as an aside, we tried to record an episode at that point. I know you were having some technical struggles with that and I've also struggled, but that I remember very distinctly. You trying to work through that.

• 15:57Daniel Whitenack?

  Yeah. And again, you know, these are personal preferences. So we'd love to hear what your guys setup is and what value you find from things. But I'm kind of moving on from our personal local setup and talking about, okay, from our local setup, that's pretty, maybe pretty light, at least a lot of people's local setup is kind of light. You know, what are the things that we're connecting to that are hosted or in the cloud? What enables our AI workflows, the things that we use that don't run on our book? And to kind of switch to that topic, I think probably there's, there's something we need to talk about first, which is there's an infinite number of ways that we could, that we could enable AI workflows in the cloud or on prem.

• 16:41Daniel Whitenack?

  And a lot of that's going to be driven by the organization that you're working for and what their concerns are. So the first of those may be being kind of governance issues. So I know probably with you working for Lockheed, Chris, there's There's a lot of things related to that, but I think anywhere now there's going to be a lot of governance-related issues.

• 17:01Chris Benson?

  Yeah, you know, if it's just you, you know, as we're talking about ourselves individually, you just kind of plug in whatever tools you like and you create your workflow out of that which is available. And there's different ways of doing that and it's pretty simple because it's really based on your personal preference. But as soon as you get to even a small team level and certainly as you get to multiple teams across an organization, it gets pretty complicated pretty quick and you have to start thinking about all the issues that go into making something that will work, not only technically for you as a data scientist doing the work, but also accommodate the various laws, regulations, you know, things like GDPR in Europe

• 17:42Chris Benson?

  that have to be accounted for and how you do different types of workflows and testing. And some of those topics that, you know, just off the cuff, and we can dive into a few of them, or things like data discovery, how are you going to know what's available to you beyond, you know, just things that your own team might be producing? What are some of the trust and certification issues out there? Provenance and lineage, you know, and I know that when you were in your previous position at Packard, I know provenance is one of the features that you had there and maybe talk a little bit toward that. But data management, who owns data? What's the sovereignty of the data? How do you access the data and the different tools in the workflow?

• 18:23Chris Benson?

  And what kinds of data science processes do you build around that? So that's a lot to think about, you know? What about you, Daniel? You know, if you want to dive into a couple of the things that are of interest to you there?

• 18:35Daniel Whitenack?

  Yeah, I mean, I think the big one, maybe a big one to emphasize is just, you know, all of the problems in this area in my mind and all of the major blockers that I've hit as a practitioner are mostly having to do with the data side of things, not whether I had the compute power to train a specific model or something. So for example, there's always issues, you know, if you're using someone else's data, there's privacy issues depending on if it's personally identifiable. Identifying data. Of course, if you're working in healthcare, there's issues about, you know, even where you're allowed to store certain data and whether that is on-prem,

• 19:17Daniel Whitenack?

  at a facility, in the cloud. There's a lot of issues there. And so I think the main issues are that you're being, some of those data sources that you might like to use might force you into using certain infrastructure, might force you into staying on-premise in your own company's infrastructure, or it might allow you to be in the cloud. And even if you are in the cloud, you might still have to maintain certain audit trails and that sort of thing for the data that you're using, especially if you're using data that's been generated by EU citizens and all of that sort of thing with GDPR.

• 19:52Chris Benson?

  Yeah.

• 19:53Daniel Whitenack?

  So I think all of this is, that's really, in my mind, the major factor.

• 19:56Chris Benson?

  Yeah, I would agree with that. And that was a great point you made in that just the laws and the way laws will affect your own organization's strategy in terms of where you're housing data. And I've actually seen over the last couple of years, you know, as people were prepping for GDR and then it came into being, I've had conversations with people where they chose where to keep data from a nationality standpoint. And so they might literally relocate their operations from a, in terms of the, the training and the data storage and stuff, into a completely different country to accommodate those laws. And also to figure out with different countries having different laws, how are they going to approach that from a strategy standpoint? You know, where should those operations be? So it can, it can increase your cost on having enough equipment to go to different places

• 20:40Chris Benson?

  and to have to think through that. So there's multiple rabbit holes here. And I know we're just kind of skimming over the top of some of these issues on this episode. And we can certainly do episodes and actually have done some in the past where we kind of do some deep dives. I'll kind of leave it there and let us kind of proceed on down the infrastructure route.

• 20:58Daniel Whitenack?

  Yeah, I think the main takeaway is before you decide on a specific, oh, I'm going to run in GCP or I'm going to run in AWS or I'm going to run on-prem, you need to have a process in place that'll help you decide what are the data concerns related to where I am allowed to store data, how much data I can share, even within your, your own organization.

• 21:29Speaker 03

  The Data Engineering Podcast is a weekly deep dive on modern data management with the engineers and entrepreneurs who are shaping the industry. Go behind the scenes on the tools, techniques, and difficulties of data engineering so you can learn and keep up with the knowledge to make you and your business successful.

• 21:45Speaker 04

  Can you give a bit of an outline about the motivation for choosing Jupiter notebooks in particular as the core interface for your data teams?

• 21:53Speaker 05

  Yeah, and actually, when I first joined Netflix, it was a sort of toss at me. And I was definitely like, are we crazy? And the answer was like, we might be a little crazy.

• 22:02Speaker 03

  Go to dataengineeringpodcast.com to listen, subscribe, and share it with your friends and colleagues.

• 22:23Daniel Whitenack?

  So we've kind of been through so far, talking about, you know, our personal setup, what an AI practitioner might need locally, and then what organizational concerns might go into choices around whether your AI workflow runs in the cloud, or where you store your data and that sort of thing. But let's go ahead and, you know, jump into how AI practitioners are running their AI workflows in the cloud or on-prem. What are the sorts of frameworks and infrastructure and tools that they're using to actually enable those AI workflows? And so, again, this is from our personal experience and, you know, what we've seen

• 23:05Daniel Whitenack?

  other people doing and what we've done ourself. But, you know, maybe we can start with what sorts of resources do we need in terms of compute and storage? So what sorts of resources do you need to run your AI workflows, Chris?

• 23:20Chris Benson?

  Well, kind of going back to a little while ago, I'm very focused on Docker, just because it makes it a lot easier. Having said that, I do keep TensorFlow installed locally, but since I'm not running on a local GPU, I'm not sure that I necessarily need to do that. I think for any real workflow I do, I have a Docker container and I know for especially at work, we have a specific production containers that we use for our workflows there. And so we'll pull down one of those. There are a lot of options on that. I know Nvidia, for instance, has a whole bunch of production workload containers that you can use as a base for your company. And this is assuming that you are running on Nvidia equipment in that case, that are already optimized for that. And so for me, it's really easy to

• 24:03Chris Benson?

  grab one of those production containers and then do the customization I need, add my model into it, figure out how I'm going to get the data into that for training. I've done that a couple of different ways over time, depending on what the resources available to me are. But when I'm serious about doing work and I'm not just playing around, I'm starting Docker from the get go.

• 24:23Daniel Whitenack?

  Yeah. And what I kind of often, so I love using Docker as well, how I kind of think about the layers that I need to deploy to enable my AI workflow. A lot of times what I'm running will be in Docker. But then, under the hood or a layer down, I like to think of kind of two primary types of resources that I need, and those being compute and storage. So if you just have compute, you might be able to run your Docker container, but then you're going to have, where are you going to put your data? It's not so great to put 200 terabytes of data in a Docker container. I don't know that anyone's actually done that. Although, some people have put, you know, like to put data in containers.

• 25:04Daniel Whitenack?

  But so I like to think of kind of under the hood or a layer down, we need kind of two sets of resources, those being compute and storage. Now with compute, of course, you have some choices as far as whether that's going to be in the cloud or on-prem, similar to basically any engineering workflow that any company does. And then, storage wise, I think mostly what I've interacted with is pretty agnostic to my AI workflow. A lot of times, you don't have the choice of where your data is stored. You might be working with a production MySQL database or Postgres, or you might be working with data that's just dumped. To an object store like S3 or something like that.

• 25:48Daniel Whitenack?

  So typically, the storage options in my cases are oftentimes driven by things already existing within a company. So you might not have a lot of say in that. But then, compute wise, maybe you do have a little bit more say in that. So when you're running those Docker containers and that sort of thing, you mentioned that, Chris, sometimes you run on Nvidia hardware. So when you're saying that, I think what you're meaning is kind of like on premise.

• 26:12Chris Benson?

  Correct.

• 26:12Daniel Whitenack?

  Nvidia workstation. So how is that different from running on a GPU like in the cloud? Could you kind of go into that a little bit?

• 26:21Chris Benson?

  Sure. So I mean, it really comes down to the constraints, as you said, that you have. If you're in, I don't have a particular preference, but let's say AWS and you're using SageMaker and you're pulling your data out of S3. You know, it is what it is in terms of that's the service they're offering. It's a great way of doing it. But to contrast against that, we have a lot of DGX equipment at Lockheed. And my previous employer had DGXs too. And that means that you're running, you're cutting into a data center where you have the DGX set up. And then you have a set of equipment with storage and such around that to enable your operations. So, you know, it's great if you have a DGX two that you're operating on, but you're going to need the storage around that to pull from and to push out to. There may be some processing around there.

• 27:06Chris Benson?

  So you end up essentially creating a whole build around your DGX to enable those operations. And so it's not so different really from the cloud environment. You know, either way, you have storage, you're pulling data from it, you're running it through, assuming that it's been pre-processed and it's ready for training. And then you got the output somewhere. And so, and then you have to have access to all of that from wherever you're coming in. So the AWS or Google or Azure world, each has their own ways of doing all those pieces. If you're running your own data center, then it really depends on the company. Whereas I've worked for two companies with DGX equipment that I was able to use. That piece of it was the same, but how they built around the DGXs was different in both companies.

• 27:51Chris Benson?

  So just because you have the luxury of buying an AI supercomputer like that, doesn't mean that your setup is going to be the same. It's very distinct on how your organization wants to configure it.

• 28:01Daniel Whitenack?

  Yeah, I think your experiences are almost on a total different side of the spectrum from mine, probably. Only because, you know, right now I'm working with a nonprofit, so I think this is a good, a good contrast. So obviously, the companies that you've worked for and do have, you know, embedded AI research teams, maybe they invest in some of this Nvidia hardware. But for me, doing sort of machine learning and AI data-related work with a nonprofit, I would basically be left out of every room I was in if I, if I tried to get anyone to buy a 200 grand Nvidia box. So I've pretty much relied on everything in the cloud when I've needed it.

• 28:43Daniel Whitenack?

  And I think that wherever you fall on that spectrum for your AI team, there is a route forward. So it's great if you're able to afford that kind of dedicated hardware and you have that commitment level within your organization. But maybe you're not at that point yet, or you're just, you know, you're a startup trying to get into this space or another organization that doesn't have a huge AI team. You can do very similar things in the cloud. So every cloud provider has instances that are available with specialized hardware. They also have a lot of services that are, you know, that will allow you to spin up clusters, to do distributed computing, you know, like Kubernetes clusters. And there's frameworks on top of that that we can talk about in a bit.

• 29:25Daniel Whitenack?

  So I think wherever you fall on that spectrum, there's a route forward. But in either case, you're going to have some number of compute nodes on premise or in the cloud that maybe some have just regular CPUs, maybe some have GPUs. And then you're going to have some storage that is storing the data that you're working with for your training data sets and that sort of thing. So now thinking about like, let's say that's the base. So you've figured out whether you're going to be in the cloud or on premise, whether you have dedicated hardware using the cloud stuff. What do AI people actually run on top of this compute and storage infrastructure?

• 30:03Daniel Whitenack?

  So maybe let's first think about what do people run on this for kind of model development and experimentation? So What's your experience there, Chris? What are some of the, maybe you mentioned like notebook environments like Jupiter? Do you run those off of your laptop? Do you have experience hosting those within your infrastructure for model development? Or how does that work for you?

• 30:25Chris Benson?

  So often I will start locally. So it kind of depends on how I, at this point with me being more Docker focused, I found that it's easier to go ahead. And in the beginning, I used to open up a Jupiter notebook locally, but then I had to package it up and go put it into the Docker. And I've gotten to where I've, I just start off with a Docker container these days because there's a little bit more to do in that slightly than just opening up a notebook. But that way I don't have to do the, you know, package it all up later. It's easier because once I get into my workflow, I can start just building. And then when it makes time, I can run the container on the infrastructure. And so for me, that is personally, that's an easier way to go. There's also the, how do you set up the resources? What do you want to select?

• 31:08Chris Benson?

  I've used Domino Data Labs and that is enabling you. It kind of gives you a very nice front end when you have different types of equipment out there. It's something obviously you don't need. If you're, if you're in a cloud environment, then you have those interfaces that you're going to be used to from that provider, whichever one you want to choose to do that. But Domino kind of gives me that on the front end if I have our own infrastructure backend. And then, you know, at that point, it's just moving it, you know, scheduling it and moving it over there. And again, that there's, there's a lot of variability on how you want to do that.

• 31:41Daniel Whitenack?

  Yeah, I think there's a, I think you're right, Chris. I think that there's, that variability also falls onto a spectrum. So we talked about the spectrum of like the hardware that you might use that all this is running on in terms of, you know, specialized hardware versus things available in the cloud. I think there's a spectrum here too. There's a lot of, on one side of the spectrum, there's a lot of open source free tooling that will allow you to do kind of interactive model development and run it on pretty much any hardware, like in the cloud, on Kubernetes, in Docker, on prem, things like Jupiter Lab. It's like Jupiter, but multi-user Jupiter. So you can have multiple Jupiter. kernels and all of this stuff and run a lot of different notebooks.

• 32:24Daniel Whitenack?

  But there's also other free options. There's Google's Colaboratory or co lab, which has a bunch of kind of free GPU resources and other things and notebooks that you can manage. There's things like binder that will spin up Jupiter notebooks from a GitHub repo. So that's kind of one side of the spectrum where you're using a lot of these free kind of environments. On the other side of the spectrum, there's kind of data science platforms like you were talking about, Chris, which are things like Domino and Data Robot and Databricks H2O. Some of these are not free. In fact, some of them are not very cheap. Some of them are a little bit more moderately priced, depending on how you, how many users you have and what workloads you're running.

• 33:09Daniel Whitenack?

  But a lot of these kind of give you, like you were saying, a really nice interface, maybe to track your data, track different experiments that you're doing. My experience is that a lot of them are centered around the idea of experiments and running experiments and iterating on those experiments. They're not necessarily meant for running production AI services, but very much for model development and experimentation.

• 33:33Chris Benson?

  Yeah, you know, by the way, as an aside, while we're talking about some of these different providers, I've noted over the past year in particular that there's a real battle between the cloud providers to draw in entry level students in this AI world in terms of doing your initial training. Because any course you may select is going to have a cloud provider that you get used to. So for instance, deeplearning.ai, which is, you know, uses Coursera, is right now, they have the Coursera for the classroom stuff, but they're also using Google Colab in the current set of courses. And that way you kind of get used to that environment. So you get a little bit of buy in. If you're taking an Nvidia Deep Learning Institute course, they're using their own Nvidia GPU cloud.

• 34:18Chris Benson?

  You know, we tend not to talk about that. We tend to talk about Microsoft, Google and AWS in general. But Nvidia has theirs and there are other providers out there as well. So there, as we're talking about what you're buying into, that very well may be impacted by how you were trained and what your comfort level is.

• 34:35Daniel Whitenack?

  Yeah, for sure. Yeah, I know that I'm very comfortable with certain things that other people just don't like and laugh at me for using. So I think that you have to experiment as well and find where you're comfortable. So, okay, so we have the base of compute and storage. On top of that compute, we are running certain things for experimentation and model development, like somewhere on that spectrum of open source, sort of notebooky things like Jupyter Lab, or maybe less open source or not open source things like Domino for data science platforms. So the next thing that we might want to run on top of the

• 35:14Daniel Whitenack?

  compute and storage is some way to automate model training and the preprocessing and post-processing of data. So automatically, when new data is brought in, you might want to update a training data set, retrain your model, export a serialized version of that model, and then export that into some serving framework. So this is typically called pipelining and automation. There's a whole lot of tools for this. I'm a Pachyderm user, and I work for them full-time for a while, so I'm definitely biased in that way. And I love Pachyderm. But that's certainly not the only thing you can use. There's things like Luigi and Airflow that are commonly used for this. I don't see quite as much Hadoop and Spark stuff

• 35:59Daniel Whitenack?

  going on these days. But I don't know, what is your impression of the landscape and where things are headed with this side of things?

• 36:06Chris Benson?

  Well, you still, in the enterprise, you're still seeing a lot of Hadoop, especially Spark more so in the enterprise environments. And I don't really, even though I'm in big companies now, I really come from smaller companies. And so it has been interesting. I almost bypassed those particular technologies along the way, and then I've kind of come back. And so as you're in a large organization, you do have to accommodate those in those data flows, given the choice. I mean, and this is, I'm probably heavily influenced by you in terms of liking Pachyderm for that. Obviously, I think you mentioned Cubeflow is another tool that is used, and that one's good. Kubernetes, everything, from my perspective, because it kind of, one, I don't have to think about that too much anymore and just

• 36:51Chris Benson?

  say, let's go that route. I know Pachyderm is built on top of Kubernetes as well. How about you?

• 36:56Daniel Whitenack?

  Yeah, so I mean, I think you're right. Obviously, people have invested a lot in Hadoop over time. And so you have to deal with that in certain cases. And this is another one of those concerns that we were talking about that might drive your infrastructure choice. If you have to write on top of HDFS and have a ton of stuff written in, you know, Hive queries and all of that, you might be stuck with using that for whatever reasons. You know, thankfully, I kind of have some flexibility in my projects. So I like to do things a little bit differently, but there's a lot of choices out there. But I think in general, kind of circling back to kind of how I have this architecture laid out in my head, you've got the compute and storage, you've got those experimentation pieces on top of that, maybe like JupyterLab or something.

• 37:41Daniel Whitenack?

  Then you have some type of automated, like non-interactive tool that will allow you to automate the retraining of your models or updating of data sets or updating of databases that drive certain services or that sort of thing. And that, if it involves large data sets, it might involve distributed processing. If it involves model training, it might involve, you know, specialized hardware. But, you know, having a pipelining tool, something like Airflow or Pacaderm, Qflow, Spark, these sorts of things will allow you to update those large data sets over time. But that's definitely not the end of the story. So let's say that we're updating our model over time.

• 38:22Daniel Whitenack?

  How are we going to then serve that model on top of, so maybe we've trained it, now we want to use that serialized trained model to run many, many inferences. How are people doing that?

• 38:35Chris Benson?

  So it's kind of funny, leading into that, that's the part that most people don't think their way through all the way is, is how do you get to deployment? How do you make this thing that you've created actually work in real life with the rest of your software and hardware out there? And so some of the things that that need to be thought about there are what technologies you could argue, because I use TensorRT. And, you know, you have to be thinking about how you're serving. And there are different approaches to that, as well as something that's often forgotten. In this space is C I C D, which is continuous integration, continuous deployment. We're so used to in the software world, thinking about that. But the kind of the data science world and the thinking that dominates the AI space

• 39:19Chris Benson?

  often forgets that altogether. Or in some cases, you'll find data scientists who aren't really familiar with it at all. And they they're so used to doing things that are going to stay on a server, you know, or website, or something that's internal to an organization that now that we are getting to where you have AI models being pushed out for inference on the edge all over the place and eventually may far outnumber things even in the servers as we move forward over the years. That's going to be really critical. So, you know, things like thinking of your way through things like TensorFlow serving, you know, TensorRT and different things like MXNet has an approach to doing that as well. What about yourself?

• 39:57Daniel Whitenack?

  Yeah, I mean, the I think what you want to be thinking about is, yeah, you have that period of experimentation and model development and that may be local, that may be in the cloud, but ultimately you're going to want to run an automated, non-interactive pipeline that takes in some amount of training data, trains your model using some framework, maybe TensorFlow or PyTorch on a workstation or on a cluster like a Kubernetes cluster, and then outputs a serialized version of that to some resource, like maybe an object store like S3 or something like that. And then you may have a this may be plugged into C I C D in some way where, you know,

• 40:38Daniel Whitenack?

  maybe I push new model training code to GitHub and a C I C D tool like Jenkins is listening on that GitHub repo, knows that my code has changed, automatically runs my pipeline to retrain my model. That model is retrained as soon as it's retrained, then there's a call made to update the model that's being used in maybe TensorFlow Serving or another serving framework like MXNet or Seldon or something like that, or maybe your own custom service that's using that that model, something that other software engineers have written or something like that. Or it could even be a JavaScript app or a mobile app that is updated based on that.

• 41:19Daniel Whitenack?

  I think the more pieces of that that you can automate, the better in my experience, assuming that you're going to be wanting to update those things fairly regularly.

• 41:29Chris Benson?

  Yeah, I think to kind of package the last couple of minutes up, the takeaway there is if you're a data scientist and you may be very comfortably starting off in your Jupiter Notebook world where you're creating a model that has to go be trained, but at some point it has to come back to software. At some point, it has to be packaged up into a software package via a deployable code in the programming language that you or your organization is using for that. You know, I know you and I have a bias toward Go, obviously C, C++, Java are all really popular in that space. You have to, that model though, has to become essentially a microservice for inferencing to where it's a, it's a library, it's in

• 42:12Chris Benson?

  a larger architecture, and it is, whether it be a server that is supporting your business operations, or whether it be in the, you know, billions of IoT devices that we're going to have out there, or, you know, whether it be in your, on your phone, or our future mobile devices that we haven't gotten to yet, that's where it's going to live, and that's where it's going to be doing the thing that you're creating it for. And so thinking of that end, right, you know, early in the process is really crucial. It only matters, this thing you're working on only matters if it's usable in the real world out there as a piece of software.

• 42:47Daniel Whitenack?

  I think that's a great way to, to close things out. Obviously, there are a ton of things that we did not have time to talk about, so please join our Slack channel at changelog.com/community. We'd love to chat with you about the sort of infrastructure related things and practical things about your setup. Before we leave, I just wanted to share a couple of relevant learning resources. We always like to give some learning resources for people that are wanting to level up in the areas that we're talking about. So we've mentioned Google's Colab a couple of times. I would highly recommend, if you're just wanting to experiment with TensorFlow or PyTorch on GPUs and on TPUs, or these sorts

• 43:30Daniel Whitenack?

  of specialized types of hardware, you can do that for absolutely free on Google's Colab. There's a ton of examples. There's a great intro video. To that. Also, there's Intel's AI Dev Cloud. I've played around on there a good bit. That's a great place to experiment with hardware other than GPUs like Intel's Xeon processors. They have optimized really fast versions of like TensorFlow and PyTorch that you can use there. And without a lot of commitment, so just jump in, try some examples. A lot of these frameworks and tools that we've mentioned have great examples that you can try out with relatively low cost or no cost in some cloud environment. So try some things out, get your hands dirty,

• 44:14Daniel Whitenack?

  and let us know what you build.

• 44:16Chris Benson?

  Well, sounds good. Thanks for hopping on this episode, Daniel. This was a really good conversation. And I think actually we'll probably have some spinoff episodes of deep diving in some of the the topics that we've hit today that we just didn't have time for.

• 44:29Daniel Whitenack?

  For sure. Yeah. See you next week, Chris.

• 44:31Chris Benson?

  Take care. Bye bye.

• 44:35Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoy the show, do us a favor. Go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do here at changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode Cloud servers at linode.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address.

• 45:17Speaker 00

  Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.

• 45:45Speaker 00

  Because you've stuck in here to the end of this show, we got a special surprise for you. This is a preview of our new upcoming show called Brain Science. This podcast is for the curious. We explore the inner workings of the human brain to understand behavior change, habit formation, mental health, and the complexities of the human condition. It's hosted by myself, Adam Stokoviac, and my good friend, Muriel Reese, a doctor in clinical psychology. It's about brain science applied, not just how the brain works, but how we apply what we know about the brain to better our lives. There we go. So where do we begin to understand the mind? Humans have brains with all this neural activity. And I'm just thinking about what I know about my brain. I understand that it's up there, what it is.

• 46:29Speaker 00

  I understand it's very important to me and without it, I couldn't function. But you know, my mind isn't my brain's activity. How can we begin to break down the brain and the mind to really understand the operations behind our mind?

• 46:44Speaker 06

  Well, one of the things that is really important when we're looking at the brain and the mind is actually the words that we use to describe different things. And so I think it's really important to be as clear as possible. And so I think we want to differentiate the brain from the mind. And the brain is made up of different structures. And then the mind is sort of the inner workings of the physical structures, which is not observable. But when we're looking at the brain, there's some primary structures that are fundamental to being human. And that involves sort of three different brains. Well, we have the brain stem, the limbic brain, and the prefrontal cortex. I know it might get a little petty in talking about some of these things, but.

• 47:30Speaker 06

  I think it's helpful when we can have a visual. So if you put your right hand up in the air like you're being sworn in with all five fingers next to each other, go ahead and

• 47:38Speaker 00

  Okay.

• 47:39Speaker 06

  Fold your thumb across the palm of your hand and then close your four fingers over the top of your thumb.

• 47:46Speaker 00

  Okay, I got that.

• 47:47Speaker 06

  And so in order to correlate these with different structures, your wrist would be synonymous with your brainstem, which is the reptile brain. Then your thumb is the limbic brain or mammalian brain, which means all mammals have that part of the brain. And then your four fingers are what we refer to as the frontal lobe or part of the free frontal cortex. Okay, so we sort of have three brains in one and all do different things in our brain to help us be able to live and move and be safe.

• 48:19Speaker 00

  So if we have three brains in one, they all have their different roles. It sounds like, you know, the reptilian seems, I don't know, like it can't think very well. When it comes to the reptilian brain, I'm assuming it's just sort of like, you know, gut reactions, you know, very, very quick thinking, you know, almost subconscious kind of stuff potentially. Is that right?

• 48:39Speaker 06

  Yeah, you're spot on. Sometimes I think, again, it's helpful to parallel things with what we do know and do understand. So thinking of different animals, reptiles, right? Lizards, turtles. So the brainstem is really only responsible for these key functions within the body. So breathing, heart rate, the essentials and fight or flight. If a lizard is afraid, right? It needs to figure out what it needs to do to survive. So the brainstem is just preoccupied with the function of survival. How do I not die? And then if we move up to that mammal brain, right? We can think about, you know, cats or dogs, bats, and

• 49:20Speaker 06

  that mammal brain or limbic brain is really the feeling center of our brain. There's two key brain structures as part of that. And that is involves the amygdala and the hippocampus, which is responsible for memory. The one thing I think is super fascinating about the mammal brain is really the way in which we bank memories whenever things have the most emotion associated with it.

• 49:47Speaker 06

  We're more likely to remember that. Okay, so it doesn't matter whether it's positive or negative. So be it a wedding, a birth of a child, you know, or something super traumatic. Our brain goes, oh, that's so important to remember. It vacuum seals it so that we hold on to that. And so this is why, too, our lives have different meaning. And being able to to feel is a fundamental part of being human. The mammal brain is really the feeling center. So as opposed to more of the fight or flight from the reptile brain, our mammal brains, they're still more of unconscious, subconscious things. But imagine that the Dewey Decimal System of your brain sorts things according to feelings when we're mammals.

• 50:31Speaker 00

  That's a preview of Brain Science. If you love where we're going with this, send us an email to get on the list to be notified the very moment this show gets released. Email us at [email protected]. In the subject line, put in all caps, "Brain Science", with a couple bangs if you're really excited. You can also subscribe to our master feed to get all of our shows in one single feed. Head to changelog.com/master or search in your podcast app for Change Law Master. You'll find it. Subscribe, get all of our shows, and even those that only hit the master feed. Again, changelog.com/master.

• 51:25Speaker 00

  I want to be remembered for my gourmet line of frozen seafood dinners.