Escaping the "dark ages" of AI infrastructure

Evan Sparks, from Determined AI, helps us understand why many are still stuck in the “dark ages” of AI infrastructure. He then discusses how we can build better systems by leveraging things like fault tolerant training and AutoML. Finally, Evan explains his optimistic outlook on AI’s economic and environmental health impact. Sponsors: DigitalOcean https://do.co/changelog – The simplest cloud platform for developers and teams Whether you’re running one virtual machine or ten thousand, makes managing your infrastructure too easy. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. Featuring: Evan Sparks – Website http://etrain.github.io, GitHub https://github.com/etrain, X https://x.com/evanrsparksChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Determined AI https://determined.ai/Previous episode on Alpha pilot https://changelog.com/practicalai/59Blog post - AI Leadership And The Positive Impacts On Economy, Privacy, Environmental Health https://www.forbes.com/sites/evansparks/2019/11/14/ai-leadership-and-the-positive-impacts-on-economy-privacy-environmental-health/#2e0243a9327eBlog post - Announcing the future of AI infrastructure https://determined.ai/blog/announcing-the-future-of-ai-infrastructure/Apache Spark https://spark.apache.org/MLlib https://spark.apache.org/mllib/Antique Candle Co. https://antiquecandleco.com/Joel Grus on AI code that facilitates good science https://changelog.com/practicalai/51Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at ChangeLaw because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com/changelog. This episode is brought to you by Digital Ocean. Guess what? Digital Ocean recently added MySQL and Redis to their list of managed databases. Their full managed databases lineup now includes the three most popular databases out there for developers: Postgres, MySQL, and Redis. Eliminate the complexity involved in managing, scaling, and securing your database infrastructure, and instead get back to focusing on building value for your users.

• 0:41Speaker 00

  Learn more and get started for free with a $50 credit at do.co/changelog. Again, do.co/changelog.

• 0:59Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slag with us around various topics of the show at changelog.com/community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:24Chris Benson?

  Welcome to another episode of the Practical AI Podcast. My name is Chris Benson. I'm a Principal AI Strategist at Lockheed Martin. And with me as always is my co-host Daniel Whitenack, who is a data scientist with SIL International. How's it going today, Daniel?

• 1:39Daniel Whitenack?

  It's, uh, it's going really good. It's Thanksgiving week here in the States for those that are listening from the States. So a little bit shorter week. I'm working through Wednesday, so it's a good week and feel like I've been reasonably productive. So what about you?

• 1:55Chris Benson?

  Same for me. I'm, I'm working through Wednesday and, but I'm looking forward to having a long weekend ahead. Do you have any special plans for Thanksgiving?

• 2:04Daniel Whitenack?

  Well, just Thanksgiving dinner, but then I'm gonna help out my wife who has a candle business and Cyber Monday weekend is pretty insane for them. It's a company called Antique Candle Co. And they're gonna ship out a kind of insane number of orders. So I'll probably be packing boxes with candles, which will be a nice, you know, break from staring at a screen and something completely different.

• 2:28Chris Benson?

  No AI in that one?

• 2:30Daniel Whitenack?

  Uh, not as of yet, although, I mean, so I helped them with some like marketing and Facebook ad stuff. And obviously in advertising, it is interesting to kind of come from, from the AI perspective, because you see certain things like in Facebook ads where it's talking about, you know, optimization and, and learning as you kind of kick off the ad, there's like a learning phase where it's kind of figuring out how to optimize the placement, the placements and the, and the cost and all of that. And so, so it's interesting to think about it from that perspective, for sure.

• 3:02Chris Benson?

  Fantastic. Well, you know, we need to include a link to your wife's business in the show notes and

• 3:07Daniel Whitenack?

  I certainly will. Shameless plug.

• 3:09Chris Benson?

  There you go. And in case there are any AI people who want to jump into candles. So I guess for me, I'm just taking a breather. Next week, I'm at Carnegie Mellon University for an AI conference to do a panel and

• 3:19Daniel Whitenack?

  Yeah, that sounds great.

• 3:20Chris Benson?

  Yeah. And then finishing up the week with two things in a row. I'll be in Philadelphia doing an AI and ethics talk as a keynote at an ethics conference. And then I'm finishing Friday night in Austin, where the final Alpha Pilot, which we've had an episode on, the world championship race will be there. And at the end of that race that evening, we're gonna hand out a $1 million check to the winner.

• 3:44Daniel Whitenack?

  Exciting stuff.

• 3:45Chris Benson?

  Yeah, pretty big deal. If anyone's interested in hearing more about that, we have an Alpha Pilot episode from not long ago, and you're welcome to, to tune into that. But turning to today, we have a fantastic guest. We have Evan Sparks, who is the co-founder and CEO at Determined AI. Evan, welcome to the show.

• 4:07Speaker 03

  Thanks so much for having me, guys. It's a pleasure to be speaking with you today.

• 4:10Chris Benson?

  Pleasure to have you on the show. If you could just kind of start us off giving us a little bit of background about yourself, kind of how you got to where you're at at this point before we dive into determined AI.

• 4:20Speaker 03

  Yeah, absolutely. So as it pertains to my sort of career around machine learning and AI, I really kind of got my start in that space, kind of fresh out of college in quantitative finance. So this is in mid 2000s. I was working for an asset manager based in Boston where we were doing applied machine learning to the stock market to pick stocks and trade client portfolios.

• 4:41Daniel Whitenack?

  When I was in, I did my PhD in physics and this was like before the sort of data science hype, but the rumor I always heard for like people that got out of academia was there was like, oh, you can go do all this cool math stuff in finance, but I never quite figured out how to do that.

• 4:59Speaker 03

  Yeah, I mean, there definitely was a very common career path. And it's funny, you know, probably a few years later, everybody then went into ad tech or something like that. And that, and you know, now it's probably autonomous vehicles or something. There's always an interesting corner in a hot area to be, to be doing this stuff, which is one of the things that I find super fascinating. So there are a few years of that quant finance thing, found that other people really, you know, liked looking at PE ratios all day and that wasn't for me. I was much more interested in sort of the technology problems we were solving. Ended up going to work for a startup in the, in the NLP space called Recorded Future. So we were, you know, building, taking the web and throwing it through this massive NLP engine and building kind of structured data products based on it.

• 5:41Speaker 03

  And trying to figure out how we sell that kind of structured data to places like trading firms, but also like the federal government and so on. And ultimately that company sort of found a good niche in, in threat intelligence, basically trying to build predictive indicators of where cyber attacks are going to happen and so on. But again, with the same kind of data-driven sort of machine learning technology. And so, you know, In many ways, the roles were were pretty similar, one being kind of in financial services, but the other being in this kind of totally different startup kind of environment. But, you know, always building models and kind of driving kind of forward data products. And in both cases, though, I found I was spending much more time building and maintaining my own

• 6:23Speaker 03

  infrastructure than I was kind of worrying about the modeling problems. And so, you know, and it was really the case in those days, this is kind of like, figure around 2010, 2012, as Hadoop was kind of becoming popular and so on, where as soon as I was tasked with analyzing a data set that didn't fit in memory on my laptop, my world just like collapsed, right? And you were forced to like, figure out how to write mapreduce jobs and so on. And I took that as kind of a good signal to go back and invent the world that that I wanted to live in, in grad school. So I had the good fortune to join the amp lab at UC Berkeley right around the time that Apache Spark was born. And my co-founder at Determined AI, I mean, Tullacher and I got to work kind of right

• 7:06Speaker 03

  away, building out kind of the machine learning ecosystem around Spark. So we were among the designers and initial contributors to MLlib, which is the standard library for machine learning in that ecosystem. And the rest of my PhD was really kind of focused on how do we build sort of, how do we give people tools to build end-to-end machine learning applications and optimize them in, you know, a large scale and sort of distributed fashion?

• 7:29Daniel Whitenack?

  So this is a slightly less formal question, but it must have been a perfect fit in terms of of working on Spark and and being named Sparks. I'm assuming it, Spark was not named for Evan Spark.

• 7:44Speaker 03

  No, absolutely not. I, you know, it was funny, I I sat next to Mateusz Haria, who who was the creator of Spark in the in the lab, sort of, Spark was around 0.3 when when I when I landed in the lab. And the first like few days, we sat next to each other, there were these kind of like long, weird stares going on back and forth until finally, we kind of broke the ice and made it, made a joke about it. But, yeah, it was kind of fortunate coincidence from from my perspective, I guess. You know, there was a long-running joke that my real name was Evan Apache Sparks, but

• 8:18Speaker 03

  not so much. So yeah, it was good timing and honestly, the AMP Lab was a great place to be for what I wanted to study, which is really thinking about where does this intersection of huge volumes of data and machine learning really get real and how do we build out kind of supporting systems to enable this? And so also while at Berkeley, I met my other co-founder at Determined AI, Neil Conway, who's more kind of from the pure distributed systems kind of part of the world. So he'd been a Postgres committer, he was working at kind of core Apache Mesos for a while around kind of distributed resource management. Meanwhile, me, on the other hand, is more kind of dyed-in-the-wool theoretical ML student. And he's now a professor at CMU in the machine learning department.

• 9:02Speaker 03

  In some ways, you think of me as the person who takes what those guys do individually, figures out how to mash them together and then hopefully can figure out how to build interesting applications on top of kind of that intersection of system and machine learning. So while at Berkeley, and I promise this is getting into what we do here at Determined, while at Berkeley, one of the big things that we saw, the big mega trends that was happening within academia at first, was this shift to deep learning as a primary way that people wanted to be doing machine learning, particularly in industrial settings. And so it started with computer vision and speech and obviously more recently, we've seen amazing advances in things like NLP and text. And this meant people retooling, you know, learning how to use tools like TensorFlow,

• 9:45Speaker 03

  buying GPUs en masse, figuring out how to take this, what had been like a tiny corner of academic machine learning and really kind of make it into an industrially viable technology and stubbing their toe on a lot of serious problems along the way, right? So you go from, you know, logistic regression that trains on my Spark cluster in a couple of minutes to like, you know, big week-long training runs for large-scale image classifiers on a, you know, massive cluster of GPUs, for example. You start to have a lot of design decision baked into your modeling choices that you didn't have before. Things like, you know, just how many layers should this architecture have? How does the model capacity relate to my training data set and so on?

• 10:27Speaker 03

  And in ways that are sort of not really intuitive and end up being really highly empirical. So we saw that and we also saw that the frameworks, the TensorFlow and the PyTorch and so on of the world are really good at sort of their individual task they're tasked with, which is helping you describe what your model is and get it training on, say, a single or maybe several GPUs on a machine, but really bad at helping model developers through the rest of the workflow associated with getting one of these applications into production. And stuff that you guys have covered on your show before around data labeling and so on. We don't do any of that at Determined AI, but there are other pieces of the workflow around hyperparameter optimization, architecture search,

• 11:10Speaker 03

  getting your models to train really fast across a wide variety of different hardware platforms, dynamically managing resources in the cloud, say, so that you can, you know, pay for the GPUs only while you're really using them. All of that stuff is sort of handled right now on a manual basis, honestly with bash scripts and duct tape in many cases. And people don't really have a good way to support their more general workflows as they're in this model development process. And so at Determined AI, that's really the gap that we serve to fill. How do we enable you to do the rest of the pieces of your workflow while still using the tools that you know and love, your TensorFlow, your PyTorch, your Paris, et cetera, but make you much more productive as kind of an individual engineer,

• 11:54Speaker 03

  but more importantly, as a team of engineers? How do you share your results in a reproducible fashion? And how do you make sure that I can get the same model out of my infrastructure as you do? And so, you know, at Determined AI, that's really what our mission is.

• 12:07Daniel Whitenack?

  Yeah, so I'm curious to dive into a few pieces of that, but you mentioned in one of the blog posts on on the term AI about, you know, people still kind of living in the dark age of of AI infrastructure, where certain larger companies have built sophisticated AI-native infrastructure for their own use, but everybody else is kind of struggling. I'm curious if that sort of dark age that you're seeing is due to the fact that, like you say, that there's all these other pieces of the AI workflow that might be data preprocessing, model deployment, model optimization, all of these other things, data labeling, like

• 12:48Daniel Whitenack?

  you mentioned, is it that the tools for those other pieces of the workflow are not, there aren't good tools for those, or is it that, like, they don't play well together in a sort of all-in-one workflow, or just that, you know, like, people haven't developed, had enough time to develop standardized methodologies around these things? What do you see as kind of the main contributor there?

• 13:12Speaker 03

  Yeah, I think it's a it's a little bit of both. So I think that, you know, the, you hit the nail on the head with the, in many cases, there are individual tools and point solutions to some of the problems that you mentioned. So there are toolkits for model compression. There, you know, are services and and open source libraries for just hyperparameter optimization and so on. Even sometimes whole companies built around these things. But in our view, you know, what ends up being a result of that is that you get these tools that are isolated and don't, aren't designed to work well with one another. And more importantly, you then miss sort of broader opportunities that might exist around optimizing sort of

• 13:54Speaker 03

  the entire workflow. If you can kind of step back and look at that, rather than, you know, individually, like, how do I make this particular piece of the puzzle go absolutely as fast as possible? Sure, you eliminate that bottleneck, but you might still be completely bottlenecked on ETL or data collection or training time, for example. And so you have to be careful as an organization about where you're investing your time and your resources in terms of making those things better. We think that a more holistic design, that is, one where the pieces are kind of designed to know about each other, opens the door for certain types of optimizations. So to give you an example, we have our Resource manager that is built into our product at Determined AI is totally AI aware.

• 14:38Speaker 03

  It's it's aware of the fact that what you're doing with running your jobs on our system. All of the jobs are somehow related to training or running inference on deep learning models. And you can start to make a bunch of interesting assumptions about the workload that you couldn't if this was just general purpose compute. For example, the idea that these things are iterative and that they have intermediate state like model weights and and state of the optimizer that can be used to sort of checkpoint and understand where the computation was and then reschedule it, say, on to run on another device. Now, we have that kind of design in the in the resource management section, but then when we're designing our hyperparameter tuning algorithms, for example, and implementing

• 15:23Speaker 03

  them, we can take full advantage of knowing what that internal scheduling layer looks like and use properties of that scheduling layer that we couldn't if we were just running this as like a black box job on top of something like Spark or or Kubernetes or whatever. And that power of sort of these components being designed with one another in mind allows us to do this job much more efficiently, in a much more fault tolerant and resource aware kind of way than we would be able to otherwise. If you're spending ninety percent of your time kind of starting up the cluster and and getting it done, that's a lot of wasted cycles for your GPUs that your data scientists really want to be putting to work, you know, finding good models and solving their problems.

• 16:01Daniel Whitenack?

  So I'm curious, you mentioned kind of a more holistic view of AI infrastructure. And I know that something that can happen, because there are so many pieces to this, that you can end up in with scenarios in companies where you have a data engineering team or something that's in charge of all of this like preprocessing and getting data sets ready, and then you have like the modeling group, and then you have like deployment and app people, app integration group people. Do you see that trend disappearing as as things are kind of tighter, more tightly and better integrated together?

• 16:42Daniel Whitenack?

  Or do you think it's reasonable that, you know, a data scientist could take something all the way through that whole cycle, I guess could and should, should they be doing that?

• 16:53Speaker 03

  I think it depends on the on the company and kind of the the scale of the application that's under development. For example, if you're building a self-driving car, that's probably not a job for a single data scientist. I would really hope, right? You know, that's a it's sort of, I call it a generational moonshot, if you will. And there it makes perfect sense that you're going to have massive team of people just worrying about data labeling and data ingest and ETL, another set of people just working on kind of the perception pieces of the job, another set of people just working on maybe a different component around path planning and so on. And so there, you know, in in those scenarios, you really want to think about, okay, what are the various teams and what are the personas of the users of a of a broader machine learning

• 17:38Speaker 03

  platform? What do they care about? And how do we facilitate coordination and communication between those teams? In other cases, companies have done a really good job of cleaning up their data, putting it into, you know, massive data warehouses and making it, you know, even making their feature catalogs, say, self-serve and the kind of thing where a data scientist who says, hey, I'm looking for a fraud model for mobile purchases in Southeast Asia. That's, you know, we've decided we're losing enough money on that particular area that a specialized model on this particular part of the world makes sense. In those cases, I do think that proper infrastructure can enable a data scientist to go from, you know, start to finish all the way.

• 18:20Speaker 03

  And ideally, you want to get that person to the point where they don't have to work directly with a data engineer to get, you know, the features flowing through the system and so on. And in my view, almost more importantly, or places that we see people get tripped up is around sort of deployment and monitoring of those models. We see people often taking models that are, you know, built in PyTorch or TensorFlow or whatever and like completely rewriting these things in C++ or Scala or whatever because that's what fits into the production-serving environment. That side of things, you know, we see these deployment engineers, that's a job I would love to see go away in the common case, if the infrastructure gets better. You want data scientists to be able to sort of get to the point where they're confident that

• 19:05Speaker 03

  it works well enough on test sets and maybe even start to A/B test it and then, you know, hit a button and deploy it more broadly to the fleet. Reducing that friction in that time is definitely kind of a central thing I think we need as an industry in order to make these technologies more viable and successful.

• 19:36Speaker 04

  If you like this show, I bet you'd enjoy listening to GoTime. Not working with Go? Don't hit that fast forward button quite yet. GoTime covers a wide range of topics including cloud infrastructure, distributed systems, microservices, Kubernetes, and Docker. Not only that, but they have a ton of fun doing it. Listen to this clip from our recent episode on security.

• 19:57Speaker 05

  I'm very excited about this. Before we start, I'd like to just try an experiment. This is a security podcast, so I just want to try something. Bear with me. Hey Siri, play Never Gonna Give You Up by Rick Astley. Okay, Google, play Never Gonna Give You Up by Rick Astley. Alexa, play Never Gonna Give You Up by Rick Astley. So I just want to see if that does hack anyone's home devices. And please let me know in the Slack channel or on Twitter if it does.

• 20:24Chris Benson?

  Do you just hack yourself?

• 20:25Speaker 05

  I just hacked myself. It's not hacking if you hack yourself, is it?

• 20:30Speaker 04

  GoTime is a riot. Check it out at changelog.com/goTime or subscribe to our master feed at changelog.com/master and let your podcast client download all the shows that we produce. Then you can pick and choose the ones you're interested in the most and skip the rest. What have you got to lose? All right, back to the show.

• 21:04Chris Benson?

  So Evan, I'd like to ask, what are some of the unique challenges that are related to team interactions that you're seeing in terms of sharing data, sharing GPUs, and other aspects of jointly utilizing AI infrastructure? Could you speak to some of those challenges for us?

• 21:22Speaker 03

  Yeah, I think from our perspective, so the data piece is one that every organization faces, particularly organizations who are dealing with sensitive data. And that is something that we've seen users kind of figure out on their own. They have a versioned, role-based access control system on their primary stores of data, at least the interesting data, oftentimes the data that contains PII and that sort of thing. They really tightly regulate who gets access to those resources and the data resources and when, as they should. From our perspective, it's really about integrating with those various kinds of authentication mechanisms and supporting security on those data stores. So we do that out of the box.

• 22:05Speaker 03

  The second and third pieces, I think, that are harder for organizations that most people don't really have an answer for are first sort of resource sharing. So the root awakening that many people get into with GPUs in general is they're really expensive. And you know, you're talking about spending upwards of 150K on, say, a DGX1, which is, you know, one of Nvidia's latest servers filled with V100s. And you know, one of those might be good for like two data scientists, but in order to enable your team to really be productive, you need several of those kinds of servers. And we see people doing really immature things with these systems. We see people managing them with either like static allocation, meaning Joe gets GPUs

• 22:50Speaker 03

  1 through 4 on this box and Kyle gets 5 through 8, you know, kind of forevermore, or they've got like some kind of Google Calendar system set up. And this is some really sophisticated organizations that we run into where that's the way they're managing this expensive resource.

• 23:07Daniel Whitenack?

  Do you think that's just because of like the mixed background of people working on this sort of technology that a lot of people are coming from, you know, science or maybe non, non-computer science or non-software engineering background? Or do you think it's more than that?

• 23:22Speaker 03

  Yeah, totally. I think that is a, so that's a big piece of it. And honestly, people who are really good at thinking about convex holes and the right shape of your loss function and so on, probably shouldn't be wasting their time, honestly, thinking about like the right way to do resource management. That problem has been solved in a bunch of different domains. And we, you know, that should be a layer of abstraction. And that's one that we provide to folks. There are other solutions to this problem as well. Some of the cluster resource managers that I mentioned earlier, like Kubernetes, or we see people using, you know, queuing systems like Slurm from the HPC world. Those things all have their drawbacks. But, you know, in general, this is like, this is a problem that modelers don't want to be thinking about. And more generally, I think we need better, you know, abstractions for these folks.

• 24:06Chris Benson?

  So that's certainly a challenge. I mean, I've been at two large organizations, one that I'm still at, Lockheed Martin, where we have many DGX systems within the enterprise. And we are from a kind of an AI oriented, high-performance computing context, trying to make these resources as broadly available as possible. Kind of conceptually, how do you think about that? Obviously, you will see organizations that start off doing this, you know, you get a GPU and you get a GPU and all that. But that's not, that doesn't scale against the workloads that, you know, certain teams, they only need one GPU at a time, and it may not take very long. And others might need dozens for a much longer period of time and everything in between. Conceptually speaking, how do you approach

• 24:51Chris Benson?

  differentiating between users and the various differentiated workloads that they're having to contend with?

• 24:57Speaker 03

  We love to see people that try and plan for this sort of thing, right? They're trying to get a sense of, okay, I know I have this data volume coming in next year. I know roughly speaking, it's going to take me this long on this many GPUs to train my models. Let's set aside budget and bring those resources on-prem or secure them, you know, with with long-term leases on one of the cloud providers for the most part. Now, that does a good job at kind of helping you plan for your baseload, right? But then, as always, there's going to be things that come up, like towards the end of the quarter, or a new model family comes out, or a new project takes really high priority that you've just got to ship. In which case, we see, you know, real benefits to bursting onto cloud resources.

• 25:40Speaker 03

  And so, within the context of our system, that's a that's a core feature that we offer. We call it elastic AI infrastructure. And the basic idea is that if the system is configured and there's budget within the organization and so on, you can do that dynamic sort of provisioning of those those cloud resources, spilling work over onto them. We handled sort of the data transfer and other aspects of that planning for you. And then, you know, as the workload goes down, those resources are released and the organization can save money. So we think it's a it's a combination of, you know, having good planning, but also maintaining some flexibility in your systems and in your processes are required to really help AI scale within the within the enterprise.

• 26:20Daniel Whitenack?

  I know one of the things that I've talked to people about as they've talked about this particular problem is the fact that the data transfer, as you're trying to scale new like GPU nodes in the cloud or something, if you have to, you know, transfer 200 gigabytes of of data very frequently, that could be could be a downside. Are there ways around the sort of, you know, data management piece while still keeping things elastic?

• 26:48Speaker 03

  Yeah, so when we see people kind of in sort of hybrid cloud and on-premise environments, we like to take a look at what their their infrastructure is for for replicating that data. And we'd like to see it be sort of continuous, where the copy of the data that lives on the cloud and the copy of the data that lives on-premise are are maintained in a way that they're not exactly identical necessarily, but but very, very close. Or there is a path for them to be become identical very quickly. So that's sort of incremental. Process ends up being important. The other side of things I'd say is, you know, with all this discussion about just how big the data sets have gotten and how much data you need to fuel deep learning and so on.

• 27:26Speaker 03

  We are mostly looking at customers where like the upper bound on the size of the training set they're they're dealing with is like order of terabytes. And that is a lot easier to manage and transfer and move around. It's still hard. You don't want to do it 100 times a day or whatever. But it's easier than than say moving petabytes, uh, which is, you know, the the scales that that, you know, a lot of people are in the Hadoop space and so on. Uh, we'll talk about. And so that gives you a little bit more flexibility and and makes the data transfer being the big bottleneck. In our experience is often the exception, not the rule.

• 28:01Daniel Whitenack?

  Yeah. So i- it's good to hear that terabytes is small data now. It's only it's only big data when we get to to petabytes, I guess.

• 28:09Chris Benson?

  I I'm just kind of curious. Um, I'm pretty fascinated with kind of as you've taken us through the approach. I I'm curious, as you're looking out, uh, kind of at at the competitive landscape as as you see different organizations tooling up. Um, you know, everything from the giant companies like Google and Microsoft and Amazon and such to smaller startups in the space like you. How do you think about yourself in a competitive advantage mode? Like, what do you really think differentiates yourself from those out there? How do you think about that in your head?

• 28:40Speaker 03

  I think there are a couple of key things. One, we've got some pretty unique expertise on the team kind of in this space. These are problems we've been thinking about really deeply, uh, both in an academic, but also professional setting for collectively the team dozens of years, right? And we've got a track record of of delivering some really popular and influential technologies in the space. The other thing I'd say is I think the cloud vendors are there to build their platforms to help monetize their hardware. You know, the GPUs that they've invested in, they want to get people using and uh, and so on. And so all of it is, you know, Google pushing Google's cloud or uh, Amazon pushing their cloud and so on. Where we differentiate ourselves is by being really neutral to the vendor.

• 29:24Speaker 03

  We will give people access to the best, cheapest, you know, correct technology for their particular workloads. And you're already seeing signs of vendors getting kind of custom hardware for these particular tasks. So Google has TPUs now, Microsoft just announced a partnership with GraphCore. There are sort of, and you can bet that Amazon is working on AI-specific hardware. There are going to be a bigger menu of hardware choices to be available to help you solve these problems down the road. And we think that developers, in the same way they don't want to be worrying about like the resource management and the calendar system. They definitely don't want to be worrying about reprogramming their applications and figuring out

• 30:08Speaker 03

  which chip is best for this version of my language model and so on. And we think that layer of abstraction from a systems level can offer that kind of flexibility. So you submit your job to us, we figure out what the best hardware to run it on is, we go acquire that for you. Your job gets built and run, and then those resources are released. That basic idea, I think, is something that we can do and will be able to do better than the larger cloud vendors because we won't have these exclusive ties to one or the other.

• 30:35Daniel Whitenack?

  So that kind of leads me a little bit into a next question, which is around automation. And I guess more specifically around AutoML methods. So I see AutoML mentioned quite a few times on the site. And also, I mean, there's been kind of a general trend of sort of AutoML platforms being released, like Google Cloud AutoML or H2O Driverless AI. And there seems to be a lot of focus in this area. I was wondering, you know, it probably makes sense to people, like, one problem that AI people are going to have is managing their GPU infrastructure. But maybe people think that the like hyperparameter tuning and the modeling side of things is kind of their baby

• 31:18Daniel Whitenack?

  and they don't want to mess with things like that. What do you see as some of the major advantages of automating some of that piece of things and utilizing some of these AutoML methods to kind of automatically figure out architectures, automatically figure out the right hyperparameters or automatically do other things? What role do you see that playing in the sort of future of AI infrastructure?

• 31:41Speaker 03

  Yeah, so I think the way we think about it right now is that you've got these experts who are highly trained in their particular fields, you know, maybe they're really great at understanding the physics of solar flares or, you know, understanding how robotics works or whatever it is. And yet, they're spending a lot of their time doing highly tedious tasks. So manually, you know, hitting, looking, telling the end of the log files, figuring out what the loss looked like, deciding is this an area I want to keep investing in or should I try a radically different model architecture, that sort of thing. And then writing the same, you know, fifty nested for loops to tune over my parameters and over and over again.

• 32:26Speaker 03

  And when there are better algorithms out there for this stuff, either they don't know about them or they don't have time or interest in implementing them, but they don't quite realize, it's easy in the narrow, to miss the fact that much of this work could be, you know, totally automated away or at least partially automated away. And so our view is really, we want to give these practitioners power tools, right? Instead of saying like, we're going to build a robot that builds the house for you, let's take a carpenter and equip him with power hammer and the circular saw and so on. That's kind of the phase where we think that we're in with, when it comes to AI development. And so if you can equip experts with tools, again, new layers of abstraction that they

• 33:07Speaker 03

  can reason about, say, move from, you know, fiddling with the knobs individually to reasoning about search spaces and budgets around how many, say, GPU hours you want to put into solving a particular problem. And then letting the system pick the right algorithm for, say, hyperparameter optimization or the right way to approach that problem, we see really terrific gains. We've had customers tell us that they were able to replicate what had been, you know, a two-month process of manually tuning hyperparameters and selecting model architectures in a single overnight run of our system. And that's, you know, leveraging kind of best of breed algorithms from active learning, developed primarily by my co-founder Amital Walker, around hyperparameter optimization and architecture search.

• 33:48Speaker 03

  And that to me, sure, if you could do that, you know, fifty times a year, I'd be printing money right now. But even if you can save somebody a couple of months, a few times a year, that ends up being really powerful in the way that they get their work done and how quickly they ship their applications. And again, they start thinking about The data problems and the modeling problems that they have and not so much, how do I, you know, write out this infrastructure and that sort of thing?

• 34:13Chris Benson?

  So I know one of the things that that determined AI is working on has been, you know, a lot about making AI work reproducible and being able to track experiments. And, you know, within the larger body of literature in AI, we're always hearing about explainability and transparency and such as that in AI. So I guess what I'm asking is, you know, why do you think that this is important to have this reproducibility built into AI infrastructure going forward? What kind of benefits do you see it offering? And what do you think might be missing in terms of the things that we are tracking or parts of the conversation that haven't really been addressed yet?

• 34:50Speaker 03

  Yeah, I mean, I think that if you told a software engineer that their code wasn't going to be tracked and that, you know, even if their code was tracked, they were going to check it out from GitHub and try and build the system. And, you know, there was only like a 2% chance that they were going to get the same artifact out at the end of the day as their peer who downloaded the same repo that afternoon. They would look at you like you were completely crazy, right? But that is very much the state of reality in the world with, when it comes to machine learning practice. And it's because we have all this stuff under the hood that we need to track and get just right in order to get our algorithms to converge to the same level. Doesn't help that the optimization problems we're solving these days are non-convex.

• 35:32Speaker 03

  And so there's a bunch of stochasticity embedded in them and so on. But the idea that I need to collect and understand every randomness that lives anywhere in my system. I need to understand what are the right hyperparameters for this particular run? What are the settings of the optimizer and so on? And how is my model even initialized in the first place? Those are all necessary ingredients. I also have to keep track of what my data is. Now, once you have built a solution or a system for ensuring reproducibility across runs of different machine learning models, and this gets to your point of why it is important. Now you have the kernel of something that can be used to enable, like, very direct and repeated collaboration among.

• 36:16Speaker 03

  Data scientist, you can say, hey, download my my version of the model and you can reproduce it exactly. Okay, great reproducible, done. That's that's cool reproducible build. But now I can also use that to say, hey, why don't you extend my model? Try turning it on a different data set. Try running it on sixty four GPUs and make sure that it converges in the same way. And I can begin to sort of riff with my my colleagues on the next great idea. And I think that's sort of the dream. It's one thing for a single developer to be able to continue to innovate. But once somebody has a good idea, and now you can broadcast that idea to the entire rest of the the organization, and everybody incorporates that into their solutions, now you've got a flywheel going that can really help an organization

• 36:58Speaker 03

  accelerate. And, you know, again, we see these these kinds of best practices and properties emerging at places that are really sophisticated in in our AI infrastructure, the bigger, you know, companies, the Googles of the world and so on. But that hasn't yet hit the mainstream yet because our tools don't have support for that. And so that's one of the the main things that we try to drive at the term of AI.

• 37:20Daniel Whitenack?

  All right, Evan, I I'd like to kind of switch gears a little bit here. So we've been talking a lot about practical things around infrastructure, which I think is great because this is practical AI after all, and those things are super important. But I was also curious to hear some of your thoughts on another subject. I saw that you wrote a recent blog post about AI leadership and positive impacts on things like the economy, on on human labor, and other things. I was wondering if you could share a little bit about the motivation behind that article and why you thought some optimism needed to be brought into that conversation?

• 37:57Speaker 03

  Yeah, I mean, it's it's funny. The company is headquartered in in San Francisco, and and as I get outside of the San Francisco kind of AI bubble, or whatever you want to call it, you know, at dinner parties with friends outside of this this world, a common theme that comes up is, isn't AI all about automating jobs away? Isn't it all about taking away kind of my livelihood? And, you know, it's scary as we move into for people who are even in in skilled jobs, they're looking at, hey, Is your algorithm that is really good at tech summarization gonna replace the need for, you know, the training programs in my law firm of, you know, an army of freshly amended attorneys doing discovery work and that sort of thing, right? And, you know, the answer is, is right. It's like, it's maybe.

• 38:42Speaker 03

  But when I think about technology, I like to look back on kind of what has technology done for the economy over time and how has this story played out previously? So in the blog post, I, I use an, an example of, of how Japan recognized that their population demographics were gonna shift in the eighties and started plowing a lot of money into robotics. And of course, now they're, they're world leader in robotics. But it was in, in service of kind of planning for a world where the majority of the population was gonna be over sixty five, right? And building out infrastructure to support that. So I think that a similar kind of view needs to be taken of AI here. We look at the industrial revolution. We've been automating things for like a century and a half at this point. And probably longer than that, depending on how you wanna think about it.

• 39:26Speaker 03

  It always does lead to sort of short-term job displacement. But in the long run, quality of life and standard of, of living across the, the globe has risen dramatically. And so I think we kinda need to take that view on technology as a whole in that we have to be careful about what it does in the short term to people and making sure that we've got social policies in place to, to help folks out. But it's good to be optimistic. These, these technologies can be really, they can enable things that felt like science fiction ten years ago to be real, like the self-driving cars that we see off the streets and so on. But they can also, uh, really help, you know, in a bunch of ways that are otherwise unexpected around helping environmental health.

• 40:09Speaker 03

  We've got a customer in kind of the waste management space that specifically uses AI to help, you know, do recycling much more, more sort of, uh, effectively. We've also got, uh, we're working with folks in, in pharmaceutical drug discovery that are using AI to cure new diseases. And so there are, there are ways that, uh, that these technologies can be used broadly for the social good. And that was really the motivation behind this piece that I put together.

• 40:34Chris Benson?

  Yeah, that it's really great to hear that. Actually, I know just because as a brief tangent, Daniel and I are both very focused on using AI for good. And we talk a lot about it during various episodes. And Daniel is focusing on bringing making language availability with an AI, you know, more broadly available because there's so many languages out there in the world that are that are not getting attention from technology. And I focus on animal welfare issues and such. And so I love your optimism in this space. So I guess turning to the next thing is obviously with the potential for AI to continue to increase productivity at large, despite some of the bumps in the road, obviously for society that you already addressed.

• 41:15Chris Benson?

  And given the fact that there is a tremendous concern right now about privacy issues, how do you look at that dynamic tension between productivity and privacy? Are the two, are they always at odds with each other? Are they mutually exclusive in the context of AI? Or do you see a more optimistic path where you can be productive and and yield privacy at the same time?

• 41:37Speaker 03

  It's a really interesting question on a broad area. And with my kind of recovering academic hat on, I think it's a really interesting question from fundamental research where we can set up this, what you're calling perhaps, we can formally study whether there is fundamentally a privacy or product and productivity trade off. And first, we try and answer that question. And then if there is indeed this trade off, maybe there are ways that we can come up with that will give us precise control over that trade off as we make it. So an example I like to talk about is federated learning, where users could potentially remain completely in control over their data and it stays, you know, on

• 42:18Speaker 03

  their edge devices. And yet, the collective wisdom of all of the users through AI and things like homomorphic encryption and and so on, could be used to in a differentially kind of private way, help update models that globally make use of lots of users' data without leaking individually private sensitive pieces of information.

• 42:41Speaker 03

  I don't think this stuff has been completely figured out, which is why I think it's still a really interesting research area. But I'm I'm hopeful that as consumers demand that their data be kept private and and so on, which I think we're seeing a lot of and and look no further than GDPR and in the European Union. Uh, as evidence of this, that we will start to have to get clever with how we navigate that trade off space. Uh, and I'm really excited. Uh, you know, I I watch the the research coming out in the field pretty closely because I think there's some really exciting stuff happening.

• 43:12Daniel Whitenack?

  Yeah, I know that in the most recent versions of of TensorFlow and a bunch of other projects, there were very certain things around around privacy. And of course you have things like federated learning like you're talking about. I was wondering, kind of as we get nearer to the end of our uh, conversation here, in terms of practicalities for AI practitioners, whether that be someone that's, you know, working on some of their first AI projects, maybe as part of a startup or something, or a larger company. What are some of the best things that, you know, we could implement to help our workflow?

• 43:51Daniel Whitenack?

  What what's the biggest bang for the buck that we can do? Maybe that's looking into things like AutoML, or maybe that's implementing experiment tracking. Where where do you think people should start changing their workflows first to make the biggest impact?

• 44:05Speaker 03

  So I think if you're if you're kind of in the early days of of your project, um, and and just kind of getting your feet well with the technologies. My advice would not be to go try an AutoML solution off the shelf. It might work for you, but you're going to be in a position very quickly where you don't understand what's going on one layer of the stack beneath you. And as uh, as data problems come up, or the next model needs needs a new tweak to it, or something like that, you might be at a loss and might be at a place where you're stuck. Instead, what I tell people is invest heavily in your data production, tracking, versioning, to make sure that you're in a spot where you can go back and replay the

• 44:49Speaker 03

  past as it was exactly at that point in the past and build your models in that particular way. And begin to invest in tracking and understanding your workflows from a code and data and kind of models perspective. So that is some level of experiment tracking. The other thing I'd say is start simple. So start with the simplest model that could possibly work and solve your problem. And that will do two things. It will both, one, maybe your problem is really simple and you don't need, you know, a fancy 50-layer convolutional neural network with an LSTM bolted on the side to solve it, which is a good thing to learn. But at the very least, it gives you a baseline for, okay, this is the, you know, baseline sort of no signal to noise ratio kind of place I need to be.

• 45:32Speaker 03

  I need to make sure my models are at least as good as this. And it gets you in the habit of targeting a metric that you can use to evaluate whether or not your model is good enough. And I think that's a really important lesson for people getting started.

• 45:44Daniel Whitenack?

  Yeah, and I think those are amazing tips. In terms of like the experiment tracking one, I think you're right on the money. That's a huge benefit that people can have. For people that are maybe not coming from a software engineering background, in your experience, maybe they're not quite to where they're ready to invest in the full determined AI solution, but what would be some like practical ways for them to track certain experiment, you know, initially just a matter of sort of metadata and naming things correctly or getting into good version control habits with GitHub? Or where do you see people struggling the most? Or what are some simple ways that maybe they can benefit themselves?

• 46:25Speaker 03

  Yeah, so I, you know, I would say that for sure, get used to using software version control tools for your code and versions of the models that you got. For data, things like S3, for example, on Amazon, they offer a version data store. You can turn it on on your bucket and start using the version numbers as you're pulling data off of it. And then for the last piece, honestly, or a big piece is around metrics. And so that in the early days can be recorded through either some pretty ad hoc processes. So structured log files where you write down what you think are the key parameters of a particular experiment or run.

• 47:05Speaker 03

  So think of it as maybe a JSON blob that records, you know, the keys and values that you care about and store that somewhere where you you're sure you can get access to it and so on. There are also projects, open source projects out there like MLflow tracking, which which can help facilitate this and give you dashboards around this as well. And so that might be another place that I'd recommend people check out if they're interested in another open source option in this area.

• 47:31Daniel Whitenack?

  Awesome. Yeah, that's great. I should also mention Joel Gruce was on the podcast and I will link his episode in the show notes as well. He talked a good deal about responsible AI development practices, bringing some of that expertise from software engineering into the the AI research and AI development workflow. So we'll definitely link that.

• 47:53Chris Benson?

  And I guess to close out, for listeners who might not necessarily have all the the skills and infrastructure and backend engineering, and they're wanting to kind of level up, and they maybe they're even a little bit intimidated by kind of diving into this new area. Do you have any other any other ideas to close out with on how they can level up those infrastructure skills?

• 48:13Speaker 03

  There are a number of great, you know, online resources. It's funny, I've never really like thought about that side of things needing to be leveled up. In fact, that's kind of why we provide the software platform that we do to try and keep people from worrying about that.

• 48:28Chris Benson?

  That's fair enough.

• 48:29Speaker 03

  But yeah, you know, I think that the the various cloud providers do a good job of providing education around things like Kubernetes and so on that can be helpful as you're thinking about what's the modern way of of building out this infrastructure. But I don't have specific resource recommendations in mind right now.

• 48:46Chris Benson?

  No worries. Well, Evan, thank you so much for coming on to the show and telling us all about Determined AI and infrastructure and it was a fantastic conversation.

• 48:55Speaker 03

  Sure thing. Great. Great speaking with you guys. Thanks so much for having me.

• 49:01Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go on your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for change law is provided by Fastly. Learn more at fastly.com. And we catch our errors before our users do hear change law because of Rollbar. Check them out at rollbar.com/changelaw. And we're hosted on Linode cloud servers. Head to linode.com/changelaw. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address, get our weekly email, keeping you up to date with the news

• 49:46Speaker 00

  and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.