Pachyderm's Kubernetes-based infrastructure for AI

Joe Doliner (JD) joined the show to talk about productionizing ML/AI with Pachyderm, an open source data science platform built on Kubernetes (k8s). We talked through the origins of Pachyderm, challenges associated with creating infrastructure for machine learning, and data and model versioning/provenance. He also walked us through a process for going from a Jupyter notebook to a production data pipeline. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean is simplicity at scale. Whether your business is running one virtual machine or ten thousand, DigitalOcean gets out of your way so your team can build, deploy, and scale faster and more efficiently. New accounts get $100 in credit to use in your first 60 days. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We catch our errors before our users do because of Rollbar. Resolve errors in minutes, and deploy your code with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Linode https://linode.com/changelog – Our cloud server of choice. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2018. Start your server - head to linode.com/changelog https://linode.com/changelogFeaturing: Joe Doliner – Website http://joedoliner.com/, GitHub https://github.com/jdoliner, X https://x.com/jdolinerChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Pachyderm http://pachyderm.io/Pachyderm on GitHub https://github.com/pachyderm/pachydermPachyderm tutorials https://pachyderm.readthedocs.io/en/latest/examples/README.htmlDoD challenge built using Pachyderm http://xviewdataset.org/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Beta for change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at change law because of roll bar. Check them out at rollbar.com and we're hosted on Lino servers. Head to linode.com slash change log. This episode is brought to you by digital ocean. They now have CPU optimized droplets with dedicated hyper threads from best in class. Intel CPUs for all your machine learning and batch processing needs. You can easily spin up their one click machine learning and AI application image. This gives you immediate access to Python three, R, Jupiter notebook, TensorFlow, side kit and PyTorch. Use our special link to get a hundred dollar credit for digital ocean and try it today for free.

• 0:45Speaker 00

  At the do.co slash change log. Once again, do.co slash change log.

• 1:03Speaker 00

  Welcome to practical AI, a weekly podcast about making artificial intelligence, practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show at change.com slash community, follow us on Twitter, we're at practical AI FM. And now into the show.

• 1:29Daniel Whitenack?

  Well, welcome to practical AI. Hey Chris, how's it going, man?

• 1:32Speaker 00

  Pretty good. How you doing, Daniel?

• 1:34Daniel Whitenack?

  Doing, doing really good. I'm really happy today with, with the conversation that we're going to have because we're going to be talking to my, my old colleague and still great friend, Joe Doliner, or as I call him JD, welcome, Joe.

• 1:49Speaker 02

  Hey Dan, it's great to be here. And hey, Chris, it's great to meet you on your show.

• 1:52Speaker 00

  Great to meet you too.

• 1:54Daniel Whitenack?

  Yeah, thank you so much for, for joining us.

• 1:57Speaker 02

  Thank you for having me.

• 1:58Daniel Whitenack?

  Yeah, why don't you give us a little bit of background about what you're, you're currently involved with and, and how you got there?

• 2:04Speaker 02

  Yeah, absolutely. So as you said, I'm Joe Doliner. Everyone calls me JD. I am the CEO and founder of Paketerm, which is a company that builds data science tools that we'll be talking about today. Before that, I've worked at a number of startups. Probably the most relevant one to this conversation is that I also worked at Airbnb as a data infrastructure engineer, basically just managing their AI and data infrastructure for the company. And so I have a lot of experience on the infrastructure side of data science, less so as an actual practitioner. And so that's most of what we're going to be talking about today.

• 2:43Daniel Whitenack?

  Awesome. Yeah, that's a perfect setup. I think that we've done a lot of talking about AI, but we really haven't got into a ton of infrastructure stuff yet, I don't think. Have we, have we, Chris?

• 2:53Chris Benson?

  Not really. And it's, I think this is an episode long overdue. And it's, it's just to note to the listeners, I know you had said that you had previously worked with JD at Paketerm. I have not been familiar with Paketerm as a newbie, so it'll be an interesting conversation for me having a couple of experts on here. And I'm gonna, I'm gonna, I'm gonna ask all the stupid questions, okay?

• 3:15Speaker 02

  Well, you know, he's not my inside man. Dan, Dan might be, but Chris definitely isn't.

• 3:23Daniel Whitenack?

  Yeah, yeah. Full disclosure, I might be a little bit biased, but only I don't officially work for Paketerm anymore, although I am a huge fan. I am actually using Paketerm on my current project. So I'm a huge fan and have that bias, but I'm, I'm excited to dive into the details and and have you learn a little bit more too, Chris.

• 3:42Chris Benson?

  Yep, absolutely. Over the time that we've known each other since we first met and, and you've been talking about it, I've adopted it. I have a long way to go to catch up to where you guys are in terms of using it as a tool. But as a beginner, it's definitely something I'm interested in. So I can't wait to hear more from JD.

• 3:55Daniel Whitenack?

  Definitely. Yeah. So with that, JD, why don't you give us just kind of a high level overview of what Paketerm is and kind of the, the needs that it's fulfilling or what it's trying to do for data scientists and people working in machine learning and AI?

• 4:11Speaker 02

  Yeah, absolutely. So Pachyderm is basically designed to be all, everything that you need to do high-level production data infrastructure in a box. And so what that means, if you're used to, you know, doing AI workloads in, you know, Jupyter Notebooks on your laptop, or maybe just in Python directly, using something like TensorFlow, something like that. Pachyderm is not in any way saying that you should stop doing that. Pachyderm is just giving you a way to take that code and deploy it on the cloud in a distributed fashion so that you know it's going to run every single night, or, you know, hook it up with its processing steps so that you can have everything sort of going in a pipeline end to end. And this is what companies turn to when they sort of need

• 4:53Speaker 02

  to make that leap from a model that's on somebody's laptop to something that's like a core part of their business that's going to run every single night. This sort of all came out of my experiences at Airbnb, where I was basically trying to make a platform that did that for our data scientists. And while I was working there, I had a couple of sort of novel ideas for what I thought that the world of data infrastructure was missing and what I wanted to bring to it. So the first really unique thing that we did with Pachyderm is, you know, we needed a way to store data. So we have a distributed file system. It's called the Pachyderm file system. If you're familiar with the Hadoop ecosystem, this is probably something pretty similar to HDFS or Tachyon or something

• 5:33Speaker 02

  like that. What's different about our file system is that it's capable of version-controlling large data sets in addition to storing them. And so you can have, you know, your training dataset, it can be terabytes of data, and this data is constantly coming in from your users on a website, from satellite imagery or something like that. And the Pachyderm file system will actually give you discrete commits, like in Git, where you can see, okay, this is what my training dataset looked like a week ago, this is what it looked like a month ago, and things like that. And what's really important for AI, that is, not only do we keep these different versions, but we actually link them to their outputs using a system that we call provenance. And so at any

• 6:14Speaker 02

  time when you've trained a model in Pachyderm, you can ask the system, what is the provenance for this model? And it'll trace you back to all of the different pieces of training data that went in it into it, and all of the different pieces of code that went into training this model, so that you can, you can basically see where it came from, and you can reproduce your results. Does that make sense to you guys?

• 6:34Chris Benson?

  It does. I'm going to dive in since I'm the newbie on this and ask.

• 6:38Speaker 02

  Please do.

• 6:38Chris Benson?

  So kind of, and I'm asking this on behalf of the listeners and partly for myself. First of all, a quick question, is it a proprietary system? Is it open source?

• 6:49Speaker 02

  This is all open source. We do have an enterprise system that goes on on top of it. And I'll talk to you later about what what features are limited to the enterprise system. But nothing that I've talked about up until this point is in that. This is all open source. You can download it yourself.

• 7:03Chris Benson?

  Okay. And to kind of wrap our heads around it a little bit, you kind of mentioned the file system inversioning and this, what sounds like a kind of a feature called Providence, where you can go back and do that. Could you kind of describe for someone who has never heard of Packarderm, kind of what the feature set is and what kind of a typical use case might be? So that in their own shop where they're doing data science, they can kind of figure out how it fits in with what they're already doing.

• 7:28Speaker 02

  Yeah, yeah, absolutely. So I think it's, I think it's easiest to sort of focus in on a use case here. So one that I can talk about very publicly, because it was a public competition, was the Department of Defense was until recently running a competition where they were basically having people write image detection algorithms for satellite imagery that they had. Right? So they had a bunch of satellite images that they had taken and they wanted people to write models that would detect this is a hospital right here. This is a school. This is a bus, things like that. Interesting AI problem, also an interesting architecture problem for them, right? Because they have people just basically throwing code at them through this web interface.

• 8:11Speaker 02

  And they need to take that and run it through their pipeline and get results out the other end and give those to the users. So the way that they set that up in Packarderm is first they spun up an instance of it. And they deployed it on AWS. They used as the backing store, they used S3. So ultimately, all of this was stored in object storage, which made it very, very easy for them to manage. And then they loaded all of the satellite images into the Packarderm file system. And so that's, you know, you can, you can get stuff in there in a number of ways. You can get it in there directly from object storage. You can push it over HTTP. I'm not sure exactly which one they used. But from there, they now had a system where all of the data was just sitting there in different versions. They could update it and have a new version. And then anytime that a user's code came in, they just deployed a new pipeline on Pachyderm.

• 8:56Speaker 02

  And that would then slurp up all of those images and process them in parallel. And out the other end, after some processing would come just a score report that they could report back to the user. And that might include your code failed on these five images, so you don't get a score. Or it might be your code succeeded on these five images and here's how accurate you were. And it would get them full reports about like, here's what you did well on, here's what you didn't do well on, things like that. Does that answer your question? Or do you want to know more about sort of specific features within Pachyderm?

• 9:22Chris Benson?

  No, that that does help a little bit. I guess, I guess as a follow up, you talked about file system and its ability of versioning. Are there any other kind of high level, like key things that that you want to name that that you really can't use Pachyderm without considering those features?

• 9:36Speaker 02

  So in terms of the file system, that that really basically covers it. It does basically all the standard things that you'd accept from a distribute, expect from a distributed file system, plus the versioning and provenance component. And that's really the only quirk to it. Now, on the processing side, things also start to get interesting. And here is where we need to start introducing maybe a few jargony words that I will explain. So one of the sort of key things that we use in Pachyderm is containers. And I'm sure most, most listeners at this point have heard of the company Docker, which has been a very successful Silicon Valley company. And they make this thing called a container, which is basically just a standard way to ship around code, right?

• 10:18Speaker 02

  Think of the, the problem that you've had where, you know, you write some script in Python that trains a model. Then you send it over to your friend and they've got the wrong version of Python, or they've got the wrong version of TensorFlow installed or something like that, and it's all incompatible. A Docker container is a way to ship code that's going to work anywhere, regardless of what the user has got installed on their machine or regardless of it in the cluster. Packet processing is all built on Docker containers. And so what that means is that you as a data scientist, when you want to productionize your code and take it off of your laptop and into the cluster, then all you need to do is package it up into a Docker container, which means that there's a little bit of a learning curve there to understand the tooling of Docker.

• 11:01Speaker 02

  But once you've got that, you as a data scientist are now completely in control of the environment that your code runs in and all of the dependencies and everything like that. And so, once people grok this, it's actually very, very liberating. And the reason that I wanted to build this on top of containers was because when I was at Airbnb, we would have these problems all the time where a data scientist would come to me and they'd written some new piece of processing that they wanted to be in the company's pipeline. Could be a machine learning model or could just be something as simple as data cleaning or something like that. And they would send me the Python script and then I would realize, oh, this isn't quite compatible with what I've got on the cluster. And we didn't have Docker containers there. We just had one big monolithic cluster. And so if we didn't have the right versions of Python installed,

• 11:46Speaker 02

  I actually would have to either redeploy the entire cluster just to run that one user's code, which was very untenable, or I would have to have them change their code to use different versions, things like that. And so it was this constant back and forth where the data scientist couldn't quite use the tools they wanted. Our infrastructure people couldn't quite like maintain a cluster with a consistent set of tools. And so I had this aha moment when I realized if these guys could just use Docker containers, then this impetus mismatch would totally go away and we could both do our jobs a lot more easily. Does that make sense, Chris?

• 12:16Daniel Whitenack?

  I was just gonna say it like, following up on that. So it's kind of like the, whether you're using Python or R or Java or whatever the different tool you're using is or the language you're using, essentially these containers unify the way that you treat each processing step. Would that be an accurate way to see it or to say it?

• 12:37Speaker 02

  Absolutely, yeah. So it allows us to basically handle the infrastructure the same way, no matter what code it's written in. And so we have a lot of companies where one of the things that's really appealing about PacketDerm is that they, all of their data scientists just know different languages. And they're looking for some sane way to have everybody writing code in their own language and pull it tie it all together into a system that they can understand. And Pachyderm allows them to do that. Now, the key thing about this, of course, is that because we have the provenance tracking, like you can still see the fact that like, oh, this data came followed through all of these steps and came out the other end, even though like one step was Python, one step was Ruby, one step was Java, one step was C++ and you didn't have to write any special tooling within those languages to track the data.

• 13:22Daniel Whitenack?

  Yeah, that's awesome. So I'm going to kind of pose a problem. I want to see if you would kind of go about things the same way as I would JD. So let's say that we have a, you know, we have a Jupyter notebook and I like how you kind of brought that up before, because that's where a lot of data scientists kind of start out. So let's say that Chris and I have been working on this Jupyter notebook that has some preprocessing for images and then we train a particular model, let's say in TensorFlow, and then we output results and then maybe do some post-processing. And to test it out, we've just kind of downloaded like a sample data set of images locally. And then we've kind of proven that, yeah, this is like a good way that we think we should do this kind of in this Jupyter notebook.

• 14:07Daniel Whitenack?

  So in order to for us to kind of get that scenario off of our, you know, off of our laptops and into into Pachyderm, what would be the things that we would need? What would be the steps that we should do, both on the data and the processing side?

• 14:24Speaker 02

  That is a great question, and I think will be a really illustrative answer. I'm going to sort of try to answer this with rather than jumping straight to the like, so here's the nth state of this, where I think it's like you're using all of the Pachyderm features. I'm sort of going to like build it up piece by piece, which is how we recommend data scientists to do it. So the first kind of problem that you need to solve when you want to put a Jupiter notebook into Pachyderm is the fact that Jupiter notebooks are meant to be interactive, right? They're meant to have a user like opening up the browser and actually like clicking the run button and stuff like that. And so the first thing that you can do is you can actually run sort of Jupiter inside of a Pachyderm service and you can just run Jupiter notebooks all by themselves, but they can't just turn into a pipeline that runs without any human intervention, right?

• 15:09Speaker 02

  Because Jupiter isn't designed that way. And so

• 15:11Daniel Whitenack?

  Like in an automated and triggered sort of way?

• 15:13Speaker 02

  Right, right. So the first step to do is just to extract the code from Jupiter. I'm pretty sure Jupiter makes it very easy to export as a Python script at this point. And so you would do that and then you would put that in a Python container with whatever dependencies you need. And to start, I wouldn't even tease apart these different steps, the preprocessing, the model training and the post-processing. You could just do all of those in one container and you wouldn't even necessarily need to parallelize the data. Because if it was running on your laptop, it could probably run on a BP EC2 node as well. And so that process, I think, would take you, you know, if you had packet of setup to begin with, you could probably do that in 20 minutes.

• 15:54Speaker 02

  And then you would have gone from a system that you can run manually on your laptop and edit to a system that now runs every single time a new image comes into the repository or you change the code or something like that. And so, and also, of course, now it's deployed on the cloud. So you can easily throw a GPU in there if you want. You can easily throw more memory at it and stuff like that. And so, so now you have sort of the first step of a productionized pipeline. Now, the next step is figuring out which of these steps does it make sense to tease apart? So that maybe their outputs can be used by other steps, you know, in the future, you might want to do the same preprocessing and then train multiple different models and then do the same post-processing on them or something like that.

• 16:33Speaker 02

  And so I would separate out the preprocessing step, the training step and the post-processing step into their own individual pipelines. And so now I've got a chain of like three steps and each of these is doing something different. And now I get the opportunity to sort of optimize each of these steps individually, right? So the preprocessing step, for the most part, the preprocessing steps that I've seen can be done completely in parallel, right? You're doing things like cleaning up the images. You don't need to see all of the other images to clean up one image.

• 17:07Chris Benson?

  Parallel as far as like in the sense of distributed processing, like processing things in isolation.

• 17:14Speaker 02

  Exactly, exactly. So that's another of the like, sort of important, important things that we get from a container is that it's very, very easy for us to scale that up. Right? So you can say, I need to process all of these images. Here's a container that does it, but don't just spin up one copy of this container. Give me a thousand. And so you're now cranking through a thousand images at the same time rather than one. And so you'll get done much, much faster and you can handle much, much bigger loads. So I would do that with that step. The training step, training, making, making training happen in parallel is definitely a much more complicated question than making something like preprocessing happen in parallel. So normally we would still keep that as a, a non-parallel thing because your code needs to see all of the data to train on it.

• 17:59Speaker 02

  If that is not true, if you really want to start parallelizing that, that is when you want to start looking at things like Kubeflow, which we integrate with as, as you know, Dan, although we're, we're still working on making that integration better. And then the last step, the post-processing step, that one could sort of stay as is unless you were anticipating having a lot of things that you wanted to post-process in parallel. So for example, when the DOD did their pipelines, theirs is all designed around the fact of we have one data set, but we have, you know, thousands of different people submitting models that they want to get tested. And so actually the post-processing step could be pretty expensive because they were just doing it for so many different entries. And so that was happening in parallel as well. From an infrastructure perspective, that's basically the idea of these pipelines is

• 18:43Speaker 02

  that when you segment these steps off into little pipelines, you then get complete control over the infrastructure on a pipeline by pipeline basis. So you get the ability to say like, this one needs to run in parallel with a thousand copies of the container up and each of those containers needs to have a GPU accessible to it and this much memory and stuff like that. And this one over here is not doing it really much at all. So like, it just needs one container and we'll fit that in somewhere. And the system sort of automatically figures out how to make all of this work with the resources that it has.

• 19:12Chris Benson?

  Okay. Hey JD, that was a great explanation. As a beginner, I have a few questions I'd like to follow up with. First of all, you mentioned Kubeflow, so I take it that Kubernetes is part of the architecture that you're deploying onto?

• 19:26Speaker 02

  Yes, I guess I jumped the gut on that one mentioning cube float before Kubernetes. But yes, this is, I think, now when we need to bring in one more jargony word, and this will probably be our last infrastructure jargony word, which is Kubernetes. If you've heard of Docker, you've probably heard of Kubernetes as well. Actually, at this point, I think if you install Docker, it just has Kubernetes built into it. You should think of Kubernetes as kind of the puppet master for your containers, right? So a container is a really, really good way to deploy a single piece of code, like a program. It's literally just a process inside of a box. To deploy complicated distributed applications, you need to deploy a bunch of programs on different machines and make sure that they can all talk to each other and that they have the right resources and everything like that. And that's the piece that Kubernetes handles.

• 20:11Speaker 02

  So Kubernetes allows you to speak in very high-level terms that were a lot of the terms I was talking about, Pachyderm, speaking in of basically being able to say, I want you to make sure that there is a copy of this container running somewhere. You have 1,000 machines, you have the code to run. Just make sure that this is always up somewhere and I can talk to it consistently when I hit like this IP address or something like that. And Kubernetes will figure all of that out in the background for you. And, you know, it can be instead of one copy, it can be 1,000 copies and they can have specific infrastructure requirements like GPUs and stuff like that. And Kubernetes just solves all of that and deploys all of these containers. And so that's how we accomplish that with Pachyderm, is we basically just take these Kubernetes semantics

• 20:52Speaker 02

  and then augment them with knowledge of the data that needs to be processed and capture how that data gets processed and where it goes.

• 21:01Chris Benson?

  Gotcha. So just to kind of catch up a little bit and make sure I'm on the right track, you have Kubernetes deployed for infrastructure and you're deploying Pachyderm on top of that. And you have the file system that it brings with the versioning and your capability for providence tracking. And you've talked about the pipelines and stuff. I take, just to ensure that I'm on the right track, I assume that the data is in the containers that you're deploying specifically?

• 21:23Speaker 02

  Yeah, so that's where it starts to get interesting. The data is in the containers, but it's kind of ephemeral in the containers because containers themselves are kind of ephemeral. Part of the point of a system like Kubernetes and the reason that you give it a thousand nodes to operate on is that any of those nodes could die at any time, right? And this is the sort of thing where like, this is technically always true. You know, even when you're just running your code on your laptop, your laptop can die at any time. It's a physical machine, but this isn't such a concern when you have one computer. But when you're running on a thousand, it's almost guaranteed to happen once a day just because you've got so many machines there. And so we put the data into your container for you to process.

• 22:08Speaker 02

  And then when you finish processing it, we write it back out to object storage. And that's where, once it's in object storage, that's when it's actually persisted within our architecture. Because nothing that's stored on a disk in a container, any of that stuff could disappear at any moment, is basically how we operate. This is also a great opportunity for me to talk to you about what the actual interface that your code gets to the packetized data is. We really, really wanted to build a system that was going to be language agnostic. One of the things that really bugged me about the Hadoop ecosystem was that you sort of had to write in Java to really get the most comfortable semantics. Like, you could kind of use Python, but it was always a little bit cludgy.

• 22:49Speaker 02

  And so when your code that you've put in a container boots up, and because PacketRm wants it to process some data, you will just find your data sitting on the local file system under a directory called PFS. And these are just totally normal files. You can open them with the system call open and you can read from them and write to them and stuff like that. And so this, we thought, was just the most natural interface that your code could possibly have. And users often have the experience when they've just written a Jupiter notebook to process some stuff on their laptop, normally they're just getting that data from local disk too. And so they have the experience when they're getting onto PacketRm and like, okay, like, I'm gonna need to learn the PacketRm API, I'm gonna need to like import PacketRm into my Python code or something like that.

• 23:32Speaker 02

  Like, no, you can just use your normal OS system calls to open data and write data out. And that's the entire system. That's all you need to do.

• 23:39Daniel Whitenack?

  Yeah, so I have a follow up there and maybe there have been some updates that I'm not aware of, but I think one of the common kind of maybe struggles that I've seen people ask about is, you know, this is definitely fundamentally different than something like Hadoop or Spark, where you have like some concept of data locality here. You're kind of like putting data into the container and then taking it out, but it actually lives somewhere else. Are there concerns with that? Are there like trade offs? What is, what are the sort of trade offs that you're playing with there? Especially as you get into kind of larger data sets and that sort of thing.

• 24:18Speaker 02

  Yeah, so there's absolutely trade offs, right? Because each time, that means that the data needs to be downloaded from S3, written to local disk, which is normally faster than S3. So that doesn't really incur a penalty. And then it needs to be pushed back into S3. And so basically what you're trading off here is that this system could be more performant if it was entirely using hard drives, but it would be basically harder to, for admins to maintain, right? Because the thing that people like about object storage is that it's just really dumb and simple. You've just got a bucket sitting there with all of the data in it. There's no like, which hard drive is this on? Like, do we have all of the hard drives? Are they linked up to the right things and stuff like that?

• 25:00Speaker 02

  The reason that we chose this architecture as sort of our initial architecture is that this was a lot of the direction that we saw. We saw people basically making this same trade off in Hadoop, even though they didn't have to. So by far, the most common Hadoop cluster that we see today, and this applies to Spark as well, is basically everything stored in object storage, almost always S3, and then MapReduce on top of that. And a lot of people are just bypassing actual HDFS at this point. We have been making, over the last release, and we're going to do a lot more of this in the upcoming 1.9 release, a lot of progress toward using hard drives to cache stuff. And so we're sort of going the other way that Hadoop went, where they were first a hard drive only solution,

• 25:45Speaker 02

  and then they started having like S3 as a way to like checkpoint stuff out to long term storage. And then eventually that started becoming the only way that people ran stuff. We're always going to have object storage as like the long term place that we checkpoint stuff out to. And then we're going to use hard drives on top as like a cache. And that'll also allow us to use boatloads of memory as a cache too, similar to Tachyon if people want like really, really low latency stuff.

• 26:07Daniel Whitenack?

  Cool. Yeah. The times that I've interacted with Spark, I kind of like, I always defaulted to that S3 option anyway, because it was hard for me to figure out other things. I don't know if that's just my own, you know, my own ignorance or whatever it is, but I definitely, I definitely hear you on that, on that front. But yeah, it's kind of like, there's, there's always trade offs, right? You don't get anything for free, but it's really kind of what you want. What do you want to optimize for?

• 26:32Speaker 02

  Yep. It's always trade offs. And actually, one of the things that we, we do a lot of is trying to counsel people to not worry as much about like, performance on the margins in the early days, because we've seen a lot of like, infrastructure deployments and like, data science projects that just get really bogged down and thinking like, well, there's going to be this extra cost of data getting copied from S3 and getting back and stuff like that. And we always try to tell people like, worry about these things if it's truly going to make it impossible for you to accomplish your, your goals. Like if this absolutely needs to be a low latency system, because you're doing like, algorithmic trading or something like that. But in a lot of cases,

• 27:12Speaker 02

  we feel like people get better results by just focusing on getting something that works. And that's, you know, I think exactly the trade off that you were making when you were setting up Spark is that like, yeah, if you really bang your head against the wall, like, you can figure out how to set up S3 on like, solid state drives on AWS, and it's going to be faster than what you're doing, what you're doing with S3. But if you consider the amount of time that you spent setting that up as like, performance time until you actually get your results, you might actually get them much, much slower. So there's a huge amount of value in just having infrastructure that you understand top to bottom and that is simple.

• 27:45Chris Benson?

  So wanted to ask about that. We've kind of talked about a lot of different technologies, you know, in these, in these potential use cases. And I know that kind of getting back to teams and individual skills, blood teams where the skills were, you know, varied fairly widely. Some people like myself came from software engineering into the AI world and machine learning world, and others came straight out of school and, you know, with data science degrees and had not, had not done some of those. Do you ever find that there is any challenge or intimidation where people come out? And they may know their data science, but you know, they may not have even heard of Kubernetes or not be familiar with containerization. I kind of wanted to call that out because like, you know, me and you and Daniel are all incredibly

• 28:28Chris Benson?

  familiar with containerization and Kubernetes and such, but not everybody is. Kind of, how do you speak to that? Do you, do you recommend a data engineer or infrastructure engineer get involved? Or what have you run into in real life?

• 28:39Speaker 02

  Yeah, so that's definitely a challenge for us. And we really see the full gamut and it's just very, very interesting. You see some people who like build themselves as like, look, I'm a, I'm a data science person. Like I'm, I've never really done any serious software engineering. Like I don't really keep up on this stuff. And then you sort of just sit them down and explain like, all right, well, here's what Docker is. Like here's how you install and stuff. And they're like, oh, this basically seems to make sense. Like I can get by here. And then there's some people for whom we, we do like education sessions and basically just try to teach people the basics of containers so that they can work with it. I would say that actually when we really have challenges, it has, it's less about software engineering expertise and probably more about DevOps expertise.

• 29:22Speaker 02

  To be honest, like a lot of the, the types of issues that we hit are just like the permissioning on the Kubernetes cluster is wrong. And so when you go to deploy like your code, everything works until it starts trying to like talk to S3 and then like the network just doesn't work or something. Cause like the bucket is rejecting it or something like that. And like, there's just a lot of DevOps complication in there. And so, you know, we always, we always sort of try to like, keep our feet on the ground a little bit on this stuff because you know, our whole goal with Paketer was when I was at Airbnb is like, well, this data infrastructure is really hard and my team is 25 people just keeping this darn thing running. And so what are all of the, the companies that don't have a team of 25 people to keep their data infrastructure running, doing?

• 30:06Speaker 02

  And so we wanted to make something where you didn't need that team. Like a data scientist could just do it by themselves. And I think we're closer, but you know, then when we go into companies and talk to them like, well, we've got like one person working on this full time. And you know, they're feeling like they have to do a lot of DevOps to keep the Paketerm cluster up and running. I sort of realized like, okay, you know, we haven't, we've made an improvement here. We haven't just magically eliminated this, you know? We haven't gone from you need 25 DevOps people to keep big infrastructure running to. You need zero DevOps people to do it. And so we're trying to make that better in every release. We're trying to make that as easy as possible. And one of the big steps forward on that will be having our own hosted solutions. So people don't have to deploy everything on their own cloud just to try it out. Short answer is, is that's definitely a challenge.

• 30:48Speaker 02

  Is that there's a bit of an infrastructure leap that needs to be made, which can be uncomfortable for a lot of people that I think could ultimately benefit from the feature set of PackerDerm. It's just they can't quite get the activation energy.

• 31:00Chris Benson?

  So I was wondering, is there anything else? You know, and another question that you commonly find is people have existing infrastructure in place. They might be a Hadoop shop, a Spark shop, or one of several other technologies. You know, they might have big databases like Cassandra. What are you trying to replace and how are you trying to fit in? I know we talked about the data locality issue, but are there any other big considerations that you would say is, is, you know, why you should go PackerDerm versus what they already have in-house?

• 31:27Speaker 02

  Yeah, I mean, I would say the things we're trying to replace are sort of HDFS and then the computation layers on top of that. So like MapReduce is a common one, but like Hive and Spark and stuff like that, we're also trying to speak to. Those are the main things that we're trying to replace. We constantly have the challenge of with people who have existing data infrastructure and want us to sort of fit into that well. And that's always a bit of a back and forth because some things can work really well in PackerDerm because you can just, you have the flexibility of a container. And so you could put whatever you want in there. So, you know, people will have containers that include code so that they can go and talk to HBase somewhere else in the cluster. Right?

• 32:11Speaker 02

  And so then you have sort of a natural like shim to put between your existing infrastructure and PackerDerm, which is the container code, which is totally flexible. It doesn't work beautifully for everything. Right? Like what you wind up doing with like Spark or something is you wind up having like, here's your data, it's stored in PackerDerm. Now you boot up a job and you want to talk to Spark. So now I need to push all of this data into Spark or somewhere where I can access it or something like that. So we're sort of constantly trying to figure out how to make these integrations better. Better, but the users that always excite us the most are the people who who basically come in and say like, we don't want to go down the Hadoop route. Like, we know that there is a lot of just pain required to get a working Hadoop cluster and to get stuff functional on it.

• 32:56Speaker 02

  And so we want to try something different and just build from on Pachyderm from scratch. And so, long term for our company, we're focused on how can we make things really good for people who just see the Pachyderm vision and commit to it from scratch? Because those are, you know, if we're successful in 10 years, then those are going to be the people that have really made the company successful. And the sort of the integrations will will help us along the way to onboard more people. But it's really going to depend on that core use case.

• 33:22Daniel Whitenack?

  Yeah, so the team that I'm working on now, the organization is pretty big, but it's kind of on this project that I'm working on. It's like myself, who has some type of data science background. And then another guy who is somewhat technical, but he's a linguist. And so our ability to spin up a like a working Hadoop infrastructure is probably like less than 0% probability. And so, I mean, even just like if there's one thing I could say to to listeners, like even if you just get to like where you can use containers themselves is like a huge benefit also to like reproducibility in the in the space of machine learning and AI, which is is awesome. So I kind of wanted to follow up.

• 34:05Daniel Whitenack?

  You've already mentioned, JD, that Pachyderm, at least what we've talked about up to this point, is is free. But I also know, like, you're a company, right? And I should give you some congratulations because you just kind of hit a big accomplishment, isn't that right?

• 34:20Speaker 02

  Yeah, and and thank you for the congratulations. We just raised a Series A, which means that we have a ton more funding to basically pursue our vision for data science infrastructure. And it also means that you can commit to Pachyderm as your infrastructure with a lot more peace of mind now. Because, you know, the company is going to be around for quite a ways to come. That also sort of leads, as you said, we are, we are a company, which means that we need a way to make money and that for that we have an enterprise product. So let me just sort of tell you what's in that that you won't find in the enterprise. We try to really make it so that our open source product contains everything that's going to be really useful to sort of individuals.

• 35:00Speaker 02

  And people who, you know, just want to get some, some data science done, but they're not running within a gigantic organization where they have all of those concerns. So the types of things that go into that enterprise product are the permissioning system. And so that's, you know, the ability to say like, this data right here is owned by Dan, this data right here is owned by JD. This data right here is owned by Steve, things like that, and make sure that nobody is getting data that they don't have access to. And what's cool and what we think is a very crucial feature for these types of systems is that it's informed by our provenance model. Right? This is a big problem that you'll run into in big data organizations is that it's very easy to have some data that nobody's allowed to see.

• 35:43Speaker 02

  That then gets turned into a model or some sort of an aggregation or something like that, that everyone's allowed to see that is accidentally leaking the data that went into it. And so we have our provenance tracking system, it formed the permissioning system. So if you don't have access to the provenance of data, then by default, you don't have access to the data itself, because it might contain that information that you're not allowed to see. Other things that go into the enterprise product are like a sort of wizard UI builder for building new pipelines and things like that. And visualizing how they're working and the ability to sort of track and really optimize your, your pipelines. See like where they're spending all of their time and like squeeze every last little bit of performance out of your hardware. The other main thing that we sell is basically just support at our time.

• 36:27Speaker 02

  And you know, the, the ability to like, talk to us and have us prioritize features and stuff like that, which is, you know, every open source project does that.

• 36:33Daniel Whitenack?

  Yeah, it's really interesting. It's a, I always love to hear like different people's perspectives on their, their open source models as well. I was just talking to someone the other day, a friend who's starting a new business and like considering how they should approach open source, but yet also be a company and like, and survive. So I think there is definitely people out there that are interested in that question. So I appreciate you sharing that.

• 36:55Speaker 02

  Yeah, absolutely. And you know, it's tricky and it's very imperfect because I really think that this is a system that that really should exist. There's a lot of need for a system like this. It basically has to be open source for it to actually fill that need. In my mind, I just couldn't see a proprietary system becoming like the standard data infrastructure layer. But it's very, very hard to get the funding to work when you're open source. You know, it's this huge asset because people can so easily try your product and you get so much adoption and stuff like that. But it really anchors people of just like an unwillingness to pay for for software when it's open source. And so you always sort of need to cross that

• 37:36Speaker 02

  threshold. And one of the the things that we're looking to do in the future, now that we've raised more money, is basically build the hosted version of our software, because that just sort of totally, it totally changes the value proposition. But it also, I think, has some sort of psychological effects on people wherein like, nobody would ever pay for Git. But the idea that you're going to pay seven bucks a month to have like private repos on GitHub or something like that is just totally palatable to people.

• 38:02Chris Benson?

  I think that's a fantastic idea. I love the hosted idea. I know that when Daniel first introduced me to Packarderm a while back, and I was kind of initially learning the fact that coming from the software engineering world, that it was built on containerization and Kubernetes was a huge plus for me. If I recall correctly, a lot of it's in Go, which I thought was pretty, pretty amazing, as is Docker and Kubernetes. I guess if you're just hearing about it and you've kind of come away from this episode today, and you want to learn more about it, and maybe want to dive in, get your hands dirty and figure out if it's right for your organization, how do people get started with that?

• 38:37Speaker 02

  Yeah, so we've got a bunch of tutorials and like quick start guides online. And so, you know, if you want to just sit down with a guide and start hacking away, then then that's the way to do it. We also have a very active user Slack channel, where all of our engineers and everyone on the team is just always hanging out and ready to ask questions. And you know, those questions range from like, I hit this error, what do I do? And you know, we just, we just give you a simple response, if it's simple, hopefully it's simple. And to people also asking us, you know, I'm looking at Packarderm for a new project, talk to me about the feature set, you know, talk to me about how you think this could be helpful here and just like, talking to us. And so I think that's, that's really the best way, if you want someone to talk about, to talk to about stuff, is just stop by the Slack channel.

• 39:21Daniel Whitenack?

  Awesome. Well, thank you so much for taking time to talk with us, JD. Of course, we'll put the links to like the tutorials and the docs and the Slack channel and all of that in our show notes. So, so go check those out. But it's been awesome to hear from you and really excited to hear about the progress with Pachyderm and all the good things you're doing.

• 39:39Speaker 02

  Yeah, thanks so much for having me, man. I love appearing on podcasts.

• 39:43Daniel Whitenack?

  All right. Well, look forward to seeing great things from Pachyderm. Thanks again. Thanks for coming on the show.

• 39:48Speaker 02

  Thanks, guys.

• 39:51Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for change log is provided by Fastly. Learn more at fastly.com. And we catch our errors before users do here at change law because of roll bar. Check them out at rollbar.com slash change log. And we're hosted on Linode cloud servers. Head to linode.com slash change law. Check them out. Support the show. This episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in

• 40:34Speaker 00

  your email address, get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.

• 40:51Speaker 04

  I'm Nick Niesi. This is K-Ball.

• 40:52Speaker 05

  And I'm Rachel White.

• 40:54Speaker 04

  We're panelists on JS Party, a community celebration of JavaScript and the web.

• 40:58Speaker 06

  Every Thursday at noon Central, a few of us get together and chat about JavaScript, Node, and topics ranging from practical accessibility to weird web APIs.

• 41:06Speaker 07

  You could just eval the text that you're given and then, and that's basically what it's doing.

• 41:12Daniel Whitenack?

  What could go wrong?

• 41:13Speaker 04

  Yeah, exactly. This is not legal advice to eval text as it comes in. Join us live on Thursdays at noon Central. Listen and Slack with us in real time, or wait for the recording to hit. New episodes come out each Friday.

• 41:26Speaker 05

  Find the show at changelog.com/jsparty or wherever you listen to podcasts.