From notebooks to Netflix scale with Metaflow

As you start developing an AI/ML based solution, you quickly figure out that you need to run workflows. Not only that, you might need to run those workflows across various kinds of infrastructure (including GPUs) at scale. Ville Tuulos developed Metaflow while working at Netflix to help data scientists scale their work. In this episode, Ville tells us a bit more about Metaflow, his new book on data science infrastructure, and his approach to helping scale ML/AI work. Sponsors: RudderStack https://rudderstack.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog – Smart customer data pipeline made for developers. RudderStack is the smart customer data pipeline. Connect your whole customer data stack. Warehouse-first, open source Segment alternative. SignalWire https://signalwire.com – Build what’s next in communications with video, voice, and messaging APIs powered by elastic cloud infrastructure. Try it today at signalwire.com https://signalwire.com and use code AI for $25 in developer credit. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFeaturing: Ville Tuulos – LinkedIn https://www.linkedin.com/in/villetuulos, X https://x.com/vtuulosDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Metaflow https://metaflow.org/Books “Effective Data Science Infrastructure” by Ville Tuulos https://www.manning.com/books/effective-data-science-infrastructureUse code podpracticalAI19 for 40% off! Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We can potentially apply ML and AI to all aspects of the business. So I think like we are entering a world where like no area of life and business is totally kind of a safe from ML in a sense that like we can apply ML to all kinds of things. And now I don't mean like a crazy general AI, but I mean like a tiny little optimization problems here and there, and they are like really everywhere in all lines of business.

• 0:23Speaker 01

  Big thanks to our partners, Leno, Fastly, and Launch Darkly. We love Leno, they keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com. This episode is brought to you by our friends at RudderStack. And we're calling all data engineers to check out RudderStack Cloud and start building smart customer data pipelines. RudderStack is warehouse-first, no more silos. RudderStack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API-first. RudderStack can be easily integrated into your existing development processes.

• 1:07Speaker 01

  And because they're open source, you can see all their code, so you don't have to worry about vendor lock-in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. RudderStack is free up to 500,000 events and pricing scales transparently from there. Learn more and get started at rudderstack.com. Again, rudderstack.com. That's R U D D E R S T A C K dot com.

• 1:42Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI.

• 2:07Daniel Whitenack?

  Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International. And I'm usually joined by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin, but he's got the week off. But I do have a wonderful guest for today's discussion, which I think is very timely and also very practical. Vila Tulos is with us, who is CEO at Outer Bounds and was previously leading the Data Science Infrastructure Group at Netflix. He's also the author of a really interesting and new book called Effective Data Science Infrastructure:

• 2:50Daniel Whitenack?

  How to Make Data Scientists More Productive. Well, that sounds super practical, Vila. Welcome to the show. Great to have you here.

• 2:58Speaker 00

  Yeah, thanks for having me.

• 2:59Daniel Whitenack?

  Well, I know even before the show, we were talking about your background and how your sort of background has shaped how you think about data science infrastructure, but also how you think about AI and what really is new, what isn't new, what trends have happened and developed over time. So could you just give us a little bit of an idea about your background and how you started in this field?

• 3:23Speaker 00

  Yeah, well, I wonder how far back I should go. But impressively enough, I started at my first startup that worked on artificial neural networks already back in 2000. So I mean, that's kind of a long time ago. And believe it or not, I mean, this of course like predates all deep learning and so forth. But people did artificial neural networks at the time. And this startup in particular, it was focusing on a very peculiar kind of neural networks called self-organizing maps that were quite popular like in late 1990s, early 2000s. And the commercial idea we had at the time is that we could help enterprises with kind of a search, like enterprise search, the internet search, stuff like that. So it happened that, yes, I mean, companies needed help with that, but I mean, they didn't need neural networks. But interestingly enough, ever since then, I've been really like focusing on this question

• 4:08Speaker 00

  that how can we actually like improve the tooling and the infrastructure for people who built these models? Already like back in the day, I I saw that like a big challenge that we had is that like we had all these amazing researchers, data scientists, whose task was to kind of build these models that were supposedly useful in practice and it took many iterations. It took a lot of thinking and like trying out experimenting and like, of course, like back then, I mean, the kind of the tooling was really quite horrible. And now it's much better, but I think it's really interesting kind of to see the kind of the whole like trajectory, like where we came from and like where we'll be going in the future. And and yeah, I mean, it's kind of scary to see that, like, also, there are like many things that like haven't actually changed that much and like kind of we have much work ahead of us.

• 4:49Daniel Whitenack?

  Yeah, maybe could you share some of those examples? Like, what is the same now? Maybe a challenge that you're still facing now that was the same as back in those days? And maybe something or a couple of examples of things that have really dramatically changed.

• 5:07Speaker 00

  Yeah, well, I mean, starting with the some things that are the same, amazingly enough, like already back in the day, like we were using Python. And of course, that was highly controversial at the time. So I mean, Java was pretty new and like, of course, like actually like everything that needed to be high performance was written in C and C++. So that's actually like a big change like between then and now that, of course, like we had to build all the libraries by ourselves and it was even seen as a big kind of like a differentiator and a competitive mode that like we had a more performant, more scalable algorithms to optimize these models than anyone else. Nowadays, of course, everybody can just use off the shelf PyTorch, TensorFlow, even like XGBoost, scikit-learn just released 1.0. So the library ecosystem, I mean, like is way ahead like what it used to be.

• 5:51Speaker 00

  Now, another like a big difference is that we actually spent considerable amount of time just setting up the hardware. Amazingly enough, like we had a rack of servers like kind of at the company and like we had to set up the networking and the storage and like the compute and the operating systems and all that stuff. And now with the cloud, I mean, again, I mean, that's another thing that like has massively changed. That like you can get this infrastructure that like used to be available only for the largest companies, even at the startup and like you can get like clusters of machines with GPUs and it's just like mind blowing. And now the what's really exciting today is that on the one hand, like you have this Like a high level modeling libraries. On the other hand, like you have all the hardware available. And now I think that like kind of still it's kind of mind boggling that still, I mean, things aren't that easy. It kind of feels that everything is possible, but I mean, nothing is really easy enough.

• 6:36Speaker 00

  And I think that that is still a kind of something that like we need to work on. And of course, like one interesting challenge is also that there are way more people working on these problems. It used to be only a handful of people. I think we had three people at the company at the time who could actually like build these models. And now I mean, it's like probably a hundred, if not thousand times more people. So definitely feels like a new field in that sense.

• 6:53Daniel Whitenack?

  Yeah. And how did your perception of data science and the infrastructure in particular needed for data science, how did that shift as then you led the team at Netflix? And I mean, now we're talking, of course, about a a much bigger scale, right? But also like, yeah, these models are, I assume even at that point, a really critical piece of what makes Netflix Netflix and the value that's added by these models. So how did your perception of infrastructure change as you worked on those sorts of problems?

• 7:28Speaker 00

  I think like one interesting thing that maybe not everybody realizes is that like the few things that like many people know about Netflix, especially the recommendation and personalization systems, they are kind of the tip of the iceberg. And there are like so many other things like where Netflix wants to apply potentially ML and related technologies. And like also like wants to experiment with new ideas. I think like one thing that's really interesting is just the kind of the diversity of different use cases. If you think about computer vision, natural language processing, even things that are not technically machine learning, like operations research, how we can optimize schedules, stuff like that. So it's just the fact that like there are like so many different types of problems and then they come in all shapes and sizes. It's not always about scale.

• 8:09Speaker 00

  Also, I mean, it's not always that everything has to be like a super kind of a high SLA business critical. I mean, they're like experiments, like a crazy experiments. But the interesting challenge is really that like, how do you manage like kind of like the diversity of all these things? So that's really interesting. Now, when it comes to the more technical side of things, Netflix, like for the longest time, has been hundred percent on AWS. Like it's a cloud first company. And Netflix has been trailblazing. Like many architectural patterns, like when it comes to storing data in the cloud, way before like other companies were doing it, also microservices, chaos engineering. So it was really fascinating to be at this company that had, on the engineering side, like kind of some of the world's leading cloud infrastructure. I think honestly, I mean, what makes Netflix interesting is that it all like runs on AWS,

• 8:53Speaker 00

  in contrast to, let's say, Google and Facebook, who have their own infrastructure, which is really kind of an island and nobody else can do it. But I mean, Netflix is in that sense like a closer to everybody else in the world that like it is the same AWS at the end of the day that Netflix uses, that everybody else can use as well. And it's just that like kind of that they have all these like practices and ways of thinking about things and like ways of about building services that like makes them really effective. And now when you layer something like a data science and machine learning on top of it, I mean, it's really interesting. And of course, I mean, that is kind of all those like learnings are reflected in Metaflow, which is the open source library that we started building there.

• 9:25Daniel Whitenack?

  Yeah, I do want to get into Metaflow, but before I do, maybe what is your perception of how like maybe back in those days, when there was sort of like a building hype around data science, a lot of people initially getting into it, companies experimenting. What is your perception of like from then to now? How has like the average data scientist's ability to work with these different pieces of infrastructure that maybe come across their path? Like the various services in AWS, whether it be EC2 or, you know, object storage or things even all the way up to like Kubernetes and EKS and that sort of stuff. How has like the average data scientist, from your perspective,

• 10:09Daniel Whitenack?

  what they're required to know or maybe what they come in with knowledge of? How has that shifted over time?

• 10:16Speaker 00

  Well, my kind of initial reaction is that I still think that like we are in the early days. Yes, I mean, of course, I mean, the fact is that like maybe five, 10 years back, if you wanted to do anything in this field, you basically had to know C++ and like you had to know the depth of knowledge about, let's say, distributed systems, what's much deeper. And even let's say the kind of recommendation systems at Netflix, they run on Spark and like many people use Scala. And that's a bit of a different persona, bit of a different profile. Like what we see now amongst data scientists who are building this like a new set of very diverse models, like using these Python-based libraries, maybe directly using the cloud and so forth. At the same time, I do think that all these things, and especially like being able to leverage the cloud, is still harder than it really should be.

• 11:01Speaker 00

  Just thinking about the kind of the amazing amount of like computational power that you have there. And like still, I mean, there are like most companies I talk with, there's the feeling that like, well, data scientists kind of need to know about Docker files and maybe they need to know about the C I C D systems. And I think that there are like a different points of view. I mean, like if that's actually a feature or a bug. I think that there are like so many questions related to the modeling itself that like, at the end of the day, all of us, like all human beings, like you kind of need to manage your cognitive bandwidth. And as interesting as it might be to kind of for everybody to know about C I C D systems, I do think that it kind of takes a bit away from the bandwidth that like you should have available for like thinking about the modeling problem itself. So I do think that and I do hope that like we manage to kind of raise that like a level of abstraction even more.

• 11:45Daniel Whitenack?

  Yeah, I guess in this case abstraction could be a good thing. And even though it is interesting to dive into these different systems and containerization and all that, it does take a lot of time, like you said.

• 11:57Speaker 00

  Yeah.

• 11:58Daniel Whitenack?

  I remember there's like a tweet from Eric Bernadson, who said like having a data scientist sort of learn about some of these things, like Kubernetes and Docker and Terraform and these things is kind of like having like web app developers learn about the Linux kernel. It's just like so far apart. It's very tough to expect that. Yeah. At the same time, like I have benefited from those times where I've been able to maybe push something further on my own, at least into like a prototyping stage within a company.

• 12:31Speaker 00

  Yeah.

• 12:31Daniel Whitenack?

  And get it in front of people to see that value without sort of like reliance on, you know, passing something off to a data engineering or a software engineering organization to even create a prototype. So maybe there is some like tooling that's improving around that. Like I know there's things like Streamlit and other things where you can create something that's very compelling very quickly in terms of like a prototype. But I don't know, one of the things Chris and I discussed on the podcast a little while back was maybe why many data projects fail in certain cases is because people aren't able to push a project far enough into a prototype stage for people to see like the value of something

• 13:16Daniel Whitenack?

  and actually get buy-in from the organization. From your perspective, I know like also being the CEO of a company that is trying to help people with their ML infrastructure, where do you often see like when you first maybe engage with clients or when you're maybe just making an observation about the industry, where do you see the problems in like people not being able to get value out of machine learning and AI? Where do things get blocked most often from your perspective?

• 13:45Speaker 00

  I think it's definitely, it's a combination of maybe three different factors. Well, I mean, like maybe starting from the kind of the easiest one is technical, that there are technical hurdles still. I mean, like it's just like putting the infrastructure together, like it's just building the models. Although like technically all the ingredients are there, like many companies are still like struggling, like kind of putting the pieces of the puzzle together. But I do think that that is in a way, I mean, like the easiest one of the three. So then the second one is definitely the kind of the skill set of the people involved. And it's not only that they wouldn't be skilled, I mean, I don't think that that's oftentimes the problem, but also like kind of what other things they should be focusing on. And especially like when we start talking about actually producing business value using machine learning, like really understanding the problem domain, like understanding the business needs.

• 14:30Speaker 00

  I mean, that takes a lot of bandwidth and like still much of the time, like these practitioners, they spend on like kind of either like engineering problems or like maybe modeling problems that may be fun, but I mean, ultimately might not really kind of affect the company's bottom line so much. But then the last problem, I think it's the big one, is really the organizational. It's kind of like a leadership question that kind of what you mentioned before is that, okay, so why do many of these projects fail? Well, I mean, they don't get like a close enough to production. I think that that is absolutely a key, and I think that that was one thing that Netflix did really, really well, that they have, even at the highest level of the organization, they have this experimentation culture and they have this idea. And understanding that like well now first we can potentially apply ML and AI to all aspects of the business.

• 15:14Speaker 00

  So I think like we are entering a world where like no area of life and and business is totally kind of a safe like from ML. In a sense that like we can apply ML to all kinds of things. And now I I don't mean like a crazy general AI, but I mean like a tiny little optimization problems here and there. And they are like really everywhere in all all lines of business. But now the problem is that like you may have thousand ideas that, okay, we could do this and that. But how do you know which one of those work? And like, nobody really knows in advance and like you can't really ask anyone. And the only way to know is that like you really need to kind of start experimenting. And like not only experimenting in a sense that like you hack something, a prototype in a notebook. But oftentimes, really, the only way to know is that you basically push these things to production. And now the production meaning not so that like you have a huge team working on something for in six months.

• 15:58Speaker 00

  But actually, like kind of getting something to A B test, let's say. I mean, like if you have an honestly, I I know that like this is really like not that easy. But I mean, the idea that like you can test these ideas, you can test different like prototypes and pipelines in in production alongside like whatever system you have in place today. And then you compare the results that it's hugely, hugely powerful. And then like have the understanding that like you can interpret the results and decide what to do with that. And like have the understanding that like actually it is by design that like most of these things fail. I mean, that's kind of the whole point of experimentation. If you knew that everything is going to succeed, I mean, you wouldn't have to experiment. But the idea is that like you can afford making so many of these like a tiny experiments that then you can quickly decide that, okay, this doesn't seem worth it. And like maybe then like you like redirect resources to something else.

• 16:43Speaker 00

  But I mean, this is also like a question of product management oftentimes having like product managers who really understand how to work with these ML systems. I mean, all these like organizational muscles are like missing at at many, many companies.

• 17:11Speaker 03

  Signal Wire is realtime video tech to help you create interactive video experiences previously not possible. It gives you access to broadcast quality ultra-low latency video that's proven and trusted by Amazon, Ring Doorbell, Zoom, and others. See why the future of video communication is being built on Signal Wire. They have easy-to-deploy APIs, SDKs for the most popular programming languages, and expert support from the OGs of software-defined telecom tech. Try it today at signalwire.com and use code AI for $25 in developer credit. Just visit signalwire.com, that's signalwire.com, and use code AI to receive that 25 bucks. Once

• 17:55Speaker 03

  again, that's signalwire.com, code AI.

• 18:11Daniel Whitenack?

  So, Vile, you started to mention, as we were talking, about trends that you're seeing in infrastructure and ideas around where things get stuck in production. You mentioned Metaflow a couple of times, which I know is a big piece of the puzzle in terms of how you solve these problems, but also a big part of your career in terms of what you've developed. So could you give us a little bit of the backstory of Metaflow, sort of the origin story, I guess?

• 18:39Speaker 00

  Yeah, I guess like the nice thing is that it's actually like a quite pragmatic, quite bottoms up in a manner that, as I mentioned, Netflix, when I joined Netflix back in 2017, this was before SageMaker, this was before MLflow, this was before Qflow. This idea of having any kind of, let's say, especially open source machine learning infrastructure was quite new. I mean, of course, there were products around, like you had DataRobot, you had Domino Data Labs, obviously, like you had Databricks and Spark and so forth. But I mean, like, the idea that, what does the full stack for ML look like? I mean, that was quite new. So now, when I joined Netflix back in the day, I saw that, okay, so, well, obviously, like, the company had, like, All these like a basic foundational pieces of infrastructure in place. So they had a large data warehouse and S3 based data lake.

• 19:23Speaker 00

  They had a team managing a large scale compute infrastructure, basically something like Kubernetes. They had also teams who had been thinking about workflow orchestration for a long time. So you had all these pieces. So again, I mean, like technically, everything is impossible. So it didn't seem like the challenge was that, okay, we need to invent some new pieces of tech that like we could do something that nobody else has done before. That didn't seem to be the problem. But then the problem was really that they had the organization of data scientists there like who constantly complained that like, well, getting anything done was too hard. Because exactly for the reasons we discussed that like, okay, how do I run compute? I mean, you go to the compute team, they say that, oh, I mean, you just have a Docker container and like you put the container here, image here and tag here. And then like, maybe you better go through the C I C D system. And then it already at that point in time, you had like a lost the data scientist.

• 20:07Speaker 00

  Like, does this make any sense? And the workflow systems, of course, needed like lots and lots of YAML to kind of define what you want to do. And even like kind of thinking like, what are the kind of the patterns? Because remember, like, these people are not software engineers by training. So the kind of how you actually architect software like this was hard. Really, the origin story and the idea for Metaflow was that, okay, assuming that like you have this like foundational infrastructure available. Like, how can you kind of stitch them together in a manner that would present an API to data scientists that like would kind of like help them to build these applications? That they have been asked to build for the company. And now the other interesting side of the coin was that Netflix has this culture of freedom and responsibility, which meant that like, we didn't want to take away

• 20:49Speaker 00

  all the freedom from people saying that like, well, I mean, like, there's like a training API and like you can only call this one API to train your model. It was well known that like different people prefer different tools for the job. I mean, some people prefer TensorFlow, some people prefer XGBoost. I mean, depends on the application, of course. So the idea was that, okay, we should allow them to kind of at the high level, like kind of exercise that freedom and exercise that domain knowledge and expertise in like choosing those like modeling tools. So like, we kind of started having this idea that, okay, we need to be quite opinionated about the lower layers of the stack. How do you do compute? How do you access data? How do you do orchestration? And then like leave a lot of space at the top of the stack that, okay, so what kind of modeling libraries you use? And how do you do your feature engineering? And like, maybe even like, what are the KPIs that matter for you when it comes to monitoring models in production?

• 21:35Speaker 00

  And then like we started like crafting that stack. And again, I mean, like Netflix, there's no top down like anything. I mean, there's no CTO, no VP of Engineering saying that like everybody must use this thing. But we started like solving these very practical problems. And then kind of like in a very organic manner, I mean, Metaflow started spreading inside the organization. Because people thought that, oh, I mean, this is like a quite like a no nonsense tool that like helped them to solve exactly the types of problems that like they had been facing on a, on a day to day basis.

• 22:00Daniel Whitenack?

  Yeah, so there's a whole variety of things that people have created around workflows. But then there's also a whole set of platforms and projects out there related to like ML Ops and other things. There's like all sorts of things that maybe data scientists care about from like making sure that they can run their workflow not on their laptop, which is more maybe infrastructure compute related all the way to like, hey, how do I version and control experiments? How do I access data? So how far does Metaflow reach in terms of these different things that data scientists might want to do? Like what pieces of the puzzle does it try to solve?

• 22:43Daniel Whitenack?

  And how can data scientists think about it in terms of those various buckets of things they're trying to do?

• 22:49Speaker 00

  Yeah, that's a good question. I think like because we were faced with this great diversity of different applications, we couldn't think, let's say, I mean, at my previous company before Netflix, I mean, we were doing a real time bidding and like for targeted advertising. And then like in a context like that, you know exactly the application and you know that, okay, maybe we build a feature store, like maybe these are exactly the workflows everybody follows. And that's like one type of a challenge. It actually like might be a great engineering challenge, but I mean, different type of a challenge. In our case with Metaflow, the challenge was that like we didn't know exactly the type of machine learning applications people wanted to build. So we started thinking very much like from the bottom up that like, what are commonalities across all applications? And, and like really it starts with that, okay, so I mean, really the question of data and like now the discipline ways of accessing data.

• 23:33Speaker 00

  Okay, so how do we do it quickly enough? Let's say you have a, some kind of a data warehouse, you have a database, how do you get the data out quickly? So you don't have to wait for 40 minutes for your SQL to execute. So that's one thing that like we were thinking about. Like working with Arrow, like Metaflow comes with the custom S3 library. So you can get your data super fast like from S3. Small things like that. Then like on the compute, like again, I mean, like all these models. I mean, not even the ones that like require huge amounts of data, require still a lot of compute. So you may want to do hyper parameter search. So maybe you have a model ensemble, like maybe you want to build a separate model for every country or maybe for every customer. So you need to be able to fan out this compute to the cloud. So definitely we need to solve that part as well. Then we saw that oftentimes it's really, really sensible idea to structure these applications as a workflow.

• 24:18Speaker 00

  So I mean, there's a lot of confusion about these DAGs and workflows these days that, okay, what does it even mean? And like, there are so many different workflow systems, but purely as a way to kind of express these ideas, the idea that you structure things as a DAG, I mean, it makes a lot of sense. So we kind of took that as a kind of a core way of implementing things. But then like, we definitely wanted to separate the idea that once you deploy these workflows to production, like running workflows that scaling production is actually an engineering challenge of its own. And we didn't want to claim that like, well, I mean, Metaflow is the most like production, great scalable workflows, get all right in the world. So we integrated with other systems out there and also the E-step path due to production. So that was really another thing that like, what we discussed earlier about how important it is to kind of test these ideas as close to production as possible.

• 25:01Speaker 00

  We knew that, okay, we need to provide a path like all the way to the end. And that's why we integrated with the existing system. So we wouldn't get the resistance from engineering teams saying that, oh, you have this piece of Python code, but I mean, no way we are going to run this in production. So kind of really like thinking about the production-based practices, studying like with very mundane questions like dependency management, like what if you need like a very specific version of TensorFlow? Again, I mean, we don't want you to kind of write Docker files by hand. It's surprisingly hard to do it in a reproducible manner. But I mean, like, how do you let you use the exact version of the library you need? And then like, yeah, I mean, you mentioned versioning as well. I mean, like, first, like, there's the idea that, well, you should maybe version your code, maybe like using Git. But I mean, like, how do you version your models? How do you version your experiment? How do you version your data even?

• 25:43Speaker 00

  We felt that like these are such kind of foundational concerns that like we should also provide an out of the box solution for them. I think that that's definitely kind of like helps because then the data scientists don't have to think about it too much. That's kind of like what we have been like doing this far. So if you think about like compute data orchestration, versioning, and like kind of all kinds of questions related to pushing things to production, then they are like things at the top of the stack that like we haven't been so opinionated about. So I mean, many of our users today, they use other model monitoring tools. They are amazing model monitoring tools. You mentioned Streamlit, weights and biases, many others out there. Of course, like a specific tools for model explainability, if that's important to you. And then like, of course, like a feature engineering, like that's a complex topic of its own.

• 26:27Speaker 00

  Like there are some customers who use Metaflow with some feature stores that works. And of course, I mean, like the modeling libraries is something that like you should absolutely use the kind of the best of the breed tools off the shelves.

• 26:37Daniel Whitenack?

  Yeah, so it sounds like part of the philosophy here is people are going to be opinionated in their own teams about like, oh, we use weights and biases to do this bits of the monitoring and experiment management, but that's not going to solve these sort of scale and infrastructure problems and the like workflow running problems that you mentioned as well. So being able to pull in what you need, I think, is really a cool idea. And having that sort of modular nature of it is really great. So I do want to get into like the actual workflow with Metaflow. But in terms of how it works under the hood, like let's say that you're setting up a flow

• 27:19Daniel Whitenack?

  and you've got a series of steps, that processing steps. Eventually, something has to run on a server, right? And maybe certain, like you're saying, maybe I'm running TensorFlow over here and it needs a GPU, or maybe I'm doing this like preprocessing of images and I just need to crank through a bunch of stuff in a sort of batch or maybe even parallel way on CPUs. How do things in code that is using Metaflow, how do those things eventually end up running on the servers? Is there some sort of containerization or something going on under the hood? Or how have you built that abstraction layer?

• 27:58Speaker 00

  Good question. Well, I mean, now Metaflow is like since the day one, I mean, it has been built with this cloud-first mindset. I think that like kind of when it comes to things like compute and storage, I think like we live in a bit of a like a post scarcity world. That it's actually like interesting when you think about it that like many of the systems that we even use today, like databases, even things like Spark and Hadoop. They were built with this idea that you have a constrained resources. And like the really the engineering challenge is that, okay, how do you allow people to kind of run compute given that like you have only like a 200 servers or something like that? And you have to do the resource management very carefully. Then the mindset that like we adapt to to Metaflow, which I think it's really, really useful, especially in the people's productivity point of view, is that you work with the cloud.

• 28:40Speaker 00

  The cloud provides you at least this like a kind of abstraction of having like basically infinite scalability. So you can use some cloud based platform. Again, I mean, like we rely on like existing systems like Kubernetes, like AWS Batch, like what have you. Do you kind of farm out the containers to the cloud? You can specify the resources you need. If your function needs GPUs, you can say that I need GPUs in this case. And in other cases, maybe you need a lot of memory. And I think that like the interesting challenge with machine learning, I think that this also, like I said, machine learning apart like from like many previous data intensive applications is that the the needs are so different. And like in some types of models are really I/O sensitive. Like maybe you need to read like a tons of images, tons of videos, but the model itself may be simple. In other cases, it might be something super compute intensive, but not I/O sensitive.

• 29:25Speaker 00

  In some cases, you absolutely GPU or maybe even like some custom crazy hardware. In other cases, it is cost prohibitive to do that. And it's very hard to have any kind of like a uniform, one size fits all. So I think like having that like scalability that goes in all the different dimensions, like vertical, horizontal, you name it. So that's super useful and cloud makes that possible. So we rely on these systems and we take care of like then packaging the code, sending it to the cloud, executing on the container, handling retries, all that like kind of a basic, basic plumbing there. And then, yeah, I mean, the same thing like with the orchestration. I mean, like the DAG execution at scale, if you have 100,000 DAGs running in parallel. I mean, it's a thing of its own and like there are some systems that do it well. We integrate with AWS step functions. Now we are integrating with Argo, maybe one day with Airflow. I mean, like what have you.

• 30:10Speaker 00

  And the idea is that you should be able to test these things locally. So Metaflow always comes with the local mode. So you can kind of test and iterate the same way you do in a notebook. But then like kind of when you want scale, like when you want to kind of something that's like production ready, you can use your existing production infrastructures.

• 30:24Daniel Whitenack?

  So you were just getting into talking about, hey, you know, maybe you're doing some experimentation locally in a notebook. And then like eventually you kind of go beyond that and scale up and all of that. I guess first off, to set the stage, Metaflow is an open source project and people can go ahead and try it out. And we'll include links in our show notes to where people can find it and try it out. But let's say I am a data scientist. I understand what we've been talking about so far that, hey, I'm going to experiment locally. But then eventually I need to run all of this sort of workflow and series of processing steps on infrastructure that is in the cloud.

• 31:08Daniel Whitenack?

  Could you maybe just walk through like, what does it look like for a practitioner to use Metaflow? Let's say they've written some Python code, they're used to working with notebooks. Maybe they're sometimes used to writing Python scripts that they run on. Maybe they log into a server and they run it. What does it look like for them to install and integrate Metaflow into their workflow? What are the prerequisites and how does the integration happen?

• 31:38Speaker 00

  Let me start with the kind of a data scientist's point of view before getting into deployment and stuff. So I can give you a timely example that just yesterday I was actually creating an example like for the book using Keras. And like this was a new dataset. I was actually like using the NYC taxi dataset, like for the example, it's a fun dataset publicly available. Everybody probably in this situation, I actually started exploring the data in a notebook. Of course, notebooks are still great for like visualizing, like exploring data and so forth. And even like when I started like a drafting, the kind of the model architecture in Keras. I mean, it's really quite nice to be able to iterate that in a notebook quickly. I was like very conscious about, I wanted to kind of introspect that. Okay. So I mean, what are the things that work well in notebooks? What are the things that work well on the Metaflow side?

• 32:19Speaker 00

  And like I kind of faced the same problem that like many other people face, like when using notebooks is that like after maybe three hours of prototyping, my notebook had this like kind of a mixture of cells that I had been executing out of order and like.

• 32:32Daniel Whitenack?

  Who knows what the state is?

• 32:33Speaker 00

  Yeah, exactly. It was super convenient. It was like I kind of felt that I'm in this like a garage hacking something together and like everything is like kind of on the table and it's kind of a messy setup. At that moment, I mean, it made me super productive, but I mean, it was absolutely hundred percent obvious that like there was nothing in that notebook that I would dare to run in production. I mean, like even the idea that I would somehow like run that because it was the kind of my experimentation process that was reflected in that notebook. I would like to kind of think about like production a bit differently. So, so then the kind of the idea like what happened at Netflix as well and like what we recommend people doing is that they have by all means use notebooks for experimentation, for exploration, like for building prototypes. But then at one point, like when you kind of have like a rough idea, like what that workflow could look like, and then really the threshold for that shouldn't be too high.

• 33:18Speaker 00

  You can almost start copy pasting like kind of the snippets. Let's say in this case, I mean, just the kind of fifteen lines of code that define the kind of the grass architecture. Like kind of in a step in a file. And now if you use something like a Visual Studio code, actually it's really easy to have like both the kind of the IDE as well as the notebook side by side. So I can use notebooks for exploration and then like I can still have that like really quote unquote proper IDE like for writing Python code. And again, the idea with Metaflow, what we have had since the very beginning is that like it doesn't require that you know anything more than like what you would need to know in a notebook. So there are like a no new concepts, no new paradigms. You don't have to change the code, the same libraries and all that stuff. So I was able to then like kind of take the best parts of like my experimentation and like kind of put them to this workflow.

• 34:01Speaker 00

  And now like thanks to Metaflow, I'm able to kind of start running it at scale. So I mean, of course, in the notebook, it was rather small data set that I was testing with. And now I could take the kind of the same concepts, the same code, then like start testing that with larger scale. I mean, like I didn't have to wait for my tiny workstation to kind of crunch all the data, but I could like affirm it to the cloud. So overall, I mean, that is quite nice pattern. Like you kind of get the best of the both worlds. I mean, you can use notebooks where they really shine. And then, and then like in the end, an artifact that like you really dare to run in production. Now, I guess the other side of your question was that, okay, so the deployment. So yeah, I mean, like indeed, I mean, the easiest way to get started is that you run pip install Metaflow on your laptop. It works out of the box. There's nothing else that needs to be done. We have also had this belief that the kind of the needs of an organization like

• 34:44Speaker 00

  Grow over time, you don't necessarily like have to kind of have the most battle hard and most scalable setup on the day one. But you can start with something simple and like probably one of the simplest thing is that like you can sign up to AWS batch. And like there are like a three configuration things that you have to set. Or like if you don't want to do it by hand, I mean, there's the Terraform Cloud Formation template. You go to the UI, you click a button and it sets up the kind of the stack for you. And then you can like start like running compute at scale and that's really great. So if you have more than one person working on these things, you probably want to have this centralized metadata tracking. So people can share their results. That's quite convenient. It comes as a part of the cloud formation, not too hard. And then like there's the orchestration system again, I mean, part of the stack, depending like how you want to do it.

• 35:28Speaker 00

  I mean, like you have freedom there to set up in a few different ways. And then like, of course, like the larger the organization, now the largest organizations there, they might care about like setting up the data governance rules, like lifecycle policies. Thinking about that, oh, I mean, like, how do we like harden the kind of the service deployment so it's highly available, stuff like that. But I think like realistically, these infrastructure stacks like they need to start small and they need to be able to grow like with the organization. I think like many systems have the problem that either they are super easy, but then they don't scale. I mean, like as your like, as your company grows, so I mean, you kind of outgrow them at some point. Or then they are like way too enterprisy and like you have to scratch your head about the Kubernetes deployments and whatnot before you can get like even the simplest thing done, so.

• 36:09Daniel Whitenack?

  Yeah, and I'm looking through your documentation, which is great, but it seems like there's a sort of concept in Metaflow where, like you said, in a lot of these workflow management systems, you're writing like YAML, maybe you're writing JSON. You're writing config files and Docker files to sort of manage these various steps, which is definitely doable if you want to get into that. But in terms of an approach that you're taking is this sort of decorator pattern in Python where like you're defining maybe a class that's your data flow and some steps within that class that are decorated with a Metaflow decorator of step.

• 36:50Daniel Whitenack?

  And then you're connecting those different steps within your actual Python code to create your workflow, which is then maybe farmed out to some infrastructure. Did I get some of that right?

• 37:01Speaker 00

  Yep. And like, the interesting thing is that, I mean, there are like so many kind of systems that look like that. And I always stated like the devil is in the details. And many systems like, let's say, I mean, like there are tons of systems that let you specify workflows in YAML. And like oftentimes, they say that, oh, I mean, like you can run any Docker container. And like you can run any code inside the Docker container that makes a step, which kind of like, as a user, like, well, I mean, it's kind of like a push is the hardest problem like to you that, okay, wait a minute. So I mean, like, how do I define what code runs in this container? And like, where do I push my container? And like, what are the dependencies I need inside the container? And how do I move data like between these containers? So in a way, I mean, just like having a workflow, I mean, that's kind of the easy part. That's like why I wanted to have this like a self-contained thing in Metaflow that you have everything in one place.

• 37:44Speaker 00

  So you define the code, you define the dependencies, you define the resources, you define the workflow. Because then you kind of get the unit that you can actually like run the full thing in production and it does something useful. And by the way, I mean, like one thing that I definitely want to mention, which is really important is that this is never a waterfall. I mean, it's never so that you prototype and then you deploy and then you declare mission accomplished. But I mean, if the project is successful at all, I mean, what inevitably happens is that either something fails in production, in which case you have to go back and start debugging, or like the business stakeholder, whoever comes to you saying that, okay, now we want better results. Can we improve accuracy? Can we add this new dataset, whatnot? So you kind of have to start iterating again. And that's really the challenge, like with many systems that like, even if you are able to do that one deployment, how do you come back from production?

• 38:29Speaker 00

  Everybody always talks about going to production, but how do you come back from production? And then like keep iterating and like maybe start having multiple versions running in parallel? So it's all these like a small things that like really matter a lot, like when you think about like running ML like for real.

• 38:43Daniel Whitenack?

  Yeah, definitely. I totally agree with that. So we talked about how you might set up your workflow in Metaflow. You know, you can pip install Metaflow locally, connect it to AWS cloud resources. In terms of that step from like, let's say, because you talked about being in the notebook and then kind of moving into this Metaflow workflow. But then eventually, like, let's say that I create a pipeline that I really like, and I know that I need to run it like triggered when this happens, or I need to run it like every Friday to, you know, do this or that. What does that look like? Because I assume like you could run your Python code locally with these Metaflow decorators.

• 39:27Daniel Whitenack?

  That gets sort of farmed out to resources in the cloud and tracked in various ways. But what does it look like to go from that to some sort of automation or something that's running kind of hands off and you're not running, you know, Python whatever locally?

• 39:43Speaker 00

  Yeah, I can paint you a picture that I saw at Netflix that worked really well. Now, I know that like not many companies are yet at this stage. I do hope that like kind of the world will advance over the next three years or so. But I do think like what's really, really useful is that you have some kind of a centralized workflow scheduler. I know that like many organizations are struggling with this question that like, should they have like a many different infrastructure stacks? And like kind of a different departments have their own and ML has its own and data engineers have their own. But the fact is that like ML is not an island. And especially if you want to use these things in production. I mean, like producing like real business value, you have to integrate with whatever is the outside reality out there. So there's a lot of value in having kind of a decentralized system.

• 40:26Speaker 00

  And like what the centralized system needs to do is only to take care of like this seemingly simple task, which is that like you have workflows and like this system needs to kind of keep the workflows running, farming out then the compute to to kind of whatever like is your compute backend. I mean Kubernetes or AWS patch or whatever. And now the beauty of this setup is that oftentimes these data science workflows run in tight conjunction with data engineering workflows. So you have an ETL and like you can imagine that like you have a maybe daily ETL that like takes some raw data, takes some like whatever streaming event data and like kind of a MoSh assist that to new tables. And whenever that table updates, then like maybe you want to update your models. And then in the best case, like there is a triggering mechanism that automatically whenever the data updates, then like triggers the ML update as well.

• 41:11Speaker 00

  And there's maybe some piece of information like carried around like saying that. Get these other new partitions available, or this is the latest hour, or like however you want to do this. And then like if you have this like a centralized scheduler, if you have this triggering mechanism in place, you can start constructing this almost like a web of workflows that like comprises both of the data engineering side of the house ETL. And even now, if you want to use something like a DBT or create expectations, you can kind of tie that really nicely upstream. So you have like a really nice ETL, like always data quality is there. And then like you have something like, let's say, the ML workflows managed by Metaflow. And then like oftentimes even like what happened at Netflix is that there's kind of the ETL after that it might be that let's say you do

• 41:52Speaker 00

  produce some batch predictions and now you kind of have to load those batch predictions to another place. Or let's say in some cases, like some decision support systems, even like wanting to have those predictions in Tableau or Airtable or something. So you have another piece that then takes those results and pushes them to something else. And now, of course, like as the complexity grows, then like you want to layer like observability tools on top of these alerting tools that, okay, what if somebody, something is late? I mean, like, how can you trace like what's going on? And of course, I mean, there's like a lot of additional like infrastructure that you need there. And now, if I'm looking at this like workflow orchestration landscape in the kind of the world overall, I mean, I think we are not quite yet there, like in many companies, many companies have maybe multiple air flows that are not connected.

• 42:33Speaker 00

  Many companies still use this like cron based scheduling that like it's always like runs at 3am, no matter what, it like runs at the same time, which is kind of silly. Also, the observability parts are missing, but I think that that vision is really great. I think it really helps a lot in like kind of tying the ML like really close to the rest of the organization. So it's not like Ireland, like in some world garden somewhere. So.

• 42:54Daniel Whitenack?

  Yeah, well, I appreciate that very much. I know that we've covered a lot of different topics today. I do want to mention again your book, which is Effective Data Science Infrastructure, How to Make Data Scientists More Productive. We'll include a link in our show notes to that because this is something our listeners, I think, will really get into because it is so practical. I'm just looking at the flow of your book right now, which goes all the way from sort of notebooks to workflows to Metaflow to production. Scaling up, which I think is a super practical book. So thank you for your work on that. Our listeners, you'll definitely want to check it out because we do have a 40% off discount code for the book from Manning.

• 43:38Daniel Whitenack?

  You can use pod practical AI 19, the code pod practical AI 19 for 40% off of the book, which is pretty cool. Well, maybe to end, we usually like to ask our guests some future looking question. And I think you've already started to go there in terms of where you would love to see infrastructure go. But maybe in terms of data scientists and the infrastructure that they're working with, what is what you're hoping to see maybe in a couple of years in terms of data scientist workflow? How do you see that abstraction layer advancing and changing over that time period?

• 44:16Speaker 00

  Yeah, I think that there is the work to be done at all layers of the stack. Again, I mean, as mentioned a few times, like during this episode, I mean, like I'm excited about the fact that like we have so much compute available. So I think like we can make that even easier. That's exciting. Definitely a lot of work to be done on the orchestration side, like tying together ETL, ML. I mean, there's a, there's like a lot of work to be done there. Overall, I think like at the higher levels, I think that of course, the fact is that like many companies, and if not most companies out there are still like struggling with the questions that like, how do they use ML to power their business, not only like produce demos? And I think that that goes back to that, like even like organizational like mindset change, like with the experimentation culture and like, how do you divide work between engineers and data scientists, data engineers?

• 45:00Speaker 00

  So I'm super curious to see how that evolves. And like I'm already now, and I'm talking to companies, I mean, I'm always fascinated by all kinds of ideas and like, all kinds of like kind of a business opportunities that like people are coming up with. And like some ideas, of course, like don't end up working so well. But I mean, there are like some like amazingly like promising ideas out there. And I'm sure that like kind of this will only grow 10 fold, 100 fold over the next three years. So I think it's like pretty much inevitable. And like the kind of a parallel that I always draw is this, like kind of a to e-commerce and like the web. I mean, like back in 2000, like even setting up the e-commerce store, like it took a lot of engineering work. Today you just like a sign up to Shopify or like kind of a you go to Squarespace. And like, you don't have to write a line of code and you can get something that works amazingly well.

• 45:43Speaker 00

  And I think it's inevitable that like we will enter like with machine learning infrastructure as well. But maybe it will take another five to 10 years.

• 45:50Daniel Whitenack?

  Yeah, well, I definitely look forward to that time. That'll be a good time. But thank you so much for joining us, Vila. It's been really wonderful to chat about your projects and your thoughts on data science infrastructure. I look forward to seeing how Metaflow grows and what you do in the coming years. Thank you so much. Thanks, Daniel.

• 46:11Speaker 03

  Thank you for listening to Practical AI. We have a bundle of awesome podcasts for you at changelog.com, including our brand new show, Ship It with Gerhard Lazu, a podcast about getting your best ideas into the world and seeing what happens. It's about the code, the ops, the infra, and the people that make it happen. Yes, we focus on the people because everything else is in implementation detail. Subscribe now at changelog.com/shipit or simply search for Ship It in your favorite podcast app. You'll find it. Of course, the Galaxy Brain move is to subscribe to our master feed. It's all changelog podcasts, including Practical AI and Ship It in one place. Search changelog master feed or head to changelog.com/master and subscribe today. Practical AI is hosted by Daniel Whitenack

• 46:56Speaker 03

  and Chris Benson with music by Breakmaster Cylinder. We're brought to you by Fastly, OnStarfleet, and Linode. That's all for now. We'll talk to you again next week.