MLOps and tracking experiments with Allegro AI

DevOps for deep learning is well… different. You need to track both data and code, and you need to run multiple different versions of your code for long periods of time on accelerated hardware. Allegro AI is helping data scientists manage these workflows with their open source MLOps solution called Trains https://github.com/allegroai/trains. Nir Bar-Lev, Allegro’s CEO, joins us to discuss their approach to MLOps and how to make deep learning development more robust. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean’s developer cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, team accounts, worldwide availability with a 99.99% uptime SLA, and 24/7/365 world-class support to back that up. Get your $100 credit at do.co/changelog https://do.co/changelog. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We move fast and fix things because of Rollbar. Resolve errors in minutes. Deploy with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Featuring: Nir Bar-Lev – LinkedIn https://www.linkedin.com/in/nirbarlevChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Allegro AI https://allegro.ai/The “Trains” Platform https://github.com/allegroai/trainsTrains demo server https://demoapp.trains.allegro.ai/Trains video tutorials on YouTube http://youtube.com/c/AllegroAIUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  When we talk about ML ops, we talk about the ability to move from the data scientist on machine or laptop to training models at scale on some remote machine cluster. We're talking about the ability to orchestrate that and do that within a larger team, not just a single data scientist. And we're talking about the ability to automate that process. That in general is what we talk about when we talk about ML ops. How is it different than DevOps? Well, it's actually very different.

• 0:31Speaker 01

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 0:49Speaker 01

  This episode is brought to you by Digital Ocean, droplets, managed Kubernetes, managed databases, spaces, object storage, volume block storage, advanced networking, like virtual private clouds and cloud firewalls, developer tooling, the robust API and CLI to make sure you can interact with your infrastructure the way you want to. Digital Ocean is designed for developers and built for businesses. Join over 150,000 businesses that develop, manage, and scale their applications with Digital Ocean. Head to do.co/changelog to get started with a $100 credit. Again, do.co/changelog.

• 1:36Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:01Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International. And I'm joined, as always, by my co-host, Chris Benson, who is a Principal AI Strategist at Lockheed Martin. How are you doing, Chris?

• 2:20Chris Benson?

  I am doing OK. It's summertime here in Georgia, and so it is hot and humid. And so I'm just trying to keep from melting.

• 2:27Daniel Whitenack?

  Yeah, not unexpected for where you are.

• 2:31Chris Benson?

  Happens on a regular basis. Hot, humid, not the melting part.

• 2:35Daniel Whitenack?

  Yeah, so I think I've mentioned this a couple of times on the podcast, but my wife owns a candle business. So there's always this, like, during the summer, you've got to figure out the right shipping and tracking so that you kind of minimize the likelihood of candles melting on people's porch before they actually get into their house. If you're sending them to like Texas or Arizona or that sort of thing.

• 2:56Chris Benson?

  That's a good point.

• 2:57Daniel Whitenack?

  Yeah, so it's an interesting thing. On another, you know, shipping front, I've got a pile of boxes sitting next to me and a computer case. All the components for a computer are here at my house. So I'm about to build a first AI workstation of my very own. So I'm excited about that.

• 3:17Chris Benson?

  Very nice.

• 3:17Daniel Whitenack?

  Yeah, it would be fun to have an episode, you know, detailing all of the mishaps that happen along the way as I hopefully, hopefully don't ruin it, but get this thing running, I'm sure.

• 3:28Chris Benson?

  So I'm curious, since you brought it up. Yeah, I know in the past, when we've talked, we both have typically gone to cloud services, especially for personal things that we're doing at home for our own interests. What caused you to decide to go this way this time with a desktop?

• 3:42Daniel Whitenack?

  I think it was twofold. I think partly it was like, I haven't built a computer since I was in college, probably, which would have... I don't know, that's over 13 or 14 years, probably 15 years since I built a computer, maybe. I thought it would be fun to just do it again. So that's partly just fun. But then also I'm getting into a lot more of audio models, so speech recognition things and spoken language identification.

• 4:08Chris Benson?

  And the data sets associated with those are quite large.

• 4:12Daniel Whitenack?

  And so sort of carting those around to various cloud machines and also running models for, you know, maybe days instead of hours starts to get fairly expensive. So I think those two things made sense to me.

• 4:28Chris Benson?

  Well, good luck with it. Yeah, we'll definitely have to get an update from you to share with us all what happened and what went wrong and what went well.

• 4:35Daniel Whitenack?

  For sure. And today we're going to keep the practical train moving with some more topics that are extremely practical. Actually, I had seen what we're going to be talking about today, which is some tools from a company called Allegro AI. One of my friends pointed me to that, which I'll mention maybe a little bit later on. But I also saw PyTorch mention recently that Allegro Trains, which is one of the ML Ops and experiment managing versioning things that we're going to be talking about today, joined the PyTorch ecosystem project. And I thought that sounded really exciting, also very practical.

• 5:14Daniel Whitenack?

  So today we've got with us Nir Barle, who is the CEO and co-founder of Allegro AI. Welcome, Nir.

• 5:22Speaker 00

  Thank you for having me, guys.

• 5:24Daniel Whitenack?

  Yeah. Before we jump into all of those exciting things about experiment tracking and versioning and ML Ops and all of that, it'd be great to hear just a little bit about your background and how you got involved in this field.

• 5:38Speaker 00

  Sure. You know, I've been in the high tech industry for longer than I'd care to actually say, probably three decades. I started as an engineer, actually. And spent about a decade on large ERP systems and that kind of thing. This is way back. And then by way of an MBA warden, I joined Google and a decade at Google, doing everything from working on the mobile team. This is right after Google had bought Android and, you know, before the iPhone went out. And to actually helping setting up Google's Tel Aviv R and D Center to leading Google's European search advertising, product and strategy, and a number of other roles.

• 6:21Speaker 00

  And after all, I was a GM of mobile payments. And yeah, when I decided to look for something else to do, I joined two folks who are, you know, actually my partners now to basically start Allegro AI. The way I came at it is I was looking to do something big that can impact the world and that would involve cutting edge technology. I mean, after being at Google and doing everything it did, you know, you don't want to do anything less than that, really.

• 6:49Daniel Whitenack?

  Yeah, I was gonna say, being at Google, if you're thinking of projects that make an impact, a sort of worldwide impact or innovative, it seems like that sets a pretty good trend for, for your path forward or a high bar to reach for sure.

• 7:03Speaker 00

  That's correct. So, you know, I don't know if I'll be able to build a company as large as Google, but certainly that's the target you, you want to put for yourself.

• 7:10Daniel Whitenack?

  Yeah. Yeah, and it is interesting. I mean, it seems, and I don't know, maybe you have a perspective on this as a CEO and founder, but it seems like there are a number of really innovative startups in the AI space that are kind of playing at the same level as the major players of, you know, OpenAI and Google and Microsoft and these at the same level at major research conferences, you see kind of startups, I'm thinking of like Hugging Face or those sorts of startups that are really right there. And it seems like such a huge impact for a small team. And so, I don't know, as a CEO, if you think about those things, but it, it seems really

• 7:49Daniel Whitenack?

  interesting to me that there, there can be these small, really focused teams that make a very large impact on that level.

• 7:59Speaker 00

  Yeah, you know, being at Google, I have to tell you, you know, you kind of think that you can probably do anything, right? The reality is that. And I've seen that personally on some of the big projects that I was involved with is that, you know, Google didn't execute as well as a startup or as fast. Google ended up, you know, acquiring them for less or even a lot money, right? And there are a number of examples for that. And as a company grows larger, the targets get bigger, right? And so doing anything requires a very, very high bar. I remember at some point I'm talking about like 2007, I think. I remember pitching something to the time with Susan Wojcicki, who was at the time, I mean,

• 8:43Speaker 00

  now she's the CEO of YouTube. She was the head of advertising. And the bar was, you know, if it's not about a hundred million dollars revenue, don't talk to me about it, right? And you can imagine this is back in 2007. So imagine today. And this gives an opportunity for small companies who are very nimble to identify opportunities. There's also a different perspective when you're outside of Google as when you're in, especially I think in the B two B space, there are opportunities where, you know, at least Google specifically is still relatively behind companies such as Amazon, for example, or Microsoft, right? So you can identify a small company's niches, and if you understand that those are going to grow, then that's opportunity.

• 9:27Chris Benson?

  So kind of curious, as you were at Google and, you know, how did you come up with this idea for what would become Allegro? And so you're kind of, you had been doing that at Google for a while, moving through your position. So what made you think, I have this idea, I'm going to make a major change in my life? You know, what gave you the motivation to go off and do a startup, find partners? Can you give us a little bit of that backstory?

• 9:52Speaker 00

  So first of all, I can't really take credit for the original idea behind Allegro AI. That's actually one of my partners. You know, I guess I can take credit in what we formed out of it and what it became because obviously, as in any company, especially startups, you know, we change and we adapt quickly to find product market fit. So, so obviously, the vision as it was set or thought of by my partner needed to improve and get better. And, you know, that's something that I was involved in. But the original idea was not mine. It was more of, for my position, you know, I'd felt like I had, you know, I'm Tel Aviv, Israel, and it was about relocating my family back to the US.

• 10:35Speaker 00

  And at the time, this was about four years ago, it didn't make sense. And coupled with the fact that, you know, I joined Google when it was 3,000 people. I think it's about 100,000 now or so, you know, on that kind of scale. It's a different company in many ways. And I'd felt like it was an amazing experience and I learned so much, you know, especially, you know, being at Google in that time of growth. But, you know, when I left Google, it was a big company with all the things that we all less like about big companies. And I felt like, you know, this was an opportunity to do something different and really go out and try to build something on my own. And as I mentioned, I looked for something really big that could change the world.

• 11:18Speaker 00

  And, you know, basically as a potential founder, I started, you know, quote unquote, dating people, right, to find partners that we could, you know, come up with something that we'd like to do. And through that quote unquote dating process, I met my current partners. And it was, you know, we hit it off, as you'd like to say, really quickly. They're amazing guys. When you're in a startup and you have partners, it's, you're practically in a Catholic marriage for the time until the exit. And so you want to make sure that you have people that you can trust and that they're, you know, great people that you can work with and obviously amazingly capable and talented. And I found all of that with them. And basically, you know, as I mentioned, one of my partners was the one that, you know,

• 12:02Speaker 00

  was bringing that idea and it came to him. He's a long time serial entrepreneur. He's a very interesting profile where he has both a very, very strong engineering background as well as a data science background. So the most prestigious lab today in AI in Israel. So it's run by Professor Lear Wolf, who's actually now at Facebook. And my co-founder, his name is Moses Goodman, was his first, you know, PhD student, which basically means that they, you know, they set up their lab together. And so he's really one of the pioneers of deep learning, machine learning, computer vision in Israel. And he basically saw what Allegro really is all about is the fact that we need to bring in engineering.

• 12:46Speaker 00

  Methodologies into the AI in the process, right? That was not the way that he said it at the time, but basically that's, that's the idea, right? How do we actually scale things up?

• 12:56Daniel Whitenack?

  And I'm kind of curious on that front and you've stated it well. And I, I think that this has been brought up on our show multiple times from different perspectives. So I definitely think it is a theme that's kind of surging through the community that we, we need to be more rigorous in terms of the engineering we put into, into our workflows and, and the AI driven products that we're building and putting out and the tools and all of that sort of thing. I was wondering from your perspective, what you see as the challenges to kind of, what are the sort of main challenges to getting people on board that are currently in data science and AI positions? And kind of convincing them that they need to start doing things differently? What are some of those challenges?

• 13:41Daniel Whitenack?

  Does it have to do with this kind of variety of backgrounds that people come from? That it's not just engineers or is it more than that?

• 13:48Speaker 00

  Yeah, that's a great question. And the answer actually, it's a moving target because our industry, right? The one that you guys are talking about and the one that I'm squarely in is rapidly evolving and changing as we speak at an amazing rate. I've never experienced that kind of rate before my career. So generally say this, right? I mean, basically, right? It's a very different paradigm, right? It's scientific paradigm, right? And initially people thought, well, you know, I'll get data scientists or research scientists and that's what I need, right? And then they'll be able to do the job. Obviously we all know that's not enough. The thing is that there's still, you know, a core and critical part of a team that needs to build something.

• 14:28Speaker 00

  But data scientists, research scientists have a very different, you know, mindset and outlook, right? I mean, they've been trained differently, right? I mean, they're, at the end of the day, they're scientists, right? And if you actually, you know, take it to the extreme, think of that, you know, mad scientist, nothing is in order. You know, everything is hectic. It's all about the creativity and finding the, the solution. And there's a lot of truth in that. Obviously that's an extreme exemplification, but there's a lot of that and that's changing. But we found, you know, throughout the course of the last three years of, you know, data scientists, research scientists have been very much against adopting any tools because, you know, they came out of university. They were focused on, you know, on the science.

• 15:12Speaker 00

  Tooling, they didn't understand the value of tooling, the value of processes. In some ways, you might say they, you know, maybe even, you know, were a little bit wary, right, of tools. And is that going to be good for them or bad for them? It wasn't even something that they were exposed to during their curriculum of their training. On the flip side, you know, they felt like, you know, I'm a, for example, PhD out of whatever Stanford, I mean, I should know everything. A lot of times, we saw relatively very junior data scientists leading AI teams, not just in small companies, in very large companies, right? Because if you're not a Google or a Microsoft or a Facebook, you're not going to get the cream of the crop.

• 15:53Speaker 00

  And the last thing is, you know, their bosses didn't know what the heck they were doing. They didn't even know how to actually measure what they were doing. And as I mentioned, they thought that, you know, bringing those people in would be enough. And so a lot of that created the situation where, you know, the background of, you know, why do I need tools? And a lot of that still exists now, I think, but, you know, a lot of people who have engineering background have actually sort of doing data science or, you know, ML engineering and data engineering because, you know, because it's new, it's interesting, salaries might be higher, et cetera. Companies have realized that they're not seeing productivity out of the data science teams. And so that shift has been happening in the last, I guess, year or year and a half.

• 16:36Speaker 00

  We've seen companies integrate, right? Their data scientists and research scientists into a larger product team, right? That has the engineers and the product leadership, et cetera, DevOps to really, you know, push them to ultimately build a product. Because it's not about coming out with a research paper, right? Ultimately, if you're sitting in a company, most of the time, it's about building a product or a service. And so I think now, what we've seen is oftentimes a situation where there's a very big underappreciation.

• 17:12Speaker 00

  Of what it takes to build a state of the art tool chain to support you. And I remember talking to someone that was way back in the day was pushing, you know, SQL databases, imagine that, right? I mean, this is prehistoric times. And he was telling me how he had trouble pushing that into organizations because they thought they were going to build it themselves. Obviously, you know, anyone who tried to do that fell flat on their faces. Same thing here. And we've had situations, you know, that's changing a lot, but we've had situations where, you know, a couple of years ago, companies would tell us, what are you talking about? I can build this in three weeks. I mean, and they believed that, right? After we showed them what we built.

• 17:56Speaker 00

  You know, today, a lot of these companies, because tools didn't necessarily exist, or they weren't aware of them, or they thought they could build it, you know, have invested internally and built something. And you know how it is, right? Not invented here. And once you've built like a small toy, you're enamored with it, especially, you know. But for this sake, I'll say us engineers, right? I remember as an engineer, I was enamored with some of the things that I built. And so that's kind of the hurdle that we as an industry that, you know, is building and pushing tools need to try to get around or over.

• 18:44Speaker 04

  Changelog News is the best way to keep up with the fast-moving software world. We track, blog, and contextualize the coolest projects, the best practices, and the biggest stories each and every week. Make changelog.com your daily destination, or hit the snooze button and subscribe to our weekly newsletter that hits inboxes on Sunday mornings. Join more than 15,000 enthusiastic readers. It'll cost you exactly $0 and you can subscribe right now at changelog.com/weekly.

• 19:22Chris Benson?

  So Nir, I guess as we were, we were starting to get into tools and you were talking about, you know, whether organizations were starting to recognize the need for tools and how did they get productive and measure that productivity and that, you know, in a world that, you know, already has things like DevOps and ML engineering and data engineering and such. We're kind of moving into that area. I noticed that, you know, kind of front and center on your website, you have this concept of ML Ops. And as you were mentioning DevOps in passing and the tooling before, it really kind of triggered that. I'm wondering if you can kind of tell us what ML Ops means to you and the organization and kind of how does that differentiate itself from DevOps on the software side and other types of ML engineering and data engineering?

• 20:08Speaker 00

  Absolutely. So actually, that's a great question because it actually touches on one point where, you know, ML Ops itself as a term is not something that is set already. And different companies are using it to mean slightly different things. You know, that's one of the, I guess, issues again with our industry. It's so, so early on that terminology is not set. When we talk about ML Ops, you know, we talk about the ability to move from the data scientist on machine or laptop, right, to training models at scale on some, right, cluster, right, remote machine cluster. We're talking about the ability to orchestrate

• 20:51Speaker 00

  that and do that within, right, a larger team, not just a single data scientist. And we're talking about the ability to automate that process. That, I guess, in general is what we talk about when we talk about ML Ops. How is it different than DevOps? Well, it's actually very different. So I guess, you know, let's define DevOps at a very high level, right? I mean, basically, the idea behind DevOps is that you want to make sure that a piece of software That usually is already tested, weighed and stable, right? Uh, that has left development uh and is now going into production to serve

• 21:33Speaker 00

  users or, you know, uh, workloads need to work at scale uh and needs to stay up all the time, right? And you need to make sure that it can sell across many machines, et cetera. That at the end of the day is what DevOps is tasked to do. And so basically, what did we say here, right? We said that there was a single piece of software that it was tested and it works and that you need to take that and you need to scale that up and replicate that, right? And that happens only in production. Well, in AI, everything around that is actually different. So first of all, as you guys know, right? Um, machine learning, deep learning experiments can be very, very heavy workloads. I mean, you actually mentioned that yourself when you're talking about building your own computer at the beginning, right?

• 22:18Speaker 00

  Um, um, you're going to run things that are going to take hours, right? Or even days. And so unlike regular software, you need to be able to run stuff on large machines from day one, right? In terms of development. Uh, so that's one big difference. The second thing is what you're doing is you're running software that's not tested because you're doing it during development. The third thing is you're running experiments. And what you're trying to do, right? Obviously is want a lot of experiments because that's the whole process, right? You're doing lots of experimentation until you reach your goal. And so with experiments, you're basically running pieces of code that are slightly different from each other. And that's a different thing than running the same piece of code, right? On lots, lots of machines.

• 23:03Speaker 00

  And so basically, this is a very different problem. How do I, as a data science team, manage my workloads on, you know, clusters of machines? Uh, how do I handle lots of experiments that I need to run from, you know, one or more data scientists or a team of data scientists? Uh, and do that effectively when we're talking about pieces of code that continually change? How do I actually take the environment that I built? Because in AI, again, the piece of code that you're running actually is much more complex on one dimension than a regular software. Because really, it's an amalgamation of the model, right? The neural network, for example, right?

• 23:45Speaker 00

  The code that wraps it with the data, right? How do I actually take that environment that I built? My research, you know, X Reech researcher, right? And the model that she built and then run it on a remote machine that has a different environment? And so all of these are different challenges. And this is the challenges that we attempt to solve. And this is what we call ML ops.

• 24:08Daniel Whitenack?

  Yeah, that's a really good summary. I like how you set that up in terms of the comparison to DevOps, because it is kind of maybe a shock for people starting to get into this field where, like you say, from day one, in order to actually make progress on their things, they might have to know about, oh, spinning up this GPU instance in the cloud or, you know, CUDA libraries and running things in a repeatable way. It seems like a really high barrier for people to overcome, you know, from day one to get things working and also do it in a, in a repeatable way.

• 24:47Speaker 00

  Exactly.

• 24:48Daniel Whitenack?

  Yeah, I also wonder on the, like, you're talking about experiments and that sort of thing. I know one thing that is definitely true of myself and, you know, my wife could confirm is that I'm not very good at remembering what I've done or what needs to be done, right? In terms of the experiment tracking side of things, of course, there's like the running of things, which is definitely important. I think that's what maybe you focus on mostly, but there's also kind of this weird documentation almost piece of the puzzle. It's not quite documentation, because it's like a very specific type of documentation that's really documenting like what have I done and what haven't I done and how successful was that?

• 25:32Daniel Whitenack?

  And it's not really like you want to have a research paper necessarily, especially if you're developing these things as a product or maybe even a trade secret. Especially if you're on a team, want to have that common understanding of what has been done and hasn't been done. How soon do you see teams encountering that issue when they start working on this problem? And what? Are those kind of essential elements of, I guess, more of the documentation or tracking side of things that need to be in stone somewhere over time?

• 26:02Speaker 00

  Yeah. Well, you know, as you were saying about documenting and how it's not exactly documented, if you come up with a term, please let me know.

• 26:13Daniel Whitenack?

  Naming things is the hardest thing.

• 26:15Speaker 00

  Exactly.

• 26:15Daniel Whitenack?

  Doc Ops.

• 26:18Speaker 00

  Yeah.

• 26:19Daniel Whitenack?

  Doc Ops. That's probably already taken. That has to be taken.

• 26:24Speaker 00

  But that's, that's a real issue. So, you know, that's a real struggle to name it. So, a prefix and say, actually, you were saying, you know, we were focusing more on the ML Ops. Actually, you know, one area where we're pretty unique is that we have a very highly integrated solution where we think that you can't focus on just one thing. If you don't have a highly integrated platform that actually takes care of both of the experiment management, the data management, the versioning, and the ML Ops, you don't have the best scalable solution. But we can talk about that later if you'd like. The experiment management part of it, the documentation part of it, when do people realize that they need it? The answer is actually when someone in the team that usually has some sort of engineering background says, "Stop. This is crazy." You know?

• 27:09Daniel Whitenack?

  Yeah.

• 27:09Speaker 00

  That's exactly the point.

• 27:11Daniel Whitenack?

  I remember Doug, if you're out there, his name was Doug. He's a great engineer. One of the startups I work with. He was my wake up call.

• 27:21Speaker 00

  And so, and so, you know, I mean, it could happen with a team of one, and it can happen. We've seen it happen. Well, we've actually seen teams of tens of data scientists that didn't have that, right? And it really depends if you have that person who realizes that and has the influence and or power to actually say, you know, "We need to change this."

• 27:42Daniel Whitenack?

  Yeah, yeah. And

• 27:45Daniel Whitenack?

  I guess this is something that we've kind of talked about in passing, but that's this interaction between AI developers or data scientists and the rest of an engineering organization. So maybe a follow-up question to Chris's question about differentiating ML Ops and Dev Ops. What is the kind of integration point from your perspective between the two worlds? Because if things eventually end up in a product, right? Like I'm importing a model into some API handler in some code that is production product code. There has to be an integration point somewhere. Where does that exist and what challenges are at that integration point?

• 28:26Speaker 00

  That's a great question. Actually, the integration is something that happens continuously if you're actually running things well. So it's exactly as I said, right? I mean, ultimately, what you want to do is you want to take this model, right? That you build to predict something or to solve something and then integrate that into, call it a wrapper or, you know, some larger piece of code that actually carries out the ultimate task of that product. The thing is, oftentimes, you know, you could test your model a lot, kind of like in a very environment that's kind of clean. But ultimately, you're going to want to test it in the field, then you're going to have to have that wrapper. The other point also is that once you want to get into automation, right?

• 29:08Speaker 00

  And even if you're still within the data science part, if you want to get into automation and create lots and lots of experiment and you want to, maybe you're actually fielding in continuously new data that's coming in. Right? Let's say you're a, you know, you're building an autonomous vehicle and you're, you're getting constantly new, new videos from your cars driving around. And you want to actually improve your, your models based on that, then that also creates an integration point. So the integration points is on those two levels. One is, you know, when you have to. Hand over the code so that it gets wrapped. And two, when you actually want to integrate those experiments within a larger pipeline that helps improve them.

• 29:50Speaker 00

  And there's another point that we actually try to facilitate with our product, which is how can I lower the barrier to entry? And I'll explain. Let's say you're a company and you're doing, you're building a solution to, I don't know. Let's take computer vision. It's easy. I mean, let's say you're building, you know, something to identify cats. Right? Let's take the ultimate example. Right? But you also need to identify dogs because you're building a pet detector, whatever.

• 30:19Daniel Whitenack?

  You're speaking right to Chris's heart. Chris is a man, the lover, so.

• 30:23Chris Benson?

  I was keeping my mouth shut this time. Yes, I am. Daniel can't normally shut me up on that, so go for it. Let's hear.

• 30:30Speaker 00

  All right. You know, as data scientists, you understand that if you've built a model, and I'm talking about the code now right now, right? That facilitates object detection for cats. Well, if you now want to do the same thing for dogs, what you need to do, right, is you need to take that code that you built for the experiment and probably the same neural network that is the one that you chose for, you know, identifying objects in whatever scenario and now marry it with a different data set. That's it. Why would you necessarily need a data scientist for that? Why couldn't an engineer do that? And that's behind a lot of the stuff that we're doing also. Right?

• 31:11Speaker 00

  The ability to actually have the data scientists work on the core pet detector model and then have engineers facilitate optimizing that for the different objects.

• 31:26Daniel Whitenack?

  Yeah, and I think that also actually, that example itself illustrates another kind of unique feature of this. I think you're right in that those later stages could be. Kind of the popular word, I guess, is democratized to other people within the organization, right? But also, it's still not quite the same as like a normal DevOps in that like if you're running with a different data set, somehow you need to have a kind of unique tracking that's going on with like what data set was used to train this particular artifact or serialized model, you know, at what time? Because the code might actually be exactly the same, right? The difference might be in the data.

• 32:09Speaker 00

  Exactly, exactly.

• 32:11Daniel Whitenack?

  Yeah, so I see so many people like develop really sophisticated kind of naming for their files and such, which you'd probably need your own documentation to document that. What about the data side? We mostly kind of talked about process and the operations, infrastructure. What about the data side of things?

• 32:31Speaker 00

  So the data is the holy grail. At the end of the day, and I think that, you know, obviously experienced and senior data scientists get this, right? It's all about the data. Novices are focused more on the models. But at the end of the day, you know, the difference between a product that meets the threshold of, you know, whatever KPIs you want it to hit and something that doesn't is about your ability to, you know, train it on the right data set, right? And be on top of your data and be able to feed the exact, what we call data view, to train that model. And so iterating on the data, right? Identifying the SKUs within the data and handling those, identifying the holes where

• 33:16Speaker 00

  you need to add more data or, you know, build synthetic data, right? Or augmentations around that. That is the key piece. And as, you know, we talked about this, right? It's an experiment process. And so being able to actually version that and track that, because as an experiment process, you know, you're going one track and then you realize, you know, what actually you want to go back to the model I built two months ago and actually take a different direction? You have to be able to version that or not just the model, you want to be able to version the data set. I mean, if you have enough experience as a data scientist, you know that you're always going to find data sets that work better for whatever reason, and you don't even know why. You don't know why, but whatever did.

• 34:01Speaker 00

  I mean, you know, there are so many examples of data sets that are quote unquote wrong because you know, the metadata on them isn't necessarily correct, but somehow they produce better results and you know, data set that's better, right? And so you have to version your data. You have to version the data, not just the files, but the metadata around that so that you can effectively go through that process and make sure that you're building the best solution that you can.

• 34:45Speaker 04

  If you like this show and you aren't listening to the change log, hey, let's fix that bug. The change log is our flagship show and we've been doing it for over a decade. Adam and I seek out and interview the people who are pushing the world forward with software. We dive deep into the hacks, the innovations and the leadership required to do what these amazing people do. One recent example is our conversation with Anders Damsgaard, a climate scientist from Denmark who gave us a peek inside his work and how he scratched a common itch he has when gathering academic research from around the web. Here's a dorky moment from that episode.

• 35:20Speaker 01

  Are you trying to be right or are you trying to solve the world's problems?

• 35:23Speaker 00

  Exactly.

• 35:24Speaker 01

  If you're a scientist trying to be right, well then your right may not actually be the right.

• 35:28Speaker 00

  Yeah, exactly. There's another saying, all models are wrong, but some are useful.

• 35:34Speaker 04

  I like that one. There's another saying, all models are wrong, except for mine. Mine's correct.

• 35:42Speaker 04

  We had a lot of fun with Anders. He's a fascinating guy. Continue listening at changelog.com/podcast/378 or search for the change log on your favorite podcast app and find the episode called open source meets climate science.

• 36:18Daniel Whitenack?

  So before we got to the break, you were talking about versioning the data and I wanted to kind of let you finish that thought. And then I actually wanted to also explore kind of how Allegro is moving ML Ops in a practical way, like what you're actually focusing on and how you're implementing ML Ops. But if you'd finish your thought on data versioning, we'd love to hear it.

• 36:40Speaker 00

  Sure. You know, with respect to data versioning, at the end of the day, you know, we think that that's, as I mentioned, the holy grail, being able to have a set of tools that enables you to effectively manage your data sets and their versions, and effectively also be able to obfuscate the connection between the code and the data, so that we can facilitate, for example, the ability to move from a cat detector to a dog detector, because now you're using a different data set. And again, as a data scientist, you all know that taking one data set with a code and actually switching it to a different data set is not as trivial as one would like it to be. And so, those are some of the goals

• 37:23Speaker 00

  that we set out about to do with Allegro. Trains and really the ability to actually switch between the data sets and the code and the models as easy as plug and play.

• 37:34Daniel Whitenack?

  Yeah. So what is the kind of, I guess, range of things that Allegro focuses on in its actual offerings? So I know that there's the Trains project, which was mentioned in that tweet that got me interested, that joined the PyTorch ecosystem project. So how does that fit into the wider scheme of what Allegro's offering? And how does a data scientist interact with it, I guess?

• 38:00Speaker 00

  Sure. What we provide is we provide, you know, a platform or a tool chain or a set of tools that basically takes care of the experiment process.

• 38:12Speaker 00

  The ML Ops part of it, the ability to actually scale and actually run things effectively, and the data. And the full platform, which isn't completely available as open source, basically has all these key pieces together, highly, highly integrated. What we've open sourced, or what's available as an open source project, is the experiment management part of things, which is all about the documentation we talked about, the ability to document things, version your models, your experiments, your hyperparameters, everything around that, reproduce, compare, et cetera, and everything that has to do with the, I

• 38:51Speaker 00

  guess, basic ML Ops, the ability to actually manage a cluster, whether it's on-prem, or on-cloud, or combination, by a team of data scientists, and really self-help themselves with orchestration and scheduling, et cetera, and automation on top of that. And then some basic, actually, it's not basic because it's on par with whatever else is out there, but data management, or what we call data tracking at least, is available in the open source, where the enterprise version adds on top of that much more sophisticated data management, more sophisticated scaling and data pipelining on top of the platform,

• 39:33Speaker 00

  so that companies can actually build a specific pipeline for what they need. And obviously, the standard enterprise-relevant features like user management, permissions, managed services, all that stuff.

• 39:47Chris Benson?

  So I'm curious, as you're kind of describing this, and I appreciated you kind of talking a little bit toward what was open source versus what was the enterprise offering. As you look at different potential customers out there, and there is a variety of ways they may implement how they are allocating resources for their own ML Ops prior to you coming into the picture with them, some people are strictly cloud-based, they may be doing Google, or AWS, or Azure, some people are, or organizations is more specific, maybe buying like a bunch of DGXs from Nvidia, and have a cluster set up locally, or some hybrid form. Which of these scenarios does Allegro fit into?

• 40:31Chris Benson?

  And if multiple, how does it change how you would implement Allegro?

• 40:34Speaker 00

  Actually, we fit into every one of those scenarios. Any hybrid scenario that you can think of. And actually, the more complex your environment is, the more Allegro Trains shines. And I'll explain. Basically, the way that Allegro Trains is set up is you have a server backend that basically manages the processes and records and logs everything, and then sets up the instructions for, you know, the clients that are basically what the data scientists connect with, as well as the agents that run on the machines that do the actual training. The system is built that you can set it up on any type of machine for training.

• 41:18Speaker 00

  It could be, you know, DGX. It could be any type of GPU by Nvidia. It could actually be CPU. It doesn't really matter. It can sit on the cloud, on-prem, any combination, on any cloud that you'd like, and it all works. In fact, a significant portion of our customers have a hybrid solution where they have on-prem systems and then they actually burst into the cloud, right? When they have a specific times where they need actually more processing power. And that becomes really effective for them. We have other customers that are completely on the cloud, you know, and everything in between. Why Allegro Trains actually shines? The more that you have a more complex environment is because...

• 42:01Speaker 00

  So on the first level is that the interface to manage these clusters is really, really simple. You can actually try it out. We have a demo server up on the web. The data scientists actually manage queues where they can set up, you know, the machines. I want one, you know, GPU, or I need a cluster of eight GPUs or whatever. And it's completely invisible to them where those machines sit.

• 42:27Speaker 00

  With the enterprise version, we go even further and we provide three layers of software caching and what we call zero data move. So if you have a complex system where you have data in multiple locations, you know, we'll make sure the data goes to the right machine to train that's, you know, close by to it. We'll make sure that there's local caching to it, that it doesn't have to go back again and again. And so the data moves as little as it can. And we can actually, we go even further. You can actually do federated learning on our platform. And so you can actually have data being trained in multiple locations geographically around the world and then combined into a single model.

• 43:06Daniel Whitenack?

  Really interesting. And I think you're kind of getting or hinted at some of these things, but just to, for my own understanding, it sounds like there's the Allegro Trains server, which kind of aggregates all this information, does that kind of experiment management is kind of, I guess, maybe the central brain is a way to think of it?

• 43:26Speaker 00

  Yep.

• 43:27Daniel Whitenack?

  In my understanding, like if I'm, let's say, running, uh, let's say I just have a machine, my own machine, and I have some code on it and I want that to be tracked by the Allegro Train server. I think based on what I was reading, you just kind of decorate that code with a certain snippet that connects to the centralized trains server? Is that kind of the workflow for that scenario?

• 43:52Speaker 00

  Exactly.

• 43:53Daniel Whitenack?

  Okay.

• 43:54Speaker 00

  We try to make it as simple as possible. We dubbed it automagical. Uh, there's a snippet of code. It's basically two lines of code. Uh, you put just once in your code and the header basically, and that's it. You're done. Everything is in track for you. And, you know, that, you could potentially have the server, the training, and your client be on the same physical machine if you'd like. Uh, it doesn't matter.

• 44:15Daniel Whitenack?

  Yeah, actually, um, I'll reveal some of my cards now because a little while ago, so I have a friend here close by geographically in Indiana and we, we kind of have, uh, regular calls to just talk about AI things because, you know, we both work in companies where there's not that many AI type people. And so, um, we like to share things that we're learning and all that stuff. His name's Will. Shout out to Will out there if you're listening. But I asked him, one of the first kind of times we were talking about his workflow and all of those things, we got into this topic of ML Ops and all of that. And he's like, oh, I use this Allegro Trains thing. It's amazing. And I was just talking to him earlier today, actually. And I was like, hey, I'm going to talk to the Allegro Train,

• 44:59Daniel Whitenack?

  the Allegro AI people later today. What do you want me to say? And one of the things he said was, it's just, for him, it's super easy. Like you were saying, pretty low barriers to, you know, add this snippet to your code and kind of things happen automatically, like you were talking about. And the other thing he definitely wanted to mention was that the team is super responsive. And he mentioned raising various things on GitHub and all of that. And the team is very responsive. So, great job. You've got a very happy user in Will here in Indiana.

• 45:32Speaker 00

  Well, thank you, Will.

• 45:35Daniel Whitenack?

  Awesome. But yeah, he was kind of telling me about how some of that works. And then there's also, you mentioned the agents. The agents, those have to do with kind of the more automated runs that happen across a set of shared resources? Or where does that fit in?

• 45:50Speaker 00

  So, the agents are, if you basically want to run your code on a remote machine, you basically set up an agent on that machine, like whether it's a DJX or a GPU or whatever you have it. And that agent is tasked with basically, that agent is then associated with certain queues that you create. It could be associated with either one or more queues. So, it's a little piece of code that sits on any machine that is potentially a target for running your experiments on.

• 46:18Daniel Whitenack?

  So, one of the things I'm curious about, and I meant to ask you this a little bit a while ago when you were touching on it, was with some of the motivation that you had for going with an open source business model that builds an enterprise business on top of that? And did you always know that that was going to be the approach you guys were going to take? Or did you consider any others? And how has that model worked out for you?

• 46:40Speaker 00

  That's a very revealing question for us. When we started out, we probably erred on where the market

• 46:49Speaker 00

  You know, so I guess one of the things that you do in a startup is right, you try to time the market, which I think I saw, you know, several articles talking about, you know, timing being the number one critical aspect of a, you know, startup success and actually one of the hardest to hit, right? And sometimes even VCs call it luck. But, um, you know, we were trying to time the market because what we had built initially was around the holy grail about the data. And we basically built a system with the thought in mind of, well, you know, companies are now doing development, but they're going to get to scale and they're going to have to be able to manage huge data sets that constantly change. You have to version that you have lots of experiments. You've got these things running on multiple clusters. How do you handle all of that? And so we actually set

• 47:32Speaker 00

  out to build this really big, robust system. And then we found out that very few companies were at the stage where they needed this or realized its value. And so we got back and started thinking, you know, where is the industry now and how can we help the industry progress? And we figured that the right thing to do is to meet the industry where it is, which was, you know, before that scale and come up and say, you know, all right, so what are the low hanging fruit of things that can bring immediate value to data scientists out there? And that was first thing was the experiment management. And then immediately after that, the ML ops, or at least the ML ops in its lighter form, right? Not a, don't think of a huge conglomerate running, you know, hundreds and thousands

• 48:17Speaker 00

  of experiments, but you know, even small teams. And we thought that the best way to do that, to really contribute to community, help spur that along, make that, uh, you know, something that a lot of people can do stuff better in the way we think it would be better and helpful. And ultimately, obviously, you know, we're a company, we're about making money, but being able to do that was something that we felt was the right thing to do that will, you know, be a win-win for everyone. It will be a win for the community and ultimately it'll be a win for us because, you know, when, you know, the larger companies that do have money to pay and do feel like they need to get, you know, more, they're going to come back to us.

• 48:54Daniel Whitenack?

  Yeah, that's great. And I think as evidenced by, I think, users and also attention and kind of joining in with the PyTorch ecosystem, like in that blog post and other things, I think. That that really allows people to, you know, solve a pain point that they really have really, really quickly and, and hopefully does eventually spur them on to, especially if they're part of larger companies or teams, you know, integrate more with your enterprise systems. But it's been amazing to talk today. The topic's very close to what I'm super passionate about, I think, Chris, as well. And part of the reason why we do this podcast is to talk about those practicalities of how people do their AI development. So really appreciate you joining.

• 49:35Daniel Whitenack?

  We'll link the demo server and the links to Allegro Trains on GitHub and also your, your main website, which talks about all of your offerings. We'll put that in the show notes for sure. And I encourage people to go there and check out those things and let us know in Slack or on LinkedIn or other places, what you think and how you like what they're doing. But really appreciate you joining, Nir. It's been a great conversation.

• 50:01Speaker 00

  Thank you so much. I mean, it was, it was a pleasure. It was fun and really thank you so much for having me.

• 50:09Speaker 04

  Have you joined the free changelog community yet? I'm not sure what you're waiting for. You get changelog news, email notifications of new podcast episodes, access to our community Slack and Practical AI channel where fun and interesting AI discussions take place all the time, all for the price of a free hot dog. Check us out at changelog.com/community. We'd love to have you. Practical AI is hosted by Daniel White neck and Chris Benson. It's produced by me, Jared Santo, and our music is provided by the mysterious Breakmaster Cylinder. We're brought to you by some amazing companies who get it. Thanks to Fastly, Linode and Rollbar. That's all for now. We'll talk to you again next week.