Machine learning at small organizations

Why is ML is so poorly adopted in small organizations (hint: it’s not because they don’t have enough data)? In this episode, Kirsten Lum from Storytellers shares the patterns she has seen in small orgs that lead to a successful ML practice. We discuss how the job of a ML Engineer/Data Scientist is different in that environment and how end-to-end project management is key to adoption. Sponsors: The Changelog https://changelog.fm – Conversations with the hackers, leaders, and innovators of the software world Featuring: Kirsten Lum – LinkedIn https://www.linkedin.com/in/kirsten-lum, X https://x.com/machsciChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: storytellers.ai https://storytellers.ai/Trello https://trello.comUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I tend to think of these roles, there's not even a letter that describes this, because there's so many, you know, you need to have a relatively shallow but working knowledge of the entire cycle. And so instead of thinking of your role as a data scientist as training models, or even producing models, the role of the data scientist is to convert the data into some business value using data science techniques.

• 0:36Speaker 01

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive, and accessible to everyone. Subscribe now if you haven't already. Head to practicalai.fm for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out at fastly.com. And to our friends at fly.io. We deploy our app servers close to our users and you can too. Learn more at fly.io.

• 1:12Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist at S I L International, and I'm joined as always by my co-host, Chris Benson, who's a tech strategist at Lockheed Martin. How are you doing, Chris?

• 1:26Chris Benson?

  I'm doing just fine. How are you today, Daniel?

• 1:28Daniel Whitenack?

  I can't complain. It was definitely the first meeting-heavy day of the new year for me.

• 1:36Chris Benson?

  Me too.

• 1:37Daniel Whitenack?

  I was really enjoying those like, not everyone knows that I'm back at work and so I can get stuff done days. So yeah, now everybody knows. But all good things. I'm working on fun stuff, so.

• 1:49Chris Benson?

  Hey, we're getting to talk AI now for the next few minutes, so we're good.

• 1:54Daniel Whitenack?

  Exactly, yeah. And really excited to get to have Kirsten Lum with us. She is co-founder and CPO of Storytellers AI. Welcome, Kirsten.

• 2:05Speaker 00

  Thank you for having me.

• 2:06Daniel Whitenack?

  Yeah, it was great to get to connect with you on Twitter and get you scheduled for the show. One of the things that we were chatting about when I was first talking to you about potential topics for the show was machine learning at small organizations, which I definitely like the idea of discussing this. One, because I don't think we've coherently discussed this on the show in the past and alluded to it at certain points. But also, I got my start as a data scientist, working at startups, at smaller organizations. So I definitely know both some joys and some pains

• 2:47Daniel Whitenack?

  from trying to do machine learning or do data science at a smaller organization. What got you started thinking about this topic in particular? And what I understand from storytellers, also engaging with a lot of these small organizations in this type of work?

• 3:04Speaker 00

  So kind of like you, I started in this field, in this range of companies, from small to large. In particular, one of the things, I'll go a little bit into my background, how I got into data science and why small organizations ended up being my passion. I don't actually have a degree in data science. I have a degree in English is my background. So I came in, I'm a transplant into this field. But I came up through startups. That's how I got into tech, was working in startups. And one of the things that you learn at startups, right, is to do the task that's at hand. You just figure out what that task is, you will do it, you make things work. And so I'm so grateful that that's where I started. My journey in tech was in startups,

• 3:47Speaker 00

  because that really is the underpinning of how ML at small organizations works. I ended up going into data science through analytics. I was doing marketing for a long time, got my feet wet and like, oh wow, if I have data about my marketing campaigns, I can do these things that, if I didn't have the data, I wouldn't be able to be nearly as successful. So that's where I really found my passion for data. And my first real data science project, I took this marketing process that was a bidding algorithm that was being run out of Excel, and I converted it into a Python script. And that Excel process was taking like 30 hours a week. And with Python, it took eight seconds.

• 4:32Speaker 00

  It was magical.

• 4:33Daniel Whitenack?

  Talk about return on investment. Yeah. Exactly.

• 4:38Speaker 00

  Exactly. I actually learned Python to do it. It took me two weeks to learn enough Python just to convert this like process into a Python process. And that was like, this is just too fun and too powerful of a tool to not spend all my time doing this stuff. So when I think about that project was actually at a large company, but that large company didn't have a lot of access to data scientists. It was a pretty nascent field at the time. And so my knowing Python, being a marketing analyst and just being like, I'm going to roll up my sleeves, I'm going to stand up this process in Python and just do it myself, had a huge impact on the business. They actually changed this entire part of the business to be based within those tools

• 5:21Speaker 00

  because of how much more powerful it was to not have people clicking buttons in Excel for hours a week. So that's where I got really passionate about it. I could, I saw how one person who had these tools could come into an organization and make meaningful change, not just organizationally, but actually for the business itself for growth by using these techniques.

• 5:43Daniel Whitenack?

  And what would you say, so like out of those experiences, because you being a single data scientist in that context were able to make a big impact. And so a small organization, I can imagine a lot of small organizations saying things like, well, you know, we're not a big tech company, like we can't support this type of work, or like, we're not in a position to do like predictive things, or we're not in a position, like, we don't have enough data, or whatever it is. What are some of those stories that you've heard or cases that you've heard where maybe a company is selling themselves short in terms of the opportunity that's there around data science and machine learning?

• 6:26Speaker 00

  Yeah, I love that question because I think it is really selling short, especially now. Maybe 10 years ago, I think that those were very valid reasons to not take a step into data science or predictive modeling. Now, the tools are so much better in terms of being able to have, you know, a single person who's making a big impact with these techniques. It's much, much easier, even than when I started, to be able to do that. I would say the top reasons that I tend to hear are, one, we don't know how to even start in terms of like, how to hire someone. That's a big, big barrier. Knowing how to evaluate if someone's going to be able to come into your organization

• 7:09Speaker 00

  and make an impact is pretty tough. I also hear a lot about not knowing if their data infrastructure is ready or if their data quality is ready. Those are two big questions that are fairly hard to answer. Like, how do you know you're ready, your data is ready for data science? Um, do I have enough data? Is it clean enough? Is it stored the right way? Those are all big questions. And then the final one is, I don't know how I would integrate this person into my existing business such that their output gets fed in as an input to all the things that are already running. All of my marketing campaigns, all of my, you know, website analytics, all of that. How do I get this new discipline integrated with all the other technology? Especially with small companies having, you know, those people are always stretched very thin.

• 7:53Speaker 00

  You know, your database administrators are stretched very thin, your engineers are stretched very thin. So how can I, you know, do I have the margin to incorporate this new technique?

• 8:01Chris Benson?

  Do you think to that last point, that just kind of, you know, FUD, fear, uncertainty, and doubt really kind of play in from a management standpoint? I had a similar experience and I really walked away from that. It was the last small company I worked at. And I just don't think they thought they could do it. And they didn't. And I ended up leaving as a result of that. But do you think that's a common situation that small companies run into?

• 8:27Speaker 00

  I totally do. And to be fair, I think that, interestingly, I don't think the data science community Is doing a fantastic job at creating literature that is accessible to someone who's in a small business to be able to say, here's how you get started. I think a lot of the literature in data science, we're still really in this experimental phase of like, what new models can we build? How can we push like the state of the art in terms of accuracy? And that kind of thing. But the literature really doesn't have an angle for, say, a CEO of a small company that has great analytics, but is actually ready to take the step. There's not that bridge that says, you know, here's how you do it. Here's how you go from an analytics,

• 9:11Speaker 00

  you know, data-driven company to a data science-driven company or predictive company. And I think that's right. That it's hard to dispel that fear when it feels so mysterious, you know. So that's one of the things I wish that the data science community had more of is non-data science-facing literature about data science.

• 9:29Daniel Whitenack?

  It's a great point. So I just had this interaction. My, my wife is an entrepreneur and owns a small business and we just had the conversation because she just saw Jasper, which is a copywriting, like assistant, like generative language solution that helps you write and that sort of thing. That was the first time, like she's of course been married to me for, for quite some time, but it was the first time where she kind of made the connection like, oh, I could have my own people be augmented by this sort of technology in a way that's like non-threatening or not a lot of work. And she was able to like talk to her team about

• 10:09Daniel Whitenack?

  that. I'm curious because in that scenario, it's like people that are already inside the company that are kind of now that tooling sort of like machine learning or NLP tooling is getting more user-friendly and marketed in that way, they're seeing how they can use those tools to advance their business. But then there is a need where like still at her company, like I help also do kind of some of the forecasting or like other things that really there isn't a great kind of off the shelf tool for, but it also isn't that hard. Like if you know Python and you're like, you can import like Facebook's Prophet,

• 10:52Daniel Whitenack?

  you know, then you're like, good and boom, there it is. Okay, I've got it. But it's not the same sort of approachable thing as like a Jasper or something like that. So where do you see kind of moving into the future, like the limits of this like kind of low no code kind of people leveling themselves up versus the things that are going to be really valuable for a data scientist to do in a small organization moving into the future?

• 11:18Speaker 00

  I love that question because it reminds me back when I was learning to be an analyst, there was always this idea that eventually BI tools will get good enough, you wouldn't need a BI, right? Like you'd have these like, like Tableau is just a step towards whatever the no code interfaces and some of these companies would be like, your marketer can just go in an interface and click some buttons and they'll get a report that really just answers all their questions. No BI needed. And that didn't prove out to be true, right? Like BI's are still like this very important role, but there are these other roles and other tools that can fill some gaps around there. I love the idea of Jasper as this friendly interface to be able to do this particular task of content generation.

• 12:01Speaker 00

  I think it's a fantastic use case. And also your example of Prophet needing someone to come in, use a pre-built library for doing forecasting in order to get this very simple output that really changes a business's ability to guide itself as that example of that's a hard task to get rid of. And the real core of that that I see is that data is different from organization to organization. When I was at Amazon, we'd always talk about solve for the constants. That's a constant. Like you go from one business to another, someone's using Stripe, someone's using Square, you know, this person's using Salesforce, this person's using Hotspot. And you have that problem that just multiplies when you look at all of these different technologies

• 12:43Speaker 00

  that come together to underpin their business. That's where it's like the BI task. You're always going to need someone who's able to reconcile, make sense of the data, and then maybe push it through something that to them is like, oh, this is, this is easy, right? Like training a forecasting model in Prophet is easy for a data scientist. But it's unreachable for a, you know, a CEO of a company. So that role, I think, just like we've seen, BIs were always needed, even when we had all these fantastic low or no-code tools for analytics. We still needed BIs that were able to do that heavy lifting of reconciling, of telling the story of creating those interfaces. I think data scientists have that same kind of role.

• 13:24Speaker 00

  They're always going to need to pull the data together, build that model, explain it to the person, explain how to integrate it. And that role, I think, is the most fun. If I'm honest, like, that's the most fun data science role to me, is that particular role.

• 13:38Daniel Whitenack?

  Yeah, I've always really enjoyed that. I've used this analogy with Chris a lot of times, that oftentimes I really enjoy this sort of idea that data science is more like cooking than at the, like, you're at the chalkboard, like, doing math problems. So you have a recipe, but your recipe doesn't quite work the way that the tutorial on Medium, like, is telling you, because you don't have Stripe, you have Square. But you kind of modify the recipe just enough to make it work for your scenario. And then, you know, you don't have the same ingredients, but you adjust the recipe and you go from there. I think that's a really good framework to think about, and I also really enjoy that.

• 14:19Daniel Whitenack?

  It does make me wonder, though, for, like, a data scientist or a machine learning person at a small company. Like, I could be a machine learning engineer at Google, like, on the Translate team, and my everyday is, like, Translate, right? What I'm thinking about is machine translation. Whereas at a small company, like you said, you're in an environment where you are the data science resource, or there's a small number of those. How can a data scientist or a machine learning person at a small company deal with that sort of variability of, like, one day you're dealing with dashboarding, and another day you're dealing with, like, sales forecasting and that sort of thing?

• 14:59Speaker 00

  Yeah, it's a fantastic question. I go back to, you know, when we used to talk about T-shaped data scientists, right? Like, you're across the top, you need to know a little bit about pretty much everything, when everything was a little bit narrower than it is today. But you know a little bit across the top, and then you'd have one area. You're, you know, again, translation, the large language, or you're the computer vision. I tend to think of these roles, there's not even a letter that describes this. There's so many, you know, you need to have a relatively shallow but working knowledge of the entire cycle. And so instead of thinking of your role as a data scientist as training models, or even producing models, the role of the data scientist is to convert the data

• 15:43Speaker 00

  into some business value using data science techniques. And that means having a working understanding of all of the elements of the machine learning workflow, like data infrastructure, having a working knowledge of how to stand up a simple database, how to pull data from various sources into that simple database. And then your feature engineering layer, which is really ETL, that's one of the areas I hear a lot of data scientists say like, no, that's not my job, right? ETL is actually, at a small company, your job. It'll be in a database, but you have to do your own really heavy duty ETL to build your features. Then you've got your training, the thing that everyone kind of gets into data science for, it feels like, build training your models.

• 16:23Speaker 00

  But then afterwards, you also need a simple way of deploying your models and a simple way of monitoring and testing the impact of your models. And so I think about a role at a small company is needing to encompass all of those components, but it doesn't need to be to the level that you would see if you were in a large company for each of those components. So sometimes people hear that and they're like, wow, that just sounds awful. But the reality is, it's because they've seen someone who's specialized in ML ops and they're thinking, I have to do that too? It's like, no, no, no, you don't have to do ML ops like your ML ops peers. You just need to know ML ops well enough to where you could deploy your own models, have a very simple batch pipeline that you know how to stand up in the technology that's available to you at your company,

• 17:04Speaker 00

  and a very simple, you know, real time, if you can do that, a real time, you know, inference pipeline. And those are your recipes. Just plug your data into those two patterns and you're good to go. Very rarely do you actually need to come up with a whole new way of doing ML ops at a small company. You can usually stick to a few simple patterns.

• 17:33Speaker 01

  The Change Log is deep discussions in and around the world of software, and it's been going for over a decade. We interview hackers like Chris Anderson from 3D Robotics.

• 17:43Speaker 04

  At the time, drones were like predators and global hawks and military industrial. They were classified and super, you know, $10 billion things. And we had just built a drone with Lego pieces around the dining room table programmed by a nine-year-old. And it's like, okay, that should not be possible. When a nine-year-old can do something that is classified, that literally export controlled as ammunition with Lego, with toy pieces, it was something important in this world has changed.

• 18:12Speaker 01

  Leaders like Devin Zougal from GitHub.

• 18:15Speaker 05

  In the like 10 to 15-year range or 20-year range, what I would really like is for, if you have like three 12-year-olds hanging out, and one of them's like, "I want to be a firefighter." Another one's like, "I want to be a lawyer." I want one of them to say, "I want to be an open source developer."

• 18:30Speaker 01

  And innovators like Amel Hussein.

• 18:32Speaker 06

  I've yet to kind of see applications at scale that don't use multiple languages, that don't have just arcane stories behind why this weirdo thing exists, you know? Like, all right, when you open this file, you're going to have to turn around three times and tap your nose once.

• 18:49Speaker 01

  There be dragons.

• 18:50Speaker 06

  Just the most hilarious stories, you know? But applications are living, breathing, they have craft, that's normal. So I want to normalize weirdness because that's just how applications evolve over time.

• 19:05Speaker 01

  Welcome to The Change Log. Please listen to an episode from our catalog that interests you and subscribe today. We'd love to have you with us.

• 19:16Chris Benson?

  I'd like to extend a little bit what we were just talking about and kind of ask, we were kind of talking about patterns or recipes if you're in small business. And you addressed a little bit about, you know, ML ops and the fact that you don't necessarily need to go to what the large company person who specializes entirely. But that does raise the question of there's a lot of those tasks to dive into. And so for a person to be successful in the small company environment, where they're by necessity forced to be a little bit more generalist and cover a lot more things, but maybe not at that depth. What are some of the patterns or other recipes other than MOOPS that you've identified? That like if you were bringing somebody in new, you would say,

• 20:00Chris Benson?

  focus on that and that and that and your life off the bat will be a lot better than just diving into the deep end without any help. What would you tell that person?

• 20:08Speaker 00

  Fantastic question. So one unintuitive thing I would tell folks is build your project or product management skills. Having a very strong framework for how you manage a project from end to end is pretty much, you're going to need that muscle to be quite strong. In particular, because you're going to be shepherding sometimes from all the way at the beginning, like data isn't even in a database and you're just talking to the salesperson who has a problem you're trying to solve and like, all right, we're starting from the very beginning from scratch. So having a very strong framework that really goes all the way end to end from data, you know, in to data out to your product is critical.

• 20:51Speaker 00

  And a lot of people use CRIS-DM as sort of their framework for that. I find it's not even quite specific enough for as a practitioner to move things from end to end. But that's going to be a little variable by company. But I do tend to break it up into these five stages of, you know, having an interview format. How do you know that you have gotten the requirements from the person who's going to use this such that you have the inputs you need for your model, having a framework for interviews? Having a simple recipe for standing up a database? Or if you need to stand up your own database or have a very simple architecture for that, that you can plug all of your projects into so they're sort of centralized. The other thing that I really recommend is

• 21:35Speaker 00

  Well, here's a, here's just a side note. Almost all problems in this space are tabular. So like, it's tabular data. That's what you're gonna do. And if you've been on Twitter about tabular data, you know, it's gradient boosted trees. Just use gradient boosted trees. You know, that's your baseline. Don't worry about baselining with linear regression or, you know, the simpler models or random forest. Don't worry about that. Just stick to a very simple baseline with your gradient boosted trees. You'll probably get a pretty good model out of that. And then the last part that I didn't mention in that earlier list that I think is super critical is having a very clear baselining process. How do I know when I'm done? And then when you're done, put down your pen. You know, don't worry about trying to get to state of the art on every problem. Just know what your baseline is.

• 22:20Speaker 00

  How do I know when I've built a model that's actually gonna improve this business? So that I can stop working on this one and work on the next one. Cause two models will have a much better impact on the business than one perfect model.

• 22:32Daniel Whitenack?

  I'm having all sorts of flashbacks to my, my work in, in various organizations all the time while you're talking. Some, some good and some painful. Uh, and, uh, I'm thinking about a couple, uh, so like, one of my experiences in a, like a small startup environment is like getting in this sort of, I don't know what to call it. Like you become the replacement for Excel function where like, oh, email Daniel. Like I think he can like merge two columns together, right?

• 23:04Speaker 00

  Right.

• 23:05Daniel Whitenack?

  And then you'd like, they send you to like Excel sheets and you're like, okay, like I can do that in like one minute. So you do it, but then you get this increasing number of tasks and, and then you just do that all the time. And the second scenario is like, okay, I'm, I'm gonna try to build out this roadmap and like this project plan. But things like get shaken up all the time in small companies. And it's like, okay, I had this plan to like optimize my pricing model. Over the next six months, using like these A B tests or like whatever I was doing, right? And then the CEO is like, "Oh, you know, like, we're not like meeting revenue this month. Like, we need to like just change our pricing structure entirely," right?

• 23:50Daniel Whitenack?

  Does that spark anything in your mind in terms of, yeah, like, the level at which you can do product or project planning within a small company while still managing that sort of flexibility and random tasks that that come up? Any recommendations there?

• 24:07Speaker 00

  Yeah. It's funny, because you sparked a whole bunch of memories too of doing that. I feel like that's a universal experience. Once you know how to use data, people will not let you do anything with data. Yeah.

• 24:18Daniel Whitenack?

  Right.

• 24:19Speaker 00

  I think number one is going back to the idea of solving for a constant. That is a constant in small business, is that strategy is changing very rapidly. And so, building that into the way that you approach data science, I think, is really important. I find that one thing that helps with both of those scenarios is being able to deliver a result that baselines everyone about what good data science does for the company. So, if a good data scientist is able to build a model that increases open rates on emails 50%, it's much less likely that they'll email you to ask you to merge the columns, because you're the person who optimizes email open rates 50%.

• 25:03Speaker 00

  So, that tends to work super well, which kind of goes back to the idea of like, you need to have an end-to-end process. You need to be able to deliver relatively quickly on a quick timeline and know how to measure your results, so that when you do start, you know, getting those questions like, "You know what? I'm super busy on this pricing model. If you can wait until such and such a time, I'd be happy to. Otherwise, here's a Google search." You know, like, that can not quite as, you know, abrupt as that. But here is a tutorial on how to merge columns in Excel. And so, that I tend to point to in a lot of scenarios as something that solves the instability problem in a small business, is make sure that you're really focused on results and that people know what the results you're driving are. But roadmapping, I think, is one of those things.

• 25:48Speaker 00

  I used to, as a project manager, hold on to my roadmaps very tightly. But as you learn, even in, you know, as working in large companies, we would have those where you're kind of feeling a little whiplash and it's like, how do I solve for a changing roadmap? What's, what is, how do I solve for that constant? And really clear prioritization frameworks tend to really help with that too. Especially if they're known way up the chain, like this is the metric I'm trying to optimize for this business. And therefore this is what the, how it reads into my roadmap. And being able to give those cost trade offs all the way up to the management chain tends to be a really effective solution as well.

• 26:25Chris Benson?

  I have a follow up question for you as we talk about process. I love the way that you've kind of worked out the end in system and you have kind of go to things that you can utilize and you're kind of simplifying and making sure everyone understands what is is, what the result, how to measure it and stuff like that. In so many small businesses, they'll have the data scientist, but they'll also have the software person. And then they'll have the infrastructure person or whatever you want to call it, systems or, or whatever.

• 26:56Daniel Whitenack?

  Dev ops Doug.

• 26:57Chris Benson?

  Yeah, exactly. Or Dev Ops Diana, whatever. Just that person is there. And so I like what you're saying about kind of going through that system. Where do you bump in either two different ways of looking at it? Bump into that person or find a way to integrate and everything is Nirvana together. How do you navigate that in a small business where there are persons like, okay, you're on my toes now?

• 27:20Speaker 00

  Yeah, I love it. I love it because it gets to this skill that I feel like is not talked about a ton in data science education, which is how much people are actually the mechanism by which things get done. More than any code or any framework, it's people that get things done. And so knowing how to work in an organization. With folks that have sort of their territory, how do I earn trust with them? How do I think about handoffs between the components of this overall system managed by people is is so important. I think one thing that I tend to recommend for data scientists is that earn trust part.

• 28:05Speaker 00

  How do you break down the process of earning trust inside of an organization? And make that a repeatable thing. Overall, just knowing the architecture people wise of your organization is a task that some care should be taken with. Who are the people that are over these various systems? And meeting with them and knowing who they are is like baseline. Even then, knowing what their goals are, what their blockers are, and how your work can actually make their life better is huge. There is a big advantage to knowing, hey, software team got this problem where they're meant to optimize this part of the app flow. And they're struggling with that.

• 28:46Speaker 00

  Well, actually, that's a place where machine learning could come in and actually solve part of that problem for them. Can I actually do something that helps them with their own KPIs such that I build this trust with this organization? And so it's not like, you know, as much as we like to talk about like code and if it was just faster, if it was just easier, if it was just simpler, then we could get data science done. Focusing on that data part and being part of the team is actually the skill that stands on its own beyond any technical skill that you develop over your career.

• 29:17Daniel Whitenack?

  This is Practical AI and I have a very practical question, which I think it is sort of a boring question, but I think it actually could be really helpful to people. So like you were talking about in part of data science education, machine learning, like we don't talk a lot about this like project management side of things. And I'm guessing there's probably even listeners out there, data scientists, who maybe they have, let's say it's a data scientist who has like a background in science or something or academia. And their idea of project management is like. Like I have like a notebook with some things written down in it.

• 29:58Daniel Whitenack?

  And then on the other end, you have like maybe data scientists coming from like a software engineering background. And their idea of project management is, okay, I have a Jira board or a Sana, these sorts of tools. And maybe there's, you know, other backgrounds as well. I could see like how the notebook isn't going to get you totally to a good place. I could also see how some of these other systems like a Jira or a Sana. It could be overkill for like managing what you need, especially if you're like a solo data scientist working on the projects. Do you have any recommendations in terms of like some things that are not overwhelming?

• 30:35Daniel Whitenack?

  It doesn't have to be a system or like a product, but like things to look into that can just make your project management workflow work for a data science scenario.

• 30:46Speaker 00

  Yeah, it's a good question. I'll start with, I really like Trello. So if we're talking about products, just products generally, like Trello is a fantastic place to start. If you are that person that's just like written things down in a physical notebook, Trello tends to be like a good step forward in terms of something that's shareable and everyone can see it.

• 31:05Daniel Whitenack?

  Not overwhelming.

• 31:06Speaker 00

  Exactly. It's, it's really great. You can build templates in it. And so you know, like, these are the things I need to put together for my, you know, data science project. But beside that, the thing that I find is very hard to beat is Google Sheets. Google Sheets is a fantastic tool for almost any workflow. Google Sheets is great. Like usually when I come into a new organization that doesn't have sort of a project management muscle, I start with a spreadsheet because it's so easy to change, so easy to update. It's, you know, you can add a new column, remove a column you're not using, super, super easy. And you do that for like a quarter, like manage 10 projects through a Google Sheet and see what actually is helpful in terms of making sure everyone knows when things are due,

• 31:51Speaker 00

  what the deliverable actually is, how do we know we're done? Like what stages does our project go through? Like iterate on those in a Google Sheet for a while. And then you'll have this system that really makes sense to everyone because everyone's been using it. And then you can level it up with an interface like a Trello if you wanted to. So that is my secret to almost every workflow is put a Google Sheet somewhere. That's, you know, connecting some pieces for a little while. And that's what's going to teach you what you actually need in order to manage that in the long term.

• 32:19Daniel Whitenack?

  Yeah, that's awesome. I wonder too, like with that, so project management is one thing, but then there's like, we were just talking about the people side as well. Communication wise, within a small company, you know, I've had experiences in the past where I am maybe managing my own thing and I think I'm managing it well, but I'm doing that in a silo and I sort of crank on something for like a month and then try to lob something over the fence or something like that. So do you have any recommendations with regard to that and really kind of developing that empathy and good communication of data science within a smaller organization between like key stakeholders?

• 33:03Speaker 00

  It's a fantastic question as well. I love like you're hitting on all of the things, all the pitfalls of working in a very small organization.

• 33:10Daniel Whitenack?

  It's only because I've hit the mind.

• 33:12Speaker 00

  That's exactly it. Yeah. The thing that comes to mind with that is really the understanding that success in a project, you can do as well as you want in a project. Your product itself can be super good, but at the end of the day, that product won't be able to make it to your end customer without passing through a few more hands. And when you really tie your project success, not to the trained model, but to the deployed model that is in front of your customer, your end customer, it starts to really raise the priority of understanding what happens downstream of the output of your workflow. So the output of our workflow is like this trained model, right? Even like a pipeline

• 33:56Speaker 00

  of inference, an inference pipeline for this trained model, that's the output, but that pipeline has to connect somewhere and there's got to be someone who's doing that connection. And that is one of the tricks that I use to really raise in my own mind, the priority of having good relationships with people up and downstream of me. And so good relationships downstream, as you've kind of pointed out, regular communication really helps with that. Having a project management framework where you've got deadlines and you're meeting your deadlines and you're giving people regular updates along the way really goes a long way in earning trust in that up and downstream relationship. With anything like that, when you're, as you've, we've mentioned a couple of times, that means that's another task. Like I can imagine someone listening and be like, oh my gosh,

• 34:39Speaker 00

  not only do I need to figure out how to do my pipeline, but I also need to figure out like project management and I need to figure out a communication strategy. Making all of those things as light touch as possible is really important. So if you have your project management framework in your Google Sheet, something as simple as updating that Google Sheet, copying those rows out, putting it in an email and saying, this is my update for the week. Here's where we are. Really leaning on those like agile frameworks of simple stand up. Here's what I did last week. Here's what I'll do next week and here's when I'll be done. Those very simple mechanisms and training yourself to make them simple and keep them regular is really, really critical.

• 35:16Chris Benson?

  I have a follow up, which I'll get to in a second, but I, just something that you said that was really resonating with me is I'm currently, though I've spent a lot of years in small companies, I'm currently in a large company. But just before we started this conversation, I was in a work meeting. I was with a, a development team and I was thinking exactly that. I was literally saying, no, no, we need to lighten this up. We're too heavy handed. Um, I think it's one of those small company things that could be used very well in many large companies is don't overdo it. Light as well. So I just wanted to say, I really resonated with that when you were saying that. You've done a really good job of kind of talking about the need for trust and finding

• 35:59Chris Benson?

  those opportunities and communicating. But as we move forward with data science and small organizations, and there are all these opportunities for growth of the small organization by really absorbing data science beyond just your role as the data scientist, but kind of affecting all of the other functions. As you build that baseline of trust and you're actively communicating that, how do you approach getting people who aren't thinking about the benefits of data in a business context? So they're not doing your job, they're doing their job, but there's so many ways that if you kind of get data centric about in a non-technical way that you can get benefit from that. How do you bring people along on that journey?

• 36:43Chris Benson?

  Because it's a hard thing to do. It takes a very savvy touch to bring people to see something that's not normally their forte. That's your forte, but they can benefit tremendously. What are some of the ways of bringing all of those other people along in their own right?

• 37:00Speaker 00

  Yeah. The reason I like this question so much is in a small organization, when you're kind of one of a few data scientists, you have this responsibility that you're not just representing your work. You're actually sort of representing the discipline within that company. And I think there's a lot of companies going back to like the beginning of our conversation that are not doing data science because they maybe like dipped their toe in and really got burned. And that one project where there was a lot of promise, but it never panned out, turned them off of wanting to do data science kind of like for a long time. So these cycles tend to be really long. Trust cycles like that tend to be super, super long. Maybe like two or three years later, they're like, maybe something's changed in, you know, the landscape that would make me want to try it one more time.

• 37:44Speaker 00

  So knowing that is sort of serious. Like when you're the one data scientist, it's kind of serious to be like, I not only have to do my work well, I have to convince folks in this organization that this discipline can do something for their organization beyond what they're doing today. So with that in mind, one of the things that I think is really critical is thinking of yourself as not just delivering a product, but educating about what that product is and its benefit, which is why having a very strong A B testing framework, maybe unintuitively, maybe intuitively, is so important. If you're not familiar with how to deploy something and A B test it at the same time, such that you can describe its impact. That's one of the biggest differentiators I've seen in small organizations

• 38:29Speaker 00

  that really just love their data science team versus like, I don't know why we are doing this. Right? It's that's a really big difference. Yeah. And so that education point is very hard. I used to say this quite a bit when I would review resumes. There is a primacy to delivering results. Anything else when you're reviewing someone's resume? A lot of times, that's the output of kind of all of these, the inputs in their resume. What the results have been driven. That can tell you a lot about how well do they manage projects? How well do they work in a team? Like, all those things kind of ladder up to delivering results. And so when you think about earning trust within a small organization, thinking about delivering results as being the output of earning trust, doing your work well, you know, training

• 39:14Speaker 00

  your models well, having good relationships with the people along the pipeline, that's what will really point you there. And once an organization sees those results, it's actually very tough. You'll have more work on your plate than you'll know what to do it. That's what I've found.

• 39:26Daniel Whitenack?

  We've talked a lot about challenges related to being a data scientist in a small organization, or like, things that you need to be thinking about. I'm wondering, what, from your perspective, because I've definitely seen some of these things, what advantages does like, a small machine learning organization, or machine learning practitioners in a small organization? What advantage do they have compared with machine learning engineers, let's say, at a really big tech company, in terms of what they're able to do? Because I think oftentimes, that's not highlighted, like, some of maybe what you can do in that

• 40:08Daniel Whitenack?

  scenario, it's maybe harder to do in a large tech company. Have you run across those things?

• 40:14Speaker 00

  Interestingly, that's part of the reason why I like doing it in small companies, because when you do data science at a large company, you have to think about things like, how do I parallelize the compute for this, so that I can actually get this pipeline to run?

• 40:28Daniel Whitenack?

  Through three billion users, or whatever.

• 40:30Speaker 00

  That's right. So there's this part of the machine learning tech stack that I find the most complex, the hardest to understand, the hardest to become expert in, the hardest to deploy, are that edge where you have very, very high number of users, an incredibly large amount of data, latency requirements that are very stringent, like, this inference needs to happen in 300 milliseconds, or like, everything is off. Those constraints don't tend to happen in small businesses. You tend to be looking at tabular data. You tend to be looking at batch inference. And those are things that are actually fairly straightforward to learn when you like sit down and you look at like what technologies do I have to do batch tabular inference. It's fairly straightforward in most platforms.

• 41:13Speaker 00

  And so I think the advantage is that you actually get this vista of the machine learning discipline at a small company that you don't tend to get at a larger organization in a larger team. You tend to have a fairly narrow aperture of like, I'm looking at my features have been engineered by my data engineering team. I'm then doing some last mile stuff on that data to put it in my model and train. And then I'm handing that model artifact off to my ML ops team and they're doing all of the C I C D stuff. So your aperture is very narrow. And so you tend to not be able to see the innovation that's happening in ML ops or the innovation that's happening in, you know, data engineering leading up to your model training that I think is fascinating. It's so interesting to do and be a part of.

• 41:59Speaker 00

  So, and then you get the chance, like if it turns out for some people who did modeling for a while that, for instance, have worked for me, they will get a chance to do some of the ML ops stuff and like, you know what? I like ML ops. That's what I want to do. And so you get the chance to actually see these different roles and try them out. And then you could go deep. You could say, I'm going to do ML ops and I'm going to be an expert in ML ops. That's what I want to spend my time on and then go do that at any size of organization.

• 42:25Chris Benson?

  So you got me thinking about this because as you're talking about that and you're really making me think back to my small organization time. But I'm also in a large one right now. And as a comment before I ask the question, in large companies, you're often at the mercy of arbitrary decisions of others that may not be as informed as you are as the data scientist. And that happens all the time. But you've kind of differentiated these kind of different opportunities at the different sized companies. If someone came and I was asking you for guidance on like, what kind of company should I target? There's a clear set of kind of generalists, but a lot of opportunity at small companies to try different things.

• 43:06Chris Benson?

  And then there's this opportunity at large companies that you may have to accept what they give you. But within that scope, you can go deep. How would you recommend someone try this or that? You know, if someone's looking to you for that mentorship, how do you steer them the right way?

• 43:21Speaker 00

  I think it takes a combination of things to be a data scientist in a small organization. I actually tend to recommend that less often than at a more established organization. In particular, if folks are coming just out of school, I tend to not recommend looking at smaller companies as their first job. Particularly if that company is just starting their data science muscle, which usually you can find out when you're doing an interview with them. Like, am I employee one or sub 10, you know, of doing data science at this organization? Usually I'll steer people away from doing those roles if they're early in their career because there's so much about that role that's not taught at, say, a university.

• 44:02Speaker 00

  I tend to find people just out of college don't know how to set up their own data science pipeline from end to end. How to interview someone really rigorously to understand, how does this requirement? Tell me how accurate this model needs to be, you know, making that translation. I tend to recommend sort of mid or larger sized companies for a first role. So that you can learn from other people that have done this for a little longer and have built these sort of intuitive ways of doing data science. You can pick up some of those frameworks from them. But if, one, it is a startup that is led by a CTO or CPO or a CEO

• 44:41Speaker 00

  that has deep expertise in data science, has seen it elsewhere, that can be a fantastic opportunity to be mentored by someone really directly. So I tend to sort of put them on that spectrum of like, if you're brand new in data science, I tend to recommend a more seasoned organization. So you can pick up some of this stuff that simply is learned on the job. Sadly, like as much as I wish I could point at like a blog or, you know, a course or something like that, that would teach you end to end data science workflows, I actually don't, haven't found one. Please send one to me if you have found one. But because that doesn't exist, I tend to say you need to see it, you need to see it and really take that opportunity to learn not just your your narrow aperture, but really try and observe what the people upstream of you and downstream of you are doing so that you know that end to end workflow.

• 45:25Speaker 00

  And from there, a smaller company can benefit from what you've learned at that larger organization, as long as you have this desire to really hands on own it end to end. If your desire is that the person upstream of you is doing a lot of the data preparation, please do not go into a small company. If you hope that someone's going to tell the story of how good your work is, don't go to a small company. That's going to be your job. But if you really love this, like I want to be the one that takes this whole thing end to end, a small company is where I would tend to send people.

• 45:55Daniel Whitenack?

  That's a great perspective. As we kind of wrap up here and get to the end, I'm just curious, as you look forward to the future, that could be something with storytellers and the work that you're doing there or just generally in the industry, what what's exciting to you and encouraging to you as you look to the future?

• 46:12Speaker 00

  I'm most excited as we built this company and seen how much need there is in smaller companies for data science techniques and how hard it still is for these companies to find, hire, you know, keep data science talent. I'm super excited for the opportunity to show how much data science can help a relatively small organization and prove out that case, like data science really works here. And from that, I think that will spark some ideas around, you know, MO Ops tools tend to really focus on enterprise use cases, large teams that are solving very complex problems.

• 46:53Speaker 00

  I'm excited to see tools that are really aimed at solving these constants that small businesses run into. Lots of disparate data that needs to be brought together in order to build these models and deploy them simply, like a very simple sort of layer for that. I'm super excited about that. I also, as you mentioned Jasper, I'm really excited about how like large language models will play into this idea of deploying data science within organizations. Will it make people more familiar with the concept of data science to where they're more ready? They're like, I see this value other people are getting. Can I try this in my organization? And then lastly, I would say, I'm really excited for how I see the data science community

• 47:35Speaker 00

  generally maybe starting to pivot away from excellence as being measured as state of the art performance towards excellence as being measured as impact within some sort of vertical. Like, there's so many areas that data science could really, I mean, not to sound like the Silicon Valley like stereotype, but really could make the world a better place.

• 47:57Daniel Whitenack?

  Of course.

• 47:58Speaker 00

  You know, you think about like, you think about like, how data science can help universities identify when a student needs some sort of service in order to help them get their degree. Like, that's a very practical thing and something that I haven't seen many universities really embrace yet. It's one of those small organizations that still tends to have a little reticence around adopting data science techniques. But if we can point towards data science outputs as being driving impact, not just the accuracy of our models, then I think we'll see that adoption really start to grow within these smaller organizations.

• 48:37Daniel Whitenack?

  That's awesome. Well, thank you so much for joining us, Kirsten. It's been a great conversation. And yeah, I know I've got, I've got a lot of tips I think that I can articulate better now with, even with my own team after the conversation. So thank you so much for joining us.

• 48:53Speaker 00

  Thank you. I really appreciate it.

• 49:04Speaker 01

  All right. That is our show for this week. If you dig it, don't forget to subscribe. Head to practicalai.fm for all the ways. And if Practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets, to fly.io for backing our dynamic requests, to Breakmaster Cylinder for the beats, and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.