Hidden Door and so much more

Hilary Mason is building a new way for kids and families to create stories with AI. It’s called Hidden Door https://www.hiddendoor.co, and in her first interview since founding it, Hilary reveals to Chris and Daniel what the experience will be like for kids. It’s the first Practical AI episode in which some of the questions came from Chris’s 8yo daughter Athena. Hilary also shares her insights into various topics, like how to build data science communities during the COVID-19 Pandemic, reasons why data science goes wrong, and how to build great data-based products. Don’t miss this episode packed with hard-won wisdom! Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean’s developer cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, team accounts, worldwide availability with a 99.99% uptime SLA, and 24/7/365 world-class support to back that up. Get your $100 credit at do.co/changelog https://do.co/changelog. Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We move fast and fix things because of Rollbar. Resolve errors in minutes. Deploy with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Featuring: Hilary Mason – Website https://hilarymason.com, LinkedIn https://www.linkedin.com/in/hilarymason, X https://x.com/hmasonChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Hidden Door https://www.hiddendoor.coUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  And then the last thing I'll say, because I know this audience is probably deeply familiar with things like GPT-3 and a lot of the issues with natural language generation tech, is that there is a real need to build a structure, safety, coherence, and memory into these systems before they can be deployed for any human-facing application, much less one that you're willing to put in front of young people.

• 0:29Speaker 01

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 0:47Speaker 01

  This episode is brought to you by Digital Ocean. Droplets, managed Kubernetes, managed databases, spaces, object storage, volume block storage, advanced networking like virtual private clouds and cloud firewalls, developer tool that are robust API and CLI to make sure you can interact with your infrastructure the way you want to. Digital Ocean is designed for developers and built for businesses. Join over 150,000 businesses that develop, manage, and scale their applications with Digital Ocean. Head to do.co/changelog to get started with a $100 credit. Again, do.co/changelog.

• 1:33Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We're at Practical AI FM.

• 2:00Chris Benson?

  Welcome to another episode of the Practical AI Podcast. We're going to explore today a lot of interesting things in AI in a little bit of a different way. My name is Chris Benson and with me as always is Daniel Whitenack, my co-host. How's it going today, Daniel?

• 2:15Daniel Whitenack?

  Oh, it's going great. Other than I woke up this morning, my computer, we had a power outage last night. So my training run abruptly ended at some point in the night. I'm not sure when. So getting that restarted, I guess I'm like learning those sort of things about having an AI workstation locally. Whereas most of the time before I just ran stuff in the cloud. So, you know, upsides and downsides, I guess, but yeah.

• 2:42Chris Benson?

  Sounds good. So yeah, I guess for listeners who have been tuning in for a while, there has been Daniel put to get built his own workstation for, for doing deep learning and has been going through the trials and tribulations. So I'm waiting till he has it all figured out before I try it myself. So that.

• 2:58Daniel Whitenack?

  On some fully connected episode, we'll have to chat about fun that has been had through that process. So.

• 3:05Chris Benson?

  There you go. Kind of like everyone can learn from your pain on that one.

• 3:10Daniel Whitenack?

  Yes, yes, please do.

• 3:12Chris Benson?

  Well, you know, today we have a bit of a different episode from the usual in that we have someone who's joined us who has done lots of interesting things in the past, which we'll hear about as well as some things that she is currently engaged in. With us today is Hilary Mason, who is currently the co-founder at Hidden Door. Welcome to the show, Hillary.

• 3:35Speaker 00

  Thank you. I'm pretty excited to be joining you today.

• 3:38Chris Benson?

  Well, thank you very much. Daniel and I have been familiar with your work for many years because you've done a lot of stuff that is, that within the context of the data science world has been very much in the public eye. And so we are really interested in finding out some of what you've done in the past, some of which our listeners may possibly, those that familiar with you may already be familiar with, but also some of the cool stuff that you're doing now. I guess to start off with, would you just kind of give us a little bit of background about yourself, how you got into the field and, you know, highlight some of the stuff that you've done up till now?

• 4:13Speaker 00

  Sure. So first, that's really kind of you to say, and hopefully we'll talk about some things today that you aren't familiar with.

• 4:21Chris Benson?

  Okay, looking forward to it.

• 4:23Speaker 00

  Yeah, so I've been doing this for quite a long time, as you say, and I try to be very prolific because if you do enough things, at least some of them are likely to be interesting. I started in computer science and machine learning over twenty years ago. It's been a long-term interest of mine. As a kid, I loved science fiction, I still do. Love writing stories, love thinking about machines that could really be partners to us. And obviously, that wasn't entirely possible in the nineties, and I'd say that it's maybe just now becoming possible. But I started in academia, realized that, you know, I have several personality traits

• 5:03Speaker 00

  that make me a little bit of a mediocre academic, but I think a pretty good entrepreneur.

• 5:10Daniel Whitenack?

  That's pretty interesting. What, uh, could you, like, just give a bit of detail about that? I'm kind of interested.

• 5:16Speaker 00

  Sure.

• 5:17Daniel Whitenack?

  Because there are a lot of people that, you know, are maybe sitting in academia a bit disillusioned, or maybe they're like in industry, they don't know if they should, like, go do research, or, you know, all of those things.

• 5:30Speaker 00

  Yeah, I know, it's, and it's a hugely emotional process, whatever side of it you're on, to think about where you might go. And there's definitely, I have a lot of conversations with folks, even now, who are considering whether there are opportunities to use their academic skills in industry, if they'll find something that's nearly as intellectually fulfilling. So yes, this is something I have a ton of empathy for, especially because I went through it. So was, you know, in a faculty role, the things that I was interested in, first in computer science, we still tend to give a little bit more status to theory work than we do to engineering work. But I really like to build things. And, you know, I can force myself to go through and do some proofs.

• 6:15Speaker 00

  And, you know, math is something that I have, I can get along with and I do enjoy. But if you give me the choice between spending two weeks, you know, at a chalkboard, thinking about math, and spending two weeks at a whiteboard and at my keyboard, actually trying to build it, I will always choose the latter. So that's one, you know, personality flaw of mine. And then another one is that I have a relatively short attention span. In the sense that, you know, as an academic researcher, I find that some of the the best work requires a persistence over, you know, not just months, but over, you know, four, five, 10 years. And I tend to, uh, you know, after one year,

• 7:01Speaker 00

  something has to change for me to to continue to keep up that persistence and interest in it. So I have a relatively short attention span. I like to pay attention to a lot of different things at the same time and try to figure out where they connect. That's also something that, uh, you know, is a real asset in data science, where you're often, you know, facing some sort of problem. You have certain technical tools that you understand, certain data assets, maybe a product platform to build on top of. Uh, and you need to figure out how to pull all those pieces together in something that'll work. And if you're doing it in a startup context, you need to do that quickly. You don't really have the luxury of a year to take the best approach. And so that's another personality trait that I have.

• 7:45Speaker 00

  And it's also something I look for when I hire folks out of academia. One of my favorite questions to ask is to lay out one of the technical challenges that we're working on and sort of say, how would you approach this? And usually they give a wonderful answer. And I say, great, now what if I told you you had two weeks to build something? What would you do? And then that, you know, the sweat starts to pour down. And then I say, okay, now you've got two days. Like, what are you gonna do? What is the stupid, simple thing you can pull together? And really looking for that kind of agility of thinking and being able to make decisions about where you're gonna prioritize that that simple thing versus the right thing. And I think, uh, you know, part of that is that I am more of a hacker than I am a perfectionist.

• 8:28Speaker 00

  And again, these are are just things that I've realized that I need to work with people who are perfectionists and who have that that other, you know, sort of perspective on things, because that way we end up building really great things. Um, but those are my personality flaws.

• 8:45Daniel Whitenack?

  No, I, yeah, I appreciate you going into detail there because I think there's, those are a lot of things that people think internally, but they, they don't voice them a lot when they're going through that or like ask people like, hey, I feel this way like, you know, what, where should I go? What should I do? A lot of times people just kind of struggle with that inwardly and don't really voice it.

• 9:08Speaker 00

  Yeah, I know. I spent so many years feeling like there was something wrong with me because I preferred to write code to write math, even though it is ultimately the same thing and it's easy enough to go from code to math. I thought there was something, you know, wrong with me because I had these particular traits and preferences about the environments I'm working in, but I realized now that I've been doing this for 20 years. It's a long time.

• 9:33Chris Benson?

  So as you were saying all that, you and I share a bunch of those same characteristics. I have, you know, over my career, I've also had the same thing where I've been like, you know, I'm feeling like I was doing it, maybe not the best way in every case. But hearing you doing it and with the, with the successes that you've had over the years, it kind of validated like, oh, maybe things aren't so bad after all. So I think it's a really good message to get out there because I suspect that if you are, and I am, probably quite a few other people out there are experiencing the same.

• 10:04Speaker 00

  I'm sure.

• 10:05Daniel Whitenack?

  That's great. I, I'm kind of curious. I know one of the, of course, aspects that we've been talking about is kind of jumping into industry and building products and those sorts of things. But how did you get to the point where you really started getting this interest in like data science and data science products? How did that develop?

• 10:23Speaker 00

  Yeah, it's a good question. So I, like I said, I started in academic machine learning. I've always been interested in using that as a tool to build useful things. But I actually learned the product lesson the hard way. So there's a bit of a longer story here, but the short version is that I, you know, left my academic position and came to work for a startup. And the startup at the time, we were building statistical models of career progressions off of millions of resumes we crawled off the web. This was in 2007, 2008, I think. So it was something that was novel. It was not, you couldn't go on LinkedIn and see that kind of analysis done for you.

• 11:07Speaker 00

  Uh, it was obviously something I had a personal need for because I was going through a career crisis myself at the time. And that company ended up failing in nine months and it failed because of two things. One is that we built some beautiful data science models, put a UI on them and nobody actually wanted to use them. And the second thing was that everybody thought a website about, you know, data about careers was meant for college students. But the whole idea here was that you could say, you know, okay, I'm a lawyer and I don't want to be a lawyer anymore. What do people like me go on to do? Or you could say, I'm a software engineer and I want to be a CEO. How do I get there?

• 11:51Speaker 00

  What are the career paths that other people who have made this transition have taken to get there? What are the, the stages they go through? What other fields do they explore? And college students, of course, have no data. So it actually didn't work for them at all. Um, so there were some flawed business assumptions, some flawed product assumptions. And that was a wonderful lesson. Um, and I learned it thankfully on, you know, somebody else's company and somebody else's money. In that you really have to build a product that is useful to people, which of course sounds so obvious when you say it out loud, but it at least was not obvious to me at the time. And so I went from being primarily interested in the modeling to being primarily interested

• 12:34Speaker 00

  in building useful things and have spent quite a bit of time studying, um, and practicing and trying to figure out how to do that, uh, in the intervening years.

• 12:44Chris Benson?

  And so I'm curious, just to that point, how do you look at that now? Because that's always a hard thing for companies, uh, to do. It's, you know, regardless of the industry, people trying to figure out what their customers want and, and their needs and, and what do people actually want to use and are willing to commit to? How have you addressed that as you've learned that over the years?

• 13:06Speaker 00

  That could be, you know, a whole course.

• 13:09Chris Benson?

  Yeah, sure.

• 13:09Speaker 00

  There's a lot to talk about there. Um, but I think the sort of principles of it are really to clearly identify the problem you're trying to solve and not rush to like, try to get to the question. Don't rush to the answer. Um, and then, you know, figure out the people who you're answering this question for. And figure out what's actually useful to them. And this honestly is a mix of both quantitative analysis and qualitative analysis and really trying to have empathy for the people who you're trying to build for.

• 13:50Speaker 00

  And understanding where this fits into their lives, which are always, you know, interesting and chaotic. And then it comes back to what we were actually talking about before, where you start to build the simple things that can start to potentially address the problem to understand if it's even worth investing in the best, the best things. And so it, it really is a process that sort of merges the practice of product management and product design with the practice. If you're building something that depends on, that I would call a data product, which is something that really depends on some sort of data science or machine learning capability to build the core feature of the product. So something like a weather prediction, like a, you know, a Google Maps is always my favorite

• 14:35Speaker 00

  data product, um, the navigation stuff. These products could not exist without the underlying representation of the data. Um, and so you're really trying to figure out what decisions are people making and what context, how do I get that information to them in that context so they make a better decision, how accurate, how good does it have to be for them to first get benefit from it, but then second, actually use it. You know, if your weather predictor is one percent more accurate than another one, that's not a compelling reason for me to use it if the other one's more convenient. Um, so really thinking about all of that, and one of the gaps I see broadly in our practice right now is that we often have product designers and product managers who

• 15:19Speaker 00

  are in the position to make the best decisions about the use of data science and machine learning in their products, but they don't have necessarily the background, the knowledge, the access to that talent or tools. They're certainly not going to build their own deep learning rig in their house. And then you have, on the other hand, your data scientists who often are not connected to the customers or to the ultimate, you know, people who will benefit from the work they're doing. And so I think there's, in our field of practice, there's actually a lot for us to figure out about, not just from a, you know, I'm a startup and I want to build something new point of view, but really, you know, from your day-to-day practice as somebody in this field, like, how do you think about doing this work in the organizational

• 16:03Speaker 00

  structure you're in? I think it's kind of a mess right now, and that's a big opportunity for us.

• 16:08Chris Benson?

  So I know you and I have met previously and have talked and stuff about business, and you have really become, you know, a powerhouse in developing entrepreneurial opportunities. But before we get too far past that, I am curious, I know, you know, I alluded to it in the beginning, we talked about the fact that our first awareness was when you were producing content and stuff and, and learning from you as students. And I'm, I'm kind of curious, how did you integrate teaching others, your expertise, integrate that into your entrepreneurship? Does it still have a role? Has that role evolved or changed? Was that just a step along the way? I'm just kind of curious about the development there.

• 16:47Speaker 00

  Yeah, it's a good question, because I think people take a variety of approaches to this. And, you know, the honest truth is that I love teaching and I love talking to people. It's why I'm here this morning. And I also really like sharing opinions that are useful. And the company I founded about six years ago is called Fast Forward Labs. And, you know, we met at the Fast Forward Labs office a while back. That company was built on the idea of doing independent applied research and sharing as much of it as possible. But it's not, the, the sharing piece is not exclusive to my work at Fast Forward Labs.

• 17:30Speaker 00

  It's been a thread through, you know, my, all of the different data science jobs or, or management jobs I've had, or things I've gotten involved in. And so- I think it's really important in this field to talk about what we do and what works, and more importantly, what doesn't work. Because the field is so young, like, people have only been able to get a degree in data science for maybe four or five years. And that still astounds me. The fact that you could have a job with that title has been a thing for about a decade. There's a lot for us still to figure out, and that's why I love it, by the way. It's, um, it is not a solved problem.

• 18:12Speaker 00

  Um, and our technology is also not a solved problem. It is really weird. So if you look at the change in capabilities of machine learning technology and how you have to manage using them and investing in them, it is completely different than most technologies that folks are familiar with. And yet, we tend to shoehorn it into existing structures and processes that come from software engineering. And so when I talk about the stuff I like to work on, it, for one thing, um, I'd say it's a, it's a two directional exchange. One is that, um, I and the teams I've been fortunate enough to work with, we try to have a unique or our own point of view.

• 18:53Speaker 00

  Um, and we try to be, and this is really something I can't help, honestly, but deeply pragmatic about what something is, how it works, where we think it's useful. Uh, this is really important in AI and machine learning because there is so much hype, there is so much salesmanship, there is so much marketing that is designed to get people who don't themselves have a deep expertise, uh, to believe something that's not quite true. And because the tech is weird and changing so quickly, it's very easy to believe that stuff. And so it's really important, if I can say this, as a technologist, to have those pragmatic points of view and then share them where you can. Because we need to build a consensus in the community around what is possible, what

• 19:37Speaker 00

  isn't, the best ways to approach certain kinds of problems. Um, and we only do that by sharing. The other thing I'll say is that I love working in the data science community because the more that you succeed, the more I succeed. We are not directly competing with each other. And for the most part, if we're data scientists or we're machine learning engineers at different companies. I can help you out. I can hear about what you're think you're working on. I can share what I'm working on. We can give each other feedback. And, you know, within a certain ecosystem, the more one company succeeds, the more another is likely to as well. This is really different than if you ever have had the, the, well, I don't want to be unkind, but if you've ever hung out with

• 20:21Speaker 00

  a bunch of like hedge fund quants, I live in New York City, so occasionally I end up, well, not anymore, but I used to end up at those events. These folks talk about the weather and about sports because they do work in a community where there is a significant competitive dynamic. So I, I think that one of the things I really appreciate is the ability to share and having that actually be supportive for all of us. And then I'll also say it's worth sharing because you will get feedback. You will meet people who are interested in the same things you're interested in. And I'm an introvert. Um, it is very hard for me to talk to a lot of people, which may seem a little bit counterintuitive because here I am talking to you.

• 21:03Chris Benson?

  That surprises me. Yeah.

• 21:05Speaker 00

  But by talking to you once, and now this discussion will be out in the world, folks who are interested in the things that we're talking about today will reach out to me. I don't have to go out and talk to a thousand people to find the two or three who are going to share these interests. And so it's a good hack also to find the people who, uh, you know, you really can brainstorm with and share with.

• 21:43Speaker 04

  What up nerds? Jared Santo here, your humble producer. I'd like to tell you about something new we're beta testing around Practical AI. It's a membership program, which we think could be really valuable for the whole community. We call it Change Dog Plus Plus, and it's the best way to directly support Practical AI and all of the podcasts, videos, and other stuff we create here at Change Log. We have big plans and ambitions for this, but we are experimenting for now to make sure there's interest. And that means when you sign up today, you get Practical AI and whatever Change Dog shows you listen to now, except no ads. I guess that means this part you're listening to right now, it'll be gone. We also have some extended episodes planned, bonus content, merch store discounts, and a lot of ideas. But since it's such early days, we are offering memberships at

• 22:27Speaker 04

  a 40% discount for early adopters. That disappears at the end of August, so head to changelog.com/plusplus to join today. Lock in that discount, get closer to the metal, and make the ads disappear. Once again, that's changelog.com/plusplus. We'd love to have you supporting us as a member.

• 22:57Chris Benson?

  So you were talking quite a bit, used the word "community" a bunch of times as we were talking through that last section, and really interested in a couple of things. First of all, your insight into kind of how to do data science well, because there are, as we know, there are so many ways to go off the rails in a variety of ways. But I also want to throw in the fact that as we are talking today, we are still in this world that is dominated by the COVID-19 epidemic. And that has completely changed what we have been doing for years as professionals. And we've had to adjust workflow, we've had to adjust the way we communicate with people. What the word "community" means to us in

• 23:39Chris Benson?

  terms of implementation has adjusted. Could you give us some insight into how you're adjusting and how you help people think about doing data science well in this new environment?

• 23:51Speaker 00

  Yeah, I actually think this is an unsolved problem, and I'm glad we can talk about it because hopefully it'll get someone who is creative and excited about it thinking. Because this, you know, we're recording this podcast on Zoom, like, this is not great. This is not the endgame. This is in no way as rewarding or, you know, as helpful as having a personal or in-person connection.

• 24:20Speaker 00

  And our, you know, data science community of practice has, there are a bunch of events people go to. And I always find that, you know, if you want to know what somebody did, you read their white papers or their publications. If you want to know what they're, you know, doing now, you take them to coffee. If you want to know what, you know, ideas they're thinking about, but they haven't quite decided if they're good or bad yet. You have to really talk to them in a way where they're comfortable, usually at some sort of event. And now we're missing all of that. So we've lost that layer of connection. And I feel like we're also burning down a lot of the social capital we've built up before the COVID crisis.

• 25:03Speaker 00

  And so I, I think there is a wide open space for people who can figure out how we do data science together. How we continue to have this sort of open space to share and learn. When we're dealing with the fact that we can't travel and mostly we have to, you know, stay apart from each other. And I wouldn't claim to have an answer, uh, but I think it's an area where I would love to see a lot more attention.

• 25:33Chris Benson?

  I, I, I agree. It's, um, you know, last year as we were recording and talking about things. I know Daniel and I were always off to conferences. I live in Atlanta, uh, but I was in New York often, occasionally meeting with you and moving around and having great conversations. And so it has definitely been a, a challenge trying to bring the, the same level of quality into the conversation. And the same level of sharing insights, because when you're around the conference table, uh, you know, it is so much easier to, to just hop up and hit the whiteboard and have those ideas sharing. If you take that then to the next level and you're actually talking about, you know, producing, creating data science products.

• 26:16Chris Benson?

  And informing, uh, other products with your data science and, and contributing to that whole development effort. It's definitely gotten, uh, a little bit harder to get to those points quicker, uh, since then. And I was wondering, you know, as someone who is, who is certainly actively doing that now, you know, how have you adjusted to that? How have you tried to ensure that you're able to get there successfully in the same way you have in the years before we got to this point?

• 26:43Speaker 00

  Yeah, I wish I could say I'd solved it, but the things I've been thinking about are trying to observe what's missing. So, you know, one thing that has come up for me is that, you know, now that I have all of my meetings in this form through a screen, I've started to forget who I talk to about what. And it's because we're missing, at least for me, the physical cues that were tied to the storage of memory. So I have a couple of folks that we had a monthly meeting in a diner in Park Slope. And I realized I couldn't, now we're doing it on Zoom and I couldn't remember what we talked about last because we didn't have that, uh, you know, the physical cue of we were at this table and I had this like diner coffee cup and the eggs were overdone.

• 27:26Speaker 00

  Um, I love diners, so I say that with the biggest amount of affection. And so it's trying to say, okay, I've noticed this is missing. How do I create that context? So can I use different Zoom backgrounds for different uh groups of people I'm meeting with? Can I physically alter the space I'm in, which is always hard in a New York City apartment, but at least turn around or try and find another corner to sit in? And uh, when it comes to the, the work of data science, one thing I've noticed is missing is the casual brainstorming and relationship building. And I think these things are tied together. But it's really easy to talk about the work that's clear, but it's almost too easy to get caught up in the details of what's obvious

• 28:11Speaker 00

  and not to spend the time on what isn't obvious or those ideas you have that are just a little weird and you're not sure if they're the, the best idea or the worst idea. I have a lot of those. And you need to share with somebody to get their impression, um, to really know if it's worth thinking about or exploring further. So really trying to create the space for that kind of discussion, which is generally less structured, may involve talking about like, what do you see out your window in Atlanta?

• 28:40Chris Benson?

  Rain today.

• 28:41Speaker 00

  Yeah, I can show you what I see. Rain. It's a beautiful sunny day in Brooklyn. Yeah, so I'm trying to be very thoughtful about that. And then thinking about, you know, as we move into playtesting some of our new stuff. Um, you know, how we do that in a way that gives us the same kind of information that in prior years, you could just get by sitting down next to somebody and watching them try to use the product. I don't have an answer, but I think the first step is to be really thoughtful about what's missing and then try to create the space for it.

• 29:14Daniel Whitenack?

  Yeah, I really resonate with what you're saying, especially as a person in an organization that doesn't have like a big data science team or something like that. I oftentimes feel like I have all of these, what to me seem like really great ideas, but I really have no gauge on are they good or are they not? And I'm not like, now I'm not regularly going to meetups or anything like that in like a physical format. There's a guy that works down in Indianapolis that I've known for a while and we just have like a standing meeting on our calendars like every couple of weeks, like labeled ML water cooler and we just like chat about random things like you're talking about. So I really resonate

• 29:58Daniel Whitenack?

  with that. I think that's even like pre-COVID for people that were in that sort of organization where they're like hired in as maybe the first data scientist or like establishing some sort of data science initiative. That can be really tough. So yeah, I resonate with that a lot.

• 30:16Speaker 00

  You're right in that same area and it is definitely something that was an issue before COVID too. I think this has just exacerbated the pain of it because it was a lot easier if you were the only data scientist at a company to go out and have data scientist friends at other companies. And now you have to be very deliberate about it.

• 30:37Chris Benson?

  And you know, that raises some of the questions is that, you know, you still have customers, you know, right now that you're developing in your new stuff that we'll talk about and you have done that repeatedly over the years. How do you adjust to that right now? So your ability to, you know, once upon a time, you could go places where your customers would be and you could engage them in various ways that were familiar that are all gone at this point. How do you, as you're trying to build things now and you've already talked about how critical that customer and user feedback is, how do you achieve that today?

• 31:12Daniel Whitenack?

  You get into like survey science?

• 31:16Speaker 00

  I mean, yeah, it uh, we're really trying to hack together the closest approximation. So, you know, I don't have an answer, but figuring out like, can I give you something to run in your own environment and then be on video and try and observe you that way? You know, can I ask you a bunch of questions that seem adjacent to the thing I'm actually interested in to try and understand the context around, you know, how you think about what you're doing. And so, for example, for Hidden Door, we've been asking a lot of families, like, how do you tell stories together? Like, how do you feel about it as a parent for kids? It's like, why is this fun for you? Like, how many characters do you have? Do you always make up new characters?

• 31:57Speaker 00

  And so it's really just trying to develop that empathy through the means we have access to right now. But, you know, honestly, it is a challenge.

• 32:08Chris Benson?

  So I guess at this point, you you started to allude to characters. So it might be a good moment here to talk about your current work and the current project you're doing. I know that you announced on LinkedIn, which is when I reached back out to you about Hidden Door. Could you tell us a little bit about Hidden Door at this point?

• 32:25Speaker 00

  And? Yeah, I'm happy to. Um, I mean, Hidden Door is a product for using machine learning and AI as a tool for creative assistance, primarily focused on kids and the people who like to tell stories and write with them. And what I mean by that is that I think the tools of NLP and natural language generation are tremendously powerful, but they're primarily useful to provide the things that are repeated in narrative and to accelerate other human creative efforts. And by that, I mean,

• 33:06Speaker 00

  Kids are deeply creative, but they are not necessarily familiar yet with things like the standard narrative arc, where you introduce tension, um, how you manage conflict and narrative. In genre writing, they may not understand the things you get for free in a genre versus the things you have to explain. They may have the spark of an idea, but but find that it's actually a lot of work to create the story around it. And we believe that the technical tools around NLP and language generation are just now starting to become powerful enough to be supportive tools for that sort of storytelling. And by the way, they're really fun.

• 33:50Speaker 00

  Um, and I think anyone who spends time with kids or has kids of their own knows that kids are endlessly creative. And they often demand endless creativity of their parents, too. Like so many parents I've talked to, and it's my own experience as well, you know, it's like, can we tell yet another story about the wombat who really wants to play basketball? Um, or the grape that rolled down the stairs onto the subway and went on an adventure? And so we're building essentially tools that can be part of this creative process. But there are a couple of things I want to be very clear about. One of which is that I don't expect the technology itself to be creative.

• 34:33Speaker 00

  The creativity comes from people. And the second is that, uh, this does not do all of the work for you. It's more of a partner in that, uh, as you explore and create and play, uh, it starts to fill in the structures and the gaps and the descriptions and does it in a way that is really fun and is actually more like a game than like a homework assignment. And then the last thing I'll say, because I know this audience is probably deeply familiar with things like GPT three and a lot of the issues with natural language generation tech, is that there is a real need to build, uh, structure, safety, coherence, and memory

• 35:17Speaker 00

  into these systems before they can be deployed for any human-facing application, much less one that you're willing to put in front of young people. And so there is a huge amount of engineering that we're thinking through right now around how you build those systems such that we're confident letting people use them in a way that is, you know, unsupervised. And I mean that in both senses of the word. And the last thing I'll say is that I grew up playing Dungeons and Dragons and a bunch of other role-playing games. And so if you need a metaphor for this, it really is the dungeon master. Like, finally, can we have our computer system play the role of a dungeon master

• 35:59Speaker 00

  in structuring and guiding a story without being, you know, deterministic about where it starts or where it ends?

• 36:32Daniel Whitenack?

  So you mentioned a couple of things that I would really love to dig into. One of one of those is you mentioned how you thought that we were kind of at a point where NLP technology could augment some of these things. And so I want to dig into that a little bit. And also, you mentioned like the safety aspect, which is of course, like a big topic. But maybe we can dig into that first one first. So from your perspective, as you've kind of seen a lot of different kind of subfields within data science and AI growing, what really catches you catches your attention about like the growth in NLP right now? And, you know, maybe why it's crossing into some areas that can augment more sophisticated workflows like this?

• 37:16Speaker 00

  Yeah. So my last company was Fast Forward Labs, which I mentioned earlier. We did a lot of independent applied research. Our very first research effort at Fast Forward Labs in 2014 was around natural language generation. It's an area I've been interested in following since then. And at the time, the state of practice was essentially that you pre-generated template sentences. It was more like a pre-generated Mad Libs-style thing. And then you had a process that would dynamically assemble those sentences. And it meant that, at the time, the tech was really good for

• 37:53Speaker 00

  Taking quantitative data, so something like a weather report, for example, or, you know, a company's quarterly earnings report, or even scores in a sports game, and writing an article off of that. If you showed it something that it had not seen before, it just did not have the language and did not have, like, the sentences just didn't exist to be generated. And at the time, we built, at Fast Word Labs, a system that would take structured data about real estate. So apartments in New York City, we crawled sixty thousand apartment listings. And then we built a little system where you could say, okay, this is a two-bedroom, two-bathroom,

• 38:37Speaker 00

  right by Central Park. It has a doorman and washer dryer, and it would write the ad for you. And it worked well enough for things that were common. So things like that one I just described. But you could also put in things like an eighty-bathroom, one-bedroom apartment. And it would try, like, it would put a few sentences together, but it would sound like, you know, it wouldn't sound very good. What has changed since then is the use of transformers and the ability to build these incredibly large-scale pre-trained models that excel at the token prediction task. So that means that you essentially take, at an intuitive level, all of the internet that's mostly English, a bunch of books,

• 39:22Speaker 00

  and kind of whatever other commentary data sets we can throw in there, you know, train a, you know, multi-billion parameter model against that. And then you use that thing now to, given a prompt or a series of tokens, predict what the next token is going to be. Now, there are a couple of things that are really interesting about this, right? So one is that these models are huge. And this is both a good thing and a bad thing. We'll come back to this in your second question. The second part of it is that what I actually think is really transformative here is not that it's solving a problem that couldn't be solved before, but rather that before you would have to train, let's say you wanted to do a translation from English to French and

• 40:06Speaker 00

  classification of something as, you know, I don't even know, what are we classifying these days? You're something that's about everything, right? Happy or sad. Maybe we have a hope of actually solving sentiment analysis for real now. And you want to generate some language, like you would have to build a system that was custom-built for each one of these applications. Now you have a general model that can be used for all of them. That's pretty mind-blowing. Second thing is that the ability to describe a task with P-shot learning. So to give a couple of examples of what you want and then have the predictor be able to actually follow that, that's really amazing. It actually says to me that we will likely change our

• 40:48Speaker 00

  expectations of how we interact with NLP systems in the future, where rather than, you know, sort of building these custom-purpose pipelines for one task, we'll expect the, and we'll be able to create these sort of general systems that we can tune for a task locally, sort of at the backend. There are a bunch of things implied by that about infrastructure you need. Like, I don't think everyone's gonna have a deep learning box in their house. I do have one. It's more of a pain probably than it's worth.

• 41:20Daniel Whitenack?

  Speaking to my current pain.

• 41:23Speaker 00

  Yeah, it looks good on the spreadsheet cost-wise, and then you think about the hours you spend trying to get some driver installed for something. It's just not fun anymore. That's my opinion. I'm glad you're having fun with it. There's a step function improvement in the capabilities of these NLP systems broadly, which means that, actually, again, this isn't something that's completely new, but the speed of development, the ability to play and use them flexibly in different parts of a product has changed. Like, the cost function's changed. And so that's really exciting to me. And then the ability to think about, like, when you think about what these systems are

• 42:06Speaker 00

  good for, they aspire to create the most mediocre drivel that humanity would create and put on the internet. And that is not what we need for writing brilliant stories. And it's not what what kids need for learning to write brilliant stories. And so I'm also really excited about the opportunity that building on this foundation opens up um to actually create something that is able to encourage that brilliance.

• 42:36Daniel Whitenack?

  Yeah, and I guess to tie in that last part of where we're going with the the safety side of things, you mentioned kind of the large size of these models and that, you know, we're now scraping much of the internet. Of course, there's a lot of things on the internet that at least I wouldn't want to show to a child, I think. So there's that aspect, I guess, which is around safety and, you know, context and and all of those things. There's also like, you know, you're mentioning much of the internet, which is mostly English, right? And there's, of course, a lot of kids out there that don't speak English. So, yeah, I definitely see there's there's some potential issues. And of course,

• 43:18Daniel Whitenack?

  you you can't tackle everything at once. But what are some of the main challenges that you're thinking about as you're trying to leverage these these models for the particular audience that you have in mind?

• 43:31Speaker 00

  Yeah, I mean, largely, I'm thinking about it as allowing the maximum flexibility and creativity within a constrained problem space. And just to be very clear about what it means to be safe, like, it took me two tries to get something deeply misogynist out of GPT three, and I was not trying. And that's not good. Like, I

• 43:52Daniel Whitenack?

  Disturbing.

• 43:52Speaker 00

  As of today, I don't think you can put this in front, the raw output in front of people at all, unless you're constraining the domain in which it's able to produce text. So I do think there you could use it today for things like translating from language to code. I built a thing that writes really shitty SQL queries. That seems pretty safe, like. Maybe, but for things that are generating or hallucin- deliberately inciting the model to hallucinate, you know, fictional worlds, it's not safe now. So constraining the problem space such that we're able to manage, say, descriptions of characters and descriptions of items

• 44:37Speaker 00

  in a way that we can then run another layer of machine learning classification. And even in many cases, human review and human feedback to ensure that what is coming out of the system beats some notion of content standards. We also have, you know, other issues around getting what you expect. So, you know, GPT-3 and other models sort of let you dial down or up the randomness of the output. But when you dial it down, it actually ends up being quite boring. And when you dial it up, it's completely random and you can't direct it where you want it to go because it doesn't have taste. We have taste, but as far as the math knows, you're just sort of randomly exploring a space.

• 45:20Speaker 00

  And any particular set of tokens is, you know, just as likely as the next one. And so we need to build systems that learn and reflect that taste. And I believe, you know, at least for our approach right now, those are systems built on top of, you know, fine-tuned GPT-2 systems in this case. And so I also want to be very clear, uh, because we're recording this, you know, here in August. We're at the beginning of this work, so I can't speak to it as if I have solved this problem. I don't want anyone to point to this discussion and say, oh, they did it, so it's great. It should be easy. It's not easy. Um, and I think there's almost as much work involved in what we have ahead as there is in getting to this point in the first place.

• 46:05Chris Benson?

  So I actually want to pull us back a little bit from, from the technical conversation. Because if, uh, if we finish up the conversation in a few minutes without me asking uh, what I need to on behalf of my daughter, I am in deep, deep trouble. So I, I'm curious, I totally recognizing that you're still fairly early in the process of building what you're building. With hidden door. The first thing that I know when I go downstairs in a few minutes is she's going to be asking, what is it? What's it about? And you know, she has these other games in virtual worlds and she's going to be kind of trying to compare it with that. And so that's kind of what I, if you could kind of, from that kid's perspective, what, you know, we've, we've talked about, you know,

• 46:47Chris Benson?

  the ability to use NLP in this and, and you've mentioned it kind of being the AI driven dungeon master. But could you tell us for a moment what your vision is for what, when you get to a point where you're ready? What that's like for the kid? What, what, what you think they're going to experience? Maybe give us a little, a little quick example.

• 47:05Speaker 00

  I'm so glad you asked. I'm hoping she'll be one of our play testers too, if she's up for that.

• 47:10Chris Benson?

  Oh, she's very up for that.

• 47:13Speaker 00

  Yeah. So from the kid's point of view, there are really two different things going on. So one is to give them essentially a buddy that is co-writing with them, but where they are in control. So, you know, the kid can say today, you know, I want to be a wizard who's getting on a rocket, going to the moon. And the system will say, yes, and on the moon, we're going to find, you know, a pizza shack. And obviously I need this help too, because I'm just making this up off the top of my head.

• 47:43Chris Benson?

  That's fine. No, it's fine. It's good. Keep going.

• 47:46Speaker 00

  Then the kid can say, oh no, it's not pizza. I don't like pizza. It's spaghetti, you know, or it's, you know, it's porridge, whatever fits in their model and the system will start to generate things that adapt. So it's really sort of this partner buddy where they are able to, you know, hit a button or swipe, or have something that can support their creativity. And it reduces the, the work of, you know, creative play by providing that support. And they can say, oh yeah, this is great. You know, we, you know, I opened the box and I did find, you know, a shard of rainbow inside. Or they could say, no, you know, I hate rainbows. It was a horseshoe.

• 48:28Speaker 00

  And then the system will adapt to that. And then the next thing will, will be that as well. At the same time, this game is encouraging and rewarding creativity and bravery and certain behaviors. So you also have it playing the role of a writing coach. Sort of helping them think through, you know, what do you think should happen next? Where should this character go? Um, you know, sort of encouraging them um to branch out a bit if they get stuck, giving them that guidance on where they need to go. Um, but always leaving them, you know, sort of in the driver's seat. I also, you know, as I said, we're still prototyping and playtesting. But I think that from the kid's point of view,

• 49:13Speaker 00

  this is something that helps them explore these worlds um that they have in their own minds and they already want to explore through text. And it's something where their characters can represent their individual experience. So, you know, every kid has things in their lives that they don't see reflected in media. And this is, of course, particularly true for um, you know, some people over others, or I have a friend who's an immigrant who said, you know, my kid uh is not going to see any stories about the clashes of cultures that you see in our family, because our family is so unique. But she wants to create these stories for herself.

• 49:57Speaker 00

  Um, so I think one of the the real exciting things is letting kids uh sort of show their own experiences through these stories in a way that maybe they're not seeing in traditional media. And then also, because this is a dynamic system, these characters can grow with them. I know that, you know, especially young kids, like, you know, an eight-year-old, they're changing all the time, and they're learning so much. They're having all these, you know, new and amazing experiences. And the character that they wanted to play with six months ago, you know, can grow up with them and can have experiences along with them, which I think is something pretty, you see it in video games, but it's not something we typically see in books.

• 50:42Speaker 00

  So I'm I'm hopeful they'll be able to create those sorts of experiences.

• 50:47Chris Benson?

  It's interesting. It sounds like she has kind of single-player games that she does currently and she has or experiences in all games. They can be educational. And then there are multiplayers with other kids. To some degree, it feels like it's a bit of a hybrid where you kind of have this AI-enabled buddy that creates a multiplayer experience in what would otherwise be a single-player engagement. Is that a fair way of representing it?

• 51:12Speaker 00

  It is, though. I'll also put a little asterisk on this.

• 51:15Chris Benson?

  Okay.

• 51:16Speaker 00

  And so we're thinking deeply about what this looks like to play with other kids and with your parents as well. And so I think, yes, it is having, as we've conceived it right now, the vision is having that AI as a second player, but also not an equal one to you. So I don't know if you have an Amazon Echo or Google Home.

• 51:39Chris Benson?

  We do both.

• 51:39Speaker 00

  But the way that, yes, so the way that kids are able to use those devices to play music, to ask questions, they don't think those are people. They don't think those are peers. They're very much the one who's driving where you go with that interaction. I see a very similar interaction here where they have the ownership of the story. They're the one creating, but this is a tool that they're having a dialogue with that is helping them, you know, sort of think through and explore those stories.

• 52:08Chris Benson?

  Okay, I guess, final question is just any thought toward the future about where, I'm really interested, especially in this application of NLP. We hit it a lot, but just kind of where where this may take us going forward, you know, what are some of the future visions that you have? That maybe not strictly things that you're planning to do, but things that you could see happening in the industry that might affect kids in this positive way to kind of finish out.

• 52:34Speaker 00

  Yeah, I mean, I love thinking about these questions broadly. I think that machines that can understand language well and can respond to us in language are incredibly powerful because language is the interface we use to talk to each other. And the ability to take information and represent it in language, whether it's for children or for anybody, is something that is really powerful.

• 53:02Speaker 00

  And we haven't yet seen what the set of products and interactions. Like, we haven't seen the end of that yet. I don't think chatbots are the height of innovation around interacting with systems through, you know, natural language. So that's something where I don't know what exactly it looks like, but I do think that a decade from now, it's quite feasible to think that it'll be a commodity to interact with a lot of systems and not just in a like, you know, Alexa, play me the songs sort of way. But in a, you know, like, can you take a look at this data set and actually tell me something meaningful out of it? So that I can make a better decision sort of way. So that's something I'm pretty excited about.

• 53:46Speaker 00

  I also think we're seeing this proliferation of sort of ways for people to create with this technology. And again, this is actually starting at the professional level. So with Auto ML tools. But we're seeing much greater democratization of access to the tech. And honestly, as much as I love data scientists and I consider myself to be one, I think the most creative applications come when you take that capability and you give it to people who have some other expertise or some other world they're living in. And so I'm pretty excited to see what people are able to do with it when it doesn't take a huge investment or a huge amount of technical skill to actually start to play with this stuff.

• 54:31Daniel Whitenack?

  Awesome. Well, I think that's a great place to wrap up. Really good perspective. I know that there's so much, so much to talk about in these areas. There's only so much time to cover, but we're gonna for sure put the link to Hidden Door and some of these other things we've discussed in in our show notes. So if you're curious and you're listening in on this later, maybe, and and curious about Hidden Door, check out those links. We really appreciate you joining us, Hillary. It's been a great conversation.

• 55:00Speaker 00

  Thank you both. This has been a lot of fun.

• 55:05Speaker 04

  Don't forget, we have a giveaway going on in celebration of episode 100. Enter for your chance to win some awesome AI hardware from Nvidia, Intel and Google, plus Practical AI and Packyderm Swag. We're giving away three bundles, so you have a good shot at them. Check your show notes for details on entering. You have until the end of the month. Speaking of Packyderm, a little birdie told me they have a big announcement coming soon, and you should join their Slack channel to stay tuned. Learn more about that at Packyderm.com. Thanks to our longtime sponsors for their continued support. Shout out to Fastly, Linode and Rollbar, and to the mysterious BrakeMaster Cylinder for these awesome beats. That's all for now, we'll talk to you again next week.