AI code that facilitates good science

We’re talking with Joel Grus, author of Data Science from Scratch, 2nd Edition, senior research engineer at the Allen Institute for AI (AI2), and maintainer of AllenNLP. We discussed Joel’s book, which has become a personal favorite of the hosts, and why he decided to approach data science and AI “from scratch.” Joel also gives us a glimpse into AI2, an introduction to AllenNLP, and some tips for writing good research code. This episode is packed full of reproducible AI goodness! Sponsors: DigitalOcean https://do.co/changelog – The simplest cloud platform for developers and teams Whether you’re running one virtual machine or ten thousand, makes managing your infrastructure too easy. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. DataEngPodcast https://www.dataengineeringpodcast.com?utm_source=changelog&utm_medium=podcast&utm_campaign=practicalai – A podcast about data engineering and modern data infrastructure. Brain Science https://changelog.com/brainscience – For the curious! Brain Science is our new podcast exploring the inner-workings of the human brain to understand behavior change, habit formation, mental health, and being human. It’s Brain Science applied — not just how does the brain work, but how do we apply what we know about the brain to transform our lives. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Joel Grus – X https://x.com/joelgrusChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Allen AI https://allenai.org/AllenNLP https://allennlp.org/AllenNLP demo website https://demo.allennlp.orgWriting Code for NLP Research​ https://docs.google.com/presentation/d/17NoJY2SnC2UMbVegaRCWA7Oca7UCZ3vHnMqBV4SUayc/edit?usp=sharingI don’t like Notebooks https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit?usp=sharingJoel’s website https://joelgrus.comAdversarial learning podcast http://adversariallearning.com/Books “Data Science from Scratch” by Joel Grus https://learning.oreilly.com/library/view/data-science-from/9781492041122/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com slash change log.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front end web servers that need to be full duty CPU all day, every day. Then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co slash change log. Again, do.co slash change log.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slay with us around various topics of the show at change.com slash community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:46Daniel Whitenack?

  Well, welcome to another episode of Practical AI. I'm Daniel White neck. I'm a data scientist and I'm joined by Chris Benson, who is a chief strategist for AI and high performance computing at Lockheed Martin. Hey, Chris, how are you doing? I'm doing good. How are you today? Doing pretty good. Made it through the through the 4th of July without blowing myself up or or anything like that. So how about, how about you? Did you have a good holiday?

• 2:12Chris Benson?

  I did. I had just family stuff. I got kissed by a llama. I'll have you know, I went to a friend's house. They had llamas and the llama came up and planted one right on my face. So that was an interesting experience. I was not sure what to do about that. I declined to take it further with the llama. I decided that that was not the relationship I wanted to pursue. But that was, that was the highlight of my four sadly.

• 2:33Daniel Whitenack?

  Probably, probably a good thing, but uh, yeah. Well, um, on, on that note, maybe a more, more relevant note. Given that this is, uh, this is practical AI. In our, in our community on Slack, uh, which you can find by, uh, going to changelog.com slash community. But oftentimes we're, we're asked maybe what are good practical ways to start learning about AI? Maybe from, from software engineers or those with an engineering background who are getting into AI and data science. And pretty much I always tell them that a great place to start is with, uh, Joel Grus' book, uh, Data Science From Scratch.

• 3:15Daniel Whitenack?

  That was one of the kind of instrumental things and helping me understand what data science is. And I frequently go back and reference that book. Well, uh, Joel has joined us today. I'm really excited to, to talk to him about his book, which now has a second edition. And also some things he's doing, uh, at the Allen Institute for AI in Seattle. So welcome, Joel.

• 3:38Speaker 03

  All right. Thanks for having me here.

• 3:39Daniel Whitenack?

  Yeah. Uh, maybe, uh, you could start by just giving us a little bit of, uh, background about yourself. You know, uh, how you got into data science, uh, how you got to Allen AI and what you're working on now.

• 3:53Speaker 03

  Right. So originally I studied math and economics. This was a long time ago, uh, well before data science was a thing. Uh, and so what I did, what a lot of people with math backgrounds do is that I went into quantitative finance, options, pricing, uh, financial risk, things like that. Uh, and I discovered I didn't like it very much. Um, and so I was working at a hedge fund and it went out of business and I got laid off. And I was lucky enough, and this was 2006.

• 4:20Speaker 03

  To get hooked up with a startup called Faircast, which was an online travel site doing basically price predictions on airfares. They'd use machine learning and you'd say, I want to fly from Seattle to Los Angeles on these dates. It would tell you, you know, the best price is 300 bucks on Alaska and we predict the price is going up, so you should buy now. Or we predict the price is going down, so you should wait. So they were doing a lot of machine learning before putting machine learning in consumer products was particularly popular. I was not doing machine learning. I was doing BI, writing SQL queries, building spreadsheets, things like that. But as time went on, I started getting a little bit into the machine learning side of things, a little bit into the Python scripting side of things, and just started to grow my skills in that area and learn from, learn by osmosis

• 5:05Speaker 03

  from some of the data miners, as we called them. And so by the time it was, you know, 2011 or so, and data science was just starting to become a thing, I said, you know, that's like what I've been doing, but kind of better. And now there's a name for it. So I want to become a data scientist. And I sort of pushed myself in that direction, managed to talk my way into a data science job, and spent three years basically running data science at a startup called Volometrics. While I was at Volometrics, I ended up writing a fair amount of production code, for better or for worse, because that was a startup doing analytics on enterprise collaboration data. And so the data science kind of was the product. And so that meant that the data scientists end up writing a lot of production code. And I found that I really liked it. I liked building things. That was a lot of

• 5:49Speaker 03

  fun for me. And so I wanted to look for opportunities to really grow out that skillset. And I had the opportunity to interview at Google. So I sort of crammed for a while, teaching myself all the computer science that I should have learned, but never did, because I didn't study computer science. Somehow passed the interview, went to Google for a couple of years, and really kind of leveled up in terms of software engineering, you know, writing code, building things. But after a couple of years there, I sort of felt like I wasn't excited about the projects I was working on, you know, building tools to help ad salespeople sell more ads, or writing backend services in C++ to replace spreadsheets, things like that. And so, you know, I had the opportunity to

• 6:30Speaker 03

  Interview at AI2, which is a research nonprofit doing fundamental AI research. And I decided to move over here. I've worked on a couple of different projects here. The first one was called Aristo, which is basically teaching computers to pass science exams. And now I work on a team called Alan NLP, which does fundamental NLP research as well as we build a library called Alan NLP, which is a deep learning library for doing NLP research. And so most of, most of my job is working on building that library, supporting people using that library, adding features, adding models, fixing bugs, giving tutorials, giving talks, and so on and so forth.

• 7:09Daniel Whitenack?

  Yeah, it's, it's interesting that you mentioned kind of having to level up your, your computer science skills. I remember kind of like getting into data science from a physics background and I, I feel kind of similar to you in that like, it was around, around the same time, a little bit later, but data science was still kind of the hype was growing. And I, I felt like similar that I kind of talked myself into a position and then learned a bunch of these coding things later and like learned how to do a bunch of things like on, on the fly. I'm not sure now, like if I was in the same place, things are so competitive now in like data science and like, it was really hard interviews and all of these things.

• 7:49Daniel Whitenack?

  I'm not really sure if I would have like made it in, given, given where I was back, back in the day. So it's, it's interesting that you had a, had a similar experience.

• 8:01Speaker 03

  Yeah. So basically my, my interview to be the, basically the data scientist of Volometrics, was me meeting with the CEO, him handing me a printed out SQL query and asking, do you understand this? And, and I said, yes. And then he hired me. But, uh, so he hired me basically in like a contract to hire position because it was so early in the company that they weren't set up to have full-time employees yet. So in that sense, he wasn't taking a ton of risk because he could have easily not converted me to full-time employee. But still, it was not a very rigorous process.

• 8:35Daniel Whitenack?

  You don't happen to remember that SQL query, do you?

• 8:37Speaker 03

  It was about. Five pages long.

• 8:40Chris Benson?

  Oh, boy. Okay, serious sequel right there. So I guess to turn toward the book, I guess I'd like to start off by just asking you, why do you think it's important to think about data science from scratch, since you use "from scratch" in the title of the book? And, you know, why is that approach useful to get people into data science and AI? What's your take on it?

• 9:03Speaker 03

  So, the practical answer is that, you know, it's very easy to install scikit-learn, points, you know, random forest, classification at some data set, and spit out a result and sort of have no idea what's going on, which, you know, is good in some ways, and it's also bad in some ways, that if it goes wrong, it's very hard for you to figure out why it's gone wrong. Or it's possible you've made some conceptual mistake, or you're cheating by looking at the test data, or things like that. And so if you don't know those things, it's very easy to lead yourself astray. So that's kind of the practical answer. The moral answer is that, as I mentioned, my background is mathematics.

• 9:45Speaker 03

  And one thing that's a very strong, almost moral principle in mathematics is that you're not allowed to use theorems if you haven't proved them yet. So, you know, math classes are set up so that the first quarter will prove these theorems, and then the second quarter will build on them, and the third quarter will build on them. So much so that, you know, when I was in grad school, I had a year-long class, but the professor changed between the first semester and the second semester. And the second semester, he came in on the first day, and he says, "I'm so glad you proved, like, theorem X, because I need to use it." And with the premise being that, like, if we hadn't gone through that proof, he couldn't use it, and then we'd be, like, in a bad spot. And so somehow that, like, burned really deeply into me,

• 10:27Speaker 03

  this idea that you shouldn't use things like that if you don't really understand how they work. And so, you know, when I want to sit down and say, "What is linear regression? You know, what is a random forest? What is a neural network?" I feel like I need to really understand under the covers, what's going on here? What is it doing behind the scenes? Otherwise, I don't feel comfortable using those things. And so that's, that's kind of the, maybe moral, you'd call it, reason for doing it that way. I feel like it's a good approach.

• 11:00Daniel Whitenack?

  Yeah, so I mean, I definitely get what you're saying. I remember someone telling me when I was in physics, like, one of the, one of the ways that you can tell, like, the difference between, like, a math physics person and like a physics person. So there's this, like, group of physics people called math physics people is that, like, physics people are okay with just, like, trying something, and if it works, they're, they're happy. Whereas, like, math physics people, they would never, like, use something unless they, they had proved it first. So, like, I definitely, I definitely get that, but also, I, I love how, in, in this book in particular, like, I've looked at some kind of statistical learning, machine learning, AI sorts of books

• 11:45Daniel Whitenack?

  that kind of start from the beginning and explain everything, and it is kind of all very, it's very mathy, right, which is, which is good, and it's, it's good to learn those things as well, but then it's, like, somehow disconnected from the coding side of things. And I think the, one of the reasons why I latched on to a lot of your content was, like, the code is, is right there, and you can go ahead and see, oh, yeah, like, there's, like, this abstract thing, but this is how it's directly connected to, like, Python code, and, like, I can see that it's, you know, it's not hard, that hard to implement this thing in a small number of lines of code, and, and it kind of makes sense. Was, was that kind of something you saw as, as a gap? And part of the reason why you, why you wrote this book?

• 12:29Daniel Whitenack?

  Or why was it important for you to kind of take this code-first approach?

• 12:33Speaker 03

  So, here's what I'll say. I think that when you put a lot of math equations in a technical book like that, people will zone out and not follow it, and for good reasons. You don't want to be reading, you know, a ton of math equations in a tech book. But the other thing is that, a lot of times, those equations are actually not the best way to explain something if you're using it in kind of an applied context, right? So, you know, basically, what I tried to do was, okay, here's a mathematical concept I want to teach, but instead of teaching it by giving you, you know, some latexed-up equations, I want to teach it by writing code that embodies it. And so that way, hopefully, you understand the math that's going on,

• 13:15Speaker 03

  but also, you can see from a Python point of view exactly what's happening. And so it's sort of it's sort of this middle ground where I want the math to be part of, you know, the text and part of the pedagogy. But I also want the math to be framed in terms of Python code, not in terms of equations.

• 13:35Chris Benson?

  So I guess in the book, when you're when you're building from scratch, what is what? What do you kind of start with and what is that? That order upon which you're building layer after layer so that the reader is is kind of rocking it all in their head? How do you approach that?

• 13:52Speaker 03

  You know, it took me a long time to figure out a good order for for doing this. But in essence, the structure is, you know, a brief introduction to Python. And then now we've got Python. Let's talk about linear algebra, which is mostly just working with vectors. And like I went serious from scratch where we're going to represent vectors as lists of floats and we're going to do arithmetic on list of floats. So starting from pure Python and just what is a list? Now we've got linear algebra, you know, I can take vectors, I can multiply them, I can do their dot product. And now that I'm equipped to do those things, I can say, let's start talking about doing statistics. So, you know, take them into these vectors and I can start talking about probability. What does it mean to have a distribution over things?

• 14:35Speaker 03

  And I can start talking about hypothesis testing and drawing inferences and just building step by step. And now, once, once I've spoken about all these things, there's a slight detour and let's talk about how do I get data? How do I work with data? How do I clean data? How do I process data? And because of the machinery we've built, we can talk about things like, okay, now I want to do principal component analysis. What does that mean? And now, once I've done sort of linear algebra and statistics and, you know, probability fundamentals, and once I've done working with data fundamentals, now I can start to push into machine learning. What is machine learning? Like, what do I need to understand about machine learning, independent of what model I'm using? And then just start building up. Okay, what is, you know, simple linear regression?

• 15:19Speaker 03

  What is multiple regression? What is logistic regression? What is a decision tree? What are neural networks? And then, once you build up all those things, then you can start talking about even further applications. What, you know, what does it mean to do natural language processing? What does it mean to do kind of recommender systems, things like that? And so that's kind of the kind of that is pretty much the progression of the book.

• 15:40Daniel Whitenack?

  And why, so why the, why the second edition? I guess, you know, were there, were there things that you felt like, you know, were, were left out? I know that there's, of course, now we're kind of in, we're, I don't know if we're past the data science hype, but we're definitely into the AI hype, right? And I think, I think if I'm not mistaken, you have a little bit more content about neural networks and that sort of thing. Was that part of the reason for, for the second edition?

• 16:06Speaker 03

  There was one overriding reason for the second edition, and that's that the first edition was written in Python 2.7.

• 16:12Daniel Whitenack?

  Ah, that would do it.

• 16:14Speaker 03

  And basically told people to use Python 2.7. And every day I woke up and felt guilty as hell that there was a book out there with my name on it, telling people to use Python 2.7. And, and so I said, I'm going to do a second edition that's Python 3. And then once I was like, so that was like 99% of the reason I did it. And then once I was doing that, I thought, okay, well now maybe I'll add, you know, a chapter on deep learning, cause that's a pretty important thing to know. And let me flesh out some of the other chapters and add some new stuff and I'll take the time to clean up the code and, and so on and so forth. But really the Python 2, Python 3 thing was, was 99% of my motivation.

• 16:50Daniel Whitenack?

  I like, I like your answer, by the way. I like the mea culpa. It's, it's funny.

• 16:54Speaker 03

  I don't even want to call it a mea culpa because I think at the time the first edition came out, probably Python 2.7 was the right choice. I, what I should have done is I should have written the code in the first edition in a way that was compatible with either Python 2 or Python 3, which wouldn't have been a ton of work, but I didn't do it. And so that was, that was a bad call on my part. But the reality is, is that even if I had written, you know, a Python 3 version, it would have been like Python 3.4 maybe, like at the time I wrote that book or 3.3. And that's not like a great long-term version to be on anyway, so.

• 17:37Speaker 04

  The Data Engineering Podcast is a weekly deep dive on modern data management with the engineers and entrepreneurs who are shaping the industry. Go behind the scenes on the tools, techniques, and difficulties of data engineering so you can learn and keep up with the knowledge to make you and your business successful.

• 17:53Speaker 05

  Can you give a bit of an outline about the motivation for choosing Jupiter Notebooks in particular as the core interface for your data teams?

• 18:01Speaker 06

  Yeah, and actually, when I first joined Netflix, it was a sort of toss at me. And I was definitely like, are we crazy? And the answer was like, we might be a little crazy.

• 18:09Speaker 04

  Go to dataengineeringpodcast.com to listen, subscribe, and share it with your friends and colleagues.

• 18:28Daniel Whitenack?

  So Joel, we've talked a little bit about data science from scratch and what you put into the book and like the code-first approach. I've also noticed like with with some of the other content that you've put out, like both in the in the book, but also in like, you know, talks that have gotten like a good bit of attention, like I don't like notebooks. And I saw another one writing code for NLP research. It seems like you put a lot of focus on sort of bringing good software engineering practices into like data science and AI. Is there a particular reason why that's kind of become a passion of yours and like a driving force?

• 19:09Speaker 03

  Yes, one, yes, that's that is kind of like my my big pet cause. So the selfish reason is that I've become sort of over my best judgment, sort of a software design geek. And what I mean is I spend a lot of time thinking about what is the difference between good code and bad code, even if both work. And in particular, you know, as someone who uses code to teach, I spend a lot of time thinking about what is the clearest way I can express the idea I'm trying to express in code. And what I Found is that going through that exercise over and over again has really drilled into the importance of doing that and has spilled over into, you know, my actual work.

• 19:51Speaker 03

  So even when I'm not teaching data science, but I'm writing code for my job, I'm still always thinking, what is the clearest way I can express what I'm trying to do? Because, you know, in a month I'm going to come back and look at it, or someone else is going to come back and look at it. And, uh, you know, if it's not clear, they want people to understand it. So that's kind of the selfish reason. The practical reason is there's a lot of bad data science code out there. And I think some data scientists even sort of take pride in, you know, I'm a data scientist, I write data science code. And I think, I think that's unfortunate because for a couple of reasons. One, I think that by not understanding software engineering best practices or how to write good code, you're basically putting yourself in a position where you cannot contribute to, or you should not be contributing to production code, say at your job.

• 20:35Speaker 03

  And you get this artificial division where I'm the data scientist, I'll write a model and then I'll hand it off to someone who's not a data scientist to, you know, put it in production or make it work in code. And I think that's unfortunate for a couple of reasons. One, I think data scientists can have a lot more impact if they can cross, you know, that divide. And I think they also become better data scientists if they can cross that divide because the ideas from good software engineering kind of flow back. Uh, you know, a second reason why, or maybe third, another reason why it's really important, I think, is that a lot of the things that are software engineering best practices are intended to help you, one, write correct code and two, get better at writing code. And those are both things that are equally important for data scientists.

• 21:19Speaker 03

  If you're a data scientist, even if you're writing code, you know, to run some experiment or something, you want that code to be correct. And you should be writing unit tests for that code because unit tests help make sure your code is correct. And you should be getting code reviews on that code. These code reviews are a second set of eyes and they'll find bugs and they'll point out things that may, maybe they're not wrong, but you could have done better or hated, you know, that psych it learn has this thing that would have done that for you. And so all of these best practices will make you a better data scientist.

• 21:48Chris Benson?

  You know, it's funny, uh, I'm sorry, keep going.

• 21:50Speaker 03

  No, no, that's it.

• 21:51Chris Benson?

  I was just gonna say, I actually sympathize tremendously with that take on it because I came into this from a software development, software engineering background. And as I moved fully into the AI space and was, and started working with data scientists, in a lot of cases, straight out of school and into the job, and they, they kind of, they were weak on the programming side, that, that was definitely, created a tension that I noticed, you know, where you had people that had different skillsets, kind of working and bringing different things to bear. That certainly was the case for us. I know that like, as we were thinking about the product of what we were doing having to be productized and, and getting into, you know, the, the data science pipelines that had been created at the company I was at at the time.

• 22:36Chris Benson?

  With that tension, have you experienced that, where, you know, people are like, well, I'm a data scientist, I just, I just don't care? It sounds sort of like you may have. What, what kind of, how did you get through those dynamics in terms of dealing with that, those different perspectives together?

• 22:51Speaker 03

  I think you really have to do a sales job, right? And so that's one thing I've been trying to do with some of my talks, and also in the second edition of my book to some degree, is to say, look, writing a test sounds like a hassle, doing code review sounds like a hassle, but I'm going to show you how your life will be better if you adopt these practices. And I think it's really hard to convince people to do it without having that sales pitch, this is going to make your life better. And that's not like me talking BS. I do think, as a data scientist, it will make your life better to do those things.

• 23:23Daniel Whitenack?

  Yeah, I definitely, like, I remember when I was in my first data science position, and I realized, like, you know, people started talking to me about, like, you know, not cutting over to master before you, like, have a review and a pass test and all these things. And I was like, I was very confused by a lot of that, because I did not come from a software engineering background. And so I was like terrified when I first went into code reviews. But over time, like, I realized that, yeah, I could, like, skip those things, and then, like, the whole system crashes down and it's my fault, and then it's like a big situation where, like, I caused this big thing, or I could go through a code review and like, Have some assurance about how this thing, you know, have some like knowledge sharing,

• 24:05Daniel Whitenack?

  but also have some assurance that like I, my logic was sound on this or that. And now I feel like I don't feel good pushing anything into a place where it's going to be used without having someone review it. So I think it is like a shift of perspectives. You have to, I'm glad that there's kind of this, this effort going on to educate people on these things.

• 24:27Speaker 03

  You know, that's one angle. Another angle is that by doing code reviews, you learn a lot, both in terms of having a second set of eyes on your code that can tell you, you know, here's another way you could have done this, or by reviewing someone else's code and saying, wow, I never would have thought to do it that way, but that's a cool way to do it. And so doing code reviews is a great way to, you know, keep learning and building skills and getting better at what you do. And hopefully if someone's a data scientist, that's, you know, something that they're interested in.

• 24:55Daniel Whitenack?

  Yeah, I think hopefully a lot of people in this field have, have that passion for learning. And it's definitely true that like, as you have a more diverse set of, of people and ideas working together, you can definitely learn a lot of things that you wouldn't have learned otherwise. So in particular with your writing code for NLP research talk, I was kind of just looking through that as we were preparing for this interview. And you talk about how to write code in a way that facilitates good science and reproducible experiments. Given that this is like practical AI, you know, that is a lot of our focus is, is giving people like practical discussions on this topic.

• 25:35Daniel Whitenack?

  I was wondering if you could just give us a few, like maybe the, the, some of the most important points from, from that, from that talk, or maybe other talks in terms of what are some like immediate wins that you can have in terms of writing code that is reproducible or writing code that facilitates good science?

• 25:54Speaker 03

  Yeah, so that talk is, is actually a tutorial from EMLP last year that wasn't just me. I presented it and wrote it with two of my colleagues here at AI2. But since then, I've sort of gone on the circuit of academic AI conferences, sort of beating this software engineering practices for AI researchers drum with reproducibility in mind. So, you know, what the first thing is just what I said before.

• 26:22Speaker 03

  Write unit tests, and that sounds weird to say write unit tests for your, you know, research code, but if your model is not doing what you think it's doing, I mean, that's bad science out of the gate, right? Like, if your model is like accidentally ignoring one of its inputs, then it's not going to be doing the right thing. So if you write tests, and the way we tend to do it is take a data set like the one you want to use, get a very small version, you know, maybe it has two or three instances in it, and now write a unit test that takes that data, reads it in, puts it through your model, checks that, you know, checks that output looks right, that the model can run on it without crashing, checks that the model can learn to predict it perfectly, because if you have three instances, your model should be able to learn it perfectly, things

• 27:04Speaker 03

  like that. And you make these actual unit tests that that run along with your code, then that gives you this really simple but basic guarantee that code is doing at least sort of what you think it's doing. So that's one, I would say, really important aspect. Another one is that typically you're going to run a lot of experiments, you know, maybe with minor variations, I'll vary some hyperparameters, or I'll try different kinds of word embeddings, or I'll try a different number of hidden units. And so what you want to do is you want to structure your code that makes it as easy as possible to run these experiments. And so what that speaks to then is making a real clean division between what I'll call library code and experiment code. So I want one code that's

• 27:46Speaker 03

  like, this is the details of how my model works, and this is the details of how data gets read off disk. And I want this code in a module that's like well-tested and it's not going to change. And now I want to run an experiment, and maybe that experiment needs to point in a specific data path, and it needs to use, you know, some other parameters and various other parameters. And I want the experiment code to basically lean very heavily on that module and just, you know, all it's really doing is taking the parameters that I fed in and calling that model. And so when you separate things out this way, it makes it really easy to, you know, run an experiment. You don't have to be saying, oh, gosh, when I did this one, what version of the code was I using? When I did this one, which version of the code was I using? In all cases, you're using the same code.

• 28:32Speaker 03

  You're just feeding in different parameters to it. And so this was a little bit, you know, this was one of my gripes against notebooks is that notebooks don't really lend themselves to this kind of division very easily. Either you have to, you know, pull all your library code out into a module, in which case you're only half working in the notebook and half not working in the notebook, or you have to do something really elaborate like the system that Netflix uses where you parameterize your notebooks and have, you know, a 50-piece system that runs them and instantiates them and saves them and so on and so forth. So, so, so I think making that clear division is another one. It's Python, so depending-

• 29:11Chris Benson?

  Not, not letting the notebook be a crutch in essence, or, you know, it's, it's inhibiting you in terms of good practices.

• 29:17Speaker 03

  Yeah, I mean, I, I could talk about that in, in various other contexts at, at great length. But yes, that is certainly one aspect of it. The other one, it's sort of obvious, but a couple more. One is write good instructions for how to use your code, obviously, because someone's going to want to rerun it and reproduce it. And, and if it's not obvious how to do that, then, then that's no good. And, you know, finally, the other one that's important is Python dependency management, like in most other languages, is a real pain in the ass. And whatever experiment you're running is going to depend in very intricate ways on, oh, you know, I use this version of PyTorch and I use this version of Spacy. And being as clear as possible about here's how to set up the environment. And, you know, that

• 29:58Speaker 03

  can be your requirements.txt, that could be a Docker container, it could be anything like that. But if you go out and look on GitHub at like data science projects, you can find a ton of them where there's literally no indication of how did I set up the environment to do this. It just starts with import PyTorch and it goes from there. And your chances of being able to reproduce that are very small.

• 30:19Daniel Whitenack?

  Yeah, I definitely, I definitely have felt that pain. I, I was curious on another point too. Like you've mentioned tests a couple of times when, when I'm like teaching workshops or something, a question that comes up a lot of times when we start talking about testing and, and this sort of thing is like, oh, well, you know, people will tell me like, oh, well, machine learning and- AI is is kind of different, you see, because like there's these like non deterministic pieces and there's like randomness. So you're not always going to get the same same answers. So like, how do you how do you test that? Or like they give the example like in software engineering, you could have a bunch of like table driven tests and you expect all those things to pass.

• 30:59Daniel Whitenack?

  But when you're testing on a test set in in machine learning, you don't expect all of them to to pass and that sort of thing. And I kind of like I kind of take issue with that because like if there was no way to test like randomness and non deterministic sort of processes, then like physics wouldn't exist, basically. But but I was curious to hear like your perspective on that. Like, there's definitely cases where you have like some randomness and other things in in your code. But it seems to be a point that people get hung up on when they start talking about testing.

• 31:30Speaker 03

  So there's a there's a couple of answers to that. One is that there are a number of things you can test that should be there are invariance of your code that don't depend on the randomness. So for instance, given an input, it should run without crashing. Like, that's not a function of randomness. Given a model and an input that has the right shape, the output should have the right shape and it should have the right fields or whatever. Again, that's not a function of randomness. That should be an invariant of your code. And so those things are easy and they don't depend on randomness. As I mentioned earlier, there are other things that depend on randomness, but you can test them in ways that are sort of randomness invariant. So, you know, I trained my model for on a small data set for 10 epochs. And I checked that the loss goes down each epoch.

• 32:15Speaker 03

  So is it possible that there's some weird random setup that would make that not be true? Possibly, but that test is basically always going to pass. And if it's going to fail, you know, one time out of a million, you mark it with a at flaky decorator and that's fine. Similarly, it's true that I have a big neural network. It has random weights, who knows what it's going to do. But again, if I take a data set that has three examples in it and I train that data set for, you know, 100 epochs with this big model, I can be sure that that model is going to learn to predict perfectly on those three data points. Like, it just has to. And so, again, you can come up with these tests in a way that give you some confidence that the model is doing what it's supposed to do

• 32:55Speaker 03

  and are robust enough that they don't care, you know, what the random initialization of the weights happened. They should work regardless.

• 33:11Speaker 00

  Well, hello there, listeners of Practically I. How are you? This is Adam Stokoviac. If you haven't heard yet, we're launching a new show called Brain Science. It's a podcast for the curious. Are you curious? Because if so, we're exploring the inner workings of the human brain to understand things like behavior change, habit formation, mental health, and what it means to be human. It's brain science applied, not just how does the brain work, but how do we apply what we know about the brain that can transform our lives? Learn more about the show and subscribe at changelog.com/brainscience. Until then, here's a preview of episode one, where we talk about the fundamentals of being human.

• 33:51Speaker 07

  We're also all designed to be in relationship. We are fundamentally hardwired to have social groups and, and this sense of attachment. And because I'm sort of a geek when it comes to research, what researchers have found is that attachment, which that's what we label, how we relate and connect with others, attachment is 100% learned, which means our genetics don't actually contribute to how we learn to stay in proximity with other people. And with that, that we all develop ways to manage the threat of the loss of a relationship. But nobody gets to opt out of going, I need to be in relationship with others.

• 34:37Speaker 00

  It's almost like we need to have that echo from another human being to let us know that we

• 34:41Speaker 07

  Yeah.

• 34:42Speaker 00

  were, were there or were alive, or just some sort of feedback loop. I'm not really sure how to describe that.

• 34:47Speaker 07

  Well, it really is this sense of being with, right? Like, I can't fight battles on my friends' behalfs or on my kids' behalfs, right? But the simple fact that I know of what's going on makes a difference because I would contend it's sort of like, I help them hold that weight emotionally. And so that actually leads me into the third thing. And the third thing that I would say in regards to the fundamentals of being human is that we all struggle.

• 35:17Speaker 00

  Oh, yes.

• 35:18Speaker 07

  Right?

• 35:19Speaker 00

  Big time.

• 35:20Speaker 07

  And that, you know, we don't always get to pick the way in which we struggle, but we all struggle.

• 35:27Speaker 00

  Well, if you like what you hear, you should go to changelog.com/brainscience. The show is not out yet, so don't get too excited, but you can subscribe and be notified as soon as the show launches. Once again, changelog.com/brainscience.

• 35:59Chris Benson?

  So, turning toward Allen AI, could you tell us a little bit about the purpose of the organization, what its mission is, and the types of problems it's oriented on?

• 36:09Speaker 03

  Yeah, so Allen Institute is a nonprofit research institute who is founded by Paul Allen, and it's funded by him and I guess now his estates.

• 36:19Chris Benson?

  Yeah, you may want to note who Paul Allen is, just in case there's anyone out there who's not familiar with him.

• 36:24Speaker 03

  Paul Allen was one of the founders of Microsoft, so he got pretty wealthy that way. And then after he left, he started a company called Vulcan, which does like a million different things. And, you know, he was a big Seattle person, he was a big everywhere person. And so our mission, at least if you look at like the sign on the website, it says AI for the common good. What that means is a little bit vague. I once interviewed someone who was very disappointed to find out that it didn't mean that we used like, we were working on like the homeless problem or anything like that. But what we do is fundamental research in various areas of AI. So, as I mentioned, the team that I work on, it's called Allen NLP, and we work on sort of fundamental research and NLP.

• 37:06Speaker 03

  So not super different from what people who are NLP instructors in a computer science department would be working on. In fact, we collaborate with them, we go to the same conferences they do, we present with them, things like that. There's another team called Pryor, which works on problems in computer visioning and visual reasoning. We have a project called Semantic Scholar, which is an AI-enabled search engine for academic papers where it indexes the academic papers, but it also tries to figure out what papers were influenced by this one and what information can I extract from the paper programmatically and things like that. And we have a project called Aristo, which I mentioned earlier, which does science question answering. So can we get an AI to answer science questions?

• 37:47Speaker 03

  And we have a project called Mosaic, which does, tries to get an AI to do common sense reasoning. So you ask questions about, science question is like, you know, when a plant makes, you know, food, what is that called? A, photosynthesis, B, whatever. But the common sense reasoning is more around understanding, you know, folk physics and folk psychology. You know, Billy did X, Y, Z. Why is it likely that he would have done this? And so those are kind of the different areas that we focus on.

• 38:14Daniel Whitenack?

  Yeah. So at Alan AI, is there like, there's obviously a lot of sort of academic-like research going on, but there's also, like you've mentioned, projects like Alan and LP. And there's like, I guess, products, like, you know, Semantic Scholar that are, people are using and out there. So there's, there's obviously like some mix of like researchers and engineers. I was just kind of curious like how the teams are structured there, like, and how that might be different from like maybe regular academic research where thing, maybe it's that things are more geared towards open source or like, I don't know, I was curious what that sort of looked like.

• 38:50Speaker 03

  So I would say, one, it varies a lot by team. So like my team, Alan and LP, is actually very researcher-heavy. It's mostly researchers and only a handful of engineers. Whereas Semantic Scholar, you know, they're running a website that has to be up. So they have a lot more engineers, you know, they have to be on call and they have a lot more front-end people and things like that that we don't really have. So it really varies by team. In terms of comparison with the academic department, I would say the engineering support that the researchers get here is, I would say, mostly not comparable to what people get in an academic research department. I mean, the engineers here are You know, I'm Google, other people are ex-Google, ex-Amazon, ex-Microsoft and so on.

• 39:31Speaker 03

  And my understanding, I mean, I've never been in a computer science department, but my understanding is that if you're working, you know, as a professor in a computer science department, typically you don't have these kind of engineers who are there to collaborate with you and support you. So I think that's a real deep difference between the way we do things and the way that an academic department does things. Now, if you go to somewhere like, you know, Google Research or Facebook AI Research, they do, or Microsoft Research, they do, those places do have this kind of engineering support.

• 40:01Chris Benson?

  So I noticed, I'm looking at the, at the webpage for Alan NLP, and you were talking about that being your own project, and it describes that it's an open source NLP research library built on PyTorch. I'm wondering, like, who is that geared for? You know, what is the audience for using it? What are the main use cases that you might apply it to? Any thoughts there?

• 40:22Speaker 03

  Yeah, so I mean, officially, the target customer is someone doing NLP research. So, you know, originally, originally, our target customers were the researchers here at AI2, but it was always an open source project, you know, from day one. And so then we quickly started, you know, taking on as customers, researchers over at the UW, researchers at other universities, people at more corporate research institutes. And then, you know, the line between research and work is really narrowing a lot in a lot of ways. So there's a lot of companies, you could go to them, startups or big companies or whatever, and they're doing things that are, say, at the cutting edge of NLP for their own problems.

• 41:03Speaker 03

  And so their workflows for solving these problems look a lot like the workflow of an academic researcher trying to solve an NLP problem. And so we do have some customers who, you know, work in companies but still use it anyway. And it's always, it's sort of an ongoing source of debate to what extent we should, you know, be actively soliciting, supporting more corporate customers versus academic customers. I'm giving A tutorial on Alan NLP and NLP more in general at the O'Reilly AI Conference in September, which is more of a practitioner focused conference, not an academic conference. And, and when this was mentioned, uh, you know, someone asked me, are we focusing on those practitioners now as customers for the library?

• 41:47Speaker 03

  And I'm like, well, you know, we've always sort of wink winked that those are our customers, but they're not officially who we're focused on, but we want them using it too.

• 41:56Daniel Whitenack?

  Yeah. And what are, um, so I'm just trying to think of like the workflow wise, like if I'm an NLP researcher, like where Alan NLP might, might fit in. So I'm thinking of like a comparable, people might be familiar with something like Spacy or something like that where Spacy has like, you know, pre-trained things that are available, right? Like I can do, um, some types of NER and other things and it's all pre-trained, but I can also train custom models, but I'm not really like messing necessarily with a lot of the like architecture that's, um, the architectures of the neural nets or something like that is, is Alan NLP kind of like a level down from that to where like you have more of the ability

• 42:36Daniel Whitenack?

  to, to mess with your models and all of that sort of stuff, or I'm just trying to kind of like gauge where, where it sits.

• 42:42Speaker 03

  Yeah, that's exactly right. So Alan NLP is not, for the most part, it would not be a substitute for Spacy. Instead, it would be a substitute for someone who's doing NLP stuff in PyTorch or in TensorFlow and says, you know what? There are a lot of problems that are caused by my working with text. And so I would like to, you know, use a library that has in it abstractions that are really intended at people tackling NLP problems and allow me to not worry about some of the nitpicky details around masking and padding and recurrent layers and things like that. But working in Alan NLP is much more similar to, I mean, when you work at NLP, you basically are working in PyTorch.

• 43:27Speaker 03

  You're just working in a PyTorch that has a bunch of higher level abstractions in it.

• 43:32Daniel Whitenack?

  Gotcha. So you're, so it's built on PyTorch essentially, right? So if you were coming at it, I'm just curious from TensorFlow, do you need to kind of go learn PyTorch first and then move into it as an intermediate thing? Or, you know, what, what would be with, with all the people out there that are also doing, you know, uh, TensorFlow in addition to, to PyTorch? What, what's kind of the migration path that you would see into that?

• 43:53Speaker 03

  Um, that's an interesting question. Uh, you would want to learn some PyTorch because Alan L P is not really insulating you from understanding how PyTorch works. The way I would describe it is, and this is going to be a terrible analogy and I'm going to regret saying it as soon as I say it, but, but imagine you have like a big bag of Legos, right? And just like only the squares and the rectangles, right?

• 44:16Chris Benson?

  Okay.

• 44:16Speaker 03

  And then you're building houses or whatever, cars or whatever. Um, and you build everything using the little square blocks. Then someone comes in and says, hey, you know what? Uh, I noticed you're making like a lot of walls. Uh, so here's a piece that's just a wall. And now you can use this one piece to make a wall instead of having to build each wall out of like little blocks, right? And so that's kind of my analogy for what we're doing. And so that allows you to, you know, now I don't have to worry about walls or now I don't have to worry about doors. But you still have to know how Legos work and you still have to know how to build with Legos and you still, you know what I mean? Does that make sense?

• 44:49Chris Benson?

  I do, and it's a good analogy. Uh, I have a seven year old, so we have Legos all over the place. That was perfect for me.

• 44:54Daniel Whitenack?

  So, uh, I'm kind of curious, like being, uh, you know, I've been involved in certain open source projects over, over time, but always geared towards kind of like the, I don't know what you want to call, like the regular software engineering community. Or what have you found to be kind of like unique challenges of creating an open source project for the AI community or more for the research side, side of things? Um, have, have there been unique challenges with that?

• 45:20Speaker 03

  So this is like the biggest open source project I've ever really worked on. In fact, it's probably the only like multi-person open source project I've ever worked on. So it's hard for me to say whether the challenges we face are unique. I tell you, some of the challenges that we face are people who submit issues without enough information about what they're doing or how to reproduce the problem they're running into. But I imagine most open source projects probably face that issue. The one that we struggle with a lot, I think, is that we have extremely high coding standards. So we have just a ton of continuous integration checks, uh, before you can merge your code. So we use Python three type annotations and we actually run.

• 46:03Speaker 03

  Mypi on the code base. And so if you submit a PR that doesn't type check or mypi, the CI system will spit it out. And we use PyLent with, uh, PyLent is sometimes the bane of my existence. But again, if you don't format your code right, PyLent's going to choke on it. And we have some code to make sure that you've actually written, you know, Sphinx documentation for every class you've added. So basically, in order to maintain this very high bar for code, what that means is that contributing to libraries is sort of a pain in the ass. Because most people who want to contribute, they might not know about mypi, they might not know about PyLent, especially if they're, you know, researchers. And so getting them over that hurdle can be pretty, uh, difficult sometimes. So that's one aspect.

• 46:46Speaker 03

  The other aspect that I think we always struggle with is when you own an open source project and, you know, it's an open source project, but it's also our project, you have to support people and you have to maintain it. So every time someone wants to contribute something to the library, someone who's not on our team, say, like on one hand, that's awesome. I love that you want to contribute to my library. The other hand, like you're putting this piece of code in the library. And now if it breaks, I'm the one who's going to have to fix it in a month. And maybe I didn't even understand it when you wrote it. And when people have questions with it, they're not going to go to you, they're going to go to me. And so it's this constant tension between, yes, I love that you want to contribute to the library. But at the same time, do I want, you know, your contribution to be something that I'm going to sign up to,

• 47:28Speaker 03

  you know, support and maintain forever, basically. And, and that's, that's another ongoing struggle that I don't think we found the right answer to exactly.

• 47:36Chris Benson?

  You know, it's funny, that last answer, I'm, I'm gonna, I had a question for you, but I'm actually going to ask you too. Because since you're working on a team that is, you know, it's more than just yourself and, and you have to juggle multiple people and their perspectives and all. What I was gonna ask you, which I'd like to address, but I'm going to tag on to is what is the team working on right now? You know, what are some of the cool things? But, but also as an individual on the team, what are you excited about in terms of, you know, the future for AI and natural language and the types of things you guys are working on? So, you know, kind of team perspective and also your own. I'm curious.

• 48:10Speaker 03

  Uh, so our team, like I said, is mostly researchers and they're doing research around things like paragraph, understanding, semantic parsing.

• 48:20Speaker 03

  Question answering, language modeling, sort of wide variety of NLP topical research kind of things. We have a handful of, like I said, we have a lot more researchers than we do engineers. What I'm working on right now is mostly, I don't know if you got a chance to check out the Alan NLP demo, but we have this demo dot alan nlp.org. We have these interactive demos for a lot of our models. And that's one of the value propositions of the library is that, you know, once you've trained a model, we make it sort of easy to get an interactive demo. And so one of the things I'm working on right now is basically, once you've trained a model, basically for free, you can get a really ugly demo, which is like text in and JSON out. How can I make it so that you can get for almost free, like a beautiful interactive demo that's like text in and widgets out and make it so that

• 49:05Speaker 03

  someone who doesn't know, you know, a ton of JavaScript or React can get it working very easily. So that's, that's kind of one of my main areas of focus. Right now, we have other people who are working on things like, how can I train more efficiently? You know, how can I speed up these models? How can I use less compute? As well as just adding various other stuff.

• 49:23Daniel Whitenack?

  Well, I for one would love to see this, uh, this demo functionality that you're, you're talking about. I would be super excited to use that. It would like make me seem way cooler than I actually am. So You're cool, buddy. I would always appreciate things like that. So to kind of wrap up, uh, first off, thank you so much for joining us. Um, as I already mentioned, like I've been a, a fan for quite some time. So it's been a real honor to, to get to chat with you a little bit and get to know you a little bit. Um, just as we wrap up, where can people find your book, first of all? And then maybe also, what's the best place where people could start learning about Alan NLP? I know there's, it's probably on GitHub, but are there other places? Do you have like a website? Are there like slot groups or anything like that?

• 50:07Daniel Whitenack?

  What's the best way to get engaged?

• 50:09Speaker 03

  Yeah. So, um, you can get my book anywhere books are sold. I, I went in the Barnes and Noble a couple of weeks ago and they had a copy there. So I took it out from, you know, where it was hiding and put it right in the front of the rack right prominently.

• 50:20Daniel Whitenack?

  Of course.

• 50:21Speaker 03

  You can get it from Amazon. Uh, O'Reilly shut down their bookstore, so you can't buy it directly from them. But if you want to buy a PDF version, apparently those are sold through something called ebooks.com, which is where you can buy PDFs of O'Reilly, but really any, any online bookstore, you should be able to get it. If you want to find out about Alan NLP, the best place is to start just at our website, alan nlp.org. That will take you to demos and tutorials.

• 50:46Speaker 03

  And so on. If you want to look at the actual code itself, that's on GitHub. Alan AI slash Alan LP. You can go there and see our enormous backlog of issues and pull requests. Laugh at us for having such a big one. Can I pitch all my other presences?

• 51:00Daniel Whitenack?

  Yes, please do.

• 51:01Speaker 03

  So I have a website. It's just my name, joelgrus.com. I blog very infrequently, but when I do, it's pretty good. I am on Twitter all the time. That's at JoelGruce and you can find me there. And it so happens that I have my own podcast with Andrew Mussumman called Adversarial Learning, which we rarely record, but sometimes we record it. And that's at adversariallearning.com.

• 51:22Daniel Whitenack?

  Awesome. Yeah. And I must say it's both on the blogs and the podcast, when they do come out, I highly recommend them. So thanks for, thanks for sharing those.

• 51:30Speaker 03

  Oh, and one more. I like to make live coding videos and those are all up on YouTube. And I think that's just youtube.com/JoelGruce. So if you want to watch me live coding frequently, problems that I haven't looked at before I started live coding them, go there and check those out.

• 51:44Daniel Whitenack?

  Cool. Awesome. Definitely will. Thanks again for joining us. Hope we can meet up at a conference sometime, but look forward to more great content online and thank you so much for the discussion.

• 51:55Speaker 03

  Yeah, thanks for having me.

• 51:59Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor. Go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do here at changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Leno cloud servers. Head to leno.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address.

• 52:42Speaker 00

  Get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.