Green AI 🌲

Empirical analysis from Roy Schwartz (Hebrew University of Jerusalem) and Jesse Dodge (AI2) suggests the AI research community has paid relatively little attention to computational efficiency. A focus on accuracy rather than efficiency increases the carbon footprint of AI research and increases research inequality. In this episode, Jesse and Roy advocate for increased research activity in Green AI (AI research that is more environmentally friendly and inclusive). They highlight success stories and help us understand the practicalities of making our workflows more efficient. Sponsors: The Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Code-ish by Heroku https://www.heroku.com/podcasts/codeish – A podcast from the team at Heroku, exploring code, technology, tools, tips, and the life of the developer. Check out episode 98 https://www.heroku.com/podcasts/codeish/98-the-ethical-side-of-deep-fakes and episode 99 https://www.heroku.com/podcasts/codeish/99-the-technical-side-of-deep-fakes for insights on the ethical and technical sides of deep fakes. Subscribe on Apple Podcasts https://itunes.apple.com/us/podcast/code-ish/id1457879893 and Spotify https://open.spotify.com/show/5VvkHRhd9E9SQf7i9HsPMd. Knowable https://www.knowable.fyi – Learn from the world’s best minds, anytime, anywhere, and at your own pace through audio. Get unlimited access to every Knowable audio course right now. Click here https://www.knowable.fyi to check it out and use code CHANGELOG for 20% off! Featuring: Roy Schwartz – Website https://schwartz-lab-huji.github.io, X https://x.com/royschwartzNLPJesse Dodge – X https://x.com/jessedodgeChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Green AI article in the communications of the ACM https://cacm.acm.org/magazines/2020/12/248800-green-ai/fulltextTraining a single AI model can emit as much carbon as five cars in their lifetimes https://www.technologyreview.com/2019/06/06/239031/training-a-single-ai-model-can-emit-as-much-carbon-as-five-cars-in-their-lifetimes/Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping https://arxiv.org/abs/2002.06305Parameter-Efficient Transfer Learning for NLP https://arxiv.org/pdf/1902.00751.pdfReproducibility at EMNLP 2020 https://2020.emnlp.org/blog/2020-05-20-reproducibilityUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We're researchers. We mostly communicate with the research community, but I mean, there's stuff to be done everywhere. Thinking about, you know, thinking about efficiency, you don't have to persuade anybody that if all other things being equal, if your tool runs twice as fast or takes half the amount of memory, everybody wins.

• 0:17Speaker 01

  Bama for change log is provided by Fastly. Learn more at fastly.com. Our feature flags are powered by Launch Darkly. Check them out at launchdarkly.com. And we're hosted on Leno cloud servers. Get $100 in hosting credit at leno.com/Changelog.

• 0:33Speaker 02

  We deserve a better internet and the brave team has the recipe for bringing it to us. Start with Google Chrome, keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits. We don't need them. Mix in ad and tracker blocking by default. Quick access to the Tor network for true private browsing and an opt-in reward system. So you can get paid to view privacy respecting ads. Then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give tipping a try on changelog.com.

• 1:14Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We're at Practical AI FM.

• 1:40Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist at S I L International and I'm joined as always by my co-host, Chris Benson, who is a principal emerging technology strategist at Lockheed Martin. How are you doing, Chris? I am doing very well. How's it going, Daniel? It's going great. It's warmer now in the US. A lot of people have been having some, some issues, particularly down in Texas and other, other areas. So this is for those listening later in the pod. This is February of 2021, a lot of uh, snow and cold weather in the US here, so.

• 2:17Chris Benson?

  A couple of people in our team at work are in Texas and we've been getting all of the, all the stories when they're able to connect and stuff. So I think,

• 2:25Daniel Whitenack?

  Yeah.

• 2:25Chris Benson?

  I think they're, they're getting through it finally. Thank goodness. It was pretty, pretty horrible, but

• 2:30Daniel Whitenack?

  Yeah.

• 2:30Chris Benson?

  In the meantime, I am enjoying my 70 something, you know, my 70 degree plus weather outside, spring like, and I'm kind of sticking my tongue out on them on Zoom meetings.

• 2:40Daniel Whitenack?

  Yeah, it's always interesting during these particular types of events because you kind of just assume that people have all of this like redundant fault tolerant, like infrastructure going on for like their APIs and other things. And these sorts of events really reveal like that is not the case. Like I know, like one of the APIs we, we frequently use is like, apparently on a on-prem server in Dallas and they did not have power and you know, you, you learn new and interesting things like that.

• 3:11Chris Benson?

  You know what? After the past year, there's nothing that surprises me anymore. Not now. Global pandemics, all sorts of strife. You name it. I mean, I'm just, yeah.

• 3:22Daniel Whitenack?

  Yeah.

• 3:22Chris Benson?

  Nothing, nothing phases me now.

• 3:24Daniel Whitenack?

  Yeah, I'm glad to hear you've, you've built a lot of robustness into your personal life there.

• 3:29Chris Benson?

  There we go. I laugh a lot. I snicker a lot. That's how I cope.

• 3:33Daniel Whitenack?

  Yeah. Well, a few months ago, actually, I think it was one of the researchers at S I L that I work with called Gary Simons. He's been a linguist and programmer, computational linguist, translator type researcher for decades. And um, he sent me this link and our Skype communication said, hey, this is a really cool article. You should think about having this on your podcast. And there's an article called Green AI from Communications of the ACM. And I'm really happy today because we get to make, materialize what uh, what Gary saw and what he recommended to me. And we've got uh, Roy Schwartz and uh, Jesse Dodge with us.

• 4:17Daniel Whitenack?

  Roy is a senior lecturer at Hebrew University of Jerusalem and Jesse Dodge is a postdoc at the Allen Institute for AI. And they were both authors on that article. Welcome, guys. Thanks for having us.

• 4:28Speaker 05

  Thank you.

• 4:29Daniel Whitenack?

  Yeah. If both of you could just uh, give us a little bit of a background about yourselves, uh, that'd be great. Why don't we uh, start with Jesse? Sure. So.

• 4:38Speaker 05

  I finished my PhD from Carnegie Mellon in the Language Technologies Institute last year in 2020 in the pandemic. Although I spent most of my PhD at the University of Washington in Seattle. And part of that time I spent working at the Allen Institute for AI, where after I graduated, now I'm back as a postdoc full-time. So we wrote this article, I think, back in, we were thinking about this for quite a while and then wrote this back in 2019 and really got it out in 2020. So yeah, now here, even though the offices are closed,

• 5:21Speaker 05

  I'm still here in Seattle and I am on the Allen NLP team once again.

• 5:26Daniel Whitenack?

  Awesome. And what are you specifically working on?

• 5:29Speaker 05

  So my research sort of falls under two broad umbrellas. The first is related to efficiency, similar to this green AI idea that we'll get into. I work on making models more efficient along the number of dimensions that they have in terms of the complexity, in terms of inference, generally related to any way that you can measure like the total computational cost of getting some kind of experimental result. And then the second pillar of my research relates to reproducibility, where I created the natural language processing reproducibility checklist that was used at, I think, four major NLP conferences now.

• 6:13Speaker 05

  And I've published some work on how we can make the science of machine learning and natural language processing more reproducible.

• 6:21Daniel Whitenack?

  Yeah, that's awesome. Well, you're working on two things that are just like desperately needed in terms of focus. So yeah, I commend you in terms of that. And yeah, it's really great, great to hear. Roy, what about yourself?

• 6:35Speaker 00

  Hi, so I'm Roy Schwartz. I'm a senior lecturer, which is an equivalent to assistant professor at the Hebrew University of Jerusalem. I'm currently in Jerusalem. I joined the Hebrew University last summer and before that I spent four years. In Seattle, where I got to meet Jesse, fortunately. And I was a postdoc and then a research scientist at the University of Washington and the Allen Institute for AI. And these were four wonderful years, but now I'm back home. Similar to Jesse to some extent, I also came from the university where I did my PhD and so and took a break and came back. My research also spans two or maybe three dimensions.

• 7:17Speaker 00

  One of them is similar to Jesse, efficiency and trying to think about ways to reduce the cost of AI and NLP in particular. And the other is trying to get better understanding of this technology. Now that we have models that are becoming so big and so good at what they're doing, but at the same time, it's very hard to know why are they doing certain things? Why think some things work and some don't? Why do models reach certain decisions? I'm particularly interested in the role of data in all of this. How do our data sets look? What do they contain? What kind of phenomena are encoded in them? And I like to make connections between all of these goals, between understanding and between understanding our data and between making things more efficient.

• 8:02Speaker 00

  And these are some of the things that I'm most excited about.

• 8:04Daniel Whitenack?

  Awesome. Before we move on, what is your general impression about sort of progress in this process of trying to make our models more interpretable and understand more about them? Obviously, you're doing work in the field, so hopefully like you see progress in that. But as an industry as a whole, where do you think we are on that journey?

• 8:25Speaker 00

  That's a great question. So as you said, on one end, we're making tons of progress. I mean, lots of very smart people are working towards developing methods to probe models to kind of kind of poke them and ask them, I mean, do you know syntax? Do you know world knowledge? Do you know this and you know that? And we're developing methods that are more and more sophisticated to get this information. At the same time, the core questions that I think will make a huge impact if we're able to solve them. And I'm not sure if these questions are even solvable to some extent, and I'm happy to talk about it, even though it's not the topic of today's talk is can, how do we get metal models to explain what they're doing? To explain it in a reliable way, in a way that's, I mean,

• 9:08Speaker 00

  I'll just say one thing, I mean, when you ask a person, why did they do something like that that they did? The explanations are often also not, I mean, they might be. Post rationale of things that it's hard even for us to say what we're why we're doing certain things and we're conscious creatures. So machines are it's much harder to get this, but we're trying.

• 9:30Chris Benson?

  I appreciate that. As we as we look at this, we're talking, I'm looking at your green AI article here again. And I'm just kind of curious, you know, what was your motivation for putting this out? And probably I should ask as part of that, you know, what is green AI? You know, initially, and how did you decide that this was the thing that you needed to get out there to the world? And because it's this is a topic that often gets left out of AI ethics and such, having worked in that field for a while. We can go back to that in a little bit. I'm curious what your motivation was, though.

• 10:04Speaker 05

  Yeah, so I think part of it was some conversations that Roy and I had. Again, this was back in 2019, when we were both at the Allen Institute for AI. And we noticed that there was this increasing trend of larger and larger computational budgets used for some of the research papers that were published in NLP. We looked around and found not only did we notice this, but there were a couple other pieces of work that had also noticed this trend. So back, you know, when I started my PhD back in 2013, I could run my experiments often on a used laptop that I had purchased off of Amazon. And it was kind of slow, but

• 10:45Speaker 05

  I would, you know, I could run most of my I could train my models in a few minutes or an hour, maybe. And it worked. And that was okay. And then we noticed in, you know, in 2019, we were like, wow, a lot of these models don't even fit on a single GPU. And we have to like rent like cloud instances to be able to actually use some of these models. Plus, in some cases, Papers would do, for example, a tremendous amount of hyperparameter optimization, or they would train on a huge amount of data, well beyond what we could do even at, you know, a good institution like, like the University of Washington or AI2. And one interesting thing, and this has really been followed up by some concrete

• 11:29Speaker 05

  research, is that we do find significant improvements in performance across a lot of tasks just by scaling up these models. So, language modeling, for example, has been a pretty foundational task in NLP. What we found is that training models to do well at this task of language modeling, if you train a large enough model on enough language data, then that model can do some other tasks that we're interested in as well. So it somehow learns some kind of representation of language that's useful across a wide variety of tasks, but to get there, we saw just huge computational budgets used for a number

• 12:13Speaker 05

  of, of these papers. And interestingly, I mean, we wrote this a while ago, but the trend has not slowed down. So this is something, like Roy and I are still working on similar motivated pieces about how this is really driving a lot of research in our field. Like these, these massive scaling laws, for example, are pushing state of the art and also getting a lot of attention and having, you know, our, our field is interesting. You can view our field through that lens now and see some interesting results.

• 12:45Daniel Whitenack?

  Yeah, so I'm curious, I have my own thoughts about how I might answer this question, but I also haven't done the amount of thinking that, that both of you have. So I don't know, maybe Roy, if you want to comment on this or kick it back to Jesse. So that trend has been continuing and like we're seeing those sort of improved results in some areas along that trend, like in language modeling. So why is that a problem? Or like, what sorts of problems or red flags does that bring up, I guess?

• 13:18Speaker 00

  Yeah, I mean, I think there it's interesting because Jesse and I bring complementary motivations for tackling this problem. So, I mean, when I started thinking about these things, I mean, yes, I was having discussions with Jesse about this, but I'm a person that cares about the environment and I try to make personal choices that, you know, I ride my bike to work because it's healthy, but also because it allows me to not drive my car. And I try to, you know, turn the light off when I leave the room, you know, do these things that don't, you know, don't matter much at the global scale, but you know, I make my personal choices. And then I go to my office and I, I don't know if you've ever seen a GPU, but this is a very loud machine, very, a machine that emits a lot of heat.

• 14:00Daniel Whitenack?

  Hot.

• 14:01Speaker 00

  Yeah, and kind of we're running stuff, you know, like, okay, let's just push a button and suddenly, you know, the five degrees or 10 degrees up in your room, maybe, but not in your planet, hopefully. And kind of, it's been something that I've been thinking about quite a bit. I mean, what's the total impact of our field? And Jesse and I have been talking about this. And then I think in mid 2019 or early mid 2019, a paper came out from the University of Massachusetts led by Emma Struble and her colleagues that tried to quantify the CO2 impact of large scale NLP experiments. And she came to the, she and her colleagues came to the conclusions that one of the most expensive experiments that run the train a model in a process called neural architecture

• 14:46Speaker 00

  search, which basically means we're going to train a bunch of models and select the best one. But when I say a bunch, I'm talking about thousands or tens of thousands of experiments. And she computed, using some rough estimations, to be said, that the amount of CO2 emitted by this process is equivalent to the amount of the lifetime omission of five cars or several flights, or I don't, I don't remember the full details, but I mean, something that's

• 15:17Daniel Whitenack?

  I think it was five car. I remember this coming out and I was also shocked.

• 15:21Speaker 00

  Yeah.

• 15:22Chris Benson?

  Daniel and I actually talked about this in an episode way back when that came out. I remember us just commenting on it.

• 15:27Speaker 00

  Yeah, everybody was talking about it and really hit me in a place that I this is something that I was thinking about and I was really happy to. I mean, I was sad to see that that my intuitions were right in some sense. I was kind of hoping that maybe it's, you know, it's not that bad, but and and kind of then Jesse and I were having discussions along with other people at AI two. And kind of we were saying that, you know, this is something we need to do something or, you know, to make the community more aware of it. And we were thinking about, I mean, AI two is an institution that our goal is, I mean, I'm not, I'm no longer working there. But at the time I was working there to do AI for the common good. And, you know, this feels like a natural fit for the goals of the organizations.

• 16:07Speaker 00

  And we got um organization, the CEO and Noah Smith, who was my manager and Jesse's advisor at the time. Um, on board and, you know, we wrote this piece and just hoping to get people, um, you know, thinking about this, you know, not necessarily thinking about this in terms of. Uh, finding more accurate ways to quantify how much energy and is omitted. And how much are the costs of these experiments? And trying to encourage the community to work on more efficient solutions that would allow us to reduce these costs.

• 16:43Speaker 05

  Yeah, that's one way that Roy and I think like one thing that Roy just mentioned. Um, is that we brought different perspectives to this. I completely agree with everything that Roy just said, like, that's super motivational. I think that's, you know, very important going forward that we keep track of CO2 estimates and we do a great job at that. There's another um side to this also, which we write about in our green AI paper. Uh, where we talk about the the sort of research inequality or inequality in the research community. Where some of these experiments really could only be done by sort of the 1% of the research community. Those that have access to tremendous numbers of GPUs or just like lots of machines.

• 17:25Speaker 05

  So one question that we address in our paper is, is this valuable research that we should treat on the same level? As like other types of research that can be done primarily motivated by just a good idea rather than. Really expensive experiments. And so both of these are sort of negative consequences of this increasing trend that we observed. And one interesting thing, I think this is an interesting thing, sort of back in 2019, going back to that Strubel et al. Paper, I found that through a number of conversations that I had had, and also just like the general information I saw online, when before Emma and her colleagues wrote

• 18:10Speaker 05

  that paper estimating the CO2 emissions, there was an understanding of like how some work was very expensive, how some work was quote boiling the ocean, for example, just to get a 1% improvement or half a percent improvement on some task. And so when Emma wrote that paper, I was surprised. But again, I mean, I felt similarly to Roy. I was surprised. I wish I hadn't been, you know, surprised by the results that I saw. I wish they had claimed that people were emitting less CO2. But it really did capture like her paper and then our paper as well. I think these got so much traction, partly because we were outlining a trend that other people had also noticed.

• 18:52Speaker 05

  And yeah, like I said, that trend really does, I think we focus on two facets, there are probably others, but the CO2 emissions and also the sort of this research inequality are both direct consequences of that, that increasing trend.

• 19:16Speaker 01

  Hey friends, this episode of Practical AI is brought to you by Codish, a podcast from the team at Heroku that explores code, technology, tools, tips, and developer life. There's tons of great conversations on the Codish podcast, so I would encourage you to check it out and subscribe. But in particular, I wanted to bring to your attention two episodes, episode 98 and 99, where Julian Dukey explores the ethical and technical sides of deep fakes. The rise of manipulated pictures and videos and other forms of computer generated media are able to cause uncertainty and doubt in what we see in here online. So how are we able to use these tools for good, if at all? Here's a sneak peek.

• 19:55Speaker 06

  Let's say we want to do a deep fake of my voice and we train the model and we have enough data and everything. This will be also able to imitate my accent, for example, like how I pronounce English and the strong pieces of my accent? Or is not there yet?

• 20:17Speaker 07

  It really depends. If there would be a person with similar accent on the input, then it would be fine, but it's kind of cheating. You can think it's cheating because we're reusing accent of a different person that's similar to your accent. But if it would be like an American native speaker or a British, a person with a British accent, or like whatever other accent, then it will kind of be a mixture on the output. So we're not there yet in terms of converting accents. It's a little bit more difficult than we initially anticipated because like when we started the company, we thought it would be, you know, we'll kind of solve it in a year or something.

• 20:58Speaker 07

  But then it turned out that, whoa, no, we're here for much longer.

• 21:04Speaker 01

  Check these episodes out. Links are in the show notes to both episodes or head to heroku.com/podcasts to listen and subscribe. Again, check the show notes for links or go to heroku.com/podcasts.

• 21:32Chris Benson?

  So you brought up something that really kind of got my brain really going there for a minute. And I was thinking about the fact that, you know, this really can matter a lot, even if not a lot of practitioner, you know, the number of practitioners in AI relative to all the people producing CO2 is quite small. But you mentioned going through all these models. And when we're doing things like hyper parameter optimization and trying, you know, little adjustments to architectures all the way through. And then one practitioner doing work is essentially, you know, being thousands of practitioners on a per model basis as they're trying to hone in on that. It really amplified. amplifies the impact of what can happen.

• 22:16Chris Benson?

  I mean, so I guess, you know, it's not, it's less of a problem that a very few people are doing and more of a problem that, that because that amplification is, is quite outsized relative to the number of people doing it. Am I getting that right? Am I understanding the problem in the way that you're thinking about it? Or, or am I missing something there?

• 22:36Speaker 00

  So I'm not 100% sure that I understood you. So let me try to

• 22:40Chris Benson?

  Sure.

• 22:41Speaker 00

  to, to say where, where I think this is going. So, I mean, also I'm assuming you're talking about the environmental

• 22:48Chris Benson?

  Yeah, environmental impact.

• 22:50Speaker 00

  Because the inequality aspect, I think, is pretty clear that, I mean, a very small proportion of the community can afford to run these experiments.

• 23:00Speaker 00

  And kind of when we're thinking about the environmental effect, then some people argue, and I'm not sure I disagree even, that it's not so bad because these experiments are being run just a handful of time. And I might agree on that, I must say. There are different ways in which the AI community is contributing so and so, quote unquote, to the emission of CO2 to the atmosphere. And probably the one that's easiest to measure is the most expensive experiments. That's perhaps one dimension. You can also think about, I mean, the entire, the amount of training being done by the entire community. And probably most influential in this sense is the, the cost of inference, of cost of taking a model that's been trained and running it.

• 23:46Speaker 00

  And this is, one operation is very cheap, especially obviously compared to training a model. But this is something that happens at scale. If you think about, I don't know, the amount of Google search queries that are being run per day, or the translation, or the number of videos being added, or recommendations in, in various websites. So there's different dimensions to these problems. And I think what we're trying to promote is not so necessarily to say, uh, look, we're boiling the ocean, as Jesse said, quote unquote, but I mean, we don't know exactly what is it that we're doing. And let's be more honest about it. Let's do a better job at reporting and let's try to reduce these costs.

• 24:26Speaker 00

  I mean, and I mean, it's hard to argue against, uh, I mean, who doesn't want cheaper models, right? It's obviously that, uh, other things are, uh, and you know, if if cheaper models perform slightly worse and maybe this slightly worse translates to slightly less revenue, then maybe cheaper is that there are different ways to define cheap. So I think what we're trying to promote is to get more people thinking about it and not just improving another epsilon on the accuracy level.

• 24:56Daniel Whitenack?

  Yeah, that's super helpful. I think, you know, one of the things that's running through my mind is, um,

• 25:03Daniel Whitenack?

  I guess, like, talking about, you know, what are the other options? What does it mean to do green AI? And I have this parallel in my mind. So I come from a physics background and like, if you're in high energy physics now, like, there's just been a progression of larger and larger particle accelerators, right? And now if you want to do high energy physics, you're going to spend some time at CERN um, in Switzerland or whatever, just because no one has another CERN, right? Like they're, they're just not there. So like, is there another option for, and I'm thinking particularly Jesse of what you were highlighting in terms of the research inequality. I think that's a really great point. Like, what can we do in terms of reducing that inequality?

• 25:48Daniel Whitenack?

  And is there something more that we can say other than like tough luck? Go work at Google or somewhere that has these like amazing resource, you know, in seemingly endless resources to do these massive experiments?

• 26:00Speaker 05

  Yeah, so that's a great question. I think this is something that comes up a lot is sort of the relationship. When we talk about green AI, sometimes somebody will say to us, oh, but in biology, it costs so much to do any experiment because you need a wet lab and because you need, you know, some equipment and you just can't do it without that equipment. So is it bad that some experiments in our field are expensive? And I think the answer here is really that in the computational sciences and in machine learning and NLP in particular, we really can. There are a few things that we can do that make future comparisons against our work with smaller budgets easier.

• 26:38Speaker 05

  So one example of that might be, sure, I train a model on all of the language data on the entire internet, right? But I can also evaluate that same model after training on only a fraction of that data. And if I do this, let's say I train and evaluate. Evaluation in this case is typically pretty inexpensive. So your evaluation set, your data set that you evaluate on is often much smaller than it's like, you know, a tenth or even smaller of your training size. So one thing that we can do is just checkpoint our model or evaluate it regularly throughout training.

• 27:19Speaker 05

  And then a future researcher will be able to come up with a new idea. Let's say they have a new model that they want to evaluate. And they can compare against some of those sort of smaller budget evaluations. So for us, the point here is that in our field, we really do have a few ways that we can sort of build in these sort of low budget comparison opportunities. And that enables not just future comparisons, but that really drives the sort of competitive nature of our field. Where instead of trying to improve just the absolute best found performance, somebody could try to find a better performance efficiency trade off.

• 28:02Speaker 05

  Where at a low budget, their new idea, a low budget for, you know, the number of parameters in your model or the total number of experiments of hyper parameter tuning or the amount of training data you use along any of those dimensions, somebody else might come along and try to compare against your work specifically in those sort of low budget regimes. And so I think here, that's a key difference between our field and, you know, physics, like you mentioned, or we often hear biology. And really, if you think about it, if you're training a model and it costs you, say, a million dollars to train on all of the internet, spending an extra $10,000 on just evaluating that model, spending an extra, you know, tenth of 1% or some small fraction of your total budget,

• 28:48Speaker 05

  so that other people in the future, they can Have an opportunity, they've got that hook to compare against. That is what one way that we can help drive the overall cost down by promoting that kind of competition.

• 29:03Speaker 00

  Yeah, I totally agree with what Jesse said. I think presenting another angle of this. So currently, there are certain norms in our community and kind of, I mean, there are certain ways of, I mean, topics of research that gets, you know, more visibility and more credit from the community while others aren't. And I don't want to say the naive assumption is, you know, go work at Google, as you said. But I mean, the fact is that when we were thinking about this paper a couple of years back, we were doing a short survey of papers in ACL, that's the top venue for our field, and in other similar venues in other fields of AI. And we found we had a very hard time finding papers that focused on efficiency.

• 29:48Speaker 00

  Most of the papers we were looking at were trying to say, okay, we did this and this and that, and we got some better improvement here and this and that. And we got some, you know, tenth of a percent better on some accuracy or answering questions a tenth of a percent better or translating a fraction of a percent better there. And what we're trying to argue that this is not a good balance. We want to see, it's good that people are working to make our models more accurate. We're not arguing that this is not important. And similarly, we're not arguing that the big models aren't important. They're making huge contributions to our field. But we think that a larger chunk of the research effort should go towards trying to find solutions that are not epsilon better,

• 30:31Speaker 00

  but are twice as fast or take 10% of the memory or what have you. And we're trying to work with the research community by providing ways to publish this work. For instance, we've established tracks. And tracks are kind of like, you can think of it as topics in major conferences where when we were working on some of our work that tried to promote efficiency or presented an efficient solution, as I said, that works five Times faster but doesn't improve the performance. We had a hard time deciding where to send this paper to and where we would get the best audience to appreciate it.

• 31:14Speaker 00

  And what we were able to do in the past year is to set up a green NLP track or an efficient NLP track in our conferences that allow works that focus on that to get published and to get the visibility that they deserve.

• 31:30Speaker 05

  Yeah, that's great. And I think another thing to build on what Roy just said is our community, like I think one's strength of the research community is really that it's just a collection of individuals all trying to do the best work that they can. There is no overall governing body. So when we think about like, how can we get our community to focus on more efficient approaches? It's kind of tricky, you know? We can't, it's just not possible for us to say like, some fraction of the work should cover this, this topic. So instead, we thought a lot about the types of incentive structures that impact people in our field. And creating this track, as Roy just mentioned, is one of the ways that we can promote this

• 32:15Speaker 05

  and provide an opportunity, sort of lowering barriers for publishing work on work that promotes efficiency.

• 32:37Speaker 02

  Have you heard about Knowable? It is an awesome new platform for learning from the world's best minds, anytime, anywhere, at your own pace, through audio. Learn about the performance benefits of a plant-based lifestyle from NBA All-Star Chris Paul, or how to launch a startup from Reddit co-founder Alexis Ohanian. There's even a 10 lesson course from astronaut Scott Kelly. Here's a sneak peek.

• 33:02Speaker 08

  We learned a lot up there, but what can you learn from a life in space? The answers might surprise you. In this Knowable course, I want to share some of the things I've learned that you might not expect. Lessons about leadership on a dark night on an aircraft carrier in the middle of a churning sea. Lessons about the fear you feel with 7 million pounds of thrust exploding underneath you. And most of all, there's an idea out there that astronauts are always perfect. Failure's not an option, right? That's why I want to take you through some of my life experiences to show you how that's just not true. I believe everyday, regular, human failure, if we handle it right, can be one of our

• 33:45Speaker 08

  greatest opportunities to learn, grow, and succeed.

• 33:49Speaker 02

  Knowable is accessible on your phone and on the web, and each audio course is broken out into individual lessons, usually around 15 minutes long. As a Change Log listener, you can get an annual membership to Knowable for 20% off. Get unlimited access to every Knowable audio course right now. Just download the Knowable app or visit knowable.fyi and use code CHANGELOG for that 20% discount. We put a link in your show notes for easy click-ins. Check out Knowable today and start learning from hundreds of top experts from around the world. Once again, that's knowable.fyi, code CHANGELOG.

• 34:34Chris Benson?

  So, this is really interesting to me, and as I'm listening to you, I'm trying to think how I'm going to implement. So, can you kind of describe some of the good examples of how Green AI has been implemented before and any kind of guidance? So, if I'm a practitioner, you know, you've hit on some of the practices, but either go through someone else's example or something that you've described to people, because I'm just trying to really make it where when I walk out of here, I want to be able to go ahead and implement that.

• 35:09Speaker 05

  Yeah, so I guess I can, I'll talk a little bit about this. So one thing that I mentioned already was performance efficiency trade offs. And I think that the key idea here and one thing that we found when we did this survey that Roy mentioned of papers in our field is that most, most papers just don't report anything. They don't report any efficiency related metrics at all. Most papers in our field invent some new model or some new, you know, loss function, some new training scheme, something like that. And then claim in a table, here is our better performance, we beat our baselines.

• 35:48Speaker 05

  But they don't report, for example, training curves or, you know, some other measure where you can trade off efficiency and performance. Maybe accuracy could be one measure of performance. So an example of this, and I guess the first thing that I would say here is what we hope everyone in the research community starts to do, and we are seeing this happen now, is just report something. Report some measure of how maybe it's going to be the floating point operations to run your model. Maybe it's going to be a training curve. Maybe it's going to be the results from your hyperparameter optimization search, right?

• 36:29Speaker 05

  So one example of this I can point to is a paper, and I think this is, I use this as a positive example of how somebody can report this kind of information. So Roy and I wrote a paper on that used early stopping. So we partway processed an example and then potentially had our model stop early. So instead of feeding the example all the way through our model and then coming up with a prediction at the end, we had ways for our model to stop this computation early and make a decision quickly. And this method allowed us to show performance efficiency trade offs, these smooth curves, which anyone can then compare against at any point.

• 37:14Speaker 05

  And what I would hope to see is other work come along and show a better curve rather than just a single point on this performance efficiency trade off. They can report just, here's how efficient my model was and here's the performance. Potentially beating our entire curve or just a single point, you know, better along one of those dimensions. In this way, like just reporting more information allows others to compete along either of those dimensions or potentially draw a better curve.

• 37:43Daniel Whitenack?

  So I'm curious, I think a lot of what we've talked about has been focused on like, what are ways in which we can still explore this regime of like large models, but potentially be responsible about how we're reporting the cost of it and/or how we're allowing others to build on top of what we're building. I'm wondering how maybe another side of this fits into this whole discussion, which is just plain smaller and more and/or more efficient or different models. So I'm thinking of things like recently I was playing around with like QuartzNet, which is this end-to-end speech recognition model

• 38:26Daniel Whitenack?

  from Nvidia, which is very compact based on these like 1D time-separable convolutions. And it's like, like the whole model like on disk is like 90 megabytes or something like that. And like, it shows like really good performance, almost like comparable or comparable to like these really large speech recognition models. So I'm curious, maybe that also has some advantages in terms of like some of the interpretability things, Roy, that you're interested in. Where do you see that this whole regime of new and different, more efficient models fitting into this? And do you see momentum in that area or good examples in that area as well?

• 39:08Speaker 00

  Yeah, I think there's been a lot of, I mean, I think the thing that I said a few minutes ago about seeing that we saw very little work that focuses on efficiency. I think in the last couple of years, there's been more and more work that focuses on that. And we're delighted to see that. It probably has nothing to do with us. It's probably something that would have happened anyway. And I think that the main ideas that are being mostly explored are ways to make inference more efficient. And this makes sense at At least in the environmental aspect, but also, you know, just in terms of you want to put a speech recognition or an image processing or a text processing machine on your phone. And then you need for it to be, you know, small in terms of the number of parameters

• 39:53Speaker 00

  or the amount of space it requires or, you know, it doesn't require much energy. So it doesn't drain your battery and so on. So there has been a lot of effort along these dimensions. And I think the main governing technology there is to train a big model, you know, train it as big as you can, and then train another model to imitate this model to some extent or to take the large model and get the same performance using fewer resources. There are different techniques of doing that, but that's probably the most common thing that we've seen. What I think is very interesting and people aren't putting that much effort into is to make the other parts of the process more efficient, namely training and what we call model selection, basically hyperparameter tuning

• 40:39Speaker 00

  or other ways of selecting your best model. And I think this is the exciting direction that relates to the motivation. I mean, it's not like, I mean, it's not like there's Jesse thing and my thing. It's we're both excited about both of these motivations. And I think this is a really one way to improve the ability of the entire community to conduct cutting edge experiments by reducing the cost of these processes.

• 41:06Daniel Whitenack?

  So in those other parts of our process that you're talking about, I can just imagine like there have been times, and I will totally confess to this, that like whether it be hyperparameter tuning or like model selection or something, like the logically the easiest way to go about that sometimes is just to say, oh, well, I can have this run for like a week and a half and like go through all these things. You know, there may be a more. Like a smarter or better efficient way to to find the right zone that I should be in. But I can just like get this running and like come back to it in a, you know, week and a half or or whatever. Do you also find that to be a sort of like a thing that you're talking to people about and a thing that you're running into?

• 41:51Daniel Whitenack?

  Is that just sort of like, I don't want to call people lazy.

• 41:54Chris Benson?

  We're kind of spoiled in that way. That's what I was thinking, actually.

• 41:58Daniel Whitenack?

  You know, programmers and researchers are often lazy. They have like a machine.

• 42:04Chris Benson?

  Let's just run it for a while. Yeah.

• 42:06Speaker 05

  I think, I mean, the thing is, this is super common. Like, there absolutely is a trade off between how much time you put in as an engineer or as a researcher. As any kind of practitioner, there's definitely a trade off. You could really carefully narrow down your hyperparameter ranges and then spend less in GPU hours to find some good optimum. Or you could just set it up to be a super broad search, let it run for a week and it'll, you know, it'll take you personally like two days less time to run those experiments of your own hours, right? This is, the thing is, this happens, everyone does this.

• 42:46Speaker 05

  There is some way to often reduce the amount of time that you have to manually engineer something. And, you know, another way this can happen is you'll think of some algorithm to, say, implement to do inference on your model. And then later you'll be like, oh, you know what? I could make that faster by maybe 5% if I spent a full working day rewriting all of that code. Sometimes, like, it's just not worth it. The key idea, I think, behind our Green AI paper is that this happens all the time with people, and often we just don't report that. So one analogy that I use is, is that we, in our field, we don't keep lab notebooks. We just don't record a lot of the experiments that we run.

• 43:32Speaker 05

  And we treat those as like negative experiments, experiments that don't show what we're looking for. And then we only report the positive experiments at the end, right? So we just report the single best performance that we found. But with our Green AI paper, what we argue is that we should be reporting Even if it's not always like the most optimized, the most efficient approach, the best thing that we can do right now is just report something.

• 43:57Chris Benson?

  You know, it's a really good point there. And I wanted to ask, Roy, I wanted to bring you back into it for a moment. The, one of the things that you say in your paper is you say, finally, we note that the, the trend of releasing pre-trained models publicly is a green success, and we'd like to encourage organizations to continue to release their models in order to save others the cost of retraining them. So, you know, how far can you really get with pre-trained models? Do you feel that that will, will do that? And is that kind of the way we should get people to start thinking about it? Because it seems like there's certainly a, a, a training component here in terms of driving people down the right path.

• 44:36Speaker 00

  Yeah, that's a great point. I mean, so again, this, we, we struggled a lot in the paper when we were writing it, how to not, you know, I mean, what we call red AI, kind of, there is a kind of the negative connotation there. But, I mean, basically, I think there, there's tons of value in these large pre-trained models. And definitely, I mean, once you release them, other people can train models much more efficiently. Because if you build of models like, I don't know if their names are BERT or BERT, I mean, I mean, lots to many of the listeners, but I mean, these are typical models that are pre-trained. I mean, some company in this case, Google or Facebook, put a lot of effort into training them.

• 45:20Speaker 00

  And now they release them and other people can take them and use them for their own task. And the result will be much cheaper than if people train their own model from scratch. So this is definitely something that we encourage companies to do. A company, I say companies because the companies are basically the only entities that are, can afford to, to do this. And again, kind of what our point is that these organizations shouldn't stop training these huge models, but we should be thinking about the negative consequences. And one way to mitigate the negative consequences is to make these models public. Again, to reduce the overall costs for everyone to run their experiments.

• 45:58Daniel Whitenack?

  Yeah, so that has a huge benefit for those that are able to use those pre-trained models and utilize model hubs and that sort of thing. But of course, there's like this element of companies where, of course, they're driven by money. Companies make money and they often want to keep their models proprietary or something like that. But I think also, like, some of the things you highlighted earlier is that there, you know, in terms of like, commercial benefit and cost savings, there's also a cost-saving element to being able to utilize something that's pre-trained and maybe fine-tune it. And that's a huge saving in labor, right?

• 46:40Daniel Whitenack?

  But also, in utilizing these more efficient or smaller models, like maybe for inferencing, like, you get less latency, you have less computational costs, all of those things. Do you think there is that sort of commercial or cost-based argument to be made to companies?

• 47:00Speaker 05

  I think so. There's one thing that we saw recently, there was a citation, I think it was from Nvidia, that claimed about 90% of the cloud cost for machine learning was for inference, and only 10 to 20% was for training. So if you can spend a bit extra during the training phase, but end up with a model that's a bit more computationally efficient for inference, then potentially that could lead to savings in terms of like, the amount of dollars spent renting instances in the cloud or GPU hours for inference, for example. I think that a lot of our focus has been on the research community.

• 47:41Speaker 05

  So you asked a question about like, are companies motivated to keep their pre-trained models proprietary? While that's true to some extent, my guess is, it's hard to know. It's hard for me to know if a company has done that. It's definitely possible. It's almost surely happened that some company has spent a lot of money training a model and then hasn't released it because it's part of their business. At the same time, what we do know about is the research community, and this has grown... Exponentially, like not just the size of our experiments has grown dramatically in recent years, but the the number of people in our field and also the number of papers that are written and the size of our conferences. So across, you know, we are already seeing

• 48:25Speaker 05

  such a tremendous growth there. I think it's very worth it to focus on helping save computational costs across inference, training, what have you. Yeah.

• 48:34Chris Benson?

  Yeah, I guess I think I'm the only person on the call working at a for-profit, you know, commercial entity. And certainly, there are times when we aren't releasing that, you know, the way you would in the research community. So maybe I'm kind of curious, you know, would it make sense for us to, you know, you still have a group of people working in the organization that want to do the right thing always, you know, so they're, you know, they're no different in that way. So maybe still having internal targets for efficiency, kind of like what you talked about earlier, and those internal metrics, so that even if you aren't publishing them, publishing them for competitive reasons, or whatever it may be, that you're, you have

• 49:14Chris Benson?

  a set of metrics that you're trying to achieve. And that might be, that might be something they could spread through the commercial space, even when they're not willing to do a full release. Does that sound like a reasonable, you know, plan, you know, for those of us who who do want to strive toward that, but maybe don't have the freedom to just release?

• 49:33Speaker 00

  Yeah, definitely. I mean, people have reached out to us from for-profit companies and with similar stories to what you're telling. And they want to, you know, they work in a for-profit company, so they're limited in what they can do, but they want to promote this, they they sympathize with the motivation, and they want to do the right thing within the scope of their, you know, what they can do inside a company.

• 49:57Chris Benson?

  Within commercial constraints. Yeah.

• 49:58Speaker 00

  Exactly.

• 49:59Chris Benson?

  Yeah, I get it.

• 49:59Speaker 00

  So, yeah, I mean, you know, as Jesse said, I mean, most of, we're researchers, we're not part of, I mean, any company is different, I guess, with its own set of norms and rules. But and we mostly communicate with the research community. But I mean, you know, there's there's stuff to be done everywhere, you know, thinking about, you know, thinking about efficiency, you don't have to persuade anybody that. You know, if if all all other things being equal, if your tool runs twice as fast or takes the half the amount of memory, then it's, you know, everybody wins.

• 50:30Daniel Whitenack?

  Great point.

• 50:31Speaker 00

  It's harder when you say, okay, I I I wanna give up a a fraction of percent or one percent or 10% and and get it run twice as fast. And there, you know, it's it's hard to goes into, you know, questions of politics and regulations. And then what do these companies, what is the price for them to have expensive models running? Again, more on the environmental side, because this is not, doesn't relate to the research community, because it's not open anyway.

• 50:59Speaker 05

  Yeah, I think another thing to build on that, one thing that we're hoping with our, for example, the track that we have at these upcoming conferences and the conferences that have happened, is a place where you can look for research that does directly aim to improve efficiency metrics. So as Roy mentioned earlier, distillation is one approach that's pretty popular about taking a large model and making it smaller and more efficient. There are a ton of ways to do this. So model compression, using the lottery ticket hypothesis, or like Roy and I had a model compression paper. There's a lot of ways that people are taking existing work and making it more efficient.

• 51:41Speaker 05

  And with this track at these conferences, or just in general, you know, promoting these ideas, hopefully, one thing that you can take away from this is a snapshot of ways that you can improve efficiency that have a good track record in the research community.

• 51:57Daniel Whitenack?

  Awesome. As we kind of close out here, I'm curious, since you both have like a very close pulse on the research community, in particularly your your own areas of research, but also sort of more generally, I'm curious if we were to imagine in the future, and there's a world where like green AI is the thing that that everyone's doing. So some of we've reached some of those goals. What else in the AI research world, or maybe like things, ways in which people are applying AI? What gets you excited as you as you look to the future of of the industry?

• 52:37Speaker 00

  That's a great question, you know, something that keeps me busy thinking about, I mean, you know, thinking about the horizon of where I want to take my work and where would I like to be in 10, 20, 30 years. So I'm excited about a few things. One, I think I started with, and two, you know, taking this amazing technology that does things that are far beyond our reach. And we, seriously, I mean, as someone who's been around, you know, not a ton of time, but I mean, even five, seven years back, nobody would even imagine that we'd be anywhere close to solving the task we're currently solving very successfully. And the questions that remain open are: How are we doing this?

• 53:20Speaker 00

  I mean, are we doing this because the models are very good at memorizing and they're just learning everything and kind of are very good at retrieving the information that they've learned? Are they really doing some sort of inference that requires some logic? Or some, you know, I don't want to use the word thinking, but you know, something that requires some processing, that requires things that we as humans do. And could we generate models that explain why they reached a certain conclusion rather than others? And could we trust? I mean, we obviously can do whatever, we can generate an explanation, but is this explanation faithful? And another thing that, you know, gets me excited is to use this technology for all the good things that it can do.

• 54:01Speaker 00

  And in particularly, thinking about doctors nowadays that, you know, how can we take things off their plate? Allow them to do more of what they're, you know, there's tons of applications of, you know, starting from doing better analysis of x-rays for radiologists and to, you know, to transcribe their patient summaries in a more efficient ways and to be able to extract information from that. There are tons of applications here that this technology can be used to make things better for lots of people. So that's things that I'm excited about.

• 54:41Daniel Whitenack?

  Awesome. Yeah, us too. I know Chris and I both resonate with those points. So what about yourself, Jesse?

• 54:48Speaker 05

  There's a lot of things I'm excited about. I think Roy, I mean, things Roy brought up, I even just now, I'm like, those are all really cool. I want to work on that stuff, too. I think for me, you know, continuing to work in these sort of two pillars of of my research so far, which has been reproducibility and efficiency. These are pretty broad categories. So along the efficiency line, one thing that I have been continuing to think about is, at least in NLP, what we've seen is like larger and larger language models, which are pre-trained on tremendous amounts of data. And then right now, what we've been doing is fine tuning these models, so updating all of the weights in the model so that we can perform well on some downstream task.

• 55:30Speaker 05

  That could be, you know, sentiment analysis, or some kind of other types of text classification, or whatever. My guess is, as these models become larger and larger, there's probably going to be some other way that we can apply them to problems that we're interested in. An example of this that has recently been popular is adapters. So that's like adding a small number of parameters to one of these large pre-trained models, and then only updating that small fraction of the total number of parameters. I think the high level motivation here is that if these models are huge, and we want to take a, you know, one massive pre-trained model and adapt it to 100 different tasks, we don't want to have to have 100 different copies of this model. We want to have

• 56:15Speaker 05

  some smaller fraction. So I think that that is a pretty motivational idea, exactly what the next big thing in NLP is going to be. The next, you know, big idea about how we take our pre-trained models and apply them to many different tasks in a relatively efficient way. I'm excited to see what that is. I think one similar idea, and one way that we might do that is through probing tasks, so being able to probe our models without updating the weights in them to understand the kinds of Inferences that they can make, I think that's a particularly interesting topic. That's very active right now. I've seen, you know, too many papers to read just in the

• 56:57Speaker 05

  last month and a half on trying to probe existing models. And then on the reproducibility side, you know, we've had the reproducibility checklist now used for every submission at, I think, four conferences. That's a huge success. I'm pretty happy with the way that's worked out. The reproducibility checklist, I guess to give a little more information on that, is a checklist that's designed to remind authors of the kinds of information they should include to make their work reproducible. So it has like, did you include the number of parameters in your model? And did you include the, you know, what, the size of your data sets, for example? I'm excited and thinking about what we can do next with that information and also with the checklist.

• 57:41Speaker 05

  So now conferences are adopting it on their own. I've had to advocate in the past, you know, reaching out to the conference chairs and saying, hey, I think we should do this. Now conferences have picked it up on their own, which is pretty exciting. So, you know, I'm thinking a lot about how we can continue to measure the sort of quality of the research that the community produces at that community-wide level and what we can do going forward. What's the next iteration of the checklist going to be, for example? So that's what I'm thinking about.

• 58:14Daniel Whitenack?

  That's awesome. Yeah, and congrats on the success with that and getting that out there and sort of self-propagating at this point. I also agree with you, there's a lot of papers, even you've mentioned in this conversation, too many papers for me to read in a lifetime. There's so much exciting stuff going on, but really appreciate both of you taking time to join us and discuss this really important topic. I hope that people check out your paper, which we'll link in our show notes, and we'll link a bunch of the other things that Roy and Jesse talked about. So be sure to check those things out and definitely spend some time, I hope our listeners spend some time thinking about this topic and how it influences their workflow and

• 58:59Daniel Whitenack?

  other things. So, thank you both and hope to talk to you again soon.

• 59:04Speaker 05

  Thank you so much. It was so much fun. Thanks for having us.

• 59:07Daniel Whitenack?

  Thanks for coming on the show.

• 59:11Speaker 02

  Thank you for listening to Practical AI. If this is your first time, make sure you subscribe so you don't miss a thing. Head to practicalai.fm to subscribe or find us in Apple Podcasts, Spotify, or wherever you listen to podcasts. And if you get value from the show, please do share it with a friend or a colleague. We appreciate you spreading the word. Practical AI is hosted by Daniel Whiteneck and Chris Benson. It's produced by Jared Santo and our music is provided by Breakmaster Cylinder. We are brought to you by some awesome sponsors. Shout out to Fastly, Linode, and Launch Darkly. That is our show. We hope you enjoyed it and we'll talk to you again next week.