AI-powered scientific exploration and discovery

Daniel and Chris explore Semantic Scholar https://www.semanticscholar.org with Doug Raymond of the Allen Institute for Artificial Intelligence. Semantic Scholar is an AI-backed search engine that uses machine learning, natural language processing, and machine vision to surface relevant information from scientific papers. Featuring: Douglas Raymond – GitHub https://github.com/daraymond, LinkedIn https://www.linkedin.com/in/draymond, X https://x.com/daraymondChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Semantic Scholar https://www.semanticscholar.orgAllen Institute for Artificial Intelligence https://allenai.orgELMo https://allennlp.org/elmoSciBERT: A Pretrained Language Model for Scientific Text https://arxiv.org/abs/1903.10676Semantic Scholar API http://api.semanticscholar.org/corpusUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com slash change log.

• 0:17Speaker 00

  This episode is brought to you by Digital Ocean. Digital Ocean's developer cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, TM accounts, worldwide availability with a 99.99 uptime SLA and 24/7 365 world-class support to back that up. Digital Ocean makes it easy to deploy, scale, store, secure, and monitor your cloud environments. Head to do.co slash change log to get started with a $100 credit. Again, do.co slash change log.

• 0:58Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slag with us around various topics of the show at change.com slash community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:24Daniel Whitenack?

  Welcome to another episode of Practical AI. I'm Daniel Whitenack, a data scientist with S I L International, and I'm joined as always by my co-host, Chris Benson, who is a principal AI strategist at Lockheed Martin. How are you doing, Chris?

• 1:39Chris Benson?

  Hey, doing great, Daniel. How's it going today?

• 1:42Daniel Whitenack?

  It's going pretty good. A busy week and lots to work on, which is good and tiring all at the same time, but but mostly good. What about on your end?

• 1:51Chris Benson?

  Nothing much, just the usual work and we're finally back to some nice weather here in Atlantis. So I'm enjoying that.

• 1:58Daniel Whitenack?

  Cool, cool. I know that in our last Fully Connected episode, one of the things that we talked about was the increase in AI-related publications and also increases in publications on the archive. And I think that's true in general. And I alluded to the fact that we would be talking about that more in a future episode, and that's this episode. So I'm really happy to introduce Doug Raymond, who's joining us from the Allen Institute for AI, where he's the General Manager of Semantic Scholar. Welcome, Doug.

• 2:32Speaker 03

  Thank you. I'm really glad to be here.

• 2:35Daniel Whitenack?

  Yeah, great to have you. Before we jump into Semantic Scholar and all about scientific publications and searching them and all of that stuff, I'd be interested to just hear a little bit about your background and how you ended up at the Allen Institute and working with AI and Semantic Scholar.

• 2:54Speaker 03

  Yeah, thanks, Daniel. My background is mostly in the product and business of artificial intelligence and machine learning. Before I was at the Allen Institute, I was at Amazon working on the Alexa machine learning platform. And prior to that, I've done a series of startups in the machine learning space and advertising and commodity trading, and then had a five-year stint at Google, working on the AdWords platform.

• 3:26Daniel Whitenack?

  Yeah, that's quite an experience with AI. What stands out to you over the years in terms of how AI or applying AI in the product sense or in an applied way has changed during that time?

• 3:40Speaker 03

  Well, from my perspective, what's changed is that we've become, as consumers, more conscious about how these models are influencing our lives and replacing various aspects of human cognition. So when I started out in the advertising space at Google, and this is now, you know, close to 15 years ago, we didn't think of it as AI. We thought of it as an efficient way to match supply and demand. But as the models have gotten more sophisticated and more capable, we, as business and product people, are thinking more carefully about how we can actually help users with some problem. And that's where the AI part of

• 4:23Speaker 03

  the technology becomes super relevant, because if you're not solving a problem that a user actually has, it's not really artificial intelligence. It could just be an interesting feature. And on the other side, I think the concerns about how these products and these AI models are using our data and potentially influencing us in unforeseen ways has become a much bigger part of what we think about and the considerations for what we build in the product.

• 4:52Daniel Whitenack?

  So were some of those considerations as related to, you know, how we use AI and think about it, was that what kind of motivated you to join with an organization like the Allen Institute? And maybe for those that aren't familiar, the Allen Institute is kind of a different sort of organization than like a tech company like Google, right? So could you kind of explain that and how you got involved with them?

• 5:16Speaker 03

  Absolutely. So I've been at the Allen Institute for almost two years now. And what motivated me to join was the mission. And we were founded by Microsoft co-founder Paul Allen about five years ago with the mission to build AI for the common good. So it's a core part of our mission to identify areas where AI can help the public in general. And I found that to be really compelling. I definitely enjoyed startup life and working at Amazon and Google, and those were great experiences. But, you know, my real motivation,

• 5:59Speaker 03

  I think, to continue my career is to do something that has a positive impact, especially with so much political discord and challenge in the world, especially with respect to the environment and other areas where citizens really need to have the accurate and relevant access to information to make good decisions. And so

• 6:23Speaker 03

  That is a part of our mission and it's something that seemed like a much more impactful use of my time than continuing to work on commercial products.

• 6:32Chris Benson?

  Gotcha. And just out of curiosity, do you have any insight into why Paul Allen wanted to to make this investment into AI, especially at the time that he did? Any insight into that?

• 6:42Speaker 03

  Yes, you know, Paul Allen was a visionary man. He had a variety of passions and interests. And when AI2 was founded, obviously, this I joined later, but the story of AI2's founding is related to Paul's interest in how AI could solve really fundamental problems in terms of how people access information. So one of our first projects at AI2 is a project called Aristo, which is a project designed to create an AI model that could answer scientific questions

• 7:23Speaker 03

  in a conversational format. And we recently reached a milestone where Aristo project is able to ace the eighth grade New York region's science test. And so the vision that Paul set years before has resulted in an AI that can actually help answer scientific questions. And that project continues and is taking on new challenges. With respect to semantic scholar, Paul's vision and is expressed by our CEO, Ornetzioni, was there is so much scientific literature out there, and it's so difficult to access and understand what's relevant, that the cure for cancer might be latent in the scientific literature.

• 8:04Speaker 03

  But with AI tools, potentially, we could make the connections and allow scholars to discover those connections and lead to breakthroughs.

• 8:12Daniel Whitenack?

  So I'm curious, now that you've kind of got into semantic scholar, given the Allen Institute's mission and how it's structured and in general, why would it be important or why should Allen Institute maybe be the one that kind of provides this assistance in parsing through the scientific literature versus maybe a for-profit organization or something like that?

• 8:37Speaker 03

  Absolutely. So I think that Semantic Scholar exists in a really unique place. And of course, there are many tools out there designed to help access the scientific literature. But when you think about broad coverage tools in the sense that they cover all scientific domains and try to solve this discovery problem in a generalized way. There's Google Scholar, there are tools like ResearchGate, and they don't have a really robust business model. So Google Scholar does continue to release features, but the pace of innovation has been quite slow. And our users always tell us that they want a better discovery experience than what they can find in Google Scholar.

• 9:19Speaker 03

  And other smaller startups like ResearchGate have the imperative of a business model. So they tend to be focused on, you know, social networking aspects or other ways to generate ad revenue, and not really on solving this fundamental discovery problem that basically all scientific disciplines face, which is there's just an information overload in terms of the number of scientific publications that are published each year. And then I guess, just to add to that on the other dimension, there are a lot of special purpose tools which try to solve a problem in a particular scientific domain, and they tend to be point solutions but aren't well-integrated with other domains or the rest of the research lifecycle. And so we think we're in a relatively unique place

• 10:03Speaker 03

  where our mission is to have the greatest impact possible on science. And with Paul's backing, we're able to pursue that in a generalized way, which makes me think that we have a great opportunity to have a huge impact on the progress of science overall.

• 10:21Chris Benson?

  That sounds cool. So, I mean, what would you say were the main problems that you were targeting, you know, to solve with Semantic Scholar when you started out? And how were you trying to make it different from what was already out there and what people were using? And how did you choose the type of interface that you wanted to kind of realize that in?

• 10:41Speaker 03

  Yeah, absolutely. So let me start by talking about the problem. That's sort of core to our mission. And we define a problem as information overload in science.

• 10:53Speaker 03

  And the characteristics of that problem are that as the number of scientists around the world have grown and the number of research institutions and publications have grown, the number of potentially relevant scientific papers for each individual scholar to read has grown at an exponential pace since World War II. So we're now at 3.5 million new publications each year. It grows about five or six percent. Yeah, it grows at five or six percent a year. And the number of new journals also grows, so proliferation of new publications. And if you're a scholar in a particular domain, your ability to read

• 11:36Speaker 03

  papers is somewhat static, at least in the short term. So our research indicates that the average scholar reads approximately 250 papers a year. The time they spend per paper is about 30 to 45 minutes. And that comes out to, you know, up to 15 hours a week just trying to understand what's new or relevant in their domain. And so they don't really have a good way to read more papers without the help of tools like semantic scholar. So the way that we think of our solution to information overload is if the scholar's attention is fixed and we want to, at least the amount of time they have is fixed, we want to make it possible for them to overcome information overload by discovering the

• 12:21Speaker 03

  relevant papers much more easily and with much higher quality in terms of what they decide to read. And then we want to make it easier for them to understand what's interesting and salient to their research in each paper they read.

• 12:36Daniel Whitenack?

  So while you were talking, I was just kind of contemplating some of the numbers that you mentioned. And at least if I did my calculation right, so 45 minutes per paper, 3.5 million per year, that would take me about 300 years to just read all of the papers for a single year. So obviously, no one's going to do that. You mentioned like, you know, scholars read about, what was it, 250 or something per year?

• 13:04Speaker 03

  Yes.

• 13:05Daniel Whitenack?

  So it's called semantic scholar. So is the idea really around like a semantic or a text-centric, like natural text-centric way to search through the literature?

• 13:17Speaker 03

  Yeah, partially, yes. And if it's okay, I'll take a minute to explain why we're called Semantic Scholar. So you hit the nail on the head, Daniel, in terms of describing the challenge. Three hundred years' worth of reading every year is obviously untenable. And so Semantic Scholar, we think of semantics as the science of how do we understand, extract the meaning from the scientific literature. So when I talked earlier about the evolution of AI throughout my career, Semantic Scholar is an AI application because we're trying to use our AI models and technology to survey and read the papers in advance for you, so that as a scholar, instead of spending three hundred hours reading a bunch of papers,

• 14:02Speaker 03

  most of which aren't relevant, you can focus on only the papers which are most relevant to your interest at that moment. That's our vision in terms of how AI can solve this problem of information overload.

• 14:15Chris Benson?

  I'm just curious, how do you match up the user who's using Semantic Scholar with that process? How do you know what is the right research and how to present it to them?

• 14:25Speaker 03

  Absolutely. So we think of our product as having kind of three core attributes that help the user find the relevant science. And at a high level, and I can go into more detail in terms of how we use AI in each of these areas, one thing that we've done is create a very rich knowledge graph that represents all of scientific literature. So through mapping all the papers and citations and indexing full-text PDF of the scientific literature, created a very rich representation of science at this point, over one hundred and eighty million scientific papers. The second aspect of that, which is, I think, more related to your question around how does a scholar use us to find the relevant

• 15:08Speaker 03

  literature, is our discovery experience. And so Semantic Scholar is, the initial experience is pretty much like traditional search engine. However, because we've extracted the semantics from all the underlying literature, and it's in a structured knowledge graph format, it's much easier For the scholar to define their interest in terms of this area of science from these journals in this date range and have a comprehensive representation, not only of what papers meet that interest, but all the other extracted information that we build with our models, such as the influence of the paper, how that paper has been discussed in social media,

• 15:52Speaker 03

  the associated data sets and GitHub repositories that are used in that research. So we try to create a very rich representation of not only what's in that scholar's scope of interest, but within each paper, you know, what are the points that would allow them to understand, is this paper relevant? What is new and interesting about my area of interest that's expressed in this paper?

• 16:27Speaker 04

  Hello there. This is Jared Santo, Managing Editor here at ChangeLog. The fact that you're listening to this means you are actively investing in your future in this industry. Things move fast and keeping up is hard work. Help us help you stay relevant by subscribing to ChangeLog Weekly. We track, log, and contextualize what's happening in software throughout the week and deliver it directly to your inbox on Sunday mornings. Head to changelog.com/weekly to browse the archives, subscribe, and push the easy button on your continuing education. That's all from me. Once again, that's changelog.com/weekly.

• 17:18Daniel Whitenack?

  So Doug, it was really great to hear about kind of the underlying structure of semantic scholar and how you kind of start to access its features, but then kind of get some of the interesting benefits of it. I was wondering, as you were talking through that, so there's the sort of knowledge graph that you mentioned, and then there's the search and discovery, I don't know if it's right to call it like a recommendation or notification type of stuff. I was wondering, there's kind of, at least I'm aware that there's some work going on pretty widely around using AI to generate or automatically build knowledge bases or knowledge graphs. So I'm wondering if that's one place where you're utilizing AI to kind

• 18:03Daniel Whitenack?

  of extract this and kind of automatically build the knowledge graph. But then it sounds like maybe there's other opportunities for AI usage on the user interactivity side. And I was wondering how much effort you're kind of placing in those two areas and where you think the main benefits, at least in this application are for AI, at least that you've leveraged so far.

• 18:25Speaker 03

  Sure. So it's true that we do use models to build our knowledge graph. And there are some efforts going on there to increase the quality and coverage of it. In terms of what are the areas that we think are most exciting to us from a research standpoint, we're focused on this discovery experience in terms of how do we help you identify what's new and relevant. And we have several research efforts in terms of, you know, creating a personalized representation of what's new, of creating explanations or recommendations that are actionable in terms of how we explain to you why we've recommended particular papers. And we have a number of other research areas there that I'd love to talk about that I think are quite exciting.

• 19:09Daniel Whitenack?

  Yeah, definitely. I'd love to hear more about those. I know specifically, because I've used Semantic Scholar, and maybe you can describe this a little bit more and how it fits into some of those discovery things. But as you're searching things, you can, I believe, tag certain content and create sort of tagged collections of content that you're looking for and organizing?

• 19:28Speaker 03

  Yes.

• 19:29Daniel Whitenack?

  Does that fit into this sort of discovery model?

• 19:31Speaker 03

  It does. I think, you know, we have a library where we enable our users to organize and tag their research. And so I think that's part of how we help them use it in their work and have a greater impact in their work with the research they've found through Semantic Scholar. There's also a lot of work we've done to help you identify, out of the thousands of new papers published each day, which of those are relevant to me, which of those are worth reading. That's another area of research for us.

• 20:04Daniel Whitenack?

  So, does that fit more within a sort of traditional recommender system sort of thing? Or in what ways are those being generated? And one of the things that's going through my mind, too, is it seems like there's these giant benefits to this sort of approach, but then also, if you're kind of amplifying certain signals within the scientific community, you kind of have to be pretty right on with those, because you could kind of, like you said, the cure for cancer could still be sitting somewhere below these sort of amplified signals.

• 20:36Speaker 03

  Yes. So, maybe I could talk about how do we recommend papers? So, I think, if I understand your point, Daniel, there's a phenomenon in science where, which I would describe as, you know, the rich get richer, which is, if you have a institutional backing, if you're publishing your papers in prestigious journals and you're generating a lot of citations, that citation count can be used as a proxy for quality. And those scholars who don't have the institutional backing, don't have as many citations, their research will get overlooked. And I think we are very conscious of that phenomenon, and it is one of the

• 21:17Speaker 03

  challenges we hope to overcome with our approach to discovery and recommending papers. So, I could go a little bit more into that.

• 21:26Daniel Whitenack?

  Yeah, that'd be fantastic if you would.

• 21:27Speaker 03

  Yeah, absolutely. So I think in the Semantic Scholar experience, we do use citation count as a as an interesting piece of metadata. But from our perspective, one of the challenges with the growth of the scientific literature is that there are great scholars out there and great science being done in places where they aren't in a prestigious conference or prestigious institution or published in a prestigious journal. And so therefore, they may be overlooked if that's the only thing you're looking at. So for us, we really think a lot about how can you discover science that's relevant before it has a rich citation history or or science that's relevant without a citation history. And so one of the the big efforts that we've made is trying to understand the relevance of of papers at

• 22:11Speaker 03

  a at a very fundamental level. And that starts with us at the language model level. And I think perhaps in a previous podcast, you talked about some of our work on on different language models. At AI2, we developed a language model called ELMO, which was subsequently developed further into a model called BERT through Google. We have created a pre-trained language model called Cybert, which is trained on 3 billion words from a host of scientific documents. So it is particularly good for trying to understand what a paper is about in a way that a model trained on Wikipedia or other texts would not be quite quite as

• 22:55Speaker 03

  good at. And we've used that scientific language model, which we call Cybert, to build a host of of discovery experiences that helps scholars find relevant papers, even if the citation count doesn't necessarily indicate that that paper is highly regarded. So one of the things we've done with that is create a personalized feed of papers, sort of a Spotify for research, if you will, which is available in Semantic Scholar now. And the idea there is that if we use this language model and create a neural network to understand the similarity of papers,

• 23:34Speaker 03

  We then allow the user to indicate what they like and what they don't like. And through that process, in a few clicks, they can create a highly relevant feed of research papers that's tuned exactly to their personal interests in a way that you could never, you could never do it with a search engine or just looking at citation counts.

• 23:53Chris Benson?

  And how do you make that available to the user? How are they able to actually kind of specify what their interests are?

• 24:00Speaker 03

  Certainly. So the way the product works is you go to Semantic Scholar, you select papers that you believe are relevant to you, and you can use the traditional search interface to do that, or just, you know, type in the title of the paper you already have high regard for, and then it will automatically generate recommendations of related works. And by indicating, I like this paper, I don't like this paper, you can tune that feed in real time to be highly relevant to your interests. So you're only seeing papers that are directly relevant to the interests you're pursuing.

• 24:36Daniel Whitenack?

  I'm curious, in this process, like, how long approximately, or how many papers do I have to go through and kind of tag before I start seeing some of this benefit? And I guess also, on that front, is this amplified, like, between users? So I guess there's like personalization at the user level, but there's also, in science, there's communities of people that are working together and collaborations, and does that fit into the recommendation at all, or is it mostly at the language model level?

• 25:07Speaker 03

  Yeah, at this point, it's really at the language model and what that individual user has indicated. I think what you described is very interesting in terms of how do we build recommenders that service to a community. But what we've done so far is create a model that looks at the paper similarity based on Cybert, a model trained on scientific text, and then tune it so that scholars can get papers that are relevant to their interests. In terms of your initial question, how many papers does it take? What we find is that most of our users are able to get a highly relevant feed within about by rating between three and five papers. Depending on how generalized or specific your interest is, it could take more.

• 25:50Speaker 03

  So if you have a very specific narrow interest, you might need to rate more papers before your feed becomes highly, highly relevant. But in most cases, it takes a minute or two to identify three to five papers that match your interest.

• 26:06Daniel Whitenack?

  So I'm also curious, because when I'm thinking about this sort of similarity matching with a language model, I'm kind of thinking about it like, oh, you perform, you know, you provide input data to this language model. You get out some representation, maybe you compare distances or something like that. But it's not really related to, in my mind, the sort of graph structured data that you mentioned before. Are those both utilized? And is the graph stuff mostly utilized for like just search? Like you type in a query and then you get entities out of that and match those entities to entities in the graph? Versus like the language model is utilized mostly for recommendation? Or is there any interplay between those?

• 26:50Speaker 03

  There is. So to be clear, the graph structure is kind of our core data structure. So when you search semantic scholar, you're essentially trying to identify, you know, a vector within this knowledge graph that's within your scope of interest. For the recommendation experience, the adaptive recommendations I was describing, we do use the graph information, but what we do is use a citation graph of these different papers as a feature in that similarity model. So understanding what papers have cited each other helps determine, you know, how close they are in that similarity space based on what the user has indicated is of interest to them.

• 27:31Daniel Whitenack?

  Gotcha. So it's kind of like the language, am I correct in saying like the language model would give you a sort of learned representation of a paper and then you're matching that in terms of distance in some space and using a feature from the graph like the citations like you're mentioning to further refine that? Or is it different than that?

• 27:53Speaker 03

  I think that's pretty close, yes. In the sense that, you know, the language model is just how we understand what the paper is about. In terms of understanding how similar one paper is to another, the language model, the Cybert, is one aspect of understanding, okay, here's a vector that represents the meaning of this paper. But similarity is also indicated by citation graph, a neural model that we built to map those papers in some vector space to understand how similar they are. And then the aspect that makes this a personalized experience is the user being able to indicate what papers are of interest and are not of interest. And that becomes an input to the model to define what papers should be presented.

• 28:52Speaker 04

  This episode is brought to you by Brave. We deserve a better internet. That's why the team behind Brave reimagined what a browser could be. Brave is like Chrome, the good parts. Even your extensions will just work. It has built-in add-in tracker blocking, easy anonymization with the Tor network, earn tokens while you browse, and use them to tip your favorite creators. And did I mention it's lightning fast? Turns out the web is super fast when you remove all the cruft. Download Brave today using the link in the show notes and give tipping a try on chainsaw.com.

• 29:34Daniel Whitenack?

  So I'm curious, as we are kind of talking through the ins and outs of discoverability in Semantic Scholar and also the, you know, how things are working under the hood, I got to thinking, like, given that you've processed so many scientific papers, are there any efforts within Semantic Scholar to sort of analyze in a more exploratory way the scientific community as a whole? Like, I'm guessing that, you know, from this knowledge graph, you're able to maybe extract collaborations and other things that might be sort of under-described in a general search interface, if you're just searching on a journal or something like that,

• 30:14Daniel Whitenack?

  where you're kind of representing more about a paper than is known. So, I don't know, have you explored anything like that in terms of, you know, exploring what collaborations exist and how to represent those? And maybe, I was thinking of like duplicate or highly-related work in terms of like reviewing new work and that sort of thing? Are there any efforts like that?

• 30:38Speaker 03

  Yes, that is an interest of ours. And I could, a few efforts come to mind. So, one thing that we've done recently is use our summary of authors as a way to help conference organizers disambiguate reviewers. So this is a problem in academic conferences where if many papers are being submitted, you need to select the people who will review those papers to decide whether they will be accepted at the conference in an optimal way. And you can imagine how that problem becomes exponentially more difficult as the number of papers submitted

• 31:20Speaker 03

  increases exponentially year over year. So that is, the submissions to conferences in computer science have grown multiple fold just in the last five or 10 years in fast-growing areas of AI. So, one partnership we did in the fall was with ACL, which is a large Conference that's going to be based in Seattle this year. And they use the semantic scholar knowledge graph to help disambiguate reviewers from the papers that are being submitted for review. Because you don't want someone that you co-authored with, or you know, someone who's potentially, you know, on your same faculty to be reviewing your paper for conference because it creates conflicts of interest. That's an example of a recent application.

• 32:04Speaker 03

  I could talk about the other uses of semantic scholar in terms of understanding science overall.

• 32:10Chris Benson?

  Oh, please do. That would be good. I was actually gonna ask something very similar to that.

• 32:13Speaker 03

  Oh, absolutely. So because we've created this rich representation of science, it allows us to do what we would describe as meta research in terms of the trends in science and what potential opportunities or challenges are emerging. So we've published articles about the growth of open access publishing, about gender, the trends in gender equality in computer science publishing, looked at biomedical research and identified areas of bias in clinical studies. And in each of these cases, because we've created this rich and structured representation of science,

• 32:53Speaker 03

  we're able to do research on orders of magnitude larger data sets than any previous research.

• 33:00Chris Benson?

  So do you have any, I'm just kinda curious. You just started to go there, but my mind's kinda wondering on different possible use cases of that. You know, because you're really in your structured graph, you're kind of capturing the shape of science, if you will, you know, based on the papers and the citations and kind of where it's flowing and where it's not flowing a little bit. Do you have some ideas when you're talking about that kind of that meta analysis? Any thoughts on different use cases where you guys have thought that that would be particularly useful?

• 33:30Speaker 03

  Yes. So we've already published a number of studies where we've identified areas of bias or potential insights of social impact. For instance, gender equality in computer science. I think in the future, we'd like to do More of these studies and focus them on areas where we can identify opportunities to increase the impact of science overall. So a big area of interest for us is climate change research in terms of what's being funded, where are the areas that are either overserved or underserved, and how can we surface that

• 34:10Speaker 03

  information in a way that helps scholars, but also potentially policymakers or politicians, invest in the areas that can have the greatest impact?

• 34:19Daniel Whitenack?

  I'm curious as well, as you've done this sort of work, are there, and you've obviously processed a lot, so there's a lot there already, but I was wondering if there's certain areas of science or areas of research that are harder to probe in terms with this approach than others? So I'm thinking, I work with a bunch of linguists at my organization, and I found that like there's all of these sort of archived systems that are really hard to access and search and all of those things, but that's where a lot of the linguistic research is, and

• 34:58Daniel Whitenack?

  it's all sort of documented in really odd and conflicting ways in terms of like what languages it applies to and all these things. So I was wondering if there's like systems like that or areas of science that have proved harder to integrate in with this sort of approach? And you know, what sort of ways you're approaching, you know, the diversity of how science is represented for different areas?

• 35:25Speaker 03

  Yeah, you know, I would say that there are definitely opportunities to increase our coverage in certain areas of science. At the highest level, we are optimistic that our generalized approach seems to work pretty well across all domains of science that we cover. There are definitely Issues that you alluded to in terms of older publications where we may not be able to get access to a PDF or the data that allows us to figure out how to integrate it under Knowledge Graph is hard to come by.

• 36:06Speaker 03

  But I wouldn't say that there's sort of an obvious, you know, major problem to overcome. There are a lot of smaller problems to overcome, which we kind of address in our planning based on how much impact we think we can have for our scholars.

• 36:20Daniel Whitenack?

  Yeah, and I guess that there's various considerations in terms of how actively or active and rapidly developing areas of science are and, you know, how they maybe apply to certain things that the Allen Institute is also interested in, like climate change and that sort of thing. Yeah, I mean, I guess you have to, you have to start somewhere and put your efforts somewhere. But I think probably just assuming that you can get a PDF covers a large majority of cases. Is that right?

• 36:53Speaker 03

  Yes, and a lot of our effort is focused on partnerships with the major academic publishers, on integration with the, some of the major pre-print servers like Archive and Open Access Journals. And so if we get a high-quality PDF, in most cases, we're able to fully index that content and make it discoverable to our users. You know, there are other challenges in terms of quality of extraction and how our models work to fully extract the content from different fields of science, but they tend to be, you know, fairly minor to the challenge of just getting access to the science and making it possible for scholars to discover it.

• 37:33Daniel Whitenack?

  Awesome. Well, I was curious, in addition to the Semantic Scholar product and the discovery tool, I think, if I'm not mistaken, some of what has happened within Semantic Scholar has been open sourced in terms of things that people can use. So are things like pre-trained models and maybe tools, I think there's like a PDF parsing tool, is that right? What's come out of Semantic Scholar in terms of open source things that maybe others can build on as they're thinking more about science and PDF parsing and those sorts of things?

• 38:11Speaker 03

  Absolutely, and I think that's a it's a great illustration of what makes AI2 and the Allen Institute for AI and Semantic Scholar special is that our mission is to have a positive impact on society at large. And so, most of the things that we build that we think are valuable and unique are available as open source projects. So our knowledge graph, the aspects of it that we can release that are not restricted by the various agreements we have with publishers is something we release as a public resource for the research community. This language model, Cybert, that I that I mentioned earlier is available on GitHub. And so, we try to make it possible for others to build on our work.

• 38:53Speaker 03

  In addition to that, we have a public API where other projects can access Semantic Scholar features and our knowledge graph to further science in their own way.

• 39:04Chris Benson?

  So I guess as we as we wind up and kind of bringing it back to a to a very practical side of things, I'm kind of curious is as users, maybe maybe they've heard the episode here and they decide to try it out and get into some of the features we've been talking about, what should they be expecting on your roadmap ahead in the relatively near term over the next few years? You know, where do you expect to grow the product so that they can kind of take advantage of some of the things we've talked about as you've talked about these big problems that you're trying to tackle and expanding that?

• 39:36Speaker 03

  Absolutely. Our vision is to be a solution for information overload. So we'd like scholars to come back to us whenever they're trying to understand what the scientific literature says about some issue that is of interest to them. And so, a lot of our work in the upcoming year and beyond is around making that discovery experience higher quality, adding new models and new AI-driven features that allow you to understand the highlights of a paper, to understand the intent and to get a summary of it in a very succinct and high-quality way. And by doing that, we hope that we can make Every semantic scholar user, higher impact in their work.

• 40:21Speaker 03

  They'll be able to spend, if they still want to spend 15 hours a week, they can do that, but it'll be much higher quality of reading. If they want to spend less time so they can focus on other aspects of their work, we hope to enable that too. And so our research is really designed to make information overload a problem of the past and allow scholars to focus on what they can do best, which is delving into new unknown areas of science and creating breakthroughs.

• 40:47Daniel Whitenack?

  Awesome. Well, I'm excited about that future and definitely, you know, I'm excited about semantic scholar and a lot of the things that the Allen Institute for AI is doing. I know we had Joel Gruess on a previous episode talking about Allen NLP, which I've used personally. And so, thank you so much for working on semantic scholar, but also, you know, pass along our thanks to the Allen Institute for all the great work that they're doing and the contributions to the community. I think it's a really great thing to see so many efforts that have made contributions, practical contributions that people can use. So yeah, thank you so much and thank you for taking time to join us.

• 41:29Speaker 03

  Excellent. Thank you, Daniel, and thank you, Chris. It was a pleasure. Thank you.

• 41:35Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for ChangeLog is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do hear ChangeLog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Lino cloud servers. Head to lino.com/changelog, check them out, support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address, get our weekly email, keeping you up to

• 42:19Speaker 00

  date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.