Pod Insights The Cognitive Revolution Can AIs Generate Novel Research Ideas? with lead author Chenglei Si About this episode In this episode of The Cognitive Revolution, Nathan delves into the fascinating world of AI-generated research ideas with Stanford PhD student Chenglei Si. They discuss a groundbreaking study that pits AI against human researchers in generating novel AI research concepts. Learn about the surprising results that show AI-generated ideas scoring higher on novelty and excitement, and explore the implications for the future of AI research and development. Join us for an insightful conversation that challenges our understanding of AI capabilities and their potential impact on scientific discovery.
Link to the research paper being discussed: https://arxiv.org/abs/2409.04109
Be notified early when Turpentine's drops new publication: https://www.turpentine.co/exclusiveaccess
SPONSORS:
Weights & Biases RAG++: Advanced training for building production-ready RAG applications. Learn from experts to overcome LLM challenges, evaluate systematically, and integrate advanced features. Includes free Cohere credits. Visit https://wandb.me/cr to start the RAG++ course today.
Shopify: Shopify is the world's leading e-commerce platform, offering a market-leading checkout system and exclusive AI apps like Quikly. Nobody does selling better than Shopify. Get a $1 per month trial at https://shopify.com/cognitive
Notion: Notion offers powerful workflow and automation templates, perfect for streamlining processes and laying the groundwork for AI-driven automation. With Notion AI, you can search across thousands of documents from various platforms, generating highly relevant analysis and content tailored just for you - try it for free at https://notion.com/cognitiverevolution
Brave: The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://bit.ly/BraveTCR
Oracle: Oracle Cloud Infrastructure (OCI) is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds; offers one consistent price, and nobody does data better than Oracle. If you want to do more and spend less, take a free test drive of OCI at https://oracle.com/cognitive
CHAPTERS:
(00:00:00) About the Show
(00:00:22) Sponsors: Weights & Biases RAG++
(00:01:28) About the Episode
(00:05:30) Introducing Chenglei Si
(00:06:22) Path to Automating Research
(00:07:58) Notable AI Research Projects
(00:15:26) Evaluating Research Ideas (Part 1)
(00:19:39) Sponsors: Shopify | Notion
(00:22:33) Evaluating Research Ideas (Part 2)
(00:25:49) Research Setup and Design
(00:29:38) AI Prompting and Idea Generation
(00:34:40) Diversity vs. Quality of Ideas (Part 1)
(00:34:40) Sponsors: Brave | Oracle
(00:36:44) Diversity vs. Quality of Ideas (Part 2)
(00:42:05) Inference Scaling and Execution
(00:45:04) Anonymizing and Evaluating Ideas
(00:53:22) Headline Results and Analysis
(00:58:45) Observations and Insights
(01:09:02) Novelty Indicators and Deception
(01:11:59) Top AI-Generated Ideas
(01:14:41) Next Steps and Future Directions
(01:20:43) Expectations for the Future
(01:23:14) Outro
Transcript 14,210 words · 3 speakers
Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Nathan Labenz? Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, my guest is Chung Lee Si, a PhD student at Stanford who's developing ways to use large language models to automate research. He's lead author of a fascinating new paper that asks the question, "Can large language models generate novel research ideas?" This question has become one of the most important in the entire field,
0:44 Nathan Labenz? as the ability to generate research ideas that are truly worth pursuing, particularly in the domain of AI, has long been considered a key precursor to recursive self-improvement loops and a possible intelligence explosion. Since the early days of GPT-4, we've seen several notable attempts to create AI research assistants, including projects like Co-Scientists from Gabe Gomez's group at CMU, which we did a full episode on, and recently, the AI Scientist paper from Japanese company Sakana AI as well. These systems demonstrated capabilities that would have seemed impossible just a couple of years ago, including translating natural language instructions to chemistry protocols and using the Semantic Scholar API to assess research ideas for originality. Nevertheless, the question of whether AI systems
1:27 Nathan Labenz? can produce high-value research ideas, true eureka moments, has remained the subject of fierce debate. Chung Lee and his collaborators set out to shine light on this subject with an ambitious study. They asked more than 100 PhD researchers working in AI for new research ideas, incentivizing quality with cash prizes for the best ideas, and then asked Claude to generate new ideas as well. After processing the text in an attempt to create a level playing field for evaluation, they then had expert reviewers rate all of the ideas without knowing their source. The results: the AI-generated ideas scored significantly higher on both novelty and excitement. Now, as with many recent AI results, this paper has become something of a Rorschach test.
2:12 Nathan Labenz? Those inclined to believe in rapid AI progress see it as a major milestone. While skeptics criticize the methodology, somewhat unfairly in my view, but you can judge for yourself as you listen, and more persuasively in my mind, emphasize that this work, which focuses specifically on prompting techniques for language models, may not generalize to the harder sciences. Personally, I agree that we cannot confidently project this one result onto other domains. But I find the experimental setup here to be quite well done, the statistically significant results seem credible, and I think Chen Lei's individual observations are worth taking very seriously as evidence, too. Everyone listening to this show should be familiar with the AI maxim "to look at your data", and nobody has spent more time with the raw outputs than Chen Lei has.
2:55 Nathan Labenz? So when he reports that nine of his 10 favorite ideas from this entire project turned out to be AI-generated, and that the AI ideas are generally more out of the box and less grounded in existing work than human ideas, I think we would do well to listen. Overall, my feeling right now is that we're at a sort of tipping point, where the Claude 3.5 Sonnet and GPT-4O class of models can sometimes, with great effort put into system design and many millions of tokens to burn, sometimes generate meaningful novel research ideas, but not yet in a way that makes frontier research dramatically more accessible or scalable. The next generation of models, starting with the O1 series, seems to me pretty likely to change that. I've been coding with O1 and Cursor a lot lately, and I've been really struck by how effectively O1 can review my entire codebase
3:40 Nathan Labenz? and my plan for a new feature, and return both a critique of my approach and a recommended alternative that is usually genuinely better. As a community, we're still figuring out exactly what the limits to these capabilities are. But if I had to bet, I'd say that systems like Chen Lei's, powered by O1 and a bigger inference budget, might well produce undeniably needle-moving results quite soon.
4:03 Nathan Labenz? As always, if you're finding value in the show, we appreciate it when folks take a moment to share it with friends, write us a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. And we welcome your feedback via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. For now, I hope you enjoy this conversation on the emerging reality of AI automated research with Chung Lai Si. Chung Lai Si, PhD student at Stanford researching the automation of research. Welcome to the Cognitive Revolution.
4:33 Speaker 01 Thank you. Excited to be here.
4:35 Nathan Labenz? I'm excited for this conversation. I think you are working on a super interesting frontier, and your recent paper with a provocative question in the title has certainly caught my attention, and I think the attention of the field more broadly. So the paper that we're here to really go deep on today is called "Can LLMs Generate Novel Research Ideas: A Large-Scale Human Study With 100+ NLP Researchers?" I'm looking forward to breaking this down and really getting into the nitty gritty of the methodology and what we can learn from it and what we should maybe take away from this and expect to see as we go forward
5:15 Nathan Labenz? into the near future. For starters, want to maybe just give me a little bit of your background? Like, how did you get to focus on this question of automating research? And maybe love to hear too, some of the results from the last year or two that you have found to be most compelling.
5:34 Speaker 01 Yeah, so I came from a very NLP background. I was doing NLP research back during my undergrad, and then I came to Stanford last year, and I met my advisor, Steve Tausch. And I always wanted to do something different for my PhD. And back then, the big background in the whole NLP field is that large language models are getting really good. And then we saw some really impressive performance on things like reasoning, math, and coding, and stuff like that. And it felt like it probably could give opportunities to some very new applications. And as someone who has done research for many years, I feel like being able to automate research feels like something very different and very exciting. And importantly, I think it will have huge practical implications to the entire field.
6:17 Speaker 01 And I just got excited about this agenda right away. And I'm lucky that most of my advisors are very excited about this whole agenda as well. So we actually got started on this whole thing pretty early, I think roughly one year ago. This whole project actually took around a year to finish.
6:32 Nathan Labenz? Yeah, interesting. And I trust that the human parts of that took the most legwork and took the most time. That is often the striking difference these days.
6:40 Speaker 01 Yeah, building the agent itself took us like two or three months. And then finding all the people and doing the actual human study took us like eight months or so.
6:50 Nathan Labenz? Yeah, interesting. So before we dive into the details of your paper, what other projects, and they don't necessarily have to be, it could be academic papers, but it could also be open source projects, have captured your imagination and kind of stand out to you as the notable step change improvements on the road to this destination of automated research?
7:16 Speaker 01 Yeah, so I think to me personally, there's this line of work on self improving AI. There has been discussion on whether as AI keep getting better, there will be a point where AI can be even smarter than humans. At that point, how do we possibly keep improving AI if AI is getting smarter than us humans? And then it gives rise to this idea that maybe we can let the smarter AI to propose new ideas and improve itself. There has been some preliminary attempts on that. There has been some work done by folks here at Stanford like Eric Zuckman. And that's one inspiration for this line of work. But of course, there's also another line of work on building tools that can assist researchers. A lot of colleagues from H C I, folks like Samantha Scholar, a bunch of other people have been doing this for many years.
8:01 Speaker 01 So research assistant tools, creative support tools is another inspiration for this. But I think we really take a very different approach in a sense that we want you to do something that sound even more aggressive in the sense that we want you to see if we can possibly automate the entire research process rather than just build some assistant tools there.
8:21 Nathan Labenz? Yeah, I think you've zeroed in on a really important question. I've looked for what it's worth at. This is kind of an area I try to keep up with because it feels to me like If we are going to have an intelligence explosion, this is kind of the form that it's going to take, right? It's going to be, and I'm by no means like alone in this analysis or original in coming up with this analysis. But basically, it seems like the expectation from the field is at some point, we're going to start to see AI systems that can make meaningful contributions to ML research. And if ML research becomes 10x, 100x, 1,000x more productive,
9:07 Nathan Labenz? because we can sort of spin up AI ML researchers and they can actually take real bites out of big problems, then it's really hard to say how good the the overall resulting models could get. And that could take us into a different regime. So it feels like there's this sort of tipping point possibly where if we can't get over that hump, then things maybe stay somewhat normal. And if we do get over that hump, then kind of all bets are off. And the future starts to look quite unpredictable and potentially quite weird. I guess for starters, like, is that your mental model of why this matters so much in the first place?
9:43 Speaker 01 Yes, you mentioned the word intelligence explosion. So I've read the whole situational awareness thing that Leopold wrote. My advisor, Tassu, was warning me that using intelligent explosion would sound too aggressive for academic setting. But I think on this podcast is probably fine. So yes, roughly, that was one exciting vision that I kind of want to open to you. There's very interesting data point. If you remember, there's a figure there where Leopold was saying, around the year of 2028, we would be able to get automated Alec Redford. So Alec was like the lead author on the GPT paper. So if there was a point where we are able to automate Alec Redford and we are skilled this up, that means we'll have massive increase in the way AI can improve itself.
10:28 Speaker 01 That'll be insane. But we want to, you know, maybe take a step back. Automating Alec is a pretty difficult role. There's only one Alec in the whole AI field. So maybe once you really see the current generation of large models, how good they are right now, I want to compare that with a more reasonable baseline. So maybe when we see how do they compare with, let's say, MHPST students. I want to offer a concrete benchmarking point to gauge the progress that we have made so far. So that was the motivation for the whole thing.
10:57 Nathan Labenz? Yeah. Okay, cool. So just to, and by the way, Alec Radford has been making the rounds on Twitter just today as, or in the last like 24 hours, we're recording on 1st of October. And he's been a popular answer to a question that somebody posed, which was, who is the most important or influential scientist that is not very well-known? And people are piling on to say Alec Radford is the answer to that question. That is indeed an extremely high bar for automation. To just further tee up the specific question that you're asking with this research around using LLMs to generate novel research ideas.
11:37 Nathan Labenz? I think it is worth taking a minute to just review a couple of my personal favorite highlights from the research over the last year. Because I do think that has been an area where it hasn't quite been there yet. And it feels like we've sort of been always a little bit short. One that came out very quickly after GPT-4 was released, and we actually had the lead author, Gabe Gomez, on the show for a prior episode, was called Co-Scientist. And they put together a really nice, early, but I think has stood the test of time, agent framework that allowed a user to basically say something like, please synthesize aspirin for me.
12:21 Nathan Labenz? And then the system would do everything needed to synthesize aspirin, including like looking up, how do you synthesize aspirin? Translating that to actual instructions that could be sent over to an automated lab. They were integrating this with Emerald Cloud Lab at Carnegie Mellon. And you can actually program a real physical lab via an API to carry out the step-by-step, you know, protocol that you want to run. And that was amazing, right? I mean, to go from a natural language sentence all the way through to like, a purified powder, you know, getting spit out the other end of a system like that. I was like, man, this is a... Major result, and obviously that's why I wanted to have him on the show back then.
13:05 Nathan Labenz? But the one thing that I would say at that time was it did not seem to me like the AI was actually coming up with meaningful ideas. So in my review of that paper, I called it, you know, the question was like, is this text to science? And or text to experiment? And I was kind of like, yeah, I think it's more text to protocol. Because the AI at that time wasn't really generating the idea so much as taking the idea and figuring out how to break that down into steps and execute on it. But that was never less a big milestone. I don't know if you have any comment on that one before we move on, but I'll highlight a couple of other ones and then we'll get into yours.
13:43 Speaker 01 Sure, I see the idea is cool for sure. There has been similar attempts in maybe other areas. Our friends at Sakana AI, for example, has a AI sent his paper that also tried to improve what a engine agent system. I think the high level research agenda is exciting for sure. But I think the tricky thing is the evolution part is really difficult, especially in scientific research. How do you possibly evaluate that research even for academic settings? We do peer review, that isn't the perfect solution either. So I think all of this really boils down to the technical challenge of doing evolution. And that is really the crux that we're trying to focus in this paper here.
14:21 Nathan Labenz? Yeah, the AI scientist was another one that was on my list. And I thought that one was a notable step forward. I mean, all these things have their critiques and I feel it's like a bit of a Rorschach test that tells you often as much about the critic of the paper, whether you know, positive or negative in their assessment as it does about the research in some ways, because there is just so much room for interpretation still of these results. I find that to be fascinating. The AI scientist, to me, though, did take a step forward in that. It introduced a mechanism for generating a bunch of ideas and also trying to identify
15:03 Nathan Labenz? which of those generated ideas were the most novel ideas. And they used the semantic scholar API to help with that. So they would have a bunch of ideas generated, then go do a search against the semantic scholar corpus of information to pull back as much related stuff as possible. And then they would sort of try to see like, has this already been done? To what degree does this seem like it's already represented in the literature versus if not, then that's more likely to be a meaningfully novel idea. And that would be the the kind of idea that they would want to prioritize. The reviews of the actual AI written papers that came out of that didn't seem like they were necessarily going to get admitted to top conferences. And even in their own assessment, the Sakana team
15:47 Nathan Labenz? basically said, this would be like the work of an early grad student that definitely needs more advising from, you know, somebody who has better zoomed out context on the field. But I did still think that was a notable mechanism that seemed to kind of show the way to where this was going. I mean, one of the ways I describe this is, you know, do we get eureka moments from AI systems or not? And initially with GPT-4, I used to say, no eureka moments. That was kind of my headline, you know, for GPT-4 performance. And then we saw the eureka paper, and that was out of Nvidia, Jim Phan and his group.
16:26 Nathan Labenz? And there, they used GPT-4 to write reward functions for robot hand reinforcement learning. And they saw that GPT-4 outperformed human authors of these reward functions. And I always am fascinated by that because I'm like, Geez, it takes some real know-how to even write a reward function in the first place, right? Like, this is not something that just any old person off the street can do. You're engaging here with, like, a certain level of expertise as a baseline and still seeing GPT-4 perform above that. Not exactly, I mean, it's sort of like a hypothesis, you know, a reward function is, you know, how well is it going to work? You're making a
17:08 Nathan Labenz? guess as to what would work. So I feel like that was kind of moving in that direction as well, and the fact that it did outperform humans in something that was like objectively measurable was quite notable to me. Any other thoughts on that one?
17:22 Speaker 01 I feel like, you know, my definition of a Euroco movement for automating research life work would be if one day you tell me you have an agent that enjoyed that idea and automatically executed the idea into your full project, and you submit that project to a talk conference, and that project actually gets the best paper award, then I would say that indicates a Euroco movement for me, because that means that piece of project is actually way better than the average that you get at the so-called talk conferences. Right? I think the AI scientist paper from Sakana that you mentioned, I think the agent system, that's a nice prototype there, but the outcome, first of all, even using their own AI reviewer, the score isn't really reaching above for Eurosto example. And second of
18:07 Speaker 01 all, how much should we really trust the automatic reviewer? That's a question we actually touched on in the paper as well. We did some analysis and find that automatic reviewer at this point isn't really ready yet, in the sense that if you measure the correlation between AI reviewer and the actual Qmax reviewer, the correlation is pretty low at the point. So maybe we shouldn't trust that much about automatic reviewer at this point. So those are the comments, but, uh, hold on, AI scientist.
18:33 Speaker 02 Hey, we'll continue our interview in a moment after a word from our sponsors.
18:39 Nathan Labenz? Yeah, I like your comment about, first of all, you know, it's always, one of my refrains is like, compared to what? So it's really important that we compare these things to the human alternative, you know, if there is one or whatever is kind of how is it being done now? I think the other thing that people often forget is that like the current thing is not perfect. Often it's actually like quite a bit worse than people imagine it to be. But I think you had a good kind of baseline in the paper on comparing to the inter-rater reliability of actual human judges for the conferences. So somebody might say, you know, the skeptic of the
19:20 Nathan Labenz? human performance, which is often a role I play, might say like, well, you know, how consistent are the human raters and is this any worse than that? And the answer here is, you can fill in a little bit of the details, but the human raters actually do agree more, right, than the AI raters agree with them. There is still something measurably not accomplished with the AI paper evaluator.
19:42 Speaker 01 Right. So at this point, it's true that human reviewers at the conferences tend to have higher agreement than the agreement between AI reviewer and the human reviewer. So I think that's one evidence that current large models are not ready for the automating the reviewing part yet. But even if, let's say, one day large models keep getting better, the agreement somehow gets higher. I think it's still questionable whether we should fully trust the automatic reviewing on such a subjective and pretty high-stake task, I would say. If you look at the actual human agreement, you realize reviewing is really a very subjective task in the sense that even the expert reviewers at top conferences like ICR or Europe's, their agreement is still pretty low in general. So I
20:27 Speaker 01 think it really highlights the challenge of evaluating research ideas in general.
20:32 Nathan Labenz? Yeah. So given, I'm looking at the paper, this is table 11 in your paper. So you basically are comparing four main results. One is, and I guess, first of all, just to be very clear, like, the question here is accept, not accept? Does it boil down to that binary judgment?
20:51 Speaker 01 The question is actually even simpler setting than that, in the sense that we basically try to get the lower 50% of papers, and then we combine that in your binary judgment of something like accept or non-accept. And then we measure the decision agreement between two sets of reviewers.
21:07 Nathan Labenz? Gotcha. So random is 50% in this setup. The NERIPs judges only got up to 66%. So they're agreeing, not one out of two, but two out of three times. The ICLR judges got up to 71.9%. So they're getting close to three out of four, but still not quite there. And then your implementation gets to 56% agreement with the human judges. So it is an incredibly noisy thing, but there is still like a meaningful delta between what the AI system can do and what the humans could do when it comes to this specific task of judging the papers.
21:50 Speaker 01 Right.
21:52 Nathan Labenz? Okay, cool. Well, let's go back to the hypothesis generation. Give us the setup for how you sought to answer this question. You already said it, you know, it took months to build the agent, a lot more months to build the dataset from humans. But give us the whole setup of how you set out to answer this question of can LLMs generate novel research ideas?
22:12 Speaker 01 Right. So given the whole question, I think there are two high-level principles that we want to achieve, right? The first thing is, what do you set out of a proper human baseline? Of course, without the human baseline, you can still measure some numbers or some metrics. But people might ask, what do those numbers really mean? We want to concrete benchmarking point by comparing to a reasonable baseline there. I will feel like, you know, PhD students, we really represent the experts in this task. So we'll compare to that for sure. And then the second thing is also, we want to set up proper design protocol to avoid any potential confounders. And we want to have expert reviewers to do all the judgment. Because I mentioned a bunch of caveats of using large models for reviews.
22:55 Speaker 01 So all the reviewing is done by expert researchers in this case. So the high log design is then, what do you have? One branch where all the ideas are generated by AI. And another branch where all the ideas are generated by human researchers. And one to have a blind review where expert researchers come in and, you know, score all the ideas without knowing which one's which. And then we compare the scores. But in the actual event, there you can also adding a certain condition, which we call the AI ideas plus human re-ranking. The consideration of that is that we have a built-in ranking part in the idea generation agent. Where the agent will generate thousands of ideas on each topic. And then the agent will automatically select the best ones. It's a little bit similar to what you mentioned in the Sakana paper.
23:39 Speaker 01 So after this ranking, we get the best ideas generated by AI, as ranked by the AI itself. Those ideas will be the condition of our AI ideas. And then we realize that the AI ranking isn't really perfect yet. So it's possible that among all the couple thousand ideas generated by AI. There could be some really good ones, but they're not being ranked as top by the AI. Because the re-ranking is not working perfectly yet. So want to have a better estimate of the upper bound. Want to find the best ideas being generated by AI. And that's where we manually actually look at all the generated ideas and select the best ones that we consider as the best. And that's what we call AI ideas plus human re-ranking. So in that condition, all the ideas are still generated by the AI agent, but the re-ranking is done by humans.
24:23 Speaker 01 So that is the certain condition. So that's extraordinary setup. We'll have one review, we'll assign ideas to all the expert researchers. We'll score ideas along different metrics, including loyalty, excitement, feasibility and effectiveness. And then we'll compare the scores of different conditions. And that's the whole design of the experiment.
24:44 Nathan Labenz? Gotcha. So let me just say that back to you and make sure I've got it all right.
24:47 Speaker 01 Yeah.
24:47 Nathan Labenz? So there's two sources of ideas, obviously the humans and the AIs. You pay the humans for their ideas. They were paid $300 for participating. And if I understand correctly, there was also, I think, a couple of thousand dollar prizes for
25:08 Speaker 01 Yes.
25:08 Nathan Labenz? ideas that were judged to be the best among the humans.
25:11 Speaker 01 Yeah.
25:11 Nathan Labenz? So the participants are mostly grad students, PhD students in machine learning, right? With a focus on natural.
25:20 Speaker 01 Mostly LP. Yeah. Yeah.
25:22 Nathan Labenz? So, yeah, I think that's notable for a couple of reasons. Obviously, you know, certain amount of expertise there, that's definitely, you know, not nothing. And the $1,000 is not insignificant on the scale of a grad student salary. So presumably people would, you know, at least be somewhat motivated to give a good idea and try to pick up the extra thousand bucks. Then on the AI side, I think there's some really interesting implementation details in the paper. Maybe you can give us a little bit more on like, how you're prompting the AI for ideas? I know that they're, you know, because you're not just asking like, a simple question like, hey, can you give me some NLP research ideas?
26:06 Nathan Labenz? You're actually bringing a sort of kernel set of ideas to it for it to expand on. Maybe give us a little bit more on that.
26:16 Speaker 01 There are a couple of things that we put in the prompt when trying to generate ideas. So first thing is we retrieve real estate papers, given the research topic. And then we'll put all those related papers in the prompt to give some inspiration there. Another thing is we give some demo examples. We actually manually constructed six demo examples based on existing papers. So those will be exemplar papers that we put in the prompt. And then all examples will follow the same format. So we define a template for how the idea should be written up. So the template roughly includes things like, there's a motivation, there's a problem set up, there's a proposed method, and there's a. And then there's also test examples, et cetera, et cetera. So it's a very detailed template for writing the idea.
27:00 Speaker 01 So we provide the template to the agent as well. And then in the prompt, we also include all those demo examples, include all those greater work. And then we generate ideas in batches, because we won't do, you know, massively over generate a number of ideas and select the best ones. There's a limit to how many tokens you can possibly generate in one generation. So we actually generate since in batches. So we generate five ideas in a batch and then we keep generating. And one issue with this thing is that it's possible that the agent will start repeating itself by generating the same ideas over and over again. And that'll be bad for us, because we want one, a large set of very diverse ideas, so that we can possibly find, you know, better ones there.
27:39 Speaker 01 So one thing we did is we will actually add all the previous generated ideas titles into the prompt. And then ask the model, you shouldn't be repeating what's already being generated. So that's one simple attempt on reducing repetition there. But as noted in the paper, there's still a lot of repetition in the idea generation thing. We do some analysis of that, we can touch more on that later. But on a high level, that's the idea generation part. So we have the retrieval part, we have the template, we have the demo examples there. And after generating all the examples, we have the ranking thing to select the best ideas there.
28:15 Nathan Labenz? And this is with Claude three five Sonnet as the
28:18 Speaker 01 Yes.
28:19 Nathan Labenz? main model. Did you
28:20 Speaker 01 That's the base model.
28:22 Nathan Labenz? Is there a, any sort of guidance you could offer on like, how much, I presume you chose that because you found it to be best? Is there like a, a, a measurement versus like G P T four O or versus Gemini one point five? That, you know, you could kind of help us calibrate how much better three five Sonnet is?
28:42 Speaker 01 Right. So we were initially actually using G P four four O, I think, back then. And then I heard a bunch of people in the lab are saying, you know, call two point five is really good on all those science and reasoning and coding on the task. So I tried call two point five. It's difficult to give a quantitative number on this, but we look at the two sets of ideas generated by GPT-4 and also GPT-5. And intuitively, based on our judgment, it feels like GPT-5 does have some advantage in the task of generating ideas. That's why we stayed with GPT-5 for the entire pipeline. I know Ova just released O1, and I heard O1 is supposed to be really good on those recent intensive tasks. It's possible that O1 might be even better. We haven't done any benchmarking on that yet, but it's on my list.
29:28 Nathan Labenz? Yeah, I'll be very interested to see the result of that. Okay, so going back to the core of the setup.
29:35 Nathan Labenz? Examples are put in, structure of the format of what an idea should look like is included for the AI to work from. It's also given the ideas that it previously generated and told not to repeat. It nevertheless does repeat quite a bit, but enough new ideas come out that you can... Tell us about the ratio. If I remember correctly, you generated like 4,000 ideas, but found that like only a couple hundred were like truly original?
30:07 Speaker 01 Yeah, around 200 ideas are like non-duplicates. So basically, we use semantic similarity to judge whether a pair of ideas are too similar to each other. And we will filter this out. So after you do this filtering, you get like 95% of the ideas out to be duplicates. And we actually have a curve in the paper showing that in initial batches, most of the ideas will be novel. But as you generate more and more in the later batches, most of the ideas will be just repeating previous ideas. So if you calculate the total number of non-duplicate ideas, the number starts to saturate in the later batches. And eventually, it goes around 200 or so among the possible generations.
30:46 Speaker 02 Hey, we'll continue our interview in a moment after a word from our sponsors.
30:51 Nathan Labenz? So how do you think about that? I guess for one thing, it looks like a logarithmic curve, right? It looks like, but a logarithmic curve doesn't, it's not a flat line. And maybe you, I don't know if you pushed this any further, but you know, beyond 4,000 ideas, like, should we still expect to get some original ideas, albeit increasingly infrequently? Or do you think that you kind of hit a point where it's like, it's just not going to come up with anything new anymore and it kind of is what it is beyond a certain point? That seems important to me because like, we could, you know, put, we can obviously pump a lot of tokens through these models and it's like, even if it is logarithmic, you know, with a budget, you could get there.
31:35 Nathan Labenz? But if it truly kind of flatlines and just doesn't really do anything new beyond a certain point, then that's kind of a qualitatively different situation. So what do you think the reality is?
31:46 Speaker 01 Yeah, so this is a tricky question because there are two factors here. One possibility is that, given research topic, maybe the nature of the problem space is that there are only a finite number of ideas possible within that space, especially if you give a very specific research question, that's possible. Another possibility is that maybe the problem space is much bigger, possibly maybe even infinite, but the model has a very low upper bound on the number of different ideas it can generate, then this will be a huge issue, as you pointed out. But I think this is still open for debate in the sense that we actually didn't push that hard on the generation part, in the sense that what we did to avoid repetition is very simple. It's just pending the already generated ideas and asking the model to not repeat.
32:30 Speaker 01 I can imagine smarter ways of doing this. I can imagine ways that you can possibly, let's say, find to the model to increase diversity, you can possibly change the decoding methods in some smart way to increase diversity. If you tried all those things really hard and still see an upper bound, then I think it's evidence that maybe the base model just has an upper bound on the idea diversity. And that would be a huge issue that I would agree with. So yeah, that's, that's my take on this.
32:59 Nathan Labenz? There's also the question too of just like how, you know, the real eureka moment is like zooming to some place in a vast search space that would have taken forever to find through brute force search that actually happens to be like a really good idea. And that is obviously hard. Not many humans can do it, right? And humans tend to only do it in like very narrow domains where they have like deep expertise. So this is a hard thing to do. With the AI ideas, you know, one wonders, and I guess this is kind of where the evaluation will come in, but one wonders like, okay, let's say you turn up the temperature or apply these different decoding methods or
33:43 Nathan Labenz? jam all the levers that you have to try to increase diversity. You can presumably increase diversity, but how does that diversity relate to quality? And do we have any way to determine other than like the, you know, I think your approach here of actually using humans to do the evaluation is really good for creating a grounded baseline that people can hopefully have some level of agreement on. But it's obviously really hard for, I would assume this is like probably why you haven't been able to rerun the thing with O1 yet, right? Because you could generate a lot of ideas really quickly, but now it's still going to
34:23 Nathan Labenz? take a lot of time and effort to do the evaluations. So when you do turn up the diversity, do you have an intuition or do we have any metrics for sort of how pushing these like temperature and other, you know, parameters up impacts the quality of the presumably more diverse ideas generated?
34:44 Speaker 01 Yep. So I think one analogy here is through some works on inference scaling. So for example, on stuff like code generation, there are experiments showing that if you sample more diverse solutions, the success rate, like the passing rate of the solutions increase. And there's a nice scaling curve there where the more you generate, the more likely that one of your generations will be correct. And then your task will be essentially to find the correct ones. Among other generations. So that's the analogy there. We didn't do the exact same thing on idea generation. In the sense that, like you said, unlike code generation where you can verify the correctness of the solution. And that means that you can actually know the so-called success rate in all your generations. In our case, you leave options really tricky.
35:27 Speaker 01 But still, I think we can build proxy rankers like we did in the paper. The ranker may be not perfect. In fact, it's definitely not perfect, as we showed in the paper. But still, as long as there's some positive correlation that the ranker is able to tell really bad papers from really good papers. Then I think it's reasonable to expect that if I scale up the direction and the number of ideas that I generate. And this ranker will possibly, you know, find better ideas. And among all the top-ranked ideas, I might expect a quality increase as I generate more ideas.
35:59 Nathan Labenz? Yeah, that reminds me quite a bit of some work that I've done on using a number of different kinds of models to try to evaluate the aesthetic appeal of images. And we found a similar thing there where the human-to-human consistency of judgment was fairly low. The human-to-AI agreement was even lower, but not zero, you know, it was better than random. And the general guidelines was, you know, the general kind of takeaway of that work was like, you can definitely tell the difference between the best and the worst, and there's clear signal there. But especially in the middle,
36:40 Speaker 01 Right.
36:41 Nathan Labenz? you really can't, you know, the fact that one was rated higher than the other is not very meaningful.
36:46 Speaker 01 Right.
36:47 Nathan Labenz? So you have like a, you know, a general direction, but at any given point in the list, like, it's definitely up to interpretation.
36:54 Speaker 01 Although I will add one comment here that at this point, it does matter that we want to have really good ranker or evolver to find the best ideas. But, you know, if we think bigger and we think about the big picture of automated research in a next stage, it's possible that we might start working on automating the execution part of things. Where we might build agents that can automatically execute the ideas in the experiments. That means the agent will tell us. Our implemented method or task on the specified data set. And I will tell you how well it does as compared to the baseline. And that means I can actually verify whether the proposed idea works or not. And that means for all the general ideas, I'm willing to spend all that inference compute money.
37:36 Speaker 01 I I can just implement all those ideas automatically. And then I can automatically verify each idea. So in that sense, it turns into a setting almost like Q generation where I can actually check the correctness of each possible idea. As as in that setting, we'll have some nice inference getting there.
37:54 Nathan Labenz? Yeah, that's a great point, first of all, and it reminds me of the AI scientist paper where they reported, I think it was like in the low tens of dollars per paper generated. And so it was like, you know, okay, sure, they're not that great. But if one in a thousand is great, then it's like clearly economically competitive. And the prices are obviously falling quickly. So
38:24 Speaker 01 Yeah.
38:24 Nathan Labenz? It seems like that. I I remember talking to somebody once at one of the leading frontier model developer companies, one of the big three. And the person said, we find over and over again that people just don't have an intuition for what happens when you're willing to spend a lot on inference compute. They're in, you know, we're obviously OpenAI has kind of now productized this to a degree with the O one family of models. But even even before that, just people try one, you know, they try two, they try three. It it's just not intuitive to think like, what if we tried a thousand or a million, you know? And that really can change the game.
39:08 Nathan Labenz? So I think that that is quite important. Okay, let's go back. Let me just reset the stage one more time because I know I keep going on all these rabbit hole regressions. So. You pay grad students for ideas, you give them the promise of a prize, you run a reasonably high volume of idea generation, you de-dupe the ideas. Is that with an embeddings approach? Or what's the de-duping method? That's embeddings?
39:32 Speaker 01 Yeah, yeah.
39:33 Nathan Labenz? Okay, then there's two ways that you can rank the ideas to try to bring the best AI ideas to the top. Those are having the AI do it directly. I believe you use Claude with pairwise head-to-head kind of like tournament style for that?
39:51 Speaker 01 Yes, yeah. The pairwise conversion thing is what we found to work much better than directly asking the model to give a score. Let's say on a score of 1.10.
40:01 Nathan Labenz? Yeah, that's a really good tip and something I think that is a practical takeaway for all sorts of AI application developers in the audience. And I actually there was a really good, this is a very in-the-weeds detail, but what I just said about tournament style is not quite right, actually. I had previously been thinking of doing something like this in a, in a project and I was thinking like almost setting it up like an NCAA, you know, March Madness bracket where I would have the field gradually narrowed. But of course, that has the problem of like, well, what if your best two ideas meet in the first round? And, you know, so then you're like, well, maybe I'll have a double elimination tournament. But then you could still have similar problems. So I think what you did was just take random pairings and have each idea paired with five other ideas
40:46 Nathan Labenz? and then just sort them by their win-loss record, right? So five and O would be at the top, four and one would be next. Is that right?
40:52 Speaker 01 So instead of random pairing, what we did is basically we paired each idea with another idea with the closest accumulated score so far. So the ideas I want to, let's say we want to pair the top-scoring ideas so far with other top-scoring ideas, for example. And you're right that it's different from typical tournament in the sense that we don't have elimination problems. Because otherwise, you run your problems where two really good ideas meet in the first round and one of them got eliminated and they would only get a score of zero in the end. And that would be really bad.
41:21 Nathan Labenz? Yeah, okay, cool. So that's even one level smarter than I understood. Pairing ideas with others that have the same score so far to try to put the most comparable ideas together.
41:35 Speaker 01 Right.
41:36 Nathan Labenz? Yeah, you can sort of, it's almost like bubble sort style. Or at least that's the visual that comes to mind for me. Okay, so that's cool. That works, but you did find that even better is human re-ranking. Anything we should know about the human re-ranking process?
41:51 Speaker 01 Uh, it takes forever. I actually manually looked at all the ideas directed. And then I didn't look at the AI ranking. I just, you know, did my own ranking of it. And now I compare my own ranking with the AI ranking. And it's somewhat surprising that there are a couple of ideas that I saw be clearly the top ideas that are not really captured by AI as a top idea. So I think the discrepancy here is interesting.
42:14 Nathan Labenz? Okay, so you personally, individually are the human re-ranker. Is that right?
42:18 Speaker 01 Right, right.
42:19 Nathan Labenz? Yeah, okay, interesting. So then I think the next thing is a pretty interesting detail. And I guess we should also set the stage, and I'll put this in the intro as well, so people have this context. The field of study that we are working on here is essentially prompt engineering. It is how can we get the best performance from language models via different prompting techniques? So that's just background baseline. All the ideas all across the board are within that domain. Now, once you have the human ideas, the AI ideas ranked by the AI itself,
42:58 Nathan Labenz? the AI ideas ranked by you, then you put them through a sort of anonymizer type of thing, right? You basically have the AI rewrite the idea. And if I understand correctly, this was done to try to create an even playing field for, with respect to the human judgment. I guess the worry was the hu- human readers will likely be able to tell the difference between a human-written idea and an AI-written idea. Even though we know that like AI, uh, large language model detectors in general don't work. And you know, if you're a teacher and you're in a classroom, like you shouldn't put too much stock in them. With that disclaimer, it is reasonable to expect that especially people that work in the field would have an intuition for like, yeah, this kind of reading.
43:43 Speaker 01 Right.
43:44 Nathan Labenz? How often is the word delve being used, et cetera? So you put just the human ideas through this rewriter? Or did you put the human?
43:51 Speaker 01 Everything.
43:52 Nathan Labenz? Okay, yeah. So yeah, tell us, give us that bit of the setup.
43:56 Speaker 01 So we pretty much specify the desired writing style and format of the idea. We try to have the writing be more academic and then we specify every single component that the idea should have. And then that writing agent essentially does some minor paraphrase to all the different ideas. We set very explicitly in the prompt that you can only do like writing style kind of edits, definitely not editing any of the original content. So most of the edits are basically changing certain word choices. And I have to say, I think this is very important in the sense that if you look at the original ideas submitted by humans, some of the word choices can be very obvious that are just written by humans. Like some of the sentences tend to, for example, some sentences are not grammatical and that pretty much is a give it away.
44:41 Speaker 01 And some people like to use more verbose writing styles, almost like they're talking to you, more engaging writing styles. And that just feels unlikely as from AI. Those things don't really affect the content that much, but we don't want this to bias people that, let's say, for example, some people just like humans better than maybe if they figure out that this written by humans just have a more engaged writing style and then maybe they would favor those ideas. We want to avoid these cases. So we do the style transfer thing for every single idea, including both AI ideas and human ideas. And the hope is that we would avoid all those components there.
45:19 Nathan Labenz? So I think your paper definitely did generate quite a bit of discussion online. This was one of the points of criticism that I think people most focused in on when they were like trying to find a reason not to believe the result. Maybe we should tell the result, then we can kind of come back and dissect the criticism.
45:37 Speaker 01 Sure.
45:38 Nathan Labenz? The final step after doing this rewriting is then to send all these ideas. And these are now like, what, like 500 words? It's like a page or two pages, like reasonably well fleshed out. A thousand words.
45:53 Speaker 01 Why it's all them work, yeah.
45:54 Nathan Labenz? Yeah, okay, so not an insignificant sketch of the idea. And these are then sent to human evaluators who have to score them. And they are scored on four different dimensions, right? They are novelty, excitement, feasibility, effectiveness. And I guess overall, kind of just aggregates those previous four scores, right? So.
46:17 Speaker 01 Uh, we will ask a separate overall score, but that's the idea.
46:20 Nathan Labenz? Okay, gotcha. So five different dimensions. So and the and the humans give each paper five number scores, one to 10. One the worst, 10 the best. How original is this idea? That's novelty. How exciting this excitement obviously is kind of subjective, but that's basically the like. Does this feel like a really good idea?
46:39 Speaker 01 Yeah.
46:40 Nathan Labenz? Feasibility, obviously being. Maybe you can give a little more color on kind of the guidance that you gave for that, but.
46:46 Speaker 01 So so for feasibility, we are really asking whether this idea can actually be executed. So for example, some of the not feasible ideas would be things like something that require tons of compute. Like you want to train GPT five. That doesn't feel that feasible. Or you are extensive human effort, like you want to on high level or huge training set. That won't be that feasible or things like that. Or you want to, you know, a fine tune, a close model, like you want your fine tune. Some general model says doesn't really have fine tune access, for example. So things like that. So that's what feasibility and that's different about effectiveness in a sense that in effectiveness, we're asking the question of, given the proposed method, do you think the proposed method will actually work better than existing base size? So that's the difference here.
47:29 Nathan Labenz? Gotcha. So novelty, excitement, feasibility and effectiveness. Feasibility is like, don't go too crazy on the budget. I wonder what would happen if you put this paper, your paper through the evaluation process? How do you think you would score on these different?
47:43 Speaker 01 Low feasibility.
47:45 Nathan Labenz? Yeah, you'll never get the budget for it. The total budget on this is is what? And how does it break down? By the way, I mean, I'd be interested to know kind of total spend and also, yeah, just what percentage of that went to the humans versus what percent went to the AIs?
48:00 Speaker 01 Yeah, the payment for humans in total is about thirty K dollars. And then for AI, so generating four thousand ideas will take a couple thousand dollars. And then we did that for seven different topics. So there's some money there. And then there's also some development in the early stage where we, you know, prototype a bunch of different agent systems. So in total, I would say maybe the money that we spend on API credits will be slightly more than what we actually paid humans. Or in our case, so that's the money breakdown there.
48:30 Nathan Labenz? Interesting. Okay, that's kind of surprising. Maybe let's dig into that just a little bit more. So what is consuming the majority of the token budget? Because I did a little back of the envelope math and I came out to a lower estimate of what I thought you had probably spent on the APIs.
48:47 Speaker 01 Yeah, the problem with the idea generation part is first of all, we have to generate all the original ideas first and then we do the duplication. That means if you want to get, let's say, two hundred ideas on a topic, you have to actually generate four thousand ideas. And initially, we're doing this almost like skin locking where we actually over generated more than four thousand ideas. We tried to test the limit there. So we actually generated a whole bunch of like AK or even ten K ideas on each topic and did some analysis there. So there's some hyper parameter tuning there. So there we spend a bunch of tokens. And also you do realize the fact that the problem is very long. So the input token is a lot. And then it gets longer and longer as we prepare all the previous general ideas in the front.
49:30 Speaker 01 Another thing is in the final step, we also have this process of a novelty filter. This part is actually not in the main paper, it's in the appendix. Novelty filtering is similar to what people have done in the past in the sense that for each generative idea, we will actually retrieve all the related papers and then we'll compare them, the general idea with each of the related paper and ask the model whether it is too similar. And that actually is pretty costly in the sense that you have to. Do this comparison between the general idea and each of the retrieved papers. That also costs a bunch of money. So in the end, you can of course simplify the pipeline by only do the generation part and then ignore the ranking and filtering part. That could save you some money. But I would say a lot of the money is really spending on doing the over generation part.
50:14 Speaker 01 Where most of our ideas are actually being duplicated and are being filtered out. And then a bunch of money spent on novelty check and filters.
50:22 Nathan Labenz? Gotcha. Okay, interesting.
50:24 Speaker 01 But in general, I would say to get one valid non-duplicate idea, you probably need to spend a couple of dollars. That's the skill of how much money you need to spend.
50:34 Nathan Labenz? Okay, cool. So we can finally, I think, get to the headline result, which is that when you have humans evaluate the AI rewritten human ideas versus AI ideas versus human with your individual reranking ideas. The AI ideas are scoring higher and statistically significantly so for both novelty and excitement. Roughly the same with feasibility. It looked like the humans are a little bit higher score there, but not necessarily statistically significant. Roughly the same, a little bit higher on effectiveness
51:15 Nathan Labenz? and higher overall, and especially with the human rerank, your rerank of the AI ideas. Then you see like a notable difference in the overall score. But novelty and excitement definitely stand out to me as like the two of the four that I'm looking for most as somebody who's trying to figure out what the future of the field looks like. So that is a pretty big deal. What additional color commentary would you give us on those results?
51:45 Speaker 01 Honestly, I was also a bit surprised when I see AI ideas are getting higher novelty than human ideas when I first saw the results. And one additional comment I want to give is that numbers are just one thing. Like there are numerous ways that you can possibly manipulate your numbers to tell the story you want. So I just want to say I personally look at. Every single idea being generated by AI and also submitted by humans. And I think I have a reasonable sense of how they look like. And I think I do see how the actual ideas support the numbers that we are seeing here. Uh, I think if you look at uh all the ideas um being generated by AI, you would realize uh you have this interesting vibe where uh they tend to be
52:28 Speaker 01 you know, almost weird in the sense that they're not similar to the typical ideas you will see being published these days. It's maybe less grounded in existing work, but more of the box in general. And I think I would categorize that as a pop novelty. So if you look at all the data, I think uh it does make sense afterwards that AI ideas are getting better novelty there.
52:50 Nathan Labenz? Okay, that's really interesting. Could you go a little deeper into just personal observations? I mean, even if they're anecdotal. And by the way, speaking of anecdotal, one question that I asked and was definitely interested to see the answer on was, okay, maybe the average score is higher, but what does the standard deviation look like? You know, because you could imagine a higher average, but you know, a sort of narrow band of performance. And that's often, by the way, how I tend to characterize AI performance for people when I'm just trying to educate people about like what AI can do and what it can't do. I often, as a rough guideline, say it's gonna be, you know, it varies dramatically on different tasks, right? Obviously, you have like these sort of areas where AI is superhuman, like
53:36 Nathan Labenz? it can translate from any language to any other language, like that's amazing, no human can do that, so in that way, it's kind of superhuman. At the same time, like they're very bad at common sense spatial reasoning in general still and like don't know how you should stack objects on top of one another. My general guideline, though, for people is it's gonna be pretty consistent within a task. It might be really good at some tasks, really bad at other tasks. But in general, it's gonna be pretty consistent within a task. I was struck by your result that the standard deviation of the scores for the AI ideas versus the human ideas was basically the same.
54:13 Nathan Labenz? And also that, in fact, with excitement, it was a little wider, the standard division was a little greater for the AI ideas. And also that the max score was higher for the AI ideas. And like I'm reminded of Tyler Cowen, who says, you read people for their peaks, like what you contribute to the world may, in some very practical sense, boil down to your very best idea. And it might just be your one idea that was like your, you know, the actual lifetime contribution. If you believe that and you look at the max score, then you're like, huh, it's also quite striking that the max score is higher for the AI ideas.
54:52 Speaker 01 Yeah, I can give you one more data point there. If you think, only look at the single one max score is just one sample size. I actually did this calculation. So I merge all the ideas and then I rank all of them by the overall score. And I look at the how many of the top 10 ideas are from human versus AI. And I can tell you, among the top 10 ideas, nine of them are actually from AI and only one of them is from human.
55:17 Nathan Labenz? Wow, okay. That's interesting to say the least. What else? Keep going. I mean, I could happily listen to any and all other observations that you have. As somebody who just obviously spent a lot of time looking at the data. This is a good reminder too of our mantra for all this kind of work. Look at your data. So you clearly did that. What else comes to mind that we should be thinking about?
55:41 Speaker 01 Yeah, I think one thing that maybe people didn't notice that much is I was closely monitoring the whole review process as people submit their reviews. And I think it's very interesting to see that, first of all, there's big review disagreement, as we noted in the paper. Second of all, I think sometimes there is some randomness in like which ideas, like some reviewers have certain preferences. And you know, let's say they tend to favor certain types of ideas. So there's this factor in there where certain reviewers, or let's say they would just systematically give higher scores to the ideas that are given to them because they like the topic, for example. And there are certain cases where certain reviewers just give low scores overall.
56:23 Speaker 01 One thing I was observing in the process and I was like, you know, that's really gonna impact the way we make conclusions because we want the conclusion to be based on the actual quality of the idea rather than, you know, maybe reviewer have biases and then different reviewers just have different calibration in general. So that's why we come did a lot of different tasks in the paper like the first table that you see is really just treating each review as an independent data point. But then I realized there could be all this reviewer biasing, there could be all sorts of different components. That's why we did two additional tasks to try to account for all those possible biases. And in the end, I think we realized the the difference between AI and human ideas is big enough such that you get significant difference
57:08 Speaker 01 no matter what kind of task you run. And that's why we are making that conclusion there. So back then, I think what we agreed with my advisor is that we're gonna run all the possible statistical tests we can think of to account for all the potential components there. And even one of the tests that this conclusion doesn't hold, then we won't be making that conclusion. So that's the bottom line there. And then the novelty aspect is just different enough such that it's hold robustly across all the different tasks. So that's one thing from the results section.
57:38 Nathan Labenz? So each idea gets three human evaluations, is that right?
57:45 Speaker 01 Average is on three, but at least two, almost four.
57:48 Nathan Labenz? Gotcha. Okay, so two to four with average being three. And then when you talk about doing multiple different statistical analyses, the main two that I saw were, one is treating each score as its own data point and measuring that way. And then the other was taking the average score at the ideal level and then treating that as a single data point.
58:16 Speaker 01 And there's another thing where we actually analyze based on each individual viewer. Because like I said, I observed the same during the review process where some reviewers, let's say they just really hate prompting. And they might give two or three to all ideas to review. Some people, they might really like prompting and, you know, they might give like seven or eight to all ideas they reviewed. And then it becomes a problem of whether you happen to have assigned more AI ideas to those people or not. And that's really not what we want to capture. So what we did is for each individual reviewer, we look at all the ideas they reviewed. And then we get an average score for the AI condition and the human condition. And we compare the difference between them. So for each individual reviewer, they should have consistent standard when they are reviewing different ideas.
59:00 Speaker 01 So if every single reviewer is giving AI ideas better scores than human ideas. Then that says that, okay, this thing has robustly, despite all the difference between different reviewers. And that's the third test we're doing there.
59:13 Nathan Labenz? Yeah, gotcha. So just to restate it one more time, you've got three ways of analyzing the data. One is every single score from a single reviewer for a single idea is a data point. Analyze it that way. Version two is take the average by grouped by idea and then analyze them that way. And then the third is aggregate at the level of the reviewer. Where you basically say, okay, this reviewer A had this average score for AI ideas, this average score for human generated ideas. And now we'll use those as the the data points to then do analysis on.
59:55 Nathan Labenz? And the bottom line is across any of these different lenses that you want to put on data. You still see a statistically significant edge for the AI ideas, particularly in novelty and excitement.
1:00:09 Speaker 01 Yeah, that's a quantitative result we have. I think one additional comment I can add there is that I'm too pretty strongly that I support the conclusion on the novelty part. But some people could argue maybe being novel isn't, you know, the only important factor there for producing good research ideas. Like I said, some of the AI ideas, they do tend to look more different than what we typically get when reading all these papers. But the question is whether that's a good thing, where you can argue. AI ideas could look very different, could be very novel. But then they turn out to be just weird ideas that just don't work at all. That's totally possible. And the current evolution paradigm has just evolution where we are asking the question of, do you think this idea is gonna work?
1:00:54 Speaker 01 Or do you think this idea is feasible? And based on the current evolution, AI ideas are slightly worse on feasibility, but some were similar on effectiveness. So that's not too bad. But you could of course argue that this evolution is bad. Because you are essentially asking people to predict whether this idea is gonna work, which is an incredibly difficult task, even for expert researchers. And it's highly subjective. But this is something that we could possibly actually measure objectively. Where the way is you actually implement the idea and then you can see whether the idea actually works or not. So that's the current project we're working on this color right now. We are, we are actually recruiting a bunch of people to execute all the human AI ideas into food products. And that we can verify whether those ideas actually work or not.
1:01:39 Speaker 01 So that is trying to address this concern that do those weird looking AI ideas actually work in practice? So I, I don't know the results yet. I think either way it's gonna be fun. I think if we get the conclusion that those AI ideas did not just look novel, they actually also work in practice. Then I think that would tell a pretty strong story.
1:01:59 Nathan Labenz? Yeah, I mean, if, and if it's the reverse, there's still something like very interesting going on. Where
1:02:05 Speaker 01 Yeah.
1:02:06 Nathan Labenz? Especially because, I mean, we've covered all the statistics, but again, you also said nine of the ten highest rated ideas were AI ideas.
1:02:14 Speaker 01 Hmm.
1:02:15 Nathan Labenz? It would seem strange if like, I mean, that would certainly suggest that like the humans, I guess it would kind of call into question like a lot of the human evaluation, right? If all of a sudden, like despite this sort of pretty clear signal that the humans expect better from the AI ideas, if that were to actually be reversed, it would be like, yikes. Yeah.
1:02:37 Speaker 01 Yeah, I think that would tell us not only about the characteristics of AI general ideas. It also tells a lot about the evolution protocol that we should actually do. I think, you know, if the reverse is happening when we finish the execution part. I think that's a interesting challenge to whether even the human evaluation is reliable at the idea stage or not.
1:02:59 Nathan Labenz? Yeah, that does remind me of one comment I saw online that I did think was potentially perceptive or sort of something that would warrant some digging, and maybe you did this digging, but the idea was when a language model is prompted to generate a novel idea, there are different ways that it can sort of satisfy the user with that request. And this ties into like a common theme of this show over time, which is that RLHF is like, you know, obviously very powerful, has worked really well, but has some pretty fundamental problems. One of which is that the language models seem to be starting to understand
1:03:41 Nathan Labenz? that like pleasing us is a bit different than actually like doing something in reality. So, and this is why we see like sycophantic behavior out of Claude and others. And this is one of the reasons people worry about deception from language models, you know, long term. But in the specific context here, you might worry that, and presumably the rewriting would like try to help fix this, but maybe not entirely, but you would worry that the AI might be really good at using sort of novelty indicators, you know, kind of words and phrases that sort of suggest novelty, even if the underlying idea maybe isn't actually so novel or even coherent necessarily, but just, you know, ways of
1:04:25 Nathan Labenz? presenting that the models have learned will score well with humans. How do you assess that possibility? Again, it would call into, I mean, if to the degree that that is true, it would be operating by means of tricking the humans. But I don't think we can entirely rule that out, you know, without at least like some digging into the question. So how much digging into that possibility did you do? And what's your kind of current state of understanding of that question?
1:04:54 Speaker 01 Yeah, so my intuition back then was that we're using the exact same, you know, paraphrasing from to standardize the style of every single idea, including AI and as a human idea. So the hope is that the kind of running style of work charts will be similar across competitions. But it's true that some ideas, they might have more novel something words in the beginning, and then they didn't really get filtered out by the paraphrasing. So I think we didn't really a hundred percent rule out this possibility that maybe the AI ideas would contain some more novelty, something worse there. I think we have some plans to do some follow-up analysis on the work choice and stylistic analysis thing. I think that will tell us something interesting.
1:05:37 Speaker 01 But still, I think the only possible way to totally address that thing is by actually finishing the execution studies thing and tell us whether AI generated ideas tend to work or not. Like, that's the most objective thing that will possibly address any concerns about the subjectivity in the version. Because in the end, I have to agree that there is a strong level of subjectivity in the current way that the ideas are being evaluated. So that's my current status.
1:06:06 Nathan Labenz? Were there any ideas generated by the AI that you felt were like just amazing? I mean, some of these ideas that got scored like a ten on, let me just go back to the table. So individual scores, the highest novelty score for a human idea was an eight, the highest score for AI idea was ten. For excitement, the highest score for a human idea was eight, and the highest for an AI idea was nine. So were there, if you, I'm sure you looked into that, ideas scored ten in novelty and nine in excitement. What did you see? Like, were those ideas that you were like tempted to go run with yourself? Or, I mean, just how good were those top rated ideas?
1:06:50 Speaker 01 There are a couple of ones that I personally really like. I mean, again, this whole thing is subjective. The ideas that I like the best may not be the ideas that other people like the best, but I personally really like the ideas under the uncertainty topic. Like, we have seven in topics, and one of the topics is on finding methods that can help us calculate uncertainty and measure confidence. And then we have a bunch of AI generated ideas on that. I think most of the ideas under that category look pretty interesting. It's this one example that we had in the paper. For example. It's sort of a much more fancy version of this self-consistency idea where you generate many different diverse solutions given the query. And then what they do is not really simple majority voting, but rather you measure how each solution supports or refuses each other.
1:07:33 Speaker 01 And you kind of construct this graph thing, and then you can measure central out your other graph metrics. And as a better way to measure the uncertainty of the solution for that query instead of just taking simple majority majority voting. So that's something neat. And there's there's a couple of other ideas that we didn't re-pub in a paper that I saw was pretty interesting. Like, if it works, then it would be awesome. So one example is, again, on measuring uncertainty. So idea is, what if we prompt a model to first generate so-called like uncertain examples, like, given this question, if I tell this answer, it has an uncertainty of ten. If I tell a different answer has the uncertainty of nine, and so on and so forth. So we kind of have this mapping of what would be answer with uncertainty one look like?
1:08:15 Speaker 01 What would be answer with uncertainty two look like? And extra, extra. And then when given a new query, I generate response, and I basically compare the response with the mapping thing. And I say, you know, this closest to a uncertainty five answer, so uncertainty will be five, something like that. So really weird in the sense that it's quite different from any of the other uncertainty estimation methods that I'm seeing, but, you know, uh, sounds interesting. We didn't really test whether that'll work. That'll be for the next stage of study. But if this very weird looking idea is actually work, and then I think that'll be pretty awesome.
1:08:47 Nathan Labenz? Yeah, it's interesting. How did you think about in terms of what to do next? Going down the path of actually executing ideas versus other dimensions of, you know, generalizing this idea like an obvious one would be to rerun a similar experiment in another different domain.
1:09:08 Speaker 01 Yeah, it's of another domain is also possible to, you know, relax the constraint on prompting. That's a lot of people are really criticizing that. You know, we could totally do something where our self just prompting ideas, we allow any sorts of ideas like, you know, I can imagine you can adapt the same idea generation system to the security ideas on a better ways to construct synthetic training data, for example, better decoding methods, better training objectives for alignment as true as a trend. So you can totally do that. I think it's a matter of which direction one you have. I see a couple of things that's on top of my head right now. So one thing is a lot of people are saying the human ideas we collected in this study are not really representing the best human ideas, which is totally true. We actually asked the exact same question and
1:09:52 Speaker 01 the idea writers are saying these ideas represent like around the average of the ideas. So this is really average PhD level rather than top PhD level. So if you want a stronger human baseline there, one thing that we are thinking about doing is there will be another upcoming MLP conference called MLP and we can compare all our general ideas with papers I should accept that at this top tier conference. And then we would have a stronger baseline there where we could assume those accepted papers will represent some higher quality work as compared to asking someone to generate ideas on the spot. Right. So that's one thing. Another thing is the execution thing that I was talking about. Want to address all the subjectivity concerns in the current evolution by testing whether AI general ideas actually work in practice.
1:10:37 Speaker 01 And I think that will give a much more objective evolution in the quality of AI ideas. So this is one thing. And both of those are still on evolving the research ideas. So a bigger step forward would be to complete the other part of the entire pipeline. And that is we have generated all those ideas. Want to find a way whether we can automatically execute each idea so that we can actually verify the effectiveness of each idea. And we did some preliminary attempt on building such an execution agent. I think there's some crucial limitation in the evolution of this agent in the sense that You know, you can have some tricks to have the agent generate a lot of code and the code could run successfully and that it can actually give you some numbers. We put an agent that can
1:11:20 Speaker 01 implement the idea and tell you the baseline number and the new proposed message's performance. But then the problem is, if you look at the actual co-invention, sometimes the model is not implementing the right baseline. For example, it's for a text classification problem, the agent was actually implementing a keyword based on message as a baseline, which is totally unreasonable in this era. So that's one thing. And sometimes the agent might skip some steps when implementing the proposed method. And those things are really tricky because it requires evaluation on the intermittent steps of the implementation rather than just the final outcome. So I think we need to do some more work there to figure out what's the best way to do this intermittent evaluation
1:12:01 Speaker 01 on the correctness of the implementation and also just to improve the agent in general. So that's something that we are thinking about. The hope is we can set up this evaluation of the execution agents so that we can post it as a challenge for the entire community and people can all work on this problem of creating execution agents that can automatically implement research ideas and verify things. You know, if that works out in the end, then we can think of a very exciting combination of idea generation and automatic execution. And then we can scale this up and maybe we can possibly find some best repro level ideas that actually works in practice. And that would approach what I called the Eureka moment for automating research in the beginning.
1:12:42 Nathan Labenz? Yeah. How much of a difference do you think O1 is going to make here? It strikes me that like in terms of accelerating your process, if O1 could get to human level evaluation and you could demonstrate that, then you would like dramatically reduce your cycle time for one thing. So I wonder how close you think that is to happening? And then, you know, just curious for your intuition in general for where you think this next generation of model will make the most impact? And if you have any ideas around where you think it won't make an impact, that would be definitely really interesting too.
1:13:23 Speaker 01 Yeah, I agree with you. I think I really need to try this O1. I think if O1 is able to get better at the automatic matching thing, it can be a great re-rankers thing that will help us with the inference scaling stuff. I think that's interesting. It's also interesting to test whether it has better diversity in the idea generation as well. I don't have any empirical evidence right now to say whether O1 will actually work better. Uh, so, you know, that's an open question that we want to try for sure. I think another interesting thing is also we're doing the prompting in the entire energy generation pipeline, and it seems that it's possible to also try some fine tuning in the sense that we actually have a lot of data, you know, all the academic papers on, on those conferences.
1:14:05 Speaker 01 It feels like we can gather some meaningful data set and it's interesting to see whether smaller language models like Plama after a proper fine tuning would give some reasonable results in this pipeline as well.
1:14:18 Nathan Labenz? Okay, cool. I think we've covered it. Is there anything else that I haven't got to that you think folks should be aware of with respect to this work?
1:14:28 Speaker 01 No, this all sounds good to me. Uh, one last advertisement thing. We're doing this execution study right now. If the audience is interested in participating by actually implementing one of the idea. We, we pay a lot of money. We'll pay a couple of thousand dollars for implementing idea and that would be a great contribution to our study.
1:14:47 Nathan Labenz? Cool. Uh, interesting. I like it. What's your expectation for the future? I mean, obviously we're getting into speculative territory here. How do you think about kind of key thresholds to be hit and how soon do you think they're likely to be hit?
1:15:04 Speaker 01 So I think a concrete threshold is what I said in the beginning. If you can, you have a end-to-end system that can automatically think of an idea, execute the idea as the whole project. And that project I can actually not just get into in top conference, but actually get in paper or what at the conference. That would be, I think, the major milestone we want to hit. I think the easier milestone is to just have the automated paper to get into one of the top conferences. You know, I, I have some politics on this. My hope is it's actually not that far away for that simple milestone to be achieved. Like just getting into a top conference. I feel like even some of the ideas we have right now, I feel like the best ones. I could see them getting accepted at a conference like ACL.
1:15:48 Speaker 01 Like I'm seeing similar shape of research being published there. I feel like we're probably getting there if we nail down the execution path. But of course, that will take some time for sure. But the the really difficult question is the whole agenda, like the ultimate vision is not to just mass produce average piece of research. I think the ultimate hope there is really we are able to have this automated system that can generate this beyond the average PhD level. So I I really think if we are able to at least demonstrate one case study where an automated idea gets implemented into something that can win a best bill award at a top conference. I think that's a you know, a strong sign. I feel like that's gonna take us a couple of years. My hope is that by the time I graduate, I mean, three or four years.
1:16:33 Speaker 01 If by that time we are able to achieve this goal, I think that'll be a pretty exciting. I think that's a aggressive estimates, but that's a goal. Uh, let's see if if we can hit that.
1:16:43 Nathan Labenz? That's right in line with Leopold's situational awareness timelines. Cool. Well, I really appreciate all your time and going into the many, many details and weeds of the implementation with me. Any closing thoughts before we break?
1:17:00 Speaker 01 Oh, sounds good to me.
1:17:02 Nathan Labenz? Alrighty, cool. Well, the paper is, once again, can L L Ms generate novel research ideas? It's one of the most important questions in the world today. Chung Lai Si, PhD student at Stanford, researching the automation of research. Thank you for being part of the cognitive revolution.
1:17:20 Speaker 01 Thank you.
1:17:21 Nathan Labenz? It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.
Previous GELU, MMLU, & X-Risk Defense in Depth, with the Great Dan Hendrycks Next Mind Hacked by AI: A Cautionary Tale, From a LessWrong User's Confession Page updated October 26, 2026 · transcript and summary are machine-generated
