🔍 Read the full analysis: A Practical Guide To Using LLMs For Research Novelty Checks on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Ai2 researchers tested six LLMs as judges of scientific idea novelty and found that results could change sharply with evaluation setup. In one pairwise test, small prompt wording changes shifted accuracy by more than 50 percentage points; retrieval and added reasoning did not consistently help. The team released a paper, benchmark data, code and an evaluation pipeline, while warning that some benchmark labels are imperfect.
Ai2 researchers report that six large language models gave unstable judgments of scientific idea novelty when prompts and other evaluation choices changed, as detailed in the original analysis. In one pairwise experiment, changing a few prompt words shifted accuracy by more than 50 percentage points, raising concerns about using a single model score to filter research proposals; the team released a paper, benchmark data, code and an automated evaluation pipeline.
The study tested GPT-5.1, GPT-5.2 and GPT-5.4, alongside Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. The models assessed ideas using both absolute novelty ratings and pairwise comparisons. The authors also tested whether adding literature retrieval or allowing more reasoning improved judgments; neither change produced consistent gains, according to the study description.
For one benchmark, the team selected 154 ideas from accepted ICLR 2026 papers rated in the top 10% of their research areas, where reviewers praised originality and no reviewer disputed it. The comparison group comprised 145 ideas from rejected papers in the bottom 10%, selected where reviewers raised originality concerns and none praised novelty. In a check of 50 extracted signals, an annotator agreed with the signal direction in 47 cases; the authors say the other three did not change the review classifications.
A second setting compared the 154 human-authored ideas with 154 proposals from a basic language-model generator. The researchers treated the generated ideas as less novel but described those labels as weak and not individually verified. In a separate expert check of 30 pairs, selected largely from cases the strongest judge got wrong, the expert preferred the human-written idea in every pair and judged none of the generated proposals novel. The authors also report that two purpose-built novelty systems performed worse than a simple prompted Claude Opus 4.6 baseline.
Why Research Teams Need Stable Scores
Research teams may use LLM judges to screen ideas before spending time on literature searches, experiments or expert review. If small prompt edits can substantially change a verdict, a score may reflect evaluation choices as much as the proposal’s originality. That makes comparisons between projects or research groups harder to interpret, particularly if teams report results from only one prompt or model configuration.
The findings also matter to efforts to automate parts of scientific discovery. Systems that generate hypotheses or synthesize papers need credible ways to assess whether an idea adds something beyond existing work. The study does not show that every LLM novelty assessment fails; it does show why accuracy on one selected setup is not enough to establish reliable performance. Replication across prompts, models and evaluation formats would be needed before treating such scores as dependable screening evidence.
Products Worth Considering
DJI Osmo Mobile 8 3-Axis Smartphone Gimbal Stabilizer
Sony WH-1000XM4C Wireless Noise-Cancelling Headphones, Black
Logitech MX Master 4 for Mac, Ergonomic Bluetooth Mouse with Haptic Feedback, Ultra-Fast Scrolling, USB-C Charging, for MacOS, iPadOS, Windo
As an affiliate, we earn on qualifying purchases.
How Ai2 Tested Novelty Judges
Ai2 presents the work as part of a broader effort to build tools that assist scientific discovery, including systems for exploring structured data, generating and testing hypotheses, and synthesizing research literature. A novelty judge is a model asked to assess whether an idea is new relative to prior work, either on its own or in comparison with another idea.
The authors say there is no standard recipe for building or evaluating these judges: researchers can vary the model, prompt, reasoning budget, comparison format and access to literature. They also argue that evaluations often use human-written papers even when a tool is intended to assess AI-generated proposals. The benchmark therefore includes a human-and-generated setting as well as ideas selected using reviewer comments. The paper cautions that acceptance or rejection alone is not a reliable novelty label, since papers can succeed or fail for reasons other than originality.
“Seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.”
— Ai2 study authors
Limits of the Benchmark Labels
The benchmark’s novelty labels rely on reviewer comments and selection criteria as evidence of relative originality; they are not definitive measures of novelty. In the human-versus-generated set, the researchers assumed the basic model-generated ideas were less novel than the selected human ideas, while acknowledging that each label was not independently checked. The expert review covered only 30 pairs, not the full set.
The supplied study description does not provide full results for every model, prompt, reasoning budget or evaluation format, or say when the materials were released. It also does not establish whether the findings carry over to other academic fields, different idea generators or systems with broader literature access. The authors say some conclusions about disagreement between evaluation setups do not depend on benchmark labels, but the available description does not detail how widely that result held across settings.
Further Tests Across Research Fields
Ai2 has released the paper, benchmark data, code and automated pipeline for researchers to build and test additional evaluation sets. Other teams can use those materials to check whether prompt sensitivity appears in different disciplines, with different sources of ideas and under other judge designs.
The release description does not announce a follow-up study or a timeline. Open questions include whether any combination of prompting, retrieval and model design can produce more consistent judgments, and how such methods should be validated against expert assessment. Until those tests are available, the benchmark offers a way to compare evaluation choices, not evidence that one novelty-scoring setup is ready to serve as a reliable standard.
Key Questions
What did the Ai2 researchers find?
They report that six LLMs’ novelty judgments varied across evaluation setups. In one pairwise experiment, changing a few prompt words shifted accuracy by more than 50 percentage points.
Which models were tested?
The study tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, using absolute ratings and pairwise comparisons.
Did literature retrieval or more reasoning improve results?
According to the authors, neither consistently improved judgments across their tests. The supplied description does not give detailed results for every setting.
Are the benchmark’s novelty labels definitive?
No. The human-idea labels draw on reviewer comments and selection criteria, while the generated-idea labels were not individually verified. The researchers describe some of those labels as weak.
What did Ai2 release?
Ai2 released the study paper, benchmark data, code and an automated pipeline intended to help researchers build further evaluation sets. No specific follow-up timeline was announced in the supplied material.
Primary source: Hugging Face · via ThorstenMeyerAI.com
