Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation
What it is
This paper presents a systematic study showing that Large Language Model (LLM) judges used to evaluate the novelty of automated ideas are highly unstable and sensitive to minor prompt variations. It demonstrates that simply changing the context provided to the judge can shift its verdicts by over 50 points, often making the evaluation unreliable or below chance level.
The problem it solves
LLM-based novelty judges are currently built ad hoc and are highly sensitive to prompt design, leading to inconsistent and unreliable evaluations of generated ideas.
What you could use it for
- Designing robust evaluation protocols for automated ideation systems
- Benchmarking the reliability of LLM-based judges in scientific research
- Identifying failure modes in AI-driven novelty assessment
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.