← Back to all news
Hugging Face Trending Papers

Do LLMs Know a Good Hypothesis When They See One? Logit-Based Energy Scoring Outperforms Prompted LLM-as-Judge for Scientific Hypothesis Ranking

Large language models (LLMs) are increasingly used for scientific hypothesis generation. However, evaluating generated hypotheses remains a challenge for trustworthy AI-enabled scientific workflows.

Related stories

arXiv AI
Aug 19

Do LLMs Know a Good Hypothesis When They See One? Logit-Based Energy Scoring Outperforms Prompted LLM-as-Judge for Scientific Hypothesis Ranking

The paper investigates whether large language models (LLMs) can reliably assess scientific hypotheses by using a logit-based energy scoring method that leverages the model’s intrinsic confidence. Across 1,323 papers in 12 disciplines, this intrinsic scoring achieved a 33.0% Hit@1 rate, outperforming a prompted listwise ranking approach that scored 16.6%. The best result, a 1‑billion‑parameter model with logit-based energy scoring, reached 53.1% Hit@1, suggesting that confidence‑based evaluation could improve trustworthy AI‑enabled scientific discovery.

By Swati Rajwal, Sanjay Das, Tirthankar Ghosal
llms
More like this →
arXiv AI
Sep 1

Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment

Ideation Arena is a battle-style platform that evaluates research ideas generated by large language models (LLMs) and research agents through pairwise human assessment. The system builds shared literature contexts, collects over 6,000 double-blind comparisons from 105 computer science researchers, and constructs an Elo rating leaderboard to rank proposal-stage expert preferences. It also introduces Ideation Arena Eval, a benchmark to test whether automated evaluators align with human preferences, finding that current LLM judges achieve at best 72.56% Soft Accuracy on overall quality.

By Zhiyu Chen, Keyu Zhao, Jigao Fu, Dong Liang, Yanbiao Wu, Jiaoyang Li, Haidong Xue, Xinhua Zeng, Yuanyi Zhen, Fengli Xu, Yong Li
llmsagentsbenchmarks
More like this →
arXiv Computation and Language
Aug 27

IDEAlign: Comparing Ideas of Large Language Models to Domain Expert

IDEAlign introduces a new protocol for evaluating the similarity of large language model (LLM) annotations to expert judgments. It uses pick‑the‑odd‑one‑out tasks to capture expert similarity and benchmarks various similarity methods—including text embeddings, topic models, and LLM-as-a-judge—against these human ratings. Applied to educational datasets, the study finds that most metrics miss nuanced expert dimensions, with LLM-as-a-judge performing best yet still insufficient for full expert alignment.

By Hyunji Nam, Lucia Langlois, James Malamut, Mei Tan, Dorottya Demszky
llmsragbenchmarkssafety
More like this →