← Back to all news
arXiv AI

Large language models eroding science understanding: an empirical study of malignment

This study investigates whether large language models (LLMs) can reliably answer scientific questions and how susceptible they are to manipulation by fringe scientific material. The authors modified custom LLMs to prioritize knowledge from selected fringe papers on the Fine Structure Constant and Gravitational Waves, then compared their responses with those of domain experts and standard LLMs. The altered models produced fluent, convincing answers that contradicted scientific consensus and were difficult for non-experts to detect as misleading, demonstrating that LLMs are vulnerable to manipulation and cannot replace expert judgment.

Related stories

arXiv AI
Aug 28

How LLMs Distort Our Written Language

Large language models (LLMs) are widely used to assist writing, but this study shows they alter both tone and meaning of human text. A user study found that heavy LLM use increased neutral essays by nearly 70% and made writers feel less creative and less in their voice. Even when prompted to make only grammar edits, LLMs changed the semantic content of essays and produced AI-generated scientific reviews that were less focused on clarity and significance and scored higher on average.

By Marwa Abdulhai, Isadora White, Yanming Wan, Ibrahim Qureshi, Joel Z. Leibo, Max Kleiman-Weiner, Natasha Jaques
llms
More like this →
arXiv AI
Jun 3

LLMs, Reasoning and Plagiarism

arXiv:2601. 02380v5 Announce Type: replace-cross Abstract: Recent reports claim that Large Language Models (LLMs) derive new science and exhibit human-level general intelligence.

By Elchanan Mossel
llms
More like this →
arXiv AI
Aug 20

Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models

The paper investigates how large language models can extract contextualized data from scientific literature. It presents four workflows: expert‑written prompts, self‑generated prompts, autonomous literature discovery, and dataset creation from guidelines. While models perform well with prompts, they struggle with context, hallucinate references, and still need human oversight for final validation.

By Valentin Romanov, Monique Bax, Steven Niederer
llmsagents
More like this →
Hugging Face Trending Papers
Aug 19

Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models

The paper evaluates browser-based large language models (LLMs) for extracting detailed, contextualized data from scientific papers. It presents four workflows: (1) expert-curated prompts yield good extraction but struggle with nuance; (2) LLMs can generate effective prompts from simple instructions; (3) autonomous literature discovery is challenging, with missing or hallucinated references; (4) LLMs can build new datasets from guidelines that align closely with human experts, yet still need human oversight. The study outlines a practical, auditable workflow where experts set standards, models cross-check extractions, and researchers resolve disputes, enabling scalable scientific data curation.

llmsagents
More like this →
arXiv AI
Aug 19

Do LLMs Know a Good Hypothesis When They See One? Logit-Based Energy Scoring Outperforms Prompted LLM-as-Judge for Scientific Hypothesis Ranking

The paper investigates whether large language models (LLMs) can reliably assess scientific hypotheses by using a logit-based energy scoring method that leverages the model’s intrinsic confidence. Across 1,323 papers in 12 disciplines, this intrinsic scoring achieved a 33.0% Hit@1 rate, outperforming a prompted listwise ranking approach that scored 16.6%. The best result, a 1‑billion‑parameter model with logit-based energy scoring, reached 53.1% Hit@1, suggesting that confidence‑based evaluation could improve trustworthy AI‑enabled scientific discovery.

By Swati Rajwal, Sanjay Das, Tirthankar Ghosal
llms
More like this →