arXiv AI
Aug 19

Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases

The study examines whether large language models (LLMs) can perform legally meaningful reasoning by testing OpenAI GPT 5.4 on European Court of Human Rights case forecasting. Using various prompting strategies, the authors find that the model produces structurally complete but substantively shallow analyses, and that LLM-as-a-Judge evaluators are internally consistent yet only weakly aligned with human annotators. The expert-curated prompt yields more comprehensive reasoning but does not improve prediction accuracy, leading the authors to caution against relying solely on automated LLM evaluation or using task accuracy as a proxy for reasoning quality.

By Amogh Raina, Ilias Chalkidis, Daniel Hershcovich, Henrik Palmer Olsen
arXiv AI
Oct 16

Incentivizing Tool-Use Reinforcement Learning for Post-Arrival Emergency Care Prediction

The paper introduces TIR-EC, a tool‑integrated reinforcement learning framework designed to improve large language models (LLMs) for emergency department prediction tasks. TIR-EC exposes groups of electronic health record (EHR) features as callable tools and trains the LLM using rewards from a gradient‑boosted tree teacher, while a reverse curriculum gradually removes prompt‑based skills to embed task‑specific clinical knowledge into the model weights. Experiments on three clinical prediction tasks show that a 4B TIR‑EC agent achieves the highest macro F1 among compared models, surpasses conventional machine‑learning baselines, and competes with much larger frontier LLMs, while also selecting more useful evidence with fewer tool calls.

By Pengfei Hu, Chenwei Wu, Qiao Jin, Jun Deng, Zhiyong Lu, Liyue Shen, Yue Ning