← Back to all news
arXiv AI By Chenhao Hong, Jiakai Xu, Kaushiki Singh, Yusen Zhang, Eugene Wu

CIRCE Divines LLM Task Complexity from Constraint Structure

Read the original on arXiv AI →

CIRCE is a method that estimates the difficulty of a task for a large language model (LLM) by extracting the problem’s constraint structure into a Constraint Intermediate Representation, which is then compiled into a constraint graph. Unlike prior approaches that rely on the problem text or model outputs, CIRCE uses a lightweight LLM to build this representation and feeds it to a graph neural network to predict whether the LLM will solve the problem correctly. Experiments on seven reasoning benchmarks and four LLMs show that even a randomly initialized GNN improves prediction accuracy, lowering negative log‑likelihood from 0.465 to 0.423 and raising AUROC from 0.691 to 0.727 compared to the best baseline.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 4

RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Data

RECAST is a new framework that generates datasets with far more constraints per example than existing benchmarks, aiming to push large language models (LLMs) to better follow complex instructions. The authors built RECAST-30K, a 30,000‑instance dataset covering 19 constraint types extracted from real prompt‑response pairs, and showed that fine‑tuning on it improves LLMs’ ability to handle complex tasks without harming general performance. RECAST also provides rule‑based and LLM‑based validators for automatic constraint verification, enabling reward‑based reinforcement learning to further enhance model performance on challenging tasks.

By Zhengkang Guo, Wenhao Liu, Mingchen Xie, Jingwen Xu, Zisu Huang, Muzhao Tian, Jianhan Xu, Yuanzhe Shen, Qi Qian, Muling Wu, Xiaohua Wang, Changze Lv, He-Da Wang, Hu Yao, Xiaoqing Zheng, Xuanjing Huang
llmsreinforcement-learningbenchmarks
More like this →
arXiv AI
Jun 3

Bridging Auxiliary Constraints to Resolve Instruction Following in Large Reasoning Models

arXiv:2606. 03624v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) have demonstrated impressive capabilities in many tasks, yet they struggle with reliably following multiple instructions, either by failing to satisfy individual constraints or by struggling to balance competing constraints simultaneously.

By Zhengyi Zhao, Shubo Zhang, Huimin Wang, Zezhong Wang, Yutian Zhao, Yefeng Zheng, Binyang Li, Yulan He, Kam-Fai Wong, Xian Wu
llms
More like this →