arXiv AI
Jun 2
Uncovering Competency Gaps in Large Language Models and Their Benchmarks
arXiv:2512. 20638v2 Announce Type: replace-cross Abstract: The evaluation of large language models relies heavily on standardized benchmarks.
Query Circuits: Explaining How Language Models Answer User Prompts
arXiv:2509. 24808v2 Announce Type: replace Abstract: Explaining why a language model produces a particular output requires local, input-level explanations.
Related stories
arXiv Machine Learning
Jul 10
Prompt Compression via Activation Aggregation
arXiv:2607. 08399v1 Announce Type: cross Abstract: Large language models process prompts by propagating activations through dozens of layers before generating a response.
arXiv AI
Jun 30
Most Current Model Organisms Are Leaky: Perplexity Differencing Often Reveals Finetuning Objectives
arXiv:2605. 00994v2 Announce Type: replace-cross Abstract: Finetuning can significantly modify the behavior of large language models, including introducing harmful or unsafe behaviors.
arXiv AI
Jun 24
Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability?
arXiv:2606. 24026v1 Announce Type: new Abstract: Mechanistic interpretability has made substantial progress in automatically localizing circuits, but explaining what localized components do remains labor-intensive and difficult to standardize.
arXiv AI
2d ago
Never the Number: Structural Abstention for AI Systems Whose Answers Are Consumed as Fact
arXiv:2608. 13926v1 Announce Type: new Abstract: Large language models have made natural language interfaces to databases (NLIDB) newly credible, but LLM text-to-SQL systems fail in a way that matters for deployment: a hallucinated column or a mis-aggregated total yields a fluent wrong answer, indistinguishable at the point of use from a right one.
arXiv AI
Jun 12
Language Model Circuits Are Sparse in the Neuron Basis
arXiv:2601. 22594v2 Announce Type: replace-cross Abstract: The high-level concepts that a neural network uses to perform computation need not be aligned to individual neurons (Smolensky, 1986).
arXiv AI
Jul 10
Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
arXiv:2601. 22588v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are widely used as reference-free evaluators via prompting, but this "LLM-as-a-Judge" paradigm is costly, opaque, and sensitive to prompt design.
arXiv AI
Jul 28
A Mechanistic Perspective and Circuit-Guided Difficulty Metric for Unlearning
arXiv:2601. 09624v2 Announce Type: replace-cross Abstract: Machine unlearning is becoming essential for building trustworthy and compliant language models.
arXiv AI
Jul 9
MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning
arXiv:2607. 07391v1 Announce Type: new Abstract: Mathematical reasoning benchmarks typically provide all facts needed to solve each problem, while interactive benchmarks often mix reasoning with tools, retrieval, and long-horizon dialogue.
arXiv AI
6d ago
Behavior and Representation in Open-Weight Large Language Models for Combinatorial Optimization: From Feature Extraction to Algorithm Selection
arXiv:2512. 13374v2 Announce Type: replace Abstract: Recent advances in Large Language Models (LLMs) open new perspectives for automation in optimization, yet little is known about whether their internal representations capture problem structure or algorithmic behavior.
arXiv AI
Jul 22
Circuit Claims Depend on What Is Extracted and How It Is Compared
arXiv:2607. 18921v1 Announce Type: cross Abstract: Circuit extraction identifies a small set of model components whose presence preserves a target behavior under ablation, and the resulting circuit is often read as the mechanism behind that behavior.
arXiv Machine Learning
Jul 30
Towards Verifiable Transformers: Solver-Checkable Circuit Explanations
arXiv:2605. 24033v2 Announce Type: replace Abstract: Mechanistic interpretability typically discovers circuits and then argues what they do from examples and ablations.