← Back to all news
arXiv Machine Learning By Harsh Kumar, Tanmay Shailesh Joshi, Rahul Maity, Anusa Saha

Causal Syntax Circuits in CodeLMs: Budget and Probe Decide What Looks Localised

Read the original on arXiv Machine Learning →

The paper investigates whether the tendency of code language models to default to Python when no language is specified is driven by a small, identifiable set of internal components. Using a comprehensive pipeline—including ablation, activation patching, greedy circuit extraction, and significance testing—across 55 model-language configurations, the authors find that this preference is large, grows with model size, and persists after instruction tuning. They also show that the apparent localization of this behavior depends heavily on analysis choices such as component budget and probe design, and that interventions can sometimes alter the model’s output.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 18

Measurement Under Selection: Decoy-Calibrated Failure Audits for Language Models

The paper introduces Janus, a method for validating error patterns in language models by comparing error rates across predefined yes/no properties and using shuffled decoy labels to set significance thresholds. Janus requires that a pattern’s error difference surpasses the decoy-derived threshold and is replicated on held‑out data before reporting. Experiments on a controlled code‑finding task confirm several meaningful error patterns, while on MuSiQue and LongBench v2 Janus reports no confirmed patterns for the tested properties, contrasting with standard shuffling tests that sometimes confirm patterns.

By Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh
llmsbenchmarks
More like this →
arXiv AI
Jul 22

Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models

arXiv:2607. 19257v1 Announce Type: cross Abstract: Practitioners make three prompt-design decisions with almost no controlled evidence behind them: how to format instructions and context (markdown, plain text, prose, or tabular), how many simultaneous instructions a system prompt can carry before compliance degrades, and how much context a model can hold before recall and honesty degrade.

By Netanel Eliav
llmssafety
More like this →