arXiv AI
Aug 11
On the Robustness of LLMs' Internal Representation of Code Correctness
arXiv:2608. 08266v1 Announce Type: cross Abstract: Code generated by modern language models often reads naturally.
Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs
arXiv:2606. 28574v1 Announce Type: cross Abstract: When a large language model (LLM) codes a construct in text as a human annotator would, that agreement makes the LLM a reliable coder.
Related stories
arXiv AI
Jul 10
Validity of LLMs as data annotators: AMALIA on authority
arXiv:2607. 08731v1 Announce Type: cross Abstract: A national language model offers a linguistic community its own instrument for measuring what its citizens say and value.
arXiv AI
Jul 21
Trusting sovereign language models as scientific instruments: evidence from Portugal's AMALIA
arXiv:2607. 08731v2 Announce Type: replace-cross Abstract: National language models are becoming publicly funded epistemic infrastructure.
arXiv AI
Jul 23
Measuring LLM Trust Allocation Across Conflicting Software Artifacts
arXiv:2604. 03447v2 Announce Type: replace-cross Abstract: LLM-based software engineering assistants often reason over multiple artifacts, including code, documentation, signatures, and tests, even when those artifacts are incomplete or mutually inconsistent.
arXiv AI
Jun 8
SWE-IF: Aligning Code Evaluation with Human Preference
arXiv:2510. 07315v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check.
Hugging Face Trending Papers
Jul 9
Validity of LLMs as data annotators: AMALIA on authority
A national language model offers a linguistic community its own instrument for measuring what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, appears competitive on agreement alone: asked to code the moral foundation of authority, it agrees with trained human coders to within six F1 points of open models eight to thirteen times its size.
arXiv AI
Jul 16
Faithful Autoformalization of Natural Language Assertions
arXiv:2607. 13303v1 Announce Type: cross Abstract: Formal contracts are essential for software testing and verification, yet writing them remains labor-intensive and error-prone.
arXiv AI
Aug 12
Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents
arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.
arXiv Machine Learning
Jun 9
Decoy-Calibrated Failure Audits for Language Models
arXiv:2606. 09046v1 Announce Type: new Abstract: Useful audits reveal not only how often a model fails, but also where its failures concentrate.
arXiv Machine Learning
1d ago
Certifying Compressed Language Models: An Audit and a Statistical Toolkit
arXiv:2608. 15046v1 Announce Type: new Abstract: A fraction of a point of benchmark accuracy is the usual evidence that a compressed model is equivalent to its original.
arXiv AI
Jun 29
CalBrief: A Pilot Diagnostic Benchmark for Evidence-Calibrated Scientific Briefing with Large Language Models
arXiv:2606. 27383v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as research assistants, yet it remains unclear whether they can calibrate research takeaways to the strength and scope of the supporting evidence.
arXiv AI
1d ago
What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models
arXiv:2608. 16852v1 Announce Type: new Abstract: Regulatory compliance monitoring in deployed language models is increasingly implemented as a legal and audit control, checking model outputs against written rules spanning data protection, healthcare, financial regulation, and platform policy.