arXiv AI
MemRiskBench: Trace-Aware Risk-Preserving Evaluation for Long-Horizon LLM Agents
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 16
SkillVetBench: LLM-as-Judge for Multi-Dimensional Security Risk Evaluation in Open-Source LLM Agent Skills
arXiv:2606. 15899v1 Announce Type: cross Abstract: Open-source LLM agent ecosystems are growing rapidly, yet the security of community-contributed skills - modular tool definitions that extend agent capabilities - remains largely unvetted.
arXiv AI
Jun 19
Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software
arXiv:2606. 20502v1 Announce Type: cross Abstract: Whether LLMs scoring well on vulnerability benchmarks genuinely reason about security or merely pattern-match on contaminated data remains unresolved.
arXiv Computation and Language
Oct 12
AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis
AMLworld-Compact introduces an importance‑weighted downsampling protocol for evaluating large language models on AMLworld’s test splits, retaining all illicit transactions while sampling benign ones from supervised‑score strata. The HT‑Coreset method reduces evaluation targets by 271× on HI‑Small and 611× on LI‑Small, providing unbiased confusion‑count estimates with exact recall and ratio estimates for precision and F1. Experiments with boosted‑tree strata and Graph Feature Preprocessor features show lower F1 RMSE compared to uniform sampling, and audit tools reveal that lexical checks do not reliably indicate correctness of model outputs.
arXiv Machine Learning
Aug 4
Why Formal Monitors Fail: Attack Distribution Entropy as a Coverage Bound for LTL-Based LLM Agent Safety
arXiv:2608. 01388v1 Announce Type: cross Abstract: Runtime safety monitors based on Linear Temporal Logic (LTL) and finite automata (FSA) are increasingly deployed to intercept unsafe tool-call sequences in LLM agents.
Hugging Face Trending Papers
Jun 10
Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness
End-to-end task-success is the dominant way to evaluate LLM agents, but one aggregate number tells you that an agent regressed, not where. We present layer-isolated evaluation: a deployed ordering agent is decomposed into a fixed taxonomy of layers (ontology, intent, routing, decomposition, escalation, safety, memory, and cross-cutting envelope/defense), each exercised by its own assertion slice in a deterministic, no-LLM "pure" mode.
arXiv AI
Aug 21
ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis
arXiv:2604. 02022v4 Announce Type: replace Abstract: Evaluating the safety of LLM-based agents is increasingly important because risks in realistic deployments often emerge over multi-step interactions rather than isolated prompts or final responses.