AMLworld-Compact introduces an importance‑weighted downsampling protocol for evaluating large language models on AMLworld’s test splits, retaining all illicit transactions while sampling benign ones from supervised‑score strata. The HT‑Coreset method reduces evaluation targets by 271× on HI‑Small and 611× on LI‑Small, providing unbiased confusion‑count estimates with exact recall and ratio estimates for precision and F1. Experiments with boosted‑tree strata and Graph Feature Preprocessor features show lower F1 RMSE compared to uniform sampling, and audit tools reveal that lexical checks do not reliably indicate correctness of model outputs.
By Natapong Nitarach, Phume Ngampornsukswadi, Pittawat Taveekitworachai, Surapon Nonesung, Warit Sirichotedumrong, Kunat Pipatanakul
arXiv:2607. 02825v1 Announce Type: cross Abstract: We release \textsc{JavaVulBench}, a benchmark dataset and evaluation harness for Java vulnerability detection.
By Norbert Sandor Szolnoki, Gabor Antal
arXiv:2605. 17561v2 Announce Type: replace-cross Abstract: Issues faced when using software are reported in the form of bug reports.
By Mahmut Furkan Gon, Emre Dinc, Tevfik Emre Sungur, Eray Tuzun
Saidzoda‑Tajik‑LFM‑2.6B is a 2.6‑billion‑parameter hybrid model adapted for Tajik tool calling. The adaptation uses two stages: LoRA continued pretraining on 1.49 million cleaned Tajik web documents followed by one epoch of supervised fine‑tuning on machine‑translated Tajik tool calls, English tool calls, and Tajik instructions. On a 174‑item Tajik tool‑calling test set, strict full‑call accuracy rises from 20.69 % for the base model to 64.14 %, while English performance remains unchanged; however, continued pretraining removes the native tool‑call format and part of the skill, and the model shows limited general assistant capabilities.
By Tohir Saidzoda, Nekruz Maxudov, Bedil Karimov, Abubakr Abdullozoda, Amin Jalilov, Jonibek Kishvarov, Rahmonali Sharipov, Mariyamkhon Shodyeva
arXiv:2610.02267v1 Announce Type: new
Abstract: Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input...
By Jiawei Li
The paper presents Baszta, a Polish multi‑label content‑safety classifier trained by fine‑tuning the 124M‑parameter allegro/herbert‑base‑cased model on five categories (hate, vulgarity, sexual content, crime, self‑harm) using a Focal + R‑Drop objective. In out‑of‑distribution evaluation on the Gadzi Język benchmark, Baszta achieves a small but statistically significant improvement in micro‑F1 over the Bielik Guard system, though the macro‑F1 advantage disappears when both models are properly tuned. The study also explores calibration techniques, showing that per‑category temperature scaling can recover performance lost by Platt scaling or isotonic regression, and discusses the trade‑offs between robust calibration and adversarial recall.
By Adam G\'orski, Mateusz J\k{a}kalak, Rafa{\l} Jakubowski
arXiv:2606. 31630v1 Announce Type: new Abstract: Language models increasingly write probabilistic programs (in NumPyro, Stan, or Pyro), but a program that compiles, runs, and passes every unit test can still be \emph{statistically} wrong -- a Gaussian likelihood for heavy-tailed data, a Poisson for over-dispersed counts, an invalid prior support, or a pathological parameterization.
By Jian Xu, Delu Zeng, John Paisley, Qibin Zhao
The paper investigates whether the hidden activations of large language models (LLMs) contain signals about the vulnerability of C/C++ code when the code is provided as context. By extracting prefill token activations from four LLMs and training small MLP probes, the authors achieve an average F1 score of 41.7% across four benchmarks, with the best probe matching state‑of‑the‑art fine‑tuned classifiers on the Devign dataset. The results suggest that a coding LLM’s internal representation can inform vulnerability detection, opening the door to lightweight, model‑native screening methods.
By Alizishaan Khatri
arXiv:2606. 20502v1 Announce Type: cross Abstract: Whether LLMs scoring well on vulnerability benchmarks genuinely reason about security or merely pattern-match on contaminated data remains unresolved.
By Arastoo Zibaeirad, Marco Vieira
arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.
By Bhaskar Gurram
arXiv:2607. 22545v1 Announce Type: cross Abstract: Deploying large language models in financial-services and agentic settings requires safety classifiers that simultaneously handle prompt injection, regulatory compliance, and general harm, a combination no existing open guardrail addresses in a single inference pass.
By Tejasvi C. Addagada
The paper introduces Conditional Accuracy Profiling (CAP), a diagnostic framework that breaks down LLM‑judge accuracy into eight deployment conditions covering content sensitivity, robustness, and rationale quality. CAP is benchmark‑agnostic and can be applied directly when annotations are available, via task‑subset proxies, or through controlled augmentation. Applied to seven judges across six benchmarks, CAP reveals that aggregate accuracy masks important differences—for example, judges highly sensitive to omitted qualifications rank poorly on a Hard‑Constructed subset, and Position Robustness is generally stable but can degrade sharply under adversarial stress.
By Wenqi Li, Bin Liu, Mindi Ruan, Chuanbo Hu, Minglei Yin, Xin Li