← Back to all news
arXiv Machine Learning By Asset Kanatov, Talgar Bayan, Askar Boranbayev

Bug Severity Classification with Fine-Tuned Open LLMs: A Paired Comparison with TF-IDF and CodeBERT Baselines on Eclipse Reports

Read the original on arXiv Machine Learning →

The study evaluates the effectiveness of small, fine‑tuned open large language models (LLMs) for predicting bug severity in Eclipse bug reports, comparing them against traditional TF‑IDF classifiers and a fine‑tuned CodeBERT encoder. Using a balanced sample of 20,000 reports, eleven model configurations—including zero‑shot and QLoRA‑fine‑tuned Llama 3 8B and Qwen 2.5 7B—were tested on 2,000 held‑out reports, revealing that fine‑tuned LLMs achieve macro‑F1 scores around 0.52, comparable to CodeBERT, while zero‑shot LLMs lag behind TF‑IDF baselines. The analysis also highlights class‑specific performance differences and the impact of duplicate reports on model evaluation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Oct 12

AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis

AMLworld-Compact introduces an importance‑weighted downsampling protocol for evaluating large language models on AMLworld’s test splits, retaining all illicit transactions while sampling benign ones from supervised‑score strata. The HT‑Coreset method reduces evaluation targets by 271× on HI‑Small and 611× on LI‑Small, providing unbiased confusion‑count estimates with exact recall and ratio estimates for precision and F1. Experiments with boosted‑tree strata and Graph Feature Preprocessor features show lower F1 RMSE compared to uniform sampling, and audit tools reveal that lexical checks do not reliably indicate correctness of model outputs.

By Natapong Nitarach, Phume Ngampornsukswadi, Pittawat Taveekitworachai, Surapon Nonesung, Warit Sirichotedumrong, Kunat Pipatanakul
llms
More like this →
arXiv Computation and Language
4d ago

Saidzoda-Tajik-LFM-2.6B: Adapting a 2.6B Hybrid Model to Tajik Tool Calling, and What Continued Pretraining Takes Away

Saidzoda‑Tajik‑LFM‑2.6B is a 2.6‑billion‑parameter hybrid model adapted for Tajik tool calling. The adaptation uses two stages: LoRA continued pretraining on 1.49 million cleaned Tajik web documents followed by one epoch of supervised fine‑tuning on machine‑translated Tajik tool calls, English tool calls, and Tajik instructions. On a 174‑item Tajik tool‑calling test set, strict full‑call accuracy rises from 20.69 % for the base model to 64.14 %, while English performance remains unchanged; however, continued pretraining removes the native tool‑call format and part of the skill, and the model shows limited general assistant capabilities.

By Tohir Saidzoda, Nekruz Maxudov, Bedil Karimov, Abubakr Abdullozoda, Amin Jalilov, Jonibek Kishvarov, Rahmonali Sharipov, Mariyamkhon Shodyeva
agentsfine-tuningbenchmarks
More like this →
arXiv AI
Sep 25

Baszta: Data-Centric Fine-Tuning of a Polish Multi-Label Safety Classifier

The paper presents Baszta, a Polish multi‑label content‑safety classifier trained by fine‑tuning the 124M‑parameter allegro/herbert‑base‑cased model on five categories (hate, vulgarity, sexual content, crime, self‑harm) using a Focal + R‑Drop objective. In out‑of‑distribution evaluation on the Gadzi Język benchmark, Baszta achieves a small but statistically significant improvement in micro‑F1 over the Bielik Guard system, though the macro‑F1 advantage disappears when both models are properly tuned. The study also explores calibration techniques, showing that per‑category temperature scaling can recover performance lost by Platt scaling or isotonic regression, and discusses the trade‑offs between robust calibration and adversarial recall.

By Adam G\'orski, Mateusz J\k{a}kalak, Rafa{\l} Jakubowski
fine-tuningbenchmarkssafety
More like this →