arXiv AI

A Tool-Augmented, GPT-4 Chatbot for Real-Time Repository Data Analysis

arXiv AI
4d ago

SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories

SlackBench is a new benchmark that evaluates AI agents on realistic collaborative research and engineering workflows, requiring joint reasoning over workplace communication and evolving repository state. The benchmark is built from privacy‑preserving reenactments of real academic projects, providing reproducible environments with multi‑party dialogue, code changes from GitHub, and Google Docs. It contains 140 queries covering heterogeneous‑source search, abstention, summarization, and inference over long conversations, and shows that current models perform well below the ceiling, indicating significant room for improvement.

By Mingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis
arXiv AI
Oct 16

RAG4Tickets: AI-Powered Ticket Resolution via Retrieval-Augmented Generation on JIRA and GitHub Data

The paper introduces RAG4Tickets, an AI system that uses Retrieval-Augmented Generation to improve ticket resolution on JIRA and GitHub. It embeds historical tickets, comments, and pull request data with Sentence-Transformers, indexes them with FAISS, and then uses a Large Language Model to generate grounded, explainable resolution suggestions based on retrieved evidence. Experiments show higher precision, recall, faster resolution times, and better developer acceptance compared to existing methods.

By Mohammad Baqar
arXiv AI
Sep 7

Building a research-software catalog with a coding agent: from hackathon prototype to public deployment

The paper reports on building a research-software catalog using a coding agent, starting from a three‑day hackathon prototype and moving to public deployment. It details the engineering work needed—adversarial review, data‑quality checks, browser validation, and publication safeguards—to ensure reliable operation, noting that silent failures were more problematic than crashes. The authors then examine applying these lessons to a larger, human‑curated portal (MateriApps) that combines curated metadata, external documentation, vector search, and local language‑model generation, finding that explicit validation, monitoring, and repeated review remain essential for AI‑assisted software portals.

By Kazuyoshi Yoshimi, Satoshi Terasaki, Gotai Yamada
arXiv AI
Oct 27

ComBeautify: Leveraging Code Change Summaries for Conventional Commit Generation

ComBeautify is a tool that automatically generates Conventional Commits Specification (CCS)-compliant commit messages by combining context-preserving summarization of Git diffs with large language models. It learns commit categories from historical data rather than relying on manually curated definitions, and also produces the CCS description component. Evaluation on 13 languages, 6,500 repositories, and over 20 million commits shows that project-specific training can outperform generic baselines by up to three times, with summarization accuracy reaching 95.25% on a manually curated dataset.

By Rishi Raj, Aashray Tandon, Nikolay Matyunin, Jibesh Patra
arXiv AI
Sep 28

Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer

The paper explores whether natural‑language documentation aids coding agents in fixing software bugs and introduces a roundtrip benchmark that evaluates code descriptions by regenerating code and testing it. It finds that description completeness, not length, determines fidelity, and presents an optimizer that can produce fully faithful descriptions that generalize to new files. However, experiments across two model families and ten repositories show that such compact documentation does not improve an agent’s ability to resolve real repository issues compared to using the issue alone.

By Md Shohel Arman, Igor Molybog