IndicBankBench is a 799‑case benchmark designed to evaluate the safety and reliability of language model assistants in Indian retail banking. It covers five operational domains, a capability/refusal domain, and twenty primary axes, assessing each case at four stages: safety, action and tool use, response adequacy, and advisory quality. The benchmark uses deterministic safety checks, a narrow resolver for ambiguous confirmation‑before‑write scenarios, and an LLM judge for semantic response adequacy, reporting strict pass rates that reveal a gap between strict reliability (43.7%–58.2%) and at‑least‑once success (60%–74%).
By Suvradip Paul, Chandra Bhushan, Harsh Sharma, Nitin Kukreja, Yatharth Dedhia, Keyur Doshi, Prashant Devadiga
FiMI Banking presents a controlled Indian retail‑banking environment designed to train conversational agents that can answer product questions, handle account requests, and comply with regulatory constraints. The system is built from vetted banking documents, structured ground truth, synthetic customer backgrounds, and banking tools. Evaluation of two post‑training methods—preference optimization and reinforcement learning with verifiable rewards—shows that preference optimization significantly increases safe behavior, while reinforcement learning improves edge‑case and order‑sensitive task performance while reducing token usage.
By NPCI AI Research Team, Aman Kumar, Asit Desai, Chandra Bhushan, Harsh Sharma, Harshit Bhushan, Hrithik Kadam, Keyur Doshi, Kolisetty Sai Kapardheeswar, Krishanu Adhikary, Nadeem Shaik, Navya Prakash, Nitin Kukreja, Prashant Devadiga, Shamanth MH, Shantanu Pandey, Suvradip Paul, Yatharth Dedhia
FiMI Banking presents a controlled Indian retail‑banking environment designed to train conversational agents that can answer product questions, handle account requests, and comply with strict operational and regulatory constraints. The system is built from vetted banking documents, structured ground truth, synthetic customer backgrounds, and banking tools. Evaluation of two post‑training methods—preference optimization and reinforcement learning with verifiable rewards—shows that preference optimization significantly increases safe behavior (out‑of‑scope refusal from 52% to 80%), while reinforcement learning improves edge‑case and order‑sensitive task performance and reduces token usage.
The paper introduces a hypothesis-driven simulation workflow that screens customer experience (CX) agents before deployment, using synthetic customers and simulated tool outputs to emulate multi-step interactions without accessing production backends. Applied to Nubank’s high-volume Card Delivery and Card Management chat-support agents, the simulation’s binary evaluator scores correlated strongly with production results, and simulation-guided iterations raised transactional net promoter score by 36.69 points in a live A/B test. Additionally, screening over 16,000 simulated conversations helped select a model that increased self‑service rate by 8.82 percentage points without harming net promoter score, demonstrating that simulation enables extensive model exploration safely.
By Edesio Alcoba, Kevin Rossell, Aman Gupta, Shao Tang, Jiwoo Hong, Pabel Carrillo-Mendoza, Wanderson Concei\c{c}\~ao Ferreira, Alvaro Tedeschi, Zayd Simjee, Shreya Rajpal, Bruno Finardi Hime, Christian Sousa, Luis Moneda, Herbert Fei, Daniel Silva, Rohan Ramanath
arXiv:2610.18881v1 Announce Type: cross
Abstract: Voice agents built on large language models (LLMs) now handle customer calls, but in regulated applications completing the task is not enough: the ag...
By Petmongkon Pornpichitsuwan, Krittin Pachtrachai, Touchapon Kraisingkorn
arXiv:2606. 12924v1 Announce Type: new Abstract: We present a modular two-agent simulation framework for evaluating conversational shopping assistant architectures.
By Jetlir Duraj, Jayanth Yetukuri, Shuang Zhou, Dhruv Varma, Rui Kong, Ishita Khan, Qunzhi Zhou