Decoupling What from Where: How Should a Small GUI Grounding Model Receive the Action Type?
Read the original on arXiv Machine Learning →The paper investigates how a small grounding model for a GUI agent should receive the action type. Using Qwen2‑VL‑2B fine‑tuned with LoRA on Android, the authors compare a flat baseline with five conditioning methods—auxiliary loss, hard‑routed action word, additive learned embedding, prepended learned token, and prompt word—under matched data, compute, and decoding settings. Results show that auxiliary loss, additive embedding, and prompt word each improve hit@0.10 by five to seven points over the baseline, while hard routing and prepended token perform similarly to the baseline; however, removing a problematic off‑screen touch point class lifts the baseline by nearly seven points, after which no method outperforms it. "whyItMatters":"The study identifies effective strategies for encoding action type in small grounding models, which could improve the accuracy of GUI agents in real‑world Android applications."
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.