← Back to all news
arXiv Computation and Language By Hatice Merve Vural, Doga Kukul, Ege Erdem Ozlu, Demir Ekin Arikan, Bob Mankoff, Erkut Erdem, Aykut Erdem

Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding

Read the original on arXiv Computation and Language →

The paper introduces IRS (Incongruity-Resolution Supervision), a framework that breaks humor understanding into three parts: identifying mismatches in a visual scene, creating coherent reinterpretations of those mismatches, and aligning these interpretations with human preferences. IRS uses structured reasoning traces to guide models from visual perception to humorous interpretation, and it is evaluated on the New Yorker Cartoon Caption Contest. Experiments on 7B, 32B, and 72B models show that IRS improves caption matching and ranking, with the 72B model achieving 76.10% ranking accuracy—outperforming non-expert humans and all other multimodal baselines—and demonstrates transferable reasoning patterns in zero‑shot settings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Oct 28

From Perception to Punchline: Empowering VLM with the Art of In-the-wild Meme

The paper introduces HUMOR, a framework that enhances vision‑language models (VLMs) for generating humorous memes by guiding them through a hierarchical, multi‑path Chain‑of‑Thought reasoning process. HUMOR first identifies a template‑level intent, explores diverse reasoning paths under different contexts, and then anchors onto a high‑quality, context‑specific path, thereby increasing reasoning diversity. Additionally, a pairwise reward model trained on groups of memes sharing the same template captures subjective humor, enabling group‑wise reinforcement learning that guarantees monotonic improvement within a trust region. Experiments demonstrate that HUMOR improves reasoning diversity, preference alignment, and overall meme quality, and the authors suggest the approach generalizes to other open‑ended, human‑aligned multimodal generation tasks.

By Xueyan Li, Yingyi Xue, Mengjie Jiang, Qingzi Zhu, Yazhe Niu
reinforcement-learningmultimodalsafety
More like this →
arXiv AI
Jun 2

Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling

arXiv:2606. 02578v1 Announce Type: cross Abstract: Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers.

By Seojeong Park, Jiho Choi, Junyong Kang, Seonho Lee, Jaeyo Shin, Hyunjung Shim
llmsreinforcement-learningmultimodalbenchmarkssafety
More like this →