arXiv AI

Multi-Context Fusion Transformer for Pedestrian Crossing Intention Prediction in Urban Environments

The paper introduces the Multi-Context Fusion Transformer (MFT), a model that predicts pedestrian crossing intentions in urban settings by integrating four types of contextual information—pedestrian behavior, environment, localization, and vehicle motion—through a progressive fusion strategy. MFT uses intra-context attention for reciprocal interactions within each context, cross-context attention to combine these contexts into a global representation, and guided attention mechanisms to refine both context tokens and the global token. Experiments on JAADbeh, JAADall, and PIE datasets show MFT outperforms existing methods with accuracies of 73%, 93%, and 90% respectively, and ablation studies confirm the importance of each network component and input context.

arXiv Computer Vision

Sep 11

TrajFusionNet+: Transformer-Based Prediction of Pedestrian Crossing Intention via Fusion of Trajectory Representations and Scene Graphs

TrajFusionNet+ is a transformer-based model that predicts pedestrian crossing intention by fusing sequential trajectory data, visual trajectory overlays, and graph-based scene context. It extends the earlier TrajFusionNet with three attention modules—Sequence, Visual, and Graph—to capture temporal, visual, and relational cues. The model outperforms state‑of‑the‑art methods on the PIE and JAAD datasets and shows better generalization under a joint‑training, separate‑evaluation protocol.

By Fran\c{c}ois G. Landry, Moulay A. Akhloufi

llms agents benchmarks

arXiv Machine Learning

Jun 18

Where Will They Go? Modelling Multimodal Pedestrian Manoeuvres from Ego-centric Videos

arXiv:2606. 18824v1 Announce Type: cross Abstract: Pedestrian trajectory prediction from an ego-centric camera is challenging since it depends on complex interactions with vehicles and scene context, as well as the intention of the pedestrian.

By Yuxuan Xie, Nicolas Pugeault, Chongfeng Wei, Hubert P. H. Shum, Edmond S. L. Ho

multimodal benchmarks

arXiv AI

Jun 2

DeepIPCv3: Event-Aware Multi-Modal Sensor Fusion for Sudden Pedestrian Crossing Avoidance

arXiv:2606. 01277v1 Announce Type: cross Abstract: Current end-to-end autonomous driving systems predominantly rely on frame-based sensors, which suffer from inherent perception latency and motion blur during highly dynamic encounters, specifically sudden pedestrian crossings.

By Oskar Natan, Andi Dharmawan, Aufaclav Zatu Kusuma Frisky, Jazi Eko Istiyanto, Jun Miura

llms agents robotics multimodal benchmarks safety

arXiv AI

Jun 17

DiffAttn: Diffusion-Based Drivers' Visual Attention Prediction with LLM-Enhanced Semantic Reasoning

arXiv:2603. 28251v3 Announce Type: replace-cross Abstract: Drivers' visual attention provides critical cues for anticipating latent hazards and directly shapes decision-making and control maneuvers, where its absence can compromise traffic safety.

By Weimin Liu, Qingkun Li, Jiyuan Qiu, Wenjun Wang, Joshua H. Meng

llms diffusion benchmarks safety

arXiv AI

3d ago

Pedestrian Crossing Intent Classification From Event-Based Vision Using Convolutional Spiking Neural Networks With Temporal Augmentation

The paper presents an end‑to‑end system that converts driving footage into dynamic vision sensor (DVS) event streams, augments training with simulated DVS data, and trains a convolutional spiking neural network (Conv‑SNN) to classify pedestrian crossing intent as crossing or non‑crossing. The Conv‑SNN, trained with a class‑balanced loss and surrogate‑gradient learning, achieves high accuracy and F1 scores on JAAD and CARLA DVS datasets, outperforming or matching prior frame‑based methods while operating on sparse temporal representations. The study details architectural choices, neuron dynamics, and training protocols, and provides a convergence analysis and domain‑transfer evaluation.

By Henok Teklu, Mustafa Sakhai, Maciej Wielgosz, Matej Mertik

agents robotics safety

arXiv Machine Learning

Jun 2

UrbanFusion: Stochastic Multimodal Fusion for Contrastive Learning of Robust Spatial Representations

arXiv:2510. 13774v2 Announce Type: replace Abstract: Forecasting urban phenomena such as housing prices and public health indicators requires the effective integration of various geospatial data.

By Dominik J. M\"uhlematter, Lin Che, Ye Hong, Martin Raubal, Nina Wiedemann

llms multimodal benchmarks

arXiv Computer Vision

2d ago

CFGPNet: Cross-Attention-Based Fused Gradient Programmed Network Framework for Multispectral Object Detection

arXiv:2608.06205v2 Announce Type: replace Abstract: Multispectral object detection combines visible and thermal imagery to improve perception under challenging illumination and environmental conditio...

By Nima Hatami, Karim Faez, Saeed Sharifian, Hamidreza Amindavar

computer-vision

arXiv Computer Vision

Sep 10

CLFTv2: Efficient Camera-LiDAR Fusion for Semantic Segmentation via Hierarchical Feature Pyramids

arXiv:2609.09881v1 Announce Type: new Abstract: Semantic segmentation for autonomous driving requires reliable detection of vulnerable road users (VRUs) despite heavy class imbalance. We introduce CL...

By Toomas Tahves, Mauro Bellone, Raivo Sell

agents computer-vision robotics efficiency

arXiv Computer Vision

3d ago

Contour-Guided Spectral Routing for Robust Real-Time Pedestrian Detection

arXiv:2609.14383v1 Announce Type: new Abstract: Real-time pedestrian detection in driving scenes is constrained by three coupled failure modes: tiny targets lose discriminative evidence, occlusion we...

By Sam Williams, Yuan Xiang

arXiv AI

Jul 29

Eliminating Propagation Delay: Attention-Based Spatial-Temporal Fusion Graph Convolution Network for Traffic Flow Prediction

arXiv:2607. 24885v1 Announce Type: cross Abstract: Predicting traffic flow is crucial to optimizing transportation systems and improving urban mobility.

By Jinpeng Chen, Ziyu Yu, Tao Wang, Jun Ma, Hongbo Gao, Senzhang Wang, Zufeng Zhang, Kaimin Wei

arXiv Machine Learning

Sep 11

M3-Former: Multimodal Transformer with Mixture-of-Experts for Long-Term Vessel Trajectory Prediction

M3-Former is a multimodal transformer framework that uses large language models to encode vessel static attributes and navigational intent as semantic priors for long‑term trajectory prediction. It builds a unified multimodal representation space, aligns static semantic information with dynamic trajectory features via self‑attention, and employs a dual‑granularity Mixture‑of‑Experts architecture to capture both global route planning and fine‑grained maneuvering behaviors. A Steering‑Weighted Cross‑Entropy loss further improves accuracy on sparse turning events, and experiments on a Danish AIS dataset show consistent improvements over state‑of‑the‑art baselines, reducing ADE and FDE by up to 5.1% in 4‑hour predictions.

By Wenzhe Jin, Haina Tang

llms multimodal benchmarks

Hugging Face Trending Papers

Jul 27

MATS: A novel multi-modality multi-task learning framework for 3D perception in autonomous driving

Multi-modality data from different sensors provides rich complementary information for 3D perception, becoming an essential component in reliable autonomous driving systems. Current research typically designs intricate and complex fusion strategies to integrate information from multimodal data on a unified bird's-eye-view (BEV) feature map for the joint learning of multiple perception tasks.

agents robotics multimodal benchmarks

Related stories