Train your first Decision Transformer

Mixture of Experts (MoEs) in Transformers

Symmetry-Aware Transformer Training for Automated Planning

arXiv:2508. 07743v2 Announce Type: replace Abstract: While transformers excel in many settings, their application in the field of automated planning is limited.

Introducing Optimum: The Optimization Toolkit for Transformers at Scale

Differential Transformer V2

Trust Guided Decision Transformer

The paper introduces Trust Guided Decision Transformer (TGDT), a method that mitigates performance degradation in Decision Transformers during long rollouts by monitoring the model’s next‑state prediction error. TGDT evaluates multiple recent context suffixes, filters out those whose prediction error exceeds a calibrated threshold, and then selects the highest‑value action from the remaining trusted suffixes using a frozen critic. Experiments on D4RL navigation and locomotion tasks demonstrate that TGDT reduces persistent high‑error runs and improves returns compared to vanilla Decision Transformer and other context‑control baselines.

Vision Transformer Finetuning Benefits from Non-Smooth Components

arXiv:2602. 06883v3 Announce Type: replace Abstract: The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness.