Can I read Architectural and Inferential Inductive Biases For Exchangeable Sequence Modeling on EtoBox?

Architectural and Inferential Inductive Biases For Exchangeable Sequence Modeling by Mittal, Daksh; Li, Ang; Yen, Tzu-Ching; Guetta, Daniel; Namkoong, Hongseok is a scholarly article available to read on EtoBox.

What is Architectural and Inferential Inductive Biases For Exchangeable Sequence Modeling about?

Autoregressive models have emerged as a powerful framework for modeling exchangeable sequences - i.i.d. observations when conditioned on some latent factor - enabling direct modeling of uncertainty from missing data (rather than a latent). Motivated by the critical role posterior inference plays as a subroutine in decision-making (e.g., active learning, bandits), we study the inferential and architectural inductive biases that are most effective for exchangeable sequence modeling. For the inference stage, we highlight a fundamental limitation of the prevalent single-step generation approach: inability to distinguish between epistemic and aleatoric uncertainty. Instead, a long line of works in Bayesian statistics advocates for multi-step autoregressive generation; we demonstrate this "correct approach" enables superior uncertainty quantification that translates into better performance on downstream decision-making tasks. This naturally leads to the next question: which architectures are best suited for multi-step inference? We identify a subtle yet important gap between recently proposed Transformer architectures for exchangeable sequences (Muller et al., 2022; Nguyen & Grover, 2022

Author

Mittal, Daksh; Li, Ang; Yen, Tzu-Ching; Guetta, Daniel; Namkoong, Hongseok

Published

2025

Language

EN