Can I read Pretraining Data Mixtures in Transformers on EtoBox?

Pretraining Data Mixtures in Transformers by djame seddah is a document available to read on EtoBox.

What is Pretraining Data Mixtures in Transformers about?

- The document studies how effectively transformers can perform in-context learning (ICL) on tasks that are both within and outside their pretraining distribution. - It finds that transformers demonstrate near-optimal unsupervised model selection for tasks well-represented in their pretraining data, selecting the correct task family and learning tasks within it. - However, transformers show failure modes and degraded generalization for extrapolation tasks outside their pretraining distribution, even if si

Author

djame seddah

Language

EN