Can I read Pre-training and Transfer Learning on EtoBox?

Pre-training and Transfer Learning is a book available to read on EtoBox.

What is Pre-training and Transfer Learning about?

We have thus far used dense embeddings to represent words, characters, parts-of-speech, dependency arcs and other sources of information in neural models. Such vectors can be randomly initialised, and then tuned along with other parameter matrices and vectors during training, as a part of the model. But as mentioned in Chapter 13, a better method is to pre-train word embeddings, typically over large-scale raw texts. The resulting vectors are then used as the initial embedding values for a model, which empirically serve as better starting points for fine-tuning. Containing distributed knowledge from a corpus that is significantly larger than that of a typical labelled training set (e.g., a treebank), pre-trained word embedding values can also be used without being fine-tuned during model training. Seminal work on training word embeddings arises from investigation of neural language models. Starting with a simple neural LM, this chapter discusses various approaches to training word and character embeddings, which allow very fast pre-training of word embeddings over large raw texts. We then discuss more complex neural LMs, showing that hidden vectors from these language models can als

Publisher

Cambridge University Press

Published

2021

Language

EN