Can I read Probing Context Localization of Polysemous Words in Pre-trained Language Model Sub-Layers on EtoBox?

Probing Context Localization of Polysemous Words in Pre-trained Language Model Sub-Layers by Vijayakumar, Soniya; van Genabith, Josef; Ostermann, Simon is a scholarly article available to read on EtoBox.

What is Probing Context Localization of Polysemous Words in Pre-trained Language Model Sub-Layers about?

In the era of high performing Large Language Models, researchers have widely acknowledged that contextual word representations are one of the key drivers in achieving top performances in downstream tasks. In this work, we investigate the degree of contextualization encoded in the fine-grained sub-layer representations of a Pre-trained Language Model (PLM) by empirical experiments using linear probes. Unlike previous work, we are particularly interested in identifying the strength of contextualization across PLM sub-layer representations (i.e. Self-Attention, Feed-Forward Activation and Output sub-layers). To identify the main contributions of sub-layers to contextualisation, we first extract the sub-layer representations of polysemous words in minimally different sentence pairs, and compare how these representations change through the forward pass of the PLM network. Second, by probing on a sense identification classification task, we try to empirically localize the strength of contextualization information encoded in these sub-layer representations. With these probing experiments, we also try to gain a better understanding of the influence of context length and context richness on

Author

Vijayakumar, Soniya; van Genabith, Josef; Ostermann, Simon

Published

2024

Language

EN