Can I read Poolingformer: Long Document Modeling with Pooling Attention on EtoBox?

Poolingformer: Long Document Modeling with Pooling Attention by Zhang, Hang; Gong, Yeyun; Shen, Yelong; Li, Weisheng; Lv, Jiancheng; Duan, Nan; Chen, Weizhu is a scholarly article available to read on EtoBox.

What is Poolingformer: Long Document Modeling with Pooling Attention about?

In this paper, we introduce a two-level attention schema, Poolingformer, for long document modeling. Its first level uses a smaller sliding window pattern to aggregate information from neighbors. Its second level employs a larger window to increase receptive fields with pooling attention to reduce both computational cost and memory consumption. We first evaluate Poolingformer on two long sequence QA tasks: the monolingual NQ and the multilingual TyDi QA. Experimental results show that Poolingformer sits atop three official leaderboards measured by F1, outperforming previous state-of-the-art models by 1.9 points (79.8 vs. 77.9) on NQ long answer, 1.9 points (79.5 vs. 77.6) on TyDi QA passage answer, and 1.6 points (67.6 vs. 66.0) on TyDi QA minimal answer. We further evaluate Poolingformer on a long sequence summarization task. Experimental results on the arXiv benchmark continue to demonstrate its superior performance.

Author

Zhang, Hang; Gong, Yeyun; Shen, Yelong; Li, Weisheng; Lv, Jiancheng; Duan, Nan; Chen, Weizhu

Published

2021

Language

EN