Can I read VL-Reader: Vision and Language Reconstructor is an Effective Scene Text Recognizer on EtoBox?

VL-Reader: Vision and Language Reconstructor is an Effective Scene Text Recognizer by Zhong, Humen; Yang, Zhibo; Li, Zhaohai; Wang, Peng; Tang, Jun; Cheng, Wenqing; Yao, Cong is a scholarly article available to read on EtoBox.

What is VL-Reader: Vision and Language Reconstructor is an Effective Scene Text Recognizer about?

Text recognition is an inherent integration of vision and language, encompassing the visual texture in stroke patterns and the semantic context among the character sequences. Towards advanced text recognition, there are three key challenges: (1) an encoder capable of representing the visual and semantic distributions; (2) a decoder that ensures the alignment between vision and semantics; and (3) consistency in the framework during pre-training, if it exists, and fine-tuning. Inspired by masked autoencoding, a successful pre-training strategy in both vision and language, we propose an innovative scene text recognition approach, named VL-Reader. The novelty of the VL-Reader lies in the pervasive interplay between vision and language throughout the entire process. Concretely, we first introduce a Masked Visual-Linguistic Reconstruction (MVLR) objective, which aims at simultaneously modeling visual and linguistic information. Then, we design a Masked Visual-Linguistic Decoder (MVLD) to further leverage masked vision-language context and achieve bi-modal feature interaction. The architecture of VL-Reader maintains consistency from pre-training to fine-tuning. In the pre-training stage,

Author

Zhong, Humen; Yang, Zhibo; Li, Zhaohai; Wang, Peng; Tang, Jun; Cheng, Wenqing; Yao, Cong

Published

2024

Language

EN