Can I read A Fast Randomized Algorithm for Massive Text Normalization on EtoBox?

A Fast Randomized Algorithm for Massive Text Normalization by Jiang, Nan; Luo, Chen; Lakshman, Vihan; Dattatreya, Yesh; Xue, Yexiang is a scholarly article available to read on EtoBox.

What is A Fast Randomized Algorithm for Massive Text Normalization about?

Many popular machine learning techniques in natural language processing and data mining rely heavily on high-quality text sources. However real-world text datasets contain a significant amount of spelling errors and improperly punctuated variants where the performance of these models would quickly deteriorate. Moreover, real-world, web-scale datasets contain hundreds of millions or even billions of lines of text, where the existing text cleaning tools are prohibitively expensive to execute over and may require an overhead to learn the corrections. In this paper, we present FLAN, a scalable randomized algorithm to clean and canonicalize massive text data. Our algorithm relies on the Jaccard similarity between words to suggest correction results. We efficiently handle the pairwise word-to-word comparisons via Locality Sensitive Hashing (LSH). We also propose a novel stabilization process to address the issue of hash collisions between dissimilar words, which is a consequence of the randomized nature of LSH and is exacerbated by the massive scale of real-world datasets. Compared with existing approaches, our method is more efficient, both asymptotically and in empirical evaluations, a

Author

Jiang, Nan; Luo, Chen; Lakshman, Vihan; Dattatreya, Yesh; Xue, Yexiang

Published

2021

Language

EN