Can I read Offensive Language Detection with BERT-based models, By Customizing Attention Probabilities on EtoBox?

Offensive Language Detection with BERT-based models, By Customizing Attention Probabilities by Alavi, Peyman; Nikvand, Pouria; Shamsfard, Mehrnoush is a scholarly article available to read on EtoBox.

What is Offensive Language Detection with BERT-based models, By Customizing Attention Probabilities about?

This paper describes a novel study on using `Attention Mask' input in transformers and using this approach for detecting offensive content in both English and Persian languages. The paper's principal focus is to suggest a methodology to enhance the performance of the BERT-based models on the `Offensive Language Detection' task. Therefore, we customize attention probabilities by changing the `Attention Mask' input to create more efficacious word embeddings. To do this, we firstly tokenize the training set of the exploited datasets (by BERT tokenizer). Then, we apply Multinomial Naive Bayes to map these tokens to two probabilities. These probabilities indicate the likelihood of making a text non-offensive or offensive, provided that it contains that token. Afterwards, we use these probabilities to define a new term, namely Offensive Score. Next, we create two separate (because of the differences in the types of the employed datasets) equations based on Offensive Scores for each language to re-distribute the `Attention Mask' input for paying more attention to more offensive phrases. Eventually, we put the F1-macro score as our evaluation metric and fine-tune several combinations of BE

Author

Alavi, Peyman; Nikvand, Pouria; Shamsfard, Mehrnoush

Published

2021

Language

EN