Can I read Infusing Biomedical Knowledge into BERT for Chinese Biomedical NLP Tasks with Adversarial Training on EtoBox?

Infusing Biomedical Knowledge into BERT for Chinese Biomedical NLP Tasks with Adversarial Training by Shan Jiang; Huanhuan Wu; Lingyun Luo is a scholarly article available to read on EtoBox.

What is Infusing Biomedical Knowledge into BERT for Chinese Biomedical NLP Tasks with Adversarial Training about?

Biomedical text mining is becoming increasingly important. Recently, biomedical pre-trained language models such as BioBERT and SciBERT, which can capture biomedical knowledge from text, have achieved promising results in biomedical NLP tasks. However, most biomedical pre-trained language models rely on the traditional masked language model (MLM) pre-training strategy, which cannot fully capture the semantic relations of context. It is challenging to learn biomedical knowledge via language models in the Chinese biomedical fields due to the lack of training resources and the extreme complexity and diversity of Chinese medical terminologies. To this end, we propose MedBERT-adv, which utilizes a biomedical knowledge infusion method that can effectively complement BERT-like models. Instead of using time-consuming medical expert annotation and inaccurate automatic annotation, we use the article structure in Baidu Encyclopedia as a weakly supervised signal, utilizing each medical term and its category as labels to pre-train the model. We also leverage adversarial training strategies like FGM for fine-tuning downstream tasks to further improve the performance of MedBERT-adv. We experiment

Author

Shan Jiang; Huanhuan Wu; Lingyun Luo

Publisher

ACM

Published

2022

Language

EN