Can I read DoPAMine: Domain-specific Pre-training Adaptation from seed-guided data Mining on EtoBox?

DoPAMine: Domain-specific Pre-training Adaptation from seed-guided data Mining by Arannil, Vinayak; Narwal, Neha; Bhabesh, Sourav Sanjukta; Thirandas, Sai Nikhil; Wang, Darren Yow-Bang; Horwood, Graham; Chirayath, Alex Anto; Pandeshwar, Gouri is a scholarly article available to read on EtoBox.

What is DoPAMine: Domain-specific Pre-training Adaptation from seed-guided data Mining about?

Large Language Models (LLMs) have shown remarkable ability to generalize effectively across numerous industry domains while executing a range of tasks. Many of these competencies are obtained from the data utilized during the pre-training phase of the Language Models (LMs). However, these models exhibit limitations when tasked with performing in specialized or low-resource industry domains. More recent approaches use LLMs for generating domain-specific synthetic data but most often they lack in truthfulness and complexity. Alternatively, in cases where domain data is available like healthcare and finance most of the LMs are proprietary necessitating the need for a scalable method to curate real world industry specific pre-training data. In this work, we propose an automated and scalable framework - DoPAMine:Domain-specific Pre-training Adaptation from seed-guided data Mining, to mine domain specific training data from a large data corpus for domain adaptation of a LM. The framework leverages the parametric knowledge of a LLM to generate diverse and representative seed data tailored to a specific domain which is then used to mine real world data from a large data corpus like Common

Author

Arannil, Vinayak; Narwal, Neha; Bhabesh, Sourav Sanjukta; Thirandas, Sai Nikhil; Wang, Darren Yow-Bang; Horwood, Graham; Chirayath, Alex Anto; Pandeshwar, Gouri

Published

2024

Language

EN