Can I read Enabling Inclusive Systematic Reviews: Incorporating Preprint Articles with Large Language Model-Driven Evaluations on EtoBox?

Enabling Inclusive Systematic Reviews: Incorporating Preprint Articles with Large Language Model-Driven Evaluations by Yang, Rui; Tong, Jiayi; Wang, Haoyuan; Huang, Hui; Hu, Ziyang; Li, Peiyu; Liu, Nan; Lindsell, Christopher J.; Pencina, Michael J.; Chen, Yong; Hong, Chuan is a scholarly article available to read on EtoBox.

What is Enabling Inclusive Systematic Reviews: Incorporating Preprint Articles with Large Language Model-Driven Evaluations about?

Background. Systematic reviews in comparative effectiveness research require timely evidence synthesis. Preprints accelerate knowledge dissemination but vary in quality, posing challenges for systematic reviews. Methods. We propose AutoConfidence (automated confidence assessment), an advanced framework for predicting preprint publication, which reduces reliance on manual curation and expands the range of predictors, including three key advancements: (1) automated data extraction using natural language processing techniques, (2) semantic embeddings of titles and abstracts, and (3) large language model (LLM)-driven evaluation scores. Additionally, we employed two prediction models: a random forest classifier for binary outcome and a survival cure model that predicts both binary outcome and publication risk over time. Results. The random forest classifier achieved AUROC 0.692 with LLM-driven scores, improving to 0.733 with semantic embeddings and 0.747 with article usage metrics. The survival cure model reached AUROC 0.716 with LLM-driven scores, improving to 0.731 with semantic embeddings. For publication risk prediction, it achieved a concordance index of 0.658, increasing to 0.667

Author

Yang, Rui; Tong, Jiayi; Wang, Haoyuan; Huang, Hui; Hu, Ziyang; Li, Peiyu; Liu, Nan; Lindsell, Christopher J.; Pencina, Michael J.; Chen, Yong; Hong, Chuan

Published

2025

Language

EN