Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records

Systematic Reviews

Research

Gerbrich Ferdinands¹  Jelle Jasper Teijema¹  Ayoub Bagheri¹  Daniel L. Oberski¹  Rens van de Schoot¹  Raoul Schram²  Jonathan de Bruin²  Lars Tummers³ 

[1] Department of Methodology and Statistics, Faculty of Social and Behavioral Sciences, Utrecht University, Utrecht, Netherlands;Department of Research and Data Management Services, Information Technology Services, Utrecht University, Utrecht, The Netherlands;School of Governance, Faculty of Law, Economics and Governance, Utrecht University, Utrecht, The Netherlands;

关键词: Systematic reviews;    Active learning;    Screening prioritization;    Machine learning;    Data mining;    Computer simulation;   

DOI  :  10.1186/s13643-023-02257-7

 received in 2020-08-20, accepted in 2023-05-16,  发布年份 2023

来源: Springer

【 摘 要 】

BackgroundConducting a systematic review demands a significant amount of effort in screening titles and abstracts. To accelerate this process, various tools that utilize active learning have been proposed. These tools allow the reviewer to interact with machine learning software to identify relevant publications as early as possible. The goal of this study is to gain a comprehensive understanding of active learning models for reducing the workload in systematic reviews through a simulation study.MethodsThe simulation study mimics the process of a human reviewer screening records while interacting with an active learning model. Different active learning models were compared based on four classification techniques (naive Bayes, logistic regression, support vector machines, and random forest) and two feature extraction strategies (TF-IDF and doc2vec). The performance of the models was compared for six systematic review datasets from different research areas. The evaluation of the models was based on the Work Saved over Sampling (WSS) and recall. Additionally, this study introduces two new statistics, Time to Discovery (TD) and Average Time to Discovery (ATD).ResultsThe models reduce the number of publications needed to screen by 91.7 to 63.9% while still finding 95% of all relevant records (WSS@95). Recall of the models was defined as the proportion of relevant records found after screening 10% of of all records and ranges from 53.6 to 99.8%. The ATD values range from 1.4% till 11.7%, which indicate the average proportion of labeling decisions the researcher needs to make to detect a relevant record. The ATD values display a similar ranking across the simulations as the recall and WSS values.ConclusionsActive learning models for screening prioritization demonstrate significant potential for reducing the workload in systematic reviews. The Naive Bayes + TF-IDF model yielded the best results overall. The Average Time to Discovery (ATD) measures performance of active learning models throughout the entire screening process without the need for an arbitrary cut-off point. This makes the ATD a promising metric for comparing the performance of different models across different datasets.

【 授权许可】

CC BY   
© The Author(s) 2023

【 图 表 】

Fig. 2

12936_2023_4634_Article_IEq5.gif

12936_2023_4634_Article_IEq2.gif

Fig. 3

Fig. 1

Fig. 2

Fig. 4

Fig. 2

Fig. 1

Fig. 13

Fig. 7