Assessor burden, inter-rater agreement and user experience of the RoB-SPEO tool for assessing risk of bias in studies estimating prevalence of exposure to occupational risk factors: An analysis from the WHO/ILO Joint Estimates of the Work-related Burden of Disease and Injury

Environment International

Monique H.W. Frings-Dresen¹  Kathrine Sørensen²  Lode Godderis³  Rebecca L. Morgan⁴  Daniela Pachito⁵  Diana Gagliardi⁶  Tom Loney⁷  Alberto Modenese⁸  Thomas Tenkate⁹  Daria Sgargi⁹  Paul T.J. Scheepers^1,10  Marília Silva Paulo^1,11  Natalie C. Momen^1,12  Kai N. Streicher^1,12  Vivi Schlünssen^1,13  Daniele Mandrioli^1,14  Frank Pega^1,15  Grace Sembajwe^1,16  Liliane R. Teixeira^1,17  Alexis Descatha^1,18  Denise T.C. da Silva^1,19 

[1] AP-HP (Paris Hospital), Occupational Health Unit, Poincaré University Hospital, Garches, France;Fundação Getúlio Vargas, Bela Vista, São Paulo, SP, Brazil;Inserm, U1168 UMS 011, Villejuif, France;;KIR Department (Knowledge, Information &Research), IDEWE, External Service for Prevention and Protection at Work, Leuven, Belgium;Versailles St-Quentin Univ-Paris Saclay Univ (UVSQ), UMS 011, UMR-S 1168, France;Amsterdam UMC, University of Amsterdam, Department Public and Occupational Health/Coronel Institute of Occupational Health, Amsterdam Research Institute, Amsterdam, the Netherlands;Centre for Environment and Health, KU Leuven, Leuven, Belgium;Cesare Maltoni Cancer Research Center, Ramazzini Institute, Bologna, Italy;College of Medicine, Mohammed Bin Rashid University of Medicine and Health Sciences, Dubai, United Arab Emirates;Department of Biomedical, Metabolic and Neural Sciences, University of Modena and Reggio Emilia, Modena, Italy;Department of Environment, Climate Change and Health, World Health Organization, Geneva, Switzerland;Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada;Inail, Department of Occupational and Environmental Medicine, Epidemiology and Hygiene, Rome, Italy;Institute of Public Health, College of Medicine and Health Sciences, United Arab Emirates University, Al Ain, United Arab Emirates;Núcleo de Avaliação de Tecnologias em Saúde, Hospital Sírio-Libanês, Bela Vista, São Paulo, SP, Brazil;Radboud Institute for Health Sciences, Radboudumc, Nijmegen, the Netherlands;UNIV Angers, CHU Angers, Univ Rennes, Inserm, EHESP, Irset (Institut de recherche en santé, environnement et travail) - UMR_S 1085, Angers, France;Workers' Health and Human Ecology Research Center, National School of Public Health Sergio Arouca, Oswaldo Cruz Foundation, Rio de Janeiro, RJ, Brazil;

关键词: Bias;    Systematic review methods;    Prevalence;    Occupational exposure;    Occupational epidemiology;   

DOI  :  

来源: DOAJ

【 摘 要 】

Background: As part of the development of the World Health Organization (WHO)/International Labour Organization (ILO) Joint Estimates of the Work-related Burden of Disease and Injury, WHO and ILO carried out several systematic reviews to determine the prevalence of exposure to selected occupational risk factors. Risk of bias assessment for individual studies is a critical step of a systematic review. No tool existed for assessing the risk of bias in prevalence studies of exposure to occupational risk factors, so WHO and ILO developed and pilot tested the RoB-SPEO tool for this purpose. Here, we investigate the assessor burden, inter-rater agreement, and user experience of this new instrument, based on the abovementioned WHO/ILO systematic reviews. Methods: Twenty-seven individual experts applied RoB-SPEO to assess risk of bias. Four systematic reviews provided a total of 283 individual assessments, carried out for 137 studies. For each study, two or more assessors independently assessed risk of bias across the eight RoB-SPEO domains selecting one of RoB-SPEO’s six ratings (i.e., “low”, “probably low”, “probably high”, “high”, “unclear” or “cannot be determined”). Assessors were asked to report time taken (i.e. indicator of assessor burden) to complete each assessment and describe their user experience. To gauge assessor burden, we calculated the median and inter-quartile range of times taken per individual risk of bias assessment. To assess inter-rater reliability, we calculated a raw measure of inter-rater agreement (Pi) for each RoB-SPEO domain, between Pi = 0.00, indicating no agreement and Pi = 1.00, indicating perfect agreement. As subgroup analyses, Pi was also disaggregated by systematic review, assessor experience with RoB-SPEO (≤10 assessments versus > 10 assessments), and assessment time (tertiles: ≤25 min versus 26–66 min versus ≥ 67 min). To describe user experience, we synthesised the assessors’ comments and recommendations. Results: Assessors reported a median of 40 min to complete one assessment (interquartile range 21–120 min). For all domains, raw inter-rater agreement ranged from 0.54 to 0.82. Agreement varied by systematic review and assessor experience with RoB-SPEO between domains, and increased with increasing assessment time. A small number of users recommended further development of instructions for selected RoB-SPEO domains, especially bias in selection of participants into the study (domain 1) and bias due to differences in numerator and denominator (domain 7). Discussion: Overall, our results indicated good agreement across the eight domains of the RoB-SPEO tool. The median assessment time was comparable to that of other risk of bias tools, indicating comparable assessor burden. However, there was considerable variation in time taken to complete assessments. Additional time spent on assessments may improve inter-rater agreement. Further development of the RoB-SPEO tool could focus on refining instructions for selected RoB-SPEO domains and additional testing to assess agreement for different topic areas and with a wider range of assessors from different research backgrounds.

【 授权许可】

Unknown