The Society for Implementation Research Collaboration Instrument Review Project: A methodology to promote rigorous evaluation

Implementation Science

Katherine A Comtois²  Melanie Barwick⁵  Mimi Kim³  Bryan J Weiner⁶  Ruben G Martinez⁴  Cameo F Stanick¹  Cara C Lewis² 

[1] Department of Psychology, University of Montana, 32 Campus Dr., Skaggs Bldg. 362, Missoula 59812, MT, USA;Department of Psychiatry and Behavioral Sciences, Harborview Medical Center, School of Medicine, University of Washington, 325 9th Ave, Seattle 98104, WA, USA;North Carolina Translational and Clinical Sciences Institute, University of North Carolina at Chapel Hill, 160 N. Medical Drive, Brinkhous-Bullitt, 2nd Floor, CB# 7064,, Chapel Hill 27599-7064, NC, USA;Department of Psychology, Virginia Commonwealth University, 806 West Franklin St, Richmond 23220, VA, USA;Child Health Evaluative Sciences, Research Institute, The Hospital for Sick Children, 555 University Avenue, Toronto M5G 1X8, ON, Canada;University of North Carolina at Chapel Hill, 1102-C McGavran-Greenberg Hall, 135 Dauer Drive, Campus Box 7411, Chapel Hill 27599-7411, NC, USA

关键词: Psychometrics;    Evidence-based assessment;    Instruments;    Dissemination;    Implementation;   

Others  :  1139402
DOI  :  10.1186/s13012-014-0193-x

 received in 2014-07-03, accepted in 2014-12-12,  发布年份 2015

【 摘 要 】

Background

Identification of psychometrically strong instruments for the field of implementation science is a high priority underscored in a recent National Institutes of Health working meeting (October 2013). Existing instrument reviews are limited in scope, methods, and findings. The Society for Implementation Research Collaboration Instrument Review Project’s objectives address these limitations by identifying and applying a unique methodology to conduct a systematic and comprehensive review of quantitative instruments assessing constructs delineated in two of the field’s most widely used frameworks, adopt a systematic search process (using standard search strings), and engage an international team of experts to assess the full range of psychometric criteria (reliability, construct and criterion validity). Although this work focuses on implementation of psychosocial interventions in mental health and health-care settings, the methodology and results will likely be useful across a broad spectrum of settings. This effort has culminated in a centralized online open-access repository of instruments depicting graphical head-to-head comparisons of their psychometric properties. This article describes the methodology and preliminary outcomes.

Methods

The seven stages of the review, synthesis, and evaluation methodology include (1) setting the scope for the review, (2) identifying frameworks to organize and complete the review, (3) generating a search protocol for the literature review of constructs, (4) literature review of specific instruments, (5) development of an evidence-based assessment rating criteria, (6) data extraction and rating instrument quality by a task force of implementation experts to inform knowledge synthesis, and (7) the creation of a website repository.

Results

To date, this multi-faceted and collaborative search and synthesis methodology has identified over 420 instruments related to 34 constructs (total 48 including subconstructs) that are relevant to implementation science. Despite numerous constructs having greater than 20 available instruments, which implies saturation, preliminary results suggest that few instruments stem from gold standard development procedures. We anticipate identifying few high-quality, psychometrically sound instruments once our evidence-based assessment rating criteria have been applied.

Conclusions

The results of this methodology may enhance the rigor of implementation science evaluations by systematically facilitating access to psychometrically validated instruments and identifying where further instrument development is needed.

【 授权许可】

   
2015 Lewis et al.; licensee BioMed Central.

【 图 表 】

Figure 1.

Figure 2.

【 参考文献 】

• [1]Cook DA, Beckman TJ: Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med 2006, 119:166.e7.
• [2]Martinez RG, Lewis CC, Weiner BJ: Instrumentation issues in implementation science. Implement Sci 2014, 9:118. BioMed Central Full Text
• [3]Weiner B, Amick H, Lee S-Y: Conceptualization and measurement of organizational readiness for change: a review of the literature in health services research and other fields. Med Care Res Rev 2008, 65(4):379-436.
• [4]Chor KHB, Wisdom JP, Olin S-CS, Hoagwood KE, Horwitz SM: Measures for predictors of innovation adoption. Adm Policy Ment Health Ment Health Serv Res 2014, 1–29:.. doi:10.1007/s10488-014-0551-7
• [5]Chaudoir SR, Dugan AG, Barr CH: Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci 2013, 8:22. BioMed Central Full Text
• [6]Rabin BA, Purcell P, Naveed S, Moser RP, Henton MD, Proctor EK, Brownson RC, Glasgow RE: Advancing the application, quality and harmonization of implementation science measures. Implement Sci 2012, 7:119. BioMed Central Full Text
• [7]Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC: Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009, 4:50. BioMed Central Full Text
• [8]Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B: Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health Ment Health Serv Res 2009, 36:24-34.
• [9]Tabak RG, Khoong EC, Chambers DA, Brownson RC: Bridging research and practice: models for dissemination and implementation research. Am J Prev Med 2012, 43:337-350.
• [10]Rycroft-Malone J: The PARIHS framework – a framework for guiding the implementation of evidence-based practice. J Nurs Care Qual 2004, 19:297-304.
• [11]Rogers EM: Diffusion of Innovations. Free Press, New York; 2003.
• [12]Feldstein AC, Glasgow RE: A practical, robust implementation and sustainability model (PRISM). Jt Comm J Qual Patient Saf 2008, 34:228-243.
• [13]Damschroder LJ, Goodrich DE, Robinson CH, Fletcher CE, Lowery JC: A systematic exploration of differences in contextual factors related to implementing the MOVE! weight management program in VA: a mixed methods study. BMC Health Serv Res 2011, 11:248. BioMed Central Full Text
• [14]Arksey H, O’Malley L: Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005, 8:19-32.
• [15]Straus S, Tetroe J, Graham ID: Knowledge Translation in Health Care: Moving from Evidence to Practice. Wiley, New York; 2013.
• [16]McKibbon KA, Lokker C, Wilczynski NL, Ciliska D, Dobbins M, Davis DA, Haynes RB, Straus SE: A cross-sectional study of the number and frequency of terms used to refer to knowledge translation in a body of health literature in 2006: a Tower of Babel. Implement Sci 2010, 5:16. BioMed Central Full Text
• [17]Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, Glass JE, York JL: A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev 2012, 69:123-157.
• [18]Hunsley J, Mash EJ: Introduction to the special section on developing guidelines for the evidence-based assessment (EBA) of adult disorders. Psychol Assess 2005, 17:251.
• [19]Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, Bouter LM, de Vet HC: Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007, 60:34-42.
• [20]Aarons GA: Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Ment Health Serv Res 2004, 6:61-74.
• [21]Addis ME, Krasnow AD: A national survey of practicing psychologists’ attitudes toward psychotherapy treatment manuals. J Consult Clin Psychol 2000, 68:331.
• [22]Beidas RS, Edmunds JM, Marcus SC, Kendall PC: Training and consultation to promote implementation of an empirically supported treatment: a randomized trial. Psychiatr Serv 2012, 63:660-665.
• [23]Michie S, Fixsen D, Grimshaw JM, Eccles MP: Specifying and reporting complex behaviour change interventions: the need for a scientific method. Implement Sci 2009, 4:1-6. BioMed Central Full Text
• [24]Walsh WB, Betz NE: Tests and Assessment. Prentice-Hall, Upper Saddle River; 1995.
• [25]Aarons GA, Ehrhart MG, Farahnak LR: The implementation leadership scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci 2014, 9:45. BioMed Central Full Text
• [26]Glasgow RE: What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Educ Behav 2013, 40:257-265.
• [27]Eccles MP, Mittman BS: Welcome to implementation science. Implement Sci 2006, 1:1-3. BioMed Central Full Text
• [28]Weisz JR, Ng MY, Bearman SK: Odd couple? Reenvisioning the relation between science and practice in the dissemination-implementation era. Clin Psych Sci 2014, 2:58-74.