The development of ORACLe: a measure of an organisation’s capacity to engage in evidence-informed health policy

Health Research Policy and Systems

Sue Brennan²  Sally Green²  Abby Haynes⁴  Sally Redman³  Jordan Louviere⁵  Anna Williamson³  Tari Turner¹  Steve R Makkar³ 

[1]World Vision Australia, 1 Vision Drive, Burwood East, Melbourne 3151, Victoria, Australia

[2]School of Public Health and Preventive Medicine, Monash University, Level 6, The Alfred Centre, 99 Commercial Road, Melbourne 3004, VIC, Australia

[3]The Sax Institute, Level 13, Building 10, 235 Jones Street, Ultimo, Sydney 2007, NSW, Australia

[4]Sydney School of Public Health, University of Sydney, Edward Ford Building, Fisher Road, Sydney 2006, NSW, Australia

[5]Institute for Choice, University of South Australia, Level 13, 140 Arthur Street, North Sydney 2060, NSW, Australia

关键词: Research use;    Research;    Policymaker;    Organisation;    Measure;    Knowledge translation;    Health policy;    Evidence;    Discrete choice experiments;    Capacity;    Assessment;   

Others  :  1235362
DOI  :  10.1186/s12961-015-0069-9

 received in 2015-03-13, accepted in 2015-12-04,  发布年份 2016

【 摘 要 】

Background

Evidence-informed policymaking is more likely if organisations have cultures that promote research use and invest in resources that facilitate staff engagement with research. Measures of organisations’ research use culture and capacity are needed to assess current capacity, identify opportunities for improvement, and examine the impact of capacity-building interventions. The aim of the current study was to develop a comprehensive system to measure and score organisations’ capacity to engage with and use research in policymaking, which we entitled ORACLe (Organisational Research Access, Culture, and Leadership).

Method

We used a multifaceted approach to develop ORACLe. Firstly, we reviewed the available literature to identify key domains of organisational tools and systems that may facilitate research use by staff. We interviewed senior health policymakers to verify the relevance and applicability of these domains. This information was used to generate an interview schedule that focused on seven key domains of organisational capacity. The interview was pilot-tested within four Australian policy agencies. A discrete choice experiment (DCE) was then undertaken using an expert sample to establish the relative importance of these domains. This data was used to produce a scoring system for ORACLe.

Results

The ORACLe interview was developed, comprised of 23 questions addressing seven domains of organisational capacity and tools that support research use, including (1) documented processes for policymaking; (2) leadership training; (3) staff training; (4) research resources (e.g. database access); and systems to (5) generate new research, (6) undertake evaluations, and (7) strengthen relationships with researchers. From the DCE data, a conditional logit model was estimated to calculate total scores that took into account the relative importance of the seven domains. The model indicated that our expert sample placed the greatest importance on domains (2), (3) and (4).

Conclusion

We utilised qualitative and quantitative methods to develop a system to assess and score organisations’ capacity to engage with and apply research to policy. Our measure assesses a broad range of capacity domains and identifies the relative importance of these capacities. ORACLe data can be used by organisations keen to increase their use of evidence to identify areas for further development.

【 授权许可】

   
2016 Makkar et al.

【 图 表 】

Fig. 1.

Fig. 2.

Fig. 3.

【 参考文献 】

• [1]Black N. Evidence based policy: proceed with care. Brit Med J. 2001; 323(7307):275-9.
• [2]Brownson RC, Chriqui JF, Stamatakis KA. Understanding evidence-based public health policy. Am J Public Health. 2009; 99(9):1576-83.
• [3]Jacobson N, Butterill D, Goering P. Development of a framework for knowledge translation: understanding user context. J Health Serv Res Policy. 2003; 8(2):94-9.
• [4]Liverani M, Hawkins B, Parkhurst JO. Political and institutional influences on the use of evidence in public health policy. PLoS One. 2013; 8(10):e77404.
• [5]Lavis JN, Lomas J, Hamid M, Sewenkambo NK. Assessing country-level efforts to link research to action. Bull World Health Organ. 2006; 84(8):620-8.
• [6]Bosch-Capblanch X, Lavis JN, Lewin S, Atun R, Røttingen J-A, Dröschel D et al.. Guidance for evidence-informed policies about health systems: rationale for and challenges of guidance development. PLoS Med. 2012; 9(3):e1001185.
• [7]Hanney SR, Gonzalez-Block MA. Yes, research can inform health policy; but can we bridge the 'Do-Knowing It's Been Done' gap? Health Res Policy Syst. 2011; 9:23. BioMed Central Full Text
• [8]Lemay MA, Sa C. The use of academic research in public health policy and practice. Res Evaluat. 2014; 23:79-88.
• [9]Brownson RC, Fielding JE, Maylan CM. Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health. 2009; 30:175-201.
• [10]Ham C, Hunter DJ, Robinson R. Evidence based policymaking: research must inform health policy as well as medical care. Brit Med J. 1995; 310:71-2.
• [11]Orton L, Lloyd-Williams F, Taylor-Robinson D, O’Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PLoS One. 2011; 6(7):e21704.
• [12]Weiss C. Knowledge creep and decision accretion. Sci Commun. 1980; 1(3):381-404.
• [13]Amara N, Ouimet M, Landry R. New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Sci Commun. 2004; 26(1):75-106.
• [14]Campbell DM, Redman S, Jorm L, Cooke M, Zwi AB, Rychetnik L. Increasing the use of evidence in health policy: practice and views of policy makers and researchers. Aust New Zealand Health Policy. 2009; 6:21. BioMed Central Full Text
• [15]Chagnon F, Poullot L, Malo C, Gervais MJ, Pigeon ME. Comparison of determinants of research knowledge utilization by practitioners and administrators in the field of child and family social services. Implement Sci. 2010; 5:41. BioMed Central Full Text
• [16]El-Jardali F, Lavis JN, Ataya N, Jamal D. Use of health systems and policy research evidence in the health policymaking in eastern Mediterranean countries: views and practices of researchers. Implement Sci. 2012; 7:2. BioMed Central Full Text
• [17]Ritter A. How do drug policy makers access research evidence? Int J Drug Policy. 2009; 20:70-5.
• [18]Trostle J, Bronfman M, Langer A. How do researchers influence decision makers? Case studies of Mexican policies. Health Policy Plan. 1999; 14(2):103-14.
• [19]Ellen ME, Lavis JN, Ouimet M, Grimshaw J, Bedard PO. Determining research knowledge infrastructure for healthcare systems: a qualitative study. Implement Sci. 2011; 6:60. BioMed Central Full Text
• [20]Glisson C, Landsverk J, Schoenwald S, Kelleher K, Hoagwood KE, Mayberg S et al.. Assessing the organizational social context (OSC) of mental health services: implications for research and practice. Adm Policy Ment Health. 2008; 35(1–2):98-113.
• [21]Glisson C, Schoenwald SK, Kelleher K, Landsverk J, Hoagwood KE, Mayberg S et al.. Therapist turnover and new program sustainability in mental health clinics as a function of organizational culture, climate, and service structure. Adm Policy Ment Health. 2008; 35(1–2):124-33.
• [22]Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014; 14:2. BioMed Central Full Text
• [23]Oxman AD, Vandvik PO, Lavis JN, Fretheim A, Lewin S. SUPPORT Tools for evidence-informed health Policymaking (STP) 2: Improving how your organisation supports the use of research evidence to inform policymaking. Health Res Policy Syst. 2009; 7 Suppl 1:S2. BioMed Central Full Text
• [24]Dobbins M, DeCorby K, Twiddy RN. A knowledge transfer strategy for public health decision makers. Worldviews Evid Based Nurs. 2004; 1(2):120-8.
• [25]Ettelt S, Mays N. Health services research in Europe and its use for informing policy. J Health Serv Res Policy. 2011; 16 Suppl 2:48-60.
• [26]Evans BA, Snooks H, Howson H, Davies M. How hard can it be to include research evidence and evaluation in local health policy implementation? Results from a mixed methods study. Implement Sci. 2013; 8:17. BioMed Central Full Text
• [27]Gold M. Pathways to the use of health services research in policy. Health Res Educ Trust. 2009; 44(4):1111-36.
• [28]Helmsey-Brown J. Facilitating research utilisation: a cross-sector review of research evidence. Int J Publ Sect Manage. 2004; 17(6):534-52.
• [29]Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003; 1(1):2. BioMed Central Full Text
• [30]Squires JE, Estabrooks CA, O’Rourke HM, Gustavsson P, Newburn-Cook CV, Wallin L. A systematic review of the psychometric properties of self-report research utilization measures used in healthcare. Implement Sci. 2011; 6(1):83. BioMed Central Full Text
• [31]McCaughan D, Thompson C, Cullum N, Sheldon TA, Thompson DR. Acute care nurses' perceptions of barriers to using research information in clinical decision-making. Issues Innov Nursing Prac. 2002; 39(1):46-60.
• [32]Kothari A, Edwards N, Hamel N, Judd M. Is research working for you? Validating a tool to examine the capacity of health organizations to use research. Implement Sci. 2009; 4:46. BioMed Central Full Text
• [33]Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003; 1:2. BioMed Central Full Text
• [34]Redman S, Turner T, Davies H, Haynes A, Williamson A, Milat A et al.. The SPIRIT Action Framework: A structured approach to selecting and testing strategies to increase the use of research in policy. Soc Sci Med. 2015; 136–137:147-55.
• [35]Supporting Policy In health with Research: an Intervention Trial (SPIRIT)-protocol for a stepped wedge trial. BMJ Open. 2014; 4(7):e005293.
• [36]Ryan M, McIntosh E, Shackley P. Methodological issues in the application of conjoint analysis in health care. Health Econ. 1998; 7:373-8.
• [37]Ryan M. Using conjoint analysis to take account of patient preferences and go beyond health outcomes: an application to in vitro fertilisation. Soc Sci Med. 1999; 48:535-46.
• [38]Ryan M. A role for conjoint analysis in technology assessment in health care? Int J Technol Assess Health Care. 1999; 15(3):443-57.
• [39]Ryan M, Hughes J. Using conjoint analysis to assess preferences for miscarriage management. Health Econ. 1997; 6:216-73.
• [40]San Miguel F, Ryan M, McIntosh E. Applying conjoint analysis in economic evaluations: an application to menorrhagia. Appl Econ. 2010; 32:823-33.
• [41]Hair JF, Black WC, Babin BJ, Anderson RE, Tatham RL. Multivariate Data Analysis. 6th ed. Pearson, New Delhi; 2006.
• [42]Carson RT, Louviere J. A common nomenclature for stated preference elicitation approaches. Environ Resource Econ. 2011; 49:539-59.
• [43]Lancsar E, Louviere J. Conducting discrete choice experiments to inform healthcare decision making: a user’s guide. Pharmacoeconomics. 2008; 26(8):661-77.
• [44]Huckel Schneider C, Campbell D, Milat A, Haynes A, Quinn E. What are the key organisational capabilities that facilitate research use in public health policy? Public Health Res Pract. 2014;25(1).
• [45]Louviere J, Flynn TN, Carson RT. Discrete choice experiments are not conjoint analysis. J Choice Modelling. 2010; 3(3):57-72.
• [46]Louviere J, Cong K, Flynn T, Turner T, Williamson A. Using discrete choice experiments to model experts’ decisions about effective organisational use of medical research evidence in medical policy decisions. Med Decis Making. 2016. In press.
• [47]Meijers JM, Maaike AP, Cummings GG, Wallin L, Estabrooks CA, Halfens RYG. Assessing the relationships between contextual factors and research utilization in nursing: systematic literature review. J Adv Nurs. 2006; 55(5):622-35.
• [48]El-Jardali F, Lavis JN, Moat KA, Pantoja T, Ataya N. Capturing lessons learned from evidence-to-policy initiatives through structured reflection. Health Res Policy Syst. 2014; 12:2. BioMed Central Full Text
• [49]Denis JL, Lomas J, Stipich N. Creating receptor capacity for research in the health system: the Executive Training for Research Application (EXTRA) program in Canada. J Health Serv Res Policy. 2008; 13 Suppl 1:1-7.
• [50]Lavis JN, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005; 10 Suppl 1:S1:35-48.
• [51]Mirzoev TN, Omar MA, Green AT, Bird PK, Lund C, Ofori-Atta A et al.. Research-policy partnerships - experiences of the Mental Health and Poverty Project in Ghana, South Africa, Uganda and Zambia. Health Res Policy Syst. 2012; 10:30. BioMed Central Full Text
• [52]Ross S, Lavis J, Rodriguez C, Woodside J, Denis J. Partnership experiences: involving decision-makers in the research process. J Health Serv Res Policy. 2003; 8 Suppl 2:26-34.
• [53]Haynes A, Brennan S, Carter S, O’Connor D, Huckel Schneider C, Turner T et al.. Protocol for the process evaluation of a complex intervention designed to increase the use of research in health policy and program organisations (the SPIRIT study). Implement Sci. 2014; 9:113. BioMed Central Full Text
• [54]Lavis JN, Boyko JA, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 14: Organising and using policy dialogues to support evidence-informed policymaking. Health Res Policy Syst. 2009; 7 Suppl 1:S14. BioMed Central Full Text
• [55]Orme D. Getting Started with Conjoint Analysis: Strategies for Product Design and Pricing Research. 2nd ed. Research Publishers LLC, Madison, Wisconsin; 2010.
• [56]Marshall D, Bridges JF, Hauber AB, Cameron R, Donnalley L, Fyie K et al.. Conjoint analysis applications in health – how are studies being designed and reported? Patient. 2010; 3(4):249-56.
• [57]Thomas K, Allen S. The learning organisation: a meta-analysis of themes in literature. Learn Organ. 2006; 13(2):123-39.
• [58]Ferlie E, Dopson SUE, Fitzgerald L, Locock L. Renewing policy to support evidence-based health care. Public Adm. 2009; 87(4):837-52.
• [59]Greenhalgh T, Robert G, Bate P, Kyriakidou O, Macfarlane F, Peacock R. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004; 82(4):581-629.
• [60]Makkar SR, Brennan S, Williamson A, Turner T, Redman S, Green S. The development of SAGE: a tool to evaluate how policymakers’ engage with and use research in health policymaking. Res Evaluat. 2016. In press.
• [61]Makkar SR, Williamson A, Turner T, Redman S, Louviere J. Using conjoint analysis to develop a system to score research engagement actions by health decision makers. Health Res Policy Syst. 2015; 13:22. BioMed Central Full Text
• [62]Makkar SR, Williamson A, Turner T, Redman S, Louviere J. Using conjoint analysis to develop a system of scoring policymakers’ use of research in policy and program development. Health Res Policy Syst. 2015; 13:35. BioMed Central Full Text
• [63]Contandriopoulos D, Lemire M, Denis JL, Tremblay E. Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature. Milbank Q. 2010; 88(4):444-83.
• [64]Hyder AA, Corluka A, Winch PJ, El-Shinnawy A, Ghassany H, Malekafzali H et al.. National policymakers speak out: are researchers giving them what they need? Health Policy Plan. 2011; 26:73-82.
• [65]Hennink M, Stephenson R. Using research to inform health policy: barriers and strategies in developing countries. J Health Commun. 2006; 10:163-80.
• [66]Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J. How can research organizations more effectively transfer research knowledge to decision makers? Milbank Q. 2003; 81(2):221-48.
• [67]Buse K, Mays N, Walt G. Making health policy. Open University Press, Glasgow; 2005.
• [68]Koon ADRKD, Tran NT, Ghaffar A. Embedding health policy and systems research into decision-making processes in low- and middle-income countries. Health Res Policy Syst. 2013; 11:30. BioMed Central Full Text
• [69]Petticrew M, Whitehead M, Macintyre SJ, Graham H, Egan M. Evidence for public health policy on inequalities: 1: The reality according to policymakers. J Epidemiol Community Health. 2004; 58(10):811-6.
• [70]Albert M, Fretheim A, Maiga D. Factors influencing the utilization of research findings by health policy-makers in a developing country: the selection of Mali’s essential medicines. Health Res Policy Syst. 2007; 5:2. BioMed Central Full Text
• [71]Dobbins M, Rosenbaum P, Plews N, Law M, Fysh A. Information transfer: what do decision makers want and need from researchers? Implement Sci. 2007; 2:20. BioMed Central Full Text
• [72]Burris H, Parkhurst J, Adu-Sarkodie Y, Mayaud P. Getting research into policy - Herpes simplex virus type-2 (HSC-2) treatment and HIV infection: international guidelines formulation and the case of Ghana. Health Res Policy Syst. 2011; 9 Suppl 1:S5. BioMed Central Full Text
• [73]Anderson M, Cosby J, Swan B, Moore H, Broekhoven M. The use of research in local health service agencies. Soc Sci Med. 1999; 49(8):1007-19.
• [74]Danseco E, Sundar P, Kasprzak S, Witteveen T, Woltman H, Manion I. Are we there yet? Evaluation and the knowledge translation journey. J Can Acad Child Adolesc Psychiatry. 2009; 18(1):7-15.
• [75]Beyer JM, Trice HM. The utilization process: a conceptual framework and synthesis of empirical findings. Adm Sci Q. 1982; 27(4):591-622.
• [76]Gerhardus A, Kielmann K, Sanou A. The use of research for decision making in the health sector: the case of ‘shared care’ in Burkina Faso. In: Lessons in Research to Action and Policy. COHRED, Document, The Council on Health Research for Development, Geneva; 2000: p.19-27.
• [77]Ginsburg LR, Lewis S, Zackheim L, Casebeer A. Revisiting interaction in knowledge interaction. Implement Sci. 2007; 2:34. BioMed Central Full Text
• [78]Innvaer S, Vist G, Trommald M, Oxman A. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002; 7(4):239-44.
• [79]Jacobson N, Butterill D, Goering P. Consulting as a strategy for knowledge transfer. Milbank Q. 2005; 83(2):299-321.
• [80]Jansen MW, De Leeuw E, Hoejimakers M, De Vries NK. Working at the nexus between public health policy, practice and research. Dynamics of knowledge sharing in the Netherlands. Health Res Policy Syst. 2012; 10:33. BioMed Central Full Text
• [81]Kothari A, Birch S, Charles C. ‘Interaction’ and research utilisation in health policies and programs: does it work? Health Policy. 2005; 71(1):117-25.
• [82]Lavis J. Research, public policymaking, and knowledge-translation processes: Canadian efforts to build bridges. J Contin Educ Heal Prof. 2006; 26:37-45.
• [83]Lomas J. Connecting Research and Policy. ISUMA Canadian J Policy Res. 2000; Spring:140-4.
• [84]Mbonye AK, Magnussen P. Translating health research evidence into policy and practice in Uganda. Malar J. 2013; 12:274. BioMed Central Full Text
• [85]Milat A, Laws R, King L, Newson R, Rychetnik L, Rissel C et al.. Policy and practice impacts of applied research: a case study analysis of the New South Wales Health promotion Demonstration Research Grants Scheme 2000–2006. Health Res Policy Syst. 2013; 11:5. BioMed Central Full Text
• [86]Patera N, Wild C. Linking public health research with policy and practice in three European countries. J Public Health. 2013; 21(5):473-9.
• [87]Rich R. Utilization of social science related information by federal policy-makers. Inquiry. 1975; 12(3):239-45.
• [88]Whitehead M, Petticrew M, Graham H, Macintyre S, Bambra C, Egan M. Evidence for public health policy on inequalities 2: assembling the evidence jigsaw. J Epidemiol Community Health. 2004; 58:817-21.
• [89]McGrath PJ, Lingley-Pottie P, Emberly DJ, Thurston C, McLean C. Integrated knowledge translation in mental health: family help as an example. J Can Acad Child Adolesc Psychiatry. 2009; 18(1):30-7.
• [90]Campbell D, Donald B, Moore G, Frew D. Evidence check: knowledge brokering to commission research reviews for policy. Evidence Policy. 2011; 7(1):97-107.
• [91]Lomas J. The in-between world of knowledge brokering. Br Med J. 2007; 334(7585):129-32.