Using conjoint analysis to develop a system of scoring policymakers’ use of research in policy and program development

Health Research Policy and Systems

Jordan Louviere¹  Sally Redman³  Tari Turner²  Anna Williamson³  Steve R Makkar³ 

[1] School of Marketing, University of South Australia, Level 4, Yungondi Building, North Terrace, Adelaide 5000, South Australia, Australia;World Vision Australia, 1 Vision Drive, Burwood East, Melbourne 3151, Victoria, Australia;The Sax Institute, Level 13, Building 10, 235 Jones Street, Ultimo 2007, NSW, Australia

关键词: Utilisation;    Use;    Research;    Policymaker;    Measurement;    Knowledge translation;    Health policy;    Evidence-informed policy;    Evidence-based policy;    Conjoint analysis;   

Others  :  1222738
DOI  :  10.1186/s12961-015-0022-y

 received in 2014-11-26, accepted in 2015-07-07,  发布年份 2015

【 摘 要 】

Background

The importance of utilising the best available research evidence in the development of health policies, services, and programs is increasingly recognised, yet few standardised systems for quantifying policymakers’ research use are available. We developed a comprehensive measurement and scoring tool that assesses four domains of research use (i.e. instrumental, conceptual, tactical, and imposed). The scoring tool breaks down each domain into its key subactions like a checklist. Our aim was to develop a tool that assigned appropriate scores to each subaction based on its relative importance to undertaking evidence-informed health policymaking. In order to establish the relative importance of each research use subaction and generate this scoring system, we conducted conjoint analysis with a sample of knowledge translation experts.

Methods

Fifty-four experts were recruited to undertake four choice surveys. Respondents were shown combinations of research use subactions called profiles, and rated on a 1 to 9 scale whether each profile represented a limited (1–3), moderate (4–6), or extensive (7–9) example of research use. Generalised Estimating Equations were used to analyse respondents’ choice data, which calculated a utility coefficient for each subaction. A large utility coefficient indicated that a subaction was particularly influential in guiding experts’ ratings of extensive research use.

Results

Utility coefficients were calculated for each subaction, which became the points assigned to the subactions in the scoring system. The following subactions yielded the largest utilities and were regarded as the most important components of each research use domain: using research to directly influence the core of the policy decision; using research to inform alternative perspectives to deal with the policy issue; using research to persuade targeted stakeholders to support a predetermined decision; and using research because it was a mandated requirement by the policymaker’s organisation.

Conclusions

We have generated an empirically derived and context-sensitive means of measuring and scoring the extent to which policymakers used research to inform the development of a policy document. The scoring system can be used by organisations to not only quantify the extent of their research use, but also to provide them with insights into potential strategies to improve subsequent research use.

【 授权许可】

   
2015 Makkar et al.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

Figure 4.

【 参考文献 】

• [1]Hausman A. Implications of evidence-based practice for community health. Am J Community Psychol. 2002; 30(3):453-67.
• [2]Black N. Evidence based policy: proceed with care. Brit Med J. 2001; 323(7307):275-9.
• [3]Lemay MA, Sa C. The use of academic research in public health policy and practice. Res Evaluat. 2014; 23:79-88.
• [4]Jacobson N, Butterill D, Goering P. Consulting as a strategy for knowledge transfer. Milbank Q. 2005; 83(2):299-321.
• [5]Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA et al.. Examining the role of health services research in public policymaking. Milbank Q. 2002; 80(1):125-54.
• [6]LaRocca R, Yost J, Dobbins M, Ciliska D, Butt M. The effectiveness of knowledge translation strategies used in public health: a systematic review. BMC Public Health. 2012; 12:751. BioMed Central Full Text
• [7]El-Jardali F, Lavis JN, Moat KA, Pantoja T, Ataya N. Capturing lessons learned from evidence-to-policy initiatives through structured reflection. Health Res Policy Syst. 2014; 12:2. BioMed Central Full Text
• [8]Brownson RC, Chriqui JF, Stamatakis KA. Understanding evidence-based public health policy. Am J Public Health. 2009; 99(9):1576-83.
• [9]Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003; 1:2. BioMed Central Full Text
• [10]Buchan H. Gaps between best evidence and practice: causes for concern. Med J Australia. 2004; 180:S48-9.
• [11]Bowen S, Erickson T, Martens PJ, Crockett S. More than “using research”: the real challenges in promoting evidence-informed decision-making. Health Policy. 2009; 4(3):87-102.
• [12]Fielding JE, Briss PA. Promoting evidence-based public health policy: can we have better evidence and more action? Health Aff (Millwood). 2006; 4:969-78.
• [13]Andre FE, Booy R, Bock HL, Clemens J, Datta SK, John TJ et al.. Vaccination greatly reduces disease, disability, death and inequity worldwide. Bull World Health Organ. 2008; 86(2):140-6.
• [14]Hanna JN, Hills SL, Humphreys JL. Impact of hepatitis A vaccination of Indigenous children on notifications of hepatitis A in north Queensland. Med J Aust. 2004; 181(9):482-5.
• [15]Morrato EH, Elias M, Gericke CA. Using population-based routine data for evidence-based health policy decisions: lessons from three examples of setting and evaluating national health policy in Australia, the UK and the USA. J Public Health. 2007; 19(4):463-71.
• [16]Milat A, Laws R, King L, Newson R, Rychetnik L, Rissel C et al.. Policy and practice impacts of applied research: a case study analysis of the New South Wales Health promotion Demonstration Research Grants Scheme 2000-2006. Health Res Policy Syst. 2013; 11:5. BioMed Central Full Text
• [17]Chagnon F, Poullot L, Malo C, Gervais MJ, Pigeon ME. Comparison of determinants of research knowledge utilization by practitioners and administrators in the field of child and family social services. Implement Sci. 2010; 5:41. BioMed Central Full Text
• [18]Lavis JN, Lomas J, Hamid M, Sewenkambo NK. Assessing country-level efforts to link research to action. Bull World Health Organ. 2006; 84(8):620-8.
• [19]Amara N, Ouimet M, Landry R. New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Sci Commun. 2004; 26(1):75-106.
• [20]Campbell DM, Redman S, Jorm L, Cooke M, Zwi AB, Rychetnik L. Increasing the use of evidence in health policy: practice and views of policy makers and researchers. Aust New Zealand Health Policy. 2009; 6:21. BioMed Central Full Text
• [21]Dobbins M, Cockerill R, Barnsley J. Factors affecting the utilization of systematic reviews: a study of public health decision makers. Int J Technol Assess Health Care. 2001; 17:203-14.
• [22]El-Jardali F, Lavis JN, Ataya N, Jamal D. Use of health systems and policy research evidence in the health policymaking in eastern Mediterranean countries: views and practices of researchers. Implement Sci. 2012; 7:2. BioMed Central Full Text
• [23]Ritter A. How do drug policy makers access research evidence? Int J Drug Policy. 2009; 20:70-5.
• [24]Elshaug AG, Hiller JE, Tunis SR, Moss JR. Challenged in Australian policy processes for disinvestment from existing, ineffective health care practices. Aust New Zealand Health Policy. 2007; 4:23. BioMed Central Full Text
• [25]Jackson WL, Sales JF. Potentially ineffective care: time for earnest reexamination. Crit Care Res Pract. 2014; 2014:1-6.
• [26]Erixon F, van der Marel E. What is driving the rise in health care expenditures? An inquiry into the nature and causes of the cost disease. ECIPE working papers. European Centre for International Political Economy, Brussels, Belgium; 2011.
• [27]Hansen J. Health services research in Europe: evaluating and improving its contribution to health care policy. J Health Serv Res Policy. 2011; 16 Suppl 2:1.
• [28]Redman S, Turner T, Davies H, Haynes A, Williamson A, Milat A et al.. The SPIRIT Action Framework: a structured approach to selecting and testing strategies to increase the use of research in policy. Soc Sci Med. 2015; 136–137:147-55.
• [29]Brownson RC, Fielding JE, Maylan CM. Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health. 2009; 30:175-201.
• [30]Dobbins M, Ciliska D, Cockerill R, Barnsley J, DiCenso A. A framework for the dissemination and utilization of research for health-care policy and practice. Online J Knowl Synth Nurs. 2002; 9:149-60.
• [31]de Goede J, van Bon-Martens MJ, Putters K, van Oers HA. Looking for interaction: quantitative measurement of research utilization by Dutch local health officials. Health Res Policy Syst. 2012; 10:9. BioMed Central Full Text
• [32]Landry R, Amara N, Lamari M. Utilization of social science research knowledge in Canada. Res Policy. 2001; 30:333-49.
• [33]Landry R, Amara N, Lamari M. Climbing the ladder of research utilization. Sci Commun. 2001; 22(4):396-422.
• [34]Landry R, Lamari M, Amara N. The extent and determinants of the utilization of university research in government agencies. Public Adm Rev. 2003; 63(2):192-205.
• [35]Squires JE, Estabrooks CA, Newburn-Cook CV, Gierl M. Validation of the conceptual research utilization scale: an application of the standards for educational and psychological testing in healthcare. BMC Health Serv Res. 2011; 11:107. BioMed Central Full Text
• [36]Squires JE, Estabrooks CA, O’Rourke HM, Gustavsson P, Newburn-Cook CV, Wallin L. A systematic review of the psychometric properties of self-report research utilization measures used in healthcare. Implement Sci. 2011; 6:83. BioMed Central Full Text
• [37]Zardo P, Collie A. Measuring use of research evidence in public health policy: a policy content analysis. Implement Sci. 2014; 14:496.
• [38]Walker I, Hulme C. Concrete words are easier to recall than abstract words: evidence for a semantic contribution to short-term serial recall. J Exper Psychol. 1999; 25(5):1256-71.
• [39]Wattenmaker WD, Shoben EJ. Context and the recallability of concrete and abstract sentences. J Exper Psychol. 1987; 13(1):140-50.
• [40]Weiss C. Knowledge creep and decision accretion. Sci Commun. 1980; 1(3):381-404.
• [41]Beyer JM, Trice HM. The utilization process: a conceptual framework and synthesis of empirical findings. Adm Sci Q. 1982; 27(4):591-622.
• [42]Haynes A, Turner T, Redman S, Milat AJ, Moore G. Developing definitions for a knowledge exchange intervention in health policy and program agencies: reflections on process and value. Int J Soc Res Meth. 2014; 18(2):145-59.
• [43]Weiss C, Bucuvalas MJ. Social science research and decision-making. Columbia University Press, New York; 1980.
• [44]Beyer JM. Research utilisation: bridging a gap between communities. J Manage Inquiry. 1997; 20:385.
• [45]Sumner A, Crichton J, Theobald S, Zulu E, Parkhurst J. What shapes research impact on policy? Understanding research uptake in sexual and reproductive health policy processes in resource poor contexts. Health Res Policy Syst. 2011; 9 Suppl 1:53. BioMed Central Full Text
• [46]Weiss CH. The many meanings of research utilization. Public Adm Rev. 1979; 39(5):426-31.
• [47]Liverani M, Hawkins B, Parkhurst JO. Political and institutional influences on the use of evidence in public health policy. PLoS ONE. 2013; 8(10):e77404.
• [48]Makkar SR, Williamson A, Turner T, Redman S, Louviere J. Using conjoint analysis to develop a system to score research engagement actions by health decision makers. Health Res Policy Syst. 2015; 13:22. BioMed Central Full Text
• [49]Ginsburg LR, Lewis S, Zackheim L, Casebeer A. Revisiting interaction in knowledge interaction. Implement Sci. 2007; 2:34. BioMed Central Full Text
• [50]Hair JF, Black WC, Babin BJ, Anderson RE, Tatham RL. Multivariate data analysis. 6th ed. New Delhi, Pearson; 2006.
• [51]Ryan M, McIntosh E, Shackley P. Methodological issues in the application of conjoint analysis in health care. Health Econ. 1998; 7:373-8.
• [52]Ryan M. Using conjoint analysis to take account of patient preferences and go beyond health outcomes: an application to in vitro fertilisation. Soc Sci Med. 1999; 48:535-46.
• [53]Ryan M. A role for conjoint analysis in technology assessment in health care? Int J Technol Assess Health Care. 1999; 15(3):443-57.
• [54]Bridges JF, Hauber AB, Marshall D, Lloyd A, Prosser LA, Regier DA et al.. Conjoint analysis applications in health – a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value Health. 2011; 14(4):403-13.
• [55]Carson RT, Louviere J. A common nomenclature for stated preference elicitation approaches. Environ Resource Econ. 2011; 49:539-59.
• [56]Ryan M, Farrar S. Using conjoint analysis to elicit preferences for health care. Brit Med J. 2000; 320:1530-3.
• [57]Ryan M, Hughes J. Using conjoint analysis to assess preferences for miscarriage management. Health Econ. 1997; 6:216-73.
• [58]San Miguel F, Ryan M, McIntosh E. Applying conjoint analysis in economic evaluations: an application to menorrhagia. Appl Econ. 2010; 32:823-33.
• [59]Farrar S, Ryan M. Response-ordering effects: a methodological issue of conjoint analysis. Health Econ. 1999; 8:75-9.
• [60]Lancsar E, Louviere J. Conducting discrete choice experiments to inform healthcare decision making: a user’s guide. Pharmacoeconomics. 2008; 26(8):661-77.
• [61]Oppewal H, Louviere JJ, Timmermans HJP. Modeling hierarchical conjoint processes with integrated choice experiments. J Market Res. 1994; 31(1):91-105.
• [62]Louviere J. Hierarchical information integration: a new method for the design and analysis of complex multiattribute judgment problems. Adv Consum Res. 1984; 11:148-55.
• [63]Haynes A, Turner T, Redman S, Milat A, Moore G. Developing definitions for a knowledge exchange intervention in health policy and program agencies: reflections on process and value. Int J Soc Res Meth. 2015; 18(2):145-59.
• [64]Bak A, Bartlomowicz T. Conjoint analysis method and its implementation in conjoint R package. University of Economics, Wroclaw; 2009.
• [65]Survey Monkey Inc. http://www. surveymonkey.com webcite
• [66]Norusis MJ. SPSS 15.0 Advanced statistical procedures companion. Upper Saddle River, NJ, Prentice Hall; 2007.
• [67]Ghisletta P, Spini D. An introduction to generalized estimating equations and an application to assess selectivity effects in a longitudinal study on very old individuals. J Educ Behav Stat. 2004; 29(4):421-37.
• [68]Lipsitz SR, Fitzmaurice GM, Orav EJ, Laird NM. Performance of generalized estimating equations in practical situations. Biometrics. 1994; 50(1):270-8.
• [69]Orme D. Getting started with conjoint analysis: strategies for product design and pricing research. 2nd ed. Research Publishers LLC, Madison, WI; 2010.
• [70]Elliott H, Popay J. How are policy makers using evidence? Models of research utilisation and local NHS policy making. J Epidemiol Commun Health. 2000; 54:461-8.
• [71]Lavis JN, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005; 10 Suppl 1:35-48.
• [72]Weiss CH, Bucuvalas MJ. Truth tests and utility tests: decision-makers’ frames of reference for social science research. Am Sociol Rev. 1980; 45(2):302-13.
• [73]Anderson M, Cosby J, Swan B, Moore H, Broekhoven M. The use of research in local health service agencies. Soc Sci Med. 1999; 49(8):1007-19.
• [74]Orton L, Lloyd-Williams F, Taylor-Robinson D, O’Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PLoS ONE. 2011; 6(7):e21704.
• [75]Hennink M, Stephenson R. Using research to inform health policy: barriers and strategies in developing countries. J Health Commun. 2006; 10:163-80.
• [76]Buse K, Mays N, Walt G. Making health policy. Open University Press, Glasgow; 2005.
• [77]Ellen ME, Lavis JN, Ouimet M, Grimshaw J, Bedard PO. Determining research knowledge infrastructure for healthcare systems: a qualitative study. Implement Sci. 2011; 6:60. BioMed Central Full Text
• [78]Ettelt S, Mays N. Health services research in Europe and its use for informing policy. J Health Serv Res Policy. 2011; 16 Suppl 2:48-60.
• [79]Hyder AA, Corluka A, Winch PJ, El-Shinnawy A, Ghassany H, Malekafzali H et al.. National policymakers speak out: are researchers giving them what they need? Health Policy Plann. 2011; 26:73-82.
• [80]Kothari A, Edwards N, Hamel N, Judd M. Is research working for you? Validating a tool to examine the capacity of health organizations to use research. Implement Sci. 2009; 4:46. BioMed Central Full Text
• [81]Green J, Thorogood N. Qualitative methods for health research. SAGE, London; 2005.
• [82]Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care. 1986; 2(1):53-3.
• [83]Hsu CC, Sanford BA. The Delphi technique: making sense of consensus. Pract Assess Res Evaluation. 2007; 12:10.
• [84]Shekelle P. The appropriateness method. Med Decis Making. 2004; 24:228.
• [85]Wortman PM, Smyth JM, Langenbrunner JC, Yeaton WH. Consensus among experts and research synthesis: a comparison of methods. Int J Technol Assess Health Care. 1998; 14(1):109-22.
• [86]Investigators CIPHER. Supporting policy in health with research: an intervention trial (SPIRIT)-protocol for a stepped wedge trial. BMJ Open. 2014; 4(7):e005293.
• [87]Oliver K, Lorenc T, Innvaer S. New directions in evidence-based policy research: a critical analysis of the literature. Health Res Policy Syst. 2014; 12:34. BioMed Central Full Text
• [88]Marshall D, Bridges JF, Hauber AB, Cameron R, Donnalley L, Fyie K et al.. Conjoint analysis applications in health – how are studies being designed and reported? Patient. 2010; 3(4):249-56.