experiment priority setting

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

What attributes should be included in a discrete choice experiment related to health technologies? A systematic literature review

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Writing – original draft, Writing – review & editing

Affiliations Department of Nursing, Physiotherapy and Occupational Therapy, University of Castilla La-Mancha (UCLM), Talavera de la Reina (Toledo), Spain, Research Institute for Evaluation and Public Policies (IRAPP), Universitat Internacional de Catalunya (UIC), Barcelona, Spain

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliations Department of Nursing, Physiotherapy and Occupational Therapy, University of Castilla La-Mancha (UCLM), Talavera de la Reina (Toledo), Spain, Faculty of Health Sciences, University College Dublin, Dublin, Ireland

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Supervision, Writing – original draft, Writing – review & editing

Affiliation Department of Nursing, Physiotherapy and Occupational Therapy, University of Castilla La-Mancha (UCLM), Talavera de la Reina (Toledo), Spain

• Marta Trapero-Bertran, 
• Beatriz Rodríguez-Martín, 
• Julio López-Bastida

• Published: July 18, 2019
• https://doi.org/10.1371/journal.pone.0219905
• Reader Comments

Discrete choice experiments (DCEs) are a way to assess priority-setting in health care provision. This approach allows for the evaluation of individuals’ preferences as a means of adding criteria to traditional quality-adjusted life year analysis. The aim of this systematic literature review was to identify attributes for designing a DCE in order to then develop and validate a framework that supports decision-making on health technologies. Our systematic literature review replicated the methods and search terms used by de Bekker-Grob et al. 2012 and Clark et al. 2014. The Medline database was searched for articles dated between 2008 and 2015. The search was limited to studies in English that reflected general preferences and were choice-based, published as full-text articles and related to health technologies. This study included 72 papers, 52% of which focused on DCEs on drug treatments. The average number of attributes used in all included DCE studies was 5.74 (SD 1.98). The most frequently used attributes in these DCEs were improvements in health (78%), side effects (57%) and cost of treatment (53%). Other, less frequently used attributes included waiting time for treatment or duration of treatment (25%), severity of disease (7%) and value for money (4%). The attributes identified might inform future DCE surveys designed to study societal preferences regarding health technologies in order to better inform decisions in health technology assessment.

Citation: Trapero-Bertran M, Rodríguez-Martín B, López-Bastida J (2019) What attributes should be included in a discrete choice experiment related to health technologies? A systematic literature review. PLoS ONE 14(7): e0219905. https://doi.org/10.1371/journal.pone.0219905

Editor: Gianni Virgili, Universita degli Studi di Firenze, ITALY

Received: October 12, 2018; Accepted: July 4, 2019; Published: July 18, 2019

Copyright: © 2019 Trapero-Bertran et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting Information files.

Funding: This research is funded by the European Commission’s FP7 Framework Programme and is undertaken under the auspices of Advance-HTA (Grant number 305983). The results presented here reflect the author’s views and not the views of the European Commission. The European Comission is not liable for any use of the information communicated. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

In health care systems around the world, decision-makers are faced with competing demands and insufficient resources, even in the wealthiest countries. In these circumstances, it is not possible to provide all available and potentially beneficial health care to those who could benefit from it, and priority-setting is therefore needed. Policy-makers should take into account the views of the general population in setting health priorities, as is done in the United Kingdom [ 1 ]. Public involvement in health care decision-making should be a policy objective, although there is an absence of empirical evidence on how society might value different health interventions [ 2 ].

There is substantial literature on the different methods available to engage the public in health care decision-making [ 3 – 5 ]. As noted by Whitty et al. [ 6 ], Ryan and colleagues provide a comprehensive systematic review and comparative assessment of the methods that can be used to elicit public preferences for health care [ 7 ], concluding that “there is no single, best method to gain public opinion”. Nevertheless, they do make recommendations regarding the appropriateness of selected qualitative and quantitative techniques. Two of their preferred methods, the citizens’ jury and discrete choice experiments (DCEs), have been gaining prominence in the health literature in recent years [ 6 , 8 – 12 ]. Each is associated with a number of features that make them particularly attractive for public engagement [ 6 ], rendering them worthy of further consideration. The DCE approach was chosen for this systematic literature review because it was aimed at informing a pilot study using a discrete choice experiment to quantify individual preferences regarding the use of public funding for orphan drugs.

The DCE has become a useful instrument for quantifying preferences related to health care priority-setting [ 6 , 8 , 13 – 16 ]. It has been used to (1) measure preferences regarding a health care service, (2) measure preferences regarding the distribution of health care within a population and (3) to assess preferences for the funding of health care [ 17 – 19 ]. Although use of the method by policy-makers has not yet become widespread, it has been applied to elicit social preferences for health care funded with public money [ 20 ].

A DCE survey can be administered relatively easily to a large, randomly selected representative sample of the population [ 7 ]. It is arguably a less resource-intensive method of community engagement than many other approaches, although resources and costs would likely be high for large sample sizes. A DCE measures not only the direction of preferences around a topic (e.g., Should health gains attributed to young children be weighted more heavily than those attributed to older people?), but also the relative strength of preferences for one policy choice alternative compared with another (e.g., How much extra weight should be attributed to young children?), as well as the trade-offs that respondents would be willing to make between different characteristics of that choice. The usefulness of most preference-based approaches (including DCEs) may be limited when the respondents represent what might be called a naïve sample of the general public–that is, one comprising individuals who lack personal knowledge or experience on the issue, and thus little weight can be given to the results [ 7 , 21 ].

A DCE provides a different way–compared with other approaches, such as small‐scale discussions or focus groups–to assess priority-setting based on the valuation of some attributes. DCEs are based on the assumption that health care interventions, services or policies can be described by their attributes and that an individual’s valuation depends upon the levels of these attributes. In a DCE, respondents are asked to choose between two or more alternatives. The resulting choices reveal an underlying utility function. For example, the DCE approach allows for the evaluation of individuals’ preferences for adding criteria to traditional quality-adjusted life year (QALY) analysis. The DCE approach also facilitates greater knowledge of the relative importance of the various attributes and the trade-offs that individuals are willing to make between these attributes.

This type of research and its applications are crucial for identifying the current impact of new health technologies on health and economics and, therefore, for assessing their effectiveness. DCEs also afford an opportunity to assess societal preferences. This type of research could serve as the basis for an integrated and harmonized approach to assessing public policies on new health technologies in the European Union.

De Bekker Grob et al. [ 8 ] published a recent review on preferences of consumers, patients and health professionals for all types of health care resources. They focused on the experimental design of DCEs, estimation procedures, the validity of responses and the definition of the attributes and their respective levels that should be used for DCEs on health technologies. The attributes found in their review were monetary measure, time, risk, health status domain, health care, and other. No further description of the attributes was given, so there were no well-defined inputs to be used to design a DCE for a particular context. Clark et al. [ 22 ] published a more recent DCE review. This paper updated the paper by de Bekker Grob et al. [ 8 ] and explored trends in DCEs used in health economics. It concluded that the use of DCEs in health care continues to grow dramatically across a broad range of countries. Thus, DCE results may be influencing decisions in a wider range of geographical settings. Little description and detail regarding attributes and their respective levels were provided for inclusion in future DCE exercises. There have been several literature reviews of DCEs in health care in general (such as de Bekker Grob et al. and Clark et al.) [ 8 , 22 ], but not of DCEs in health technology assessment (HTA). Decisions regarding HTA also involve public resources, however, and it is therefore important to establish approaches for prioritizing health technology resources. Accordingly, determining the attributes that should be considered in DCEs to inform HTA decisions should be a current research concern.

Materials and methods

This systematic literature review was conducted using the search terms and methods used in two recent published systematic reviews on DCEs covering the periods 2001–2008 [ 8 ] and 2009–2015 [ 22 ]. These methods involved the use of the Medline Ovid database to identify DCE studies on health care or health economics. These studies used the same search terms used by Ryan and Gerard [ 23 ], reflecting the different terms applied to refer to DCEs. The search terms included were “discrete choice experiment(s)”, “discrete choice model(l)ing”, “stated preference”, “part-worth utilities”, “functional measurement”, “paired comparisons”, “pairwise choices”, “conjoint analysis”, “conjoint measurement”, “conjoint studies” and “conjoint choice experiment(s)”.

In this study, the same database used in de Bekker Grob et al. [ 8 ] and Clark et al. [ 22 ] was used to search for articles published from January 2008 to December 2015. The same key words were also used. Papers in English and Spanish were retrieved, although the search terms used were in English only. Any paper explaining a DCE on health technologies was included. Review papers were excluded from the analysis but kept for the discussion section of this paper. De Bekker-Grob et al. [ 8 ] and Clark et al. [ 22 ] included studies that were choice-based and published as full-text articles and that applied to health care or health economics in general. Our review focused on health technologies and thus had a more limited scope. The search was extensive with respect to health care and health economics in general, but only papers related to health technologies were included in our systematic review. The objective was to evaluate DCEs on health technologies that reflected the preferences of patients, policy-makers, providers and the general public. Papers were excluded if they had the following characteristics: (a) they were duplicates; (b) they were not choice- or preference-based or they merely provided measurements but no descriptions of attributes; (c) neither the full text nor an abstract was found; (d) they did not apply to health technologies or to rural areas of developing countries; and (e) they did not involve human respondents. Grey literature was also searched using Google Scholar, although unfortunately no results were found. Each abstract and paper selected was carefully peer-reviewed, and data extraction was systematically and independently performed by two researchers. Whenever there was a discrepancy, papers were reviewed a second time to reach a consensus. Excel was used to summarize the results of this systematic literature review. A data extraction form included questions on the following: background (e.g., quartile of impact factor); sampling and sample characteristics (e.g., illness of respondents); general design of the DCE (e.g., number of attributes and description of attributes and levels covered); experimental design (e.g., method for creating choice sets); design validity (e.g., estimation procedure, model); and qualitative methods for enhancing the DCE process and results (e.g., pretesting of the DCE questionnaire). However, only the general design and experimental design features were presented in the results and discussion sections. Considerations relating to design validity and qualitative methods used to enhance the DCE process were beyond the scope of this paper.

The specific details of the template were dynamically adjusted during the piloting process, which included the revision of a few papers. Data were extracted in free-text form with no limitations on the number of free-text fields and as little categorization of data as possible to avoid the loss of detailed information. Descriptive analysis was undertaken to describe the most common attributes used and their corresponding levels. The attributes and levels for the DCE questionnaire on HTA are presented in a summary table. The table shows the attributes found in this literature review and the attributes identified in the previously published systematic literature reviews [ 8 , 22 ].

To assess the methodological quality of the systematic literature review, the PRISMA checklist was used [ 24 ]. In addition, a DCE quality assessment tool [ 25 ] was used to assess the validity of the studies and their attributes and levels.

This approach was devised following the guidance of Mandeville et al. [ 25 ], who covered all four key stages of a DCE (choice task design; experimental design; conduct; and analysis) using a list of 13 criteria drawn from an earlier study [ 26 ]. The authors assessed whether each criterion for each study was met. If the criterion was met, it was indicated with a green colour. If there was insufficient information to judge whether a criterion was met, then an orange colour was used. A red colour indicated that the criterion was not met. This type of qualitative analysis is important for validating the results from this systematic literature review.

Fig 1 shows the flowchart for the identification of studies, with reasons for exclusion.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0219905.g001

Overall, the search strategy identified 384 titles (after duplicates were excluded) from a pool of studies with the potential for inclusion in this review. Based on the abstracts, 160 papers were ordered and manually reviewed. Of these 160 articles, 72 were included in this study [ 14 , 27 – 96 ]. S1 Table , included in the supplemental material, provides details of the PRISMA checklist used to assess the methodological quality of the systematic literature reviews.

The number of studies published on DCEs on general preferences regarding health technologies over time is as follows: 18 articles (25%) were published before 2010, 30 articles (41.6%) appeared between 2010 and 2011, and the remaining 21 articles (29%) were published between 2012 and 2013. The years with the greatest research output were 2011 and 2012 (16 articles and 15 articles published, respectively). The average sample used across the 72 studies included 299 individuals with a mean age of 59.6 years; an average of 44.88% of the respondents were female. In seven studies, only the age interval was reported; in those cases, the average of the age interval was taken. The largest number of the DCEs identified were conducted in the Netherlands, although significant numbers were also conducted in the United Kingdom, Canada, Germany and the United States. The 72 papers covered 30 different diseases, such as chronic obstructive pulmonary disease (COPD), depression and hepatitis B. The most studied preferences related to cancer (26%), followed by attention deficit disorder (4%) and osteoporosis (4%). Only one paper [ 61 ] was found that examined preferences relating to orphan drugs for rare diseases. S2 Table , included in the supplemental material, provides more detailed information about the health technologies, the attributes and the levels for each paper.

Fig 2 presents the validity assessment for all included studies. Overall, while the choice task design and the experimental design of the studies were more robust than expected, there were significant weaknesses with regard to conduct and analysis of the studies. In terms of choice task, attributes and levels were identified through qualitative work with the target population in 24.8% of the studies. In 16% of the studies, there was no opt-out or status quo option, nor any justification of a forced choice for the attributes selected. In terms of the experimental design, 27% of papers did not have a design that was optimal or statistically efficient. However, most of the relevant problems with the validity of the papers pertained to the pilot testing conducted among the target population and the lack of a pooled analysis from different subgroups: in 42% of the papers the authors did not conduct a pilot test to inform the design of the questionnaire, and 63% did not include any pooled analysis from different population subgroups. In an assessment of the validity of experimental design and analytic approach, it is necessary to examine current practices in DCEs.

https://doi.org/10.1371/journal.pone.0219905.g002

Most of the studies (53%) assessed societal preferences regarding pharmacological treatments; the rest focused on medical devices. Of the latter, 17% related to diagnostic technologies and 26% to therapeutic technologies (specifically, 13% were on assistive technology devices used directly by patients, 5% were on medical devices to assist medical professionals, and 4% were on artificial body parts implanted through medical procedures). The average number of attributes was 5.74 (SD 1.98), with a minimum value of 2 and a maximum value of 12. Each attribute had an average of 3.26 levels (SD 1.11), with a minimum value of 1 and a maximum value of 18. The six most common attributes used in the DCEs (n = 72) were (a) improvement in health (78%), (b) side effects (57%), (c) costs (53%), (d) waiting time for treatment or duration of treatment (25%), (e) severity of disease (7%), and (f) value for money (4%). When the focus was only on papers that assessed preferences in relation to drugs (n = 36), the relative importance of the attributes remained the same: improvement in health (55.56%), costs (50%), adverse events (41.67%), and mode of administration (22.22%), followed by discomfort and pain (16.67%), treatment duration (19.44%) and waiting time (2.78%). These attributes reflected the general preferences of several groups, including patients, the general public patients and other stakeholders. The majority of the papers (n = 60) referred to patients’ preferences. In terms of attributes revealing preferences related to efficiency, availability of other treatments and value for money were considered relevant attributes [ 14 ]. Availability of other treatments refers to the existence of alternative treatments for the same disease. Value for money refers to how efficiently resources are used (e.g., doctor time, hospital beds, drugs) in the national health system and is based on the relationship between the cost of treatment and the health benefits it provides. Hence, although these two attributes were not the most commonly used, they were also considered for inclusion in the DCE survey.

In terms of levels, the papers reviewed most commonly referred to the following levels of administration: oral, subcutaneous, intravenous or injection. These papers also referred to the following levels of pain or discomfort: none, mild, moderate or severe. Therefore, the terms mild, moderate, and severe were adopted to describe the levels for as many attributes as possible. See Table 1 for details on the most used attributes and their respective levels. This table also includes the attributes and levels described in the two previously published systematic reviews [ 8 , 22 ]. Both studies highlight the monetary measure and the time- and health care-related attributes as the most frequent ones to be considered in a DCE.

https://doi.org/10.1371/journal.pone.0219905.t001

Only 10 papers (13.8%) offered partial access to the questionnaire used to carry out the DCE and the specific questions formulated for respondents. The rest of the papers did not offer access to the survey. The most common means of administering the survey was a self-report questionnaire (35.29%); questionnaires were administered through an interviewer in 7.35% of cases or a computer in 8.82%. The rest of the papers did not report the mode of survey administration.

Regarding experimental design, 43% (n = 31) of the studies had a fractional factorial design, whereas 13% (n = 10) had a full factorial design. The remaining 27 studies did not report the type of DCE design. Thirty-one studies measured main effects only, whereas seven studies reported main effects with 2-way interactions. In 32% of the DCEs reviewed, the effects evaluated were not reported. More than half of the papers (55%) used orthogonal arrays to create choice sets, while 14% used D-efficiency methods. Five papers combined two methods: either orthogonal arrays with D-efficiency or other methods. Three papers used other methods to create choice sets, and 18 did not explain the method used. In terms of the estimation procedure, the multinomial logit was the most common model used to analyse DCE preferences (22%), followed by random effects logit (13%) and logit (12%).

This study fills a gap in systematic reviews of the literature aimed at identifying the most relevant attributes and levels for measuring public preferences regarding health technologies by means of a DCE. Six attributes were identified from de Bekker Grob et al. (2012) [ 8 ] and Clark et al. (2014) [ 22 ]: health status domain, monetary measure, time, risk, health care and other, some of which were too subjective to build questions for a DCE survey. No levels were defined or detailed in these papers and no additional information was given concerning the definitions of these attributes. In contrast, our literature review found the following attributes: improvement in health, side effects, cost (price) of treatment, waiting time for treatment or treatment duration, severity of disease and value for money. The attributes included in the previous systematic reviews are too wide and general to understand. No definitions were provided by the authors, so it was difficult to evaluate the complementarity between the results of the two systematic literature reviews, even though they used the same literature review methods. It will be important for future research to describe these attributes and their respective levels in as detailed a manner as possible, so that they can be applied with no uncertainties regarding what is encompassed in their definition. In addition, a complete description will be helpful in providing information for the design of future DCEs on HTA. Because public preferences might change greatly over time, depending on current situations worldwide, it was decided to incorporate papers published between 2008 and 2015 –i.e., a period of 7 years. Regarding the optimum number of attributes to include in a DCE, Marshal et al. [ 97 ] identified and described recent applications of conjoint analysis to determine what combination of a limited number of attributes was most influential on respondent choice or decision-making. In their review, they found that most surveys included 6 attributes, with the number ranging from 3 to 16. Therefore, it seems that a larger number of attributes should be used to better capture the criteria on which people base their preferences related to health care. However, many attributes make the decision task more difficult and hence render the outcomes less reliable. The number of attributes found in this systematic review–six–seems a sensible and adequate number to be potentially included in a DCE.

Orphan drugs are unlikely to be efficient (provide value for money) due to the high price paid for often modest effectiveness. It is important to identify all appropriate criteria that will help in the “correct” evaluation of the potential impact and benefit generated in society. Unfortunately, only one paper [ 61 ] was found that studied preferences relating to orphan drugs for rare diseases. The authors investigated public preferences regarding public funding for orphan drugs used to treat both rare and common diseases, using a convenience sample of university students. They found that when all other variables were held constant, the respondents did not prefer to have the government spend more for orphan drugs used to treat rare diseases and that they weighted the relevant attributes of coverage decisions similarly for both rare and common diseases. More DCEs on orphan drugs should be conducted to generate more evidence on the particular attributes and levels for this kind of drug.

The inclusion of either cost or improvement in health and value for money as attributes helps to capture the preferences of respondents, although it could lead to double-counting. None of the papers found in this systematic review included either combination of those attributes; however, it is important to be aware of the potential for double-counting that can occur as a result of the inclusion of such similar attributes.

Although there were significant weaknesses in terms of the validity assessment of the included studies, important and essential issues–such as no overlap between the attributes, the use of unidimensional attributes in the questionnaires, the use of the correct target population and the appropriate use of an econometric model for the choice task design–were common characteristics for most of the studies. Hence, despite some weaknesses regarding validity, the most important criteria for these types of studies were included overall.

DCEs have been previously used in other published studies to gain insight into the criteria that were important for decision-makers in health care priority-setting [ 14 , 40 , 52 , 64 , 65 ]. These five papers were included in this literature review. The rest of the papers (n = 67) focused on patient preferences. The type of attributes used in the papers that focused on policy-makers’ opinions were quite different from those that sought to identify patients’ preferences. For instance, six intervention-related attributes were included in a paper [ 73 ] that measured the preferences of policy-makers and other health professionals, including disease severity, budget impact, cost-effectiveness (incremental cost per QALY and number of QALYs gained per patient), uncertainty regarding the probability of doubling costs per QALY, national savings in costs related to absence from work per year and the composition of the health gain. Disease severity and cost of treatment are included in both literature reviews, as is health improvement; however, questions related to cost-effectiveness or national savings when the target audience includes patients are more difficult to ask. For that reason, availability of other treatments and value for money were included as relevant attributes. Attributes might differ, depending on the survey target audience. In this case, all audiences have been included.

Subsequent research is needed to further develop DCE attributes and levels for various specific technologies and diseases. One possible approach might be to investigate in a more in-depth manner the methods that led to the selection and identification of attributes in DCE studies (e.g., focus groups, interviews, literature, expert opinion), which could then be informative for future DCEs. Another approach might be to conduct DCEs for different diseases among different types of audiences to assess and validate attributes and thus help to inform future priority-setting decisions.

Conclusions

This systematic literature review was performed to identify the attributes that may better help decision-makers and patients to identify the criteria leading to decisions about health technologies. This study revealed that attributes such as improvements in health, treatment side effects, treatment cost (price), waiting time for treatment or treatment duration, severity of disease and value for money can be considered to better capture and describe societal preferences in relation to HTA. This topic is of interest for preference practitioners, as it can help them, first, to build the best survey on health technology and, then, to aid public decision-makers in identifying the treatments that should be implemented or funded, in accordance with the population’s preferences.

Supporting information

S1 table. prisma 2009 checklist..

https://doi.org/10.1371/journal.pone.0219905.s001

S2 Table. Description of attributes and levels by device or HTA.

https://doi.org/10.1371/journal.pone.0219905.s002

• 1. Department of Health. Involving patients and the public in healthcare: a discussion document. London: Practitioners RCOG; 2001.
• View Article
• Google Scholar
• PubMed/NCBI
• 11. Coote A, Lenaghan J. Citizens’ Juries: Theory into Practice. London: Institute for Public Policy Research; 1997.
• 21. Ryan M, Gerard K, Amaya-Amaya M. Using Discrete Choice Experiments to Value Health and Health Care. Dordrecht, Netherlands: Springer; 2008.

Cochrane Methods Priority Setting

Guidance Documents

• An introductory blog post on priority setting by Sumant Kumbargere
• What is research priority setting and what you need to do before starting?   Guidance Sheet 1.
• Top Tips for review groups
• Quick guidance on priority setting (draft)
• Checklist for health research priority setting
• Equity lens for setting priorities for research
• Specific guidance for Cochrane review groups
• Chapter - 2.7 Prioritization of research by Mona Nasser, Roderik Floris Viergever and John Martin.   The chapter is part of the WHO Guidance on Research Methods for Health Emergency and Disaster Risk Management 
• An updated framework to set priorities for population-level priority setting exercises  
• Engaging with multiple stakeholders in the priority setting process (a presentation of initial results – the final results will be available soon link 
• An adapted version of AMSTAR For systematic reviews of research priority setting exercises ( Appendix B: Adapted AMSTAR-2 for use with reviews of priority setting ). There is also  blog post on the topic.
• REPRISE reporting guidelines for research priority setting

Examples of Cochrane groups that conducted priority setting exercises:  link

Examples of systematic reviews on research priorities

• Systematic Review of priorities for tuberculosis research
• Systematic review of priorities in kidney disease
• Systematic review of priorities in Organ transplantation
• Systematic review of priorities in stroke research
• Two overviews of systematic reviews of priority setting - ( Tan 2022 ) ( Lund 2022 )

Examples of research priority setting exercises in the literature (under development)

Overview of methods used in priority setting

The Ludwig Boltman Gesellschaft – Open innovation in science centre is hosting a database of priority setting exercises

Montorzi, G., S. de Haan, and C. IJsselmuiden, Priority Setting for Research for Health: a management process for countries. 

Grill, C. Involving stakeholders in research priority setting: a scoping review.  Res Involv Engagem   7 , 75 (2021). https://doi.org/10.1186/s40900-021-00318-6

Using Guidelines to inform priority setting of Systematic Reviews

Yu T, Li T, Friedman DS, Dickersin K, Puhan M. Asia-Pacific clinicians prioritize comparative effectiveness research for the management of primary angle closure. Journal of Glaucoma (in press).

Li T, Vedula SS, Scherer R, Dickersin K. What comparative effectiveness research is needed? A framework for using guidelines and systematic reviews to identify evidence gaps and research priorities . Ann Intern Med. 2012;156(5):367-77.

Li T, Ervin A, Scherer R, Jampel H, Dickersin K. Setting priorities for comparative effectiveness research: A case study using primary open-angle glaucom a. Ophthalmology. 2010;117(10):1937-45.

Using GRADE to inform priority setting of Systematic Reviews

 Siegfried N, Narasimhan M, Kennedy CE, Welbourn A, Yuvraj A. Using GRADE as a framework to guide research on the sexual and reproductive health and rights (SRHR) of women living with HIV – methodological opportunities and challenges .AIDS Care. 2017 Apr 27:1-6. doi: 10.1080/09540121.2017.1317711.  [Epub ahead of print]

Equity in Priority setting Process

Nasser M, Ueffing E, Welch V, Tugwell P. An equity lens can ensure an equity-oriented approach to agenda setting and priority setting of Cochrane Reviews . J Clin Epidemiol. 2013 May;66(5):511-21. doi: 10.1016/j.jclinepi.2012.11.013.

Bhaumik S, Rana S, Karimkhani C, Welch V, Armstrong R, Pottie K, Dellavalle R, Dhakal P, Oliver S, Francis DK, Nasser M, Crowe S, Aksut B, Amico RD. Ethics and equity in research priority-setting: stakeholder engagement and the needs of disadvantaged groups. Indian J Med Ethics. 2015 Apr-Jun;12(2):110-3.

Jaramillo A, Welch VA, Ueffing E, Gruen RL, Bragge P, Lyddiatt A, Tugwell P. Prevention and self-management interventions are top priorities for osteoarthritis systematic reviews . J Clin Epidemiol. 2013 May;66(5):503-510.e4. doi: 10.1016/j.jclinepi.2012.06.017

Evidence mapping

Clavisi O, Bragge P, Tavender E, Turner T, Gruen RL. Effective stakeholder participation in setting research priorities using a Global Evidence Mapping approach . J Clin Epidemiol. 2013 May;66(5):496-502.e2.

Bragge P, Clavisi O, Turner T, Tavender E, Collie A, Gruen RL. The Global Evidence Mapping Initiative: scoping research in broad topic areas . BMC Med Res Methodol. 2011 Jun 17;11:92.

Miake-Lye IM, Hempel S, Shanman R, Shekelle PG. What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products . Syst Rev. 2016 Feb 10;5:28. doi: 10.1186/s13643-016-0204-x.

Stakeholder engagement in priority setting

Madden M, Morley R. Exploring the challenge of health research priority setting in partnership: reflections on the methodology used by the James Lind Alliance Pressure Ulcer Priority Setting Partnership. Research Involvement and Engagement2016 2 :12 DOI: 10.1186/s40900-016-0026-y

Pollock A, St George B, Fenton M, Crowe S, Firkins L. Development of a new model to engage patients and clinicians in setting research prioritie s. J Health Serv Res Policy. 2014 Jan;19(1):12-8. doi: 10.1177/1355819613500665.

Elwyn G, Crowe S, Fenton M, Firkins L, Versnel J, Walker S, Cook I, Holgate S, Higgins B, Gelder C. Identifying and prioritizing uncertainties: patient and clinician engagement in the identification of research question s. J Eval Clin Pract. 2010 Jun;16(3):627-31.

Abma TA, Pittens CA, Visse M, Elberse JE, Broerse JE. Patient involvement in research programming and implementation: A responsive evaluation of the Dialogue Model for research agenda settin g. Health Expect. 2015 Dec;18(6):2449-64. doi: 10.1111/hex.12213. Epub 2014 May 30. PubMed PMID: 24889933.

Elberse JE, Caron-Flinterman JF, Broerse JE. Patient-expert partnerships in research: how to stimulate inclusion of patient perspectives . Health Expect. 2011 Sep;14(3):225-39. doi: 10.1111/j.1369-7625.2010.00647.x. Epub 2010 Dec 22.

Abma TA, Broerse JE. Patient participation as dialogue: setting research agendas . Health Expect. 2010 Jun;13(2):160-73. doi: 10.1111/j.1369-7625.2009.00549.x.

Caron-Flinterman JF, Broerse JE, Teerling J, Bunders JF. Patients' priorities concerning health research: the case of asthma and COPD research in the Netherlands . Health Expect. 2005 Sep;8(3):253-63.

Broerse JE, Zweekhorst MB, van Rensen AJ, de Haan MJ. Involving burn survivors in agenda setting on burn research: an added value? Burns. 2010 Mar;36(2):217-31. doi: 10.1016/j.burns.2009.04.004. Epub 2009 Jul 4.

Chalmers I, Atkinson P, Fenton M, Firkins L, Crowe S, Cowan K. Tackling treatment uncertainties together: the evolution of the James Lind Initiative, 2003-2013 . J R Soc Med. 2013 Dec;106(12):482-91. doi: 10.1177/0141076813493063. Epub 2013 Jul 3.

Elberse JE, Pittens CA, de Cock Buning T, Broerse JE. Patient involvement in a scientific advisory process: setting the research agenda for medical products. Health Policy. 2012 Oct;107(2-3):231-42. doi: 10.1016/j.healthpol.2012.05.014. Epub 2012 Jun 25.

Cromwell I, Peacock SJ, Mitton C. 'Real-world' health care priority setting using explicit decision criteria: a systematic review of the literature. BMC Health Serv Res. 2015 Apr 17;15:164.

Economic Approaches to Priority Setting: 

The following are a list of references to papers describing quantitative approaches to research priority setting:

Programme Budgeting and Marginal Analysis:

• Brambleby P. A survivor's guide to programme budgeting . Health Policy1995;33(2):127-45.
• Cohen DR. Messages from Mid Glamorgan: a multi-programme experiment with marginal analysis. Health Policy1995;33(2):147-55.
• Craig N, Parkin D, Gerard K. Clearing the fog on the Tyne: programme budgeting in Newcastle and North Tyneside Health Authority .Health Policy1995;33(2):107-25.
• Holmes RD, Bate A, Steele JG, Donaldson C. Commissioning NHS dentistry in England: issues for decision-makers managing the new contract with finite resources.Health Policy2009;91(1):79-88.
• Holmes RD, Steele J, Exley CE, Donaldson C. Managing resources in NHS dentistry: using health economics to inform commissioning decisions.BMC Health Serv Res2011;11:138.
• Madden L, Hussey R, Mooney G, Church E. Public health and economics in tandem: programme budgeting, marginal analysis and priority setting in practice.Health Policy1995;33(2):161-8.
• Mitton C, Donaldson C. Twenty-five years of programme budgeting and marginal analysis in the health sector, 1974-1999.J Health Serv Res Policy2001;6(4):239-48.
• Peacock SJ, Richardson JR, Carter R, Edwards D. Priority setting in health care using multi-attribute utility theory and programme budgeting and marginal analysis (PBMA).Soc Sci Med2007;64(4):897-910.
• Posnett J, Street A. Programme budgeting and marginal analysis: an approach to priority setting in need of refinement.J Health Serv Res Policy1996;1(3):147-53.
• Twaddle S, Walker A. Programme budgeting and marginal analysis: application within programmes to assist purchasing in Greater Glasgow Health Board.Health Policy1995;33(2):91-105. 
• Wilson E, Sussex J, Macleod C, Fordham R. Prioritizing health technologies in a Primary Care Trust.J Health Serv Res Policy2007;12(2):80-5.

Value of Information Analysis:

• Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies.J Health Econ1999;18(3):341-64.
• Claxton K, Cohen JT, Neumann PJ. When is evidence sufficient?Health Aff (Millwood)2005;24(1):93-101.
• Willan AR, Pinto EM. The value of information and optimal clinical trial design.Stat Med2005;24(12):1791-806.
• Wilson E, Abrams K. From Evidence Based Economics to Economics Based Evidence: Using Systematic Review to inform the design of future research. In: Shemilt I, Mugford M, Vale L, Marsh K, Donaldson C, editors.Evidence Based Economics. London: Blackwell Publishing, 2010.

Measuring Impact - this websit e provides a great database of impact studies to help you plan for future research priority setting studies 

• Research article
• Open access
• Published: 05 March 2009

Priority setting: what constitutes success? A conceptual framework for successful priority setting

• Shannon L Sibbald 1 , 2 ,
• Peter A Singer 3 ,
• Ross Upshur 2 , 4 &
• Douglas K Martin 1 , 2  

BMC Health Services Research volume  9 , Article number:  43 ( 2009 ) Cite this article

36k Accesses

170 Citations

12 Altmetric

Metrics details

The sustainability of healthcare systems worldwide is threatened by a growing demand for services and expensive innovative technologies. Decision makers struggle in this environment to set priorities appropriately, particularly because they lack consensus about which values should guide their decisions. One way to approach this problem is to determine what all relevant stakeholders understand successful priority setting to mean. The goal of this research was to develop a conceptual framework for successful priority setting.

Three separate empirical studies were completed using qualitative data collection methods (one-on-one interviews with healthcare decision makers from across Canada; focus groups with representation of patients, caregivers and policy makers; and Delphi study including scholars and decision makers from five countries).

This paper synthesizes the findings from three studies into a framework of ten separate but interconnected elements germane to successful priority setting: stakeholder understanding, shifted priorities/reallocation of resources, decision making quality, stakeholder acceptance and satisfaction, positive externalities, stakeholder engagement, use of explicit process, information management, consideration of values and context, and revision or appeals mechanism.

The ten elements specify both quantitative and qualitative dimensions of priority setting and relate to both process and outcome components. To our knowledge, this is the first framework that describes successful priority setting. The ten elements identified in this research provide guidance for decision makers and a common language to discuss priority setting success and work toward improving priority setting efforts.

Peer Review reports

Priority setting, also known as rationing or resource allocation, is a complex and difficult problem faced by all decision makers at all levels of all health systems, including macro (e.g. governments), meso (e.g. regional health authorities (RHAs), hospitals), and micro (e.g. clinical programs) levels. There is relatively little interaction between decision makers at the three levels, or among institutions, regarding the setting of priorities. Consequently, priority setting has been described as a series of unconnected experiments with no systematic mechanism for capturing the lessons, or evaluating the strengths and weaknesses, of each experiment [ 1 ]. Hospital administrators, constrained by budget restrictions and confronted by increasing demand, find it a particularly difficult challenge to maintain services and quality, while controlling costs; decision makers (or leaders) lack guidance and information for priority setting and are unaware of priority setting tools available to them [ 2 – 4 ]. Mitton and Donaldson found decision makers were "frustrated with the lack of an explicit priority setting framework" and questioned "the credibility of resource allocation decision-making" ([ 4 ]p. 1660). Several studies have reported that leaders desire an explicit framework to guide priority setting [ 4 – 6 ] and acknowledged leadership as a key area where improvement can make the most difference [ 7 ].

The sustainability of healthcare systems worldwide is threatened by a growing demand for services and expensive innovative technologies. Decision makers struggle in this environment to set priorities appropriately, particularly because they lack consensus about which values should guide their decisions [ 8 ]. One way to approach this problem is to determine what relevant stakeholders understand successful priority setting to mean. Greater insight into stakeholders' attitudes and perceptions of achieving successful priority setting could improve the way in which institutions and healthcare organizations set priorities.

Successful priority setting is a desirable goal for decision makers; however there is no agreed upon definition for successful priority setting, so there is no way of knowing if an organization achieves it. Priority setting is extremely complex – choosing between competing values makes priority setting fundamentally an ethical issue [ 9 ]. Different disciplines offer their own perspective on how priority setting 'ought' to be done, defining 'good' (or successful) priority setting through values such as efficiency, equity, or justice. Discipline specific approaches and priority setting frameworks can help decision makers with priority setting: health economics encourages a focus on efficiency, policy approaches focus on legitimacy, evidence-based medicine looks to effectiveness. Daniels and Sabin created 'accountability for reasonableness' (A4R) with legitimacy and fairness as two key goals of priority setting [ 10 ]. Interdisciplinary approaches are also available such as program-budgeting and marginal analysis (PBMA)[ 11 ], health technology assessment (HTA)[ 12 ]. Menon et al. described priority setting in four steps: (1) identification of health care needs, (2) allocation of resources, (3) communication of decisions to stakeholders, and (4) management of feedback from them [ 13 ]. Still, there is no consensus that any one framework provides the 'correct' or 'best' comprehensive definition of successful priority setting.

These normative approaches are necessary because they help identify important values and underlying assumptions in priority setting, however alone they are insufficient and provide only a piece of a definition of successful priority setting. The problem that remains is: priority setting involves the adjudication between many relevant values and that people (and disciplines) will disagree about which values should dominate in any specific priority setting context and there is no agreed upon normative approach for resolving the disagreement. When relevant values conflict, decision makers must rely on developing context-specific agreement in order to achieve priority setting success.

Empirical studies are also important for understanding current decision making practices within healthcare organizations [ 14 , 15 ]; since they identify current priority setting practices, they provide insight into defining successful priority setting. In recent years, there have been empirical descriptions of priority setting in various contexts (e.g. waitlists [ 16 – 18 ], hospitals [ 19 – 21 ], and RHAs [ 13 , 22 ]). Other empirical studies have evaluated actual priority setting against an ethical framework (e.g. [ 19 , 23 ]). Studies have been done detailing factors that influence priority setting practices, including technical factors (such as clinical practice guidelines), non-technical factors (such as alignment with goals) [ 13 ] and clusters of factors [ 24 ]. Several studies have brought forth components for improving priority setting or ensuring success in priority setting, such as stakeholder engagement [ 13 ] increased dialogue [ 4 ], a culture supporting explicit priority setting [ 6 ], decision maker/group composition (size and representation) [ 25 ], clear information management and clarity of process [ 5 ], and local ownership and awareness of local politics [ 26 ].

Only a few studies have presented ideas for evaluating the success of priority setting including: economic evaluations [ 27 , 28 ], checklists looking at both pragmatic (such as establish organizational objectives and ensure implementation) and ethical considerations (such as publicity and appeals)[ 29 ], success parameters (effect on organizational priorities and budgets, effect of staff, and effect on community, efficiency of priority setting process, fairness, conformity with conditions of accountability for reasonableness)[ 2 ], a criteria-based framework (including objectives and context, methodology, process issues, and study outcomes)[ 30 ], outputs-based measures (such as usefulness re-allocation, improved patient outcomes) [ 31 ], and a model for ethical standards (including health of patients, professional (clinical) expertise, public health, unmet health needs, advocacy for social policy reform, relationships of special ethical importance (with employees), organizational solvency/survival, and benefit to community) [ 32 ]. Together, these studies contribute to our understanding of successful priority setting; but on their own, do not provide a comprehensive definition.

Evaluating success of health care (and other sectors) is possible through many of the aforementioned tools/processes, and different instruments may elicit different results [ 33 ]. The problem with these studies is their limited focus (narrow organizational study and/or small range of stakeholders). While we are more cognizant of important factors in successful priority setting, we still do not have a complete picture of it.

Normative approaches tell us what ought to be done, empirical studies tell us what is being done, and we are still left with a lack of consensus on an appropriate approach to successful priority setting. There is a need to define successful priority setting, to provide a common language, and to come to some agreement on conceptual basis for the concept.

A first step to ground such a definition is to collect and synthesize the views of stakeholders with direct priority setting knowledge and experience. Stakeholders include decision makers (particularly in publicly funded health systems, who are under growing pressure to base their decisions on available evidence and to demonstrate the effectiveness of their decision), patients (since the health system exists for them and because they fund the health system through taxes, insurance premiums or out-of-pocket payments), and priority setting scholars (who can provide different theoretical viewpoints on decision making). Creating a framework that defines success in priority setting is a necessary step toward improving priority setting practices in healthcare organizations [ 34 , 35 ].

The purpose of this paper is to present a synthesized definition of successful priority setting brought together from the findings of three empirical studies describing successful priority setting from the viewpoint of stakeholders (decision makers, patients, and priority setting scholars). The definition is presented as a conceptual framework with ten elements. The framework we describe here is a new development for evaluating priority setting; it can provide guidance to decision makers and scholars interested in successful priority setting.

The findings reported here were derived from three separate but related empirical studies that used different data collection methods, but similar data analysis techniques. The overarching goal for the three studies was to create a conceptual framework for achieving successful priority setting. Study 1 gathered international perspectives through a Delphi consensus building initiative [ 36 ]. Study 2 used qualitative interviews to capture the views of decision makers across the Canadian healthcare system. Study 3 included the perspective of Canadian public/patients and policy makers and used multiple interconnected focus groups called the "circle within a circle" approach (table 1 ) [ 37 ]. Each study used a unique set of participants; there was no overlap. By bringing these three data sets together, we tapped into a diverse and rich knowledge base and captured what we feel to be a comprehensive description of successful priority setting. What follows is a combined description of the methods for all three studies.

Participants

Study 1 (Delphi panel) consisted of 12 priority setting scholars and healthcare decision makers from five different health systems (table 2 ), chosen for their experience and interest in priority setting (i.e., published work in the field, different disciplinary approaches to priority setting and international perspectives).

Study 2 consisted of senior or executive level decision makers in healthcare organizations across Canada sampled using two methods: (1) theoretical sampling – people who were involved in a significant aspect of priority setting and (2) 'snowball' sampling – participants were asked to identify others (colleagues) who might have knowledge or insight into priority setting and who should be interviewed. Participants were sought out until conceptual saturation was reached (i.e. until no new concepts were identified in successive interviews). Participants came from 45 different organizations with representation from every province except Newfoundland and Prince Edward Island (table 3 ). Attempts were also made to ensure there was representation within provinces – interviews did not focus solely on the capital regions of each province.

Study 3 consisted of 13 patients/health system users (one from every province and territory). Patients were identified and approached through various health networks, organizations and associations. In addition, 13 health policy makers representing different levels of government and different health care contexts (at least one policy maker from each province and territory) were recruited (table 4 ).

Sample size was not formally calculated for any of the three studies since our goal was not to generate generalizable conclusions, but instead to describe characteristics of successful priority setting from the point of view of decision makers.

Data Collection

Study 1 spanned three Delphi 'rounds'. Round 1 was conducted in May/June 2003 via email; the ethical framework 'accountability for reasonableness' (A4R) acted as the starting point for discussions [ 10 ]. (A4R was chosen as a starting point for discussions because it has traction among decision makers and it is an established framework of priority setting researchers internationally; moreover, it is a useful tool and a practical guide to develop, implement, and evaluate fair priority setting processes.)

Round 2 had all participants face-to-face; the input was a list of 39 items, generated from Round 1 (see appendix). Round 3 was conducted by email four months after Round 2. Results of Round 2 (now 14 succinct and prioritized items) were circulated; panelists were asked to make final suggestions and revisions to sharpen the list. Subsequently, the list was revised down to six items (table 5 ).

Study 2 interviews were conducted in person or by telephone from July 2003 to May 2004 and used an interview guide based on previous research and relevant literature. All interviews were audio taped and transcribed – over 800 pages of transcripts were generated.

Study 3 was set around an existing event: an Alberta-based Provincial Health Ethics Network (PHEN) conference (April 2003). The study utilized this conference as a unique opportunity to bring together patients and policy makers in one location. All study participants participated in the PHEN conference. All focus groups were video taped and the discussions were transcribed. Observations were recorded by the researchers in field notes which provided context to the data analysis.

Data analysis

Data from the three studies was first analyzed separately and then synthesized and analyzed in aggregate. Analysis was done using a modified thematic analysis that proceeded in two steps: open and axial coding [ 38 ]. In open coding, the data was read and then fractured by identifying chunks of data that related to a concept or idea. In axial coding, similar ideas were organized into overarching themes by grouping similar codes. The analysis was facilitated by, and culminated in, writing, which served as an important tool in formalizing elements and making explicit assumptions that influence data interpretation [ 39 ].

The validity of the findings was addressed in three ways [ 40 ]. First, two researchers (SS and DKM) coded the raw data to ensure accuracy and that one person's biases did not unduly skew the interpretation – differences were resolved through ongoing discussion. Second, all research activities were rigorously documented by the researcher to permit a critical appraisal of the methods [ 41 ]. Third, throughout all three studies, participants verified the reasonableness of the findings in "member checks" – participants were invited to read the results from the data analysis and comment on misinterpretations or missing information. Revisions from participants were incorporated into the findings, or where disagreement occurred, were discussed by the research team to determine further action.

Research Ethics

All three studies received research ethics board approval from The Committee on the Use of Human Subjects of the University of Toronto. Where appropriate, written informed consent was obtained from each individual. All data was protected as confidential and available only to the research team. No individuals have been identified in reports without their explicit agreement.

Synthesis of 3 Studies

When analyzed independently, the three studies provided insight into key elements which could define successful priority setting; together they provided 21 elements (table 5 ). In order to make one comprehensive list of elements of successful priority setting, we re-read and re-analyzed raw data to look for similarities; similar items were merged and an amalgamated list was created (e.g., within the views of Canadian decision makers, 'context considerations' and 'consideration for values' were merged; in the focus group list, 'consideration of context' and 'consideration of values' were merged; combined they created 'consideration of context and values'). Effort was made to ensure that the merged list captured the original description and meaning.

In the end, a list of ten elements was created (table 6 ). The research team created the element labels (left column) based on the results of the three studies; where possible, we used labels that were verbatim from the original raw data (i.e. participants themselves used the words).

When there was disagreement or uncertainty about merging items (i.e. can they legitimately be combined, or should they remain separate), we went back to the original data and re-analyzed the individual and specific meaning of the element and how it originally emerged in the data. There were not many inconsistencies between the derived success elements in each of the three studies. Given the controversial nature of priority setting, this finding might seem out of place; however, it showed that there are common elements reasonable people can agree on [ 10 ]. By providing a forum to discuss priority setting, different stakeholders were able to come to some agreement on elements important to any priority setting activity. Further, the aim of this conceptual framework was to identify higher-level elements of success, about which there seems to have been a certain amount of consensus across healthcare settings and stakeholder groups.

There were some contradictions within study 2, between the focus groups (patients/health system users and decision/policy makers) mainly to do with procedural elements of priority setting. For example, patients were less concerned with procedural efficiency, but more focused on partnership in public consultation and education. Decision makers saw the importance of public consultation, but spent more time discussing the priority setting process, highlighting (among other things) the importance of efficiency.

We circulated the conceptual framework and an explanation of the elements using electronic communications to a selection of participants from the three studies as well as a group of interdisciplinary scholars, for their comments and refinements – a type of 'member check'. Across the studies, 15 participants were invited to comment. Additionally, eight scholars were asked to comment on the framework. Seven participants and all eight scholars replied via email with comments and questions of clarity. Most of the comments had to do with wording of the elements. For example 'information management' was clarified and further qualified as 'clear and transparent information management'. Another example: 'improved' was added to 'stakeholder understanding' to reflect the idea of change over time. Revisions were made accordingly.

Each element is important individually, but is also related to the others, thus forming a robust and comprehensive definition of successful priority setting in a broad conceptual framework; each of the ten elements is described below.

Process Concepts

1. stakeholder engagement.

Stakeholder engagement refers to an organization's efforts to identify the relevant internal and external stakeholders and to involve these stakeholders effectively in the decision-making process. This should include, at a minimum, administrators, clinicians, members of the public and patients. To ensure adequate engagement, identifying and engaging stakeholders should involve multiple techniques, such as round tables, open forums, departmental meetings. There should be a genuine commitment from the organization to engage stakeholders effectively through partnership and empowerment. Stakeholder engagement is also concerned with stakeholder satisfaction regarding the level of their involvement in the decision-making process.

2. Use of Explicit Process

An explicit process is one that is transparent not only to decision makers, but also to other stakeholders. Adhering to a predetermined process can enhance trust and confidence in the process. Transparency means knowing who is making the decision, how the decision will be made, and why decisions were made. Communication needs to be well coordinated, systematic and well-planned. All stakeholders (internal and external) should be probed for relevant information to the priority setting decisions and information should be communicated effectively using multiple vehicles (town-hall, departmental meetings, memos, emails, etc.).

3. Information Management

Information management refers first to the information made available to decision makers during the priority setting process, including what was used and what was perceived to be lacking. Second, information management considers how the information was managed, including how it was collected and collated. Relevant information includes, but is not restricted to: health outcomes data, economic data (such as cost effectiveness analyses), community needs assessment, current policies or policy reports, and the experiences of both clinicians and patients.

4. Consideration of Values and Context

Values and context are important considerations in any priority setting process, including the values of the organization, the values of staff within that organization, and the values of other stakeholders (such as patients, policy makers, politicians, and members of the community). The mission, vision and values of the organization should guide priority setting. Priority setting decisions should be based on reasons that are grounded in clear value choices, and those reasons should be made explicit. This also involves not only looking within the organization at previous priority setting decisions, but also observing what other healthcare organizations are doing. This would involve looking at organizations in the local community, at other healthcare organizations with similar mandates, as well as looking at the other levels of healthcare provision. Context is distinct from values and considers the organization's goals in the health care environment articulated in its strategic directions.

5. Revision or Appeal Mechanism

A revision process is a formal mechanism for reviewing decisions and for addressing disagreements constructively. It is important to have such a mechanism and to ensure its rules and requirements are communicated clearly ahead of time. The dual purposes of a revision process are to: 1) improve the quality of decisions by providing opportunities for new information to be brought forward, errors to be corrected, and failures in due process to be remedied; and 2) to operationalize the key ethical concept of responsiveness.

Outcome Concepts

1. improved stakeholder understanding.

Stakeholder understanding implies more than basic knowledge of the process. It assumes stakeholders have gained insight into the priority setting (e.g. goals of the process, rationale for priority setting and rationale for priority setting decisions) and/or the organization (e.g. mission, vision, values, and strategic plan). As stakeholder understanding increases, stakeholder acceptance and confidence should also increase.

2. Shifted Priorities and/or Reallocated Resources

A successful priority setting process results in the allocation of budgets across portfolios, changes in utilization of physical resources (e.g. operating theatre schedules, bed allocations) or possibly changes in strategic directions. Effort that does not result in change may encourage the perception among stakeholders that the process was an inefficient use of time or mere window-dressing for pre-determined outcomes. A reaffirmation of previous resource allocation decisions (e.g. the previous year's budget) may, in some circumstances, be seen as a success.

3. Improved Decision Making Quality

Decision making quality relates to appropriate use of available evidence, consistency of reasoning, institutionalization of the priority setting process, alignment with the goals of the process and compliance with the prescribed process. It also captures the extent to which the institution is learning from its experience to facilitate ongoing improvement. This component is most obvious as subsequent iterations of priority setting are evaluated; where consistency and building on previous priority setting would be indicative of a successful process. Institutional learning, increased institutionalization of the priorities, more efficient decision making, more consistent decision making, or increased compliance with decisions (i.e. 'buy-in') are valuable, hard to achieve outcomes of successful priority setting. Institutional learning from its experiences facilitates ongoing institutional improvement, which appears as subsequent iterations of priority setting are evaluated.

4. Stakeholder Acceptance and Satisfaction

It is important to consider the satisfaction of all stakeholder groups, both internal to the hospital and external to the hospital (community groups/public and governmental health agencies/ministries of health). Successful priority setting leads to increases in satisfaction over multiple decision cycles. Stakeholder acceptance is indicated by continued willingness to participate in the process (i.e. 'buy-in') as well as the degree of contentment with the process. Stakeholders may be able to accept priority setting decisions, even if they may not always agree with the outcomes.

5. Positive Externalities

Positive externalities can act as a sort of check and balance, ensuring information is made transparent to stakeholders through various avenues, and/or establishing good practices for budgeting in other healthcare organizations. As an indicator of success, externalities may include positive media coverage (which can contribute to public dialogue, social learning, and improved decision making in subsequent iterations of priority setting), peer-emulation or health sector recognition (e.g. by other health care organizations, accreditation bodies, etc), changes in policies, and potentially changes to legislations or practice.

The primary purpose of this study was to develop a conceptual framework for successful priority setting. This research has helped elucidate elements of successful priority setting which can be used to assist organizations in their priority setting efforts.

Priority setting is complex, difficult, contentious and often controversial. Developing a conceptual framework is a necessary first step to approaching the evaluation of successful priority setting. The findings of the three studies described here have been synthesized into a conceptual framework which aims to provide guidance to decision makers (and other stakeholders) in better understanding successful priority setting.

The conceptual framework contains ten elements germane to successful priority setting. It is an advance in knowledge because it is the first attempt to comprehensively describe elements of successful priority setting from the point of view of stakeholders. It provides a way of thinking about successful priority setting and the considerations, or components, essential to achieving successful priority setting. It also provides a basis for decision makers to think specifically about successful priority setting and how to achieve it. Finally, it offers a common language for decision makers and stakeholders, within and between institutions, to discuss successful priority setting.

This research is complementary to previous studies that identified pieces of successful priority setting, and it builds and expands upon these previous works by describing a broad range of stakeholders' views about successful priority setting and synthesizes them into one conceptual framework that can be used by decision makers to improve priority setting. Further, this framework focuses both on process and outcomes of priority setting – other descriptive frameworks (e.g. accountability for reasonableness) focus only on the fairness of the process.

The ten elements identified in this framework are interconnected and often interdependent, it is difficult to use these elements in isolation. Elements were not weighted since there was no empirical evidence to suggest one element was more important than another. All elements are relative – that is, as conditions, they may be more or less met, and each may be improved.

Although the ten elements are not directly derived from moral theory, they hold normative relevance because they are derived from overlapping consensus of empirical observations involving the participants' reported values. Many of the participants were actual priority setting decision makers who are motivated to improve priority setting because they are directly involved in it. It is important to distinguish here between normative versus positive. This 'fact/value distinction' differentiates statements about what is the case from statements about what ought to be the case. Facts are descriptive, telling us what was done; values are prescriptive, telling us what should be done. The value-relevance of this study comes from the participants' values – i.e. their normative reasoning – not from the data analysis. In this research, we have 'described' participants' views; the participants have provided what they thought 'should be'.

The ten elements of successful priority setting in our framework have been organized into two types: process concepts and outcome concepts. Traditionally in health care, outcome measures refer to health outcomes (e.g. morbidity and mortality) in a selected population. However, the respondents in our three studies did not mention health outcomes as an element of successful priority setting. Thus, our framework does not include health outcomes in its list of priority setting outcomes. Our framework may then be criticized for not including health outcomes. A critic may ask: In an organization dedicated to health care, can a priority setting exercise that results in poorer health outcomes really be considered successful? Future scholarship should examine these divergent views. Health outcomes may be influenced by priority setting decisions, but are also influenced by a myriad other factors, such as quality of care. Outcome measures, such as mortality rates, may be helpful in evaluating the success of a healthcare organization, but there are many complicating factors – e.g. What about organizations that deliberately treat very complex cases? According to our respondents, achieving priority setting success is possible by focusing directly on priority setting outcomes, such as improved stakeholder understanding, shifted priorities, improved decision making, stakeholder acceptance, and positive externalities. Ultimately, we suspect that future research may find strong associations between health outcomes and priority setting outcomes such as stakeholder acceptance.

Our study is supported by previous studies that have reported on pieces of our framework [ 2 , 5 , 6 , 20 ]. While other frameworks exist to help decision makers with priority setting, our framework is more comprehensive. For example, Gibson, Mitton et al showed that while PBMA can be effective, it is not comprehensive; they suggested combining PBMA with A4R to achieve optimal benefits with available resources [ 42 ]. This framework is an advance because it was derived directly from conversations with people involved in priority setting about priority setting, and because it is the first time they have all been connected together in a comprehensive framework (table 6 ).

Our study builds and expands upon previous works by, for the first time, describing a broad range of stakeholder's views about successful priority setting and synthesizing them into one conceptual framework that can be used by actual decision makers to improve priority setting.

Limitations

The primary limitation of this research is its generalizability. The results of this research reflect the views of a wide range of key stakeholders, but most are from the Canadian health system, and they may not represent the views of stakeholders in other countries or cultures. It is also important to note that each country may have additional contextual elements of success. Moreover, participants were not sampled by a statistical method designed to yield generalizable results. The sampling technique was designed to probe a range of perspectives. Further research is required to determine the wider applicability of the concepts described here. Second, it is possible that the views provided by participants were shaped by social desirability bias, and responses given in the interviews might not correspond to what their organization actually does. However, we found no glaring inconsistencies between the interview data and the documentary support.

Using an innovative and robust mixed-methods approach, we have created a framework which attempts to provide much needed guidance to decision makers (and other stakeholders) to begin to improve the success of priority setting. Health care decision makers need guidance to set priorities. This study has helped elucidate the elements of successful priority setting which can be used to assist organizations in their priority setting efforts. Further research is needed to determine how best to utilize them to evaluate success of priority setting.

Appendix DELPHI ROUND ONE LIST OF ITEMS

Directly related to a4r.

Assessments of the health needs or other interests of the affected populations have been determined and documented. Other interests could take into account concessions on health needs for other gains or advantages (job security, education) as result from collective bargaining or political processes.

Representatives of different stakeholders groups are represented and meaningfully participate in the allocation decision-making process.

Data or generally accepted opinion exist that support specific allocation policies and management practices.

No policies or management practices (e.g., requirements for patients or providers) are in place that can frustrate access to the allocated health care services either purposely or inadvertently.

A systematic search and evaluation of evidence

Conformance with evidence would require expert judgment

The quality of decisions should be higher because rationales are required, there is less scope for decisions to be based on considerations other than the available evidence e.g. lobbying and political pressure, though lobbying will still occur.

Wide professional consultations

Communication materials and mechanisms made available by policy makers, and by surveys of stakeholders and direct observation approaches.

Decisions are public and accessible

Reasons are given in non-technical language

Policies, rationales, and requirements can be revised as made necessary by changes in objectives to providing allocations or new information or arguments that have a bearing on allocation decisions.

Policies and procedures in place addressing surveillance needs to determine when changes are necessary to general allocation policies and to adjudicate individual requests from stakeholders for revisions in general policies or individual decisions.

Documentation exists showing responses to new information or stakeholder requests for changes in policies or practices

Enforcement

Mechanisms exist that ensure the processes are available and function properly

Governmental regulatory requirements for compliance to processes.

Internal policies and procedures (including auditing functions) to ensure compliance.

Voluntary arrangements with independent third-parties exist to assess compliance with processes and/or to adjudicate stakeholder requests for changes in policies or for appeals of individual decisions.

Other forms of outcome indicators

Available through interested observers such as governmental agencies, courts, news media, and cultural apparatus; could include, but not be limited to the following:

Qualitative and quantitative measures of federal and local legislation and regulation targeting problems meant to be addressed by the main ideas of accountability for reasonableness

Qualitative and quantitative measures of complaints and grievances about health care service allocation policies and management practices brought by stakeholders in the process

Number of appeals submitted for unavailable health care services that can be tied to insufficient conformance to the main ideas of accountability for reasonableness

The number of lawsuits filed and the size of awards provided for problems that correspond to the main ideas of accountability for reasonableness

The number and nature of news media accounts of problems with health care service allocation policies and management practices

The frequency and nature of content in common cultural media (plays, movies, books)

Principles or criteria are explainable and justifiable to lay audiences need to have at their core the overriding responsibility to make decisions consistent with the public's health needs as well as available resources – both present and future.

Evaluation that has structure and is somewhat generic

An evaluation framework for measuring effectiveness of the given priority setting process that provides structure for evaluation but is also generic enough to be adapted in the local context

Tool provides guidance but is at the same time not overly prescriptive

Resource inequalities are compensated

Re-allocation of resources; improved patient outcomes

Relevant Stakeholders: consideration of the differing roles of governing bodies, executive management, operational management, and (in some situations) physicians and other health care professionals – but also alignment with the decision-making structure of the affected organization (who gets to decide what?).

The organization must be inclusive enough for the participation of key stakeholders, to be accepted by all parties; The organization must be exclusive enough to reach a limit-setting decision within reasonable time and resources; All key stakeholders have equal access and voice.

Stakeholder understanding: greater knowledge of why decisions have been made

Impact on stakeholder understanding of limits and their rationales

Measured in surveys in natural experiments

Measured in use of web pages or other devices for explaining limits, eg: of pharmacy benefits

Satisfaction of the participants: self-rated usefulness by participants; important to draw on the judgments of decision makers themselves and of key stakeholders; whether decisions 'felt fair' – as assessed by decision makers and stakeholders, and in the context of what has been achieved in other settings.

Policies and mechanisms in place to make affected populations aware of

Objectives to providing covered health care services

Health services available and specific conditions/requirements

Mechanisms available that facilitate access to covered health services, including appeals processes

Rationales for allocations, conditions, and requirements

Degree to which main ideas become embedded in culture: improvement could be measured by the nature and number of enhancements to the original process

Enhancement of market perception: of provider in situations where some providers promote themselves as abiding by A4R

High degree of stakeholder acceptance

High degree of reasonable public acceptance

Indirectly Related to A4R (but relevant to effectiveness)

There needs to be clear objectives/purpose: decision makers need to have clear objectives upon which they agree.

Commitment to implementation: without a commitment to implementation/follow-through based on the results, the process is incomplete and its credibility may be undermined for any subsequent use.

Maximization of benefits and minimization of opportunity costs

Effectiveness measured by efficiency:

An efficiently timed process that provides for meaningful involvement without demanding excessive time or effort.

a lengthy time for stakeholder involvement, etc., crucial energy and sustained knowledge/understanding and commitment can be compromised.

Martin D, Singer P: Priority Setting and health care technology assessment: beyond evidence-based medicine and cost-effectiveness analysis. The Global Challenge of Health Care Rationing. Edited by: Ham C, Coulter A. 2000, Buckingham: Open University Press, 135-145.

Google Scholar  

Gibson JL, Martin DK, Singer PA: Setting priorities in health care organizations: criteria, processes, and parameters of success. BMC Health Serv Res. 2004, 4 (1): 25-10.1186/1472-6963-4-25.

Article   PubMed   PubMed Central   Google Scholar  

Lomas J, Woods J, Veenstra G: Devolving authority for health care in Canada's provinces: 1. An introduction to the issues. CMAJ. 1997, 156 (3): 371-377.

CAS   PubMed   PubMed Central   Google Scholar  

Mitton C, Donaldson C: Setting priorities in Canadian regional health authorities: a survey of key decision makers. Health Policy. 2002, 60 (1): 39-58. 10.1016/S0168-8510(01)00190-7.

Article   PubMed   Google Scholar  

Mitton C, Patten S: Evidence based priority setting: what do the decision makers think?. Journal of Health Services Research and Policy. 2004, 9 (3): 146-152. 10.1258/1355819041403240.

Teng F, Mitton C, Mackenzie J: Priority Setting in the Provincial health services authority: survey of key decision makers. BMC Health Serv Res. 2007, 7 (84).

Reeleder D, Martin D, Keresztes C, Singer P: What do hospital decision-makers in Ontario, Canada, have to say about the fairness of priority setting in their institutions?. BMC Health Serv Res. 2005, 5 (8).

Holm S: Goodbye to the simple solutions: the second phase of priority setting in health care. British Medical Journal. 1998, 317: 1000-1002.

Article   CAS   Google Scholar  

Singer PA, Mapa J: Ethics of resource allocation: dimensions for healthcare executives. Hosp Q. 1998, 1 (4): 29-31.

CAS   PubMed   Google Scholar  

Daniels N, Sabin J: Limits to health care: Fair procedures, democratic deliberation and the legitimacy problems for insurers. Philosophy and Public Affairs. 1997, 26: 303-350. 10.1111/j.1088-4963.1997.tb00082.x.

Mitton C, Donaldson C: Twenty-five years of programme budgeting and marginal analysis in the health sector, 1974–1999. Journal of Health Services Research and Policy. 2001, 6 (4): 239-248. 10.1258/1355819011927558.

Article   CAS   PubMed   Google Scholar  

Battista RN, Hodge MJ: The evolving paradigm of health technology assessment: reflections for the millennium. Canadian Medical Association Journal. 1999, 160 (19): 146-147.

Menon D, Stafinski T, Martin D: Priority-setting for healthcare: Who, how, and is it fair?. Health Policy. 2007, 84 (2–3): 220-233. 10.1016/j.healthpol.2007.05.009.

Martin D, Singer P: A Strategy to Improve Priority Setting in Health Care Institutions. Health Care Analysis. 2003, 11 (1): 59-68. 10.1023/A:1025338013629.

Mitton C, Donaldson C: Priority Setting Toolkit: A guide to the use of economics in healthcare decision making. 2004, London: BMJ Publishing Group

Coster C, McMillan S, Brant R, McGurran J, Noseworthy T: Primary Care Panel of the Western Canada Waiting List Project: The Western Canada Waiting List Project: development of a priority referral score for hip and knee arthroplasty. Journal of Evaluation in Clinical Practice. 2007, 13: 192-197. 10.1111/j.1365-2753.2006.00671.x.

Gauld , Derrett : Solving the Surgical Waiting List Problem? New Zealand's 'Booking System'. International Journal of Health Planning and Management. 2000, 15: 259-272. 10.1002/hpm.596.

McKeen M, MacKenzie M: Report of the Provincial Wait Time Monitoring Project Steering Committee. Nova Scotia. 2004

Bell JA, Hyland S, DePellegrin T, Upshur RE, Bernstein M, Martin DK: SARS and hospital priority setting: a qualitative case study and evaluation. BMC Health Serv Res. 2004, 4 (1): 36-10.1186/1472-6963-4-36.

Madden S, Martin DK, Downey S, Singer PA: Hospital priority setting with an appeals process: a qualitative case study and evaluation. Health Policy. 2005, 73 (1): 10-20.

Reeleder D, Goel V, Singer P, Martin D: Accountability Agreements in Ontario Hospitals: Are They Fair?. Journal of Public Administration Research and Theory. 2008, 18 (1): 161-175. 10.1093/jopart/mul024.

Article   Google Scholar  

Mitton C, Mackenzie J, Cranston L, Teng F: Priority Setting in the Provincial Health Services Authority: Case Study for the 2005/06 Planning Cycle. Healthcare Policy. 2006, 2 (1): 91-106.

PubMed   PubMed Central   Google Scholar  

Martin D, Shulman K, Santiago-Sorrell P, Singer P: Priority-setting and hospital strategic planning: a qualitative case study. J Health Serv Res Policy. 2003, 8 (4): 197-201. 10.1258/135581903322403254.

Martin DK, Pater JL, Singer PA: Priority-setting decisions for new cancer drugs: a qualitative case study. Lancet. 2001, 358 (9294): 1676-1681. 10.1016/S0140-6736(01)06714-9.

Cohen D: Messages from Mid Glamorgan: a multi-programme experiment with marginal analysis Health Policy. Health Policy. 1995, 33 (2): 147-155. 10.1016/0168-8510(95)93674-P.

Chappel D, Bailey J, Stacy R, Rodgers H, Thomson R: Implementation and evaluation of local-level priority setting for stroke. Public Health. 2001, 115 (1): 21-9. 10.1016/S0033-3506(01)00409-7.

Bate A, Donaldson C, Ray H: Resource Allocation in Orthopedics. Clinical Orthopaedics and Related Research. 2007, 457: 49-56.

PubMed   Google Scholar  

Williams I, Bryan S: Understanding the limited impact of economic evaluation in health care resource allocation: A conceptual framework. Health Policy. 2007, 80 (1): 135-143. 10.1016/j.healthpol.2006.03.006.

Peacock S, Ruta D, Mitton C, Donaldson C, Bate A, Murtagh M: Using economics to set pragmatic and ethical priorities. British Medical Journal. 2006, 332: 482-485. 10.1136/bmj.332.7539.482.

Peacock S: An Evaluation of Program Budgeting and Marginal Analysis Applied in South Australian Hospitals. 1998, West Heidelberg, Australia: Centre for Health Program Evaluation

Mitton CR, Donaldson C: Setting priorities and allocating resources in health regions: lessons from a project evaluating program budgeting and marginal analysis (PBMA). Health Policy. 2003, 64 (3): 335-348. 10.1016/S0168-8510(02)00198-7.

Ozar D, Berg J, Werhane P, Emanuel L: Organizational Ethics in Health Care: Toward a Model for Ethical Decision Making by Provider Organization In Institute for Ethics National Working Group Report. American Medical Association. 2001

Peck , Asch , Goold , Roter , Ubel , McIntyre , Abbott , Koropchak , Tulsky : Measuring patient expectations: does the instrument affect satisfaction or expectations?. Medical Care. 2001, 39 (1): 100-108. 10.1097/00005650-200101000-00011.

Feinstein AR: Clinical Judgment. 1967, Baltimore: Williams & Wilkins

Mitton C, Donaldson C: Tools of the trade: a comparative analysis of approaches to priority setting in healthcare. Health Services Management Research. 2003, 2: 96-105. 10.1258/095148403321591410.

Adler M, Ziglio E: Gazing Into the Oracle: The Delphi Method and it's Implications to Social Policy and Public Health. 1996, London: Jessica Kingsley Publishers

Gallagher TH, Waterman AD, Ebers AG, Fraser VJ, Levinson W: Patients' and physicians' attitudes regarding the disclosure of medical errors. Jama. 2003, 289 (8): 1001-1007. 10.1001/jama.289.8.1001.

Strauss A, Corbin J: Basics of Qualitative Research: Techniques and Procedures of Developing Grounded Theory. 1998, Thousand Oaks: Sage Publications

Richardson L: Writing: a method of inquiry. Handbook of Qualitative Research. Edited by: Denzin NK, Lincoln YS. 1994, London: Sage Publications Inc

Altheide DL, Johsnson JM: Criteria for Assessing Interpretive Validity in Qualitative Research. Handbook of Qualitative Research. Edited by: Denzin NK, Lincoln YS. 1994, Thousand Oaks: Sage Publications Inc, 485-499.

Mays N, Pope C: Rigour and Qualitative Research. British Medical Journal. 1995, 311: 109-112.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Gibson J, Mitton C, Martin D, Donaldson C, Singer P: Ethics and economics: does programme budgeting and marginal analysis contribute to fair priority setting?. J Health Serv Res Policy. 2006, 11 (1): 32-37. 10.1258/135581906775094280.

Pre-publication history

The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1472-6963/9/43/prepub

Download references

Acknowledgements

Funding for study provided by a grant from an Interdisciplinary Capacity Enhancement grant from the Canadian Institutes of Health Research.

The views expressed herein are those of the authors, and do not necessarily reflect those of the supporting groups. This research was supported by the Canadian Institutes of Health Research and the Charles E. Frosst Foundation. SLS is supported by the University of Toronto and through a CIHR grant of the Canadian Priority Setting Research Network. DKM is supported by a Canadian Institutes of Health Research New Investigator award.

All correspondence should be sent to: Dr. Douglas K. Martin, University of Toronto Joint Centre for Bioethics, 88 College Street, Toronto, Ontario, M5G 1L4, Canada, tel: 416-978-6926, fax: 416-978-1911, e-mail: [email protected]

Author information

Authors and affiliations.

Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada

Shannon L Sibbald & Douglas K Martin

University of Toronto Joint Centre for Bioethics, Toronto, Canada

Shannon L Sibbald, Ross Upshur & Douglas K Martin

The McLaughlin-Rotman Centre for Global Health, Toronto, Canada

Peter A Singer

Sunnybrook Health Science Centre, Toronto, Canada

Ross Upshur

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Shannon L Sibbald .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors' contributions

SS was the primary analyst and principal author of the manuscript. DKM conceived the research, was involved in data collection and analysis, and was co-author of the manuscript. RU and PAS were involved in study conception, analysis and drafting the manuscript. All authors read and approved the final manuscript.

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Sibbald, S.L., Singer, P.A., Upshur, R. et al. Priority setting: what constitutes success? A conceptual framework for successful priority setting. BMC Health Serv Res 9 , 43 (2009). https://doi.org/10.1186/1472-6963-9-43

Download citation

Received : 17 June 2008

Accepted : 05 March 2009

Published : 05 March 2009

DOI : https://doi.org/10.1186/1472-6963-9-43

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Decision Maker
• Priority Setting
• Healthcare Organization
• Stakeholder Engagement
• Priority Setting Process

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Search Menu
• Sign in through your institution
• Advance Articles
• Editor's Choice
• Supplements
• Open Access Articles
• Research Collections
• Review Collections
• Author Guidelines
• Submission Site
• Open Access Options
• Self-Archiving Policy
• About Health Policy and Planning
• About the London School of Hygiene and Tropical Medicine
• HPP at a glance
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Introduction, what is a discrete choice experiment, how to design a dce.

• < Previous

How to do (or not to do) … Designing a discrete choice experiment for application in a low-income country

• Article contents
• Figures & tables
• Supplementary Data

Lindsay J Mangham, Kara Hanson, Barbara McPake, How to do (or not to do) … Designing a discrete choice experiment for application in a low-income country, Health Policy and Planning , Volume 24, Issue 2, March 2009, Pages 151–158, https://doi.org/10.1093/heapol/czn047

• Permissions Icon Permissions

Understanding the preferences of patients and health professionals is useful for health policy and planning. Discrete choice experiments (DCEs) are a quantitative technique for eliciting preferences that can be used in the absence of revealed preference data. The method involves asking individuals to state their preference over hypothetical alternative scenarios, goods or services. Each alternative is described by several attributes and the responses are used to determine whether preferences are significantly influenced by the attributes and also their relative importance. DCEs are widely used in high-income contexts and are increasingly being applied in low- and middle-income countries to consider a range of policy concerns. This paper aims to provide an introduction to DCEs for policy-makers and researchers with little knowledge of the technique. We outline the stages involved in undertaking a DCE, with an emphasis on the design considerations applicable in a low-income setting.

Conducting a discrete choice experiment in a developing country context can involve issues not encountered in developed countries.

Selection of attributes in key. Although secondary literature can be used to identify an initial set of attributes, in low-income settings additional primary research is almost always necessary to ensure that the final set of attributes is appropriate and valid.

Pre-testing the questionnaire is likely to be particularly important where there are cultural and language differences between the researchers and study participants, or where the population surveyed has a low level of literacy.

A discrete choice experiment (DCE) is a quantitative technique for eliciting individual preferences. It allows researchers to uncover how individuals value selected attributes of a programme, product or service by asking them to state their choice over different hypothetical alternatives. DCEs have been applied to a range of health policy, planning and resource allocation decisions in high-income settings. These include the elicitation of views on diagnosis, treatment and care (Coast et al. 2006 ; King et al. 2007 ; Lancsar et al. 2007 ; Kjaer and Gyrd-Hansen 2008 ), access to services (Gerard and Lattimer 2005 ; Gerard et al. 2006 ; Longo et al. 2006 ) and the employment preferences of health personnel (Scott 2001 ; Ubach et al. 2003 ; Wordsworth et al. 2004 ).

Although DCEs have been frequently applied in health economic research in high-income countries, there are comparatively few examples of DCEs being used elsewhere. Searches of the published literature identified only four papers that had employed a DCE to consider a health issue in a low- and middle-income country (LMICs) (Hanson et al. 2005 ; Baltussen et al. 2006 ; Christofides et al. 2006 ; Baltussen et al. 2007 ) and a further three papers were identified by undertaking selective searches of the internet (Chomitz et al. 1997; Penn-Kekana et al. 2005 ; Baltussen and Niessen 2006 ). Moreover, we are aware of several DCEs currently underway in LMICs. These include: an examination of public preferences for maternity services in Ghana (Ternent et al. 2006 ) and an estimation of women's demand for barrier methods for HIV prevention in South Africa (Terris-Prestholt et al. 2008a , b ). In addition, there are ongoing DCEs to elicit the employment preferences of health workers, including of doctors and nurses in Ethiopia (Hanson and Jack 2007 ), clinical officers in Tanzania (O Maestad and J Riise Kolstad, personal communication) and several cadres in Malawi, South Africa and Tanzania (C Normand, personal communication). In LMICs, DCEs have been more frequently used outside of the health sector, with academic articles reporting preferences in the agriculture, water, transport and tourism sectors (Baidu-Forson et al. 1997 ; Tiwari and Kawakami 2001 ; Tano et al. 2003 ; Hope and Garrod 2004 ). Internet searches also returned several discussion papers, reports and other sources of grey literature that had applied the technique in these settings (Carlsson et al. 2005 ; Porras and Hope 2005 ). Nonetheless, the use of DCEs in LMICs remains relatively recent.

This paper aims to provide an introduction to the DCE methodology for those working on health issues in low-income countries. We describe the technique, outline the stages involved in its application and emphasize the design considerations pertinent to conducting a DCE in a low-income setting. In discussing the design process, we draw on our experiences of conducting DCEs to elicit patient preferences of hospital services in Zambia (Hanson et al. 2005 ) and employment preferences of health workers in Malawi (Mangham and Hanson, 2008 ).

Techniques for eliciting preferences have primarily emerged from a desire to understand consumer demand for goods and services where it was not possible to use revealed preference data on the actual choices made by individuals. This would arise for products and services not traded on a market, such as for a new product under development and not yet commercially available. Similarly, where there is no variation in the products available (or services provided), it is not possible to isolate the contribution of each product attribute to the overall utility derived from the product.

DCEs require respondents to state their choice over sets of hypothetical alternatives. Each alternative is described by several characteristics, known as attributes, and responses are used to infer the value placed on each attribute. In comparison to other stated preference techniques that require the individual to rank or rate alternatives, a DCE presents a reasonably straightforward task and one which more closely resembles a real-world decision.

DCEs are used to determine the significance of the attributes that describe the good or service and the extent to which individuals are willing to trade one attribute for another (Drummond et al. 2005 ). Information on the relative importance of the selected attributes can be useful for those involved in policy decisions and setting resource allocation priorities, and may be designed with that in mind (Baltussen and Niessen 2006 ; Baltussen et al. 2007 ). For example, a DCE on the employment preferences of registered nurses in Malawi was designed in light of the Malawi government's programme to recruit and retain health personnel, which included salary increases to improve motivation and discourage attrition (Palmer 2006 ). The results of the DCE showed that remuneration had a significant impact on how nurses viewed their employment, and that they were willing to trade-off pay increases to obtain improvements in non-monetary benefits or working conditions (Mangham and Hanson, 2008 ).

The method can be used to estimate the marginal valuations of attributes or the willingness to pay (WTP) for a unit change in each attribute estimated (Drummond et al. 2005 ). In comparison with standard techniques, which estimate WTP for the good or service as a whole, this more detailed information on WTP by attribute may be useful, though some evidence suggests that the levels of the cost attribute can affect the estimates (Radcliffe 2000 ; Drummond et al. 2005 ). DCE results can also be analysed by subgroup and it is possible to consider the extent to which individuals’ characteristics impact on the marginal valuations. For instance, a DCE applied to elicit the patient preferences for hospital quality in Zambia estimated WTP for the different attributes for the whole sample and by socio-economic group. They found the highest willingness to pay was for a thorough, rather than superficial, examination and that the WTP for this attribute in the lowest socio-economic group was only about half that in the highest group.

Discrete choice analysis involves the construction of an experimental design to study the effects of the attribute levels on the stated preference (or dependent variable). The attributes of an experimental design are variables that have two or more fixed levels.

There are several stages to the design of a DCE, which we outline below. In our discussion the design considerations pertinent to conducting a DCE on a health-related issue in a low-income setting are highlighted. We also use examples from our own experience of undertaking DCEs in sub-Saharan Africa, to elicit patient preferences on the quality of hospital services in Zambia (Hanson et al. 2005 ), and the employment preferences of registered nurses in Malawi (Mangham and Hanson, 2008 ).

Establishing attributes

The first stage of a DCE involves identifying the attributes relevant to the stated research question and then assigning levels for each of these attributes (Ryan 2001 ; Hensher et al. 2005 ). Since these attributes and attribute levels describe the hypothetical scenarios under consideration in the DCE, this is a critical aspect of the design. The underlying validity of the study depends, therefore, on the researcher's ability to correctly specify the relevant attributes. Despite the importance of this stage in the design, there is often sparse explanation in the DCE literature of how attributes and levels are established (Coast and Horrocks 2007 ).

Selecting and defining the attributes requires a good understanding of the target population's perspective and experience (Hall et al. 2004 ; Coast and Horrocks 2007 ). Researchers working in a different cultural or language setting may find obtaining the necessary depth of understanding a challenge and may want to involve local experts. Policy concerns may also shape the choice of attributes, and it is often advisable to engage local institutions and policy-makers during this preparatory stage (Baltussen and Niessen 2006 ). For example, in the Malawi study on nurses’ employment preferences, the decision to include an attribute on the provision of government housing was influenced by the Ministry of Health's interest in how the availability and quality of government housing affects the retention of health personnel.

Published and grey literature, such as policy documents and government reports, are a useful starting point for identifying attributes. For example, the literature on the global shortage of health workers (Buchan and Sochalski 2004 ; Joint Learning Initiative 2004 ; World Health Organization 2006 ) provided a background to the Malawi study, while Malawi government policy and programme documents provided detail on the country-specific situation (Buchan and Sochalski 2004 ; Joint Learning Initiative 2004 ; World Health Organization 2006 ; Coast and Horrocks 2007 ).

It is often necessary, however, to augment secondary sources with primary data to ensure that the DCE is tailored to the study setting. Primary qualitative data were essential for selecting and defining the attributes that registered nurses considered important. Semi-structured interviews were conducted with public sector nurses to obtain information on their current working conditions, preferences, reasons for the shortage of nurses and priorities for reform. The sample included views of nurses based in urban and in rural settings in three geographically distinct districts, working in primary, secondary and tertiary health facilities, and with differing degrees of seniority and experience. Additional interviews were also held with senior officials in the Ministry of Health to obtain their perceptions, policy concerns and remuneration data. In Zambia, qualitative data were collected using focus group discussions on what dimensions of the quality of hospital services were important to individuals when choosing a hospital. In total, 16 separate discussions were held with men and women from two districts, and from different socio-demographic groups: university students, market sellers, residents from high and low density areas (Hanson et al. 2005 ).

In both settings, the discussions were facilitated by a sociologist from a local research institute and were tape recorded, transcribed, translated and supplemented by detailed written notes. Local researchers brought a valuable perspective, knowledge and experience to the design of the DCE. They led the collection of qualitative data, which not only encouraged respondents to be more open, but also allowed them to express themselves in their local languages.

In both studies content analysis was used to obtain possible attributes for the DCE. This involved reading all transcripts and written notes to identify major themes, followed by a more detailed review, during which additional themes and sub-themes emerged. The transcripts were coded manually and relevant extracts recorded in a summary table. Software packages can also assist the coding and retrieval of data, and are particularly useful where the number of interviews is large or for more complex qualitative analysis (Lewando-Hundt et al. 1997 ).

There are no design restrictions on the number of attributes that could be included in a DCE, though in practice most DCEs have contained fewer than 10 to ensure that respondents are able to consider all attributes listed when making their choice (DeShazo and Fermo 2002 ). The greater the number of attributes, the greater the cognitive difficulty of completing a DCE. With too many attributes, the participants may be encouraged to apply a simple decision rule in which they base their response on a single or subset of attributes. In establishing attributes it is also important to avoid conceptual overlap between two or more of the attributes, known as inter-attribute correlation, since it would prevent the accurate estimation of the main effect of a single attribute on the dependent variable.

In both the Malawi and Zambia studies, six attributes were established that were important to the respondents and were policy relevant. For example, the attributes relating to the quality of hospital services identified in the Zambia study were: (1) the likelihood that the child will receive all the drugs s/he needs at the hospital, (2) the likelihood that the hospital staff will examine the child properly, (3) staff attitudes, (4) cleanliness of the wards and toilets, and (5) the waiting time between arrival at the outpatients department and admission to the ward. In addition a sixth cost attribute was included that covered the costs of the child's examination and treatment. Other issues frequently mentioned by discussants were not included in the DCE because of inter-attribute correlation with the established attributes. They were: the availability of diagnostic services, quality of nursing care, staff dedication and availability of staff (Hanson et al. 2005 ).

In specifying attributes, care should be taken to ensure that definitions are appropriate for the setting and are not ambiguous. For example, the Malawian nurses frequently used the term ‘upgrading’ when referring to obtaining additional professional qualifications. These long-term educational opportunities are distinct from in-service training courses, and in defining the attribute on access to training it was therefore important to use the ‘upgrading’ expression to avoid attribute ambiguity.

Assigning attribute levels

Once the attributes are established, attribute levels need to be assigned. Typically the levels chosen should reflect the range of situations that respondents might expect to experience, and again qualitative data are valuable. Ensuring the levels are realistic and meaningful will increase the precision of parameter estimates (Hall et al. 2004 ).

In the Malawi study, qualitative data were used to determine base levels that reflected the prevailing working conditions for the public sector registered nurses. Additional levels were then established that represented a reasonable improvement from the base. For ease of cognition we sought to establish no more than three levels for each attribute, initially opting for two levels and then adding a third where there was no single base level. For example, variation in the actual provision of government housing meant both ‘no government housing’ and ‘basic government housing’ were possible base levels. Three levels were also used for the pay attribute as we were interested in the preferences over the magnitude of the pay increase. A net monthly pay of K30,000 Malawi kwacha (approximately US$240) was an average prevailing salary for public sector registered nurses, and two higher levels were included, K40,000 ($320) and K50,000 ($400). These higher pay levels were similar to those in the private sector and in line with what the interviewed nurses had indicated they would consider a reasonable salary.

Designing the choice sets

The next stage in the design of a DCE is to generate the hypothetical alternatives and to combine them to create choice sets. A full factorial design can be generated which consists of all possible combinations of the levels of the attributes, and permits estimation of main effects and interactions. A main effect refers to the direct independent effect on the choice variable of the difference in attribute levels (e.g. difference in price). An interaction effect is the effect on the choice variable obtained by varying two or more attribute levels together (e.g. difference in price combined with difference in colour).

In most practical situations it is considered too cost-prohibitive and tedious to have subjects rate all possible combinations in a full factorial design (Kuhfeld 2005 ). A design with five attributes, each with three levels would, for example, generate 243 possible alternatives (3 5 ). Thus, fractional factorial designs are often used to consider a selection of possible alternatives.

In selecting a fractional factorial design, researchers should seek to obtain a design that is both orthogonal and balanced (Huber and Zwerina 1996 ; Kuhfeld 2005 ). In orthogonal fractional factorial designs, the parameter estimates in the linear model are uncorrelated, which means that the attributes of the design are statistically independent of each other (Hensher et al. 2005 ; Kuhfeld 2005 ). A balanced design has each attribute level occurring equally often, and this minimizes the variance in the parameter estimates (Kuhfeld 2005 ). Fractional factorial designs that are both orthogonal and balanced are known as orthogonal arrays and these can be obtained from design websites such as http://www.research.att.com/~njas/oadir (Burgess and Street 2005 ). However, orthogonal arrays only exist for certain combinations of attributes and attribute levels (Kuhfeld 2005 ). For other combinations there will be a trade-off between the degrees of orthogonality and balance. Researchers should select the most efficient design, using a measure known as D-efficiency (Carlsson and Martinsson 2003 ; Burgess and Street 2005 ; Kuhfeld 2005 ; Street et al. 2005 ).

A third property that characterizes efficient choice designs is minimal overlap (Huber and Zwerina 1996 ; Maddala et al. 2003 ). Each attribute level is only meaningful in comparison to others within the choice set, or in other words no information is obtained on an attribute's value when its levels are the same across all alternatives within a choice set. Researchers should, therefore, seek to minimize the probability that an attribute level repeats itself in each choice set. Finally, Huber and Zwerina ( 1996 ) have argued for the importance of utility balance, which refers to balancing the utilities of the alternatives offered in the choice set, though in practice the lack of prior information on the utility of attributes limits the applicability of this criterion (Huber and Zwerina 1996 ).

The recommended approach to constructing an experimental design continues to evolve (DeShazo and Fermo 2002 ; Street et al. 2005 ; Adamowicz 2006 ; Louviere 2006 ) and ensuring that the generated DCE meets best practice can be a challenge. Consequently, we encourage researchers planning to undertake a DCE to review the latest publications, within and beyond the discipline of health economics. We are also aware that access to the latest information may be an obstacle for those undertaking research in LMICs.

Generating and pre-testing the questionnaire

The created choice sets form the basis for the DCE questionnaire. The number of choice sets presented to each individual will depend on the size of the fractional factorial design and the strategy employed in designing the choice sets (Street et al. 2005 ). Typically DCEs ask respondents to consider up to 18 choice sets, with 18 representing a practical limit of how many comparisons can be completed before boredom sets in (Hanson et al. 2005 ; Christofides et al. 2006 ). This boredom threshold level is likely to vary, and will depend on the number of choice sets, their complexity and the characteristics of the target population.

In both the Zambia and the Malawi studies we applied a pairwise design, such that respondents were asked to consider a choice set with two alternatives and state their preference for each pair. For illustration, an example of a choice set from the Malawi study is shown in Figure 1 . A dichotomous choice has frequently been applied in health services research and the statistical information obtained represents the demand conditional on accepting one of the two scenarios.

Example of a choice set from the study of employment preferences of Malawian registered nurses

In the Malawi study we also asked a second question that introduced a ‘choose neither’ option. This allowed respondents to reject both alternatives and provided data to estimate unconditional demand. In many situations the inclusion of a non-choice option will more closely resemble a real world context, since individuals are not required to make a choice, though these non-choice responses do not provide any information on how individuals trade-off the attribute levels of the available alternatives. In our design, the motivation behind asking both which job they considered superior and which one they would choose was to ascertain the extent to which the personal circumstances of respondents were an explicit constraint in their decision-making. For example, we hypothesized that married women would be more constrained in making their choice over place of work.

The internal consistency of responses can be considered by including one or two choice pairs in which one option is superior to the other on all characteristics. In the Zambia study this assumed that people prefer a lower cost, more thorough examination, a shorter waiting time, and better drug availability, cleanliness and staff attitudes. Individuals that fail to choose the superior hospital job may have misunderstood the task or were unable to provide consistent answers because of problems of communication or translation. While it is useful to know the extent to which individuals respond rationally, Lancsar and Louviere ( 2006 ) advise against excluding apparently ‘irrational’ choices from the analysis of DCEs as that may cause statistical bias and/or affect statistical efficiency.

The questionnaire should be clearly presented and contain a standard introduction to the DCE with choice set examples. To minimize any bias caused by the order in which the choice sets occur or the attributes are described, it is good practice to produce several versions of the questionnaire in which choice sets and attributes are presented in different orders (Kjaer et al. 2006 ). Pictures, diagrams and symbols may aid comprehension, and are particularly relevant for conducting a DCE in low-income countries where literacy cannot be assumed. Similarly, in some settings the questionnaire will need to be translated into one or more local languages. For example, picture boards and verbal descriptions were used when eliciting preferences on hospital quality in Zambia and interviews were administered in both Bemba and English (Hanson et al. 2005 ). Finally, it is usual to collect data on socio-economic indicators to allow analysis of the impact of individual characteristics on the choices made.

In preparing the questionnaire it is also important to consider how the DCE will be administered. It is possible for the questionnaire to be self-administered, undertaken in examination conditions (Chomitz et al. 1997) or, as we opted, to have trained fieldworkers administer the questionnaire to respondents individually. In many low-income countries, postal or online surveys will not be feasible because of the infrastructure, and given the lower levels of literacy, there are also likely to be practical advantages to having fieldworkers explain what is asked of consenting respondents and work through examples.

Piloting the questionnaire is a key stage in most survey designs and is particularly relevant when working across cultures and several languages. Moreover, pre-testing provides an opportunity to review several elements of the design process, including the selection and definition of attributes and their levels (Hall et al. 2004 ). This is important given the extensive role of the researcher in the design of the DCE. For example, pre-testing in the Malawi study identified the conceptual overlap between the type of hospital and the typical workload, which led to the rewording of the levels relating to place of work (Mangham and Hanson, 2008 ). Minor modifications were also made to the definition of the attribute levels for typical workload. Other aspects of the design which should be reviewed during pre-testing include the respondent's understanding of the task, their ease of comprehension and whether the number of choice sets can be managed by the target population (Hall et al. 2004 ).

Analysing of DCE data

Once the choice sets and questionnaire design are finalized, the DCE questionnaire can be administered to collect data for subsequent analysis. Our discussion on data analysis is restricted to a few key aspects, since the focus of this paper is on how to design a DCE. Furthermore, the same methods for analysing DCE data apply irrespective of the study context and are well documented elsewhere (Louviere et al. 2000 ; Hensher et al. 2005 ).

Moreover, as respondents are asked to consider multiple choice pairs, it cannot be assumed that the error terms are independent and panel data estimation techniques are required. The estimated parameters represent the marginal utility associated with a change in the attribute level in moving from one alternative to the other.

In developing countries the application of DCEs to consider questions of health policy and planning is relatively recent, but appears to be of growing interest (Chomitz et al. 1997; Hanson et al. 2005 ; Penn-Kekana et al. 2005 ; Baltussen et al. 2006 ; Christofides et al. 2006 ; Ternent et al. 2006 ; Mangham and Hanson 2008 ). Moreover, Baltussen and Niessen ( 2006 ) argue that a multi-criteria approach to health priority setting is essential and DCEs, as a technique for undertaking multi-attribute analysis, should be used more routinely to guide resource allocation decisions.

The stages involved in the design of a DCE are the same regardless of study setting: establishing attributes and attribute levels, designing choice sets, and generating and pre-testing the DCE questionnaire. There are, however, some particular challenges in conducting a DCE in a developing country context, and in describing each element of the design we have sought to highlight the additional design considerations. These challenges may relate to working in different cultural or language settings, or surveying populations that have a lower level of literacy or are less accustomed to market research techniques. Many of these challenges will be common across a range of research methodologies and will be familiar to those experienced in undertaking research in LMICs. Nevertheless, we believe that there is merit in considering aspects of the DCE design that may require more attention when the method is to be applied in a low-income setting.

For example, the use of primary data when establishing attributes and attribute levels may be more critical in a low-income setting. The validity of the research findings depends on the analyst's ability to correctly specify the relevant programme, product or service attributes and levels, and this requires a detailed understanding of the target populations’ experience and point of view (Hall et al. 2004 ). Access to relevant information on the policy context or specific health programmes can be a challenge, particularly in developing countries, and the perspectives and concerns of individuals are often poorly articulated. Thus, although secondary literature can be used to identify an initial set of attributes, in low-income settings additional primary research is almost always necessary to ensure that the final set of attributes is appropriate and valid.

Similarly, pre-testing the questionnaire is likely to be even more important in a context where there are cultural and language differences between the researchers and study participants, or where the population surveyed has a lower level of literacy. Pre-testing provides an opportunity to ensure that the information is presented in a comprehensible way, and that the choices are realistic and meaningful. It also provides an opportunity to observe how cognitively demanding the questionnaire is for respondents to complete. Researchers may, for example, need to reduce the number of choice sets reviewed, adjust the number of attributes in each alternative or include pictorial information and verbal descriptions. Such amendments should help to ensure that responses are a better reflection of individual preferences and improve the precision in the parameter estimates.

Given some of these additional challenges of designing a DCE for a low-income country, it is reasonable to ask whether a research methodology that was originally developed to understand demand for consumer products in high-income countries readily translates to questions of public policy and health resource allocation in low-income countries. Our experience, and that from the literature, suggests that participants are able to state their preferences over health service provision and areas for policy reform (Chomitz et al. 1997; Hanson et al. 2005 ; Penn-Kekana et al. 2005 ; Baltussen et al. 2006 ; Christofides et al. 2006 ; Ternent et al. 2006 ; Mangham and Hanson, 2008 ). The results also suggest that the preferences are reasoned and deliberate. For instance, the internal consistency of responses was high in both the Zambia and Malawi studies, from which we inferred that the vast majority of individuals were making rational choices. Similarly, the theoretical validity of the valuations, which is assessed by determining whether the estimated coefficients are of the anticipated sign, found that the results were consistent with prior expectations.

DCEs that elicit preferences on the provision of health services or strategies for policy reform should be useful for health policy-makers and planners involved in identifying priorities for resource allocation. Although the task of choosing between alternative scenarios is reasonably straightforward to comprehend, some elements of the design are complex. It will be important, therefore, that researchers are able to communicate the research findings and policy implications in a way that can be easily understood.

DCEs, and other stated preference techniques, also have the advantage that they are a means of obtaining preferences on situations that are not traded explicitly in markets, as is often the case with health care, or have public good characteristics, such as a vaccination programme (Pokhrel 2006 ). Similarly, their ability to be used in situations that are purely hypothetical means that it is possible to elicit preferences over potential policy reform or health system changes prior to their implementation. Moreover, as the design of DCEs is controlled and the attributes are varied systematically, it is straightforward to identify the effect of an attribute on the choice variable (Baltussen and Niessen 2006 ; Pokhrel 2006 ).

As the benefits of the DCEs become more widely understood we expect the technique to be increasingly applied to health policy and planning questions in developing countries. We hope this paper provides a useful introduction for those wanting to gain a better understanding of the methodology and the process of designing a DCE.

Google Scholar

Google Preview

• developing countries
• health personnel
• health policy
• patient preferences
• quantitative methods

Email alerts

Citing articles via.

• Recommend to Your Librarian

Affiliations

• Online ISSN 1460-2237
• Copyright © 2024 The London School of Hygiene and Tropical Medicine and Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• Systematic review
• Open access
• Published: 23 November 2017

Application of discrete choice experiments to enhance stakeholder engagement as a strategy for advancing implementation: a systematic review

• Ramzi G. Salloum 1 ,
• Elizabeth A. Shenkman 1 ,
• Jordan J. Louviere 2 &
• David A. Chambers 3  

Implementation Science volume  12 , Article number:  140 ( 2017 ) Cite this article

One of the key strategies to successful implementation of effective health-related interventions is targeting improvements in stakeholder engagement. The discrete choice experiment (DCE) is a stated preference technique for eliciting individual preferences over hypothetical alternative scenarios that is increasingly being used in health-related applications. DCEs are a dynamic approach to systematically measure health preferences which can be applied in enhancing stakeholder engagement. However, a knowledge gap exists in characterizing the extent to which DCEs are used in implementation science.

We conducted a systematic literature search (up to December 2016) of the English literature to identify and describe the use of DCEs in engaging stakeholders as an implementation strategy. We searched the following electronic databases: MEDLINE, Econlit, PsychINFO, and the CINAHL using mesh terms. Studies were categorized according to application type, stakeholder(s), healthcare setting, and implementation outcome.

Seventy-five publications were selected for analysis in this systematic review. Studies were categorized by application type: (1) characterizing demand for therapies and treatment technologies ( n  = 32), (2) comparing implementation strategies ( n  = 22), (3) incentivizing workforce participation ( n  = 11), and (4) prioritizing interventions ( n  = 10). Stakeholders included providers ( n  = 27), patients ( n  = 25), caregivers ( n  = 5), and administrators ( n  = 2). The remaining studies ( n  = 16) engaged multiple stakeholders (i.e., combination of patients, caregivers, providers, and/or administrators). The following implementation outcomes were discussed: acceptability ( n  = 75), appropriateness ( n  = 34), adoption ( n  = 19), feasibility ( n  = 16), and fidelity ( n  = 3).

Conclusions

The number of DCE studies engaging stakeholders as an implementation strategy has been increasing over the past decade. As DCEs are more widely used as a healthcare assessment tool, there is a wide range of applications for them in stakeholder engagement. The DCE approach could serve as a tool for engaging stakeholders in implementation science.

Peer Review reports

Implementation science promotes methods to integrate scientific evidence into healthcare practice and policy. Traditionally, it has taken 15–20 years for academic research to translate into evidence-based program and policies, and implementation science is focused on narrowing time for translation of knowledge into practice [ 1 ]. One major component of implementation science is stakeholder engagement [ 2 ]. Successful implementation of healthcare interventions relies on stakeholder engagement at every stage, ranging from assessing and improving the acceptability of innovations to the sustainability of implemented interventions. In order to optimize the implementation of healthcare interventions, researchers, administrators, and policymakers must weigh the benefits and costs of complex multidimensional arrays of healthcare policies, strategies, and treatments.

As the field of implementation science matures, conceptualizing and measuring implementation outcomes becomes inevitable, particularly as it relates to the context of understanding the demand for evidence-based programs [ 3 , 4 ]. One strategy for systematically evaluating implementation outcomes involves the assessment of patient health preferences. As a multidisciplinary field, implementation science should leverage health economics tools that assess alternative implementation strategies and communicate the preferences of relevant stakeholders around the characteristics of healthcare programs and interventions.

One dynamic tool for appraising choices in health-related settings is the discrete choice experiment (DCE), which elicits preferences from individual decision makers over alternative scenarios, goods, or services. Each alternative is characterized by several attributes; and the choices subsequently determine how preferences are influenced by each attribute, as well as their relative importance. Health economists increasingly rely on DCEs (also referred to as conjoint analysis) [ 5 ] to elicit preferences for healthcare products and programs, which then can be used in outcome measurement for economic evaluation [ 6 ]. Despite their utility in improving our understanding of health-related choices, the extent to which DCEs have been applied in implementation research is unknown. In this paper, we explore and document potential applications of DCEs and how these applications can contribute to the field of implementation science by enhancing stakeholder engagement.

There is limited guidance on how to tailor implementation strategies in order to address the contextual needs of change efforts in health-related settings [ 7 ]. A recent study identified four methods to improve the selection and tailoring of implementation strategies: (1) concept mapping (i.e., visual mapping using mixed methods); (2) group model building (i.e., causal loop diagrams of complex problems); (3) intervention mapping (i.e., systematic multi-step development of interventions); and (4) DCEs [ 7 ]. Although all four methods could be used to match implementation strategies to recognize barriers and facilitators for a particular evidence-based practice or process change being implemented in a given setting, DCEs were identified as having the clear advantages of (1) providing a clear stepwise method for selecting and tailoring strategies to unique settings, while (2) guiding stakeholders to consider attributes of strategies at a granular level, enhancing the precision with which strategies are tailored to context.

Discrete choice experiments are a commonly used technique to address a range of important healthcare questions. DCEs constitute an attribute-based measure of benefit, with the assumptions that first, healthcare interventions, services or policies can be described by their attributes or characteristics and second, the levels of these attributes drive an individual’s valuation of the healthcare good. Within a DCE, respondents are asked to choose between two or more alternatives. The resulting choices reveal an underlying utility function (i.e., an economic measure of preferences over a given set of goods or services). The approach combines econometric analysis with experimental design theory, consumer theory, and random utility theory, which posits that consumers generally choose what they prefer, and where they do not, this can be explained by random factors [ 6 , 8 , 9 ]. Meanwhile, conjoint analysis originated in psychology to address the mathematical representation of the behavior of rankings observed as an outcome of systematic, factorial manipulation of multiple measures. Although there is a distinction between conjoint analysis and DCE, the two terms are used interchangeability by many researchers [ 5 ].

Advancing methods to capture stakeholder perspectives is essential for implementation science [ 10 ], and consequently, research is needed to document choice experiment methods for assessing the feasibility, acceptability, and validity of stakeholder perspectives. Whereas the use of DCEs in healthcare settings is well documented, there is a knowledge gap in characterizing whether DCE methodology is being applied to improve stakeholder engagement in implementation science. Therefore, the aim of this systematic review was to provide a synthesis of the use of DCEs as a stakeholder engagement tool. Specific objectives were to (1) identify published studies using DCEs in stakeholder engagement; (2) categorize these studies by application subtype, stakeholder group, and healthcare setting; and (3) provide recommendations for future use of DCEs in implementation science.

Identification of eligible publications

To be included, studies must have reported on original research using the DCE methodology and include a discussion of at least one implementation outcome. Studies must also have been available in English and occurred in a health-related setting. Duplicate abstracts were excluded from the review, as were abstracts describing reviews, editorials, commentaries, protocols, conference abstracts, and dissertations.

Search strategy

A search of MEDLINE, EconLit, PsycINFO, and CINAHL databases was conducted using the following search terms: (“discrete choice” OR “discrete rank” OR “conjoint analysis”) AND (implement*). These four databases were selected as they index journals from the fields of implementation science and include applications of DCEs across a range of health-related contexts or environments including the following: healthcare practice (e.g., clinical, public health, community-based health settings), health policy (e.g., interactions with health decision-makers at local, regional, provincial/state, federal, or international levels), health education (e.g., interactions with health educators in clinical or academic settings), and healthcare administration (e.g., interactions with health system organizations). The keyword search terms were repeated for all four databases. Keyword searches were limited to the English language, covering all published work available up to December 2016 (Additional file  1 ).

Coding and data synthesis

Retrieved abstracts were initially assessed against the eligibility criteria by one reviewer (RS) and rejected if the reviewer determined from the title and abstract that the study did not meet inclusion criteria. Full-text copies of the remaining publications were retrieved and further assessed against eligibility criteria to confirm or refute inclusion. Studies meeting the eligibility criteria were then coded by two reviewers. Disagreements were resolved by consensus or by a third reviewer. For all included studies, we recorded the mode of administration (i.e., electronic, paper-based, or via telephone), whether ethics board approval was obtained, the study sponsor, the incentives provided to participants, and the average duration of surveys. Included studies were categorized as follows:

Application type: A formative process was used to identify categories of applications used in the studies that met inclusion criteria. All studies were classified according to one of four application types, as follows: (1) characterizing demand for therapies and treatment technologies; (2) comparing implementation strategies; (3) incentivizing workforce participation; and (4) prioritizing interventions. Studies were further coded based on whether implementation science was a primary focus in the research vs. studies that casually discuss one or more implementation outcome.

Implementation outcome and stage: All studies were classified based on one more implementation outcomes discussed in the paper. The implementation outcomes were derived from Proctor’s Conceptual Framework for Implementation Outcomes [ 3 , 4 ] and include acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability. We also assessed implementation stage—whether early, mid, or late.

Stakeholder: All studies were classified according to the stakeholder(s) involved. These included the patient, stakeholder, provider (including physician, nurse, community health worker, and health educator), and administrator (including health system leader, information technology administrator, and policy maker). Sample size (i.e., number of participants in the DCE) was also recorded.

Setting: Studies were further classified based on the healthcare setting where the research was conducted, as either primary care (including community-based settings), specialty care, or research that involved the broader health system (including research related to health information technology). Studies were also classified based on the country or countries where the research was conducted. Countries were then categorized as either “high income” or “low and middle income” according to the World Bank income classification [ 11 ].

An electronic search yielded a total of 284 titles and abstracts which were judged to be potentially relevant based on title and abstract reading. Of these, 69 records were excluded for being duplicates. Full texts of the remaining 215 articles were reviewed. We finally selected 75 studies that met our inclusion criteria and excluded 140 studies. A flow chart through the different steps of study selection is provided in Fig.  1 .

PRISMA flow diagram of study selection

Excluded studies

A total of 140 studies were excluded. Of these, 47 were not conducted in health-related settings, 38 did not discuss any implementation outcomes, 23 were either commentaries or systematic reviews, 13 were methodological studies without empirical applications, 12 were study protocols, and 7 did not use the DCE methodology. A table with references and reasons for exclusion can be found in Additional file  2 .

Summary of included publications

Of the 75 included studies, 38 were administered as paper-based surveys, 23 were administered electronically, 5 were available in both paper and electronic formats, and 3 were administered via telephone. Administration mode was missing for 6 studies. Overall, 57 studies received institutional review board approval and 38 were exempted.

In terms of sponsorship, 37 studies were supported with government funding, 17 received funding from non-profit organizations, 3 were funded by healthcare delivery systems, and 2 had industry funding. No funding source was listed for the remaining 16 studies. In addition, only 10 studies reported the distribution of financial incentives to participants. The incentives ranged from US$1 or equivalent to US$25 (mean = US$12). Only 7 studies reported the average time it took participants to complete the survey (range 15–30 min).

Summary of publications over time

The earliest DCE study addressing stakeholder engagement in our systematic review was published in 2005. The annual number of publications has steadily increased over the past decade to reach 18 articles in 2016 (Fig.  2 ).

Number of studies, by year: 2005–2016

Summary of publications by country

Figure  3 shows the distribution of publications by country. Canada had the largest number of studies that met our inclusion criteria ( n  = 13), followed by the UK ( n  = 11), the Netherlands ( n  = 10), the USA ( n  = 6), Australia ( n  = 4), and South Africa ( n  = 4). Overall, 56 studies were conducted in high-income countries and 19 studies were from low- and middle-income countries (results now shown).

Number of studies, by country

Summary of publications by healthcare setting

Table  1 shows the number of studies by application type and healthcare setting. The majority of included studies were conducted in the primary care setting ( n  = 46), followed by specialty care ( n  = 22), and across the broader healthcare system ( n  = 7). Primary care studies were distributed as follows, according to application type: characterizing demand for therapies and treatment technologies ( n  = 21); incentivizing workforce participation ( n  = 11); comparing implementations strategies ( n  = 8); and prioritizing interventions ( n  = 4). The most common application in specialty care was comparing implementation strategies ( n  = 12) followed by characterizing demand ( n  = 10). The majority of health system studies focused on prioritizing interventions ( n  = 4), followed by comparing implementation strategies ( n  = 2), and characterizing demand ( n  = 1).

Summary of publications by stakeholder

Table  2 shows the number of studies by stakeholder type and healthcare setting. A total of 59 studies involved one stakeholder group, distributed as follows: provider ( n  = 27; mean sample size = 408), patient ( n  = 25; mean sample size = 717), caregiver ( n  = 5; mean sample size = 408), and administrator ( n  = 2; mean sample size = 60). The remainder ( n  = 16) involved multiple stakeholders. These were distributed as follows: patient and provider ( n  = 9; mean sample size = 740); provider and administrator ( n  = 3; mean sample size = 532); patient and caregiver ( n  = 2; mean sample size = 492); patient, caregiver, and provider ( n  = 2; mean sample size = 393); and patient, caregiver, provider, and administrator ( n  = 1; samples size = 102). Only 7 of the 46 studies (15%) in primary care involved more than one stakeholder; whereas 9 of 22 studies (41%) in specialty care had multiple stakeholders.

Summary of publications by implementation outcome

Figure  4 shows the documentation of implementation outcomes across the included studies. All included studies were conducted prior to implementation and focused on outcomes associated with early phases of implementation [ 4 ]. All 75 publications discussed acceptability. Other outcomes that were discussed include appropriateness ( n  = 34), adoption ( n  = 19), feasibility ( n  = 16), and fidelity ( n  = 3). Outcomes associated with later phases of implementation (i.e., implementation cost, penetration, and sustainability) were not discussed.

Number of studies, by implementation outcome

Summary of publications by application type

In terms of application type (Fig.  5 ), 32 studies were classified as characterizing demand for therapies and treatment technologies [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 ] (16 of 32 [50%] had a primary focus on implementation) [ 13 , 15 , 17 , 18 , 19 , 21 , 22 , 25 , 26 , 29 , 33 , 34 , 35 , 36 , 37 , 43 ]; 22 studies compared implementation strategies [ 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 ] (22 of 22 [100%] had a primary focus on implementation) [ 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 ]; 11 studies were concerned with incentivizing workforce participation [ 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 ] (6 of 11 [55%] had a primary focus on implementation) [ 66 , 68 , 71 , 74 , 75 , 76 ]; and 10 studies involved prioritizing health-related interventions [ 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 ] (4 of 10 [40%] had a primary focus on implementation) [ 80 , 82 , 83 , 85 ]. Overall, 48 of the 75 studies (64%) had a primary focus on implementation. The following paragraphs summarize findings by application type:

Number of studies, by application type

Application 1: characterizing demand for therapies and treatment technologies

Characterizing demand was the most common application among studies included in the current systematic review ( n  = 32) [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 ]. In these studies, decision makers used DCEs in predicting demand for new innovations in healthcare products and services prior to implementation. Because DCEs rely on hypothetical (but realistic) scenarios, they have been used to model the demand for treatment options before they become available to healthcare consumers. Advances in medical technology stipulate that patients and their caregivers choose among alternative scenarios (i.e., traditional therapy vs. new innovation). Forecasting demand for new healthcare technologies has been of great interest to various stakeholders, including public and private payers, healthcare systems, and various health programs and implementing agencies. These studies were overwhelmingly focused on exploring acceptability and appropriateness of health-related product or service, and all 32 of them included the perspectives of either patients or caregivers [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 ].

Application 2: comparing implementation strategies

Although the bulk of health-related DCEs examine healthcare preferences and resource allocation, DCEs have also been used in producing decision-making information to guide organizational strategies for implementation of evidence-based practices. Of the 22 studies comparing implementation strategies that were included in the systematic review [ 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 ], 13 examined the perspective of the provider only [ 44 , 48 , 49 , 51 , 52 , 53 , 54 , 57 , 58 , 59 , 62 , 63 ], 2 focused only on the patient perspective [ 56 , 60 ], and 7 examined the perspectives of multiple stakeholders [ 46 , 47 , 50 , 55 , 61 , 64 , 65 ]. Furthermore, implementation of patient-centered healthcare provision and the integration of patient priorities into healthcare decision-making require methods for measuring their preferences with respect to health and process outcomes. Therefore, DCEs can be used within the implementation process as tools that elicit stakeholder feedback to ensure the adoption of effective implementation strategies.

Application 3: incentivizing workforce participation

Our review found 11 publications that examined the question of incentivizing workforce participation [ 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 ]. Healthcare providers, including healthcare organizations and the health professionals employed within them, represent key stakeholders in the implementation and delivery of effective interventions. Providers play leading roles in activities that are essential to implementation, including training, supervision, quality assurance, and improvement. All 11 publications in this category took the provider perspective and were conducted in the primary care setting. The range of topics in these studies covered strategies to incentivize community health personnel in low resource settings within low-income countries [ 66 , 67 , 68 , 70 , 71 , 74 , 76 ] and primary care providers in rural settings within high-income countries [ 69 , 72 , 73 , 75 ]. These studies were mainly concerned with investigating the acceptability and appropriateness of their proposed solutions.

Application 4: prioritizing delivery of evidence-based interventions

The systematic review found 10 studies that used DCEs to inform the prioritization of health-related interventions [ 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 ]. Policy makers have long used economic tools, such as cost-effectiveness analysis, to prioritize healthcare service delivery [ 87 ]. However, prioritizing healthcare services on the basis of cost-effectiveness alone overlooks other important factors. Among the 10 studies in this category, 4 were conducted at the health system level [ 79 , 82 , 83 , 84 ] and 6 were in the primary care setting [ 77 , 78 , 80 , 81 , 85 , 86 ]. In terms of stakeholder engagement, 5 studies involved providers [ 77 , 78 , 80 , 85 , 86 ], 4 involved administrators [ 77 , 78 , 79 , 84 ], and 3 involved patients [ 81 , 82 , 83 ]. This category encompassed a wide range of implementation science topics, including the examination of strategies for approving new medicines in Wales [ 79 ], strategies for improving treatment of acute respiratory infections in the USA [ 86 ], and priority setting for HIV/AIDS interventions in Thailand [ 78 ].

This systematic review identified and synthesized the literature on the use of DCEs to enhance stakeholder engagement as a strategy to improve implementation. Findings suggest that the use of DCE methodology in implementation science has been scarce but growing steadily over the past decade. The current review documented research studies investigating multiple applications of DCEs, namely characterizing demand for therapies and treatment technologies, comparing implementation strategies, incentivizing workforce participation, and prioritizing interventions. The studies were conducted across diverse primary care and specialty care settings and involved several stakeholder groups, including patients, caregivers, providers, and administrators. All studies included in this systematic review were conducted pre-implementation and therefore focused on the investigation of early-stage implementation outcomes (e.g., acceptability and appropriateness).

The systematic review included studies that engaged various stakeholders, including patients, caregivers, providers, and administrators. Successful implementation of evidence-based strategies and programs depend largely on the fit of the interventions with the values and priorities of stakeholders who are shaping and participating in healthcare service delivery and consumption [ 1 ]. For example, healthcare recipients and their family members contribute a wide range of perspectives to the evaluation of healthcare services [ 88 ], underscoring the importance of systematically assessing their perspectives with respect to evidence-based alternatives. Choosing which evidence-based programs to implement and how to implement them are key decision points for health systems.

Moreover, the preferences of healthcare providers, administrators, and payers within the context of stakeholder engagement inevitably impact the priority attached to healthcare decisions. Effective implementation efforts focusing on individual providers require changes in professional norms and changes in individual providers’ knowledge and beliefs, economic incentives, and other factors [ 89 , 90 ]. Both financial and non-financial job characteristics can influence the recruitment and retention, as well as the attitudes and perceptions of healthcare professionals toward emerging evidence and innovations. Therefore, understanding the preferences of individual providers can improve the effectiveness of such efforts. However, existing data using revealed preferences are limited in their ability to address provider-level characteristics, and DCEs can be used to better inform this issue [ 91 , 92 ].

Preference measurement approaches, such as DCEs, are effective instruments for understanding stakeholders’ decision-making. DCEs have been used to engage patients prior to the implementation of cancer screening and tobacco cessation programs [ 93 , 94 ]. In such studies, researchers were able to gain valuable information about the demand for healthcare services prior to their provision and implementation. Although the choices presented to participants are hypothetical and the responses to them are potentially different from actual behavior, this hypothetical nature has its advantages over actually exposing the participant to the condition, with the researcher having complete control over the experimental design. Combined with advanced statistical techniques, the ability to model hypothetical conditions within the experimental design of DCEs ensures statistical robustness [ 95 ]. DCEs also allow the inclusion of attribute levels that do not yet exist, and are ideal for pre-intervention testing. Accordingly, marketing professionals have widely used DCEs in new product development, pricing, market segmentation, and advertising processes [ 96 ].

The aforementioned features of DCE studies can be useful in the design of interventions because they can enhance concordance with stakeholder preferences prior to, and during their implementation. The process of integrating research findings into population-level behaviors occurs in context [ 97 ]. Context in many healthcare systems includes scarce resources, variability in adoption of existing innovations, and ways of changing behavior that often incur their own costs but are rarely factored into the final estimate of the cost-effectiveness of innovation adoption [ 98 , 99 ]. DCEs can integrate the assessment of contextual factors, including cost, in the implementation of evidence-based prevention programs.

As the DCE becomes more widely used in healthcare preference assessment, the potential arises for a broad range of applications in implementation science. This systematic review sheds light on the current applications that have been documented in the peer-reviewed literature to date. Implementation science and DCEs are both rapidly emerging concepts in health services research. DCEs are becoming more accepted as an evaluation tool in healthcare while implementation science is now a growing scientific field with funding announcements, annual conferences, training programs, and a growing portfolio of studies globally [ 100 ]. Nevertheless, the two areas seldom cross paths. As implementation science advances, there is an opportunity for the field to harness the power of DCEs as a widely accepted tool for engaging stakeholders. The ability of DCEs to present and evaluate attributes and strategies prior to implementation, and their robustness in simultaneously examining these criteria within a decision framework can greatly enhance their value for implementation science.

Strengths and limitations

To our knowledge, our study is the first to highlight the use of DCEs as a stakeholder engagement strategy to improve implementation. Our study has several strengths, including its explicit and transparent methodology. We conducted a systematic and comprehensive search of the peer-reviewed literature across the relevant databases that resulted in a comprehensive representation of the published research in this area. Further, articles included in this systematic review were categorized into different application types and further classified using the Implementation Outcomes framework to shed light on practical applications for DCEs in implementation science. The identification of these application types and linking them with an implementation science framework provides strong guidance for future studies in this area.

However, our results should be considered in light of several limitations. First, gray literature such as reports, policy documents, and dissertations were not included in the review, nor were protocol papers. Although such reports may be relevant to the topic of interest, gray literature is not peer-reviewed and therefore may not rise to the high standards of quality associated with peer-reviewed publications. Inclusion of gray literature would also have biased the results given that papers related to work known by the authors and their network would have been more likely to have been identified than other works. Second, there are limitations to using volume of research output as a measure of research effort. Due to publication bias, studies with unfavorable results may not be published, leading to under-representation of the actual volume of work carried out in the field. Finally, it is unclear if DCE use effectively influenced implementation strategies and subsequent outcomes, due to the lack of follow-up data in these studies. Whether stakeholder DCEs direct implementation activities to the best approach or outcome remains to be demonstrated in future studies.

DCEs offer an opportunity to address an underrepresented challenge in implementation science—that of the “demand” side. By bringing key stakeholders to the forefront, we can not only focus on the push of scientific innovations but also understand how best they may be desired, demanded, and valued by patients, families, providers, and administrators. Understanding these dimensions will help us improve how to implement evidence-based interventions and programs in such ways that they will be effectively taken up and the gap in translation of evidence to practice and policy will be shortened.

Abbreviations

Discrete choice experiment

Brownson RC, Colditz GA, Proctor EK. Dissemination and implementation research in health: translating science to practice; 2012. doi: 10.1093/acprof:oso/9780199751877.001.0001.

Book   Google Scholar  

Lobb R, Colditz GA. Implementation science and its application to population health. Annu Rev Public Health. 2013;34:235–51. doi: 10.1146/annurev-publhealth-031912-114444;10.1146/annurev-publhealth-031912-114444.

Article   PubMed   PubMed Central   Google Scholar  

Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Heal Ment Heal Serv Res. 2009;36(1):24–34. doi: 10.1007/s10488-008-0197-4 .

Article   Google Scholar  

Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Heal Ment Heal Serv Res. 2011;38(2):65–76. doi: 10.1007/s10488-010-0319-7 .

Louviere JJ, Flynn TN, Carson RT. Discrete choice experiments are not conjoint analysis. J Choice Model. 2010;3(3):57–72. doi: 10.1016/S1755-5345(13)70014-9 .

Lancsar E, Louviere J. Conducting discrete choice experiments to inform healthcare decision making: a user’s guide. PharmacoEconomics. 2008;26(8):661–77. doi: 10.2165/00019053-200826080-00004.

Article   PubMed   Google Scholar  

Powell BJ, Beidas RS, Lewis CC, et al. Methods to improve the selection and tailoring of implementation strategies. J Behav Health Serv Res. 2015;44(2):177-94.

Louviere JJ, Pihlens D, Carson R. Design of discrete choice experiments: a discussion of issues that matter in future applied research. J Choice Model. 2011;4(1):1–8. doi: 10.1016/S1755-5345(13)70016-2 .

Lancaster KJ. A new approach to consumer theory. J Polit Econ. 1966;74(2):132–57.

Chambers DA. Advancing the science of implementation: a workshop summary. Adm Policy Ment Heal Ment Heal Serv Res. 2008;35(1–2):3–10. doi: 10.1007/s10488-007-0146-7 .

World Bank Data Team. World Bank country and lending groups–World Bank data help desk. World Bank. 2017:1–8. https://datahelpdesk.worldbank.org/knowledgebase/articles/906519 . Accessed 15 Oct 2017.

Marshall DA, Johnson FR, Phillips KA, Marshall JK, Thabane L, Kulin NA. Measuring patient preferences for colorectal cancer screening using a choice-format survey. Value Heal. 2007;10(5):415–30. doi: 10.1111/j.1524-4733.2007.00196.x .

Fitzpatrick E, Coyle DE, Durieux-Smith A, Graham ID, Angus DE, Gaboury I. Parents’ preferences for services for children with hearing loss: a conjoint analysis study. Ear Hear. 2007;28(6):842–9. doi: 10.1097/AUD.0b013e318157676d .

Bridges JF, Searle SC, Selck FW, Martinson NA. Engaging families in the choice of social marketing strategies for male circumcision services in Johannesburg, South Africa. Soc Mar Q. 2010;16:60–76. doi: 10.1080/15245004.2010.500443 .

Eisingerich AB, Wheelock A, Gomez GB, Garnett GP, Dybul MR, Piot PK. Attitudes and acceptance of oral and parenteral HIV preexposure prophylaxis among potential user groups: a multinational study. PLoS One. 2012;7(1) doi: 10.1371/journal.pone.0028238 .

Naik-Panvelkar P, Armour C, Rose JM, Saini B. Patients’ value of asthma services in Australian pharmacies: the way ahead for asthma care. J Asthma. 2012;49(3):310–6. doi: 10.3109/02770903.2012.658130 .

Naik-Panvelkar P, Armour C, Rose JM, Saini B. Patient preferences for community pharmacy asthma services: a discrete choice experiment. PharmacoEconomics. 2012;30(10):961–76. doi: 10.2165/11594350-000000000-00000 .

Benning TM, Kimman ML, Dirksen CD, Boersma LJ, Dellaert BGC. Combining individual-level discrete choice experiment estimates and costs to inform health care management decisions about customized care: the case of follow-up strategies after breast cancer treatment. Value Heal. 2012;15(5):680–9. doi: 10.1016/j.jval.2012.04.007 .

Hill M, Fisher J, Chitty LS, Morris S. Women’s and health professionals’ preferences for prenatal tests for down syndrome: a discrete choice experiment to contrast noninvasive prenatal diagnosis with current invasive tests. Genet Med. 2012;14(11):905–13. https://doi.org/10.1038/gim.2012.68 .

Rennie L, Porteous T, Ryan M. Preferences for managing symptoms of differing severity: a discrete choice experiment. Value Heal. 2012;15(8):1069–76. doi: 10.1016/j.jval.2012.06.013 .

Wheelock A, Eisingerich AB, Ananworanich J, et al. Are Thai MSM willing to take PrEP for HIV prevention? An analysis of attitudes, preferences and acceptance. PLoS One. 2013;8(1) doi: 10.1371/journal.pone.0054288 .

Burton CR, Fargher E, Plumpton C, Roberts GW, Owen H, Roberts E. Investigating preferences for support with life after stroke: a discrete choice experiment. BMC Health Serv Res. 2014;14(1):63. doi: 10.1186/1472-6963-14-63.

Baxter J-AB, Roth DE, Al Mahmud A, Ahmed T, Islam M, Zlotkin SH. Tablets are preferred and more acceptable than powdered prenatal calcium supplements among pregnant women in Dhaka, Bangladesh. J Nutr. 2014;144(7):1106–12. doi: 10.3945/jn.113.188524 .

Article   CAS   PubMed   Google Scholar  

Pechey R, Burge P, Mentzakis E, Suhrcke M, Marteau TM. Public acceptability of population-level interventions to reduce alcohol consumption: a discrete choice experiment. Soc Sci Med. 2014;113:104–9. doi: 10.1016/j.socscimed.2014.05.010 .

Deal K, Keshavjee K, Troyan S, Kyba R, Holbrook AM. Physician and patient willingness to pay for electronic cardiovascular disease management. Int J Med Inform. 2014;83(7):517–28. doi: 10.1016/j.ijmedinf.2014.04.007 .

Veldwijk J, Lambooij MS, Bruijning-Verhagen PCJ, Smit HA, de Wit GA. Parental preferences for rotavirus vaccination in young children: a discrete choice experiment. Vaccine. 2014;32(47):6277–83. doi: 10.1016/j.vaccine.2014.09.004 .

Fraenkel L, Cunningham M, Peters E. Subjective numeracy and preference to stay with the status quo. Med Decis Mak. 2015;35(1):6–11. doi: 10.1177/0272989X14532531 .

Hollin IL, Peay HL, Bridges JFP. Caregiver preferences for emerging Duchenne muscular dystrophy treatments: a comparison of best-worst scaling and conjoint analysis. Patient. 2014;8(1):19–27. https://doi.org/10.1007/s40271-014-0104-x .

Zickafoose JS, DeCamp LR, Prosser LA, et al. Parents’ preferences for enhanced access in the pediatric medical home. JAMA Pediatr. 2015;169(4):358. doi: 10.1001/jamapediatrics.2014.3534 .

Agyei-Baffour P, Boahemaa MY, Addy EA. Contraceptive preferences and use among auto artisanal workers in the informal sector of Kumasi, Ghana: a discrete choice experiment. Reprod Health. 2015;12(1):32. doi: 10.1186/s12978-015-0022-y.

Powell G, Holmes EAF, Plumpton CO, et al. Pharmacogenetic testing prior to carbamazepine treatment of epilepsy: patients’ and physicians’ preferences for testing and service delivery. Br J Clin Pharmacol. 2015;80(5):1149–59. doi: 10.1111/bcp.12715 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Spinks J, Janda M, Soyer HP, Whitty J a. Consumer preferences for teledermoscopy screening to detect melanoma early. J Telemed Telecare. 2015;0(0):1–8. doi: 10.1177/1357633X15586701.

Google Scholar  

Becker MPE, Christensen BK, Cunningham CE, et al. Preferences for early intervention mental health services: a discrete-choice conjoint experiment. Psychiatr Serv. 2015;67:appips201400306. doi: 10.1176/appi.ps.201400306 .

Tang EC, Galea JT, Kinsler JJ, et al. Using conjoint analysis to determine the impact of product and user characteristics on acceptability of rectal microbicides for HIV prevention among Peruvian men who have sex with men. Sex Transm Infect. 2016;92(3):200–5. doi: 10.1136/sextrans-2015-052028 .

Hill M, Johnson J-A, Langlois S, et al. Preferences for prenatal tests for down syndrome: an international comparison of the views of pregnant women and health professionals. Eur J Hum Genet. 2015;44:968–75. doi: 10.1038/ejhg.2015.249 .

Lock J, de Bekker-Grob EW, Urhan G, et al. Facilitating the implementation of pharmacokinetic-guided dosing of prophylaxis in haemophilia care by discrete choice experiment. Haemophilia. 2016;22(1):e1–e10. doi: 10.1111/hae.12851.

Herman PM, Ingram M, Cunningham CE, et al. A comparison of methods for capturing patient preferences for delivery of mental health services to low-income Hispanics engaged in primary care. Patient. 2016;9(4):293–301. doi: 10.1007/s40271-015-0155-7 .

Morel T, Ayme S, Cassiman D, Simoens S, Morgan M, Vandebroek M. Quantifying benefit-risk preferences for new medicines in rare disease patients and caregivers. Orphanet J Rare Dis. 2016;11(1):70. doi: 10.1186/s13023-016-0444-9.

Johnson DC, Mueller DE, Deal AM, et al. Integrating patient preference into treatment decisions for men with prostate cancer at the point of care. J Urol. 2016;196(6):1640–4. doi: 10.1016/j.juro.2016.06.082 .

Uemura H, Matsubara N, Kimura G, et al. Patient preferences for treatment of castration-resistant prostate cancer in Japan: a discrete-choice experiment. BMC Urol. 2016;16(1):63. doi: 10.1186/s12894-016-0182-2.

Determann D, Lambooij MS, Gyrd-Hansen D, et al. Personal health records in the Netherlands: potential user preferences quantified by a discrete choice experiment. J Am Med Inform Assoc. 2016;0(0):ocw158. doi: 10.1093/jamia/ocw158 .

Harrison M, Marra CA, Bansback N. Preferences for “new” treatments diminish in the face of ambiguity. Health Econ (United Kingdom). 2016;26(6):743-52.

Bridges JFP, Searle SC, Selck FW, Martinson NA. Designing family-centered male circumcision services: a conjoint analysis approach. Patient. 2012;5(2):101–11. doi: 10.2165/11592970-000000000-00000 .

Huis in’t Veld MHA, van Til JA, Ijzerman MJ, MMR V-H. Preferences of general practitioners regarding an application running on a personal digital assistant in acute stroke care. J Telemed Telecare. 2005;11(Suppl 1):37–9. doi: 10.1258/1357633054461615 .

Berchi C, Dupuis JM, Launoy G. The reasons of general practitioners for promoting colorectal cancer mass screening in France. Eur J Health Econ. 2006;7(2):91–8. doi: 10.1007/s10198-006-0339-0 .

Oudhoff JP, Timmermans DRM, Knol DL, Bijnen AB, Van der Wal G. Prioritising patients on surgical waiting lists: a conjoint analysis study on the priority judgements of patients, surgeons, occupational physicians, and general practitioners. Soc Sci Med. 2007;64(9):1863–75. doi: 10.1016/j.socscimed.2007.01.002 .

Goossens A, Bossuyt PMM, de Haan RJ. Physicians and nurses focus on different aspects of guidelines when deciding whether to adopt them: an application of conjoint analysis. Med Decis Mak. 2008;28(1):138–45. doi: 10.1177/0272989X07308749 .

Van Helvoort-Postulart D, Dellaert BGC, Van Der Weijden T, Von Meyenfeldt MF, Dirksen CD. Discrete choice experiments for complex health-care decisions: does hierarchical information integration offer a solution? Health Econ. 2009;18(8):903–20. doi: 10.1002/hec.1411 .

van Helvoort-Postulart D, van der Weijden T, Dellaert BGC, de Kok M, von Meyenfeldt MF, Dirksen CD. Investigating the complementary value of discrete choice experiments for the evaluation of barriers and facilitators in implementation research: a questionnaire survey. Implement Sci. 2009;4(1):10. doi: 10.1186/1748-5908-4-10 .

Davison SN, Kromm SK, Currie GR. Patient and health professional preferences for organ allocation and procurement, end-of-life care and organization of care for patients with chronic kidney disease using a discrete choice experiment. Nephrol Dial Transplant. 2010;25(7):2334–41. doi: 10.1093/ndt/gfq072 .

Hinoul P, Goossens A, Roovers JP. Factors determining the adoption of innovative needle suspension techniques with mesh to treat urogenital prolapse: a conjoint analysis study. Eur J Obstet Gynecol Reprod Biol. 2010;151(2):212–6. doi: 10.1016/j.ejogrb.2010.03.026 .

Grindrod KA, Marra CA, Tsuyuki RT, Lynd LD. Pharmacists ’ preferences for providing patient-centered services: a discrete choice experiment to guide. Health Policy. 2010;44:1554–64.

Wen K-Y, Gustafson DH, Hawkins RP, et al. Developing and validating a model to predict the success of an IHCS implementation: the readiness for implementation model. J Am Med Inform Assoc. 2010;17(6):707–13. doi: 10.1136/jamia.2010.005546 .

Lagarde M, Paintain LS, Antwi G, et al. Evaluating health workers’ potential resistance to new interventions: a role for discrete choice experiments. PLoS One. 2011;6(8) doi: 10.1371/journal.pone.0023588 .

Cunningham CE, Henderson J, Niccols A, et al. Preferences for evidence-based practice dissemination in addiction agencies serving women: a discrete-choice conjoint experiment. Addiction. 2012;107(8):1512–24. doi: 10.1111/j.1360-0443.2012.03832.x.

Philips H, Mahr D, Remmen R, Weverbergh M, De Graeve D, Van Royen P. Predicting the place of out-of-hours care-a market simulation based on discrete choice analysis. Health Policy (New York). 2012;106(3):284–90. https://doi.org/10.1016/j.healthpol.2012.04.010 .

Deuchert E, Kauer L, Meisen Zannol F. Would you train me with my mental illness? Evidence from a discrete choice experiment. J Ment Health Policy Econ. 2013;16(2):67–80.

PubMed   Google Scholar  

Cunningham CE, Barwick M, Short K, et al. Modeling the mental health practice change preferences of educators: a discrete-choice conjoint experiment. School Ment Health. 2014;6(1):1–14. doi: 10.1007/s12310-013-9110-8 .

Struik MHL, Koster F, Schuit AJ, Nugteren R, Veldwijk J, Lambooij MS. The preferences of users of electronic medical records in hospitals: quantifying the relative importance of barriers and facilitators of an innovation. Implement Sci. 2014;9(1):1–11. doi: 10.1186/1748-5908-9-69.

Dixon PR, Grant RC, Urbach DR. The impact of marketing language on patient preference for robot-assisted surgery. Surg Innov. 2015;22(1):15–9. doi: 10.1177/1553350614537562 .

Nicaise P, Soto VE, Dubois V, Lorant V. Users’ and health professionals’ values in relation to a psychiatric intervention: the case of psychiatric advance directives. Adm Policy Ment Heal Ment Heal Serv Res. 2014:384–93. doi: 10.1007/s10488-014-0580-2 .

Grudniewicz A, Bhattacharyya O, McKibbon KA, Straus SE. Redesigning printed educational materials for primary care physicians: design improvements increase usability. Implement Sci. 2015;10(1):156. doi: 10.1186/s13012-015-0339-5.

Bailey K, Cunningham C, Pemberton J, Rimas H, Morrison KM. Understanding academic clinicians’ decision making for the treatment of childhood obesity. Child Obes. 2015;11(6):696–706. doi: 10.1089/chi.2015.0031 .

Barrett AN, Advani HV, Chitty LS, et al. Evaluation of preferences of women and healthcare professionals in Singapore for implementation of noninvasive prenatal testing for down syndrome. Singap Med J. 2016:1–31. 10.11622/smedj.2016114 .

Jennifer Anne Whitty C, Whitty BPharm GradDipClinPharm JA, Spinks BPharm MPHJ, Bucknall TR, Tobiano RNGB, Chaboyer WR. Patient and nurse preferences for implementation of bedside handover: do they agree? Findings from a discrete choice experiment. Health Expect. 2016;1(9):1–9. doi: 10.1111/hex.12513 .

Lagarde M, Blaauw D, Cairns J. Cost-effectiveness analysis of human resources policy interventions to address the shortage of nurses in rural South Africa. Soc Sci Med. 2012;75(5):801–6. doi: 10.1016/j.socscimed.2012.05.005 .

Miranda JJ, Diez-Canseco F, Lema C, et al. Stated preferences of doctors for choosing a job in rural areas of Peru: a discrete choice experiment. PLoS One. 2012;7(12) doi: 10.1371/journal.pone.0050567 .

Huicho L, Miranda JJ, Diez-Canseco F, et al. Job preferences of nurses and midwives for taking up a rural job in Peru: a discrete choice experiment. PLoS One. 2012;7(12) doi: 10.1371/journal.pone.0050315 .

Li J, Scott A, McGrail M, Humphreys J, Witt J. Retaining rural doctors: doctors’ preferences for rural medical workforce incentives. Soc Sci Med. 2014;121:56–64. doi: 10.1016/j.socscimed.2014.09.053 .

Yaya Bocoum F, Koné E, Kouanda S, Yaméogo WME, Bado A. Which incentive package will retain regionalized health personnel in Burkina Faso: a discrete choice experiment. Hum Resour Health. 2014;12(Suppl 1):S7. doi: 10.1186/1478-4491-12-S1-S7 .

Song K, Scott A, Sivey P, Meng Q. Improving Chinese primary care providers’ recruitment and retention: a discrete choice experiment. Health Policy Plan. 2015;30(1):68–77. doi: 10.1093/heapol/czt098 .

Holte JH, Kjaer T, Abelsen B, Olsen JA. The impact of pecuniary and non-pecuniary incentives for attracting young doctors to rural general practice. Soc Sci Med. 2015;128:1–9. doi: 10.1016/j.socscimed.2014.12.022.

Kjaer NK, Halling A, Pedersen LB. General practitioners’ preferences for future continuous professional development: evidence from a Danish discrete choice experiment. Educ Prim Care. 2015;26(1):4–10. doi: 10.1080/14739879.2015.11494300 .

Kasteng F, Settumba S, Källander K, Vassall A. Valuing the work of unpaid community health workers and exploring the incentives to volunteering in rural Africa. Health Policy Plan. 2016;31(2):205–16. doi: 10.1093/heapol/czv042 .

Chen TT, Lai MS, Chung KP. Participating physician preferences regarding a pay-for-performance incentive design: a discrete choice experiment. Int J Qual Heal Care. 2016;28(1):40–6. doi: 10.1093/intqhc/mzv098 .

Shiratori S, Agyekum EO, Shibanuma A, et al. Motivation and incentive preferences of community health officers in Ghana : an economic behavioral experiment approach. Hum Resour Health. 2016:1–26. doi: 10.1186/s12960-016-0148-1.

Baltussen R, ten Asbroek AHA, Koolman X, Shrestha N, Bhattarai P, Niessen LW. Priority setting using multiple criteria: should a lung health programme be implemented in Nepal? Heal Policy Plan. 2007;22(3):178–85.

Article   CAS   Google Scholar  

Youngkong S, Baltussen R, Tantivess S, Koolman X, Teerawattananon Y. Criteria for priority setting of HIV/AIDS interventions in Thailand: a discrete choice experiment. BMC Health Serv Res. 2010;10(1):197–197 1p. doi: 10.1186/1472-6963-10-197 .

Linley WG, Hughes DA. Decision-makers’ preferences for approving new medicines in wales: a discrete-choice experiment with assessment of external validity. PharmacoEconomics. 2013;31(4):345–55. doi: 10.1007/s40273-013-0030-0 .

Farley K, Thompson C, Hanbury A, Chambers D. Exploring the feasibility of conjoint analysis as a tool for prioritizing innovations for implementation. Implement Sci. 2013;8:56. doi: 10.1186/1748-5908-8-56 .

Seghieri C, Mengoni A, Nuti S. Applying discrete choice modelling in a priority setting: an investigation of public preferences for primary care models. Eur J Health Econ. 2014;15(7):773–85. doi: 10.1007/s10198-013-0542-8 .

Erdem S, Thompson C. Prioritising health service innovation investments using public preferences: a discrete choice experiment. BMC Health Serv Res. 2014;14(1):360. doi: 10.1186/1472-6963-14-360.

Honda A, Ryan M, Van Niekerk R, McIntyre D. Improving the public health sector in South Africa: eliciting public preferences using a discrete choice experiment. Health Policy Plan. 2015;30(5):600–11. doi: 10.1093/heapol/czu038 .

Paolucci F, Mentzakis E, Defechereux T, Niessen LW. Equity and efficiency preferences of health policy makers in China-a stated preference analysis. Health Policy Plan. 2015;30(8):1059–66. doi: 10.1093/heapol/czu123 .

Ammi M, Peyron C. Heterogeneity in general practitioners’ preferences for quality improvement programs: a choice experiment and policy simulation in France. Health Econ Rev. 2016;6(1):44. doi: 10.1186/s13561-016-0121-7.

Gong CL, Hay JW, Meeker D, Doctor JN. Prescriber preferences for behavioural economics interventions to improve treatment of acute respiratory infections: a discrete choice experiment. BMJ Open. 2016;6(9):e012739. doi: 10.1136/bmjopen-2016-012739 .

Drummond MF, Sculpher MJ, Torrance GW, O’Brien BJ, Stoddart GL. Methods for the economic evaluation of health care programmes, vol. 3; 2005. http://econpapers.repec.org/RePEc:oxp:obooks:9780198529453 . Accessed 1 Dec 2016.

Aarons GA, Covert J, Skriner LC, et al. The eye of the beholder: youths and parents differ on what matters in mental health services. Adm Policy Ment Heal Ment Heal Serv Res. 2010;37(6):459–67. doi: 10.1007/s10488-010-0276-1 .

Aarons GA. Mental health provider attitudes toward adoption of evidence-based practice: the evidence-based practice attitude scale (EBPAS). Ment Health Serv Res. 2004;6(2):61–74. https://doi.org/10.1023/B:MHSR.0000024351.12294.65 .

Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8(22):20. doi: 10.1186/1748-5908-8-22.

Lagarde M, Blaauw D. A review of the application and contribution of discrete choice experiments to inform human resources policy interventions. Hum Resour Health. 2009;7:62. doi: 10.1186/1478-4491-7-62 .

Scott A. Eliciting GPs’ preferences for pecuniary and non-pecuniary job characteristics. J Health Econ. 2001;20(3):329–47. doi: 10.1016/S0167-6296(00)00083-7 .

Pignone MP, Crutchfield TM, Brown PM, et al. Using a discrete choice experiment to inform the design of programs to promote colon cancer screening for vulnerable populations in North Carolina. BMC Health Serv Res. 2014;14(1):611–29. doi: 10.1186/s12913-014-0611-4.

Salloum RG, Abbyad CW, Kohler RE, Kratka a K, Oh L, Wood K a. Assessing preferences for a university-based smoking cessation program in Lebanon: a discrete choice experiment. Nicotine Tob Res. 2015;17(5):580–5. doi: 10.1093/ntr/ntu188 .

De Bekker-Grob EW, Ryan M, Gerard K. Discrete choice experiments in health economics: a review of the literature. Health Econ. 2012;21(2):145–72. doi: 10.1002/hec.1697 .

Schlereth C, Eckert C, Schaaf R, Skiera B. Measurement of preferences with self-explicated approaches: a classification and merge of trade-off- and non-trade-off-based evaluation types. Eur J Oper Res. 2014;238(1):185–98. doi: 10.1016/j.ejor.2014.03.010 .

Tomoaia-Cotisel A, Scammon DL, Waitzman NJ, et al. Context matters: the experience of 14 research teams in systematically reporting contextual factors important for practice change. Ann Fam Med. 2013;11(SUPPL. 1) doi: 10.1370/afm.1549 .

Mason J, Freemantle N, Nazareth I, Eccles M, Haines a, Drummond M. When is it cost-effective to change the behavior of health professionals? JAMA. 2001;286(23):2988–92. doi: 10.1001/jama.286.23.2988 .

Sculpher M. Evaluating the cost-effectiveness of interventions designed to increase the utilization of evidence-based guidelines. Fam Pract. 2000;17(Suppl 1):S26–31. doi: 10.1093/fampra/17.suppl_1.S26.

Sampson UKA, Chambers D, Riley W, Glass RI, Engelgau MM, Mensah GA. Implementation research the fourth movement of the unfinished translation research symphony. Glob Heart. 2016;11(1):153–8. doi: 10.1016/j.gheart.2016.01.008 .

Download references

Acknowledgements

We are grateful to Laura Ramirez, Evan Johnson, and Antonio Laracuente for their contributions during the data synthesis phase of this study. We thank Dr. Angela Stover for her comments on a prior draft.

This research was supported in part by the Mentored Training for Dissemination and Implementation Research in Cancer (MT-DIRC) Program, National Cancer Institute grant (R25-CA171994). Dr. Chambers is a MT-DIRC faculty member. Dr. Salloum is a MT-DIRC fellow from 2016 to 2018. The content of this paper is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies. No financial disclosures were reported by the authors of this paper.

Availability of data and materials

Please contact the authors for data requests.

Author information

Authors and affiliations.

Department of Health Outcomes and Policy, College of Medicine, University of Florida, 2004 Mowry Road, Gainesville, FL, 32610, USA

Ramzi G. Salloum & Elizabeth A. Shenkman

Institute for Choice, School of Marketing, University of South Australia, Adelaide, SA, Australia

Jordan J. Louviere

Division of Cancer Control and Population Sciences, National Cancer Institute, Rockville, MD, USA

David A. Chambers

You can also search for this author in PubMed   Google Scholar

Contributions

All authors were involved in various stages of the study design. RS and DC conceptualized the study. RS wrote the first draft. JL, ES, and DC gave methodological advice and commented on subsequent drafts of the paper. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ramzi G. Salloum .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:.

List of included studies. (DOCX 40 kb)

Additional file 2:

List of excluded studies based on full text evaluation. (DOCX 62 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Salloum, R.G., Shenkman, E.A., Louviere, J.J. et al. Application of discrete choice experiments to enhance stakeholder engagement as a strategy for advancing implementation: a systematic review. Implementation Sci 12 , 140 (2017). https://doi.org/10.1186/s13012-017-0675-8

Download citation

Received : 17 May 2017

Accepted : 15 November 2017

Published : 23 November 2017

DOI : https://doi.org/10.1186/s13012-017-0675-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Discrete choice
• Conjoint analysis
• Preferences
• Stakeholder

Implementation Science

ISSN: 1748-5908

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Advertisement

Priority setting in the German healthcare system: results from a discrete choice experiment

• Research article
• Open access
• Published: 15 May 2023
• Volume 23 , pages 411–431, ( 2023 )

Cite this article

You have full access to this open access article

• V. Meusel 1 ,
• E. Mentzakis 2 ,
• P. Baji 3 ,
• G. Fiorentini 4 &
• F. Paolucci   ORCID: orcid.org/0000-0001-6173-5324 5 , 6  

Worldwide, social healthcare systems must face the challenges of a growing scarcity of resources and of its inevitable distributional effects. Explicit criteria are needed to define the boundaries of public reimbursement decisions. As Germany stands at the beginning of such a discussion, more formalised priority setting procedures seem in order. Recent research identified multi-criteria decision analysis (MCDA) as a promising approach to inform and to guide decision-making in healthcare systems. In that regard, this paper aims to analyse the relative weight assigned to various criteria in setting priority interventions in Germany. A discrete choice experiment (DCE) was employed in 2015 to elicit equity and efficiency preferences of 263 decision makers, through six attributes. The experiment allowed us to rate different policy interventions based on their features in a composite league table (CLT). As number of potential beneficiaries, severity of disease, individual health benefits and cost-effectiveness are the most relevant criteria for German decision makers within the sample population, the results display an overall higher preference towards efficiency criteria. Specific high priority interventions are mental disorders and cardiovascular diseases.

Similar content being viewed by others

Priority setting in health: development and application of a multi-criteria algorithm for the population of New Zealand’s Waikato region

Public engagement in setting healthcare priorities: a ranking exercise in Cyprus

Multi-criteria Decision Analysis Software in Healthcare Priority Setting: A Systematic Review

Avoid common mistakes on your manuscript.

Introduction

Priority setting in healthcare decision-making is inevitable. Limited health resources, growing health expenditures, combined with technological advances and an ageing population, continue to put pressure on healthcare systems (Fleck, 2001 ; Emanuel, 2000 ; Fuchs, 2010 , Borck et al., 2012 ). Policymakers are encouraged to consider priority setting in healthcare, albeit such rationing decisions may not always be based on transparent processes, but are ad-hoc (Baltussen & Niessen, 2006 ) or history-based (Kapiriri & Martin, 2007 ), possibly leading to a suboptimal use of resources.

Complex developments in healthcare are widening the gap between new technological possibilities and advances in therapy and diagnosis and the financial feasibility of the German Healthcare system (Borck, 2012 ; di Costanzo, 2020 ). In that regard, concepts such as rationalisation and prioritisation are intensely debated in the quest to implement an equitable allocation of resources (Schmitz-Luhn & Katzenmeier, 2016 ).

For Germany, the controversial debate on healthcare expenditures and limits on the availability of healthcare services offered by the system started relatively late compared to other countries (Sabik & Lie, 2008 ). Footnote 1 Germany’s health expenditures have steadily increased from 9.4 per cent of its Gross Domestic Product (GDP) in 1992–11.7% in 2019, placing Germany in 6th place in the world rank and on top of the EU27 (8,3%) (OECD, 2021 , Statistische Bundesamt, 2022 ). In international terms however, the German healthcare system stands out for a generous benefit package with high levels of capacity and relatively low cost-sharing (Beske & Drabinski, 2005 ). As in most high income countries, the future levels of expenditure on GDP, due to demographic and epidemiologic changes (e.g. increase in life expectancy, a diminishing mortality rate) (Institute for Health Metrics & Evaluation, 2016 ; World Bank, 2016 ) combined with the increasing costs of medicine are likely to be significantly higher. This has implications for the statutory health insurance benefits as the Federal Joint Committee (G-BA) decides over what adequate, appropriate, economic healthcare entails (Federal Joint Committee, 2017 ). However, rationalisation in terms of efficiency gains alone will hardly be enough to prevent a further divergence between the feasible and the financially viable.

The complexity of such decisions suggests the need for explicit criteria to be used (Alliance for Health Policy & Systems Research, 2004 ; Baltussen & Niessen, 2006 ; Chalkidou et al., 2016 ; Cromwell et al., 2015 ). In this respect, discrete choice experiments (DCEs) allow for the simultaneous assessment of multiple policy criteria and preferences elicitation of decision-makers when facing trade-off decisions (de Bekker-Grob et al., 2012 ; Hansen & Devlin, 2019 ; Lancsar & Louviere, 2008 ; Ryan & Gerard, 2003 ; Thokala et al., 2016 ).

Past studies support the feasibility and acceptability of DCEs in eliciting public preferences towards societal values and attribute-driven interventions (Genie et al., 2020 ; Green & Gerard, 2009 ; Krauth et al., 2021 ; Krinke et al., 2019 ; López-Bastida et al., 2019 ; Luyten et al., 2015 ). Empirical studies in Germany eliciting priority rankings for the treatments of determined groups, find treatments for children are ranked highest (Raspe & Stumpf, 2013 ) whereas Diederich ( 2011 ) finds little evidence that the German public accepts age as a priority criterion for healthcare services, although there is wide agreement to prioritise specific age groups.

A combination of efficiency and societal values tends to be predominantly considered in priority setting (Baeten et al., 2010 ; Kenny & Joffres, 2008 ). With the advent of numerous new initiatives in the health sector, decision-makers are expected to choose between competing healthcare interventions and explicitly consider equity and efficiency trade-off (Baltussen & Niessen, 2006 ). Transparent and informed decision-making in healthcare requires national level criteria are to be established (Mitton & Donaldson, 2004 ; Ottersen et al., 2016 ; WHO, 2007 ) encouraging a more open social and political discourse on the priorities in medical care to guide policy making on which healthcare technologies should be publicly financed at the different levels of the Social Health Insuracne (SHI) system (Diederich, ; Gerst, 2014 ; Norheim, 2016 ).

This paper presents the results of a DCE conducted in 2015, aiming to explore key stakeholders’ preferences for different features of healthcare policies and interventions in Germany, and to show how the latter are prioritised according to such preferences. The paper builds on past work (Defchereux et al., 2012 ; Mirelman et al., 2012 ; Mentzakis et al., 2014 ; Paolucci et al., 2015 ; Baji et al., 2016 ) and extends the pool of countries with available data for cross country comparisons.

Background on the German health care system

The main pillar of the German system is the Statutory Health Insurance which is inspired by strong solidarity principles providing the normative basis to pursue the objectives of equity and comprehensiveness, and which represents the framework of regulations on the provision and the financing of healthcare services (Oduncu, 2013 ). The SHI regulations aim for cost-containment and sustainable financing mechanisms, managed competition, as well as improved efficacy and quality (Blümel & Busse, 2017 ; Sauerland, 2001 ).

At the federal level, the Social Code Book V gives a foundation for entitlements, rights and duties of insureds covered by SHI. However, it does not lay down specific guidelines but, instead, sets a framework for policy interfaces (Dannecker, 2009 ). The scope of benefits is subject to negotiations between the latter bodies and the associations of payers and providers (Blümel et al., 2020 ). In an international comparison, the German benefit catalogue shows to be quite extensive, leading to reform efforts and the question about how far the solidarity should go. The Reform Act in 2004 showed early attempts of rationing certain benefits from the catalogue, for instance medications for the treatment of erectile dysfunction, hair loss or smoking cessation (Burkhardt, 2012 ).

Ever since its resumption, the various aspects of prioritisation have been discussed following different political strategies and institutional procedures (Friedrich et al., 2009 ; Groß et al., 2010 ; Heil et al., 2010 ; Müller & Groß, 2009 ; Oduncu, 2012 ; Peacock et al., 2006 ; Raspe & Meyer, 2009 ; Schöne-Seifert, 2006 , Heyers, 2016 , Petri, 2015 ). The Central Ethics Committee for Observance of Ethical Principles in Medicine (ZEKO) and the German Medical Association have promoted the concept of prioritisation (ZEKO, 2007 ; ZEKO, 2000 ; Bundesärztekammer, 2014 ; Borck et al., 2012 ; Raspe & Schulze, 2013 ; Diederich et al., 2011 ) with specific focus on its legal, ethical and economic aspects and have supported the use of pre-defined criteria to evaluate medical services and benefits (Marckmann, 2009 ; Gordijn & Have, 2013 ; Oduncu, 2012 ; Borck et al., 2012 ; Storz & Egger, 2010 ; Kliemt, 2006 ).

As such, the three major criteria of prioritisation ‘‘medical need’ ’ (severity and urgency), ‘‘proven benefit and fitness for purpose’ ’, and ‘‘cost–benefit-effectiveness’ ’ have been proposed by the central ethics committee (ZEKO, 2007 ).

Methodology

A discrete choice experiment (DCE) was employed to assess the relative weight of various criteria in setting priorities in the German health arena. DCE are commonly used in healthcare for prioritisation decisions (Lancsar & Louviere, 2008 ; Ryan & Gerard, 2003 ; Ryan et al., 2008 ) using public preferences. Health economists have acknowledged the benefit of the approach especially when facing health policy, planning and resource allocation decisions in high-income countries. In that regard DCEs are widely applied to a range of policy questions (Baji et al., 2016 ; de Bekker-Grob et al., 2012 ; Ryan & Gerard, 2003 ; Whitty et al., 2011 ) and priority setting frameworks (Baltussen et al., 2006 ; Peacock et al., 2010 ; Razavi et al., 2020 ). These include the elicitation of views on diagnosis, treatment and care (van de Schoot et al., 2017 , Koopmanschap et al., 2010 , Clark et al., 2017 , King et al., 2007 ; Kjaer & Gyrd-Hansen, 2008 ), access to services (Longo et al., 2006 ; Mengoni et al., 2013 ), consumer (health) preferences (Czoli et al., 2016 ) and the employment preferences of health personnel (Mandeville et al., 2014 ; Wordsworth et al., 2004 ).

Respondents’ preferences are elicited in a survey adapted from previous studies (Defechereux et al., 2012 ; Mirelman et al., 2012 ; Paolucci et al., 2015 ). Respondents’ were asked to choose among a set of hypothetical alternative interventions presented in choice sets, with each alternative described in terms of six criteria. To every criterion, values have been assigned over a range of pre-defined levels.

Experimental and instrument design

In the first stage, the decision-making context was defined and a set of key attributes was narrowed down accordingly. Here, a combination of relevant efficiency/equity-related factors was included, where efficiency is mainly referred to as the maximisation of health gains within society at lowest cost, including non-health outcomes. Equity criteria, on the other hand, are related to the distributional effects of interventions, aiming for the reduction of inequalities in health status or targeting disadvantaged groups (James et al., 2005 ).

An existing standardized questionnaire reported earlier for other countries (Baltussen et al., 2006 ; Koopmanschap et al., 2010 ; Mirelman et al., 2012 ) was used comprising a core set of preference criteria as attribute. Those have been identified based on literature reviews and were verified by national focus groups of health programmers and experts within the initial three settings. These were in Ghana, Nepal, as well as a working session with 28 leading HTA experts at the HTAi conference in 2008 (Baeten et al., 2010 ; Baltussen et al., 2007 ; Defechereux et al., 2012 ; Mirelman et al., 2012 ; Noorani et al., 2007 ; Paolucci et al., 2015 ; Tanios et al., 2013 ).

Overall, six attributes have been identified as comprising key criteria in health decision making for our DCE: one with three levels and five with two levels (Table 1 ). This set of criteria describes the most generic aspects of a health intervention. The chosen criteria were grouped under the equity (willingness to subsidise others, severity of disease, age of the target group) and efficiency (number of beneficiaries, cost-effectiveness, individual health benefits) categories. Further, the selected attributes have been consistent with those used in previous studies in which they proved to be important preference criteria for priority setting (Baltussen et al., 2010 ).

The full factorial design resulted in 96 possible combinations. To avoid cognitive burden and facilitate administration, a fractional factorial design was used with 16 forced-choice pair-wise choice sets, ensuring orthogonality (all attributes are orthogonal except for the three-level attribute age that exhibits correlations with the rest of the attribute but all smaller than 0.04), level balance and minimum overlap. For the experimental design Sawtooth Software was used.

Sample and data collection

The data collection for this study focused on expert stakeholders in the German healthcare sector. An online questionnaire was administered that entailed a detailed description of the survey purpose and guidance on how to interpret and handle the questionnaire. This was followed by 16 choice sets and socio-demographic questions about gender, age, profession, working institution and experience in the healthcare sector (Tables 2 , 3 ).

Out of 2153 individuals contacted, 263 complete and valid questionnaires were returned giving a response rate of about 12%. The sample targeted individuals involved in the macro-, meso- and micro-levels of healthcare decision-making in Germany and included healthcare academics, members of various legislative and political decision-making bodies accountable for strategy, implementation, funding and supervision, executives of national research and planning institutions, as well as leaders and senior staff members of individual healthcare providers.

Statistical analysis

Data from respondents who failed to answer all choice sets were dropped. The remaining observations were analysed through a mixed logit regression model (Hole, 2007 ). This modeling approach allows for multiple observations being obtained from individuals that do not exhibit the restrictive independence from irrelevant alternatives and account for correlations in unobserved heterogeneity of preferences (Hensher & Greene, 2003 ; Kjær & Gyrd-Hansen, 2008 ; Revelt & Train, 1998 ). All coefficients are specified as normally distributed parameters with zero-correlations between random parameters. The DCE model captured the main effects of each domain level. Interaction terms between attributes and individual characteristics (i.e., Sex dummy taking value of 1 if male; Age dummy taking value of 1 if age > 45, Work experience dummy taking value of 1 for > 10 year of experience; and two Profession dummies with reference category Researcher/Academia) were excluded from final model since earlier likelihood-ratio tests on conditional logits found them to be not statistically significant (results remained similar for different individual characteristics threshold values). Similarly, restricting analysis to the policy-makers sub-sample (as the group who is more likely to be involved and influence decision-making) produces very similar results to the full sample analysis and as such sub-group results are omitted (results given in Appendix I ).

Equity/ efficiency profiles and ratios

As the magnitude of estimated parameters cannot be directly interpreted, results are discussed in terms of percentage changes in predicted probabilities for each attribute (Lancsar et al., 2007 ; Mentzakis et al., 2014 ; Ryan et al., 2008 ) as well as for the equity/efficiency groups in aggregate (i.e. summing up all efficiency (or equity) attributes for a fully efficient (or equitable) alternative). Criteria with a higher probability of being chosen will be more likely to influence the selection of the interventions.

Moreover, the difference between the predicted probabilities for the equity-only and efficiency-only interventions were calculated by subtraction as well as the percentage change with respect to a baseline, defined as a hypothetical intervention for which all attributes are set at their sample mean values. The results provide an estimate of the size of contribution of the efficiency and equity components and denote the implicit willingness to trade-off these components with each other. Table 5 presents the (changes in) predicted probabilities. Furthermore, changes and percentage changes in predicted probabilities for the aggregated criteria along with the calculated equity-efficiency trade-off (i.e. calculated as the ratios of percentage changes in predicted probabilities of the aggregate Efficiency over the corresponding aggregate Equity value) were separately measured for each age group: interventions targeting young, middle-age and old groups. For all predicted probabilities 95% confidence intervals are calculated through the Delta method.

Composite league table

To further operationalize estimation results and place them more aptly in a policy relevant context, a composite league table (CLT) is used for illustration. Health interventions are classified and ranked within the context of country-specific clinical conditions. Each intervention is mapped along the attributes of our experiment (an example is given in the Table notes of Appendix II ). The information on the mapping of interventions is based on information used in the epidemiology disease models developed and employed in the CHOICE ( CHO osing I nterventions that are C ost- E ffective) program of the World Health Organization (WHO). The ‘severity of disease’ and ‘individual benefit’ criteria were decided based on primary/secondary preventive and inpatient/outpatient treatment. Willingness to subsidise others is considered to be equally high across interventions due to universal coverage.

According to the disease burden of high-income countries (Mathers et al., 2008 ; WHO, 2003 ), 24 types of interventions were considered (cf. Appendix II ), i.e. health interventions across the major disease areas, including control of non-communicable, chronic disease threat and risk factors that are of interest in Germany. The main data sources used to choose the clinical conditions were developed by the WHO and partner communities (Alwan, 2011 ; Murray & Lopez, 2013 ; Murray & Lopez, 1998 ; Whiteford et al., 2013 ), as well as by national guidelines and statistics issued by the German Ministry of Health and associated organisations (Robert Koch Institut, 2006 ; Federal Ministry of Health, 2007 ; Berufsverband Deutscher Psychologinnen & Psychologen, 2012 ; Lademann & Kolip, 2005 ; Robert Koch Institut, 2014 ; NVL, 2015 ).

Given the mapping of interventions to the six attributes that enter the model and the attribute coefficients obtained from the estimation, the probability of selection of each intervention is calculated, often termed “composite index” score (CIs) in the literature, which measures its priority level as determined by its characteristics (Baltussen & Niessen, 2006 ; Baltussen et al., 2007 ). Subsequently, all interventions are rank ordered according to their CIs which produces the final CLT ordering. The aim of the CLT is to identify those interventions that should be prioritized for public reimbursement (high-income countries) or health initiative (low-income countries) (Defechereux et al., 2012 ).

Table 4 presents the mixed logit estimates for equity and efficiency attributes among German decision-makers. Magnitude is not directly interpretable and therefore we discuss sign and significance of the coefficients in the first instance; a positive sign suggesting utility increasing characteristics and conversely for a negative sign. With the exception of ‘middle age group’ and ‘willingness to subsidise’, all coefficients were statistically significant at 1% ( p  < 0.01).

Respondents appear to prefer interventions addressing the young (baseline) over those targeting high age groups. Moreover, interventions requiring public support are not favoured. On the other hand, there seems to be a strong preference towards interventions that target the severely-ill as well as towards interventions with substantial health effects for those treated. Not surprisingly, interventions that are beneficial for a larger proportion of the population and those which prove to be cost-effective are favoured.

Moving on to Table 5 , with regards to equity criteria, ‘severity of disease’ increases the probability of selection for an intervention by 7.23% (95% CI 6.23–8.22) as compared to the baseline. All other equity attributes reduce the probability of selection as compared to the baseline. However, the effects for ‘middle aged’ and ‘willingness to subsidise others’ are statistically insignificant. Looking at efficiency criteria, all three criteria exhibit large, significant and positive effects. The probability of selecting interventions that entail substantial health benefits increases by 6.11% (95% CI 5.17–7.05) compared to the baseline, while the corresponding probability for interventions that provide benefits to a larger share of the population is 6.25% (95% CI 5.28–7.21), and 5.83% (95% CI 5.00–6.66) for interventions that are cost-effective. With regards to aggregate criteria along with the calculated equity/efficiency trade-off, interventions appear to be strongly preferred when improving efficiency, independent of the age group that is targeted. This, however, is especially true for interventions targeting young and high age groups. Except for aggregated equity criteria for high age groups, all coefficients are significant.

Based on the estimated coefficients, an overall ranking is presented in the Appendix II . Several interventions have similar characteristics with respect to our efficiency and equity criteria, resulting in similar scores and, hence, rankings. According to the results of the CLT, interventions aimed at mental disorders and CVDs are among those ranked the highest.

Overall, interventions targeting psychological and behavioural disorders as well as cardiovascular diseases exhibit the highest-ranking scores, closely followed by neoplasms and diabetes (endocrine, metabolic diseases). Intervention “Education, promote individual, family, community connectedness” targeting the condition “Suicide and intentional self-harm” is the highest ranked intervention for the German stakeholders.

This study draws attention on the use of discrete choice experiments to devise rational frameworks for priority setting, taking explicitly into account the concerns for different societal objectives. In this regard, the results of the experiment on a sample of relevant stakeholders in the German health system allow one to discuss some interesting findings.

German decision makers consider severity of disease, individual health benefits, cost-effectiveness, as well as number of potential beneficiaries as important criteria for priority setting. The absolute values of the regressions reflect their relative importance in priority setting. Based on their respective weights, severity of disease, number of potential beneficiaries, individual health benefits and cost-effectiveness appear as the most important criteria for German decision makers within the sample population, displaying great preference towards efficiency.

Besides the general pro cost-effectiveness attitude, respondents associate a higher utility to interventions targeting younger age groups. This is in line with empirical studies in Germany, such as Raspe and Stumpf ( 2013 ), which find priority setting in favour of treatments for children. Willingness to subsidise others appears insignificant, which again confirms a priori expectations in high-income countries such as Germany that are characterised by universal coverage (cf. Norway and Austria). Overall, all efficiency attributes are favoured over equity criteria, except for severity of disease (Diederich et al., 2012 ). Although equity concerns seem to be comparably less important in healthcare resources allocation decisions in Germany, the two objectives (i.e. efficiency and equity) are in conflict with each other and are equally needed in a deliberative process (Culyer, 2006 , 2015 ). The estimated ratios between equity and efficiency support a general preference for efficiency over equity criteria for all age groups, with much stronger results for interventions targeting younger and higher age groups.

These findings show a large overlap with the prioritisation discussion in Germany and are aligned with what was proposed by influential bodies in the German health community. For instance, in a second plea in favour of a priority setting debate in 2007 (first in 2000) the ZEKO addresses the importance of defining the best relative weight for the much-needed prioritisation criteria. Almost all revealed criteria, namely proven benefit/fitness of purpose, cost–effectiveness and medical need (urgency and severity), support our findings. Nevertheless, the general preference assigned to efficiency does not involve a lack of concern for equity. Indeed, basic equity in terms of financial protection is guaranteed through the basic solidarity principle grounded within the SHI. This principle entitles every individual to the same services irrespective of their insurance status or the contributions paid (Deutsche Sozialversicherung, n.d.).

The CLT results can be considered indicative when prioritizing among interventions. Largely, the resulting rankings reflect the National Health Goals (Bundesministerium für Gesundheit, Gesundheitsziele.de, 2022 ) concerning Type 2 diabetes, breast cancer, depressive disorder, healthy ageing, reduction of alcohol and tobacco consumption and enhancing health competence (Federal Ministry of Health, 2007 ). These main goals are a complementary governance tool in healthcare and seek to improve the health of individuals or specific groups to tackle the conditions of highest urgency. Yet, the high prioritization of mental disorders is not in line with Schröter and Diederich ( 2013 ) who reported that the German population considers mental health of lower importance for prioritisation of medical resources compared with physical health. Nevertheless, such discrepancies in preferences for mental health could largely depend on the specific context.

Together with the National Health Goals, the Information System of the Federal Health Monitoring and the Federal Joint Committee identified the disease burden as one of the most relevant determinants in setting healthcare priorities. This approach is in line with the WHO “2013–2020 Global Action Plan for Prevention and Control of Non-communicable Diseases” that underlines the need to urgently address prevention problems, and to allocate more resources to the early treatment of chronic NCDs and mental illnesses (WHO, 2013 ).

Similarly to the German results, preferences for efficiency and equity criteria elicited across countries have displayed individual benefits, severity of disease and cost-effectiveness as the most significant priorities in high income countries, HICs (Baji et al., 2016 ; Defechereux et al., 2012 ; Mentzakis et al., 2014 ). In low-income countries (LICs) like Ghana or Nepal (Baltussen et al., 2006 , 2007 ) instead, number of beneficiaries, individual benefits, cost-effectiveness, severity of disease, and middle-aged people are found to be the preferred criteria—showing more balanced equity/efficiency preferences. Results of a Chinese study (Paolucci et al., 2015 ) disclose a much closer profile resembling that of the mentioned high income countries where universal health coverage is in place.

Comparing the CLTs obtained for Germany with those for Austria and Norway (as instances of comparable HICs) findings are largely comparable (Defechereux et al., 2012 ; Mentzakis et al., 2011 ). In those studies, countries share a similar disease burden, comprising mainly mental disorders and NCD, including diabetes, cancer, and cardiovascular diseases. This holds true for the CLT that assigns relatively high rankings for respective disease areas. Compared to other HICs, German decision makers seem to rank higher those interventions affecting young or middle-aged people.

One of the limitations of DCEs is their hypothetical nature. Due to the explorative nature of the study the findings cannot be directly implemented into national policy making but could act as a first step and guide. In fact, results can contribute towards the German debate on setting priorities in healthcare. Further, we note that while our sample size is not small and allows for robust estimation, the low response rate suggests caution in inference and limited generalizability, while future research should explore the congruence of preference between stakeholders and general public. Nonetheless, the methodology is generalizable and can be transported to other countries and settings when the required conditions for successful multi-criteria decision analysis (MCDA) in health are met. Apart from age, non-linear effects were not incorporated in the analysis. Our design identifies individuals’ direction of preferences rather than the exact shape of their function, while future research could focus on non-linearities. The survey and attribute levels were taken from a larger DCE project targeting many countries and as such reference levels in the dichotomization of the attributes were taken to meet international standards and ensure cross-country comparability. Yet, such dichotomization and use of attributes with different scales could introduce vagueness and affect their perception by respondents and conceal potential difference in the relative importance of attributes. Future studies could increase the number of levels for relevant attributes and obtain preferences over a range of discrete attribute values.

Establishing criteria for equitable and efficient resource allocation in healthcare is a political task with a number of dimensions including medical, economic, ethical and legal ones. The complexity of the issue makes it impossible to achieve a complete consensus between all those involved. Nevertheless, principles ought to be formulated in which existing structures and processes must be measured, not at least in the sense of a future-oriented perspective.

In conclusion, this explorative study details how multiple criteria can guide a transparent and systematic priority-setting process by allowing for the simultaneous assessment of multiple policy objectives of decision-makers. With German decision makers stating a preference for efficiency, such an approach can help to support the priority setting processes and may contribute to a more informed and participated debate on priority setting between different health interventions in Germany.

Commissions working on prioritisation in other countries are: Norway (Lønning Committee I und II, 1987/1997), Orgeon, USA (Oregon Health Service Commission, 1989), Netherlands (Dunning Committee, 1992/1995), Sweden (Commission of Parliament and Experts, 1993/1995), New Zealand (Core Services Committee/National Health Committee, 1993), Israel (Medical Technical Forum and National Advisory Committee, 1995), Denmark (Danish Counsil of Ethics, 1997) and England, Wales (National Institute for Health and Clinical Excellence [NICE], 1999).

Alliance for Health Policy and Systems Research. (2004). Strengthening health systems: The role and promise of policy and systems research. Alliance for Health Policy and Systems Research.

Alwan, A. (2011). Global status report on noncommunicable diseases 2010. World Health Organization.

Baeten, S. A., Baltussen, R. M., Uyl-de Groot, C. A., Bridges, J., & Niessen, L. W. (2010). Incorporating equity-efficiency interactions in cost-effectiveness analysis—three approaches applied to breast cancer control. Value in Health, 13 (5), 573–579.

Article   PubMed   Google Scholar  

Baji, P., García-Goñi, M., Gulácsi, L., Mentzakis, E., & Paolucci, F. (2016). Comparative analysis of decision maker preferences for equity/efficiency attributes in reimbursement decisions in three European countries. The European Journal of Health Economics, 17 (7), 791–799.

Baltussen, R., et al. (2006). Towards a multi-criteria approach for priority setting: An application to Ghana. Health Economics, 15 (7), 689–696.

Baltussen, R., & Niessen, L. (2006). Priority setting of health interventions: The need for multi-criteria decision analysis. Cost Effectiveness and Resource Allocation: C/e, 4 , 14.

Baltussen, R., Ten Asbroek, A., Koolman, X., Shrestha, N., Bhattarai, P., & Niessen, L. W. (2007). Priority setting using multiple criteria: Should a lung health programme be implemented in Nepal? Health Policy and Planning, 22 (3), 178–185.

Article   CAS   PubMed   Google Scholar  

Baltussen, R., Youngkong, S., Paolucci, F., & Niessen, L. (2010). Multi-criteria decision analysis to prioritize health interventions: Capitalizing on first experiences. Health Policy, 96 (3), 262–264.

Beske, F., & Drabinski, T. (2005). Leistungskatalog des Gesundheitswesens im internationalen Vergleich. Eine Analyse von 14 Ländern. Schmidt & Klaunig, Kiel. Bd. I: Struktur, Finanzierung und Gesundheitsleistungen. ISBN 3-88312-330-7. Bd: II: Geldleistungen. ISBN 3-88312-331-5.

Blümel, M., Busse, R. (2017). In E. Mossialos, A. Djordjevic, R. Osborn, & D. Sarnak (Eds.), International profiles of health care systems . The Commonwealth Fund, May 2017.

Blümel, M., Spranger, A., Achstetter, K., Maresso, A., & Busse, R. (2020). Germany: Health system review. Health Systems in Transition, 22 (6), 1–272.

PubMed   Google Scholar  

Borck, C. (2012). What kind of debate on prioritisation do we need for the benefit of our health system and society? What kind of debate on prioritisation do we need for the benefit of our health system and society? Welche Debatte um Priorisierung in der medizinischen Versorgung brauchen wir? Zeitschrift Für Evidenz, Fortbildung Und Qualität Im Gesundheitswesen, 106 (6), 383–385.

Bundesärztekammer. (2014). Ärztetag fordert: “Priorisierung enttabuisieren”. Retrieved Mar 5, 2019 from https://www.bundesaerztekammer.de/aerztetag/aerztetage-der-vorjahre/117-daet-2014-in-duesseldorf/presseinformationen/priorsierung/ .

Bundesministerium für Gesundheit, Gesundheitsziele.de. (2022). https://www.bundesgesundheitsministerium.de/themen/gesundheitswesen/gesundheitsziele.html . Accessed 12 April 2023.

Burkhardt, B. (2012). Der Leistungskatalog oder Wie weit soll die Solidarität gehen? In Bundeszentrale für politische Bildung (Ed.). Gesundheitspolitik.

Chalkidou, K., Glassman, A., Marten, R., Vega, J., Teerawattananon, Y., Tritasavit, N., Gyansa-Lutterodt, M., Seiter, A., Kieny, M. P., Hofman, K., & Culyer, A. J. (2016). Priority-setting for achieving universal health coverage. Bulletin of the World Health Organization, 94 (6), 462.

Article   PubMed   PubMed Central   Google Scholar  

Clark, M. D., Szczepura, A., Gumber, A., Howard, K., Moro, D., & Morton, R. L. (2017). Measuring trade-offs in nephrology: a systematic review of discrete choice experiments and conjoint analysis studies. Nephrology Dialysis Transplantation, 33 (2), 348–355.

Article   Google Scholar  

Cromwell, I., Peacock, S. J., & Mitton, C. (2015). ‘Real-world’health care priority setting using explicit decision criteria: A systematic review of the literature. BMC Health Services Research, 15 (1), 164.

Culyer, A. J. (2015). Efficiency, equity and equality in health and health care (No. 120cherp). Centre for Health Economics, University of York.

Culyer, A. J. (2006). The bogus conflict between efficiency and vertical equity. Health Economics, 15 (11), 1155–1158.

Czoli, C. D., Goniewicz, M., Islam, T., Kotnowski, K., & Hammond, D. (2016). Consumer preferences for electronic cigarettes: Results from a discrete choice experiment. Tobacco Control, 25 (e1), e30–e36.

Dannecker, G. (2009). Priorisierung: Notwendiger rechtlicher gestaltungsspielraum. Deutsches Arzteblatt-Arztliche Mitteilungen-Ausgabe B, 106 (41), 1721.

Google Scholar  

de Bekker-Grob, E. W., Ryan, M., & Gerard, K. (2012). Discrete choice experiments in health economics: A review of the literature. Health Economics, 21 (2), 145–172.

Defechereux, T., Paolucci, F., Mirelman, A., Youngkong, S., Botten, G., Hagen, T. P., & Niessen, L. W. (2012). Health care priority setting in Norway a multicriteria decision analysis. BMC Health Services Research, 12 (1), 39.

Di Costanzo, C. (2020). Healthcare resource allocation and priority-setting. A European Challenge, European Journal of Health Law, 27 (2), 93–114.

Diederich, A., du Bois, G. & Dörr, D., 2015. Einstellungen zu Priorisierungen in der medizinischen Versorgung: Ergebnisse einer Befragung des Deutschen Ärztinnenbundes (DÄB) (p. 40).

Diederich, A., et al. (2009). Fragebogen zur Erhebung von Präferenzen in der Bevölkerung bezüglich der Verteilung von Gesundheitsleistungen in der GKV (p. 655). Jacobs University.

Diederich, A., Swait, J., & Wirsik, N. (2012). Citizen participation in patient prioritization policy decisions: An empirical and experimental study on patients’ characteristics. PLoS ONE, 7 (5), e36824.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Diederich, A., Winkelhage, J., & Wirsik, N. (2011). Age as a criterion for setting priorities in health care? A survey of the German public view. PLoS ONE, 6 (8), e23930.

Emanuel, E. J. (2000). Justice and managed care: Four principles for the just allocation of health care resources. The Hastings Center Report, 30 (3), 8.

Federal Joint Committee 2017. The Federal Joint Committee Decisions on Healthcare benefits. Retrieved Mar 5, 2019 from http://www.english.g-ba.de/downloads/17-98-3769/2017-02-02_G-BA_Infobrosch%C3%BCre_EN_web_bf.pdf .

Federal Ministry of Health. (2007). Maßnahmen des Bundesministeriums für Gesundheit zur Umsetzung nationaler Gesundheitsziele . Bundesministerium für Gesundheit.

Fleck, L. (2001). Healthcare justice and rational democratic deliberation. The American Journal of Bioethics: AJOB, 1 (2), 20–21.

Friedrich, D., Buyx, A., & Schöne-Seifert, B. (2009). Priorisierung: Marginale Wirksamkeit als Ausschlusskriterium. Dtsch Arztebl, 106 , 1562–1564.

Fuchs, C. (2010). Demografischer Wandel und Notwendigkeit der Priorisierung im Gesundheitswesen. Bundesgesundheitsblatt-Gesundheitsforschung-Gesundheitsschutz, 53 (5), 435–440.

Genie, M. G., Loría-Rebolledo, L. E., Paranjothy, S., Powell, D., Ryan, M., Sakowsky, R. A., & Watson, V. (2020). Understanding public preferences and trade-offs for government responses during a pandemic: A protocol for a discrete choice experiment in the UK. British Medical Journal Open, 10 (11), e043477.

Schöne-Seifert B (ed.). (2006). Gerecht behandelt: Rationierung und Priorisierung im Gesundheitswesen

Gerst T. (2014). Priorisierung: Ärzte sollen überzeugt werden. Deutsches Ärzteblatt , 111(44), A-1893 / B-1619 / C-1551.

Gordijn, B., & Have, H. T. (2013). International experiences with priority setting in healthcare. Medicine, Health Care, and Philosophy, 16 (3), 325–326.

Green, C., & Gerard, K. (2009). Exploring the social value of health-care interventions: A stated preference discrete choice experiment. Health Economics, 18 (8), 951–976.

Groß, D., Schäfer, G., & Westermann, S. (2010). Möglichkeiten und Grenzen von Rationierung und Priorisierung im Gesundheitswesen. Gesundheit und Wirtschaftswachstum (pp. 133–149). Springer.

Chapter   Google Scholar  

Hansen, P., & Devlin, N., (2019). Multi-criteria decision analysis (MCDA) in healthcare decision-making. In Oxford Research Encyclopedia of Economics and Finance.

Heil, S., Schreier, M., Winkelhage, J., & Diederich, A. (2010). Explorationsstudien zur Priorisierung medizinischer Leistungen: Kriterien und Präferenzen verschiedener Stakeholdergruppen. FOR655 , 26 (3).

Hensher, D. A., & Greene, W. H. (2003). The mixed logit model: The state of practice. Transportation, 30 (2), 133–176.

Heyers, J. (2016). Möglichkeiten und Grenzen einer Ökonomisierung des Sozialrechts am Beispiel der Priorisierung in der Gesetzlichen Krankenversicherung. Medizinrecht, 34 (11), 857–866.

Hole, A. R. (2007). Fitting mixed logit models by using maximum simulated likelihood. The Stata Journal, 7 (3), 388–401.

Institut, R. K. (Ed.). (2014). Daten und Fakten: Ergebnisse der Studie ‘Gesundheit in Deutschland aktuell 2012.’ Robert-Koch-Inst.

Institute for Health Metrics and Evaluation. (2016). Global burden of disease (GBD)—Germany , Retrieved Nov 15, 2018 from http://www.healthdata.org/germany .

James, C., Carrin, G., Savedoff, W., & Hanvoravongchai, P. (2005). Clarifying efficiency-equity tradeoffs through explicit criteria, with a focus on developing countries. Health Care Analysis, 13 (1), 33–51.

Kapiriri, L., & Martin, D. (2007). A strategy to improve priority setting in developing countries: health care analysis. Health Care Analysis, 15 (3), 159–167.

Kenny, N., & Joffres, C. (2008). An ethical analysis of international health priority-setting. Health Care Analysis: HCA: Journal of Health Philosophy and Policy, 16 (2), 145–160.

King, M. T., Hall, J., Lancsar, E., Fiebig, D., Hossain, I., Louviere, J., & Jenkins, C. R. (2007). Patient preferences for managing asthma: Results from a discrete choice experiment. Health Economics, 16 (7), 703–717.

Kjær, T., & Gyrd-Hansen, D. (2008). Preference heterogeneity and choice of cardiac rehabilitation program: Results from a discrete choice experiment. Health Policy, 85 (1), 124–132.

Kliemt, H. (2006). Ethik und Politik der Rationierung im Gesundheitswesen. In C. Wendt & C. Christof (Eds.), Soziologie der Gesundheit (pp. 343–363). VS Verlag für Sozialwissenschaften.

Koopmanschap, M. A., Stolk, E. A., & Koolman, X. (2010). Dear policy maker: Have you made up your mind? A discrete choice experiment among policy makers and other health professionals. International Journal of Technology Assessment in Health Care, 26 (2), 198–204.

Krauth, C., Oedingen, C., Bartling, T., Dreier, M., Spura, A., de Bock, F., & Robra, B. P. (2021). Public preferences for exit strategies from COVID-19 lockdown in Germany—a discrete choice experiment. International Journal of Public Health, 66 , 6.

Krinke, K. S., Tangermann, U., Amelung, V. E., & Krauth, C. (2019). Public preferences for primary care provision in Germany–a discrete choice experiment. BMC Family Practice, 20 (1), 1–9.

Lademann, J., & Kolip, P. (2005). Gesundheit von Frauen und Männern im mittleren Lebensalter . Robert-Koch-Institut.

Lancsar, E., & Louviere, J. (2008). Conducting discrete choice experiments to inform healthcare decision making. PharmacoEconomics, 26 (8), 661–677.

Lancsar, E., Louviere, J., & Flynn, T. (2007). Several methods to investigate relative attribute impact in stated preference experiments. Social Science & Medicine, 64 (8), 1738–1753.

Longo, M. F., Cohen, D. R., Hood, K., Edwards, A., Robling, M., Elwyn, G., & Russell, I. T. (2006). Involving patients in primary care consultations: Assessing preferences using discrete choice experiments. British Journal of General Practice, 56 (522), 35–42.

López-Bastida, J., Ramos-Goñi, J. M., Aranda-Reneo, I., Trapero-Bertran, M., Kanavos, P., & Martin, B. R. (2019). Using a stated preference discrete choice experiment to assess societal value from the perspective of decision-makers in Europe. Does it work for rare diseases? Health Policy, 123 (2), 152–158.

Luyten, J., Kessels, R., Goos, P., & Beutels, P. (2015). Public preferences for prioritizing preventive and curative health care interventions: A discrete choice experiment. Value in Health, 18 (2), 224–233.

Mandeville, K. L., Lagarde, M., & Hanson, K. (2014). The use of discrete choice experiments to inform health workforce policy: a systematic review. BMC Health Services Research . https://doi.org/10.1186/1472-6963-14-367

Marckmann, G. (2009). Prioritisation in health care: Learning from international experiences. Zeitschrift Für Evidenz, Fortbildung Und Qualität Im Gesundheitswesen, 103 (2), 85–91.

Mathers, C., Fat, D. M., & Boerma, J. T., (2008). The global burden of disease: 2004 update , World Health Organization.

Mengoni, A., Seghieri, C., & Nuti, S. (2013). Heterogeneity in preferences for primary care consultations: Results from a discrete choice experiment. International Journal of Statistics in Medical Research, 2 (1), 67.

Mentzakis, E., Paolucci, F., & Rubicko, G. (2014). Priority setting in the Austrian healthcare system: Results from a discrete choice experiment and implications for mental health. The Journal of Mental Health Policy and Economics, 17 (2), 61–73.

Mentzakis, E., Ryan, M., & McNamee, P. (2011). Using discrete choice experiments to value informal care tasks: Exploring preference heterogeneity. Health Economics, 20 (8), 930–944.

Mirelman, A., Mentzakis, E., Kinter, E., Paolucci, F., Fordham, R., Ozawa, S., & Niessen, L. W. (2012). Decision-making criteria among national policymakers in five countries: A discrete choice experiment eliciting relative preferences for equity and efficiency. Value in Health, 15 (3), 534–539.

Mitton, C., & Donaldson, C. (2004). Health care priority setting: Principles, practice and challenges. Cost Effectiveness and Resource Allocation: C/e, 2 , 3.

Müller, S., & Groß, D. (2009). Zur Akzeptanz von Leistungsbegrenzungen im Gesundheitswesen: Strategien, Kriterien und Finanzierungsmodelle unter Berücksichtigung ethischer Aspekte. Gesundheitsmonitor , 258–279.

Murray, C. J., & Lopez, A. (Eds.), 1998. The global burden of disease . Harvard School of Public Health on behalf of the World Health Organization and the World Bank.

Murray, C. J., & Lopez, A. (2013). Measuring the global burden of disease. New England Journal of Medicine, 369 (5), 448–457.

Noorani, H. Z., Husereau, D. R., Boudreau, R., & Skidmore, B. (2007). Priority setting for health technology assessments: A systematic review of current practical approaches. International Journal of Technology Assessment in Health Care, 23 (03), 310–315.

Norheim, O. F. (2016). Ethical priority setting for universal health coverage: Challenges in deciding upon fair distribution of health services. BMC Medicine, 14 (1), 75.

NVL. (2015). NVL—Nationale Versorgungsleitlinien. Retrieved Nov 20, 2018 from http://www.leitlinien.de/nvl/

Oduncu, F. S. (2012). Verteilungsgerechtigkeit, Rationierung und Priorisierung–das Gesundheitswesen im Spannungsfeld zwischen Medizin, Ökonomie. Ethik Und Recht. Medr Medizinrecht, 30 (6), 359–367.

Oduncu, F. S. (2013). Priority-setting, rationing and cost-effectiveness in the German health care system. Medicine, Health Care, and Philosophy, 16 (3), 327–339.

OECD. (2021) . Health statistics 2021, WHO global health expenditure database . https://doi.org/10.1787/888934016816 . Retrieved Aug 20, 2022.

Ottersen, T., Førde, R., Kakad, M., Kjellevold, A., Melberg, H. O., Moen, A., Ringard, Å., & Norheim, O. F. (2016). A new proposal for priority setting in Norway: Open and fair. Health Policy, 120 (3), 246–251.

Paolucci, F., Mentzakis, E., Defechereux, T., & Niessen, L. W. (2015). Equity and efficiency preferences of health policy makers in China—a stated preference analysis. Health Policy and Planning, 30 (8), 1059–1066.

Peacock, S., Mitton, C., Ruta, D., Donaldson, C., Bate, A., & Hedden, L. (2010). Priority setting in healthcare: towards guidelines for the program budgeting and marginal analysis framework. Expert Review of Pharmacoeconomics & Outcomes Research, 10 (5), 539–552.

Peacock, S., Ruta, D., Mitton, C., Donaldson, C., Bate, A., & Murtagh, M. (2006). Using economics to set pragmatic and ethical priorities. BMJ, 332 (7539), 482–485.

Petri, B. (2015). Rationalisierung, Mittelerhöhung, Rationierung und Priorisierung von Gesundheitsleistungen in der Sozialversicherung mit ethischen Aspekten. In L. Mülheims, K. Hummel, S. Peters-Lange, E. Toepler, & I. Schuhmann (Eds.), Handbuch Sozialversicherungswissenschaft. Springer VS.

Berufsverband Deutscher Psychologinnen und Psychologen ed. (2012). Die grossen Volkskrankheiten: Beiträge der Psychologie zu Prävention, Gesundheitsförderung und Behandlung (1 ed.). Deutscher Psychologen Verl

Raspe, H., & Meyer, T. (2009). Priorisierung. Vom Schwedischen Vorbild Lernen. Deutsches Ärzteblatt, 106 (21), A1036–A1039.

Raspe, H., & Stumpf, S. (2013). Kriterien und Verfahren zur Priorisierung medizinischer Leistungen: Ergebnisse und methodische Herausforderungen. In J. Böcken, B. Braun, & U. Repschläger (Eds.), Gesundheitsmonitor 2013, Bürgerorientierung im Gesundheitswesen (pp. 186–210). Verlag Bertelsmann Stiftung.

Razavi, S., Kapiriri, L., Wilson, M., & Abelson, J. (2020). Applying priority-setting frameworks: A review of public and vulnerable populations’ participation in health-system priority setting. Health Policy, 124 (2), 133–142.

Revelt, D., & Train, K. (1998). Mixed logit with repeated choices: Households’ choices of appliance efficiency level. Review of Economics and Statistics, 80 (4), 647–657.

Robert Koch Institut (ed.), (2006). Gesundheit in Deutschland: Gesundheitsberichterstattung des Bundes

Ryan, M., & Gerard, K. (2003). Using discrete choice experiments to value health care programmes: Current practice and future research reflections. Applied Health Economics and Health Policy, 2 (1), 55–64.

Ryan, M., Gerard, K., & Amaya-Amaya, M. (2008). In Using discrete choice experiments to value health and health care . Springer Science & Business Media.

Book   Google Scholar  

Sabik, L. M., & Lie, R. K. (2008). Priority setting in health care: Lessons from the experiences of eight countries. International Journal for Equity in Health, 7 (1), 1–13.

Sauerland, D. (2001). The German strategy for quality improvement in health care: Still to improve. Health Policy, 56 (2), 127–147.

Schmitz-Luhn, B., & Katzenmeier, C. (2016). The law behind priorities: Implementation of priority setting in health care—The German example. In E. Nagel & M. Lauerer (Eds.), Prioritization in medicine. Springer.

Schröter, J., & Diederich, A. (2013). Mental and physical health: Citizens’ preferences on priority-setting in healthcare (p. 655). Jacobs Univ.

Deutsche Sozialversicherung, n.d.. Basic Principles. Retrieved Mar 12, 2019 from http://www.deutsche-sozialversicherung.de/en/guide/basic_principles.html .

Statistische Bundesamt. (2022) . Gesundheitsausgaben in Deutschland in Mio. €, in Prozent, je Einwohner. Retrieved Aug 02, 2022 from Gesundheitsausgaben in Deutschland in Mio. €, in Prozent, je Einwohner .

Storz, P., & Egger, B. (2010). Die Debatte über Priorisierung und Rationierung in der Gesetzlichen Krankenversicherung: überfällig oder überflüssig? Gesundheits Sozialpolitik, 1 , 11–18.

Tanios, N., Wagner, M., Tony, M., Baltussen, R., van Til, J., Rindress, D., Kind, P., & Goetghebeur, M. M. (2013). Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers. International Journal of Technology Assessment in Health Care, 29 (4), 456–465.

Thokala, P., Devlin, N., Marsh, K., Baltussen, R., Boysen, M., Kalo, Z., Longrenn, T., Mussen, F., Peacock, S., Watkins, J., & Ijzerman, M. (2016). Multiple criteria decision analysis for health care decision making—an introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value in Health, 19 (1), 1–13.

van de Schoot, T., Pavlova, M., Atanasova, E., & Groot, W. (2017). Preferences of Bulgarian consumers for quality, access and price attributes of healthcare services—result of a discrete choice experiment. The International Journal of Health Planning and Management, 32 (1), e47–e71.

Whiteford, H. A., Degenhardt, L., Rehm, J., Baxter, A. J., Ferrari, A. J., Erskine, H. E., Charlson, F. J., Norman, R. E., Flaxman, A. D., Johns, N., & Burstein, R. (2013). Global burden of disease attributable to mental and substance use disorders: Findings from the Global Burden of Disease Study 2010. The Lancet, 382 (9904), 1575–1586.

Whitty, J. A., Scuffham, P. A., & Rundle-Thielee, S. R. (2011). Public and decision maker stated preferences for pharmaceutical subsidy decisions. Applied Health Economics and Health Policy , 73–79.

WHO. (2003). The global burden of disease concept. In Assessing the environmental burden of disease at national and local levels introduction and methods. Geneva: World Health Organization. Retrieved Nov 20, 2018 from http://public.eblib.com/choice/PublicFullRecord.aspx?p=3050142 .

WHO, (2007). Strengthening health systems to improve health outcomes. WHO’s Framework for Action .

WHO. (2013). 2013–2020 Global action plan for prevention and control of non-communicable diseases , Geneva.

Wordsworth, S., Skåtun, D., Scott, A., & French, F. (2004). Preferences for general practice jobs: A survey of principals and sessional GPs. British Journal of General Practice, 54 (507), 740–746.

World Bank. (2016). Life expectancy at birth (years). Retrieved Nov 20, 2018 from http://data.worldbank.org/indicator/SP.DYN.LE00.MA.IN/countries .

ZEKO. (2000). Zentrale Kommission zur Wahrung ethischer Grundsätze in der Medizin und ihren Grenzgebieten (Zentrale Ethikkommission)—Prioritäten in der medizinischen Versorgung im System der Gesetzlichen Krankenversicherung (GKV): Müssen und können wir uns entscheiden? Dtsch Arztebl International, 97 (15), A-1017.

Zentrale Ethikkommission der Bundesärztekammer. (2007). Stellungnahme zur Priorisierung medizinischer Leistungen im System der GKV. http://www.zentrale-ethikkommission.de/page.asp?his=0.1.53

Download references

Acknowledgements

Petra Baji’s research was supported by the Hungarian Research Fund OTKA (PD 112499)

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and affiliations.

Faculty of Medicine, FAU Erlangen-Nürnberg, Erlangen, Germany

Department of Economics, City University of London, Northampton Square, London, EC1V 0HB, UK

E. Mentzakis

Department of Health Economics, Corvinus University of Budapest, Budapest, Hungary

Department of Economics, University of Bologna, Bologna, Italy

G. Fiorentini

Sir Walter Murdoch School of Public Policy and International Affairs, Murdoch University, Perth, WA, Australia

F. Paolucci

Department of Sociology and Law & Economics, University of Bologna, Bologna, Italy

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to V. Meusel .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix I Policy Makers estimation results

Mixed logit estimation results for the policy makers subgroup.

• 1. Estimates are based on a mixed logit model with normal distribution specified for each attribute, 2. Dependent variable takes the value of 1 if individual chooses that particular alternative, 3. Every individual contributes a total of 32 observations (16 choice sets with 2 alternatives), 4. *indicates that the coefficient is significant at 10% level of significance ( p  < 0.1), **indicates that the coefficient is significant at 5% level of significance ( p  < 0.05), 6. ***indicates that the variables are significant at 1% level of significance ( p  < 0.01)

Appendix II Composite league table

• Psy psychological and behavioural disorders , CVD cardiovascular diseases, Resp respiratory diseases, Endo endocrine, nutrition and metabolic disorders, Musc diseases of the muscular and skeletal system, Neo (malignant) neoplasms, Lifest risk-related factors/Lifestyle
• Construction of the CLT begins by taking each intervention under consideration and assigning it values for each attribute. For example, an intervention to “Promote healthy eating in school” would take the value of 0 for Severity (i.e. low severity), value of 1 for Individual Benefit (i.e. large individual benefits), value of 0 for Middle-age and value of 0 for Old-age (ie as it target young people), value of 0 for cost-effectiveness (i.e. low cost-effectiveness), value of 1 for Number of potential beneficiaries (ie high number of beneficiaries) and value of 1 for willingness-to-subsidize others (i.e. more than 70%). Following such mapping and given the attribute coefficients (ie part worth utilities) obtained from the model estimation, predicted probabilities for each intervention can be computed indicating the probability with which an intervention would be chosen to be funded. Ranking interventions by their probabilities produces the CLT ranking

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Meusel, V., Mentzakis, E., Baji, P. et al. Priority setting in the German healthcare system: results from a discrete choice experiment. Int J Health Econ Manag. 23 , 411–431 (2023). https://doi.org/10.1007/s10754-023-09347-y

Download citation

Received : 29 April 2020

Accepted : 04 March 2023

Published : 15 May 2023

Issue Date : September 2023

DOI : https://doi.org/10.1007/s10754-023-09347-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Decision making
• Priority setting
• Discrete choice experiment

JEL Classification

• Find a journal
• Publish with us
• Track your research

Involving the public in priority setting: A case study using discrete choice experiments

• December 2011
• Journal of Public Health 34(2):253-60
• 34(2):253-60

• University of Aberdeen
• This person is not on ResearchGate, or hasn't claimed this research yet.

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

No full-text available

To read the full-text of this research, you can request a copy directly from the authors.

• Abdullah Meshal Mubarak Abdullah A Alsabah

• Stephan Devriese

• Carine Van de Voorde

• J PUBLIC HEALTH-UK

• Hamidreza Khankeh

• BMC MED RES METHODOL
• Soghra Aliasgharzadeh
• Mehrangiz Ebrahimi-Mameghani

• Fathollah Pourali

• Diane Smith

• AM J KIDNEY DIS

• Tarun Bansal

• Shelby Reed

• Yaser Jouyani
• Mohammad Hadiyan

• SOC SCI MED
• Fiona Leh Hoon Chuah

• Mihai PANDREA

• Emily Moore

• HEALTH POLICY

• HEALTH ECON

• Susan Mallett

• Steve Halligan
• Karen S Kauffman

• Yu-Hsiang Hsieh

• Anne L R Schuster

• Nicola B. Campoamor
• John F P Bridges

• Martin Bardsley

• Anne Marie Rafferty
• PUBLIC HEALTH

• Mohammad Hadian

• Ramin Rahimnia
• SM Halligan

• Ibrahim-Kholilul Rohman
• Fabienne Abadie

• John McAlister
• EUR J HEALTH ECON

• Francesco Paolucci
• PHARMACOECONOMICS

• Domino Determann

• Mark J Harrison

• HEALTH EXPECT

• Brian Ferguson

• Chris Heginbotham

• Craig Mitton

• Duncan Mortimer
• ORGAN BEHAV HUM DEC

• Joffre Dan Swait Jr

• Mabel Amaya-Amaya
• P.M. Mullen

• Donald Ross
• Anne Ludbrook
• TRANSPORT RES B-METH
• Sebastian Caussade

• German Fermo
• J TRANSP ECON POLICY

• Kelvin J. Lancaster

• Ann Bowling

• Jo Lenaghan
• Elizabeth Mitchell
• David R. Cohen

• Sheila N. Scott

• Andrea Litva
• Jenny L Donovan
• Kieran Morgan

• Virginia Wiseman
• Gavin Mooney

• Howard Waldner

• J HEALTH ECON

• Dan Chisholm

• Scottish Executive
• R Baltussen
• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

• Methodology
• Open access
• Published: 21 August 2006

Priority setting of health interventions: the need for multi-criteria decision analysis

• Rob Baltussen 1 , 2 &
• Louis Niessen 1 , 3  

Cost Effectiveness and Resource Allocation volume  4 , Article number:  14 ( 2006 ) Cite this article

Priority setting of health interventions is often ad-hoc and resources are not used to an optimal extent. Underlying problem is that multiple criteria play a role and decisions are complex. Interventions may be chosen to maximize general population health, to reduce health inequalities of disadvantaged or vulnerable groups, ad/or to respond to life-threatening situations, all with respect to practical and budgetary constraints. This is the type of problem that policy makers are typically bad at solving rationally, unaided. They tend to use heuristic or intuitive approaches to simplify complexity, and in the process, important information is ignored. Next, policy makers may select interventions for only political motives.

This indicates the need for rational and transparent approaches to priority setting. Over the past decades, a number of approaches have been developed, including evidence-based medicine, burden of disease analyses, cost-effectiveness analyses, and equity analyses. However, these approaches concentrate on single criteria only, whereas in reality, policy makers need to make choices taking into account multiple criteria simultaneously. Moreover, they do not cover all criteria that are relevant to policy makers.

Therefore, the development of a multi-criteria approach to priority setting is necessary, and this has indeed recently been identified as one of the most important issues in health system research. In other scientific disciplines, multi-criteria decision analysis is well developed, has gained widespread acceptance and is routinely used. This paper presents the main principles of multi-criteria decision analysis. There are only a very few applications to guide resource allocation decisions in health. We call for a shift away from present priority setting tools in health – that tend to focus on single criteria – towards transparent and systematic approaches that take into account all relevant criteria simultaneously.

Pertaining health needs and accelerating technological development put an ever-increasing demand on limited health budgets. Policy makers need to make important decisions on the use of public funds – to target which disease areas, which populations, and with which interventions. However, these choices may not be based on a rational and transparent process, and resources may not be used to an optimal extent [ 1 , 2 ]. For example, despite evidence that investing in primary health care is more effective than investing in specialized health care, allocations to primary care in Ghana have remained behind those allocated to tertiary care [ 3 ]. The underlying problem is that decisions on the choice of health interventions are complex and multifaceted [ 4 , 5 ], and the process is therefore ad-hoc or history-based [ 1 , 2 ]. Many criteria, or factors, play a role, and present the type of problem that behavioral decision research shows policy makers are typically quite bad at solving, unaided [ 6 , 7 ] (Figure 1 ).

Ad hoc priority setting and rational priority setting.

A first, and probably most important, criterion is the societal wish to maximize general population health . This has indeed been the basis of many national disease programs in the past century [ 8 ]. A second set of criteria relates to the distribution of health in the population. Societies may give high priority to interventions that target vulnerable population groups such as the poor [ 9 , 10 ], the severely ill [ 11 ], or children or women of reproductive age [ 12 ], because they are more deserving of health care than others [ 13 , 14 ]. Also, societies may give high priority to the economically productive people to stimulate economic growth [ 15 ], or low priority to people who require health care as a result from irresponsible behavior (e.g. smoking) [ 16 ]. A third set of criteria responds to specific societal preferences , e.g. for acute care in life threatening situations, or for curative over preventive services [ 17 ].

A fourth set of criteria relates to the budgetary and practical constraints that policy makers face when implementing interventions, including costs and availability of trained health workers [ 18 ], and may take these into account when choosing between interventions. Fifthly, political criteria may play an important role. Policy makers may not always be benevolent maximizers of social welfare, but may also act out of own (political) self-interest [ 19 ]. Interests groups in societies exercise their influence on policy makers to prioritise interventions according to their objectives, and policy makers may be sensitive to this in their efforts to maximize political support. For example, health expenditures in many developing countries are often focused on services for richer areas or groups at the expense of the poor, even where the latter offers greater scope for cost-effective healthcare [ 19 ]. Also, policy makers may follow funding preferences of (international) organisations, which may not always cohere with national priorities [ 20 – 22 ]. The above list may not be exhaustive, and still other criteria may be important.

When confronted with such complex problems, policy-makers tend to use intuitive or heuristic approaches to simplify complexity, and in the process, important information may be lost, and priority setting is ad-hoc. Or worse, they act out of political self-interest and prioritize interventions according to their own objectives. In other words, policy makers may not always well placed to make informed well-thought choices involving trade-offs of societal values [ 6 , 7 ].

The above indicates the need for a rational and transparent approach to priority setting that guides policy makers in their choice of health interventions, and that maximizes social welfare. This paper presents an overview of the approaches that have been developed over the past decades, and argues that these offer little guidance to policy makers. They concentrate on single criteria only, whereas in reality, policy makers need to make choices taking into account multiple criteria simultaneously. Moreover, they do not cover all criteria that are relevant to policy makers. In other disciplines, multi-criteria decision analysis (MCDA) is routinely used in similar problems, and we show its basic concepts and most important methods. We call for the application of MCDA in health, and present some first examples.

Rational approaches to priority setting

The past decades have witnessed the development of number of rational and transparent approaches to priority setting. Most prominent has been the development of evidence-based medicine , or the use of interventions with established effectiveness. This dates back to the beginning of the last century but was institutionalized by the foundation of the Cochrane Collaboration in 1993 [ 23 – 25 ]. The Cochrane Collaboration produces and disseminates systematic reviews of healthcare interventions and promotes the search for evidence in the form of clinical trials and other studies of interventions.

Because of steep increases in health interventions costs in western countries in the 1980's, economists proposed the use of cost-effectiveness analysis of health interventions. The underlying notion is that interventions should not only have established effectiveness, but should also be worth its costs [ 26 ]. For a certain budget, population health would then maximized by choosing interventions that show best value for money ('most cost-effective'). The World Bank promoted the concept in developing countries in 1993 [ 27 ] and recently the World Health Organization have made such information available at the regional level through the WHO-CHOICE project, e.g. on tuberculosis and HIV/AIDS control [ 28 – 30 ]. Work is underway to apply these cost-effectiveness estimates to the country level [ 31 ].

Also in the early 1990's, the World Bank expanded epidemiological mortality measures to the concept of burden of disease analysis [ 32 ]. Burden of disease analysis measures ill health in terms of morbidity and mortality to indicate the most important disease areas in a country. Its proponents consider the analysis as an important aid to priority setting as it would guide policy makers in targeting their intervention at the most important disease areas. Others argue that it lacks a conceptual basis for priority setting of health interventions, as the size of a disease problem has no relation to the potential for effective reduction [ 33 ]. Nevertheless, burden of disease analysis has been applied in many developed and developing countries including Eritrea, Kenya, Ethiopia, Uganda, and Tanzania in East Africa, Algeria, Morocco and Tunis in Northern Africa, and India [ 34 , 35 ].

With advances in population health in developing countries in the past decades, policy makers have increasingly become aware of disparities in health status between different groups in society. The past few years has witnessed an increased attention for equity analyses describing the distributional impact of interventions [ 9 – 12 ]. These studies aim to analyze to the extent interventions reach and benefit disadvantages groups, such as the poor or certain ethnicities, or otherwise vulnerable populations.

The need for multi-criteria decision analysis

However, the above approaches offer limited guidance to policy makers in their choice of interventions, for a number of reasons. Firstly, they were developed in isolation from each other, and concentrate on single criteria for priority setting – be it effectiveness, cost-effectiveness, burden of disease, or equity analysis, and do not advice on how to integrate or judge the relative importance of each criterion. In reality, policy makers need to make choices on interventions taking those criteria into account simultaneously. Moreover, criteria can easily conflict. For example, interventions targeting marginalized populations in remote areas of a country are likely to be more costly and therefore less cost-effective than those covering only people in urban areas [ 36 ]. Also, not all criteria are equally important: depending on the pro-poor stance of a country, policy makers may value interventions that target the poor more highly than those that stimulate economic growth.

Secondly, these approaches do not cover all criteria that are relevant to policy makers. For example, they are not able to capture preferences of society regarding 'the rule of rescue' in acute cure or regarding interventions related to irresponsible behavior of patients. A further complicating factor is that prioritisation decisions typically draw upon multidisciplinary knowledge bases, incorporating clinical medicine, public health, social sciences and ethics, and policy makers lack expertise to adequately interpret on all these aspects.

As a result, policy makers may not be able to utilize all available and necessary information in choosing between different interventions, and priority setting is ad-hoc (Figure 1 ). This stresses the need for the scientific development of MCDA to support priority setting, which has recently indeed been identified as one of the most important issues in health system research [ 5 ]. Baltussen and others have argued that MCDA should allow a trade-off between various criteria, and should establish the relative importance of criteria in a way that allows a rank ordering of a comprehensive set of interventions [ 4 , 37 ] (Figure 1 ). The underlying idea is that policy makers fund interventions according to this rank ordering until their budget is exhausted.

Methods of multi-criteria decision analysis

In stark contrast with the near-absence of applications of MCDA to allocation decisions in health care is the widespread acceptance and routine use of MCDA in other disciplines, e.g. to structure remedial decisions at contaminated sites in environmental sciences [ 38 ]. MCDA has also been applied in agricultural [ 39 ], energy [ 40 ], and marketing [ 41 ] sciences. In those disciplines, MCDA has evolved as a response to the observed inability of people to effectively analyze multiple streams of dissimilar information. The analysis establishes preferences between options by reference to an explicit set of objectives that the decision making body has identified, and for which it has established measurable criteria to assess the extent to which the objectives have been achieved [ 42 ]. MCDA offers a number of ways of aggregating the data on individual criteria to provide indicators of the overall performance of options.

This section outlines the main principles of MCDA, heavily drawing on standard works in those disciplines [ 42 – 45 ]. Wherever we use to term 'option' in this paper, this refers to 'intervention' in the context of priority setting in health, and the terms are used interchangeably. It first presents the performance matrix, which is a standard feature of every multi-criteria analysis. Next, it explains how the basic information in the performance matrix can be processed – either qualitatively or quantitatively.

The performance matrix

In a performance matrix, each row describes an option and each column describes the performance of the options against each criterion. The criteria are the measures of performance by which the options will be judged, and must be carefully selected, to assure completeness, feasibility, and mutual independence, and avoid redundancy and an excessive number of criteria. The individual performance assessments are often qualitative descriptions, or natural units, or sometimes a (crude) numerical scale [ 42 ]. Table 1 shows a simplified example, on the basis of the performance of a number of different interventions in regard to a set of criteria thought to be relevant in policy making. These criteria are cost-effectiveness, severity of disease, whether a disease is more among the poor, and age. As can be seen, some of these criteria are measured on a binary scale (a tick indicates a disease is more prevalent among the poor than among the rich), nominal scale (age), ordinal scales (severity of disease), or ratio scale (cost-effectiveness).

Qualitative analysis of the performance matrix

The performance matrix may be the final product of the analysis, allowing the decision maker to qualitatively rank the options. Such intuitive processing of the data can be quick and effective, but it may also lead to the use of unjustified assumptions, causing incorrect ranking of options [ 42 ]. The decision maker can come to a few types of comparisons.

Direct inspection of the performance matrix can show if any of the options are dominated by others. Dominance occurs when one option performs at least as well as another on all criteria and strictly better than the other on at least one criterion. In practice, dominance is likely to be rare, and the extent to which it can help to discriminate between many options and to support real decisions is correspondingly limited.

Subjective interpretation

Decision makers may also use the performance matrix to add recorded performance levels across the rows (options) to make some holistic judgment between options about which ones are better. However, this implies that all criteria contribute with equal importance to options' overall performance, when this has not been established. More generally, a subjective interpretation of the matrix is prone to many well-documented distortions of human judgments [ 6 , 7 ]. In marketing, this method is also called the 'pros and cons' or 'balance sheet' analysis, and is used by salespeople to gain commitment from a buyer by asking to think of the pros and cons of various alternatives [ 41 ].

Quantitative analysis of the performance matrix

In analytically more sophisticated MCDA techniques the information in the basic matrix is usually converted into consistent numerical values. The key idea is to construct scales representing preferences for the consequences, to weight the scales for their relative importance, and then to calculate weighted averages across the preference scales [ 42 ].

First, the expected consequences of each option are assigned a numerical score reflecting the strength of preference scale for each option for each criterion. More preferred options score higher on the scale, and less preferred options score lower. The scoring can be based on a value function, which translates a measure of achievement on the criterion in to a value score on the scale. Alternatively, when a commonly agreed scale of measurement does not exist, direct rating can be used and is based on the judgment of an expert simply to associate a number on that scale with the value of each option on that criterion. Or, scores can be obtained by eliciting from the decision maker a series of verbal pair wise assessments expressing a judgment of the performance of each option relative to each of the others (e.g. the Analytical Hierarchy Process does this (see below)). The scores are presented in Table 2 in normal figure.

Second, numerical weights are assigned to define, for each criterion, the relative valuations of a shift between the top and bottom of the chosen scale. Weights can be obtained by comparing weights of criterions to the most important criterion, e.g. on the basis of group discussions. In a next step, those weights are calculated to sum up to 100 in total. In the example in Table 2 , weights are presented in bold figure: 'cost-effectiveness' and 'disease of the poor' are both assigned a value of 40, and the other criteria a value of 10.

Mathematical routines then combine these two components to give an overall assessment of each option being appraised. At this stage, it is important to determine whether trade-offs between different criteria are acceptable, so that good performance on one criterion can in principle compensate for weaker performance on another. Most public decisions admit such trade-offs, but there may be some circumstances, perhaps where ethical issues are central, where trade-offs of this type are not acceptable. If it is not acceptable to consider trade-offs between criteria, then there are a limited number of non-compensatory MCA techniques available [ 42 ]. Where compensation is acceptable, and low scores on one criterion may be compensated by high scores on another, compensatory MCA techniques are used that involve aggregation of each option's performance across all the criteria to form an overall assessment of each option, on the basis of which the set of options can be compared. These techniques are usually based on multi-attribute utility theory [ 46 ]. The principal difference between the main families of MCA methods is the way in which this aggregation is done.

The simple linear additive evaluation model

If it can either be proved, or reasonably assumed, that the criteria are preferentially independent of each other, then the simple linear additive evaluation model is applicable. The linear model shows how an option's values on the many criteria can be combined into one overall value. This is done through multiplication of the value score on each criterion by the weight of that criterion, and then adding all those weighted scores together. For example, in Table 2 , antiretroviral treatment in HIV/AIDS scores 50 on the criterion 'cost-effectiveness', and the weight of that criterion is 40/100: the weighted score is then 50 * 40/100 = 20. In a similar way, the weighted scores on 'severity of disease', 'disease of the poor', and 'age' are respectively 10, 40, and 0. The weighted scores sum up to 70, which is shown in the final column. Treatment of childhood pneumonia has a total score of 100, and is therefore the preferred option, followed by antiretroviral treatment in HIV/AIDS, plastering for simple fractures (48), and inpatient care for acute schizophrenia (5).

The analytical hierarchy process

The analytic hierarchy process also develops a linear additive model, but, in its standard format, uses procedures for deriving the weights and the scores achieved by alternatives, which are based, respectively, on pair wise comparisons between criteria and between options. Thus, for example, in assessing weights, the decision maker is asked a series of questions, each of which asks how important one particular criterion is relative to another for the decision being addressed.

Outranking methods

A rather different approach depends upon the concept of outranking, and seeks to eliminate alternatives that are, in a particular sense, 'dominated'. However, unlike the straightforward dominance idea outlined above, 'outranked dominance' gives more influence to some criteria than others. One option is said to outrank another if it outperforms the other on enough criteria of sufficient importance (as reflected by the sum of the criteria weights) and is not outperformed by the other option in the sense of recording a significantly inferior performance on any one criterion. The outranking concept indirectly captures some of the political realities of decision-making, by downgrading options that perform badly on any one criterion (which might in turn activate strong lobbying from concerned parties and difficulty in implementing the option in question). In the example, in Table 1 , all interventions are outranked by 'treatment of child pneumonia', and this illustrates its low discriminative power and hence its limited potential for priority setting, especially in the context of many criteria and many interventions.

Applications to health care

To date, MCDA knows very few applications to guide resource allocation decisions in health care, in either western or developing countries. These applications have used MCDA to different extents: to only illustrate its principles, to identify the criteria for priority setting, to identify and weigh the criteria for priority setting, or more comprehensive approaches that result in a rank ordering of interventions.

James et al. [ 47 ] illustrated the principles of MCDA by demonstrating the potential impact of alternative weights for equity and efficiency criteria on the ranking of a number of hypothetical interventions.

The criteria for priority setting were identified by two merely qualitative studies in Uganda [ 4 , 48 ], including medical (e.g. effectiveness, cost-effectiveness, quality of evidence, severity of disease) and non-medical criteria (e.g. age, gender, and area of residence). Yet, they did not establish the weights of these criteria in a way that allows a rank ordering of interventions. Recently, a number of tools have been developed that take into account various criteria, but these do not explicitly attach weights to these criteria. Tugwell et al. [ 49 ] have proposed the 'equity effectiveness loop' to highlight equity issues inherent in assessing health needs, effectiveness and cost-effectiveness of interventions. The 'marginal budgeting for bottlenecks' tool aims to bridge between costing, cost-effectiveness and burden of disease analysis [ 50 ]. 'District health accounts' is a tool designed to help districts analyze their budgets and expenditures so that budgets can be set against priorities as defined by the prevailing burden of disease, and as such integrates budgeting, costing and burden of disease analysis [ 51 ]. In the Netherlands, Dunning identified a number of criteria for public reimbursement of health care. However, some of its criteria – especially medical need – were not well defined, and its application therefore suboptimal [ 52 ].

Further studies have quantified the scores and weights of criteria , but these are typically limited to two criteria only: e.g. on cost-effectiveness and equity [ 53 ], or on age and severity of illness [ 54 , 55 ].

Recently, two comprehensive MCDA approaches have been developed. Wilson et al. [ 56 ] developed a prioritization framework in an English Primary Care Trust. Through group discussion with policy makers, a number of criteria were identified (such as effectiveness, quality of life, access/equity, need, and prevention) and were weighed into four broad 'levels of importance'. Next, the groups scored four hypothetical interventions on those criteria on a scale from 0–10. A simple linear additive evaluation model was used to calculate overall scores, and interventions were rank ordered according to their 'cost-value' ratio (estimated by dividing the costs of an interventions by the overall score). The authors consider the framework as a promising tool for prioritizing interventions in the Primary Care Trust.

Baltussen et al. carried out explorative research to prioritize health interventions in Ghana and Nepal using discrete choice experiments [ 37 , 57 ]. In Ghana, criteria were identified through a series of group discussions with policy makers, and included 'cost-effectiveness', 'poverty reduction', 'age', 'severity of illness', 'budget impact' and 'burden of disease'. Intervention scores on those criteria were based on poverty profiles, burden of disease and cost-effectiveness analysis as presented in the World Health Report 2002 [ 58 ], and were expressed on a binary scale with arbitrary cut-off values. The relative weights of the various criteria were estimated through the use of discrete choice experiments (DCE) [ 59 ], with a large number of policy makers. In the DCE, respondents choose their preferred option from sets of hypothetical interventions, each consisting of a bundle of criteria that described the intervention in question, with each criterion varying over a range of scores (Figure 2 ). The criteria were constant in each scenario, but the scores that described each criterion varied across interventions. Analysis of the options chosen by respondents in each set revealed the extent to which each criterion was important. The work in Ghana showed that policy makers give high value to interventions that are cost-effective (score of 1.42), reduce poverty (1.25), target the young (0.84), or target severe diseases (0.38). Using a simple linear additive evaluation model, total scores were calculated for a set of interventions, and rank ordered accordingly: high priority interventions in Ghana were prevention of mother to child transmission in HIV/AIDS control, and treatment of pneumonia and diarrhea in childhood. Lower priority interventions were certain interventions to control blood pressure, tobacco and alcohol abuse. Full details are reported elsewhere [ 37 ].

Example of a question in a discrete choice experiment.

This paper has shown the basic principles of MCDA, and the need for its application in health. Whereas decisions in health care are often characterized by informal judgment unsupported by analysis, MCDA may be an important tool towards a more rational priority setting process.

This paper has introduced various approaches to MCDA, and these are mainly characterized by how the performance matrix is interpreted. Some approaches seem more useful to prioritise health interventions than others. First, the priority setting process involves many criteria and many interventions, and since intuitive processing of this complex data can lead to unjustified conclusions, quantitative rather than qualitative analyses seem apt. Second, compensatory rather than non-compensatory techniques seem apt as public decisions typically allow trade-offs between criteria (perhaps except in situations where ethical issues are central). Third, because of the need to rank order a large number of interventions rather than to identify a single (or small number of) dominant interventions, the linear additive model seems more suitable than the outranking method. As noted above, first experiments with the linear additive model have been carried out in Ghana and Nepal [ 37 , 55 ], and encouraging results indicate the potential of the approach to inform policy makers on actual priority setting of interventions.

This paper has illustrated the use of MCDA with some simplified examples. In a practical application, interventions may be need evaluated at different geographic coverage levels, to inform decisions on the choice between scaling up existing interventions, or implementing new interventions. WHO-CHOICE does evaluate interventions at coverage levels of 50%, 80%, and 95% for this purpose [ 60 , 61 ]. In addition, interventions may need to be evaluated not only in isolation, but also in combination, since interactions may exist between interventions in either costs and/or effects. For this reason, WHO-CHOICE does evaluate interventions in isolation and in combination [ 62 ].

The priority setting process should be strongly embedded in the organizational context, probably with a central role for an advisory panel [ 63 ]. An advisory panel comprises key stakeholders such as health personnel, policy makers, finance and information staff, and community representatives. The panel has an important role in the definition of the relevant criteria and their relative importance for priority setting, and making recommendations for reallocating resources on the basis of MCDA results. In the latter, the advisory panel may diverge from MCDA results because of e.g. pragmatic considerations. In other words, while MCDA suggests a rank ordering of interventions, this not necessarily means that interventions should be funded accordingly till the budget is exhausted. This is based on the notion that MCDA should not be seen as a formulaic or technocratic approach to priority setting, but rather as an aid to policy making.

MCDA will contribute to the fairness of the priority setting process. According to Daniels and Sabin's ethical framework of accountability for reasonableness, priority setting is said to be fair if the priority setting process, decisions and rationales are accessible and relevant; and an appeals and enforcement mechanism are established [ 64 ]. MCDA contributes to the first two conditions because of its systematic and transparent nature.

We call for a shift away from present tools for priority setting – that tend to focus on single criteria for priority setting – towards transparent and systematic approaches that take into account all relevant criteria simultaneously. Although very little work has been done so far on comprehensive MCDA approaches, a number of tools that aim to bridge the different analytical approaches are being developed. It is time to assess the current state of the art of the methods, and to stimulate the development of a new generation of more evidence-based priority setting tools.

Ham C: Priority setting in health care: learning from international experience. Health Policy 1997, 42: 49–66. 10.1016/S0168-8510(97)00054-7

Article   CAS   PubMed   Google Scholar  

Robinson R: Limits to rationality: economics, economists and priority setting. Health Policy 1999, 49: 13–26. 10.1016/S0168-8510(99)00040-8

MOH Republic of Ghana Ministry of Health: Health Sector Five-Year Programme of Work: 1997–2001. Accra, Ghana: Ministry of Health; 1998.

Google Scholar  

Kapiriri L, Norheim OF: Criteria for priority setting in health interventions in Uganda. exploration of stakeholders' values. Bull World Health Organ 2004, 82: 172–9.

PubMed Central   PubMed   Google Scholar  

Mills A, Bennett S, Bloom G, González-Block MA, Pathmanathan I: Strengthening health systems: the role and promise of policy and systems research. Alliance for Health Policy and Systems Research/Alliance for Health Policy and Systems Research (AHPSR) 2004.

McDaniels TL, Gregory RS, Fields D: Democratizing risk management: successful public involvement. Risk analysis 1999, 3: 497–51.

Bazerman MH: Judgment in Managerial Decision Making. Fourth edition. John Wiley, New York; 1998.

Cooter R, Neve M, Nutton V: History of Medicine 1988–1992. The Wellcome Series in the History of Medicine IISSN Routledge, London

Wagstaff A, Van Doorslaer E: Equity in Health Care: concepts and definitions. In Equity in the finance and delivery of health care: an international perspective . Edited by: Van Doorslaer E, Wagstaff A, Rutten F. Oxford University Press, Oxford; 1993.

Webster J, Lines J, Bruce J, Armstrong Schellenberg JR, Hanson K: Which delivery systems reach the poor? A review of equity of coverage of ever-treated nets, never-treated nets, and immunisation to reduce child mortality in Africa. Lancet Infect Dis 2005, 5 (11):709–17. 10.1016/S1473-3099(05)70269-3

Article   PubMed   Google Scholar  

Bennett S, Chanfreau C: Approaches to rationing antiretroviral treatment: ethical and equity implications. Bull World Health Organ 2005, 83: 541–7.

Olsen OE: Bridging the equity gap in maternal and child health: health systems research is needed to improve implementation. BMJ 331 (7520):844. 2005 Oct 8 10.1136/bmj.331.7520.844

Article   PubMed Central   PubMed   Google Scholar  

Daniel Wikler , (ed): Fairness and Goodness in Health. World Health Organization: Geneva; 2004.

McIntyre D, Gilson L: Redressing Disadvantage: Promoting Vertical Equity within South Africa. Health Care Analysis 2000, 8: 235–258. 10.1023/A:1009483700049

Commission on Macroeconomics and Health: Macroeconomics and health: investing in health for economic development. Boston: Center for International Development at Harvard University; 2001. [ http://www.cid.harvard.edu/cidcmh/CMHReport.pdf ]

McKie L, Laurier E, Taylor RJ, Lennox AS: Eliciting the smoker's agenda: implications for policy and practice. Soc Sci Med 2003, 56: 83–94. 10.1016/S0277-9536(02)00009-6

Wiseman V, Mooney G, Berry G, Tang KC: Involving the general public in priority setting: experiences from Australia. Soc Sci Med 2003, 56 (5):1001–12. 10.1016/S0277-9536(02)00091-6

Gericke CA, Kurowski C, Ranson MK, Mills A: Intervention complexity – a conceptual framework to inform priority-setting in health. Bull World Health Organ 2005, 83 (4):285–93.

Goddard M, Hauck K, Preker A, Smith PC: Priority setting in health – a polictical economy perspective. Health Economics Policy and Law 2006, 1: 79–90. 10.1017/S1744133105001040

Article   Google Scholar  

Lee K, Walt G: Linking national and global population agendas: case studies from eight developing countries. Third World Q 1995, 16: 257–72. 10.1080/01436599550036121

Ebrahim S, Smeeth L: Non-communicable diseases in low and middle-income countries: a priority or a distraction? Int J Epidemiol 2005, 34: 961–966. 10.1093/ije/dyi188

Ollila E: Global health priorities – priorities of the wealthy? Global Health 2005, 1 (1):6. 10.1186/1744-8603-1-6

Summerskill W: Cochrane Collaboration and the evolution of evidence. Lancet 366 (9499):1760. 2005 Nov 19 10.1016/S0140-6736(05)67710-0

Claridge JA, Fabian TC: History and development of evidence-based medicine. World J Surg 2005, 29 (5):547–53. 10.1007/s00268-005-7910-1

Niessen LW, Gijssels EW, Rutten FF: The evidence-based approach in health policy and health interventions delivery. Soc Sci Med 2000, 51 (6):859–69. 10.1016/S0277-9536(00)00066-6

Drummond M, McGuire A: Methods for the economic evaluation of health care programmes. 2nd edition. Oxford. Oxford University Press; 1997.

The World Bank World Development Report 1993: investing in Health New York Oxford University Press; 1993.

Hogan DR, Baltussen R, Hayashi C, Lauer JA, Salomon JA: Achieving the millennium development goals for health: Cost effectiveness analysis of strategies to combat HIV/AIDS in developing countries. BMJ 2005, 331: 1431–7. 10.1136/bmj.38643.368692.68

Baltussen R, Floyd K, Dye C: Achieving the millennium goals for health: Cost effectiveness analysis of strategies for tuberculosis control in developing countries. BMJ 2005, 333: 1364–1370. 10.1136/bmj.38645.660093.68

A description of the WHO-CHOICE project 2005. [ http://www.who.int/choice ]

Hutubessy R, Chisholm D, Edejer TT: Generalized cost-effectiveness analysis for national-level priority-setting in the health sector. Cost Eff Resour Alloc 1: 8. 2003 Dec 19 10.1186/1478-7547-1-8

Murray CJL, Lopez AD, (eds): The global burden of disease. Harvard School of Public Health, on behalf of the WHO and the World Bank. Cambridge: Harvard University Press; 1996.

Mooney G, Wiseman V: Burden of disease and priority setting. Health Econ 2000, 9 (5):369–72. 10.1002/1099-1050(200007)9:5<369::AID-HEC536>3.0.CO;2-D

Bobadilla JL, Cowley P, Musgrove P, Saxenian H: Design, content and financing of an essential national package of health services. Bull World Health Organ 1994, 72: 653–662.

CAS   PubMed Central   PubMed   Google Scholar  

Kapiriri L, Norheim OF, Heggenhougen K: Using burden of disease information for health planning in developing countries: the experience from Uganda. Soc Sci Med 2003, 56: 2433–2441. 10.1016/S0277-9536(02)00246-0

Evans DB, Lim S, Adam T, Tan-Torres Edejer T, for the WHO CHOICE MDG Team: Evaluation of current strategies and future priorities for improving health in developing countries. British Medical Journal doi 10.1136/bmj.38658.675243.94 (Nov 10 2005)

Baltussen R, Stolk E, Chisholm D, Aikins M: Towards a composite league table for priority setting: An application to Ghana. Health Economics 2006. DOI: 10.1002/hec.1092

Linkov I, Varghese A, Jamil S, Seager TP, Kiker G, Bridges T: Multi-criteria decision analysis: a framework for structuring remedial decisions at contaminated sites. In Comparative Risk Assessment and Environmental Decision Making. Kluwer Edited by: Linkov I, Ramadan A. 2004, 15–54.

LPNMS (Land and Plant Nutrition Management Service): MCDA – Multi-Criteria Decision Analysis techniques, using the Aspiration-Led Decision Support (ALDS) approach. 2004. [ http://www.fao.org/ag/agl/agll/infotech.htm ]

Hirschberg S, Dones R, Heck T, Burgherr P, Schenler W, Bauer C: Sustainability of Electricity Supply Technologies under German Conditions: A Comparative Evaluation. In PSI-Report No.04–15 . Paul Scherrer Institut, Villigen, Switzerland; 2004.

McDaniel C, Gates R, Gates RH, McDaniel CD Marketing Research Essentials by, John Wiley & Sons Inc; 2005.

National Economic Research Associated. Multi-criteria Analysis Manual 2005. [ http://www.odpm.gov.uk/index.asp?id=114 ]

Keeney RL, Raiffa H: Decisions with Multiple Objectives: Performances and Value Trade-Offs. Wiley, New York; 1976.

Olson D: Decision Aids for Selection Problems . Springer Verlag, New York; 1995.

Yoon KP, Hwang C-L: Multi-Attribute Decision Making, Sage, Beverley Hills . 1995.

Von Neumann J, Morgenstern O: Theory of Games and Economic Behaviour . second edition. Princeton University Press, Princeton; 1947.

James C, Carrin G, Savedoff W, Hanvoravongchai P: Clarifying efficiency-equity tradeoffs through explicit criteria, with a focus on developing countries. Health Care Anal 2005, 13 (1):33–51. 10.1007/s10728-005-2568-2

Kapiriri L, Arnesen T, Norheim OF: Is cost-effectiveness analysis preferred to severity of disease as the main guiding principle in priority setting in resource poor settings? The case of Uganda. Cost Eff Resour Alloc 2004, 2: 1. 10.1186/1478-7547-2-1

Tugwell P, de Savigny D, Hawker G, Robinson V: Applying clinical epidemiological methods to health equity: the equity effectiveness loop. BMJ 2006, 332 (7537):358–61. 10.1136/bmj.332.7537.358

Soucat A, Van Lerberghe W, Diop F: Marginal Budgeting for bottlenecks: A new costing and resource allocation practice to buy health results. The World Bank 2002.

de Savigny D, Munna G, Mbuya C, Mgalula L, Machibya H, Mkikima S, Mwageni E, Kasale H, Reid G: District Health Expenditure Mapping: A Budget Analysis Tool for Council Health Management Teams. In Tanzania Essential Health Interventions Project. Discussion Paper No. 1 . Dar es Salaam, Tanzania; 2001.

Stolk EA, Poley MJ: Criteria for determining a basic health services package. Recent developments in The Netherlands. Eur J Health Econ 2005, 6 (1):2–7. 10.1007/s10198-004-0271-0

Wagstaff A: Inequality aversion, health inequalities and health achievement. J Health Econ 2002, 21: 627–41. 10.1016/S0167-6296(02)00006-1

Stolk EA, Pickee SJ, Ament AH, Busschbach JJ: Equity in health care prioritisation: an empirical inquiry into social value. Health Policy 2005, 74 (3):343–55. 10.1016/j.healthpol.2005.01.018

Dolan P, Tsuchiya A: Health priorities and public preferences: the relative importance of past health experience and future health prospects. J Health Econ 2005, 24: 703–14. 10.1016/j.jhealeco.2004.11.007

Wilson ECF, Rees J, Fordham RJ: Developing a prioritisation framework in an English Primary Care Trust. Cost Effectiveness and Resource Allocation 2006, 4: 3. 10.1186/1478-7547-4-3

Baltussen R, ten Asbroek G, Shrestra N, Niessen L: A rational approach to priority setting: should a lung health program be implemented in Nepal? iMTA Discussion paper 2005.6 2005.

World Health Report: Reducing Risks, Promoting Healthy Life. World Health Organization, Geneva; 2002.

Ryan M, Gerard K: Using discrete choice experiments to value health care programmes: current practice and future research reflections. Applied Health Economics and Health Policy 2003, 2: 1.

Johns B, Baltussen R, Adam T, Hutubessy RCW: Programmeme costs in the economic evaluation of health interventions. Cost Eff Resour Alloc 2003, 1: 1. 10.1186/1478-7547-1-1

Johns B, Baltussen R: Accounting for the costs of scaling up health interventions. Health Economics 2004, 13: 1117–1124. 10.1002/hec.880

Evans DB, Lim SS, Adam T, Tan-Torres Edejer T, the WHO-CHOICE MDG Team: Achieving the Millennium Development Goals for health. Methods for assessing the costs and health impact of interventions. BMJ 2005.

Daniels N, Sabin J: The ethics of accountability in managed care reform. Health Aff (Milwood) 1998, 17: 50–64. 10.1377/hlthaff.17.5.50

Article   CAS   Google Scholar  

Peacock S, Ruta D, Mitton C, Donaldson C, Bate A, Murtagh M: Using economics to set pragmatic and ethical priorities. BMJ 2006, 332: 482–485. 10.1136/bmj.332.7539.482

Download references

Author information

Authors and affiliations.

Institute for Medical Technology Assessment (iMTA), ErasmusMC Rotterdam, Rotterdam, The Netherlands

Rob Baltussen & Louis Niessen

Department of Public Health, University Medical Centre Nijmegen, Nijmegen, The Netherlands

Rob Baltussen

Department of Health Policy and Management, ErasmusMC, Rotterdam, The Netherlands

Louis Niessen

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rob Baltussen .

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, rights and permissions.

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Baltussen, R., Niessen, L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Eff Resour Alloc 4 , 14 (2006). https://doi.org/10.1186/1478-7547-4-14

Download citation

Received : 01 March 2006

Accepted : 21 August 2006

Published : 21 August 2006

DOI : https://doi.org/10.1186/1478-7547-4-14

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Priority Setting
• Performance Matrix
• Discrete Choice Experiment
• Health System Research
• Priority Setting Process

Cost Effectiveness and Resource Allocation

ISSN: 1478-7547

• General enquiries: [email protected]

Keynote webinar | Spotlight on medication adherence

Half of all patients are non-adherent to their medication. Our experts explain the fundamentals with a practical discussion of how to manage non-adherence in two common scenarios: hypertension and asthma control.  Watch now on-demand

Springer Medicine

Open Access 15-05-2023 | Research article

Priority setting in the German healthcare system: results from a discrete choice experiment

Authors: V. Meusel, E. Mentzakis, P. Baji, G. Fiorentini, F. Paolucci

Published in: International Journal of Health Economics and Management | Issue 3/2023

Please log in to get access to this content

Other articles of this issue 3/2023.

Research article

Finding fraud: enforcement, detection, and recoveries after the ACA

How does the quality of care for type 2 diabetic patients benefit from gps-nurses’ teamwork a staggered difference-in-differences design based on a french pilot program, hospital cost efficiency: an examination of us acute care inpatient hospitals, impact of covid-19 on hospital screening, diagnosis and treatment activities among prostate and colorectal cancer patients in canada.

Research Article

Does the market reward quality? Evidence from India

Analyzing the effect of health reforms on the efficiency of ecuadorian public hospitals.

• Medical Journals
• Webcasts & Webinars
• CME & eLearning
• Newsletters
• Clinical Trial Summaries
• ESMO Congress 2023
• 2023 ERS Congress
• ESC Congress 2023
• Advances in Alzheimer’s
• About Springer Medicine
• Anesthesiology
• Dermatology
• Diabetology
• Endocrinology
• Gastroenterology
• Geriatrics and Gerontology
• Gynecology and Obstetrics
• Infectious Disease
• Internal Medicine
• Psychiatry and Psychosomatics
• Respiratory Medicine
• Rheumatology

Center for American Progress

Vice President Kamala Harris and Migration in the Americas: Setting the Record Straight

Vice President Kamala Harris has supported bipartisan border security solutions and has demonstrated a long-standing commitment to working with regional partners to address the root causes of irregular migration.

Restoring Social Trust in Democracy, Biden Administration, Foreign Policy, Immigration, Latin America, Mexico +3 More

Media Contact

Julia cusick.

Vice President, Communications

[email protected]

Government Affairs

Peter gordon.

Director, Federal Affairs

Jerry Parshall

Senior Director, Safety and Justice Campaign; Director, State and Local Government Affairs

Vice President Kamala Harris has shown a long-standing commitment to the rule of law and supports a bipartisan border security bill. On the other hand, anti-immigration MAGA extremists in Congress, including House Speaker Mike Johnson (R-LA), have played politics with the issue of immigration—even making up a nonexistent immigration role—but shown little interest in actually fixing the broken immigration system.

Contrary to what her detractors have long alleged, Vice President Harris was never placed in charge of the U.S.-Mexico border; rather, she has taken on a challenging task similar to the effort then-Vice President Joe Biden undertook during the later stages of the Obama-Biden administration: overseeing U.S. efforts to address the root causes of migration from El Salvador, Honduras, and Guatemala—the so-called Northern Triangle of Central America.

Stay informed on Immigration Policy

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

The Biden-Harris administration, working closely with partners across the Americas, has taken multiple steps over the past few years to address the historic levels of irregular migration in the Western Hemisphere. These layered efforts to mitigate, manage, and order migration have, in recent months, led to fewer encounters between ports of entry at the U.S.-Mexico border than there were in 2019 during the comparable time period.

Confronting a new, post-pandemic, hemisphere-wide irregular migration challenge

From 2014 to 2020, increases in irregular migration to the United States were driven primarily by displacement from the Northern Triangle countries ; migration was again on the rise in early 2021 as countries came out of peak pandemic shutdowns. As a result, the Biden-Harris administration focused its initial efforts there. Saddled with deeply unreliable partner governments in all three countries, Vice President Harris chose to emphasize cooperation with civil society and the private sector through the creation of the Partnership for Central America (PCA).

However, as decimated countries across Latin America and the Caribbean emerged from the pandemic and the U.S. economic recovery outpaced the rest of the world, two profound shifts occurred. First, the region faced never-before-seen levels of displacement, accounting for nearly one-fifth of the world’s displaced peoples despite being home to just 8 percent of the world’s population . Second, migration patterns shifted and a new set of countries —Cuba, Haiti, Nicaragua, and Venezuela—became major sources of irregular migration to the United States. The inability of the U.S. government to safely and effectively repatriate encountered migrants to those four countries exacerbated the challenge of managing such migration.

In response, the current administration’s focus evolved to leading a hemisphere-wide effort to mitigate, manage, and order migration through creating and adopting the landmark Los Angeles Declaration on Migration and Protection alongside 21 partners from across the region—notably the first time such a hemispheric-wide agreement directly involved the United States and Canada.

Partnering on economic and migration solutions across the Americas

Under the LA declaration, the Biden-Harris administration has secured unprecedented cooperation to curb irregular migration. Today, for example, Mexico is doing more to stem irregular migration to the United States than it has ever done before. The administration also created alternatives to irregular migration so that individuals fleeing Cuba, Haiti, Nicaragua, and Venezuela—the primary sources of instability in the hemisphere—could come to the United States in a sponsored, orderly fashion and be able to legally work shortly after arrival so as to contribute to U.S. prosperity and enhance stability in their home countries without delay.

Critically, the LA declaration built on successful migrant integration efforts in Latin America and spurred multiple countries to create new temporary legal status programs for migrants in order to stabilize these populations and to provide an alternative to migrating further. In May 2024, countries such as Ecuador, Colombia, and Costa Rica at the third ministerial meeting on the LA declaration announced regularization programs for various irregular migrant populations—building on the nearly 2 million Venezuelans who had already been provided status in Colombia alone. Today, more than 80 percent of those displaced in Latin America and the Caribbean have found a home in the region and have not proceeded to the U.S.-Mexico border.

Likewise, since its inception, the PCA has stimulated $5.2 billion in private sector investments designed to promote sustainable job creation and economic growth to help stabilize the populations across Central America’s Northern Triangle countries. So far in fiscal year 2024, compared with fiscal year 2021, there has been a 14 percent drop in the U.S. Customs and Border Protection average monthly encounters from Guatemala, a 39 percent drop from El Salvador, and a 50 percent drop from Honduras.

Combined with a recent executive order signed by President Biden related to asylum at the U.S.-Mexico border and the use of the statutory immigration parole authority to build new, lawful pathways , these layered efforts to mitigate, manage, and order migration have, in recent months, reduced encounters between ports of entry at the U.S.-Mexico border to below what they were during the same months in 2019 .

Achieving sustainable order at the U.S.-Mexico border—something all Americans rightly expect—is possible only through working on migration at all points along the migratory chain that extends into Latin America and well beyond. Vice President Harris’ work in northern Central America and the Biden-Harris administration’s efforts through the LA declaration have set the United States up for precisely that kind of sustained order—but only if elected leaders turn away from weaponizing immigration and work on real solutions.

The positions of American Progress, and our policy experts, are independent, and the findings and conclusions presented are those of American Progress alone. A full list of supporters is available here . American Progress would like to acknowledge the many generous supporters who make our work possible.

Dan Restrepo

Senior Fellow

Debu Gandhi

Senior Director, Immigration Policy

Joel Martinez

Senior Policy Analyst

Silva Mathema

Director, Immigration Policy

Immigration Policy

We aim to create a fair, humane, and workable immigration system in the United States through comprehensive data analysis, research, and advocacy.

Stay informed

on Immigration Policy

More From Forbes

4 ways servant leaders can inspire professional development in the workplace.

• Share to Facebook
• Share to Twitter
• Share to Linkedin

Edward DeAngelis, CEO, EDA Contractors , advocates emotional intelligence and psychological safety.

Servant leadership is very important to me. As business leaders, we strive to build natural and genuine relationships with our workforce, ideally to empower them, as people within the organization, and, in a collective sense, to demonstrate to each individual that the organization, as an entity, recognizes and appreciates…everyone.

To support employee retention and performance, leaders must make engaging with their teams a priority. Employees who feel appreciated and acknowledged may not only stay with an organization and be more productive and positive, but also improve customer experiences, increased sales and profitability for the entire organization.

It is my hope that my legacy, the legacy that I am building, will be one of servant leadership—that I built a company and, with the talents and support of my family, colleagues and remarkable team, gave rise to a community of care that positively impacted countless lives. Servant leaders have the opportunity to do that—to influence lives for the better. I can think of no more honorable opportunity as a business leader than to influence lives for the better.

In today's rapidly evolving work experience, providing employees with pathways to enhance professional development has emerged as a cornerstone of employee engagement and organizational success.

Apple iPhone 16, iPhone 16 Pro Release Date: New Report Reveals Extraordinary Strategy

‘the boys’ dethroned in amazon prime video’s top 10 list by a new show, will there be a ‘house of the dragon’ season 3 here’s what to know about the next chapter.

This is a very good thing.

Providing professional advancement opportunities can boost confidence, creativity and job satisfaction for employees. By focusing on professional development opportunities and creating career advancement pathways, employees have a roadmap to their personal and professional success within an organization. To achieve this, it's essential to set clear expectations, offer comprehensive training and create a positive work environment. These measures can improve employee performance and help them grasp their responsibilities and growth opportunities.

Here are four ways servant leaders can inspire and support employee development.

Cultivate a culture of continuous learning.

Thought leaders understand that learning doesn't end with formal education but is a lifelong journey. To cultivate a culture where curiosity and a hunger for knowledge are celebrated, leaders must foster an environment that values continuous learning, inspire employees to seek out new skills, stay abreast of industry trends and pursue opportunities for growth. Lead by example when it comes to professional development. Show commitment to your own growth and development by continuously learning from others, developing key relationships and working with coaches. Modeling a dedication to learning and self-improvement can inspire your teams to do the same.

Provide resources and mentorship.

As a thought leader, you must recognize the importance of providing employees with the resources and support they need to succeed in their professional development efforts. Invest in your employees' growth and development—whether it's offering access to online courses, hosting lunch-and-learn sessions or providing financial assistance for further education. Mentorship and coaching are powerful tools for professional development. Help employees unlock their full potential and achieve their goals by pair them with mentors who can offer guidance, support and feedback as they navigate their career paths.

Set clear expectations and collaborative goals.

Thought leaders communicate clear expectations and goals for professional development, aligning individual growth with organizational objectives. Work with employees to identify areas for improvement and create personalized development plans that support both short-term skill enhancement and long-term career advancement. Leaders can foster a culture of collaboration by creating a supportive community where employees can learn and grow together. Create opportunities for employees to collaborate on projects, share expertise and learn from one another's experiences.

Recognize achievements and empower everyone.

Thought leaders understand the importance of recognizing and celebrating employees' achievements in their professional development journey. Whether it's a promotion, completion of a certification program or mastering a new skill, take the time to acknowledge and celebrate milestones to foster a sense of accomplishment and motivation among your teams. Empowered employees who have opportunities for growth and development are more engaged and motivated. Provide opportunities for employees to take on new responsibilities and lead projects, which helps them grow and gain confidence.

Servant leaders have the opportunity to empower their workforces to thrive and succeed, one vibrant career at a time.

Servant leadership represents a commitment to shift from more traditional leadership philosophies. With servant leadership, the primary goal is to serve and empower employees. This approach, rooted in the principles of listening, empathy and stewardship, transforms the traditional power dynamics of organizations—resulting in the emergence of positive and inclusive workplaces nurtured by empathetic servant leaders committed to cultivating work environments in which all feel empowered, respected and served. At the end of the day, I find that there is no greater legacy for a business leader to aspire to achieve.

Forbes Business Council is the foremost growth and networking organization for business owners and leaders. Do I qualify?

• Editorial Standards
• Reprints & Permissions

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

What attributes should be included in a discrete choice experiment related to health technologies? A systematic literature review

Marta trapero-bertran.

1 Department of Nursing, Physiotherapy and Occupational Therapy, University of Castilla La-Mancha (UCLM), Talavera de la Reina (Toledo), Spain

2 Research Institute for Evaluation and Public Policies (IRAPP), Universitat Internacional de Catalunya (UIC), Barcelona, Spain

Beatriz Rodríguez-Martín

3 Faculty of Health Sciences, University College Dublin, Dublin, Ireland

Julio López-Bastida

Associated data.

All relevant data are within the manuscript and its Supporting Information files.

Discrete choice experiments (DCEs) are a way to assess priority-setting in health care provision. This approach allows for the evaluation of individuals’ preferences as a means of adding criteria to traditional quality-adjusted life year analysis. The aim of this systematic literature review was to identify attributes for designing a DCE in order to then develop and validate a framework that supports decision-making on health technologies. Our systematic literature review replicated the methods and search terms used by de Bekker-Grob et al. 2012 and Clark et al. 2014. The Medline database was searched for articles dated between 2008 and 2015. The search was limited to studies in English that reflected general preferences and were choice-based, published as full-text articles and related to health technologies. This study included 72 papers, 52% of which focused on DCEs on drug treatments. The average number of attributes used in all included DCE studies was 5.74 (SD 1.98). The most frequently used attributes in these DCEs were improvements in health (78%), side effects (57%) and cost of treatment (53%). Other, less frequently used attributes included waiting time for treatment or duration of treatment (25%), severity of disease (7%) and value for money (4%). The attributes identified might inform future DCE surveys designed to study societal preferences regarding health technologies in order to better inform decisions in health technology assessment.

Introduction

In health care systems around the world, decision-makers are faced with competing demands and insufficient resources, even in the wealthiest countries. In these circumstances, it is not possible to provide all available and potentially beneficial health care to those who could benefit from it, and priority-setting is therefore needed. Policy-makers should take into account the views of the general population in setting health priorities, as is done in the United Kingdom [ 1 ]. Public involvement in health care decision-making should be a policy objective, although there is an absence of empirical evidence on how society might value different health interventions [ 2 ].

There is substantial literature on the different methods available to engage the public in health care decision-making [ 3 – 5 ]. As noted by Whitty et al. [ 6 ], Ryan and colleagues provide a comprehensive systematic review and comparative assessment of the methods that can be used to elicit public preferences for health care [ 7 ], concluding that “there is no single, best method to gain public opinion”. Nevertheless, they do make recommendations regarding the appropriateness of selected qualitative and quantitative techniques. Two of their preferred methods, the citizens’ jury and discrete choice experiments (DCEs), have been gaining prominence in the health literature in recent years [ 6 , 8 – 12 ]. Each is associated with a number of features that make them particularly attractive for public engagement [ 6 ], rendering them worthy of further consideration. The DCE approach was chosen for this systematic literature review because it was aimed at informing a pilot study using a discrete choice experiment to quantify individual preferences regarding the use of public funding for orphan drugs.

The DCE has become a useful instrument for quantifying preferences related to health care priority-setting [ 6 , 8 , 13 – 16 ]. It has been used to (1) measure preferences regarding a health care service, (2) measure preferences regarding the distribution of health care within a population and (3) to assess preferences for the funding of health care [ 17 – 19 ]. Although use of the method by policy-makers has not yet become widespread, it has been applied to elicit social preferences for health care funded with public money [ 20 ].

A DCE survey can be administered relatively easily to a large, randomly selected representative sample of the population [ 7 ]. It is arguably a less resource-intensive method of community engagement than many other approaches, although resources and costs would likely be high for large sample sizes. A DCE measures not only the direction of preferences around a topic (e.g., Should health gains attributed to young children be weighted more heavily than those attributed to older people?), but also the relative strength of preferences for one policy choice alternative compared with another (e.g., How much extra weight should be attributed to young children?), as well as the trade-offs that respondents would be willing to make between different characteristics of that choice. The usefulness of most preference-based approaches (including DCEs) may be limited when the respondents represent what might be called a naïve sample of the general public–that is, one comprising individuals who lack personal knowledge or experience on the issue, and thus little weight can be given to the results [ 7 , 21 ].

A DCE provides a different way–compared with other approaches, such as small‐scale discussions or focus groups–to assess priority-setting based on the valuation of some attributes. DCEs are based on the assumption that health care interventions, services or policies can be described by their attributes and that an individual’s valuation depends upon the levels of these attributes. In a DCE, respondents are asked to choose between two or more alternatives. The resulting choices reveal an underlying utility function. For example, the DCE approach allows for the evaluation of individuals’ preferences for adding criteria to traditional quality-adjusted life year (QALY) analysis. The DCE approach also facilitates greater knowledge of the relative importance of the various attributes and the trade-offs that individuals are willing to make between these attributes.

This type of research and its applications are crucial for identifying the current impact of new health technologies on health and economics and, therefore, for assessing their effectiveness. DCEs also afford an opportunity to assess societal preferences. This type of research could serve as the basis for an integrated and harmonized approach to assessing public policies on new health technologies in the European Union.

De Bekker Grob et al. [ 8 ] published a recent review on preferences of consumers, patients and health professionals for all types of health care resources. They focused on the experimental design of DCEs, estimation procedures, the validity of responses and the definition of the attributes and their respective levels that should be used for DCEs on health technologies. The attributes found in their review were monetary measure, time, risk, health status domain, health care, and other. No further description of the attributes was given, so there were no well-defined inputs to be used to design a DCE for a particular context. Clark et al. [ 22 ] published a more recent DCE review. This paper updated the paper by de Bekker Grob et al. [ 8 ] and explored trends in DCEs used in health economics. It concluded that the use of DCEs in health care continues to grow dramatically across a broad range of countries. Thus, DCE results may be influencing decisions in a wider range of geographical settings. Little description and detail regarding attributes and their respective levels were provided for inclusion in future DCE exercises. There have been several literature reviews of DCEs in health care in general (such as de Bekker Grob et al. and Clark et al.) [ 8 , 22 ], but not of DCEs in health technology assessment (HTA). Decisions regarding HTA also involve public resources, however, and it is therefore important to establish approaches for prioritizing health technology resources. Accordingly, determining the attributes that should be considered in DCEs to inform HTA decisions should be a current research concern.

Materials and methods

This systematic literature review was conducted using the search terms and methods used in two recent published systematic reviews on DCEs covering the periods 2001–2008 [ 8 ] and 2009–2015 [ 22 ]. These methods involved the use of the Medline Ovid database to identify DCE studies on health care or health economics. These studies used the same search terms used by Ryan and Gerard [ 23 ], reflecting the different terms applied to refer to DCEs. The search terms included were “discrete choice experiment(s)”, “discrete choice model(l)ing”, “stated preference”, “part-worth utilities”, “functional measurement”, “paired comparisons”, “pairwise choices”, “conjoint analysis”, “conjoint measurement”, “conjoint studies” and “conjoint choice experiment(s)”.

In this study, the same database used in de Bekker Grob et al. [ 8 ] and Clark et al. [ 22 ] was used to search for articles published from January 2008 to December 2015. The same key words were also used. Papers in English and Spanish were retrieved, although the search terms used were in English only. Any paper explaining a DCE on health technologies was included. Review papers were excluded from the analysis but kept for the discussion section of this paper. De Bekker-Grob et al. [ 8 ] and Clark et al. [ 22 ] included studies that were choice-based and published as full-text articles and that applied to health care or health economics in general. Our review focused on health technologies and thus had a more limited scope. The search was extensive with respect to health care and health economics in general, but only papers related to health technologies were included in our systematic review. The objective was to evaluate DCEs on health technologies that reflected the preferences of patients, policy-makers, providers and the general public. Papers were excluded if they had the following characteristics: (a) they were duplicates; (b) they were not choice- or preference-based or they merely provided measurements but no descriptions of attributes; (c) neither the full text nor an abstract was found; (d) they did not apply to health technologies or to rural areas of developing countries; and (e) they did not involve human respondents. Grey literature was also searched using Google Scholar, although unfortunately no results were found. Each abstract and paper selected was carefully peer-reviewed, and data extraction was systematically and independently performed by two researchers. Whenever there was a discrepancy, papers were reviewed a second time to reach a consensus. Excel was used to summarize the results of this systematic literature review. A data extraction form included questions on the following: background (e.g., quartile of impact factor); sampling and sample characteristics (e.g., illness of respondents); general design of the DCE (e.g., number of attributes and description of attributes and levels covered); experimental design (e.g., method for creating choice sets); design validity (e.g., estimation procedure, model); and qualitative methods for enhancing the DCE process and results (e.g., pretesting of the DCE questionnaire). However, only the general design and experimental design features were presented in the results and discussion sections. Considerations relating to design validity and qualitative methods used to enhance the DCE process were beyond the scope of this paper.

The specific details of the template were dynamically adjusted during the piloting process, which included the revision of a few papers. Data were extracted in free-text form with no limitations on the number of free-text fields and as little categorization of data as possible to avoid the loss of detailed information. Descriptive analysis was undertaken to describe the most common attributes used and their corresponding levels. The attributes and levels for the DCE questionnaire on HTA are presented in a summary table. The table shows the attributes found in this literature review and the attributes identified in the previously published systematic literature reviews [ 8 , 22 ].

To assess the methodological quality of the systematic literature review, the PRISMA checklist was used [ 24 ]. In addition, a DCE quality assessment tool [ 25 ] was used to assess the validity of the studies and their attributes and levels.

This approach was devised following the guidance of Mandeville et al. [ 25 ], who covered all four key stages of a DCE (choice task design; experimental design; conduct; and analysis) using a list of 13 criteria drawn from an earlier study [ 26 ]. The authors assessed whether each criterion for each study was met. If the criterion was met, it was indicated with a green colour. If there was insufficient information to judge whether a criterion was met, then an orange colour was used. A red colour indicated that the criterion was not met. This type of qualitative analysis is important for validating the results from this systematic literature review.

Fig 1 shows the flowchart for the identification of studies, with reasons for exclusion.

Overall, the search strategy identified 384 titles (after duplicates were excluded) from a pool of studies with the potential for inclusion in this review. Based on the abstracts, 160 papers were ordered and manually reviewed. Of these 160 articles, 72 were included in this study [ 14 , 27 – 96 ]. S1 Table , included in the supplemental material, provides details of the PRISMA checklist used to assess the methodological quality of the systematic literature reviews.

The number of studies published on DCEs on general preferences regarding health technologies over time is as follows: 18 articles (25%) were published before 2010, 30 articles (41.6%) appeared between 2010 and 2011, and the remaining 21 articles (29%) were published between 2012 and 2013. The years with the greatest research output were 2011 and 2012 (16 articles and 15 articles published, respectively). The average sample used across the 72 studies included 299 individuals with a mean age of 59.6 years; an average of 44.88% of the respondents were female. In seven studies, only the age interval was reported; in those cases, the average of the age interval was taken. The largest number of the DCEs identified were conducted in the Netherlands, although significant numbers were also conducted in the United Kingdom, Canada, Germany and the United States. The 72 papers covered 30 different diseases, such as chronic obstructive pulmonary disease (COPD), depression and hepatitis B. The most studied preferences related to cancer (26%), followed by attention deficit disorder (4%) and osteoporosis (4%). Only one paper [ 61 ] was found that examined preferences relating to orphan drugs for rare diseases. S2 Table , included in the supplemental material, provides more detailed information about the health technologies, the attributes and the levels for each paper.

Fig 2 presents the validity assessment for all included studies. Overall, while the choice task design and the experimental design of the studies were more robust than expected, there were significant weaknesses with regard to conduct and analysis of the studies. In terms of choice task, attributes and levels were identified through qualitative work with the target population in 24.8% of the studies. In 16% of the studies, there was no opt-out or status quo option, nor any justification of a forced choice for the attributes selected. In terms of the experimental design, 27% of papers did not have a design that was optimal or statistically efficient. However, most of the relevant problems with the validity of the papers pertained to the pilot testing conducted among the target population and the lack of a pooled analysis from different subgroups: in 42% of the papers the authors did not conduct a pilot test to inform the design of the questionnaire, and 63% did not include any pooled analysis from different population subgroups. In an assessment of the validity of experimental design and analytic approach, it is necessary to examine current practices in DCEs.

Most of the studies (53%) assessed societal preferences regarding pharmacological treatments; the rest focused on medical devices. Of the latter, 17% related to diagnostic technologies and 26% to therapeutic technologies (specifically, 13% were on assistive technology devices used directly by patients, 5% were on medical devices to assist medical professionals, and 4% were on artificial body parts implanted through medical procedures). The average number of attributes was 5.74 (SD 1.98), with a minimum value of 2 and a maximum value of 12. Each attribute had an average of 3.26 levels (SD 1.11), with a minimum value of 1 and a maximum value of 18. The six most common attributes used in the DCEs (n = 72) were (a) improvement in health (78%), (b) side effects (57%), (c) costs (53%), (d) waiting time for treatment or duration of treatment (25%), (e) severity of disease (7%), and (f) value for money (4%). When the focus was only on papers that assessed preferences in relation to drugs (n = 36), the relative importance of the attributes remained the same: improvement in health (55.56%), costs (50%), adverse events (41.67%), and mode of administration (22.22%), followed by discomfort and pain (16.67%), treatment duration (19.44%) and waiting time (2.78%). These attributes reflected the general preferences of several groups, including patients, the general public patients and other stakeholders. The majority of the papers (n = 60) referred to patients’ preferences. In terms of attributes revealing preferences related to efficiency, availability of other treatments and value for money were considered relevant attributes [ 14 ]. Availability of other treatments refers to the existence of alternative treatments for the same disease. Value for money refers to how efficiently resources are used (e.g., doctor time, hospital beds, drugs) in the national health system and is based on the relationship between the cost of treatment and the health benefits it provides. Hence, although these two attributes were not the most commonly used, they were also considered for inclusion in the DCE survey.

In terms of levels, the papers reviewed most commonly referred to the following levels of administration: oral, subcutaneous, intravenous or injection. These papers also referred to the following levels of pain or discomfort: none, mild, moderate or severe. Therefore, the terms mild, moderate, and severe were adopted to describe the levels for as many attributes as possible. See Table 1 for details on the most used attributes and their respective levels. This table also includes the attributes and levels described in the two previously published systematic reviews [ 8 , 22 ]. Both studies highlight the monetary measure and the time- and health care-related attributes as the most frequent ones to be considered in a DCE.

Only 10 papers (13.8%) offered partial access to the questionnaire used to carry out the DCE and the specific questions formulated for respondents. The rest of the papers did not offer access to the survey. The most common means of administering the survey was a self-report questionnaire (35.29%); questionnaires were administered through an interviewer in 7.35% of cases or a computer in 8.82%. The rest of the papers did not report the mode of survey administration.

Regarding experimental design, 43% (n = 31) of the studies had a fractional factorial design, whereas 13% (n = 10) had a full factorial design. The remaining 27 studies did not report the type of DCE design. Thirty-one studies measured main effects only, whereas seven studies reported main effects with 2-way interactions. In 32% of the DCEs reviewed, the effects evaluated were not reported. More than half of the papers (55%) used orthogonal arrays to create choice sets, while 14% used D-efficiency methods. Five papers combined two methods: either orthogonal arrays with D-efficiency or other methods. Three papers used other methods to create choice sets, and 18 did not explain the method used. In terms of the estimation procedure, the multinomial logit was the most common model used to analyse DCE preferences (22%), followed by random effects logit (13%) and logit (12%).

This study fills a gap in systematic reviews of the literature aimed at identifying the most relevant attributes and levels for measuring public preferences regarding health technologies by means of a DCE. Six attributes were identified from de Bekker Grob et al. (2012) [ 8 ] and Clark et al. (2014) [ 22 ]: health status domain, monetary measure, time, risk, health care and other, some of which were too subjective to build questions for a DCE survey. No levels were defined or detailed in these papers and no additional information was given concerning the definitions of these attributes. In contrast, our literature review found the following attributes: improvement in health, side effects, cost (price) of treatment, waiting time for treatment or treatment duration, severity of disease and value for money. The attributes included in the previous systematic reviews are too wide and general to understand. No definitions were provided by the authors, so it was difficult to evaluate the complementarity between the results of the two systematic literature reviews, even though they used the same literature review methods. It will be important for future research to describe these attributes and their respective levels in as detailed a manner as possible, so that they can be applied with no uncertainties regarding what is encompassed in their definition. In addition, a complete description will be helpful in providing information for the design of future DCEs on HTA. Because public preferences might change greatly over time, depending on current situations worldwide, it was decided to incorporate papers published between 2008 and 2015 –i.e., a period of 7 years. Regarding the optimum number of attributes to include in a DCE, Marshal et al. [ 97 ] identified and described recent applications of conjoint analysis to determine what combination of a limited number of attributes was most influential on respondent choice or decision-making. In their review, they found that most surveys included 6 attributes, with the number ranging from 3 to 16. Therefore, it seems that a larger number of attributes should be used to better capture the criteria on which people base their preferences related to health care. However, many attributes make the decision task more difficult and hence render the outcomes less reliable. The number of attributes found in this systematic review–six–seems a sensible and adequate number to be potentially included in a DCE.

Orphan drugs are unlikely to be efficient (provide value for money) due to the high price paid for often modest effectiveness. It is important to identify all appropriate criteria that will help in the “correct” evaluation of the potential impact and benefit generated in society. Unfortunately, only one paper [ 61 ] was found that studied preferences relating to orphan drugs for rare diseases. The authors investigated public preferences regarding public funding for orphan drugs used to treat both rare and common diseases, using a convenience sample of university students. They found that when all other variables were held constant, the respondents did not prefer to have the government spend more for orphan drugs used to treat rare diseases and that they weighted the relevant attributes of coverage decisions similarly for both rare and common diseases. More DCEs on orphan drugs should be conducted to generate more evidence on the particular attributes and levels for this kind of drug.

The inclusion of either cost or improvement in health and value for money as attributes helps to capture the preferences of respondents, although it could lead to double-counting. None of the papers found in this systematic review included either combination of those attributes; however, it is important to be aware of the potential for double-counting that can occur as a result of the inclusion of such similar attributes.

Although there were significant weaknesses in terms of the validity assessment of the included studies, important and essential issues–such as no overlap between the attributes, the use of unidimensional attributes in the questionnaires, the use of the correct target population and the appropriate use of an econometric model for the choice task design–were common characteristics for most of the studies. Hence, despite some weaknesses regarding validity, the most important criteria for these types of studies were included overall.

DCEs have been previously used in other published studies to gain insight into the criteria that were important for decision-makers in health care priority-setting [ 14 , 40 , 52 , 64 , 65 ]. These five papers were included in this literature review. The rest of the papers (n = 67) focused on patient preferences. The type of attributes used in the papers that focused on policy-makers’ opinions were quite different from those that sought to identify patients’ preferences. For instance, six intervention-related attributes were included in a paper [ 73 ] that measured the preferences of policy-makers and other health professionals, including disease severity, budget impact, cost-effectiveness (incremental cost per QALY and number of QALYs gained per patient), uncertainty regarding the probability of doubling costs per QALY, national savings in costs related to absence from work per year and the composition of the health gain. Disease severity and cost of treatment are included in both literature reviews, as is health improvement; however, questions related to cost-effectiveness or national savings when the target audience includes patients are more difficult to ask. For that reason, availability of other treatments and value for money were included as relevant attributes. Attributes might differ, depending on the survey target audience. In this case, all audiences have been included.

Subsequent research is needed to further develop DCE attributes and levels for various specific technologies and diseases. One possible approach might be to investigate in a more in-depth manner the methods that led to the selection and identification of attributes in DCE studies (e.g., focus groups, interviews, literature, expert opinion), which could then be informative for future DCEs. Another approach might be to conduct DCEs for different diseases among different types of audiences to assess and validate attributes and thus help to inform future priority-setting decisions.

Conclusions

This systematic literature review was performed to identify the attributes that may better help decision-makers and patients to identify the criteria leading to decisions about health technologies. This study revealed that attributes such as improvements in health, treatment side effects, treatment cost (price), waiting time for treatment or treatment duration, severity of disease and value for money can be considered to better capture and describe societal preferences in relation to HTA. This topic is of interest for preference practitioners, as it can help them, first, to build the best survey on health technology and, then, to aid public decision-makers in identifying the treatments that should be implemented or funded, in accordance with the population’s preferences.

Supporting information

Funding statement.

This research is funded by the European Commission’s FP7 Framework Programme and is undertaken under the auspices of Advance-HTA (Grant number 305983). The results presented here reflect the author’s views and not the views of the European Commission. The European Comission is not liable for any use of the information communicated. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Data Availability

• SI SWIMSUIT
• SI SPORTSBOOK

5-Star WR Kaliq Lockett Breaks Down Longhorns Interest Ahead of Commitment

Matt galatzan | 2 hours ago.

• Texas Longhorns

The Texas Longhorns have a big stretch of recruiting coming up, with multiple five-star targets set to make their decisions before the end of the month.

The first prospect of that group to make their decision will be five-star Sasche (TX) wide receiver Kaliq Lockett , who will pick between Texas, Texas A&M, Alabama, LSU, and Florida State on Wednesday.

Ahead of his announcement, Lockett sat with On3 recruiting expert Steve Wiltfong, to discuss his contenders, and what exactly he is looking for at the next level.

“Really just an extension of my family,” Lockett said to Wiltfong. “I’m really big on family. It’s going to be hard to leave them. I wanted the school to look after me and take care of me and I want the coaches to basically be my parents but not my parents if you get what I’m saying. My plan is to go three years and done. The school that I chose is the best way to be one of the best receivers in the country and to be a first-round receiver.”

In other words, Lockett wants to stay close to home and his family, and develop into a top-line NFL receiver prospect.

On paper, those are all things areas in which the Longhorns excel.

In terms of NFL talent, the Longhorns had three receivers taken in this spring's NFL Draft Xavier Worthy, Adonai Mitchell, and Jordan Whittington, and they look primed to put more in the league next spring as well. As for proximity to his family, Austin is just over a three-hour drive from his home.

Lockett sees that in the Horns as well, and it could give them the advantage heading into his Wednesday decision.

”I like Coach Sark’s offensive mastermind style of offense," Lockett told Wiltfong. "I like Coach Jackson’s perspective as an NFL receiver coach coming from the NFL to college and translating his NFL perspective to college. I like how Texas is a little bit more close (to home) than everybody else. And also my past dealings with Texas.”

As it stands, Lockett ranks as the No. 17 player nationally, and No. 3 wide receiver in the 2025 class, and had 59 receptions for 1,299 yards and 13 touchdowns last season as a junior for Sasche. He also had 29 catches for 492 yards and five touchdowns in his sophomore season, and earned snaps as a freshman on the varsity squad.

Texas currently stands as the favorite to land Lockett, with a score of 83.6 percent per the On3 Recruiting Prediction Machine.

MATT GALATZAN

Matt Galatzan is the Publisher of LonghornsCountry.com, AllAggies.com, and the Managing Editor of BuckeyesNow.com and TheGroveReport.com He is also the Editor-In-Chief of RamDigest.com and TexansDaily.com. Galatzan graduated from the University of Mississippi, where he studied integrated marketing communications, with minors in journalism and business administration.  Galatzan started in the sports journalism industry under Mike Fisher at DallasBasketball.com in 2014, which at the time was part of the 247Sports network. He also spent two years covering the SMU Mustangs for PonyStampede.com on the 247Sports network.  When DallasBasketball.com and CowboysCountry.com moved over to Sports Illustrated's Fan Nation network in 2020, Galatzan followed suit, eventually being taking over as the Managing Editor and Publisher of LonghornsCountry.com and AllAggies.com a year later.  Through the years, Galatzan has conducted a handful of high-profile one-on-one interviews to add to his resume — in both writing and podcasting. Some of his biggest interviews have been with Mavs owner Mark Cuban, former Longhorns players Dan Neil and Phil Dawson, and many other recruits, and current/former players for each of the teams he has covered.  Galatzan is also a full-time employee in the digital media department for Audacy and KRLD FM's 105.3 The Fan in Dallas, which is the official radio home of the Dallas Cowboys.  You can find Galatzan on all major social media channels, including Twitter on @MattGalatzan For any inquiries, please email [email protected]

Follow MattGalatzan

Portland’s sweeping overhaul of government, elections nears. No one knows what will unfold

• Updated: Aug. 05, 2024, 10:00 a.m.
• | Published: Aug. 04, 2024, 6:00 a.m.

Portland City Hall is undergoing renovations as officials prepare to accommodate an expanded City Council and new administrative structure that are part of a sweeping set of government changes approved by voters in 2022. Beth Nakamura

• Shane Dixon Kavanaugh | The Oregonian/OregonLive

Come early January, after months of painstaking work and an historic election of a new mayor and City Council, a first full view of Portland’s radically transformed government and political power structures will emerge inside the council’s chambers at City Hall.

A dozen seats — not five — will dominate the dais, enough for an expanded legislative body. Its members will hail from four separate geographic districts rather than the city at large and be tasked with policy making and constituent services. At the peak of the u-shaped rostrum will sit a council president endowed with powers that observers say could be on par with the mayor.

Stories by Shane Dixon Kavanaugh

• After repeated setbacks, Oregon hires Measure 110 program director
• Portland pays Water Bureau director $141,000 to resign
• Portland watchdog walks back lobbying violation against homeless services nonprofit Urban Alchemy
• Portland explores repealing standalone arts tax, pushing pricey new ‘parks and arts’ levy in 2025
• After months of roadblocks, Multnomah County chair will allow vote on critic’s ambulance staffing plan

If you purchase a product or register for an account through a link on our site, we may receive compensation. By using this site, you consent to our User Agreement and agree that your clicks, interactions, and personal information may be collected, recorded, and/or stored by us and social media and other third-party partners in accordance with our Privacy Policy.

• Our Mission

Setting the Tone for the First Lesson of the Year

To start the academic year off well, approach your high school students in ways that make them feel seen and create excitement about the subject matter.

There are few lessons as special as the first lesson of a new academic year. While success in teaching happens in the aggregate—and is rarely derailed by any single lesson, even those we feel could have gone better—the very first lesson feels a little different. This is because it is different. 

The first lesson is an opportunity to welcome our students to our classrooms and to share with them the excitement (and trepidation) of a new academic year. It’s an opportunity to get to know our students and for them to get to know us. We set the tone for the year by beginning to build the routines and habits that shape our classrooms. What we do in this first lesson reverberates in all of those that follow. 

In this article, I’ll outline four of my best strategies for getting the most out of your first lesson. Whether you’re teaching an entirely new class for the first time or rejoining a class that you taught last year, these strategies are sure to set you and your students up for success. 

1. Meet Them at the Door

Make sure that you’re waiting at the door of the classroom to greet your new students . As they arrive, say hello to each student personally, asking for their name if you haven’t taught the class before. Direct them to head into the classroom and to find their desk, which you should label beforehand.  

There are a few reasons for doing this: 

• Most obviously and fundamentally, it gives you the opportunity to acknowledge each student personally. It tells them that they matter to you and creates, from the first second, an inclusive and welcoming space. They are seen and they matter. 
• If you’re asking for their name, it gives students the chance to tell you how they prefer to be known. Often, a name as it appears on a register is not the student’s preference. Jot down what they say on the class roster (which I have with me on a clipboard) so that you can begin to address them in the manner they prefer as soon as the lesson begins. 
• A more practical reason for meeting students at the door is to control the flow of entry into the classroom. It means the door doesn’t become a bottleneck and students will enter, one by one, in a calm and orderly fashion. 

2. Give Students Something to Think About 

If you’re waiting at the door for students to enter, this means you’re not in the classroom teaching. Make sure that there is something on the desk for students to complete. It should be self-explanatory and self-contained so students don’t feel the need to get back up and ask you a question. While they’re working on this, you can continue to greet students as they arrive. 

My preference for this first task is either a retrieval activity from the last academic year (so long as I am confident that they will know it) or a list of some books I read over summer, with space for students to write down what they read and/or what they most enjoyed about your subject in the previous year. 

3. Make Names a Priority 

Aiming to learn the names of your students as quickly as possible is always one of my top priorities. It helps to build a strong relationship with the class, but also, just as important, it enables a whole host of routines I rely on in my teaching. It is hard to cold call if you don’t know the name of the person you wish to ask. 

The important work of learning names begins in the first lesson. Keep your clipboard with you during the lesson (with their preferred names now added), and make a conscious effort to refer to it during questioning. Let students know this is what you’re doing and why you’re doing it. Invite them to correct you anytime you get a name wrong. At the end of the first lesson, I always make a big deal of moving around the room and trying to name every student correctly. I don’t always get this right, but it’s surprising to find how quickly you can do it. 

4. Make the Subject Matter 

As well as sending a signal that each student matters as an individual, you also want them to understand that your subject matters. It’s why you’re all gathered together, after all. Therefore, make sure that you always begin with something that’s academically substantive. 

In my own subject of English, I always begin by teaching a poem. It is discrete and exemplifies many of the skills that characterize the overall discipline; I’m able to build a quick snapshot of the class’s current ability in order to inform future planning; and it allows plenty of scope for rich and interesting discussion.  

During the discussion, continue to systematically use and rehearse student names, but also begin to introduce students to high-leverage routines you know you’re likely to use a lot: turn and talk, cold call, mini-whiteboards. As you do it for the first time, narrate what you’re doing and how you’d like the routine to work in all future lessons. 

The first lesson matters, and you can ensure that it does in all the right ways. Set the tone you want to permeate all the lessons that will follow it: inclusive and purposeful, welcoming and rigorous.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Criteria for priority setting of HIV/AIDS interventions in Thailand: a discrete choice experiment

Affiliation.

• 1 Nijmegen International Center for Health Systems Research and Education (NICHE), Department of Primary and Community Care, Radboud University Nijmegen Medical Center, Nijmegen, The Netherlands. [email protected]
• PMID: 20609244
• PMCID: PMC2912896
• DOI: 10.1186/1472-6963-10-197

Background: Although a sizeable budget is available for HIV/AIDS control in Thailand, there will never be enough resources to implement every programme for all target groups at full scale. As such, there is a need to prioritize HIV/AIDS programmes. However, as of yet, there is no evidence on the criteria that should guide the priority setting of HIV/AIDS programmes in Thailand, including their relative importance. Also, it is not clear whether different stakeholders share similar preferences.

Methods: Criteria for priority setting of HIV/AIDS interventions in Thailand were identified in group discussions with policy makers, people living with HIV/AIDS (PLWHA), and community members (i.e. village health volunteers (VHVs)). On the basis of these, discrete choice experiments were designed and administered among 28 policy makers, 74 PLWHA, and 50 VHVs.

Results: In order of importance, policy makers expressed a preference for interventions that are highly effective, that are preventive of nature (as compared to care and treatment), that are based on strong scientific evidence, that target high risk groups (as compared to teenagers, adults, or children), and that target both genders (rather than only men or women). PLWHA and VHVs had similar preferences but the former group expressed a strong preference for care and treatment for AIDS patients.

Conclusions: The study has identified criteria for priority setting of HIV/AIDS interventions in Thailand, and revealed that different stakeholders have different preferences vis-à-vis these criteria. This could be used for a broad ranking of interventions, and as such as a basis for more detailed priority setting, taking into account also qualitative criteria.

PubMed Disclaimer

Similar articles

• Criteria for prioritization of HIV programs in Viet Nam: a discrete choice experiment. Safarnejad A, Pavlova M, Son VH, Phuong HL, Groot W. Safarnejad A, et al. BMC Health Serv Res. 2017 Nov 13;17(1):719. doi: 10.1186/s12913-017-2679-0. BMC Health Serv Res. 2017. PMID: 29132355 Free PMC article.
• Multi-criteria decision analysis for setting priorities on HIV/AIDS interventions in Thailand. Youngkong S, Teerawattananon Y, Tantivess S, Baltussen R. Youngkong S, et al. Health Res Policy Syst. 2012 Feb 17;10:6. doi: 10.1186/1478-4505-10-6. Health Res Policy Syst. 2012. PMID: 22339981 Free PMC article.
• IMPORTANCE OF MULTIPLE CRITERIA FOR PRIORITY SETTING OF HIV/AIDS INTERVENTIONS. Tromp N, Prawiranegara R, Siregar A, Sunjaya D, Baltussen R. Tromp N, et al. Int J Technol Assess Health Care. 2015;31(6):390-8. doi: 10.1017/S0266462316000039. Int J Technol Assess Health Care. 2015. PMID: 26961721 Free PMC article.
• Human immunodeficiency virus infection and AIDS in east Africa: challenges and possibilities for prevention and control. Mhalu FS, Lyamuya E. Mhalu FS, et al. East Afr Med J. 1996 Jan;73(1):13-9. East Afr Med J. 1996. PMID: 8625855 Review.
• Cost-effectiveness of HIV/AIDS interventions in Africa: a systematic review of the evidence. Creese A, Floyd K, Alban A, Guinness L. Creese A, et al. Lancet. 2002 May 11;359(9318):1635-43. doi: 10.1016/S0140-6736(02)08595-1. Lancet. 2002. PMID: 12020523 Review.
• Stated-preference research in HIV: A scoping review. Humphrey JM, Naanyu V, MacDonald KR, Wools-Kaloustian K, Zimet GD. Humphrey JM, et al. PLoS One. 2019 Oct 30;14(10):e0224566. doi: 10.1371/journal.pone.0224566. eCollection 2019. PLoS One. 2019. PMID: 31665153 Free PMC article. Review.
• Developing attributes and levels for a discrete choice experiment on basic health insurance in Iran. Kazemi Karyani A, Rashidian A, Akbari Sari A, Emamgholipour Sefiddashti S. Kazemi Karyani A, et al. Med J Islam Repub Iran. 2018 Mar 28;32:26. doi: 10.14196/mjiri.32.26. eCollection 2018. Med J Islam Repub Iran. 2018. PMID: 30159277 Free PMC article.
• Application of discrete choice experiments to enhance stakeholder engagement as a strategy for advancing implementation: a systematic review. Salloum RG, Shenkman EA, Louviere JJ, Chambers DA. Salloum RG, et al. Implement Sci. 2017 Nov 23;12(1):140. doi: 10.1186/s13012-017-0675-8. Implement Sci. 2017. PMID: 29169397 Free PMC article. Review.
• Developing attributes for discrete choice experiments in health: a systematic literature review and case study of alcohol misuse interventions. Helter TM, Boehler CE. Helter TM, et al. J Subst Use. 2016 Nov 1;21(6):662-668. doi: 10.3109/14659891.2015.1118563. Epub 2016 Mar 10. J Subst Use. 2016. PMID: 27695386 Free PMC article. Review.
• United Nations Development Programme. Thailand's response to HIV/AIDS: Progress and Challenges. Bangkok: United Nations Development Programme; 2004.
• UNAIDS/WHO working group on global HIV/AIDS and STI. Epidemiological fact sheet on HIV and AIDS: Core data on epidemiology and response. Geneva: World Health Organization; 2008.
• The National Committee for HIV and AIDS Prevention and Alleviation. The National Plan for Strategic and Integrated HIV and AIDS Prevention and Alleviation 2007 - 2011: Key contents. The Agricultural Co-operative Federation of Thailand; 2007.
• Phoolchareon W. Evolution of Thailand's strategy to cope with the HIV/AIDS epidemic. Food, Nutrition and Agriculture. 2005;34:16–23.
• World Health Organization. Scaling up antiretroviral treatment: Lessons learnt from Thailand. New Delhi, India: World Health Organization, Regional Office for South-East Asia; 2007.

Publication types

• Search in MeSH

Related information

• PubChem Compound
• PubChem Substance

LinkOut - more resources

Full text sources.

• BioMed Central
• Europe PubMed Central
• PubMed Central
• Genetic Alliance
• MedlinePlus Health Information

Research Materials

• NCI CPTC Antibody Characterization Program

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.