conflicts of interest literature review

Jump to navigation

Cochrane Training

Chapter 7: considering bias and conflicts of interest among the included studies.

Isabelle Boutron, Matthew J Page, Julian PT Higgins, Douglas G Altman, Andreas Lundh, Asbjørn Hróbjartsson; on behalf of the Cochrane Bias Methods Group

Key Points:

• Review authors should seek to minimize bias. We draw a distinction between two places in which bias should be considered. The first is in the results of the individual studies included in a systematic review. The second is in the result of the meta-analysis (or other synthesis) of findings from the included studies.
• Problems with the design and execution of individual studies of healthcare interventions raise questions about the internal validity of their findings; empirical evidence provides support for this concern.
• An assessment of the internal validity of studies included in a Cochrane Review should emphasize the risk of bias in their results, that is, the risk that they will over-estimate or under-estimate the true intervention effect.
• Results of meta-analyses (or other syntheses) across studies may additionally be affected by bias due to the absence of results from studies that should have been included in the synthesis.
• Review authors should consider source of funding and conflicts of interest of authors of the study, which may inform the exploration of directness and heterogeneity of study results, assessment of risk of bias within studies, and assessment of risk of bias in syntheses owing to missing results.

Cite this chapter as: Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. Available from www.training.cochrane.org/handbook .

7.1 Introduction

Cochrane Reviews seek to minimize bias. We define bias as a systematic error , or deviation from the truth, in results. Biases can lead to under-estimation or over-estimation of the true intervention effect and can vary in magnitude: some are small (and trivial compared with the observed effect) and some are substantial (so that an apparent finding may be due entirely to bias). A source of bias may even vary in direction across studies. For example, bias due to a particular design flaw such as lack of allocation sequence concealment may lead to under-estimation of an effect in one study but over-estimation in another (Jüni et al 2001).

Bias can arise because of the actions of primary study investigators or because of the actions of review authors, or may be unavoidable due to constraints on how research can be undertaken in practice. Actions of authors can, in turn, be influenced by conflicts of interest. In this chapter we introduce issues of bias in the context of a Cochrane Review, covering both biases in the results of included studies and biases in the results of a synthesis. We introduce the general principles of assessing the risk that bias may be present, as well as the presentation of such assessments and their incorporation into analyses. Finally, we address how source of funding and conflicts of interest of study authors may impact on study design, conduct and reporting. Conflicts of interest held by review authors are also of concern; these should be addressed using editorial procedures and are not covered by this chapter (see Chapter 1, Section 1.3 ).

We draw a distinction between two places in which bias should be considered. The first is in the results of the individual studies included in a systematic review . Since the conclusions drawn in a review depend on the results of the included studies, if these results are biased, then a meta-analysis of the studies will produce a misleading conclusion. Therefore, review authors should systematically take into account risk of bias in results of included studies when interpreting the results of their review.

The second place in which bias should be considered is the result of the meta-analysis (or other synthesis) of findings from the included studies . This result will be affected by biases in the included studies, and may additionally be affected by bias due to the absence of results from studies that should have been included in the synthesis. Specifically, the conclusions of the review may be compromised when decisions about how, when and where to report results of eligible studies are influenced by the nature and direction of the results. This is the problem of ‘non-reporting bias’ (also described as ‘publication bias’ and ‘selective reporting bias’). There is convincing evidence that results that are statistically non-significant and unfavourable to the experimental intervention are less likely to be published than statistically significant results, and hence are less easily identified by systematic reviews (see Section 7.2.3 ). This leads to results being missing systematically from syntheses, which can lead to syntheses over-estimating or under-estimating the effects of an intervention. For this reason, the assessment of risk of bias due to missing results is another essential component of a Cochrane Review.

Both the risk of bias in included studies and risk of bias due to missing results may be influenced by conflicts of interest of study investigators or funders . For example, investigators with a financial interest in showing that a particular drug works may exclude participants who did not respond favourably to the drug from the analysis, or fail to report unfavourable results of the drug in a manuscript.

Further discussion of assessing risk of bias in the results of an individual randomized trial is available in Chapter 8 , and of a non-randomized study in Chapter 25 . Further discussion of assessing risk of bias due to missing results is available in Chapter 13 .

7.1.1 Why consider risk of bias?

There is good empirical evidence that particular features of the design, conduct and analysis of randomized trials lead to bias on average, and that some results of randomized trials are suppressed from dissemination because of their nature. However, it is usually impossible to know to what extent biases have affected the results of a particular study or analysis (Savović et al 2012). For these reasons, it is more appropriate to consider whether a result is at risk of bias rather than claiming with certainty that it is biased. Most recent tools for assessing the internal validity of findings from quantitative studies in health now focus on risk of bias, whereas previous tools targeted the broader notion of ‘methodological quality’ (see also Section 7.1.2 ).

Bias should not be confused with imprecision . Bias refers to systematic error , meaning that multiple replications of the same study would reach the wrong answer on average. Imprecision refers to random error , meaning that multiple replications of the same study will produce different effect estimates because of sampling variation, but would give the right answer on average. Precision depends on the number of participants and (for dichotomous outcomes) the number of events in a study, and is reflected in the confidence interval around the intervention effect estimate from each study. The results of smaller studies are subject to greater sampling variation and hence are less precise. A small trial may be at low risk of bias yet its result may be estimated very imprecisely, with a wide confidence interval. Conversely, the results of a large trial may be precise (narrow confidence interval) but also at a high risk of bias.

Bias should also not be confused with the external validity of a study, that is, the extent to which the results of a study can be generalized to other populations and settings. For example, a study may enrol participants who are not representative of the population who most commonly experience a particular clinical condition. The results of this study may have limited generalizability to the wider population, but will not necessarily give a biased estimate of the effect in the highly specific population on which it is based. Factors influencing the applicability of an included study to the review question are covered in Chapter 14 and Chapter 15 .

7.1.2 From quality scales to domain-based tools

Critical assessment of included studies has long been an important component of a systematic review or meta-analysis, and methods have evolved greatly over time. Early appraisal tools were structured as quality ‘scales’, which combined information on several features into a single score. However, this approach was questioned after it was revealed that the type of quality scale used could significantly influence the interpretation of the meta-analysis results (Jüni et al 1999). That is, risk ratios of trials deemed ‘high quality’ by some scales suggested that the experimental intervention was superior, whereas when trials were deemed ‘high quality’ by other scales, the opposite was the case. The lack of a theoretical framework underlying the concept of ‘quality’ assessed by these scales resulted in tools mixing different concepts such as risk of bias, imprecision, relevance, applicability, ethics, and completeness of reporting. Furthermore, the summary score combining these components is difficult to interpret (Jüni et al 2001).

In 2008, Cochrane released the Cochrane risk-of-bias (RoB) tool, which was slightly revised in 2011 (Higgins et al 2011). The tool was built on the following key principles:

• The tool focused on a single concept: risk of bias. It did not consider other concepts such as the quality of reporting, precision (the extent to which results are free of random errors), or external validity (directness, applicability or generalizability).
• The tool was based on a domain-based (or component) approach, in which different types of bias are considered in turn. Users were asked to assess seven domains: random sequence generation, allocation sequence concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting, and other sources of bias. There was no scoring system in the tool.
• The domains were selected to characterize mechanisms through which bias may be introduced into a trial, based on a combination of theoretical considerations and empirical evidence.
• The assessment of risk of bias required judgement and should thus be completely transparent. Review authors provided a judgement for each domain, rated as ‘low’, ‘high’ or ‘unclear’ risk of bias, and provided reasons to support their judgement.

This tool has been implemented widely both in Cochrane Reviews and non-Cochrane reviews (Jørgensen et al 2016). However, user testing has raised some concerns related to the modest inter-rater reliability of some domains (Hartling et al 2013), the need to rethink the theoretical background of the ‘selective outcome reporting’ domain (Page and Higgins 2016), the misuse of the ‘other sources of bias’ domain (Jørgensen et al 2016), and the lack of appropriate consideration of the risk-of-bias assessment in the analyses and interpretation of results (Hopewell et al 2013).

To address these concerns, a new version of the Cochrane risk-of-bias tool, RoB 2, has been developed, and this should be used for all randomized trials in Cochrane Reviews ( MECIR Box 7.1.a ). The tool, described in Chapter 8 , includes important innovations in the assessment of risk of bias in randomized trials. The structure of the tool is similar to that of the ROBINS-I tool for non-randomized studies of interventions (described in Chapter 25 ). Both tools include a fixed set of bias domains, which are intended to cover all issues that might lead to a risk of bias. To help reach risk-of-bias judgements, a series of ‘signalling questions’ are included within each domain. Also, the assessment is typically specific to a particular result. This is because the risk of bias may differ depending on how an outcome is measured and how the data for the outcome are analysed. For example, if two analyses for a single outcome are presented, one adjusted for baseline prognostic factors and the other not, then the risk of bias in the two results may be different. 

MECIR Box 7.1.a Relevant expectations for conduct of intervention reviews

7.2 Empirical evidence of bias

Where possible, assessments of risk of bias in a systematic review should be informed by evidence. The following sections summarize some of the key evidence about bias that informs our guidance on risk-of-bias assessments in Cochrane Reviews.

7.2.1 Empirical evidence of bias in randomized trials: meta-epidemiologic studies

Many empirical studies have shown that methodological features of the design, conduct and reporting of studies are associated with biased intervention effect estimates. This evidence is mainly based on meta-epidemiologic studies using a large collection of meta-analyses to investigate the association between a reported methodological characteristic and intervention effect estimates in randomized trials. The first meta-epidemiologic study was published in 1995. It showed exaggerated intervention effect estimates when intervention allocation methods were inadequate or unclear and when trials were not described as double-blinded (Schulz et al 1995). These results were subsequently confirmed in several meta-epidemiologic studies, showing that lack of reporting of adequate random sequence generation, allocation sequence concealment, double blinding and more specifically blinding of outcome assessors tend to yield higher intervention effect estimates on average (Dechartres et al 2016a, Page et al 2016).

Evidence from meta-epidemiologic studies suggests that the influence of methodological characteristics such as lack of blinding and inadequate allocation sequence concealment varies by the type of outcome. For example, the extent of over-estimation is larger when the outcome is subjectively measured (e.g. pain) and therefore likely to be influenced by knowledge of the intervention received, and lower when the outcome is objectively measured (e.g. death) and therefore unlikely to be influenced by knowledge of the intervention received (Wood et al 2008, Savović et al 2012).

7.2.2 Trial characteristics explored in meta-epidemiologic studies that are not considered sources of bias

Researchers have also explored the influence of other trial characteristics that are not typically considered a threat to a direct causal inference for intervention effect estimates. Recent meta-epidemiologic studies have shown that effect estimates were lower in prospectively registered trials compared with trials not registered or registered retrospectively (Dechartres et al 2016b, Odutayo et al 2017). Others have shown an association between sample size and effect estimates, with larger effects observed in smaller trials (Dechartres et al 2013). Studies have also shown a consistent association between intervention effect and single or multiple centre status, with single-centre trials showing larger effect estimates, even after controlling for sample size (Dechartres et al 2011).

In some of these cases, plausible bias mechanisms can be hypothesized. For example, both the number of centres and sample size may be associated with intervention effect estimates because of non-reporting bias (e.g. single-centre studies and small studies may be more likely to be published when they have larger, statistically significant effects than when they have smaller, non-significant effects); or single-centre and small studies may be subject to less stringent controls and checks. However, alternative explanations are possible, such as differences in factors relating to external validity (e.g. participants in small, single-centre trials may be more homogenous than participants in other trials). Because of this, these factors are not directly captured by the risk-of-bias tools recommended by Cochrane. Review authors should record these characteristics systematically for each study included in the systematic review (e.g. in the ‘Characteristics of included studies’ table) where appropriate. For example, trial registration status should be recorded for all randomized trials identified.

7.2.3 Empirical evidence of non-reporting biases

A list of the key types of non-reporting biases is provided in Table 7.2.a . In the sections that follow, we provide some of the evidence that underlies this list.

Table 7.2.a Definitions of some types of non-reporting biases

7.2.3.1 Selective publication of study reports

There is convincing evidence that the publication of a study report is influenced by the nature and direction of its results (Chan et al 2014). Direct empirical evidence of such selective publication (or ‘publication bias’) is obtained from analysing a cohort of studies in which there is a full accounting of what is published and unpublished (Franco et al 2014). Schmucker and colleagues analysed the proportion of published studies in 39 cohorts (including 5112 studies identified from research ethics committees and 12,660 studies identified from trials registers) (Schmucker et al 2014). Only half of the studies were published, and studies with statistically significant results were more likely to be published than those with non-significant results (odds ratio (OR) 2.8; 95% confidence interval (CI) 2.2 to 3.5) (Schmucker et al 2014). Similar findings were observed by Scherer and colleagues, who conducted a systematic review of 425 studies that explored subsequent full publication of research initially presented at biomedical conferences (Scherer et al 2018). Only 37% of the 307,028 abstracts presented at conferences were published later in full (60% for randomized trials), and abstracts with statistically significant results in favour of the experimental intervention (versus results in favour of the comparator intervention) were more likely to be published in full (OR 1.17; 95% CI 1.07 to 1.28) (Scherer et al 2018). By examining a cohort of 164 trials submitted to the FDA for regulatory approval, Rising and colleagues found that trials with favourable results were more likely than those with unfavourable results to be published (OR 4.7; 95% CI 1.33 to 17.1) (Rising et al 2008).

In addition to being more likely than unpublished randomized trials to have statistically significant results, published trials also tend to report larger effect estimates in favour of the experimental intervention than trials disseminated elsewhere (e.g. in conference abstracts, theses, books or government reports) (ratio of odds ratios 0.90; 95% CI 0.82 to 0.98) (Dechartres et al 2018). This bias has been observed in studies in many scientific disciplines, including the medical, biological, physical and social sciences (Polanin et al 2016, Fanelli et al 2017).

7.2.3.2 Other types of selective dissemination of study reports

The length of time between completion of a study and publication of its results can be influenced by the nature and direction of the study results (‘time-lag bias’). Several studies suggest that randomized trials with results that favour the experimental intervention are published in journals about one year earlier on average than trials with unfavourable results (Hopewell et al 2007, Urrutia et al 2016).

Investigators working in a non-English speaking country may publish some of their work in local, non-English language journals, which may not be indexed in the major biomedical databases (‘language bias’). It has long been assumed that investigators are more likely to publish positive studies in English-language journals than in local, non-English language journals (Morrison et al 2012). Contrary to this belief, Dechartres and colleagues identified larger intervention effects in randomized trials published in a language other than English than in English (ratio of odds ratios 0.86; 95% CI 0.78 to 0.95), which the authors hypothesized may be related to the higher risk of bias observed in the non-English language trials (Dechartres et al 2018). Several studies have found that in most cases there were no major differences between summary estimates of meta-analyses restricted to English-language studies compared with meta-analyses including studies in languages other than English (Morrison et al 2012, Dechartres et al 2018).

The number of times a study report is cited appears to be influenced by the nature and direction of its results (‘citation bias’). In a meta-analysis of 21 methodological studies, Duyx and colleagues observed that articles with statistically significant results were cited 1.57 times the rate of articles with non-significant results (rate ratio 1.57; 95% CI 1.34 to 1.83) (Duyx et al 2017). They also found that articles with results in a positive direction (regardless of their statistical significance) were cited at 2.14 times the rate of articles with results in a negative direction (rate ratio 2.14; 95% CI 1.29 to 3.56) (Duyx et al 2017). In an analysis of 33,355 studies across all areas of science, Fanelli and colleagues found that the number of citations received by a study was positively correlated with the magnitude of effects reported (Fanelli et al 2017). If positive studies are more likely to be cited, they may be more likely to be located, and thus more likely to be included in a systematic review.

Investigators may report the results of their study across multiple publications; for example, Blümle and colleagues found that of 807 studies approved by a research ethics committee in Germany from 2000 to 2002, 135 (17%) had more than one corresponding publication (Blümle et al 2014). Evidence suggests that studies with statistically significant results or larger treatment effects are more likely to lead to multiple publications (‘multiple (duplicate) publication bias’) (Easterbrook et al 1991, Tramèr et al 1997), which makes it more likely that they will be located and included in a meta-analysis.

Research suggests that the accessibility or level of indexing of journals is associated with effect estimates in trials (‘location bias’). For example, a study of 61 meta-analyses found that trials published in journals indexed in Embase but not MEDLINE yielded smaller effect estimates than trials indexed in MEDLINE (ratio of odds ratios 0.71; 95% CI 0.56 to 0.90); however, the risk of bias due to not searching Embase may be minor, given the lower prevalence of Embase-unique trials (Sampson et al 2003). Also, Moher and colleagues estimate that 18,000 biomedical research studies are tucked away in ‘predatory’ journals, which actively solicit manuscripts and charge publications fees without providing robust editorial services (such as peer review and archiving or indexing of articles) (Moher et al 2017). The direction of bias associated with non-inclusion of studies published in predatory journals depends on whether they are publishing valid studies with null results or studies whose results are biased towards finding an effect.

7.2.3.3 Selective dissemination of study results

The need to compress a substantial amount of information into a few journal pages, along with a desire for the most noteworthy findings to be published, can lead to omission from publication of results for some outcomes because of the nature and direction of the findings. Particular results may not be reported at all ( ‘selective non-reporting of results’ ) or be reported incompletely ( ‘selective under-reporting of results’ , e.g. stating only that “P>0.05” rather than providing summary statistics or an effect estimate and measure of precision) (Kirkham et al 2010). In such instances, the data necessary to include the results in a meta-analysis are unavailable. Excluding such studies from the synthesis ignores the information that no significant difference was found, and biases the synthesis towards finding a difference (Schmid 2016).

Evidence of selective non-reporting and under-reporting of results in randomized trials has been obtained by comparing what was pre-specified in a trial protocol with what is available in the final trial report. In two landmark studies, Chan and colleagues found that results were not reported for at least one benefit outcome in 71% of randomized trials in one cohort (Chan et al 2004a) and 88% in another (Chan et al 2004b). Results were under-reported (e.g. stating only that “P>0.05”) for at least one benefit outcome in 92% of randomized trials in one cohort and 96% in another. Statistically significant results for benefit outcomes were twice as likely as non-significant results to be completely reported (range of odds ratios 2.4 to 2.7) (Chan et al 2004a, Chan et al 2004b). Reviews of studies investigating selective non-reporting and under-reporting of results suggest that it is more common for outcomes defined by trialists as secondary rather than primary (Jones et al 2015, Li et al 2018).

Selective non-reporting and under-reporting of results occurs for both benefit and harm outcomes. Examining the studies included in a sample of 283 Cochrane Reviews, Kirkham and colleagues suspected that 50% of 712 studies with results missing for the primary benefit outcome of the review were missing because of the nature of the results (Kirkham et al 2010). This estimate was slightly higher (63%) in 393 studies with results missing for the primary harm outcome of 322 systematic reviews (Saini et al 2014).

7.3 General procedures for risk-of-bias assessment

7.3.1 collecting information for assessment of risk of bias.

Information for assessing the risk of bias can be found in several sources, including published articles, trials registers, protocols, clinical study reports (i.e. documents prepared by pharmaceutical companies, which provide extensive detail on trial methods and results), and regulatory reviews (see also Chapter 5, Section 5.2 ).

Published articles are the most frequently used source of information for assessing risk of bias. This source is theoretically very valuable because it has been reviewed by editors and peer reviewers, who ideally will have prompted authors to report their methods transparently. However, the completeness of reporting of published articles is, in general, quite poor, and essential information for assessing risk of bias is frequently missing. For example, across 20,920 randomized trials included in 2001 Cochrane Reviews, the percentage of trials at unclear risk of bias was 49% for random sequence generation, 57% for allocation sequence concealment; 31% for blinding and 25% for incomplete outcome data (Dechartres et al 2017). Nevertheless, more recent trials were less likely to be judged at unclear risk of bias, suggesting that reporting is improving over time (Dechartres et al 2017).

Trials registers can be a useful source of information to obtain results of studies that have not yet been published (Riveros et al 2013). However, registers typically report only limited information about methods used in the trial to inform an assessment of risk of bias (Wieseler et al 2012). Protocols, which outline the objectives, design, methodology, statistical consideration and procedural aspects of a clinical study, may provide more detailed information on the methods used than that provided in the results report of a study. They are increasingly being published or made available by journals who publish the final report of a study. Protocols are also available in some trials registers, particularly ClinicalTrials.gov (Zarin et al 2016), on websites dedicated to data sharing such as ClinicalStudyDataRequest.com , or from drug regulatory authorities such as the European Medicines Agency. Clinical study reports are another highly useful source of information (Wieseler et al 2012, Jefferson et al 2014).

It may be necessary to contact study investigators to request access to the trial protocol, to clarify incompletely reported information or understand discrepant information available in different sources. To reduce the risk that study authors provide overly positive answers to questions about study design and conduct, we suggest review authors use open-ended questions. For example, to obtain information about the randomization process, review authors might consider asking: ‘What process did you use to assign each participant to an intervention?’ To obtain information about blinding of participants, it might be useful to request something like, ‘Please describe any measures used to ensure that trial participants were unaware of the intervention to which they were assigned’. More focused questions can then be asked to clarify remaining uncertainties.

7.3.2 Performing assessments of risk of bias   

Risk-of-bias assessments in Cochrane Reviews should be performed independently by at least two people ( MECIR Box 7.3.a ). Doing so can minimize errors in assessments and ensure that the judgement is not influenced by a single person’s preconceptions. Review authors should also define in advance the process for resolving disagreements. For example, both assessors may attempt to resolve disagreements via discussion, and if that fails, call on another author to adjudicate the final judgement. Review authors assessing risk of bias should have either content or methodological expertise (or both), and an adequate understanding of the relevant methodological issues addressed by the risk-of-bias tool. There is some evidence that intensive, standardized training may significantly improve the reliability of risk-of-bias assessments (da Costa et al 2017). To improve reliability of assessments, a review team could consider piloting the risk-of-bias tool on a sample of articles. This may help ensure that criteria are applied consistently and that consensus can be reached. Three to six papers should provide a suitable sample for this. We do not recommend the use of statistical measures of agreement (such as kappa statistics ) to describe the extent to which assessments by multiple authors were the same. It is more important that reasons for any disagreement are explored and resolved.

MECIR Box 7.3.a Relevant expectations for conduct of intervention reviews

The process for reaching risk-of-bias judgements should be transparent. In other words, readers should be able to discern why a particular result was rated at low risk of bias and why another was rated at high risk of bias. This can be achieved by review authors providing information in risk-of-bias tables to justify the judgement made. Such information may include direct quotes from study reports that articulate which methods were used, and an explanation for why such a method is flawed. Cochrane Review authors are expected to record the source of information (including the precise location within a document) that informed each risk-of-bias judgement ( MECIR Box 7.3.b ).

MECIR Box 7.3.b Relevant expectations for conduct of intervention reviews

Many results are often available in trial reports, so review authors should think carefully about which results to assess for risk of bias. Review authors should assess risk of bias in results for outcomes that are included in the ‘Summary of findings’ table ( MECIR Box 7.1.a ). Such tables typically include seven or fewer patient-important outcomes (for more details on constructing a ‘Summary of findings’ table, see Chapter 14 ).

Novel methods for assessing risk of bias are emerging, including machine learning systems designed to semi-automate risk-of-bias assessment (Marshall et al 2016, Millard et al 2016). These methods involve using a sample of previous risk-of-bias assessments to train machine learning models to predict risk of bias from PDFs of study reports, and extract supporting text for the judgements. Some of these approaches showed good performance for identifying relevant sentences to identify information pertinent to risk of bias from the full-text content of research articles describing clinical trials. A study showed that about one-third of articles could be assessed by just one reviewer if such a tool is used instead of the two required reviewers (Millard et al 2016). However, reliability in reaching judgements about risk of bias compared with human reviewers was slight to moderate depending on the domain assessed (Gates et al 2018).

7.4 Presentation of assessment of risk of bias

Risk-of-bias assessments may be presented in a Cochrane Review in various ways. A full risk-of-bias table includes responses to each signalling question within each domain (see Chapter 8, Section 8.2 ) and risk-of-bias judgements, along with text to support each judgement. Such full tables are lengthy and are unlikely to be of great interest to readers, so should generally not be included in the main body of the review. It is nevertheless good practice to make these full tables available for reference.

We recommend the use of forest plots that present risk-of-bias judgements alongside the results of each study included in a meta-analysis (see Figure 7.4.a ). This will give a visual impression of the relative contributions of the studies at different levels of risk of bias, especially when considered in combination with the weight given to each study. This may assist authors in reaching overall conclusions about the risk of bias of the synthesized result, as discussed in Section 7.6 . Optionally, forest plots or other tables or graphs can be ordered (stratified) by judgements on each risk-of-bias domain or by the overall risk-of-bias judgement for each result.

Review authors may wish to generate bar graphs illustrating the relative contributions of studies with each of risk-of-bias judgement (low risk of bias, some concerns, and high risk of bias). When dividing up a bar into three regions for this purpose, it is preferable to determine the regions according to statistical information (e.g. precision, or weight in a meta-analysis) arising from studies in each category, rather than according to the number of studies in each category.

Figure 7.4.a Forest plot displaying RoB 2 risk-of-bias judgements for each randomized trial included in a meta-analysis of mental health first aid (MHFA) knowledge scores. Adapted from Morgan et al (2018).

7.5 Summary assessments of risk of bias

Review authors should make explicit summary judgements about the risk of bias for important results both within studies and across studies (see MECIR Box 7.5.a ). The tools currently recommended by Cochrane for assessing risk of bias within included studies (RoB 2 and ROBINS-I) produce an overall judgement of risk of bias for the result being assessed. These overall judgements are derived from assessments of individual bias domains as described, for example, in Chapter 8, Section 8.2 .

To summarize risk of bias across study results in a synthesis, review authors should follow guidance for assessing certainty in the body of evidence (e.g. using GRADE), as described in Chapter 14, Section 14.2.2 . When a meta-analysis is dominated by study results at high risk of bias, the certainty of the body of evidence may be rated as being lower than if such studies were excluded from the meta-analysis. Section 7.6 discusses some possible courses of action that may be preferable to retaining such studies in the synthesis.

MECIR Box 7.5.a Relevant expectations for conduct of intervention reviews

7.6 Incorporating assessment of risk of bias into analyses

7.6.1 introduction.

When performing and presenting meta-analyses, review authors should address risk of bias in the results of included studies ( MECIR Box 7.6.a ). It is not appropriate to present analyses and interpretations while ignoring flaws identified during the assessment of risk of bias. In this section we present suitable strategies for addressing risk of bias in results from studies included in a meta-analysis, either in order to understand the impact of bias or to determine a suitable estimate of intervention effect (Section 7.6.2 ). For the latter, decisions often involve a trade-off between bias and precision. A meta-analysis that includes all eligible studies may produce a result with high precision (narrow confidence interval) but be seriously biased because of flaws in the conduct of some of the studies. However, including only the studies at low risk of bias in all domains assessed may produce a result that is unbiased but imprecise (if there are only a few studies at low risk of bias).

MECIR Box 7.6.a Relevant expectations for conduct of intervention reviews

7.6.2 Including risk-of-bias assessments in analyses

Broadly speaking, studies at high risk of bias should be given reduced weight in meta-analyses compared with studies at low risk of bias. However, methodological approaches for weighting studies according to their risk of bias are not sufficiently well developed that they can currently be recommended for use in Cochrane Reviews.

When risks of bias vary across studies in a meta-analysis, four broad strategies are available to incorporate assessments into the analysis. The choice of strategy will influence which result to present as the main finding for a particular outcome (e.g. in the Abstract). The intended strategy should be described in the protocol for the review.

(1) Primary analysis restricted to studies at low risk of bias

The first approach involves restricting the primary analysis to studies judged to be at low risk of bias overall. Review authors who restrict their primary analysis in this way are encouraged to perform sensitivity analyses to show how conclusions might be affected if studies at a high risk of bias were included.

(2) Present multiple (stratified) analyses

Stratifying according to the overall risk of bias will produce multiple estimates of the intervention effect: for example, one based on all studies, one based on studies at low risk of bias, and one based on studies at high risk of bias. Two or more such estimates might be considered with equal prominence (e.g. the first and second of these). However, presenting the results in this way may be confusing for readers. In particular, people who need to make a decision usually require a single estimate of effect. Furthermore, ‘Summary of findings’ tables typically present only a single result for each outcome. On the other hand, a stratified forest plot presents all the information transparently. Though we would generally recommend stratification is done on the basis of overall risk of bias, review authors may choose to conduct subgroup analyses based on specific bias domains (e.g. risk of bias arising from the randomization process).

Formal comparisons of intervention effects according to risk of bias can be done with a test for differences across subgroups (e.g. comparing studies at high risk of bias with studies at low risk of bias), or by using meta-regression (for more details see Chapter 10, Section 10.11.4 ). However, review authors should be cautious in planning and carrying out such analyses, because an individual review may not have enough studies in each category of risk of bias to identify meaningful differences. Lack of a statistically significant difference between studies at high and low risk of bias should not be interpreted as absence of bias, because these analyses typically have low power.

The choice between strategies (1) and (2) should be based to large extent on the balance between the potential for bias and the loss of precision when studies at high or unclear risk of bias are excluded.

(3) Present all studies and provide a narrative discussion of risk of bias

The simplest approach to incorporating risk-of-bias assessments in results is to present an estimated intervention effect based on all available studies, together with a description of the risk of bias in individual domains, or a description of the summary risk of bias, across studies. This is the only feasible option when all studies are at the same risk of bias. However, when studies have different risks of bias, we discourage such an approach for two reasons. First, detailed descriptions of risk of bias in the Results section, together with a cautious interpretation in the Discussion section, will often be lost in the Authors’ conclusions, Abstract and ‘Summary of findings’ table, so that the final interpretation ignores the risk of bias and decisions continue to be based, at least in part, on compromised evidence. Second, such an analysis fails to down-weight studies at high risk of bias and so will lead to an overall intervention effect that is too precise, as well as being potentially biased.

When the primary analysis is based on all studies, summary assessments of risk of bias should be incorporated into explicit measures of the certainty of evidence for each important outcome, for example, by using the GRADE system (Guyatt et al 2008). This incorporation can help to ensure that judgements about the risk of bias, as well as other factors affecting the quality of evidence, such as imprecision, heterogeneity and publication bias, are considered appropriately when interpreting the results of the review (see Chapter 14 and Chapter 15 ).

(4) Adjust effect estimates for bias

A final, more sophisticated, option is to adjust the result from each study in an attempt to remove the bias. Adjustments are usually undertaken within a Bayesian framework, with assumptions about the size of the bias and its uncertainty being expressed through prior distributions (see Chapter 10, Section 10.13 ). Prior distributions may be based on expert opinion or on meta-epidemiological findings (Turner et al 2009, Welton et al 2009). The approach is increasingly used in decision making, where adjustments can additionally be made for applicability of the evidence to the decision at hand. However, we do not encourage use of bias adjustments in the context of a Cochrane Review because the assumptions required are strong, limited methodological expertise is available, and it is not possible to account for issues of applicability due to the diverse intended audiences for Cochrane Reviews. The approach might be entertained as a sensitivity analysis in some situations.

7.7 Considering risk of bias due to missing results

The 2011 Cochrane risk-of-bias tool for randomized trials encouraged a study-level judgement about whether there has been selective reporting, in general, of the trial results. As noted in Section 7.2.3.3 , selective reporting can arise in several ways: (1) selective non-reporting of results, where results for some of the analysed outcomes are selectively omitted from a published report; (2) selective under-reporting of data, where results for some outcomes are selectively reported with inadequate detail for the data to be included in a meta-analysis; and (3) bias in selection of the reported result, where a result has been selected for reporting by the study authors, on the basis of the results, from multiple measurements or analyses that have been generated for the outcome domain (Page and Higgins 2016).

The RoB 2 and ROBINS-I tools focus solely on risk of bias as it pertains to a specific trial result. With respect to selective reporting, RoB 2 and ROBINS-I examine whether a specific result from the trial is likely to have been selected from multiple possible results on the basis of the findings (scenario 3 above). Guidance on assessing the risk of bias in selection of the reported result is available in Chapter 8 (for randomized trials) and Chapter 25 (for non-randomized studies of interventions).

If there is no result (i.e. it has been omitted selectively from the report or under-reported), then a risk-of-bias assessment at the level of the study result is not applicable. Selective non-reporting of results and selective under-reporting of data are therefore not covered by the RoB 2 and ROBINS-I tools. Instead, selective non-reporting of results and under-reporting of data should be assessed at the level of the synthesis across studies. Both practices lead to a situation similar to that when an entire study report is unavailable because of the nature of the results (also known as publication bias). Regardless of whether an entire study report or only a particular result of a study is unavailable, the same consequence can arise: bias in a synthesis because available results differ systematically from missing results (Page et al 2018). Chapter 13 provides detailed guidance on assessing risk of bias due to missing results in a systematic review.

7.8 Considering source of funding and conflict of interest of authors of included studies

Readers of a trial report often need to reflect on whether conflicts of interest have influenced the design, conduct, analysis and reporting of a trial. It is therefore now common for scientific journals to require authors of trial reports to provide a declaration of conflicts of interest (sometimes called ‘competing’ or ‘declarations of’ interest), to report funding sources and to describe any funder’s role in the trial.

In this section, we characterize conflicts of interest in randomized trials and discuss how conflicts of interest may impact on trial design and effect estimates. We also suggest how review authors can collect, process and use information on conflicts of interest in the assessment of:

• directness of studies to the review’s research question;
• heterogeneity in results due to differences in the designs of eligible studies;
• risk of bias in results of included studies;
• risk of bias in a synthesis due to missing results.

At the time of writing, a formal Tool for Addressing Conflicts of Interest in Trials (TACIT) is being developed under the auspices of the Cochrane Bias Methods Group. The TACIT development process has informed the content of this section, and we encourage readers to check http://tacit.one for more detailed guidance that will become available.

7.8.1 Characteristics of conflicts of interest

The Institute of Medicine defined conflicts of interest as “ a set of circumstances that creates a risk that professional judgment or actions regarding a primary interest will be unduly influenced by a secondary interest” (Lo et al 2009). In a clinical trial, the primary interest is to provide patients, clinicians and health policy makers with an unbiased and clinically relevant estimate of an intervention effect. Secondary interest may be both financial and non-financial.

Financial conflicts of interest involve both financial interests related to a specific trial (for example, a company funding a trial of a drug produced by the same company) and financial interests related to the authors of a trial report (for example, authors’ ownership of stocks or employment by a drug company).

For drug and device companies and other manufacturers, the financial difference between a negative and positive pivotal trial can be considerable. For example, the mean stock price of the companies funding 23 positive pivotal oncology trials increased by 14% after disclosure of the results (Rothenstein et al 2011). Industry funding is common, especially in drug trials. In a study of 200 trial publications from 2015, 68 (38%) of 178 trials with funding declarations were industry funded (Hakoum et al 2017). Also, in a cohort of oncology drug trials, industry funded 44% of trials and authors declared conflicts of interest in 69% of trials (Riechelmann et al 2007).

The degree of funding, and the type of the involvement of industry funders, may differ across trials. In some situations, involvement includes only the provision of free study medication for a trial that has otherwise been planned and conducted independently, and funded largely, by public means. In other situations, a company fully funds and controls a trial. In rarer cases, head-to-head trials comparing two drugs may be funded by the two different companies producing the drugs.

A Cochrane Methodology Review analysed 75 studies of the association between industry funding and trial results (Lundh et al 2017). The authors concluded that trials funded by a drug or device company were more likely to have positive conclusions and statistically significant results, and that this association could not be explained by differences in risk of bias between industry and non-industry funded trials. However, industry and non-industry trials may differ in ways that may confound the association; for example due to choice of patient population, comparator interventions or outcomes. Only one of the included studies used a meta-epidemiological design and found no clear association between industry funding and the magnitude of intervention effects (Als-Nielsen et al 2003). Similar to the association with industry funding, other studies have reported that results of trials conducted by authors with a financial conflict of interest were more likely to be positive (Ahn et al 2017).

Conflicts of interest may also be non-financial (Viswanathan et al 2014). Characterizations of non-financial conflicts of interest differ somewhat, but typically distinguish between conflicts related mainly to an individual (e.g. adherence to a theory or ideology), relationships to other individuals (e.g. loyalty to friends, family members or close colleagues), or relationship to groups (e.g. work place or professional groups). In medicine, non-financial conflicts of interest have received less attention than financial conflicts of interest. In addition, financial and non-financial conflicts are often intertwined; for example, non-financial conflicts related to institutional association can be considered as indirect financial conflicts linked to employment. Definitions of what should be characterized as a ‘non-financial’ conflict of interest, and, in particular, whether personal beliefs, experiences or intellectual commitments should be considered conflicts of interest, have been debated (Bero and Grundy 2016).

It is useful to differentiate between non-financial conflicts of interest of a trial researcher and the basic interests and hopes involved in doing good trial research. Most researchers conducting a trial will have an interest in the scientific problem addressed, a well-articulated theoretical position, anticipation for a specific trial result, and hopes for publication in a respectable journal. This is not a conflict of interest but a basic condition for doing health research. However, individual researchers may lose sight of the primacy of the methodological neutrality at the heart of a scientific enquiry, and become unduly occupied with the secondary interest of how trial results may affect academic appearance or chances of future funding. Extreme examples are the publication of fabricated trial data or trials, some of which have had an impact on systematic reviews (Marret et al 2009).

Few empirical studies of non-financial conflicts of interest in randomized trials have been published, and to our knowledge there are none that assess the impact of non-financial conflicts of interest on trial results and conclusions. However, non-financial conflicts of interests have been investigated in other types of clinical research; for example, guideline authors’ specialty appears to have influenced their voting behaviour while developing guidelines for mammography screening (Norris et al 2012).

7.8.2 Conflict of interest and trial design

Core decisions on designing a trial involve defining the type of participants to be included, the type of experimental intervention, the type of comparator, the outcomes (and timing of outcome assessments) and the choice of analysis. Such decisions will often reflect a compromise between what is clinically and scientifically ideal and what is practically possible. However, when investigators have important conflicts of interest, a trial may be designed in a way that increases its chances of detecting a positive trial result, at the expense of clinical applicability. For example, narrow eligibility criteria may exclude older and frail patients, thus reducing the possibility of detecting clinically relevant harms. Alternatively, trial designers may choose placebo as a comparator despite an effective intervention being in regular use, or they may focus on short-term surrogate outcomes rather than clinically relevant long-term outcomes (Estellat and Ravaud 2012, Wieland et al 2017).

Trial design choices may be more subtle. For example, a trial may be designed to favour an experimental drug by using an inferior comparator drug when better alternatives exist (Safer 2002) or by using a low dose of the comparator drug when the focus is efficacy and a high dose of the comparator drug when the focus is harms (Mann and Djulbegovic 2013). In a typical Cochrane Review with fairly broad eligibility criteria aiming to identify and summarize all relevant trials, it is pertinent to consider the degree to which a given trial result directly relates to the question posed by the review. If all or most identified trials have narrow eligibility criteria and short-term outcomes, a review question focusing on broad patient categories and long-term effects can only be answered indirectly by the included studies. This has implications for the assessment of the certainty of the evidence provided by the review, which is addressed through the concept of indirectness in the GRADE framework (see Chapter 14, Section 14.2 ).

If results in a meta-analysis display heterogeneity, then differences in design choices that are driven by conflicts of interest may be one reason for this. Thus, conflicts of interest may also affect reflections on the certainty of the evidence through the GRADE concept of inconsistency.

7.8.3 Conflicts of interest and risk of bias in a trial’s effect estimate

Authors of Cochrane Reviews have sometimes included conflicts of interest as an ‘other source of bias’ while using the previous versions of the risk-of-bias tool (Jørgensen et al 2016). Consistent with previous versions of the Handbook , we discourage the inclusion of conflicts of interest directly in the risk-of-bias assessment. Adding conflicts of interest to the bias tool is inconsistent with the conceptual structure of the tool, which is built on mechanistically defined bias domains. Also, restricting consideration of the potential impact of conflicts of interest to a question of risk of bias in an individual trial result overlooks other important aspects, such as the design of the trial (see Section 7.8.2 ) and potential bias in a meta-analysis due to missing results (see Section 7.8.4 ).

Conflicts of interest may lead to bias in effect estimates from a trial through several mechanisms. For example, if those recruiting participants into a trial have important conflicts of interest and the allocation sequence is not concealed, then they may be more likely to subvert the allocation process to produce intervention groups that are systematically unbalanced in favour of their preferred intervention. Similarly, investigators with important conflicts of interests may decide to exclude from the analysis some patients who did not respond as anticipated to the experimental intervention, resulting in bias due to missing outcome data. Furthermore, selective reporting of a favourable result may be strongly associated with conflicts of interest (McGauran et al 2010), due to either selective reporting of particular outcome measurements or selective reporting of particular analyses (Eyding et al 2010, Vedula et al 2013). One study found that use of modified-intention-to-treat analysis and post-randomization exclusions occurred more often in trials with industry funding or author conflicts of interest (Montedori et al 2011). Accessing the trial protocol and statistical analysis plan to determine which outcomes and analyses were pre-specified is therefore especially important for a trial with relevant conflicts of interest.

Review authors should explain how consideration of conflicts of interest informed their risk-of-bias judgements. For example, when information on the analysis plans is lacking, review authors may judge the risk of bias in selection of the reported result to be high if the study investigators had important financial conflicts of interest. Conversely, if trial investigators have clearly used methods that are likely to minimize bias, review authors should not judge the risk of bias for each domain higher just because the investigators happen to have conflicts of interest. In addition, as an optional component in the revised risk-of-bias tool, review authors may reflect on the direction of bias (e.g. bias in favour of the experimental intervention). Information on conflicts of interest may inform the assessment of direction of bias.

7.8.4 Conflicts of interest and risk of bias in a synthesis of trial results

Conflicts of interest may also affect the decision not to report trial results. Conflicts of interest are probably one of several important reasons for decisions not to publish trials with negative findings, and not to publish unfavourable results (Sterne 2013). When relevant trial results are systematically missing from a meta-analysis because of the nature of the findings, the synthesis is at risk of bias due to missing results. Chapter 13 provides detailed guidance on assessing risk of bias due to missing results in a systematic review.

7.8.5 Practical approach to identifying and extracting information on conflicts of interest

When assessing conflicts of interest in a trial, review authors will, to a large degree, rely on declared conflicts. Source of funding may be reported in a trial publication, and conflicts of interest may be reported in an accompanying declaration, for example the International Committee of Medical Journal Editors ( ICMJE ) declaration. In a random sample of 1002 articles published in 2016, authors of 229 (23%) declared having a conflict of interest (Grundy et al 2018). Unfortunately, undeclared conflicts of interest and sources of funding are fairly common (Rasmussen et al 2015, Patel et al 2018).

It is always prudent to examine closely the conflicts of interest of lead and corresponding authors, based on information reported in the trial publication and the author declaration (for example, the ICMJE declaration form). Review authors should also consider examining conflicts of interest of trial co-authors and any commercial collaborators with conflicts of interest; for example, a commercial contract research organization hired by the funder to collect and analyse trial data or the involvement of a medical writing agency. Due to the high prevalence of undisclosed conflicts of interest, review authors should consider expanding their search for conflicts of interest data from other sources (e.g. disclosure in other publications by the authors, the trial protocol, the clinical study report, and public conflicts of interest registries (e.g. Open Payments database)).

We suggest that review authors balance the workload involved with the expected gain, and search additional sources of information on conflicts of interest when there is reason to suspect important conflicts of interest . As a rule of thumb, in trials with unclear funding source and no declaration of conflicts of interest from lead or corresponding authors, we suggest review authors search the Open Payments database, ClinicalTrials.gov , and conflicts of interest declarations in a few previous publications by the study authors. In trials with no commercial funding (including no company employee co-authors) and no declared conflicts of interest for lead or corresponding authors, we suggest review authors not bother to consult additional sources. Also, for trials where lead or corresponding authors have clear conflicts of interest, little additional information may be gained from checking conflicts of interest of co-authors.

Gaining access to relevant information on financial conflicts of interest is possible for a considerable number of trials, despite inherent problems of undeclared conflicts. We expect that the proportion of trials with relevant declarations will increase further.

Access to relevant information on non-financial conflicts of interest is more difficult to gain. Declaration of non-financial conflicts of interest is requested by approximately 50% of journals (Shawwa et al 2016). The term was deleted from ICMJE’s declaration in 2010 in exchange for a broad category of “Other relationships or activities” (Drazen et al 2010). Therefore, non-financial conflicts of interests are seldom self-declared, although if available, such information should be considered.

Non-financial conflicts of interest are difficult to address due to lack of relevant empirical studies on their impact on study results, lack of relevant thresholds for importance, and lack of declaration in many previous trials. However, as a rule of thumb, we suggest that review authors assume trial authors have no non-financial conflicts of interest unless there are clear suggestions of the opposite. Examples of such clues could be a considerable spin in trial publications (Boutron et al 2010), an institutional relationship pertinent to the intervention tested, or external evidence of a fixated ideological or theoretical position.

7.8.6 Judgement of notable concern about conflict of interest

Review authors should describe funding information and conflicts of interest of authors for all studies in the ‘Characteristics of included studies’ table ( MECIR Box 7.8.a ). Also, review authors may want to explore (e.g. in a subgroup analysis) whether trials with conflicts of interest have different intervention effect estimates, or more variable effect estimates, than trials without conflicts of interest. In both cases, review authors need to aim for a relevant threshold for when any conflict of interest is deemed important. If put too low, there is a risk that trivial conflicts of interest will cloud important ones; if set too high, there is the risk that important conflicts of interest are downplayed or ignored.

This judgement should take into account both the degree of conflicts of interest of study authors and also the extent of their involvement in the study. We pragmatically suggest review authors aim for a judgement about whether or not there is reason for ‘notable concern’ about conflicts of interest. This information could be displayed in a table with three columns:

• trial identifier;
• judgement (e.g. ‘notable concern about conflict of interest’ versus ‘no notable concern about conflict of interest’); and
• rationale for judgement, potentially subdivided according to who had conflicts of interest (e.g. lead or corresponding authors, other authors) and stage(s) of the trial to which they contributed (design, conduct, analysis, reporting).

A judgement of ‘notable concern about conflict of interest’ should be based on reflected assessment of identified conflicts of interest. A hypothetical possibility for undeclared conflicts of interest is, as a rule of thumb, not considered sufficient reason for ‘notable concern’. By ‘notable concern’ we imply important conflicts of interest expected to have a potential impact on study design, risk of bias in study results or risk of bias in a synthesis due to missing results. For example, financial conflicts of interest are important in a trial initiated, designed, analysed and reported by drug or device company employees. Conversely, financial conflicts of interest are less important in a trial initiated, designed, analysed and reported by academics adhering to the arm’s length principle when acquiring free trial medication from a drug company, and where lead authors have no conflicts of interest. Similarly, non-financial conflicts of interest may be important in a trial of a highly controversial and ideologically loaded question such as the adverse effect of male circumcision. Non-financial conflicts of interest are less concerning in a trial comparing two treatments in general use with no connotation to highly controversial scientific theories, ideology or professional groups. Mixing trivial conflicts of interest with important ones may mask the latter and will expand review author workload considerably.

MECIR Box 7.8.a Relevant expectations for conduct of intervention reviews

7.9 Chapter information

Authors: Isabelle Boutron, Matthew J Page, Julian PT Higgins, Douglas G Altman, Andreas Lundh, Asbjørn Hróbjartsson

Acknowledgements: We thank Gerd Antes, Peter Gøtzsche, Peter Jüni, Steff Lewis, David Moher, Andrew Oxman, Ken Schulz, Jonathan Sterne and Simon Thompson for their contributions to previous versions of this chapter.

7.10 References

Ahn R, Woodbridge A, Abraham A, Saba S, Korenstein D, Madden E, Boscardin WJ, Keyhani S. Financial ties of principal investigators and randomized controlled trial outcomes: cross sectional study. BMJ 2017; 356 : i6770.

Als-Nielsen B, Chen W, Gluud C, Kjaergard LL. Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events? JAMA 2003; 290 : 921-928.

Bero LA, Grundy Q. Why Having a (Nonfinancial) Interest Is Not a Conflict of Interest. PLoS Biology 2016; 14 : e2001221.

Blümle A, Meerpohl JJ, Schumacher M, von Elm E. Fate of clinical research studies after ethical approval--follow-up of study protocols until publication. PloS One 2014; 9 : e87184.

Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA 2010; 303 : 2058-2064.

Chan A-W, Song F, Vickers A, Jefferson T, Dickersin K, Gøtzsche PC, Krumholz HM, Ghersi D, van der Worp HB. Increasing value and reducing waste: addressing inaccessible research. The Lancet 2014; 383 : 257-266.

Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004a; 291 : 2457-2465.

Chan AW, Krleža-Jeric K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. Canadian Medical Association Journal 2004b; 171 : 735-740.

da Costa BR, Beckett B, Diaz A, Resta NM, Johnston BC, Egger M, Jüni P, Armijo-Olivo S. Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study. Systematic Reviews 2017; 6 : 44.

Dechartres A, Boutron I, Trinquart L, Charles P, Ravaud P. Single-center trials show larger treatment effects than multicenter trials: evidence from a meta-epidemiologic study. Annals of Internal Medicine 2011; 155 : 39-51.

Dechartres A, Trinquart L, Boutron I, Ravaud P. Influence of trial sample size on treatment effect estimates: meta-epidemiological study. BMJ 2013; 346 : f2304.

Dechartres A, Trinquart L, Faber T, Ravaud P. Empirical evaluation of which trial characteristics are associated with treatment effect estimates. Journal of Clinical Epidemiology 2016a; 77 : 24-37.

Dechartres A, Ravaud P, Atal I, Riveros C, Boutron I. Association between trial registration and treatment effect estimates: a meta-epidemiological study. BMC Medicine 2016b; 14 : 100.

Dechartres A, Trinquart L, Atal I, Moher D, Dickersin K, Boutron I, Perrodeau E, Altman DG, Ravaud P. Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: research on research study. BMJ 2017; 357 : j2490.

Dechartres A, Atal I, Riveros C, Meerpohl J, Ravaud P. Association between publication characteristics and treatment effect estimates: A meta-epidemiologic study. Annals of Internal Medicine 2018.

Drazen JM, de Leeuw PW, Laine C, Mulrow C, DeAngelis CD, Frizelle FA, Godlee F, Haug C, Hébert PC, Horton R, Kotzin S, Marusic A, Reyes H, Rosenberg J, Sahni P, Van der Weyden MB, Zhaori G. Towards more uniform conflict disclosures: the updated ICMJE conflict of interest reporting form. BMJ 2010; 340 : c3239.

Duyx B, Urlings MJE, Swaen GMH, Bouter LM, Zeegers MP. Scientific citations favor positive results: a systematic review and meta-analysis. Journal of Clinical Epidemiology 2017; 88 : 92-101.

Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991; 337 : 867-872.

Estellat C, Ravaud P. Lack of head-to-head trials and fair control arms: randomized controlled trials of biologic treatment for rheumatoid arthritis. Archives of Internal Medicine 2012; 172 : 237-244.

Eyding D, Lelgemann M, Grouven U, Harter M, Kromp M, Kaiser T, Kerekes MF, Gerken M, Wieseler B. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ 2010; 341 : c4737.

Fanelli D, Costas R, Ioannidis JPA. Meta-assessment of bias in science. Proceedings of the National Academy of Sciences of the United States of America 2017; 114 : 3714-3719.

Franco A, Malhotra N, Simonovits G. Social science. Publication bias in the social sciences: unlocking the file drawer. Science 2014; 345 : 1502-1505.

Gates A, Vandermeer B, Hartling L. Technology-assisted risk of bias assessment in systematic reviews: a prospective cross-sectional evaluation of the RobotReviewer machine learning tool. Journal of Clinical Epidemiology 2018; 96 : 54-62.

Grundy Q, Dunn AG, Bourgeois FT, Coiera E, Bero L. Prevalence of Disclosed Conflicts of Interest in Biomedical Research and Associations With Journal Impact Factors and Altmetric Scores. JAMA 2018; 319 : 408-409.

Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008; 336 : 924-926.

Hakoum MB, Jouni N, Abou-Jaoude EA, Hasbani DJ, Abou-Jaoude EA, Lopes LC, Khaldieh M, Hammoud MZ, Al-Gibbawi M, Anouti S, Guyatt G, Akl EA. Characteristics of funding of clinical trials: cross-sectional survey and proposed guidance. BMJ Open 2017; 7 : e015997.

Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, Tsertsvadze A, Hempel S, Shekelle P, Dryden DM. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. Journal of Clinical Epidemiology 2013; 66 : 973-981.

Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JAC. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011; 343 : d5928.

Hopewell S, Clarke M, Stewart L, Tierney J. Time to publication for results of clinical trials. Cochrane Database of Systematic Reviews 2007; 2 : MR000011.

Hopewell S, Boutron I, Altman D, Ravaud P. Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study. BMJ Open 2013; 3 : 8.

Jefferson T, Jones MA, Doshi P, Del Mar CB, Hama R, Thompson MJ, Onakpoya I, Heneghan CJ. Risk of bias in industry-funded oseltamivir trials: comparison of core reports versus full clinical study reports. BMJ Open 2014; 4 : e005253.

Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Medicine 2015; 13 : 282.

Jørgensen L, Paludan-Muller AS, Laursen DR, Savovic J, Boutron I, Sterne JAC, Higgins JPT, Hróbjartsson A. Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews. Systematic Reviews 2016; 5 : 80.

Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999; 282 : 1054-1060.

Jüni P, Altman DG, Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ 2001; 323 : 42-46.

Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, Williamson PR. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010; 340 : c365.

Li G, Abbade LPF, Nwosu I, Jin Y, Leenus A, Maaz M, Wang M, Bhatt M, Zielinski L, Sanger N, Bantoto B, Luo C, Shams I, Shahid H, Chang Y, Sun G, Mbuagbaw L, Samaan Z, Levine MAH, Adachi JD, Thabane L. A systematic review of comparisons between protocols or registrations and full reports in primary biomedical research. BMC Medical Research Methodology 2018; 18 : 9.

Lo B, Field MJ, Institute of Medicine (US) Committee on Conflict of Interest in Medical Research Education and Practice. Conflict of Interest in Medical Research, Education, and Practice . Washington, D.C.: National Academies Press (US); 2009.

Lundh A, Lexchin J, Mintzes B, Schroll JB, Bero L. Industry sponsorship and research outcome. Cochrane Database of Systematic Reviews 2017; 2 : MR000033.

Mann H, Djulbegovic B. Comparator bias: why comparisons must address genuine uncertainties. Journal of the Royal Society of Medicine 2013; 106 : 30-33.

Marret E, Elia N, Dahl JB, McQuay HJ, Møiniche S, Moore RA, Straube S, Tramèr MR. Susceptibility to fraud in systematic reviews: lessons from the Reuben case. Anesthesiology 2009; 111 : 1279-1289.

Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. Journal of the American Medical Informatics Association 2016; 23 : 193-201.

McGauran N, Wieseler B, Kreis J, Schuler YB, Kolsch H, Kaiser T. Reporting bias in medical research - a narrative review. Trials 2010; 11 : 37.

Millard LA, Flach PA, Higgins JPT. Machine learning to assist risk-of-bias assessments in systematic reviews. International Journal of Epidemiology 2016; 45 : 266-277.

Moher D, Shamseer L, Cobey KD, Lalu MM, Galipeau J, Avey MT, Ahmadzai N, Alabousi M, Barbeau P, Beck A, Daniel R, Frank R, Ghannad M, Hamel C, Hersi M, Hutton B, Isupov I, McGrath TA, McInnes MDF, Page MJ, Pratt M, Pussegoda K, Shea B, Srivastava A, Stevens A, Thavorn K, van Katwyk S, Ward R, Wolfe D, Yazdi F, Yu AM, Ziai H. Stop this waste of people, animals and money. Nature 2017; 549 : 23-25.

Montedori A, Bonacini MI, Casazza G, Luchetta ML, Duca P, Cozzolino F, Abraha I. Modified versus standard intention-to-treat reporting: are there differences in methodological quality, sponsorship, and findings in randomized trials? A cross-sectional study. Trials 2011; 12 : 58.

Morgan AJ, Ross A, Reavley NJ. Systematic review and meta-analysis of Mental Health First Aid training: Effects on knowledge, stigma, and helping behaviour. PloS One 2018; 13 : e0197102.

Morrison A, Polisena J, Husereau D, Moulton K, Clark M, Fiander M, Mierzwinski-Urban M, Clifford T, Hutton B, Rabb D. The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. International Journal of Technology Assessment in Health Care 2012; 28 : 138-144.

Norris SL, Burda BU, Holmer HK, Ogden LA, Fu R, Bero L, Schunemann H, Deyo R. Author's specialty and conflicts of interest contribute to conflicting guidelines for screening mammography. Journal of Clinical Epidemiology 2012; 65 : 725-733.

Odutayo A, Emdin CA, Hsiao AJ, Shakir M, Copsey B, Dutton S, Chiocchia V, Schlussel M, Dutton P, Roberts C, Altman DG, Hopewell S. Association between trial registration and positive study findings: cross sectional study (Epidemiological Study of Randomized Trials-ESORT). BMJ 2017; 356 : j917.

Page MJ, Higgins JPT. Rethinking the assessment of risk of bias due to selective reporting: a cross-sectional study. Systematic Reviews 2016; 5 : 108.

Page MJ, Higgins JPT, Clayton G, Sterne JAC, Hróbjartsson A, Savović J. Empirical evidence of study design biases in randomized trials: systematic review of meta-epidemiological studies. PloS One 2016; 11 : 7.

Page MJ, McKenzie JE, Higgins JPT. Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review. BMJ Open 2018; 8 : e019703.

Patel SV, Yu D, Elsolh B, Goldacre BM, Nash GM. Assessment of conflicts of interest in robotic surgical studies: validating author's declarations with the open payments database. Annals of Surgery 2018; 268 : 86-92.

Polanin JR, Tanner-Smith EE, Hennessy EA. Estimating the difference between published and unpublished effect sizes: a meta-review. Review of Educational Research 2016; 86 : 207-236.

Rasmussen K, Schroll J, Gøtzsche PC, Lundh A. Under-reporting of conflicts of interest among trialists: a cross-sectional study. Journal of the Royal Society of Medicine 2015; 108 : 101-107.

Riechelmann RP, Wang L, O'Carroll A, Krzyzanowska MK. Disclosure of conflicts of interest by authors of clinical trials and editorials in oncology. Journal of Clinical Oncology 2007; 25 : 4642-4647.

Rising K, Bacchetti P, Bero L. Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation. PLoS Medicine 2008; 5 : e217.

Riveros C, Dechartres A, Perrodeau E, Haneef R, Boutron I, Ravaud P. Timing and completeness of trial results posted at ClinicalTrials.gov and published in journals. PLoS Medicine 2013; 10 : e1001566.

Rothenstein JM, Tomlinson G, Tannock IF, Detsky AS. Company stock prices before and after public announcements related to oncology drugs. Journal of the National Cancer Institute 2011; 103 : 1507-1512.

Safer DJ. Design and reporting modifications in industry-sponsored comparative psychopharmacology trials. Journal of Nervous and Mental Disease 2002; 190 : 583-592.

Saini P, Loke YK, Gamble C, Altman DG, Williamson PR, Kirkham JJ. Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ 2014; 349 : g6501.

Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, St John PD, Viola R, Raina P. Should meta-analysts search Embase in addition to Medline? Journal of Clinical Epidemiology 2003; 56 : 943-955.

Savović J, Jones HE, Altman DG, Harris RJ, Jüni P, Pildal J, Als-Nielsen B, Balk EM, Gluud C, Gluud LL, Ioannidis JPA, Schulz KF, Beynon R, Welton NJ, Wood L, Moher D, Deeks JJ, Sterne JAC. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Annals of Internal Medicine 2012; 157 : 429-438.

Scherer RW, Meerpohl JJ, Pfeifer N, Schmucker C, Schwarzer G, von Elm E. Full publication of results initially presented in abstracts. Cochrane Database of Systematic Reviews 2018; 11 : MR000005.

Schmid CH. Outcome Reporting Bias: A Pervasive Problem in Published Meta-analyses. American Journal of Kidney Diseases 2016; 69 : 172-174.

Schmucker C, Schell LK, Portalupi S, Oeller P, Cabrera L, Bassler D, Schwarzer G, Scherer RW, Antes G, von Elm E, Meerpohl JJ. Extent of non-publication in cohorts of studies approved by research ethics committees or included in trial registries. PloS One 2014; 9 : e114023.

Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273 : 408-412.

Shawwa K, Kallas R, Koujanian S, Agarwal A, Neumann I, Alexander P, Tikkinen KA, Guyatt G, Akl EA. Requirements of Clinical Journals for Authors’ Disclosure of Financial and Non-Financial Conflicts of Interest: A Cross Sectional Study. PloS One 2016; 11 : e0152301.

Sterne JAC. Why the Cochrane risk of bias tool should not include funding source as a standard item [editorial]. Cochrane Database of Systematic Reviews 2013; 12 : ED000076.

Tramèr MR, Reynolds DJ, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: a case study. BMJ 1997; 315 : 635-640.

Turner RM, Spiegelhalter DJ, Smith GC, Thompson SG. Bias modelling in evidence synthesis. Journal of the Royal Statistical Society Series A, (Statistics in Society) 2009; 172 : 21-47.

Urrutia G, Ballesteros M, Djulbegovic B, Gich I, Roque M, Bonfill X. Cancer randomized trials showed that dissemination bias is still a problem to be solved. Journal of Clinical Epidemiology 2016; 77 : 84-90.

Vedula SS, Li T, Dickersin K. Differences in reporting of analyses in internal company documents versus published trial reports: comparisons in industry-sponsored trials in off-label uses of gabapentin. PLoS Medicine 2013; 10 : e1001378.

Viswanathan M, Carey TS, Belinson SE, Berliner E, Chang SM, Graham E, Guise JM, Ip S, Maglione MA, McCrory DC, McPheeters M, Newberry SJ, Sista P, White CM. A proposed approach may help systematic reviews retain needed expertise while minimizing bias from nonfinancial conflicts of interest. Journal of Clinical Epidemiology 2014; 67 : 1229-1238.

Welton NJ, Ades AE, Carlin JB, Altman DG, Sterne JAC. Models for potentially biased evidence in meta-analysis using empirically based priors. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009; 172 : 119-136.

Wieland LS, Berman BM, Altman DG, Barth J, Bouter LM, D'Adamo CR, Linde K, Moher D, Mullins CD, Treweek S, Tunis S, van der Windt DA, Zwarenstein M, Witt C. Rating of Included Trials on the Efficacy-Effectiveness Spectrum: development of a new tool for systematic reviews. Journal of Clinical Epidemiology 2017; 84 .

Wieseler B, Kerekes MF, Vervoelgyi V, McGauran N, Kaiser T. Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publications. BMJ 2012; 344 : d8141.

Wood L, Egger M, Gluud LL, Schulz K, Jüni P, Altman DG, Gluud C, Martin RM, Wood AJG, Sterne JAC. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 2008; 336 : 601-605.

Zarin DA, Tse T, Williams RJ, Carr S. Trial Reporting in ClinicalTrials.gov - The Final Rule. New England Journal of Medicine 2016; 375 : 1998-2004.

For permission to re-use material from the Handbook (either academic or commercial), please see here for full details.

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• For authors
• Browse by collection
• BMJ Journals

You are here

• Volume 6, Issue 4
• Conflicts of interest and spin in reviews of psychological therapies: a systematic review
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Klaus Lieb 1 ,
• Jan von der Osten-Sacken 1 ,
• Jutta Stoffers-Winterling 1 ,
• Neele Reiss 2 ,
• Jürgen Barth 3
• 1 Department of Psychiatry and Psychotherapy , University Medical Center Mainz , Mainz , Germany
• 2 Department of Differential Psychology and Psychological Assessment , Goethe-University Frankfurt , Frankfurt , Germany
• 3 Institute for Complementary and Integrative Medicine , University Hospital Zurich and University Zurich , Zurich, Switzerland
• Correspondence to Dr Klaus Lieb; klaus.lieb{at}unimedizin-mainz.de

Objective To explore conflicts of interest (COI) and their reporting in systematic reviews of psychological therapies, and to evaluate spin in the conclusions of the reviews.

Methods MEDLINE and PsycINFO databases were searched for systematic reviews published between 2010 and 2013 that assessed effects of psychological therapies for anxiety, depressive or personality disorders, and included at least one randomised controlled trial. Required COI disclosure by journal, disclosed COI by review authors, and the inclusion of own primary studies by review authors were extracted. Researcher allegiance, that is, that researchers concluded favourably about the interventions they have studied, as well as spin, that is, differences between results and conclusions of the reviews, were rated by 2 independent raters.

Results 936 references were retrieved, 95 reviews fulfilled eligibility criteria. 59 compared psychological therapies with other forms of psychological therapies, and 36 psychological therapies with pharmacological interventions. Financial, non-financial, and personal COI were disclosed in 22, 4 and 1 review, respectively. 2 of 86 own primary studies of review authors included in 34 reviews were disclosed by review authors. In 15 of the reviews, authors showed an allegiance effect to the evaluated psychological therapy that was never disclosed. Spin in review conclusions was found in 27 of 95 reviews. Reviews with a conclusion in favour of psychological therapies (vs pharmacological interventions) were at high risk for a spin in conclusions (OR=8.31 (1.41 to 49.05)). Spin was related in trend to the inclusion of own primary studies in the systematic review (OR=2.08 (CI 0.83 to 5.18) p=0.11) and researcher allegiance (OR=2.63 (0.84 to 8.16) p=0.16).

Conclusions Non-financial COI, especially the inclusion of own primary studies into reviews and researcher allegiance, are frequently seen in systematic reviews of psychological therapies and need more transparency and better management.

• psychotherapy
• conflict of interest
• systematic review

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

https://doi.org/10.1136/bmjopen-2015-010606

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

This study addresses a widely neglected research topic, that is, spin introduced by non-financial conflicts of interest (COI), for example, the researcher's allegiance to a specific therapy, in reviews on psychotherapy studies.

Although authors of reviews of psychological therapies frequently show COI (which mainly are not declared), the relationship to spin in review conclusions is less clear and has to be interpreted with caution.

We decided to use the term ‘spin’ instead of ‘bias’, although we cannot make claims about the nature of the influence which might be mere bias or more intentional spin.

The selection of studies up to 2013 does not reflect possible changes in COI declarations in recent years.

Introduction

Conflicts of interest (COI) are defined as a set of circumstances that creates a risk that a professional judgement or action regarding a primary interest will be unduly influenced by a secondary interest. 1 , 2 Research on COI has so far mainly focused on financial COI such as close financial relationships between researchers or medical doctors and pharmaceutical companies, or the financing of drug trials by pharmaceutical companies. 3–7 Such research has shown that studies funded by pharmaceutical companies more often yield results or conclusions in favour of the sponsoring company as compared to non-industry-funded trials; 8 , 9 that close relationships of researchers to pharmaceutical companies are linked to biased assessments of drug safety and efficacy; 10 , 11 that positive trials are more likely to be published than trials unfavourable to sponsors; 12 and that COI are under-reported in meta-analyses of pharmacological treatments. 13 , 14

The influence of non-financial COI, however, on the framing of research questions, the data analysis and interpretation of results, or the decision which results are being published, has been much less extensively studied. 15 With respect to outcome research of psychological therapies, researcher allegiance constitutes an important non-financial COI. Allegiance covers the belief of a researcher in the superiority of a treatment. 16 , 17 Allegiance may be due to a special training in one specific psychological therapy, the involvement in previous efficacy research about this psychological therapy or the involvement in development of aetiological models via basic research. 18–20 Empirical studies showed a strong impact of researcher allegiance on outcome in psychotherapy studies: a recent meta-analysis showed a robust and moderate allegiance outcome association (r=0.26), 21 and such an association is also present in equally effective treatments. 22 Taking allegiance into account for the explanation of effect differences between two active treatments studies with balanced allegiance for two different treatments show no difference in the effectiveness. 23

Since nothing is known about the extent and nature of non-financial COI in systematic reviews of psychological therapies, the aim of this study was to investigate how often non-financial COI are present and disclosed in systematic reviews of psychological therapies, and to analyse whether these COI increase the risk of spin in the conclusions of the reviews.

Search strategy and eligibility criteria of systematic reviews

We searched the MEDLINE and PsycINFO data bases for systematic reviews or meta-analyses of randomised controlled trials (RCT) on psychological therapies. Reviews were selected if they fulfilled the following inclusion criteria: (1) inclusion of psychological therapies to treat patients with anxiety disorders, personality disorders and/or major depressive disorders in adults, (2) active control groups with either other forms of psychological therapy or pharmacological interventions, (3) inclusion of at least one randomised study and (4) English language. Searches were last run on 3 February 2014, covering the publication period of January 2010–December 2013. For exact MEDLINE and PsycINFO search strategies, refer online supplementary table s S1 and S2.

Supplemental material

Screening and inclusion of systematic reviews and primary studies.

Retrieved references were initially screened for inclusion by title and abstract by two independent researchers. In a second step, full texts of relevant reviews were retrieved and assessed for inclusion by two independent researchers. These reviews were used to rate COI and their disclosure.

Primary studies included in these reviews were identified from the reference lists of the systematic reviews and retrieved if one of the coauthors of the review was an author of the respective primary study. These primary studies were then used to rate researcher allegiance.

Assessment of disclosed and undisclosed COI

All disclosed COI were extracted: financial COI (honoraria, eg, for consulting, lectures, scientific articles, training courses, or money for research projects), non-financial COI (eg, researcher allegiance to a psychological therapy, special qualification in a psychological therapy, enthusiasm for a psychological therapy in scientific publications, lectures and research, or inclusion of own primary studies in reviews), and personal COI (eg, employee or private relationship to an employee of a company—regularly addressed as relationships to pharmaceutical companies). If no COI was reported, the websites of the respective journals as well as the guidelines for authors were screened for requirements of COI disclosures at the time of the publication of the review. In addition, we assessed whether review authors included own studies on psychological therapies into their review, and whether this inclusion was disclosed.

Rating of researcher allegiance

In case that a review author included at least one own primary study (which he or she coauthored) into the review, we retrieved these primary studies and rated the researcher allegiance according to information presented in the primary study (note that a rating of researcher allegiance was not possible in the reviews since these do not provide essential information to rate researcher allegiance according to established standards). 18–20 Researcher allegiance was rated in 73 of the 86 included primary studies since 13 reviews did not compare psychological therapies to other treatments and were therefore excluded.

Researcher allegiance was defined to be present if the author (1) recommended the respective psychological therapy over another therapy and was (2) either involved in the development of the respective psychological therapy, or (3) was involved in research of/development of the aetiological model of the psychological therapy. Two independent researchers (JvdOS, JB) assessed allegiance in the primary studies, and disagreements were resolved with a third rater (KL). If researcher allegiance was rated to be present in at least one of the primary studies included in a review, this review was rated as afflicted by researcher allegiance. κ Statistics showed substantial inter-rater reliability (k=0.62; agreement 82%).

Assessment of spin in review conclusions

To assess spin in review conclusions, we evaluated whether the conclusion of the review as expressed in the abstract or the discussion section was inconsistent or consistent with the empirical results described in the results section of the review. If the conclusion was consistent with the empirical results, the review was considered as showing no spin. If it was inconsistent, the review was rated as showing spin. Two researchers (KL, JvdOS), who both were blind to the author names of the review as well as the journal having published the review, independently assessed review conclusions and results, and rated whether a spin in review conclusions was present or not. If no consensus was achieved, disagreements were resolved by a third person (JB). κ Statistics showed substantial inter-rater reliability (k=0.70; agreement 87%).

Statistical analyses

The percentage of disclosed COI, researcher allegiance and reviews with spin was calculated. For the first two indicators, the number of reviews was the denominator, the latter indicator was calculated with the number of studies as denominator. The association of researcher allegiance with a spin in the conclusion of reviews is presented as OR with 95% CI. The same procedure was used for the association of the inclusion of own primary studies of the authors in the review, and the disclosure of COI with a spin in the review.

Our search yielded 936 references. After screening and retrieving full-text articles, 95 reviews remained which met our inclusion criteria. A detailed flow chart with a schedule of the reasons for exclusions is found in figure 1 . The reviews and meta-analyses addressed anxiety disorders (n=42), depressive disorders (n=48), and/or personality disorders (n=13), and allowed conclusions about the following interventions: 59 reviews compared psychological therapies with other forms of psychological therapies, and 36 compared psychological therapies with pharmacological interventions.

• Download figure
• Open in new tab
• Download powerpoint

Flow chart of study selection. RCT, randomised control trial.

Required COI disclosure by journal and disclosed COI

The references of the 95 reviews included in this study are listed in online supplementary table S3. online supplementary table S4 gives an overview how many reviews were published per year and in which journal. Forty of the 50 journals regularly requested a disclosure of COI at the time of publication of the respective review. online supplementary table S5 demonstrates which journal asked for which kind of COI disclosure in the respective year of publication of the review article. In sum: of the 50 publishing journals, 40 requested a disclosure of financial COI (80%), 28 of personal COI (56%), and 17 of non-financial COI (34%).

In 37 of 95 reviews (38.9%), the authors disclosed that no competing interests exist. Authors in 25 of 95 reviews (26.3%) made COI statements as follows: own study included in the review (n=2), research activities in relation to one psychological therapy (n=2), research support (n=18), author has served as consultant (n=4), served as speaker on congresses (n=1), get honoraria (n=5), have holdings (n=2), have patents (n=1), served as a trainer for a psychological therapy (n=1), being influenced as employer (n=1). In other words, financial, non-financial and personal COI were disclosed in 22 reviews (23.1%), 4 reviews (4.2%), and 1 review (1%), respectively. One of the disclosures of financial COI was given in a journal which does not request declaration of COI; the non-financial and personal COI were all given in journals requesting such disclosures. In 33 of 95 reviews (34.7%), no COI statement was made.

Inclusion of own studies into the reviews and researcher allegiance

We also looked at the frequency of the inclusion of own primary studies into the review, and the allegiance of the researcher. Thirty-four of 95 reviews (35.8%) included at least one primary study of one of the review authors. In sum, 86 primary studies (all addressing psychological therapies) were identified which were included in 34 reviews (see online supplementary table S6 for references of these included studies). Twenty reviews included 1 study, 4 reviews 6 studies, 4 reviews 3 studies, 4 reviews 2 studies, 1 review 4 studies and 1 review 18 studies. In 15 of the 34 reviews which included at least one own primary study, we found a researcher allegiance.

Since both the inclusion of own primary studies and researcher allegiance can be described as non-financial COI, we further assessed the disclosure of such COI in relation to the requests of the journal to declare non-financial COI. Regarding the inclusion of own studies into the review, we found: of 34 reviews including own primary studies, inclusion of own studies by review authors was declared in two reviews according to the policy of the journal which specifically asked for inclusion of own studies; was not declared in 16 reviews published in journals requesting the disclosure of non-financial COI (but not defining inclusion of own studies as non-financial COI specifically); and was not declared in 16 reviews published in journals not requesting the disclosure of non-financial COI at all. Regarding researcher allegiance, we found that researcher allegiance was never disclosed: of 15 reviews with a researcher allegiance, researcher allegiance was not declared in nine reviews published in journals requesting the disclosure of non-financial COI (but not defining researcher allegiance as non-financial COI specifically), and was not declared in six reviews published in journals not requesting the disclosure of non-financial COI at all.

Spin in review conclusions

Spin in the interpretation of review results was rated to be present in 27 of 95 reviews (28%). Within the 36 reviews comparing psychological therapies to pharmacological interventions, nine (25%) showed a spin. In reviews comparing psychological therapies and pharmacological interventions, spin in favour of a specific psychological therapy was more often present as compared with spin in favour of a pharmacological intervention ( figure 2 ). Reviews with a favourable conclusion about psychological therapies (vs pharmacological interventions) are at high risk for a spin in conclusions (OR=8.31 (1.41 to 49.05)), whereas favourable conclusions about effects of pharmacological interventions showed no spin in our sample (OR=1.00 (0.16 to 6.14). Also the conclusion of equal effects of psychological therapies and pharmacological interventions does not face a risk of spin (OR=0.12 (0.01 to 1.08). The conclusion of the equality of effects of psychological therapies, however, showed a trend for a spin, which means that for the primary outcome of interest the review more often states equality despite inequality of treatment effects (OR=2.69 (0.86 to 8.41).

Risk of spin in review conclusions in comparison with different treatments.

We further explored whether spin in review conclusions is associated with a disclosed COI, the inclusion of own primary studies of the authors, or the researcher allegiance of the authors. To do so, we first investigated these associations in all 95 reviews ( table 1 ). Conclusions with spin were not associated with disclosed COI. However, spin, in conclusions, was related in trend to the inclusion of own studies in the systematic review. Reviews with inclusion of own primary studies showed more often spin than reviews without inclusion of own primary studies of the review authors (OR=2.08 (CI 0.83 to 5.18) p=0.11; table 1 ). The odds for spin in conclusions in systematic reviews including studies with researcher allegiance was similarly increased, but statistically non-significant (OR=2.63(0.84 to 8.12), p=0.16; table 1 ).

• View inline

Association between disclosed conflicts of interest (COI) and other forms of COI (ie, inclusion of primary studies in reviews, researcher allegiance) and spin in review conclusions in all 95 reviews

Since we were especially interested in spin in favour of psychological therapies, we also investigated whether spin in review conclusions in favour of psychological therapies is associated with a disclosed COI, the inclusion of own primary studies of the authors, or the researcher allegiance of the authors ( table 2 ). However, none of the associations were statistically significant or showed trends.

Association between disclosed conflicts of interest (COI) and other forms of COI (ie, inclusion of primary studies in reviews, researcher allegiance) and spin in review conclusions in the reviews rated as having spin in favour of psychological therapies as compared with all other reviews

This study is—at least to our knowledge—the first that systematically assessed the extent and nature of reporting of financial and non-financial COI in systematic reviews of psychological therapies, and that investigated how often these conflicts are disclosed, and whether they may lead to spin in review conclusions. Financial and non-financial COI were disclosed only in 23.1% and 4.2% of the reviews, respectively, although non-financial COI were much more often detectable: review authors had included 86 own studies in approximately one-third of the reviews, and authors of at least 16% of the reviews had allegiance for the evaluated psychological therapy. Spin in review conclusions was found in 27 of 95 reviews (28%), and was associated in trend with a non-financial COI, that is, the inclusion of own primary studies in the systematic review.

Disclosure of financial, non-financial and personal COI

The disclosure of financial COI was requested by 80% of the journals which published the reviews in our study, but only 22 reviews (23.2%) disclosed any financial COI. This may be explained by two reasons: first, systematic reviews focusing on effectiveness of psychological therapies are most often written by psychologists who have rather seldom financial ties to pharmaceutical companies as compared to physicians who often show these relationships, 3–7 and second, the minority of reviews (36 of 95 reviews) compared psychological therapies with pharmacological interventions (in 10 of those reviews, financial COI were disclosed). Although psychologists may mostly judge themselves as free of financial COI, however, researcher allegiance, as well as the inclusion of own studies into a review (which we both rated as non-financial COI), may well lead to financial gains indirectly. 15 Since psychologists who develop new psychological treatments are often the ones who distribute and train other psychologists in those therapies, the demonstration of effectiveness of a specific psychotherapy in a review may potentially lead to high financial incentives. The promotion of the respective therapy might be easier, and the number of trained psychotherapists with high course fee increases. Showing the effectiveness of a treatment can also be an important step for patents and for the implementation in treatment guidelines. The fact that researchers developing and evaluating the effectiveness of psychological therapies are mostly allied with a specific psychotherapy (eg, cognitive–behavioural therapy or psychoanalysis), makes the issue of COI in psychology therapy research very complex and much more complicated than in pharmacological research. Psychotherapy researchers who realise that the effect of the therapy to which they are allied is less beneficial than another therapy cannot easily switch their research programme to another therapy (since they have often been trained in that therapy for many years), in contrast with a researcher addressing pharmacotherapy who can more easily change his or her research agenda to another drug if a drug proves to be less effective than previously thought. Therefore, researcher allegiance might be present in primary studies in any case to some extent, but needs to be carefully declared in systematic reviews.

Non-financial COI were disclosed only in a very small number of reviews (4.2%) although non-financial COI, such as the inclusion of own primary studies of the review authors (in 34 of 95 reviews) and researcher allegiance (in 15 of 95 reviews), were detectable in a considerable number of them. This low disclosure rate may be explained by three factors: first, only a minority of journals (34% at the time of assessment) request a disclosure of non-financial COI, and all four declarations of non-financial COI were done in these journals; second, only two journals ( Perm J and Cochrane Database Syst Rev ) specifically asked the authors for the inclusion of own primary studies, and only two others ( Psychol Trauma and J Psychiatr Res ) asked for circumstances related to the presence of researcher allegiance at the point of our assessment; third, researchers may not see the necessity to declare such COI, although present and requested by the journal asking for non-financial COI. We conclude from this finding that the necessity to declare non-financial COI should be made more transparent in journal articles. The following strategies may be effective: journals should consequently ask their authors to disclose any non-financial COI, should exactly define such conflicts, and should include examples of common causes of non-financial COI, such as the inclusion of own primary studies into review articles or researcher allegiance. Even the International Committee of Medical Journal Editors mainly focuses on financial COI and their disclosures, but gives little emphasis on and advice to the disclosure of non-financial COI.

Similar to non-financial COI, also personal COI were very seldom disclosed (only in one review). This is probably due to the common definition of personal COI meaning any relationship to a person working in a pharmaceutical company. This, of course, is a less relevant COI for psychotherapist assessing treatment effects of psychological therapies. However, psychotherapists, especially the ones who develop new therapies, are very often personally involved in institutes promoting the distribution and training of new psychological therapies. Such personal COI may indirectly lead to considerable financial gains.

Previous research of our group and others has identified different risks increasing the likelihood of bias in psychotherapeutic outcome research. 21 , 24 In our study, we investigated whether researcher allegiance, an important risk factor of moderate effect size, 21 the inclusion of own primary studies into the review, or any declared COI, may be associated with spin in review conclusions, which we found in 27 of the 95 reviews. Both reviews with inclusion of own primary studies and reviews with researcher allegiance showed more often a spin (statistical trend). Since researcher allegiance has been shown to be significantly related to outcome of psychological therapies, 21 authors should be transparent in disclosing their own psychotherapeutic training background and the inclusion of own outcome studies in systematic reviews to make an assessment of COI and allegiance easier. The allegiance indicators of our study might be an initial step for such a statement (development of treatment or basic research on the aetiological model for a specific treatment).

Shortcomings

This study has several shortcomings. First, we restricted our search to systematic reviews and meta-analyses of anxiety disorders, personality disorders and major depressive disorders. This may limit the generalisability of our findings. Second, our study is limited to published reports from 2010 onwards. This limits generalisability to earlier reviews, but is justified since COI reporting has become more regular nowadays and authors might not have been asked for a COI statement in earlier submissions. Third, our indicators of COI and allegiance are based on publications, and reporting quality on some indicators was rather low. The inter-rater reliability of both ratings might be much better if reporting standards in journals would be implemented. Fourth, we only checked the disclosed COI, but did not investigate whether authors might have more COI than the disclosed ones. We also did not investigate which authors of a review might be responsible for the evaluation and interpretation of studies addressing different types of interventions (ie, pharmacotherapy and psychotherapy), since such investigations are at high risk of being inaccurate and incomplete.

Conclusions and suggestions for the management of COI in psychotherapy outcome research

We conclude that non-financial COI, especially the inclusion of own primary studies into reviews and researcher allegiance, are frequently seen in systematic reviews of psychological therapies and need more transparency. Most policies and journal requirements for COI disclosure focus on the importance of financial COI for risks of bias, and fail to capture the risk of spin associated with an allegiance. Therefore, if journals place more emphasis on the declaration of non-financial COI, declaration rates of non-financial COI by authors will most likely increase. If spin effects of non-financial COI in psychotherapy outcome research are confirmed in further studies, journals should do more than simply providing transparency of COI in order to better manage the impact of COI on research outcomes and publications. 15 Strategies to mitigate biases may include the detection and removal of spin at the editorial stage, using independent authors and reviewers interpreting the findings of meta-analyses, the rejection of systematic reviews that demonstrate selective citation biases, and providing free access to all data of systematic reviews to ensure that systematic reviews can be more easily replicated.

Acknowledgments

The study was funded by intramural funds from the University Medical Center Mainz, Department of Psychiatry and Psychotherapy, Mainz, Germany; there was no extramural funding. The authors thank V Stancheva for helping in study extraction.

• Thompson DF
• Field MJ , eds
• Campbell EG
• Campbell EG ,
• DesRoches CM , et al
• Weissman JS ,
• Ehringhaus S , et al
• Brandtönies S
• Scheurich A
• Bekelman JE ,
• Limbach U , et al
• Murad MH , et al
• Hudgins J , et al
• de Jonge P ,
• Williams CD , et al
• Roseman M ,
• Milette K ,
• Bero LA , et al
• Turner EH ,
• Lexchin J , et al
• Ainsworth K , et al
• DeRubeis RJ
• Gaffan EA ,
• Tsaousis I ,
• Kemp-Wheeler SM
• Trelle S , et al
• Wampold B ,
• Brütsch O ,
• Leonhart R , et al
• Flückiger C ,
• Gerger H , et al
• Cuijpers P ,
• Driessen E ,
• Hollon SD , et al
• van Straten A ,
• Bohlmeijer E , et al

Contributors KL designed the study, analysed data, monitored study extraction, data analysis and interpretation, and drafted and revised the manuscript. He is guarantor. JvdOS and NR extracted and analysed data and revised the draft of the manuscript. JS-W analysed data and revised the draft of the manuscript. JB analysed data, monitored study extraction, data analysis and interpretation, and revised the manuscript. All authors gave final approval of the version to be published.

Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests JB was involved in the development of indicators of allegiance. KL, NR, JS-W and JB are psychotherapists trained in CBT, KL and NR also in schematherapy. NR and JB received money from institutes providing training in schematherapy and CBT within the last 3 years. JS-W and KL are coauthors on two reviews included into the study (Gibbon et al , 2010 and Stoffers et al , 2012; see online supplementary table S3), and NR and KL are coauthors on one primary study (Reiss et al , 2014; see online supplementary table S6) included in one of the reviews.

Provenance and peer review Not commissioned; externally peer reviewed.

Data sharing statement Additional data is available by emailing the corresponding author at [email protected].

Read the full text or download the PDF:

• Open access
• Published: 18 November 2014

Conflicts of interest and critiques of the use of systematic reviews in policymaking: an analysis of opinion articles

• Susan R Forsyth 1 ,
• Donna H Odierna 2 ,
• David Krauth 2 &
• Lisa A Bero 2 , 3 , 4  

Systematic Reviews volume  3 , Article number:  122 ( 2014 ) Cite this article

7710 Accesses

37 Citations

36 Altmetric

Metrics details

Strong opinions for or against the use of systematic reviews to inform policymaking have been published in the medical literature. The purpose of this paper was to examine whether funding sources and author financial conflicts of interest were associated with whether an opinion article was supportive or critical of the use of systematic reviews for policymaking. We examined the nature of the arguments within each article, the types of disclosures present, and whether these articles are being cited in the academic literature.

We searched for articles that expressed opinions about the use of systematic reviews for policymaking. We included articles that presented opinions about the use of systematic reviews for policymaking and categorized each article as supportive or critical of such use. We extracted all arguments regarding the use of systematic reviews from each article and inductively coded each as internal or external validity argument, categorized disclosed funding sources, conflicts of interest, and article types, and systematically searched for undisclosed financial ties. We counted the number of times each article has been cited in the “Web of Science.” We report descriptive statistics.

Articles that were critical of the use of systematic reviews ( n = 25) for policymaking had disclosed or undisclosed industry ties 2.3 times more often than articles that were supportive of the use ( n = 34). We found that editorials, comments, letters, and perspectives lacked published disclosures nearly twice as often (60% v. 33%) as other types of articles. We also found that editorials, comments, letters, and perspectives were less frequently cited in the academic literature than other article types (median number of citations = 5 v. 19).

Conclusions

It is important to consider whether an article has industry ties when evaluating the strength of the argument for or against the use of systematic reviews for policymaking. We found that journal conflict of interest disclosures are often inadequate, particularly for editorials, comments, letters, and perspectives and that these articles are being cited as evidence in the academic literature. Our results further suggest the need for more consistent and complete disclosure for all article types.

Peer Review reports

Systematic reviews are often used to inform practice guidelines and public and private sector health policy decisions and are often used as an alternative to expert opinion or consensus conferences [ 1 – 4 ]. For example, systematic reviews have been used to guide policymaking around such issues as tobacco control and to set blood alcohol levels at which drivers are considered intoxicated [ 2 ]. States have used systematic reviews to evaluate whether policies for managing prison populations are working and cost effective [ 2 ]. Such use is not without controversy. While some authors argue that systematic reviews are the highest form of evidence and provide a solid basis for policymaking [ 1 , 2 ], others argue that these studies are methodologically flawed or limited in scope [ 5 ]. Editorials, letters, and other opinion pieces are important because they can inform debate about controversial topics. Furthermore, these articles are sometimes cited as if they are original research; a letter in the New England Journal of Medicine critical of a systematic review on secondhand smoke [ 5 ] has been cited 30 times as evidence of the flaws of systematic reviews, according to the Web of Science.

Systematic reviews synthesize the results of primary qualitative and quantitative research, using strategies to reduce bias [ 6 , 7 ]. Systematic reviews of qualitative research summarize and narratively synthesize results using a range of methods [ 8 – 10 ]. Meta-analyses are systematic reviews that summarize results quantitatively. This technique enables researchers to combine the results of several studies into a single effect estimate [ 11 ]. We use the term “systematic review” to refer to both systematic reviews and meta-analyses. Depending on the rigor of the protocol, the quality of the results of systematic reviews can be quite variable. High-quality reviews are characterized by a pre-defined protocol, clearly stated research questions, pre-defined eligibility criteria for studies, reproducible methodology, comprehensive search strategies to identify all studies and unpublished research that meet eligibility criteria, quality assessments including assessment of risk of bias, and systematic syntheses and presentation of findings [ 11 ]. Previous research has demonstrated that study results and conclusions may be influenced by funding sources and author conflicts of interest (COI) [ 12 ]. Primary studies or reviews funded by the pharmaceutical or tobacco industries are more likely to having findings or conclusions that favor the sponsor’s product than those funded by non-industry sources [ 12 – 14 ].

However, little research has been done on whether industry ties influence the conclusions of articles that contain opinions on the use of systematic reviews for health policy decisions. Therefore, we examined financial and other COI disclosed by authors of articles with opinions about the use of systematic reviews in policymaking. We investigated whether affiliations, funding sources, financial ties, and other COI of authors were associated with support or criticism of the use of systematic reviews. Further, we qualitatively summarized the types of arguments presented.

Despite the importance of knowing a study’s funding source and authors’ COI, it is often difficult to find complete information. Disclosure policies vary among journals and may be different for research articles and opinion pieces [ 15 – 17 ]. Therefore, we examined the extent of disclosure present in editorials, comments, letters, and perspectives as compared to other article types that expressed an opinion. In addition, authors’ COI statements and funding disclosures may not be entirely accurate [ 18 , 19 ]. To explore this, we searched for undisclosed industry ties, examining whether such ties were also associated with an article’s stance on the use of systematic reviews.

The primary objective of this study was to determine whether industry ties—disclosed or undisclosed—were associated with the conclusions of opinion articles on the use of systematic reviews for health policy decisions. Our secondary objectives were to determine the substance of the arguments that authors were using to support their positions and how disclosure policies vary by journal and by article type and to determine the extent that these articles are being cited in the scientific literature. This study was not intended to be a systematic review, as we did not undertake a comprehensive search with multiple databases, nor did we contact authors for unpublished papers. This is an analysis of a representative sample of available opinion articles.

Search strategy

Opinion pieces are often difficult to locate, are inconsistently indexed, and may have uninformative titles [ 20 , 21 ]. For our systematic investigation, we conducted a preliminary MEDLINE text search (“Systematic reviews for health policy decision making”) and consulted a reference librarian to further develop our search strategy. We then conducted four more searches of MEDLINE for all articles published between January 1, 1946, and July 31, 2010, with the following MeSH terms and topics: Meta-Analysis as Topic [Mesh] AND health policy [Mesh]; Review Literature as Topic [Mesh] AND health policy [MAJR]; Review literature as topic [Mesh] AND health policy [Mesh]; and “Meta-analysis” and “problem” as topics.

We included editorials, letters, commentaries, and other articles expressing opinions about the use of systematic reviews for policy. Research articles, including studies, reviews, and case examples, were included if the authors expressed an opinion on the use of systematic reviews in policymaking within the body of the article. For included papers, we examined the MEDLINE summary page for associated comments for possible inclusion into our sample. The purpose of this was to understand the debate surrounding a particular publication on the use of systematic reviews in policymaking. In addition, we also included relevant articles from our files.

Selection criteria

Two researchers (SRF and DHO) screened search result titles to identify articles that appeared in peer-reviewed journals that criticized or supported the use of systematic reviews for making health policy decisions. Titles were excluded if they did not include at least one of the terms: systematic reviews, meta-analysis, evidence-based, Cochrane, medical effectiveness research, drug effectiveness review program, or policy and evidence. Titles were additionally excluded if they appeared to be entirely related to statistical methodology or handling of data or only related to the use of systematic reviews solely for clinical decision-making.

All other titles were included. Each author (SRF and DHO) independently screened each search and compared results. The abstracts and full text of all titles selected by either author were reviewed for inclusion. We also included relevant articles from the author’s files that did not appear in the original searches.

Two authors (SRF and DHO) independently screened the abstracts and/or full text of each included title. Discrepancies were resolved by consensus. We included articles that contained significant arguments for or against the use of systematic reviews in policymaking. We considered arguments to be significant if they did not focus solely on a single systematic review, but rather commented on whether systemic reviews did or did not have a role in policymaking. We excluded articles that made no mention or only a brief mention of the use of systematic reviews for policymaking, systematic reviews of a particular topic, tools or instructional articles, articles that were entirely clinically focused, and articles/abstracts that were not available in English or online.

Coding of articles as supportive or critical of the use of systematic reviews to inform policy

We coded articles as supportive if they promoted well-conducted systematic reviews as a strong method for gathering, evaluating, and disseminating bodies of research. Articles that included suggestions for improvement but still advocated systematic reviews as an important knowledge generation tool for policymakers were also coded as supportive. We coded articles as critical if they argued that systematic reviews did not generate useful or accurate knowledge or argued that the method is currently too limited to be useful.

Coding of individual arguments in each article

We initially coded each article inductively, abstracting each argument the authors made to support their positions. After analysis, we then thematically grouped similar arguments together for analysis and coded them as “internal reliability” arguments or “external validity” arguments using adapted definitions from previous studies [ 14 , 22 ]. Although each article was coded overall as supportive or critical, an individual article could contain both critical and supportive arguments or arguments about internal or external validity.

Two authors (DHO and SRF) coded in duplicate 14% (37/264) of the arguments. We agreed 92% (34/37) of the time. We resolved disagreements by consensus. One author (SRF) coded the remainder of the articles. We also abstracted illustrative quotes from the articles, qualitatively summarized the arguments, and conducted a content analysis on the abstracted arguments [ 23 ].

Internal reliability arguments

If the authors argued that a well-conducted systematic review was an internally reliable and valid method of collecting and analyzing data, the argument was coded as supportive of internal reliability. If the authors argued that systematic reviews were internally flawed because of issues of selection bias, misclassification, confounding, study heterogeneity, flaws in data pooling and data analysis, and publication bias, the article was coded as critical of internal reliability.

External validity arguments

If the authors argued that well-conducted systematic reviews were externally valid, useful for highlighting gaps in the literature, or a strong summary source of evidence useful in policy contexts, the argument was coded as a supportive external validity argument. If the authors argued that systematic reviews were not useful for highlighting gaps in the literature, a weak summary source of evidence and/or not useful in policy contexts, and/or not generalizable, the argument was coded as a critical external validity argument.

Coding of disclosures

Affiliation of authors.

We coded each included article by the author affiliations stated in the article. If authors claimed more than one affiliation, each type was coded.

Conflicts of interest of authors

We coded each article for COI by examining disclosures using categories adapted from the International Committee of Medical Journal Editors [ 24 ]. We considered study sponsorship/funding separately from COI. If there was an explicit COI statement contained within the article, it was considered to have a disclosed COI. We then divided COI statements into three categories: no disclosed COI, industry-related COI, and other, which included all non-industry-related COI such as a school or a government. We coded an article as having an industry-related COI if at least one author reported explicit financial ties to a for-profit industry. If an article did not have an explicit conflict of interest statement, we coded it as “no information provided.”

Funding sources of articles

We coded all articles by disclosed funding sources. If there was an explicit statement about funding, or that the article had no funding, the paper was considered to include disclosure of its funding source. We coded funding statements into five categories: self-funded or no funding, private non-profit funding, industry funding, government funding, and mixed funding. If an article did not have an explicit funding statement, we coded it as “no information provided.”

Industry ties

For analysis, we collapsed affiliations, COI, and funding sources into one dichotomous variable, “industry tie or no industry tie”. Our primary outcome was presence of industry ties by supportive or critical argument.

Extent of disclosure

We abstracted the date of publication and extent of disclosure statement in each included article. Some journals have only recently required and/or printed COI statements and funding disclosure statements [ 17 , 25 , 26 ]. We identified current COI and funding disclosure policies for each of the journals in which our study articles appeared. Articles were coded into three categories: no disclosure of either COI or funding source, disclosure of COI or funding source, or disclosure of COI and funding source.

Identification of undisclosed COI

For authors of articles with no disclosed industry ties, we searched MEDLINE for any articles they had published within the 3 years before publication of the included study article. We assessed the disclosures in these articles using criteria set forth by the International Committee of Medical Journal Editors [ 24 ]. We classified the disclosures of the authors as: (a) publication of a study sponsored by industry; (b) affiliation with industry at the time of publication, including, but not limited to, employment, board membership, etc.; (c) disclosure of a financial relationship to industry, including patents, stock, research funding, grants, gifts, consultancy, royalties, expert testimony, service on industry speaker’s bureau, payments for manuscript preparation or review, travel, lectures, etc., or other relevant financial activities/relationships [ 24 ]. If an industry tie was identified in an article published within the 3 prior years of the study article, we categorized the study article as having an undisclosed industry tie. We then discontinued our search for the remaining publications related to the study article, including publications by co-authors.

We also searched the Integrity in Science (ISS) database that contains over 4,000 scientists with corporate ties [ 27 ]. Up until 2009, when the database stopped collecting data, the researchers at ISS routinely scrutinized more than 200 science-based federal advisory committees for undisclosed conflicts of interest and monitored the media and scientific literature for failure to disclose. While the database is not comprehensive, it does provide an additional resource to examine corporate ties of scientists. If an author of a study article was listed as having an industry tie within the last 3 years in the ISS database, we categorized the study article as having an undisclosed industry tie.

Citations of included articles in the academic literature

We searched the academic search engine “Web of Science” and recorded the number of citations for each article included in our study as of November 2013. We used this number as a proxy measure for the extent that the article was being used as evidence within the academic community.

After articles were selected for inclusion, each article was closely read by two authors (SRF and DHO) and a decision was reached about whether the article was supportive or critical of the use of systematic reviews in policymaking. Discrepancies were resolved by a third author (LAB). Using an author-generated data collection sheet, we coded full citation and article type, whether the article was supportive or critical of the use of systematic reviews to inform policy, individual argument types, affiliations of authors, stated conflicts of interest of all authors, funding source of articles, and extent of disclosure, as described below. Once all data from the article had been extracted, we then searched for undisclosed industry ties and individual journal disclosure policies. Finally, we identified the number of times each article had been cited in the academic literature.

All data were analyzed using descriptive statistics, as the use of inferential statistics was inappropriate due to the non-randomness of the sample.

We screened a total of 533 article titles including ten from the author’s files and seven from comments associated with the articles (see Figure  1 ). Twenty-two articles were excluded because they were not in English (see Additional file 1 ). Of the 59 included articles (see Additional file 2 ), nearly 34% (20/59) of the included articles were editorials, comments, letters, and perspectives, while 66% (39/59) of the articles were research, reviews, and case studies. All included articles were published between 1991 and July 2010. We coded 58% (34/59) articles as supportive of using systematic reviews to inform policy and 42% (25/59) as critical.

PRISMA 2009 flow diagram.

Disclosure of affiliation, conflict of interest, and funding source

Table  1 shows the extent of disclosure found in the study articles. Forty-two percent (25/59) of both supportive and critical articles contained no COI or funding source disclosures, whereas 21% (7/34) and 24% (6/25) of the articles, respectively, had both COI and funding disclosures. Disclosure status appeared to improve over time. The median publication date for articles without disclosures was 2003 for both supportive and critical articles, compared to 2008 for articles with COI and funding disclosures.

Table  2 shows the extent of disclosures found in the study articles by article type. Sixty percent (12/20) of the editorials, comments, letters, or perspectives did not contain either COI statements or funding disclosures, compared to 33% of the other article types considered.

Table  3 shows disclosed affiliations, COI, and funding sources of the study articles. We found similar rates of disclosed university affiliations among supportive and critical authors. However, more supportive authors were affiliated with government and considerably more critical authors were affiliated with the pharmaceutical, tobacco, or insurance industries, the three industries represented in our sample. The majority of papers did not have explicit COI statements. However, 26% (9/34) of the supportive authors explicitly stated that they did not have COI, compared to 8% (2/25) of the critical authors. Conversely, 32% (8/25) of the critical authors had an explicitly stated industry-related COI as compared to 3% (1/34) of the supportive authors. Funding statements were almost equally scarce among supportive and critical articles, and no article listed more than one funding source. Of those articles that disclosed funding sources, more articles critical of systematic reviews were funded by industry.

Industry ties—disclosed and undisclosed

Table  4 shows industry ties that were disclosed in the included article and industry ties that were found in the 3-year retrospective search for undisclosed ties. Critical articles were about six times more likely to have a disclosed industry tie than supportive articles; 6% (2/34) of supportive articles and 40% (10/25) of critical articles disclosed industry ties. While this gap narrowed when we included industry ties found in the retrospective 3-year search, critical articles had an industry tie more than twice as often as did supportive articles (35%, 12/34 v. 80%, 20/25).

Journal disclosure policies

At the time we conducted our study, most journals had current disclosure policies. The 15 articles critical of systematic reviews that did not disclose industry ties were published in 12 distinct journals. Among these journals, 9 of the 12 currently require authors to disclose their conflicts of interest and 6 of the 12 require authors to disclose their funding sources. The 32 supportive study articles that did not disclose industry ties were published in 20 distinct journals. Among these journals, 17 of the 20 currently require authors to disclose their conflict of interest and 13 of the 20 require authors to disclose their funding sources (data not shown in tabular form).

Article citations

Table  5 shows the number of times that the included articles have been cited within the academic literature by article type and by whether the article was supportive or critical. Interestingly, 48% (12/25) of articles critical of systematic reviews were editorials, comments, letters, and perspectives, while only 24% (8/34) of supportive articles were editorials, comments, letters, and perspectives. Taken together, editorials, comments, letters, and perspectives were cited less often than other article types (median citations 5 v. 19); however, they still received a number of citations within the literature.

Argument analysis

We abstracted 151 individual arguments from the 34 supportive articles. Of these 151, 37% (56/151) were arguments in support of the internal reliability of well-conducted systematic reviews. Authors typically argued that the methods of well-conducted systematic reviews reduce bias. Lavis et al. contended that this lack of bias, as well the transparency and comprehensiveness of systematic review methodology, provides policymakers with more complete and valid information than they are likely to encounter in individual studies [ 28 ].

We abstracted 113 individual arguments from the 25 critical articles. Of these 113, 62% (70/113) were arguments that were critical of the internal reliability of systematic reviews. Typically, authors argued that the research studies used in systematic reviews are too dissimilar to be combined into a summary statement or statistic, included studies are cherry-picked; and that studies are excluded because of investigator bias, not because of inherent problems with the study. These authors argued that systematic reviews magnify publication bias or that they are often dependent on small numbers of underpowered and methodologically inadequate trials. Critical authors also contend that the methods used are not transparent, that research questions asked by systematic reviews are too narrowly focused to be of any use, or that meta-analyses may use analyses of information from subgroups collected after randomization, resulting in the possibility that the confounding variables may no longer be distributed at random. In a recent critical article that reviewed four published systematic reviews, the authors note that weaknesses encountered in many meta-analyses often stem not from the method itself, but from the poor design and reporting of the trials that make up the body of evidence available to answer the particular research question. They argued that this is exacerbated when the authors of meta-analyses fail to exclude poor-quality studies or to account for the variability in study quality by performing sensitivity analyses [ 29 ]. A commentary in the New England Journal of Medicine questioning the conclusions of several meta-analyses that found that secondhand smoke is harmful to health presented many common criticisms of these studies, including authors’ bias and lack of subject matter expertise, subject matter experts’ misunderstanding of meta-analytic methods, and such methodological flaws as failure to include relevant co-variables or account for heterogeneity or varying effect sizes in different populations [ 5 ].

External validity arguments occurred almost twice as often in supportive arguments as did internal validity arguments (63% v. 37%). The authors of these articles generally argued that well-conducted systematic reviews were the highest level of evidence available [ 30 , 31 ] and effectively summarized large bodies of evidence, resulting in strong external validity [ 32 ]. Supportive authors also asserted that systematic reviews were useful in identifying gaps in the literature and highlighting research priorities.

Thirty-eight percent (43/113) articles critical of systematic reviews contained external validity arguments; however, they generally were extensions of arguments made that were critical of the internal reliability of systematic reviews, arguing that there can be no external validity because of the poor internal validity and reliability. For example, Eysenck [ 33 ] argued that the results of a meta-analysis on the toxicity of secondhand smoke by the National Research Council were scientifically meaningless and should not be used in policymaking because of detection bias resulting from unreliable cause of death information on death certificates, authors’ reliance on effect size, and exclusion of “evidence relevant to the paradigm of research.”

In five critical articles [ 34 – 38 ], and six supportive [ 30 , 39 – 43 ], authors expressed concerns about generalizability of systematic review results. The critical articles primarily argued that results were too general to be applied with confidence to individuals or typical patient populations and warned that the results might not capture complexities of disease and treatment or biological, environmental, and contextual variability. One [ 37 ] raised questions about the utility to physicians of summary statistics; another [ 36 ] described the process of pooling heterogeneous patient data as “the risks run in pooling data from different studies to determine care guidelines are enormous”. Two critical articles discussed social context [ 34 , 38 ]. Neumann et al. criticized the Drug Effectiveness Review Project, which produces systematic reviews of drug-drug comparisons that are used by some US states to develop Medicaid preferred drug lists for not adequately considering costs and a “full societal perspective” [ 34 ]. Most of the critical articles concluded that intractable flaws in systematic review methods greatly decreased their external validity and suggested that policymakers use caution and take other sources of information into account when making health policy decisions. While most of the critical articles did not raise issues of health equity, Ahmad et al. [ 38 ] expressed concerns about the use by public health policymakers in low-income countries of systematic reviews of studies that are conducted primarily in high-income countries. The authors were concerned that systematic reviews on HIV prevention and tobacco cessation lacked cultural and socioeconomic context, rendering the findings potentially less useful for developing policies and practice guidelines in low-income countries. The authors identified steps that systematic review authors could take to increase the strength of review-based recommendations for developing countries.

Six otherwise supportive articles also expressed concerns about generalizability. One [ 30 ] advised policymakers to consider contextual factors and study populations when applying systematic review results. Five articles [ 39 – 43 ] argued that while systematic reviews provided the best available evidence for development of health policies and practice guidelines, they did not adequately address issues of health equity or account for differences in socio-economic status, racial/ethnic diversity, and other social health determinants. All of these articles made recommendations to authors and policymakers for tools and approaches, many of them interdisciplinary, that they can use to enhance the utility of systematic reviews for the development of equitable health policy. One article advocating for evidence-based health promotion recommends that authors increase applicability of their reviews by taking into consideration the salience of the topic and outcome measures and the practical and cost implications of the interventions that are being considered [ 43 ]. Other authors stress the importance of providing credible evidence that is convincing within the health sector and also among non-health collaborators [ 40 ].

Articles without industry ties were more likely to accept systematic review methods as internally reliable and valid, focusing on the relevance of systematic reviews for policymaking, making arguments to support why and how policymakers should consider the results of systematic reviews in their policy decision-making. Articles that were critical of systematic reviews, but did not have industry ties, were more likely to make highly nuanced arguments expressing concern about the generalizability of systematic review results to diverse populations.

Our findings suggest that articles critical of the use of systematic reviews for policymaking are more likely to have industry ties than supportive articles. One possible explanation is that authors of well-conducted systematic reviews, such as those done by The Cochrane Collaboration, examine all of the data available about a topic, including unpublished data when possible, and select for inclusion studies that meet rigorous criteria [ 11 ]. Systematic reviews that are conducted according to strict pre-defined protocols leave industry with less ability to direct or criticize the review findings. By considering all of the available data, the summary effect estimate from a well-done systematic review should represent the current state of the science on the topic in question. Studies that are funded by industry are considered as part of the evidence, but usually not as the only evidence. This may leave industry with less control over the discourse, introducing variability and the possibility that their product, process, or service may not be the most efficacious. For example, unpublished data are more likely to show that a study drug is ineffective than data that are published [ 44 – 47 ]. Systematic reviews attempt to capture and include this information, thus leading to results that may ultimately conflict with the results of industry-sponsored randomized controlled trials.

Our findings also suggest that current journal disclosure requirements for articles are frequently inadequate, especially for editorials, comments, letters, and perspectives. These types of articles lacked any type of disclosure three times as often as the other article types. Editorials, comments, letters, and perspectives play an active role in the academic discourse, as evidenced by the number of times they have been cited in the literature. Together, the included articles have been cited 2,079 times in the literature with editorials, comments, letters, and perspectives cited 366 times. However, it should be noted that citations can either be favorable or unfavorable and that the total number of citations does not necessarily equate with the influence of the article.

Overall, 42% (25/59) of all the included articles, including some that were recently published, did not have any disclosures, leaving readers with no information on the author’s affiliations, funding sources, and COI. Others only had partial disclosures. Our detailed 3-year retrospective search found that 43% (20/47) of articles that had no disclosed industry ties had at least one author with an undisclosed industry tie from the previous 3 years. Having this information readily available is important when readers evaluate the strength of an argument and possible bias. By not requiring the information, or not printing it, journals may leave readers inadequately prepared to accurately judge an article’s accuracy or usefulness.

Our results are consistent with previous studies that showed a positive relationship between industry ties and favorable conclusions in research articles, even when the results of the study do not support the conclusions drawn [ 13 , 14 , 45 , 48 ]. Our research suggests that opinion pieces may be subject to similar biases as other article types and provides evidence that funding sources and COI for all article types should be made transparent.

Our research has several limitations. Because of the inherent difficulties of locating opinion articles, we acknowledge that it is likely that our list of included papers was not comprehensive. Rather, our aim was to perform a systematic search of the topic for a representative sample of articles. As our sample was not a random sample, we elected not to do inferential statistics with the results so that there would be no suggestion of false precision. Because of variable disclosure requirements of journals, we likely have under-estimated the number of industry ties in our sample. For example, one critical article, written during the debates over secondhand smoke [ 33 ], did not list any disclosures, nor did we find any industry ties during our 3-year retrospective search. This paper was catalogued in our study as not having any industry ties. However, a British newspaper [ 49 ] wrote that this particular researcher had received about 800,000 pounds from the tobacco industry. Finally, researchers who do not directly receive payments from industry may work at institutions in which industry funds infrastructure. We did not capture these relationships.

Our findings are important because they demonstrate that industry ties may play a role in opinions expressed in the scientific literature. Moreover, opinion articles are often subsequently cited in the debate as evidence of the controversy over the use of systematic reviews in policymaking. Our results suggest the need for more consistent and complete disclosure for all article types, including articles that may not go through the traditional peer review process, such as editorials, commentaries, letters, and perspectives.

It is important to consider the industry ties of the authors when evaluating arguments regarding the use of systematic reviews by health policymakers and other decision makers. The conduct of systematic reviews is far from a perfect science, and there are substantial and nuanced criticisms of the generalizability of the results. While these arguments may be valid and should not be ignored, it is essential that we consider how knowledge generated by scientists can be critically summarized and subsequently translated for use in policymaking and population health decision-making. Well-conducted systematic reviews provide a rigorous and transparent method for knowledge dissemination and should be improved, not discarded.

Bero LA, Jadad AR: How consumers and policymakers can use systematic reviews for decision making. Ann Intern Med. 1997, 127: 37-42. 10.7326/0003-4819-127-1-199707010-00007.

Article   CAS   PubMed   Google Scholar  

Sweet M, Moynihan R, Fund MM: Improving Population Health: The Uses of Systematic Reviews. 2007, New York: Milbank Memorial Fund

Google Scholar  

Fox DM: The governance of standard-setting to improve health. Prev Chronic Dis. 2010, 7: A123-

PubMed   PubMed Central   Google Scholar  

Fox DM: Systematic reviews and health policy: the influence of a project on perinatal care since 1988. Milbank Q. 2011, 89: 425-449. 10.1111/j.1468-0009.2011.00635.x.

Article   PubMed   PubMed Central   Google Scholar  

Bailar JC: Passive smoking, coronary heart disease, and meta-analysis. N Engl J Med. 1999, 340: 958-959. 10.1056/NEJM199903253401211.

Article   PubMed   Google Scholar  

Chalmers I, Hedges LV, Cooper H: A brief history of research synthesis. Eval Health Prof. 2002, 25: 12-37. 10.1177/0163278702025001003.

Meerpohl JJ, Herrle F, Antes G, von Elm E: Scientific value of systematic reviews: survey of editors of core clinical journals. PLoS One. 2012, 7: e35732-10.1371/journal.pone.0035732.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Greenhalgh T, Potts HW, Wong G, Bark P, Swinglehurst D: Tensions and paradoxes in electronic patient record research: a systematic literature review using the meta-narrative method. Milbank Q. 2009, 87: 729-788. 10.1111/j.1468-0009.2009.00578.x.

Thomas J, Harden A: Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008, 8: 45-10.1186/1471-2288-8-45.

Barnett-Page E, Thomas J: Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009, 9: 59-10.1186/1471-2288-9-59.

Cochrane Handbook for Systematic Reviews of Interventions Version 5.1. 0 [updated March 2011]. The Cochrane Collaboration. Edited by: Higgins JPT, Green S. 2011, [ http://www.cochrane-handbook.org ]

Lundh A, Sismondo S, Lexchin J, Busuioc Octavian A, Bero L: Industry sponsorship and research outcome. Cochrane Database Syst Rev. 2012, Issue 12. Art. No.: MR000033. doi:10.1002/14651858.MR000033.pub2. 2012

Bartels RH, Delye H, Boogaarts J: Financial disclosures of authors involved in spine research: an underestimated source of bias. Eur Spine J. 2012, 21: 1229-1233. 10.1007/s00586-011-2086-x.

Yank V, Rennie D, Bero LA: Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study. BMJ. 2007, 335: 1202-1205. 10.1136/bmj.39376.447211.BE.

Forbes TL: Author disclosure of conflict of interest in vascular surgery journals. J Vasc Surg. 2011, 54: 55S-58S. 10.1016/j.jvs.2011.06.019.

Kesselheim AS, Lee JL, Avorn J, Servi A, Shrank WH, Choudhry NK: Conflict of interest in oncology publications. Cancer. 2012, 118: 188-195. 10.1002/cncr.26237.

Blum JA, Freeman K, Dart RC, Cooper RJ: Requirements and definitions in conflict of interest policies of medical journals. JAMA. 2009, 302: 2230-2234. 10.1001/jama.2009.1669.

Okike K, Kocher MS, Wei EX, Mehlman CT, Bhandari M: Accuracy of conflict-of-interest disclosures reported by physicians. N Engl J Med. 2009, 361: 1466-1474. 10.1056/NEJMsa0807160.

Bero LA, Glantz S, Hong M-K: The limits of competing interest disclosures. Tob Control. 2005, 14: 118-126.

CAS   PubMed   PubMed Central   Google Scholar  

Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A: Applying systematic review methods to studies of people’s views: an example from public health research. J Epidemiol Community Health. 2004, 58: 794-800. 10.1136/jech.2003.014829.

Woodman J, Harden A, Thomas J, Brunton J, Kavanagh J, Stansfield C: Searching for systematic reviews of the effects of social and environmental interventions: a case study of children and obesity. J Med Libr Assoc. 2010, 98: 140-10.3163/1536-5050.98.2.006.

Schotland MS, Bero LA: Evaluating public commentary and scientific evidence submitted in the development of a risk assessment. Risk Anal. 2002, 22: 131-140. 10.1111/0272-4332.t01-1-00011.

Weber RP: Basic Content Analysis. 1990, Newbury Park: Sage

Book   Google Scholar  

International Committee of Medical Journal Editors: International Committee of Medical Journal Editors (ICMJE): uniform requirements for manuscripts submitted to biomedical journals: writing and editing for biomedical publication. Haematologica. 2004, 89: 264-

Ancker JS, Flanagin A: A comparison of conflict of interest policies at peer-reviewed journals in different scientific disciplines. Sci Eng Ethics. 2007, 13: 147-157. 10.1007/s11948-007-9011-z.

Meerpohl JJ, Wolff RF, Niemeyer CM, Antes G, von Elm E: Editorial policies of pediatric journals: survey of instructions for authors. Arch Pediatr Adolesc Med. 2010, 164: 268-272. 10.1001/archpediatrics.2009.287.

Integrity in Science. [ http://www.cspinet.org/integrity/index.html ]

Lavis JN, Posada FB, Haines A, Osei E: Use of research to inform public policymaking. Lancet. 2004, 364: 1615-1621. 10.1016/S0140-6736(04)17317-0.

Coyne JC, Thombs BD, Hagedoorn M: Ain’t necessarily so: review and critique of recent meta-analyses of behavioral medicine interventions in health psychology. Health Psychol. 2010, 29: 107-

Chan KS, Morton SC, Shekelle PG: Systematic reviews for evidence-based management: how to find them and what to do with them. Am J Manag Care. 2004, 10: 806-812.

PubMed   Google Scholar  

Coffman JM, HONG MK, Aubry WM, Luft HS, Yelin E: Translating medical effectiveness research into policy: lessons from the California Health Benefits Review Program. Milbank Q. 2009, 87: 863-902. 10.1111/j.1468-0009.2009.00582.x.

Brownson RC, Fielding JE, Maylahn CM: Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health. 2009, 30: 175-201. 10.1146/annurev.publhealth.031308.100134.

Eysenck H: Meta-analysis or best-evidence synthesis?. J Eval Clin Pract. 1995, 1: 29-36. 10.1111/j.1365-2753.1995.tb00005.x.

Neumann PJ, Drummond MF, Jönsson B, Luce BR, Schwartz JS, Siebert U, Sullivan SD: Are key principles for improved health technology assessment supported and used by health technology assessment organizations?. Int J Technol Assess Health Care. 2010, 26: 71-78.

Laupacis A, Straus S: Systematic reviews: time to address clinical and policy relevance as well as methodological rigor. Ann Intern Med. 2007, 147: 273-274. 10.7326/0003-4819-147-4-200708210-00180.

Marini JJ: Meta-analysis: convenient assumptions and inconvenient truth. Crit Care Med. 2008, 36: 328-329. 10.1097/01.CCM.0000297959.02114.2B.

Goodman SN: Have you ever meta-analysis you didn’t like?. Ann Intern Med. 1991, 114: 244-246. 10.7326/0003-4819-114-3-244.

Ahmad N, Boutron I, Dechartres A, Durieux P, Ravaud P: Applicability and generalisability of the results of systematic reviews to public health practice and policy: a systematic review. Trials. 2010, 11: 20-10.1186/1745-6215-11-20.

Tugwell P, Robinson V, Grimshaw J, Santesso N: Systematic reviews and knowledge translation. Bull World Health Organ. 2006, 84: 643-651. 10.2471/BLT.05.026658.

Rychetnik L, Wise M: Advocating evidence-based health promotion: reflections and a way forward. Health Promot Int. 2004, 19: 247-257. 10.1093/heapro/dah212.

Petticrew M, Tugwell P, Welch V, Ueffing E, Kristjansson E, Armstrong R, Doyle J, Waters E: Better evidence about wicked issues in tackling health inequities. J Public Health. 2009, 31: 453-456. 10.1093/pubmed/fdp076.

Article   Google Scholar  

Odierna DH, Bero LA: Systematic reviews reveal unrepresentative evidence for the development of drug formularies for poor and nonwhite populations. J Clin Epidemiol. 2009, 62: 1268-1278. 10.1016/j.jclinepi.2009.01.009.

Gruen RL, Morris PS, McDonald EL, Bailie RS: Making systematic reviews more useful for policy-makers. Bull World Health Organ. 2005, 83: 480-

Rising K, Bacchetti P, Bero L: Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation. PLoS Med. 2008, 5: e217-10.1371/journal.pmed.0050217.

Hart B, Lundh A, Bero L: Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ. 2012, 344: d7202-10.1136/bmj.d7202.

Gøtzsche PC: Why we need easy access to all data from all clinical trials and how to accomplish it. Trials. 2011, 12: 249-10.1186/1745-6215-12-249.

Doshi P, Jefferson T, Del Mar C: The imperative to share clinical study reports: recommendations from the Tamiflu experience. PLoS Med. 2012, 9: e1001201-10.1371/journal.pmed.1001201.

Boutron I, Dutton S, Ravaud P, Altman DG: Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010, 303: 2058-2064. 10.1001/jama.2010.651.

Pringle P: Eysenck took pounds 800,000 tobacco funds. The Independent. 1996, London: Independent Print Limited

Download references

Acknowledgements

We would like to thank Mark Gibson for the assistance with development of the study protocol. We are also grateful to the research staff at the UCSF Library, the UCSF Tobacco Center, and the Drug Industry Document Archive for the assistance with search strategies.

This project was funded by the Flight Attendant Medical Research Institute (FAMRI), Miami, FL, USA. Susan Forsyth is funded by a dissertation award from the Tobacco-Related Disease Research Program (TRDRP; grant #22DT-0003).

Author information

Authors and affiliations.

Department of Social and Behavioral Sciences, School of Nursing, University of California, San Francisco, CA, USA

Susan R Forsyth

Department of Clinical Pharmacy, University of California, San Francisco, CA, USA

Donna H Odierna, David Krauth & Lisa A Bero

Institute for Health Policy Studies, University of California, San Francisco, CA, USA

Lisa A Bero

Charles Perkins Centre, University of Sydney, Sydney, Australia

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Lisa A Bero .

Additional information

Competing interests.

Susan Forsyth, Donna Odierna, and Lisa Bero are affiliated with The Cochrane Collaboration. The authors have no other competing interests to declare.

Authors’ contributions

SRF assisted with the project development, collected and analyzed the data, and prepared drafts of the manuscript. DHO collaborated on the project development, developed the initial coding scheme, collected the data, wrote sections of the manuscript, and reviewed the drafts. DK collected the data and reviewed the drafts. LAB collaborated on the project development, contributed to the analysis of data, and reviewed the drafts. All authors read and approved the final manuscript.

Electronic supplementary material

13643_2014_303_moesm1_esm.docx.

Additional file 1: Study articles excluded for being in a language other than English. Additional file 1 lists all studies from the original sample that were excluded because they were not in English. (DOCX 105 KB)

13643_2014_303_MOESM2_ESM.docx

Additional file 2: Included study articles. Additional file 2 lists all studies included in the sample, including studies obtained from database searching, related commentary, and author’s files. (DOCX 112 KB)

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Rights and permissions.

Open Access  This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/ .

The Creative Commons Public Domain Dedication waiver ( https://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Forsyth, S.R., Odierna, D.H., Krauth, D. et al. Conflicts of interest and critiques of the use of systematic reviews in policymaking: an analysis of opinion articles. Syst Rev 3 , 122 (2014). https://doi.org/10.1186/2046-4053-3-122

Download citation

Received : 06 June 2014

Accepted : 14 October 2014

Published : 18 November 2014

DOI : https://doi.org/10.1186/2046-4053-3-122

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Systematic reviews
• Health policy
• Conflicts of interest

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Association between conflicts of interest and favourable recommendations in clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews: systematic review

Affiliations.

• 1 Centre for Evidence-Based Medicine Odense (CEBMO), Odense University Hospital, Odense, Denmark [email protected].
• 2 Department of Clinical Research, University of Southern Denmark, Odense, Denmark.
• 3 Open Patient data Exploratory Network (OPEN), Odense University Hospital, Odense, Denmark.
• 4 Nordic Cochrane Centre, Rigshospitalet, Copenhagen, Denmark.
• 5 Center for Bioethics and Humanities, University of Colorado, CO, USA.
• 6 Centre for Evidence-Based Medicine Odense (CEBMO), Odense University Hospital, Odense, Denmark.
• 7 ENT Clinic Hobro, Hobro, Denmark.
• 8 Stasjonsgata Legekontor, Hokksund, Norway.
• 9 Department of Infectious Diseases, Hvidovre Hospital, Hvidovre, Denmark.
• PMID: 33298430
• PMCID: PMC8030127
• DOI: 10.1136/bmj.m4234

Objective: To investigate the association between conflicts of interest and favourable recommendations in clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews.

Design: Systematic review.

Eligibility criteria: Studies that compared the association between conflicts of interest and favourable recommendations of drugs or devices (eg, recommending a drug) in clinical guidelines, advisory committee reports, opinion pieces (eg, editorials), or narrative reviews.

Data sources: PubMed, Embase, Cochrane Methodology Register (from inception to February 2020), reference lists, Web of Science, and grey literature.

Data extraction and analysis: Two authors independently extracted data and assessed the methodological quality of the studies. Pooled relative risks and 95% confidence intervals were estimated using random effects models (relative risk >1 indicates that documents with conflicts of interest more often had favourable recommendations than documents with no conflicts of interest). Financial and non-financial conflicts of interest were analysed separately, and the four types of documents were analysed separately (preplanned) and combined (post hoc).

Results: 21 studies that analysed 106 clinical guidelines, 1809 advisory committee reports, 340 opinion pieces, and 497 narrative reviews were included. Unpublished data were received for 11 studies (eight full datasets and three summary datasets). 15 studies showed risk of confounding because the compared documents could differ in factors other than conflicts of interest (eg, different drugs used for different populations). The relative risk for associations between financial conflicts of interest and favourable recommendations for clinical guidelines was 1.26 (95% confidence interval 0.93 to 1.69; four studies of 86 clinical guidelines), for advisory committee reports was 1.20 (0.99 to 1.45; four studies of 629 advisory committee reports), for opinion pieces was 2.62 (0.91 to 7.55; four studies of 284 opinion pieces), and for narrative reviews was 1.20 (0.97 to 1.49; four studies of 457 narrative reviews). An analysis of all four types of documents combined supported these findings (1.26, 1.09 to 1.44). In one study that investigated specialty interests, the association between including radiologists as authors of guidelines and recommending routine breast cancer was: relative risk 2.10, 95% confidence interval 0.92 to 4.77; 12 clinical guidelines).

Conclusions: We interpret our findings to indicate that financial conflicts of interest are associated with favourable recommendations of drugs and devices in clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews. Limitations of this review were risk of confounding in the included studies and the statistical imprecision of individual analyses of each document type. It is not certain whether non-financial conflicts of interest influence recommendations.

Systematic review registration: Cochrane Methodology Review Protocol MR000040.

© Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

PubMed Disclaimer

Conflict of interest statement

Competing interests: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf and declare no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; and no other relationships or activities that could appear to have influenced the submitted work. LB is coauthor of one of the included studies. LB was not involved in the study inclusion, data extraction, and methodological quality assessment of any studies.

Flow chart of study inclusion

Methodological quality in included studies

Meta-analysis of association between financial…

Meta-analysis of association between financial conflicts of interest and favourable recommendations for each…

Meta-analysis of association between financial conflicts of interest and favourable votes of committee…

Similar articles

• Conflicts of interest in clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews: associations with recommendations. Nejstgaard CH, Bero L, Hróbjartsson A, Jørgensen AW, Jørgensen KJ, Le M, Lundh A. Nejstgaard CH, et al. Cochrane Database Syst Rev. 2020 Dec 8;12(12):MR000040. doi: 10.1002/14651858.MR000040.pub3. Cochrane Database Syst Rev. 2020. PMID: 33289919 Free PMC article.
• Financial conflicts of interest in systematic reviews: associations with results, conclusions, and methodological quality. Hansen C, Lundh A, Rasmussen K, Hróbjartsson A. Hansen C, et al. Cochrane Database Syst Rev. 2019 Aug 5;8(8):MR000047. doi: 10.1002/14651858.MR000047.pub2. Cochrane Database Syst Rev. 2019. PMID: 31425611 Free PMC article. Review.
• Financial conflict of interest disclosure and voting patterns at Food and Drug Administration Drug Advisory Committee meetings. Lurie P, Almeida CM, Stine N, Stine AR, Wolfe SM. Lurie P, et al. JAMA. 2006 Apr 26;295(16):1921-8. doi: 10.1001/jama.295.16.1921. JAMA. 2006. PMID: 16639051
• Reporting of financial and non-financial conflicts of interest by authors of systematic reviews: a methodological survey. Hakoum MB, Anouti S, Al-Gibbawi M, Abou-Jaoude EA, Hasbani DJ, Lopes LC, Agarwal A, Guyatt G, Akl EA. Hakoum MB, et al. BMJ Open. 2016 Aug 10;6(8):e011997. doi: 10.1136/bmjopen-2016-011997. BMJ Open. 2016. PMID: 27515760 Free PMC article.
• Association between industry affiliation and position on cardiovascular risk with rosiglitazone: cross sectional systematic review. Wang AT, McCoy CP, Murad MH, Montori VM. Wang AT, et al. BMJ. 2010 Mar 18;340:c1344. doi: 10.1136/bmj.c1344. BMJ. 2010. PMID: 20299696 Free PMC article. Review.
• Non-research payments to board-certified cardiologists from pharmaceutical industry in Japan from 2016 to 2019: a retrospective analysis. Murayama A, Higuchi K, Senoo Y. Murayama A, et al. BMJ Open. 2024 Jul 31;14(7):e083445. doi: 10.1136/bmjopen-2023-083445. BMJ Open. 2024. PMID: 39089711 Free PMC article.
• Cross-sectional analysis of pharmaceutical industry payments to authors of clinical practice guidelines for bipolar disorder and major depressive disorder in Japan. Murayama A, Kugo H, Senoo Y. Murayama A, et al. BMJ Open. 2024 Jun 21;14(6):e086396. doi: 10.1136/bmjopen-2024-086396. BMJ Open. 2024. PMID: 38908845 Free PMC article.
• Identifying and Minimizing Incentives for Competing Interests in Sports Medicine Publications. Shrier I, Impellizzeri FM, Stovitz SD. Shrier I, et al. Sports Med. 2024 Aug;54(8):1991-2000. doi: 10.1007/s40279-024-02037-w. Epub 2024 May 7. Sports Med. 2024. PMID: 38714641
• Conflicts of Interest Among Cardiology Clinical Practice Guideline Authors in Japan. Murayama A, Aizawa M, Byreddy KR, Higuchi K, Senoo Y. Murayama A, et al. J Am Heart Assoc. 2024 Apr 16;13(8):e034506. doi: 10.1161/JAHA.124.034506. Epub 2024 Apr 12. J Am Heart Assoc. 2024. PMID: 38606773 Free PMC article.
• Financial Relationships Between Pharmaceutical Companies and Internal Medicine Societies. Murayama A, Higuchi K, Senoo Y. Murayama A, et al. JAMA Netw Open. 2024 Apr 1;7(4):e244777. doi: 10.1001/jamanetworkopen.2024.4777. JAMA Netw Open. 2024. PMID: 38568694 Free PMC article.
• Grundy Q, Dunn AG, Bourgeois FT, Coiera E, Bero L. Prevalence of Disclosed Conflicts of Interest in Biomedical Research and Associations With Journal Impact Factors and Altmetric Scores. JAMA 2018;319:408-9. 10.1001/jama.2017.20738 - DOI - PMC - PubMed
• Xu J, Emenanjo O, Ortwerth M, Lurie P. Association of Appearance of Conflicts of Interest With Voting Behavior at FDA Advisory Committee Meetings-A Cross-sectional Study. JAMA Intern Med 2017;177:1038-40. 10.1001/jamainternmed.2017.1917 - DOI - PMC - PubMed
• Bindslev JB, Schroll J, Gøtzsche PC, Lundh A. Underreporting of conflicts of interest in clinical practice guidelines: cross sectional study. BMC Med Ethics 2013;14:19. 10.1186/1472-6939-14-19 - DOI - PMC - PubMed
• Bero LA, Grundy Q. Why Having a (Nonfinancial) Interest Is Not a Conflict of Interest. PLoS Biol 2016;14:e2001221. 10.1371/journal.pbio.2001221 - DOI - PMC - PubMed
• Lundh A, Lexchin J, Mintzes B, Schroll JB, Bero L. Industry sponsorship and research outcome. Cochrane Database Syst Rev 2017;2:MR000033. - PMC - PubMed

Publication types

• Search in MeSH

Related information

Linkout - more resources, full text sources.

• Europe PubMed Central
• Ovid Technologies, Inc.
• PubMed Central
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Managing Conflicts of Interest and Fiduciary Duties

Managing conflicts of interest and fiduciary duties requires a multifaceted approach that prioritizes transparency, accountability, and stakeholder trust. Effective management involves understanding fiduciary duties, identifying potential conflicts of interest, and establishing clear boundaries and roles. Disclosing conflicts in a timely and transparent manner is also vital. By implementing policies and procedures that promote integrity and accountability, individuals and organizations can mitigate risks and liability associated with conflicts of interest. A thorough understanding of these concepts is necessary for maintaining the integrity of fiduciary relationships and upholding obligations to stakeholders, and further exploration of these principles can provide valuable insights.

Table of Contents

Understanding Fiduciary Duties

Fiduciary duty is a cornerstone of trust relationships, obligating individuals or organizations to prioritize the interests of others above their own. In managing fiduciary obligations, individuals must adhere to the duty standards of care, loyalty, and confidentiality. This involves avoiding actions that could compromise their position, adhering to transparency in their dealings, and always maintaining accountability to those whom they owe fiduciary duties.

These standards, along with laws governing specific contexts (such as probate law for personal representatives, securities laws for broker-dealers), play critical roles in framing what these individuals may do with resources, managing principal-stakeholder and -owner duties where related individuals who undertake additional caretaker-type of trust service relationship-related employment require critical vetting protocols including by experts representing required vested concerns – – key checks through proactive responsibility means if directly the expert doing real asset discovery responsibilities vested.

Strategies guiding formulation guarantee constant renewal upsurges applied support operations bolster capacity serving targeted fulfillment standard outcomes during said official use phases guarantee balanced check equity set amid obligations; Fiduciary and statutory (power-law under delegation means having), balancing goals performance relative applicable full operating pre-standard compliant level guidelines operating officially being legally law recognized principle-driven guide model forms further strengthening provisions operationsAdapterManager )

Correcting using previous paragraphs

Fiduciary duty is a cornerstone of trust relationships, obligating individuals or organizations to prioritize the interests of others above their own. In managing fiduciary obligations, individuals must adhere to the duty standards of care, loyalty, and confidentiality. This involves avoiding actions that could compromise their position, adhering to transparency in their dealings, and always maintaining accountability to those whom they owe fiduciary duties. In governance by authority standard vested organizations recognized it imposes major significance relationship given mutual decision entrusting vital real-value financial type possessions be responsibility owed prioritization expected both conduct dealing interaction performed compliance on guidance code when this subject standards codes other.

Ongoing emphasis here typically the individuals owing respective have other functions support staff leadership directing activity processes always adhere above

Identifying Conflicts of Interest

Identifying conflicts of interest is a critical step in managing fiduciary duties, as it enables individuals and organizations to recognize and address potential biases and competing interests. There are several types of conflicts that can arise, including personal, financial, and organizational conflicts, each with distinct characteristics and implications. Effective identification of conflicts requires a nuanced understanding of personal interests, organizational dynamics, and the interplay between individual and institutional goals.

Types of Conflicts

One key factor in maintaining objectivity in both professional and personal life is acknowledging that numerous relationships, activities, and situations have the potential to generate conflicts of interest. These conflicts can be broadly categorized into several types, each with distinct characteristics and implications. Self-dealing, for instance, is a type of conflict where an individual engages in a transaction or decision-making process that benefits themselves directly or indirectly. This can manifest in various forms, such as a fiduciary acquiring assets or property that belongs to the entity they represent, or using their position to secure personal advantages.

Another common type of conflict is driven by financial motives. This occurs when an individual's financial interests or those of their associates influence their decision-making, potentially compromising their objectivity and impartiality. Examples of financial motive-driven conflicts include insider trading, accepting bribes or kickbacks, and engaging in transactions that yield personal financial gains. Recognizing these types of conflicts is vital for maintaining the integrity of professional and personal relationships, as well as upholding fiduciary duties and responsibilities. Effective management of these conflicts is critical to preventing harm to individuals, organizations, and society as a whole.

Recognizing Personal Interests

Recognizing personal interests is a vital step in identifying conflicts of interest, as it allows individuals to pinpoint potential biases and take proactive measures to mitigate their impact. Personal interests can manifest in various forms, including financial, emotional, or familial ties. Individuals must acknowledge these interests to guarantee that their decision-making processes remain impartial and objective.

Personal bias can profoundly influence an individual's judgment, often unconsciously. Family ties, in particular, can create conflicts of interest, as individuals may prioritize the interests of their relatives over those of their organization or stakeholders. It is vital to recognize these biases and establish measures to manage them effectively.

To identify personal interests, individuals should conduct a thorough self-assessment, considering their relationships, financial dealings, and personal affiliations. This introspection enables them to recognize potential conflicts and develop strategies to mitigate their impact. By acknowledging and addressing personal interests, individuals can maintain their integrity and uphold their fiduciary duties, ultimately protecting the interests of their organization and stakeholders. Effective management of personal interests is pivotal in preventing conflicts of interest and guaranteeing sound decision-making.

Identifying Organizational Conflicts

Within an organizational context, conflicts of interest can arise from a variety of structural, cultural, or operational factors. Identifying these conflicts is vital to maintaining the integrity of the organization and upholding fiduciary duties. Organizational silos, where departments or teams operate independently with little communication or collaboration, can create an environment ripe for conflicts of interest. Departmental rivalries can also lead to conflicting priorities and interests, as each department may prioritize its own goals over the organization's overall objectives.

To identify organizational conflicts, it is vital to analyze the organization's structure, policies, and procedures. This includes examining the organization's governance framework, decision-making processes, and communication channels. Identifying potential conflicts of interest requires a thorough understanding of the organization's operations, including its financial and operational systems. By recognizing these potential conflicts, organizations can take proactive steps to mitigate them and guarantee that the interests of the organization and its stakeholders are protected. Effective identification and management of organizational conflicts of interest are critical to maintaining trust, integrity, and accountability within the organization.

Establishing Clear Boundaries

Establishing clear boundaries is a critical component of managing conflicts of interest, as it enables individuals to distinguish between their professional and personal roles. Defining professional roles involves outlining specific responsibilities, authority, and expectations to prevent overlap or confusion. By setting personal limits, individuals can also maintain a clear separation between their personal interests and professional obligations, thereby reducing the risk of conflicts of interest.

Defining Professional Roles

Defining Professional Roles (Establishing Clear Boundaries)

Defining professional roles is a crucial step in maintaining the integrity of an organization, as it enables individuals to understand their responsibilities and the boundaries that govern their actions. Clear role definitions promote accountability, transparency, and professionalism, thereby minimizing the potential for conflicts of interest.

To achieve role clarity, organizations must establish professional expectations that are concise, well-communicated, and aligned with their mission and values. Key aspects of defining professional roles include:

• Establishing clear job descriptions: Define specific duties, responsibilities, and expectations for each role to avoid ambiguity and overlap.
• Defining reporting relationships: Identify supervisors, subordinates, and peers to guarantee that decision-making authority is clear.
• Specifying professional obligations: Outline expectations for professional conduct, including confidentiality, discretion, and loyalty.
• Identifying potential conflicts of interest: Anticipate and address potential conflicts that may arise due to an individual's professional or personal relationships.

Setting Personal Limits

Managing Conflicts of Interest

Setting personal limits is a crucial component of maintaining professional integrity, as it enables individuals to navigate complex situations and make informed decisions that align with their organizational responsibilities. Establishing clear boundaries is necessary in preventing conflicts of interest and ensuring that fiduciary duties are upheld. Personal boundaries serve as a framework for decision-making, guiding individuals to prioritize their professional obligations over personal interests.

Through self-reflection, individuals can identify areas where personal boundaries may be compromised and take proactive steps to establish clear limits. By doing so, they can maintain their professional integrity, uphold their fiduciary duties, and prevent conflicts of interest from arising.

Disclosing Conflicts Effectively

Effective conflict disclosure is a crucial aspect of maintaining the integrity of an organization or individual's decision-making processes. It involves providing clear and timely information about potential conflicts of interest to relevant stakeholders. Disclosure timing is critical, as it can profoundly impact the effectiveness of the disclosure. Ideally, conflicts should be disclosed as soon as they arise, allowing stakeholders to make informed decisions.

When disclosing conflicts, it is vital to ponder conflict nuances, such as the nature and extent of the conflict, as well as the potential impact on decision-making processes. Effective disclosure should be transparent, accurate, and exhaustive. Key considerations for effective conflict disclosure include:

• Providing clear and concise information about the conflict
• Disclosing conflicts in a timely manner, ideally as soon as they arise
• Pondering the potential impact of the conflict on decision-making processes
• Ensuring that disclosure is transparent, accurate, and exhaustive

Managing Conflicting Interests

Effective conflict management necessitates a nuanced understanding of the complex dynamics at play when conflicting interests arise. In situations where multiple stakeholders are involved, interest alignment becomes a critical consideration. This involves identifying and prioritizing the interests of various stakeholders, including shareholders, clients, employees, and the organization itself. Stakeholder prioritization is vital in managing conflicting interests, as it enables decision-makers to allocate resources and make decisions that balance competing demands.

To manage conflicting interests effectively, organizations must establish clear policies and procedures that promote transparency and accountability. This includes defining roles and responsibilities, setting boundaries, and establishing protocols for managing conflicts of interest. By doing so, organizations can minimize the risk of conflicts and guarantee that decisions are made in the best interests of all stakeholders. Effective interest alignment and stakeholder prioritization are critical components of a robust conflict management framework, enabling organizations to navigate complex conflicts and maintain the trust of their stakeholders. By prioritizing stakeholder interests, organizations can promote a culture of integrity and accountability.

Mitigating Risks and Liability

Mitigating risks and liability is crucial in the context of conflicts of interest, as unresolved or poorly managed conflicts can lead to severe reputational damage, financial losses, and regulatory penalties. Effective risk management is critical to minimize potential harm and guarantee compliance with regulatory requirements. A thorough risk assessment should be conducted to identify potential conflicts of interest and evaluate their likelihood and potential impact.

To mitigate risks and liability, organizations can implement the following strategies:

• Conduct regular risk assessments to identify potential conflicts of interest and evaluate their likelihood and potential impact.
• Develop and implement compliance strategies to prevent or manage conflicts of interest, including policies, procedures, and training programs.
• Establish a system for reporting and addressing conflicts of interest, including a clear escalation process for potential issues.
• Regularly review and update compliance strategies to guarantee they remain effective and aligned with regulatory requirements.

Maintaining Transparency Accountability

Transparency and accountability are crucial components in the effective management of conflicts of interest, as they enable organizations to demonstrate their commitment to ethical conduct and compliance with regulatory requirements. To maintain transparency, organizations should establish open communication channels that facilitate the free flow of information among stakeholders. This can be achieved through regular reporting, disclosure of potential conflicts, and provision of clear guidelines on conflict management procedures.

Organizations should also establish a culture of accountability, where individuals are held responsible for their actions and decisions. This can be achieved by setting clear expectations, providing training on ethical standards, and enforcing consequences for non-compliance. Effective accountability mechanisms should include regular audits, monitoring of compliance, and prompt investigation of alleged breaches. By maintaining transparency and accountability, organizations can guarantee that conflicts of interest are identified and managed in a fair and transparent manner, thereby upholding their fiduciary duties and maintaining stakeholder trust. This approach also helps to promote a culture of integrity and adherence to ethical standards.

Frequently Asked Questions

Can personal relationships impact fiduciary duties.

Personal relationships may compromise objectivity, as social ties and emotional bonds can influence decision-making. Fiduciaries must separate personal connections from professional obligations to guarantee impartiality and maintain the integrity of their duties.

How Often Should Conflicts Be Reassessed?

Regular reassessment of conflicts is vital, ideally following significant events or changes. Periodic risk assessments, typically quarterly or bi-annually, can help identify emerging conflict triggers, ensuring proactive mitigation and minimizing potential risks and liabilities.

Are Gifts From Clients Considered Conflicts?

Gifts from clients can be perceived as conflicts if not managed properly. Establishing clear gift policies and acceptance criteria is vital to maintain objectivity and prevent undue influence, ensuring impartial decision-making and transparency.

Can Family Members Be Fiduciaries Too?

Family members can indeed assume fiduciary roles, often due to family dynamics and inherited responsibility, necessitating a delicate balance between personal relationships and objective decision-making, which can be challenging to maintain.

Do Fiduciary Duties Apply to Volunteers?

Volunteer roles often involve governance structures that imply fiduciary duties, as volunteers' motivations may align with organizational goals, necessitating a duty of care and loyalty, particularly in decision-making capacities or when overseeing organizational assets.

• Business & Money

Sorry, there was a problem.

Download the free Kindle app and start reading Kindle books instantly on your smartphone, tablet, or computer - no Kindle device required .

Read instantly on your browser with Kindle for Web.

Using your mobile phone camera - scan the code below and download the Kindle app.

Image Unavailable

• To view this video download Flash Player

Conflicts of Interest in Initial Public Offerings (IPOs): A Systematic Review of Literature Paperback – December 3, 2009

• Print length 96 pages
• Language English
• Publication date December 3, 2009
• Dimensions 5.91 x 0.22 x 8.66 inches
• ISBN-10 3639180313
• ISBN-13 978-3639180312
• See all details

Product details

• Publisher ‏ : ‎ VDM Verlag Dr. Müller (December 3, 2009)
• Language ‏ : ‎ English
• Paperback ‏ : ‎ 96 pages
• ISBN-10 ‏ : ‎ 3639180313
• ISBN-13 ‏ : ‎ 978-3639180312
• Item Weight ‏ : ‎ 5.4 ounces
• Dimensions ‏ : ‎ 5.91 x 0.22 x 8.66 inches

Customer reviews

• 5 star 4 star 3 star 2 star 1 star 5 star 0% 0% 0% 0% 0% 0%
• 5 star 4 star 3 star 2 star 1 star 4 star 0% 0% 0% 0% 0% 0%
• 5 star 4 star 3 star 2 star 1 star 3 star 0% 0% 0% 0% 0% 0%
• 5 star 4 star 3 star 2 star 1 star 2 star 0% 0% 0% 0% 0% 0%
• 5 star 4 star 3 star 2 star 1 star 1 star 0% 0% 0% 0% 0% 0%

Customer Reviews, including Product Star Ratings help customers to learn more about the product and decide whether it is the right product for them.

To calculate the overall star rating and percentage breakdown by star, we don’t use a simple average. Instead, our system considers things like how recent a review is and if the reviewer bought the item on Amazon. It also analyzed reviews to verify trustworthiness.

No customer reviews

• About Amazon
• Investor Relations
• Amazon Devices
• Amazon Science
• Sell products on Amazon
• Sell on Amazon Business
• Sell apps on Amazon
• Become an Affiliate
• Advertise Your Products
• Self-Publish with Us
• Host an Amazon Hub
• › See More Make Money with Us
• Amazon Business Card
• Shop with Points
• Reload Your Balance
• Amazon Currency Converter
• Amazon and COVID-19
• Your Account
• Your Orders
• Shipping Rates & Policies
• Returns & Replacements
• Manage Your Content and Devices

• Conditions of Use
• Privacy Notice
• Consumer Health Data Privacy Disclosure
• Your Ads Privacy Choices

• 02074940118
• [email protected]

Dealing with Conflicts of Interest at Work

Anne morris.

• 23 August 2024

IN THIS SECTION

In this guide for employers, we consider the risks associated with conflicts of interest at work, and the steps you can take to manage conflicts of interest and to mitigate legal and commercial risk.

What are conflicts of interest at work?

A conflict of interest arises where an employee’s personal interests are at odds with those of the business or organisation they work for. At times, these interests may even undermine or disadvantage the business or organisation.

These conflicts are inevitable and can occur at any level of the business – whether the individual is a permanent employee (full-time and part-time), casual employee, director or board member, or independent contractor.

Examples of conflicts of interest at work

Employee actions can cause a conflict of interest either intentionally or unintentionally. Instances of conflicts of interest include:

• Working for a competitor.
• Opening a business which is in competition with your own.
• Sharing confidential and/or damaging information about your company on social media.
• Failing to disclose hiring or working with a relative.
• Providing consulting advice and pay from a competitor.
• Not investigating a coworker’s misbehavior due to a personal relationship.
• Using knowledge or information about the company for their own financial benefit.

Why you need to prevent conflicts of interest

A conflict of interest can have a negative impact on the workplace in a number of ways.

It can create an unfair competitive advantage for those involved in the conflict. It can also lead to decreased productivity as employees focus on their own interests rather than the work at hand. Additionally, a conflict of interest can erode trust between employees and employers.

If things get really bad, a conflict of interest can result in legal action being taken against an individual or organisation. Basically, a severe conflict of interest can create a hostile work environment and cause morale, along with a company’s bottom line, to suffer.

Types of conflicts of interest

There are four main types of conflicts of interests you should be aware of when running your business:

Actual conflict of interest

This occurs where there is a real clash between an employee’s personal interests and the best interests of your business. For example:

• One of your employees starts their own business which offers similar products as your company.
• Your business is expanding and you have advertised for a job opening publicly. However, you employ your cousin without undertaking a competitive and formal selection process.

Potential conflict of interest

Potential conflicts happen in cases where there is no actual conflict right now, but it is foreseeable that a conflict of interest may arise in the future. For example:

• Your friend tells you that their marketing agency is considering applying for a contract to provide services to your business.
• One of your employees works as a freelance designer. One of your business’ clients asks that employee to provide paid services outside of business hours.

Where there is a potential conflict, it is important to take steps to mitigate the future risk.

Perceived conflict of interest

Perceived conflicts of interest arise where there is no actual or potential conflict, however it would be reasonable for someone else to think that a conflict does indeed exist. As such, identifying and dealing with these types of conflict can sometimes be slightly more difficult. For example:

• Your partner won an Instagram giveaway run by a business that has a contract to supply goods to your company.
• A friend of your relative applies and is successful in landing a job at your company, however you played no role in the hiring decision.

Conflict of duty

A conflict of duty generally arises in situations where a person has to fulfil two or more roles that are incompatible or clash with each other. For example:

• An employee begins casual employment at another business that operates in the same market as your company and offers the same or similar goods and services as you.

How to deal with conflicts of interest at work

If you suspect a conflict of interest, it can damage your company in a number of ways. From financial implications, leaked data or damage to overall morale, leaving these situations unmanaged can have long-term impacts on your business.

There are a number of steps that employers can take to prevent conflicts of interest in the workplace.

First, it is important to have clear policies and procedures in place regarding conflicts of interest. These policies should be communicated to all employees and enforced consistently. Additionally, employers should provide training on what constitutes a conflict of interest and how to avoid them.

A conflict of interest is not always easy to recognise. If your company policies and procedures are not clearly laid out, they may not understand what is and is not appropriate information to share. They may also not grasp the ways in which their personal relationships at work (with relatives, for example) create a conflict of interest that may be problematic for the structure and morale of their workplace.

Second, employers should encourage employees to disclose any potential conflicts of interest. This will help to ensure that conflicts are dealt with before they become a problem.

Finally, employers should monitor the situation closely and take disciplinary action if necessary. This may include anything from verbal and written warnings to termination of employment.

Learning about a conflict of interest in the workplace can be stressful for everyone involved, whether you are an employer or an employee. So do your best to have a protocol set for what to do when you find a circumstance of conflict.

As a manager, how to handle an employee with a conflict of interest

If you are a manager and you have an employee with a conflict of interest, there are a few steps you can take to handle the situation.

First, talk to the employee about the conflict of interest and explain why it is a problem. It is important to be clear and concise in your explanation.

Next, give the employee an opportunity to correct the situation. If they are unable to do so, then you may need to take disciplinary action, up to and including termination of employment.

It is important to document everything throughout the process. This will help to protect you and your organization if the situation escalates.

As an employee, what do if you see another employee committing a conflict of interest

If you are an employee and you see another employee committing a conflict of interest, there are a few steps you can take.

First, try to talk to the employee about the conflict of interest and explain why it is a problem. It is important to be clear and concise in your explanation.

If the employee does not listen or refuses to correct the situation, then you can report the issue to your manager or HR department. They will investigate the situation and take appropriate disciplinary action, up to and including termination of employment. It is important to document everything throughout the process. This will help to protect you and your organisation if the situation escalates.

But gathering evidence to back up your suspicions can be a difficult process. Can you be sure that there is a conflict of interest at play? What do you need to support what you suspect? And, perhaps most importantly, how can you do this without alienating your employees?

Is a conflict of interest a legal issue?

Depending on the circumstance, a conflict of interest can turn into legal action. More times than not, a conflict of interest is not a legal issue, but is more of an ethical or moral dilemma. If the potential for a conflict of interest exists, it’s best to err on the side of caution and avoid the conflict altogether. However, if you are ever unsure whether a situation constitutes a conflict of interest, it is best to seek legal advice.

The role of the conflict of interest policy

We know employment relationships often rely on mutual trust and it is important to ensure your employee’s commitment to the business. The best way you can manage and get on top of any conflicts of interests is to have a written Conflict of Interest Policy that underpins how you handle and resolve these situations.

Such a policy can help set out what an employee can and can’t do, especially when it comes to working with another business in the same industry.

Having a written Conflict of Interest Policy means that you can outline your business’ expectations with regards to conflicts of interests, rather than waiting until a conflict arises and subsequently responding to it.

A good Conflict of Interest Policy should set out clear guidelines as to a worker’s obligations and responsibilities when it comes to having – and disclosing – any conflict of interests. It should also provide clarity around what it means to have a conflict of interest, communicate how employees can disclose conflicts, and the procedures for dealing with them.

From a management perspective, having a written Conflict of Interest Policy makes it easy and convenient to provide this information to those involved within your business, and helps you to avoid needing to deal with complex disputes relating to competing duties and interests.

What to include in a conflict of interest policy

As a starting point, we recommend thinking about your Conflict of Interest Policy as something that is tailored specifically to fit the needs of your business – unfortunately, there isn’t a one-size-fits-all template that you can insert seamlessly into your business operations. Below are some key sections you may want to include in your Conflict of Interest Policy.

Your Conflict of Interest Policy should state its purpose. This may include:

• Obligations of workers (such as employees and contractors) to notify the company of any conflicts of interest
• Clarification around your company’s approach to dealing with and resolving any conflicts of interest

Definitions

One of the best ways to ensure that your workers engage with your Conflict of Interest Policy is to make it easy-to-understand. Therefore, you may include a list of key terms and definitions used in the Policy.

Your Conflict of Interest Policy should clearly set out what your business considers a conflict of interest. If it is unclear what the policy means for them, it’s less likely that they will be able to comply with and follow your company’s procedures properly.

You should also clarify the scope of who your Conflict of Interest Policy applies to. Does it cover former, current and prospective employees? What about independent contractors and agents acting on your company’s behalf? Some sections of your business’ Conflict of Interest Policy may not apply to every person working for you, so it is crucial that the Policy explains who it applies to and what it means for them.

TIP: It may be helpful to use specific examples and situations that are likely to arise so that your workers are able to visualise the types of conflicts that may be prohibited and apply them to their own personal circumstances.

Processes and Procedures

It’s important to inform your workers how your business manages and deals with conflicts of interests that have been disclosed. This includes details around how your business’ decision-making process, and the factors influencing these outcomes.

We recommend that you Conflict of Interest Policy covers the following topics:

• Procedures for declaring and disclosing a conflict of interest. This includes steps an individual should take if they believe a conflict exists, whether the conflict is their own or someone else’s. If your business is introducing a new Conflict of Interest Policy, you may also want to think about providing a way for current employees, contractors, and other relevant parties, to disclose any potential conflicts of interest they may have. Disclosures made during this time should generally not result in any disciplinary action – applying it retrospectively may damage your relationship with your workers, and lead to potential legal consequences against you.
• Investigation and review. This includes how conflicts are investigated and by whom. It can also address the methods in which certain conflicts of interest may be resolved.
• Disciplinary action. Under some circumstances, if a person has knowingly acted on a conflict of interest, it may be necessary for you to take disciplinary action. Your Conflict of Interest Policy should set out these circumstances and consequences so your workers are fully informed. This is particularly important as it could determine the severity of the disciplinary action applied. For example, suspension and termination in serious cases.

Need Assistance?

DavidsonMorris’ team of employment law and HR specialists provide advice to employers on all aspects of workforce management. For expert guidance on dealing with, and helping to prevent, conflicts of interest in the workplace, contact us .

Conflicts of Interest at work FAQs

What is a conflict of interest in the workplace?

A conflict of interest occurs when an employee’s personal interests, relationships, or activities outside of work could potentially interfere with their professional responsibilities or the interests of the company. This can include financial interests, family connections, or outside employment.

How should an employee disclose a conflict of interest?

Employees should follow their company’s established procedures, typically involving filling out a conflict of interest disclosure form and submitting it to HR or their line manager. Transparency is key, and the disclosure should be made as soon as a potential conflict is identified.

Can conflicts of interest be resolved without disciplinary action?

In most cases, conflicts of interest can be resolved without the need for disciplinary measures. The resolution might involve reassigning tasks, altering reporting lines, or placing certain restrictions on the employee’s activities. The goal is to manage the conflict while maintaining fairness and trust.

What are the legal consequences of not managing conflicts of interest?

Failing to manage conflicts of interest can lead to legal issues, such as claims of unfair dismissal, discrimination, or breaches of employment contracts. It can also harm the company’s reputation and result in financial losses.

Is it necessary for every company to have a conflict of interest policy?

While not legally required, it is highly advisable for companies to have a clear conflict of interest policy. This ensures that both employers and employees understand their responsibilities and the procedures for handling potential conflicts.

What should be included in a conflict of interest policy?

A conflict of interest policy should include definitions of what constitutes a conflict, the process for disclosure, how conflicts will be managed, and the consequences of failing to disclose a conflict. It should also outline the roles and responsibilities of both employees and management in maintaining ethical standards.

Can an employee be dismissed for a conflict of interest?

Dismissal is possible, but it should be a last resort. Employers are encouraged to address conflicts of interest through other means first, such as reassignment or mediation. If the conflict results in serious misconduct or a breach of trust, dismissal may be warranted.

How can conflicts of interest be prevented?

Prevention strategies include regular training, fostering a culture of transparency, clear communication of expectations, and having robust policies in place. Regular reviews and updates to these policies can also help prevent potential conflicts.

Founder and Managing Director Anne Morris is a fully qualified solicitor and trusted adviser to large corporates through to SMEs, providing strategic immigration and global mobility advice to support employers with UK operations to meet their workforce needs through corporate immigration.

She is a recognised by Legal 500 and Chambers as a legal expert and delivers Board-level advice on business migration and compliance risk management as well as overseeing the firm’s development of new client propositions and delivery of cost and time efficient processing of applications.

Anne is an active public speaker, immigration commentator , and immigration policy contributor and regularly hosts training sessions for employers and HR professionals

• Anne Morris https://www.davidsonmorris.com/author/anne/ Contracts Hours and Pay
• Anne Morris https://www.davidsonmorris.com/author/anne/ A Guide to the Temporary Contract for Employers
• Anne Morris https://www.davidsonmorris.com/author/anne/ Understanding the Permanent Contract for Employers
• Anne Morris https://www.davidsonmorris.com/author/anne/ Holiday Sickness and Leave

About DavidsonMorris

As employer solutions lawyers, DavidsonMorris offers a complete and cost-effective capability to meet employers’ needs across UK immigration and employment law, HR and global mobility .

Led by Anne Morris, one of the UK’s preeminent immigration lawyers, and with rankings in The Legal 500 and Chambers & Partners , we’re a multi-disciplinary team helping organisations to meet their people objectives, while reducing legal risk and nurturing workforce relations.

Legal Disclaimer

The matters contained in this article are intended to be for general information purposes only. This article does not constitute legal advice, nor is it a complete or authoritative statement of the law, and should not be treated as such. Whilst every effort is made to ensure that the information is correct at the time of writing, no warranty, express or implied, is given as to its accuracy and no liability is accepted for any error or omission. Before acting on any of the information contained herein, expert legal advice should be sought.

Contact DavidsonMorris

Sign up to our award winning newsletters, find us on:.

Trending Services

DavidsonMorris Ltd t/a DavidsonMorris Solicitors is a company Registered in England & Wales No. 6183275

Regulated by the Solicitors Regulation Authority No. 542691

Registered Office: Level 30, The Leadenhall Building, 122 Leadenhall Street, London, EC3V 4AB

© Copyright 2024

Terms of Use | Privacy Policy | Cookies Notice

Website design by Prof Services Limited . 

A Structural Analysis of Eurasian Economic Union States’ Gross Domestic Product: A Conservation of a Middle-Income Trap

• Published: 27 August 2024

Cite this article

• Mayya Dubovik   ORCID: orcid.org/0000-0001-8843-1398 1 ,
• Sergey Dmitriev   ORCID: orcid.org/0000-0003-2621-5870 2 &
• Natalya Obidovskaya   ORCID: orcid.org/0000-0002-7426-9710 3  

The research employs a comprehensive analysis of existing literature, statistical data, and expert opinions to identify the primary challenges and opportunities for economic development within the EAEU. It develops a new conceptual framework for understanding the dynamics of economic integration and development in the EAEU. The study proposes a set of policy recommendations designed to promote innovation, enhance competitiveness, and foster sustainable economic development within the EAEU. The findings can be applied by policymakers, business leaders, and other stakeholders in the EAEU region, providing them with new insights into the challenges and opportunities of economic integration and development.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Data Availability

Data will be available on request.

Agenor, P. R. (2017). Caught in the Middle? The economics of middle-income traps. Journal of Economic Surveys , 31 , 771–791. https://doi.org/10.1111/joes.12175

Article   Google Scholar  

Aghion, P., & Bircan, C. (2017). The Middle-Income trap from a Schumpeterian Perspective. Asian Development Bank Economics Working Papers . https://doi.org/10.22617/WPS179029-2

Aghion, P., & Howitt, P. (1990). A model of growth through creative destruction. Econometrica , 60 (2), 323–351. https://doi.org/10.3386/w3223

Aghion, P., & Williamson, J. G. (2004). Growth, Inequality, and globalization. Theory, history, and policy . Cambridge University Press.

Google Scholar  

Aghion, P., Antonin, C., & Bunel, S. (2021). A power of Creative Destruction. Economic upheaval and the Wealth of nations . Belknap Press of Harvard University.

Book   Google Scholar  

Aiyar, S., Duval, R., Puy, D., Wu, Y., & Zhang, L. (2013). Growth Slowdowns and the Middle-Income Trap . IMF Working Papers. Vol. 071. IMF. https://doi.org/10.5089/9781484330647.001

Ashfaq, F., Abid, G., Ilyas, S., & Elahi, A. R. (2024). Sustainable leadership and work engagement: Exploring sequential mediation of organizational support and gratitude. Public Organization Review , in press . https://doi.org/10.1007/s11115-024-00778-w

Aslund, A. (2013). Why Growth in Emerging Economies Is Likely to Fall. Peterson Institute for International Economics. Working Paper Series WP 13– 10.

Atlas (2021). Country & Product Complexity Rankings. Retrieved July 22, 2024, from https://atlas.cid.harvard.edu/rankings

Ayadi, O. F., Chatterjee, A., & Ademiluyi, A. (2004). The O-Ring theory, geographical distribution of misery and corruption. International Business & Economics Research Journal , 3 (4), 61–66. https://doi.org/10.19030/iber.v3i4.3682

Clemens, W. C., & Kauffman, S. A. (2019). Is the world chaos, a machine, or evolving complexity? How well can we understand life and world affairs? NETSOL: New Trends in Social and Liberal Sciences , 4 (2), 24–43. https://doi.org/10.24819/netsol2019.06

Dubovik, M. V., & Degtyareva, O. A. (2015). «Middle-income trap» as a hedge of economic development and growth. Electronic Scientific Journal , 1 (1), 438–442.

Eurasian Economic Commission (2021). Statistics of external and mutual trade of goods. https://bit.ly/3lPiIu1x

Eurasian Bank of Development (2020). Directorate for analytical work. Main trends in the integration development of Russia in 2019 Center for Integration Studies. Retrieved July 22, 2024, from https://eabr.org/upload/iblock/ad8/EABR_RF_07_2020_RU.pdf

Fedyunina, A., Simachev, Y., Kuzyk, M., & Averyanova, Y. (2020). Structural features of Russian economy integration into global value chains and lessons for structural policy. New Economic Association Journal , 3 (47), 106–127. https://doi.org/10.31737/2221-2264-2020-47-3-5/

Fieller, E. C., Hartley, H. O., & Pearson, E. S. (1957). Tests for rank correlation coefficients. I Biometrika , 44 (3/4), 470–481. https://doi.org/10.1093/biomet/48.1-2.29

Georgiev, Y., Nagy-Mohacsi, P., & Plekhanov, A. (2017). Structural Reform and Productivity Growth in Emerging Europe and Central Asia. Asian Development Bank. Economics Working Papers. https://doi.org/10.22617/WPS179102-2

Glawe, L., & Wagner, H. (2016). The middle-income trap: Definitions, theories and countries concerned—A literature survey. Comparative Economic Studies , 58 (4), 507–538. https://doi.org/10.1057/s41294-016-0014-0

Haddad, M. A. (2020). Residential income segregation and commuting in a latin American city. Applied Geography , 117 , 102186.

Hausmann, R., Hidalgo, C., Bustos, S., Coscia, M., Simoes, A., & Yildirim, M. A. (2011). The Atlas of Economic Complexity: Mapping paths to Prosperity . Center for International Development, Harvard University, MIT.

He, Q., Liu, J., Xue, C., & Zhou, S. (2020). Bureaucratic integration and synchronization of regional economic growth: Evidence from China. China Economic Review , 63 , 101310. https://doi.org/10.1016/j.chieco.2019.101310

Hidalgo, C. A., & Hausmann, R. (2009). The building blocks of economic complexity. Proceedings of the National Academy of Sciences , 106 (26), 10570–10575. https://doi.org/10.1073/pnas.0900943106

Horn, P., & Grugel, J. (2018). The SDGs in middle-income countries: Setting or serving domestic development agendas? Evidence from Ecuador. World Development , 109 , 73–84. https://doi.org/10.1016/j.worlddev.2018.04.005

International Monetary Fund (2021). GDP per capita, current prices. Retrieved July 22, 2024, from https://www.imf.org/external/datamapper/NGDPDPC@WEO/OEMDC/ADVEC/WEOWORLD

Kremer, M. (1993). The O-ring theory of economic development. The Quarterly Journal of Economics , 108 (3), 551–575. https://doi.org/10.2307/2118400

Larson, G., Loayza, N., & Woolcock, M. (2016). The Middle-Income Trap: Myth or Reality? Research and Policy Briefs 104230. The World Bank. Retrieved July 22, 2024, from https://ideas.repec.org/p/wbk/wbkrpb/104230.html

Liang, Y., Zhou, K., Li, X., Zhou, Z., Sun, W., & Zeng, J. (2020). Effectiveness of high-speed railway on regional economic growth for less developed areas. Journal of Transport Geography , 82 , 102621. https://doi.org/10.1016/j.jtrangeo.2019.102621

Liebman, A. M. (2008). Theoretical and empirical researches in modern economics: Communications problems. Economic Issues , 6 , 4–19. https://doi.org/10.32609/0042-8736-2008-6-4-19

Liou, K. T. (2024). Applying cost reduction managerial strategies in government agencies: Lessons from the Chinese government reform. Public Organization Review , (in press). https://doi.org/10.1007/s11115-024-00767-z

Liu, H. (2019). The communication and European Regional economic growth: The interactive fixed effects approach. Economic Modelling , 83 , 299–311. https://doi.org/10.1016/j.econmod.2019.07.016

London School of Economics and Political Science (2021). Avoiding the Middle-income Trap. Retrieved July 22, 2024, from https://www.lse.ac.uk/iga/initiatives/middle-income-trap

Makowski, D., Ben-Shachar, M. S., Patil, I., & Lüdecke, D. (2020). Methods and algorithms for correlation analysis in R. Journal of Open Source Software , 5 (51), 2306. https://doi.org/10.21105/joss.023061

Ministry of Internal Affairs of the Russian Federation (2021). Migration situation statistical data. Retrieved July 22, 2024, from https://xn--b1aew.xn--p1ai/dejatelnost/statistics/migracionnaya

Nguyen, C. P., & Su, T. D. (2021). Economic integration and economic complexity: The role of basic resources in absorptive capability in 40 selected developing countries. Economic Analysis and Policy , 71 , 609–625. https://doi.org/10.1016/j.eap.2021.07.001

Nguyen, C. P., Schinckus, C., & Su, T. D. (2020). The drivers of economic complexity: International evidence from financial development and patents. International Economics , 164 , 140–150. https://doi.org/10.1016/j.inteco.2020.09.004

Nilsson, I., & Delmelle, E. C. (2020). On the link between rail transit and spatial income segregation. Applied Geography , 125 , 102364. https://doi.org/10.1016/j.apgeog.2020.102364

Oloyede, B. M., Osabuohien, E. S., & Ejemeyovwi, J. O. (2021). Trade openness and economic growth in Africa’s regional economic communities: Empirical evidence from ECOWAS and SADC. Heliyon , 7 (5), e06996. https://doi.org/10.1016/j.heliyon.2021.e06996

Our World in Data (2019). GDP, 1820 to 2018. GDP adjusted for price changes over time (inflation) and price differences between countries– it is measured in international-$ in 2011 prices. Retrieved July 22, 2024, from https://ourworldindata.org/search?q=gdp

Pan, H., Cong, C., Zhang, X., & Zhang, Y. (2020). How do high-speed rail projects affect the agglomeration in cities and regions? Transportation Research Part D: Transport and Environment , 88 , 102561. https://doi.org/10.1016/j.trd.2020.102561

Rekiso, Z. S. (2017). Rethinking regional economic integration in Africa as if industrialization mattered. Structural Change and Economic Dynamics , 43 , 87–98. https://doi.org/10.1016/j.strueco.2017.10.001

United Nations (2020). Falling into the middle-income trap? LSE Consulting; European Commission Directorate-General for Regional and Urban Policy . https://doi.org/10.2776/02363

Ushkalova, D. I., & Nikitina, S. A. (2019). The influence of external factors on Russia’s export and import. The Bulletin of the Institute of Economics of the Russian Academy of Sciences , 2 , 110–122.

Wang, C., Zhang, X., Ghadimi, P., Liu, Q., Lim, M. K., & Stanley, H. E. (2019). The impact of regional financial development on economic growth in Beijing–Tianjin–Hebei region: A spatial econometric analysis. Physica A: Statistical Mechanics and its Applications , 521 , 635–648. https://doi.org/10.1016/j.physa.2019.01.103

World Bank Open Data (2021). Official web site. Retrieved July 22, 2024, from https://data.worldbank.org

World Data Atlas (2021). Gross domestic product in current prices growth rate. Retrieved July 22, 2024, from https://bit.ly/2XEUvhz

World Bank. (2021). Russia integrates: Deepening the Country’s integration in the Global Economy . World Bank.

Yang, Z., Li, C., Jiao, J., Liu, W., & Zhang, F. (2020). On the joint impact of high-speed rail and megalopolis policy on regional economic growth in China. Transport Policy , 99 , 20–30. https://doi.org/10.1016/j.tranpol.2020.08.007

Ziliak, T., & McCloskey, D. (2008). The cult of statistical significance: How the standard error costs us Jobs, Justice and lives . University of Michigan Press.

Download references

Acknowledgements

We are indebted to valuable comments from the Editor and the anonymous reviewers.

This study received no funding.

Author information

Authors and affiliations.

Department of Economic Theory, Plekhanov Russian University of Economics, Moscow, Russian Federation

Mayya Dubovik

Department of Economics, Customs, IT, Plekhanov Russian University of Economics, Moscow, Russian Federation

Sergey Dmitriev

Department of Civil, Criminal Law and Procedure and Financial and Legal Disciplines, Plekhanov Russian University of Economics, Moscow, Russian Federation

Natalya Obidovskaya

You can also search for this author in PubMed   Google Scholar

Contributions

Mayya Dubovik: Conceptualization; Data curation; Methodology; Project administration; Funding acquisition. Sergey Dmitriev: Software; Supervision; Validation; Writing - review & editing. Natalya Obidovskaya: Formal analysis Investigation; Resources; Visualization; Writing - original draft.

Corresponding author

Correspondence to Mayya Dubovik .

Ethics declarations

Ethics approval.

Not applicable.

Informed Consent

Conflict of interest.

Authors declare that they have no conflict of interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

RStudio Code (data used from Table  1 )

library(easystats)

#Data for analysis

ARM <- c(31.321, 247.884, 196.98, 399.576, 503.774, 521.869, 608.415, 597.241, 620.638, 692.287, 779.122, 923.387, 1180.399, 1628.063, 2128.151, 3079.031, 3913.436, 2911.763, 3121.778, 3417.172, 3575.529, 3732.035, 3889.004, 3529.026, 3524.00, 3868.906, 4195.963, 4604.772, 4155.273)

BLR <- c(1252.792, 1114.104, 1533.721, 1034.931, 1429.059, 1396.112, 1514.669, 1210.965, 1079.61, 1283.273, 1524.49, 1873.667, 2450.9, 3221.463, 3968.753, 4887.275, 6581.175, 5345.325, 6023.149, 6472.739, 6938.067, 7977.147, 8316.012, 5941.24, 5022.472, 5757.286, 6322.317, 6798.349, 6398.87)

KAZ <- c(168.645, 304.849, 751.98, 1061.475, 1358.804, 1459.418, 1480.094, 1132.124, 1230.491, 1491.653, 1657.155, 2062.289, 2862.502, 3753.441, 5261.026, 6733.45, 8349.287, 7116.371, 9005.039, 11552.574, 12300.184, 13789.172, 12713.564, 10435.17, 7662.006, 9186.712, 9749.069, 750.427, 8732.644)

KGZ <- c(207.976, 148.155, 245.851, 329.934, 396.495, 379.056, 344.404, 258.467, 280.635, 309.913, 323.343, 382.888, 436.496, 478.77, 546.621, 725.487, 971.678, 876.933, 884.846, 1131.471, 1189.517, 1295.225, 1292.621, 1132.843, 1131.841, 1254.51, 1321.954, 1323.465, 1146.393)

RUS <- c(618.067, 1322.006, 1979.411, 2264.343, 2787.022, 2935.03, 1949.801, 1427.307, 1901.956, 2255.253, 2552.788, 3197.573, 4403.962, 5708.838, 7426.005, 9761.368, 12464.244, 9156.97, 11431.148, 14306.432, 15287.967, 15928.699, 14007.509, 9257.935, 8723.523, 10723.996,

11261.716, 11511.507, 10037.239)

EU <- c(18246.609, 16340.128, 17296.298, 19547.146, 19842.306, 18201.994, 18745.16, 18624.437, 17062.735, 17301.218, 18866.845, 23112.538, 26469.091, 27528.973, 29280.352, 33817.753, 37318.372, 33698.988, 33244.584, 35908.051, 33311.904, 34746.002, 35493.882, 30652.249,

31352.626, 33257.679, 35944.307, 35127.435, 34047.123)

#Make a data frame

CORR.DATA <- c(ARM, BLR, EU, KAZ, KGZ, RUS)

#Correlation EEU to EU - Pearson

cor(ARM, EU, method = “pearson”)

cor(BLR, EU, method = “pearson”)

cor(KAZ, EU, method = “pearson”)

cor(KGZ, EU, method = “pearson”)

cor(RUS, EU, method = “pearson”)

> #Correlation EEU to EU - Pearson

> cor(ARM, EU, method = “pearson”)

[1] 0.9605379

> cor(BLR, EU, method = “pearson”)

[1] 0.9561961

> cor(KAZ, EU, method = “pearson”)

[1] 0.9230231

> cor(KGZ, EU, method = “pearson”)

[1] 0.9177746

> cor(RUS, EU, method = “pearson”)

[1] 0.9454607

> #Correlation EEU to Russia - Pearson (Results)

> cor(BLR, RUS, method = “pearson”)

[1] 0.979467

> cor(ARM, RUS, method = “pearson”)

[1] 0.9345104

> cor(KAZ, RUS, method = “pearson”)

[1] 0.9825186

> cor(KGZ, RUS, method = “pearson”)

[1] 0.9347083

#Correlation EEU to EU - Spearman

cor(ARM, EU, method = “spearman”)

cor(BLR, EU, method = “spearman”)

cor(KAZ, EU, method = “spearman”)

cor(KGZ, EU, method = “spearman”)

cor(RUS, EU, method = “spearman”)

#Correlation EEU to EU - Spearman (Results)

> cor(ARM, EU, method = “spearman”)

[1] 0.8985222

> cor(BLR, EU, method = “spearman”)

[1] 0.9054187

> cor(KAZ, EU, method = “spearman”)

[1] 0.873399

> cor(KGZ, EU, method = “spearman”)

[1] 0.9004926

> cor(RUS, EU, method = “spearman”)

[1] 0.920197

#Correlation EEU to Russia - Spearman

cor(BLR, RUS, method = “spearman”)

cor(ARM, RUS, method = “spearman”)

cor(KAZ, RUS, method = “spearman”)

cor(KGZ, RUS, method = “spearman”)

> #Correlation EEU to Russia– Spearman (Results)

> cor(BLR, RUS, method = “spearman”)

[1] 0.953202

> cor(ARM, RUS, method = “spearman”)

[1] 0.8871921

> cor(KAZ, RUS, method = “spearman”)

> cor(KGZ, RUS, method = “spearman”)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Dubovik, M., Dmitriev, S. & Obidovskaya, N. A Structural Analysis of Eurasian Economic Union States’ Gross Domestic Product: A Conservation of a Middle-Income Trap. Public Organiz Rev (2024). https://doi.org/10.1007/s11115-024-00790-0

Download citation

Accepted : 16 August 2024

Published : 27 August 2024

DOI : https://doi.org/10.1007/s11115-024-00790-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Economic Complexity
• Economic Growth
• Gross Domestic Product
• Middle-income trap
• Regional Integration
• Theory of Percolation
• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Conflicts of interest and critiques of the use of systematic reviews in policymaking: an analysis of opinion articles

Susan r forsyth.

1 Department of Social and Behavioral Sciences, School of Nursing, University of California, San Francisco, CA, USA

Donna H Odierna

2 Department of Clinical Pharmacy, University of California, San Francisco, CA, USA

David Krauth

Lisa a bero.

3 Institute for Health Policy Studies, University of California, San Francisco, CA, USA

4 Charles Perkins Centre, University of Sydney, Sydney, Australia

Associated Data

Strong opinions for or against the use of systematic reviews to inform policymaking have been published in the medical literature. The purpose of this paper was to examine whether funding sources and author financial conflicts of interest were associated with whether an opinion article was supportive or critical of the use of systematic reviews for policymaking. We examined the nature of the arguments within each article, the types of disclosures present, and whether these articles are being cited in the academic literature.

We searched for articles that expressed opinions about the use of systematic reviews for policymaking. We included articles that presented opinions about the use of systematic reviews for policymaking and categorized each article as supportive or critical of such use. We extracted all arguments regarding the use of systematic reviews from each article and inductively coded each as internal or external validity argument, categorized disclosed funding sources, conflicts of interest, and article types, and systematically searched for undisclosed financial ties. We counted the number of times each article has been cited in the “Web of Science.” We report descriptive statistics.

Articles that were critical of the use of systematic reviews ( n = 25) for policymaking had disclosed or undisclosed industry ties 2.3 times more often than articles that were supportive of the use ( n = 34). We found that editorials, comments, letters, and perspectives lacked published disclosures nearly twice as often (60% v. 33%) as other types of articles. We also found that editorials, comments, letters, and perspectives were less frequently cited in the academic literature than other article types (median number of citations = 5 v. 19).

Conclusions

It is important to consider whether an article has industry ties when evaluating the strength of the argument for or against the use of systematic reviews for policymaking. We found that journal conflict of interest disclosures are often inadequate, particularly for editorials, comments, letters, and perspectives and that these articles are being cited as evidence in the academic literature. Our results further suggest the need for more consistent and complete disclosure for all article types.

Systematic reviews are often used to inform practice guidelines and public and private sector health policy decisions and are often used as an alternative to expert opinion or consensus conferences [ 1 - 4 ]. For example, systematic reviews have been used to guide policymaking around such issues as tobacco control and to set blood alcohol levels at which drivers are considered intoxicated [ 2 ]. States have used systematic reviews to evaluate whether policies for managing prison populations are working and cost effective [ 2 ]. Such use is not without controversy. While some authors argue that systematic reviews are the highest form of evidence and provide a solid basis for policymaking [ 1 , 2 ], others argue that these studies are methodologically flawed or limited in scope [ 5 ]. Editorials, letters, and other opinion pieces are important because they can inform debate about controversial topics. Furthermore, these articles are sometimes cited as if they are original research; a letter in the New England Journal of Medicine critical of a systematic review on secondhand smoke [ 5 ] has been cited 30 times as evidence of the flaws of systematic reviews, according to the Web of Science.

Systematic reviews synthesize the results of primary qualitative and quantitative research, using strategies to reduce bias [ 6 , 7 ]. Systematic reviews of qualitative research summarize and narratively synthesize results using a range of methods [ 8 - 10 ]. Meta-analyses are systematic reviews that summarize results quantitatively. This technique enables researchers to combine the results of several studies into a single effect estimate [ 11 ]. We use the term “systematic review” to refer to both systematic reviews and meta-analyses. Depending on the rigor of the protocol, the quality of the results of systematic reviews can be quite variable. High-quality reviews are characterized by a pre-defined protocol, clearly stated research questions, pre-defined eligibility criteria for studies, reproducible methodology, comprehensive search strategies to identify all studies and unpublished research that meet eligibility criteria, quality assessments including assessment of risk of bias, and systematic syntheses and presentation of findings [ 11 ]. Previous research has demonstrated that study results and conclusions may be influenced by funding sources and author conflicts of interest (COI) [ 12 ]. Primary studies or reviews funded by the pharmaceutical or tobacco industries are more likely to having findings or conclusions that favor the sponsor’s product than those funded by non-industry sources [ 12 - 14 ].

However, little research has been done on whether industry ties influence the conclusions of articles that contain opinions on the use of systematic reviews for health policy decisions. Therefore, we examined financial and other COI disclosed by authors of articles with opinions about the use of systematic reviews in policymaking. We investigated whether affiliations, funding sources, financial ties, and other COI of authors were associated with support or criticism of the use of systematic reviews. Further, we qualitatively summarized the types of arguments presented.

Despite the importance of knowing a study’s funding source and authors’ COI, it is often difficult to find complete information. Disclosure policies vary among journals and may be different for research articles and opinion pieces [ 15 - 17 ]. Therefore, we examined the extent of disclosure present in editorials, comments, letters, and perspectives as compared to other article types that expressed an opinion. In addition, authors’ COI statements and funding disclosures may not be entirely accurate [ 18 , 19 ]. To explore this, we searched for undisclosed industry ties, examining whether such ties were also associated with an article’s stance on the use of systematic reviews.

The primary objective of this study was to determine whether industry ties—disclosed or undisclosed—were associated with the conclusions of opinion articles on the use of systematic reviews for health policy decisions. Our secondary objectives were to determine the substance of the arguments that authors were using to support their positions and how disclosure policies vary by journal and by article type and to determine the extent that these articles are being cited in the scientific literature. This study was not intended to be a systematic review, as we did not undertake a comprehensive search with multiple databases, nor did we contact authors for unpublished papers. This is an analysis of a representative sample of available opinion articles.

Search strategy

Opinion pieces are often difficult to locate, are inconsistently indexed, and may have uninformative titles [ 20 , 21 ]. For our systematic investigation, we conducted a preliminary MEDLINE text search (“Systematic reviews for health policy decision making”) and consulted a reference librarian to further develop our search strategy. We then conducted four more searches of MEDLINE for all articles published between January 1, 1946, and July 31, 2010, with the following MeSH terms and topics: Meta-Analysis as Topic [Mesh] AND health policy [Mesh]; Review Literature as Topic [Mesh] AND health policy [MAJR]; Review literature as topic [Mesh] AND health policy [Mesh]; and “Meta-analysis” and “problem” as topics.

We included editorials, letters, commentaries, and other articles expressing opinions about the use of systematic reviews for policy. Research articles, including studies, reviews, and case examples, were included if the authors expressed an opinion on the use of systematic reviews in policymaking within the body of the article. For included papers, we examined the MEDLINE summary page for associated comments for possible inclusion into our sample. The purpose of this was to understand the debate surrounding a particular publication on the use of systematic reviews in policymaking. In addition, we also included relevant articles from our files.

Selection criteria

Two researchers (SRF and DHO) screened search result titles to identify articles that appeared in peer-reviewed journals that criticized or supported the use of systematic reviews for making health policy decisions. Titles were excluded if they did not include at least one of the terms: systematic reviews, meta-analysis, evidence-based, Cochrane, medical effectiveness research, drug effectiveness review program, or policy and evidence. Titles were additionally excluded if they appeared to be entirely related to statistical methodology or handling of data or only related to the use of systematic reviews solely for clinical decision-making.

All other titles were included. Each author (SRF and DHO) independently screened each search and compared results. The abstracts and full text of all titles selected by either author were reviewed for inclusion. We also included relevant articles from the author’s files that did not appear in the original searches.

Two authors (SRF and DHO) independently screened the abstracts and/or full text of each included title. Discrepancies were resolved by consensus. We included articles that contained significant arguments for or against the use of systematic reviews in policymaking. We considered arguments to be significant if they did not focus solely on a single systematic review, but rather commented on whether systemic reviews did or did not have a role in policymaking. We excluded articles that made no mention or only a brief mention of the use of systematic reviews for policymaking, systematic reviews of a particular topic, tools or instructional articles, articles that were entirely clinically focused, and articles/abstracts that were not available in English or online.

Coding of articles as supportive or critical of the use of systematic reviews to inform policy

We coded articles as supportive if they promoted well-conducted systematic reviews as a strong method for gathering, evaluating, and disseminating bodies of research. Articles that included suggestions for improvement but still advocated systematic reviews as an important knowledge generation tool for policymakers were also coded as supportive. We coded articles as critical if they argued that systematic reviews did not generate useful or accurate knowledge or argued that the method is currently too limited to be useful.

Coding of individual arguments in each article

We initially coded each article inductively, abstracting each argument the authors made to support their positions. After analysis, we then thematically grouped similar arguments together for analysis and coded them as “internal reliability” arguments or “external validity” arguments using adapted definitions from previous studies [ 14 , 22 ]. Although each article was coded overall as supportive or critical, an individual article could contain both critical and supportive arguments or arguments about internal or external validity.

Two authors (DHO and SRF) coded in duplicate 14% (37/264) of the arguments. We agreed 92% (34/37) of the time. We resolved disagreements by consensus. One author (SRF) coded the remainder of the articles. We also abstracted illustrative quotes from the articles, qualitatively summarized the arguments, and conducted a content analysis on the abstracted arguments [ 23 ].

Internal reliability arguments

If the authors argued that a well-conducted systematic review was an internally reliable and valid method of collecting and analyzing data, the argument was coded as supportive of internal reliability. If the authors argued that systematic reviews were internally flawed because of issues of selection bias, misclassification, confounding, study heterogeneity, flaws in data pooling and data analysis, and publication bias, the article was coded as critical of internal reliability.

External validity arguments

If the authors argued that well-conducted systematic reviews were externally valid, useful for highlighting gaps in the literature, or a strong summary source of evidence useful in policy contexts, the argument was coded as a supportive external validity argument. If the authors argued that systematic reviews were not useful for highlighting gaps in the literature, a weak summary source of evidence and/or not useful in policy contexts, and/or not generalizable, the argument was coded as a critical external validity argument.

Coding of disclosures

Affiliation of authors.

We coded each included article by the author affiliations stated in the article. If authors claimed more than one affiliation, each type was coded.

Conflicts of interest of authors

We coded each article for COI by examining disclosures using categories adapted from the International Committee of Medical Journal Editors [ 24 ]. We considered study sponsorship/funding separately from COI. If there was an explicit COI statement contained within the article, it was considered to have a disclosed COI. We then divided COI statements into three categories: no disclosed COI, industry-related COI, and other, which included all non-industry-related COI such as a school or a government. We coded an article as having an industry-related COI if at least one author reported explicit financial ties to a for-profit industry. If an article did not have an explicit conflict of interest statement, we coded it as “no information provided.”

Funding sources of articles

We coded all articles by disclosed funding sources. If there was an explicit statement about funding, or that the article had no funding, the paper was considered to include disclosure of its funding source. We coded funding statements into five categories: self-funded or no funding, private non-profit funding, industry funding, government funding, and mixed funding. If an article did not have an explicit funding statement, we coded it as “no information provided.”

Industry ties

For analysis, we collapsed affiliations, COI, and funding sources into one dichotomous variable, “industry tie or no industry tie”. Our primary outcome was presence of industry ties by supportive or critical argument.

Extent of disclosure

We abstracted the date of publication and extent of disclosure statement in each included article. Some journals have only recently required and/or printed COI statements and funding disclosure statements [ 17 , 25 , 26 ]. We identified current COI and funding disclosure policies for each of the journals in which our study articles appeared. Articles were coded into three categories: no disclosure of either COI or funding source, disclosure of COI or funding source, or disclosure of COI and funding source.

Identification of undisclosed COI

For authors of articles with no disclosed industry ties, we searched MEDLINE for any articles they had published within the 3 years before publication of the included study article. We assessed the disclosures in these articles using criteria set forth by the International Committee of Medical Journal Editors [ 24 ]. We classified the disclosures of the authors as: (a) publication of a study sponsored by industry; (b) affiliation with industry at the time of publication, including, but not limited to, employment, board membership, etc.; (c) disclosure of a financial relationship to industry, including patents, stock, research funding, grants, gifts, consultancy, royalties, expert testimony, service on industry speaker’s bureau, payments for manuscript preparation or review, travel, lectures, etc., or other relevant financial activities/relationships [ 24 ]. If an industry tie was identified in an article published within the 3 prior years of the study article, we categorized the study article as having an undisclosed industry tie. We then discontinued our search for the remaining publications related to the study article, including publications by co-authors.

We also searched the Integrity in Science (ISS) database that contains over 4,000 scientists with corporate ties [ 27 ]. Up until 2009, when the database stopped collecting data, the researchers at ISS routinely scrutinized more than 200 science-based federal advisory committees for undisclosed conflicts of interest and monitored the media and scientific literature for failure to disclose. While the database is not comprehensive, it does provide an additional resource to examine corporate ties of scientists. If an author of a study article was listed as having an industry tie within the last 3 years in the ISS database, we categorized the study article as having an undisclosed industry tie.

Citations of included articles in the academic literature

We searched the academic search engine “Web of Science” and recorded the number of citations for each article included in our study as of November 2013. We used this number as a proxy measure for the extent that the article was being used as evidence within the academic community.

After articles were selected for inclusion, each article was closely read by two authors (SRF and DHO) and a decision was reached about whether the article was supportive or critical of the use of systematic reviews in policymaking. Discrepancies were resolved by a third author (LAB). Using an author-generated data collection sheet, we coded full citation and article type, whether the article was supportive or critical of the use of systematic reviews to inform policy, individual argument types, affiliations of authors, stated conflicts of interest of all authors, funding source of articles, and extent of disclosure, as described below. Once all data from the article had been extracted, we then searched for undisclosed industry ties and individual journal disclosure policies. Finally, we identified the number of times each article had been cited in the academic literature.

All data were analyzed using descriptive statistics, as the use of inferential statistics was inappropriate due to the non-randomness of the sample.

We screened a total of 533 article titles including ten from the author’s files and seven from comments associated with the articles (see Figure  1 ). Twenty-two articles were excluded because they were not in English (see Additional file 1 ). Of the 59 included articles (see Additional file 2 ), nearly 34% (20/59) of the included articles were editorials, comments, letters, and perspectives, while 66% (39/59) of the articles were research, reviews, and case studies. All included articles were published between 1991 and July 2010. We coded 58% (34/59) articles as supportive of using systematic reviews to inform policy and 42% (25/59) as critical.

PRISMA 2009 flow diagram.

Disclosure of affiliation, conflict of interest, and funding source

Table  1 shows the extent of disclosure found in the study articles. Forty-two percent (25/59) of both supportive and critical articles contained no COI or funding source disclosures, whereas 21% (7/34) and 24% (6/25) of the articles, respectively, had both COI and funding disclosures. Disclosure status appeared to improve over time. The median publication date for articles without disclosures was 2003 for both supportive and critical articles, compared to 2008 for articles with COI and funding disclosures.

Extent of disclosures made within the articles

Table  2 shows the extent of disclosures found in the study articles by article type. Sixty percent (12/20) of the editorials, comments, letters, or perspectives did not contain either COI statements or funding disclosures, compared to 33% of the other article types considered.

Extent of disclosures in editorials, comments, letters, and perspectives compared to other article types

Table  3 shows disclosed affiliations, COI, and funding sources of the study articles. We found similar rates of disclosed university affiliations among supportive and critical authors. However, more supportive authors were affiliated with government and considerably more critical authors were affiliated with the pharmaceutical, tobacco, or insurance industries, the three industries represented in our sample. The majority of papers did not have explicit COI statements. However, 26% (9/34) of the supportive authors explicitly stated that they did not have COI, compared to 8% (2/25) of the critical authors. Conversely, 32% (8/25) of the critical authors had an explicitly stated industry-related COI as compared to 3% (1/34) of the supportive authors. Funding statements were almost equally scarce among supportive and critical articles, and no article listed more than one funding source. Of those articles that disclosed funding sources, more articles critical of systematic reviews were funded by industry.

Disclosed affiliations, conflicts of interest, and funding source by opinion on the use of systematic reviews for policymaking

a Totals add to more than 100 as some papers had more than one affiliation.

Industry ties—disclosed and undisclosed

Table  4 shows industry ties that were disclosed in the included article and industry ties that were found in the 3-year retrospective search for undisclosed ties. Critical articles were about six times more likely to have a disclosed industry tie than supportive articles; 6% (2/34) of supportive articles and 40% (10/25) of critical articles disclosed industry ties. While this gap narrowed when we included industry ties found in the retrospective 3-year search, critical articles had an industry tie more than twice as often as did supportive articles (35%, 12/34 v. 80%, 20/25).

Industry ties (disclosed and not disclosed) by opinion on the use of systematic reviews for policymaking

Journal disclosure policies

At the time we conducted our study, most journals had current disclosure policies. The 15 articles critical of systematic reviews that did not disclose industry ties were published in 12 distinct journals. Among these journals, 9 of the 12 currently require authors to disclose their conflicts of interest and 6 of the 12 require authors to disclose their funding sources. The 32 supportive study articles that did not disclose industry ties were published in 20 distinct journals. Among these journals, 17 of the 20 currently require authors to disclose their conflict of interest and 13 of the 20 require authors to disclose their funding sources (data not shown in tabular form).

Article citations

Table  5 shows the number of times that the included articles have been cited within the academic literature by article type and by whether the article was supportive or critical. Interestingly, 48% (12/25) of articles critical of systematic reviews were editorials, comments, letters, and perspectives, while only 24% (8/34) of supportive articles were editorials, comments, letters, and perspectives. Taken together, editorials, comments, letters, and perspectives were cited less often than other article types (median citations 5 v. 19); however, they still received a number of citations within the literature.

Citations of editorials, comments, letters, and perspectives in the academic literature as compared to other article types (from the Web of Science by November 2013)

Argument analysis

We abstracted 151 individual arguments from the 34 supportive articles. Of these 151, 37% (56/151) were arguments in support of the internal reliability of well-conducted systematic reviews. Authors typically argued that the methods of well-conducted systematic reviews reduce bias. Lavis et al. contended that this lack of bias, as well the transparency and comprehensiveness of systematic review methodology, provides policymakers with more complete and valid information than they are likely to encounter in individual studies [ 28 ].

We abstracted 113 individual arguments from the 25 critical articles. Of these 113, 62% (70/113) were arguments that were critical of the internal reliability of systematic reviews. Typically, authors argued that the research studies used in systematic reviews are too dissimilar to be combined into a summary statement or statistic, included studies are cherry-picked; and that studies are excluded because of investigator bias, not because of inherent problems with the study. These authors argued that systematic reviews magnify publication bias or that they are often dependent on small numbers of underpowered and methodologically inadequate trials. Critical authors also contend that the methods used are not transparent, that research questions asked by systematic reviews are too narrowly focused to be of any use, or that meta-analyses may use analyses of information from subgroups collected after randomization, resulting in the possibility that the confounding variables may no longer be distributed at random. In a recent critical article that reviewed four published systematic reviews, the authors note that weaknesses encountered in many meta-analyses often stem not from the method itself, but from the poor design and reporting of the trials that make up the body of evidence available to answer the particular research question. They argued that this is exacerbated when the authors of meta-analyses fail to exclude poor-quality studies or to account for the variability in study quality by performing sensitivity analyses [ 29 ]. A commentary in the New England Journal of Medicine questioning the conclusions of several meta-analyses that found that secondhand smoke is harmful to health presented many common criticisms of these studies, including authors’ bias and lack of subject matter expertise, subject matter experts’ misunderstanding of meta-analytic methods, and such methodological flaws as failure to include relevant co-variables or account for heterogeneity or varying effect sizes in different populations [ 5 ].

External validity arguments occurred almost twice as often in supportive arguments as did internal validity arguments (63% v. 37%). The authors of these articles generally argued that well-conducted systematic reviews were the highest level of evidence available [ 30 , 31 ] and effectively summarized large bodies of evidence, resulting in strong external validity [ 32 ]. Supportive authors also asserted that systematic reviews were useful in identifying gaps in the literature and highlighting research priorities.

Thirty-eight percent (43/113) articles critical of systematic reviews contained external validity arguments; however, they generally were extensions of arguments made that were critical of the internal reliability of systematic reviews, arguing that there can be no external validity because of the poor internal validity and reliability. For example, Eysenck [ 33 ] argued that the results of a meta-analysis on the toxicity of secondhand smoke by the National Research Council were scientifically meaningless and should not be used in policymaking because of detection bias resulting from unreliable cause of death information on death certificates, authors’ reliance on effect size, and exclusion of “evidence relevant to the paradigm of research.”

In five critical articles [ 34 - 38 ], and six supportive [ 30 , 39 - 43 ], authors expressed concerns about generalizability of systematic review results. The critical articles primarily argued that results were too general to be applied with confidence to individuals or typical patient populations and warned that the results might not capture complexities of disease and treatment or biological, environmental, and contextual variability. One [ 37 ] raised questions about the utility to physicians of summary statistics; another [ 36 ] described the process of pooling heterogeneous patient data as “the risks run in pooling data from different studies to determine care guidelines are enormous”. Two critical articles discussed social context [ 34 , 38 ]. Neumann et al. criticized the Drug Effectiveness Review Project, which produces systematic reviews of drug-drug comparisons that are used by some US states to develop Medicaid preferred drug lists for not adequately considering costs and a “full societal perspective” [ 34 ]. Most of the critical articles concluded that intractable flaws in systematic review methods greatly decreased their external validity and suggested that policymakers use caution and take other sources of information into account when making health policy decisions. While most of the critical articles did not raise issues of health equity, Ahmad et al. [ 38 ] expressed concerns about the use by public health policymakers in low-income countries of systematic reviews of studies that are conducted primarily in high-income countries. The authors were concerned that systematic reviews on HIV prevention and tobacco cessation lacked cultural and socioeconomic context, rendering the findings potentially less useful for developing policies and practice guidelines in low-income countries. The authors identified steps that systematic review authors could take to increase the strength of review-based recommendations for developing countries.

Six otherwise supportive articles also expressed concerns about generalizability. One [ 30 ] advised policymakers to consider contextual factors and study populations when applying systematic review results. Five articles [ 39 - 43 ] argued that while systematic reviews provided the best available evidence for development of health policies and practice guidelines, they did not adequately address issues of health equity or account for differences in socio-economic status, racial/ethnic diversity, and other social health determinants. All of these articles made recommendations to authors and policymakers for tools and approaches, many of them interdisciplinary, that they can use to enhance the utility of systematic reviews for the development of equitable health policy. One article advocating for evidence-based health promotion recommends that authors increase applicability of their reviews by taking into consideration the salience of the topic and outcome measures and the practical and cost implications of the interventions that are being considered [ 43 ]. Other authors stress the importance of providing credible evidence that is convincing within the health sector and also among non-health collaborators [ 40 ].

Articles without industry ties were more likely to accept systematic review methods as internally reliable and valid, focusing on the relevance of systematic reviews for policymaking, making arguments to support why and how policymakers should consider the results of systematic reviews in their policy decision-making. Articles that were critical of systematic reviews, but did not have industry ties, were more likely to make highly nuanced arguments expressing concern about the generalizability of systematic review results to diverse populations.

Our findings suggest that articles critical of the use of systematic reviews for policymaking are more likely to have industry ties than supportive articles. One possible explanation is that authors of well-conducted systematic reviews, such as those done by The Cochrane Collaboration, examine all of the data available about a topic, including unpublished data when possible, and select for inclusion studies that meet rigorous criteria [ 11 ]. Systematic reviews that are conducted according to strict pre-defined protocols leave industry with less ability to direct or criticize the review findings. By considering all of the available data, the summary effect estimate from a well-done systematic review should represent the current state of the science on the topic in question. Studies that are funded by industry are considered as part of the evidence, but usually not as the only evidence. This may leave industry with less control over the discourse, introducing variability and the possibility that their product, process, or service may not be the most efficacious. For example, unpublished data are more likely to show that a study drug is ineffective than data that are published [ 44 - 47 ]. Systematic reviews attempt to capture and include this information, thus leading to results that may ultimately conflict with the results of industry-sponsored randomized controlled trials.

Our findings also suggest that current journal disclosure requirements for articles are frequently inadequate, especially for editorials, comments, letters, and perspectives. These types of articles lacked any type of disclosure three times as often as the other article types. Editorials, comments, letters, and perspectives play an active role in the academic discourse, as evidenced by the number of times they have been cited in the literature. Together, the included articles have been cited 2,079 times in the literature with editorials, comments, letters, and perspectives cited 366 times. However, it should be noted that citations can either be favorable or unfavorable and that the total number of citations does not necessarily equate with the influence of the article.

Overall, 42% (25/59) of all the included articles, including some that were recently published, did not have any disclosures, leaving readers with no information on the author’s affiliations, funding sources, and COI. Others only had partial disclosures. Our detailed 3-year retrospective search found that 43% (20/47) of articles that had no disclosed industry ties had at least one author with an undisclosed industry tie from the previous 3 years. Having this information readily available is important when readers evaluate the strength of an argument and possible bias. By not requiring the information, or not printing it, journals may leave readers inadequately prepared to accurately judge an article’s accuracy or usefulness.

Our results are consistent with previous studies that showed a positive relationship between industry ties and favorable conclusions in research articles, even when the results of the study do not support the conclusions drawn [ 13 , 14 , 45 , 48 ]. Our research suggests that opinion pieces may be subject to similar biases as other article types and provides evidence that funding sources and COI for all article types should be made transparent.

Our research has several limitations. Because of the inherent difficulties of locating opinion articles, we acknowledge that it is likely that our list of included papers was not comprehensive. Rather, our aim was to perform a systematic search of the topic for a representative sample of articles. As our sample was not a random sample, we elected not to do inferential statistics with the results so that there would be no suggestion of false precision. Because of variable disclosure requirements of journals, we likely have under-estimated the number of industry ties in our sample. For example, one critical article, written during the debates over secondhand smoke [ 33 ], did not list any disclosures, nor did we find any industry ties during our 3-year retrospective search. This paper was catalogued in our study as not having any industry ties. However, a British newspaper [ 49 ] wrote that this particular researcher had received about 800,000 pounds from the tobacco industry. Finally, researchers who do not directly receive payments from industry may work at institutions in which industry funds infrastructure. We did not capture these relationships.

Our findings are important because they demonstrate that industry ties may play a role in opinions expressed in the scientific literature. Moreover, opinion articles are often subsequently cited in the debate as evidence of the controversy over the use of systematic reviews in policymaking. Our results suggest the need for more consistent and complete disclosure for all article types, including articles that may not go through the traditional peer review process, such as editorials, commentaries, letters, and perspectives.

It is important to consider the industry ties of the authors when evaluating arguments regarding the use of systematic reviews by health policymakers and other decision makers. The conduct of systematic reviews is far from a perfect science, and there are substantial and nuanced criticisms of the generalizability of the results. While these arguments may be valid and should not be ignored, it is essential that we consider how knowledge generated by scientists can be critically summarized and subsequently translated for use in policymaking and population health decision-making. Well-conducted systematic reviews provide a rigorous and transparent method for knowledge dissemination and should be improved, not discarded.

Competing interests

Susan Forsyth, Donna Odierna, and Lisa Bero are affiliated with The Cochrane Collaboration. The authors have no other competing interests to declare.

Authors’ contributions

SRF assisted with the project development, collected and analyzed the data, and prepared drafts of the manuscript. DHO collaborated on the project development, developed the initial coding scheme, collected the data, wrote sections of the manuscript, and reviewed the drafts. DK collected the data and reviewed the drafts. LAB collaborated on the project development, contributed to the analysis of data, and reviewed the drafts. All authors read and approved the final manuscript.

Supplementary Material

Study articles excluded for being in a language other than English. Additional file 1 lists all studies from the original sample that were excluded because they were not in English.

Included study articles. Additional file 2 lists all studies included in the sample, including studies obtained from database searching, related commentary, and author’s files.

Acknowledgements

We would like to thank Mark Gibson for the assistance with development of the study protocol. We are also grateful to the research staff at the UCSF Library, the UCSF Tobacco Center, and the Drug Industry Document Archive for the assistance with search strategies.

This project was funded by the Flight Attendant Medical Research Institute (FAMRI), Miami, FL, USA. Susan Forsyth is funded by a dissertation award from the Tobacco-Related Disease Research Program (TRDRP; grant #22DT-0003).

• Bero LA, Jadad AR. How consumers and policymakers can use systematic reviews for decision making. Ann Intern Med. 1997; 127 :37–42. doi: 10.7326/0003-4819-127-1-199707010-00007. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Sweet M, Moynihan R, Fund MM. Improving Population Health: The Uses of Systematic Reviews. New York: Milbank Memorial Fund; 2007. [ Google Scholar ]
• Fox DM. The governance of standard-setting to improve health. Prev Chronic Dis. 2010; 7 :A123. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Fox DM. Systematic reviews and health policy: the influence of a project on perinatal care since 1988. Milbank Q. 2011; 89 :425–449. doi: 10.1111/j.1468-0009.2011.00635.x. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Bailar JC III. Passive smoking, coronary heart disease, and meta-analysis. N Engl J Med. 1999; 340 :958–959. doi: 10.1056/NEJM199903253401211. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Chalmers I, Hedges LV, Cooper H. A brief history of research synthesis. Eval Health Prof. 2002; 25 :12–37. doi: 10.1177/0163278702025001003. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Meerpohl JJ, Herrle F, Antes G, von Elm E. Scientific value of systematic reviews: survey of editors of core clinical journals. PLoS One. 2012; 7 :e35732. doi: 10.1371/journal.pone.0035732. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Greenhalgh T, Potts HW, Wong G, Bark P, Swinglehurst D. Tensions and paradoxes in electronic patient record research: a systematic literature review using the meta-narrative method. Milbank Q. 2009; 87 :729–788. doi: 10.1111/j.1468-0009.2009.00578.x. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008; 8 :45. doi: 10.1186/1471-2288-8-45. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009; 9 :59. doi: 10.1186/1471-2288-9-59. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Higgins JPT, Green S, editor. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1. 0 [updated March 2011]. The Cochrane Collaboration. 2011. [ http://www.cochrane-handbook.org ] [ Google Scholar ]
• Lundh A, Sismondo S, Lexchin J, Busuioc Octavian A, Bero L. Industry sponsorship and research outcome. Cochrane Database Syst Rev. 2012. Issue 12. Art. No.: MR000033. doi:10.1002/14651858.MR000033.pub2. 2012. [ PubMed ]
• Bartels RH, Delye H, Boogaarts J. Financial disclosures of authors involved in spine research: an underestimated source of bias. Eur Spine J. 2012; 21 :1229–1233. doi: 10.1007/s00586-011-2086-x. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Yank V, Rennie D, Bero LA. Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study. BMJ. 2007; 335 :1202–1205. doi: 10.1136/bmj.39376.447211.BE. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Forbes TL. Author disclosure of conflict of interest in vascular surgery journals. J Vasc Surg. 2011; 54 :55S–58S. doi: 10.1016/j.jvs.2011.06.019. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kesselheim AS, Lee JL, Avorn J, Servi A, Shrank WH, Choudhry NK. Conflict of interest in oncology publications. Cancer. 2012; 118 :188–195. doi: 10.1002/cncr.26237. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Blum JA, Freeman K, Dart RC, Cooper RJ. Requirements and definitions in conflict of interest policies of medical journals. JAMA. 2009; 302 :2230–2234. doi: 10.1001/jama.2009.1669. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Okike K, Kocher MS, Wei EX, Mehlman CT, Bhandari M. Accuracy of conflict-of-interest disclosures reported by physicians. N Engl J Med. 2009; 361 :1466–1474. doi: 10.1056/NEJMsa0807160. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Bero LA, Glantz S, Hong M-K. The limits of competing interest disclosures. Tob Control. 2005; 14 :118–126. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A. Applying systematic review methods to studies of people’s views: an example from public health research. J Epidemiol Community Health. 2004; 58 :794–800. doi: 10.1136/jech.2003.014829. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Woodman J, Harden A, Thomas J, Brunton J, Kavanagh J, Stansfield C. Searching for systematic reviews of the effects of social and environmental interventions: a case study of children and obesity. J Med Libr Assoc. 2010; 98 :140. doi: 10.3163/1536-5050.98.2.006. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Schotland MS, Bero LA. Evaluating public commentary and scientific evidence submitted in the development of a risk assessment. Risk Anal. 2002; 22 :131–140. doi: 10.1111/0272-4332.t01-1-00011. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Weber RP. Basic Content Analysis. Newbury Park: Sage; 1990. [ Google Scholar ]
• International Committee of Medical Journal Editors. International Committee of Medical Journal Editors (ICMJE): uniform requirements for manuscripts submitted to biomedical journals: writing and editing for biomedical publication. Haematologica. 2004; 89 :264. [ PubMed ] [ Google Scholar ]
• Ancker JS, Flanagin A. A comparison of conflict of interest policies at peer-reviewed journals in different scientific disciplines. Sci Eng Ethics. 2007; 13 :147–157. doi: 10.1007/s11948-007-9011-z. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Meerpohl JJ, Wolff RF, Niemeyer CM, Antes G, von Elm E. Editorial policies of pediatric journals: survey of instructions for authors. Arch Pediatr Adolesc Med. 2010; 164 :268–272. doi: 10.1001/archpediatrics.2009.287. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Integrity in Science. [ http://www.cspinet.org/integrity/index.html ]
• Lavis JN, Posada FB, Haines A, Osei E. Use of research to inform public policymaking. Lancet. 2004; 364 :1615–1621. doi: 10.1016/S0140-6736(04)17317-0. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Coyne JC, Thombs BD, Hagedoorn M. Ain’t necessarily so: review and critique of recent meta-analyses of behavioral medicine interventions in health psychology. Health Psychol. 2010; 29 :107. [ PubMed ] [ Google Scholar ]
• Chan KS, Morton SC, Shekelle PG. Systematic reviews for evidence-based management: how to find them and what to do with them. Am J Manag Care. 2004; 10 :806–812. [ PubMed ] [ Google Scholar ]
• Coffman JM, HONG MK, Aubry WM, Luft HS, Yelin E. Translating medical effectiveness research into policy: lessons from the California Health Benefits Review Program. Milbank Q. 2009; 87 :863–902. doi: 10.1111/j.1468-0009.2009.00582.x. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Brownson RC, Fielding JE, Maylahn CM. Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health. 2009; 30 :175–201. doi: 10.1146/annurev.publhealth.031308.100134. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Eysenck H. Meta-analysis or best-evidence synthesis? J Eval Clin Pract. 1995; 1 :29–36. doi: 10.1111/j.1365-2753.1995.tb00005.x. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Neumann PJ, Drummond MF, Jönsson B, Luce BR, Schwartz JS, Siebert U, Sullivan SD. Are key principles for improved health technology assessment supported and used by health technology assessment organizations? Int J Technol Assess Health Care. 2010; 26 :71–78. [ PubMed ] [ Google Scholar ]
• Laupacis A, Straus S. Systematic reviews: time to address clinical and policy relevance as well as methodological rigor. Ann Intern Med. 2007; 147 :273–274. doi: 10.7326/0003-4819-147-4-200708210-00180. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Marini JJ. Meta-analysis: convenient assumptions and inconvenient truth. Crit Care Med. 2008; 36 :328–329. doi: 10.1097/01.CCM.0000297959.02114.2B. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Goodman SN. Have you ever meta-analysis you didn’t like? Ann Intern Med. 1991; 114 :244–246. doi: 10.7326/0003-4819-114-3-244. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ahmad N, Boutron I, Dechartres A, Durieux P, Ravaud P. Applicability and generalisability of the results of systematic reviews to public health practice and policy: a systematic review. Trials. 2010; 11 :20. doi: 10.1186/1745-6215-11-20. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Tugwell P, Robinson V, Grimshaw J, Santesso N. Systematic reviews and knowledge translation. Bull World Health Organ. 2006; 84 :643–651. doi: 10.2471/BLT.05.026658. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Rychetnik L, Wise M. Advocating evidence-based health promotion: reflections and a way forward. Health Promot Int. 2004; 19 :247–257. doi: 10.1093/heapro/dah212. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Petticrew M, Tugwell P, Welch V, Ueffing E, Kristjansson E, Armstrong R, Doyle J, Waters E. Better evidence about wicked issues in tackling health inequities. J Public Health. 2009; 31 :453–456. doi: 10.1093/pubmed/fdp076. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Odierna DH, Bero LA. Systematic reviews reveal unrepresentative evidence for the development of drug formularies for poor and nonwhite populations. J Clin Epidemiol. 2009; 62 :1268–1278. doi: 10.1016/j.jclinepi.2009.01.009. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Gruen RL, Morris PS, McDonald EL, Bailie RS. Making systematic reviews more useful for policy-makers. Bull World Health Organ. 2005; 83 :480. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Rising K, Bacchetti P, Bero L. Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation. PLoS Med. 2008; 5 :e217. doi: 10.1371/journal.pmed.0050217. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ. 2012; 344 :d7202. doi: 10.1136/bmj.d7202. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Gøtzsche PC. Why we need easy access to all data from all clinical trials and how to accomplish it. Trials. 2011; 12 :249. doi: 10.1186/1745-6215-12-249. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Doshi P, Jefferson T, Del Mar C. The imperative to share clinical study reports: recommendations from the Tamiflu experience. PLoS Med. 2012; 9 :e1001201. doi: 10.1371/journal.pmed.1001201. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010; 303 :2058–2064. doi: 10.1001/jama.2010.651. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Pringle P. Eysenck took pounds 800,000 tobacco funds. The Independent. London: Independent Print Limited; 1996. [ Google Scholar ]