grey literature review article

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings
My Bibliography
Collections
Citation manager

Save citation to file

Email citation, add to collections.

Create a new collection
Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

Search in PubMed
Search in NLM Catalog
Add to Search

Gray literature: An important resource in systematic reviews

Affiliation.

1 Nuffield Department of Primary Care Health Sciences, Department for Continuing Education, The University of Oxford, Oxford, UK.
PMID: 28857505
DOI: 10.1111/jebm.12266

Systematic reviews aide the analysis and dissemination of evidence, using rigorous and transparent methods to generate empirically attained answers to focused research questions. Identifying all evidence relevant to the research questions is an essential component, and challenge, of systematic reviews. Gray literature, or evidence not published in commercial publications, can make important contributions to a systematic review. Gray literature can include academic papers, including theses and dissertations, research and committee reports, government reports, conference papers, and ongoing research, among others. It may provide data not found within commercially published literature, providing an important forum for disseminating studies with null or negative results that might not otherwise be disseminated. Gray literature may thusly reduce publication bias, increase reviews' comprehensiveness and timeliness, and foster a balanced picture of available evidence. Gray literature's diverse formats and audiences can present a significant challenge in a systematic search for evidence. However, the benefits of including gray literature may far outweigh the cost in time and resource needed to search for it, and it is important for it to be included in a systematic review or review of evidence. A carefully thought out gray literature search strategy may be an invaluable component of a systematic review. This narrative review provides guidance about the benefits of including gray literature in a systematic review, and sources for searching through gray literature. An illustrative example of a search for evidence within gray literature sources is presented to highlight the potential contributions of such a search to a systematic review. Benefits and challenges of gray literature search methods are discussed, and recommendations made.

Keywords: evidence; gray literature; publication bias; search; systematic review.

PubMed Disclaimer

LinkOut - more resources

Full text sources, other literature sources.

scite Smart Citations

Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Langson Library
Science Library
Grunigen Medical Library
Law Library
Connect From Off-Campus
Accessibility
Gateway Study Center

Email this link

Systematic reviews & evidence synthesis methods.

Schedule a Consultation / Meet our Team
What is Evidence Synthesis?
Types of Evidence Synthesis
Evidence Synthesis Across Disciplines
Finding and Appraising Existing Systematic Reviews
0. Preliminary Searching
1. Develop a Protocol
2. Draft your Research Question
3. Select Databases
4. Select Grey Literature Sources
5. Write a Search Strategy
6. Register a Protocol
7. Translate Search Strategies
8. Citation Management
9. Article Screening
10. Risk of Bias Assessment
11. Data Extraction
12. Synthesize, Map, or Describe the Results
Evidence Synthesis Resources & Tools

Grey Literature

Grey literature is literature produced by individuals or organizations outside of commercial and/or academic publishers. This can include information such as government reports, conference proceedings, graduate dissertations, unpublished clinical trials, and much more. The sources you select will be informed by your research question and field of study, but should likely include, at a minimum, theses and dissertations.

Why Search the Grey Literature?

The intent of an evidence synthesis is to synthesize all available evidence that is applicable to your research question. There is a strong bias in scientific publishing toward publishing studies that show some sort of significant effect. Meanwhile, many studies and trials that show no effect end up going unpublished. But knowing that an intervention had no effect is just as important as knowing that it did have an effect when it comes to making decisions for practice and policy-making. While not peer-reviewed, grey literature represents a valuable body of information that is critical to consider when synthesizing and evaluating all available evidence.

Searching the Grey Literature

Finding grey literature and searching it systematically is challenging. But there are a few approaches that you can take to add some structure to your search of this type of information:

Refer to grey literature sources used for related evidence syntheses. Refer to both published evidence syntheses and registered protocols.
Ask experts in the field for relevant grey literature sources. If you are an expert, include important grey literature sources, and ask colleagues for their recommendations.
Search databases that specialize in grey literature: See the "Grey Literature Sources" box at the bottom of this page for more information.
Search for theses and dissertations: There are a number of databases dedicated to theses and dissertations, which you can search using your search terms. See the "Grey Literature Sources" box at the bottom of this page for links to these resources.
Search clinical trials: There may be clinical trials being conducted that are relevant to your research question, but that haven't been published yet or never were published. See the "Grey Literature Sources" box at the bottom of this page for links to these resources.
Identify government agencies and international and non-governmental organizations that might publish technical papers and reports on your topic. Search their websites or any online libraries that they may provide. See the "Grey Literature Sources" box at the bottom of this page for links to some examples.
Search conference proceedings and newsletters: Identify professional organizations that have and/or conferences at which researchers might be presenting work related to your topic. Search those conference proceedings or newsletters on the organization's website or by contacting organizational boards for access to past proceedings that may not be online. See the "Grey Literature Sources" box at the bottom of this page for some examples.
Contact known researchers in the field to determine if there are any ongoing or unpublished studies that s/he may be aware of.
Search professional and trade magazines. Professional magazines contain literature that is written by professionals in the field for other professionals in the field, but that may not be about research. Trade magazines contain advertisements and news very specific to a topic or industry.

How to Manage the Grey Literature Search

Identify and record the sources you will search. The sources you search will be informed by your research question and where you expect to find information related to your question.
Document where you are searching and your search strategies, including document resource name, URL, search terms, and date searched.
Collect citation information as you go.
Adhere to your established inclusion and exclusion criteria when selecting sources.

See below for guidance documents specific to grey literature searching.

Grey Literature Sources

Grey Literature Databases
Theses and Dissertations
Clinical Trials
Public Policy
Conference Proceedings
Preprint Repositories
Other Resources
NY Academy of Medicine Grey Literature Report This report is a bimonthly publication of The New York Academy of Medicine (NYAM) alerting readers to new grey literature publications in health services research and selected public health topics. The database platform is keyword searchable and serves as an archive for the cataloged reports.
WHO Library Database The WHO library database includes governing documents, reports and technical documentation.
Global Think Tanks A database of reports, papers, and other resources published by NGOs, think tanks, IGOs and research centers from across the world. It also includes a directory of thousands of organizations involved in public and social policy in most subject area. A module in the Policy Commons database.
MedNar MedNar searches across more than 60 medical research sources, including commercial databases, medical societies, NIH resources, and other government resources.
Global Index Medicus The Global Index Medicus (GIM) provides worldwide access to biomedical and public health literature produced by and within low-middle income countries. The main objective is to increase the visibility and usability of this important set of resources. The material is collated and aggregated by WHO Regional Office Libraries on a central search platform allowing retrieval of bibliographical and full text information.
National Grey Literature Collection - UK Funded by Health Education England this holds the catalogue for the hard copy grey literature collection of the former North Grey Literature Service. It is collecting an increasing range of electronic Grey Literature.

Access available to all on campus. Off-campus access requires VPN (active UCInetID).

Networked Digital Library of Theses and Dissertations (NDLT) Free international resource for theses and dissertations.
Center for Research Libraries A resource for institutions outside of the U.S. and Canada.
EThOS EThOS is currently out of service due to a 2023 cyber attack. To find UK manuscripts, contact the author/university directly. Theses from Wales are available at Theses Collection Wales more... less... EThOS is the UK’s national thesis service which aims to maximize the visibility and availability of the UK’s doctoral research theses.
Open Access Theses and Dissertations Resource for finding open access graduate theses and dissertations published around the world. Metadata (information about the theses) comes from over 1100 colleges, universities, and research institutions. OATD currently indexes over 4 million theses and dissertations.
DART European - Digital Access to Research Theses Europe The aim of DART is to provide researchers with a single European Portal for the discovery of Electronic Theses and Dissertations (ETDs). Note that only theses available in electronic form are included.
Australia New Zealand Clinical Trials Registry Online register of clinical trials being undertaken in Australia, New Zealand and elsewhere. The ANZCTR includes trials from the full spectrum of therapeutic areas of pharmaceuticals, surgical procedures, preventive measures, lifestyle, devices, treatment and rehabilitation strategies, and complementary therapies.
ClinicalTrials.gov The US registry for clinical trials. Includes new, ongoing, and completed human clinical trials both in the US and countries around the world.
Cochrane Central Register of Controlled Trials A highly concentrated source of reports of randomized and quasi-randomized controlled trials. The majority of CENTRAL records are taken from bibliographic databases (mainly MEDLINE and Embase), but records are also derived from other published and unpublished sources.
EU Clinical Trials Register The European Union Clinical Trials Register allows you to search for protocol and results information on interventional clinical trials that are conducted in the European Union (EU) and the European Economic Area (EEA) and clinical trials conducted outside the EU / EEA that are linked to European paediatric-medicine development.
WHO International Clinical Trials Registry Platform ICTRP provides access to a central database containing the trial registration data sets provided by numerous international registries. It also provides links to the full original records.
Wikipedia Clinical Trial Registries List This Wikipedia entry contains a list of links to individual clinical trial registries by country.
WHO Institutional Repository for Information Sharing (IRIS) Institutional WHO database of intergovernmental policy documents and technical reports. Can search by IRIS by region (Africa, Americas, Eastern Mediterranean, Europe, South-East Asia, Western Pacific).
Health Research Web A wiki with descriptors of national health research systems, ethics review committees, local and regional policies and research priorities and more.

This list of conference proceedings is not comprehensive, but are ideas of where to search. We recommend that you search proceedings for the three most relevant conferences for your evidence synthesis.

OCLC PapersFirst OCLC PapersFirst is an OCLC index of papers presented at conferences worldwide.
BIOSIS Previews A subcollection of Web of Science, BIOSIS Previews searches across journals, meetings, patents, and books in the life sciences and biomedical fields.

An abstract and citation database of peer-reviewed literature, scientific journals, books and conference proceedings. Content covers research output in the fields of physical science, health science, life science, technology, medicine, and social sciences.

arXiv arXiv is a free distribution service and an open-access archive for scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.
medRxiv medRxiv (pronounced "med-archive") is a free online archive and distribution server for complete but unpublished manuscripts (preprints) in the medical, clinical, and related health sciences.
PsyArXiv A free preprint service for the psychological sciences. It is maintained by The Society for the Improvement of Psychological Science.
OSF Preprints A general preprint repository that covers architecture, business, engineering, life sciences, physical sciences and mathematics, arts and humanities, education, law, medicine and health sciences and social and behavioral sciences, among others.
TechRxiv Hosted by IEEE
Engineering Archive Created in 2016. Accepts all types of manuscripts, including but not limited to systematic reviews, hypotheses, ‘negative’ results, and data and methods papers.Includes 2 major types of articles: (1) Preprints of articles in parallel to traditional journal submissions. (2) Previously published articles that you wish to make Open Access
Preprints.org This is a multidiscipline platform providing preprint service that is dedicated to making early versions of research outputs permanently available and citable. We post original research articles and comprehensive reviews, and papers can be updated by authors as long as the updated content has not been published online.
Grey Matters: A practical search tool for evidence-based medicine This online manual provides a thorough list of sources for grey literature in medicine and a helpful checklist to help systematize your process.
Searching the grey literature: A handbook for searching reports, working papers, and other unpublished research Searching the Grey Literature is for librarians and information professionals interested in learning more about grey literature. This book will aid with crafting a grey lit search successfully, from start to finish. Many types of librarians will find the content of this book useful, particularly those in health or social science.
Duke University Medical Center Guide to Resource for Searching the Grey Literature Resources for trial registries, pharmacological studies, conference abstracts, government document and more.
Gray Literature Resources for Agriculture Evidence Syntheses This is a comprehensive list of agricultural grey literature sources, one of several evidence synthesis worksheets and tools provided to help in non-medical evidence synthesis development and implementation.
Searching for studies: A guide to information retrieval for Campbell Systematic Reviews This document gives comprehensive guidance in searching for evidence syntheses, and offers a robust list of grey literature sources for social science disciplines.
Finding Grey Literature Evidence and Assessing for Outcome and Analysis Reporting Biases When Comparing Medical Interventions: AHRQ and the Effective Health Care Program An introduction to clinical resources and strategies in approaching systematic reviews.

Video: Grey literature (3:10 minutes)

<< Previous: 3. Select Databases
Next: 5. Write a Search Strategy >>
Last Updated: Sep 4, 2024 10:30 AM
URL: https://guides.lib.uci.edu/evidence-synthesis

Off-campus? Please use the Software VPN and choose the group UCIFull to access licensed content. For more information, please Click here

Software VPN is not available for guests, so they may not have access to some content when connecting from off-campus.

Research article
Open access
Published: 19 April 2017

Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews

Lisa Hartling ORCID: orcid.org/0000-0001-8341-3991 1 , 2 ,
Robin Featherstone 2 ,
Megan Nuspl 2 ,
Kassi Shave 2 ,
Donna M. Dryden 2 &
Ben Vandermeer 2

BMC Medical Research Methodology volume 17 , Article number: 64 ( 2017 ) Cite this article

11k Accesses

194 Citations

61 Altmetric

Metrics details

Systematic reviews (SRs) are an important source of information about healthcare interventions. A key component of a well-conducted SR is a comprehensive literature search. There is limited evidence on the contribution of non-English reports, unpublished studies, and dissertations and their impact on results of meta-analyses.

Our sample included SRs from three Cochrane Review Groups: Acute Respiratory Infections (ARI), Infectious Diseases (ID), Developmental Psychosocial and Learning Problems (DPLP) ( n = 129). Outcomes included: 1) proportion of reviews that searched for and included each study type; 2) proportion of relevant studies represented by each study type; and 3) impact on results and conclusions of the primary meta-analysis for each study type.

Most SRs searched for non-English studies; however, these were included in only 12% of reviews and represented less than 5% of included studies. There was a change in results in only four reviews (total sample = 129); in two cases the change did not have an impact on the statistical or clinical significance of results. Most SRs searched for unpublished studies but the majority did not include these (only 6%) and they represented 2% of included studies. In most cases the impact of including unpublished studies was small; a substantial impact was observed in one case that relied solely on unpublished data. Few reviews in ARI (9%) and ID (3%) searched for dissertations compared to 65% in DPLP. Overall, dissertations were included in only nine SRs and represented less than 2% of included studies. In the majority of cases the change in results was negligible or small; in the case where a large change was noted, the estimate was more conservative without dissertations.

Conclusions

The majority of SRs searched for non-English and unpublished studies; however, these represented a small proportion of included studies and rarely impacted the results and conclusions of the review. Inclusion of these study types may have an impact in situations where there are few relevant studies, or where there are questionable vested interests in the published literature. We found substantial variation in whether SRs searched for dissertations; in most reviews that included dissertations, these had little impact on results.

Peer Review reports

Knowledge syntheses, including systematic reviews (SRs), provide essential evidence to inform healthcare decision making [ 1 ]. A key component of a well-conducted SR is an objective, sensitive, and reproducible literature search of multiple sources [ 2 ]. Methodological standards for SRs recommend extensive searching to address the potential for publication bias, to reflect the totality of evidence on a given question and produce accurate and valid estimates of effect [ 3 – 6 ]. Current Cochrane guidance recommends searching in grey literature sources beyond conventional bibliographic biomedical databases (e.g., Medline or Embase) for unpublished data, including trials registries, government regulatory documents and conference proceedings [ 4 , 7 ]. The Institute of Medicine and the Agency for Healthcare Research and Quality (AHRQ) also recommend (in addition to electronic databases) searching grey literature databases, clinical trials registries and other sources of unpublished information [ 5 , 6 ]. However, there is limited empiric evidence about the potential impact of selective searching and inclusion on the results of SRs.

Direct and indirect evidence confirm that studies which report large effect sizes or findings of beneficial interventions are published more frequently [ 3 ]. The impact of unpublished trials on the results of 60 meta-analyses on different medical interventions found that unpublished trials were less likely to produce statistically significant or beneficial results compared with published trials [ 8 ]. In most instances, the exclusion of these unpublished trials in meta-analyses had relatively small impact on estimates of treatment effectiveness and the changes in effect were inconsistent [ 8 ]. Other methods research has also observed the contribution of unpublished studies to reduce or change effect estimates of SRs [ 9 – 12 ] and to expose underestimates of harms in published studies [ 10 , 12 , 13 ].

Dissertations and theses are also recognized as sources of study data that, while published by academic institutions who award degrees, are not routinely published in commercial journals or indexed in conventional bibliographic databases [ 14 ]. As a sub-category of grey literature, dissertations may report studies of relevance to SR producers. However, we are unaware of investigations that have attempted to quantify the contributions of dissertations and theses to SR results.

Language bias is also a concern for systematic reviewers, [ 15 ] although empiric evidence of the existence and direction of a bias is inconsistent when non-English language publications are excluded [ 6 ]. Research suggests that German investigators are more likely to publish positive trial results in English-language publications [ 16 , 17 ]. But contrary evidence from a study of acupuncture trials found that researchers from some countries (e.g., China, Russia) were more likely to publish positive findings in non-English publications [ 18 ]. Research suggests that language bias in trials is variable by topic, and that SRs of complementary and alternative medicine (CAM) interventions are more likely to be significantly impacted by the exclusion of non-English studies [ 19 ]. An investigation of SRs on herbal medicines found that relevant Chinese studies would have been missed if reviewers included only Medline-indexed journal articles [ 20 ]. Non-English language trials were also found to be prevalent in fields of psychiatry, rheumatology and orthopaedics; but SRs of exclusively English trials in these areas were found to produce similar results to reviews with no language restrictions [ 8 ]. Regardless of the impact of language bias, Cochrane guidance supports identification and inclusion of trials in all languages [ 2 , 4 ] while the Institute of Medicine recommends searching in languages other than English if appropriate for the review topic [ 5 ].

Comprehensive literature searching is widely recommended; however, searching additional sources is associated with diminishing returns, [ 21 ] and the effect of continuing to search new sources has an unknown impact on the final results and conclusions. Further, inclusion of non-English studies can add substantially to the resources required to complete SRs. Given the existing environment of limited resources and the push for increased efficiencies, particularly in the context of decision-makers who require quicker production of reviews to meet their needs, [ 22 ] limits on the number of sources searched and types of studies included are necessary. The objective of this study was to examine the impact of searching for and including non-English studies, unpublished studies, and dissertations on the results of existing SRs.

This paper reports on additional analyses that were conducted as part of a broader methodological research initiative to generate empiric evidence about the impact of searching on the results of SRs. The methods have been described in detail previously [ 23 ].

We derived our sample from the Cochrane Child Health register of SRs which is organized in a REDCap database. The rationale for choosing reviews from the CDSR for this analysis is provided in our previous publication [ 23 ]. We exported all available SR records ( n = 1345) in the register to screen in Excel in December 2013. 51 records were excluded for having the status “Withdrawn” ( n = 45) or for having no status given ( n = 6); 294 records were excluded for not having performed at least one meta-analysis; 234 records were excluded for not having an effect size given for the first outcome; and 217 records were excluded for being incomplete. We organized the remaining 549 SR records by the Cochrane Review Groups (CRGs) responsible for their production. SRs were collected from three CRGs: Acute Respiratory Infections (ARI), Infectious Diseases (ID), and Development, Psychosocial and Learning Problems (DPLP). These three CRGs were selected by the research team as they contained the most SRs for analysis compared to other CRGs (ARI = 57; ID = 38; DPLP = 34), and represented three distinct clinical topic areas. All SRs from these three review groups were included in our final sample if they contained one or more meta-analysis (both continuous and/or dichotomous outcomes were eligible).

Data extraction

For each SR, all of the studies included in the primary meta-analysis were listed, forming our reference standard. The primary meta-analysis was either for the identified primary outcome, or the meta-analysis presented first in the review, if the primary outcome was not specified. For each study included in each meta-analysis, we checked the authors’ list of references to studies included in the review and categorized them as published or unpublished (i.e., authors indicated “published data only” vs. “published and unpublished data” or “unpublished data only”). We examined the citations and associated database records to identify any non-English studies. We made particular effort to identify any non-English studies when searching the databases examined for our companion methods paper [ 23 ]. As it was possible that titles had been translated into English for the reviews’ reference lists, we checked full-text whenever the title of a study database record appeared in square brackets or when the source title was not English. Further, we identified any included dissertations by examining the citations (dissertations typically included “PhD” in the citation and place of publication was a university or academic institution). Finally, we examined the section of the reviews describing the searching methods and documented if the review authors specifically indicated that they searched for unpublished studies (or grey literature), dissertations, or non-English studies.

Data analysis

For each review group and overall, we calculated: 1) the number of reviews that indicated they searched for each study type and the percentage relative to the total number of reviews; 2) the number of reviews that included each study type and the percentage relative to the total number that searched for the study type; and, 3) the number of each study type included and the percentage relative to the total number of studies included across the reviews.

For each meta-analysis that included at least one study representing one of our study types of interest, we re-analyzed the data excluding each study type. We used the same methods as the original meta-analysis (i.e., same summary measure and model). We calculated the percent of studies lost to each meta-analysis due to removal of each study type. We calculated the percent change in the point estimate and confidence interval width between the original and revised analysis. For point estimate, we ignored the direction of effect and considered only the magnitude. We categorized the change in point estimate as negligible (<5% change), small (5–10% change), moderate (11–20% change), large (more than 20%), and substantial (i.e., it was not possible to calculate an effect estimate as all the studies were categorized as a given study type, e.g., all were non-English). We also examined whether or not the result changed in statistical significance.

Data were managed using Excel Version 14.4.8 (Microsoft Corporation, Redmond, WA USA). Statistical analyses were conducted using RevMan 5.3 and SAS 9.4 (SAS institute Inc., Cary, NC USA).

Our analyses are based on 129 meta-analyses: 57 from ARI, 38 from ID, and 34 from DPLP. Citations and a description of the SRs used in our analysis are included in the Additional file 1 . The median year of publication was 2007 (IQR 1996–2013). The median number of studies and participants were 3 (IQR 1–35) and 636.5 (IQR 30–4,400,266).

Table 1 shows the number of SRs that searched for and included non-English studies, unpublished studies, and dissertations. Table 1 also shows the number of each type of study that was included and the percentage relative to the total number of studies.

The majority of reviews across review groups searched for non-English studies; however, only 12% of reviews included non-English studies and these represented only 4% of the total studies contained across all reviews ( n = 34/780). (See Additional file 1 for complete list of systematic reviews that included non-English language studies in their analyses). Among reviews that searched for non-English studies, there was a substantial difference across topic areas in terms of actually including non-English studies, with almost 20% of reviews in ARI compared to 0 in DPLP. Further, non-English studies represented 7% of all included studies in ARI reviews compared to 0 and 0.8% in DPLP and ID, respectively. The non-English studies included in ARI reviews were in the following languages: 32% French, 21% Spanish, 7% Chinese, 18% Italian, 14% German, 4% Turkish, and 4% Swedish. In ID reviews that included non-English studies, 50% of studies were Spanish, 33% were French, and 17% were Chinese.

Table 2 shows the results of the 15 meta-analyses where non-English studies were included, and the results when the non-English studies were removed. In nine cases the change in the effect estimate was negligible or small, and in two cases the change was moderate but no change in statistical significance. In two cases the change in effect estimate was large. There was a change in statistical significance in only one of these cases: the lower confidence interval changed from significant (0.01) to not significant (0.00); however, the conclusions of the review would not have changed as the authors concluded no clinically important difference between the two interventions. In two cases, all included studies were non-English; therefore, no effect estimate was available without the non-English studies. The topics of these reviews were acetylcysteine and carbocysteine for acute upper and lower respiratory tract infections in pediatric patients without chronic bronchopulmonary disease, and Chinese medicinal herbs for influenza. Change in confidence interval tended to be similar to change in point estimate with two exceptions. One study had a relatively small CI change despite a large point estimate difference, while another showed the opposite.

The majority of reviews searched for unpublished studies with some variation across groups (i.e., 91% for DPLP vs. 100% for ID). Only a very small percentage of reviews included unpublished studies ( n = 8/124; 5.6%); likewise, these reflected a very small percentage of the total studies ( n = 15/780; 1.9%). There was little variation across topics with very small numbers of unpublished studies included in each.

Table 3 shows the results of the eight meta-analyses that included unpublished studies. In four cases the change in effect estimates was negligible. In three cases the change was large, but none of the results changed in statistical significance. One study showed a small change in CI width, despite a large change in point estimate; otherwise changes in CI tended to be similar to changes in point estimate. One review included only unpublished studies; therefore, no effect estimate was available without the unpublished studies. The topic of the review was neuroaminidase inhibitors for preventing and treating influenza, a particularly high profile topic that relied heavily on industry reports and other regulatory documents [ 24 ].

There was wide variation across topics in searching for dissertations from 64.7% of reviews in DPLP compared to 2.6% in ID. Among the reviews that searched for dissertations, there was wide variation in the percentage that included dissertations from 20% in ARI to 100% in ID (although the latter was based on only one review that searched for dissertations). Overall, dissertations represented a very small percentage of included studies ( n = 15/780; 1.9%), with variability across topics: for ARI and ID, dissertations represented less than 1% of included studies, while for DPLP they represented 7.6%.

Table 4 shows the results of the nine meta-analyses that included dissertations, and the results when the dissertations were removed. In all but one case the change in effect estimates was negligible or small, and changes in CI width tended to be similar to changes in magnitude to changes in point estimate. In one case the change in effect estimate was large and the statistical significance changed: the result with the dissertations was statistically significant (SMD −0.24, 95% CI −0.35, −0.13; 10 studies) and without dissertations the upper confidence interval rested on the null (SMD −0.19, 95% CI −0.38, 0.00; 5 studies). In both cases (with and without dissertations), the effect estimate was small in magnitude but was smaller (more conservative) without the dissertations. The topic of this review was kinship care for the safety, permanency, and well-being of children removed from the home for maltreatment.

This study provides empiric evidence on the impact of searching for and including studies published in languages other than English, unpublished studies, and dissertations. The majority of SRs in our sample searched for non-English studies; however, these were included in a minority of reviews (12%) and represented less than 5% of all included studies. Moreover, there was a large or substantive change in results in only four reviews (among the total sample of 129). In two of these cases there were few included studies (2 and 3, respectively) and all were non-English. In two other cases the large change did not have an impact on the statistical or clinical significance of the findings. These results indicate that restricting the search and inclusion to English-only studies may not have an impact on the results of meta-analyses in the vast majority of cases. Searching for non-English studies should be considered on a case-by-case basis considering the topic area (due to the nature of the topic experts might expect evidence to be published or not in other languages, e.g., complementary and alternative medicine or diseases common in low-middle income countries) and volume of evidence (i.e., may be more necessary in areas where there is little evidence).

Likewise, the majority of SRs in our sample searched for unpublished studies but the vast majority did not include these (only 6% of reviews) and they represented only 2% of all included studies. In most cases the impact of including unpublished studies was small; only two of the eight meta-analyses including unpublished studies showed a large change in point estimates but in both cases there was no change in statistical significance of the result. In an additional case (of the eight meta-analyses), there were no published studies included; this review relied on unpublished industry reports and regulatory documents because of questions raised about the credibility of the published reports. The authors of this review had been questioned about the findings of an earlier version for which the conclusions were based on a pooled analysis, conducted by the manufacturer of the manufacturer-sponsored trials [ 24 ]. To address the concern, the authors set out to obtain the unpublished data from the drug manufacturer [ 24 ]. We might argue that this was a special case, and advise reviewers to seek unpublished data when the manufacturer has been heavily involved in a substantial proportion of published reports identified for inclusion. Further, we would advise that reviewers follow guidance on presenting information about vested interests (e.g., industry sponsored trials), and use this information when interpreting results and drawing conclusions [ 25 ].

Dissertations are a specific type of grey literature defined as a document supporting candidature for a doctorate degree that presents the candidate’s research and results [ 26 ]. One can assume that these documents undergo some extent of external review by content experts. We found variation across review groups with very few reviews in the ARI (9%) and ID (3%) groups searching for and including dissertations compared to 65% of DPLP reviews. Overall dissertations were included in only nine SRs (seven of these in DPLP, one each for ARI and ID) and represented less than 2% of all included studies. In the majority of cases the change in results was small or negligible. In one case, dissertations represented half of the included studies (five of ten) and there was a large change in the point estimate when dissertations were removed. Further, the result changed from statistically significant to not significant (lower confidence interval on the null). However, removing the dissertations resulted in a more conservative estimate, which may indicate that authors should carefully consider results when dissertations are included and the potential for overestimating treatment effects.

This study builds on work we recently published on the potential impact of prioritizing particular databases on the results of SRs, wherein we found that a limited number of databases provided the majority of relevant studies. Moreover, the results of meta-analyses based on studies contained in fewer databases did not differ, in the majority of cases, from the results of meta-analyses that included all identified studies [ 23 ]. However, we noted that the choice of database, and likewise decisions around searching for and including non-English studies, unpublished studies, and dissertations are likely topic dependent. Our results generally support reviewers to limit their searches in the interests of efficiencies without an important impact on results (in the vast majority of cases). Our results may be particularly relevant for rapid reviews, which are intended to produce evidence reports more quickly and efficiently than traditional SRs [ 27 – 29 ]. In rapid reviews, searching is one step that is typically modified to create efficiencies [ 29 , 30 ]. Changes include searching fewer databases, limiting the search for grey literature, and restricting by language of publication (e.g., English only) [ 23 ]. Recent research found that end-users of SRs identified restrictions to searching as an acceptable trade-off in the interests of creating efficiencies in the review process [ 22 ].

Our study had several limitations; many of these were cited in our previous related publication [ 23 ]. First, we derived our sample from the CDSR; further, the SRs came from three review groups and focused on healthcare interventions and randomized controlled trials. Results may not be generalizable to all clinical areas, for non-traditional interventions, or for SRs of alternative research questions (e.g., diagnostic, prognostic) or study designs (e.g., observational or qualitative studies). Second, our sample included SRs that were already complete and we used as our reference standard the original search strategies and the included studies from completed Cochrane reviews. We cannot confirm the sensitivity of the original search strategies to retrieve all potentially relevant studies; however, Cochrane reviews are recognized as having gold standard methods (including criterion related to searching) and searching for unpublished studies and dissertations and inclusion of non-English studies are mandatory expectations of Cochrane reviews. Third, we used completed analyses of the primary outcomes from SRs. Focusing on the primary outcomes provided us with the most data from which to test our hypotheses; however, results from the SRs may have varied across outcomes. Fourth, to determine if included studies were non-English or dissertations, we examined reference lists from reviews, the studies’ database records and, when necessary, full-text manuscripts. Despite our efforts to accurately represent the contribution of non-English studies and dissertations, we may not have accounted for all instances of these study types due to inaccurate reference lists in original reviews and incorrect metadata in database records. Fifth, there are other sources of unpublished studies, such as clinical trials registries and conference proceedings that we did not specifically examine. Future research on these specific sources of study data will contribute empiric evidence to guide this important aspect of knowledge synthesis. Further, additional studies conducted prospectively and in different clinical areas would be valuable; our study provides data based on a small proportion of published Cochrane reviews (129 meta-analyses from approximately 7000 total reviews in the CDSR). Sixth, we used a pragmatic approach to classify the extent of change as negligible, small, moderate, large or substantial. This may be too simplistic from a clinical point of view where other factors may be considered such as the nature of the outcome and the extent of heterogeneity. An alternative approach to classifying the change would be to ask clinical experts about the clinical significance of the change; however this was beyond the scope of the present work. Finally it should be noted that when looking at differences in results between meta-analyses, we analyzed reviews which identified non-English and unpublished studies and dissertations. It is possible these types of studies did exist even in the reviews that were unable to find them—they were simply too difficult for the searching to locate. We are unable to know if there is an inherent difference between removing studies when they are found (what we did) and adding them when they could not be found. Future research is needed to investigate the impact on SRs of including data from sources that are typically unavailable to reviewers.

This study provides quantitative data regarding the potential impact on meta-analysis results of excluding studies published in non-English languages, as well as unpublished studies and dissertations. We found that the vast majority of SRs searched for non-English and unpublished studies; however, these represented a very small proportion of included studies and rarely impacted the results and conclusions of the review. Inclusion of these study types may have an impact in situations where there are very few relevant studies, or where there are questionable vested interests identified in the published literature. We found substantial variation in whether SRs searched for dissertations; in the majority of reviews that included dissertations, these had little impact on the results and in fact may overestimate treatment effects. The findings from this study may be useful to optimize the conduct of SRs and guide the development of methods for rapid reviews. Future research in different clinical areas, and for other select types of grey literature, will help establish best practices for literature searching to support evidence syntheses.

Lavis JN, Davies HT, Gruen RL, Walshe K, Farquhar CM. Working within and beyond the Cochrane Collaboration to make systematic reviews more useful to healthcare managers and policy makers. Healthc Policy. 2006;1(2):21–33. [cited 2017 Jan 6] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/19305650/ .

PubMed PubMed Central Google Scholar

Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration. 2011. [cited 2016 Jun 7]. Available from: http://handbook.cochrane.org/ .

Google Scholar

Sterne JA, Egger M, Moher D. Chapter 10: Addressing reporting biases. In: Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration. 2011. [cited 2017 Jan 3]. Available from: http://handbook.cochrane.org/ .

Higgins JPT, Lasserson T, Chandler J, Tovey D, Churchill R. Methodological Expectations of Cochrane Intervention Reviews (MECIR): standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates. Cochrane; 2016 [cited 2017 Jan 3]. Available from: http://methods.cochrane.org/sites/default/files/public/uploads/mecir_printed_booklet_final.pdf .

Institute of Medicine. Finding what works in health care: standards for systematic reviews. Washington, DC: The National Academies Press; 2011. [cited 2016 Jun 7]. Available from: http://www.nationalacademies.org/hmd/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews.aspx .

Balshem H, Stevens A, Ansari M, Norris S, Kansagara D, Shamliyan T, Chou R, Chung M, Moher D, Dickersin K. Finding grey literature evidence and assessing for outcome and analysis reporting biases when comparing medical interventions: AHRQ and the Effective Health Care Program. Methods Guide for Comparative Effectiveness Reviews. Rockville: Agency for Healthcare Research and Quality (US); 2013. [cited 2017 Jan 3]; Contract Nos.: 290-2007-10057-I and 290-2007-10059-I. Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/24404628 .

Lefebvre C, Manheimer E, Glanville J. Chapter 6.2.3: Unpublished and ongoing studies. In: Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration. 2011. [cited 2017 Jan 4]. Available from: http://handbook.cochrane.org/ .

Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews?: Empirical study. Health Technol Asses. 2003;7(1). [cited 2017 Jan 4] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/12583822 .

Jefferson T, Jones MA, Doshi P, Del Mar CB, Heneghan CJ, Hama R, Thompson MJ. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database of Syst Rev. 2012 [cited 2017 Jan 3];Jan 18. Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/22258996 .

Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ. 2012;344:d7202. [cited 2017 Jan 3] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/22214754 .

Article PubMed Google Scholar

Turner EH, Knoepflmacher D, Shapley L. Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database. PLoS Med. 2012;9(3):e1001189. [cited 2017 Jan 4] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/22448149 .

Article PubMed PubMed Central Google Scholar

Golder S, Loke YK, Wright K, Norman G. Reporting of adverse events in published and unpublished studies of health care interventions: a systematic review. PLoS Med. 2016;13(9):e1002127. [cited 2017 Jan 4] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/27649528 .

Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E. Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet. 2004;363(9418):1341–5. [cited 2017 Jan 3] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/15110490 .

Article CAS PubMed Google Scholar

Lefebvre C, Manheimer E, Glanville J. Chapter 6.2.1.7 Dissertations and theses databases. In: Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration. 2011. [cited 2017 Jan 4]. Available from: http://handbook.cochrane.org/ .

Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technol Assess. 2003;7(41):1–90. [cited 2017 Jan 4] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/14670218 .

Egger M, Zellweger-Zähner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet. 1997;350(9074):326–9. Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/9251637 .

Heres S, Wagenpfeil S, Hamann J, Kissling W, Leucht S. Language bias in neuroscience—is the Tower of Babel located in Germany? Eur Psychiatry. 2004;19(4):230–2. [cited 2017 Jan 3] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/15196606 .

Vickers A, Goyal N, Harland R, Rees R. Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials. 1998;19(2):159–66. [cited 2017 Jan 3] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/9551280 .

Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol. 2005;58(8):769–76. [cited 2017 Jan 3] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/16086467 .

Royle PL, Bain L, Waugh NR. Sources of evidence for systematic reviews of interventions in diabetes. Diabet Med. 2005;22(10):1386–93. [cited 2016 Jun 7] Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/16176201 .

Royle P, Waugh N. Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technol Assess. 2003;7(34):iii, ix-x, 1–51. [cited 2016 Jun 7] Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/14609481

Hartling L, Guise JM, Hempel S, Featherstone R, Mitchell MD, Motu’apuaka ML, et al. EPC Methods: AHRQ End-User Perspectives of Rapid Reviews. Rockville: Agency for Healthcare Research and Quality (US); 2016. [cited 2016 Jun 7]; Contract No.: 16-EHC014-EF. Available from NCBI Bookshelf: http://www.ncbi.nlm.nih.gov/books/NBK362003/ .

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Med Res Methodol. 2016;16(1):127. [cited 2017 Jan 4] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/27670136 .

Jones M, Jefferson T, Doshi P, Del Mar C, Heneghan C, Onakpoya I. Commentary on Cochrane review of neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Clin Microbiol Infect. 2015;21(3):217–21. [cited 2017 Jan 3] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/25658541 .

Higgins JPT, Altman DG, Sterne JAC. Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration. 2011. [cited 2016 Jun 7]. Available from: http://handbook.cochrane.org/ .

Wikipedia contributors. Thesis. Wikipedia, The Free Encyclopedia. 2017, [cited 2017 Jan 3] Available from: https://en.wikipedia.org/w/index.php?title=Thesis&oldid=758058152 .

Ganann R, Ciliska D, Thomas H. Expediting systematic reviews: methods and implications of rapid reviews. Implement Sci. 2010;5:56. [cited 2016 Jun 7] Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/20642853 .

Khangura S, Polisena J, Clifford TJ, Farrah K, Kamel C. Rapid review: an emerging approach to evidence synthesis in health technology assessment. Int J Technol Assess Health Care. 2014;30(1):20–7. [cited 2016 Jun 7] Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/24451157 .

Hartling L, Guise J-M, Kato E, Anderson J, Belinson S, Berliner E, et al. A taxonomy of rapid reviews links report types and methods to specific decision-making contexts. J Clin Epidemiol. 2015;68(12):1451–62. [cited 2016 Jun 7] Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/26278023 .

Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, et al. A scoping review of rapid review methods. BMC Med. 2015;13:224. [cited 2016 Jun 7] Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/26377409 .

Download references

Acknowledgements

The authors gratefully acknowledge the contributions of Katy Moore and Michele Pagliuso who assisted with identifying studies from Cochrane meta-analyses in each of the investigated databases.

This project was funded in part through the Canadian Institutes of Health Research (CIHR). Dr. Hartling holds a New Investigator Salary Award from CIHR.

Availability of data and materials

The datasets analyzed during the current study are available at http://dx.doi.org/10.7939/DVN/10866 .

Authors’ contributions

LH, RF, DD, BV designed the study. RF, MN and KS collected and managed the data. BV conducted the statistical analyses. LH, RF, DD and BV interpreted results. LH and RF drafted the manuscript. All authors critically reviewed and edited the manuscript and approved the final version for submission. LH is the guarantor of the study and affirms that the manuscript is an honest, accurate, and transparent account of the study being reported, that no important aspects of the study have been omitted, and that there were no discrepancies from the study as planned.

Competing interests

Authors have no financial competing interests. All authors are involved in the production of systematic reviews.

Consent for publication

Not applicable (the manuscript does not contain individual data).

Ethics approval and consent to participate

Not applicable (the study did not involve human participants).

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Author information

Authors and affiliations.

Cochrane Child Health, University of Alberta, ECHA4-472, 11405-87 Avenue, Edmonton, AB, T6G 1C9, Canada

Lisa Hartling

Alberta Research Center for Health Evidence (ARCHE), University of Alberta, 11405-87 Avenue, Edmonton, AB, T6G 1C9, Canada

Lisa Hartling, Robin Featherstone, Megan Nuspl, Kassi Shave, Donna M. Dryden & Ben Vandermeer

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lisa Hartling .

Additional file

Additional file 1:.

Description of Included Systematic Reviews. This file contains a detailed listing of the systematic reviews that were included in the analysis, as well as a summary of the reviews, by Cochrane group, which included non-English language studies, unpublished studies and/or dissertations. (DOCX 62 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Hartling, L., Featherstone, R., Nuspl, M. et al. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol 17 , 64 (2017). https://doi.org/10.1186/s12874-017-0347-z

Download citation

Received : 20 January 2017

Accepted : 12 April 2017

Published : 19 April 2017

DOI : https://doi.org/10.1186/s12874-017-0347-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Systematic reviews
Literature searching
Meta-analysis
Knowledge synthesis
Publication bias
Language bias
Grey literature
Unpublished studies
Dissertations
Non-English publications

BMC Medical Research Methodology

ISSN: 1471-2288

General enquiries: [email protected]

An official website of the United States government

Here's how you know

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Literature Search: Databases and Gray Literature

The literature search.

A systematic review search includes a search of databases, gray literature, personal communications, and a handsearch of high impact journals in the related field. See our list of recommended databases and gray literature sources on this page.
a comprehensive literature search can not be dependent on a single database, nor on bibliographic databases only.
inclusion of multiple databases helps avoid publication bias (georaphic bias or bias against publication of negative results).
The Cochrane Collaboration recommends PubMed, Embase and the Cochrane Central Register of Controlled Trials (CENTRAL) at a minimum.
NOTE: The Cochrane Collaboration and the IOM recommend that the literature search be conducted by librarians or persons with extensive literature search experience. Please contact the NIH Librarians for assistance with the literature search component of your systematic review.

Cochrane Library

A collection of six databases that contain different types of high-quality, independent evidence to inform healthcare decision-making. Search the Cochrane Central Register of Controlled Trials here.

European database of biomedical and pharmacologic literature.

PubMed comprises more than 21 million citations for biomedical literature from MEDLINE, life science journals, and online books.

Largest abstract and citation database of peer-reviewed literature and quality web sources. Contains conference papers.

Web of Science

World's leading citation databases. Covers over 12,000 of the highest impact journals worldwide, including Open Access journals and over 150,000 conference proceedings. Coverage in the sciences, social sciences, arts, and humanities, with coverage to 1900.

Subject Specific Databases

APA PsycINFO

Over 4.5 million abstracts of peer-reviewed literature in the behavioral and social sciences. Includes conference papers, book chapters, psychological tests, scales and measurement tools.

CINAHL Plus

Comprehensive journal index to nursing and allied health literature, includes books, nursing dissertations, conference proceedings, practice standards and book chapters.

Latin American and Caribbean health sciences literature database

Gray Literature

Gray Literature is the term for information that falls outside the mainstream of published journal and mongraph literature, not controlled by commercial publishers
hard to find studies, reports, or dissertations
conference abstracts or papers
governmental or private sector research
clinical trials - ongoing or unpublished
experts and researchers in the field
Library catalogs
Professional association websites
Google Scholar - Search scholarly literature across many disciplines and sources, including theses, books, abstracts and articles.
Dissertation Abstracts - dissertation and theses database - NIH Library biomedical librarians can access and search for you.
NTIS - central resource for government-funded scientific, technical, engineering, and business related information.
AHRQ - agency for healthcare research and quality
Open Grey - system for information on grey literature in Europe. Open access to 700,000 references to the grey literature.
World Health Organization - providing leadership on global health matters, shaping the health research agenda, setting norms and standards, articulating evidence-based policy options, providing technical support to countries and monitoring and assessing health trends.
New York Academy of Medicine Grey Literature Report - a bimonthly publication of The New York Academy of Medicine (NYAM) alerting readers to new gray literature publications in health services research and selected public health topics. NOTE: Discontinued as of Jan 2017, but resources are still accessible.
Gray Source Index
OpenDOAR - directory of academic repositories
International Clinical Trials Registery Platform - from the World Health Organization
Australian New Zealand Clinical Trials Registry
Brazilian Clinical Trials Registry
Chinese Clinical Trial Registry -
ClinicalTrials.gov - U.S. and international federally and privately supported clinical trials registry and results database
Clinical Trials Registry - India
EU clinical Trials Register
Japan Primary Registries Network
Pan African Clinical Trials Registry

Systematic Review and Evidence Synthesis

What is grey literature.

Grey (or gray) literature is literature produced by individuals or organizations outside of commercial and/or academic publishers. This can include information such as government reports, conference proceedings, graduate dissertations, unpublished clinical trials, and much more. The sources you select will be informed by your research question and field of study, but should likely include, at a minimum, theses and dissertations.

Why search grey literature?

How do I search for grey literature?

Finding grey literature and searching it systematically is challenging. But there are a few approaches that you can take to add some structure to your search of this type of information:

Refer to grey literature sources used for related evidence syntheses . Refer to both published evidence syntheses and registered protocols.

Ask experts in the field for relevant grey literature sources . If you are an expert, include important grey literature sources, and ask colleagues for their recommendations.

Search databases that specialize in grey literature .

This Preprint Repository Syntax Table can help you to identify potential databases to search, and aid you in successfully constructing a search strategy

Search for theses and dissertations : There are a number of databases dedicated to theses and dissertations, which you can search using your search terms.

Search clinical trials : There may be clinical trials being conducted that are relevant to your research question, but that haven't been published yet or never were published.

Identify government agencies and international and non-governmental organizations that might publish technical papers and reports on your topic. Search their websites or any online libraries that they may provide .

Search conference proceedings and newsletters : Identify professional organizations or conferences at which researchers might be presenting work related to your topic. Search those conference proceedings or newsletters on the organization's website or by contacting organizational boards for access to past proceedings that may not be online.

Contact known researchers in the field about whether there are any relevant ongoing or unpublished studies.

Search professional and trade magazines. Professional magazines contain literature that is written by professionals in the field for other professionals in the field, but that may not be about research. Trade magazines contain advertisements and news very specific to a topic or industry.

Grey literature resources

Clinical trials.

Clinical trials may go unreported in the published literature. One useful method to identify unpublished clinical trials is to search clinical trials registries. The results may be available within the registries or you may need to contact the researchers associated with the trial for further information.

ClinicalTrials.gov

International Clinical Trials Registry Platform (WHO)

Clinical Trials Registers

Conference Proceedings

OCLC PapersFirst OCLC PapersFirst is an OCLC index of papers presented at conferences worldwide.

Dissertations and Theses

Dissertations and theses are potentially rich sources of grey literature and are also usually considered grey literature themselves.

Dissertations & Theses - How to find resources by format

Grey Literature Databases

WHO Library Database

The WHO library database includes governing documents, reports and technical documentation.

Preprint Repositories

Preprint repositories are free online archives that allow researchers to identify studies in rapidly developing fields and can be used to identify studies that have not gone through the more formalized and traditional peer review process. Search and export functionality varies from site to site.

Preprints in Europe PMC

Includes bioRxiv, Research Square, Preprints.org, MedRxiv, ChemRxiv, PeerJ Preprints, F1000Res, SSRN, and others. Easy to export the basic citation to citation management software.

Free online archive of health sciences preprints (unpublished manuscripts) including original research articles, systematic reviews and meta-analyses, data articles, and articles describing methodological research/investigations and clinical research design protocols.

OSFPreprints: Preprint Archive Search

Aggregated search that covers preprint servers using the OSF infrastructure, e.g., AfricArXiv, AgriRxiv, Arabixiv, EcoEvoRxiv, EdArXiv, engRxiv, Frenxiv, IndiaRxiv, NutriXiv, PsyArXiv, SocArXiv and others, in addition to other preprint sources, e.g., arXiv, bioRxiv, cogPrints.

"SSRN is a platform for the dissemination of early-stage research. From its initial focus on the social sciences in 1994, SSRN has grown to become the most interdisciplinary service of its kind, representing disciplines across the full research spectrum, including the applied sciences, health sciences, humanities, life sciences, physical sciences, and social sciences."

Open access to over 1.9M e-prints in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.

bioRxiv: The Preprint Server for Biology

bioRxiv (bio-archive) is a free online archive and distribution service for unpublished preprints in the life sciences. Articles are not peer-reviewed, edited, or typeset before being posted online.

Preprint archive for engineering articles.

Research Preprints: Server List [Google Sheet]

Public Policy

The World Bank now makes all of their publications openly available online.

WHO Institutional Repository for Information Sharing (IRIS)

Institutional WHO database of intergovernmental policy documents and technical reports. Can search by IRIS by region (Africa, Americas, Eastern Mediterranean, Europe, South-East Asia, Western Pacific).

Health Research Web

A wiki with descriptors of national health research systems, ethics review committees, local and regional policies and research priorities and more.

Other Resources

Grey Matters: A practical search tool for evidence-based medicine

This online manual provides a thorough list of sources for grey literature in medicine and a helpful checklist to help systematize your process.

Searching the grey literature: A handbook for searching reports, working papers, and other unpublished research

Searching the Grey Literature is for librarians and information professionals interested in learning more about grey literature. This book will aid with crafting a grey lit search successfully, from start to finish. Many types of librarians will find the content of this book useful, particularly those in health or social science.

Duke University Medical Center Guide to Resource for Searching the Grey Literature

Resources for trial registries, pharmacological studies, conference abstracts, government documents and more.

Gray Literature Resources for Agriculture Evidence Syntheses

This is a comprehensive list of agricultural grey literature sources, one of several evidence synthesis worksheets and tools provided to help in non-medical evidence synthesis development and implementation.

Searching for studies: A guide to information retrieval for Campbell Systematic Reviews

This document gives comprehensive guidance in searching for evidence syntheses, and offers a robust list of grey literature sources for social science disciplines.

<< Previous: Modify Search to Other Databases
Next: Citation Management >>

Research Guides

Grey Literature

What is Grey Literature? Activities
Why Use Grey Literature?
Types of Grey Literature
Sources of Grey Literature
Searching for Grey Literature
Evaluating Grey Literature Activity
How to Incorporate & Cite Grey Literature

This guide includes content adapted with permission from the University of Illinois Library .

Evaluating Grey Literature

Grey literature presents a viable alternative to conventional scholarly and peer-reviewed literature, albeit necessitating heightened scrutiny in information assessment compared to the latter. The evaluation of grey literature entails a rigorous examination of various factors, including the document's intrinsic characteristics, the credibility and reputation of the producing organization, and the veracity of the information conveyed. One established framework facilitating this critical evaluation process is the AACODS checklist, devised by Jess Tyndall.

Tyndall, J. (2010). AACODS checklist . Flinders University. https://fac.flinders.edu.au/dspace/api/core/bitstreams/e94a96eb-0334-4300-8880-c836d4d9a676/content

Tyndall, J. (2008). How low can you go? Toward a hierarchy of grey literature . Conference: Dreaming08: Australian Library and Information Association Biennial Conference, Alice Springs. https://core.ac.uk/download/pdf/14931482.pdf

<< Previous: Searching for Grey Literature
Next: Evaluating Grey Literature Activity >>
Last Updated: Jul 31, 2024 10:19 AM
URL: https://researchguides.library.vanderbilt.edu/greyliterature

Systematic Reviews: Grey Literature & Handsearching

Before You Begin
Define Topic & Register Protocol
PubMed Tips
Search Results Delivery
Grey Literature & Handsearching
Screening & Analysis

In this Guide

What is grey literature?
Where to find grey literature
Read more about grey literature

Handsearching

Cache that content: keeping track

What is Grey Literature

Grey literature refers to both published and unpublished research material that is not available commercially. in general, grey literature publications are non-conventional and sometimes ephemeral publications that are not indexed in databases such as pubmed and embase..

Grey literature includes:

Clinical trials
Dissertations and theses
Conference proceedings
Government reports and documents
Newsletters
Social media: Twitter, Facebook, LinkedIn, etc.
Statistics and data
Technical specifications and standards
Technical and commercial documentation
Including unpublished results reduces publication bias, which includes the phenomenon that studies reporting a positive result are more likely to be published than those finding a negative result.
Incorporating unpublished trial data can change statistical results.
Global literature should be included.
Grey literature is often more current.
Current IOM systematic review standards call for grey literature inclusion.

Referencing and Citing Grey Literature

Consider using a citation management program to organize your search results.

Author or (organizational author)
Year of publication
Publisher information (for printed documents)
Retrieved from and URL (for online material)

Where to Find Grey Literature

Clinical Trials
Conference Papers
Databases & Search Engines
Dissertations
US Government
International

Advanced Search

Australian Clinical Practice Guidelines
CMS Measures Inventory Tool

Statistical Resources in Medicine and Healthcare by Rachel Schwartz Last Updated Jul 1, 2024 527 views this year

International Network of Agencies for Health Technology Assessment (INAHTA)

IOM Standard 3.2.4 states: that researcher should "Handsearch selected journals and conference abstracts." Handsearching can be done of either paper or electronic journals and involves a page-by-page search. Reasons for handsearching include:

Indexing inconsistencies in bibliographic databases
Selective inclusion of articles
Difficulty of retrieving non-English articles in a database search

Critical Appraisal: AACODS Framework

Objectivity
Significance
Can you identify who is responsible for the intellectual content?
Is the work associated with a reputable organization?
Is the organization reputable? (e.g. W.H.O.)?
Can you easily identify the producing source?
Is he organization an authority in the field?
Does the item have a detailed reference list or bibliography?
Does the item have a clearly stated aim? If so, is it met?
Does it have a stated methodology?
Has it been peer-reviewed?
Has it been edited by a reputable authority?
The item refers to a particular population group, or excludes certain types of publications.
Are any limits clearly stated?
Opinion, expert or otherwise, is still opinion: is the author’s standpoint clear?
Does the work seem to be balanced in presentation?
Is there a conflicts of interest?
Does the item have a clearly stated date related to the content?
For the item to inform your research, it needs to have a date that confirms relevance
No easily visible date is a strong concern
Have key contemporary material been included in the bibliography?
Is the item meaningful?
Does it enrich or negate a current position?
Is the publication relevant?

Critical Appraisal: CRAP or CRAAP Test

Purpose (Point of View)

The CRAAP test is a method for evaluating research based on the following criteria

When was the information published or posted?
Has the information been revised or updated?
Is the information current or too out-of-date for my topic?
Are all the links functional or are there dead links?*

Developed by Sarah Blakeslee and her team of librarians at California State University, Chico (CSU Chico) to check the reliability of sources across academic disciplines .

Does the information relate to my topic or answer my question?
Who is the intended audience?
Is the information at an appropriate level (i.e. not too simple or advanced) for my needs?
Did I look at a variety of sources before deciding to use this one?
Would I be comfortable using this source for my college research paper?
Who is the author/publisher/source/sponsor?
Are the author's credentials or organizational affiliations given?
What are the author's credentials or organizational affiliations?
What are the author's qualifications to write on the topic?
Is there contact information, such as a publisher or e-mail address?
Does the URL reveal anything about the author or source? Examples: .com .edu .gov .org .net*
Where does the information come from?
Is the information supported by evidence?
Has the information been reviewed by anyone else?
Can I verify any of the information in another source or from personal knowledge?
Does the language or tone seem biased? Or is it free of emotion?
Are there spelling, grammar, typographical, or other errors?
What is the purpose of the information? to inform? teach? sell? entertain? persuade?
Do the authors/sponsors make their intentions or purpose clear?
Is the information fact? opinion? propaganda?
Does the point of view appear objective and impartial?
Are there political, ideological, cultural, religious, institutional, or personal biases?

Record Keeping

Keep track of the search terms or search methods used
Tools to Capture Results
When grey literature is searched, a detailed account of the approach should be documented
It is important that your protocol documents the types of grey literature you will use and where you will look
When searching, you should record exactly what you do from the minute you start
This is important to ensure that you can reproduce your search later if necessary
Print/save the complete record and content of the results

Managing and Reporting Results

Google Docs

List any trials and other research registries, web search engines, specific web sites, conference proceedings, or other resources searched, including their dates of coverage

List all websites searched, search methods (advanced or basic)
File a print copy or save a screen shot of items found on the internet, rather than simply ‘book-marking’ the site, in case the record of the trial is removed or altered at a later stage (beware of "link rot."
Which search engines, search terms used?
It’s important to keep a record of the date the web site was accessed, dates of coverage and Limitations, for citation purposes. - - Cochrane handbook guidelines

Writing Up Search Methods

BE TRANSPARENT

Focused or broader
Inclusion/exclusion criteria
Databases, structured internet searching
Unpublished clinical trials, dissertations
Search terms
Number of results
Limitations

Avoid Predatory Publishers

Is your inbox inundated with invitations from unfamiliar publishers to submit your manuscript to a journal you've never heard of? The number of journals in the biomedical sciences has increased exponentially in the past few years. How can you tell if a journal is legit?

Here are a few questions you can ask yourself to help guide your decision about publishing in a particular journal:

Does our library have a subscription ?
Is the journal indexed in PubMed ?
Is the explanation of fees clear?
Who is on the editorial board?
Is it listed in Directory of Open Access Journals (DOAJ) ?
Does it belong to the Committee on Publication Ethics (COPE) ?
Does it belong to the Open Access Scholarly Publishers’ Association (OASPA) ?

<< Previous: Search Results Delivery
Next: Screening & Analysis >>
Last Updated: Sep 6, 2024 3:49 PM
URL: https://library.einsteinmed.edu/SystematicReviews

D. Samuel Gottesman Library Albert Einstein College of Medicine Jack and Pearl Resnick Campus Forchheimer Building, 1st Floor 1300 Morris Park Avenue Bronx, NY 10461-1924 Phone: 718.430.3108 - Fax: 718.430.8795 © Albert Einstein College of Medicine

Today's Hours:

Locations and Hours
UCLA Library
Research Guides
Biomedical Library Guides

Systematic Reviews

Grey Literature
Types of Literature Reviews
Planning Your Systematic Review
Database Searching
Creating the Search
Search Filters and Hedges

What is grey literature and why is it important?

How do you search for grey literature, sources for grey literature searching.

Managing and Appraising Results
Further Resources

What is Grey Literature? Examples of Grey literature include: conference abstracts, presentations, proceedings; regulatory data; unpublished trial data; government publications; reports (such as white papers, working papers, internal documentation); dissertations/theses; patents; and policies & procedures. The Twelfth International Conference on Grey literature in Prague in 2010 arrived at the following definition: " Grey literature stands for manifold document types produced on all levels of government, academics, business and industry in print and electronic formats that are protected by intellectual property rights, of sufficient quality to be collected and preserved by libraries and institutional repositories, but not controlled by commercial publishers; i.e. where publishing is not the primary activity of the producing body ." (Adapted from https://guides.mclibrary.duke.edu/sysreview/greylit )

Why is it important? It is important to look outside of information published in databases in order to reduce publication bias where "results from significant positive findings are more likely to be submitted and accepted for publication." Several manuals including the Cochrane Handbook and PRISMA for systematic review protocol recommend and incorporate the use of grey literature searching.

Searching for grey literature can be overwhelming since it is not as easy as looking into one database. What you search for depends upon your research question. It is recommended to at least search for Abstracts and Conferences. If you are researching drugs or other interventions, search for clinical trial registries and pharma data. Here are some helpful resources about the grey literature searching process:

CADTH Grey Matters
Finding Grey Literature Evidence and Assessing for Outcome and Analysis Reporting Biases When Comparing Medical Interventions: AHRQ and the Effective Health Care Program
Abstracts and Conferences
Pharma Data
Clinical Trial Registries
Repositories and Reports
Dissertations and Theses

conference proceeedings index; FirstSearch; First search; OCLC

F1000 Research Posters
Devices@FDA
European Public Assessment Reports
Health Canada Drug Product Database (DPD)
AHRQ Grants Online Database
HSRProj (Health Services Research Projects in Progress)
HSRR (Health Services and Sciences Research Resources)
AstraZeneca Clinical Trials
Bristol-Myers Squibb Clinical Trials Registry
Eli Lilly Company Clinical Trials
GSK Clinical Studies
Novartis Clinical Trials
Pfizer Clinical Trials
ClinicalTrials.gov
WHO (World Health Organization) ICTRP (International Clinical Trials Registry Program)
ISRCTN Registry
Grey Literature Report (1999-2016)
Directory of Open Access Repositories
PROSPERO — International prospective register of systematic reviews
Registry of Open Access Repositories
Virginia Henderson International Nursing Library
<< Previous: Search Filters and Hedges
Next: Managing and Appraising Results >>
Last Updated: Jul 23, 2024 3:40 PM
URL: https://guides.library.ucla.edu/systematicreviews

A Guide to Evidence Synthesis: 3. Select Grey Literature Sources

Meet Our Team
Our Published Reviews and Protocols
What is Evidence Synthesis?
Types of Evidence Synthesis
Evidence Synthesis Across Disciplines
Finding and Appraising Existing Systematic Reviews
0. Develop a Protocol
1. Draft your Research Question
2. Select Databases
3. Select Grey Literature Sources
4. Write a Search Strategy
5. Register a Protocol
6. Translate Search Strategies
7. Citation Management
8. Article Screening
9. Risk of Bias Assessment
10. Data Extraction
11. Synthesize, Map, or Describe the Results
Evidence Synthesis Institute for Librarians
Open Access Evidence Synthesis Resources

Video: Grey literature (3:10 minutes)

The Grey Literature Background

What is grey literature .

Why Search the Grey Literature

Why search the grey literature .

How to Search the Grey Literature

How Do I Search the Grey Literature?

Finding grey literature and searching it systematically is challenging. But there are a few approaches that you can take to add some structure to your search of this type of information:

Refer to grey literature sources used for related evidence syntheses: Refer to both published evidence syntheses and registered protocols.
Ask experts in the field for relevant grey literature sources: If you are an expert, include important grey literature sources, and ask colleagues for their recommendations.
Search databases that specialize in grey literature: See the "Grey Literature Sources" box at the bottom of this page for more information.
Search for theses and dissertations : There are a number of databases dedicated to theses and dissertations, which you can search using your search terms. See the "Grey Literature Sources" box at the bottom of this page for links to these resources.
Search clinical trials: There may be clinical trials being conducted that are relevant to your research question, but that haven't been published yet or never were published. See the "Grey Literature Sources" box at the bottom of this page for links to these resources.
Identify government agencies and international and non-governmental organizations that might publish technical papers and reports on your topic. Search their websites or any online libraries that they may provide. See the "Grey Literature Sources" box at the bottom of this page for links to some examples.
Search conference proceedings and newsletters : Identify professional organizations that have and/or conferences at which researchers might be presenting work related to your topic. Search those conference proceedings or newsletters on the organization's website or by contacting organizational boards for access to past proceedings that may not be online. See the "Grey Literature Sources" box at the bottom of this page for some examples.
Contact known researchers in the field to determine if there are any ongoing or unpublished studies that s/he may be aware of.
Search professional and trade magazines: Professional magazines contain literature that is written by professionals in the field for other professionals in the field, but that may not be about research. Trade magazines contain advertisements and news very specific to a topic or industry.

How to Manage the Grey Literature Search

How do i manage the grey literature search.

Identify and record the sources you will search. The sources you search will be informed by your research question and where you expect to find information related to your question.
Document where you are searching and your search strategies, including document resource name, URL, search terms, and date searched.
Collect citation information as you go.
Adhere to your established inclusion and exclusion criteria when selecting sources.

See below for guidance documents specific to grey literature searching.

Grey Literature Sources Header

Grey literature resources, grey literature sources.

Grey Literature Databases
Theses and Dissertations
Clinical Trials
Public Policy
Conference Proceedings
Preprint Repositories
Other Resources
WHO Library Database The WHO library database includes governing documents, reports and technical documentation.
MedNar MedNar searches across more than 60 medical research sources, including commercial databases, medical societies, NIH resources, and other government resources.
Global Index Medicus The Global Index Medicus (GIM) provides worldwide access to biomedical and public health literature produced by and within low-middle income countries. The main objective is to increase the visibility and usability of this important set of resources. The material is collated and aggregated by WHO Regional Office Libraries on a central search platform allowing retrieval of bibliographical and full text information.
NY Academy of Medicine Grey Literature Report (Last Updated 2016) This report is a bimonthly publication of The New York Academy of Medicine (NYAM) alerting readers to new grey literature publications in health services research and selected public health topics. The database platform is keyword searchable and serves as an archive for the cataloged reports. Although it is no longer updated, existing reports can be found through the NYAM catalog.
ProQuest Dissertations and Theses With more than 2 million entries, PQD&T is the single central, authoritative resource for information about doctoral dissertations and master's theses.
Networked Digital Library of Theses and Dissertations (NDLT) NDLT is a free international resource for theses and dissertations.
Center for Research Libraries (CRL) CRL is a resource for institutions outside of the U.S. and Canada.
OCLC WorldCat Dissertations and Theses OCLC WorldCat Dissertations and Theses "provides fast and convenient access to the dissertations and theses available in OCLC member libraries. Many theses are available electronically, at no charge, directly from the publishing institution." This database can be searched in FirstSearch, which provides Basic, Advanced, and Expert search options. Access may be limited to members of participating institutions.
Guide to Theses and Dissertations Resources This guide provides information on finding and accessing both Cornell and non-Cornell theses and dissertations, including many resources for non-US literature.
EThOS (offline) Electronic Theses Online Service (EThOS) contains doctoral theses from candidates in the UK and is maintained by the British Library. Due to the October 2023 cyber attack, this resource is currently unavailable and is estimated to return in 2025.
Australia New Zealand Clinical Trials Registry Online register of clinical trials being undertaken in Australia, New Zealand and elsewhere. The ANZCTR includes trials from the full spectrum of therapeutic areas of pharmaceuticals, surgical procedures, preventive measures, lifestyle, devices, treatment and rehabilitation strategies, and complementary therapies.
ClinicalTrials.gov The US registry for clinical trials. Includes new, ongoing, and completed human clinical trials both in the US and countries around the world.
Cochrane Central Register of Controlled Trials A highly concentrated source of reports of randomized and quasi-randomized controlled trials. The majority of CENTRAL records are taken from bibliographic databases (mainly MEDLINE and Embase), but records are also derived from other published and unpublished sources.
EU Clinical Trials Register The European Union Clinical Trials Register allows you to search for protocol and results information on interventional clinical trials that are conducted in the European Union (EU) and the European Economic Area (EEA) and clinical trials conducted outside the EU / EEA that are linked to European paediatric-medicine development.
WHO International Clinical Trials Registry Platform ICTRP provides access to a central database containing the trial registration data sets provided by numerous international registries. It also provides links to the full original records.
Wikipedia Clinical Trial Registries List This Wikipedia entry contains a list of links to individual clinical trial registries by country.
World Bank The World Bank now makes all of their publications openly available online.
WHO Institutional Repository for Information Sharing (IRIS) Institutional WHO database of intergovernmental policy documents and technical reports. Can search by IRIS by region (Africa, Americas, Eastern Mediterranean, Europe, South-East Asia, Western Pacific).
Health Research Web Health Research Web is an interactive resource containing information about national health research systems, ethics review committees, local and regional policies and research priorities and more. It is built by user contributors, and the data may be incomplete.

OCLC PapersFirst OCLC PapersFirst is an OCLC index of papers presented at conferences worldwide.
BIOSIS Previews A subcollection of Web of Science, BIOSIS Previews searches across journals, meetings, patents, and books in the life sciences and biomedical fields.
arXiv arXiv is a free distribution service and an open-access archive for scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.
medRxiv medRxiv (pronounced "med-archive") is a free online archive and distribution server for complete but unpublished manuscripts (preprints) in the medical, clinical, and related health sciences.
bioRxiv Another complementary archive to arXiv, bioRxiv is a preprint repository that where authors can share unpublished preprints in the life sciences (including biology, ecology, neuroscience and more).
OSF Preprints OSF Preprints is a general preprint repository that covers architecture, business, engineering, life sciences, physical sciences and mathematics, arts and humanities, education, law, medicine and health sciences and social and behavioral sciences, among others.
Grey Matters: A practical search tool for evidence-based medicine Grey Matters is an online manual that provides a thorough list of sources for grey literature in medicine and a helpful checklist to help systematize your process. (Requires free account)
Searching the grey literature: A handbook for searching reports, working papers, and other unpublished research Searching the Grey Literature is for librarians and information professionals interested in learning more about grey literature. This book will aid with crafting a grey lit search successfully, from start to finish. Many types of librarians will find the content of this book useful, particularly those in health or social science.
Duke University Medical Center Guide to Resource for Searching the Grey Literature Scroll down to the bottom of this Duke University page to find resources for trial registries, pharmacological studies, conference abstracts, government documents and more.
Gray Literature Resources for Agriculture Evidence Syntheses This OSF-hosted sheet is a comprehensive list of agricultural grey literature sources, one of several evidence synthesis worksheets and tools provided to help in non-medical evidence synthesis development and implementation.
Searching for studies: A guide to information retrieval for Campbell Systematic Reviews This document gives comprehensive guidance in searching for evidence syntheses, and offers a robust list of grey literature sources for social science disciplines.
<< Previous: 2. Select Databases
Next: 4. Write a Search Strategy >>
Last Updated: Sep 3, 2024 4:30 PM
URL: https://guides.library.cornell.edu/evidence-synthesis

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

Advanced Search
Journal List
Saudi J Anaesth
v.17(2); Apr-Jun 2023
PMC10228866

Significance of including grey literature search in systematic reviews and meta-analyses

Abhijit nair.

Department of Anaesthesiology, Ibra Hospital, Ministry of Health-Oman, Ibra-414, Sultanate of Oman

Nitin K. Borkar

1 Department of Paediatric Surgery, All India Institute of Medical Sciences, Raipur, Chhattisgarh, India

Dear Editor,

A systematic review is considered as the strongest evidence and occupies the top place in the research pyramid. A researcher involved in conducting a systematic review and/or meta-analysis (SRMA) needs to perform a literature review that involves a search for articles with relevant keywords in various databases. The popular and recommended databases are PubMed/Medline, EMBASE (Excerpta Medical Database), SCOPUS, CENTRAL (The Cochrane Central Register of Controlled Trials), CINAHL (The Cumulative Index to Nursing and Allied Health), and Web of Science.[ 1 ]

A meticulous SRMA also involves something which is known as grey literature which refers to research that is produced on all levels of government, academics, business, and industry in print and electronic formats, but not controlled by commercial publishers.[ 2 ] Grey literature is not indexed by major databases but can be searched through various search engines. Grey literature basically comprises of reports, conference abstracts and posters, theses, dissertations, and white papers. In 2011, the Institute of Medicine suggested inclusion of grey literature search while performing any SRMA.[ 3 ]

The reason for performing a grey literature search is to reduce publication bias (by including data which is not peer reviewed or in an indexed database probably because of negative findings). Grey literature search also helps to gather data which is not available commercially so as to comprehensively review and analyze a research question and for a more precise effect size estimate. Experts feel that excluding grey literature from analyses could compromise the validity and reliability of meta-analyses and the specificity of systematic reviews.[ 4 ] McAuley et al. [ 5 ] found that 33% of meta-analyses reviewed included grey literature, accounting for 4.5% to 75% of studies in meta-analysis. The Cochrane Handbook for Systematic Reviews of Interventions also mentions that a failure to identify and analyze studies from various grey literature search could impact the results of an SRMA.[ 6 ]

There are several ways of searching documents available in the grey literature. The following are the websites: https://www.cadth.ca/grey-matters-practical-tool-searching-health-related-grey-literature , https://opengrey.eu/ , https://www.nyam.org/library/collections-and-resources/grey-literature-report/ , https://www.tripdatabase.com/ , https://www.oclc.org/en/oaister.html , http://mednar.com/ . The grey literature can also be searched on clinical trial registries ( http://clinicaltrials.gov/ ), patent databases, company and industry repositories, digital archives ( https://www.fda.gov/ ), clinical practice guidelines of various regions, factsheets, clinical communications, websites of various organizations and institutes, and newspapers.[ 7 ] As there are no recommended strategies for performing a grey literature search, it could turn out to be time consuming and at times frustrating if the search does not yield anything definitive. Taking the help of a librarian is recommended for a grey literature search.

In conclusion, searching grey literature and mentioning the search strategies are important while performing SRMA. However, the search strategies are heterogenous, are not streamlined, and take a lot of time. Nevertheless, searching it is essential for arriving at a definitive conclusion after performing a comprehensive SRMA.

Abhijit Nair: This author helped in concept, design, definition of intellectual content, literature search, data acquisition, data analysis, statistical analysis, manuscript preparation, manuscript review.

Nitin Kumar Borkar: This author helped in definition of intellectual content, literature search, data acquisition, manuscript review.

Financial support and sponsorship

There are no conflicts of interest.

Skip to Guides Search
Skip to breadcrumb
Skip to main content
Skip to footer
Skip to chat link
Report accessibility issues and get help
Go to Penn Libraries Home
Go to Franklin catalog

Critical Writing Program: Multilingualism and Identity Fall 2024: Grey Literature

Getting started
News and Opinion Sites
Academic Sources

Grey Literature

Substantive News Sources
What to Do When You Are Stuck
Understanding a citation
Examples of Quotation
Examples of Paraphrase
Chicago Manual of Style: Citing Images
Researching the Op-Ed
Researching Prospective Employers
Resume Resources
Cover Letter Resources

Grey Literature is not academic/scholarly piece of writing. It includes government documents, foundation reports, white papers, working papers, issue briefs, organizational (versus academic) research, and other kinds of policy reports. It can be challenging to distinguish grey literature from academic sources, and often enough scholars are authors of this grey literature, blurring the lines all the more. An important distinction between grey literature and academic publications are that the latter are peer reviewed by scholars with expertise in the field.

Searching for Grey Literature:

You may find references to this kind of literature in bibliographies of articles and books (e.g., citation chasing).
You can often find it by means of a Google search, experimenting with different key words and adding genres such as “foundation report," "issue brief," "policy report," "funding report,” “working paper,” “study,” or “white paper.”
Pew Research Center is a "a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world"
Ford Foundation -- The Ford Foundation's mission is to, "reduce poverty and injustice, strengthen democratic values, promote international cooperation, and advance human achievement."
United Nations -- Penn Libraries subscribes to the UN iLibrary . You can also search the UN site directly.
Public Affairs Information Service (PAIS) is a database that indexes and provides access to government documents, statistical directories, grey literature, research reports, conference reports, publications of international agencies.
HeinOnline provides access to legal periodicals, international law, treaties, court reports, historical and contemporary legislation, and more
Policy Commons -- a one-stop community platform for objective, fact-based research from the world's leading policy experts, nonpartisan think tanks, IGOs and NGOs.

Some social science databases will index this material and it may appear in government or organizational websites or through other means rather than formal publication. The creators of these documents may be teams or individuals, and the authors are not always listed.

Unlike academic articles, grey literature doesn't necessarily go through a vetting or peer review process. As such, when using this material, you will want to have an understanding of the organizational body sponsoring the work. Who funds them? What is their mission? Are they a lobbying or political group? What kinds of information do they publish? Do other organizations cite them? Who and for what reasons? Google Scholar is a great way to find out who is citing a particular publication.

<< Previous: Academic Sources
Next: Substantive News Sources >>
Last Updated: Sep 4, 2024 2:17 PM
URL: https://guides.library.upenn.edu/c.php?g=1422116

Relapse prevention following guided self-help for common health problems: A Scoping Review

Original Article
Open access
Published: 03 September 2024

Cite this article

You have full access to this open access article

Saher Nawaz ORCID: orcid.org/0009-0007-8188-220X 1 ,
Penny Bee 1 ,
Hannah Devaney 1 &
Cintia Faija 1 , 2

To gain an in-depth understanding of interventions, tools, and resources available focused on maintaining recovery and preventing relapse for patients with anxiety and/or depression symptoms following guided self-help (GSH).

The literature search was conducted on four electronic databases from inception until May 2024 (PsycINFO, CINAHL Plus, PubMed and Web of Science). Additional searches were also conducted through other sources, including Grey Literature Databases, Google Search Engine, Citations and contacting experts in the field. All identified articles were screened for eligibility by two independent reviewers and quality appraised.

A total of 1277 records across databases and other sources were identified. After removing duplicates, 511 were screened for eligibility. A total of six references met the inclusion criteria and were included in the review. Three peer-reviewed publications were identified, and all reported an intervention targeting relapse prevention following GSH which included monthly telephone follow-up calls. The other three sources included two workbooks and a mobile application developed and completed towards the final GSH sessions and used following treatment. All of the articles highlight the importance of independently learning and practicing skills and strategies to ease an individuals’ symptoms following discharge from GSH.

Conclusions

The current review found limited evidence surrounding recovery and relapse prevention interventions following GSH. Developing this field of research by further creating and testing relapse prevention interventions can provide an understanding of the core components needed in such tools, to successfully maintain treatment gains over time and support individuals to continue their recovery journey.

User Experiences of CBT for Anxiety and Depression: A Qualitative Systematic Review and Meta-synthesis

The GET READY relapse prevention programme for anxiety and depression: a mixed-methods study protocol

The Effectiveness of Low-Intensity Psychological Interventions for Comorbid Depression and Anxiety in Patients with Long-Term Conditions: A Real-World Naturalistic Observational Study in IAPT Integrated Care

Explore related subjects.

Artificial Intelligence

Avoid common mistakes on your manuscript.

Introduction

Anxiety and depression are two of the most common mental health disorders globally (The Lancet, 2021 ). They have significant implications for an individual’s social and occupational functioning (Saris et al., 2017 ; Gunnarsson et al., 2021 ), physical health and mortality (Ferrari et al., 2013 ; Roy-Byrne et al., 2008 ), leading to increased health and economic burden. Research reports that the costs of mental health problems across the United Kingdom (UK) pertained to approximately £117.9 billion, in 2019, with 72% of these costs resulting from the loss of productivity of people living with mental health problems and from unpaid informal carers (McDaid & Park, 2022 ).

Various pharmacological and psychological treatments are available for depression and anxiety. However, patients often prefer psychotherapy treatments as opposed to medication (Backenstrass et al., 2006 ; McHugh et al., 2013 ; van Schaik et al., 2004 ), which are also the recommended first step of treatment for anxiety and/or depression symptoms (NICE, 2014 ; 2022a ). Guided Self-Help (GSH) interventions are low-intensity evidence-based psychological treatments (Baguley et al., 2010 ; Clark, 2018 ) that involve supporting patients to understand and overcome their anxiety and/or depression symptoms using techniques based in cognitive behavioural therapy (CBT).

GSH interventions offer individuals dealing with anxiety and/or depression symptoms an opportunity to access cost-effective and evidence-based psychological treatment that caters to their preferences (Palacios et al., 2023 ). This helps reduce barriers to care, such as limited access to high-quality treatment (Alonso et al., 2018 ; Thornicroft et al., 2017 ). The efficacy of GSH is substantiated by multiple systematic reviews (Cuijpers et al., 2010 ; Coull & Morris, 2011 ; Lewis et al., 2012 ; Powell et al., 2024 ), which present compelling evidence of its potential to alleviate symptoms of anxiety and/or depression. Notably, a recent systematic review involving various types of GSH demonstrated superior efficacy in reducing anxiety, depression, and worry compared to control groups (Powell et al., 2024 ). Furthermore, GSH interventions have exhibited positive outcomes in addressing social phobia and panic disorders, particularly when complemented with self-help materials such as books, leaflets, websites, and videos (Lewis et al., 2012 ).

In England, psychological wellbeing practitioners (PWPs), are qualified practitioners and are responsible for administering GSH (i.e., low-intensity interventions) (Coull & Morris, 2011 ; Falbe-Hansen et al., 2009 ) using written materials like booklets or digital tools. PWPs play a vital role in delivering GSH within talking therapy services, motivating a client throughout their treatment (McDevitt-Petrovic, 2019 ). Regardless of the mode of delivery of the GSH intervention, whether it is online, face-to-face, by telephone or group, PWPs provide patients with personalized tools and techniques for self-managing their symptoms of anxiety and/or depression (Stonebank, 2014 ).

For this scoping review, GSH refers to any low intensity psychological interventions involving support from trained practitioners and utilising self-help materials (Shafran et al., 2021 ). According to the National Institute of Health and Care Excellence (NICE), step 2 care for anxiety and/or depression symptoms involves individual facilitated self-help, computerised Cognitive Behavioural Therapy (CBT), self-help groups, and psychoeducational groups (NICE, 2011 ). Low–intensity treatments (i.e., GSH) are typically delivered across six to eight sessions lasting up to 30 min (Papworth & Marrinan, 2018 ), adhering to NICE guidelines (NICE, 2023 ).

The effectiveness of GSH treatments is evidenced in past research (Coull & Morris, 2011 ; Gellatly et al., 2007 ; Salomonsson et al., 2018 ). A systematic review of 21 studies demonstrated comparable effects of GSH with traditional face-to-face psychotherapies after a one-year follow-up (Cuijpers et al., 2010 ). Another systematic review exploring the effects of GSH via computerised CBT, provides evidence demonstrating the efficacy and acceptability of GSH interventions as treatment for anxiety and/or depression symptoms (Etzelmueller et al., 2020 ). In England, the effectiveness of low intensity interventions, like GSH, administered within NHS Talking Therapies services is evidenced by recovery rates of 50% meeting national standards (NHS, 2022 ).

Although psychological interventions are effective at reducing symptoms of anxiety and/or depression symptoms (Hoffman & Gomez., 2017 ), rates of relapse i.e., the recurrence of symptoms after a period of improvement (Bockting et al., 2015 ) following clinically and cost-effective psychological treatments are fairly high. Research indicates that the prevalence of a second episode after treatment for depression is 50%, this figure rising to 90% following three episodes (Burcusa & Lacono., 2007 ). The relapse rate for anxiety is also fairly high as demonstrated by a meta-analysis of nine studies which found an average of 23.8% relapse following CBT (Lorimer et al., 2021 ).

Considering long-term effectiveness of low-intensity interventions, longitudinal research conducted in NHS Talking Therapies services indicates that both low and high-intensity treatments are associated with sustained increases in depression and anxiety symptoms post-treatment (Clark et al., 2009 ). Further research exhibits the high rates of relapse by discharged patients who achieved recovery, with significant deterioration evidenced within six months post discharge (Ali et al., 2017 ). Additionally, a systematic evaluation focusing on low-intensity interventions within NHS Talking Therapies services showed that 65.8% of patients experience relapse or recurrence within one year (Delgadillo et al., 2018 ). Notably, the risk of relapse following low-intensity interventions (i.e., GSH) is heightened for younger individuals, those unemployed, and reporting residual symptoms at discharge (Lorimer et al., 2021 ). Therefore, future research should contribute to further understand challenges faced by patients following end of treatment and design interventions accordingly to maintain treatment gains over time and mitigate the risk of relapse.

Furthermore, research shows that those who have previously experienced an episode of depression or anxiety, continue to experience impaired functions and work disability (Hendriks et al., 2015 ), showcasing the important need for support (NICE, 2022b ) beyond the therapy setting.

Relapse prevention has been defined in many ways within a therapy setting for different disorders (Marlatt & George, 1984 ; Melemis, 2015 ; Menon & Kandasamy, 2018 ), but they all share two key principles: (1) the aim of relapse prevention is to identify and address any situation which could lead to deterioration, (2) it requires individuals to successfully develop and implement coping skills to address high risk situations. Through relapse prevention, individuals are able to maintain recovery and consolidate changes and acquisition of skills in the long term (Melemis, 2015 ).

The high rates of relapse following different psychological interventions demonstrates the need for preventative approaches to tackle the challenges of relapse and recurrence which prevent long-term recovery. By understanding the causes and processes of relapse, advances in the long-terms efficacy of psychological therapies can be made (Brandon et al., 2007 ; Scholten et al., 2013 ). Some practitioners delivering low-intensity treatment may discuss relapse prevention throughout the course of treatment to ensure sufficient time is provided to any relapse prevention work (Papworth & Marrinan, 2018 ). Additionally, providing patients with personalised treatment plans and increasing the ability for patients to make informed choices, are some of the ways in which relapse prevention programmes can be designed, as demonstrated through a discrete choice experiment (Muntingh et al., 2019 ).

Maintaining wellbeing over time can decrease the financial burden and the distress associated with living with issues associated with conditions such as depression and anxiety (Mental Health Foundation, 2016 ). Through the development of programmes focussed on supporting individuals after their treatment to help increase self-management, rates of relapse can decrease along with associated burdens.

Understanding what relapse prevention interventions, tools or resources are available following GSH would equip practitioners and researchers with the knowledge to determine effective strategies to address relapse and identify areas for improvement. This scoping review aims to explore available evidence on how to maintain treatment gains and prevent relapse following GSH. It will present a detailed inventory of the available relapse prevention materials/resources/interventions along with measures of effectiveness (if available) and it will highlight the gaps in the evidence, informing future research.

A scoping review was chosen rather than a standard systematic review to provide an overview and synthesise the research evidence on a certain topic (Lockwood et al., 2019 ; Tricco et al., 2016 ). A standard systematic review, differently to a scoping review, aims to answer a specific research question (Tokgöz et al., 2021 ) and can be undertaken to confirm or refute hypotheses, report on the quality of the evidence base or address any variations or uncertainty (Munn et al., 2018 ). Unlike systematic reviews, scoping reviews do not aim to inform practice or consider questions related to feasibility, appropriateness, and effectiveness, instead seeking to clarify key concepts, provide background and context behind specific phenomena (Pollock et al., 2021 ) and identify knowledge gaps (Munn et al., 2018 ). Unlike systematic reviews, scoping reviews are helpful when identifying, reporting, and discussing specific characteristics and concepts across papers or studies (Munn et al., 2018 ). They provide baseline data about the availability of evidence on a particular topic (Lockwood et al., 2019 ), such as relapse prevention following GSH. Scoping reviews are exploratory in nature, aiming to address broader questions about the current literature rather than specific inquiries and questions (Munn et al., 2022 ). Hence, a scoping review was chosen to provide an overview of relapse prevention interventions designed following GSH and their core components without reporting on quality and effectiveness.

This scoping review follows the guidelines provide by Arksey and O’Malley ( 2005 ): (1) identification of a research question through discussions with the research team where the question was refined, (2) identifying relevant studies using electronic databases and grey literature searches, (3) study selection through the use of an inclusion and exclusion criteria, (4) charting data through a narrative review of included studies and resources and (5) collating, summarising and reporting results by providing an overview of the material reviewed and reporting basic characteristics of the included literature.

Protocol and Registration

This scoping review follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines for scoping reviews (PRISMA-ScR; Tricco et al., 2018 ). A scoping review protocol was registered on FigShare.

( https://figshare.manchester.ac.uk/articles/preprint/What_do_we_know_about_relapse_prevention_following_guided-self-help_for_anxiety_and_depression_A_scoping_review_protocol_/23179181?file=42906871 ).

Eligibility Criteria

The inclusion criteria were (1) studies conducted in adults aged 18 or over, (2) studies reporting on symptoms of depression and/or anxiety, (3) studies reporting on relapse prevention interventions/tools/resources used for depression and/or anxiety following GSH. There were no restrictions regarding the methodology of the studies, allowing for the inclusion of qualitative and quantitative research.

The exclusion criteria were (1) participants where the main health disorder described is not anxiety and/or depression symptoms but this exists as a comorbidity, (2) studies not reporting information on relapse prevention, (3) studies that report information on relapse prevention work completed during GSH (i.e., not after completion of GSH) (4) studies where the primary outcomes are not related to anxiety and/or depression, (5) studies where the intervention is designed to replace existing psychotherapy treatment for anxiety and/or depression.

Information Sources

The original literature search was conducted on the 9th of May 2023 and updated on 30th May 2024. There was no limit for years and all results were from inception until the date of the search. Identification of relevant studies was achieved by searching four different electronic databases including PsycINFO, CINAHL Plus, PubMed and Web of Science. The search strategy was discussed and curated with the research team and comprised key subject terms associated with the research question which were combined using Boolean operators. The following search strategy was used: (“self -help” OR “self-management” OR “low intensity”) AND (“relapse prevention” OR “remission” OR “recurrence”) AND (“Anxiety” OR “depression”). The reference list of included studies was hand searched by two independent reviewers (SN and HD) to identify additional relevant studies not found through the search strategy. In addition, the reference list of any reviews or meta-analyses deemed relevant were also hand-searched.

The title and abstract of all retrieved searches were independently reviewed by SN and HD against the inclusion and exclusion criteria. Any articles that were deemed relevant or where there was not enough information during screening of the title/abstract, the articles were subject to a full text review by two independent reviewers against the inclusion and exclusion criteria. Further clarification regarding whether an article met the inclusion and exclusion criteria was sought through contacting the authors where available. In the case of disagreement between reviewers, the opinion of a third reviewer (CF) was sought.

Other Sources of Evidence

In addition to the literature search identifying peer-reviewed articles, this scoping review explored grey literature. Although there are many definitions of grey literature (Adams et al., 2016 ), the most widely agreed upon definition is literature that is “produced on all levels of government, academics, business and industry in electronic and print formats not controlled by commercial publishers” (Auger, 1998 ). Therefore, it is anything that is not formally published in a peer-reviewed journal (Cooper et al., 2009 ; Godin et al., 2015 ) and can include websites, policy documents, conference proceedings, and unpublished research (Higgins & Green, 2011 ). Incorporating grey literature in any forms of evidence synthesis, such as scoping reviews, is seen as good practice as it reduces aspects of publication bias (Hopewell et al., 2007 ), providing a more comprehensive review.

Following Godin et al. ( 2015 ), a systematic approach was used when searching for grey literature via three different methods: Google searches, grey literature databases and consultation with experts.

The google search strategy involved inputting the search string used for the literature search into the Google search engine. The first ten pages of results were reviewed by SN using the title and the accompanying short description appearing beneath the search. The grey literature databases strategy, repeated the process used for the Google search strategy, inserting the same search string into the National Grey Literature Collection Database ( https://allcatsrgrey.org.uk/tematres3.2/vocab2/ ) and reviewing all of the results according to the title of the search and accompanying description beneath. An updated search using this grey literature database could not be conducted as the database has since been discontinued, resulting in limited access (Health Education England, 2023 ). The link to any records that appeared relevant across both search strategies were extracted into a separate file for further screening against the inclusion and exclusion criteria.

The consultation with experts’ strategy, involved contacting experts across the world with knowledge on relapse prevention in the context of GSH, including clinical academics and NHS project managers and practitioners who shared their knowledge regarding existing resources and/or tools routinely used.

Data Charting

Data from each article was extracted using a standardized table format developed apriori by the research team. The data extracted comprised the key article details (e.g., author, year and country), patient characteristics, content of the relapse prevention interventions/resources described following GSH and any relevant outcome data (where available).

Quality Appraisal

A quality appraisal of all included studies was carried out to provide an overview of the quality of the evidence. A scoping review does not seek to assess the quality of the evidence (Arksey & O’Malley, 2005 ; Pham et al., 2014 ) but rather provide an overview of all existing evidence related to a certain topic (Tricco et al., 2016 ). For this reason, the included articles will not be excluded irrespective of the results from the quality appraisal or allocated different weight in the results section.

The quality of the individual studies was assessed using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018 ) and evaluated against the appropriate methodological quality criteria pending on study design. This tool was chosen as it allows the appraisal of empirical studies across different study designs including qualitative, quantitative randomised controlled trials, quantitative non-randomised controlled trials, quantitative descriptive and mixed methods (Hong et al., 2018 ). Two authors (SN and HD) rated each criterion for each study and any disagreement over ratings between reviewers were resolved through discussion and consensus.

Synthesis of Results

Results were synthesised using a narrative descriptive synthesis following the guidance by Popay et al. ( 2006 ) due to the variation in type of studies included, allowing for the investigation of any similarities or differences between studies to provide a summary of the knowledge available (Lisy & Porritt, 2016 ). The synthesis provided a descriptive summary of the findings across the included studies, consistent with an integrative approach to synthesising data. The relationship between and across studies was explored to (i) identify any similarities or discrepancies across the studies and (ii) identify the key characteristics of studies describing resources/interventions to prevent relapse following GSH.

Electronic Databases

The search strategy identified 954 records. These records were exported to Endnote for easy management and after removing 467 duplicates across databases automatically using EndNote, 511 records were screened for eligibility at title and abstract, from which 28 were reviewed at full text. Two papers screened at full text were review articles (Moriarty et al., 2020 ; Tokgöz et al., 2021 ) and their reference lists were hand searched by two independent reviewers (SN and HD) identifying 14 additional papers which were subject to screening at title and abstract. Three of the 28 papers, identified via the electronic databases, and screened at full text were book reviews (Kannis, 2010 ; Rosenquist, 2012 ; Watt, 2006 ) and the references of these reviews resulted in the inclusion of three additional papers subject to screening at title and abstract. Of these 17 additional papers identified via citation searching, three were reviewed at full text. Figure 1 illustrates the study selection procedure using a PRISMA-ScR flow diagram and includes the reasons for exclusion at different stages of the scoping review. A total of 26 papers identified via electronic databases were screened at full text and five authors were contacted for further information regarding six different papers (Biesheuvel-Leliefeld et al., 2017 ; Bockting et al., 2011 ; Krijnen-de Bruin et al., 2019 , 2022 ; Lucock et al., 2018 ; Malins et al., 2020 ). Of the five authors contacted two responded providing clarification regarding sample population characteristics, and whether the treatment received prior to the relapse prevention intervention was GSH. For the narrative synthesis, three peer-reviewed publications were included, and their data was extracted and presented in Table 1 . The quality of the three published articles was assessed and findings are presented in Table 2 . All three publications met at least 60% or more of the MMAT criteria.

Adapted from Page et al. ( 2021 ). PRISMA diagram detailing screening process

Other Resources

The first search strategy for identifying evidence via different resources involved searching the first ten pages of results equating to 100 resources at each search time point. Through a manual search by SN, this google search strategy resulted in six hits. The majority of results that were excluded comprised publications that had been reviewed from the electronic databases search or did not meet inclusion criteria upon the initial scope of the title and accompanying description, websites detailing self-management strategies and information regarding GSH but not specifically focussing on relapse prevention, and manuals or guides related to the treatment of anxiety and/or depression. Upon further exploration of the six hits, four results were excluded as they did not focus on relapse prevention following guided self-help. The second search strategy, via the grey literature database, resulted in 102 hits. However, upon screening using the methods above and inclusion criteria, none of the results were relevant to this scoping review and all were excluded. An exploration of the grey literature database used and additional databases that may have been appropriate but were inaccessible is listed in the Appendix. Through consultation with two experts, the third strategy resulted in two pieces of grey literature – a webinar and an app. Both items were further explored through a targeted search on google to identify any other accompanying resources and through consultation with experts associated with the relapse prevention app. This resulted in access to two unpublished reports, facilitated by members of the team involved in the relapse prevention app, which described the development of the app and a workbook pdf of the document described in the webinar. However, the unpublished reports surrounding the app focused on its development not implementation and thus are not included in this review. Instead, information regarding the app is limited to everything listed on the website. Therefore, three items identified via different resources were included in the scoping review.

The characteristics of the included studies and resources are presented in Table 2 . From the electronic databases, three published peer-reviewed studies were identified (Wright et al., 2000 ; Lucock et al., 2018 , 2021 ) and three other resources were identified through the grey literature (Chellingsworth et al., 2013 ; Paddle, 2019 ; OxCADAT, n.d. ). The three peer-reviewed publications, CEDAR relapse prevention booklet (Chellingsworth et al., 2013 ) and the Paddle app ( 2019 ) were published online and developed between 2000 and 2021. Information regarding the year of development for the staying well resources (OxCADAT, n.d. ) was not available. One study (Wright et al., 2000 ) was conducted in the United States whereas the two peer-reviewed publications and three other resources were conducted and produced in the UK. The three published studies examined a total of 164 different participants between the ages of 19–83 years. One study (Wright et al., 2000 ) explored panic disorder, two studies (Lucock et al., 2018 , 2021 ) focussed on depression and the additional resources did not specify whether they focussed on anxiety or depression alone or both. Two studies exploring the Self-Management after Therapy (SMArT) intervention (Lucock et al., 2018 , 2021 ) reported qualitative findings derived from interviews.

Relapse Prevention Description

The study by Wright et al. ( 2000 ) focussed on adults who had formerly received bibliotherapy for panic attacks. It examined the effect of a relapse prevention programme that utilised a seven-chapter manual for participants to practice independently for six months. The program comprised information about the booklet, hypothetical scenarios, examples of multiple relapse prevention techniques and exercises. Participants were encouraged to focus on the most relevant techniques/exercises and received brief telephone calls across six months to enhance treatment compliance.

The SMArT (Self-Management after Therapy) intervention described in both publications by Lucock et al. ( 2018 , 2021 ) explored the same relapse prevention intervention for depression but with different participants. It involved creating up to five different implementation intentions unique to each participant, decided collaboratively with a psychological wellbeing practitioner during a face-to-face session up to 4 weeks following discharge. Using' if-then' statements, these implementation intentions linked any situation (i.e., an internal or external cue) to a response, such as a behaviour, feeling or cognition, and accompanied any relapse prevention plan discussed during therapy. Additionally, the intervention involved using diary sheets to monitor the usage of implementation intentions and any issues that occurred. Participants also received three monthly telephone support calls from a practitioner following end of treatment.

The Staying Well Workbook (OxCADAT, n.d. ) is a resource for use within and beyond low-intensity therapy to enable patients to record information during therapy sessions, to prevent relapse. It consists of seven chapters across nine pages, and encourages patients to reflect on their therapeutic journey. This includes differentiating between lapse and relapse, reviewing techniques used during treatment, creating if-then plans for early warning signs, monitoring and signposting for further support. The therapist and patient collaboratively, create if-then plans and the patient is encouraged to continue monitoring change and referring to this workbook following end of treatment. There is also a guide for professionals on using this workbook, including information on scheduling a follow-up session.

The paddle therapy support app (Paddle, 2019 ) was designed to help patients stay well after treatment by storing therapy-related information for easy access. It allows users to upload workbooks, record session details, complete questionnaires, visualise progress, create a resource library, and add emergency information. The app is accessible online and offline.

The relapse prevention workbook, known as CEDAR, was designed by Chellingsworth et al. ( 2013 ) and comprises eight chapters with information on how to differentiate between a lapse and relapse, identifying early warning signs, reflecting on progress, rating goals, well-being action plans, areas for improvement and further support resources. It also includes information about activities for patients to refer to during difficult times.

Across all these relapse prevention interventions and resources, participants were asked to refer to a manual or workbook consistently following the completion of GSH to prevent relapse and practice various techniques independently. A checklist of the key components is summarised in Table 3 .

Qualitative Findings

Qualitative feedback regarding the SMArT intervention was also collected for both studies (Lucock et al., 2018 , 2021 ) through a brief telephone call after completing the relapse prevention intervention. The first study (Lucock et al., 2018 ) reported feedback received via telephone from six patients who completed the intervention. The supplementary material highlighted eight themes from the data surrounding experiences of different aspects of the intervention and the difficulties in implementing their relapse prevention plans. This includes a patient reporting that they were not inclined to implement their plans due to a lack of connection with the practitioner delivering the intervention and it was also reported that difficulties in carrying out their relapse prevention plans were influenced by external factors which were not specified. The second study (Lucock et al., 2021 ) included 16 semi-structured interviews following the end of the intervention and identified seven themes– four surrounding relapse prevention and self-management, one concerning the delivery of the SMArT intervention and two themes relating to the suitability of the intervention for patients and services. For both studies, patients reported that contact with the practitioner via follow-ups provided reassurance and was useful for continued support. Features described as helpful by patients in both studies included having an initial face-to-face session to develop an implementation plan, which was seen as necessary in building the therapeutic relationship and the social support received from sharing plans with family and friends. Both studies interviewed practitioners delivering low intensity interventions, who reported that the interventions aligned with their role and training. The second study (Lucock et al., 2021 ) also reported that some challenges expressed by practitioners included increasing awareness of the intervention to other staff and incorporating it into the service demands. The qualitative results across both studies highlighted how patients viewed the inclusion of follow-ups as positively influencing their recovery by providing a safety net following discharge and allowing patients to consolidate their learning in practice.

The relapse prevention interventions described across the three peer-reviewed publications (Lucock et al., 2018 , 2021 ; Wright et al., 2000 ) all included monthly telephone follow-up calls. The guidance accompanying the staying well booklet (OxCADAT, n.d. ) also included one follow-up; however, it was not specified whether this was via telephone or face-to-face. The duration of these follow-up sessions varied with the study by Wright et al. ( 2000 ), limiting the telephone calls to a maximum of 15 min, whereas the follow-up telephone calls described in both papers by Lucock et al. ( 2018 , 2021 ) were each 30 min long. The Staying Well booklet (OxCADAT, n.d. ) did not include any details regarding the length of the follow-up. The purpose of the follow-up sessions across these four relapse prevention interventions/tools was to encourage compliance and provide social support if needed. Excluding the Staying Well booklet (OxCADAT, n.d. ), the other three relapse prevention interventions used follow-ups to assess the effectiveness of the intervention and collect outcome measure data.

Measures of Effectiveness

The three peer-reviewed publications (Lucock et al., 2018 ; 2021 ; Wright et al., 2000 ) tested the effectiveness of their interventions. Whereas data concerning the effectiveness of the relapse prevention intervention across the grey literature documents (OxCADAT, n.d. ; Chellingsworth et al., 2013 ) was not reported or has yet to be collected (Paddle, 2019 ).

The effectiveness of the SMArT intervention (Lucock et al., 2018 , 2021 ) was measured by the Patient Health Questionnaire-9 (PHQ-9; Kroenke et al., 2001 ) and Generalized Anxiety Disorder-7 (GAD; Spitzer et al., 2006 ) which was collected at six different time points, including before therapy, at discharge, during the face-to-face session and at each of the three telephone follow-up sessions.

To test the effectiveness of the relapse prevention manual, Wright et al. ( 2000 ) compared the frequency of full and limited-symptom panic attacks of the relapse prevention group with a waiting list control group.

Within the ever-evolving field of clinical psychology, this is the first comprehensive summary of the relapse prevention resources and interventions administered to patients after completing a GSH intervention in primary care. This scoping review provides an overview of the knowledge available, and the key features considered in developing a resource to tackle the global concern of patients with anxiety and depression relapsing following treatment. Additionally, essential gaps in the literature were identified predominantly surrounding the scarcity of relapse prevention interventions or tools specifically for individuals who have completed low-intensity treatments. To provide a comprehensive overview of the advances and gaps in the literature surrounding relapse prevention following GSH, this section will discuss the main findings in accordance with the Patterns, Advances, Gaps, Evidence of practice, and Research Recommendations (PAGER) framework (Bradbury-Jones et al., 2021 , 2022 ). The PAGER framework (Table 4 ) is used to guide the discussion and planning for future research and uses model themes (Bradbury-Jones et al., 2021 ).

The importance of additional support after GSH - Incorporating an additional element following the end of treatment that focuses primarily on relapse prevention would be beneficial to ensuring patients can maintain their recovery for a more extended period. The three peer-reviewed studies reporting on the evaluation of their interventions have shown to be effective at reducing the risk of relapse for depression symptoms (Lucock et al., 2018 , 2021 ) or anxiety symptoms (Wright et al., 2000 ). This suggests that interventions supporting patients following GSH can maximise long term benefits of treatment. These findings corroborate a previous systematic review exploring how interference using a range of psychological therapies following recovery from depression can help reduce the risk of relapse (Clarke et al., 2015 ). In another systematic review (Moriarty et al., 2020 ) findings demonstrate that relapse prevention interventions are highly effective at reducing the risk of relapse for patients with depression; however, both reviews do not focus on administering these interventions following GSH treatment. Nonetheless, findings from this scoping review look promising due to the effects reported from two relapse prevention interventions for patients with anxiety or depression treated in short-term, low-intensity care. Although this review does not report on the effects of the intervention, adding a relapse prevention component following the treatment phase has beneficial properties for reducing the likelihood of relapse as patients continue to feel supported following discharge. However, due to the scarcity of evidence highlighted in this review and the restricted evidence evaluating their impact in reducing long-term relapse, further research is needed before concrete conclusions regarding the effectiveness of relapse prevention interventions can be drawn.

Providing accessible tools and resources - Despite the limited evidence discussed in this scoping review, many commonalities existed. To aid relapse prevention, an integral feature across all tools and resources was helping individuals to develop the skills necessary to overcome and prevent any incidence of relapse, such that patients become independent in their recovery journey. The process of developing and implementing skills to address situations that could result in relapse is a key principle of relapse prevention (Marlatt & George, 1984 ; Melemis, 2015 ; Menon & Kandasamy, 2018 ), which was addressed across all studies and identified resources. The development and ongoing use of skills are common strategies in materials created to help overcome depression and anxiety. This is evident in workbooks available to the public, which can be utilized by anyone, whether they have received professional help, to support their recovery journey (Williams, 2012 ) and in practitioner guides for conducting relapse prevention work in low-intensity care (Papworth & Marrinan, 2018 ). Regardless of whether these skills were developed in collaboration with a therapist at the start of treatment or at the start of the relapse prevention intervention, or through a more generalised plan, they remained helpful to maintain recovery over time. Most importantly, ensuring that the skills are recorded in an accessible format – written or digital – for access outside of the therapy session without the aid of a therapist is an important feature evidenced across all the resources listed in this review, supporting relapse prevention. Having access to written information that is tailored to an individual’s problems has formally been identified as helpful in enabling self-management across various conditions (Dineen-Griffin et al., 2019 ; Liddy et al., 2014 ). By ensuring individuals have the appropriate resources to help them independently manage their condition, the immediate benefits of recovery following GSH can be maintained. However, it is important to note that the lack of information about the development of the relapse prevention interventions deters understanding of the development process. In addition, the benefits and challenges of different intervention development approaches in this topic is scarce. Future research which explores the development of relapse prevention interventions should consider the involvement of multiple stakeholders and report each step in the development of the intervention to ensure transparency and improve understanding of how best it is to create relapse prevention interventions and resources.

Continued brief support after treatment - A follow-up was also an integral feature for relapse prevention as this component was included in five of six relapse prevention tools and interventions. To ensure that patients continuously apply the skills learnt during their GSH treatment outside of the clinical setting, checking in with patients for a brief amount of time reinforces the implementation of the relapse prevention tool or resource and enables the therapist to track their progress or identify any early signs of relapse before they develop into something bigger (National Collaborating Centre for Mental Health, 2018 ). Having additional contact with a practitioner, regardless of whether an individual relapses following treatment, is a preferred feature of relapse prevention programmes influencing the rate of engagement with the programme as evidenced in previous research (Muntingh et al., 2019 ; Scholten et al., 2021 ). Follow-ups are also recommended for low-intensity treatment to ensure patients are maintaining treatment gains, continue to implement their techniques and identify signs of relapse early (Papworth & Marrinan, 2018 ; NHS, 2024 ). However, continuous contact with a practitioner is often not feasible in routine care in the long term and previous research has demonstrated that motivation can influence the continuity of treatment (Runge et al., 2022 ). Therefore, future research should explore patient motivation to increase engagement and ensure sustained positive effects of therapy.

Strengths and Limitations

To our knowledge, this is the first scoping review to explore relapse prevention interventions explicitly administered following a specific type of psychological treatment, i.e., GSH. The publication of a review protocol accessible on a publicly available research data repository ensures transparency of the research design and demonstrates whether the findings of the scoping review align with the researchers’ objectives set out at the start, which can be seen as a strength of this review.

This review addresses relapse prevention for individuals with anxiety and/or depression symptoms after GSH through the inclusion of peer reviewed publications and other resources. However, there are limitations that warrant discussion. Firstly, our search strategy was not exhaustive, and the search string used was narrow, focussing on the most relevant terms to address the aim of this scoping review considering the time constraints, which may have meant that the papers extracted through the reference lists did not appear during the electronic database search, as a result. Nonetheless the inclusion of reference checking supported the electronic database search, strengthening the findings of this scoping review. Secondly, despite the inclusion of evidence beyond what is published in the literature, the use of a singular grey literature database limited the comprehensiveness of the search and the use of a database, which yielded no results suggests that the chosen database may not have been appropriate for this scoping review. However, access to other grey literature databases was restricted; thus, future reviews may consider including multiple grey literature databases. Finally, it is crucial to consider that the inclusion/exclusion criteria focussed on GSH, which, for many countries such as the Netherlands, is not commonly administered, unlike within England, where it is often used within talking therapy services, which may be why the majority of the includes studies/resources are from England. Here, it is essential to note that although this review explores relapse after treatment, discussions regarding relapse prevention may occur during and throughout GSH treatment by PWPs in step 2 care (Papworth & Marrinan, 2018 ), and the resources or tools focused on relapse utilised during treatment may differ from those discussed (as evidenced by the Staying Well Resource (OxCADAT, n.d. ) which is used within treatment and beyond). Nonetheless, this review focussed on identifying what is available for patients when they are no longer in treatment and do not have access to the support of their practitioners instead of understanding the relapse prevention discussions that may occur during GSH.

This scoping review sought to provide insight into ways in which relapse prevention is approached following GSH and revealed a shortage of resources and interventions specifically tailored to target relapse prevention among people with anxiety and/or depression symptoms who received GSH. Nonetheless, many commonalities were observed in the characteristics of available resources including continued utilisation of learnt techniques and further development on acquired skills beyond GSH, alongside follow-ups to enhance compliance with relapse prevention interventions. However, further investigation is warranted to assess their effectiveness at reducing relapse in the long-term. Providing additional interventions and resources following treatment aimed to prevent relapse may be integral in facilitating long-term positive change in mental health by supplying individuals with all the necessary tools to independently continue their recovery journey. However, further research is still needed to inform how relapse prevention can be best and efficiently addressed following GSH treatment to ensure the skills learnt during GSH transition to lifelong skills.

Data Availability

Publicly available datasets were analysed in this study. The author confirms that all data generated or analysed during this study are included in this manuscript

Adams, J., Hillier-Brown, F. C., Moore, H. J., Lake, A. A., Araujo-Soares, V., White, M., & Summerbell, C. (2016). Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: Critical reflections on three case studies. Systematic Reviews , 5 (1), 1–11.

Article Google Scholar

Ali, S., Rhodes, L., Moreea, O., McMillan, D., Gilbody, S., Leach, C., Lutz, W., & Delgadillo, J. (2017). How durable is the effect of low intensity CBT for depression and anxiety? Remission and relapse in a longitudinal cohort study. Behaviour Research and Therapy , 94 , 1–8. https://doi.org/10.1016/j.brat.2017.04.006

Article PubMed Google Scholar

Alonso, J., Liu, Z., Evans-Lacko, S., Sadikova, E., Sampson, N., Chatterji, S., & WHO World Mental Health Survey Collaborators. (2018). Treatment gap for anxiety disorders is global: Results of the World Mental Health Surveys in 21 countries. Depression and Anxiety , 35 (3), 195–208. https://doi.org/10.1002/da.22711

Article PubMed PubMed Central Google Scholar

Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology , 8 (1), 19–32. https://doi.org/10.1080/1364557032000119616

Auguer, P. (1998). Information sources in grey literature. De Gruyter . https://doi.org/10.1515/9783110977233 BOOK 4TH ED.

Backenstrass, M., Schwarz, T., Fiedler, P., Joest, K., Reck, C., Mundt, C., & Kronmueller, K. T. (2006). Negative mood regulation expectancies, self-efficacy beliefs, and locus of control orientation: Moderators or mediators of change in the treatment of depression? Psychotherapy Research , 16 (02), 250–258. https://doi.org/10.1080/10503300500485474

Baguley, C., Farrand, P., Hope, R., Leibowitz, J., Lovell, K., Lucock, M., O’Neill, C., Paxton, R., Pilling, S., Richards, D., Turpin, G., White, J., & Williams, C. (2010, March). Good practice guidance on the use of self-help materials within IAPT services . http://eprints.hud.ac.uk/id/eprint/9017/

Biesheuvel-Leliefeld, K. E., Dijkstra-Kersten, S. M., Van Schaik, D. J., Van Marwijk, H. W., Smit, F., Van Der Horst, H. E., & Bockting, C. L. (2017). Effectiveness of supported self-help in recurrent depression: A randomized controlled trial in primary care. Psychotherapy and Psychosomatics , 86 (4), 220–230. https://doi.org/10.1159/000472260

Bockting, C. L., Hollon, S. D., Jarrett, R. B., Kuyken, W., & Dobson, K. (2015). A lifetime approach to major depressive disorder: The contributions of psychological interventions in preventing relapse and recurrence. Clinical Psychology Review , 41 , 16–26. https://doi.org/10.1016/j.cpr.2015.02.003

Bockting, C. L., Kok, G. D., Kamp, L. V. D., Smit, F., van Valen, E., Schoevers, R., Van Marwijk, H., Cuijpers, P., Riper, H., Dekker, J., & Beck, A. T. (2011). Disrupting the rhythm of depression using Mobile Cognitive Therapy for recurrent depression: Randomized controlled trial design and protocol. Bmc Psychiatry , 11 (1), 1–9. https://doi.org/10.1186/1471-244X-11-12

Bradbury-Jones, C., Aveyard, H., Herber, O. R., Isham, L., Taylor, J., & O’malley, L. (2022). Scoping reviews: The PAGER framework for improving the quality of reporting. International Journal of Social Research Methodology , 25 (4), 457–470. https://doi.org/10.1080/13645579.2021.1899596

Bradbury-Jones, C., Isham, L., Morris, A. J., & Taylor, J. (2021). The neglected relationship between child maltreatment and oral health? An international scoping review of research. Trauma Violence & Abuse , 22 (2), 265–276. https://doi.org/10.1177/152483801984159

Brandon, T. H., Vidrine, J. I., & Litvin, E. B. (2007). Relapse and relapse prevention. Annual Review of Clinical Psychology , 3 , 257–284. https://doi.org/10.1146/annurev.clinpsy.3.022806.091455

Burcusa, S. L., & Lacono, W. G. (2007). Risk for recurrence in depression. Clinical Psychology Review , 27 (8), 959–985. https://doi.org/10.1016/j.cpr.2007.02.005

Chellingsworth, M., Farrand, P., & Small, F. (2013). Relapse prevention toolkit recovery guide CEDAR. https://www.elft.nhs.uk/sites/default/files/2022-05/relapse-prevention.pdf

Clark, D. M. (2018). Realizing the mass public benefit of evidence-based psychological therapies: the IAPT program. Annual Review of Clinical Psychology , 14 (1), 159–183. https://doi.org/10.1146/annurev-clinpsy-050817-084833

Clark, D. M., Layard, R., Smithies, R., Rochards, D. A., Suckling, R., & Wright, B. (2009). Improving access to psychological therapy: Initial evaluation of two UK demonstration sites. Behaviour Research and Therapy , 47 (11), 910–920. https://doi.org/10.1016/j.brat.2009.07.010

Clarke, K., Mayo-Wilson, E., Kenny, J., & Pilling, S. (2015). Can non-pharmacological interventions prevent relapse in adults who have recovered from depression? A systematic review and meta-analysis of randomised controlled trials. Clinical Psychology Review , 39 , 58–70. https://doi.org/10.1016/j.cpr.2015.04.002

Cooper, H., Hedges, L. V., & Valentine, J. C. (2009). The Handbook of Research Synthesis and Meta-Analysis (2nd ed.). Russell Sage Foundation.

Coull, G., & Morris, P. G. (2011). The clinical effectiveness of CBT-based guided self-help interventions for anxiety and depressive disorders: A systematic review. Psychological Medicine , 41 (11), 2239–2252. https://doi.org/10.1017/s0033291711000900

Article PubMed CAS Google Scholar

Cuijpers, P., Donker, T., van Straten, A., Li, J., & Andersson, G. (2010). Is guided self-help as effective as face-to-face psychotherapy for depression and anxiety disorders? A systematic review and meta-analysis of comparative outcome studies. Psychological Medicine , 40 (12), 1943–1957. https://doi.org/10.1017/S0033291710000772

Delgadillo, J., Rhodes, L., Moreea, O., McMillan, D., Gilbody, S., Leach, C., Lucock, M., Lutz, W., & Ali, S. (2018). Relapse and recurrence of common mental health problems after low intensity cognitive behavioural therapy: The WYLOW longitudinal cohort study. Psychotherapy and Psychosomatics , 87 (2), 116–117. https://doi.org/10.1159/000485386

Dineen-Griffin, S., Garcia-Cardenas, V., Williams, K., & Benrimoj, S. I. (2019). Helping patients help themselves: A systematic review of self-management support strategies in primary health care practice. PloS One , 14 (8), e0220116. https://doi.org/10.1371/journal.pone.0220116

Article PubMed PubMed Central CAS Google Scholar

Etzelmueller, A., Vis, C., Karyotaki, E., Baumeister, H., Titov, N., Berking, M., Cuijpers, P., Riper, H., & Ebert, D. D. (2020). Effects of internet-based cognitive behavioral therapy in routine care for adults in treatment for depression and anxiety: Systematic review and meta-analysis. Journal of Medical Internet Research , 22 (8), e18100. https://doi.org/10.2196/18100

Falbe-Hansen, L., Le Huray, C., Phull, B., Shakespeare, C., & Wheatley, J. (2009). Using guided self-help to treat common mental health problems: The Westminster Primary Care Psychology Service. London Journal of Primary Care , 2 (1), 61–64. https://doi.org/10.1080/17571472.2009.11493246

Ferrari, A. J., Charlson, F. J., Norman, R. E., Patten, S. B., Freedman, G., Murray, C. J., Vos, T., & Whiteford, H. A. (2013). Burden of depressive disorders by country, sex, age, and year: Findings from the global burden of disease study 2010. PLoS Medicine , 10 (11), e1001547. https://doi.org/10.1371/journal.pmed.1001547

Gellatly, J., Bower, P., Hennessy, S. U. E., Richards, D., Gilbody, S., & Lovell, K. (2007). What makes self-help interventions effective in the management of depressive symptoms? Meta-analysis and meta-regression. Psychological Medicine , 37 (9), 1217–1228. https://doi.org/10.1017/S0033291707000062

Godin, K., Stapleton, J., Kirkpatrick, S. I., Hanning, R. M., & Leatherdale, S. T. (2015). Applying systematic review search methods to the grey literature: A case study examining guidelines for school-based breakfast programs in Canada. Systematic Reviews , 4 (1), 1–10. https://doi.org/10.1186/s13643-015-0125-0

Gunnarsson, A. B., Hedberg, A. K., Håkansson, C., Hedin, K., & Wagman, P. (2021). Occupational performance problems in people with depression and anxiety. Scandinavian Journal of Occupational Therapy , 30 (2), 148–158. https://doi.org/10.1080/11038128.2021.1882562

Health Education England (2023, July 26). Grey Literature . NHS England. https://library.hee.nhs.uk/learning-academy/learning-zone/grey-literature

Hendriks, S. M., Spijker, J., Licht, C. M., Hardeveld, F., de Graaf, R., Batelaan, N. M., Penninx, B. W. J. H., & Beekman, A. T. (2015). Long-term work disability and absenteeism in anxiety and depressive disorders. Journal of Affective Disorders , 178 , 121–130. https://doi.org/10.1016/j.jad.2015.03.004

Higgins, J. P. T., & Green, S. (2011). Cochrane handbook for systematic reviews o interventions . Wiley-Blackwell.

Hofmann, S. G., & Gómez, A. F. (2017). Mindfulness-based interventions for anxiety and depression. Psychiatric Clinics , 40 (4), 739–749. https://doi.org/10.1016/j.psc.2017.08.008

Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M., Vedel, I., & Pluye, P. (2018). The mixed methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Education for Information , 34 (4), 285–291. https://doi.org/10.3233/EFI-180221

Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007). Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database of Systematic Reviews . https://doi.org/10.1002/14651858.MR000010.pub3 . 2.

https://doi.org/10.1371/journal.pone.0219588

Kannis, L. (2010, February 15). The Anti-Anxiety Workbook: Proven Strategies to Overcome Worry, Phobias, Panic, and Obsessions [Review of the book The Anti-Anxiety Workbook: Proven Strategies to Overcome Worry, Phobias, Panic, and Obsessions , by P. J. Norton., & M. M. Antony]. https://www.cambridge.org/core/journals/behavioural-and-cognitive-psychotherapy/article/antianxiety-workbook-proven-strategies-to-overcome-worry-phobias-panic-and-obsessions-antony-martin-and-norton-peter-new-york-the-guilford-press-2009-pp-262-1350-pb-isbn-9781593859930/824D39B694899E84BD4EC083F06B1FE6

Krijnen-de Bruin, E., Muntingh, A. D., Bourguignon, E. M., Hoogendoorn, A., Maarsingh, O. R., van Balkom, A. J., Batelaan, N. M., van Straten, A., & van Meijel, B. (2022). Usage intensity of a relapse prevention program and its relation to symptom severity in remitted patients with anxiety and depression: Pre-post study. JMIR Mental Health , 9 (3), e25441. https://doi.org/10.2196/25441

Krijnen-de Bruin, E., Muntingh, A. D., Hoogendoorn, A. W., van Straten, A., Batelaan, N. M., Maarsingh, O. R., van Balkom, A. J. L. M., & van Meijel, B. (2019). The GET READY relapse prevention programme for anxiety and depression: A mixed-methods study protocol. Bmc Psychiatry , 19 (1), 1–11. https://doi.org/10.1186/s12888-019-2034-6

Kroenke, K., Spitze, R. L., & Williams, J. B. (2001). The PHQ-9: validity of a brief depression severity measure. Journal of General Internal Medicine,16 (9): 606 – 13. https://doi.org/10.1046/j.1525-1497.2001.016009606.x

Lewis, C., Pearce, J., & Bisson, J. I. (2012). Efficacy, cost-effectiveness and acceptability of self-help interventions for anxiety disorders: Systematic review. The British Journal of Psychiatry , 200 (1), 15–21. https://doi.org/10.1192/bjp.bp.110.084756

Liddy, C., Blazkho, V., & Mill, K. (2014). Challenges of self-management when living with multiple chronic conditions: Systematic review of the qualitative literature. Canadian Family Physician , 60 (12), 1123–1133.

PubMed PubMed Central Google Scholar

Lisy, K., & Porritt, K. (2016). Narrative synthesis: Considerations and challenges. JBI Evidence Implementation , 14 (4), 201. https://doi.org/10.1097/01.XEB.0000511348.97198.8c

Lockwood, C., Dos Santos, K. B., & Pap, R. (2019). Practical guidance for knowledge synthesis: Scoping review methods. Asian Nursing Research , 13 (5), 287–294. https://doi.org/10.1016/j.anr.2019.11.002

Lorimer, B., Kellett, S., Nye, A., & Delgadillo, J. (2021). Predictors of relapse and recurrence following cognitive behavioural therapy for anxiety-related disorders: A systematic review. Cognitive Behaviour Therapy , 50 (1), 1–18. https://doi.org/10.1080/16506073.2020.1812709

Lucock, M., Bartys, S., Cupac, J., Delgadillo, J., Denton, C., Gaines, S., McMillan, D., Prestwich, A., & Stebbings, R. (2018). Using implementation intentions to prevent relapse after psychological treatment for depression–the SMArT intervention. Behavioural and Cognitive Psychotherapy , 46 (5), 626–632. https://doi.org/10.1017/S1352465818000255

Lucock, M., Borthwick, R., Cupac, J., Elliott, R., Howell, R., Kendal, S., Khan, W., Sandford, D., & Tolley, B. (2021). Using implementation intentions to prevent relapse after remission from psychological treatment for depression: The SMArT intervention. Psychotherapy Research , 32 (4), 428–439. https://doi.org/10.1080/10503307.2021.1959079

Malins, S., Biswas, S., Patel, S., Levene, J., Moghaddam, N., & Morriss, R. (2020). Preventing relapse with personalized smart-messaging after cognitive behavioural therapy: A proof‐of‐concept evaluation. British Journal of Clinical Psychology , 59 (2), 241–259. https://doi.org/10.1111/bjc.12244

Marlatt, G. A., & George, W. H. (1984). Relapse prevention: Introduction and overview of the model. British Journal of Addiction , 79 (3), 261–273. https://doi.org/10.1111/j.1360-0443.1984.tb00274.x

McDaid, D., & Park, A. (2022, February). The economic case for investing in the prevention of mental health conditions in the UK . Mental Health Foundation . https://www.mentalhealth.org.uk/sites/default/files/2022-06/MHF-Investing-in-Prevention-Report-Summary.pdf

McDevitt-Petrovic, O. (2019). Improving Access and Treatment Effectiveness Using Low Intensity Cognitive Behavioural Therapy for Mild to Moderate Level Mental Health Disorders, Including Medically Unexplained Symptoms, in Northern Ireland (Doctoral dissertation, Ulster University).

McHugh, R. K., Whitton, S. W., Peckham, A. D., Welge, J. A., & Otto, M. W. (2013). Patient preference for psychological vs pharmacologic treatment of psychiatric disorders: A meta-analytic review. The Journal of Clinical Psychiatry , 74 (6), 595–602. https://doi.org/10.4088/jcp.12r07757

Melemis, S. M. (2015). Focus: Addiction: Relapse prevention and the five rules of recovery. The Yale Journal of Biology and Medicine , 88 (3), 325.

Menon, J., & Kandasamy, A. (2018). Relapse prevention. Indian Journal of Psychiatry , 60 (Suppl 4), S473. https://doi.org/10.4103/psychiatry.indianjpsychiatry_36_18

Mental Health Foundation. (2016). Fundamental facts about mental health 2016 . Mental Health Foundation. https://www.mentalhealth.org.uk/sites/default/files/2022-06/The-Fundamental-facts-about-mental-health-2016.pdf

Moriarty, A. S., Coventry, P. A., Hudson, J. L., Cook, N., Fenton, O. J., Bower, P., Lowell, K., Archer, J., Clarke, R., Richards, D. A., Dickens, C., Gask, L., Waheed, W., Huijbregts, K. M., van der Feltz-Cornelis, C., Ali, S., Gilbody, S., & McMillan, D. (2020). The role of relapse prevention for depression in collaborative care: A systematic review. Journal of Affective Disorders , 265 , 618–644. https://doi.org/10.1016/j.jad.2019.11.105

Munn, Z., Peters, M. D., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology , 18 (143), 1–7. https://doi.org/10.1186/s12874-018-0611-x

Munn, Z., Pollock, D., Khalil, H., Alexander, L., Mclnerney, P., Godfrey, C. M., Peters M., & Tricco, A. C. (2022). What are scoping reviews? Providing a formal definition of scoping reviews as a type of evidence synthesis. JBI Evidence Synthesis , 20(4), 950–952. https://doi.org/10.11124/JBIES-21-00483

Muntingh, A. D., Hoogendoorn, A. W., Van Schaik, D. J., Van Straten, A., Stolk, E. A., Van Balkom, A. J., & Batelaan, N. M. (2019). Patient preferences for a guided self-help programme to prevent relapse in anxiety or depression: A discrete choice experiment. PloS One , 14 (7), e0219588.

National Collaborating Centre for Mental Health (2018, June). The improving access to psychological therapies manual . NHS . https://www.england.nhs.uk/wp-content/uploads/2018/06/the-nhs-talking-therapies-manual-v6.pdf

NHS (2022, August 11). Psychological Therapies: reports on the use of IAPT services, England, May 2022 Final including a report on the IAPT Employment Advisers pilot NHS England. https://digital.nhs.uk/data-and-information/publications/statistical/psychological-therapies-report-on-the-use-of-iapt-services/may-2022-final-including-a-report-on-the-iapt-employment-advisers-pilot/outcomes

NHS (2024, March). NHS Talking Therapies for anxiety and depression Manual NHS. https://www.england.nhs.uk/wp-content/uploads/2018/06/NHS-talking-therapies-manual-version-7.pdf

NICE (2011, May 25). Common mental health problems: identification and pathways to care National Institute for Health and Care Excellence. https://www.nice.org.uk/guidance/cg123/chapter/recommendations

NICE (2014, February 06). Anxiety disorders . National Institute for Health and Care Excellence. https://www.nice.org.uk/guidance/qs53/chapter/quality-statement-2-psychological-interventions#:~:text=Rationale,in%20preference%20to%20pharmacological%20treatment

NICE (2022a, June 29). Depression in adults: treatment and management National Institute of Heath and Care Excellence. https://www.nice.org.uk/guidance/ng222/chapter/Recommendations#choice-of-treatments

NICE (2022b). Depression in adults [C] Preventing relapse. National Institute for Health and Care Excellence. https://www.nice.org.uk/guidance/ng222/evidence/c-preventing-relapse-pdf-11131004416

NICE (2023, December 14). Digitally enabled therapies for adults with anxiety disorders: early value assessment National Institute for Health and Care Excellence. https://www.nice.org.uk/guidance/hte9/chapter/2-The-technology

OxCADAT (n.d.). Step 2 staying well materials. OxCADAT. https://oxcadatresources.com/step-2-staying-well-materials/

Paddle (2019). Paddle App. https://www.paddleapp.org/

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, L. A., Stewart, L. A., Thomas, J., Tricco, A. C., Welch, V. A., Whiting, P., & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. International Journal of Surgery , 88 , 105906. https://doi.org/10.1016/j.ijsu.2021.105906

Palacios, J. E., Enrique, A., Mooney, O., Farrell, S., Earley, C., Duffy, D., Eilert, N., Harty, S., Timulak, L., & Richards, D. (2022). Durability of treatment effects following internet-delivered cognitive behavioural therapy for depression and anxiety delivered within a routine care setting. Clinical Psychology & Psychotherapy , 29 (5), 1768–1777. https://doi.org/10.1002/cpp.2743

Papworth, M., & Marrinan, T. (2018). Low Intensity Cognitive Behaviour Therapy: A Practitioner’s Guide (2nd ed.). SAGE.

Pham, M. T., Rajić, A., Greig, J. D., Sargeant, J. M., Papadopoulos, A., & McEwen, S. A. (2014). A scoping review of scoping reviews: Advancing the approach and enhancing the consistency. Research Synthesis Methods , 5 (4), 371–385. https://doi.org/10.1002/jrsm.1123

Pollock, D., Davies, E. L., Peters, M. D., Tricco, A. C., Alexander, L., McInerney, P., & Munn, Z. (2021). Undertaking a scoping review: A practical guide for nursing and midwifery students, clinicians, researchers, and academics. Journal of Advanced Nursing , 77 (4), 2102–2113. https://doi.org/10.1111/jan.14743

Popay, J., Roberts, H., Sowden, A., Petticrew, M., Arai, L., Rodgers, M., Britten, N., Roen, K., & Duffy, S. (2006). Guidance on the conduct of narrative synthesis in systematic reviews. A Product from the ESRC Methods Programme Version , 1 (1), b92.

Google Scholar

Powell, C. L., Chiu, C. Y., Sun, X., & So, S. H. W. (2024). A meta-analysis on the efficacy of low-intensity cognitive behavioural therapy for generalised anxiety disorder. BMC Psychiatry , 24 (1), 10.

Rosenquist, P. B. (2012, June). Struck by Living: From Depression to Hope. [Review of the book Struck by Living: From Depression to Hope by J.K.Hersh]. https://journals.lww.com/ectjournal/Fulltext/2012/06000/Struck_by_Living__From_Depression_to_Hope.17.aspx?casa_token=UGcewk7TbCAAAAAA:dQEVy9ngw-8j9EF_6deBRjsX_yQg24K3eSYaOLZFB7Z2DQAlFVne1pjn5JSv_QO63Z8-nYTdqcTymJTxbNZ_aB_V

Roy-Byrne, P. P., Davidson, K. W., Kessler, R. C., Asmundson, G. J., Goodwin, R. D., Kubzansky, L., Lydiard, R. B., Massie, M. J., Katon, W., Laden, S. K., & Stein, M. B. (2008). Anxiety disorders and comorbid medical illness. General Hospital Psychiatry , 30 (3), 208–225. https://doi.org/10.1016/j.genhosppsych.2007.12.006

Runge, E., Jensen, E. K., Mathiasen, K., Larsen, P. V., Hertz, S. P. T., Holmberg, T. T., Tarp, K., Linnet, J., & Lichtenstein, M. B. (2022). Early development of treatment motivation predicts adherence and symptom reduction in an internet-based guided self-help program for binge eating disorder. Frontiers in Psychiatry , 13 , 969338. https://doi.org/10.3389/fpsyt.2022.969338

Salomonsson, S., Santoft, F., Lindsäter, E., Ejeby, K., Ljótsson, B., Öst, L. G., Ingvar, M., Lekander, M., & Hedman-Lagerlöf, E. (2018). Stepped care in primary care–guided self-help and face-to-face cognitive behavioural therapy for common mental disorders: A randomized controlled trial. Psychological Medicine , 48 (10), 1644–1654. https://doi.org/10.1017/S0033291717003129

Saris, I. M. J., Aghajani, M., Van Der Werff, S. J. A., Van Der Wee, N. J. A., & Penninx, B. W. J. H (2017). Social functioning in patients with depressive and anxiety disorders. Acta Psychiatrica Scandinavica , 136 (4), 352–361. https://doi.org/10.1111/acps.12774

Scholten, W. D., Batelaan, N. M., Van Oppen, P., Smit, J. H., & Van Balkom, A. J. (2013). Discontinuation of antidepressants in remitted anxiety disorder patients: The need for strategies to prevent relapse. Psychotherapy and Psychosomatics , 82 (6), 399–400. https://doi.org/10.1159/000350450

Scholten, W., Muntingh, A., Batelaan, N., Draisma, S., De Jonge, M., & van Balkom, A. (2021). Reasons for participation and nonparticipation in Psychological Relapse Prevention for anxiety and depression: A qualitative study. Journal of Psychiatric Practice , 27 (3), 184–193. https://doi.org/10.1097/pra.0000000000000550

Shafran, R., Myles-Hooton, P., Bennett, S., & Öst, L. G. (2021). The concept and definition of low intensity cognitive behaviour therapy. Behaviour Research and Therapy , 138 , 103803. https://doi.org/10.1016/j.brat.2021.103803

Spitzer, R. L., Kroenke, K., Williams, J. B., & Löwe, B. (2006). A brief measure for assessing generalized anxiety disorder. Archives of Internal Medicine , 166 , 1092–1097.

Stonebank, H. (2017, October 2). The role of Psychological Wellbeing Practitioner and career progression NHS England. https://www.england.nhs.uk/blog/the-role-of-psychological-wellbeing-practitioner-and-career-progression/

The Lancet. (2021). Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the COVID-19 pandemic. The Lancet , 398 (10312), 1700–1712. https://doi.org/10.1016/S0140-6736(21)02143-7

Thornicroft, G., Chatterji, S., Evans-Lacko, S., Gruber, M., Sampson, N., Aguilar-Gaxiola, S., Al-Hamzawi, A., Alonso, J., Andrade, L., Borges, G., Bruffaerts, R., Bunting, B., de Almeida., Florescu, S., de Girolamo, G., Gureje, O., Haro, J. M., He, Y., Hinkiv, H., Karam, E., Kawakami, N., Lee, S., Navarro-Mateu, F., Piazza, M., Posada-Villa, J., de Galvis, Y. T., & Kessler, R. C. (2017). Undertreatment of people with major depressive disorder in 21 countries. The British Journal of Psychiatry , 210 (2), 119–124. https://doi.org/10.1192/bjp.bp.116.188078

Tokgöz, P., Hrynyschyn, R., Hafner, J., Schönfeld, S., & Dockweiler, C. (2021). Digital health interventions in prevention, relapse, and therapy of mild and moderate depression: Scoping review. JMIR Mental Health , 8 (4), e26268. https://doi.org/10.2196/26268

Tricco, A. C., Lillie, E., Zarin, W., O’brien, K., Colquhoun, H., Kastner, M., Levac, D., Ng, C., Sharpe, J. P., Wilson, K., Kenny, M., Warren, R., Wilson, C., Stelfox, H. T., & Straus, S. E. (2016). A scoping review on the conduct and reporting of scoping reviews. BMC Medical Research Methodology , 16 , 1–10. https://doi.org/10.1186/s12874-016-0116-4

Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D.,... & Straus, S. E. (2018). PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Annals of Internal Medicine, 169(7), 467-473. https://doi.org/10.7326/M18-0850

Van Schaik, D. J., Klijn, A. F., Van Hout, H. P., Van Marwijk, H. W., Beekman, A. T., De Haan, M., & Van Dyck, R. (2004). Patients’ preferences in the treatment of depressive disorder in primary care. General Hospital Psychiatry , 26 (3), 184–189. https://doi.org/10.1016/j.genhosppsych.2003.12.001

Watt, M. C. (2006). Review of It’s Not All In Your Head: How Worrying about Your Health Could Be Making You Sick–and What You Can Do about It. [Review of the book It’s Not All In Your Head: How Worrying about Your Health Could Be Making You Sick–and What You Can Do about It by G. J. G. Asmundson., & S.Taylor.] https://www.proquest.com/openview/0929166cc2dab81a4e9ff1c47a98a9d9/1?pq-origsite=gscholar&cbl=41992

Williams, C. (2012). Overcoming depression and low mood (3rd ed.). Hodder Arnold.

Wright, J., Clum, G. A., Roodman, A., & Febbraro, G. A. (2000). A bibliotherapy approach to relapse prevention in individuals with panic attacks. Journal of Anxiety Disorders , 14 (5), 483–499. https://doi.org/10.1016/S0887-6185(00)00035-9

Download references

This study is funded by the National Institute for Health Research (NIHR) Research for Patient Benefit (project reference: NIHR204037). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. The funders had no role in study design, data collection, analysis or interpretation, decision to publish, or preparation of the manuscript.

Author information

Authors and affiliations.

School of Health Sciences, Division of Nursing, Midwifery and Social Work, Manchester Academic Health Science Centre, University of Manchester, Manchester, M13 9PL, UK

Saher Nawaz, Penny Bee, Hannah Devaney & Cintia Faija

Department of Primary Care & Mental Health, Institute of Population Health, Faculty of Health and Life Sciences, University of Liverpool, Liverpool, UK

Cintia Faija

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Saher Nawaz .

Ethics declarations

Conflict of interest, additional information, publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Nawaz, S., Bee, P., Devaney, H. et al. Relapse prevention following guided self-help for common health problems: A Scoping Review. Cogn Ther Res (2024). https://doi.org/10.1007/s10608-024-10520-x

Download citation

Accepted : 29 July 2024

Published : 03 September 2024

DOI : https://doi.org/10.1007/s10608-024-10520-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Scoping review
Guided self-help
Relapse prevention
Find a journal
Publish with us
Track your research

Open access
Published: 06 September 2024

Is artificial intelligence for medical professionals serving the patients?

Protocol for a systematic review on patient-relevant benefits and harms of algorithmic decision-making

Christoph Wilhelm ORCID: orcid.org/0000-0003-0931-2134 1 , 2 ,
Anke Steckelberg 1 &
Felix G. Rebitschek 2 , 3

Systematic Reviews volume 13 , Article number: 228 ( 2024 ) Cite this article

Metrics details

Algorithmic decision-making (ADM) utilises algorithms to collect and process data and develop models to make or support decisions. Advances in artificial intelligence (AI) have led to the development of support systems that can be superior to medical professionals without AI support in certain tasks. However, whether patients can benefit from this remains unclear. The aim of this systematic review is to assess the current evidence on patient-relevant benefits and harms, such as improved survival rates and reduced treatment-related complications, when healthcare professionals use ADM systems (developed using or working with AI) compared to healthcare professionals without AI-related ADM (standard care)—regardless of the clinical issues.

Following the PRISMA statement, MEDLINE and PubMed (via PubMed), Embase (via Elsevier) and IEEE Xplore will be searched using English free text terms in title/abstract, Medical Subject Headings (MeSH) terms and Embase Subject Headings (Emtree fields). Additional studies will be identified by contacting authors of included studies and through reference lists of included studies. Grey literature searches will be conducted in Google Scholar. Risk of bias will be assessed by using Cochrane’s RoB 2 for randomised trials and ROBINS-I for non-randomised trials. Transparent reporting of the included studies will be assessed using the CONSORT-AI extension statement. Two researchers will screen, assess and extract from the studies independently, with a third in case of conflicts that cannot be resolved by discussion.

It is expected that there will be a substantial shortage of suitable studies that compare healthcare professionals with and without ADM systems concerning patient-relevant endpoints. This can be attributed to the prioritisation of technical quality criteria and, in some cases, clinical parameters over patient-relevant endpoints in the development of study designs. Furthermore, it is anticipated that a significant portion of the identified studies will exhibit relatively poor methodological quality and provide only limited generalisable results.

Systematic review registration

This study is registered within PROSPERO (CRD42023412156).

Peer Review reports

Artificial intelligence (AI) is a broad term referring to the field of computer science that develops algorithms mimicking human cognitive functions such as learning, perception, problem-solving and decision-making. AI encompasses various approaches, including machine learning (ML) and deep learning. It comprises a range of technologies and techniques, including algorithmic decision-making (ADM) ([ 9 ]: 1). ADM refers to the process of using these algorithms to gather, process, model and use input data to make or support decisions. Feedback from these decisions can then be used for improving the system ([ 2 ]: 612). An ADM can take various forms depending on how it is framed and presented to the user or decision subject. It can be a simple algorithm that has been known and used for decades, such as classification trees [ 37 ], or a more complex system like a recommender or AI that can provide recommendations to human decision-makers, nudge its users in a certain direction or perform fully automated decision-making processes without human involvement ([ 2 ]: 613). We specify AI-related algorithmic decision-making systems (AI-related ADM) as decision support systems that either apply AI (relying on ML models) or have been developed with the help of AI.

Recent advances in AI have resulted in the development of more complex and sophisticated systems that can outperform humans in certain tasks. For example, in the field of computer vision, systems like DeepMind’s AlphaFold have revolutionised protein structure prediction, solving a decades-old challenge in biology by accurately predicting 3D protein structures [ 18 ]. Additionally, AI innovations have transformed financial services, with machine learning models now being used to predict market trends, optimise trading strategies and enhance fraud detection [ 12 ]. Furthermore, generative AI has demonstrated remarkable capabilities in generating human-like text and performing a wide range of language-related tasks with unprecedented accuracy [ 13 ]. Recently, ChatGPT was evaluated for its clinical reasoning ability by testing its performance on questions from the United States Medical Licensing Examination, where it scored at or near the passing threshold on all three exams without any special training or reinforcement [ 21 ].

These advances in AI seem to have enormous potential to transform many different fields and industries, which begs the question: will AI do so in healthcare?

In clinical trials, AI systems have already shown potential to help clinicians make better diagnoses [ 3 , 22 ], help personalise medicine and monitor patient care [ 6 , 16 ] and contribute to drug development [ 7 ]. However, successful application in practice is limited ([ 30 ]: 77) and potential issues that may be responsible for this gap between research and practice should be revealed by our work.

By searching PubMed for the term ‘artificial intelligence’, we found over 2000 systematic reviews and meta-analyses published in the last 10 years, with a yearly increasing trend. These include several reviews conducted in the area of AI in healthcare that provide an overview of the current state of AI technologies in specific clinical areas, including AI systems for breast cancer diagnosis in screening programmes [ 8 ], ovarian cancer [ 38 ], early detection of skin cancer [ 17 ], COVID-19 and other pneumonia [ 15 ], prediction of preterm birth [ 1 ] or diabetes management [ 19 ]. Other reviews have focused on comparing clinicians and AI systems in terms of their performance to show their capabilities in a clinical setting [ 24 , 27 , 34 ].

Although these reviews are crucial to the further development of AI systems, they offer little insight into whether patients actually benefit from their use by medical professionals. Indeed, these studies focus on the analytical performance of these systems, rather than on healthcare-related metrics. In most of the studies mentioned here, the underlying algorithms have been evaluated using a variety of parameters, such as the F1 score for error classification, balanced accuracy, false positive rate and area under the receiver operating characteristic curve (AUROC). However, measures of a system’s accuracy often provide non-replicable results ([ 25 ]: 4), do not necessarily indicate clinical efficiency ([ 20 ]: 1), AUROC does not necessarily indicate clinical applicability ([ 10 ]: 935) and in fact, none of these measures reflects beneficial change in patient care ([ 4 ]: 1727, [ 33 ]: 1).

To summarise, as with any other new technology introduced into healthcare, the clinical effectiveness and safety of AI compared to the standard of care must be evaluated through properly designed studies to ensure patient safety and maximise benefits while minimising any unintended harm ([ 31 ]: 328). Therefore, a critical analysis of patient-relevant outcomes is needed, especially the benefits and harms of decisions informed by or made by AI systems.

To this end, this review goes beyond previous studies in several ways. First, we study clinical AI systems that enable algorithmic decision-making (AI-related ADM) in general and therefore do not limit ourselves to selected clinical problems. In particular, we focus on machine learning systems that infer rules from observations. Although we omit rule-based systems, we apply the term AI throughout our work because it is often incorrectly and redundantly used for ML and deep learning in the literature we study. Second, we focus on studies that report patient-relevant outcomes that, according to German Institute for Quality and Efficiency in Healthcare ([ 14 ]: 44), describe how patients feel, how they can perform their functions and activities or if they survive. These may include, for example, mortality, morbidity (with regard to complaints and complications), length of hospital stay, readmission, time to intervention and health-related quality of life. Third, we focus only on studies that compare medical professionals supported by AI-related ADM systems with medical professionals without AI-related ADM systems (standard care). By doing so, this review provides an overview of the current literature on clinical AI-related ADM systems, summarises the empirical evidence on their benefits and harms for patients and highlights research gaps that need to be addressed in future studies.

The aim of this review is to systematically assess the current evidence on patient-relevant benefits and harms of ADM systems which are developed or used with AI (AI-related ADM) to support medical professionals compared to medical professionals without this support (standard care).

Are there studies that compare patient-relevant effectiveness of AI-related ADM for medical professionals compared to medical professionals without AI-related ADM?

Do these studies show adequate methodological quality and are their findings generalisable?

Can AI-related ADM systems help medical professionals to make better decisions in terms of benefits and harms for patients?

Methods/design

In accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) statement [ 26 ], the study protocol for this systematic is registered on the International Prospective Register of Systematic Reviews (PROSPERO) database (CRD42023412156). If necessary, post-registration changes to the protocol will be detailed under the PROSPERO record with an accompanying rationale.

We will follow the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [ 29 ] and the Methodological Expectations of Cochrane Intervention Reviews (MECIR) standards [ 11 ].

We will search systematically using English free text terms in title/abstract, Medical Subject Headings (MeSH) terms and Embase Subject Headings (Emtree) fields for various forms of keywords related to ‘artificial intelligence’ and relevant subcategories of computer generated and processed decision-making algorithms, ‘medical professionals’ and keywords describing effectiveness parameters and outcomes as well as preferred study types. Based on the block building approach, keywords and terms are combined using the Boolean operators AND and OR and progressively checked for relevant hits.

Databases to be used for searches

MEDLINE and PubMed (via PubMed), Embase (via Elsevier) and Institute of Electrical and Electronics Engineers (IEEE) Xplore will be searched for peer-reviewed articles as well as ClinicalTrials.gov and ICTRP (via CENTRAL) for ongoing trials and protocols.

To reduce potential publication bias, additional studies will be identified by contacting authors of included studies, contacting experts in the field and through reference lists of relevant studies. Grey literature searches will be conducted in Google Scholar. For this purpose, the keywords used in the systematic search will be used in different combinations, as well as their German equivalents. Google Scholar will be searched up to the 10th hit page. The detailed search strategy for each database will be reported under the PROSPERO record once the searches have been conducted.

Search strategy

We developed our search strategy using the PICOS scheme (Table 1 ).

While doing preliminary searches for basic literature in MEDLINE and PubMed (via PubMed), we noticed that study conductors from different scientific fields (e.g. computer scientists) used different terms for the intervention outcomes we were looking for. In addition, some studies were not indexed appropriately in PubMed, which complicated our initial search strategy. To carry out the search strategy, we have created and tested the blocks consecutively to gather the best results from each block, expanding and narrowing the search strategy. To assess the right direction of the search strategy, we have used fundamental literature, such as Choudhury and Asan [ 5 ], Park et al. [ 31 ] and Nagendran et al. [ 27 ] as test sets, making sure the results of our search had common ground with these studies.

The resulting search string for MEDLINE and PubMed in the individual blocks can be found in Table 2 and describes the basis for other databases.

Types of studies to be included

For the systematic search, peer-reviewed interventional and observational studies published in German or English 10 years retrospectively from the date of the search will be considered. For the search of grey literature, scientific reports published in German or English 10 years retrospectively from the date of the search will be considered. To extract potentially relevant studies from (systematic) reviews and meta-analyses, secondary studies will be gathered and screened. However, secondary studies will not be included in the synthesis.

In contrast to studies of effectiveness and safety, pure efficacy studies (e.g. focusing on algorithms accuracy) will be excluded as these outcomes are not directly relevant for patients. Patient-relevant outcomes will be defined according to the IQEHC method paper [ 14 ]. In addition, studies that used AI systems beyond our scope, such as robotics (systems that support the implementation of decisions), will be excluded. Editorials, commentaries, letters and other informal publication types will be excluded as well.

We will provide a list of all references screened in full text including exclusion reasons in the appendix of the final study.

Participants

Our study is focusing on human patients without restriction of age or sex. Therefore, the input data for the algorithms must include real human data gathered either during routine care and saved for use in research or generated specifically for the individual study.

Intervention

Out study is focusing on medical professionals utilising an AI-related ADM system to address a clinical problem.

In our working definition, a medical professional is a qualified individual who has the authority to perform necessary medical procedures within their professional scope of practice. Their goal is to improve, maintain or restore the health of individuals by examining, diagnosing, prognosticating and/or treating clinical problems. This may include medical doctors, registered nurses and other medical professionals. Clinical problems can encompass illnesses, injuries and physical or mental disorders, among other conditions.

In our working definition, an AI-related ADM system is a clinical decision support system that either applies AI in the sense of machine learning (ML, excluding rule-based systems) or has been developed with the help of ML. Clinical decision support models without any involvement of AI will be excluded.

Medical professionals, as described in the working definition, are addressing a clinical problem without the support of an AI-related ADM system (standard care).

Patient-relevant benefits and harms, according to the IQEHC method paper [ 14 ], are gathered. These may include, for example, mortality, morbidity (with regard to complaints and complications), length of hospital stay, readmission, time to intervention and health-related quality of life.

Study types

We will collect both interventional and observational studies, which may encompass randomised controlled trials, cohort studies, case–control studies, randomised surveys, retrospective and prospective studies and phase studies, as well as non-inferiority or diagnostic studies.

Data extraction

Records arising from the literature search will be stored in the citation manager Citavi 6 (c) by Swiss Academic Software. After removing duplicates, two reviewers will independently review all titles and abstracts via the browser application Rayyan [ 28 ]. Studies potentially meeting the inclusion criteria will then be screened in full text independently by two reviewers using Citavi 6 (c). Disagreements over eligibility of studies will be discussed and, if necessary, resolved by a third reviewer. Authors of the included studies will be contacted if clarification of their data or study methods is required. The PRISMA 2020 flow diagram [ 29 ] will be used to keep the study selection process transparent.

Using a standardised data collection form, two reviewers will extract data independently from the included studies and will compare them for discrepancies. Missing data will be requested from study authors. Extracted data will include country of conduction, setting, study design, observational period, patient-relevant outcomes, intervention, comparator, characteristics of patient and medical professional populations and characteristics of the used algorithm. Additionally, studies will be classified by type of system, medical specialty or clinical area, prediction or classification goal of the AI-related ADM, supported decision, investigated benefits and harms, private or public study funding, applicable regulation (e.g. FDA, MDR), medical device classification (based on the risk and nature of the product) and whether the product is commercially available in its respective class (Table 3 ).

Risk of bias and quality assessment

Risk of bias will be assessed by using the revised Cochrane risk-of-bias tool for randomised trials (RoB 2) [ 36 ] and the risk-of-bias in non-randomised studies for interventions (ROBINS-I) tool [ 35 ]. Disagreements between the authors over the risk of bias in the included studies will be resolved by discussion or with involvement of a third author if necessary. Transparent reporting of the included studies will be assessed trough the Consolidated Standards of Reporting Trials interventions involving Artificial Intelligence (CONSORT-AI) extension by Liu et al. [ 23 ]. The CONSORT-AI extension includes 14 new items that were considered sufficiently important for AI interventions to be routinely reported in addition to the core CONSORT items by Schulz et al. [ 32 ]. CONSORT-AI aims to improve the transparency and completeness in reporting clinical trials for AI interventions. It will assist to understand, interpret and critically appraise the quality of clinical trial design and risk of bias in the reported outcomes. We will assess studies conducted prior to the introduction of the CONSORT-AI guidelines in 2020 against these standards where possible. Although these studies may not fully meet the new criteria, application of the guidelines may still identify potential reporting gaps and ensure a consistent assessment framework across studies. We will discuss limitations related to this retrospective requirement to ensure a balanced and comprehensive analysis.

Data synthesis

Given the expected likelihood of heterogeneity between studies in the different medical specialties in terms of outcome measures, study designs and interventions, we do not know if performing a meta-analysis will be possible. However, a systematic narrative synthesis will be provided of the results with an overview of the relevant effects for the outcomes, with information presented in the text and tables to summarise and explain the characteristics and findings of the included studies. We will analyse the geographic distribution, study settings and medical specialties of the included studies. Additionally, we will examine funding sources and conduct a detailed risk of bias assessment. Compliance with reporting standards, such as CONSORT-AI and TRIPOD-AI, will be evaluated. We also plan to analyse patient demographics, including age, sex and race/ethnicity, as well as the involvement and training of medical professionals. ADM systems will be categorised into applicable regulation (e.g. FDA, MDR), medical device classification (based on the risk and nature of the product) and whether the product is commercially available in its respective class. Outcome analyses will focus on assessing both benefits and harms. Furthermore, we will analyse the validation of algorithms, considering both internal and external validation, and review the data availability statements to evaluate the accessibility of data used for algorithm development. Studies with an unclear or high risk of bias are not excluded to avoid potential selection bias and to ensure that valuable findings, particularly in emerging areas, are not lost. By including them, but clearly acknowledging and discussing their limitations, we aim to provide a more comprehensive overview of the available evidence. For this reason, our narrative synthesis emphasises the qualitative aspects of the data and focuses on identifying and describing trends, patterns and inconsistencies in the studies, rather than attempting to quantify effect sizes. This is consistent with the approach of recent reviews examining the methodological quality of machine learning systems in clinical settings (e.g. [ 27 ]).

It is to be expected that there is a significant lack of suitable studies comparing healthcare professionals with and without AI-related ADM systems regarding patient-relevant outcomes. It is assumed that this is due to, first, the lack of approval regulations for AI systems, second, the prioritisation of technical and clinical parameters over patient-relevant outcomes in the development of study designs and, third, the prioritisation of AI for supporting clinical processes (e.g. administration). In addition, it is to be expected that a large proportion of the studies to be identified are of rather poor methodological quality and provide results that are rather difficult to generalise. Although reporting guidelines such as the Consolidated Standards of Reporting Trials (CONSORT) statement [ 32 ] are well-known and widely used in medical and public health research, they do not necessarily correspond to the novel protocol and study designs that are relevant for the assessment of the research questions relevant here. The extension of the Reporting Guidelines for Clinical Study Reports of Interventions Using Artificial Intelligence (CONSORT-AI) [ 23 ] may fill the gap but this guideline is relatively new and not necessarily always applied.

Availability of data and materials

Not applicable.

Abbreviations

Algorithmic decision-making
Artificial intelligence

Area under the receiver operating characteristic curve

Cochrane Central Register of Controlled Trials

Convolutional neural network

Consolidated Standards of Reporting Trials

Consolidated Standards of Reporting Trials for Artificial Intelligence

Centre for Reviews and Dissemination

Embase Subject Headings

Intensive care unit

Institute of Electrical and Electronics Engineers

German Institute for Quality and Efficiency in Healthcare

Methodological Expectations of Cochrane Intervention Reviews

Medical Subject Headings

Machine learning

Non-randomised controlled trial

Participants, Intervention, Control, Outcome

Preferred Reporting Items for Systematic Review and Meta-Analysis

Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols

International Prospective Register of Systematic Reviews

Randomised controlled trial

A convolutional neural network that is 18 layers deep

Revised Cochrane risk-of-bias tool for randomised trials

Risk-of-bias in non-randomised studies for interventions

Respiratory rate

Standard deviation

Oxygen saturation

Standards for Reporting Qualitative Research

Akazawa M, Hashimoto K. Prediction of preterm birth using artificial intelligence: a systematic review. J Obstet Gynaecol. 2022;42(6):1662–8. https://doi.org/10.1080/01443615.2022.2056828 .

Article PubMed Google Scholar

Araujo T, Helberger N, Kruikemeier S, de Vreese CH. In AI we trust? Perceptions about automated decision-making by artificial intelligence. AI & Soc. 2020;35:611–23. https://doi.org/10.1007/s00146-019-00931-w .

Article Google Scholar

Bahl M, Barzilay R, Yedidia AB, Locascio NJ, Yu L, Lehman CD. High-risk breast lesions: a machine learning model to predict pathologic upgrade and reduce unnecessary surgical excision. Radiology. 2018;286(3):810–8. https://doi.org/10.1148/radiol.2017170549 .

Brocklehurst P, Field D, Greene K, Juszczak E, Keith R, Kenyon S, et al. Computerised interpretation of fetal heart rate during labour (INFANT): a randomised controlled trial. Lancet. 2017;389:1719–29. https://doi.org/10.1016/s0140-6736(17)30568-8 .

Choudhury A, Asan O. Role of artificial intelligence in patient safety outcomes: systematic literature review. JMIR Med Inform. 2020;8(7):e18599. https://doi.org/10.2196/18599 .

Article PubMed PubMed Central Google Scholar

Ciervo J, Shen SC, Stallcup K, Thomas A, Farnum MA, Lobanov VS, Agrafiotis DK. A new risk and issue management system to improve productivity, quality, and compliance in clinical trials. JAMIA Open. 2019;2(2):216–21. https://doi.org/10.1093/jamiaopen/ooz006 .

Ekins S, Puhl AC, Zorn KM, Lane TR, Russo DP, Klein JJ, Hickey AJ, Clark AM. Exploiting machine learning for end-to-end drug discovery and development. Nat Mater. 2019;18(5):435–41. https://doi.org/10.1038/s41563-019-0338-z .

Article CAS PubMed PubMed Central Google Scholar

Freeman K, Geppert J, Stinton C, Todkill D, Johnson S, Clarke A, Taylor-Phillips S. Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy. BMJ. 2021;1(374):n1872. https://doi.org/10.1136/bmj.n1872 .

Graili P, Ieraci L, Hosseinkhah N, Argent-Katwala M. Artificial intelligence in outcomes research: a systematic scoping review. Expert Rev Pharmacoecon Outcomes Res. 2021;21(4):601–23. https://doi.org/10.1080/14737167.2021.1886083 .

Halligan S, Altman DG, Mallett S. Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: a discussion and proposal for an alternative approach. Eur Radiol. 2015;25(4):932–9. https://doi.org/10.1007/s00330-014-3487-0 .

Higgins J, Lasserson T, Chandler J, Tovey D, Thomas J, Flemyng E, Churchill R. Methodological Expectations of Cochrane Intervention Reviews (MECIR). Standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates. 2021. https://community.cochrane.org/sites/default/files/uploads/MECIR-February-2021.pdf . Accessed 12 Nov 2023.

Hilpisch Y. Artificial intelligence in finance: a Python-based guide. Sebastopol: O’Reilly Media; 2020.

Google Scholar

Hughes A. ChatGPT: Everything you need to know about OpenAI’s GPT-3 tool. BBC Science Focus Magazine. https://www.sciencefocus.com/future-technology/gpt-3/ . 2023. Accessed 12 Nov 2023.

Institute for Quality and Efficiency in Healthcare (IQEHC). Allgemeine Methoden. Version 6.1, from 24 th of January 2022. https://www.iqwig.de/methoden/allgemeine-methoden-v6-1.pdf . 2022. Accessed 12 Nov 2023.

Jia LL, Zhao JX, Pan NN, Shi LY, Zhao LP, Tian JH, Huang G. Artificial intelligence model on chest imaging to diagnose COVID-19 and other pneumonias: a systematic review and meta-analysis. Eur J Radiol Open. 2022;9:100438. https://doi.org/10.1016/j.ejro.2022.100438 .

Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, Wang Y, Dong Q, Shen H, Wang Y. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. 2017;2(4):230–43. https://doi.org/10.1136/svn-2017-000101 .

Jones OT, Matin RN, van der Schaar M, Prathivadi Bhayankaram K, Ranmuthu CKI, Islam MS, Behiyat D, Boscott R, Calanzani N, Emery J, Williams HC, Walter FM. Artificial intelligence and machine learning algorithms for early detection of skin cancer in community and primary care settings: a systematic review. Lancet Digit Health. 2022;4(6):e466–76. https://doi.org/10.1016/S2589-7500(22)00023-1 .

Article CAS PubMed Google Scholar

Jumper J, Evans R, Pritzel A, Green T, Figurnov M, Ronneberger O, Tunyasuvunakool K, Bates R, Žídek A, Potapenko A, Bridgland A, Meyer C, Kohl SA, Ballard AJ, Cowie A, Romera-Paredes B, Nikolov S, Jain R, Adler J, Back T, Petersen S, Reiman D, Clancy E, Zielinski M, Steinegger M, Pacholska M, Berghammer T, Bodenstein S, Silver D, Vinyals O, Senior A, Kavukcuoglu K, Kohli P, Hassabis D. Highly accurate protein structure prediction with AlphaFold. Nature. 2021;596(7873):583–9. https://doi.org/10.1038/s41586-021-03819-2 .

Kamel Rahimi A, Canfell OJ, Chan W, Sly B, Pole JD, Sullivan C, Shrapnel S. Machine learning models for diabetes management in acute care using electronic medical records: a systematic review. Int J Med Inform. 2022;2(162):104758. https://doi.org/10.1016/j.ijmedinf.2022.104758 .

Keane PA, Topol EJ. With an eye to AI and autonomous diagnosis. NPJ Digit Med. 2018;28(1):40. https://doi.org/10.1038/s41746-018-0048-y .

Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, Madriaga M, Aggabao R, Diaz-Candido G, Maningo J, Tseng V. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198. https://doi.org/10.1371/journal.pdig.0000198 .

Li Q, Zhao K, Bustamante CD, Ma X, Wong WH. Xrare: a machine learning method jointly modeling phenotypes and genetic evidence for rare disease diagnosis. Genet Med. 2019;21(9):2126–34. https://doi.org/10.1038/s41436-019-0439-8 .

Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AK, SPIRIT-AI and CONSORT-AI Working Group. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence the CONSORT-AI extension. Nat Med. 2020;26(9):1364–74. https://doi.org/10.1038/s41591-020-1034-x .

Liu X, Faes L, Kale AU, Wagner SK, Fu DJ, Bruynseels A, Mahendiran T, Moraes G, Shamdas M, Kern C, Ledsam JR, Schmid MK, Balaskas K, Topol EJ, Bachmann LM, Keane PA, Denniston AK. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health. 2019;1(6):e271–97. https://doi.org/10.1016/S2589-7500(19)30123-2 .

McDermott MBA, Wang S, Marinsek N, Ranganath R, Foschini L, Ghassemi M. Reproducibility in machine learning for health research: Still a ways to go. Sci Transl Med. 2021 Apr;13(586). doi: 10.1126/scitranslmed.abb1655.

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA. Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1. https://doi.org/10.1609/aimag.v27i4.1904 .

Nagendran M, Chen Y, Lovejoy CA, Gordon AC, Komorowski M, Harvey H, Topol EJ, Ioannidis JPA, Collins GS, Maruthappu M. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ. 2020;25(368):m689. https://doi.org/10.1136/bmj.m689 .

Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan - a web and mobile app for systematic reviews. Syst Rev. 2016;5:210. https://doi.org/10.1186/s13643-016-0384-4 .

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;29(372):n71. https://doi.org/10.1136/bmj.n71 .

Panch T, Mattie H, Celi LA. The “inconvenient truth” about AI in healthcare. NPJ Digit Med. 2019;16(2):77. https://doi.org/10.1038/s41746-019-0155-4 .

Park Y, Jackson GP, Foreman MA, Gruen D, Hu J, Das AK. Evaluating artificial intelligence in medicine: phases of clinical research. JAMIA Open. 2020;3(3):326–31. https://doi.org/10.1093/jamiaopen/ooaa033 .

Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010 340:c332. https://doi.org/10.1136/bmj.c332 .

Shah NH, Milstein A, Bagley PhD SC. Making machine learning models clinically useful. JAMA. 2019;322(14):1351–2. https://doi.org/10.1001/jama.2019.10306 .

Shen J, Zhang CJP, Jiang B, Chen J, Song J, Liu Z, He Z, Wong SY, Fang PH, Ming WK. Artificial intelligence versus clinicians in disease diagnosis: systematic review. JMIR Med Inform. 2019;7(3):e10010. https://doi.org/10.2196/10010 .

Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, Carpenter JR, Chan AW, Churchill R, Deeks JJ, Hróbjartsson A, Kirkham J, Jüni P, Loke YK, Pigott TD, Ramsay CR, Regidor D, Rothstein HR, Sandhu L, Santaguida PL, Schünemann HJ, Shea B, Shrier I, Tugwell P, Turner L, Valentine JC, Waddington H, Waters E, Wells GA, Whiting PF, Higgins JP. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;12(355):i4919. https://doi.org/10.1136/bmj.i4919 .

Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, Cates CJ, Cheng HY, Corbett MS, Eldridge SM, Emberson JR, Hernán MA, Hopewell S, Hróbjartsson A, Junqueira DR, Jüni P, Kirkham JJ, Lasserson T, Li T, McAleenan A, Reeves BC, Shepperd S, Shrier I, Stewart LA, Tilling K, White IR, Whiting PF, Higgins JPT. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;28(366):l4898. https://doi.org/10.1136/bmj.l4898 .

Winterfeldt DV, Edwards W. Decision analysis and behavioral research. Cambridge University Press; 1986.

Xu HL, Gong TT, Liu FH, Chen HY, Xiao Q, Hou Y, Huang Y, Sun HZ, Shi Y, Gao S, Lou Y, Chang Q, Zhao YH, Gao QL, Wu QJ. Artificial intelligence performance in image-based ovarian cancer identification: a systematic review and meta-analysis. EClinicalMedicine. 2022;17(53):101662. https://doi.org/10.1016/j.eclinm.2022.101662 .

Download references

Acknowledgements

Open Access funding enabled and organized by Projekt DEAL. This research does not receive any funding.

Author information

Authors and affiliations.

Institute of Health and Nursing Sciences, Martin Luther University Halle-Wittenberg, Magdeburger Str. 8, Halle, 06112, Germany

Christoph Wilhelm & Anke Steckelberg

Harding Center for Risk Literacy, Faculty of Health Sciences Brandenburg, University of Potsdam, Virchowstr. 2, Potsdam, 14482, Germany

Christoph Wilhelm & Felix G. Rebitschek

Max Planck Institute for Human Development, Lentzeallee 94, Berlin, 14195, Germany

Felix G. Rebitschek

You can also search for this author in PubMed Google Scholar

Contributions

Christoph Wilhelm conceptualised, developed the methodology and wrote this protocol. He is the guarantor of the manuscript. Prof. Dr Anke Steckelberg validated the methodology of this protocol. Dr Felix G. Rebitschek conceptualised, reviewed, edited and supervised this protocol.

Corresponding author

Correspondence to Christoph Wilhelm .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Wilhelm, C., Steckelberg, A. & Rebitschek, F.G. Is artificial intelligence for medical professionals serving the patients? . Syst Rev 13 , 228 (2024). https://doi.org/10.1186/s13643-024-02646-6

Download citation

Received : 17 May 2024

Accepted : 22 August 2024

Published : 06 September 2024

DOI : https://doi.org/10.1186/s13643-024-02646-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Patient relevant
Healthcare professionals
Decision support

Systematic Reviews

ISSN: 2046-4053

Submission enquiries: Access here and click Contact Us
General enquiries: [email protected]

Systematic Review
Open access
Published: 05 September 2024

Components and entities of post-disaster damage and loss assessment program in healthcare sector: a scoping review

Javad Miri ORCID: orcid.org/0000-0003-1464-6788 1 ,
Golrokh Atighechian ORCID: orcid.org/0000-0003-3290-2765 2 ,
Hesam Seyedin ORCID: orcid.org/0000-0001-5614-4052 3 &
Ahmad Reza Raeisi ORCID: orcid.org/0000-0001-8588-4340 4

BMC Public Health volume 24 , Article number: 2417 ( 2024 ) Cite this article

Metrics details

Disasters can cause casualties and significant financial loss. In accordance with the Sendai Framework for Disaster Risk Reduction, areas affected by disasters must be built back better. Accurate post-disaster damage and loss assessments are critical for the success of recovery programs. This scoping review aimed to identify the components and entities of the healthcare sector’s post-disaster damage and loss assessment program.

An comprehensive search for relevant literature was performed using several databases, including the Web of Science, PubMed, Scopus, ProQuest, and Magiran. The search was limited to papers published between 2010 and 2022. In addition, we searched the grey literature for resources related to post-disaster damage and loss assessments. Study selection and data extraction were evaluated by a third reviewer. The main themes were determined through a consensus process and agreement among team members.

A total of 845 papers were identified, 41 of which were included in the review. The grey literature search yielded 1015 documents, 23 of which were associated with the study’s purpose. The findings were classified into five main themes, 20 subthemes, and 876 codes. The main-themes include the following: Concepts and Definitions; Post-Disaster Damage and Loss Assessment Procedures; Healthcare sector procedures; Assessments Tools, and Methods; Intra-sectoral, Inter-sectoral, and cross-cutting issues.

Conclusions

The existing corpus of literature on post-disaster damage and loss assessment programs within the healthcare sector offers only limited insights into the entities and components involved. It is of great importance that stakeholders have an extensive grasp of these pivotal concepts and principles, as they are fundamental in enabling effective responses to disasters, informed decision-making, and facilitating rehabilitation and reconstruction efforts. Consequently, there is a considerable scope for further investigation in this area.

Scoping review registration number

https://osf.io/nj3fk .

Peer Review reports

Introduction

The most significant consequences of disasters are health impacts that occur in the aftermath [ 1 ]. Natural disasters such as earthquakes and floods not only have a detrimental impact on an individual’s health but also result in significant damage to the healthcare sector, reducing its capacity to respond and recover effectively. This, in turn, leads to a rise in mortality and morbidity rates [ 2 , 3 ]. Disasters directly damage the physical structure of hospitals, clinics, and healthcare centers and indirectly affect the health sector by destroying community infrastructure, such as water, electricity, fuel, transportation, and communication systems. Additionally, disasters can impact healthcare providers and their families [ 4 , 5 ].

Providing essential health services is challenging during disasters because of infrastructure failure and the inefficiency of healthcare centres [ 6 ]. It is evident that the health centres play a pivotal role in alleviating the negative consequences that arise in the aftermath of disasters. Therefore, it is of paramount importance to ensure the uninterrupted functioning of this vital infrastructures [ 7 , 8 , 9 ]. Comprehending the health consequences of disasters provides the basis for identifying demands, improving capacity, and providing opportunities for reconstruction and future disaster risk reduction [ 10 ].

The convergence of four seminal accords on disaster risk reduction, development finance, sustainable development, and climate change at the end of 2015 presented a singularly promising opportunity to achieve coherence across related policy domains. The Sendai Framework for Disaster Risk Reduction represents the global policy framework of the United Nations from 2015 to 2030. This represents a significant advance in global policy coherence concerning health, development, and climate change [ 11 , 12 , 13 ]. One of the principal objectives of the Sendai Framework is to enhance disaster preparedness for an effective response and “build back better“(BBB) in recovery [ 14 ].

The scope of disaster recovery is broader than that of response. In the context of the health system, recovery is defined as the reconstruction, restoration, and upgrading of the components of a country’s health sector and the main functions of public health, in accordance with the BBB principle and the goals of sustainable development [ 15 ]. For an optimal reconstruction, it is necessary to develop a legal, technical, and comprehensive framework. The success of a reconstruction program depends on an accurate assessment of the damage, loss, and needs of the post-disaster area to determine the approaches, goals, priorities, and measures required for reconstruction [ 2 ].

The post-disaster reconstruction of the health system in developing countries is hindered by some factors, including a lack of knowledge and expertise, limited budget and planning, political competition, fraud, and embezzlement or misuse of social benefits [ 3 , 16 ]. Considering the argument of ‘humanitarian ignorance’, In light of the argument put forth by scholars who refer to this phenomenon as “humanitarian ignorance,“ [ 17 ], it can be argued that this “knowledge” does exist and that it is purposeful ignorance of said knowledge.

In 2008, the European Union, World Bank, and United Nations Development Group implemented a standard post-disaster assessment approach and developed a comprehensive and collaborative post-disaster assessment program [ 18 ]. In damage and loss assessments, experts in each sector calculate post-disaster damage and loss, which are essential in reconstruction programs [ 19 , 20 ].

Chapin et al. (2009) studied the impact of the 2007 Ica earthquake on healthcare facilities in southern Peru. They reported that after an earthquake of magnitude 7.9 in Peru, 60% of the health centers in the region were affected to the degree that they were unable to provide client services. This study revealed that reports of damage assessments in a single disaster were sometimes not the same [ 21 ]. Achour et al. (2020) evaluated hospital performance after the 2016 Kumamoto Earthquake in Japan. Data analysis revealed that the impaired function of some healthcare centers in the affected areas significantly affected the health needs of the local communities [ 22 ].

Similar to other social sectors, the disaster impacts on the healthcare sector is considerable and is one of the concerns of managers and experts in the healthcare sector. In light of the pivotal role of the health sector in post-disaster response and recovery, as well as in the development of a post-disaster reconstruction program, it is crucial to conduct a thorough assessment of damage and losses incurred following a disaster. A post-disaster damage and loss assessment in the health sector can serve as a foundation for the creation of a coherent and integrated framework for health reconstruction. The absence of a post-disaster damage and loss assessment program may result in certain requirements being overlooked, the results of which are not deemed acceptable, facilities being allocated on a non-prioritized basis, and there being no basis for monitoring the implementation of plans and activities. Assessment is a demanding and decisive management task that is effective in decision-making, planning, monitoring, handling a program, and taking coherent actions. Post-disaster damage and loss assessment has a direct impact on decision-making, planning, monitoring of responses, and the implementation of recovery operations. Consequently, these assessments must be purposeful and scheduled.

Review objective and research questions

This scoping review was conducted to identify the entities and components of post-disaster damage and loss assessment programs in the healthcare sector. The PCC framework, which includes the participants, concepts, and context recommended by the Joanna Briggs Institute, was used to develop the research question [ 23 ]. The research question for this scoping review is as follows: what information is available about the entities and components of the healthcare sector’s post-disaster damage and loss assessment program?

A knowledge gap exists in the field of post-disaster damage and loss assessment in the healthcare sector. To address this issue, the most appropriate methodology for achieving the study’s objective was identified as a scoping review. This systematic scoping review was conducted under the proposed Joanna Briggs Institute method [ 24 ]. The study included the following steps: defining and aligning the research objectives and questions, developing the inclusion and exclusion criteria, describing the planned approach to the evidence search, study selection, data extraction, presentation of the evidence, searching for evidence, selecting the evidence, extracting the evidence, analyzing the evidence, presenting the results, and summarizing the evidence [ 23 ]. The study protocol was registered in the Open Science Framework on 4 June 2022 [ 25 ] and was published in BMJ Open [ 26 ]. The Preferred Reporting Items for Systematic Reviews extension for Scoping Reviews (PRISMA-ScR Checklist 1) [ 27 ] checklist was used to report the results of this scoping review.

Inclusion and exclusion criteria

In keeping with the scoping review methodology, our inclusion criteria (Table 1 ) were broad, and our search was comprehensive in capturing the entities and components of the healthcare sector’s post-disaster damage and loss assessment program. We included literature reviews, primary empirical articles, case studies, opinion pieces, and editorials published in English or Persian “due to geographical focus, and researcher language skills”. In addition, grey literature related to the study objective, including dissertations, organizational documents, post-disaster assessment reports, and guidelines, was searched and reviewed. Table 2 presents a distribution of studies by location, organization, and document type.

Search strategy

The search strategy was drafted with the help of an experienced informaticist librarian and was further refined through team discussion. Initially, a primary search was conducted on the Google Scholar, PubMed, World Bank, and PreventionWeb websites. The following concepts were extracted from the documents: post-conflict consequences in health systems, disaster impacts on the healthcare sector, post-disaster damage and loss assessment, post-earthquake hospital functionality, post-disaster damage and loss assessment, disaster damage, operational status of healthcare facilities during a hurricane, and the impacts of extreme events. An appropriate search strategy was used for each database (Table 3 ).

Study selection

We searched all English and Persian articles published from 2010 to 2022 on the Web of Science, PubMed, Scopus, ProQuest, Google Scholar, and Magiran databases. Our search started on 20 January 2022. The search results were imported into Endnote X9 software. After removing the duplicates, J. Miri checked all the remaining titles to remove unrelated documents. The titles and abstracts of the remaining articles were independently examined by two authors (J. Miri and A.R. Raeisi) to reach a common understanding of the selection criteria, discussion of disagreements, and definition of the inclusion and exclusion criteria. The remaining articles were uploaded to Rayyan software to facilitate record screening. The full texts of articles whose abstracts did not meet the exclusion criteria or were ambiguous were reviewed. Discrepancies in inclusion or exclusion decisions were resolved through discussion (G. Atighechian). Finally, the reference lists were checked to identify relevant studies. In the grey literature search, researchers also investigated organizations’ websites related to disaster management, such as the UNDP, World Bank, UNDRR, International Recovery Platform, PreventionWeb, WHO, and FEMA. (Fig. 1 )

Adapted from the PRISMA 2020 flow diagram from Page et al. [ 47 ].

PRISMA flow diagram of the scoping review process.

Data extraction

Supplementary Tables 1 and 2 show the characteristics of the articles and grey literature discussed in this study. General information (title, authors, publication year, study location, and key findings) regarding the questions addressed in this scoping review was extracted from the selected studies. Two independent reviewers extracted all relevant information and any discrepancies were resolved through discussion.

Data analysis and presentation

The documents were organized and analyzed by the researchers using the MAXQDA 2020 software. The data analysis strategy employed at this juncture was a thematic analysis approach. Thematic analysis is a valuable approach for elucidating experiences, thoughts, or behaviors within a data set. Additionally, researchers have proposed that thematic analysis is an optimal analytical method for novice qualitative researchers due to its transparent and straightforward procedures [ 28 , 29 ].

The search of the related electronic databases led to the identification of 845 articles. After removing the duplicates, 826 studies remained. The titles were screened, and 102 potentially eligible articles were selected. The simultaneous title and abstract review by two independent reviewers led to the selection of 80 articles that were uploaded to Rayyan software. Finally, 41 articles were selected for full-text review. The grey literature search identified 1015 documents, reports, manuals, and guidelines based on the inclusion and exclusion criteria and the study objectives, and 23 documents were selected for review.

Researchers have classified resources into five categories: articles, books, dissertations, policy documents, and reports. Studies have been conducted in different countries, half of which have been published in the last five years. The findings were categorized into five main themes, 20 subthemes, and 876 codes according to the research objectives and questions. The main themes, subthemes, and some related codes are presented in (Table 4 ).

This study provides a comprehensive perspective on post-disaster damage and loss assessment in the healthcare sector. To achieve a common understanding of post-disaster damage and loss assessment in the healthcare sector, the researchers first collected definitions and related concepts. Then, organized concepts related to damage and loss assessment teams, damage and loss assessment stages, data collection elements, assessment tools, and programs. The paper concludes with a discussion of the linkages between the healthcare sector and other sectors affected by disasters.

The health system comprises a wide range of organizations, institutions, groups, and individuals in governmental and nongovernmental sectors that policy, produce resources, finance, and provide health services to restore, promote, and maintain public health [ 30 ]. According to the WHO framework, the health system comprises six building blocks; service delivery, health workforce, information, medical products, vaccines and technologies, financing, leadership, and governance [ 31 , 32 ]. The realization and promotion of community health and fair cooperation in providing resources are crucial goals of the health system and are considered fundamental in most countries [ 33 ].

The continuity of services is critical in some businesses, such as those in the healthcare sector. However, these trends can be disrupted by disasters [ 34 ]. Achour et al. (2020) evaluated hospital performance after the 2016 Kumamoto Earthquake in Japan. The occurrence of this event resulted in a disruption to the continuity of healthcare services. The investigation revealed that the primary causes of the disruption were damage to the infrastructure, including buildings, critical systems, and medical equipment. The results of the study indicated a 15% reduction in healthcare functionality in the affected regions [ 22 ]. In the study by Gufue et al. (2024), the direct economic loss to the health system caused by war-related looting or vandalism in the Tigray region of Northern Ethiopia was quantified in excess of $511 million. The assessment revealed that 80.6% of health posts, 73.6% of health centres, 80% of primary hospitals, 83.3% of general hospitals and two specialized hospitals were damaged and/or vandalized either fully or in part due to the war [ 35 ]. Therefore, a disaster recovery plan in the healthcare sector is essential for providing necessary measures and minimizing disaster consequences, And international frameworks such as Sendai play an important role in this regard and emphasize the need to develop and implement measures for disaster risk reduction and vulnerability [ 36 , 37 ].

For reconstruction, a reliable post-disaster damage and loss assessment method is required. The diversity of approaches and assessment-related outputs have led to various challenges. A significant obstacle to post-disaster damage and loss assessment is access to consistent, dependable, and detailed data on the impact of disasters. Establishing guidelines for reporting post-disaster damage and loss assessments is necessary to help national and regional institutions collect information in a structured manner [ 38 ]. Accurate data on disaster damage and losses are crucial for effective risk management, including loss accounting, disaster forensics, and risk analysis [ 39 ]. Assessment information is pivotal for effective policy development, resource allocation, and disaster preparedness [ 40 ]. We can improve disaster management and link disaster management science to disaster risk reduction policymaking by using these data [ 38 ].

In the Kermanshah Earthquake Lessons Learned study conducted by Khankeh et al. (2018) in Iran, it was recommended that a standard protocol be established for the receipt of reports from disaster locations in the initial days and weeks following an earthquake. Moreover, the establishment of rapid assessment teams at the local, regional, and national levels, with specific guidelines, was considered a crucial step [ 41 ]. The composition of the assessment team depends on the sector to be assessed. Healthcare sector assessment teams from different disciplines, including public health experts, physicians, epidemiologists, architects, civil engineers, and health economists, can estimate the value of production losses [ 18 , 31 , 42 ].

The post-disaster damage and loss assessment methodology includes pre-disaster baseline data collection, disaster effects, impact analysis, recovery needs estimation, and strategies that recommend appropriate interventions, implementation arrangements, and policies [ 43 ]. Documentation of damage and loss assessments should begin as soon as possible after a disaster [ 44 ]. A post-disaster damage and loss assessment report is a live document that is revised as better data become available [ 45 ]. Post-disaster damage and loss assessment reports should differ according to the assessment stage and type of disaster [ 46 ].

There is the fact that all societies and countries are susceptible to disasters. The primary responsibility for disaster and emergency management is affected by local communities and countries. After a disaster, the healthcare sector faces multiple hazards, limited resources for dealing with them, and high expectations regarding their performance. Multiple stakeholders engage in post-disaster damage and loss assessments and their interventions are guided by various damage and loss assessment methods. Such variations in techniques and related assessment outputs challenge the comparability across assessments and often present conflicting images. Despite the long history of reconstruction in Iran, there are numerous challenges in assessing post-disaster damage and loss. Therefore, there is a need to develop a set of post-disaster damage and loss assessment frameworks, including methodologies and guidelines, for the healthcare sector.

Limited scientific resources for disaster damage assessment in the healthcare sector, access to imperative content, and documentation in the country were limitations of this study. As with all scoping reviews, we did not formally evaluate the quality of the evidence, and because of the varying nature of the studies, only a limited synthesis of results was possible.

In light of the pivotal role of the post-disaster healthcare sector, it is of the utmost importance to develop appropriate post-disaster damage and loss assessment programme that can be adapted to different socio-cultural contexts and varying resources. To date, there have been few studies that have discussed the entities and components of disaster damage and loss assessment programme in the healthcare sector. It was, however, determined that identifying the entities and components of the healthcare sector’s post-disaster damage and loss assessment program was a necessary step for advancing the healthcare sector in Iran. This review offers a detailed examination of post-disaster damage and loss assessment programs within the healthcare sector.

Data availability

All data generated or analysed during this study are included in this published article [and its supplementary information files].

Pourhosseini SS, Ardalan A, Mehrolhassani MH. Key aspects of Providing Healthcare Services in Disaster Response Stage. Iran J Public Health. 2015;44(1):111–8.

PubMed PubMed Central Google Scholar

UNISDR. Guidance Note on Recovery: Health. 2010. https://www.undrr.org/publication/guidance-note-recovery-health .

World Health Organization. Regional Office for the Eastern M. Implementation guide for health systems recovery in emergencies: transforming challenges into opportunities. Cairo2020. https://apps.who.int/iris/handle/10665/336472 .

Kimberley I, Shoaf SJR. Public Health Impact of Disasters: Australian Emergency Management Institute; 2000. https://search.informit.org/doi/ https://doi.org/10.3316/ielapa.369826620745919 .

Shahpari G, Ashena M, Shahpari M. How earthquakes can affect the Health Sector of the economy? Int J Economic Policy Emerg Economies. 2021;14(1):85–100.

Google Scholar

Hatami H, Razavi M-MSM, Eftekhar Ardabili MD-MPHH, Majlesi MD-MPHF, Sayed Nozadi MD-MPHM, PhD M. J. Parizadeh. Textbook of Public Health 4th Edition: Arjmand publication; 2019. http://phs.sbmu.ac.ir/uploads/VOLUME_3.htm .

Ardagh MW, Richardson SK, Robinson V, Than M, Gee P, Henderson S, et al. The initial health-system response to the earthquake in Christchurch, New Zealand, in February, 2011. Lancet. 2012;379(9831):2109–15.

Article PubMed Google Scholar

Irvin-Barnwell EA, Cruz M, Maniglier-Poulet C, Cabrera J, Rivera Diaz J, De La Cruz Perez R, et al. Evaluating disaster damages and operational status of Health-Care facilities during the emergency response phase of Hurricane Maria in Puerto Rico. Disaster Med Pub Health Prep. 2020;14(1):80–8.

Article CAS Google Scholar

Ochi S, Kato S, Kobayashi KI, Kanatani Y. The Great East Japan Earthquake: analyses of disaster impacts on Health Care clinics. Disaster Med Public Health Prep. 2018;12(3):291–5.

United Nations Development Programme - Headquarters, United Nations Office for Disaster Risk Reduction. International Recovery Platform. Guidance notes on recovery: health - Supplementary edition2017.

Carabine E. Revitalising evidence-based policy for the Sendai Framework for Disaster Risk Reduction 2015–2030: lessons from existing International Science partnerships. PLoS Curr. 2015;7.

Aitsi-Selmi A, Egawa S, Sasaki H, Wannous C, Murray V. The Sendai Framework for Disaster Risk reduction: renewing the global commitment to people’s resilience, Health, and well-being. Int J Disaster Risk Sci. 2015;6(2):164–76.

Article Google Scholar

Center ADR. Sendai framework for disaster risk reduction 2015–2030. Geneva, Switzerland: United Nations Office for Disaster Risk Reduction; 2015.

United Nations. Sendai framework for disaster risk reduction 2015. https://www.undrr.org/publication/sendai-framework-disaster-risk-reduction-2015-2030 .

United Nations General Assembly. Report of the open-ended intergovernmental expert working group on indicators and terminology relating to disaster risk reduction (A/71/644). 2016.

Kligerman M, Barry M, Walmer D, Bendavid E. International aid and natural disasters: a pre- and post-earthquake longitudinal study of the healthcare infrastructure in Leogane, Haiti. Am J Trop Med Hyg. 2015;92(2):448–53.

Article PubMed PubMed Central Google Scholar

Fejerskov AM, Clausen ML, Seddig S. Humanitarian ignorance: towards a new paradigm of non-knowledge in digital humanitarianism. Disasters. 2024;48(2):e12609.

Jovel RJM. Mohinder. Damage, Loss, and Needs Assessment Guidance Notes: Volume 1. Design and Execution of Damage, Loss, and Needs Assessment: World Bank, Washington, DC; 2010. https://openknowledge.worldbank.org/handle/10986/19047 .

Collaborative EP. Participatory planning guide for post-disaster reconstruction. EPC-Environmental Planning Collaborative, TCG International, LLC.; 2004. pp. 1–22.

Jovel JR, Mudahar MS. Conducting damage and loss assessments after disasters. The World Bank; 2010.

Chapin E, Daniels A, Elias R, Aspilcueta D, Doocy S. Impact of the 2007 Ica earthquake on health facilities and health service provision in southern Peru. Prehosp Disaster Med. 2009;24(4):326–32.

Achour N, Miyajima M. Post-earthquake hospital functionality evaluation: the case of Kumamoto Earthquake 2016. Earthq Spectra. 2020;36(4):1670–94.

Peters MD, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil H. Chapter 11: scoping reviews (2020 version). JBI manual for evidence synthesis. JBI. 2020.

Peters MDJ, Marnie C, Tricco AC, Pollock D, Munn Z, Alexander L, et al. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth. 2020;18(10):2119–26.

Components and entities of post-disaster Damage and loss Assessment Programme in the health sector: a Scoping Review Protocol [Internet]. Center for Open Science. 2022. https://doi.org/10.17605/OSF.IO/NJ3FK .

Miri J, Raeisi AR, Atighechian G, Seyedin H. Developing a conceptual model of post-disaster damage and loss assessment program in the Iranian health sector: a qualitative study protocol. BMJ Open. 2023;13(3):e065521.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for scoping reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169(7):467–73.

Clarke V, Braun V. Thematic analysis. J Posit Psychol. 2017;12(3):297–8.

Nowell LS, Norris JM, White DE, Moules NJ. Thematic analysis: striving to meet the trustworthiness criteria. Int J Qualitative Methods. 2017;16(1):1609406917733847.

Mosadeghrad AM, Rahimi-Tabar P. Health system governance in Iran: a comparative study. Razi J Med Sci. 2019;26(9):10–28.

UNDP. GFDRR, EU. Post-disaster needs assessments guidelines: Volume B - Health. 2014.

Organization WH. Monitoring the building blocks of health systems: a handbook of indicators and their measurement strategies. World Health Organization; 2010.

Haghdoost A, Dehnavieh R, Mehrolhssan MH, Abolhallaje M, Fazaeli AA, Ramezanian M. Future financing scenarios for Iran’s Healthcare System. Arch Iran Med. 2022;25(2):85–90.

Al-Harbi E, Zaghloul SS, editors. Swot analysis on cisco ® high availability virtualization clusters disaster recovery plan. 3rd International Conference on Digital Information Processing and Communications, ICDIPC 2013; 2013: Society of Digital Information and Wireless Communications (SDIWC).

Gufue ZH, Haftu HK, Alemayehu Y, Tsegay EW, Mengesha MB, Dessalegn B. Damage to the public health system caused by war-related looting or vandalism in the Tigray region of Northern Ethiopia. Front Public Health. 2024;12:1271028.

De Groeve T, Poljansek K, Ehrlich D. Recording Disaster Losses. Recommendations for a European Research JRC Scientific and Policy reports Joint Research Centre, European Commission. 2013.

Reduction UNISfD. Global Assessment Report on Disaster Risk reduction 2015. United Nations; 2015.

Corbane C, De Groeve T, Ehrlich D, Poljansek K. A European Framework for Recording and sharing disaster damage and loss data. Isprs J Photogrammetry Remote Sens. 2015;XL–3/W3:277–83.

De Groeve T, Corbane C, Poljanšek K, Ehrlich D. Current status and best practices for disaster loss data recording in the EU Member States. Publications Office of the European Union; 2014.

Giri S, Risnes K, Uleberg O, Rogne T, Shrestha SK, Nygaard OP, et al. Impact of 2015 earthquakes on a local hospital in Nepal: a prospective hospital-based study. PLoS ONE. 2018;13(2):e0192076.

Khankeh H, Kolivand PH, Beyrami Jam M, Rajabi E. Kermanshah Health Care Services: a lesson learned from Iran’s recent earthquake. Health Emergencies Disasters Q. 2018;3(4):221–33.

World B, European U, United N. Gaza Rapid Damage and needs Assessment, June 2021. Washington, DC: World Bank; 2021.

Book Google Scholar

Government of the Islamic Republic of Iran UCTiI. Post Disaster Needs Assessment (PDNA): Iran 2019 Floods in Lorestan, Khuzestan, and Golestan Provinces. 2019.

Boisvert S. Disaster recovery: mitigating loss through documentation. J Healthc Risk Manag. 2011;31(2):15–7.

Nepal government G, UNDP. Nepal Earthquake 2015: Post Disaster needs Assessment Vol. B: Sector Reports; 2015.

Institute NDR. Post-disaster Reconstruction and Rehabilitation National Plan. The National Disaster Management Organization; 2021.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

Download references

Acknowledgements

The authors express their gratitude to the Vice Chancellor for Research and Technology at Isfahan University of Medical Sciences for financial support.

This work was supported by the Vice-Chancellery of Research and Technology at Isfahan University of Medical Sciences, Isfahan, Iran [grant no. 3400686].

Author information

Authors and affiliations.

Student Research Committee, School of Management and Medical Information Sciences, Isfahan University of Medical Sciences, Isfahan, Iran

Social Determinants of Health Research Center, Isfahan University of Medical Sciences, Isfahan, Iran

Golrokh Atighechian

Department of Health in Disaster and Emergencies, School of Health Management and Information Sciences, University of Medical Sciences, Tehran, Iran

Hesam Seyedin

Health Management and Economics Research Center, Department of Health Services Management, Isfahan University of Medical Sciences, Isfahan, Iran

Ahmad Reza Raeisi

You can also search for this author in PubMed Google Scholar

Contributions

J. Miri contributed to study design, title, abstract and full-text screening, data extraction, data analysis, writing the first draft of the manuscript, and subsequent revisions of the manuscript. A.R. Raeisi contributed to study design, literature search and project management. G. Atighechian was also involved in drafting the abstract, full-text screening, data extraction and writing the manuscript at all stages. H. Seyedin contributed to the study design and drafting of the manuscript. All authors critically revised the manuscript and approved the final version for submission.

Corresponding author

Correspondence to Ahmad Reza Raeisi .

Ethics declarations

Ethics approval and consent to participate.

This study was approved by the Research Ethics Committee of the Isfahan University of Medical Sciences, Iran (IR.MUI.NUREMA.REC.1400.171).

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it.The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Miri, J., Atighechian, G., Seyedin, H. et al. Components and entities of post-disaster damage and loss assessment program in healthcare sector: a scoping review. BMC Public Health 24 , 2417 (2024). https://doi.org/10.1186/s12889-024-19523-5

Download citation

Received : 25 March 2024

Accepted : 18 July 2024

Published : 05 September 2024

DOI : https://doi.org/10.1186/s12889-024-19523-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Healthcare sector
Health planning

BMC Public Health

ISSN: 1471-2458

General enquiries: [email protected]

Research Guides

Hmb491: project in biotechnology industry.

Health Data and Statistics
Advanced Google Searching
Public Company Information
Patent Searching
Grey Literature Searching

What Is Grey Literature?

Grey Literature is any literature that has not been published through traditional means. It is often excluded from large databases and other mainstream sources. Grey literature can also mean literature that is hard to find or has inconsistent or missing bibliographic information.

Search grey literature to:

ensure that the review is as thorough as possible
find sources for negative results or brand new evidence
discover more references to published literature that your database search might have missed

CADTH's "Grey Matters" tool provides a list of organizations that produce grey literature. As well, the CRD guide and Chapter 6 of the Cochrane Handbook mentioned earlier in this guide include links to a number of grey literature sources.

Grey Literature Tutorial

Grey Literature Interactive Tutorial - 10 Minutes By the end of this interactive tutorial, learners should be able to: - define what grey lit is (and is not) - identify relevant sources of grey lit - search appropriate grey lit sources for your research question video... close... Note: some people have reported trouble accessing this module off campus. Please allow a few minutes for the module to load.

Develop your Grey Literature Search Strategy

Finding grey literature can be tricky! The strategy you'll develop to find grey literature is very question- and objective-dependent, and can require quite a bit of creativity and dogged determination. Download and use the template below to see strategies to find and document your search for grey literature.

How to Find & Document Grey Literature - Template

STEP 1 - IDENTIFY RELEVANT AUTHORITIES / STAKEHOLDERS

Look at who publishes and/or stores relevant documents
Review CADTH’s “Grey Matters” tool of health organizations
Ask research team members for suggested organizations
Browse the grey literature library page for sources of grey literature
Use the below honeycomb to brainstorm potential authorities, organizations, or stakeholders relevant to your topic

STEP 2 – SEARCH & DOCUMENTATION STRATEGIES

There are many different search strategies you can employ.

Targeted Website Browsing/Searching
Advanced Google Targeted website searching
Grey Literature Database Search
Search Engine Searching

The below links contain some grey literature resources for you to consider.

Policy Documents

Overton World's largest collection of policy documents, parliamentary transcripts, government guidance, and think tank research. Over 12M documents from 188+ countries and territories in many different languages with citing/cited by linkages.

Databases & Catalogues of Grey Literature

This list provides only a few producers and collectors of grey literature. Do a thorough search for sources in your field.

CADTH - Canadian Agency for Drugs and Technologies in Health Includes resources such as rapid response reviews, guidelines, and health technology assessments.
CIHI - Canadian Institute for Health Information
Health Canada
Health Quality Ontario (HQO), including Ontario Health Technology Assessment Series
National Technical Information Service Access to more than 3 million scientific, technical, engineering, and related business information from U.S. gov't
New York Academy of Medicine's Grey Literature Report A bimonthly collection of new grey lit resources. NYAM's website also hosts a list of international grey lit producers. NOTE: As of January 2017, the Grey Literature Report website and database will be discontinued and will no longer be updated, but the resources will still be accessible.
OAIster A catalogue with millions of records from open access sources.
OpenGrey A large database of European grey literature. APPEARS TO BE NO LONGER ACTIVE.
TRIP Pro This link opens in a new window Open-source medical search engine designed to provide evidence for clinical decision-making. Includes research evidence, images, videos, patient information leaflets, educational courses and news

Clinical Trials Databases

Health Canada's Clinical Trials Registry
ClinicalTrials.gov Produced by the U.S. National Institutes of Health.
WHO International Clinical Trials Registry Platform NGOs, such as WHO, are also great grey lit resources.
Current Controlled Trials A UK organization listing randomized controlled trials.
CenterWatch Trials are searchable by medical condition, therapeutic area, and geographical location.
Cochrane's CENTRAL A database of controlled trials compiled by Cochrane.
Best Practice Guidelines From the Registered Nurses Association of Ontario.
British Columbia
Canadian Agency for Drugs and Technologies in Health (CADTH) Unbiased, evidence-based information on drugs, devices, health care systems and best practices.
Cancer Care Ontario
Centre for Effective Practice
Disease Prevention and Control Guidelines From the Public Health Agency of Canada.
National Institute for Heath and Care Excellence - Guidance Guidance and advice based on best evidence to help you make decisions. UK.

Government Documents

Government bodies frequently publish reports and studies on topics relevant to health science. In Canada, government bodies at both the national and provincial levels produce relevant material.

Custom Google Search Formulated to retrieve Canadian government documents. Results can be filtered by the federal, provincial, and municipal levels.
Canadian Research Index This link opens in a new window
Public Policy: Key Government Resources Research Guide Lists important resources both outside and within Canada. The Canada page has a Health Policy section.
Canadian Government Publications Research Guide Also check the full list of government documents research guides for more Canadian and international resources.
U of T Library Catalogue U of T holds many individual government documents. Try searching with key terms, or use the name of a government body as an author search to find its publications.
FedSys US Government Printing Office Catalog. Will include docs from federal health related agencies.
National Technical Information Service Reports Library Through NTRL, NTIS provides a comprehensive offering that delivers high-quality US government technical content.
Publications Canada Health Canada, PHAC and others. Serials, monographs and some reports.

Institutional Repositories

Many institutions have institutional repositories , online databases of publications by their members. These can include publications by faculty and student dissertations and theses.

University of Toronto TSpace This link opens in a new window The University of Toronto's research bank. Showcases and preserves the scholarly work of the U of T community, and makes theses and dissertations publicly accessible online.
Canadian Institutional Repositories A list maintained by the Canadian Association of Research Libraries (CARL).
OpenDOAR A list of repositories (academic and otherwise) that are open access and freely available.
Registry of Open Access Repositories Another list similar to OpenDOAR.

Conferences

Conferences are nodes of new research, often featuring studies before they appear in journals.

Web of Science: Core Collection This link opens in a new window
BIOSIS Citation Index This link opens in a new window Citation information including abstracts of journals, books, conference papers, and technical reports, in the fields of life sciences and biology.
Scopus A multidisciplinary database that includes “Conference Paper” in its Document Type menu.
Proquest Conference Papers Index Focuses on life and environmental sciences. Holds citations for both papers and poster sessions.
UTM's Research Guide on Searching for Conference Proceedings
U of T's Subject Guide on Conference Papers

Theses and Dissertations

Students conduct both systematic reviews and original studies for their theses and dissertations. These texts are not usually included in major databases.

U of T’s Theses and Dissertations in the Sciences research guide is an excellent resource, with links for both U of T theses and dissertations and those from other institutions.

Sources for Authorities

When searching for grey literature, it is recommended to browse individual websites of identified authorities on your subject. Use the links below to find websites for specific governmental agencies and departments.

Public Health Offices / Regional Health Authorities Lists of public health offices and regional health authorities websites across Canada according to PHAC organizational structure
Government of Canada - Departments and Agencies Here is an alphabetical list of links to current Government of Canada Departments, Agencies, Crown Corporations, Special Operating Agencies and other related organizations.
EPA - Health and Environmental Agencies of U.S. States and Territories Links to health and environmental agencies organized alphabetically by state.
Hosp itals in selected countries Provides information on hospitals/country by Statista
HealthForceOntario - Allied Health Professional Associations A list of provincial and national associations/societies and regulatory colleges that provide additional information for allied health professionals.
Charities Listing Use this listing from the Canada Revenue Agency to find registered charities doing work in your area of research.

Contacting Experts and Authors

Often, writers of systematic reviews find that some data is left out of studies. In such cases, a good option is to contact the author of the study.

Experts in the field can also be rich sources of information. Talk to an expert to find out:

Studies or information you have missed
New aspects of your research topic to consider
Sources of grey lit

Negative Results

Journal of negative results in biomedicine Open access, peer-reviewed, online journal
PLOS: The Missing Pieces: A Collection of Negative, Null and Inconclusive Results Publication of negative, null and inconclusive results
The All Results Journals Focus on recovering and publishing negative results, valuable pieces of information in science in the fields of chemistry, biology, nanotechnology and physics.
Journal of Pharmaceutical Negative Results Peer reviewed journal
International Journal of Negative & Null Results

Newspapers and Magazines

Black and white stack of newspapers. creative commons 4.0; free to share and adapt.

Newspapers and magazines can be helpful sources of non-academic information. While the information in newspapers might not be acceptable as a source of scientific evidence, newspapers can often point you to key references or provide a source of evidence for public opinion.

U of T's research guide on Newspapers includes thorough information and some databases that also index magazines. You can also look up individual magazines in the library catalogue to determine our access.

Searching the Web

Sometimes, the best way to find grey lit is to search the Web. Different search engines have their own search algorithms that will pick up different results.

Another useful search engine is Duck Duck Go. Duck Duck Go does not collect user information and therefore results are not filtered based on your personal profile.

Advanced Google

Use advanced search in google.

Look at the "Get More Out of Google" infographic for the full details

Optimize your google searches:.

1. Combine keywords with boolean

OR : pull results that include either word [children or child or adolescent or young people]

2. Use quotation marks ( " " ) to search for phrases

Use around a phrase of two or more words ["alcohol abuse", "addictive behavior"]

3. Search a specific website rather than the entire Internet using site:

Get results from certain sites or domains [site:www.canada.ca/en/public-health]
To get results from multiple sites or domains, combine with OR [ site: www.canada.ca/en/public-health OR site:www.publichealthontario.ca]

4. Search for terms in location on webpage (intitle:)

Use intitle :term to restrict search results to documents containing term in the title.
For example, [intitle:addiction] will return documents that mention the word “addiction” in their titles

5. Search for particular types of documents (filetype:)

Specify the file type that your require (ex. pdf.) and be sure that this is no space between colon and file type [filetype:pdf]

Example search:

"alcohol abuse" (site: www.canada.ca/en/public-health OR site:www.publichealthontario.ca) filetype:pdf

This work is openly licensed via CC BY-NC-SA 4.0 . For information on this guide contact Erica Nekolaichuk , Faculty Liaison & Instruction Librarian at the Gerstein Science Information Centre.

<< Previous: Patent Searching
Last Updated: Sep 6, 2024 1:27 PM
URL: https://guides.library.utoronto.ca/HMB491

University of Toronto Libraries 130 St. George St.,Toronto, ON, M5S 1A5 [email protected] 416-978-8450 Map About web accessibility . Tell us about a web accessibility problem . About online privacy and data collection .

Connect with us

Open access
Published: 04 September 2024

Emergency pediatric patients and use of the pediatric assessment triangle tool (PAT): a scoping review

Tore A. G. Tørisen 1 ,
Julie M. Glanville 2 ,
Andres F. Loaiza 3 , 4 &
Julia Bidonde 5

BMC Emergency Medicine volume 24 , Article number: 158 ( 2024 ) Cite this article

1 Altmetric

Metrics details

We conducted a scoping review of the evidence for the use of the Pediatric Assessment Triangle (PAT) tool in emergency pediatric patients, in hospital and prehospital settings. We focused on the psychometric properties of the PAT, the reported impact, the setting and circumstances for tool implementation in clinical practice, and the evidence on teaching the PAT.

We followed the Joanna Briggs Institute methodology for scoping reviews and registered the review protocol. We searched MEDLINE, PubMed Central, the Cochrane Library, Epistemonikos, Scopus, CINAHL, Grey literature report, Lens.org, and the web pages of selected emergency pediatrics organizations in August 2022. Two reviewers independently screened and extracted data from eligible articles.

Fifty-five publications were included. The evidence suggests that the PAT is a valid tool for prioritizing emergency pediatric patients, guiding the selection of interventions to be undertaken, and determining the level of care needed for the patient in both hospital and prehospital settings. The PAT is reported to be fast, practical, and useful potentially impacting overcrowded and understaff emergency services. Results highlighted the importance of instruction prior using the tool. The PAT is included in several curricula and textbooks about emergency pediatric care.

Conclusions

This scoping review suggests there is a growing volume of evidence on the use of the PAT to assess pediatric emergency patients, some of which might be amenable to a systematic review. Our review identified research gaps that may guide the planning of future research projects. Further research is warranted on the psychometric properties of the PAT to provide evidence on the tool’s quality and usefulness. The simplicity and accuracy of the tool should be considered in addressing the current healthcare shortages and overcrowding in emergency services.

Review registration: Open Science Framework; 2022. https://osf.io/vkd5h/

Peer Review reports

Emergency medical services (EMS) are crucial to emergency care systems providing effective emergency medical care to people in need [ 1 ]. The World Health Organization (WHO) Emergency Care System Framework [ 2 ] (see Additional file 1) notes that effective emergency care involves a coordinated and integrated system of care, including the provision of prehospital care, transportation, and emergency department (ED) services. The WHO framework emphasizes the importance of early recognition of health issues and the timely provision of appropriate interventions to reduce morbidity and decrease the incidence of death and illness. Pediatric emergencies, particularly acute injuries and illnesses, generate considerable numbers of ambulance calls and ED visits in developed countries [ 3 , 4 ].

There is a general understanding that lack of pediatric emergency flow (or crowding) may lead to adverse outcomes for the child. However, the prevalence of pediatric emergencies poses significant challenges to emergency healthcare providers [ 5 , 6 ]. In the UK, pediatric emergencies represent 5–10% of all emergencies [ 7 ] and in the USA, children represent 20% of ED patients [ 8 ]. Injuries are the leading cause of morbidity and mortality among children and adolescents [ 9 , 10 ].

Caring for critically ill or injured pediatric patients can be challenging for emergency healthcare workers (EHWs) [ 11 ]. Patients’ histories may be difficult to obtain if the patient cannot provide verbal information or has been found alone without a caregiver [ 12 ]. Taking vital signs can be difficult and may not provide accurate information due to normal age-based variations [ 12 ]. Furthermore, some EHWs may have not received training in pediatric emergencies, which can be stressful [ 13 ].

Despite these challenges, EHWs need to conduct a rapid and accurate assessment of the pediatric patient to deliver timely effective emergency treatment. EHWs also need to reassure patients and caregivers and bring order to potentially chaotic situations. EHWs who lack specialized training in pediatric emergencies may unintentionally exacerbate stressful situations [ 13 ]. Emergency pediatric training for healthcare professionals inside and outside of the hospital is essential to ensure the best outcomes for critically ill or injured pediatric patients [ 14 , 15 ].

Emergency triage involves quickly identifying patients who require medical attention to prioritize treatment efficiently for those in greatest need [ 14 ]. Triage tools such as the Manchester Triage System and the Emergency Severity Index are helpful [ 16 ]. The Paediatric Canadian Triage and Acuity Scale (PaedCTAS) was developed specifically for pediatric patients [ 17 ], using the Pediatric Assessment Triage (PAT) tool as the first step in assessing emergency patients. It includes the “general impression” stage using the PAT, primary assessment with the airway, breathing, circulation, disability, and exposure (ABCDE) approach [ 18 ], secondary assessment, diagnostic assessment, and reassessment.

The Pediatric Assessment Triangle (PAT)

The PAT is used to quickly identify critically ill or injured children needing immediate medical attention. It focuses on three presenting components (“arms”): appearance, work of breathing, and circulation (Fig. 1 ). It can be used in prehospital or hospital settings for efficient rapid assessment of the patient's level of consciousness, breathing, and circulation, without requiring hands-on assessment or equipment [ 5 , 19 ]. It can help identify key pathophysiological problems and whether urgent transport or resources are needed. The PAT assessment takes 30–60 s [ 5 , 19 ] and it can be performed remotely (a “through the room” assessment).

The Pediatric Assessment Triangle components (arms). Figure adapted from Fuchs S and McEvoy M [ 20 ]]

Scoping review aim and design

Give the current shortage of healthcare personnel worldwide, and overcrowding of emergency departments, gathering of the PAT’s evidence is essential. This review aimed to identify the available scientific evidence about the PAT and its use by EMS. Our objective was to complete a scoping review within the pre-and-hospital care to synthesize:

What are the psychometric properties of the PAT (e.g., validity, reliability, applicability)?

What are the reported impact(s) of the PAT? (e.g., improved triage, cost, better clinical outcomes)

What are the requirements or circumstances for PAT implementation in clinical practice?

What is the evidence on the value of teaching EMS workers about PAT?

We followed the Joanna Briggs Institute framework for scoping reviews [ 21 ]. The review protocol was registered [ 22 ]. The review is reported according to the PRISMA extension for scoping reviews [ 23 ] (Additional file 2).

Eligibility criteria

Eligible publications (Table 1 ) reported the use of the PAT with pediatric populations in prehospital, hospital or training settings. Eligible outcomes matched our specific aims as follows: 1) psychometric performance, 2) impact(s), 3) implementation of PAT utilization, and 4) evidence on teaching the PAT.

We searched MEDLINE (PubMed), PubMed Central (via LitSense), the Cochrane Library, Epistemonikos, Scopus and CINAHL, from 1995 to July 2022, to include publications before the introduction of the PAT in the curricula of Pediatric Education for Prehospital Professionals (PEPP) and Advanced Pediatric Life Support (APLS) in 2000 [ 24 ]. The database searches were conducted from 24 to 28 July 2022. Fourteen websites of organizations involved in policy making in emergency pediatrics were searched between 6 and 10 August 2022. We searched for unpublished (grey) literature using Grey Literature Report ( http://www.greylit.org/ ) and Lens.org ( https://www.lens.org/ ). Full searches are presented in additional file 3.

Study selection process

We deduplicated records in EndNote and conducted double independent screening (TT, AFL-B) in Covidence (Veritas) against the eligibility criteria (Table 1 ). Conflicts were resolved by consensus or arbitrated by a third reviewer (JB). Additional file 4 lists records excluded at full text with reasons. Records reporting the same study were grouped and we cite the earliest publication while presenting relevant data from any of the related publications.

Data collection process

Data were extracted from eligible studies into a Microsoft 365 Excel form which was piloted on a random sample of five included studies, and modified as required based on feedback from the team [ 22 ]. One reviewer (TT) completed data extraction and a second reviewer (AFL-B) verified the extracted data. Disagreements were resolved by consensus or arbitrated by a third reviewer (JB). Risk of bias was not assessed [ 21 ].

Knowledge user (KU)/patient engagement and methodological appraisal

We defined KU/ patient engagement as individuals who may be affected by the research findings. Since this review was time sensitive, we did not recruit knowledge users or patients.

We did not appraise methodological quality or risk of bias of the included articles, which is consistent with guidance on scoping review conduct.

The synthesis included quantitative (e.g. psychometric properties) and qualitative analyses (e.g. content analysis) of the components of the impact, implementation and teaching. A word cloud was drawn for the impact of the PAT using the online program WordClouds. The team members identified, coded, and charted relevant units of text from the articles using a framework established a priori as a guide. The framework was developed through team discussions upon reviewing the preliminary results. Data were grouped by question and overviews are provided using charts and tables generated using Microsoft 365 Excel.

Search results and publication characteristics

The searches identified 548 records (Fig. 2 ). Fifty-five publications were included (full citations listed in Additional file 5) of which three were books. Sixteen publications were in non-English languages, but with English abstracts, and of these we retrieved 14 full text publications (Spanish ( n = 9), German ( n = 2), French ( n = 1), Turkish ( n = 1), and assumed Taiwanese Mandarin ( n = 1)). Of these, there were seven papers that described the psychometric properties of the PAT, 18 were about the PAT’s impact, 38 described implementation pros and cons, and 30 provided references to the PAT used in educational/training environments. The publication dates ranged from 1999 to 2022, representing 18 countries with the majority classified as "high income" (World Bank classification) [ 25 ] (see Additional file 6). Study designs were diverse: primary research ( n = 27, 49.1%), secondary research ( n = 4, 7.3%), and "other" ( n = 24, 43.6%). We identified no randomized controlled trials, systematic reviews, or scoping reviews.

PRISMA flow chart

Psychometric properties

The seven papers reporting psychometric properties were as follows. Four studies (Table 2 ) reported sensitivity and specificity, measuring test accuracy [ 26 , 27 , 28 , 29 ], of which one study reported an area under the receiver operating characteristic curve (AUROCC) [ 29 ] and four studies reported likelihood ratios (LR) [ 26 , 27 , 28 , 30 ].

PAT sensitivity (Fig. 3 ) ranged from 77.4% to 97.3% (four studies) suggesting it can accurately identify a large proportion of patients with the targeted condition [ 26 , 27 , 28 , 29 ]. Specificity, measuring a test's ability to correctly identify patients without the condition, ranged from 22.9% to 99.15% (four studies) [ 26 , 27 , 28 , 29 ].

PAT sensitivity and specificity

One study evaluated the PAT’s validity and reliability [ 31 ] by collecting data for 157 patients triaged by a single trained observer and an “enfermera clasificadora” (classifying nurse). This single pair showed high inter-observer agreement in applying the PAT and no errors associated with polypnea, pre-existing pallor, or irritability.

Likelihood ratios (LR) measure a test’s diagnostic accuracy which are less likely to change with the prevalence of a disorder. A positive LR (LR +) indicates a positive test result is more likely in people with the condition and a negative LR (LR-) indicates that a negative test result is more likely in people without the condition of interest. One study reported LR + of 5.2 (95% CI 5–7.8) [ 26 ] with a statistically significant high odds ratio (OR 111, 95% CI 73–168.6; p < 0.001), indicating the PAT has a high ability to correctly identify and classify initial severity of disease during triage. A second study reported a LR + of 7.7 (95% CI 5.9–9.1) [ 27 ]. A third study triaged 1002 children using the PAT, reporting a LR + of 0.12 (95% CI 0.06–0.25) for children deemed stable by the PAT ( n = 200) [ 28 ]. This study’s results for categories of pathophysiology (respiratory distress, respiratory failure, shock, central nervous system/metabolic disorder, and cardiopulmonary failure) highlighted the need to consider the clinical scenario when interpreting the PAT in EMS. However, the moderate LR- value (0.22, 95% CI 0.18–0.26) indicated that the test is less able to correctly identify children who do not need urgent care. The study reported a LR- of 0.12 (95% CI 0.06–0.25) for children found to be stable by the PAT ( n = 802) [ 28 ]. The LR- values for children with the five specified categories of pathophysiology suggest the PAT has relatively low LR for identifying respiratory distress and shock, indicating it is better at ruling out those conditions. However, the relatively high LR- for respiratory failure and cardiopulmonary failure suggests the PAT is less effective at ruling out those conditions.

One study (2017) found that abnormal PAT results were associated with an increased risk of admission to the hospital (OR 5.14, 95% CI 4.98–5.32; p < 0.01) [ 30 ]. Abnormal appearance (OR 3.99, 95% CI 3.63–4.38) or having one or more components of the PAT (OR 14.99, 95% CI 11.99–18.74) were significantly associated with hospital admission [ 30 ]. The study identified adjusted age (OR 4.44, 95% CI 3.77–5.24; p < 0.001) and triage (OR 1.78, 95% CI 1.72–1.84; p < 0.001) as independent risk factors for intensive care unit admission and longer stays in the pediatric ED [ 30 ]. One study reported the PAT performed similarly to the Pediatric Early Warning Score (PEWS) (AUROCC 0.963 (PAT) and 0.966 (PEWS); x 2 = 0.10; p = 0.74) [ 29 ].

Four studies reported high levels of reliability in PAT results [ 27 , 28 , 29 , 32 ]. One study reported 93.6% reliability (Kappa index 0.7, 95% CI 0.5–0.8) [ 29 ]. A second study found paramedics used the PAT highly consistently across its three arms (Kappa 0.93, 95% CI 0.91–0.95) [ 32 ] and the paramedics’ impression, completed using PAT on first contact with the patient, showed substantial agreement with the investigators’ retrospective chart review on diagnosis and disposition (Kappa 0.62, 95% CI 0.57–0.66) and categorization of stable versus unstable (Kappa 0.66, 95% CI 0.62–0.71). A third study reported substantial inter-rater reliability agreement on PAT scores ( n = 1002, two pediatric emergency physicians and a pediatric nurse practitioner) (Fleiss' κ 0.7, p < 0.001) [ 28 ]. A fourth study reported an agreement rate of 93.24% between the PAT and the condition of sick children [ 29 ].

Reported impacts of the PAT

Eighteen publications reported on impacts after PAT implementation; the word cloud of impact names is display in Fig. 4 . Terms most used were “triage –communication -vocabulary and care”.

The PAT reported impact

Impact reported were on mortality, safety, effectiveness of care, timeliness of care, triage, and communication [ 27 , 28 , 29 , 30 , 31 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 ]. Three studies showed the ability of the PAT to correctly assess critical cases (e.g. higher risk of mortality in patients with sepsis with an altered or unstable PAT) [ 33 , 34 , 36 ]. Two studies found that PAT helped to avoid unnecessary interventions or potential harm to patients [ 27 , 35 ]. One study reported that a normal PAT result did not exclude severe infections, and a proper examination was still necessary to diagnose emergency pediatric patients [ 33 ]. One study reported that the PAT was timely and rapid to apply (mean 32.4 s) [ 31 ] and two studies reported that the PAT was equally effective, but faster and easier to use, than the PEWS in predicting critical illness in pediatric patients [ 29 , 38 ].

Communication and documentation were another way the PAT’s impact were reported. The PAT’s “general impression” aided in care communication and helped prioritize management options. The specific vocabulary to describe a patient’s vital signs and physical findings allowed for easy documentation and transfer/flow of information between EHWs [ 27 , 28 , 37 ]. Two studies highlighted the power of a common vocabulary in EMS replacing subjective comments with specific assessments [ 27 , 28 ].

Studies offered insights into achieving optimal triage outcomes using the PAT. One study demonstrated the PAT’s usefulness when classifying non-urgent patients [ 40 ] and a second noted the importance of setting severity and prioritization criteria (1 to 5 depending on severity) and using the PAT to ensure proper attention [ 45 ].

Abnormal PAT findings helped to identify patients with a higher risk of hospitalization [ 30 ] and enabled earlier interventions for high-risk patients [ 42 ]. One study used the PAT for children experiencing secondary complications to hematopoietic cell transplantation [ 44 ] and reported that an unstable PAT, along with other factors, accurately predicted the need for admission (relative risk 3.4, 95% CI 2.6–4.6; p < 0.001). A study investigated features of 17,243 cases referred from in-hospital areas to the pediatric ED (median age 42 months (range: 0–120)); 65% of transferred patients were PAT-assessed as stable [ 41 ]. One study assessed the PAT as a discriminator in the triage classification system and assessed the correlation between pathophysiological diagnosis and triage classification [ 31 ]. Four studies suggested the PAT was considered practical and helpful in identifying emergency pediatric patients in need of intervention and identifying the probable underlying cause of illness [ 26 , 28 , 38 , 46 ]. Treatment priorities were met in children with fever, and to a lesser extent for pain, respiratory distress, and oxygen needs.

One study concluded that an abnormal PAT and a more severe triage level (I-III) were independent factors in identifying asthmatic children requiring hospitalization and longer stays [ 43 ]. One study suggested that the PAT did not perform well for patients with anaphylaxis and as a result patients did not receive timely interventions [ 39 ].

We found no data for impacts on pediatric readmission, patient/caregiver experience, or provider burnout.

Setting and circumstances for PAT implementation

Ten studies evaluated pre-hospital triage using the PAT [ 6 , 20 , 27 , 30 , 38 , 47 , 48 , 49 , 50 ] and 28 evaluated hospital triage [ 24 , 26 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 ]. No studies reported PAT use in emergency call centers or telemedicine services. One study noted that the PAT may be implemented by midwives working in hospitals or prehospital settings [ 37 ]. A study of 391 admissions reported PAT was considered a useful triage tool in resource-poor hospitals [ 52 ].

Four studies recommended formal training on using the PAT as necessary for effective use [ 27 , 28 , 45 , 47 ]. One study ( n = not reported) found that a low utilization rate for the PAT (patient report forms collected over a three-month period) following its introduction increased significantly following training in PAT use (12% vs 63.3%) [ 47 ]. After implementation, one study reported that the 30 emergency nurses involved preferred using the PAT over the PEWS when assessing emergency pediatric patients [ 29 ]. In a study of the Advanced Pediatric Life Support (APLS) course, attendees considered the systematic assessment approach incorporating the PAT crucial to their clinical practice, highlighting the importance of training prior implementation [ 54 ]. Studies acknowledged that applying the PAT with young infants (7–89 days old) was challenging [ 33 ], implementing the PAT requires skills, on-site senior emergency pediatric care providers, and a pediatric-friendly environment [ 59 ] and that the feasibility of the PAT is promising, but further research for “clinical validation” (not further defined) was needed [ 30 ].

We found no information about the implementation of PAT in clinical guidelines, requirements for recertification after PAT implementation, cost of implementation, or sustainability.

Teaching the PAT

Thirty studies presented data on teaching PAT to EHWs as follows: an early report suggested that the PAT was ideal for pediatric life support courses in all settings, based on its simplicity and reproducibility for both teachers and clinicians [ 60 ]. The PAT is included in one textbook of general emergency pediatrics [ 61 ] and two textbooks for emergency pediatric care in the prehospital environment [ 20 , 62 ]. Courses for EHWs on pediatric life support have incorporated the PAT for the “first impression” assessment, as well as training on the use of the PAT tool itself [ 29 , 30 , 63 ].

Methods for teaching the PAT tool included classroom-based, use of simulation, use of virtual reality and video for case training [ 54 , 64 , 65 ]. The PAT has been recommended as a teaching tool for the goal-directed management of shock in children [ 66 ].

The number of people who have received PAT training is unknown, but more than 170,000 EHWs had received formal training up to 2010 (worldwide) [ 63 ]. The numbers of EHWs trained in the studies ranged from 30 to 1520 [ 29 , 54 ].

Eighteen studies reported the care of emergency pediatric patients and provide insights into best practices for care which can, in turn, inform educational programs or be used to develop evidence-based protocols [ 30 , 37 , 48 , 49 , 50 , 56 , 57 , 59 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 ]. Four publications describe how emergency care providers use the PAT to assess emergency pediatric patients generally or with specific medical problems [ 30 , 49 , 59 , 67 ].

We identified 55 documents reporting the use of the PAT in hospital and pre-hospital emergency pediatric care. Research indicates that the PAT is a valid and reliable tool for evaluating emergency pediatric patients, prioritizing interventions, and determining the appropriate level of care. EHWs found the PAT is fast and practical, akin to the intuitive ‘gut feeling’ of experienced clinicians., but they should complete formal training before implementing the PAT. Several emergency pediatric care course curricula and key textbooks include the PAT.

We found only seven publications on the PAT’s psychometric properties, which suggest that the PAT has good sensitivity and some variability in specificity. The low research volume may reflect ethical challenges around research involving children, the unique and unpredictable nature of emergency situations, the impossibility of controlling all variables and difficulties in obtaining funding [ 77 ]. Research on psychometric properties can be expensive and funding for pediatric-focused psychometric research may not be a priority for research funders. The PAT’s ease of use may have contributed to its rapid adoption in practice before adequate psychometric testing was conducted and published. Implementing the PAT may still be challenging in terms of training or resistance to change [ 47 ]. Despite the challenges of research in the emergency setting, a third of the included studies reported positive impacts when using the PAT, suggesting its potential for triaging and improving patient outcomes in clinical settings which merits further investigation in an era of emergency department overcrowding and shortages of healthcare personnel.

Other tools are also used for emergency pediatric assessment (e.g., the Pediatric Glasgow Coma Scale, the PEWS, and the Pediatric Vital Sign Score) and each has its strengths and limitations. Choosing a tool depends on the specific circumstances and the healthcare provider's expertise. Based on the included comparative studies, the PAT is often favored for its simplicity, rapidity, and ease of use in remote or face-to-face emergency settings, since it does not require hands-on assessment or the use of specialized equipment. The available research and comparative studies merit further investigation.

Evidence was identified on training EHWs to use the PAT to assess accurately a child's appearance, work of breathing, and circulation. Proficiency is needed in using the tool and there is a need to use it regularly, to maintain their knowledge. While the PAT can provide a quick snapshot of a child's overall condition, it is only one part of a comprehensive assessment, and EHWs should use additional tools and techniques to assess a child's condition. Online courses, in-person workshops, and continuing education courses offered by professional organizations as well as guides or manuals with step-by-step instructions on how to use the PAT are all available. Healthcare providers who are considering preparing or updating their PAT training, perhaps using simulation-based approaches, should review these sources of evidence-based training [ 78 ].

The main challenges to PAT instruction noted to date are the limited provision of hands-on experience (i.e. real-life emergency situations), limited feedback on site to the EHW on their performance (to enable them to identify and correct areas of weakness in their assessment skills) and lack of standardization in the training programs. Skill decay is problematic as EHWs may forget the PAT steps without regular use. Re-certification requirements depend on the EHW’s professional organization and any employer’s certification requirements.

Although research evidence seems to show that the PAT is considered a valuable tool for rapid assessment of the status of a distressed patient, and its simplicity makes it easy to implement across a range of settings, we identified limited evidence on using the PAT in low-income settings [ 52 , 79 ]. Resource-limited settings may lack coordinated emergency systems including at the scene aid, a system of triage, emergency medical care and critical care [ 80 ]. In these situations, different approaches to pediatric assessments may be adopted, limited data may be recorded on the frequency and quality of PAT assessments [ 81 ] and access to PAT training may be limited. Workforce shortages can impact the availability of trained EHWs to provide PAT instruction. Despite the limited evidence, we anticipate that the PAT is still a feasible tool for EHWs with limited resources [ 52 ]. The PAT’s simplicity can be helpful in rural areas, remote communities, and resource-limited clinics. Based on evidence from this review, the PAT provides a practical and effective way for EHWs to assess children in emergency situations and make informed decisions about their care.

Limitations

This scoping review has limitations. Firstly, we focused on English language articles and there may be additional full text publications in non-English languages that might have provided information on low- and middle-income countries’ experiences of the PAT, its impact, or its psychometric properties. This scoping review was pragmatic, but a follow up review may identify additional studies in languages other than English. Secondly, the search for grey literature was conducted on 14 websites, was hampered by the varying quality (and sometimes absence) of website search engines and the list of websites was prepared by one author (TT). A full systematic review would ideally search a larger number of websites and other sources of grey literature to potentially identify further research, particularly for LMICs. and might have been enhanced by suggestions from experts in the field.

Options for a future systematic review and other areas of research

A full systematic review would likely focus on those research questions for which there are most data following the scoping review and would also include detailed data extraction as well as the grouping of studies by outcomes of interest to provide summaries of the evidence for each outcome. Scoping reviews typically do not conduct risk of bias assessments or evaluate publication bias. A future systematic review could include these steps to assess the strength and quality of the evidence for the use of the PAT.

Other areas for research identified are how the PAT affects pediatric readmissions, patient/caregiver experience, and provider burnout. This scoping review did not find evidence of implementation, that is requirements of recertification and costs or data on utilization for example use of the PAT by emergency call centers, assessments by videoconference or other telemedicine services. Evidence on the utilization of the PAT specific to different emergency transport services such as air medical services, disaster response, etc. was not found.

In summary, this scoping review shows that the PAT has been used in clinical settings for over 20 years. There is some evidence of its validity and reliability, impacts and that the tool is broadly accepted by EHWs. Although the PAT condenses years of experience into a practical and useful assessment suitable for use by less experienced personnel, the need for prior training and certification was highlighted. Although there are gaps in the literature, the evidence has increase in recent years. Scoping reviews are used to inform research agendas and identify implications for policy or practice. As such, psychometric tool data are imperative. Further research on impact and implementation is warranted, and in particular, there is a need to standardize the teaching of PAT teaching and its certification. The simplicity, friendliness and low resources requirement of the tool should be considered in addressing the current healthcare shortages and overcrowding in emergency services.

Availability of data and materials

All data generated or analysed during this study are included in this published article [and its additional information files].

Kobusingye OC, Hyder AA, Bishai D, Joshipura M, Hicks ER, Mock C. Emergency medical services. In: Jamison DT, Breman JG, Measham AR, et al, editors. Disease control priorities in developing countries 2nd ed. Washington, DC: International Bank for Reconstruction and Development / The World Bank; 2006. https://www.ncbi.nlm.nih.gov/books/NBK11744/ .

World Health Organization (WHO). WHO Emergency care system framework infographics Geneva: World Health Organization; 2018. Available from: https://www.who.int/publications/i/item/who-emergency-care-system-framework . Updated 2 May 2018; cited 2023 11 April.

Andersen K, Mikkelsen S, Jørgensen G, Zwisler ST. Paediatric medical emergency calls to a Danish emergency medical dispatch centre: a retrospective, observational study. Scand J Trauma Resusc Emerg Med. 2018;26(1):2.

Article PubMed PubMed Central Google Scholar

Lee LK, Porter JJ, Mannix R, Rees CA, Schutzman SA, Fleegler EW, et al. Pediatric traumatic injury emergency department visits and management in US children’s hospitals from 2010 to 2019. Ann Emerg Med. 2022;79(3):279–87.

Article PubMed Google Scholar

Fuchs S. The origins and evolution of emergency medical services for children. Pediatr Ann. 2021;50(4):e150–4.

Fuchs S, Yamamoto L, editors. APLS: the pediatric emergency medicine resource. 5th ed. Burlington: Jones & Bartlett Learning; 2012. p. 538.

Houston R, Pearson GA. Ambulance provision for children: a UK national survey. Emerg Med J. 2010;27(8):631–6.

McDermott KW, Stocks C, Freeman WJ. Overview of pediatric emergency department visits, 2015: Statistical Brief #242. Rockville: Agency for Healthcare Research and Quality; 2018.

Google Scholar

Nesje E, Valoy NN, Kruger AJ, Uleberg O. Epidemiology of paediatric trauma in Norway: a single-trauma centre observational study. Int J Emerg Med. 2019;12(1):18.

United Nations Children's Fund (UNICEF). Levels and trends in child mortality. Report 2022. New York: UNICEF; 2023. Available from: https://data.unicef.org/resources/levels-and-trends-in-child-mortality/ .

Jeruzal JN, Boland LL, Frazer MS, Kamrud JW, Myers RN, Lick CJ, et al. Emergency medical services provider perspectives on pediatric calls: a qualitative study. Prehosp Emerg Care. 2019;23(4):501–9.

Nordén C, Hult K, Engström Å. Ambulance nurses’ experiences of nursing critically ill and injured children: a difficult aspect of ambulance nursing care. Int Emerg Nurs. 2014;22(2):75–80.

Hansen M, Meckler G, Dickinson C, Dickenson K, Jui J, Lambert W, et al. Children’s safety initiative: a national assessment of pediatric educational needs among emergency medical services providers. Prehosp Emerg Care. 2015;19(2):287–91.

Li J, Roosevelt G, McCabe K, Preotle J, Pereira F, Takayesu JK, et al. Critically ill pediatric case exposure during emergency medicine residency. J Emerg Med. 2020;59(2):278–85.

Ralston ME, Zaritsky AL. New opportunity to improve pediatric emergency preparedness: pediatric emergency assessment, recognition, and stabilization course. Pediatrics. 2009;123(2):578–80.

Roukema J, Steyerberg EW, van Meurs A, Ruige M, van der Lei J, Moll HA. Validity of the manchester triage system in paediatric emergency care. Emerg Med J. 2006;23(12):906.

Article CAS PubMed PubMed Central Google Scholar

Yates MT, Ishikawa T, Schneeberg A, Brussoni M. Pediatric Canadian Triage and Acuity Scale (PaedsCTAS) as a measure of injury severity. Int J Environ Res Public Health. 2016;13(7):659.

Thim T, Krarup NH, Grove EL, Rohde CV, Lofgren B. Initial assessment and treatment with the Airway, Breathing, Circulation, Disability, Exposure (ABCDE) approach. Int J Gen Med. 2012;5:117–21.

Shah MN. The formation of the emergency medical services system. Am J Public Health. 2006;96(3):414–23.

Fuchs S, McEvoy M, editors. Pediatric education for prehospital professionals. 4th ed. Burlington: Jones & Bartlett Learning; 2021. p. 490.

Peters MDJ, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil H. Chapter 11: scoping reviews. In: JBI manual for evidence synthesis. Joanna Briggs Institute; 2020. Available from: https://synthesismanual.jbi.global .

Tørisen TAG, Glanville J, Loaiza-Betancur AF, Bidonde J. Emergency pediatric patients and use of the pediatric assessment triangle (PAT) tool. Protocol for a scoping review. Charlottesville: Open Science Framework; 2022. https://www.osf.io/vkd5h .

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Fernandez A, Benito J, Mintegi S. Is this child sick? Usefulness of the pediatric assessment triangle in emergency settings. J Pediatr (Rio J). 2017;93(Suppl 1):60–7.

World Bank. New World Bank country classifications by income level: 2022-2023. The World Bank Group; 202. Available from: https://blogs.worldbank.org/opendata/new-world-bank-country-classifications-income-level-2022-2023 .

Avilés-Martinez KI, López-Enríquez A, Luévanos-Velázquez A, Jiménez-Pérez BA, García-Armenta MB, Ceja-Moreno H, et al. Triage, priorization tools of pediatric emergency room. Acta Pediatr de Mex. 2016;37(1):4–16.

Gausche-Hill M, Eckstein M, Horeczko T, McGrath N, Kurobe A, Ullum L, et al. Paramedics accurately apply the pediatric assessment triangle to drive management. Prehosp Emerg Care. 2014;18(4):520–30.

Horeczko T, Enriquez B, McGrath NE, Gausche-Hill M, Lewis RJ. The pediatric assessment triangle: accuracy of its application by nurses in the triage of children. J Emerg Nurs. 2013;39(2):182–9.

Ma X, Liu Y, Du M, Ojo O, Huang L, Feng X, et al. The accuracy of the pediatric assessment triangle in assessing triage of critically ill patients in emergency pediatric department. Int Emerg Nurs. 2021;58.

Fernández A, Ares MI, Garcia S, Martinez-Indart L, Mintegi S, Benito J. The validity of the pediatric assessment triangle as the first step in the triage process in a pediatric emergency department. Pediatr Emerg Care. 2017;33(4):234–8.

Lugo S, Pavlicich V. Application of the pediatric assessment triangle to the triage classification system in an emergency department. Rev Bol Ped. 2014;53(2):88–93.

Fernández A, Pijoan JI, Ares MI, Mintegi S, Benito FJ. Canadian paediatric triage and acuity scale: assessment in a European pediatric emergency department. Emergencias. 2010;22(5):355–60.

Ecclesia FG, Alonso Cadenas JA, Gómez B, Gangoiti I, Hernández-Bou S, de la Torre EM. Late-onset group B streptococcus bacteremia evaluated in the pediatric emergency department and risk factors for severe infection. Pediatr Infect Dis J. 2022;41(6):455–9.

Gomez B, Hernandez-Bou S, Garcia-Garcia JJ, Mintegi S. Bacteremia in previously healthy children in emergency departments: clinical and microbiological characteristics and outcome. Eur J Clin Microbiol Infect Dis. 2015;34(3):453–60.

Article CAS PubMed Google Scholar

Sánchez IA, Cotanda CP, Casas MM, de la Maza VTS, Cubells CL. Profile of the child seen in the resuscitation room. Rev Esp Salud Publica. 2019;93.

Shiva GS, Kumar VS, Kumar PR, Subramanian SB. A study on the role of paediatric assessment triangle, clinical scoring and serum lactate in the management of septic shock in children. Int J Contemp Pediatr. 2019;6(5):2037.

Article Google Scholar

Macnab AJ. Objective assessment and communication of the physiologic status of the sick infant. Can J Midwif Res Pract. 2004;3(2):7–12.

Mierek C, Nacca N, Scott JM, Wojcik SM, D’Agostino J, Dougher K, et al. View from the door: making pediatric transport decisions based on first impressions. JEMS. 2010;35(7):68–9, 71, 3, 5, 7, 9, 81.

PubMed Google Scholar

Arroabarren E, Alvarez-Garcia J, Anda M, de Prada M, Ponce MC, Palacios M. Quality of the triage of children with anaphylaxis at the emergency department. Pediatr Emerg Care. 2021;37(1):17–22.

Alp EE, Dalgic N, Yilmaz V, Altuntas Y, Ozdemir HM. Evaluation of patients with suspicion of COVID-19 in pediatric emergency department. Sisli Etfal Hastan Tip Bul. 2021;55(2):179–87.

PubMed PubMed Central Google Scholar

Derİnöz-Güleryüz O. In-hospital pediatric patient transfers to the pediatric emergency department. Cukurova Med J. 2022;47(1):332–40.

Kawai R, Nomura O, Tomobe Y, Morikawa Y, Miyata K, Sakakibara H, et al. Retrospective observational study indicates that the paediatric assessment triangle may suggest the severity of Kawasaki disease. Acta Paediatr. 2018;107(6):1049–54.

Paniagua N, Elosegi A, Duo I, Fernandez A, Mojica E, Martinez-Indart L, et al. Initial asthma severity assessment tools as predictors of hospitalization. J Emerg Med. 2017;53(1):10–7.

Alonso Cadenas JA, Corredor Andrés B, Andina Martínez D, et al. Characteristics and risk factors for admission in children undergoing hematopoietic cell transplantation in a pediatric emergency department. Authorea. 2021. https://doi.org/10.22541/au.163253914.42579466/v1 .

Rodríguez Borbolla FJ, Sancha Herrera ML, Ortiz Angulo E, Pulido PP. Implementación del sistema de clasificación en la Unidad de Urgencias Pediátricas del Hospital Marqués de Valdecilla. Fundacion de Enfermeria de Cantabria. 2013;2(9):26–31.

Suárez M, Jaime M. Utilidad del triángulo de evaluación pediátrica en un servicio de emergencia pediátrica. Boletin Medico de Postgrado. 2018;34(2):39–45.

Ogden K. The use of the paediatric assessment triangle in the management of the sick child. Emerg Med J. 2016;33(9):e4.

Romig LE. PREP for peds-patient physiology, rescuer responses, equipment, protocols. Size-up & approach tips for pediatric calls. JEMS. 2001;26(5):24–33.

CAS PubMed Google Scholar

Horeczko T, Gausche-Hill M. The paediatric assessment triangle: a powerful tool for the prehospital provider. J Paramed Pract. 2011;3(1):20–5.

Walker A, Hanna A. Kids really are just small adults: utilizing the pediatric triangle with the classic ABCD approach to assess pediatric patients. Cureus. 2020;12(3): e7424.

Morilla L, Morel Z, Pavlicich V. Clinical characteristics of pediatric patients with COVID-19 in an emergency department. Pediatría (Asunción). 2020;47(3):124–31.

Akindolire AE, Tongo OO. Paediatric critical care needs assessment in a tertiary facility in a developing country. Niger J Paediatr. 2018;45(1):10–4.

Anitha GF, Velmurugan L, Sangareddi S, Nedunchelian K, Selvaraj V. Effectiveness of flow inflating device in providing Continuous Positive Airway Pressure for critically ill children in limited-resource settings: a prospective observational study. Indian J Crit Care Med. 2016;20(8):441–7.

Benito J, Luaces-Cubells C, Mintegi S, Manrique Martínez I, De la Torre EM, Miguez Navarro C, et al. Evaluation and impact of the “advanced pediatric life support” course in the care of pediatric emergencies in Spain. Pediatr Emerg Care. 2018;34(9):628–32.

Guerrero-Márquez G, Míguez-Navarro MC. The physiological diagnosis missing in the pediatric assessment triangle. Pediatr Emerg Care. 2021;37(11).

Jayashree M, Singhi SC. Initial assessment and triage in ER. Indian J Pediatr. 2011;78(9):1100–8.

Gonzalez Brabin A, Martín Rivada Á, Cabrero Hernández M, Cañedo Villarroya E. MIR clinical case. Make your diagnosis: a newborn with decreased intake and lethargy. Pediatr Integ. 2019;23(3):162–5.

Simon Junior H, Schvartsman C, Sukys GA, Farhat SCL. Pediatric emergency triage systems. Rev Paul Pediatr. 2022;41:e2021038.

Gehri M, Flubacher P, Chablaix C, Pediatrics Curchod P. The PAT: a simple and rapid tool for the assessment of the severely ill or injured child. Rev Med Suisse. 2011;3(277):64–5.

Dieckmann RA. New assessment model saves critical time in pediatric emergencies. AAP News. 1999;15(2):22.

Mendes M, McCormick T. Pediatric resuscitation. In: Rose E, editor. Pediatric emergencies: a practical, clinical guide. United kingdom: Oxford Oxford University Press; 2020. p. 67–74.

Chapter Google Scholar

Fuchs S. The special needs of children. In: Cone D, Brice JH, Delbridge TR, Myers JB, editors. Emergency medical services: clinical practice and systems oversight. 3rd ed. Hoboken: Wiley-Blackwell; 2021. p. 379–85.

Dieckmann RA, Brownstein D, Gausche-Hill M. The pediatric assessment triangle: a novel approach for the rapid evaluation of children. Pediatr Emerg Care. 2010;26(4):312–5.

Chiu Y-C, Liu S-Y, Yen T-A, Chen Y-Y, Yang C-W, Chu T-S, et al. Application of high-fidelity patient simulation in the teaching of pediatric primary assessment and management - is it feasible for medical students? J Med Educ. 2018;22(1):17–27.

Hansen M, Spiro DM. Teaching the pediatric assessment triangle using online video cases. Ann Emerg Med. 2013;62(5):S172.

Patten J. Goal-directed management of shock in children [thesis]. Zagreb: University of Zagreb School of Medicine; 2015.

Tagg A. Paediatric Assessment Triangle [internet]: Don't forget the bubbles; 2019 [updated 06/02/2023. Available from: https://dontforgetthebubbles.com/the-paediatric-assessment-triangle/ .

Furmick J, Malburg L, Leetch A. Pediatric airway management. Pediatr Emerg Med Rep. 2017;22(10):1–17.

Khouli M. Injuries in children and general principles of management. Mexican J Med Res ICSA. 2015;3(5). https://doi.org/10.29057/mjmr.v3i5.1835 .

Pérez LFT, Bouza MR, Valle AML, Hoyos JB, Vera CV. Emergency management: introduction. Revista Infancia y Salud. 2019;1(2). Available from: http://rinsad.uca.es/ojs3/index.php/rinsad/article/view/20 .

Rochat MK, Gehri M. Pediatric emergencies - the essential, briefly, for general practitioners. Ther Umsch. 2013;70(11):653–60.

Yock Corrales A, Starr M. Assessment of the unwell child. Aust Fam Physician. 2010;39(5):270–5.

Agbim CA, Wang NE, Lee M. Respiratory distress in pediatric patients. Pediatr Emerg Med Rep. 2018;23(4):41–55.

Güler E, Özkaya AK. Recognition of shock in children: review. Turkiye Klinikleri Pediatri. 2015;24(2):45–50.

Ramser M. The febrile child in respiratory distress. Praxis. 2017;106(4):201–7.

Singh A, Frenkel O. Evidence-based emergency management of the pediatric airway. Pediatr Emerg Med Pract. 2013;10(1):1–25.

Neumar RW, Blomkalns AL, Cairns CB, D’Onofrio G, Kuppermann N, Lewis RJ, et al. Emergency medicine research: 2030 strategic goals. Acad Emerg Med. 2022;29(2):241–51.

Grant D. The future of paediatric simulation. In: Cheng A, Grant V, editors. Comprehensive healthcare simulation: Pediatric Edition: Springer International Publishing. 2016. p. 401.

Hansoti B, Jenson A, Keefe D, De Ramirez SS, Anest T, Twomey M, et al. Reliability and validity of pediatric triage tools evaluated in low resource settings: a systematic review. BMC Pediatr. 2017;17(1):37.

Slusher T, Bjorklund A, Aanyu HT, Kiragu A, Philip C. The assessment, evaluation, and management of the critically ill child in resource-limited international settings. J Pediatr Intensive Care. 2017;6(1):66–76.

Muttalib F, González-Dambrauskas S, Lee JH, Steere M, Agulnik A, Murthy S, et al. Pediatric emergency and critical care resources and infrastructure in resource-limited settings: a multicountry survey. Crit Care Med. 2021;49(4):671–81.

Download references

Acknowledgements

This work was submitted as a thesis requirement for the Master in Pre-Hospital Critical Care, University of Stavanger, Norway in December 2022. We thank the Norwegian Institute of Public Health for supporting this publication open access fees.

This work had no financial support.

Author information

Authors and affiliations.

University of Stavanger, P.O. Box 8600, NO-4036, Stavanger, Norway

Tore A. G. Tørisen

Glanville.Info, 38 Moorgate, York, YO24 4HR, UK

Julie M. Glanville

Instituto Universitario de Educación Física, Universidad de Antioquia. , Medellín, Colombia

Andres F. Loaiza

Grupo de Investigación en Entrenamiento Deportivoy, Actividad Física Para La Salud (GIEDAF), Universidad Santo Tomás. , Tunja, Colombia

School of Rehabilitation Science, College of Medicine, University of Saskatchewan, 107 Wiggins Rd, Saskatoon, SK, S7N 5E5, Canada

Julia Bidonde

You can also search for this author in PubMed Google Scholar

Contributions

Project conceptualization was done by TT, JB, JG, and AFL. TT and JB were primarily responsible for the methodology and resources, while TT, JB, and AFL carried out the data extraction and validation process, and TT led the analysis and synthesis. TT and JB wrote the original draft, with input and feedback from JG and AFL during the review and editing process. The visualization was primarily handled by TT and AFL. TT and JB supervised the project, and TT managed project administration. JB was responsible for software acquisition.

Corresponding author

Correspondence to Julia Bidonde .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

Tore A. G. Tørisen is an instructor and medical supervisor for courses in “Pediatric Education for Prehospital Professionals” (PEPP) and he is registered with the American Academy of Pediatrics (AAP). The remaining authors declare no conflict of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: appendix b. prisma-scr, additional file 2. appendix c. search strategy, additional file 3. appendix d. excluded studies with reasons for exclusion, additional file 4. appendix e. included studies, additional file 5. tables, additional file 6. appendix a – who emergency care system framework, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Tørisen, T.A.G., Glanville, J.M., Loaiza, A.F. et al. Emergency pediatric patients and use of the pediatric assessment triangle tool (PAT): a scoping review. BMC Emerg Med 24 , 158 (2024). https://doi.org/10.1186/s12873-024-01068-w

Download citation

Received : 19 July 2023

Accepted : 08 August 2024

Published : 04 September 2024

DOI : https://doi.org/10.1186/s12873-024-01068-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Pediatrics or paediatric
Pediatric assessment triangle
Children; emergency medicine; review

BMC Emergency Medicine

ISSN: 1471-227X

General enquiries: [email protected]

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Unveiling the Exodus: A scoping review of attrition in allied health

Roles Conceptualization, Formal analysis, Methodology, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation UniSA Allied Health & Human Performance, University of South Australia, Adelaide, Australia

Roles Conceptualization, Methodology, Supervision, Writing – original draft, Writing – review & editing

Affiliation School of Allied Health, Human Services and Sport, La Trobe University, Melbourne, Australia

Su Ann Yeoh,
Saravana Kumar,
Anna Phillips,
Lok Sze Katrina Li

Published: September 6, 2024
https://doi.org/10.1371/journal.pone.0308302
Reader Comments

Efficient utilisation of allied health workforce may help address the predicted shortfall of 18 million health workers estimated by 2030. Knowledge about allied health professionals’ attrition, or intention to leave, and factors influencing attrition can assist in developing evidence-informed strategies to mitigate this issue. The review aimed to map attrition and attrition intention rates, and its attributing factors for allied health professions worldwide.

Adhering to the PRISMA-ScR guidelines, a comprehensive search was conducted across academic databases (PsycINFO, MEDLINE, Embase, Emcare, CINAHL, Scopus, and the Cochrane Library database) and grey literature (Google, Google Scholar, organisational websites). Two reviewers independently undertook a two-stage screening process along with data extraction using customised data extraction forms. A narrative synthesis was used to synthesise the data.

Thirty-two studies published between 1990 and 2024 were included. Attrition rates ranged from 0.5% to 41% across allied health disciplines. Pharmacists demonstrated the lowest attrition rates, while audiologists reported the highest. Radiographers reported the lowest intent to leave at 7.6%, while occupational therapists showed highest intent to leave, ranging from 10.7% to 74.1%. The analysis revealed three recurring themes contributing to attrition: profession-centric factors (e.g., career progression, job satisfaction, support, and professional growth), systemic-centric factors (e.g., compensation, staffing challenges, clinical practices, patient care, workload), and individual-centric factors (e.g., recognition, the need for change, and burnout).

Attrition in allied health remains a significant challenge. Addressing this issue requires a systemic, nuanced, and evidence-based approach, given the complex, interlinked, and multifaceted factors contributing to attrition. The younger workforce, characterized by changing generational values, necessitates innovative thinking, intersectoral collaboration, and the potential for co-created solutions with, for, and by the allied health workforce.

Citation: Yeoh SA, Kumar S, Phillips A, Li LSK (2024) Unveiling the Exodus: A scoping review of attrition in allied health. PLoS ONE 19(9): e0308302. https://doi.org/10.1371/journal.pone.0308302

Editor: Jenny Wilkinson, Endeavour College of Natural Health, AUSTRALIA

Received: May 9, 2024; Accepted: July 20, 2024; Published: September 6, 2024

Copyright: © 2024 Yeoh et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting Information files.

Funding: The author(s) received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

On average, human life expectancy has increased by ten years within the last five decades [ 1 ]. This respective increase in life expectancy is associated with greater burdens on the healthcare system to manage individuals with non-communicable diseases [ 2 , 3 ]. According to Philip [ 1 ], efficient utilisation of allied health workforce could potentially reduce health system cost burdens by reducing the demand and utilisation of acute health facilities. Additionally, Philip’s research describes the fundamental role of the allied health workforce in chronic disease and multimorbidity management which aligns with current healthcare needs owing to the aging demographics [ 1 ]. As morbidity rates continue to change with medical care advancements that facilitate increased life expectancies, a greater employment of allied health professionals is imperative to meet this expanding demand.

According to the United Nations High-Level Commission on Health Employment and Economic Growth, a deficit of 18 million health workers is estimated by 2030 [ 4 ]. In addition, the ongoing COVID-19 pandemic has increased the prevalence of psychological distress amongst healthcare workers; contributing to global shortages of health care workers [ 5 ]. Furthermore, the health workforce in most developed countries is highly dependent upon foreign health workers. Within Europe, foreign healthcare workers constitute 27% of doctors and 16% of nurses [ 5 ]. During the COVID-19 pandemic, the travel restrictions hindered the migration of foreign healthcare workers [ 5 ]. A study conducted by Satiani et al. [ 6 ] on attrition trends of surgical faculty within a 15-year period in the United States indicated a national turnover between six percent to 12% annually; with 40% of surgeons reporting burnout with intention to leave their current practice [ 6 ]. The attrition rates were notably higher amongst women, ethnic minorities, and academic physician professors [ 6 ]. Correspondingly, a review conducted by Lopes et al. [ 7 ] of 51 academic studies on attrition rates of healthcare workers identified relatively low attrition rates in midwives (4.5% - 16%) and doctors (1.7% - 15%) compared to nurses (4.9% to 44.3%) [ 7 ]. Previous studies on attrition amongst allied health professionals indicated higher attrition rates in contrast to other health professions such as general practitioners, dentists, and nurses [ 8 , 9 ].

At present, there is no globally recognised definition or classification for allied health professionals which presents a challenge for research in this field. In Australia, allied health professionals are described as university qualified practitioners that are not part of medical, dental, or nursing professions [ 10 ]. Commonly known allied health professionals include nutritionists, occupational therapists, pharmacists, physiotherapists, psychologists, social workers, podiatrists, audiologists, speech pathologists, and medical radiation professionals [ 10 ]. The respective lack of standardisation complicates efforts to address issues such as increased attrition rates among allied health professionals. Previous research indicates that 10% to 15% of Canadian rehabilitation professionals, predominantly occupational therapists and physiotherapists, leave their profession within two years of graduation, and in Australia, 65% of surveyed physiotherapy graduates foresee leaving their profession within the next decade [ 11 ]. Contributing factors include heavy caseloads, stress and burnout, desire for increased salary and promotional opportunities, and discrepancies between clinicians’ expectations and actual practice [ 12 ]. Understanding these factors is crucial for developing strategies to retain these professionals, thereby ensuring the sustainability of healthcare services. While research highlights important issues with regards to attrition in allied health professions, there is largely a preliminary focus on single disciplines instead of the broader, allied health collective [ 13 – 15 ].

To our knowledge no reviews have been conducted to investigate attrition among all professions within the allied health sector. Furthermore, research on the health workforce has been predominantly focused on physicians and nurses [ 7 , 16 ]. Therefore, this review aims to map attrition and attrition intention rates, and its attributing factors for allied health professions worldwide [ 17 ].

This scoping review was conducted in accordance with PCC (Population, Concept, Context) [ 18 ] framework, which informed its search strategy as recommended by the JBI methodology for scoping reviews [ 19 ]. The review followed best practice standards in the conduct and reporting of scoping reviews (PRISMA scoping review) [ 17 ]. The protocol of this scoping review has been registered on the Open Science Framework database (DOI 10.17605/OSF.IO/57T3R ).

Search strategy

Preliminary search on MEDLINE and PsycINFO was performed to explore the body of literature and establish key terms and medical subject headings (MeSH) in the field of interest. Search terms, developed based on the population, concept, and context are presented in Table 1 . To ensure a rigorous search strategy was developed, the search strategy was evaluated by an academic librarian at the University of South Australia.

PPT PowerPoint slide
PNG larger image
TIFF original image

https://doi.org/10.1371/journal.pone.0308302.t001

A comprehensive search was conducted across seven academic databases (PsycINFO, MEDLINE, Embase, Emcare, CINAHL, Scopus, and the Cochrane Library) from inception to December 2022, supplemented by an updated search on March 18, 2024, to capture current literature. The databases were selected based upon accessibility and applicability to the research question. Search syntax for each database is provided in Appendix A (Tables A1-A7) in S1 Appendix . To reduce the risk of publication bias, Google and Google Scholar were searched for grey literature and the first 10 pages of the search results generated were reviewed [ 20 ]. Additionally, organisational websites (Australian Health Practitioner Regulation Agency (APHRA)) and websites of professional societies/associations (in the United States, Europe, United Kingdom, and Australia) were searched for relevant publications. Searches were limited to publications in English. No limitations for age, gender, or country of workplace were applied.

Eligibility criteria

The following eligibility criteria were developed based upon the predetermined research question ( Table 2 ). Eligibility criteria included studies investigating attrition rates, attrition intention rates, and attrition factors for allied health professionals. Studies involving multiple professions that measured and reported the attrition components on individual allied health professions were included. Studies exploring attrition factors including retirement, disability and/or leaving a workplace but remaining in the profession were excluded.

https://doi.org/10.1371/journal.pone.0308302.t002

Study selection process

Following the search process, results obtained for the databases were uploaded to Endnote 20 TM software. Covidence TM was used to identify duplicate articles and for the screening and selection processes. To select relevant studies, a two-step process was implemented; reviewing the title and abstract followed by screening the full text. Both screening processes were conducted by two independent reviewers (SY and KL/AP/SK) with any screening discrepancies discussed and conflicting votes resolved by a third reviewer.

Data extraction

A customised data extraction form was developed based on the PCC [ 18 ] framework including author, country, health profession, design, method, sample characteristics, attrition rate, intention to leave rates, and factors contributing to attrition (Appendix B, Table B1 in S1 Appendix ). Data were extracted independently by SY and reviewed by KL or AP. Discrepancies in data extraction between reviewers were resolved through further review and discussion.

Data synthesis

Given the nature of the review question, a narrative synthesis of the literature was conducted. To consolidate the quantitative data, summary tables were used, categorising information by profession, attrition rates, intention to leave, and other relevant factors contributing to the attrition of allied health workers from their related professions. The outcomes of each category were thereafter consolidated by contrasting each individual study with their respective allied health professions.

The qualitative data were analysed thematically. Through extensive reading, key themes were compiled from each study and grouped by common traits, allowing central themes to be identified. These themes were collaboratively discussed and refined by the review team until a consensus was achieved. For this review, data from quantitative and qualitative research were broadly categorised into three major themes: i) profession centric (factors related to the profession); ii) system centric (factor related to the overall health system); iii) person centric (factors relating to self/individual). Additional details are provided in the individual sections.

The initial search identified 1028 studies, with an additional 394 studies identified from updated searches. Following the removal of duplicates, 1234 studies were screened based on titles and abstracts, resulting in the exclusion of 1046 studies. Of the 233 eligible for full-text screening, 32 studies were included in this scoping review. The results of database searching screening phases, eligibility and rationale for exclusion are presented in Fig 1 .

https://doi.org/10.1371/journal.pone.0308302.g001

Characteristics of included studies

The 32 studies included in the scoping review were published between 1990 [ 21 ] and 2024 [ 22 ] ( Table 3 ). Eleven of the 30 studies were conducted in Australia [ 8 , 14 , 23 – 31 ], six in the United Kingdom [ 15 , 32 – 36 ], five in Canada [ 13 , 37 – 40 ], four in the United States of America [ 21 , 41 – 43 ], four in New Zealand [ 22 , 44 – 46 ], and one in Ireland [ 47 ] and Romania [ 48 ] respectively. Study designs varied from cross-sectional to qualitative studies as outlined in Table 3 . Attrition data were predominantly obtained through survey questionnaires (n = 23) [ 13 , 14 , 21 – 24 , 27 , 28 , 30 , 31 , 33 – 42 , 45 , 47 , 48 ], semi-structured interviews (n = 9) [ 8 , 15 , 25 , 26 , 29 , 32 , 33 , 44 , 46 ], and relevant allied health department data (n = 1) [ 43 ]. The professions studied included physiotherapists (n = 7) [ 24 , 28 , 29 , 31 , 38 , 40 , 44 ], occupational therapists (n = 6) [ 13 , 21 , 27 , 33 , 37 , 47 ], pharmacists (n = 6) [ 22 , 26 , 32 , 34 , 36 , 42 ], nuclear medicine technologists (n = 5) [ 15 , 23 , 25 , 35 , 41 , 45 ], social workers (n = 2) [ 39 , 48 ], speech pathologists (n = 2) [ 8 , 14 ], audiologists (n = 1) [ 43 ], podiatrists (n = 2) [ 30 , 46 ] and multiple allied health professionals (n = 1) [ 25 ]. Sample sizes ranged from 12 participants [ 32 ] to 32,181 participants [ 34 ]. Further characteristics of the studies are outlined in Table 3 .

https://doi.org/10.1371/journal.pone.0308302.t003

Quantitative results

Attrition rates..

Twenty-five studies presented quantitative findings, detailed in Table 4 . Among them, nine studies provided attrition rates, ranging from 0.5% [ 34 ] to 41% [ 43 ]. The lowest attrition rates were observed among pharmacists between 0.5% [ 34 ] and 1% [ 42 ]. For physiotherapists, attrition rates varied, ranging from 1.6% [ 38 ] to 12% [ 24 ]. The highest attrition rate, at 41%, was reported in a study involving audiologists [ 43 ]. A study provided attrition rates for nuclear medicine technologists from 1996–2001, broken down by age group, with the 45–49 years category having the highest attrition rate within the profession [ 23 ]. However, attrition rates for social workers, medical radiation professionals, and speech pathologists were not reported.

https://doi.org/10.1371/journal.pone.0308302.t004

Intention to leave.

The intention to leave the profession was more frequently reported than actual attrition rates, as shown in Table 4 . Among the 14 studies [ 13 , 22 , 27 , 28 , 30 , 31 , 33 – 35 , 37 , 41 , 45 , 47 , 48 ] that examined participants’ intent to leave their profession, a study involving radiographers reported the lowest intention to leave at 7.6% [ 41 ]. Conversely, studies observing occupational therapists reported the highest intent to leave, with rates ranging from 10.7% [ 13 ] to 74.1% [ 37 ]. Three professions, including nuclear medicine technologists, audiologists, and speech pathologists, did not report any intention to leave the profession.

Factors contributing to attrition

Ten studies examined factors contributing to attrition, as summarised in Table 5 . The most commonly cited reasons for attrition included job dissatisfaction (n = 7) [ 13 , 21 , 30 , 34 , 39 , 40 , 45 ], lack of autonomy (n = 4) [ 13 , 21 , 34 , 40 ], burnout (n = 2) [ 40 , 48 ], and overburdened workload (n = 4) [ 13 , 21 , 34 , 45 ]. Studies also indicated that male pharmacists [ 34 ] and nuclear medicine [ 23 ] technologists were more inclined to leave their professions compared to their female counterparts. Furthermore, social workers who exhibited higher levels of occupational commitment reported greater job satisfaction and expressed less intent to leave their profession [ 39 ]. Age played a role in attrition, with varying findings across three studies [ 21 , 34 , 45 ]. Specifically, Bailey [ 21 ] noted that younger age groups with less than three years of professional occupational therapy experience had attrition rates of 54%. Conversely, older radiation therapists reported higher job satisfaction and lower intent to leave the profession [ 45 ]. Additionally, pharmacists in their 20s and 50s were more likely to consider leaving their profession compared to those in their 30s [ 34 ].

https://doi.org/10.1371/journal.pone.0308302.t005

Qualitative results

Qualitative data were reported by 12 of the 31 included studies [ 8 , 15 , 23 , 25 , 26 , 29 , 31 – 33 , 36 , 44 , 46 ]. Thematic analysis of the studies revealed three major overarching themes: i) profession centric; ii) system centric; iii) person centric ( Fig 2 ). As discussed, the first central theme, profession centric was consolidated from factors related to shortcomings identified by participants in relation to the profession involving subthemes of: i) lack of career pathway; ii) job dissatisfaction; and iii) lack of support and professional development. The second theme, system centric, involved factors related to the overall healthcare system with subthemes of: i) impact of workload; ii) barriers to optimal patient care; iii) staffing issues; iv) limited clinical practice; and v) remuneration. The third theme, person centric encompasses personal factors involving subthemes of: i) lack of recognition; ii) need for change; and iii) burnout. Nine of the qualitative studies reported profession-centric themes [ 8 , 15 , 23 , 25 , 26 , 32 , 33 , 36 , 44 ] while eight reported system centric themes [ 15 , 25 , 26 , 31 – 33 , 44 , 46 ] and six reported personal centric themes [ 8 , 15 , 26 , 29 , 32 , 33 ].

https://doi.org/10.1371/journal.pone.0308302.g002

Theme 1: Profession centric.

Lack of career pathway . The most frequently described profession centric factor was the lack of career pathway, and the related insufficient career opportunities within clinical practices [ 8 , 15 , 23 , 26 , 32 ]. The absence of a satisfactory career pathway prompted frustration and reduced career motivations within allied health professionals. This factor was observed in five studies expressed by nuclear medicine technologist [ 23 ], pharmacists [ 26 , 32 ], speech pathologists [ 8 ], and radiographers [ 15 ]. Lack of career progression was associated with feelings of “frustration” [ 32 ] (p.185) , “overwhelm” [ 26 ] (p.133) , and being “blocked” [ 15 ] (p.78) .

Changing professions was denoted as a result of the lack of career progression and opportunity to professionally progress. Several health professionals discussed their dissatisfaction with their available career pathways:

‘‘I was at the top of the grading, and so there was really no where for me to go except to move out of the profession…” (Speech pathologist) [ 8 ] (p.167) “Lack of career pathways in nuclear medicine . No succession planning . No chief jobs . Reach senior then a dead end so technologist looks for another career” . (Nuclear medicine technologist) [ 23 ] (p.289)

Job satisfaction . Job satisfaction factors describe the extent of contentment and fulfilment experienced by professionals in performing their job responsibilities. Two studies involving speech pathologists [ 8 ] and radiographers [ 15 ] highlighted job dissatisfaction as a key factor contributing to attrition. Speech pathologist expressed a sense of being unable to “make a difference” [ 8 ] (p.166) , leading to a desire to pursue a different profession where they could feel more “valued” [ 8 ] (p.166) and have a greater “impact” [ 8 ] (p.166) . Meanwhile, radiographers expressed discontent with the repetitive nature of their job:

“You were just kind of factory workers, and that wasn’t the buzz for the job, and I struggled doing the same thing, and working on the same machine day in day out… It wasn’t for me.” (Radiographer) [ 15 ] (p.78)

Lack of support and professional development . Lack of support and professional development factors revolves around deficiency in resources, guidance, or mentorship, professional development training, and job performance reviews. Three studies involving occupational therapists [ 33 ], pharmacists [ 36 ], and physiotherapists [ 25 ] identified issues related to the absence of support and professional development factors. Occupational therapists characterised the lack of support as "dreadful" , "devalued , " and "neglected , " [ 33 ] (p.289) . Additionally, occupational therapists associated the absence of opportunities for personal development with a sense of incompetency:

“When I asked if there were any courses, I was told ‘no way’. I didn’t seem to be able to get any further… I wouldn’t feel competent to go on to a Senior post, I wouldn’t feel that I’d learnt any more than a Basic Grade.” (Occupational therapist) [ 33 ] (p.290)

Physiotherapists and pharmacists involved in the studies cited "no support" [ 25 ] (p.8) and a "lack of protected time" [ 36 ] (p.2) for professional development as reasons for their decision to leave their respective professions.

Theme 2: System centric factors.

Impact of workload . Impact of workload entails the significant influence that the quantity and nature of work responsibilities have on individuals’ decisions to leave their professions. This theme encompasses components such as excessive work demands, overwhelming job pressures, and an imbalance between workload and available resources. Workload was a major system centric factor contributing to attrition and intent to leave amongst pharmacists [ 26 , 32 ], radiographers [ 15 ], and physiotherapists [ 31 , 44 ]. A radiographer highlighted a system that “does not accept the fact that you are getting older” [ 15 ] (p.80) and is not designed to accommodate an aging workforce in terms of workload. Pharmacists expressed concerns of carrying out their responsibilities “effectively” [ 32 ] (p.184) and “safely” [ 32 ] (p.184) within limited time constraints. Workloads were further described by pharmacists and physiotherapists as:

“You know, long hours, no breaks, pays pretty ordinary and the level of responsibility and stress with the amount of prescriptions that you’re expected to do…it’s just like working in a factory… I felt like a glorified factory worker…” (Pharmacist) [ 26 ] (p.133) “Most new graduates said they weren’t prepared for the heavy workloads or the psychosocial aspect of face-to-face client interaction . ” (Physiotherapist) [ 44 ] (p.23) “Long work hours and giving up my own sport and hobbies to pursue this career . ” (Physiotherapist) [ 31 ] (p.10)

Barriers to optimal patient care . Barriers to optimal patient care involve challenges and limitations of professionals when striving to provide optimal patient care; that is ethically and professionally acceptable. This theme emerges as the second major system centric factor expressed in four studies by physiotherapists [ 25 ], pharmacists [ 26 , 32 ], and occupational therapists [ 33 ]. Physiotherapists voiced concerns about the expectation of patient care, deeming it "professionally compromising" [ 25 ] (p.7) and "disparate to the needs of the client," [ 25 ] (p.7) leading to attrition from the profession. Pharmacists also expressed feelings of dissatisfaction regarding similar issues, highlighting the pervasion of patient care constraints [ 26 , 32 ]. Occupational therapists similarly conveyed challenges related to optimal patient care:

“I’m actually leaving the rotation to find a post somewhere else… I find it incredibly frustrating that you don’t get to follow… [patients]… through.” (Occupational therapist) [ 33 ] (p.289)

Impact of staffing . The effects and consequences of staffing-related factors on the decision of professionals to leave their respective professions is prevalent. This respective theme explores how staffing issues, such as insufficient personnel, high workload, or inadequate support, contribute to challenges and difficulties faced by professionals in delivering quality care. Both pharmacists [ 32 ] and occupational therapists [ 33 ] voiced apprehensions that revolved around the challenges posed by staff shortages:

“Yeah. I mean they want to maximise profits, the large companies, because the shareholders want maximum profits… maximum output and minimum staff.” (Pharmacist) [ 32 ] (p.184) “ … got me to take an unqualified member of staff , and teach them the ropes , and then put them in my place … without any supervision or any help , having to give information to doctors and nurses … She just filled a gap , ” (Occupational therapist) [ 33 ] (p.290)

Podiatrists additionally described that shortages in personnel resulted in insufficient time off:

“ There are not enough podiatrists … Not even close” [ 46 ] (p.6) “ I never really went away… any time I had off I worked around Christmas or long weekends ” [ 46 ] (p.6)

Limited clinical practice . Limited clinical practice refers to the impact of constrained or restricted opportunities for professionals to engage in hands-on, practical, and clinically relevant duties within their field. This theme explores how a lack of exposure to diverse and meaningful clinical practices can contribute to professionals’ dissatisfaction, reduced skill development, overall frustration and eventual decision to leave their profession. Pharmacists expressed their lack of opportunity to utilise their clinical skills [ 26 ] and characterised their role as primarily focused on “dispensing and checking” [pharmaceuticals] [ 32 ] (p.184) . Pharmacists also expressed dissatisfaction with the misalignment between the perceived role during training:

“We’re taught in University which I think is wrong, that we have a certain clinical role…but in community [pharmacy], that role doesn’t exist.” (Pharmacist) [ 26 ] (p.134)

Insufficient remuneration . Insufficient remuneration refers to the impact of low or insufficient salaries and benefits on dissatisfaction and frustration among individuals in the profession. It encompasses the economic aspects of the job, including salary structures, benefits packages, and overall compensation, and examines how these factors may influence professionals’ job satisfaction, motivation, and ultimately their decision to pursue alternative career paths. Frustration stemming from inadequate remuneration or wages, despite possessing skill competency in their respective professions, was identified in three studies involving pharmacists [ 26 ], physiotherapists [ 44 ], and speech pathologists [ 25 ]. Insufficient remuneration devalued the roles of pharmacists, as expressed:

“…when you go for a position, there’s not really much opportunities to negotiate, because what’s basically put on you is like, if you don’t want this job, some other script monkey will do it for a lot less.” (Pharmacist) [ 26 ] (p.133)

Theme 3: Person centric factor.

Lack of recognition . Lack of recognition refers to the perceived insufficiency of acknowledgment and appreciation of the contributions, skills, and efforts of professionals within their respective fields. This theme explores how professionals may feel undervalued or overlooked in terms of their expertise, achievements, and the overall impact of their work. This lack of recognition can result in decreased job satisfaction, demotivation, and contribute to the decision to leave the profession; in search of environments where their contributions are appreciated and acknowledged. Lack of recognition was expressed in two studies involving occupational therapists [ 33 ] and radiographers [ 15 ]. Occupational therapists conveyed instances of being excluded from discussions and the expectation to entertain patients, which resulted in feelings of incompetence [ 33 ]. In the meantime, radiographers highlighted the adverse impact of a lack of recognition on their decision to discontinue practice:

“And I think that if that contribution was at least acknowledged by anyone I would have felt better about things, I would have definitely felt better about staying… no appreciation for all the hard work…” (Radiographer) [ 15 ] (p.78)

Need for change . The need for change comprises professionals’ dissatisfaction with the overall structure, policies, and practices of their employment; contributing to a lack of fulfillment and prompting profession changes. This respective theme emerged in three studies, encompassing pharmacists [ 26 ], occupational therapists [ 33 ] and physiotherapist [ 29 ]. The intentions behind the decision to change professions was described by healthcare professionals:

“I’m looking forward to moving on because… there’s not enough challenge in… [this job]… I would like to be in a job where I get to use all… [my]… skills, that’s a bit more challenging…” (Occupational therapist) [ 33 ] (p.290) “I did have a second pathway that I was alternatively going to take … it was almost a flip of a coin there . And that other pathway has continued to nudge at me as well . So there’s that option to take that and it’s sort of a bit of a fear of missing out . ” (Physiotherapist) [ 29 ] (p. 6)

Burnout . The theme of burnout refers to a state of chronic physical and emotional exhaustion, often accompanied by feelings of cynicism and detachment from work. Professionals experiencing burnout find themselves overwhelmed by prolonged and intense workplace stress, resulting in a reduced sense of personal accomplishment, diminished interest in their professional roles, and ultimately the desire to leave their profession. Radiographers [ 15 ], pharmacists [ 32 ], and speech pathologists [ 8 ] in three studies highlighted the risk of burnout. Pharmacists denoted “dreading going into work” as a direct result of an overwhelming workload [ 32 ] (p.184) . A speech pathologist identified that “if the negatives outweigh your positives , then I think you’ll find people are really dissatisfied and getting burned out and leaving the profession … ” (Speech pathologist), denoting the primary consequences of burnout for many healthcare professionals [ 8 ] (p.166) .

The aim of this scoping review was to map the literature on attrition rates and contributory factors of attrition within global allied health professions. Within 32 studies identified within this scoping review, attrition and attrition intention rates were examined in research with audiologists [ 43 ], nuclear medicine technologists [ 23 ], radiographers [ 41 ], radiation therapists [ 45 ], physiotherapists [ 24 , 28 , 29 , 31 , 38 , 44 ], occupational therapists [ 13 , 27 , 33 , 37 , 47 ], pharmacists [ 22 , 34 , 42 ], podiatrists [ 30 , 46 ], social workers [ 48 ], and sonographers [ 35 ]. Occupational therapists and physiotherapists were the central focus of attrition research among allied health professionals. Accordingly, attrition studies involving nutritionists, and psychologists were not identified within this review. The rate at which allied health professionals withdrew from their professions varied considerably (0.5% [ 34 ] to 74.1% [ 37 ]) amongst professions. Thus, indicating potential factors that contribute to attrition across allied health disciplines. These factors were broadly categorised into profession-centric [ 13 , 21 , 34 , 39 , 40 , 45 ], system-centric [ 13 , 21 , 34 , 45 , 48 ], and person-centric [ 13 , 14 , 21 , 23 , 40 , 45 , 48 ] factors.

The influential interconnected nature of attrition factors within allied health professionals were a chief finding derived from this review [ 15 ]. Factors such as the lack of recognition and need for change, were expressed simultaneously, indicating an interconnected relationship between person centric factors [ 33 ]. Furthermore, the need for change was described concurrently with the absence of career pathways within profession centric factors [ 26 ]. Similarly, system centric factors, particularly the impact of workload, were concurrently expressed alongside profession-centric factors like job satisfaction and person-centric factors such as burnout. Conclusively, attrition and attrition intention amongst allied health disciplines are not solely influenced by individual factors but rather, a multitude of factors distributed across various "levels," with each factor and level influencing the others. These findings are shared by similar research amongst other health disciplines. A systematic review of 19 studies exploring attrition factors among physicians highlighted the potential to introduce a multifaceted approach to mitigate attrition; providing financial incentives, career development, sufficient staffing, maintenance of professional work environments, manageable workloads, and autonomy, with recommendations for a multifaceted approach to tackle this challenge [ 49 ].

Attrition amongst allied health professionals remains a persistent global challenge. According to McLaughlin, Lincoln [ 8 ], 52% of speech pathologists in Queensland, Australia, expect to stay in the profession for less than 10 years. Similarly, Pretorius, Karunaratne [ 9 ] reported that 60% of physiotherapy graduates in Australia anticipate leaving the workforce within 10 years. While evidence-informed strategies to address attrition should be implemented, these findings also raise the question of whether career changes among allied health professionals signify a potential "new normal" in the future of the healthcare workforce, largely a result of changes in generational personal and work values. A study examining shifts in career values across generations identified significant differences between millennials and their predecessors [ 50 ]. Notably, millennials placed greater importance on intellectual stimulation, followed by advancements, workplace social interactions, job prestige, and the importance of having fun, with a crucial need for a work-life balance, distinctive from the values of earlier generations [ 50 ]. Additionally, younger generations considered work as less central in their lives, prioritised leisure more, and exhibited different work ethics compared to older counterparts [ 51 ].

These findings have important consequences for the future of allied health workforce. First, the allied health workforce should recognise, cater to and support those who seek career changes and non-linear career paths. This could be achieved by ensuring allied health professionals are inclusive of diverse positions, have structured and supported career pathways and, include opportunities for progression outside of frontline clinical practice (such as leadership roles, joint appointments with educational institutions) [ 52 ]. Policies that establish and promote organisational values, especially by those in leadership roles, can further enhance job satisfaction and retention. Additionally, formal support programs such as mentorships in the first year of practice have shown significant value in retaining professionals by providing essential guidance and support [ 29 , 31 ]. Second, the values of self-care and work-life balance embraced by the younger generation may directly conflict with the demanding and emotionally taxing nature of the role as a health professional (such as the increasing prevalence of burnout) [ 50 ]. As a result, the conventional expectation for healthcare professionals to uphold a "lifelong" commitment to the health workforce may no longer be relevant for the younger generation of healthcare professionals. While this issue may not be completely resolvable, the younger generation’s work life may be extended by ensuring professional support and mentoring through clinical supervision, peer support and career pathways in complementary fields (such as leadership and management) [ 53 ]. Furthermore, targeted initiatives such as rural admissions schemes for training programs can help address workforce shortages and health inequalities in underserved areas [ 46 ].

The transformation, and evolving career values of, the health workforce has been expedited by the COVID-19 pandemic [ 54 ]. Research by Skelton, Harrison [ 35 ] reported that 12.5% of sonographers’ intended to leave their profession during the COVID-19 outbreak, driven by job dissatisfaction, burnout, and psychological distress. These findings are supported by previous research including Bhardwaj [ 55 ], which reflected a 10% increase in burnout amongst physicians in 2021, with 50% of physicians reporting burnout amidst the pandemic. Lou, Montreuil [ 56 ] reported an attrition intention rate of 20% among physicians in Canada during the pandemic in contrast to 3.2% amongst physicians in the United Kingdom prior to COVID-19 [ 57 ]. Similarly, burnout was identified as the major theme contributing to attrition amongst registered nurses in Italy and Greece during COVID-19 [ 58 , 59 ]. In line with this, 49.3% of nurses in Ghana intend to leave the profession because of burnout amidst the COVID-19 pandemic [ 60 ]. Collectively, these studies underscore the transformative impact of the COVID-19 pandemic on the global health workforce and advocate for a thoughtful and nuanced approach to health workforce planning [ 61 ].

Strengths and limitations

This review adhered to best practice guidelines throughout the conduct and reporting of scoping reviews (PRISMA-ScR). Despite efforts to minimise publication bias within grey literature and citation searches, the review only included studies published in English, potentially introducing publication and language bias in study selection, as relevant studies in languages other than English were excluded. While grey literature was searched via organisational and professional society websites, these were mostly confined to Western jurisdictions (e.g., United States, Europe, United Kingdom, and Australia). Therefore, this introduces a significant limitation of the review as publications and other resources from developing countries may have been overlooked. Data on certain disciplines were not identified in this review and therefore the collective extent of attrition concerning the allied health workforce could not be captured. All studies identified in this review were undertaken in developed countries and two thirds were published over a decade ago (prior to the impact of the COVID-19 pandemic), and thus, the generalisability to the global allied health workforce within the current health care context is limited. Future research should explore attrition, intention to leave, and contributing factors among nutritionists, and psychologists to mitigate the information deficit regarding these professions and increase breadth of research. Additionally, investigating attrition amongst allied health professionals in the post-COVID-19 world, will enhance understandings of the pandemic’s impact on this essential workforce.

The ongoing concerns about the allied health workforce and its capacity to meet increasing health care demands has fuelled interest and research on this topic. Despite the findings of this research reflecting variable attrition rates across health disciplines, the contributing factors are consistently complex and interlinked; thus, requiring a systemic, nuanced and evidence-informed approach.

The literature additionally posits the potential existence of a “new-normal” for the allied health workforce, involving the changes in generational values amongst younger workers; driven by the COVID-19 pandemic. These respective changes comprise a younger workforce pursuing flexibility in work conditions, fun, opportunities for career progression, self-care interests and work-life balance. Ultimately, satisfying these demands will require innovative thinking, intersectoral collaboration and introduce the potential of co-created solutions with, for, and by the allied health workforce.

Supporting information

S1 appendix..

https://doi.org/10.1371/journal.pone.0308302.s001

View Article
Google Scholar
PubMed/NCBI
10. Allied Health Professions Australia. Defining Allied Health 2023 [Available from: https://ahpa.com.au/allied-health-professions/ .
28. State of Victoria. Victorian Allied Health WorkforceResearch ProgramPhysiotherapy Workforce Report. Health and Human Services; 2016.
44. Reid A, Dixon H. Analysis of the Physiotherapy Workforce. New Zealand; 2018.
50. Ng ES, Lyons ST, Schweitzer L. Generational career shifts: how Matures, Boomers, Gen Xers and Millennials view work. First edition. ed. Bingley, [England: Emerald Publishing; 2018.

COMMENTS

Grey literature: An important resource in systematic reviews
Identifying all evidence relevant to the research questions is an essential component, and challenge, of systematic reviews. Grey literature, or evidence not published in commercial publications, can make important contributions to a systematic review. Grey literature can include academic papers, including theses and dissertations, research and ...
Gray literature: An important resource in systematic reviews
Gray literature also provides an important forum for disseminating studies that might otherwise not be disseminated, such as those with null or negative results. 18, 19 As a result, including gray literature in meta-analyses may help minimize the effects of publication bias. 11, 15, 18 Publication bias refers to the propensity for only studies ...
Searching and synthesising 'grey literature' and 'grey information' in
Public health researchers may want to include 'grey literature' in evidence syntheses for at least three reasons. Firstly, including grey literature can reduce the impact of publication bias as studies with null findings are less likely to be published in peer-reviewed journals [].Secondly, grey literature can provide useful contextual information on how, why, and in whom complex public ...
Shades of Grey: Guidelines for Working with the Grey Literature in
As with Category C articles that ignore grey literature, peer-review is used as a primary filter for inclusion as a proxy for quality in Category B articles. ... The recommendations for systematic review of grey literature developed in this paper also suggest how scientific evidence can be juxtaposed with other sources of evidence to provide a ...
Gray literature: An important resource in systematic reviews
Gray literature may thusly reduce publication bias, increase reviews' comprehensiveness and timeliness, and foster a balanced picture of available evidence. Gray literature's diverse formats and audiences can present a significant challenge in a systematic search for evidence. However, the benefits of including gray literature may far outweigh ...
Grey Literature Searching for Health Sciences Systematic Reviews: A
Our study also more specifically identified time taken to search the grey literature for a systematic review. All systematic reviews reported (n= 18) included some form of grey literature searching. The average time taken to conduct the grey literature search was approximately 7 hours, with range of 20 minutes to 58 hours, with 50% of the ...
Applying systematic review search methods to the grey literature: a
Applying systematic review search methods to the grey literature
4. Select Grey Literature Sources
Finding grey literature and searching it systematically is challenging. But there are a few approaches that you can take to add some structure to your search of this type of information: Refer to grey literature sources used for related evidence syntheses. Refer to both published evidence syntheses and registered protocols.
Searching for grey literature for systematic reviews: challenges and
The purpose of this paper is to provide a detailed account of one systematic review team's experience in searching for grey literature and including it throughout the review. We provide a brief overview of grey literature before describing our search and review approach. We also discuss the benefits and challenges of including grey literature ...
Grey literature in systematic reviews: a cross-sectional study of the
Systematic reviews (SRs) are an important source of information about healthcare interventions. A key component of a well-conducted SR is a comprehensive literature search. There is limited evidence on the contribution of non-English reports, unpublished studies, and dissertations and their impact on results of meta-analyses. Our sample included SRs from three Cochrane Review Groups: Acute ...
Literature Search: Databases and Gray Literature
Gray Literature. Gray Literature is the term for information that falls outside the mainstream of published journal and mongraph literature, not controlled by commercial publishers. includes: hard to find studies, reports, or dissertations. conference abstracts or papers. governmental or private sector research.
Locating Grey Literature
Searching the grey literature: A handbook for searching reports, working papers, and other unpublished research. Searching the Grey Literature is for librarians and information professionals interested in learning more about grey literature. This book will aid with crafting a grey lit search successfully, from start to finish.
Research Guides: Grey Literature: What is Grey Literature?
Grey literature encompasses various media, resources, documents, and data that diverge from the conventional academic or commercial publishing pathways, often termed "white literature." If a resource lacks publication in a scholarly journal, it likely falls within the realm of grey literature. Unlike white literature, grey Literature doesn't ...
Research Guides: Grey Literature: Evaluating Grey Literature
Evaluating Grey Literature. Grey literature presents a viable alternative to conventional scholarly and peer-reviewed literature, albeit necessitating heightened scrutiny in information assessment compared to the latter. The evaluation of grey literature entails a rigorous examination of various factors, including the document's intrinsic ...
Systematic Reviews: Grey Literature & Handsearching
Grey literature refers to both published and unpublished research material that is not available commercially. In general, grey literature publications are non-conventional and sometimes ephemeral publications that are not indexed in databases such as PubMed and Embase. Grey literature includes: Clinical trials; Dissertations and theses
Systematic Reviews: Searching the Gray Literature
The Grey Literature Reporter is a Google Chrome extension that you train on the search results of the current website that you are on. It will then record the results - as JSON, CSV and potentially RIS - as well as the URL of the site and the search string used. With this extension you can easily start your search, click on the extension ...
Research Guides: Systematic Reviews: Grey Literature
The Twelfth International Conference on Grey literature in Prague in 2010 arrived at the following definition: " Grey literature stands for manifold document types produced on all levels of government, academics, business and industry in print and electronic formats that are protected by intellectual property rights, of sufficient quality to be ...
Applying systematic review search methods to the grey literature: a
A grey literature search plan was developed to incorporate four different searching strategies: (1) grey literature databases, (2) customized Google search engines, (3) targeted websites, and (4) consultation with contact experts. These strategies were loosely adapted from those used in a grey literature review on drug prevention initiatives ...
A Guide to Evidence Synthesis: 3. Select Grey Literature Sources
Grey (or gray) literature is literature produced by individuals or organizations outside of commercial and/or academic publishers. This can include information such as government reports, conference proceedings, graduate dissertations, unpublished clinical trials, and much more. The sources you select will be informed by your research question ...
Searching and synthesising 'grey literature' and 'grey information' in
Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Syst Rev. 2015; 4:138. doi: 10.1186/s13643-015-0125-. [PMC free article] [Google Scholar] 14. Canadian Agency for Drugs and Technologies in Health. Grey Matters: A Practical Search Tool for Evidence ...
Significance of including grey literature search in systematic reviews
McAuley et al. [ 5] found that 33% of meta-analyses reviewed included grey literature, accounting for 4.5% to 75% of studies in meta-analysis. The Cochrane Handbook for Systematic Reviews of Interventions also mentions that a failure to identify and analyze studies from various grey literature search could impact the results of an SRMA.
Grey literature
The term grey literature acts as a collective noun to refer to a large number of publications types produced by organizations for various reasons. These include research and project reports, annual or activity reports, theses, conference proceedings, preprints, working papers, newsletters, technical reports, recommendations and technical standards, patents, technical notes, data and statistics ...
Grey Literature
An important distinction between grey literature and academic publications are that the latter are peer reviewed by scholars with expertise in the field. Searching for Grey Literature: You may find references to this kind of literature in bibliographies of articles and books (e.g., citation chasing).
Relapse prevention following guided self-help for common health
In addition to the literature search identifying peer-reviewed articles, this scoping review explored grey literature. Although there are many definitions of grey literature (Adams et al., 2016 ), the most widely agreed upon definition is literature that is "produced on all levels of government, academics, business and industry in electronic ...
Is artificial intelligence for medical professionals serving the
For the search of grey literature, scientific reports published in German or English 10 years retrospectively from the date of the search will be considered. To extract potentially relevant studies from (systematic) reviews and meta-analyses, secondary studies will be gathered and screened.
Gray literature: An important resource in systematic reviews
Gray literature also provides an important forum for disseminat- ing studies that might otherwise not be disseminated, such as those with null or negative results.18,19As a result, including gray litera- ture in meta-analyses may help minimize the effects of publication bias.11,15,18Publication bias refers to the propensity for only studies ...
Components and entities of post-disaster damage and loss assessment
An comprehensive search for relevant literature was performed using several databases, including the Web of Science, PubMed, Scopus, ProQuest, and Magiran. The search was limited to papers published between 2010 and 2022. In addition, we searched the grey literature for resources related to post-disaster damage and loss assessments.
Grey Literature Searching
Grey literature can also mean literature that is hard to find or has inconsistent or missing bibliographic information. Search grey literature to: avoid bias; ensure that the review is as thorough as possible; find sources for negative results or brand new evidence; discover more references to published literature that your database search ...
Emergency pediatric patients and use of the pediatric assessment
We followed the Joanna Briggs Institute methodology for scoping reviews and registered the review protocol. We searched MEDLINE, PubMed Central, the Cochrane Library, Epistemonikos, Scopus, CINAHL, Grey literature report, Lens.org, and the web pages of selected emergency pediatrics organizations in August 2022.
Unveiling the Exodus: A scoping review of attrition in allied health
Background Efficient utilisation of allied health workforce may help address the predicted shortfall of 18 million health workers estimated by 2030. Knowledge about allied health professionals' attrition, or intention to leave, and factors influencing attrition can assist in developing evidence-informed strategies to mitigate this issue. The review aimed to map attrition and attrition ...

Save citation to file

Add to My Bibliography

Gray literature: An important resource in systematic reviews

Similar articles

LinkOut - more resources

Email this link

Grey Literature

Why Search the Grey Literature?

Searching the Grey Literature

How to Manage the Grey Literature Search

Grey Literature Sources

Video: Grey literature (3:10 minutes)

Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews

Conclusions

Data extraction

Data analysis

Acknowledgements

Availability of data and materials

Authors’ contributions

Competing interests

Consent for publication

Ethics approval and consent to participate

Publisher’s Note

Author information

Corresponding author

Additional file

Rights and permissions

About this article

Share this article

BMC Medical Research Methodology

Literature Search: Databases and Gray Literature

Subject Specific Databases

Gray Literature

Systematic Review and Evidence Synthesis

Why search grey literature?

How do I search for grey literature?

Grey literature resources

Conference Proceedings

Dissertations and Theses

Grey Literature Databases

Preprint Repositories

Public Policy

Other Resources

Grey Literature

Evaluating Grey Literature

Systematic Reviews: Grey Literature & Handsearching

In this Guide

Handsearching

What is Grey Literature

Referencing and Citing Grey Literature

Read More about Grey Literature

Where to Find Grey Literature

Advanced Search

Critical Appraisal: AACODS Framework

Critical Appraisal: CRAP or CRAAP Test

Record Keeping

Managing and Reporting Results

Writing Up Search Methods

Avoid Predatory Publishers

Systematic Reviews

What is grey literature and why is it important?

A Guide to Evidence Synthesis: 3. Select Grey Literature Sources

Video: Grey literature (3:10 minutes)

The Grey Literature Background

Why Search the Grey Literature

How to Search the Grey Literature

How to Manage the Grey Literature Search

Grey Literature Sources Header

Significance of including grey literature search in systematic reviews and meta-analyses

Nitin K. Borkar

Financial support and sponsorship

Critical Writing Program: Multilingualism and Identity Fall 2024: Grey Literature

Grey Literature

Relapse prevention following guided self-help for common health problems: A Scoping Review

Cite this article

Conclusions

Similar content being viewed by others

User Experiences of CBT for Anxiety and Depression: A Qualitative Systematic Review and Meta-synthesis

The GET READY relapse prevention programme for anxiety and depression: a mixed-methods study protocol

The Effectiveness of Low-Intensity Psychological Interventions for Comorbid Depression and Anxiety in Patients with Long-Term Conditions: A Real-World Naturalistic Observational Study in IAPT Integrated Care